The Risks & Research of Over Reliance on AI

After a frustrating week of trying to wrangle AI outputs, we decided to explore the risks of overreliance on AI. It’s good for us to question our tools. It enhances our processes and challenges us to find the right tools.

Listen on Spotify | Listen on Apple Podcasts | Watch on YouTube

In this episode, we say the quiet parts out loud. Not only are LLMs often feeding us incorrect information, but over-trusting these systems poses a serious risk.

We can look at this Rolling Stone article headline and immediately laugh it off. It is insane to believe this will happen to anyone we know. However, in Mark Zuckerberg’s vision of the AI future, your friends will be bots.

The loneliness epidemic is real. One in three Americans feels lonely every week. Data from Harvard’s Making Caring Common Project supports that loneliness is tied to increasing feelings of anxiety, not part of this country, and being about more than social isolation. 65% of respondents blame “our society,” pointing to a lack of confidence in our way of life and institutions.

So, it should be no surprise that Harvard Business Review found that the top three use cases of 2025 involved loneliness and navigating life's stresses.

AI could quickly become the next addiction for a world desperate for solutions. The fact that there’s demand for robo-companionship shouldn’t be treated as validation for building more tools to disassociate them from life.

Let’s go back to exploring this topic from the perspective of business users.

Understanding GenAI’s Productivity Gains

As we barrel into the AI-powered era, we can take one of two perspectives:

* GenAI products are the next evolution of SaaS: Precise tools for specific workflows

* LLMs are the next evolution of social media, where instead of degrading our interpersonal relationships, AI will addict us to easy and often incorrect information

The majority of the research identified productivity gains and time savings, which would support the goal of GenAI as a professional advantage.

But when you dig into the data, there are concerns.

Many are funded by Microsoft, OpenAI, and Google, like this one showing that GitHub Copilot users completed tasks 55.8% faster than the control group. While that result was impressive, they were being assessed on their ability to complete a very basic task. The paper’s results were boosted by showing that people with less experience benefit more from coding assistants, something that should worry anyone concerned about being replaced by cheaper talent who are boosted by AI.

But these results were refuted in a separate study where no DevOps productivity gains were found from using the same GitHub Copilot. That study found the code quality to be poor, leading to a 41% increase in bugs!

Remember that GitHub is owned by Microsoft and powered by OpenAI’s foundation model, ChatGPT.

This contradictory data highlights a paradox of GenAI: The technology is increasingly more successful at a basic task-level, but shouldn’t be over-relied on to do our work for us. This Danish study hammers that point home: Time saved by AI offset by new work created.

If we shake ourselves out of the AI hype stupor, we can critically examine the current state of LLMs more like SaaS tools. 95% of SaaS tools available won’t help you and your business. Once you find the right tool for your hyper-specific use case, the AI product’s success will depend on its implementation and the first-party data entered into it.

Thanks for reading Design of AI: Strategies for Product Teams & Agencies!

More Research about Using AI

Behavioural research about AI should be considered a counterpoint to the benefits of AI. Yes, leveraging GenAI will have productivity gains in specific circumstances. But the technology also brings with it risks and considerations that built into the design and business case of “should we build this” discussions.

* Increased AI use linked to eroding critical thinking skills

* When experiencing time pressures, we’re more susceptible to misinformation

* AI systems are already capable of deceiving humans

* People Facing Life-or-Death Choice Put Too Much Trust in AI

* AI’s Trust Problem: Twelve persistent risks of AI that are driving skepticism

Catch up on Recent Design of AI Episodes

31. AI is Disrupting Architecture and Lessons for Digital Product TeamsGuest Matthew Krissel (FAIA) explores how AI is reshaping architectural design and what digital product teams can learn about process, creativity, and scale from the built environment.

30. Take Control of AI’s Predictive Power – Tyler Hochman, ForethoughtTyler Hochman shares how businesses can operationalize AI for forecasting and insights by targeting high-value, repeatable problems and unlocking underutilized data.

29. Trust is a Double‑edged Sword: AI will Transform Services – Sarah Gold, Projects by IFSarah Gold explains how AI changes our relationship with services and why it’s urgent to rethink trust, transparency, and accountability in product design.

28. AI will Transform Product Research – John Whalen, Author & PsychologistJohn Whalen outlines how AI can automate user research tasks and speed up insights while cautioning where human interpretation is still irreplaceable.

27. Implementing AI in Creative Teams: Why Adoption Will Surge – Jan Emmanuele, WondercraftJan Emmanuele shares real-world strategies for rolling out AI in creative teams, tracking ROI, and overcoming fear and friction in adoption.

26. Designing a New Relationship with AI – Critical Mindset Shifts for Success – Sara ViennaSara Vienna emphasizes why designers must rethink their role and mindset in an AI-powered world—moving from tool experts to value creators.

25. Faster, Cheaper, Better: AI’s Transformation of Insights & Strategy – David BoyleDavid Boyle discusses how AI is reducing the cost and time of gathering insights while also introducing new pitfalls that only expert interpretation can avoid.

24. Adding AI to Miro – Case Study in Improving the UX of Existing Products – Ioana TeleanuIoana Teleanu discusses the strategic thinking behind introducing AI to Miro’s design tools without compromising the user experience.

23. AI in Healthcare: Challenges of Implementing in Complex Industries – Dr. Spencer DornDr. Dorn explores why healthcare has been both cautious and optimistic with AI, offering lessons for anyone deploying AI in regulated or high-stakes fields.