LLMs and Mental Health Risks

Discussion of an ethnographic study showing ethical violations in AI-led therapy and the risks of AI counseling in schools.

Play episode from 21:43

chevron_right

Transcript

chevron_right

Transcript

Episode notes

In this episode of ChatEDU (From Sycophants to Socrates), Matt and Liz open with a laugh reacting to a study showing LLMs may suffer “brain rot” when trained on low-quality data. The Rundown starts with a listener shout-out and then covers eight stories on how educators, researchers, and policymakers are navigating the AI moment.

Story #1: Prompt Reflections from Nano in Melbourne

Nano, a loyal listener in Australia, asked about the “clerk, colleague, or coach” prompt framework. Matt and Liz revisit two Skills21 frameworks that help educators and students reflect on AI use and extend it thoughtfully.

Story #2: MA Launches a Statewide AI Module

The MA Depart. of Elementary and Secondary Education released a free AI module for educators by Dr. Rebecca Mazur at CES. Grounded in five principles and avoiding tech hype, it includes videos, PD tools, and a certificate.

Story #3: 5 Myths About Oral Assessments

Interactive Oral Assessments (IOAs) are gaining traction as essay alternatives. A University of Sydney study debunks five myths, showing how IOAs simulate real-world dialogue, save time, and expand accessibility.

Story #4: NotebookLM Gets Smarter, Gemini Grows

NotebookLM now supports custom chat goals and conversation saving. Google launched a Wellness Gem, a “vibe coding” tool, and previewed auto-generated infographics. Gemini has doubled its market share, signaling AI tool consolidation.

Story #5: LLMs Fail at Therapy

An ethnographic study of 137 AI-led mental health sessions found 15 ethical violations. Matt and Liz explain why AI counseling isn’t ready for prime time and what schools should watch for.

Story #6: Character.AI Faces Pressure After Suicides

After suicides linked to chatbot dependency, Character.AI and OpenAI are adding safety measures. Senators propose banning chatbots for minors and requiring age checks.

Story #7: Parents Get a Guide to AI Advocacy

A new guide from Common Sense Media, EdSafe, and the National Parents Union helps families ask key questions about AI in schools. Built on the SAFE framework, it includes sample questions and actions.

Story #8: Living Without AI is Harder Than It Sounds

Author A.J. Jacobs tried living without AI for 48 hours. Matt and Liz suggest this as a student project to uncover hidden algorithmic systems.

Beneath the Surface:

The Sycophant Problem: A study using MMLU shows some AI models change correct answers to wrong ones if the user suggests them. Smaller models are most compliant, dropping accuracy up to 15 percent and raising equity concerns.

The Socratic Alternative: Khan Academy’s “Explain Your Thinking” feature shows student understanding deepens when learners explain reasoning. Even with wrong answers, AI can detect sound logic.

Bright Byte: Boston Uses AI to Untangle Traffic

Mayor Michelle Wu expanded Boston’s Project Greenlight with Google. Using AI to optimize light timing, it cuts delays by 13% and stops by one-third.

Links and References

Prompt Review + Beyond the Prompt Tools

skills21.org/prompts

skills21.org/btp

Massachusetts DESE AI Module

https://tinyurl.com/yuyrw4bv

Five Myths About Oral Assessments

https://tinyurl.com/2s42a6er

NotebookLM

notebooklm.google