LLMs and Mundane Utility Limits

Zvi reviews examples where LLMs fail at legal judgments and safety-bound products like AI toys showing guardrail decay.

Play episode from 07:19

chevron_right

Transcript

chevron_right

Transcript

Episode notes

Last week had the release of GPT-5.1, which I covered on Tuesday.

This week included Gemini 3, Nana Banana Pro, Grok 4.1, GPT 5.1 Pro, GPT 5.1-Codex-Max, Anthropic making a deal with Microsoft and Nvidia, Anthropic disrupting a sophisticated cyberattack operation and what looks like an all-out attack by the White House to force through a full moratorium on and preemption of any state AI laws without any substantive Federal framework proposal.

Among other things, such as a very strong general analysis of the relative position of Chinese open models. And this is the week I chose to travel to Inkhaven. Whoops. Truly I am now the Matt Levine of AI, my vacations force model releases.

Larry Summers resigned from the OpenAI board over Epstein, sure, why not.

So here's how I’m planning to handle this, unless something huge happens.

Today's post will include Grok 4.1 and all of the political news, and will not be split into two as it normally would be. Long post is long, can’t be helped.
Friday will be the Gemini 3 Model Card and Safety Framework.
Monday will be Gemini 3 Capabilities.
Tuesday will [...]

---

Outline:

(01:50) Language Models Offer Mundane Utility

(02:43) Tool, Mind and Weapon

(06:55) Choose Your Fighter

(07:18) Language Models Don't Offer Mundane Utility

(11:31) First Things First

(12:12) Grok 4.1

(15:03) Misaligned?

(18:37) Codex Of Ultimate Coding

(20:21) Huh, Upgrades

(20:49) On Your Marks