Dealing with AI Hallucination and Dishonest Outputs

Ryan Carson (ex-Treehouse, Intel; now Builder-in-Residence at Sourcegraph’s AMP) shares his origin story and a practical playbook for shipping software with AI agents. We cover why “tokens aren’t cheap,” how AMP made pro-level coding free via developer ads, a concrete workflow (PRD → atomic dev tasks → agent execution with self-tests), and why managers should spend time as ICs “managing AI.” We close with advice for raising AI-native kids and a perspective on this moment in tech (think integrated circuit–level shift).Timestamps

00:00 – The beginning of intelligence: how LLMs changed Ryan’s view of computing

00:23 – Apple IIe → Turbo Pascal → Computer Science: the maker bug bites

03:20 – DropSend: early SaaS, Dropbox name clash, first acquisition

04:30 – Treehouse: teaching coding without a CS degree; $20M raised, acquired in 2021

05:02 – The “bigger than a computer” moment: discovering LLMs

06:15 – Joining Intel: learning GPUs and the scale of silicon (“my adult internship”)

07:09 – Building an AI divorce assistant → joining AMP as Builder-in-Residence

09:38 – AMP vs ChatGPT/Claude/Cursor: agentic coding with contextual developer ads

11:09 – Token economics: why AI isn’t really cheap

17:27 – Frontier vs Flash models (Sonnet 4.5 vs Gemini 2.5) — how costs scale

21:31 – Private startup: vertical AI for specialized domains

22:36 – The new wave of small, vertical AI businesses

23:01 – Live demo: building a news app end-to-end with AMP

28:18 – How to plan like a pro: write the PRD before you build

30:02 – “Outsource the work, not your thinking.”

32:28 – Turning PRDs into atomic tasks (1.0, 1.1…)

35:50 – Competing in an AI world = planning well

36:28 – Managers should schedule IC time to “manage AI”

37:14 – Designing feedback loops so agents can test themselves

39:47 – “AI lied to me”: why verifiable tests matter

41:11 – Raising AI-native kids: build trust, context, and agency

43:59 – “We’re living in the integrated circuit moment of intelligence.”Tools & Technologies MentionedAMP (Sourcegraph) – Agentic coding tool/IDE copilot that plans, edits, and ships code. Now offers a high-end, ad-supported free tier; ads are contextual for developers and don’t influence code outputs.Sourcegraph (Code Search) – Parent company; enterprise code intelligence/search.ChatGPT / Claude – General-purpose LLM assistants commonly used alongside coding agents.Cursor / Windsurf – AI-first code editors that integrate LLMs for completion and refactors.Bolt / Lovable – Text-to-app builders for rapid prototyping from prompts.WhisperFlow / SuperWhisper – Voice-to-text tools for fast prompting and dictation.Anthropic Sonnet 4.5 – Frontier-grade reasoning/coding model; powerful but pricier per token.Google Gemini 2.5 Flash – Fast, lower-cost model; “good enough” for many workloads.Auth0 (example) – Authentication-as-a-service mentioned as a contextual ad use case.GPUs / TPUs – Compute for training/inference; token cost drivers behind AI pricing.PRD + Atomic Tasks Workflow – Ryan’s method: record spec → generate PRD → expand to dot-notated tasks → let the agent implement.Self-testing Scripts – Ask agents to generate runnable tests/health checks and loop until passing to reduce back-and-forth and prevent “it passed” hallucinations.Family ChatGPT Accounts – Tip for raising AI-native kids; teach sourcing, context, and trust calibration.Subscribe at⁠ thisnewway.com⁠ to get the step-by-step playbooks, tools, and workflows.