Controlling inference costs via context engineering

Register here to join Founder University Japan’s kickoff! 👉🏻 https://luma.com/cm0x90mkToday’s show: On TWiST, Alex is chatting with Neubird CEO Gou Rao about the intricate complexity of IT Ops, and the vast amount of layers and components that have to be delivered very quickly to your phone in order to stream a simple video clip.Neubird has developed Hawkeye, an AI “engineer” that uses “context engineering” to isolate and troubleshoot problems, while still keeping costs down. Loading up a video to show your friends may SOUND straightforward, but it’s actually, as Alex notes, a miracle anything on the internet works at all.PLUS we’ve got Subject AI CEO Michael Vilardo stopping by to tell us how the app uses gamification to keep kids learning and engaged. NOTE TO FOUNDERS: Mike got invited on the show because he sends email updates to his “dream investors,” including Jason. It’s a solid best practice!Finally, we’re flashing back to a remarkably prescient 2019 interview with then-Scale AI chief Alexandr Wang, who tells us why he thinks the AI Doomsday scenarios — AND the elaborate AGI promises — are both way overblown.Timestamps:(3:19) Neubird asks: What if we took AI tools and applied them to IT ops?(5:17) Why does IT ops seem to get so much more complicated over time?(6:14) Gou walks us through all the various components that need to be delivered when you post a simple TikTok video(8:41) Why agentic systems (like Hawkeye) are the ideal tool for monitoring metrics, logs, traces and alerts(10:05) CLA - Get started with CLA's CPAs, consultants, and wealth advisors now at https://claconnect.com/tech(11:17) How is IT Ops different from Site Reliability Engineering (SRE)?(12:36) Is Hawkeye an AI Agent or an “Engineer”? How we decide what to call our “co-pilot.”(16:07) How “context engineering” helps Hawkeye isolate problems, and keeps inference costs down(17:27) Is Hawkeye plug and play? Or does it need to be trained on a specific system?(18:59) How Neubird measures Hawkeye’s overall effectiveness(19:59) Perspective AI: Real insights, straight from your customers, and your first two months are on us. Just go to http://getperspective.ai/twist(21:47) Hawkeye can diagnose and make recommendations, or go in and fix the problem itself, if you like!(24:27) What Gou thinks about the pace of AI innovation: Has it slowed? Are we in a lull?(25:41) LLMs already have a lot of knowledge, but how do we enhance their EQ?(28:58) TIP FOR FOUNDERS: Send updates to your “dream” investors! That’s how Mike got on the show!(30:20) Uber AI Solutions - Your trusted partner to get AI to work in the real world. Book a demo with them TODAY at https://Uber.com/twist⁠(31:21) The importance of branding… Why Mike pivoted to Subject.ai(33:43) How Subject helps bring teachers and students together for more 1-on-1 interaction(35:48) Mike walks us through how Subject works, and how gamification keeps kids engaged(39:56) Is anything lost when an AI grades a student’s test, rather than a human?(42:09) Subject can serve as a “teacher of record” in classrooms(46:28) Why Mike and Subject AI are back in the office(47:40) TWIST FLASHBACK with Alexandr Wang of Scale AI from November 2019(53:10) Alexandr explains AI to Jason (back when it was still called “Machine Learning”)(1:01:08) How AI will help people focus on “higher-value work”(1:04:48) Why Alexandr thinks apocalyptic AI scenarios are unrealistic(1:13:28) Are the big promises around AGI overblown?*Thank you to our partners:(10:05) CLA - Get started with CLA's CPAs, consultants, and wealth advisors now at https://claconnect.com/tech(19:59) Perspective AI: Real insights, straight from your customers, and your first two months are on us. Just go to http://getperspective.ai/twist(30:20) Uber AI Solutions - Your trusted partner to get AI to work in the real world. Book a demo with them TODAY at https://Uber.com/twist