Generative AI in the Real World

Justin Norman, author of Product Management for AI and co-founder of Vera, a startup focused on security for generative AI, talks with Ben Lorica about how product management has changed since Generative AI came on the scene. He discusses the issues retrieval-augmented generation (RAG) raises for product management; how reliability has become part of a product’s value; how companies that have lagged in their adoption of AI can use generative AI as a way to catch up; and the ability of open source AI in helping smaller companies compete with more established companies.Points of Interest0:00: You wrote Product Management for AI back in 2020 and 2021. How have things changed for product managers since then?3:04: Do companies that lead with operations and infrastructure for traditional AI maintain an advantage with Generative AI? Or does Generative AI allow companies that are just starting to catch up?5:09: Can new companies use open source to compete with established companies? Can open source help capture value as well as larger proprietary models?6:08: What do product managers struggle with when implementing RAG? What's the relationship between fine-tuning and RAG?10:58: RAG gives you value out of the box, but the key to success is how the data is organized.13:57: Are VCs underinvesting in certain parts of the pipeline? There is lots of investment in AI, but not as much investment in startups working on necessary technologies like ETL and data engineering.16:31: Why is reliability important for generative AI? How is generative AI different from other applications that we’re familiar with, and what implications does this have for product management?21:03: Are enterprises realizing that efficiency is important for succeeding with generative AI?23:44: We’re familiar with dashboards for monitoring and managing traditional software products. What would you imagine a dashboard for generative AI models to be? What do you need to be monitoring?28:49: Very few developers working in machine learning have also done frontend development or worked on user experience (UX). However, understanding user interaction can help you to improve your model.30:44: You're working with the father of digital forensics, Hany Farid. Should we be worried about DeepFakes?

Pete Warden, founder of Useful Sensors and co-author of TinyML, discusses use cases for artificial intelligence that we rarely think about: how can you run AI on very small systems? How can you put AI on consumer devices in ways that are actually useful and not just buzzword-compliant? AI doesn’t have to rely on massive GPU farms. Pete talks about what happens when you exchange one set of requirements (extreme power, heat, and expense) for another (minimal size, cost, and heat).Points of Interest00:00: Introductions, including Pete’s introduction to his company.2:22: What are some of the challenges and use cases for sensor-driven AI?4:11: Is sensor-driven AI relevant to industries other than hardware?6:22: Now we’re in the age of foundation models and large language models. Is “large” incompatible with “tiny”? Can you run language models on smaller devices?8:00: Will there be developments in tinyML that will benefit the broader LLM community?9:30: What’s deployable today in computer vision, speech, and language? What can be done with hardware that’s constrained by cost, size, and power consumption?11:15: How will product designers work with sensor-driven AI? Will they simply select from a palette of optional modules?12:37: Pete walks us through the development of AI-in-a-Box, from its conception to its reception.15:31: Your devices don’t have network connections. Without a network connection, how do you update models? Is it necessary?19:00: Do you do Retrieval Augmented Generation (RAG) on your devices?20:35: Our devices have user interfaces that combine voice and presence. A voice interface is central, but visual (and other channels) help to create an awareness of the speaker.21:35: What are some of your specific challenges, like power consumption and latency? How do you make tradeoffs?22:45: What is the future of large language models for sensor-driven AI?26:50: What are some of the security concerns for sensor-driven AI and what are you doing about them?28:22: What is Dark Compute and why is it important?30:48: What are the biggest opportunities for pushing AI into consumer devices? We need to start with problems that users actually care about.32:30: How can listeners connect to the broader movement around TinyML?

O’Reilly’s Generative AI in the Enterprise survey reported that people have trouble coming up with appropriate enterprise use cases for AI. Why is it hard to come up with appropriate use cases?Chip Huyen, cofounder of Claypot AI and author of Designing Machine Learning Systems, talks about why many companies have trouble coming up with appropriate use cases for AI, how to evaluate possible use cases, and the skills your company will need to put them into practice.Points of Interest0:00: Introduction0:49: O’Reilly’s Generative AI in the Enterprise survey report results.3:02: Now that generative AI is more accessible, will it be easier to come up with use cases?4:29: AI is easy to demo but hard to productize. Consistence, risk, and compliance.6:44: Is there a framework or checklist for thinking about applications?8:15: What are some of your favorite use cases?13:30: RAG is the “hello, world” of AI applications.17:24: How do you navigate between the desires and requirements of different stakeholders?19:00: When talking to stakeholders, you have to answer questions at the right level.21:10: How to think about staffing teams for generative AI.22:45: There’s less model development with generative AI, more application development.23:12: Frontend engineers and full-stack developers are very successful.26:27: What are companies’ concerns about risk?27:27: Understanding data gives a lot of clues about what it is good at and should be used for.29:00: The importance of documentation.30:25: Are there specific things you can do to ease the integration of AI into an organization?32:49: What companies that have deployed AI into products stand out?