This blogpost has been updated since original release to add more links and references.The ChatGPT Plugins announcement today could be viewed as the launch of ChatGPT’s “App Store”, a moment as significant as when Apple opened its App Store for the iPhone in 2008 or when Facebook let developers loose on its Open Graph in 2010. With a dozen lines of simple JSON and a mostly-english prompt to help ChatGPT understand what the plugin does, developers will be able to add extensions to ChatGPT to get information and trigger actions in the real world. OpenAI itself launched with some killer first party plugins for: * Browsing the web, * writing AND executing Python code (in an effortlessly multimodal way), * retrieving embedded documents from external datastores,* as well as 11 launch partner plugins from Expedia to Milo to Zapier.My recap thread was well received:But the thing that broke my brain was that ChatGPT’s Python Interpreter plugin can run nontrivial code - users can upload video files and ask ChatGPT to edit it, meaning it now has gone beyond mere chat to offer a substantial compute platform with storage, memory and file upload/download. I immediately started my first AI Twitter Space to process this historical moment with Alessio and friends of the pod live. OpenAI’s Logan (see Episode 1 from *last month*…) suggested that you might be able to link ChatGPT up with Zapier triggers to do arbitrary tasks! and then Flo Crivello, who just launched his AI Assistant startup Lindy, joined us to discuss the builder perspective.Tune in on this EMERGENCY EPISODE of Latent Space to hear developers ask and debate all the issues spilling out from the ChatGPT Plugins launch - and let us know in the comments if you want more/have further questions!SPECIAL NOTE: I was caught up in the hype and was far more negative on Replit than I initially intended as I tried to figure out this new ChatGPT programming paradigm. I regret this. Replit is extremely innovative and well positioned to help you develop and host ChatGPT plugins, and of course Amjad is already on top of it:Mea culpa.Timestamps* [00:00:38] First Reactions to ChatGPT Plugins* [00:07:53] Q&A: Keeping up with AI* [00:10:39] Q&A: ChatGPT Intepreter changes Programming* [00:12:27] Q&A: ChatGPT for Education* [00:15:21] Q&A: GPT4 Sketch to Website Demo* [00:16:32] Q&A: AI Competition and Human Jobs* [00:18:44] ChatGPT Plugins as App Store* [00:34:40] Google vs ChatGPT* [00:36:04] Nader Dabit on Selling His GPT App* [00:43:16] Q&A: ChatGPT Waitlist and Voice* [00:45:26] LangChain with Human in the Loop* [00:46:58] Google vs Microsoft vs Apple* [00:51:43] ChatGPT Plugin Ideas* [00:53:49] Not an app store?* [00:55:24] LangChain and the Future of AI* [01:00:48] Q&A: ChatGPT Bots and Cronjobs* [01:04:43] Logan Joins Us!* [01:07:14] Q&A: Plugins Rollout* [01:08:26] Q&A: Plugins Discovery* [01:10:00] Q&A: OpenAI vs BingChat* [01:11:03] Q&A: App Store Monetization* [01:14:45] Q&A: ChatGPT Plugins API* [01:17:17] Q&A: Python Interpreter* [01:19:58] The History of App Stores and Marketplaces* [01:22:40] LindyAI's Flo Crivello Joins Us* [01:29:42] AI Safety* [01:31:07] Multimodal GPT4* [01:32:10] Designing AI-safe APIs* [01:34:39] Flo's Closing CommentsTranscript[00:00:00] Hello and welcome to the Latent Space Emergency episode. This is our first ever where chatty PT just dropped a plugin ecosystem today, or at least they demoed their plugins. It's still on the wait list, but it is the app store moment for ai. And we did an emergency two hour space with Logan from OpenAI and Flo Coveo from Lin AI and a bunch of our friends.[00:00:28] And if you ever wanted to listen to what it's like to hear developers process in real time when a new launch happens, this is it. Enjoy,[00:00:38] First Reactions to ChatGPT Plugins[00:00:38] I assume everyone has read the blog post. For me the, the big s**t was do you see Greg Brockman's tweet about FFMPEG? I did not. I should check it out. It is amazing. Okay, so. So ChatGPT can generate Python code. We knew this, this is not new, and they can now run the code that it generates.[00:00:58] This is not new. I mean this is like, this is good. It's not like surprising. It's, it's fine. It can run FFMPEG code. You can upload a file, ask it to edit the video file, and it can process the video file and then it can give you the link to download the video file. So it's a general purpose compute platform.[00:01:22] Wow. Did they show how to do this? Agents? I just, I just, I just pinned it. I just, it did I, did I turn into this space? I dunno how to use it. Yeah, it's, it's showing up there. Okay. It can run like is. Is, is, is my And by, by the way hi to people. I, I don't know how to run spaces. I, I not something I normally do.[00:01:42] But You wanna say something? Please request. But yeah, reactions have a look at this video because it run, it generates and runs video editing code. You can upload any arbitrary file. It seems to have good enough compute and memory and file storage. This is not chat anymore, man. I don't know what the hell this is.[00:02:01] What, what is this?[00:02:02] Well, progress has been all faster than I expected. . That's all I can, I, I, I don't know how to respond. . Yeah. It's pretty wild. I wonder, I wonder, I'm wondering how, how this will affect, like opening up the app store different from, let's say Apple App Store when it opened up. Because there are a lot of, of big companies just building stuff already and how like a small developer will be able to, to build something that's not already there.[00:02:31] I dunno. It will be interesting. So one thing that's really nice, have you seen the installation process for the plugins? It's right at the bottom of the blog post and you have to play the video to kind of see it, but literally anybody can write your own plugin. It's a small little json file. It's, it's literally like 10 lines of code.[00:02:49] It's 10 nights of, you described what your plugin does in English, you given an open API spec. That's it. That, that's, that's the plugin. It's amazing. You can distribute your plugin. This is, this is, this is easier than extensions manifest v3, which nobody knows how to use. This is English.[00:03:15] You write English . So, so, yeah. I mean I think, I think I think there'll be a lot of people trying to develop for this if they can get access, which you know, everybody's on a wait list. I, I've, I've signed up to 200 wait lists this week. . I wonder if, if it'll be different if you, if you sign up as a, as a developer or as the chat user.[00:03:35] Hopefully it doesn't matter, right? Use different emails and sign up to both. Let's, let's just see, in fact, use t to generate like, plausible sounding reasons for why you want to build whatever. Cause they don.[00:03:47] But yeah, I mean, how do you compete? I, I don't know, man. You know, it, it's really OpenAI is definitely a partnership strategy to do what they do here which means they're essentially picking favorites. So if you're a competitor of Expedia Kayak Open Table Wolf from Zapier, you're a s**t out of luck, kind of, you know?[00:04:06] Cause these are presumptive winners of their spaces. Right. And it'll happen in too many industries, probably. Right. I was thinking about maybe summarization or, or I don't know, YouTube video summarization, but there seems to be some application of that already on the examples that you shared. Yeah, yeah, yeah.[00:04:26] They have shared that, but I think there's always room to improve the experience. It's just, you know It's interesting which platform, like sort of platform strategy, right? Like if you write an OpenAI chat plugin, you instantly gain access to a hundred million users, right? All of them can instantly use your thing.[00:04:47] Whereas if you are a standalone app or company, good luck trying to able to use OpenAI through you. There's just no point. So you much rather just be on OpenAI platform and promote there. The the fortunate thing is they don't have some kind of like popularity ranking yet. Actually, someone should go open, someone should do register, like OpenAI plugins list.com or something where like everyone can like submit their own opening app plugins and like upload them, review them cuz this like, this is not a complete app store without reviews and a rating system and a reputation system and probably monetization opening app probably doesn't care about that.[00:05:26] But I mean, I can go start that right now. F**k. I can go start it right now.[00:05:34] Yeah, it'll, it'll take a while, right? Like this is the, like the basic version of the, of the app evolving. But this is a pretty basic version. Yeah. The basic version can browse the web, it can write, write an execute code. It can retrieve you know, we can retrieve data from documents, right? So all the documents search just died.[00:06:02] There's like five of these in Y Combinator right now. Oh.[00:06:08] Examples. Pretty crazy how, how they use the FFMPEG library or, I dunno if I'm saying that correctly, but right in there. You don't need to, to write code to,[00:06:27] it's crazy. Dunno. Yeah. Any reactions? Please, please, you know, open space. Anyone can request a speaker. Oh, Ash, come on in. Ash. I have to add you a speaker. Yeah, we're, we're just reacting here. I just, I, I needed a place to talk and I'm in Japan and I don't have anyone else to talk to, so I need, I, I I just want to share this moment.[00:06:46] I think it's a special moment in history. This is the biggest new app source since ever. Yeah. Hey, Shawn. I think plugin is already taken. . Oh man. Someone, someone bought it already. Yep. , of course. Right? Of course. , what are your reactions? What how are you feeling? What's what are you seeing out there?[00:07:07] Just crowdsource all the tweeting. Yeah, man, it's, it's been wild. I mean, I get out of there to like five minutes and then anything drops, you know, , I think productivity today will be like zero. If I, if I still, like, I quit my job you know, a few weeks ago but I would not be working today. There, there's no point.[00:07:26] There's nothing else. There's nothing else that's important, like, nothing's going on. Like this is the only story. Yep. . I wonder if you have any, any frameworks or anyone that's listening any frameworks on, on how you're handling all of this new, new stuff. Like every single day if something new comes up and, or you can like get the, the wait list invitations to, to use the new products.[00:07:52] Q&A: Keeping up with AI[00:07:52] Like, for example, today I just got the, the one from GIK cli and I was just playing around with that. And then suddenly I started to see all of the, these Twitter threads with announcements. It's getting crazy just to follow up with, with the stuff. And every day something new comes up and started. I was starting to feel a lot of formal, you know, like, h how do you keep up with all of these?[00:08:12] Or how do you focus? Does anyone have any, any good frameworks for that? Well, feel free to respond. Also, we, we have some more room if anyone wants to share your feelings. This is a, this is a safe space to share your feelings because. We all dunno how to react right now. I don't know. I just, I, I, I have a few notifications on for OpenAI employees and people that I do that I think do good recaps.[00:08:37] So in other words, find the people who are high signal and who do a lot of gathering of other people's stuff for, and then just subscribe to those people and trust that that is 90% of it and forget the 10%[00:08:57] Alright. And Sean probably, I have, I have another question. So I can't really figure out like what's left for us to do, you know, without AI tools. Like what, what is we learn next? You know, there's no learning some coding stuff, because you can only do that. You know, we can't do arts, we can't do poetry.[00:09:17] Farming[00:09:17] bakery, probably making things with your hands. Enjoying the sun.[00:09:23] Do you guys think this should be regulated? Like you don't go more than like the speed is going? I don't know. I dunno. There's, there's no point. Like if, like, if you regulate OpenAI, then someone else will come along. The secret is out now that you can't do this, and at most you'll slow things down by 10 years.[00:09:44] You called the secret. This is the end. . Yeah. Yeah. I, I don't know. Secret is out. China's trying to do it right, so I don't know if people have seen, but like China was, was fairly strict on crypto, which is probably good for them. And now they're, they're also trying to clamp down on AI stuff, which is funny because oa like they're, you know, the m i t of of China Ihu, I was actually doing like producing like really good bilingual models.[00:10:10] But yeah, they, they seem to be locking this down, so we'll see. We'll see. Right? Like you know, in, in, in sort of the, the free world there, there's open innovation that may be unsafe. OpenAI, try to be safe. You know, there, there's a big part of the blog post that was talk, talking about red team meeting and all that.[00:10:24] I'm sure every one of us skipped it. I skipped it. And then and then we just care about capabilities and now that, you know, every time people have their minds opened, like, I did not know Ron. EG in chat.[00:10:38] Q&A: ChatGPT Intepreter changes Programming[00:10:38] Now that I know my conception of what a REPL is, or literate programming or what a notebook is, is completely blown outta the water, right?[00:10:44] Like there's no like this, this is a new form factor for me. So not now that I know that I won't be innovating on that or trying to, to shape this into something that I can use because I want to use this, and this is, this is clearly better. Does, does this ha have to do with, with the, like AI as backend?[00:11:00] Yeah. Ideas that have been, yeah. You know, GP as backend. So, so apparently I had a few friends reach out to those guys and they're not doing that because it's not mature enough. Like it works for a simple demo. So, so for, for those who don't know ScaleAI did a hackathon I think two months ago just before I did mine.[00:11:18] And the winner on the hackathon was, was something called GPT is all you need for backend. And they actually what in register? DBC is backend.com. But as far as I can tell, they're not gonna start a company based on that because if you even push a little bit, it falls apart, right? So GPT3 wasn't good enough for that.[00:11:36] Maybe GPT4 is maybe GPT5, but then it'll still be super slow and super expensive. Like you don't want to run, you know, a large language model on every API request. So I don't know. I think it'll be good for scaffolding. I think it'll be good for re type use cases. Like, Hey, I need to edit this video on an ad hoc basis.[00:11:53] I don't, I don't want to learn FFMPEG. I don't need to now, because I can just talk to ChatGPT. That makes sense. But if you want a reliable, scalable backend you probably don't want to use it on a large language model, but that's okay because language model can probably help you write it rather than run it.[00:12:13] Hey, Lessio. Hey guys. Oh yeah. Hey guys. What's up? Hey, yeah, we're, we're just, there's no structure. Just drop your reactions. Let's go. Awesome. Awesome, awesome guys.[00:12:26] Q&A: ChatGPT for Education[00:12:26] What do you think what if Shawn, what do you think if you could use you know AI and the education field, like, you know, like personal attribution system for students?[00:12:35] What's the thought automation education or attribution edu edu education. Yeah. That is the holy grail. This is called the Blooms two Sigma problem. Like the, the, the, one of the big issues of education is we have to teach to the slowest person in the class. And, and, you know, I'm a beneficiary of, of a gifted education system where they take out you know, nominally high IQ people and put them in a separate class.[00:12:56] And, and yeah, we did, we did do better. What if we can personalize every student's experience there's, there's some educational theory. This is called Bloom's two Sigma problem. Where the results will be better. I think that we are closer, but like, I still hope that we're pretty far , which sounds like a negative, like why do I want to deny education to students?[00:13:18] Because if we are there, then we will have achieved theory of mind for ai. The AI has a very good model, is able to develop a representation of who you are, is able to develop theories that the test who you are in, in a short amount of time. And I, it's a very dangerous path to, to go down. So I want, I want us to go slowly rather than fast on, on the education front.[00:13:41] Does that make sense? Yeah, definitely. It makes a lot sense and yeah, definitely. I think personally the education for each student and making it turn the best way would be great. And what do you think how about like, first of all, I'm, I'm having very curious, curious question, you know, like we are having, this week was full of launches, so how you guys are keeping up with if we're not, this is, I created the space though cuz I cannot handle it.[00:14:05] Today, today was my breaking point. I was like I don't know what's happening anymore. Yeah, like every single day I'm just in constant anxiety that like everything I assumed about the world is gonna be thrown up. Like I don't know how to handle it. This is a therapy session, so feel free to express.[00:14:21] Definitely. It's, it's been a very overwhelming feeling for everyone of us like that. I think, you know, like past two weeks and like the industry was definitely a lot, lot of ones we are definitely open for, you know, to discuss more about it. Thanks a lot for this space. Sean. Yeah. Appreciate. Yeah. Va one more thing.[00:14:39] So I think that the most constrained version of education use cases is language teaching. So there are a few language teachers out there speak I think is one of them that is an OpenAI partner. And they're also part of the chat GPT plugin release. , but there are also other language tutor platforms.[00:14:57] You can certainly have your news. There was one that was released maybe like four or five months ago that you can try to see what the experience is like. And you can, you can tell when the teacher has no idea who you are and it breaks the illusion that you're speaking to another human. So I, I just, you can experience that today and, and decipher yourself if we're ready for that.[00:15:14] I hope that we're not ready and it seems like we're not ready. Yeah, definitely, definitely. Thanks a lot for sharing. And guys, what do you think?[00:15:19] Q&A: GPT4 Sketch to Website Demo[00:15:19] Like I, in the launch of four we have show that we could, you know, generate apps and web apps just from you know, like a single simple sketch, you know different tent.[00:15:30] Just start from sketch. So what do you think like how, how it would be impacting the industry? It's all because it's not just like that, that sketch was very, was a very shitty sketch. Right. It was just like drawn on a piece of paper. But if you combine that with the multimodal, like it was that they had another part of that demo where they had a screenshot of the discord the opening eye discord and you're mm-hmm.[00:15:57] and they put it in and it, it like read the entire screen to you and if you can read the entire screen, you can code the entire . Screen. So it's over like[00:16:12] It's definitely, I think interaction, interaction designers, you know, like people who like, think design function still have some time. Yeah. I, I just, I just, I just tried the same thing, you know on bar today and it was like much more better than GPT3 so definitely it's you know, things are really changing.[00:16:30] Q&A: AI Competition and Human Jobs[00:16:30] Great forward. I'm, I'm really worried what we wanna do, you know? Do you think the competition will like stable everything? Like what competition? Anthropic. Well, like Google, Google won't race, I don't think. Google Race, like Google the fight. The one that, the one that launched the W links list of blog posts.[00:16:50] That, that Google.[00:16:55] Well, no, not, not the list. Not the list. Competitions will come. . I have a question. I mean I mean my fear is many of the jobs that are going away, whether it is developer and designers, because I mean, I think GPT four is very capable. So how to deal with it. I mean, it's going to replace, I mean, many of the jobs, that's for sure.[00:17:16] Yeah. It's okay. We'll find new jobs or we'll, we'll not need jobs anymore. We should, we should also, Start universal basic income. That's, that, that is something I, I do believe, yeah, I think the, the main change is going from the web of like, syntax to like the web of Symantec. So if your job is valuable because, you know, a unique syntax or like, you know, how to transform things from like words to syntax, I think that will be a lot less useful going forward.[00:17:45] But the Symantec piece is still important. So a lot of product work, it's not just writing CSS and HTML and like the backend for it. It's a lot more than that. So I just thinking about how do you change your skills to do that. But yeah, even the sketch, you know, you gotta like, you gotta draw the sketch and to draw the sketch, you gotta know where the button should go.[00:18:06] You know, you have, you know, incorrect with it. Yeah. I'm just processing this as I, I just read the whole thing as well. And Yeah, I mean, it's been a wild wild couple of weeks and it's gotten me thinking that maybe all our role was over the past couple years was we were just middlemen to talk to computers, right?[00:18:27] So we're sitting in between, it's over man PMs or business folks or whoever wanna build a product. And then as a software developer, you're just a middle manish talking to the machine and it seems like. N LP is the way forward and, oh, yeah. Yeah. It's, it's been it's been, it's been a while.[00:18:42] ChatGPT Plugins as App Store[00:18:42] Couple of weeks. It's, I feel like we all just have to move either move upstream or, or find other jobs. You just gotta move upstream, either toward product directly. Cuz right now the plugin is yeah, is, is just you know, it's still a very sanitized UI that is controlled by OpenAI. But imagine them opening up the ui portion as well.[00:19:03] So you no longer need to have a siloed product that needs to integrate. ChatGPT instead you can bring your product directly into into ChatGPT, I don't think exactly. I think that would be probably the next next logical move after this, and I'm sure they're already thinking about that.[00:19:22] So that's a great, I don't know if this is, it's wild. What are you guys think? Yeah. Yeah. Like, so before you came up, right, I was, I was talking about this like ChatGPT has at least a hundred million users. Why would you bring people to your platform rather than write a plugin for ChatGPT and use their platform?[00:19:39] It's an open question now. Zapier just launched their integration. OpenAI and OpenAI just launched their integration of Zapier. Which one is gonna be more interesting? Probably OpenAI.[00:19:50] Totally a hundred percent . this is the app store of wow, our century of our decade. Like, I don't know, maybe century. I, I think the thing with ster though, if you think about it, like how many native apps do you download every week, every month versus like how many web things you use. So I think it's all about whether or not long-term opening eyes incentivize to keep broadening the things you can do within the plugin space.[00:20:17] And I think the lab, you know, as this technology gets more widespread, they're gonna have a lot more pressure from regulators, safety, blah, blah, blah. So I'm really curious to see you know, all, all the, all the government stuff that they'll, they'll have a congressional on this in six months and by then it will be completely irrelevant.[00:20:34] It's like that beside that time, they, they, they called it the GameStop guy after he made like 20 million on GameStop. And he just, you know, he was like, yeah, you know, followed the rules, made a bunch of money for those who don't know, unless you're our co-host. On the, we were supposed to drop an episode today, which I was supposed to work on, and then Chatty Phi dropped this thing, and now I, I can't think about anything else.[00:20:59] So this, this is my excuse for not, for for not working on the podcast today. . I know it's funny, we have like three, four recorded ones and spend last week, like GP four came out and we're like, okay, everybody's talking about this is irrelevant. What else? Anything else? Like, but I'm really excited about the, I, I feel like the first, the first use case for this, and I think he tweeted it about it too, is like, before if you had to do like data reformatting and stuff like that, it was really hard to do programmatically.[00:21:32] You know, like you didn't have an natural language interface and now you have it. And before if you had to integrate things together, like you could explain it very easily, but you couldn't like, put the APIs together and now they kind of remove all that part. So I'm excited to see what this looks like.[00:21:48] For commercial use cases, you know, you could see like, is there gonna be like a collaborative ChatGPT where like you're gonna have two, three people in the same conversation working on things. I think there's a lot of ui things that will improve. And so as we have lining from OpenAI for a second, almost pulled them up, but I'm sure you cannot talk about it[00:22:07] But yeah, it'll be interesting to see. Yes, sir. We're extremely excited. Extremely excited. I, I don't, if you, I don't know what else I'm, I'm like, so as far as I can tell there's the, there's hacker and Twitter. I haven't looked at Reddit yet, but I'm sure there's a bunch of reactions on Reddit.[00:22:23] I'm sure there's the OpenAI discord that we can also check out. I got locked out of the discord at some point, but yeah, anyone, anyone else like see news, demos, tweets the whole point of this is that it's live, so please feel free to share on comments or anything like that. But yeah. Yeah, the, the craziest thing I saw was the Mitchell from Hash.[00:22:44] We tweeted about Yes. How the integrations actually work and you just write a open APIs back and then just use natural language to describe what it's supposed to do. And then their model does everything. I wonder if they're using the off-the-shelf model or they have like a fine tune model to actually run integrations.[00:23:02] I wonder, I don't think they'll ever say it. Knowing them, probably they would just use the base one cuz they want, like, I think opening eyes kind of wants a God model, right? There's no point. It's not intellectually interesting to do small models, but like, like it's trivial. Yeah. Yeah. It's, this is a minor optimization problem as far as the, the long arc of history and the, the point is to build a gi safe agi and I, I do think this is kind of safe, right?[00:23:33] Like, . One of the criticisms that people were saying on hacks was that this is very closed. Like it's, it is an app store. At any point opening, I can randomly decide to close this like they did for Codex, and then they change their minds. Whereas if you use something like Alan Chain, it is more open and something that at the same time, like clearly this is a better integration path than long-chain.[00:23:56] Like, I much rather write this kind of plugin than a long-chain plugin. So they, they've managed to, I mean, they know how to ship man, like they're an AI research lab, but they also know how to ship product. Mm-hmm. . Yeah. I, I'm curious to see what the pricing models gonna look like. Also, I mean, if I'm writing the plugin, this is great because I don't even have to take care of the compute, you know, like, I just plug it in, then they actually run everything for me.[00:24:26] Yeah, but how, how it'll be monetized. I mean if the is giving their plugin know Expedia, I mean, people will not go to their website. Yeah. I don't, I mean, yeah. I have no idea that they, I don't think they said also don't super care . Yeah. It's because in the, in the app store, it's transaction driven.[00:24:46] But on Channel G, you're just paying a flat fee every month. So like, you can't really do revenue share on a flat fee. And I don't think that we use like, the Spotify model, but it's like a why not the amount of times? No, wait, wait, wait, wait, wait. Why not , you have Spotify. I just, Spotify model works. Cause swyx has power, right?[00:25:05] Opening has power. Same thing. They have all the audience. Yeah. But every, every every song is like the same value. Like if you listen to song actor to song y. , like, you're gonna make the same money. Like if I'm calling the API to, for like the meme generator or if I'm calling the API for the, you know, business summary thing, they're probably gonna cost the firm things, you know, so it's kind of hard to model up for OpenAI to say, Hey, okay, we're charging, we're going from 20 to 35 bucks a month.[00:25:36] But then like, how do you actually do royalties on a per model basis? Like how do people decide what royalties to negotiate? This probably needs to be a flat fee, but I dunno. Or put your credit card it OpenAI and then every time you wanna use a plugin, you pay for it separately. Uvp, usage based pricing all the way, and then you just get at the end of every month.[00:25:58] Exactly the, the only question mark is like, how much does OpenAI value the training they on and like how much they wanna subsidize the usage. Canada they have, they have promised to not use any of our usage data for training. So, oh, but the, I think like the plugins, it's a, it's a different thing.[00:26:16] It's like, like how you could, you could easily see how are like requests usually structure for like these things, you know, like, are people searching? So how are people searching for flights and stuff like that. I don't know. I haven't read the terms for like the actual plugin, you know, so. Well if anyone has please come up to speak cuz we're all processing this live.[00:26:37] This is the therapy session. Yeah, go ahead. One thing I see is basically you have to change the plugin I mean, to ask anything or even if you did browsing, right? I mean I see. I mean, they are becoming directly competitor to Microsoft also, I think, because now a user can actually just see, I mean, instead of being chat or Google, I mean they, they just.[00:27:04] Basically select the browsing plugin and basically get all the updated data. And other thing I see is basically you have to change the plugins. Like if you want to use the Expedia data, I don't know how it'll fit with the browsing plugin or you can select multiple plugins. But yeah, it is interesting.[00:27:23] I mean, if we get access, yeah, there is no actual browsing plugin. The browsing is a new model. So just like you can select GT three, GT 3 45, GT four, there's a new model now that says browsing alpha. So you, you can use CHATT in browsing mode and then you can use it in plugins mode, which which is a different model again.[00:27:45] So the, the plug browsing don't cross over.[00:27:51] Oh, that's interesting. And how do you see, I mean, in this whole descending, they are becoming competitive to Microsoft or how they're playing it out. I mean, Bing is just by the way, like, yeah, this, this killed the bing wait list. Cuz you don't need to wait for Bing. You can just use the browser mode open of Chatt.[00:28:11] How does it compete? It competes for sure. I don't think Microsoft cares. I don't think OpenAI cares. This is one of those things where like, you know, they are the two, two friends, you know, and they're clearly winning, so who cares? I don't like, I don't imagine it takes any of their mental bandwidth at all.[00:28:29] Yeah. The main thing is Google is Yeah, the main, like how is Google competing? Well let's see. Right. Bard is out there. I haven't got us yet, but could be interesting. Again, like it doesn't seem like they have the shipping capacity or velocity of Open I Microsoft and. That is probably going to bite them eventually because there's already been a big brain drain.[00:28:53] Something like four researchers, four, the top Google Brain researchers left Google Brain for OpenAI in January. And you know, those are the ones that I know about. And I, I imagine there's, there's quite a bit of brain, brain drain and firing going on at Google, so who knows.[00:29:08] All right, well, any other topics, concerns? Hyperventilation, if you just wanna scream I can turn down the volume and you can just, ah, for like five minutes. , that was literally, I was like, I, I need to like scream and just, ah, because what is going on?[00:29:29] I said that I'm filling out the form right now for the Oh, yeah. Okay. So wait list. So use use chat t to fill out that form. Right. And then, and then use a fake, use a different email and fill out the form a different way. This maximizes . I'm going to ask GT for what plugin do I want to build or, right, right.[00:29:51] Exactly. Yeah. Yeah. I, we can brainstorm. My plugins can live. Yeah. I think that will be a fun exercise. Like the, the main thing that breaks my brain is just this, this whole ability to run code, right? Like this is a new notebook, a new ripple. Mm-hmm. It, it looks like it has storage and it has memory.[00:30:08] Probably it has GPUs. That, I mean, can we run Lama inside GP?[00:30:19] I don't know if that's a, a model within a model. I think for me, most of the things come to like, you know, if I have my own personal assistant, what I want the assistant to do. I think like travel is like the first thing that comes to mind. Like, if I could use pt Yeah. Expedia, plug in with my calendar.[00:30:39] Yeah, yeah, yeah, yeah. But it needs to like know where I, where I'm supposed to be going to, you know, like if I just add a calendar that's like I'm going to, you know, room this week. Yeah. And then like can automatically both send my calendar and say, okay, these are like, or like the times that you like to travel, I know that you don't like ops and yada yada, yada.[00:31:00] That's one thing that I've always, we had this thesis at my peers firm about personalized consumer. There's so many website like, . I go to a lot of basketball games and every time I open Ticketmaster or whatever, it always shows me that she's a seat. And like, I'm not gonna see, that's not what I, that's not the tickets I wanna buy, you know?[00:31:18] But doesn't matter how many tickets I buy, never remembers that. So I think a way to say, to see, take all the information in and suggest, Hey, I saw that there's actually a price drop for the specific seats that you want, not for like any seats. You know, I think that would be a, a very good use case. So I've been a personal entertainment assistant for like, travel like going to shows, going to games.[00:31:41] That would be cool. That's what I'll submit on the wait list. Then we'll see if anybody cares. Right. Did you see get Lindy? Yeah. Yeah. At the, maybe you wanna recap, get Lindy for people. I'm gonna pin it up on the. . Yeah. So basically and this is like the kind of like a assistant lend the ai, right?[00:32:03] Yeah. Lend the ai it's on the board right now. Yeah. For those who can see it through the space. Yeah. Yeah. Actually at the AI Thinkers meet up the, the other day, you can basically like create all kind of like personal workflows and you, it kind of looks like integrations like zier, but it's actually just natural language.[00:32:24] So you can pop this thing up on your desktop and say, trying to hire 10 software engineers. So go on LinkedIn and plan 10 software engineers. The next step, draft a, an email that says, I'm the CEO of this company and I'm trying to hire for my team. If you wanna talk. Then the next step is like, send emails to all these people and it's gonna use people data labs or something else that they use on the backend to get the emails.[00:32:50] Then it actually sends the emails and. This is just gonna run in the background as if it was like you actually doing it. It's pretty neat that you don't have to write the actual integrations. Like it just uses natural language so you're not bound by what they build. Like theoretically anything you wanna integrate with, you can just explain to it how it works and it's gonna figure out how to do it.[00:33:12] So there's a wait list now. Flow didn't give us any papers just because we were at the meetup, so I'm also waiting to get access to it, but it looks really, really good. Yeah, so generative AI's top use case is generating wait lists, right? Like we we're, we are, so we have never had such an easy way to generate a lot of wait lists.[00:33:30] A lot of signup for witness. Oh my God. So much interest. So much product market fit. But also you know, one thing that you, you raising this point? I think, I think, I think by the way, I also pin this up. Mindy can support complex roles like no meetings on Fridays, all one-on-ones on Monday. , I like my meetings back to back within five minutes.[00:33:47] Five minutes in between. So it's just arbitrary rules that you could not program in a normal assistant type environment without a large language model. Which is kind of exactly what you want when you're booking your travel, right? Like, hey, I only like aisle seats unless it's it's a flight that is less than one hour that I don't care, right?[00:34:02] Mm-hmm. . So stuff like that I think is, is super interesting. And but also like not a common use case. Like how many times do you travel a year? Like, you know, five, right? Like more than that, but yes, I think for, yeah, a lot of times it's not a, it's not like a super widespread thing, especially if you don't do it or work.[00:34:21] If it's infrequent, you want high value and then if it's, if it's frequents, you can do low value, right? Like that, that's the sort of binary tradeoff, like the Uber is sort of frequent and low value. Airbnb is high value in frequent there's something of that nature. . So like, you want, you want sort of inspections of that sort.[00:34:37] Google vs ChatGPT[00:34:37] But the other thing that you brought to my attention was, and, and has room for Google to do something is do you notice that OpenAI plugins, none of them are Google because they're not friends. So Open BT will probably never have first party access to Google Calendar, probably never your Gmail and probably whatever, you know, Google copies, OpenAI again.[00:35:04] They will do, Hey, we have all your docs.[00:35:10] Yeah, I, I, I'm interested in that because I don't know if you remember, but like in the first iPhone, like YouTube came, like pre-installed on the homepage and then I forgot when, but one of the early ioss, they removed it. So now obviously Google's not a friend. Who's gonna be a friend in the future, who's not gonna be like, do we all have to hail our AI overlords?[00:35:33] Yeah. To get access to the, the only plugin system. Yeah. The only winners are brown CEOs. Think you're fine. Alright. But yeah, yeah. I just invited nada. C my old boss. Hi. You can't lurk. I, I want, I want to hear from you. And but, but also, you know, yeah, I, I think the Google point is actually novel.[00:35:50] I'll probably write something about that. Yeah. I mean, I'll have to write something about this today. So please feed me things to write.[00:36:01] Nader Dabit on Selling His GPT App[00:36:01] Oh, there we go. Hey, what's up man? What are you think. I know it's like, not entirely your space, but like you're, you're all about the future, right? I mean I did build and sell an AI company about a month ago, . I did the wait, what travel app was built on GP T three Tweeted about You sold it? Yeah.[00:36:21] It was getting like a hundred thousand visitors a day, like 60 to 80,000 unique a day. And then I, whoa. Yeah, I sold it like within about 24 hours. I tweeted out that it was for sale. I had like 30 or 40 people in my inbox. Whoa, whoa, whoa, whoa. Okay. I need, so like, but you're right. This isn't my, my man like domain of expertise.[00:36:41] It's fine. You make, you may just a thousand dollars on the side. It's, it's cool. Wait, wait. So I saw you tweet your original thing, which was, Hey you know, GP three can plan your travel. I don't know what happened since then. Can you, can you fill the rest of. Yeah. Yeah. So I mean I was basically, you know, I travel a lot for work.[00:36:55] I, I do travel like once a month and, you know, but I'm also very resource constrained on my time. So I usually like to spend like one day sightseeing. So what I typically do is I go a trip advisor and then I kind of like, you know, Google around and like look at all these things and it usually takes me about an hour to figure out like what I wanna do on my day or two off to go, like sighting.[00:37:14] And then I realized GPT3, you know, you can just literally ask and, and say, okay, within X number of. Like, I'm gonna be in this city, I want to have an iter itinerary. You know, you can give all these different parameters and it gives back a really good response. This was before GPT, even three and a half or four was out.[00:37:30] So I just built like a nice UI on top. Then, like I mapped over the results and, and was linking to, you know, the the Google searches for these different items and, and kind of made it into a nice user interface and, you know, just built it out and tweeted it out. And it, it just got a lot of traction and attention.[00:37:48] Like I said, I had around a hundred thousand visitors a day, like right off the bat, 60,000 uniques like per day. So it was getting a shitload of of traction and. I don't have a lot of free time to kind of like, maintain or build something like that out. So it was costing me money, but I wasn't monetizing it.[00:38:06] So the way that I was thinking to monetize it would be to use affiliate links and stuff like that. So I could either, you know, spend time figuring out a way to monetize it or just try to make, flip it and just make some money. So I decided to sell it and that was kind of it. I just sent a tweet out and kind of said, this is for sale, who wants it?[00:38:25] And I had I had so much inbound from that that I had to delete the tweet within about two hours cuz I was just unable to keep up with all the people that were coming in. And I filled it out a couple of offers and I, I found the person with the most money that could close within the shortest amount of time and just took it.[00:38:44] Well done. Well done. Nice. Awesome. I need a, I need a, I need an applause button right here. . Okay. So with that context your thoughts on today, what you seeing? There's Expedia there, but. Comment on travel or not travel, whatever you want. . Yeah, I'm still reading up on the, the chat plugins actually.[00:39:01] And I was hoping to kind of chime into this to learn a little more about how they work. I'm here on the the page. I've had API access from fairly early on. I signed up and I've been you using it a lot. I'm trying to find some different ways to integrate AI and machine learning into the blockchain space.[00:39:20] There's a lot of stuff around civil resistance that I think are gonna be, you know, pretty interesting use cases for us. It's obviously not like a, a a type of use case that is gonna be useful to, to the general public maybe, but yeah, I'm still, actually still trying to understand how these plugins work.[00:39:35] So what have you seen the developer documentation, which developer documentation at the bottom? Yes. That's where I'm, I'm check, I'm reading through as of now, I see the examples, which are pretty cool. Yeah. Yeah. So my, my quote the, the quote I put on Hacker News was, this is OpenAI leveraging chat, GPT to write OpenAI op open API to extend OpenAI chat.[00:39:58] GPT. I'm confused, but it sounds sick, but yeah, I mean, so open api, you know, not to be confused is OpenAI is randomly the perfect spec for OpenAI to navigate because it, you know, is somewhat plain English. And then you just supply a description for model. You described a off method. So they actually provided a link to a repo where you can see some examples.[00:40:20] The examples are not very, not very flesh out. But you can do, like, bear off, I assume you can do whatever, whatever kind of off you like then you just provide like logo url, legal info url. It's not, it's not, it's not that much. This is 10 times better than Chrome manifest.[00:40:37] Like manifest v3. Yeah, I mean, I'm reading through some of these examples and a lot of them are in Python. I wish they would've more JavaScript stuff, but I would say 10 times would be kind of an understatement if I'm understanding how some of this stuff is gonna work. English is all you need, man.[00:40:53] English is all you need.[00:40:57] Well, so, so, and then I think in buried in the video is sort of the Ethan experience, right? Which is where you specify. So if you're, if you're first party congrats, you know, you're, you're inside of the the chatt ui, but if you're third party, you can just host your Js o file anywhere. It's literally a JSON file on an API spec, right?[00:41:15] You hosted Jason file anywhere. And then you just like plug it into their their, their text field here and then they, they validate a little bit and it's installed. So there is a third party app store on day one. Yeah, that open table plugin example is pretty sick. Yeah. So like yeah, I I What would you want as a developer that's missing?[00:41:41] I think that we're like in the golden age of of being a developer and I don't know if it's gonna go downhill quickly or if it's gonna go like, get better quickly or this is like the, the end of all of it. like, is OpenAI just gonna be where like we do everything like nothing else is like gonna exist.[00:42:00] I think that Okay. You know what I, I know that's not the answer for sure. I'm just kind of joking, but I think it will, this is obviously shut down a lot of companies. This is the app store moment, right? For like, just like, I mean, you and I remember the iPhone app store moment. Some people dropped everything to write apps and they made it big and some, a lot of people did not.[00:42:20] But the people who were earlier rather than later probably benefited from understanding the platform. Like imagine, imagine you, like, you know, you, you are a big React native person for a long while. Like imagine if you had the chance to drop everything and be one of the first developers on a new app store.[00:42:35] Like that's pretty huge. Yeah, a hundred percent. But I'm wondering like the, the type of mode that you'll be able to build with some of this stuff, because it seems like that OpenAI AI will just continue adding more and more features directly into the platform. But I think like for very like, Proprietary type of stuff.[00:42:54] It might make more sense, but like if you, if you want to build like an app for the general public it just seems like they'll end up integrating something like directly within their platform for a lot of different ideas like, such as this travel app that I sold. I have a feeling like they'll have a way better version of that built directly into their platform sometime soon.[00:43:13] Q&A: ChatGPT Waitlist and Voice[00:43:13] Hey, hey guys. Can I ask just to get a quick update does anyone here have access to it yet? Like is it, is it open? Cause I signed up for the wait list, but I haven't seen anything yet. Yeah, no, it's just, it's just wait list where just like 90% of the stuff that people launch, you know, she has a few, she has a few videos and demos, but yeah, it's just a wait list.[00:43:31] Who knows? I mean, thanks. Opening OpenAI Pretty has been pretty good about getting people off wait list, right? Like a lot of people got off the GT four API wait list, like the day after they launched. Mm-hmm. . This one, I feel like they're quite fully baked, like it's. I wouldn't be surprised if they started dropping tomorrow.[00:43:50] So we'll see. But like you can start developing your, your third party plugins today, because there's examples. The docs are like two paragraphs, but that's all I need really . So, so I've been, I've been working and, and I've been following a lot of projects where people are, the one thing I don't see with ChatGPT is like, why are they have, we have Whisper, we have the APIs for ChatGPT.[00:44:13] It's like, why are we not at the point where we're talking to this thing and it's talking back to us? Like, I don't know how we haven't, nobody's wrapped their head around that yet, but it's like, it seems to me like, don't you wanna be like, Hey computer build me an app that does X and it says okay and builds it for you and talks back to you.[00:44:29] Like, I just, it's like, I don't know. That'll be the first probably plugin that I try to work on, but it's just driving me a little nuts. That's all interesting. I like the voice interfaces because sometimes it gets really long, like some of the prompts get really long. They're like, I don't wanna talk that long.[00:44:46] Yeah, yeah. Yeah. I was so, so I was doing, I was messing with the system prompt, basically get it to be like, Hey look, I'm gonna be talking to you. So keep it condensed. I think like the ideal interface would be like, for like, talking to, it would be like putting that at like the system level, but also, you know, being able to type as well as speak to it is just something that I'm, I'm trying to work on.[00:45:08] And I think with Plug, you know, if we could do that with plugins, I'd be huge. Cuz I know there's already a, like a Chrome extension that allows you to talk to it. Or, or I guess you could do it natively as well, but, you know, native stuff on like iPhone and Android is not too good.[00:45:24] LangChain with Human in the Loop[00:45:24] Hey, you, you mentioned that. Hi, by the way. You mentioned the hey way of, of talking to or having way the AI talking to you as a user. So just today there was a new release to of LangChain. I know it's kind of, not really the plugin, but this is the closest thing probably. And they edit a Ask Human tool.[00:45:46] So now the model can ask you a question if it's not sure. About something[00:45:55] to share. Share what? Go ahead. So, so the ask you if it's during its chain of thought, when it's not sure. To an example. Right, right. Oh, I would love that. Yeah. Probably not gonna do that. It's too confident. Yeah, I, I've seen a little bit about. LangChain, but I haven't used it yet. Has anyone here it?[00:46:15] Oh, it's all about it.[00:46:19] I did, I did. I built the LangChain on UI too. It's pretty nice. I mean, especially when it first came out, the, the trolling, it was like so rudimentary. But it's nice to be able to change things together. I think the agent part is pretty interesting. I haven't used it myself because I didn't need it.[00:46:34] But yeah, there's a, a very big community. See, see, light chain was very smart, right? Like they picked out the open source angle first, and then the others like dust or did the closed source angle. Now they have indirect competition with ChatGPT, but Langchain still has that. It's open source, extensible, like you own your agent.[00:46:55] Google vs Microsoft vs Apple[00:46:55] Them doing business deals with OpenAI in, in closed doors, right? Like, so pretty smart, like strategic position. All things considered.[00:47:05] It's a little, isn't it? It's like a little funny to me. That, you know, it's like goo because Google just came out with Bard, right. And I don't know if you guys have messed with Bard at all, but it's at least to me another wait list. Oh, okay. Yeah. I mean, to me it was a little underwhelming. I mean, I'm, I don't know if you've seen like the same, yeah, if you've seen like the screenshots going around, like it seems like, you know, someone tweeted it was like in, in guys in a boardroom or whoever's in a boardroom just being like, s**t.[00:47:30] Like, we need to you know, we lost our first mover advantage here. But it's just kind of funny to me that like, I guess now Microsoft's gonna have like an app store, right? Like just after everything, you know, Microsoft dominated in the nineties and stuff, and then it was Apple, apple, apple. But it's just kind of funny to me that it's gonna be, I guess Microsoft now, right?[00:47:49] Bard feels like Bing does to Google. Totally. Yeah. A hundred percent. I agree with you a hundred percent. All the turntables, right?[00:47:57] Yeah. So for, for those of you who might have missed the earlier discussion the one thing that OpenAI or Microsoft will not do is integrate with your Google calendar. So, the one saving grace that Google probably has it, it probably owns your workspace, right? Like most of us have Google accounts, Gmail accounts.[00:48:14] When we work, we log into Gmail and Google, again, use Google Docs spreadsheets. So if Bard is smart, they will take advantage of that. And then slowly watch as everyone moves to Microsoft Office.[00:48:31] I think Apple should do a partnership with the OpenAI and basically Microsoft. Cause Google has huge advantage of Android. So basically having OpenAI on the, I, I mean, it would I mean having the partnership with OpenAI would make, I mean, very useful on I devices if they, I mean, Siri is really bad and if they integrate with I, I mean they've win the world I think.[00:49:00] So it would be huge, beneficial to Apple and basically the Microsoft also if they integrate together because Microsoft doesn't have any of the devices and most people, I, most ordinary people use the devices iPhone or phone and . So it would be huge advantage. And for the 10, basically Apple I, I'm very curious to see what Apple ships next.[00:49:24] You know, everyone's shipping AI stuff and then Apple was like, Hey, look at our AR glasses. . Yeah, but I mean, ar ar with, with the, with the 3D models that are, that are coming out cuz isn't it mid journeys working on like a three, like their lab, I know is, is building a 3d generative model. And I think that sort of stuff with, with AR is very, oh, is that, is that public?[00:49:45] How did, how did you know that? I don't know if it's public. I, I saw a tweet about it I don't know, like a week ago. It is a semi, semi open secret in San Francisco, but I, I don't know if it's public. Yeah, I think I, I saw them, it was some context of they were talking about text to video and they were like, well we're, we're doing our like 3D modeling first.[00:50:02] So, I mean, my assumption is, and I, I don't work in the space yet, unless anyone's hiring please, I'm looking for work. But it seems to me like Apple. Seems to have their head on straight and like it might be that if they're gonna release these ar like mixed reality ar vr glasses, like, you know, the mo the thing that makes the most sense to me is like getting with generative AI graffiti modeling.[00:50:24] It's like, you know, it would be cool to go to like a coffee house or a bar. And then, you know, when you see like the graffiti in the bathroom when people write sometimes funny stuff, sometimes, like the worst stuff you've ever read in your life and you're like, what is going on when this person's going to the bathroom where they have this much hate?[00:50:38] But it's like, it would be cool to have a component of that, you know, like in the metaverse, so to speak, right? Like, so you put on your AR glasses and it's like, oh cool, I can see like a bulletin board here that exists in the fizzled. But it's also in the, you know, it's like augmented, right? That's just, to me it seems to be like the logical next step.[00:50:57] Interesting. Well, we'll, we'll see that when that happens. I recently got a Quest Pro quest to my, and yeah, my parents love it. And any tech, any type that my parents like, I think has a real crossover appeal. You know, the thing that you, your conversation had gimme an idea for winners of every app store in the early days, like Facebook has an app store, apple had an app store, you know, the winners of an app, store games like what we need Yep.[00:51:24] Is a multi-player. Like everyone logging into chat, BT and then playing a multiplayer game line. Mpc. MPCs are gonna text you on your. , that would be kind of cool.[00:51:40] ChatGPT Plugin Ideas[00:51:40] Actually. I was thinking, I don't, I don't know if it's gonna be game games at first though. Like, it seems like games always push the envelope with tech.[00:51:47] Well, it's like pornography and games, right? But like, I don't know, I was talking to like, you, you mentioned your parents and like you know, I was talking to my mom about this stuff and I was like, you know, I'm seeing stuff that are just demos of just like, Hey, take a picture of your fridge and it'll tell you like, here's what you can make.[00:52:01] Or you know, even like talking to it and just being like, Hey, here's what I ate today. You know, what's my, how many calories I ate today? Or, you know, what's my diet plan? Just things like that. And that's why I brought up the talking to it just with na using natural language and then having it, being able to talk back to you.[00:52:17] I'm surpri I'm like really surprised that they haven't implemented that yet. Cuz it seems to me like that's a use case that a lot of people would use it for, you know? Or if you could just like, you know, call it on a phone if you built like a Twilio back in, into it or something. Like I just don't, it, it boggles my mind why they haven't.[00:52:35] Put that feature in yet? . Yeah. Yeah. I really don't think it's gonna be too long before you're, you're sitting there at work and you get a text or call on your phone from an nbc, Hey, our village is burning down. You need to come over here and help . Do, do you guys think there's gonna be different silos?[00:52:55] Like you know, with Bard coming out and you know, people implementing GP T three and four now, I guess, into all their apps, but do you think they'll be like, chat GP p chat, GP, PT will have their store and then Google will have their store? Do you think it'll be like, there's gonna be a clear Victor here and then, you know, it'll be like, okay, Google's apps or, you know, Google Docs or whatever is like part of chat GP t's plugins, right.[00:53:20] Yeah, it is gonna be like crypto. Everybody's just gonna be fighting for the top. You're gonna have the couple of dominant people, but then you're gonna have all the, the small guys who go up and down and Yeah, I I, I feel like it's gonna be pretty similar to, to how crypto was. So we're gonna have some slur juices is what you're telling me.[00:53:41] Yeah, boy. Nice, nice. I dig it.[00:53:46] Not an app store?[00:53:46] So may maybe we aren't, tell me what you guys think about this, cuz maybe we aren't thinking about this right? Because maybe this is not an app store. Cuz typically in an app store you'll go ahead and choose which plugins you want installed, like on a phone or whatever have you.[00:54:02] But the path forward seems like all the plugins are like omnipresent. I, I don't know why Google isn't shitting their, shitting their pants right now. Cuz basically you check like openly I could just force all. The big companies to write plugins and then just be a single search box for everything. So imagine if you wanna like fly somewhere or you wanna book a hotel you, we have the Expedia and booking.com.[00:54:29] Both of those plugins summoned up and it shows you both the results. And then you can click through on whichever ones you want. And then, yeah, you charge 'em based on click throughs. Like I, I think like we're, maybe we're just getting tripped over by the fact that you have to choose a plugin right now and only interact with that single plugin.[00:54:49] But I think I think the smart move forward would probably be just to have all of them omnipresent and then have this like n l p higher layer up there to summon the right plugin when need be. What, what do you guys think about that? Yeah, so, so that's like the LangChain thing. That's what I haven't used LangChain yet, but it sounds like that's, from what I was reading with LangChain, it sounds like that's kind of is how I thought that worked.[00:55:12] But I don't know, can someone here like enlighten me? I, I don't know if it, how, how LangChain works.[00:55:21] LangChain and the Future of AI[00:55:21] Yeah. I don't know how LangChain works either, but I think it's gonna be a two-way street. Everybody's gonna be making plug-ins with chat GP p t and everybody's gonna be making chat GP plug-ins for other services as well. I think there's gonna be a whole bunch of people about to make a bunch of Jira plugins and stuff like that, so I think it's kind of gonna be a, a two-way street.[00:55:45] I dunno, is anyone else, like, this is super exciting to me. I haven't been this excited about like, the internet since like, probably like the, like the web 1.0 days. Like I, I, I hate, I'm so . Yeah. Like, I hate web two. Like, this is cool. I'm glad that like spaces exist, but I hate Web 2.0, like Web 3.0. I'm about, and like, I, I consider this part of Web 3.0.[00:56:04] But it's exciting, right? Like, this is cool. Like I, I'm really, you know, I'm stoked about, about the progress that's being, like, the joke is like, you know, every day in, in AI is like, it's like way longer, right? It's like we're telescoping very quickly. Yeah, I mean, one of the things, telescope and updating.[00:56:23] Yeah. You know, I, I would say I noticed towards, maybe like three years ago when I was working at aws, it just seemed like for, for about five or or so years, everything was very stagnant and there just wasn't a lot of exciting things that were happening. Everyone was like, if you remember, all the Devrel advocates were like all creating like tutorials around creating your own CMS and your blog, and you saw like that exact same tutorial given by like hundreds of people over the course of a few years because there just wasn't any cool s**t that was happening.[00:56:52] And then I think when crypto and, and blockchain stuff like that kind of caught my attention. Caught my attention, and I'm still excited by that, that stuff. And then this seems to be just almost like when, if you were like around when the iPhone was coming out and actually realized how important it was, I think everyone now is, is seeing this and they're all like realizing how important it is.[00:57:13] And it's cool to be like part of this moment as a software engineer. Yeah, I'm, yeah, go ahead. Oh, sorry. I was gonna say, like, I'm, I'm excited for you, I'm sure you guys saw the alpaca stuff, right? And I know that they're doing D D M C A stuff, but essentially someone's gonna train one of these models and it's gonna, you know, you're gonna be able to run this stuff offline.[00:57:35] And just like the way to, if, if you have access to like I forget which one of the EAC accelerate people was talking about it, but it was like wharf in the flask. It's like you've gotten the machine offline. So if you don't need internet access to access, like, the entirety of human knowledge, whatever's in the data set up until 2021 or whatever, and you don't need internet access, like that's gonna revolutionize everything.[00:57:57] Like, that's insane to think about[00:57:59] Yeah. Oh, well we won't speculating You can run in Inside Chat runs Python. Oh, really? Is that, is that happening? I mean, it has a file system and it has file storage and CPU at memory. Yeah.[00:58:20] is turtles all the way down. Turtles all the way down, man.[00:58:23] The, I, I think the plugin system, if people can get to run their own models like the LAMA ones and the same structure for plugins, you can see like going back to the Metaverse thing like a and snow crash where people built their own like demons. You know, it's like I got the demonn that like kicks people out of the club, the, the black sun.[00:58:43] But you can see in real life it's like I have a bunch of plugins that only I have, you know, and I use them to make myself more productive, use them to make myself, you know, look like I'm working when I'm not working and I'm like responding to my emails and stuff like that. But I think like, The OpenAI releasing this today makes it so much easier to start it because you don't have to worry about any of the infrastructure.[00:59:07] You just build the plugin and then they run everything and you get the best model possible. But I think none line, you know, I would love to walk around with my own, you know, raspberry pie or whatever of my wrist, kind of like I'm fall out and say, Hey, I wanna do this, I wanna do that. I don't know, I don't think we're that far away, so I'm excited to, to keep building.[00:59:28] Shoot, the, the technology exists where you could make that now, but it'd be a little awkward to have a raspberry pie on your wrist at the moment. . Well, well, well, that's kind of what I'm saying with the, with the al alpaca thing, right? It's like if you don't need internet access to, to use the model, I mean, we're, we're still pretty far off floor.[00:59:48] I don't know if Moore's Law even applies anymore. You know, we're not that far off from being able to run this stuff on, you know, consumer hardware that's cheap and that's gonna be huge for, for, you know, the majority of the world, right? Like, that's gonna be very big. Like e e even bigger than this. Like, it's great that we can do it with the internet, but as soon as we don't need the internet to access it, like it's, it's over, but we're back.[01:00:12] Whatever, whichever one you believe. It's just, this is crazy to think about that. Yeah, you could, you could if that happens, you can go and hook it up to a coding compiler and have it sped out human readable errors, but at that point it's probably just gonna be brighten on the cup for us anyways.[01:00:30] So we have a Hey guys. Hey. Hey, Alex. Go ahead. I, one more question, but yeah. Oh, go ahead. No, no, no. I have a right in question from someone who's trying to join but was unable to Stefani. , who I met, by the way, at the LangChain Hackathon, LangChain meetup in San Francisco. She has a lot of cool insights.[01:00:45] Q&A: ChatGPT Bots and Cronjobs[01:00:45] Follow it. Yeah, go ahead, Alex. I'll, I'll cue the question up. Oh yeah, for sure. Uh, One thing that really got my mind out this stuff and, you know, high vision mode is the fact that you can kind of externalize memories now. So the main use case I was thinking about is you could basically set up crime jobs, for lack of a better word.[01:01:04] So suppose you're, I don't know, building a trading bot, right? And you can say, Hey, Chad, GPT, look at the price of wheat every day at midnight. And you can just cue that up in the background and then have that send the response back to back to the LLM at a certain time. , and, you know, that's just like one use case.[01:01:21] But here comes like the play where like there's time sensitive things that break the one by one synchronous nature of ChatGPT and adds a little more, you can say from one level more humanness to it rather than like direct response and reply with latency. So there's that level, but also you can like schedule tasks and I think that's gonna be the killer plugin, whoever creates like the, the cal.com or the you know, theron integrations for just like, Hey, look at this point in time, and they give me the response.[01:01:48] I don't know if anybody's been thinking about that. Yeah, I, I was thinking about that a lot. Like how you said the expand, it's like an expandable, it's like a portable brain. Like, it's like, Hey, here's my secondary brain and it does, it's like my secretary, or it's like my assistant, right? Like somebody had a prompt where it was, you know, you're a form of, you know, one person's wisdom, one person's, you know, thinking about.[01:02:11] things x X way. Someone's thinking about it y way and like being able to have that just on demand with the like expandable component where you're able to basically Yeah. Delegate tasks to it and be like, Hey, you know keep what's, what's like the way to think about it? Like, not like a crime job, well, sort of like Aron job, but like like, you know, like news alerts, like Google news alerts, like things like that.[01:02:33] Just being able to be like, Hey, like keep, keep an eye on this thread for me while I do other things. And then if something comes up, you know, whether, you know, you just do some NLP or whatever, search for keywords you know, alert me or do whatever. And being able to do that without having to go through, you know, setting a reminder or doing all that painful, like, pain in the ass calendar stuff.[01:02:52] Cuz I think there's so many, there's so much software for that because people just hate doing it so much. Like, that's gonna be so big. Yeah, no, I was thinking that's probably a better way to put it, right? Like asynchronous alerts or I guess you could do timed alerts also. Because the one thing I was thinking about is the Instacart api, which is what they're demoing.[01:03:10] I don't know if anybody uses Instacart, but it's pretty slow on the lookups. So that's like, you know, that's a blocking process in the current integration of chat GPT. But if they could figure out a way to make it like asynchronous and then actually interact when it's done getting the, the fetch, and then you can do stuff in between that, that's gonna really change the interface.[01:03:27] And that's like, that's really the step closer to having like a real personal assistant in your pocket, man being able to just give chat d p t all of your Ps that you cook that week, and then just have it, order all the stuff from Instacart, from you. I can't wait for that man. Oh my God, that's great.[01:03:49] Oh, okay. Okay. You know, you can ship a boat Logan, like a cook, a cookbook with like actual recipe, but yeah. Yeah. Let's introduce Logan. So does this, like physical companies that integrate with software are gonna be coming like more of a moat as opposed to just software specific companies? Every software is a software company.[01:04:10] I know[01:04:13] Yeah. But if you're just a software company, OpenAI or, or, or some, one of these companies can just build that feature in now a lot easier than they could maybe in the past. Yes. For instance, you know I don't know, like we were talking about travel and, and stuff like that. But, but let's say you have a physical, you know, product that that, that maybe you can just separate yourself from other products by building, you know a better quality user experience.[01:04:40] Logan Joins Us![01:04:40] And so we got Logan here was our first podcast guest and the first Devrel person at OpenAI actually. . So Logan, welcome. Obviously, a lot of people here are excited to talk about this. One thing I noticed from the plugins is that a lot of them are more mundane things. You know, you got travel, you got grocery.[01:04:58] Can you tell us a bit more about how you picked those and like maybe give us a sneak peek of other use cases that you all are excited about? Yeah, I, I think first of all, I think going back to the conversation about the ability to like queue up tasks for you in the background, I'm, my understanding is that Zapier actually already does this by default.[01:05:20] And I'll, I'll go play around with it after this and see, but my, I, I think Zapier has the ability to schedule things and I think this is the part. Yeah, people are sleeping on this the most is that basically Zapier is already connected.[01:05:36] Zapier's already connected to 5,000 different plugins, and now you can just integrate directly with all of those through Zapier, which is incredible. So you don't even need to wait for like the plugin or whatever to come. Zapier will already do that for you. Which is, which is super cool. And it already has the ability, I'm 90% sure to like schedule certain actions to happen which is awesome.[01:05:57] So I, I think going back to the point of like how these folks were, were specifically chosen, I think the reality was when it was initially scoped out for doing this work, there was just, we needed people who were willing to sort of deal with the idea of of sort of, we were still building this entire platform and infrastructure from the ground up.[01:06:15] And I think those. Those folks who were featured today during the blog post, did a lot of work of iterating on these things with us as we figured out a lot of the challenges. So huge shout out to all those, the engineering teams of those companies for, for working with us so closely to make it happen.[01:06:32] I just gotta say too shameless, shameless plug here. It's my birthday today and this is a super cool birthday gift. So thanks for, for doing this and the blog post. It's really awesome. happy birthday. Yeah. Thank you. Thank you. I think we all just got a, a huge gift like look like. Yeah, Logan, you don't have to speak on opening as we have here.[01:06:50] Like, we're all just like, you know, large model and Enjoyers here. I think. And this is a, this is a app store moment for like all of us. Like it's, I I'm just processing this and, and just trying to. Do therapy in public[01:07:06] There's a lot of wait list fo here, so we're all excited. Oh, yes.[01:07:11] Q&A: Plugins Rollout[01:07:11] What do we have to do to get the wait list? Yes, . I, I think the reality is yeah, it, it's, they're rolling people out really slowly and I think the intent is part of this is to understand, and I think it was one of the big highlights of the blog post about what are the new sort of accesses for, for harm here.[01:07:30] And I think we know some of those things, but there's a lot of known unknowns, so it'll be intentionally small for the time being. But hopefully we'll, we'll expand that access in.[01:07:44] bottom line, get on, get on the wait list and, and keep your, keep your fingers crossed. Come up, like come up with a cool use case. I think there's something, there's part of the wait list is like submitting what you would be interested in working on and actually in, they actually will, we will actually read that to make sure that, you know, we're bringing people in who are gonna build cool things, not stuff that's uninteresting or potentially harmful.[01:08:06] Okay. Are you using Tri GB two to analyze the wait list? ? Yeah, that was my question. It'll probably be humans to analyze the wait list would be my guess, but maybe, maybe not. I'm not sure. Very old. What's the difference? Old, like, yeah, yeah, we have a question from write in who couldn't join for technical issues.[01:08:23] Q&A: Plugins Discovery[01:08:23] Stefania, who is a researcher at Microsoft right now. and her question is about search. How what is the future of search for plugins? How do we discover new plug-ins? Do we need a schema for plug-ins with complex queries or, or complex behaviors? And does it limit the context window as well?[01:08:41] Like, do we install like a hundred different plug-ins and like, does that, does that hurt help? I don't know. . Yeah, it does. So there's a limited, and I, it's all in the developer documentation right now if you wanna read through it. But there's a bunch of limits on like your open API spec and the descriptions you use.[01:08:58] But we actually take all that information. We take a sample request, we take a sample response, we take the description of it, and it's actually all inside of the, the context window to begin with. So it is limited right now. And I think that's where some of those larger models like GPT four with 32 K contacts in the future, when that's the available will be super helpful and you'll be able to bring a lot of plug-ins in.[01:09:20] But at the current moment, the more plug-ins you add, the less, the less context to you you actually have in the conversation. Yeah, yeah, that makes sense. Makes sense. I mean with like 50 pages worth of context, that that's a lot. And you know, I was very impressed at the latency as well that that at least the demo was able to pull off, which is awesome.[01:09:39] Yeah. Any, any other like, reactions, thoughts, questions to plugin? I have a couple new people joining. Hey ar Yeah, I had a couple of them. If I can chime in. First of all, just blown away. I mean, it's a fairly interesting approach to deal with, like live data with this data that you guys train on. Couple of quick questions for you.[01:09:57] Q&A: OpenAI vs BingChat[01:09:57] How do you see this? Maybe it's too early to ask, but how do you see this starting out to something like a Bing Chat? The, the reason why I ask this is, I mean currently Bing is more of the UI that you're dealing with and chat GP t's being launched on the side. But do you see it more being like a platform or do you see it more consumer facing?[01:10:20] I mean, I dunno if this question was to me or not. Yeah, you don't, you don't have to answer that. You know, obviously Logan cannot comment on Microsoft.[01:10:31] I do think though, that the, the interesting differentiator is that the, the work, and I think this was in their public blog post, is that a lot of the stuff that Bing is doing is optimized for search specifically. So it's, it's just a fundamentally different experience. I still think that like if you're, if you want like that search first experience, I think something like B makes a ton of sense.[01:10:54] Yeah, it's just, it, it feels like a different experience to me, so, thanks.[01:11:00] Q&A: App Store Monetization[01:11:00] So I think it's been mentioned a few times that this is like the new app store or ai. What, I guess I'd, I'd like to hear thoughts of other people as well, but like, what's the, so the app store is monetized, right? So that's a big incentive for people to put their apps on there.[01:11:14] So how does in, in this case, you put a manifest and it hits, hits the API for your app maybe. So what side of the monetization strategy here? I mean, this is not a question for OpenAI, it's just like a general sort of direction for things. Yeah. I don't know if they care. , this is like trivial to OpenAIr.[01:11:34] Yeah, we were talking, you're paying for the api, right? So you're you mean like on top of, of paying for API access, like you're using your credentials, you supply your credentials when you, when you sign up to plug in. Right. So I guess you do building off platform.[01:11:50] Yeah, I guess so. So not from an OpenAI point of view. So Open of course, makes money on wins anyway. What I mean is like for an app developer to go on there. So I guess you have an app outside of OpenAIr, which is useful. And this is kind of distribution for your app. Is that, is that kind of the, the sale for the app?[01:12:07] I mean, we're three hours into it, so it's hard to say , definitely. But I think that's, I'm just waiting for someone to write a mega threat on how to make money with the app store here. Seven ways. I'm sure. I'm sure there's gonna be people on YouTube making videos with themselves streaming, and that's how they all saying, I just figure figured how, how to make millions.[01:12:27] But yeah, one model we were talking about was maybe you can do kind like Spotify or like a, you have Achen GD subscription and then people each plug in gets royalty. Or a lot of things. So like Instacart, like the Chan GD thing is more like a UI alternative rather than like an app itself.[01:12:46] So it makes a lot of sense. Do I have things like that? But yeah, it would be. . Yeah, I guess what I mean I think Dylan or somebody else said earlier that this might not be the, the app store might be like something different. I think App Store is like the closest we, we have to think about. Like that's the closest analogy, but it might be just something completely new.[01:13:06] And that's very interesting. I think that's that's a pretty, pretty exciting place to be. Well, well, I don't know how much overlap with like the web three stuff, but it seems to me, I know there's like a couple projects out there that are, I think there's one called Bit Tenser, where it's like people are you know, basically selling their you know, their, their GPU usage, right?[01:13:24] Like, you know, there's tons of gamers out there that just have, their cards are just sitting idly by, and I don't know, it seems to me like a monetization model for OpenAIr might be to, you know, they own the model, right? So it's like, I don't know if they can like, lease out the model if you could like write a smart contract that like, uses their model somehow, or, I dunno, maybe plugins could be like written into a smart contract where it's like if you, if you're using this plugin, like, I don't know how that would work specifically, but thinking ahead, like, I don't know, do you think it's gonna just be centralized this, this whole time or like, surely there's gonna be a way for this to, to spread.[01:13:58] And you know, obviously like there's a. What's the, what's the word? It's, it's kind of like you're trying to hold all this water back with like this one stone, and it's like eventually it's gonna break. So like, there's gonna be some decentralization in this at some point. So I don't know if that makes sense.[01:14:12] I'm just trying to think about like, how, how there's a monetization you know, pathway for, for this. For, for the, for these plugins.[01:14:24] Yeah. We're not gonna get the answer today.[01:14:34] Let's, it's Farmville. We're gonna, we're Farmville on ChatGPT. Let's do it. Yeah.[01:14:42] Q&A: ChatGPT Plugins API[01:14:42] . Yeah. I was interested in like if there's already an API for this or if there's like an planned, so like when chat was just a weapon interface and then we got the API later, or is this like a web only?[01:15:02] There is a API available today, but you have to have access to actually create plugins. So you won't have the interface to install a plugin or do anything like that. You can basically build all the stuff on the backend right now if you want to, and then when you get access you'll be able to actually install the plugin through the ChatGPT UI test it out and all that stuff.[01:15:23] But as of the present moment, no one beyond a very small group of people are able to actually install those developer unverified plugins. Yeah, I was I don't know if if that's what you meant, but I was thinking about like, do we have a programmatic way of calling the ChatGPT API with these plug-ins enabled and get like adjacent response back opposed to like using the weapon interface with the plug-ins enabled?[01:15:47] Yeah, so that, that doesn't exist yet today either. I think it's, it's unclear when and if that will come, but it's definitely something that folks are, are thinking about. I think there's just a little bit more a bunch more security and other challenges like that when you give the plugin access through the api, but it's, it's definitely something the team has talked and thought about internally.[01:16:09] Alright. Thanks for your insight, Leo, follow up question. Did, did you have a specific use case in mind for that that specific need that that can help to motivate things sometimes? No, not right now. It's just a general question exploring. Yeah. Well, okay. You know, you can sort of hack it together with the stuff that Diane Gross was doing in the early days of chat.[01:16:27] Bt. But then also, like, I, I feel like we could make like a mock validator for plugins such that we are ready to go when it's live. I don't think it'll be too hard. Yeah. Any clones, 20 clones out there for like chat ui, so you can sort to kind of hack it in. Maybe it's like not, not the highest fidelity, but the, the schema is out there, so there's nothing really stopping us apart from, you know, waking up tomorrow and, and seeing that Chad opening, I have done it already.[01:16:54] So , I, I think the only, the only, you could definitely do some of that today. I think some part of the challenge will be that it's a different model that's powering some of these things, which isn't available. Yeah. Yeah. I think that would be, but I still think even with probably base Sahara and just injecting some of this in there you could probably get most of the way there.[01:17:14] Q&A: Python Interpreter[01:17:14] Yeah. By the way, that, that was a misconception that I had to correct a bit early on in the space before you came on. You dropped three models today. Like there was a browsing model and then there's a separate plugins model. And the plugins model doesn't talk to the browsing model. And then there's a, you know, there's.[01:17:28] Python running, which is still going my mind by the way. . Yeah. The Python running also goes back to the piece around, if you wanna basically have things like set things up to dispatch, you can essentially have it write the code and just like plug into any third party library and like set up crime jobs and all that stuff for you.[01:17:47] So going back to sort of having chat b t do your bidding, you could, you could do all that with the code interpreter, which is super cool. And I think Greg tweeted like 20 minutes ago or an hour ago, something like that. An example of it, yes. Like doing video compression and like editing and stuff like that, which was super cool.[01:18:05] That that is the one. Like are we gonna have that or is that Greg's special box? Like No, I think that he's just running straight up interpreter is my understanding. I don't think there's anything special going on there because like that is insane, that like you have storage, you have compute you are a compute platform now.[01:18:22] Like CHATT is not a chat app. It's crazy. Like this is what made me start this space because I was like, wait, like this is not chat. This is a new thing. I don't know what this is. So yeah, I have to drop, but this was, this was awesome. Thanks for hosting this, and thanks for, thanks for having me on again.[01:18:41] Appreciate you. Happy birthday, Dylan. Hopefully this was a, a worthwhile present. , it was great. Thanks for coming on. Yeah, yeah, yeah, yeah. All right. Bye, Logan. Okay. A couple more questions. If anyone has them. These things tend to drag on a little bit, so I always like to end on a well-defined note. Anyone else have reactions, questions, see anything out there that might be interesting?[01:19:01] I did see you know, the, the, the chat partners are starting to tweet out some stuff, so Ane Patel tweeted up about the Milo plugin that they developed with OpenAI, so we can see a little bit of that. Oh, particularly, I haven't particularly like dived in. . But yeah, you know, I, I'm collecting all, all sorts of information and, and reactions.[01:19:18] I'm gonna write out something today because I think this is one of the biggest days again, in tech since, I dunno, Tuesday since last week.[01:19:30] it's hard, but I mean, does anyone agree that things were like, really boring for a while? And this is like the first exciting thing that I've seen. The, the reacts people are still talking about use effects. Like, f**k that. Like ? Yes, exactly. Like we were stuck and reacting like CMS land for like 10 years, just.[01:19:52] Thank God. Thank God. Hey Peter. Hey. Thanks for having me on.[01:19:55] The History of App Stores and Marketplaces[01:19:55] I just wanted to say something real quick to the person that was asking earlier about monetization models and, and plug-ins and touch and I just, I thought one, one thing that occurred listening was that you know, a lot of these, I've done a lot of these plug-in marketplaces over my career and I think there's obviously an opportunity to like, offer different levels of validation and sort of test compatibility kit pass.[01:20:16] And you know, there's also an ongoing component of it cuz there's, you know, potentially data streaming through and, you know, You know, concerns around, you know, the quality of that data does it, you know, circumvent or inter interfere with OpenAI safety systems. So, you know, one obvious way that they could, you know, potentially monetize, you know, any marketplace really, you know, app store, whatever, JetBrains, you know intelligent idea marketplace, right?[01:20:38] Is to have that concept of different levels of validation and, and compliance, you know, to a certain specification. And, you know, you get a little logo or something like that and, you know, so anyway, just a quick thought as I was listening. Fascinating. And thanks for having me on. Hey Peter, since I want you, you to, since you had felt like you have a bunch of experience could you list like the, the, the marketplaces that you've been a part of?[01:20:59] And like, maybe like one thing they did well, one thing they recorded. Sure. I, I'd love to get a top down view. Sure. Yeah. I, I, I don't know that I've seen all of them, but I mean, you know, obviously I'm an iPhone and Android user, so I've, I've seen the marketplace like the rest of us. But JetBrains marketplace I think was particularly good.[01:21:13] Postman has a really good API marketplace rapid. I didn't know that. Rapid ap. Yeah. You know, I think, I think a lot of platform companies have gotten the message and, and they think about marketplaces, obviously the, the hyperscalers, right? You know, you've got the, you know, the, the cloud marketplaces from Amazon.[01:21:28] From Amazon and Google and, and Azure and such. But you know, it's some of the, sometimes it's these smaller ones that are also surprisingly good, like the intelligent idea, you know you know, you go to their website and it's like, you can buy an ad banner if you're in marketing, but, you know. Yeah. Anyway, so this concept of like validated plugins, right?[01:21:44] Especi. when there's this aspect of the data that's flowing through them I think presents an interesting opportunity not only for, for developers to, to make non-st plugins, pardon my frank for you know, for for OpenAI to, to, you know, say, Hey, we looked at this and not just with chat, GPT, no offense[01:22:01] you know, we, we we're giving it a th seal of approval. Right. You know, and that'll, that'll carry weight and carry meeting and people will pay for that is my guess. Yeah. Yeah, yeah. Yeah. Awesome. Well, if I think there's an appetite for like, understanding how to do well in the marketplace right now, if you write a post about that, I think you'll be very well received.[01:22:18] Sweet. Cool. I'll try to find you on Twitter. I, I just kind of dropped in. This was sort of an instinct and then I saw like, NARS here and all these other people here, so it was just kinda like, wow, this is awesome. I know, I know, I know, I know. Well, we're all just like reacting and we need a, we need a space to, to yell because this was huge.[01:22:34] So thanks Peter. No problem. And yeah, let's, let's connect offline.[01:22:37] LindyAI's Flo Crivello Joins Us[01:22:37] Flow is here. I'm trying to invite you, Flo. Because we were talking about Lindy earlier. We're talking about what this, what judge plugins means for Lindy. I don't think it'll, it will, I, I think actually like it will help highlight the differences.[01:22:49] But Oh, you're speaker. Okay. Congrats on your launch, by the way. Very, very, very well done. Thanks. Yeah. One hell of a day. . Hi everyone. Hell of a day. Did you know this was coming by the way? We didn't know it was coming today, but yes, we knew, we knew about this and we knew it would come in the, in the viewing of future.[01:23:05] Yeah. So I'll, I'll intro, I'll reintroduce cuz like the space is like, like four x since the time I talked about. But, you know, AI, virtual assistant is able to arbitrarily respond emails and step meetings and use natural language to do all of that. I think the, the user interface also was very, very well.[01:23:22] Which you know, I, I can't, I can't imagine how long you took to, to do that, but like that is the polish that you need for personal use stuff, right? Like it, this is the, this is the table six. Thank you. I'll, I'll pass your compliments to the designers who hate me now,[01:23:38] it did take a long time to reach this point. I mean, my take is that I think like the button is being passed from the folks like the, the, the, the lab coat researchers working on the models, they're passing the button over to like the, the product teams, basically. And I think we're gonna see a new wave of aed, not just about, Hey, we have a model that is X billion parameters, but we're gonna see a new wave of startups that own a business of building great products around these models.[01:24:07] And with a very simple interface, which is well, sorry, sorry. Yeah. Well, I'll tell you about plugins, but you're talking about over the foundation model APIs. . That's, that's correct. Yeah. Yeah. So I mean, are, are you worried about competition from like, you know, chatt, like, let's, let's talk, let's talk this out, right?[01:24:22] Like what do you see sort of the products gaps that, that PTs have versus whistle? Yeah. My understanding is that chat PT is really like chatt plugin by understanding, so up on the announcement, it's like, it's really more of like a developer product. So OpenAI is remaining true to the DNA of like, you know, we're building models and we're building stuff for, for developers to build product on.[01:24:42] So the impact on companies like Lin is that it's lowering the barrier to entry, which I think you're not targeting developers. Yeah, well, it's not just, it's like, it's become easier to buildy, like a whole lot of stuff that we've built, like over AI just released for free and we're like, well, f**k, like, I guess we build that.[01:25:01] So it's, it's lowering the barrier to entry, but you know, you, you're still left with your expertise. . Yeah, that's true. That's true. Yeah. And also also commenting before you came on that open, I probably will never have Google Calendar on their list of preferred, you know, plugins. They'll never have Gmail on.[01:25:20] And, and your, your integration is already super tight like this, this plugs in exactly to where, what people use today instead of having difficulty Microsoft and Google. Yeah. I wouldn't say never. I think the, but certainly their incentives are not secure aligned. And so I think there is going to be merit in being Switzerland here.[01:25:37] Right? It's like, look, our incentives are aligned with you as the user. Like we're not embed with, with Microsoft or Google or whatever. We're not protecting an existing ecosystem. We're just like, send AI assistant and we are gonna play as well as we can with all of your product. Yeah. Yeah. Does anyone have like I'll open up, you know, obviously we have the founder of Lindy here.[01:25:55] Like, does anyone have questions about Lindy? Did you see the launch? Did you have a follow up? Like this is a very nice place to. Ask it. Unless you wanna , you wanna start? I just wanna get, I'm gonna pay you just wanna get access. yesterday. It would be cool for you to maybe talk a little about how the integrations work.[01:26:16] And I know you're using natural language for it. I think like when tools like it, they think, oh, is my tool gonna be supportive? So yeah, maybe you wanna talk about it. Yeah, definitely. So the, and so I actually tweeted about that separately. Like, the way we build integration is we literally just give the documentation of the API to Lindy and then she out how to use the APIs on her own.[01:26:36] And so it's trivial for us to build a new integration. Like it actually takes 15 minutes to build a new integration. And so the answer to will my product, like, will my thing be supported will be yes. Like in 15 minutes. Like, it'll be like, Hey, you asked us do something. And literally it's like, we couldn't do it yesterday and today we can.[01:26:52] And it's gonna be as simple as that. So, yeah. . Yeah. I, I think to me the most interesting thing is that a lot of companies, I mean, even if you think about Airbyte and Fivetran, like when it comes to connectors, there was like the whole closed source versus open source. Like the open source usually at an advantage because the community can help you build more connectors.[01:27:12] But now using natural language, like the barrier is so much lower and just, it's just super exciting to, to, to use everything right away instead of waiting like four months because I'm the only person using that one tool. So excited to, to . Yeah, 100%. Well, even considering a world in which the user creates their own integration by themselves in like 10 minutes, it's like, hey, like give us, really the only thing we need is like, we need a, a documentation and then we need like an API token.[01:27:39] Like that's the only part that right now requires like an engineer's involvement. But you know, perhaps some power users would be fine generating some developer API token and building their own integration in like 10 minutes. I mean, I, the, the sort of app store model between Google and, and Apple and it's like the bar for quality that they held, you know what I mean?[01:27:57] That, that, I don't know. It's, it's, I don't know. It makes me think of that whole race again and it's like, do you lower the bar for quality and, and go the Android route or do you keep the qual, do you keep the bar high? And especially if, if there's, you know, issues with circumventing or interfering with safety systems and, and data quality and you know, things that are inappropriate, like, I dunno, I wonder, it makes me think, well the thing is that there is a ceiling to quality here when it comes to this integration.[01:28:23] Like, how good can you make a Gmail integration? There's like, there's like 20 endpoint or something, and then the question is like, can you call this endpoint and can you support their parameters? And it's not even the user who would actually like write the endpoint and the parameters. They would literally just like point us to the right API documentation.[01:28:39] Good point. . Yeah, I do think it's a little scary when I give my, you know, if I give like my, my Gmail integration and then you have Brad access, like actually just open source, the GBD four, like email drafter. And I didn't put any auto send or anything like that because I was so scared of it. But I wrote all the code, so it's I trust it.[01:29:02] But it'll be interesting to see how people are gonna trust these systems. Yeah. So we've built some, like hard guardrails in place where certain actions especially any endpoint that is a post endpoint we, we, we flag these actions as like, we call them like a right action. So it's like read action versus right actions.[01:29:18] And if it's a right action, we require user of information in a way that I mean this is like technical details, but like, it, it is physically impossible for the model to actually take a right action without user of inform. So the user, it asks for his information and like the user through the confirmation actually issues a token that is required for the model to be able to call that, that thing.[01:29:39] AI Safety[01:29:39] How worried about you about AI safety is, is this like coming from a place of UX or AI safety? , I'm, I'm super worried about very long term AI safety, right? Yeah. I am, I am, I am moderately worried about like medium term AI safety, like the whole like misinformation thing and like, yeah, like I'm sure there are ways in which Lindy may go wrong, but like, that's not the top of my concerns, and especially because I've built this kind of system.[01:30:04] Like I see the ways in which you can build guardrails and like, this is just like an engineering challenge. Like it's, it's very solvable now. The very long term AI safety thing, like Yeah, I mean there's like an existential race and this is, this is a whole different beast. Yeah. Part, part of me, like trying to do B2B stuff, you know, in the, in the face of AI safety issues, it feels like, you know, you're just kind of rearranging textures on the Titanic.[01:30:25] Or like, you, you know, you're the four piece string quartet playing music to entertain people while the strip is thinking like . Yeah, yeah. It is discouraging a little bit because you, you don't really have a take on the problem, do you? Right. You're like, all right, I guess this is coming. And I, like, I, I, I'm my head and I'm like, I don't really see what I can do about it.[01:30:46] Sam Altman seems to think he can turn it off. Like he has his blue bag, which presume presumably has the off the off button. That that's why he, that's why he always has it with him. Dunno. Yeah. I dunno. Yeah, yeah, yeah. So, . Yeah.[01:31:04] Multimodal GPT4[01:31:04] Well, can I get your reactions just generally on like potential of like maybe multimodal GT four, like just anything that your, your, you know, US builder are looking to really take advantage of as it, as it comes down the line?[01:31:14] Yeah, I think multimodality and you know, audio and, and image especially, I think is like the next big zero to one thing, but otherwise, I think like, just language gets so far, man. So I was just having this conversation. To me it's the same thing as like the cpu, right? Where it's like Fairchild Semiconductor and like Intel, like they gave us the CPU and I think again, like the lab coat researchers passed the button to the hackers and Z garage, like the Steve Jobs and, and, and Steve Snak who now owns the business of building the pc.[01:31:42] And so that doesn't mean that like innovation in the CPU is over, like the CPU still has like four decades of ahead of it. But yeah, like we've got the cpu and now I think that the product and engineering and hacker teams have to, to take it from there. I mean, Intel did pretty well. Totally. Yeah.[01:31:59] I'm not, I'm not saying like OpenAI is going anywhere, for sure. Yeah. Cool, cool, cool. Uh, Any other yeah, does anyone else have questions? No, I see you unmuted.[01:32:07] Designing AI-safe APIs[01:32:07] Yeah. Just upon the on the like safety, AI safety side, I mean, as much as I Sure. Hit the complexity of Im I mean like permissions in AWS and GCP and so on, the server purpose, and I think like maybe in this page, like if you can hit any endpoint on the internet like how do you control which endpoint?[01:32:24] Yeah. So maybe this is, this is like a connection for flow, like one new generation of Im, which is, you know, you have a proxy sitting in front of, in front of the internet and you're only allowed to see certain parts of the internet. You said you have like, you have like right access on the post request already, but yeah, maybe there's something around.[01:32:40] Yeah. So we're looking into this kind of catchall guardrails right now. The way our must, for example, the Gmail API is, so it actually writes code, but at no point does it use a library to make rest API and, and phone calls, right? Like it actually we give it a function that's like Gmail, send email with like primaries for like two and subject and bugging and all of that stuff, right?[01:33:01] And certain of these actions, again, require an authorization token that is specific for like that one action, and these authorization tokens actually expire. So yes, in theory the model could circumnavigate that by writing code to like call the, the API endpoints directly. We've not seen it do that yet.[01:33:17] And, and that's just not the way we train the model to behave. That's pretty response. That's like general platform question for maybe you in the future, maybe OpenAI. That if you hook it up to the, how do you prevent it from, I, I'm not saying that the AI will do something malicious, but like a developer who gets it to write some code and hidden endpoint that you didn't give it permission for.[01:33:40] So for example, you can, in Deno, I I love the permission system in Deno. You can give it access to your file system or the n or you know, like the internet, but like how do you specify only a part of the internet or only a part of a domain or so on?[01:33:56] Yeah, so open by the way, I, I, I'm a little bit bearish on the Deno permissioning because it's permissioning on the whole executable. And and that's, you know, it's basically you're going to try to relax it the moment you run into errors and people just kind of relax it all the way, you know, it's kind of.[01:34:12] True. Very. I was actually I, the way I got around it, I, I was starting a new a new process subprocess and only giving it access. Really? So instead of making Yeah, it was, it was really done. Really annoying. Well done. They should go get it only. Exactly. Yeah. It's kind of overselling the security if like everybody just runs like, you know, pseudo whatever the pseudo is in, in, in Deno.[01:34:34] But yeah. Okay. Cool. Any other reactions?[01:34:36] Flo's Closing Comments[01:34:36] Flo, before I'll give you the, the last word here, just reactions to Chatt, PT and open the eye shipping velocity in general. You're, you're always a good speaker, so leaving to you for soundbites. Soundbites. No, it's great. You know, I, I, I'm excited to see this kind of product, see the light, and I, I, I don't use them as like direct competitors just yet.[01:34:51] And even if they. Look, I think the market, this is going to be the model of our market, so I think it's gonna be, it's gonna be more than fine, but maybe room full. Mini here. Blue ocean. That's right. Time to build. Let's go. What do you think swyx? What do I think? I, I, I, I don't know what to think. That's, this is why I started this space because I saw that CHE BT can run f fm Peg, which means it is a compute platform, right?[01:35:16] Like it generates Python code, it runs the Python code. It can receive files, it can store files, it has memory and then it can let you download the files. Give it some GPUs, and you can run Lama inside of chat, gbc, for whatever reason you want. It is a new compute platform now, and I want to build for it, but I don't know what I, what I can.[01:35:38] Yeah, I, I agree. I think it's, it's, these large models are like the next operating system. I'm, I'm very convinced that that's the way people are gonna interact with the computers. Like, you're no longer gonna do work at your computer, you're gonna have a conversation with your computer and the computers gonna work for you.[01:35:55] Well, you're, you're certainly building the platform for that. So everyone go check out Lindy. I think this is a great conversation. I always want spaces to end on a high note. But thanks for joining in. I know it's like zero notice. I was just DMing you. But thanks for coming on, man. Yeah, thanks everyone.[01:36:09] Yeah, all. Go out there. Bye. Thanks. Get full access to Latent Space at www.latent.space/subscribe

If Text is the Universal Interface, then Text to SQL is perhaps the killer B2B business usecase for Generative AI. You may have seen incredible demos from Perplexity AI, OSS Insights, and CensusGPT where the barrier of learning SQL and schemas goes away and you can intuitively converse with your data in natural language.But in the multi-billion dollar data engineering industry, Seek.ai has emerged as the forerunner in building a conversational engine and knowledge base that truly democratizes data insights. We’re proud to present our first remote interview with Sarah Nagy to learn how AI can help you “seek what matters”!Timestamps* 00:00: Intro to Sarah* 03:40: Seek.ai origin* 05:45: Data driven vs Data backfit* 09:15: How Enterprises adopt AI* 12:55: Patents and IP Law* 14:05: The Semantic Layer* 16:35: Interfaces - Dashboards vs Chat?* 21:05: LLM performance and selection* 26:05: LLMOps and LangChain* 30:55: Lightning roundShow notes* Sarah Nagy Linkedin* Seek.ai* Sarah on the dbt podcastLightning Rounds* Favorite AI Product: Stable Diffusion* Favorite AI Community: Eleuther* One year prediction: Things will move fast!* Request for Startup: Scheduling/Emails (shoutout Ipso.ai from our hackathon!)* Takeaway: Automate everything! Get full access to Latent Space at www.latent.space/subscribe

OpenAI just rollicked the AI world yet again yesterday — while releasing the long awaited ChatGPT API, they also priced it at $2 per million tokens generated, which is 90% cheaper than the text-davinci-003 pricing of the “GPT3.5” family. Their blogpost on how they did it is vague: Through a series of system-wide optimizations, we’ve achieved 90% cost reduction for ChatGPT since December; we’re now passing through those savings to API users.We were fortunate enough to record Episode 2 of our podcast with someone who routinely creates 90%+ improvements for their customers, and in fact have started productizing their own infra skills with Codeium, the rapidly growing free-forever Copilot alternative (see What Building “Copilot for X” Really Takes). Varun Mohan is CEO of Exafunction/Codeium, and he indulged us in diving deep into AI infrastructure, compute-optimal training vs inference tradeoffs, and why he loves suffering.Recorded in-person at the beautiful StudioPod studios in San Francisco.Full transcript is below the fold. Timestamps* 00:00: Intro to Varun and Exafunction* 03:06: GPU Efficiency, Model Flop Utilization, Dynamic Multiplexing* 05:30: Should companies own their ML infrastructure?* 07:00: The two kinds of LLM Applications* 08:30: Codeium* 14:50: “Our growth is 4-5% day over day”* 16:30: Latency, Quality, and Correctability* 20:30: Acceleration mode vs Exploration mode* 22:00: Copilot for X - Harvey AI’s deal with Allen & Overy* 25:00: Scaling Laws (Chinchilla)* 28:45: “The compute-optimal model might not be easy to serve”* 30:00: Smaller models* 32:30: Deepmind Retro can retrieve external infromation* 34:30: Implications for embedding databases* 37:10: LLMOps - Eval, Data Cleaning* 39:45: Testing/User feedback* 41:00: “Users Is All You Need”* 42:45: General Intelligence + Domain Specific Dataset* 43:15: The God Nvidia computer* 46:00: Lightning roundShow notes* Varun Mohan Linkedin* Exafunction* Blogpost: Are GPUs Worth it for ML* Codeium* Copilot statistics* Eleuther’s The Pile and The Stack* What Building “Copilot for X” Really Takes* Copilot for X* Harvey, Copilot for Law - deal with Allen & Overy* Scaling Laws* Training Compute-Optimal Large Language Models - arXiv (Chinchilla paper)* chinchilla's wild implications (LessWrong)* UL2 20B: An Open Source Unified Language Learner (20B)* Paper - Deepmind Retro* “Does it make your beer taste better”* HumanEval benchmark/dataset* Reverse Engineering Copilot internals* Quora Poe* Prasanna Sankar notes on FLOPs and Bandwidth* NVIDIA H100 specs - 3TB/s GPU memory, 900GB/s NVLink Interconnect* Optimizer state is 14x size of model - 175B params => 2.5TB to store state → needs at least 30 H100 machines with 80GB each* Connor Leahy on The Gradient PodcastLightning Rounds* Favorite AI Product: Midjourney* Favorite AI Community: Eleuther and GPT-J* One year prediction: Better models, more creative usecases* Request for Startup: Superathlete Fitness Assistant* Takeaway: Continue to tinker!Transcript[00:00:00] Alessio Fanelli: Hey everyone. Welcome to the Latent Space podcast. This is Alessio, partner and CTO in residence at Decibel Partners. I'm joined by my cohost, swyx, writer, editor of L Space Diaries.[00:00:20] swyx: Hey, and today we have Varun Mohan from Codeium / Exafunction on. I should introduce you a little bit because I like to get the LinkedIn background out of the way.[00:00:30] So you did CS at MIT and then you spent a few years at Nuro where you were ultimately tech lead manager for autonomy. And that's an interesting dive. Self-driving cars in AI and then you went straight into Exafunction with a few of your coworkers and that's where I met some of them and started knowing about Exafunction.[00:00:51] And then from out of nowhere you cloned GitHub Copilot. That's a lot of progress in a very short amount of time. So anyway, welcome .[00:00:59] Varun Mohan: That's high praise.[00:01:00] swyx: What's one thing about you that doesn't appear on LinkedIn that is a big part of what people should know?[00:01:05] Varun Mohan: I actually really like endurance sports actually.[00:01:09] Like I, I've done multiple triathlons. I've actually biked from San Francisco to LA. I like things that are like suffering. I like to suffer while I, while I do sports. Yeah.[00:01:19] swyx: Do you think a lot about like code and tech while you're doing those endurance sports or are you just,[00:01:24] Varun Mohan: your mind is just focused?[00:01:26] I think it's maybe a little bit of both. One of the nice things about, I guess, endurance athletics, It's one of the few things you can do where you're not thinking about, you can't really think about much beyond suffering. Like you're climbing up a hill on a bike and you see like, uh, you see how many more feet you need to climb, and at that point you're just struggling.[00:01:45] That's your only job. Mm-hmm. . Yeah. The only thing you can think of is, uh, pedaling one more pedal. So it's actually like a nice, a nice way to not think about work. Yeah,[00:01:53] Alessio Fanelli: yeah, yeah. Maybe for the audience, you wanna tell a bit about exa function, how that came to be and how coding came out[00:01:59] Varun Mohan: of that. So a little bit about exo function.[00:02:02] Before working at exa function, I worked at Neuro as Sean was just saying, and at neuro, I sort of managed large scale offline deep learning infrastructure. Realized that deep learning infrastructure is really hard to build and really hard to maintain for even the most sophisticated companies, and started exa function to basically solve that gap, to make it so that it was much easier for companies.[00:02:24] To serve deep learning workloads at scale. One of the key issues that we noticed is GPUs are extremely hard to manage fundamentally because they work differently than CPUs. And once a company has heterogeneous hardware requirements, it's hard to make sure that you get the most outta the hardware. It's hard to make sure you can get, get great GPU utilization and exa function was specifically built to make it so that you could get the most outta the hardware.[00:02:50] Make sure. Your GP was effectively virtualized and decoupled from your workload to make it so that you could be confident that you were running at whatever scale you wanted without burning the bank.[00:03:00] swyx: Yeah. You gave me this metric about inefficiency,[00:03:03] Varun Mohan: right? Oh, okay. Like flop efficiency. Yeah. Yeah. So basically, I think it comes down to, for most people, one of the things about CPUs that's really nice is with containers, right?[00:03:13] You can end up having a single. You can place many containers on them and all the containers will slowly start eating the compute. It's not really the same with GPUs. Like let's say you have a single. For the most part, only have one container using that gpu. And because of that, people heavily underestimate what a single container can sort of do.[00:03:33] And the GPU is left like heavily idle. And I guess the common term now with a lot of LM workloads is like the flop efficiency of these workloads. M F U, yeah. Yeah. Model flop utilization. The model flop utilization, which is basically like what fraction of the flops or compute on the hardware is actually getting used.[00:03:49] And sort of what we did at exa function. Not only make it so that the model was always running, we also built compiler technology to make it so that the model was also running more efficiently. And some of these things are with tricks like operator fusion, like basically you could imagine fusing two operations together such that the time it takes to compute.[00:04:07] the fused operation is lower than the time it takes for each individual operation. Oh my God. Yeah. .[00:04:13] Alessio Fanelli: Yeah. And you have this technique called dynamic multiplexing, which is basically, instead of having a one-to-one relationship, you have one GP for multiple clients. And I saw one of your customers, they went from three clients to just one single GPU and the cost by 97%.[00:04:29] What were some of those learning, seeing hardware usage and efficiencies and how that then played into what, what[00:04:34] Varun Mohan: you're building? Yeah, I think it basically showed that there was probably a gap with even very sophisticated teams. Making good use of the hardware is just not an easy problem. I think that was the main I, it's not that these teams were like not good at what they were doing, it's just that they were trying to solve a completely separate problem.[00:04:50] They had a model that was trained in-house and their goal was to just run it and it, that should be an easy. Easy thing to do, but surprisingly still, it's not that easy. And that problem compounds in complexity with the fact that there are more accelerators now in the cloud. There's like TPUs, inferential and there's a lot of decisions, uh, that users need to make even in terms of GPU types.[00:05:10] And I guess sort of what we had was we had internal expertise on what the right way to run the workload was, and we were basically able to build infrastructure and make it so that companies could do that without thinking. So most[00:05:21] Alessio Fanelli: teams. Under utilizing their hardware, how should they think about what to own?[00:05:26] You know, like should they own the appearance architecture? Like should they use Xlo to get it to production? How do you think[00:05:32] Varun Mohan: about it? So I think one thing that has proven to be true over the last year and a half is companies, for the most part, should not be trying to figure out what the optimal ML architecture is or training architecture is.[00:05:45] Especially with a lot of these large language models. We have generic models and transformer architecture that are solving a lot of distinct problems. I'll caveat that with most companies. Some of our customers, which are autonomous vehicle companies, have extremely strict requirements like they need to be able to run a model at very low latency, extremely high precision recall.[00:06:05] You know, GBT three is great, but the Precision Recall, you wouldn't trust someone's life with that, right? So because of that, they need to innovate new kinds of model architectures. For a vast majority of enterprises, they should probably be using something off the shelf, fine tuning Bert models. If it's vision, they should be fine tuning, resonant or using something like clip like the less work they can do, the better.[00:06:25] And I guess that was a key turning point for us, which is like we start to build more and more infrastructure for the architectures that. The most popular and the most popular architecture was the transformer architecture. We had a lot of L L M companies explicitly reach out to us and ask us, wow, our GT three bill is high.[00:06:44] Is there a way to serve G P T three or some open source model much more cheaply? And that's sort of what we viewed as why we were maybe prepared for when we internally needed to deploy transform models our.[00:06:58] Alessio Fanelli: And so the next step was, Hey, we have this amazing infrastructure. We can build kind of consumer facing products, so to speak, at with much better unit economics, much better performance.[00:07:08] And that's how code kind[00:07:10] Varun Mohan: of came to be. Yeah. I think maybe the, the play is not maybe for us to be just, we make a lot of consumer products. We want to make products with like clear ROI in the long term in the enterprise. Like we view code as maybe one of those things. Uh, and maybe we can, we can talk about code maybe after this.[00:07:27] We. Products like co-pilot as being extremely valuable and something that is generating a lot of value to professionals. We saw that there was a gap there where a lot of people probably weren't developing high intensive L L M applications because of cost, because of the inability to train models the way they want to.[00:07:44] And we thought we could do that with our own infrastructure really quickly.[00:07:48] swyx: I wanna highlight when you say high intensive, you mean basically generate models every key, uh, generate inferences on every keystroke? That's[00:07:55] Varun Mohan: right. Yeah. So I would say like, there's probably two kinds of L l M applications here.[00:07:59] There's an L L M application where, you know, it rips through a bunch of data and maybe you wait a couple minutes and then you see something, and then there's an application where the quality is not exactly what you want, but it's able to generate enough, sorry, low enough latency. It's still providing a ton of value.[00:08:16] And I will say there's like a gap there where the number of products that have hit that co-pilot spot is actually not that high. Mm. A lot of them are, are kind of like weight and, you know, just generate a lot of stuff and see what happens because one is clearly more compute intensive than the other Basically.[00:08:31] swyx: Well co uh, I don't know if we told the whole story yet, you were going to[00:08:35] Varun Mohan: dive into it. . Yeah, so I guess, I guess the story was I guess four or five months ago we sort of decided internally as a team we were like very early adopters of co-pilot. I'm not gonna sit here and say co-pilot, it's not a great tool.[00:08:45] We love co-pilot. It's like a fantastic tool. We all got on the beta. The moment it came out we're like a fairly small T, but we, like we all got in, we were showing each other completions. We end up writing like a lot of cuda and c plus plus inside the company. And I think there was probably a thought process within us that was like, Hey, the code we write is like very high aq.[00:09:04] You know? So like there's no way it can help. And one of the things in c plus plus that's like the most annoying is writing templates. Writing template programming is maybe one of those things. No one, maybe there's like some people in the C plus O standards community that can do it without looking at the, looking at anything online.[00:09:19] But we struggle. We struggle writing bariatric templates and COPA just like ripped through. Like we had a 500 line file and it was just like writing templates like, and we didn't really even test it while we were running it. We then just compiled it and it just, We're like, wow. Like this is actually something that's not just like it's completing four loops, it's completing code for us.[00:09:38] That is like hard in our brains to reach, but fundamentally and logically is not that complicated. The only reason why it's complicated is there's just a lot of rules, right. And from then we were just like, wow, this is, that was maybe the first l l m application for us internally, because we're not like marketers that would use, uh, Jasper, where we were like, wow, this is like extremely valuable.[00:09:58] This is not a toy anymore. So we wanted to take our technology to build maybe apps where these apps were not gonna be toys, right? They were not gonna be like a demo where you post it on Twitter and then you know there's hype and then maybe like a month later, no one's using.[00:10:11] swyx: There's a report this morning, um, from co-pilot where they, they were estimating the key tabs on amount of code generated by a co-pilot that is then left in code repos and checked in, and it's something like 60 to 70%[00:10:24] Varun Mohan: That's, that's nuts, but I totally believe it given, given the stats we have too. There's this flips in your head once you start using products like this, where in the beginning there's like, there's like skepticism, like how, how valuable can it be? And suddenly now like user behavior fundamentally changes so that now when I need to write a function, I'm like documenting my code more because I think it's prompting the model better, right?[00:10:43] So there's like this crazy thing where it's a self-fulfilling prophecy where when you get more value from it, more of your code is generated. From co-pilot[00:10:50] swyx: just to walk through the creation process, I actually assumed that you would have grabbed your data from the pile, which is the Luther ai, uh, open source, uh, code information.[00:11:00] But apparently you scraped your own[00:11:01] Varun Mohan: stuff. Yeah. We ended up basically using a lot of open, I guess, permissively licensed code, uh, in the public internet, mainly because I think also the pile is, is fairly a small subset. Uh, I think maybe after we started there was the, that was also came to be, but for us, we had a model for ourselves even before that, uh, was the point.[00:11:21] Ah, okay. So the timing was just a little bit off. Yeah, exactly. Exactly. But it's awesome work. It's, it seems like there's a good amount of work that's getting done Decentrally. Yeah. Which is a little bit surprising to me because I'm like more bullish on everyone needs to get together in a room and make stuff happen.[00:11:35] Like we're all in person in Mountain View. But yeah, no, it's pretty impressive. Yeah. Luther in general, like everything they've done, I'm pretty impressed with it. Yeah, and we're[00:11:42] swyx: gonna talk about that. Cause I, I didn't know you were that involved in the community[00:11:45] Varun Mohan: that early on I wasn't involved. It was more of like a, I was watching and maybe commenting from time to time.[00:11:50] So they're a very special community for sure. Yeah,[00:11:52] swyx: yeah, yeah. That's true. That's true. My impression is a bunch of you are geniuses. You sit down together in a room and you. , get all your data, you train your model, like everything's very smooth sailing. Um, what's wrong with that[00:12:02] Varun Mohan: image? Yeah, so probably a lot of it just in that a lot of our serving infrastructure was already in place, Uhhuh before then.[00:12:09] So like, hey, we were able to knock off one of these boxes that I think a lot of other people maybe struggle with. The open source serving offerings are just, I will say, not great in that. That they aren't customized to transformers and these kind of workloads where I have high latency and I wanna like batch requests, and I wanna batch requests while keeping latency low.[00:12:29] Mm-hmm. , right? One of the weird things about generation models is they're like auto regressive, at least for the time being. They're auto aggressive. So the latency for a generation is a function of the amount of tokens that you actually end up generating. Like that's like the math. And you could imagine while you're generating the tokens though, unless you batch a.[00:12:46] It's gonna end up being the case that you're not gonna get great flop utilization on the hardware. So there's like a bunch of trade offs here where if you end up using something completely off the shelf, like one of these serving thing, uh, serving frameworks, you're gonna end up leaving a lot of performance on the table.[00:13:00] But for us, we were already kind of prepared. To sort of do that because of our infrastructure that we had already built up. And probably the other thing to sort of note is early on we were able to leverage open source models, sort of bootstrap it internally within our company, but then to ship, we finally had some requirements like, Hey, we want this model to have fill in the middle capabilities and a bunch of other things.[00:13:20] And we were able to ship a model ourselves. So we were able to time it so that over the course of multiple months, different pieces were like working out properly for us. So it wasn't. . You know, we started out and we were just planning the launch materials. The moment we started there was like maybe some stuff that was already there, some stuff that we had already figured out how to train models at scale internally.[00:13:38] So we were able to just leverage that muscle very quickly. I think the one[00:13:41] swyx: thing that you had figured out from the beginning was that it was gonna be free forever. Yeah. Yeah, co-pilot costs $10[00:13:47] Varun Mohan: a month. Co-pilot costs $10 a month. I would argue significantly more value than $10 a month. The important thing for us though, was we are gonna continue to build more great products on top of code completion.[00:13:58] We think code completion is maybe day one of what the future looks like. And for that, clearly we can't be a product that's like we're $10 a month and we're adding more products. We want a user base that loves using us. And we'll continue to stay with us as we continue to layer on more products. And I'm sure we're gonna get more users from the other products that we have, but we needed some sort of a differentiator.[00:14:17] And along the way we realized, hey, we're pretty efficient at running these workloads. We could probably do this. Oh, so it wasn't,[00:14:23] swyx: it was a plan to be free from the start. You just[00:14:25] Varun Mohan: realized we, yeah. We realized we could probably, if we cut and optimized heavily, we could probably do this properly. Part of the reasoning here was we were confident we could probably build a pro tier and go to the enter.[00:14:35] But for now, originally when we, when we started, we weren't like, we're just gonna go and give every, all pieces of software away for free. That wasn't like sort of the goal there. And[00:14:43] swyx: since you mentioned, uh, adoption and, you know, traction and all that, uh, what can you disclose about user growth? Yeah, user adoption.[00:14:50] Varun Mohan: Yeah. So right now we have. We probably have over 10,000 users and thousands of daily actives, and people come back day over day. Our growth is like around, you know, four to 5% day over day right now. So all of our growth right now is sort of like word of mouth, and that's fundamentally because like the product is actually one of those products where.[00:15:08] Even use COT and use us, it's, it's hard to tell the difference actually. And a lot of our users have actually churned off of cot isn't Yeah. I,[00:15:14] swyx: I swept Yeah. Yeah. To support you guys, but also also to try[00:15:17] Varun Mohan: it out. Yeah, exactly. So the, the crazy thing is it wasn't like, Hey, we're gonna figure out a marketing motion of like, Going to the people that have never heard of co-pilot and we're gonna like get a bunch of users.[00:15:27] We wanted to just get users so that in our own right we're like a really great product. Uh, and sort of we've spent a lot of engineering time and obviously we co-wrote a blog post with you, Sean, on this in terms of like, there's a lot of engineering work, even beyond the latency, making sure that you can get your cost down to make a product like this actually work.[00:15:44] swyx: Yeah. That's a long tail of, of stuff that you referenced,[00:15:47] Varun Mohan: right? Yes. Yeah, exactly.[00:15:48] swyx: And you, you said something to the order of, um, and this maybe gets into co-pilot for X uh, which is something that everybody is keen about cuz they, they see the success of co-pilot. They're like, okay, well first of all, developer tools, there's more to do here.[00:16:00] And second of all, let's say the co-pilot idea and apply for other disciplines. I don't know if you wanna Yeah.[00:16:06] Varun Mohan: There's[00:16:06] Alessio Fanelli: gonna some. Key points that, that you touched on. Um, how to estimate, inference a scale, you know, and the latency versus quality trade-offs. Building on first party. So this is free forever because you run your own models, right?[00:16:19] That's right. If you were building on open ai, you wouldn't be able to offer it for free real-time. You know, when I first use coding, It was literally the same speed as Copi is a little bit[00:16:29] swyx: faster. I don't know how to quantify it,[00:16:31] Varun Mohan: but we are faster. But it's one of those things that we're not gonna like market as that's the reason because it's not in and of itself a right for you to like, I'm just gonna be open with you.[00:16:39] It's not a reason for you to like suddenly turn off a copilot where if our answers were trash, uh, but we were faster. You know what I mean? But your focus[00:16:46] Alessio Fanelli: was there. We used the alpha, I think prem on our discord came to us and say, you guys should try this out. So it was really fast. Even then, prompt optimization is another big thing, and model outputs and UX kind of how you bring them together.[00:17:00] Which ones of these things are maybe like the one or two that new founders should really think about first?[00:17:07] Varun Mohan: Yeah, I think, I think my feeling on this is unless you are ex, you probably should always bootstrap on top of an existing a. Because like even if you were to, the only reason why we didn't is because we knew that this product was actually buildable.[00:17:22] Probably if we worked hard enough to train a model, we would actually be able to build a great product already. But if you're actually going out and trying to build something from scratch, unless you genuinely believe, I need to fine tune on top of, you know, terabytes of data terabyte is a very large amount of data, but like tens of gigabytes of data.[00:17:37] Probably go out and build on top of an API and spend most of your time to make it so that you can hit that quality latency trade off properly. And if I were to go out and think about like the three categories of like an LM product, it's probably like latency, quality, and correct ability. The reality is, you know, if I were to take a product like co-pilot or Coum, the latency is very low.[00:17:58] The quality I think, is good enough for the task, but the correct ability is, is very easy. Credibility. What, what is correct ability? Correct ability means, let's say the quality is not there. Like you consider the the case where, The answer is wrong. How easy is it for your user to actually go and leverage parts of the generation?[00:18:16] Maybe a, a concrete example. There's a lot of things people are excited about right now where I write a comment and it generates a PR for me, and that's like, that's like really awesome in theory. I think that's like a really cool thing and I'm sure at some point we will be able to get there. That will probably require an entirely new model for what it's worth that's trained on diffs and commits and all these other things that looks at like improvements and code and stuff.[00:18:37] It's probably not gonna be just trained on generic code. But the problem with those, those sort of, I would say, applications are that, let's suppose something does change many files, makes large amounts of changes. First of all, it's guaranteed not gonna be. Because even the idea of like reviewing the change takes a long time.[00:18:54] So if the quality and the correct ability is just not there, let's say you had 10 file, a 10 file change and you modified like, you know, file two and four, and those two modifications were consistent, but the other eight files were not consistent. Then suddenly the correct ability is like really hard.[00:19:10] It's hard to correct the output of the model. And so the user interface is 100% really important. But maybe until you get the latency down or the correct ability, like correct ability, like a lot better, it's probably not gonna be shippable. And I think that's what you gotta spend your time focusing on.[00:19:26] Can you deliver a product that is actually something users want to use? And I think this is why I was talking about like demo. It's like very easy to hand to handpick something that like works, that works for a demo, exceedingly hard for something that has large scope, like a PR to work consistently. It will take a lot of engineering effort to make it work on small enough chunks so that a user is like, wow, this is value generative to me.[00:19:49] Because eroding user trust or consumer trust is very easy. Like that is, it is is much, much, it's very easy to erode user trust versus enterprise. So just be mindful of that, and I think that's probably like the mantra that most of these companies need to operate under. Have you done any[00:20:05] Alessio Fanelli: analysis on. What the ratio between code generated and latency is.[00:20:11] So you can generate one line, but you could also generate the whole block. You can generate Yeah. A whole class and Yeah. You know, the more you generate the, the more time it takes. Like what's the sweet spot that, that you[00:20:21] Varun Mohan: found? Yeah, so I think there was a great study and, and I'm not sure if it's possible to link it, but there was a great study about co-pilot actually that came out.[00:20:28] Basically what they said was there were two ways that developers usually develop with a code assistant technology. They're either in what's called like acceleration mode or exploration mode. And exploration mode is basically you're in the case where you don't even know what the solution space for the function is.[00:20:43] and you just wanna generate a lot of code because you don't even know what that looks like. Like it might use some API that you've never heard of. And what you're actually doing at that point is like you're writing a clean comment, just wishing and praying that you know, the generation is long enough and gets you, gets you far enough, right?[00:20:57] acceleration mode is basically you are doing things where you are very confident in what you're doing and effectively. Code gives you that muscle so that you can basically stay in flow state and you're not thinking about like exactly what the APIs look like, but push comes to shove. You will figure out what the APIs look like, but actually like mentally, it takes off like a load in your head where you're like, oh wow.[00:21:18] Like I can just do this. The intent to execution is just a lot, a lot lower there. And I think effectively you want a tool that captures that a little bit. And we have heuristics in terms of captur. Whether or not you're in acceleration versus exploration mode. And a good heuristic is, let's say you're inside like a basic block of a piece of code.[00:21:37] Let's say you're inside a a block of code or an IF statement, you're probably already in acceleration mode and you would feel really bad if I started generating the ELs clause. Because what happens if that else causes really wrong? That's gonna cause like mental load for you because you are the way programmers think.[00:21:51] They only want to complete the if statement first, if that makes sense. So there are things where we are mindful of like how many lines we generate if you use the product, like multi-line generations happen and we are happy to do them, but we don't want to do them when we think it's gonna increase load on developers, if that makes sense.[00:22:07] That[00:22:07] Alessio Fanelli: makes sense. So co-pilot for x. , what are access that you think are interesting for people to build[00:22:13] Varun Mohan: in? Didn't we see some, some tweet recently about Harvey ai, uh, company that, that is trying to sell legal? It's like a legal, legal assistance. That's, that's pretty impressive, honestly. That's very impressive.[00:22:23] So it seems like I would really love to see what the product looks like there, because there's a lot of text there. You know, looking at bing, bing, ai, like, I mean, it's, it's pretty cool. But it seems like groundedness is something a lot of these products struggle with, and I assume legal, if there's one thing you want them to.[00:22:39] To get right. It's like the groundedness. Yeah.[00:22:42] swyx: Yeah. I've made the analogy before that law and legal language is basically just another form of programming language. You have to be that precise. Yes. Definitions must be made, and you can scroll to find the definition. It's the same thing. Yes. ,[00:22:55] Varun Mohan: yes. Yeah. But like, I guess there's a question of like comprehensiveness.[00:22:59] So like, let's say, let's say the only way it generates a suggestion is it provides like, you know, citations to other legal. You don't want it to be the case that it misses things, so you somehow need the comprehensiveness, but also at the same time, you also don't want it to make conclusions that are not from the site, the things at sites.[00:23:15] So, I don't know, like that's, that's very impressive. It's clear that they've demonstrated some amount of value because they've been able to close a fairly sizable enterprise contract. It was like a firm with 3,500 lawyers, something nuts, honestly. Very cool. So it's clear this is gonna happen, uh, and I think people are gonna need to be clever about how they actually make it work.[00:23:34] Within the constraints of whatever workload they're operating in. Also, you, you guys[00:23:37] swyx: are so good at trading stuff, why don't you, you try[00:23:39] Varun Mohan: cloning it. Yeah. So I think, I think that's, that's, uh, preview the roadmap. Yeah, yeah, yeah, yeah. No, no, no, but I'm just kidding. I think one of the things that we genuinely believe as a startup is most startups can't really even do one thing properly.[00:23:52] Mm-hmm. Focus. Yeah. Yeah. Usually doing one thing is really hard. Most companies that go public have like maybe a couple big products. They don't really have like 10, so we're under no illusions. Give the best product experience, the amount of engineering and attention to detail, to build one good product as hard.[00:24:08] So it's probably gonna be a while before we even consider leaving code. Like that's gonna be a big step because the amount of learning we need to do is gonna be high. We need to get users right. We've learned so much from our users already, so, yeah, I don't think we'd go into law anytime soon.[00:24:22] swyx: 3,500 lawyers with Ellen and Ry, uh, is, is is apparently the, the new[00:24:27] Varun Mohan: That's actually really big.[00:24:28] Yeah. Yeah. I can congrat.[00:24:29] swyx: Yeah, it's funny cuz like, it seems like these guys are moving faster than co-pilot. You know, co-pilot just launched, just announced enterprise, uh, like co-pilot for teams or co-pilot for Enterprise. Yeah. After like two years of testing.[00:24:40] Varun Mohan: Yeah, it does seem like the co-pilot team has built a very, very good product.[00:24:44] Um, so I don't wanna like say anything, but I think it is the case to startups will be able to move faster. I feel like that is true, but hey, like GitHub has great distribution. Whatever product they do have, they will be able to sell it really. Shall[00:24:56] swyx: we go into model numbers and infra estimates? our favorite[00:25:01] Varun Mohan: topics.[00:25:02] Nice small models. Nice.[00:25:04] swyx: So this is, um, relevant to basically I'm researching a lot of skilling law stuff. You have a lot of thoughts. You, you host paper discussions[00:25:12] Varun Mohan: in your team. Yeah, we, we try to like read papers that we think are really interesting and relevant to us. Recently that's been, there's just a fire hose of papers.[00:25:21] You know, someone even just curating what papers we should read internally as a company. Yeah, I think, I think there's, there's so much good content[00:25:28] swyx: out there. You should, you guys should have a podcast. I mean, I told you this before. Should have a podcast. Just, just put a mic near where, where you guys are[00:25:33] Varun Mohan: talking.[00:25:34] We gotta, we gotta keep developing coding though, . No, but you're doing this discussion[00:25:38] swyx: anyway. You[00:25:38] Varun Mohan: might as well just, oh, put the discussion on a podcast. I feel like some of the, some of the thoughts are raw, right? Like, they're not gonna be as, as nuanced. Like we'll just say something completely stupid during our discussions.[00:25:48] I don't know, , maybe that's exciting. Maybe that's, it's kinda like a justin.tv, but for ML papers, Okay, cool. I watched that.[00:25:55] swyx: Okay, so co-pilot is 12 billion parameters. Salesforce cogen is up to 16. G P t three is 175. GP four is gonna be 100 trillion billion. Yeah. So what, what we landed on with you is with, uh, with Cilla, is that we now have an idea of what compute optimal data scaling is.[00:26:14] Yeah. Which is about 20 times parameters. Is that intuitive to you? Like what, what did that[00:26:18] Varun Mohan: unlock? I think basically what this shows is that bigger models are like more data efficient, like given the same number of tokens, a big model like trained on the same number of tokens. A bigger model is like, is gonna learn more basically.[00:26:32] But also at the same time, the way you have to look at it is there are more flops to train a bigger model on the same number of tokens. So like let's say I had a 10 billion parameter model and I trained it on on 1 million tokens, but then I had a 20 billion parameter model at the end of it will be a better.[00:26:47] It will have better perplexity numbers, which means like the probability of like a prediction is gonna be better for like the next token is gonna be better. But at the end of it, you did burn twice the amount of compute on it. Right? So Shinto is an interesting observation, which says if you have a fixed compute budget, And you want the best model that came out of it because there's like a difference here where a model that is, that is smaller, trained on the same number of tokens as fewer flops.[00:27:12] There's a a sweet spot of like number of tokens and size a model. I will say like people probably like. Are talking about it more than they should, and, and I'll, I'll explain why, but it's a useful result, which is like, let's say I have, you know, some compute budget and I want the best model. It tells you what that, what you should generate.[00:27:31] The problem I think here is there is a real trade off of like, you do need to run this model somewhere. You need to run it on a piece of hardware. So then it comes down to how much memory does that piece of hardware have. Let's say for a fixed compute budget, you could train a 70 billion parameter. What are you gonna put that on?[00:27:47] Yeah, maybe you could, could you put that on an 80 gig, A 100? It would be a stretch. You could do things like f, you know, in eight F p a, to reduce the amount of memory that's on the box and do all these other things. But you have to think about that first, right? When you want to go out and train that model.[00:27:59] The worst case is you ended up training that mo, that model, and you cannot serve it. So actually what you end up finding is for a lot of these code completion models, they are actually what you would consider over-trained . So by that I mean like, let's look at a model like Cogen. It's actually trained on, I believe, and, and I could be wrong by, you know, a hundred billion here or there.[00:28:18] I got some data. Oh, okay. Let's look at the 3 billion parameter model. It's a 2.7. I think it's actually a 2.7 billion barometer model. It's weird because they also trained on natural language on top of code, but it's trained on hundreds of billions of tokens. If you applied that chinchilla, Optimization to it, you'd be like, wow, this is, this is a stupid use of compute.[00:28:36] Right? Because three, they should be going to 60, any anything more than 60. And they're like, they should have just increased the model size. But the reality is if they had like the compute optimal one might not be one that's easy to serve, right? It could just have more parameters. And for our case, our models that we train internally, they might not be the most compute.[00:28:56] In other words, we probably could have had a better model by making it larger, but the trade off would've been latency. We know what the impact of having higher latency is, and on top of that, being able to fit properly on our hardware constraints would've also been a concern.[00:29:08] swyx: Isn't the classic stopping point when you, you see like loss kind of levels off.[00:29:12] Right now you're just letting chinchilla tell you,[00:29:16] Varun Mohan: but like you should just look at loss. The problem is the loss will like continue to go down. It'll just continue to go down like, like in a, in a way that's like not that pleasing. It's gonna take longer and longer. It's gonna be painful, but it's like one of those things where if you look at the perplexity number of difference between.[00:29:31] Let's say a model that's like 70 billion versus 10 billion. It's not massive. It's not like tens of percentage points. It's like very small, right? Mm. The reality is here, like, I mean this comes down to like IQ of like these models in some sense, like small wins at the margins are massive wins in terms of iq.[00:29:47] Like it's harder to get those and they don't look as big, but they're like massive wins in terms of reasoning. They can now do chain of thought, all these other things. Yeah, yeah, yeah.[00:29:55] swyx: It's, and, and so apparently unlocked around the[00:29:57] Varun Mohan: 20 billion. Yes. That's right. Some kind of magic. Yeah. I think that was from the UL two or maybe one of those land papers.[00:30:03] Any thoughts on why? Like is there is? I don't know. I mean, emergence of intelligence, I think. I think maybe one of the things is like we don't even know, maybe like five years from now of what we're gonna be running are transformers. But I think it's like, we don't, we don't 100% know that that's true. I mean, there's like a lot of maybe issues with the current version of the transformers, which is like the way attention works, the attention layers work, the amount of computers quadratic in the context sense, because you're like doing like an n squared operation on the attention blocks basically.[00:30:30] And obviously, you know, one of the things that everyone wants right now is infinite context. They wanna shove as much prop as possible in here. And the current version of what a transformer looks like is maybe not ideal. You might just end up burning a lot of flops on this when there are probably more efficient ways of doing it.[00:30:45] So I'm, I'm sure in the future there's gonna be tweaks to this. Yeah. Uh, but it is interesting that we found out interesting things of like, hey, bigger is pretty much always better. There are probably ways of making smaller models significantly better through better data. That is like definitely true. Um, And I think one of the cool things that the stack showed actually was they did a, like a, I think they did some ablation studies where they were like, Hey, what happens if we do, if we do decontamination of our data, what happens if we do de-duplication?[00:31:14] What happens if we do near dup of our data and how does the model get better? And they have like some compelling results that showcase data quality really matters here, but ultimately, Yeah, I think it is an interesting result that at 20 billion there's something happening. But I also think like some of these things in the future may look materially different than what they look like right now.[00:31:30] Hmm. Do you think[00:31:31] Alessio Fanelli: the token limitation is actually a real architectural limitation? Like if you think about the tokens need as kind of like atic, right? Like once you have. 50,000 tokens context, like 50,000 or infinite. For most use cases, it's like the same. Where do you think that number is, especially as you think about code, like some people have very large code bases, there's a lot.[00:31:53] Have you done any work there to figure out where the sweet[00:31:55] Varun Mohan: spot is? Yeah, look, I think what's gonna really end up happening is if people come up with a clever way and, and it, there was some result research that I believe came out of Stanford. I think the team from the Helm group, I think came out with some architecture that looks a little bit different than Transformers, and I'm sure something like this will work in the future.[00:32:13] What I think is always gonna happen is if you find a cheap way to embed context, people are gonna figure out a way to, to put as much as possible in because L LM so far have been like virtually stateless. So the only thing that they have beyond fine tuning is like just shoveling everything you can inside.[00:32:28] And there are some interesting papers, like retro, actually there are maybe some interesting pieces of thought like ideas that have come out recently. Yeah, let's go through them. So one of the really interesting ideas, I think is retro. It's this paper that came out of DeepMind and the idea is actually, let's say you send out, you send out, uh, a prompt.[00:32:44] Okay? Send out a prompt. You compute the burt embedding of that. And then you have this massive embedding database. And by massive, I'm not talking about like gigabytes, I'm talking about terabytes. Like you have, geez, you actually have 10 times the number of tokens as what was used to train the model. So like, let's say you had a model that was trained on a trillion tokens, you have a 10 trillion embed, uh, like embedding database.[00:33:04] And obviously Google has this because they have all content that ever existed in humanity and they have like the best data set and sort of, they were able to make one of these, uh, embedding databases. But the idea here, which is really cool, is you end. Taking your prompt, computing, the bird, embedding you find out the things that were nearby.[00:33:20] So you do roughly like a semantic search or an embedding search within that. And then you take those, you take the documents that were from those embeddings and you shove those in the model too, in what are called like cross chunked attention. So you like shove them in the model with it as well.[00:33:34] Suddenly now the model is able to take in external. Which is really exciting actually, because suddenly now you're able to get dynamic context in, and the model in some sense is deciding what that context is. It's not deciding it completely. In this case, because the Bert model in this case was actually frozen.[00:33:50] It wasn't trained with the retro model as well, but. The idea is you're somehow adding or augmenting context, which I think is like quite exciting. There's probably two futures. Either context becomes really cheap. Right now it's quadratic. Maybe there's a future where it becomes linear in the, in the size of the context, but the future might actually be the model itself dictates, Hey, I have this context.[00:34:10] You have this data source. Give me this. The model itself is going out into your database and like being like, I want this information, and this is kind of like. What Bing search is looking like. Right? Or bing chat is sort of looking like where it's like I, the model is probably, there's probably some model that's saying I want this information.[00:34:27] And that is getting augmented into the context. Now the model itself knows what context it sort of has and it can sort of like build a state machine of sort of what it needs. And that's probably what the future of this looks like. So you, you[00:34:37] swyx: predict monster embedding database[00:34:39] Varun Mohan: companies? Probably Monster embedding database companies or, yeah.[00:34:43] The model in some sense will need to talk to, Talk to these embedding databases. I'm actually not convinced that the current breed of embedding database companies are like ready for what the future sort of looks like. I think I'm just looking at their pricing, how much it costs per gigabyte and it's prohibitive at the scale we're talking about, like let's say you actually did want to host a 10 terabyte embedding database.[00:35:03] A lot of them were created, let's say two years ago, two, three years ago, where people were like, you know, embedding databases are small and they need to make the cost economics work. But maybe, yeah, there's probably gonna be a big workload there. I will just say for us, we will probably just build this in-house to start with, and that's because I think the technology probably isn't there.[00:35:20] And I think that the technology isn't there yet. Like waiting on point solutions to come up is a lot harder, um, than probably building it up. The way I, I like to think about this is probably the world looks on the LM space. Looks like how the early internet days were, where I think the value was accrued to probably like Google and Google needed to figure out all the crazy things to make their workload work.[00:35:41] And the reason why they weren't able to outsource is, is no one else was feeling the pain. ,[00:35:46] swyx: they're just solving their own pain points. They're just solving their own pain points. They're so far ahead of everyone else. Yes, yes. And just wait[00:35:50] Varun Mohan: for people to catch up. Yes. Yes. And that's maybe different than how things like Snowflake look where the interface has been decided for what SQL looks like 50 years ago.[00:35:58] And because of that, you can go out and build the best database and Yeah, like everyone's gonna be like, this doesn't make my beer taste better. And buy your database basically. That's[00:36:08] swyx: a great reference, by the way. Yeah. We have some friends of the, the pod that are working on embedding database, so we'll try to connect you Toroma[00:36:14] Varun Mohan: and see.[00:36:14] Yeah. Oh, I actually know Anton. I worked with him at Neuro. Oh. Although, there you go. Yeah. Uh, what do you, well, what do you think about, I mean,[00:36:20] swyx: so chromas pivoting towards an embedding[00:36:22] Varun Mohan: database. I think it's an interesting idea. I think it's an interesting idea. I wonder what the early set of workloads that.[00:36:27] They will hit our, and you know what the scaling requirements are. This is maybe the classic thing where like, the teams are great, but you need to pick a workload here that you care about the most. You could build anything. You could build anything. When you're an infrastructure company, you can go in, if I was selling, serving in for, I could build, serving for like linear aggression.[00:36:44] I could build this, but like, unless you hit the right niche for the end user, it's gonna be. . So I think it, I'm excited to see what comes out and if they're great, then we'll use it. Yeah.[00:36:54] swyx: I also like how you slowly equated yourself to Google there. Oh, we're not, we're not Google. You're, you're gonna be the Google of ai.[00:37:00] Varun Mohan: We're definitely, we're definitely not Google. But I was just saying in terms of like, if you look at like the style of companies that came out. Yeah. You know? Absolutely. Or maybe we should live in the cutting edge in[00:37:08] swyx: the future. Yeah. I think that's the pitch.[00:37:10] Varun Mohan: Okay, thanks for b***h us.[00:37:13] Alessio Fanelli: So you just mentioned the older vector embedding source are kind of not made for the L l M generation of compute size.[00:37:21] what does l LM ops look like? You know, which pieces need to be drastically different? Which ones can we recycle?[00:37:27] Varun Mohan: Yeah. One of the things that we've found, like in our own thing of building code that's been just shows how much is missing, and this is the thing where like, I don't know how much of this you can really outsource, which is like we needed to build eval infrastructure.[00:37:40] That means how do you build a great code? And there are things online like human eval, right? And uh, I was telling, which is the benchmark telling Sean about this, the idea of human eval is really neat for code. The idea is you provide a bunch of functions with Docstrings and the eval instead of being, did you predict next token?[00:37:56] It's like, did you generate the entire function and does the function run correctly against a bunch of unit tests? Right. And we've built more sophisticated evals to work on many languages, to work on more variety of code bases. One of the issues that ends up coming up with things like human eval is contam.[00:38:12] Because a lot of these, uh, things that train models end up training on all of GitHub GitHub itself has human eva, so they end up training on that. And then the numbers are tiny, though. It's gonna be tiny, right? But it doesn't matter if it's tiny because it'll just remember it. It'll remember that it's, it's not that it's that precise, but it will, it's like, it's basically like mixing your, your training and validation set.[00:38:32] It's like, oh, yeah, yeah, yeah, yeah. But we've seen cases where like online where someone is like, we have a code model that's like, they we're like, we did this one thing, and HU and human eval jumped a ton and we were just like, huh, did human eval get into your data set? Is that really what happened there?[00:38:46] But we've needed to build all this eval. And what is shown is data cleaning is massive, but data cleaning looks different by. Like code data cleaning is different than what is a high quality piece of code is probably different than what's a high quality legal document. Yeah. And then on top of that, how do you eval this?[00:39:01] How do you also train it at scale at whatever cost you really want to get? But those are things that the end user is either gonna need to solve or someone else is gonna need to solve for them. And I guess maybe one of the things I'm a little bearish on is if another company comes out and solves eval properly for a bunch of different verticals, what was the company that they were selling to really?[00:39:21] What were they really doing at that point? If they themselves were not eval for their own workload and all these other things? I think there are cases where, let's say for code where we probably couldn't outsource our eval, like we wouldn't be able to ship models internally if we didn't know how to eval, but it's clear that there's a lot of different things that people need to take.[00:39:38] Like, Hey, maybe there's an embedding piece. How large is this embedding database actually need to be? But hey, this does look very different than what classic ML ops probably did. Mm-hmm. . How[00:39:47] Alessio Fanelli: do you compare some of these models? Like when you're thinking about model upgrading and making changes, like what does the testing piece of it internally?[00:39:56] Yeah. For us look like.[00:39:56] Varun Mohan: For us, it's like old school AB testing. We've built like infrastructure to be able to say, ramp up users from one to 10 to. 50% and slowly roll things out. This is all classic software, uh, which[00:40:09] swyx: you do in-house. You don't, you don't buy any[00:40:10] Varun Mohan: services. We don't buy services for that.[00:40:13] There are good services, open source services that help you just don't need them. Uh, yeah, I think that's just like not the most complicated thing for us. Sure. Basically. Yeah. Uh, but I think in the future, maybe, we'll, obviously we use things like Google Analytics and all this other stuff, but Yeah. For things of ramping our models, finding out if they're actually better because the eval also doesn't tell the whole story because also for us, Even before generating the prompt, we do a lot of work.[00:40:36] And the only way to know that it's really good across all the languages that our users need to tell us that it's actually good. And, and they tell us by accepting completions. So, so GitHub[00:40:44] swyx: co-pilot, uh, the extension does this thing where they, they like, they'll set a timer and then within like five minutes, 10 minutes, 20 minutes, they'll check in to see if the code is still there.[00:40:54] I thought it was a[00:40:54] Varun Mohan: pretty creative way. It's, it's a very, it's honestly a very creative way. We do do things to see, like in the long term, if people did. Accept or write things that are roughly so because they could accept and then change their minds. They could accept and then change their minds. So we, we are mindful of, of things like that.[00:41:09] But for the most part, the most important metric is at the time, did they actually, did we generate value? And we want to know if that's true. And it's, it's kind of, it's honestly really hard to get signal unless you have like a non-trivial amount of usage, non-trivial, meaning you're getting, you're doing hundreds of thousands of completions, if not millions of completions.[00:41:25] That sounds like, oh wow. Like, that's like a very small amount. But like it's classic. Maybe like if you look at like when I used to be an intern at Quora, like, you know, now more than seven, eight years ago. When I was there, I like shipped a change and then Cora had like millions of daily actives and then it looked like it was good, and then a week later it was just like way worse.[00:41:43] And how is this possible? Like in a given hour we get like hundreds of thousands of interaction, just like, no, you just need way more data. So this is like one of those things where I think having users is like genuinely very valuable to us, basically. Users is all you need. . Yeah.[00:41:59] swyx: Um, by the way, since you brought out Quora, have you tried po any, any thoughts[00:42:03] Varun Mohan: on po I have not actually tried po I've not actually tried.[00:42:05] I[00:42:05] swyx: mean, it seems like a question answering website that's been around for 20 years or something. Would be very, would be very good at question answering. Yeah.[00:42:12] Varun Mohan: Also Adam, the ceo, is like incredibly brilliant. That guy is like insanely smart, so I'm sure they're gonna do,[00:42:18] swyx: they have accidentally built the perfect like data collection company for For qa.[00:42:22] Varun Mohan: Yeah. . It takes a certain kind of person to go and like cannibalize your original company like the in, I mean, it was kinda stagnant for like a few years. Yeah, that's probably true. That's[00:42:31] swyx: probably true. The observation is I feel like you have a bias to its domain specific. , whereas most research is skewed towards, uh, general models, general purpose models.[00:42:40] I don't know if there's like a, a deeper insight here that you wanna go into or, or not, but like, train on all the things, get all the data and you're like, no, no, no. Everyone needs like customized per task,[00:42:49] Varun Mohan: uh, data set. Yeah. I think I'm not gonna. Say that general intelligence is not good. You want a base model that's still really good and that's probably trained on normal text, like a lot of different content.[00:43:00] But I think probably one thing that old school machine learning, even though I'm like the kind of person that says a lot of old school machine learning is just gonna die, is that training on a high quality data set for your workload is, is always gonna yield better results and more, more predictable results.[00:43:15] And I think we are under no illusions that that's not the case. Basical. And[00:43:19] swyx: then the other observation is bandwidth and connectivity, uh, which is not something that people usually think about, but apparently is a, is a big deal. Apparently training agreed in the synchronous needs, high GPU coordination.[00:43:29] These are deleted notes from Sam Altman talking about how they think about training and I was like, oh yeah, that's an insight. And[00:43:34] Varun Mohan: you guys have the same thing. Yeah. So I guess for, for training, you're right in that it is actually nuts to think about how insane the networks are for NVIDIA's most recent hardware, it's.[00:43:46] For the H 100 boxes, you shove eight of these H 100 s on a. Between two nodes. The bandwidth is 3,200 gigabits a second, so 400 gigabytes a second between machines. That's like nuts when you just sit and think about it. That's like double the memory bandwidth of what a CPU has, but it's like between two machines.[00:44:04] On top of that, within the machine, they've created this, this fabric called envy link that allows you to communicate at ultra low latency. That's even lower than P C I E. If you're familiar, that's like the communication protocol. . Yeah, between like the CPU and the other devices or other P C I E devices.[00:44:21] All of this is to make sure that reductions are fast, low latency, and you don't need to think about it. And that's because like a lot of deep learning has sort of evolved. Uh, training has evolved to be synchronous in the OG days. There is a lot of analysis in terms of how good is asynchronous training, which is like, Hey, I have a node, it has a current state of the model.[00:44:39] It's gonna update that itself locally, and it'll like every once in a while, go to another machine and update the weights. But I think like everyone has converged to synchronous. I'm not exactly sure. There's not a lot of good research on asynchronous training right now. Or maybe there is an, I haven't read it.[00:44:52] It's just that there isn't as much research because people are just like, oh, synchronous works. Uh, and the hardware is continually upleveled to handle[00:44:59] swyx: that. Yeah. It was just un unintuitive to me cuz like the whole purpose of GPUs could train things. A lot of things in parallel. Yes.[00:45:05] Varun Mohan: But the crazy thing is also, maybe I can, I can give some dumb math here.[00:45:09] Sure. Here, which is that, uh, let's go with uh, G B T three, which is like 170 billion per. The optimizer state, so while you're training is 14 times the size of the model, so in this case, if it's like 170 billion parameters, it's probably, I'm not great at mental math here, but that's probably around 2.5 terabytes to just store the optimizer state.[00:45:30] That has gotta be sharded across a lot of machines. Like that is not a single gpu. Even if you take an H 100 with 80 gigs to just shard that much, that's like 40, at least 30 machines. So there's like something there where these things need to communicate with each other too.[00:45:44] swyx: You need to vertically scale horizontally.[00:45:46] Varun Mohan: Yeah. You gotta co-located, you gotta somehow feel like you have this massive, the, the ideal programming paradigm is you feel like you have this massive computer. That has no communication, you know, overhead at all, but it has like infinite computer and infinite memory bandwidth.[00:45:59] swyx: That's the AI cluster. Um, okay, well, uh, we want to head to the questions.[00:46:05] Alessio Fanelli: So favorite AI product that you are not[00:46:08] Varun Mohan: building? Yeah, I'm friends with some of the folks at Mid Journey and I really think the Mid Journey product is super cool, especially seeing how the team is iterating and the quality of generations. It consistently gets upleveled. I think it's like quite neat and I think internally at at exa functional, we've been trying out mid Journey for like random content to like generate images and stuff.[00:46:26] Does it bother[00:46:26] swyx: you that they have like a style. I don't know. It, it seems like they're hedging themselves into a particular, like you want mid journey art, you go there.[00:46:33] Varun Mohan: Yeah. It's a brand of art. Yeah, you're right. I think they do have a style, but it seems more predictably good for that style. Okay. So maybe that's too, so just get good at, uh, domain specific thing.[00:46:41] Yeah. Yeah. maybe. Maybe I, maybe I'm just selling, talking to a booker right now. . Yeah. Uh, okay.[00:46:46] swyx: Uh, next question. Uh, favorite AI people and[00:46:48] Varun Mohan: communities? Yeah, so I think I mentioned this before, but I think obviously the open. The opening eye folks are, are insane. Like we, we only have respect for them. But beyond that, I think Elu is a pretty special group.[00:46:59] Especially it's been now probably more than a year and a half since they released like G P T J, which was like back when open source G PT three Curri, which was comparable. And it wasn't like a model where like, It wasn't good. It was like comparable in terms of perplexity to GT three curity and it was trained by a university student actually, and it just showed that, you know, in the end, like I would say pedigree is great, but in if you have people that are motivated know how computers work and they're willing to just get their hands dirty, you can do crazy things and that was a crazy project that gave me more hope.[00:47:34] Decentral training being potentially pretty massive. But I think that was like a very cool thing where a bunch of people just got on Discord and were chatting and they were able to just turn this out. Yeah. I did[00:47:42] swyx: not know this until I looked in further into Luther, but it was not a formal organization.[00:47:45] Was a company was a startup. It's not, yeah. Bunch of guys on Discord.[00:47:48] Varun Mohan: They gotta you, they gotta keep you research grant and they somehow just wrote some codes. .[00:47:52] Alessio Fanelli: Yeah. Yeah. Listen to APAC with Connor, who's the person, and basically Open Eye at the time was like, we cannot release G P T because it's like too good and so bad.[00:48:01] And he was like, He actually said he was sick, so he couldn't leave home for like a, a few weeks. So it was like, what else am I gonna do? And ended up getting through the Google like research programs through his university and they were like, oh, we'll give you TPUs. And he was like, cool. And that's how, that's,[00:48:17] Varun Mohan: that's amazing.[00:48:18] So I came to you. I love the story. Yeah, it's a great story. .[00:48:21] Alessio Fanelli: So a year from now, what do you think people will be most surprised by[00:48:25] Varun Mohan: In ai? Yeah. I think the thing people will be most surprised by is, I think they, the models are gonna, More good at SP special tasks for sure, but even the existing models, I think people will come up with more creative ways of leveraging them to build like world class products.[00:48:39] I think that's just like human creativity is gonna go wild. It seems like Cha GBT has already kind of unleashed that. I think I'm just excited to see what the future of these products look like. I guess law was not something I expected in such a short, well,[00:48:51] swyx: totally expected. I, I, I was actually watching a different company that I thought was gonna be the winner, and then Harvey just came outta nowhere,[00:48:56] Oh, wow. Okay. Okay. Well that's, that's awesome. But yeah. So my, my takeaway from what you're saying is like, foundation models have kind of shot way too far ahead of the apps and people need to build[00:49:05] Varun Mohan: apps. Yes. I think people should be building apps, but I. The reality is the model is like probably at a state right now where it can do crazy enough things.[00:49:12] Uh, and I think great apps will, will come out of this. Yeah.[00:49:16] swyx: AI thing you would pay for if someone else built it personal or work.[00:49:20] Varun Mohan: I think if, if someone else built like a proper assistant, like a proper like fitness assistant, I would probably pay for that actually. I know that, that sounds weird, but someone that actually tells me like, how should I end up, like, you know, doing fitness today, I ended up injuring my knee from over biking.[00:49:35] I ended up biking like 150 miles a week and I ended up just injuring my knee outta nowhere. So, so you need, you need an app to tell you to exercise less. Exercise less, but tell me what my training regimen is. Uh, tell me what I should do to prepare for things. I know that this is like a big niche, but I think the fact that Strava is such a big group of people and like swyx is a big group of people, seems to suggest that I think a lot of people would be willing to pay for something like this.[00:49:57] Alessio Fanelli: what's one thing you want everyone to take away about AI and our[00:50:01] Varun Mohan: conversation? Probably the most important thing to take away is there's probably a lot out there if people continue to tinker. I think that's probably like the biggest takeaway I've had. Uh, and it's, you know, being a pure infrastructure company, I think like, uh, six to eight months ago, I think it was like very hard to watch everyone tinkering and us just, you know, building, building infrastructure.[00:50:22] But I think there's gonna be some crazy things that come out over the next year or. Um, excited to just see what that looks like. Awesome. Yeah, man. That's it. This was fantastic. Thanks so much. Thanks for coming. Get full access to Latent Space at www.latent.space/subscribe

We’re so glad to launch our first podcast episode with Logan Kilpatrick! This also happens to be his first public interview since joining OpenAI as their first Developer Advocate. Thanks Logan!Recorded in-person at the beautiful StudioPod studios in San Francisco. Full transcript is below the fold.Timestamps* 00:29: Logan’s path to OpenAI* 07:06: On ChatGPT and GPT3 API* 16:16: On Prompt Engineering* 20:30: Usecases and LLM-Native Products* 25:38: Risks and benefits of building on OpenAI* 35:22: OpenAI Codex* 42:40: Apple's Neural Engine* 44:21: Lightning RoundShow notes* Sam Altman’s interview with Connie Loizos* OpenAI Cookbook* OpenAI’s new Embedding Model* Cohere on Word and Sentence Embeddings* (referenced) What is AGI-hard?Lightning Rounds* Favorite AI Product: https://www.synthesia.io/* Favorite AI Community: MLOps * One year prediction: Personalized AI, https://civitai.com/* Takeaway: AI Revolution is here!Transcript[00:00:00] Alessio Fanelli: Hey everyone. Welcome to the Latent Space podcast. This is Alessio, partner and CTO in residence at Decibel Partners. I'm joined by my cohost, swyx writer editor of L Space Diaries. Hey.[00:00:20] swyx: Hey . Our guest today is Logan Kilpatrick. What I'm gonna try to do is I'm gonna try to introduce you based on what people know about you, and then you can fill in the blanks.[00:00:28] Introducing Logan[00:00:28] swyx: So you are the first. Developer advocate at OpenAI, which is a humongous achievement. Congrats. You're also the lead developer community advocate of the Julia language. I'm interested in a little bit of that and apparently as I've did a bit of research on you, you got into Julia through NASA where you interned and worked on stuff that's gonna land on the moon apparently.[00:00:50] And you are also working on computer vision at Apple. And had to sit at path, the eye as you fell down the machine learning rabbit hole. What should people know about you that's kind of not on your LinkedIn that like sort of ties together your interest[00:01:02] Logan Kilpatrick: in story? It's a good question. I think so one of the things that is on my LinkedIn that wasn't mentioned that's super near and dear to my heart and what I spend a lot of time in sort of wraps a lot of my open source machine learning developer advocacy experience together is supporting NumFOCUS.[00:01:17] And NumFOCUS is the nonprofit that helps enable a bunch of the open source scientific projects like Julia, Jupyter, Pandas, NumPy, all of those open source projects are. Facilitated legal and fiscally through NumFOCUS. So it's a very critical, important part of the ecosystem and something that I, I spend a bunch of my now more limited free time helping support.[00:01:37] So yeah, something that's, It's on my LinkedIn, but it's, it's something that's important to me. Well,[00:01:42] swyx: it's not as well known of a name, so maybe people kind of skip over it cuz they were like, I don't know what[00:01:45] Logan Kilpatrick: to do with this. Yeah. It's super interesting to see that too. Just one point of context for that is we tried at one point to get a Wikipedia page for non focus and it's, it's providing, again, the infrastructure for, it's like a hundred plus open source scientific projects and they're like, it's not notable enough.[00:01:59] I'm like, well, you know, there's something like 30 plus million developers around the world who use all these open source tools. It's like the foundation. All open source like science that happens. Every breakthrough in science is they discovered the black hole, the first picture of the black hole, all that stuff using numb focus tools, the Mars Rovers, NumFOCUS tools, and it's interesting to see like the disconnect between the nonprofit that supports those projects and the actual success of the projects themselves.[00:02:26] swyx: Well, we'll, we'll get a bunch of people focused on NumFOCUS and we'll get it on Wikipedia. That that is our goal. . That is the goal. , that is our shot. Is this something that you do often, which is you? You seem to always do a lot of community stuff. When you get into something, you're also, I don't know where this, where you find time for this.[00:02:42] You're also a conference chair for DjangoCon, which was last year as well. Do you fall down the rabbit hole of a language and then you look for community opportunities? Is that how you get into.[00:02:51] Logan Kilpatrick: Yeah, so the context for Django stuff was I'd actually been teaching and still am through Harvard's division of continuing education as a teaching fellow for a Django class, and had spent like two and a half years actually teaching students every semester, had a program in Django and realized that like it was kind of the one ecosystem or technical tool that I was using regularly that I wasn't actually contributing to that community.[00:03:13] So, I think sometime in 2021 like applied to be on the board of directors of the Django Events Foundation, north America, who helps run DjangoCon and was fortunate enough to join a support to be the chair of DjangoCon us and then just actually rolled off the board because of all the, all the craziness and have a lot less free time now.[00:03:32] And actually at PATH ai. Sort of core product was also using, was using Django, so it also had a lot of connections to work, so it was a little bit easier to justify that time versus now open ai. We're not doing any Django stuff unfortunately, so, or[00:03:44] swyx: Julia, I mean, should we talk about this? Like, are you defecting from Julia?[00:03:48] What's going on? ,[00:03:50] Logan Kilpatrick: it's actually felt a little bit strange recently because I, for the longest time, and, and happy to talk about this in the context of Apple as well, the Julie ecosystem was my outlet to do a lot of the developer advocacy, developer relations community work that I wanted to do. because again, at Apple I was just like training machine learning models.[00:04:07] Before that, doing software engineering at Apple, and even at Path ai, we didn't really have a developer product, so it wasn't, I was doing like advocacy work, but it wasn't like developer relations in the traditional sense. So now that I'm so deeply doing developer relations work at Open OpenAI, it's really difficult to.[00:04:26] Continue to have the energy after I just spent nine hours doing developer relations stuff to like go and after work do a bunch more developer relations stuff. So I'll be interested to see for myself like how I'm able to continue to do that work and I. The challenge is that it's, it's such critical, important work to happen.[00:04:43] Like I think the Julie ecosystem is so important. I think the language is super important. It's gonna continue to grow in, in popularity, and it's helping scientists and engineers solve problems they wouldn't otherwise be able to. So it's, yeah, the burden is on me to continue to do that work, even though I don't have a lot of time now.[00:04:58] And I[00:04:58] Alessio Fanelli: think when it comes to communities, the machine learning technical community, I think in the last six to nine months has exploded. You know, you're the first developer advocate at open ai, so I don't think anybody has a frame of reference on what that means. What is that? ? So , what do you, how did, how the[00:05:13] swyx: job, yeah.[00:05:13] How do you define the job? Yeah, let's talk about that. Your role.[00:05:16] Logan Kilpatrick: Yeah, it's a good question and I think there's a lot of those questions that actually still exist at OpenAI today. Like I think a lot of traditional developed by advocacy, at least like what you see on Twitter, which I think is what a lot of people's perception of developer advocacy and developer relations is, is like, Just putting out external content, going to events, speaking at conferences.[00:05:35] And I think OpenAI is very unique in the sense that, at least at the present moment, we have so much inbound interest that there's, there is no desire for us to like do that type of developer advocacy work. So it's like more from a developer experience point of view actually. Like how can we enable developers to be successful?[00:05:53] And that at the present moment is like building a strong foundation of documentation and things like that. And we had a bunch of amazing folks internally who were. Who were doing some of this work, but it really wasn't their full-time job. Like they were focused on other things and just helping out here and there.[00:06:05] And for me, my full-time job right now is how can we improve the documentation so that people can build the next generation of, of products and services on top of our api. And it's. Yeah. There's so much work that has to happen, but it's, it's, it's been a ton of fun so far. I find[00:06:20] swyx: being in developer relations myself, like, it's kind of like a fill in the blanks type of thing.[00:06:24] Like you go to where you, you're needed the most open. AI has no problem getting attention. It is more that people are not familiar with the APIs and, and the best practices around programming for large language models, which is a thing that did not exist three years ago, two years ago, maybe one year ago.[00:06:40] I don't know. When she launched your api, I think you launched Dall-E. As an API or I, I don't[00:06:45] Logan Kilpatrick: know. I dunno. The history, I think Dall-E was, was second. I think it was some of the, like GPT3 launched and then GPT3 launched and the API I think like two years ago or something like that. And then Dali was, I think a little more than a year ago.[00:06:58] And then now all the, the Chachi Beast ChatGPT stuff has, has blown it all outta the water. Which you have[00:07:04] swyx: a a wait list for. Should we get into that?[00:07:06] Logan Kilpatrick: Yeah. .[00:07:07] ChatGPT[00:07:07] Alessio Fanelli: Yeah. We would love to hear more about that. We were looking at some of the numbers you went. Zero to like a million users in five days and everybody, I, I think there's like dozens of ChatGPT API wrappers on GitHub that are unofficial and clearly people want the product.[00:07:21] Like how do you think about that and how developers can interact with it.[00:07:24] Logan Kilpatrick: It. It's absolutely, I think one of the most exciting things that I can possibly imagine to think about, like how much excitement there was around ChatGPT and now getting to hopefully at some point soon, put that in the hands of developers and see what they're able to unlock.[00:07:38] Like I, I think ChatGPT has been a tremendous success, hands down without a question, but I'm actually more excited to see what developers do with the API and like being able to build those chat first experiences. And it's really fascinating to see. Five years ago or 10 years ago, there was like, you know, all this like chatbot sort of mm-hmm.[00:07:57] explosion. And then that all basically went away recently, and the hype went to other places. And I think now we're going to be closer to that sort of chat layer and all these different AI chat products and services. And it'll be super interesting to see if that sticks or not. I, I'm not. , like I think people have a lot of excitement for ChatGPT right now, but it's not clear to me that that that's like the, the UI or the ux, even though people really like it in the moment, whether that will stand the test of time, I, I just don't know.[00:08:23] And I think we'll have to do a podcast in five years. Right. And check in and see whether or not people are still really enjoying that sort of conversational experience. I think it does make sense though cause like that's how we all interact and it's kind of weird that you wouldn't do that with AI products.[00:08:37] So we. and I think like[00:08:40] Alessio Fanelli: the conversational interface has made a lot of people, first, the AI to hallucinate, you know, kind of come up with things that are not true and really find all the edge cases. I think we're on the optimism camp, you know, like we see the potential. I think a lot of people like to be negative.[00:08:56] In your role, kind of, how do you think about evangelizing that and kind of the patience that sometimes it takes for these models to become.[00:09:03] Logan Kilpatrick: Yeah, I think what, what I've done is just continue to scream from the, the mountains that like ChatGPT has, current form is definitely a research preview. The model that underlies ChatGPT GPT 3.5 is not a research preview.[00:09:15] I think there's things that folks can do to definitely reduce the amount of hall hallucinations and hopefully that's something that over time I, I, again have full confidence that it'll, it'll solve. Yeah, there's a bunch of like interesting engineering challenges. you have to solve in order to like really fix that problem.[00:09:33] And I think again, people are, are very fixated on the fact that like in, you know, a few percentage points of the conversations, things don't sound really good. Mm-hmm. , I'm really more excited to see, like, again when the APIs and the Han developers like what are the interesting solutions that people come up with, I think there's a lot that can be explored and obviously, OpenAI can explore all them because we have this like one product that's using the api.[00:09:56] And once you get 10,000, a hundred thousand developers building on top of that, like, we'll see what are the different ways that people handle this. And I imagine there's a lot of low-hanging fruit solutions that'll significantly improve the, the amount of halluc hallucinations that are showing up. Talk about[00:10:11] swyx: building on top of your APIs.[00:10:13] Chat GPTs API is not out yet, but let's assume it is. Should I be, let's say I'm, I'm building. A choice between GP 3.5 and chat GPT APIs. As far as I understand, they are kind of comparable. What should people know about deciding between either of them? Like it's not clear to me what the difference is.[00:10:33] Logan Kilpatrick: It's a great question.[00:10:35] I don't know if there's any, if we've made any like public statements about like what the difference will be. I think, I think the point is that the interface for the Chachi B API will be like conversational first, and that's not the case now. If you look at text da Vinci oh oh three, like you, you just put in any sort of prompt.[00:10:52] It's not really built from the ground up to like keep the context of a conversation and things like that. And so it's really. Put in some sort of prompt, get a response. It's not always designed to be in that sort of conversational manner, so it's not tuned in that way. I think that's the biggest difference.[00:11:05] I think, again, the point that Sam made in a, a strictly the strictly VC talk mm-hmm. , which was incredible and I, I think that that talk got me excited and my, which, which part? The whole thing. And I think, I haven't been at open AI that long, so like I didn't have like a s I obviously knew who Sam was and had seen a bunch of stuff, but like obviously before, a lot of the present craziness with Elon Musk, like I used to think Elon Musk seemed like a really great guy and he was solving all these really important problems before all the stuff that happened.[00:11:33] That's a hot topic. Yeah. The stuff that happened now, yeah, now it's much more questionable and I regret having a Tesla, but I, I think Sam is actually. Similar in the sense that like he's solving and thinking about a lot of the same problems that, that Elon, that Elon is still today. But my take is that he seems like a much more aligned version of Elon.[00:11:52] Like he's, he's truly like, I, I really think he cares deeply about people and I think he cares about like solving the problems that people have and wants to enable people. And you can see this in the way that he's talked about how we deploy models at OpenAI. And I think you almost see Tesla in like the completely opposite end of the spectrum, where they're like, whoa, we.[00:12:11] Put these 5,000 pound machines out there. Yeah. And maybe they'll run somebody over, maybe they won't. But like it's all in the interest of like advancement and innovation. I think that's really on the opposite end of the spectrum of, of what open AI is doing, I think under Sam's leadership. So it's, it's interesting to see that, and I think Sam said[00:12:30] Alessio Fanelli: that people could have built Chen g p t with what you offered like six, nine months ago.[00:12:35] I[00:12:35] swyx: don't understand. Can we talk about this? Do you know what, you know what we're talking about, right? I do know what you're talking about. da Vinci oh three was not in the a p six months before ChatGPT. What was he talking about? Yeah.[00:12:45] Logan Kilpatrick: I think it's a little bit of a stretch, but I do think that it's, I, I think the underlying principle is that.[00:12:52] The way that it, it comes back to prompt engineering. The way that you could have engineered, like the, the prompts that you were put again to oh oh three or oh oh two. You would be able to basically get that sort of conversational interface and you can do that now. And, and I, you know, I've seen tutorials.[00:13:05] We have tutorials out. Yep. No, we, I mean, we, nineties, we have tutorials in the cookbook right now in on GitHub. We're like, you can do this same sort of thing. And you just, it's, it's all about how you, how you ask for responses and the way you format data and things like that. It. The, the models are currently only limited by what people are willing to ask them to do.[00:13:24] Like I really do think that, yeah, that you can do a lot of these things and you don't need the chat CBT API to, to build that conversational layer. That is actually where I[00:13:33] swyx: feel a little bit dumb because I feel like I don't, I'm not smart enough to think of new things to ask the models. I have to see an example and go, oh, you can do that.[00:13:43] All right, I'm gonna do that for now. You know, and, and that's why I think the, the cookbook is so important cuz it's kind of like a compendium of things we know about the model that you can ask it to do. I totally[00:13:52] Logan Kilpatrick: agree and I think huge shout out to the, the two folks who I work super closely with now on the cookbook, Ted and Boris, who have done a lot of that work and, and putting that out there and it's, yeah, you see number one trending repo on, on GitHub and it was super, like when my first couple of weeks at Open ai, super unknown, like really, we were only sort of directing our customers to that repo.[00:14:13] Not because we were trying to hide it or anything, but just because. It was just the way that we were doing things and then all of a sudden it got picked up on GitHub trending and a bunch of tweets went viral, showing the repo. So now I think people are actually being able to leverage the tools that are in there.[00:14:26] And, and Ted's written a bunch of amazing tutorials, Boris, as well. So I think it's awesome that more people are seeing those. And from my perspective, it's how can we take those, make them more accessible, give them more visibility, put them into the documentation, and I don't think that that connection right now doesn't exist, which I'm, I'm hopeful we'll be able to bridge those two things.[00:14:44] swyx: Cookbook is kind of a different set of documentation than API docs, and I think there's, you know, sort of existing literature about how you document these things and guide developers the right way. What, what I, what I really like about the cookbook is that it actually cites academic research. So it's like a nice way to not read the paper, but just read the conclusions of the paper ,[00:15:03] Logan Kilpatrick: and, and I think that's, that's a shout out to Ted and Boris cuz I, I think they're, they're really smart in that way and they've done a great job of finding the balance and understanding like who's actually using these different tools.[00:15:13] So, . Yeah.[00:15:15] swyx: You give other people credit, but you should take credit for yourself. So I read your last week you launched some kind of documentation about rate limiting. Yeah. And one of my favorite things about reading that doc was seeing examples of, you know, you were, you're telling people to do exponential back off and, and retry, but you gave code examples with three popular libraries.[00:15:32] You didn't have to do that. You could have just told people, just figure it out. Right. But you like, I assume that was you. It wasn't.[00:15:38] Logan Kilpatrick: So I think that's the, that's, I mean, I'm, I'm helping sort of. I think there's a lot of great stuff that people have done in open ai, but it was, we have the challenge of like, how can we make that accessible, get it into the documentation and still have that high bar for what goes into the doc.[00:15:51] So my role as of recently has been like helping support the team, building that documentation first culture, and supporting like the other folks who actually are, who wrote that information. The information was actually already in. Help center but it out. Yeah, it wasn't in the docs and like wasn't really focused on, on developers in that sense.[00:16:10] So yeah. I can't take the, the credit for the rate limit stuff either. , no, this[00:16:13] swyx: is all, it's part of the A team, that team effort[00:16:16] On Prompt Engineering[00:16:16] Alessio Fanelli: I was reading on Twitter, I think somebody was saying in the future will be kind of like in the hair potter word. People have like the spell book, they pull it out, they do all the stuff in chat.[00:16:24] GP z. When you talk with customers, like are they excited about doing prompt engineering and kind of getting a starting point or do they, do they wish there was like a better interface? ?[00:16:34] Logan Kilpatrick: Yeah, that's a good question. I think prompt engineering is so much more of an art than a science right now. Like I think there are like really.[00:16:42] Systematic things that you can do and like different like approaches and designs that you can take, but really it's a lot of like, you kind of just have to try it and figure it out. And I actually think that this remains to be one of the challenges with large language models in general, and not just head open ai, but for everyone doing it is that it's really actually difficult to understand what are the capabilities of the model and how do I get it to do the things that I wanted to do.[00:17:05] And I think that's probably where a lot of folks need to do like academic research and companies need to invest in understanding the capabilities of these models and the limitations because it's really difficult to articulate the capabilities of a model without those types of things. So I'm hopeful that, and we're shipping hopefully some new updated prompt engineering stuff.[00:17:24] Cause I think the stuff we have on the website is old, and I think the cookbook actually has a little bit more up-to-date stuff. And so hopefully we'll ship some new prompt engineering stuff in the, in the short term. I think dispel some of the myths and rumors, but like I, it's gonna continue to be like a, a little bit of a pseudoscience, I would imagine.[00:17:41] And I also think that the whole prompt engineering being like a job in the future meme, I think is, I think it's slightly overblown. Like I think at, you see this now actually with like, there's tools that are showing up and I forgot what the, I just saw went on Twitter. The[00:17:57] swyx: next guest that we are having on this podcast, Lang.[00:17:59] Yeah. Yeah.[00:18:00] Logan Kilpatrick: Lang Chain and Harrison on, yeah, there's a bunch of repos too that like categorize and like collect all the best prompts that you can put into chat. For example, and like, that's like the people who are, I saw the advertisement for someone to be like a prompt engineer and it was like a $350,000 a year.[00:18:17] Mm-hmm. . Yeah, that was, that was philanthropic. Yeah, so it, it's just unclear to me like how, how sustainable stuff like that is. Cuz like, once you figure out the interesting prompts and like right now it's kind of like the, the Wild West, but like in a year you'll be able to sort of categorize all those and then people will be able to find all the good ones that are relevant for what they want to do.[00:18:35] And I think this goes back to like, having the examples is super important and I'm, I'm with you as well. Like every time I use Dall-E the little. While it's rendering the image, it gives you like a suggestion of like how you should ask for the art to be generated. Like do it in like a cyberpunk format. Do it in a pixel art format.[00:18:53] Et cetera, et cetera, and like, I really need that. I'm like, I would never come up with asking for those things had it not prompted me to like ask it that way. And now I always ask for pixel art stuff or cyberpunk stuff and it looks so cool. That's what I, I think,[00:19:06] swyx: is the innovation of ChatGPT as a format.[00:19:09] It reduces. The need for getting everything into your prompt in the first try. Mm-hmm. , it takes it from zero shot to a few shot. If, if, if that, if prompting as, as, as shots can be concerned.[00:19:21] Logan Kilpatrick: Yeah. , I think that's a great perspective and, and again, this goes back to the ux UI piece of it really being sort of the differentiating layer from some of the other stuff that was already out there.[00:19:31] Because you could kind of like do this before with oh oh three or something like that if you just made the right interface and like built some sort of like prompt retry interface. But I don't think people were really, were really doing that. And I actually think that you really need that right now. And this is the, again, going back to the difference between like how you can use generative models versus like large scale.[00:19:53] Computer vision systems for self-driving cars, like the, the answer doesn't actually need to be right all the time. That's the beauty of, of large language models. It can be wrong 50% of the time and like it doesn't really cost you anything to like regenerate a new response. And there's no like, critical safety issue with that, so you don't need those.[00:20:09] I, I keep seeing these tweets about like, you need those like 99.99% reliability and like the three nines or whatever it is. Mm-hmm. , but like you really don't need that because the cost of regenerating the prop is again, almost, almost. I think you tweeted a[00:20:23] Alessio Fanelli: couple weeks ago that the average person doesn't yet fully grasp how GBT is gonna impact human life in the next four, five years.[00:20:30] Usecases and LLM-Native Products[00:20:30] Alessio Fanelli: I think you had an example in education. Yeah. Maybe touch on some of these. Example of non-tech related use cases that are enabling, enabled by C G B[00:20:38] T.[00:20:39] Logan Kilpatrick: I'm so excited and, and there's a bunch of other like random threads that come to my mind now. I saw a thread and, and our VP of product was, Peter, was, was involved in that thread as well, talking about like how the use of systems like ChatGPT will unlock like pretty almost low to zero cost access to like mental health services.[00:20:59] You know, you can imagine like the same use case for education, like really personalized tutors and like, it's so crazy to think about, but. The technology is not actually , like it's, it's truly like an engineering problem at this point of like somebody using one of these APIs to like build something like that and then hopefully the models get a little bit better and make it, make it better as well.[00:21:20] But like it, I have no doubt in my mind that three years from now that technology will exist for every single student in the world to like have that personalized education experience, have a pr, have a chat based experience where like they'll be able. Ask questions and then the curriculum will just evolve and be constructed for them in a way that keeps, I think the cool part is in a way that keeps them engaged, like it doesn't have to be sort of like the same delivery of curriculum that you've always seen, and this now supplements.[00:21:49] The sort of traditional education experience in the sense of, you know, you don't need teachers to do all of this work. They can really sort of do the thing that they're amazing at and not spend time like grading assignments and all that type of stuff. Like, I really do think that all those could be part of the, the system.[00:22:04] And same thing, I don't know if you all saw the the do not pay, uh, lawyer situation, say, I just saw that Twitter thread, I think yesterday around they were going to use ChatGPT in the courtroom and basically I think it was. California Bar or the Bar Institute said that they were gonna send this guy to prison if he brought, if he put AirPods in and started reading what ChatGPT was saying to him.[00:22:26] Yeah.[00:22:26] swyx: To give people the context, I think, like Josh Browder, the CEO of Do Not Pay, was like, we will pay you money to put this AirPod into your ear and only say what we tell you to say fr from the large language model. And of course the judge was gonna throw that out. I mean, I, I don't see how. You could allow that in your court,[00:22:42] Logan Kilpatrick: Yeah, but I, I really do think that, like, the, the reality is, is that like, again, it's the same situation where the legal spaces even more so than education and, and mental health services, is like not an accessible space. Like every, especially with how like overly legalized the United States is, it's impossible to get representation from a lawyer, especially if you're low income or some of those things.[00:23:04] So I'm, I'm optimistic. Those types of services will exist in the future. And you'll be able to like actually have a, a quality defense representative or just like some sort of legal counsel. Yeah. Like just answer these questions, what should I do in this situation? Yeah. And I like, I have like some legal training and I still have those same questions.[00:23:22] Like I don't know what I would do in that situation. I would have to go and get a lawyer and figure that out. And it's, . It's tough. So I'm excited about that as well. Yeah.[00:23:29] Alessio Fanelli: And when you think about all these vertical use cases, do you see the existing products implementing language models in what they have?[00:23:35] Or do you think we're just gonna see L L M native products kind of come to market and build brand[00:23:40] Logan Kilpatrick: new experiences? I think there'll be a lot of people who build the L l M first experience, and I think that. At least in the short term, those are the folks who will have the advantage. I do think that like the medium to long term is again, thinking about like what is your moat for and like again, and everyone has access to, you know, ChatGPT and to the different models that we have available.[00:24:05] So how can you build a differentiated business? And I think a lot of it actually will come down to, and this is just the true and the machine learning world in general, but having. Unique access to data. So I think if you're some company that has some really, really great data about the legal space or about the education space, you can use that and be better than your competition by fine tuning these models or building your own specific LLMs.[00:24:28] So it'll, it'll be interesting to see how that plays out, but I do think that. from a product experience, it's gonna be better in the short term for people who build the, the generative AI first experience versus people who are sort of bolting it onto their mm-hmm. existing product, which is why, like, again, the, the Google situation, like they can't just put in like the prompt into like right below the search bar.[00:24:50] Like, it just, it would be a weird experience and, and they have to sort of defend that experience that they have. So it, it'll be interesting to see what happens. Yeah. Perplexity[00:24:58] swyx: is, is kind of doing that. So you're saying perplexity will go Google ?[00:25:04] Logan Kilpatrick: I, I think that perplexity has a, has a chance in the short term to actually get more people to try the product because it's, it's something different I think, whether they can, I haven't actually used, so I can't comment on like that experience, but like I think the long term is like, How can they continue to differentiate?[00:25:21] And, and that's really the focus for like, if you're somebody building on these models, like you have to be, your first thought should be, how do I build a differentiated business? And if you can't come up with 10 reasons that you can build a differentiated business, you're probably not gonna succeed in, in building something that that stands the test of time.[00:25:37] Yeah.[00:25:37] Risks and benefits of building on OpenAI[00:25:37] swyx: I think what's. As a potential founder or something myself, like what's scary about that is I would be building on top of open ai. I would be sending all my stuff to you for fine tuning and embedding and what have you. By the way, fine tuning, embedding is their, is there a third one? Those are the main two that I know of.[00:25:55] Okay. And yeah, that's the risk. I would be a open AI API reseller.[00:26:00] Logan Kilpatrick: Yeah. And, and again, this, this comes back down to like having a clear sense of like how what you're building is different. Like the people who are just open AI API resellers, like, you're not gonna, you're not gonna have a successful business doing that because everybody has access to the Yeah.[00:26:15] Jasper's pretty great. Yeah, Jasper's pretty great because I, I think they've done a, they've, they've been smart about how they've positioned the product and I was actually a, a Jasper customer before I joined OpenAI and was using it to do a bunch of stuff. because the interface was simple because they had all the sort of customized, like if you want for like a response for this sort of thing, they'd, they'd pre-done that prompt engineering work for us.[00:26:39] I mean, you could really just like put in some exactly what you wanted and then it would make that Amazon product description or whatever it is. So I think like that. The interface is the, the differentiator for, for Jasper. And again, whether that send test time, hopefully, cuz I know they've raised a bunch of money and have a bunch of employees, so I'm, I'm optimistic for them.[00:26:58] I think that there's enough room as well for a lot of these companies to succeed. Like it's not gonna, the space is gonna get so big so quickly that like, Jasper will be able to have a super successful business. And I think they are. I just saw some, some tweets from the CEO the other day that I, I think they're doing, I think they're doing well.[00:27:13] Alessio Fanelli: So I'm the founder of A L L M native. I log into open ai, there's 6 million things that I can do. I'm on the playground. There's a lot of different models. How should people think about exploring the surface area? You know, where should they start? Kind of like hugging the go deeper into certain areas.[00:27:30] Logan Kilpatrick: I think six months ago, I think it would've been a much different conversation because people hadn't experienced ChatGPT before.[00:27:38] Now that people have experienced ChatGPT, I think there's a lot more. Technical things that you should start looking into and, and thinking about like the differentiators that you can bring. I still think that the playground that we have today is incredible cause it does sort of similar to what Jasper does, which is like we have these very focused like, you know, put in a topic and we'll generate you a summary, but in the context of like explaining something to a second grader.[00:28:03] So I think all of those things like give a sense, but we only have like 30 on the website or something like that. So really doing a lot of exploration around. What is out there? What are the different prompts that you can use? What are the different things that you can build on? And I'm super bullish on embeddings, like embed everything and that's how you can build cool stuff.[00:28:20] And I keep seeing all these Boris who, who I talked about before, who did a bunch of the cookbook stuff, tweeted the other day that his like back of the hand, back of the napkin math, was that 50 million bucks you can embed the whole internet. I'm like, Some companies gonna spend the 50 million and embed the whole internet and like, we're gonna find out what that product looks like.[00:28:40] But like, there's so many cool things that you could do if you did have the whole internet embedded. Yeah, and I, I mean, I wouldn't be surprised if Google did that cuz 50 million is a drop in the bucket and they already have the whole internet, so why not embed it?[00:28:52] swyx: Can can I ask a follow up question on that?[00:28:54] Cuz I am just learning about embeddings myself. What makes OpenAI’s embeddings different from other embeddings? If, if there's like, It's okay if you don't have the, the numbers at hand, but I'm just like, why should I use open AI emitting versus others? I[00:29:06] Logan Kilpatrick: don't understand. Yeah, that's a really good question.[00:29:08] So I'm still ramping up on my understanding of embeddings as well. So the two things that come to my mind, one, going back to the 50 million to embed the whole internet example, it's actually just super cheap. I, I don't know the comparisons of like other prices, but at least from what I've seen people talking about on Twitter, like the embeddings that that we have in the API is just like significantly cheaper than a lot of other c.[00:29:30] Embeddings. Also the accuracy of some of the benchmarks that are like, Sort of academic benchmarks to use in embeddings. I know at least I was just looking back through the blog post from when we announced the new text embedding model, which is what Powers embeddings and it's, yeah, the, on those metrics, our API is just better.[00:29:50] So those are the those. I'll go read it up. Yeah, those are the two things. It's a good. It's a good blog post to read. I think the most recent one that came out, but, and also the original one from when we first announced the Embeddings api, I think also was a, it had, that one has a little bit more like context around if you're trying to wrap your head around embeddings, how they work.[00:30:06] That one has the context, the new one just has like the fancy new stuff and the metrics and all that kind of stuff.[00:30:11] swyx: I would shout a hugging face for having really good content around what these things like foundational concepts are. Because I was familiar with, so, you know, in Python you have like text tove, my first embedding as as a, as someone getting into nlp.[00:30:24] But then developing the concept of sentence embeddings is, is as opposed to words I think is, is super important. But yeah, it's an interesting form of lock in as a business because yes, I'm gonna embed all my source data, but then every inference needs an embedding as. . And I think that is a risk to some people, because I've seen some builders should try and build on open ai, call that out as, as a cost, as as like, you know, it starts to add a cost to every single query that you, that you[00:30:48] Logan Kilpatrick: make.[00:30:49] Yeah. It'll be interesting to see how it all plays out, but like, my hope is that that cost isn't the barrier for people to build because it's, it's really not like the cost for doing the incremental like prompts and having them embedded is, is. Cent less than cents, but[00:31:06] swyx: cost I, I mean money and also latency.[00:31:08] Yeah. Which is you're calling the different api. Yeah. Anyway, we don't have to get into that.[00:31:13] Alessio Fanelli: No, but I think embeds are a good example. You had, I think, 17 versions of your first generation, what api? Yeah. And then you released the second generation. It's much cheaper, much better. I think like the word on the street is like when GPT4 comes out, everything else is like trash that came out before it.[00:31:29] It's got[00:31:30] Logan Kilpatrick: 100 trillion billion. Exactly. Parameters you don't understand. I think Sam has already confirmed that those are, those are not true . The graphics are not real. Whatever you're seeing on Twitter about GPT4, you're, I think the direct quote was, you're begging to be disappointed by continuing to, to put that hype out.[00:31:47] So[00:31:48] Alessio Fanelli: if you're a developer building on these, What's kind of the upgrade path? You know, I've been building on Model X, now this new model comes out. What should I do to be ready to move on?[00:31:58] Logan Kilpatrick: Yeah. I think all of these types of models folks have to think about, like there will be trade offs and they'll also be.[00:32:05] Breaking changes like any other sort of software improvement, like things like the, the prompts that you were previously expecting might not be the prompts that you're seeing now. And you can actually, you, you see this in the case of the embeddings example that you just gave when we released Tex embeddings, ADA oh oh two, ada, ada, whichever it is oh oh two, and it's sort of replaced the previous.[00:32:26] 16 first generation models, people went through this exact experience where like, okay, I need to test out this new thing, see how it works in my environment. And I think that the really fascinating thing is that there aren't, like the tools around doing this type of comparison don't exist yet today. Like if you're some company that's building on lms, you sort of just have to figure it out yourself of like, is this better in my use case?[00:32:49] Is this not better? In my use case, it's, it's really difficult to tell because the like, Possibilities using generative models are endless. So I think folks really need to focus on, again, that goes back to how to build a differentiated business. And I think it's understanding like what is the way that people are using your product and how can you sort of automate that in as much way and codify that in a way that makes it clear when these different models come up, whether it's open AI or other companies.[00:33:15] Like what is the actual difference between these and which is better for my use case because the academic be. It'll be saturated and people won't be able to use them as a point of comparison in the future. So it'll be important to think about. For your specific use case, how does it differentiate?[00:33:30] swyx: I was thinking about the value of frameworks or like Lang Chain and Dust and what have you out there.[00:33:36] I feel like there is some value to building those frameworks on top of OpenAI’s APIs. It kind of is building what's missing, essentially what, what you guys don't have. But it's kind of important in the software engineering sense, like you have this. Unpredictable, highly volatile thing, and you kind of need to build a stable foundation on top of it to make it more predictable, to build real software on top of it.[00:33:59] That's a super interesting kind of engineering problem. .[00:34:03] Logan Kilpatrick: Yeah, it, it is interesting. It's also the, the added layer of this is that the large language models. Are inherently not deterministic. So I just, we just shipped a small documentation update today, which, which calls this out. And you think about APIs as like a traditional developer experience.[00:34:20] I send some response. If the response is the same, I should get the same thing back every time. Unless like the data's updating and like a, from like a time perspective. But that's not the, that's not the case with the large language models, even with temperature zero. Mm-hmm. even with temperature zero. Yep.[00:34:34] And that's, Counterintuitive part, and I think someone was trying to explain to me that it has to do with like Nvidia. Yeah. Floating points. Yes. GPU stuff. and like apparently the GPUs are just inherently non-deterministic. So like, yes, there's nothing we can do unless this high Torch[00:34:48] swyx: relies on this as well.[00:34:49] If you want to. Fix this. You're gonna have to tear it all down. ,[00:34:53] Logan Kilpatrick: maybe Nvidia, we'll fix it. I, I don't know, but I, I think it's a, it's a very like, unintuitive thing and I don't think that developers like really get that until it happens to you. And then you're sort of scratching your head and you're like, why is this happening?[00:35:05] And then you have to look it up and then you see all the NVIDIA stuff. Or hopefully our documentation makes it more clear now. But hopefully people, I also think that's, it's kinda the cool part as well. I don't know, it's like, You're not gonna get the same stuff even if you try to.[00:35:17] swyx: It's a little spark of originality in there.[00:35:19] Yeah, yeah, yeah, yeah. The random seed .[00:35:22] OpenAI Codex[00:35:22] swyx: Should we ask about[00:35:23] Logan Kilpatrick: Codex?[00:35:23] Alessio Fanelli: Yeah. I mean, I love Codex. I use it every day. I think like one thing, sometimes the code is like it, it's kinda like the ChatGPT hallucination. Like one time I asked it to write up. A Twitter function, they will pull the bayou of this thing and it wrote the whole thing and then the endpoint didn't exist once I went to the Twitter, Twitter docs, and I think like one, I, I think there was one research that said a lot of people using Co Palace, sometimes they just auto complete code that is wrong and then they commit it and it's a, it's a big[00:35:51] Logan Kilpatrick: thing.[00:35:51] swyx: Do you secure code as well? Yeah, yeah, yeah, yeah. I saw that study.[00:35:54] Logan Kilpatrick: How do[00:35:54] Alessio Fanelli: you kind of see. Use case evolving. You know, you think, like, you obviously have a very strong partnership with, with Microsoft. Like do you think Codex and VS code will just keep improving there? Do you think there's kind of like a. A whole better layer on top of it, which is from the scale AI hackathon where the, the project that one was basically telling the l l m, you're not the back end of a product[00:36:16] And they didn't even have to write the code and it's like, it just understood. Yeah. How do you see the engineer, I, I think Sean, you said copilot is everybody gets their own junior engineer to like write some of the code and then you fix it For me, a lot of it is the junior engineer gets a senior engineer to actually help them write better code.[00:36:32] How do you see that tension working between the model and the. It'll[00:36:36] Logan Kilpatrick: be really interesting to see if there's other, if there's other interfaces to this. And I think I've actually seen a lot of people asking, like, it'd be really great if I had ChatGPT and VS code because in, in some sense, like it can, it's just a better, it's a better interface in a lot of ways to like the, the auto complete version cuz you can reprompt and do, and I know Via, I know co-pilot actually has that, where you can like click and then give it, it'll like pop up like 10 suggested.[00:36:59] Different options instead of brushes. Yeah, copilot labs, yeah. Instead of the one that it's providing. And I really like that interface, but again, this goes back to. I, I do inherently think it'll get better. I think it'll be able to do a lot, a lot more of the stuff as the models get bigger, as they have longer context as they, there's a lot of really cool things that will end up coming out and yeah, I don't think it's actually very far away from being like, much, much better.[00:37:24] It'll go from the junior engineer to like the, the principal engineer probably pretty quickly. Like I, I don't think the gap is, is really that large between where things are right now. I think like getting it to the point. 60% of the stuff really well to get it to do like 90% of the stuff really well is like that's within reach in the next, in the next couple of years.[00:37:45] So I'll be really excited to see, and hopefully again, this goes back to like engineers and developers and people who aren't thinking about how to integrate. These tools, whether it's ChatGPT or co-pilot or something else into their workflows to be more efficient. Those are the people who I think will end up getting disrupted by these tools.[00:38:02] So figuring out how to make yourself more valuable than you are today using these tools, I think will be super important for people. Yeah.[00:38:09] Alessio Fanelli: Actually use ChatGPT to debug, like a react hook the other day. And then I posted in our disc and I was like, Hey guys, like look, look at this thing. It really helped me solve this.[00:38:18] And they. That's like the ugliest code I've ever seen. It's like, why are you doing that now? It's like, I don't know. I'm just trying to get[00:38:24] Logan Kilpatrick: this thing to work and I don't know, react. So I'm like, that's the perfect, exactly, that's the perfect solution. I, I did this the other day where I was looking at React code and like I have very briefly seen React and run it like one time and I was like, explain how this is working.[00:38:38] So, and like change it in this way that I want to, and like it was able to do that flawlessly and then I just popped it in. It worked exactly like I. I'll give a[00:38:45] swyx: little bit more context cause I was, I was the guy giving you feedback on your code and I think this is a illustrative of how large language models can sort of be more confident than they should be because you asked it a question which is very specific on how to improve your code or fix your code.[00:39:00] Whereas a real engineer would've said, we've looked at your code and go, why are you doing it at at all? Right? So there's a sort of sycophantic property of martial language. Accepts the basis of your question, whereas a real human might question your question. Mm-hmm. , and it was just not able to do that. I mean, I, I don't see how he could do that.[00:39:17] Logan Kilpatrick: Yeah. It's, it's interesting. I, I saw another example of this the other day as well with some chatty b t prompt and I, I agree. It'll be interesting to see if, and again, I think not to, not to go back to Sam's, to Sam's talk again, but like, he, he talked real about this, and I think this makes a ton of sense, which is like you should be able to have, and this isn't something that that exists right now, but you should be able to have the model.[00:39:39] Tuned in the way that you wanna interact with. Like if you want a model that sort of questions what you're asking it to do, like you should be able to have that. And I actually don't think that that's as far away as like some of the other stuff. Um, It, it's a very possible engineering problem to like have the, to tune the models in that way and, and ask clarifying questions, which is even something that it doesn't do right now.[00:39:59] It'll either give you the response or it won't give you the response, but it'll never say like, Hey, what do you mean by this? Which is super interesting cuz that's like we spend as humans, like 50% of our conversational time being like, what do you mean by that? Like, can you explain more? Can you say it in a different way?[00:40:14] And it's, it's fascinating that the model doesn't do that right now. It's, it's interesting.[00:40:20] swyx: I have written a piece on sort of what AGI hard might be, which is the term that is being thrown around as like a layer of boundary for what is, what requires an A real AGI to do and what, where you might sort of asymptotically approach.[00:40:33] So, What people talk about is essentially a theory of mind, developing a con conception of who I'm talking to and persisting that across sessions, which essentially ChatGPT or you know, any, any interface that you build on top of GPT3 right now would not be able to do. Right? Like, you're not persisting you, you are persisting that history, but you don't, you're not building up a conception of what you know and what.[00:40:54] I should fill in the blanks for you or where I should question you. And I think that's like the hard thing to understand, which is what will it take to get there? Because I think that to me is the, going back to your education thing, that is the biggest barrier, which is I, the language model doesn't have a memory or understanding of what I know.[00:41:11] and like, it's, it's too much to tell them what I don't know. Mm-hmm. , there's more that I don't know than I, than I do know . I think the cool[00:41:16] Logan Kilpatrick: part will be when, when you're able to, like, imagine you could upload all of the, the stuff that you've ever done, all the texts, the work that you've ever done before, and.[00:41:27] The model can start to understand, hey, what are the, what are the conceptual gaps that this person has based on what you've said, based on what you've done? I think that would be really interesting. Like if you can, like I have good notes on my phone and I can still go back to see all of the calculus classes that I took and I could put in all my calculus notebooks and all the assignments and stuff that I did in, in undergrad and grad school, and.[00:41:50] basically be like, Hey, here are the gaps in your understanding of calculus. Go and do this right now. And I think that that's in the education space. That's exactly what will end up happening. You'll be able to put in all this, all the work that you've done. It can understand those ask and then come up with custom made questions and prompts and be like, Hey, how, you know, explain this concept to me and if it.[00:42:09] If you can't do that, then it can sort of put that into your curriculum. I think like Khan Academy as an example, already does some of this, like personalized learning. You like take assessments at the beginning of every Khan Academy model module, and it'll basically only have you watch the videos and do the assignments for the things that like you didn't test well into.[00:42:27] So that's, it's, it's sort of close to already being there in some sense, but it doesn't have the, the language model interface on top of it before we[00:42:34] swyx: get into our lightning round, which is like, Quick response questions. Was there any other topics that you think you wanted to cover? We didn't touch on, whisper.[00:42:40] We didn't touch on Apple. Anything you wanted to[00:42:42] Logan Kilpatrick: talk?[00:42:43] Apple's Neural Engine[00:42:43] Logan Kilpatrick: Yeah, I think the question around Apple stuff and, and the neural engine, I think will be really interesting to see how it all plays out. I think, I don't know if you wanna like ask just to give the context around the neural engine Apple question. Well, well, the[00:42:54] swyx: only thing I know it's because I've seen Apple keynotes.[00:42:57] Everyone has, you know, I, I have a m M one MacBook Cure. They have some kind of neuro chip. , but like, I don't see it in my day-to-day life, so when is this gonna affect me, essentially? And you worked at Apple, so I I was just gonna throw the question over to you, like, what should we[00:43:11] Logan Kilpatrick: expect out of this? Yeah.[00:43:12] The, the problem that I've seen so far with the neural engine and all the, the Mac, and it's also in the phones as well, is that the actual like, API to sort of talk to the neural engine isn't something that's like a common you like, I'm pretty sure it's either not exposed at all, like it only like Apple basically decides in the software layer Yeah.[00:43:34] When, when it should kick in and when it should be used, which I think doesn't really like help developers and it doesn't, that's why no one is using it. I saw a bunch of, and of course I don't have any good insight on this, but I saw a bunch of rumors that we're talking about, like a lot of. Main use cases for the neural engine stuff.[00:43:50] It's, it's basically just in like phantom mode. Now, I'm sure it's doing some processing, but like the main use cases will be a lot of the ar vr stuff that ends up coming out and like when it gets much heavier processing on like. Graphic stuff and doing all that computation, that's where it'll be. It'll be super important.[00:44:06] And they've basically been able to trial this for the last, like six years and have it part of everything and make sure that they can do it cheaply in a cost effective way. And so it'll be cool to see when that I'm, I hope it comes out. That'll be awesome.[00:44:17] swyx: Classic Apple, right? They, they're not gonna be first, but when they do it, they'll make a lot of noise about it.[00:44:21] Yeah. . It'll be[00:44:22] Logan Kilpatrick: awesome. Sure.[00:44:22] Lightning Round[00:44:22] Logan Kilpatrick: So, so are we going to light. Let's[00:44:24] Alessio Fanelli: do it. All right. Favorite AI products not[00:44:28] Logan Kilpatrick: open AI. Build . I think synthesis. Is synthesis.io is the, yeah, you can basically put in like a text prompt and they have like a human avatar that will like speak and you can basically make content in like educational videos.[00:44:44] And I think that's so cool because maybe as people who are making content, like it's, it's super hard to like record video. It just takes a long time. Like you have to edit all the stuff, make sure you sound right, and then when you edit yourself talking it's super weird cuz your mouth is there and things.[00:44:57] So having that and just being able to ChatGPT A script. Put it in. Hopefully I saw another demo of like somebody generating like slides automatically using some open AI stuff. Like I think that type of stuff. Chat, BCG, ,[00:45:10] swyx: a fantastic name, best name of all time .[00:45:14] Logan Kilpatrick: I think that'll be cool. So I'm super excited,[00:45:16] swyx: but Okay.[00:45:16] Well, so just a follow up question on, on that, because we're both in that sort of Devrel business, would you put AI Logan on your video, on your videos and a hundred[00:45:23] Logan Kilpatrick: percent, explain that . A hundred percent. I would, because again, if it reduces the time for me, like. I am already busy doing a bunch of other stuff,[00:45:31] And if I could, if I could take, like, I think the real use case is like I've made, and this is in the sense of like creators wanting to be on every platform. If I could take, you know, the blog posts that I wrote and then have AI break it up into a bunch of things, have ai Logan. Make a TikTok, make a YouTube video.[00:45:48] I cannot wait for that. That's gonna be so nice. And I think there's probably companies who are already thinking about doing that. I'm just[00:45:53] swyx: worried cuz like people have this uncanny valley reaction to like, oh, you didn't tell me what I just watched was a AI generated thing. I hate you. Now you know there, there's a little bit of ethics there and I'm at the disclaimer,[00:46:04] Logan Kilpatrick: at the top.[00:46:04] Navigating. Yeah. I also think people will, people will build brands where like their whole thing is like AI content. I really do think there are AI influencers out there. Like[00:46:12] swyx: there are entire Instagram, like million plus follower accounts who don't exist.[00:46:16] Logan Kilpatrick: I, I've seen that with the, the woman who's a Twitch streamer who like has some, like, she's using like some, I don't know, that technology from like movies where you're like wearing like a mask and it like changes your facial appearance and all that stuff.[00:46:27] So I think there's, there's people who find their niche plus it'll become more common. So, cool. My[00:46:32] swyx: question would be, favorite AI people in communities that you wanna shout up?[00:46:37] Logan Kilpatrick: I think there's a bunch of people in the ML ops community where like that seemed to have been like the most exciting. There was a lot of innovation, a lot of cool things happening in the ML op space, and then all the generative AI stuff happened and then all the ML Ops two people got overlooked.[00:46:51] They're like, what's going on here? So hopefully I still think that ML ops and things like that are gonna be super important for like getting machine learning to be where it needs to be for us to. AGI and all that stuff. So a year from[00:47:05] Alessio Fanelli: now, what will people be the most[00:47:06] Logan Kilpatrick: surprised by? N. I think the AI is gonna get very, very personalized very quickly, and I don't think that people have that feeling yet with chat, BT, but I, I think that that's gonna, that's gonna happen and they'll be surprised in like the, the amount of surface areas in which AI is present.[00:47:23] Like right now it's like, it's really exciting cuz Chat BT is like the one place that you can sort of get that cool experience. But I think that, The people at Facebook aren't dumb. The people at Google aren't dumb. Like they're gonna have, they're gonna have those experiences in a lot of different places and I think that'll be super fascinating to see.[00:47:40] swyx: This is for the builders out there. What's an AI thing you would pay for if someone built it with their personal[00:47:45] Logan Kilpatrick: work? I think more stuff around like transfer learning for, like making transfer, learning easier. Like I think that's truly the way to. Build really cool things is transfer learning, fine tuning, and I, I don't think that there's enough.[00:48:04] Jeremy Howard who created Fasted AI talks a lot about this. I mean, it's something that really resonates with me and, and for context, like at Apple, all the machine learning stuff that we did was transfer learning because it was so powerful. And I think people have this perception that they need to.[00:48:18] Build things from scratch and that's not the case. And I think especially as large language models become more accessible, people need to build layers and products on top of this to make transfer learning more accessible to more people. So hopefully somebody builds something like that and we can all train our own models.[00:48:33] I think that's how you get like that personalized AI experiences you put in your stuff. Make transfer learning easy. Everyone wins. Just just to vector in[00:48:40] swyx: a little bit on this. So in the stable diffusion community, there's a lot of practice of like, I'll fine tune a custom dis of stable diffusion and share it.[00:48:48] And then there also, there's also this concept of, well, first it was textual inversion and then dream booth where you essentially train a concept that you can sort of add on. Is that what you're thinking about when you talk about transfer learning or is that something[00:48:59] Logan Kilpatrick: completely. I feel like I'm not as in tune with the generative like image model community as I probably should be.[00:49:07] I, I think that that makes a lot of sense. I think there'll be like whole ecosystems and marketplaces that are sort of built around exactly what you just said, where you can sort of fine tune some of these models in like very specific ways and you can use other people's fine tunes. That'll be interesting to see.[00:49:21] But, c.ai is,[00:49:23] swyx: what's it called? C C I V I Ts. Yeah. It's where people share their stable diffusion checkpoints in concepts and yeah, it's[00:49:30] Logan Kilpatrick: pretty nice. Do you buy them or is it just like free? Like open. Open source? It's, yeah. Cool. Even better.[00:49:34] swyx: I think people might want to sell them. There's a, there's a prompt marketplace.[00:49:38] Prompt base, yeah. Yeah. People hate it. Yeah. They're like, this should be free. It's just text. Come on, .[00:49:45] Alessio Fanelli: Hey, it's knowledge. All right. Last question. If there's one thing you want everyone to take away about ai, what would.[00:49:51] Logan Kilpatrick: I think the AI revolution is gonna, you know, it's been this like story that people have been talking about for the longest time, and I don't think that it's happened.[00:50:01] It was really like, oh, AI's gonna take your job, AI's gonna take your job, et cetera, et cetera. And I think people have sort of like laughed that off for a really long time, which was fair because it wasn't happening. And I think now, Things are going to accelerate very, very quickly. And if you don't have your eyes wide open about what's happening, like there's a good chance that something that you might get left behind.[00:50:21] So I'm, I'm really thinking deeply these days about like how that is going to impact a lot of people. And I, I'm hopeful that the more widespread this technology becomes, the more mainstream this technology becomes, the more people will benefit from it and hopefully not be affected in that, in that negative way.[00:50:35] So use these tools, put them into your workflow, and, and hopefully that will, and that will acceler. Well,[00:50:41] swyx: we're super happy that you're at OpenAI getting this message out there, and I'm sure we'll see a lot more from you in the coming months[00:50:46] Logan Kilpatrick: and years. I'm excited that this was awesome to be on. This is actually the first, my first in-person podcast.[00:50:52] I've done so many Yeah. Virtual podcasts over the, the covid years and it's, it's super fun to be in person and where the headphones in . Yeah.[00:51:00] swyx: We gotta shout out this studio. I mean, let's, let's get them a shout out Pod on[00:51:03] Alessio Fanelli: in San Francisco, California. Where should people find you? Social media.[00:51:08] Logan Kilpatrick: Twitter. It'll be interesting to see how that, the migration or not migration.[00:51:12] I was, I was pretty sold. I'm like everyone was getting off Twitter and then that seemed like that. It sort of was a network. Network effects are hard too. Yeah, it is hard. So Twitter, I'll see you on Twitter. Thanks so much coming. Thanks. Thanks for having me. This was awesome. Thank you, Logan. Get full access to Latent Space at www.latent.space/subscribe