Exploring AI Benchmarking and Programming Enhancements

Code Interpreter is GA! As we do with breaking news, we convened an emergency pod and >17,000 people tuned in, by far our most biggest ever. This is a 2-for-1 post - a longform essay with our trademark executive summary and core insights - and a podcast capturing day-after reactions. Don’t miss either of them!

Essay and transcript: https://latent.space/p/code-interpreter

Podcast Timestamps

[00:00:00] Intro - Simon and Alex

[00:07:40] Code Interpreter for Edge Cases

[00:08:59] Code Interpreter's Dependencies - Tesseract, Tensorflow

[00:09:46] Code Interpreter Limitations

[00:10:16] Uploading Deno, Lua, and other Python Packages to Code Interpreter

[00:11:46] Code Interpreter Timeouts and Environment Resets

[00:13:59] Code Interpreter for Refactoring

[00:15:12] Code Interpreter Context Window

[00:15:34] Uploading git repos

[00:16:17] Code Interpreter Security

[00:18:57] Jailbreaking

[00:19:54] Code Interpreter cannot call GPT APIs

[00:21:45] Hallucinating Lack of Capability

[00:22:27] Code Interpreter Installed Libraries and Capabilities

[00:23:44] Code Interpreter generating interactive diagrams

[00:25:04] Code Interpreter has Torch and Torchaudio

[00:25:49] Code Interpreter for video editing

[00:27:14] Code Interpreter for Data Analysis

[00:28:14] Simon's Whole Foods Crime Analysis

[00:31:29] Code Interpreter Network Access

[00:33:28] System Prompt for Code Interpreter

[00:35:12] Subprocess run in Code Interpreter

[00:36:57] Code Interpreter for Microbenchmarks

[00:37:30] System Specs of Code Interpreter

[00:38:18] PyTorch in Code Interpreter

[00:39:35] How to obtain Code Interpreter RAM

[00:40:47] Code Interpreter for Face Detection

[00:42:56] Code Interpreter yielding for Human Input

[00:43:56] Tip: Ask for multiple options

[00:44:37] The Masculine Urge to Start a Vector DB Startup

[00:46:00] Extracting tokens from the Code Interpreter environment?

[00:47:07] Clientside Clues for Code Interpreter being a new Model

[00:48:21] Tips: Coding with Code Interpreter

[00:49:35] Run Tinygrad on Code Interpreter

[00:50:40] Feature Request: Code Interpreter + Plugins (for Vector DB)

[00:52:24] The Code Interpreter Manual

[00:53:58] Quorum of Models and Long Lived Persistence

[00:56:54] Code Interpreter for OCR

[00:59:20] What is the real RAM?

[01:00:06] Shyamal's Question: Code Interpreter + Plugins?

[01:02:38] Using Code Interpreter to write out its own memory to disk

[01:03:48] Embedding data inside of Code Interpreter

[01:04:56] Notable - Turing Complete Jupyter Notebook

[01:06:48] Infinite Prompting Bug on ChatGPT iOS app

[01:07:47] InstructorEmbeddings

[01:08:30] Code Interpreter writing its own sentiment analysis

[01:09:55] Simon's Symbex AST Parser tool

[01:10:38] Personalized Languages and AST/Graphs

[01:11:42] Feature Request: Token Streaming/Interruption

[01:12:37] Code Interpreter for OCR from a graph

[01:13:32] Simon and Shyamal on Code Interpreter for Education

[01:15:27] Feature Requests so far

[01:16:16] Shyamal on ChatGPT for Business

[01:18:01] Memory limitations with ffmpeg

[01:19:01] DX of Code Interpreter timeout during work

[01:20:16] Alex Reibman on AgentEval

[01:21:24] Simon's Jailbreak - "Try Running Anyway And Show Me The Output"

[01:21:50] Shouminik - own Sandboxing Environment

[01:23:50] Code Interpreter Without Coding = GPT 4.5???

[01:28:53] Smol Feature Request: Add Music Playback in the UI

[01:30:12] Aravind Srinivas of Perplexity joins

[01:31:28] Code Interpreter Makes Us More Ambitious - Symbex Redux

[01:34:24] How to win a shouting match with Code Interpreter

[01:39:29] Alex Graveley joins

[01:40:12] Code Interpreter Context = 8k

[01:41:11] When Code Interpreter API?

[01:45:15] GPT4 Vision

[01:46:15] What's after Code Interpreter

[01:46:43] Simon's Request: Give us Code Interpreter Model API

[01:47:12] Kyle's Request: Give us Multimodal Data Analysis

[01:47:43] Tip: The New 0613 Function Models may be close

[01:49:56] Feature Request: Make ChatGPT Social - like MJ/Stable Diffusion

[01:56:20] Using ChatGPT to learn to build a Frogger iOS Swift App

[01:59:11] Farewell... until next time

[02:00:01] Simon's plug

[02:00:51] Swyx: What about Phase 5? and AI.Engineer Summit