In this special edition of the podcast, Technical Fellow and Microsoft Research AI for Science Director Chris Bishop joins guest host Eliza Strickland in the Microsoft Booth at the 38th annual Conference on Neural Information Processing Systems (NeurIPS) in Vancouver, British Columbia, to talk about deep learning’s potential to improve the speed and scale at which scientific advancements can be made.

Jindong Wang, a researcher, and Steven Euijong Whang, an associate professor at KAIST and co-author of the ERBench paper, dive into the innovative ERBench project designed to evaluate large language models (LLMs). They discuss leveraging relational databases to tackle inaccuracies and enhance response assessments. The duo highlights the importance of integrity constraints in crafting multi-hop questions, as well as the varied performance metrics needed to ensure model trustworthiness, especially in addressing LLM hallucinations.

Next-token prediction trains a language model on all tokens in a sequence. VP Weizhu Chen discusses his team’s 2024 NeurIPS paper on how distinguishing between useful and “noisy” tokens in pretraining can improve token efficiency and model performance.Read the paperGet the code

Pranjal Chitale discusses the 2024 NeurIPS work CVQA. Spanning 31 languages and the cultures of 30 countries, this VQA benchmark was created with native speakers and cultural experts to evaluate model performance across diverse linguistic and cultural contexts.Read the paperGet the dataset

Can existing algorithms designed for simple reinforcement learning problems be used to solve more complex RL problems? Researcher Dylan Foster discusses the modular approach he and his coauthors explored in their 2024 NeurIPS paper on RL under latent dynamics.Read the paper

Nicole Immorlica, Senior Principal Research Manager at Microsoft Research New England, dives into the fascinating intersection of math, economics, and technology. She shares her journey from physics to theoretical economics, revealing how algorithms and AI are reshaping markets. Immorlica discusses the stable marriage problem and its real-world implications, particularly in medical residencies. She also explores the transformative effects of generative AI on creativity and auction design, emphasizing the balance needed for regulation in this evolving landscape.

Bikland Quinn, a principal researcher specializing in chemistry and biotechnology, Jake Smith, an automation expert in DNA synthesis, and Sergei Yakanen, a coding theory specialist, delve into the revolutionary potential of DNA for data storage. They discuss the challenges of converting digital data into DNA sequences and the vital role of error correction to ensure data integrity. Innovations like the open-sourcing of the Trellis BMA code are highlighted, as well as the surprising resilience of DNA against radiation, showcasing the promising future of this groundbreaking technology.

The efficient simulation of molecules has the potential to change how the world understands biological systems and designs new drugs and biomaterials. Tong Wang discusses AI2BMD, an AI-based system designed to simulate large biomolecules with speed and accuracy.Read the paperGet the code

Researcher Siddharth Suri and professor David Holtz give a brief history of prompt engineering, discuss the debate behind their recent collaboration, and share what they found from studying how people’s approaches to prompting change as models advance.Learn more:As Generative Models Improve, People Adapt Their Prompts | Publication, July 2024AI, Cognition, and the Economy (AICE) | Initiative page

Researchers Chris Hawblitzel and Jay Lorch share how progress in programming languages and verification approaches are bringing bug-free software within reach. Their work on the Rust verification tool Verus won the Distinguished Artifact Award at SOSP ’24.Read the paper