Machine Learning Street Talk (MLST) Prof. Melanie Mitchell 2.0 - AI Benchmarks are Broken!
15 snips
Sep 10, 2023 Prof. Melanie Mitchell, Davis Professor of Complexity at the Santa Fe Institute, dives into the murky waters of AI understanding. She argues that current benchmarks are inadequate, as machines often replicate human tasks without true comprehension. Mitchell highlights the limitations of large language models, noting their lack of common sense despite impressive statistical capabilities. She emphasizes the need for evolving evaluation methods and suggests a deeper, context-specific look at intelligence, advocating for more rigorous testing to reflect genuine understanding.
AI Snips
Chapters
Books
Transcript
Episode notes
Nature of Intelligence
- Intelligence is ill-defined and multidimensional, not a simple yes/no.
- AI systems might possess intelligence in specific ways or degrees, rather than absolute intelligence.
Grounded Understanding
- Dilip George argues a university professor understands vectors better than AI.
- This is because their knowledge is grounded in real-world situations, not just abstract concepts.
Understanding in LLMs
- Key question: Is AI understanding a category error, mistaking token associations for real-world connections?
- Or do large language models create concept-based mental models like humans, and does scaling improve them?
