Mystery AI Hype Theater 3000

Episode 7: There Are Now 15 Competing Evaluation Metrics (ft. Dr. Jeremy Kahn). December 12, 2022

Jul 26, 2023

01:03:14

AI Chapters

Episode notes

Introduction

2min

Jeremy Kahn on Streaming and Connectivity

2min

How to Evaluate Natural Language Processing in Large Language Models

4min

The GISH Galap: A Rhetorical Style That Says Nothing

4min

The Role of Scenario in AI Culture

5min

The Importance of Scenario in Language Design

5min

Building a Generative Model for Disinformation

2min

The Importance of Empowering Disinformation

4min

The Importance of Human Evaluation in Summarization

4min

10.

The Importance of Dialect in User-Phasing Applications

3min

11.

The Future of Linguistics

3min

12.

The Importance of Linguistic Knowledge

3min

13.

The Resurgence of Old School Linguistics Tools

3min

14.

The Importance of Fresh AI Hell

2min

15.

The Fresh AI Hell

2min

16.

The Uncanny Valley of Beauty

2min

17.

How to Use Language Models in Legal Context

2min

18.

The Nile Service Attack on Public Comment Periods

3min

19.

The Future of Legal Workflows

5min

20.

The Importance of Learning to Write Essays

5min

Emily and Alex are joined by Dr. Jeremy G. Kahn to discuss the distressingly large number of evaluation metrics for artificial intelligence, and some new AI hell.

Jeremy G. Kahn has a PhD in computational linguistics, with a focus on information-theoretic and empirical engineering approaches to dealing with natural language (in text and speech). He’s gregarious, polyglot, a semi-auto-didact, and occasionally prolix. He also likes comic books, coffee, progressive politics, information theory, lateral thinking, science fiction, science fact, linear thinking, bicycles, beer, meditation, love, play, and inquiry. He lives in Seattle with his wife Dorothy and son Elliott.

This episode was recorded on December 12, 2022.

Watch the video of this episode on PeerTube.

References:

XKCD: Standards

WikidataCon

Gish Gallop

The Bender Rule

DJ Khaled - You Played Yourself

Jeff Kao's interrogation of public comment periods.

Emily's blog post response to NYT piece

Check out future streams at on Twitch, Meanwhile, send us any AI Hell you see.

Our book, 'The AI Con,' comes out in May! Pre-order now.

Subscribe to our newsletter via Buttondown.