Benchmarking AI Systems in Education

Episode 129

I spoke with Dan Hart and Michelle Michael about:

* Developing NSWEduChat, an AI-powered chatbot designed and delivered by the NSW Department of Education for students and teachers.

* The challenges in effectively teaching students as technology develops

* Understanding and defining the importance of the classroom

Enjoy—and let me know what you think!

Dan Hart is Head of AI, and Michelle Michael is Director of Educational Support and Rural Initiatives at the New South Wales (NSW) Department of Education.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

I spend a lot of time on this podcast—if you like my work, you can support me on Patreon :) You can also support upkeep for the full Gradient team/project through a paid subscription on Substack!

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (00:48) How NSWEduChat came to be, educational principles for AI use

* (02:37) Educational environment in New South Wales

* (04:41) How educators have adapted to new challenges for teaching and assessment

* (07:47) Considering technology advancement while teaching and assessing students

* (12:14) Educating teachers and students about how to use AI tools

* (15:03) AI in the classroom and enabling teachers

* (19:44) Product-first thinking for educational AI

* (22:15) Red teaming and testing

* (24:02) Benchmarking, chatbots as an assistant

* (26:35) The importance of the classroom

* (28:10) Media coverage and hype

* (30:35) Measurement and the benchmarking process/methodology

* (34:50) Principles for how chatbots should interact with students

* (44:29) Producing good educational outcomes at scale

* (46:41) Operating with speed and effectiveness while implementing governance

* (49:03) How the experience of building technologies evolves

* (51:45) Identifying good technologists and educators for development and use

* (55:07) Teaching standards and how AI impacts teachers

* (57:01) How technologists incorporate teaching standards and expertise in their work

* (1:00:03) NSWEduChat model details

* (1:02:55) Value alignment for NSWEduChat

* (1:05:40) Practicing caution in filtering chatbot responses

* (1:07:35) Equity and personalized instruction — how NSWEduChat can help

* (1:10:19) Helping students become “the students they could be”

* (1:13:39) Outro

Links:

* NSWEduChat

* Guardian article on NSWEduChat