Artificial General Intelligence (AGI) Show with Soroush Pour

Ep 10 - Accelerated training to become an AI safety researcher w/ Ryan Kidd (Co-Director, MATS)

Nov 8, 2023

Ryan Kidd, Co-Director at MATS, discusses the accelerated training program for AI safety researchers, research directions, alignment gaps, and the importance of ethical decision-making in AI safety. They explore quantum decoherence, open-source innovation trade-offs, AI model training, aligning models, dangers of sycophancy, and human feedback mechanisms in AI safety research.

01:16:58

Episode guests

Ryan Kidd

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

MATS program trains AI safety researchers to align AI systems with human values for a safer future.

Research directions in MATS include scalable oversight strategies and integrating AI systems into human processes for efficiency.

Understanding misalignment patterns, agent structures, and model capabilities is crucial for preemptively mitigating risks in AI models.

Deep dives

MATTS Program Overview

The MATS program, ML Alignment and Theory Scholars Program, aims to train researchers in AI safety, emphasizing the significance of understanding and steering advanced AI systems for a safer future. By offering seminars, mentorship, and research opportunities, MATS focuses on aligning AI systems with human values to mitigate existential risks and ensure a positive future for humanity.

Introduction

2min

Navigating AI Safety Risks and Opportunities

13min

Exploring Quantum Decoherence and Neural Process Modeling in AI

3min

Balancing Open-source Innovation and AI Safety Concerns

22min

Accelerated AI Safety Research Program Overview

7min

AI Training Models and Future Research Directions

4min

Enhancing AI Training Signals and Aligning Models

4min

Exploring the Dangers of Sycophancy in AI Models

2min

Exploring Human Feedback Mechanisms in AI Safety Research

20min

We speak with Ryan Kidd, Co-Director at ML Alignment & Theory Scholars (MATS) program, previously "SERI MATS".

MATS (https://www.matsprogram.org/) provides research mentorship, technical seminars, and connections to help new AI researchers get established and start producing impactful research towards AI safety & alignment.

Prior to MATS, Ryan completed a PhD in Physics at the University of Queensland (UQ) in Australia.

We talk about:

* What the MATS program is
* Who should apply to MATS (next *deadline*: Nov 17 midnight PT)
* Research directions being explored by MATS mentors, now and in the past
* Promising alignment research directions & ecosystem gaps , in Ryan's view

Hosted by Soroush Pour. Follow me for more AGI content:
* Twitter: https://twitter.com/soroushjp
* LinkedIn: https://www.linkedin.com/in/soroushjp/

== Show links ==

-- About Ryan --

* Twitter: https://twitter.com/ryan_kidd44
* LinkedIn: https://www.linkedin.com/in/ryan-kidd-1b0574a3/
* MATS: https://www.matsprogram.org/
* LISA: https://www.safeai.org.uk/
* Manifold: https://manifold.markets/

-- Further resources --

* Book: “The Precipice” - https://theprecipice.com/
* Ikigai - https://en.wikipedia.org/wiki/Ikigai
* Fermi paradox - https://en.wikipedia.org/wiki/Fermi_p...
* Ajeya Contra - Bioanchors - https://www.cold-takes.com/forecastin...
* Chomsky hierarchy & LLM transformers paper + external memory - https://en.wikipedia.org/wiki/Chomsky...
* AutoGPT - https://en.wikipedia.org/wiki/Auto-GPT
* BabyAGI - https://github.com/yoheinakajima/babyagi
* Unilateralist's curse - https://forum.effectivealtruism.org/t...
* Jeffrey Ladish & team - fine tuning to remove LLM safeguards - https://www.alignmentforum.org/posts/...
* Epoch AI trends - https://epochai.org/trends
* The demon "Moloch" - https://slatestarcodex.com/2014/07/30...
* AI safety fundamentals course - https://aisafetyfundamentals.com/
* Anthropic sycophancy paper - https://www.anthropic.com/index/towar...
* Promising technical alignment research directions
* Scalable oversight
* Recursive reward modelling - https://deepmindsafetyresearch.medium...
* RLHF - could work for a while, but unlikely forever as we scale
* Interpretability
* Mechanistic interpretability
* Paper: GPT4 labelling GPT2 - https://openai.com/research/language-...
* Concept based interpretability
* Rome paper - https://rome.baulab.info/
* Developmental interpretability
* devinterp.com - http://devinterp.com
* Timaeus - https://timaeus.co/
* Internal consistency
* Colin Burns research - https://arxiv.org/abs/2212.03827
* Threat modelling / capabilities evaluation & demos
* Paper: Can large language models democratize access to dual-use biotechnology? - https://arxiv.org/abs/2306.03809
* ARC Evals - https://evals.alignment.org/
* Palisade Research - https://palisaderesearch.org/
* Paper: Situational awareness with Owain Evans - https://arxiv.org/abs/2309.00667
* Gradient hacking - https://www.lesswrong.com/posts/uXH4r6MmKPedk8rMA/gradient-hacking
* Past scholar's work
* Apollo Research - https://www.apolloresearch.ai/
* Leap Labs - https://www.leap-labs.com/
* Timaeus - https://timaeus.co/
* Other orgs mentioned
* Redwood Research - https://redwoodresearch.org/

Recorded Oct 25, 2023

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more