LessWrong (Curated & Popular) “Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals” by johnswentworth, David Lorell
Jan 27, 2025
Join johnswentworth, co-author on instrumental goals, as he explores the fascinating distinction between terminal and instrumental goals using the whimsical example of baking a chocolate cake. He explains how acquiring cocoa powder represents an instrumental goal that serves a greater purpose. The conversation dives into the complexities of coordination among chefs in a restaurant and the importance of corrigibility in achieving shared objectives. It's a light-hearted yet insightful look at goal-setting and collaboration!
AI Snips
Chapters
Transcript
Episode notes
The Cake Analogy
- John Wentworth uses the example of baking a cake to illustrate the difference between terminal and instrumental goals.
- If baking the cake is the only goal, then ruining the oven afterward doesn't matter.
Terminal vs. Instrumental Goals
- Terminal goals are about the end result, while instrumental goals are about the steps needed to achieve a terminal goal.
- Acquiring cocoa powder is instrumental to the terminal goal of baking a cake.
The Restaurant Analogy
- The restaurant analogy expands the concept of instrumental goals to a multi-agent setting.
- A chef's instrumental goal of baking a cake includes constraints like not wasting ingredients needed by other chefs.