LessWrong (30+ Karma) “Condensation” by abramdemski
Nov 9, 2025
Dive into a fascinating theory of concept formation with insights on Shannon's information theory and universal codes. Explore how encodings can be optimized for usability rather than just raw compression. The idea of a universal data structure emerges, enhancing retrieval efficiency with clever analogies. Discover the relationship between latent variables and givens, along with the implications for shared understanding among agents. The discussion also touches on model interpretability and potential research directions that could reshape our understanding of information processing.
AI Snips
Chapters
Books
Transcript
Episode notes
Encode For Questions Not Compression
- Condensation prioritizes making encodings useful for answering queries rather than minimizing total code length.
- This forces encodings to form conceptually meaningful latent variables instead of an uninterpretable compressed blob.
Condensation As Universal Data Structure
- Treat condensation as a universal data structure that organizes data into tagged bins for efficient retrieval.
- Retrieval cost matters while storage cost is assumed negligible in Sam's best-case setup.
Information Desk Notebook Example
- Abram frames condensation via an information desk with clerks and reference notebooks tagged by questions.
- Clerks retrieve all notebooks tagged with any asked questions and retrieval cost equals total text retrieved.
