The Nonlinear Library: LessWrong LW - Building intuition with spaced repetition systems by Jacob G-W
May 14, 2024
06:25
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Building intuition with spaced repetition systems, published by Jacob G-W on May 14, 2024 on LessWrong.
Do you ever go to a lecture, follow it thinking it makes total sense, then look back at your notes later and realize it makes no sense? This used to happen to me, but I've learned how to use spaced repetition to fully avoid this if I want. I'm going to try to convey this method in this post.
Much of my understanding of how to create flashcards comes from "Using spaced repetition systems to see through a piece of mathematics" by Michael Nielsen and "How to write good prompts: using spaced repetition to create understanding" by Andy Matuschak, but I think my method falls in between both, in terms of abstraction. Finally, I want to credit Quantum Country for being an amazing example of flashcards created to develop intuition in users.
My method is more abstract than Michael Nielsen's approach, since it does not only apply to mathematics, but to any subject. Yet it is less abstract than Andy Matuschak's approach because I specifically use it for 'academic subjects' that require deep intuition of (causal or other) relationships between concepts.
Many of Matuschak's principles in his essay apply here (I want to make sure to give him credit), but I'm looking at it through the 'how can we develop deep intuition in an academic subject in the fastest possible time?' lens.
Minimize Inferential Distance on Flashcards
A method that I like to repeat to myself while making flashcards that I haven't seen in other places is that each flashcard should only have one inferential step on it. I'm using 'inferential step' here to mean a step such as remembering a fact, making a logical deduction, visualizing something, or anything that requires thinking. It's necessary that a flashcard only have a single inferential step on it. Anki trains the mind to do these steps.
If you learn all the inferential steps, you will be able to fully re-create any mathematical deduction, historical story, or scientific argument. Knowing (and continually remembering) the full story with spaced repetition builds intuition.
I'm going to illustrate this point by sharing some flashcards that I made while trying to understand how Transformers (GPT-2) worked. I made these flashcards while implementing a transformer based on Neel Nanda's tutorials and these two blog posts.
Understanding Attention
The first step in my method is to learn or read enough so that you have part of the whole loaded into your head. For me, this looked like picking the
attention step of a transformer and then reading about it in the two blog posts and watching the section of the video on it. It's really important to learn about something from multiple perspectives. Even when I'm making flashcards from a lecture, I have my web browser open and I'm looking up things that I thought were confusing while making flashcards.
My next step is to understand that intuition is fake! Really good resources make you feel like you understand something, but to actually understand something, you need to engage with it. This engagement can take many forms. For technical topics, it usually looks like solving problems or coding, and this is good! I did this for transformers! But I also wanted to not forget it long term, so I used spaced repetition to cement my intuition.
Enough talk, here are some flashcards about attention in a transformer. For each flashcard, I'll explain why I made it. Feel free to scroll through.
Examples
I start with a distillation of the key points of the article.
I wanted to make sure that I knew what the attention operation was actually doing, as the blog posts emphasized this.
When building intuition, I find it helpful to know "the shape" or constraints about something so that I can build a more accurate mental model. In this case, th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Building intuition with spaced repetition systems, published by Jacob G-W on May 14, 2024 on LessWrong.
Do you ever go to a lecture, follow it thinking it makes total sense, then look back at your notes later and realize it makes no sense? This used to happen to me, but I've learned how to use spaced repetition to fully avoid this if I want. I'm going to try to convey this method in this post.
Much of my understanding of how to create flashcards comes from "Using spaced repetition systems to see through a piece of mathematics" by Michael Nielsen and "How to write good prompts: using spaced repetition to create understanding" by Andy Matuschak, but I think my method falls in between both, in terms of abstraction. Finally, I want to credit Quantum Country for being an amazing example of flashcards created to develop intuition in users.
My method is more abstract than Michael Nielsen's approach, since it does not only apply to mathematics, but to any subject. Yet it is less abstract than Andy Matuschak's approach because I specifically use it for 'academic subjects' that require deep intuition of (causal or other) relationships between concepts.
Many of Matuschak's principles in his essay apply here (I want to make sure to give him credit), but I'm looking at it through the 'how can we develop deep intuition in an academic subject in the fastest possible time?' lens.
Minimize Inferential Distance on Flashcards
A method that I like to repeat to myself while making flashcards that I haven't seen in other places is that each flashcard should only have one inferential step on it. I'm using 'inferential step' here to mean a step such as remembering a fact, making a logical deduction, visualizing something, or anything that requires thinking. It's necessary that a flashcard only have a single inferential step on it. Anki trains the mind to do these steps.
If you learn all the inferential steps, you will be able to fully re-create any mathematical deduction, historical story, or scientific argument. Knowing (and continually remembering) the full story with spaced repetition builds intuition.
I'm going to illustrate this point by sharing some flashcards that I made while trying to understand how Transformers (GPT-2) worked. I made these flashcards while implementing a transformer based on Neel Nanda's tutorials and these two blog posts.
Understanding Attention
The first step in my method is to learn or read enough so that you have part of the whole loaded into your head. For me, this looked like picking the
attention step of a transformer and then reading about it in the two blog posts and watching the section of the video on it. It's really important to learn about something from multiple perspectives. Even when I'm making flashcards from a lecture, I have my web browser open and I'm looking up things that I thought were confusing while making flashcards.
My next step is to understand that intuition is fake! Really good resources make you feel like you understand something, but to actually understand something, you need to engage with it. This engagement can take many forms. For technical topics, it usually looks like solving problems or coding, and this is good! I did this for transformers! But I also wanted to not forget it long term, so I used spaced repetition to cement my intuition.
Enough talk, here are some flashcards about attention in a transformer. For each flashcard, I'll explain why I made it. Feel free to scroll through.
Examples
I start with a distillation of the key points of the article.
I wanted to make sure that I knew what the attention operation was actually doing, as the blog posts emphasized this.
When building intuition, I find it helpful to know "the shape" or constraints about something so that I can build a more accurate mental model. In this case, th...