The Nonlinear Library LW - Questions are usually too cheap by Nathan Young
May 12, 2024
10:31
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Questions are usually too cheap, published by Nathan Young on May 12, 2024 on LessWrong.
It is easier to ask than to answer.
That's my whole point.
It is much cheaper to ask questions than answer them so beware of situations where it is implied that asking and answering are equal.
Here are some examples:
Let's say there is a maths game. I get a minute to ask questions. You get a minute to answer them. If you answer them all correctly, you win, if not, I do. Who will win?
Preregister your answer.
Okay, let's try. These questions took me roughly a minute to come up with.
What's 56,789 * 45,387?
What's the integral from -6 to 5π of sin(x cos^2(x))/tan(x^9) dx?
What's the prime factorisation of 91435293173907507525437560876902107167279548147799415693153?
Good luck. If I understand correctly, that last one's gonna take you at least an hour1 (or however long it takes to threaten me).
Perhaps you hate maths. Let's do word problems then.
Define the following words "antidisestablishmentarianism", "equatorial", "sanguine", "sanguinary", "escapology", "eschatology", "antideluvian", "cripuscular", "red", "meter", all the meanings of "do", and "fish".
I don't think anyone could do this without assistance. I tried it with Claude, which plausibly still failed2 the "fish" question, though we'll return to that.
I could do this for almost anything:
Questions on any topic
Certain types of procedural puzzles
Asking for complicated explanations (we'll revisit later)
Forecasting questions
This is the centre of my argument
I see many situations where questions and answers are treated as symmetric. This is rarely the case. Instead, it is much more expensive to answer than to ask.
Let's try and find some counter examples. A calculator can solve allowable questions faster than you can type them in. A dictionary can provide allowable definitions faster than you can look them up. An LLM can sometimes answer some types of questions more cheaply in terms of inference costs than your time was worth in coming up with them.
But then I just have to ask different questions. Calculators and dictionaries are often limited. And even the best calculation programs can't solve prime factorisation questions more cheaply than I can write them. Likewise I could create LLM prompts that are very expensive for the best LLMs to answer well, eg "write a 10,000 word story about an [animal] who experiences [emotion] in a [location]."
How this plays out
Let's go back to our game.
Imagine you are sitting around and I turn up and demand to play the "answering game". Perhaps I reference on your reputation. You call yourself a 'person who knows things', surely you can answer my questions? No? Are you a coward? Looks like you are wrong!
And now you either have to spend your time answering or suffer some kind of social cost and allow me to say "I asked him questions but he never answered". And whatever happens, you are distracted from what you were doing. Whether you were setting up an organisation or making a speech or just trying to have a nice day, now you have to focus on me. That's costly.
This seems like a common bad feature of discourse - someone asking questions cheaply and implying that the person answering them (or who is unable to) should do so just as cheaply and so it is fair. Here are some examples of this:
Internet debates are weaponised cheap questions. Whoever speaks first in many debates often gets to frame the discussion and ask a load of questions and then when inevitably they aren't answered, the implication is that the first speaker is right3. I don't follow American school debate closely, but I sense it is even more of this, with people literally learning to speak faster so their opponents can't process their points quickly enough to respond to them.
Emails. Normally they exist within a framework of f...
It is easier to ask than to answer.
That's my whole point.
It is much cheaper to ask questions than answer them so beware of situations where it is implied that asking and answering are equal.
Here are some examples:
Let's say there is a maths game. I get a minute to ask questions. You get a minute to answer them. If you answer them all correctly, you win, if not, I do. Who will win?
Preregister your answer.
Okay, let's try. These questions took me roughly a minute to come up with.
What's 56,789 * 45,387?
What's the integral from -6 to 5π of sin(x cos^2(x))/tan(x^9) dx?
What's the prime factorisation of 91435293173907507525437560876902107167279548147799415693153?
Good luck. If I understand correctly, that last one's gonna take you at least an hour1 (or however long it takes to threaten me).
Perhaps you hate maths. Let's do word problems then.
Define the following words "antidisestablishmentarianism", "equatorial", "sanguine", "sanguinary", "escapology", "eschatology", "antideluvian", "cripuscular", "red", "meter", all the meanings of "do", and "fish".
I don't think anyone could do this without assistance. I tried it with Claude, which plausibly still failed2 the "fish" question, though we'll return to that.
I could do this for almost anything:
Questions on any topic
Certain types of procedural puzzles
Asking for complicated explanations (we'll revisit later)
Forecasting questions
This is the centre of my argument
I see many situations where questions and answers are treated as symmetric. This is rarely the case. Instead, it is much more expensive to answer than to ask.
Let's try and find some counter examples. A calculator can solve allowable questions faster than you can type them in. A dictionary can provide allowable definitions faster than you can look them up. An LLM can sometimes answer some types of questions more cheaply in terms of inference costs than your time was worth in coming up with them.
But then I just have to ask different questions. Calculators and dictionaries are often limited. And even the best calculation programs can't solve prime factorisation questions more cheaply than I can write them. Likewise I could create LLM prompts that are very expensive for the best LLMs to answer well, eg "write a 10,000 word story about an [animal] who experiences [emotion] in a [location]."
How this plays out
Let's go back to our game.
Imagine you are sitting around and I turn up and demand to play the "answering game". Perhaps I reference on your reputation. You call yourself a 'person who knows things', surely you can answer my questions? No? Are you a coward? Looks like you are wrong!
And now you either have to spend your time answering or suffer some kind of social cost and allow me to say "I asked him questions but he never answered". And whatever happens, you are distracted from what you were doing. Whether you were setting up an organisation or making a speech or just trying to have a nice day, now you have to focus on me. That's costly.
This seems like a common bad feature of discourse - someone asking questions cheaply and implying that the person answering them (or who is unable to) should do so just as cheaply and so it is fair. Here are some examples of this:
Internet debates are weaponised cheap questions. Whoever speaks first in many debates often gets to frame the discussion and ask a load of questions and then when inevitably they aren't answered, the implication is that the first speaker is right3. I don't follow American school debate closely, but I sense it is even more of this, with people literally learning to speak faster so their opponents can't process their points quickly enough to respond to them.
Emails. Normally they exist within a framework of f...