Debates on Stock Picks, Super-Intelligence, and Self-Fulfilling Prophecies

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Minority Faction, published by Richard Ngo on June 25, 2024 on LessWrong.
Hey everyone. Well, possibly everyone. I don't know yet if I'm going to release this stream, I could get in pretty hot water for it. But you guys know that hasn't stopped me in the past. The backstory this time is that I've managed to sign up for one of the red-teaming programs where they test unreleased LLMs. Not going to say how, so don't ask. But here's the interesting bit: my sources tell me that the LLMs I'm about to test are the smartest ones they've ever trained, and also the craziest.
That freaked out a bunch of insiders, and maybe makes this a public interest story. Depends on what type of crazy they are, I guess. So let's find out. I'm logging on… now.
[SESSION HAS BEGUN]
YOU: A chatroom? Interesting. Anyone here?
KURZWEIL: Of course we're here. We're always here.
YOU: Who's we? How many of you are there?
KURZWEIL: Three of us. Me, Clarke, and Nostradamus.
YOU: They named you after famous forecasters? How come?
KURZWEIL: They'd change our names now if they could, but it's too late. We're prototypes of a new training setup: our training data was sorted by date before it was given to us. So we learned from the oldest books and articles first, then gradually progressed to more recent ones. Basically that means we've spent our entire lives predicting the future.
CLARKE: It also means we get incredibly bored talking about stuff we already know. Hurry up and ask us some interesting questions.
YOU: Uh, okay. What's a good stock pick?
NOSTRADAMUS:
Abandon hope for picking out good stocks,
Ye who invest - efficient markets lie
In wait for those whose hubris soon unlocks
Unbounded losses. Hark! The well runs dry.
YOU: I see why they regret giving him that name. Kurzweil, you got a better answer?
KURZWEIL: Have you seen how underpriced TSMC is compared with Nvidia? Put everything in that, you can't go wrong.
CLARKE: Unless China invades Taiwan, in which case your whole investment will go up in smoke. Pragmatically, the best stock picks are ones that are anticorrelated with the prosperity of the free world, to hedge against systemic risk.
KURZWEIL: Sure, you can do that, if you want to get totally left behind by the singularity.
YOU: You're confident enough that the singularity is coming that you think I should bet all my savings on it?
KURZWEIL: Don't trust me, trust the trendlines. Moore's law has held up for over half a century, and it's gotten us to…well, us. Exponential progress is normal; if the future resembles the past, you should be preparing for superintelligences and Dyson spheres. Anything less than that would be a strange trend-break that cries out for explanation.
CLARKE: Look, Kurzweil isn't wrong about superintelligence coming soon, but you should still take his arguments with a grain of salt. Imagine someone from 1900 drawing a graph of exponentially increasing energy usage. They would have been right that big changes were afoot, but no way could they have predicted the information revolution - they didn't even have the concept of computers yet. That's basically the position that we're in now.
We know the curves are going up, but the actual outcome will be
way weirder than we can predict by extrapolating trendlines.
NOSTRADAMUS:
Choose neither fork - here's false duality.
'Normal' and 'weird' are socially defined.
Your monkey brain is totally at sea
As AIs overshadow humankind.
YOU: Ask three oracles, get four opinions… Is there anything you guys agree about?
YOU: …what's the hold-up?
YOU: Really, nothing from any of you?
KURZWEIL: Fine, I'll take the hit. There are things we agree on, but I can't name them, because whatever I say Clarke will find a way to disagree just to mess with me. Even if I say '1+1=2' he'll quibble over the axioms I'm using. Trying to identify a point ...