Astral Codex Ten Podcast

https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers A Machine Alignment Monday post, 4/11/22 I. Our goal here is to popularize obscure and hard-to-understand areas of AI alignment, and surely this meme (retweeted by Eliezer last week) qualifies: So let's try to understand the incomprehensible meme! Our main source will be Hubinger et al 2019, Risks From Learned Optimization In Advanced Machine Learning Systems. Mesa- is a Greek prefix which means the opposite of meta-. To "go meta" is to go one level up; to "go mesa" is to go one level down (nobody has ever actually used this expression, sorry). So a mesa-optimizer is an optimizer one level down from you. Consider evolution, optimizing the fitness of animals. For a long time, it did so very mechanically, inserting behaviors like "use this cell to detect light, then grow toward the light" or "if something has a red dot on its back, it might be a female of your species, you should mate with it". As animals became more complicated, they started to do some of the work themselves. Evolution gave them drives, like hunger and lust, and the animals figured out ways to achieve those drives in their current situation. Evolution didn't mechanically instill the behavior of opening my fridge and eating a Swiss Cheese slice. It instilled the hunger drive, and I figured out that the best way to satisfy it was to open my fridge and eat cheese.

https://astralcodexten.substack.com/p/spring-meetups-in-seventy-cities Lots of people only want to go to meetups a few times a year. And they all want to go to the same big meetups as all the other people who only go a few times a year. In 2022, we set up one big well-telegraphed meetup in the fall as a Schelling point for these people. This year, we're setting up two. We'll have the fall meetup as usual. If you only want to go to one meetup a year, go to that one. But we'll also have a spring round. If you only go to two meetups a year, come to this one too! You can find a list of cities and times below. If you want to add your city to the list, fill in this form; if you have questions, ask meetupsmingyuan@gmail.com .

https://astralcodexten.substack.com/p/dictator-book-club-xi-jinping [Previous entries: Erdogan, Modi, Orban] The Third Revolution, by Elizabeth Economy, promises to explain "the transformative changes underway in China today". But like her namesake, Dr. Economy doesn't always allocate resources the way I would like. I came to the book with questions like: How did the pre-Xi Chinese government work? How was it different from dictatorship? What safeguards did it have against it? Why hadn't previous Chinese leaders become dictators? And: How did Xi come to power? How did he defeat those safeguards? Had previous Chinese leaders wanted more power? How come they failed to get it, but Xi succeeded? Third Revolution barely touched on any of this. It mostly explained Xi's domestic and foreign policies. Some of this was relevant: a lot of Xi's policies involve repression to prop up his rule. But none of it answered my key questions. So this is less of a book review than other Dictator Book Club entries. It's a look through recent Chinese history, with The Third Revolution as a very loose inspiration.

https://astralcodexten.substack.com/p/highlights-from-the-comments-on-self 1: Rosemary (writes Parallel Republic) says: I think a preference for the status quo has to weigh in to some extent. All else being equal, sure, I agree with the "any group large enough that it isn't ludicrous on its face has a right to self-determination" standard. But all else is almost never equal. Someone wants to secede and someone else wants to conquer—and all of that is enormously disruptive to many other someones. So I think there's an immediately obvious utilitarian bias towards the status quo of, oh, the last decade or so. Governments are heavy, complicated things, and I think a group who wants to disrupt that needs to make an affirmative argument based on something other than "self determination" that this is a good idea and all the disruption is worth it for the sake of things being better in the long run. Which unfortunately gets us nowhere because it brings us right back to debates about culture and history etc.

https://astralcodexten.substack.com/p/yudkowsky-contra-christiano-on-ai Previously in series: Yudkowsky Contra Ngo On Agents, Yudkowsky Contra Cotra On Biological Anchors Prelude: Yudkowsky Contra Hanson In 2008, thousands of blog readers - including yours truly, who had discovered the rationality community just a few months before - watched Robin Hanson debate Eliezer Yudkowsky on the future of AI. Robin thought the AI revolution would be a gradual affair, like the Agricultural or Industrial Revolutions. Various people invent and improve various technologies over the course of decades or centuries. Each new technology provides another jumping-off point for people to use when inventing other technologies: mechanical gears → steam engine → railroad and so on. Over the course of a few decades, you've invented lots of stuff and the world is changed, but there's no single moment when "industrialization happened". Eliezer thought it would be lightning-fast. Once researchers started building human-like AIs, some combination of adding more compute, and the new capabilities provided by the AIs themselves, would quickly catapult AI to unimaginably superintelligent levels. The whole process could take between a few hours and a few years, depending on what point you measured from, but it wouldn't take decades. You can imagine the graph above as being GDP over time, except that Eliezer thinks AI will probably destroy the world, which might be bad for GDP in some sense. If you come up with some way to measure (in dollars) whatever kind of crazy technologies AIs create for their own purposes after wiping out humanity, then the GDP framing will probably work fine. For transhumanists, this debate has a kind of iconic status, like Lincoln-Douglas or the Scopes Trial. But Robin's ideas seem a bit weird now (they also seemed a bit weird in 2008) - he thinks AIs will start out as uploaded human brains, and even wrote an amazing science-fiction-esque book of predictions about exactly how that would work. Since machine learning has progressed a lot faster than brain uploading has, this is looking less likely and probably makes his position less relevant than in 2008. The gradualist torch has passed to Paul Christiano, who wrote a 2018 post Takeoff Speeds revisiting some of Hanson's old arguments and adding new ones. (I didn't realize this until talking to Paul, but "holder of the gradualist torch" is a relative position - Paul still thinks there's about a 1/3 chance of a fast takeoff.) Around the end of last year, Paul and Eliezer had a complicated, protracted, and indirect debate, culminating in a few hours on the same Discord channel. Although the real story is scattered over several blog posts and chat logs, I'm going to summarize it as if it all happened at once. Gradatim Ferociter Paul sums up his half of the debate as: There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we'll see an 8 year doubling before a 2 year doubling, etc.) That is - if any of this "transformative AI revolution" stuff is right at all, then at some point GDP is going to go crazy (even if it's just GDP as measured by AIs, after humans have been wiped out). Paul thinks it will go crazy slowly. Right now world GDP doubles every ~25 years. Paul thinks it will go through an intermediate phase (doubles within 4 years) before it gets to a truly crazy phase (doubles within 1 year).

https://astralcodexten.substack.com/p/the-low-hanging-fruit-argument-models A followup to Contra Hoel On Aristocratic Tutoring: Imagine scientists venturing off in some research direction. At the dawn of history, they don't need to venture very far before discovering a new truth. As time goes on, they need to go further and further. Actually, scratch that, nobody has good intuitions for truth-space. Imagine some foragers who have just set up a new camp. The first day, they forage in the immediate vicinity of the camp, leaving the ground bare. The next day, they go a little further, and so on. There's no point in traveling miles and miles away when there are still tasty roots and grubs nearby. But as time goes on, the radius of denuded ground will get wider and wider. Eventually, the foragers will have to embark on long expeditions with skilled guides just to make it to the nearest productive land. Let's add intelligence to this model. Imagine there are fruit trees scattered around, and especially tall people can pick fruits that shorter people can't reach. If you are the first person ever to be seven feet tall, then even if the usual foraging horizon is very far from camp, you can forage very close to camp, picking the seven-foot-high-up fruits that no previous forager could get. So there are actually many different horizons: a distant horizon for ordinary-height people, a nearer horizon for tallish people, and a horizon so close as to be almost irrelevant for giants. Finally, let's add the human lifespan. At night, the wolves come out and eat anyone who hasn't returned to camp. So the the maximum distance anyone will ever be able to forage is a day's walk from camp (technically half a day, so I guess let's imagine that everyone can teleport back to camp whenever they want).

https://astralcodexten.substack.com/p/idol-words The woman was wearing sunglasses, a visor, a little too much lipstick, and a camera around her neck. "Excuse me," she asked. "Is this the temple with the three omniscient idols? Where one always tells the truth, one always lies, and one answers randomly?" The center idol's eyes glowed red, and it spoke with a voice from everywhere and nowhere, a voice like the whoosh of falling waters or the flash of falling stars. "No!" the great voice boomed. "Oh," said the woman. "Because my Uber driver said - ". She cut herself off. "Well, do you know how to get there?" "It is here!" said the otherworldly voice. "You stand in it now!" "Didn't you just say this wasn't it?"

https://astralcodexten.substack.com/p/who-gets-self-determination I. LSE: Fact-Checking The Kremlin's Version Of Russian History: The notion that Ukraine is not a country in its own right, but a historical part of Russia, appears to be deeply ingrained in the minds of many in the Russian leadership. Already long before the Ukraine crisis, at an April 2008 NATO summit in Bucharest, Vladimir Putin reportedly claimed that "Ukraine is not even a state! What is Ukraine? A part of its territory is [in] Eastern Europe, but a[nother] part, a considerable one, was a gift from us!" In his March 18, 2014 speech marking the annexation of Crimea, Putin declared that Russians and Ukrainians "are one people. Kiev is the mother of Russian cities. Ancient Rus' is our common source and we cannot live without each other." Since then, Putin has repeated similar claims on many occasions. As recently as February 2020, he once again stated in an interview that Ukrainians and Russians "are one and the same people", and he insinuated that Ukrainian national identity had emerged as a product of foreign interference. Similarly, Russia's then-Prime Minister Dmitry Medvedev told a perplexed apparatchik in April 2016 that there has been "no state" in Ukraine, neither before nor after the 2014 crisis. The article is from 2020, but the same discussion is continuing; see eg the New York Times' recent Putin Calls Ukrainian Statehood A Fiction. History Suggests Otherwise. I'm especially grateful to the Russian nationalist / far-right blogosphere for putting the case for Ukraine's non-statehood in terms that I can understand:

https://astralcodexten.substack.com/p/information-markets-decision-markets [thumbnail image credit: excellent nature photographer Eco Suparman, which is a great name for an excellent nature photographer!] Information Markets Niels Bohr supposedly said that "prediction is very difficult, especially about the future". So why not predict the past and present instead? Here's a recent market on Manifold (click image for link). Taylor Hawkins is a famous drummer who died last weekend under unclear circumstances. This market asks if he died of drug-related causes. Presumably someone will do an autopsy or investigation soon, and Chris will resolve the market based on that information. This is a totally standard prediction market, except that it's technically about interpreting past events. Same idea, only more tenuous. We know someone will do an autopsy on Taylor Hawkins soon, and we probably trust it. But how do we figure out whether COVID originated in a lab? This question's hack is to ask whether two public health agencies will claim it. If we trust the public health agencies, we can turn this mysterious past event into a forecasting question.

https://astralcodexten.substack.com/p/highlights-from-the-comments-on-justice A lot of comments on Justice Creep fell into three categories: First, people who thought some variety of: yes, all this stuff is definitely a justice issue, and it's good that language is starting to reflect that more. For example, Adnamanil: So... as someone who actually does use "___" Justice, quite frequently, I'd like to say that I think it's a good thing to reframe "helping the poor" or "saving the poor" as "pursuing economic justice." I don't think it's a good thing for people to think of themselves as saviors, to me that's a really unhealthy and unhelpful mindset which results in people who aren't themselves poor thinking they can be the experts and the decision-makers, and that there is something wrong with poor people, that they need to be "saved" or "fixed." We live in a world where there is enough food to feed everyone, yet people go hungry; enough shelter to keep everyone warm, yet people go cold. To me, that says there is something wrong with our system of resource distribution, not with the people who ended up, for one reason or another, being left out of it. Does that result in a sense of responsibility to fix the system? Yes! Does it imply that we don't live in Utopia? Yes! Because we don't. And I don't think we should pretend to. But it also implies that we *could* live in utopia. It demonstrates a real hope about the possibility of utopia. It says, "if we could figure out how to live together better, we could all have enough to eat and be warm." And Philosophy Bear, as Economic Justice And Climate Justice Are Not Metaphors: Regardless of whether it is useful -and I hope it is- I think that honesty compels a clear-eyed person to talk about many of these things in terms of justice, even in the narrowest conception of justice. The mistake in Scott's article is assuming that these forms of justice are merely metaphors or analogies on criminal justice. Many of these are about justice in exactly the same sense that crimes are about justice- no metaphor required. Of course, they are also about being just in other senses- justice was never just about crime. For example, one can detect demands for social justice in the bible that go far beyond "wouldn't it be nice to help people", but nonetheless aren't framed in terms of the criminal law. Nevertheless, yes, climate justice and economic justice- for example- are also about being just in the same way laws against murder are- no stretching of meaning is required…