The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Practically-A-Book Review: Rootclaim $100,000 Lab Leak Debate, published by trevor on March 28, 2024 on LessWrong. Lots of people already know about Scott Alexander/ACX/SSC, but I think that crossposting to LW is unusually valuable in this particular case, since lots of people were waiting for a big schelling-point overview of the 15-hour Rootclaim Lab Leak debate, and unlike LW, ACX's comment section is a massive vote-less swamp that lags the entire page and gives everyone equal status. It remains unclear whether commenting there is worth your time if you think you have something worth saying, since there's no sorting, only sifting, implying that it attracts small numbers of sifters instead of large numbers of people who expect sorting. Here are the first 11 paragraphs: Saar Wilf is an ex-Israeli entrepreneur. Since 2016, he's been developing a new form of reasoning, meant to transcend normal human bias. His method - called Rootclaim - uses Bayesian reasoning, a branch of math that explains the right way to weigh evidence. This isn't exactly new. Everyone supports Bayesian reasoning. The statisticians support it, I support it, Nate Silver wrote a whole book supporting it. But the joke goes that you do Bayesian reasoning by doing normal reasoning while muttering "Bayes, Bayes, Bayes" under your breath. Nobody - not the statisticians, not Nate Silver, certainly not me - tries to do full Bayesian reasoning on fuzzy real-world problems. They'd be too hard to model. You'd make some philosophical mistake converting the situation into numbers, then end up much worse off than if you'd tried normal human intuition. Rootclaim spent years working on this problem, until he was satisfied his method could avoid these kinds of pitfalls. Then they started posting analyses of different open problems to their site, rootclaim.com. Here are three: For example, does Putin have cancer? We start with the prior for Russian men ages 60-69 having cancer (14.32%, according to health data). We adjust for Putin's healthy lifestyle (-30% cancer risk) and lack of family history (-5%). Putin hasn't vanished from the world stage for long periods of time, which seems about 4x more likely to be true if he didn't have cancer than if he did. About half of cancer patients lose their hair, and Putin hasn't, so we'll divide by two. On the other hand, Putin's face has gotten more swollen recently, which happens about six times more often to cancer patients than to others, so we'll multiply by six. And so on and so forth, until we end up with the final calculation: 86% chance Putin doesn't have cancer, too bad. This is an unusual way to do things, but Saar claimed some early victories. For example, in a celebrity Israeli murder case, Saar used Rootclaim to determine that the main suspect was likely innocent, and a local mental patient had committed the crime; later, new DNA evidence seemed to back him up. One other important fact about Saar: he is very rich. In 2008, he sold his fraud detection startup to PayPal for $169 million. Since then he's founded more companies, made more good investments, and won hundreds of thousands of dollars in professional poker. So, in the grand tradition of very rich people who think they have invented new forms of reasoning everywhere, Saar issued a monetary challenge. If you disagree with any of his Rootclaim analyses - you think Putin does have cancer, or whatever - he and the Rootclaim team will bet you $100,000 that they're right. If the answer will come out eventually (eg wait to see when Putin dies), you can wait and see. Otherwise, he'll accept all comers in video debates in front of a mutually-agreeable panel of judges. Since then, Saar and his $100,000 offer have been a fixture of Internet debates everywhere. When I argued that Vitamin D didn't help fight...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Recent and upcoming media related to EA, published by 2ndRichter on March 28, 2024 on The Effective Altruism Forum. I'm Emma from the Communications team at the Centre for Effective Altruism (CEA). I want to flag a few media items related to EA that have come out recently or will be coming out soon, given they'll touch on topics - like FTX - that I expect will be of interest to Forum readers. The CEO of CEA, @Zachary Robinson, wrote an op-ed that came out today addressing Sam Bankman-Fried and the continuing value of EA. ( Read here) @William_MacAskill will appear on two podcasts and will discuss FTX: Clearer Thinking with Spencer Greenberg and the Making Sense Podcast with Sam Harris. The podcast episode with Sam Harris will likely be released next week and is aimed at a general audience. The podcast episode with Spencer Greenberg will likely be released in two weeks and is aimed at people more familiar with the EA movement. I'll add links for these episodes once they become available and plan to update this post as needed. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Development RCTs Are Good Actually, published by ozymandias on March 28, 2024 on The Effective Altruism Forum. This post was cross-posted from the substack Thing of Things with the permission of the author. In defense of trying things out The Economist recently published an article, "How poor Kenyans became economists' guinea pigs," which critiques development economists' use of randomized controlled trials. I think it exemplifies the profoundly weird way people think about experiments. The article says: In 2018, an RCT run by two development economists, in partnership with the World Bank and the water authority in Nairobi, Kenya's capital, tracked what happened when water supply was cut off to households in several slum settlements where bills hadn't been paid. Researchers wanted to test whether landlords, who are responsible for settling the accounts, would become more likely to pay as a result, and whether residents would protest. Hundreds of residents in slum settlements in Nairobi were left without access to clean water, in some cases for weeks or months; virtually none of them knew that they were part of an RCT. The study caused outrage among local activists and international researchers. The criticisms were twofold: first, that the researchers did not obtain explicit consent from participants for their involvement (they said that the landlord's contracts with the water company allowed for the cut-offs); and secondly, that interventions are supposed to be beneficial. The economists involved published an ethical statement defending the trial. Their research did not make the cut-offs more likely, they explained, because they were a standard part of the water authority's enforcement arsenal (though they acknowledged that disconnections in slums had previously been "ad hoc"). The statement did little to placate the critics. You know what didn't get an article in The Economist? All the times that slum dwellers in Nairobi were left without access to clean water for weeks or months without anyone studying them. By the revealed preferences of local activists, international researchers, and The Economist, the problem isn't that people are going without clean water, or that the water authority is shutting off people's water - those things have been going on for decades without more than muted complaining. The ethical problem is that someone is checking whether this unthinkably vast amount of human suffering is actually accomplishing anything. The water authority is presumably not shutting off people's water recreationally: it's shutting off people's water because they think it will get them to pay their water bills. Therefore, the possible effects of this study are: The water authority continues to do the same thing it was doing all along. The water authority learns that shutting off water doesn't get people to pay their bills, so it stops shutting off people's water, and they have enough to drink. If you step back from your instinctive ick reaction, you'll notice that this study may well improve water access for slum dwellers in Nairobi, and certainly isn't going to make it any worse. But people are still outraged because, I don't know, they have a strongly felt moral opposition to random number generators. I really don't understand the revulsion people feel about experimenting on humans. It's true that many scientists have done great evil in the name of science: the Tuskegee syphilis experiment, MKUltra, Nazi human experimentation, the Imperial Japanese Unit 731.[1] But the problem isn't the experiments. It's not somehow okay to deny people treatment for deadly diseases, force them to take drugs, or torture them if you happen to not write anything down about it. If it's fine to do something, then it's fine to randomly assign people to two groups, only do it to half of ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why we're entering a new nuclear age - and how to reduce the risks (Christian Ruhl on the 80k After Hours Podcast), published by 80000 Hours on March 28, 2024 on The Effective Altruism Forum. We just published an interview: Christian Ruhl on why we're entering a new nuclear age - and how to reduce the risks. You can click through for the audio, a full transcript, and related links. Below are the episode summary and some key excerpts. Episode summary We really, really want to make sure that nuclear war never breaks out. But we also know - from all of the examples of the Cold War, all these close calls - that it very well could, as long as there are nuclear weapons in the world. So if it does, we want to have some ways of preventing that from turning into a civilisation-threatening, cataclysmic kind of war. And those kinds of interventions - war limitation, intrawar escalation management, civil defence - those are kind of the seatbelts and airbags of the nuclear world. So to borrow a phrase from one of my colleagues, right-of-boom is a class of interventions for when "shit hits the fan." Christian Ruhl In this episode of 80k After Hours, Luisa Rodriguez and Christian Ruhl discuss underrated best bets to avert civilisational collapse from global catastrophic risks - things like great power war, frontier military technologies, and nuclear winter. They cover: How the geopolitical situation has changed in recent years into a "three-body problem" between the US, Russia, and China. How adding AI-enabled technologies into the mix makes things even more unstable and unpredictable. Why Christian recommends many philanthropists focus on "right-of-boom" interventions - those that mitigate the damage after a catastrophe - over traditional preventative measures. Concrete things policymakers should be considering to reduce the devastating effects of unthinkable tragedies. And on a more personal note, Christian's experience of having a stutter. Who this episode is for: People interested in the most cost-effective ways to prevent nuclear war, such as: Deescalating after accidental nuclear use. Civil defence and war termination. Mitigating nuclear winter. Who this episode isn't for: People interested in the least cost-effective ways to prevent nuclear war, such as: Coating every nuclear weapon on Earth in solid gold so they're no longer functional. Creating a TV show called The Real Housewives of Nuclear Winter about the personal and professional lives of women in Beverly Hills after a nuclear holocaust. A multibillion dollar programme to invent a laser beam that could write permanent messages on the Moon, and using it just once to spell out #nonukesnovember. Producer: Keiran Harris Audio Engineering Lead: Ben Cordell Technical editing: Ben Cordell and Milo McGuire Content editing: Katy Moore, Luisa Rodriguez, and Keiran Harris Transcriptions: Katy Moore "Gershwin - Rhapsody in Blue, original 1924 version" by Jason Weinberger is licensed under creative commons Highlights The three-body problem Christian Ruhl: For much of the Cold War, the US and the Soviet Union were the two nuclear superpowers. Other states eventually did acquire nuclear weapons, but in terms of arsenals, those two just towered over all of them. We're talking orders of magnitude bigger. And that had been the case for a long time, this kind of bipolar order. After the Cold War, people in many cases kind of stopped paying attention to this altogether. And what's happened in the last couple of years is that China seems poised to expand its own arsenal. So in 2020, their number of warheads, best estimate, is in the low 200s - 220 or so. Last year, that was up to 400 something. And now we're talking 500, and the projections suggest it could be as high as 1,000 by 2030 and 1,500 by 2035 - so really this massive increase. Lu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Leif Wenar's The Deaths of Effective Altruism, published by Arden on March 28, 2024 on The Effective Altruism Forum. Leif Wenar thoughtfully critiqued EA in "Poverty is No Pond" (2011) & just wrote a critique in WIRED. He is a philosophy professor at Stanford & author of Blood Oil. Edit: My initial thoughts (which are very raw & will likely change & I will accordingly regret having indelibly inscribed on the Internet): Initially, after a quick read-through, my take is he does a great job critiquing EA as a whole & showing the shortfalls are not isolated incidents. But none of the incidents were news to me. I think there's value in having these incidents/critique (well) written in a single article. But, really, I'm interested in the follow-up piece / how to reform EA or else the alternative to EA / what's next for the many talented young people who care, want to do good, & are drawn to EA. I'd love to hear y'all's thoughts on this. Edit: Share your Qs for Leif here. Thank you, M, for sharing this with me & encouraging me to connect. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Summer program for high schoolers - Guest speakers Yoshua Bengio and Peter Singer, published by Peter McIntyre on March 28, 2024 on The Effective Altruism Forum. Please share this opportunity with high schoolers you know - we'd be grateful for your help spreading the word! About Non-Trivial The Non-Trivial Fellowship is now accepting applications. It's an online summer program for high school students aged 14-20 to start an impactful research or policy project. Accepted fellows get access to: A $500 scholarship Up to $15,000 in funding Guidance from facilitators at Oxford, Cambridge, and Stanford Universities Guest speaker sessions, including Turing Award winner Yoshua Bengio and philosopher Peter Singer The summer cohort is running from July 8th - August 30th, 2024 and we will accept 200 people. The application has sections on probability, brainteasers, and game theory which many enjoy. Apply by March 31st, for increased interview chances and an early admissions decision. How you could help 1. Let high school students you may know about Non-Trivial (e.g. your old school, gifted programs you used to be a part of). 2. Share on social media Instagram: https://www.instagram.com/p/C4-9dHWo3_1/ Click to tweet: nntrvl.org/tweet Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Crises reveal centralisation, published by Vasco Grilo on March 28, 2024 on The Effective Altruism Forum. This is a crosspost for Crises reveal centralisation by Stefan Schubert, published on 3 May 2023. An important question for people focused on AI risk, and indeed for anyone trying to influence the world, is: how centralised is power? Are there dominant actors that wield most of the power, or is it more equally distributed? We can ask this question on two levels: On the national level, how powerful is the central power - the government - relative to smaller actors, like private companies, nonprofits, and individual people? On the global level, how powerful are the most powerful countries - in particular, the United States - relative to smaller countries? I think there are some common heuristics that lead people to think that power is more decentralised than it is, on both of these levels. One of these heuristics is what we can call "extrapolation from normalcy": Extrapolation from normalcy: the view that an actor seeming to have power here and now (in relatively normal times) is a good proxy for it having power tout court. It's often propped up by a related assumption about the epistemology of power: Naive behaviourism about power (naive behaviourism, for short): the view that there is a direct correspondence between an actor's power and the official and easily observable actions it takes. In other words, if an actor is powerful, then that will be reflected by official and easily observable actions, like widely publicised company investments or official government policies. Extrapolation from normalcy plus naive behaviourism suggest that the distribution of power is relatively decentralised on the national level. In normal times, companies are pursuing many projects that have consequential social effects (e.g. the Internet and its many applications). While these projects are subject to government regulation to some extent, private companies normally retain a lot of leeway (depending on what they want to do). This suggests (more so, the more you believe in naive behaviourism) that companies have quite a lot of power relative to governments in normal times. And extrapolation from normalcy implies that that this isn't just true in normal times, but holds true more generally. Similarly, extrapolation from normalcy plus naive behaviourism suggest that power is relatively decentralised on the global level, where we compare the relative power of different countries. There are nearly 200 independent countries in the world, and most of them make a lot of official decisions without overt foreign interference. While it's true that invasions do occur, they are relatively rare (the Russian invasion of Ukraine notwithstanding). Thus, naive behaviourism implies that power is decentralised under normal times, whereas extrapolation from normalcy extends that inference beyond normal times. But in my view, the world is more centralised than these heuristics suggest. The easiest way to see that is to look at crises. During World War II, much of the economy was put under centralised control one way or another in many countries. Similarly, during Covid, many governments drastically curtailed individual liberties and companies' economic activities (rightly or wrongly). And countries that want to acquire nuclear weapons (which can cause crises and wars) have found that they have less room to manoeuvre than the heuristics under discussion suggest. Accordingly, the US and other powerful nations have been able to reduce nuclear proliferation substantially (even though they've not been able to stop it entirely). It is true that smaller actors have a substantial amount of freedom to shape their own destiny under normal times, and that's an important fact. But still, who makes what official de...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Was Releasing Claude-3 Net-Negative?, published by Logan Riggs on March 28, 2024 on LessWrong. Cross-posted to EA forum There's been a lot of discussion among safety-concerned people about whether it was bad for Anthropic to release Claude-3. I felt like I didn't have a great picture of all the considerations here, and I felt that people were conflating many different types of arguments for why it might be bad. So I decided to try to write down an at-least-slightly-self-contained description of my overall views and reasoning here. Tabooing "Race Dynamics" I've heard a lot of people say that this "is bad for race dynamics". I think that this conflates a couple of different mechanisms by which releasing Claude-3 might have been bad. So, taboo-ing "race dynamics", a common narrative behind these words is As companies release better & better models, this incentivizes other companies to pursue more capable models at the expense of safety. Eventually, one company goes too far, produces unaligned AGI, and we all die". It's unclear what "at the expense of safety" means, so we can investigate two different interpretations:: If X increases "race dynamics", X causes an AGI company to Invest less in evals/redteaming models before deployment Divert resources away from alignment research & into capabilities research Did releasing Claude-3 cause other AI labs to invest less in evals/redteaming models before deployment? If OpenAI releases their next model 3 months earlier as a result. These 3 months need to come from *somewhere*, such as: A. Pre-training B. RLHF-like post-training C. Redteaming/Evals D. Product development/User Testing OpenAI needs to release a model better than Claude-3, so cutting corners on Pre-training or RLHF likely won't happen. It seems possible (C) or (D) would be cut short. If I believed GPT-5 would end the world, I would be concerned about cutting corners on redteaming/evals. Most people are not. However, this could set a precedent for investing less in redteaming/evals for GPT-6 onwards until AGI which could lead to model deployment of actually dangerous models (where counterfactually, these models would've been caught in evals). Alternatively, investing less in redteaming/evals could lead to more of a Sydney moment for GPT-5, creating a backlash to instead invest in redteaming/evals for the next generation model. Did releasing Claude-3 divert resources away from alignment research & into capabilities research? If the alignment teams (or the 20% GPUs for superalignment) got repurposed for capabilities or productization, I would be quite concerned. We also would've heard if this happened! Additionally, it doesn't seem possible to convert alignment teams into capability teams efficiently due to different skill sets & motivation. However, *future* resources haven't been given out yet. OpenAI could counterfactually invest more GPUs & researchers (either people switching from other teams or new hires) if they had a larger lead. Who knows! Additionally, OpenAI can take resources from other parts such as Business-to-business products, SORA, and other AI-related projects, in order to avoid backlash from cutting safety. But it's very specific to the team being repurposed if they could actually help w/ capabilities research. If this happens, then that does not seem bad for existential risk. Releasing Very SOTA Models Claude-3 isn't very far in the frontier, so OpenAI does have less pressure to make any drastic changes. If, however, Anthropic released a model as good as [whatever OpenAI would release by Jan 2025], then this could cause a bit of a re-evaluation of OpenAI's current plan. I could see a much larger percentage of future resources to go to capabilities research & attempts to poach Anthropic employees in-the-know. Anthropic at the Frontier is Good? Hypothe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How do LLMs give truthful answers? A discussion of LLM vs. human reasoning, ensembles & parrots, published by Owain Evans on March 28, 2024 on The AI Alignment Forum. Summary Large language models (LLMs) like ChatGPT and Claude 3 become increasingly truthful as they scale up in size and are finetuned for factual accuracy and calibration. However, the way LLMs arrive at truthful answers is nuanced. When an LLM answers a question immediately without chain-of-thought reasoning, the answer is typically not the result of the LLM reasoning about the question and weighing the evidence. Instead, the answer is based on human answers from pretraining documents that are (i) contextually relevant and (ii) resemble sources that led to truthful answers in finetuning. By contrast, when LLMs do explicit chain-of-thought reasoning before answering the question, the reasoning steps are more likely to causally determine the LLM's answer. This has parallels in human cognition. Many people can state Fermat's Theorem without having evaluated the proof themselves. Does this mean LLMs just parrot humans when answering without chain-of-thought reasoning? No. LLMs don't mimic a single human's answers. They aggregate over many human answers, weighted by relevance and whether the source is correlated with truthfulness. This is loosely analogous to mechanisms that aggregate many human judgments and outperform most individual humans, such as ensembling forecasts, markets, PageRank, and Bayesian Truth Serum. Moreover, LLMs have some conceptual understanding of their answers, even if they did not evaluate the answers before giving them. Epistemic Status: This essay is framed as a dialogue. There are no new experimental results but only my quick takes. Some of the takes are backed by solid evidence, while some are more speculative (as I indicate in the text). How do LLMs give truthful answers? Q: We'd like to have LLMs that are truthful, i.e. that systematically say true things and avoid saying false or inaccurate things wherever possible. Can we make LLMs like this? Owain: Current finetuned models like GPT-4 and Claude 3 still make mistakes on obscure long-tail questions and on controversial questions. However, they are substantially more truthful than earlier LLMs (e.g. GPT-2 or GPT-3). Moreover, they are more truthful than their own base models, after being finetuned specifically for truthfulness (or "honesty" or "factuality") via RLHF. In general, scaling up models and refining the RLHF finetuning leads to more truthful models, i.e. models that avoid falsehoods when answering questions. Q: But how does this work? Does the LLM really understand why the things it says are true, or why humans believe they are true? Owain: This is a complicated question and needs a longer answer. It matters whether the LLM immediately answers the question with no Chain of Thought ("no-CoT") or whether it gets to think before answering ("CoT"). Let's start with the no-CoT case, as in Figure 1 above. Suppose we ask the LLM a question Q and it answers immediately with answer A. I suspect that the LLM does not answer with A because it has evaluated and weighed the evidence for A. Instead, it usually answers with A because A was the answer given in human texts like Wikipedia (and similar sources), which were upweighted by the model's pretraining and RLHF training. Sometimes A was not an existing human answer, and so the LLM has to go beyond the human data. (Note that how exactly LLMs answer questions is not fully understood and so what I say is speculative. See "Addendum" below for more discussion.) Now, after the LLM has given answer A, we can ask the LLM to verify the claim. For example, it can verify mathematical assertions by a proof and scientific claims by citing empirical evidence. The LLM will also make some asse...

The podcast delves into Updateless Decision Theory, discussing the slow progress in formalizing it and insights gained. It introduces a 10-part series on decision theory in multiplayer settings, exploring AI decision-making in game theory contexts. The challenges of actions influenced by predictions of others' choices, exploring probability distributions, policies, and expected utility maximization, and navigating complexities of policy prediction environments are discussed.