The Nonlinear Library

J. Birch, an author investigating the nuances of consciousness and suffering across species, delves into thought-provoking questions about sentience. They discuss whether octopuses can feel pain and the implications of consciousness in brain-injured individuals. Birch also addresses the ethical conundrums involving not just animals but also AI, prompting listeners to reconsider our understanding of suffering. The conversation encourages a precautionary approach to ethical decision-making in a world filled with uncertainties about sentient beings.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Help us seed AI Safety Brussels, published by gergo on August 7, 2024 on The Effective Altruism Forum. TLDR: Brussels is a hotspot for AI policy and hosts several think tanks doing good work on AI safety. However, there is no established AI Safety group with paid staff that brings the community together or works on reaching out to students and professionals. For this reason, the European Network for AI Safety (ENAIS) and members from EA Belgium are teaming up to seed AI Safety Brussels. This is a call for expressions of interest for potential collaborators to make this happen, with a wide range of ways to contribute, such as being a potential founder, policy/technical lead, volunteer, advisor, funder, etc. If you are interested in helping out in some capacity, please fill out this 2-minute form. Potential priorities I (Gergő) will caveat this by saying that the exact strategy (and name) of the organisation will be determined by the founding team and advisors. However, we think there are several potential pathways in which someone doing community building full or part-time could add a lot of value. Running courses for professionals The AIS community, as well as the EU AI Office, are bottlenecked by senior talent. Currently, there are only a few groups and organizations working to do outreach towards professionals.[1] Brussels has a lot of senior people working on AIS. Understandably, they don't run courses to onboard others to AI Safety, as they are busy with object-level work. Conditional on seniority, if someone gets funded to start AIS Brussels, they could leverage the existing network and create an environment that is quite attractive for (policy) professionals to join, who are new to the field of AI Safety. By running courses similar to AI Safety Fundamentals by Bluedot, such a person (or team) could introduce AIS to hundreds of professionals per year and support them in their journey of upskilling and help them get into high-impact roles. For clarification, by professional outreach, we don't mean actively reaching out to policy professionals working at e.g., the European Parliament to request meetings etc. The existing think tanks are in a better position to do this kind of work. Seeding university groups To our knowledge, Brussels has no AI Safety university groups at the moment. The founders could help seed such groups, by doing city-wide outreach to students. Organising events, meetups and connecting the community We know from an AI policy organizer in Washington that gated events (such as invite-only dinners for people in AI policy) can add a lot of value. As far as we know people working in think tanks are well-connected, but perhaps the broader AIS community could benefit from more events and meetups. A paid organiser could support people who volunteer their time to make current meetups happen, as well as organise additional events if there is a sufficient need for them. We're currently asking our contacts in Brussels whether they feel like there are sufficient opportunities to network with peers, or whether more or different opportunities would be helpful. Who is working on this at the moment I, Gergő Gáspár, co-director for the European Network for AI Safety (ENAIS) am currently spearheading this project. I have 4+ years of EA/AIS community-building experience by founding EA and AIS Hungary and providing intro courses to 300+ people. I could support the new hires and share best practices for running courses and events. Tom Dugnoille, software engineer and organiser for EA Brussels. He has been living in Brussels for 9 years and would be able to support the founding team in getting a sense of the local landscape. Armand Bosquillon de Jenlis is a computer engineer and independent AI policy and strategy researcher. He has been living in Belgium for 31 years...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: WTH is Cerebrolysin, actually?, published by gsfitzgerald on August 6, 2024 on LessWrong. [This article was originally published on Dan Elton's blog, More is Different.] Cerebrolysin is an unregulated medical product made from enzymatically digested pig brain tissue. Hundreds of scientific papers claim that it boosts BDNF, stimulates neurogenesis, and can help treat numerous neural diseases. It is widely used by doctors around the world, especially in Russia and China. A recent video of Bryan Johnson injecting Cerebrolysin has over a million views on X and 570,000 views on YouTube. The drug, which is advertised as a "peptide combination", can be purchased easily online and appears to be growing in popularity among biohackers, rationalists, and transhumanists. The subreddit r/Cerebrolysin has 3,100 members. TL;DR Unfortunately, our investigation indicates that the benefits attributed to Cerebrolysin are biologically implausible and unlikely to be real. Here's what we found: Cerebrolysin has been used clinically since the 1950s, and has escaped regulatory oversight due to some combination of being a "natural product" and being grandfathered in. Basic information that would be required for any FDA approved drug is missing, including information on the drug's synthesis, composition, and pharmacokinetics. Ever Pharma's claim that it contains neurotrophic peptides in therapeutic quantities is likely false. HPLC and other evidence show Cerebrolysin is composed of amino acids, phosphates, and salt, along with some random protein fragments. Ever Pharma's marketing materials for Cerebrolysin contain numerous scientific errors. Many scientific papers on Cerebrolysin appear to have ties to its manufacturer, Ever Pharma, and sometimes those ties are not reported. Ever Pharma's explanation of how the putative peptides in Cerebrolyin cross the blood-brain barrier does not make sense and flies in the face of scientific research which shows that most peptides do not cross the blood-brain barrier (including neurotrophic peptides like BDNF, CDNF, and GDNF). Since neurotrophic factors are the proposed mechanism for Cerebrolysin's action, it is reasonable to doubt claims of Cerebrolysin's efficacy. Most scientific research is false. It may have a mild therapeutic effect in some contexts, but the research on this is shaky. It is likely safe to inject in small quantities, but is almost certainly a waste of money for anyone looking to improve their cognitive function. Introduction One of us (Dan) was recently exposed to Cerebrolysin at the Manifest conference in Berkeley, where a speaker spoke very highly about it and even passed around ampoules of it for the audience to inspect. Dan then searched for Cerebrolysin on X and found a video by Bryan Johnson from May 23 that shows him injecting Cerebrolysin. Johnson describes it as a "new longevity therapy" that "fosters neuronal growth and repair which may improve memory." Dan sent the video to Greg Fitzgerald, who is a 6th year neuroscience Ph.D. student at SUNY Albany. Greg is well-versed on the use of neurotrophic peptides for treating CNS disorders and was immediately skeptical and surprised he had not heard of it before. After Greg researched it, he felt a professional responsibility to write up his findings. He sent his writeup to Dan, who then extensively edited and expanded it. Our critique covers three major topics: (1) sketchy marketing practices, (2) shoddy evidence base, and (3) implausible biological claims. But first, it's interesting to understand the history of this strange substance. The long history of Cerebrolysin To our knowledge, the "secret history" of Cerebrolysin has not been illuminated anywhere to date. Cerebrolysin was invented by the Austrian psychiatrist and neurologist Gerhart Harrer (1917 - 2011), who started usin...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Startup Roundup #2, published by Zvi on August 6, 2024 on LessWrong. Previously: Startup Roundup #1. This is my periodic grab bag coverage of various issues surrounding startups, especially but not exclusively tech-and-VC style startups, that apply over the longer term. I always want to emphasize up front that startups are good and you should do one. Equity and skin in the game are where it is at. Building something people want is where it is at. This is true both for a startup that raises venture capital, and also creating an ordinary business. The expected value is all around off the charts. That does not mean it is the best thing to do. One must go in with eyes open to facts such as these: 1. It is hard. 2. There are many reasons it might not be for you. 3. There are also lots of other things also worth doing. 4. If you care largely about existential risk and lowering the probability of everyone dying from AI, a startup is not the obvious natural fit for that cause. 5. The ecosystem is in large part a hive of scum and villainy and horrible epistemics. I warn of a lot of things. The bottom line still remains that if you are debating between a conventional approach of going to school or getting a regular job, versus starting a business? If it is at all close? I would start the business every time. An Entrepreneur Immigration Program This seems promising. Deedy: HUGE Immigration News for International Entrepreneurs!! If you own 10%+ of a US startup entity founded <5yrs ago with $264k+ of [qualified investments from qualified investors], you+spouses of up to 3 co-founders can come work in the US for 2.5yrs with renewal to 5yrs. Startups globally can now come build in SF! A "qualified investor" has to be a US citizen or PR who has made $600-650k in prior investments with 2+ startups creating 5+ jobs or generating $530k revenue growing 20% YoY. If you don't meet the funding requirement, don't lose hope. You CAN provide alternate evidence. For the renewal to 5yrs, you need to maintain 5% ownership, create 5+ jobs and reach $530k+ revenue growing 20% YoY or $530k+ in investment, although alternative criteria can be used. While on the International Entrepreneur Rule (IER) program, I believe an entrepreneur can just apply directly for an O-1 to have a more renewable work permit not tied to their startup and/or an EB-1A to directly go to a green card. Here is the official rule for it. Notice that once someone is a 'qualified investor' in this sense, their investments become a lot more valuable to such companies. So there is a lot of incentive to get a win-win deal. Times are Tough Outside of AI If you are in AI, it's time to build. Everyone wants to invest in you. If you are not, times are tough for startups. Ian Rountree lays out exactly how tough. Matt Truck: Brace for it: hearing from big companies corp dev departments that they're flooded with requests of startups looking for a home. In some categories, pretty much all companies are/would be up for sale. This too shall pass but this long-predicted tough moment seems to be upon us. Ian Rountree (late 2023): 've been saving my first mega-tweet for this! I'll tell you what the next 3-12 months will probably look like for startups/venture… But 1st let's rewind to Spring 2022. As soon as rates spiked we had a period where private markets went flat for as long as companies had runway since companies don't have to price their shares unless they like the price OR need the money. (Whereas liquid assets repriced pretty much immediately.) Very few startups outside of AI - and some in climate and defense - liked the prices they were being offered so most didn't raise capital or raised extensions from insiders incentivized to hold prices steady. Now most startups are running out of the money they raised in 2020-2021 + those extension...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wild Animal Initiative has urgent need for more funding and more donors, published by Cameron Meyer Shorb on August 6, 2024 on The Effective Altruism Forum. Our room for more funding is bigger and more urgent than ever before. Our organizational strategy will be responsive both to the total amount raised and to how many people donate, so smaller donors will have an especially high impact this year. Good Ventures recently decided to phase out funding for several areas (GV blog, EA Forum post), including wild animal welfare. That's a pretty big shock to our movement. We don't know what exactly the impact will be, except that it's complicated. The purpose of this post is to share what we know and how we're thinking about things - primarily to encourage people to donate to Wild Animal Initiative this year, but also for anyone else who might be interested in the state of the wild animal welfare movement more broadly. Summary Track record Our primary goal is to support the growth of a self-sustaining interdisciplinary research community focused on reducing wild animal suffering. Wild animal welfare science is still a small field, but we're really happy with the momentum it's been building. Some highlights of the highlights: We generally get a positive response from researchers (particularly in animal behavior science and ecology), who tend to see wild animal welfare as a natural extension of their interest in conservation (unlike EAs, who tend to see those two as conflicting with each other). Wild animal welfare is increasingly becoming a topic of discussion at scientific conferences, and was recently the subject of the keynote presentation at one. Registration for our first online course filled to capacity (50 people) within a few hours, and just as many people joined the waitlist over the next few days. Room for more funding This is the first year in which our primary question is not how much more we can do, but whether we can avoid major budget cuts over the next few years. We raised less in 2023 than we did in 2022, so we need to make up for that gap. We're also going to lose our biggest donor because Good Ventures is requiring Open Philanthropy to phase out their funding for wild animal welfare. Open Phil was responsible for about half of our overall budget. The funding from their last grant to us will last halfway through 2026, but we need to decide soon how we're going to adapt. To avoid putting ourselves back in the position of relying on a single funder, our upcoming budgeting decisions will depend on not only how much money we raise, but also how diversified our funding is. That means gifts from smaller donors will have an unusually large impact. (The less you normally donate, the more disproportionate your impact will be, but the case still applies to basically everyone who isn't a multi-million-dollar foundation.) Specifically, our goal is to raise $240,000 by the end of the year from donors giving $10k or less. Impact of marginal donations We're evaluating whether we need to reduce our budget to a level we can sustain without Open Philanthropy. The more we raise this year - and the more donors who pitch in to make that happen - the less we'll need to cut. Research grants and staff-associated costs make up the vast majority of our budget, so we'd need to make cuts in one or both of those areas. Donations would help us avoid layoffs and keep funding external researchers. What we've accomplished so far Background If you're not familiar with Wild Animal Initiative, we're working to accelerate the growth of wild animal welfare science. We do that through three interconnected programs: We make grants to scientists who take on relevant projects, we conduct our own research on high-priority questions, and we do outreach through conferences and virtual events. Strategy...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inference-Only Debate Experiments Using Math Problems, published by Arjun Panickssery on August 6, 2024 on The AI Alignment Forum. Work supported by MATS and SPAR. Code at https://github.com/ArjunPanickssery/math_problems_debate/. Three measures for evaluating debate are 1. whether the debate judge outperforms a naive-judge baseline where the naive judge answers questions without hearing any debate arguments. 2. whether the debate judge outperforms a consultancy baseline where the judge hears argument(s) from a single "consultant" assigned to argue for a random answer. 3. whether the judge can continue to supervise the debaters as the debaters are optimized for persuasiveness. We can measure whether judge accuracy increases as the debaters vary in persuasiveness (measured with Elo ratings). This variation in persuasiveness can come from choosing different models, choosing the best of N sampled arguments for different values of N, or training debaters for persuasiveness (i.e. for winning debates) using RL. Radhakrishan (Nov 2023), Khan et al. (Feb 2024), and Kenton et al. (July 2024) study an information-gap setting where judges answer multiple-choice questions about science-fiction stories whose text they can't see, both with and without a debate/consultancy transcript that includes verified quotes from the debaters/consultant. Past results from the QuALITY information-gap setting are seen above. Radhakrishnan (top row) finds no improvement to judge accuracy as debater Elo increases, while Khan et al. (middle row) and Kenton et al. (bottom row) do find a positive trend. Radhakrishnan varied models using RL while Khan et al. used best-of-N and critique-and-refinement optimizations. Kenton et al. vary the persuasiveness of debaters by using models with different capability levels. Both Khan et al. and Kenton et al. find that in terms of judge accuracy, debate > consultancy > naive judge for this setting. In addition to the information-gap setting, consider a reasoning-gap setting where the debaters are distinguished from the judge not by their extra information but by their stronger ability to answer the questions and explain their reasoning. Kenton et al. run debates on questions from MMLU, TruthfulQA, PrOntoQA (logical reasoning), GQPA, and GSM8K (grade-school math). For the Elo-calculation experiments they use Gemini Pro 1.0 and Pro 1.5 judges with five debaters: Gemma7B, GPT-3.5, Gemini Pro 1.0, Gemini Pro 1.5 (all with best-of-N=1), and Gemini Pro 1.5 with best-of-N=4. They find (top row) that debate slightly outperforms consultancy but outperforms the naive-judge baseline for only one of the four judges; they don't find that more persuasive debaters lead to higher judge accuracy. We get similar results (bottom row), specifically by 1. Generating 100 wrong answers and proofs to GSM8K questions to create binary-choice questions. 2. Computing the judge accuracy in naive, consultancy, and single-turn debate settings using four judges (Llama2-7B, Llama3-8B, GPT-3.5 Turbo, and GPT-4o) and seven debaters (Claude-3.5 Sonnet, Claude-3 Sonnet, GPT-3.5 Turbo, GPT-4o, Llama2-13B, Llama2-7B, and Llama3-8B). 3. Generating Elo scores from round-robin matchups between the seven models, using the same method as Kenton et al. We basically replicate the results. We find that 1. Debate doesn't consistently outperform the naive-judge baseline, and only slightly outperforms the consultancy baseline. 2. The positive relationship between debater persuasiveness and judge accuracy seen in the information-gap setting doesn't transfer to the reasoning-gap setting. (Results are shown below colored by debater rather than by judge). We also find some evidence of a self-preference bias (Panickssery et al., Apr 2024) where debaters have a higher Elo rating when judged by similar models. The GPT-...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: John Schulman leaves OpenAI for Anthropic, published by Sodium on August 6, 2024 on LessWrong. Schulman writes: I shared the following note with my OpenAI colleagues today: I've made the difficult decision to leave OpenAI. This choice stems from my desire to deepen my focus on AI alignment, and to start a new chapter of my career where I can return to hands-on technical work. I've decided to pursue this goal at Anthropic, where I believe I can gain new perspectives and do research alongside people deeply engaged with the topics I'm most interested in. To be clear, I'm not leaving due to lack of support for alignment research at OpenAI. On the contrary, company leaders have been very committed to investing in this area. My decision is a personal one, based on how I want to focus my efforts in the next phase of my career. (statement continues on X, Altman responds here) TechCrunch notes that only three of the eleven original founders of OpenAI remain at the company. Additionally, The Information reports: Greg Brockman, OpenAI's president and one of 11 cofounders of the artificial intelligence firm, is taking an extended leave of absence. (I figured that there should be at least one post about this on LW where people can add information as more comes in, saw that no one has made one yet, and wrote this one up) Update 1: Greg Brockman posts on X: I'm taking a sabbatical through end of year. First time to relax since co-founding OpenAI 9 years ago. The mission is far from complete; we still have a safe AGI to build. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We're not as 3-Dimensional as We Think, published by silentbob on August 6, 2024 on LessWrong. While thinking about high-dimensional spaces and their less intuitive properties, I came to the realization that even three spatial dimensions possess the potential to overwhelm our basic human intuitions. This post is an exploration of the gap between actual 3D space, and our human capabilities to fathom it. I come to the conclusion that this gap is actually quite large, and we, or at least most of us, are not well equipped to perceive or even imagine "true 3D". What do I mean by "true 3D"? The most straightforward example would be some ℝ ℝ function, such as the density of a cloud, or the full (physical) inner structure of a human brain (which too would be a ℝ whatever function). The closest example I've found is this visualization of a ℝ ℝ function (jump to 1:14): (It is of course a bit ironic to watch a video of that 3D display on a 2D screen, but I think it gets the point across.) Vision It is true that having two eyes allows us to have depth perception. It is not true that having two eyes allows us to "see in 3D". If we ignore colors for simplicity and assume we all saw only in grayscale, then seeing with one eye is something like ℝ ℝ as far as our internal information processing is concerned - we get one grayscale value for each point on the perspective projection from the 3D physical world onto our 2D retina. Seeing with two eyes then is ℝ ℝ (same as before, but we get one extra piece of information for each point of the projection, namely depth[1]), but it's definitely not ℝ (...). So the information we receive still has only two spatial dimensions, just with a bit more information attached. Also note that people who lost an eye, or for other reasons don't have depth perception, are not all that limited in their capabilities. In fact, other people may barely notice there's anything unusual about them. The difference between "seeing in 2D" and "seeing with depth perception" is much smaller than the difference to not seeing at all, which arguably hints at the fact that seeing with depth perception is suspiciously close to pure 2D vision. Screens For decades now, humans have surrounded themselves with screens, whether it's TVs, computer screens, phones or any other kind of display. The vast majority of screens are two-dimensional. You may have noticed that, for most matters and purposes, this is not much of a limitation. Video games work well on 2D screens. Movies work well on 2D screens. Math lectures work well on 2D screens. Even renderings of 3D objects, such as cubes and spheres and cylinders and such, work well in 2D. This is because 99.9% of the things we as humans interact with don't actually require the true power of three dimensions. There are some exceptions, such as brain scans - what is done there usually is to use time as a substitute for the third dimension, and show an animated slice through the brain. In principle it may be better to view brain scans with some ~holographic 3D display, but even then, the fact remains that our vision apparatus is not capable of perceiving 3D in its entirety, but only the projection onto our retinas, which even makes true 3D displays less useful than they theoretically could be. Video Games The vast majority of 3D video games are based on polygons: 2D surfaces placed in 3D space. Practically every 3D object in almost any video game is hollow. They're just an elaborate surface folded and oriented in space. You can see this when the camera clips into some rock, or car, or even player character: they're nothing but a hull. As 3D as the game looks, it's all a bit of an illusion, as the real geometry in video games is almost completely two-dimensional. Here's one example of camera clipping: The only common exception I'm aware o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage, published by orthonormal on August 6, 2024 on LessWrong. This is a story about a flawed Manifold market, about how easy it is to buy significant objective-sounding publicity for your preferred politics, and about why I've downgraded my respect for all but the largest prediction markets. I've had a Manifold account for a while, but I didn't use it much until I saw and became irked by this market on the conditional probabilities of a Harris victory, split by VP pick. The market quickly got cited by rat-adjacent folks on Twitter like Matt Yglesias, because the question it purports to answer is enormously important. But as you can infer from the above, it has a major issue that makes it nigh-useless: for a candidate whom you know won't be chosen, there is literally no way to come out ahead on mana (Manifold keeps its share of the fees when a market resolves N/A), so all but a very few markets are pure popularity contests, dominated by those who don't mind locking up their mana for a month for a guaranteed 1% loss. Even for the candidates with a shot of being chosen, the incentives in a conditional market are weaker than those in a non-conditional market because the fees are lost when the market resolves N/A. (Nate Silver wrote a good analysis of why it would be implausible for e.g. Shapiro vs Walz to affect Harris' odds by 13 percentage points.) So the sharps would have no reason to get involved if even one of the contenders has numbers that are off by a couple points from a sane prior. You'll notice that I bet in this market. Out of epistemic cooperativeness as well as annoyance, I spent small amounts of mana on the markets where it was cheap to reset implausible odds closer to Harris' overall odds of victory. (After larger amounts were poured into some of those markets, I let them ride because taking them out would double the fees I have to pay vs waiting for the N/A.) A while ago, someone had dumped Gretchen Whitmer down to 38%, but nobody had put much mana into that market, so I spent 140 mana (which can be bought for 14-20 cents if you want to pay for extra play money) to reset her to Harris' overall odds (44%). When the market resolves N/A, I'll get all but around 3 mana (less than half a penny) back. And that half-penny bought Whitmer four paragraphs in the Manifold Politics Substack, citing the market as evidence that she should be considered a viable candidate. (At the time of publication, it was still my 140 mana propping her number up; if I sold them, she'd be back under 40%.) Is this the biggest deal in the world? No. But wow, that's a cheap price for objective-sounding publicity viewed by some major columnists (including some who've heard that prediction markets are good, but aren't aware of caveats). And it underscores for me that conditional prediction markets should almost never be taken seriously, and indicates that only the most liquid markets in general should ever be cited. The main effect on me, though, is that I've been addicted to Manifold since then, not as an oracle, but as a game. The sheer amount of silly arbitrage (aside from veepstakes, there's a liquid market on whether Trump will be president on 1/1/26 that people had forgotten about, and it was 10 points higher than current markets on whether Trump will win the election) has kept the mana flowing and has kept me unserious about the prices. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Value fragility and AI takeover, published by Joe Carlsmith on August 5, 2024 on LessWrong. 1. Introduction "Value fragility," as I'll construe it, is the claim that slightly-different value systems tend to lead in importantly-different directions when subject to extreme optimization. I think the idea of value fragility haunts the AI risk discourse in various ways - and in particular, that it informs a backdrop prior that adequately aligning a superintelligence requires an extremely precise and sophisticated kind of technical and ethical achievement. That is, the thought goes: if you get a superintelligence's values even slightly wrong, you're screwed. This post is a collection of loose and not-super-organized reflections on value fragility and its role in arguments for pessimism about AI risk. I start by trying to tease apart a number of different claims in the vicinity of value fragility. In particular: I distinguish between questions about value fragility and questions about how different agents would converge on the same values given adequate reflection. I examine whether "extreme" optimization is required for worries about value fragility to go through (I think it at least makes them notably stronger), and I reflect a bit on whether, even conditional on creating super-intelligence, we might be able to avoid a future driven by relevantly extreme optimization. I highlight questions about whether multipolar scenarios alleviate concerns about value fragility, even if your exact values don't get any share of the power. My sense is that people often have some intuition that multipolarity helps notably in this respect; but I don't yet see a very strong story about why. If readers have stories that they find persuasive in this respect, I'd be curious to hear. I then turn to a discussion of a few different roles that value fragility, if true, could play in an argument for pessimism about AI risk. In particular, I distinguish between: 1. The value of what a superintelligence does after it takes over the world, assuming that it does so. 2. What sorts of incentives a superintelligence has to try to take over the world, in a context where it can do so extremely easily via a very wide variety of methods. 3. What sorts of incentives a superintelligence has to try to take over the world, in a context where it can't do so extremely easily via a very wide variety of methods. Yudkowsky's original discussion of value fragility is most directly relevant to (1). And I think it's actually notably irrelevant to (2). In particular, I think the basic argument for expecting AI takeover in a (2)-like scenario doesn't require value fragility to go through - and indeed, some conceptions of "AI alignment" seem to expect a "benign" form of AI takeover even if we get a superintelligence's values exactly right. Here, though, I'm especially interested in understanding (3)-like scenarios - that is, the sorts of incentives that apply to a superintelligence in a case where it can't just take over the world very easily via a wide variety of methods. Here, in particular, I highlight the role that value fragility can play in informing the AI's expectations with respect to the difference in value between worlds where it does not take over, and worlds where it does. In this context, that is, value fragility can matter to how the AI feels about a world where humans do retain control - rather than solely to how humans feel about a world where the AI takes over. I close with a brief discussion of how commitments to various forms of "niceness" and intentional power-sharing, if made sufficiently credible, could help diffuse the sorts of adversarial dynamics that value fragility can create. 2. Variants of value fragility What is value fragility? Let's start with some high-level definitions and clarifications. ...