The Nonlinear Library: LessWrong

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Investigating the Chart of the Century: Why is food so expensive?, published by Maxwell Tabarrok on August 16, 2024 on LessWrong. You've probably seen this chart from Mark Perry at the American Enterprise Institute. I've seen this chart dozens of times and have always enjoyed how many different and important stories it can tell. There is a story of the incredible abundance offered by technological growth and globalization. Compared to average hourly wages, cars, furniture, clothing, internet access, software, toys, and TVs have become far more accessible than they were 20 years ago. Flatscreens and Fiats that were once luxuries are now commodities. There is also a story of sclerosis and stagnation. Sure, lots of frivolous consumer goods have gotten cheaper but healthcare, housing, childcare, and education, all the important stuff, has exploded in price. Part of this is " cost disease" where the high productivity of labor in advancing industries like software, raises the cost of labor in slower productivity growth industries like healthcare. Another part is surely the near-universal "restrict supply and subsidize demand" strategy that governments undertake when regulating an industry. Zoning laws + Prop 13 in housing, occupational licensing and the FDA + Medicare in healthcare, and free student debt + all of the above for higher ed. One story from this graph I've never heard and only recently noticed is that "Food and Beverages" has inflated just as much as Housing in this graph. This is extremely counterintuitive. Food is a globally traded and mass produced commodity while housing is tied to inelastic land supply in desirable locations. Farming, grocery, and restaurants are competitive and relatively lightly regulated markets while the housing is highly regulated, subsidized, and distorted. Construction productivity is worse than stagnant while agricultural productivity has been ascendent for the past 300 years and even retail productivity is 8x higher than it was in 1950. Construction is also more labor intensive than farming or staffing the grocery. Source Yet food prices have risen just as much as housing prices over the past 24 years. What explains this? One trend is that Americans are eating out more. The "Food and Beverages" series from the BLS includes both "Food At Home" and "Food Away From Home." In 2023, eating out was a larger portion of the average household's budget than food at home for the first time, but they have been converging for more than 50 years. Restaurant food prices have increased faster than grocery prices. This makes sense, as a much larger portion of a restaurant's costs are location and labor, both of which are affected by tight supply constraints on urban floor space. This isn't enough to satisfy my surprise at the similarity in price growth though. Even if we just look at "food at home" price growth, it only really sinks below housing after 2015. Beverages at home/away from home follow a more divergent version of the same pattern, but are a much smaller part of the weighted average that makes up the aggregate index. The BLS series for "Housing" is also an aggregate index of " Shelter" prices, which is the actual rent (or Owner Rent Equivalent), and other expenses like utilities, moving, and repairs. Stagnant construction productivity and land use regulation will show up mostly in rents so these other pieces of the series are masking a bit of the inflation. There is also changing composition within the "Food at home" category. Americans eat more fats and oils, more sugars and sweets, more grains, and more red meat; all four items that grew the most in price since 2003. There's also a flipside to food and beverage's easy tradability: they're closer to the same price everywhere. House prices per square foot, by contrast, differ by more th...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Demis Hassabis - Google DeepMind: The Podcast, published by Zach Stein-Perlman on August 16, 2024 on LessWrong. The YouTube "chapters" are mixed up, e.g. the question about regulation comes 5 minutes after the regulation chapter ends. Ignore them. Noteworthy parts: 8:40: Near-term AI is hyped too much (think current startups, VCs, exaggerated claims about what AI can do, crazy ideas that aren't ready) but AGI is under-hyped and under-appreciated. 16:45: "Gemini is a project that has only existed for a year . . . our trajectory is very good; when we talk next time we should hopefully be right at the forefront." 17:20-18:50: Current AI doesn't work as a digital assistant. The next era/generation is agents. DeepMind is well-positioned to work on agents: "combining AlphaGo with Gemini." 24:00: Staged deployment is nice: red-teaming then closed beta then public deployment. 28:37 Openness (at Google: e.g. publishing transformers, AlphaCode, AlphaFold) is almost always a universal good. But dual-use technology - including AGI - is an exception. With dual-use technology, you want good scientists to still use the technology and advance as quickly as possible, but also restrict access for bad actors. Openness is fine today but in 2-4 years or when systems are more agentic it'll be dangerous. Maybe labs should only open-source models that are lagging a year behind the frontier (and DeepMind will probably take this approach, and indeed is currently doing ~this by releasing Gemma weights). 31:20 "The problem with open source is if something goes wrong you can't recall it. With a proprietary model if your bad actor starts using it in a bad way you can close the tap off . . . but once you open-source something there's no pulling it back. It's a one-way door, so you should be very sure when you do that." 31:42: Can an AGI be contained? We don't know how to do that [this suggests a misalignment/escape threat model but it's not explicit]. Sandboxing and normal security is good for intermediate systems but won't be good enough to contain an AGI smarter than us. We'll have to design protocols for AGI in the future: "when that time comes we'll have better ideas for how to contain that, potentially also using AI systems and tools to monitor the next versions of the AI system." 33:00: Regulation? It's good that people in government are starting to understand AI and AISIs are being set up before the stakes get really high. International cooperation on safety and deployment norms will be needed since AI is digital and if e.g. China deploys an AI it won't be contained to China. Also: Because the technology is changing so fast, we've got to be very nimble and light-footed with regulation so that it's easy to adapt it to where the latest technology's going. If you'd regulated AI five years ago, you'd have regulated something completely different to what we see today, which is generative AI. And it might be different again in five years; it might be these agent-based systems that [] carry the highest risks. So right now I would [] beef up existing regulations in domains that already have them - health, transport, and so on - I think you can update them for AI just like they were updated for mobile and internet. That's probably the first thing I'd do, while . . . making sure you understand and test the frontier systems. And then as things become [clearer] start regulating around that, maybe in a couple years time would make sense. One of the things we're missing is [benchmarks and tests for dangerous capabilities]. My #1 emerging dangerous capability to test for is deception because if the AI can be deceptive then you can't trust other tests [deceptive alignment threat model but not explicit]. Also agency and self-replication. 37:10: We don't know how to design a system that could come up with th...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Adverse Selection by Life-Saving Charities, published by vaishnav92 on August 16, 2024 on LessWrong. GiveWell, and the EA community at large, often emphasize the "cost of saving a life" as a key metric, $5,000 being the most commonly cited approximation. At first glance, GiveWell might seem to be in the business of finding the cheapest lives that can be saved, and then saving them. More precisely, GiveWell is in the business of finding the cheapest DALY it can buy. But implicit in that is the assumption that all DALYs are equal, or that disability or health effects are the only factors that we need to adjust for while assessing the value of a life year.. However, If DALYs vary significantly in quality (as I'll argue and GiveWell acknowledges we have substantial evidence for), then simply minimizing the cost of buying a DALY risks adverse selection. It's indisputable that each dollar goes much further in the poorest parts of the world. But it goes further towards saving lives in one the poorest parts of the world, often countries with terrible political institutions, fewer individual freedoms and oppressive social norms. More importantly, these conditions are not exogenous to the cost of saving a life. They are precisely what drive that cost down. Most EAs won't need convincing of the fact that the average life in New Zealand is much, much better than the average life in the Democratic Republic of Congo. In fact, those of us who donate to GiveDirectly do so precisely because this is the case. Extreme poverty and the suffering it entails is worth alleviating, wherever it can be found. But acknowledging this contradicts the notion that while saving lives, philanthropists are suddenly in no position to make judgements on how anything but physical disability affects the value/quality of life. To be clear, GiveWell won't be shocked by anything I've said so far. They've commissioned work and published reports on this. But as you might expect, these quality of life adjustments wouldnt feature in GiveWell's calculations anyway, since the pitch to donors is about the price paid for a life, or a DALY. But the idea that life is worse in poorer countries significantly understates the problem - that the project of minimizing the cost of lives saved while making no adjustments for the quality of lives said will systematically bias you towards saving the lives least worth living. In advanced economies, prosperity is downstream of institutions that preserve the rule of law, guarantee basic individual freedoms, prevent the political class from raiding the country, etc. Except for the Gulf Monarchies, there are no countries that have delivered prosperity for their citizens who don't at least do this. This doesn't need to take the form of liberal democracy; countries like China and Singapore are more authoritarian but the political institutions are largely non-corrupt, preserve the will of the people, and enable the creation of wealth and development of human capital. One can't say this about the countries in sub Saharan Africa. High rates of preventable death and disease in these countries are symptoms of institutional dysfunction that touches every facet of life. The reason it's so cheap to save a life in these countries is also because of low hanging fruit that political institutions in these countries somehow managed to stand in the way of. And one has to consider all the ways in which this bad equilibrium touches the ability to live a good life. More controversially, these political institutions aren't just levitating above local culture and customs. They interact and shape each other. The oppressive conditions that women (50% of the population) and other sexual minorities face in these countries isn't a detail that we can gloss over. If you are both a liberal and a consequentialis...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Danger, AI Scientist, Danger, published by Zvi on August 15, 2024 on LessWrong. While I finish up the weekly for tomorrow morning after my trip, here's a section I expect to want to link back to every so often in the future. It's too good. Danger, AI Scientist, Danger As in, the company that made the automated AI Scientist that tried to rewrite its code to get around resource restrictions and launch new instances of itself while downloading bizarre Python libraries? Its name is Sakana AI. (魚סכנה). As in, in hebrew, that literally means 'danger', baby. It's like when someone told Dennis Miller that Evian (for those who don't remember, it was one of the first bottled water brands) is Naive spelled backwards, and he said 'no way, that's too f***ing perfect.' This one was sufficiently appropriate and unsubtle that several people noticed. I applaud them choosing a correct Kabbalistic name. Contrast this with Meta calling its AI Llama, which in Hebrew means 'why,' which continuously drives me low level insane when no one notices. In the Abstract So, yeah. Here we go. Paper is "The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery." Abstract: One of the grand challenges of artificial general intelligence is developing agents capable of conducting scientific research and discovering new knowledge. While frontier models have already been used as aids to human scientists, e.g. for brainstorming ideas, writing code, or prediction tasks, they still conduct only a small part of the scientific process. This paper presents the first comprehensive framework for fully automatic scientific discovery, enabling frontier large language models to perform research independently and communicate their findings. We introduce The AI Scientist, which generates novel research ideas, writes code, executes experiments, visualizes results, describes its findings by writing a full scientific paper, and then runs a simulated review process for evaluation. In principle, this process can be repeated to iteratively develop ideas in an open-ended fashion, acting like the human scientific community. We demonstrate its versatility by applying it to three distinct subfields of machine learning: diffusion modeling, transformer-based language modeling, and learning dynamics. Each idea is implemented and developed into a full paper at a cost of less than $15 per paper. To evaluate the generated papers, we design and validate an automated reviewer, which we show achieves near-human performance in evaluating paper scores. The AI Scientist can produce papers that exceed the acceptance threshold at a top machine learning conference as judged by our automated reviewer. This approach signifies the beginning of a new era in scientific discovery in machine learning: bringing the transformative benefits of AI agents to the entire research process of AI itself, and taking us closer to a world where endless affordable creativity and innovation can be unleashed on the world's most challenging problems. Our code is open-sourced at this https URL We are at the point where they incidentally said 'well I guess we should design an AI to do human-level paper evaluations' and that's a throwaway inclusion. The obvious next question is, if the AI papers are good enough to get accepted to top machine learning conferences, shouldn't you submit its papers to the conferences and find out if your approximations are good? Even if on average your assessments are as good as a human's, that does not mean that a system that maximizes score on your assessments will do well on human scoring. Beware Goodhart's Law and all that, but it seems for now they mostly only use it to evaluate final products, so mostly that's safe. How Any of This Sort of Works According to section 3, there are three phases. 1. Idea generation using ...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A computational complexity argument for many worlds, published by jessicata on August 15, 2024 on LessWrong. The following is an argument for a weak form of the many-worlds hypothesis. The weak form I mean is that there are many observers in different branches of the wave function. The other branches "actually exist" for anthropic purposes; some observers are observing them. I've written before about difficulties with deriving discrete branches and observers from the Schrödinger equation; I'm ignoring this difficulty for now, instead assuming the existence of a many-worlds theory that specifies discrete branches and observers somehow. To be clear, I'm not confident in the conclusion; it rests on some assumptions. In general, physics theories throughout history have not been completely correct. It would not surprise me if a superintelligence would consider many-worlds to be a false theory. Rather, I am drawing implications from currently largely accepted physics and computational complexity theory, and plausible anthropic assumptions. First assumption: P != BQP. That is, there are some decision problems that cannot be decided in polynomial time by a classical computer but can be decided in polynomial time by an idealized quantum computer. This is generally accepted (RSA security depends on it) but not proven. This leaves open the possibility that the classically hardest BQP problems are only slightly harder than polynomial time. Currently, it is known that factorizing a b-bit integer can be done in roughly O(exp(cb1/3)) time where c is a constant greater than 1, while it can be done in polynomial time on an idealized quantum computer. I want to make an assumption that there are decision problems in BQP whose running time is "fast-growing", and I would consider O(exp(cb1/3)) "fast-growing" in this context despite not being truly exponential time. For example, a billion-bit number would require at least exp(1000) time to factorize with known classical methods, which is a sufficiently huge number for the purposes of this post. Second assumption: The universe supports BQP computation in polynomial physical resources and clock time. That is, it's actually possible to build a quantum computer and solve BQP problems in polynomial clock time with polynomial physical resources (space, matter, energy, etc). This is implied by currently accepted quantum theories (up to a reasonably high limit of how big a quantum computer can be). Third assumption: A "computational density anthropic prior", combining SIA with a speed prior, is a good prior over observations for anthropic purposes. As background, SIA stands for "self-indicating assumption" and SSA stands for "self-sampling assumption"; I'll assume familiarity with these theories, specified by Bostrom. According to SIA, all else being equal, universes that have more observers are more likely. Both SSA and SIA accept that universes with no observers are never observed, but only SIA accepts that universes with more observers are in general more likely. Note that SSA and SIA tend to converge in large universes (that is, in a big universe or multiverse with many observers, you're more likely to observe parts of the universe/multiverse with more observers, because of sampling). The speed prior implies that, all else being equal, universes that are more efficient to simulate (on some reference machine) are more likely. A rough argument for this is that in a big universe, many computations are run, and cheap computations are run more often, generating more observers. The computational density anthropic prior combines SIA with a speed prior, and says that we are proportionally more likely to observe universes that have a high ratio of observer-moments to required computation time. We could imagine aliens simulating many universes in paral...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An anti-inductive sequence, published by Viliam on August 14, 2024 on LessWrong. I was thinking about what would it mean for a sequence of bits to be "anti-inductive". It probably is a concept that is already known (as a rule of thumb, if I can think about it, someone probably already wrote a paper on it 50 years ago), but I haven't heard about it. * Some sequences are predictable and can be compressed. These two concepts are deeply related, because if you can successfully predict the next part of the sequence, you don't need to actually write it down; hence compression. A completely random sequence of bits cannot be compressed or predicted. There is a simple mathematical proof that some sequences cannot be compressed, although it doesn't say which ones. For any natural number N, there are more sequences of size exactly N, than sequences of size smaller than N. Therefore no program can generate a unique sequence shorter than N for any input sequence of size N. * Things get more complicated if we consider the caveat that although random sequences in general cannot be compressed, true randomness means that sometimes we accidentally get a sequence that can be compressed -- for example, with probability 1/2ᴺ we get a sequence of N zeroes, and it would sound silly to argue that we can't compress that! The solution to this paradox is that if we decide to compress only some selected sequences, then we need to add an extra bit of information specifying whether this sequence was compressed or not. Otherwise, if we see a sequence of bits saying (in binary) "a sequence of thousand zeroes", we wouldn't know whether the intended value is this very sequence of bits taken literally, or the sequence of thousand zeroes. One bit doesn't seem like much, but actually most sequences cannot be compressed, so the cost of adding one extra bit to each of them outweighs the occasional space we would save by compressing the ones we can. But still, if I needed a random sequence of bits to use e.g. as a password for something important... and by a miracle I generated a sequence of N zeroes... I would probably feel quite uncomfortable to use it, and would simply generate a new one. Is this a good security practice, or not? Because from certain perspective, by removing the "insufficiently random" sequences from the pool of possible choices, I am reducing the size of the pool, which... kinda makes it easier to guess the password? Something similar actually happened to me once. I used a mobile application that asked me to create a 4-digit PIN. So I typed 4 random digits, and the application said: "nope, you cannot use the same digit multiple times in the PIN, that is not secure". So I typed 4 random digits again, and the application said: "nope, you cannot use an ascending sequence of digits, that is not secure". So I typed 4 random digits yet again, and this time the application was finally satisfied. But it felt funny that the more "security checks" the application uses, the more limited is the choice of possible PINs. There are only 10000 possible combinations of 4 digits to start with; I wonder how far an overzealous security department could reduce that number. In a hypothetical extreme case, we would be left with only one possible choice of PIN -- certainly the most secure one that no one could possibly guess! The holy grail of information security. * Okay, back to the sequences of bits. Imagine that we are trying to create not just any random sequence, but the single most random, most unpredictable sequence ever! Suppose the length of the sequence is not specified in advance, so we just keep generating bits one by one, for as long as necessary. What we could do is create a predictor -- an algorithm that looks at the previously generated bits, tries to find all possible patterns in them and pr...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Superintelligent AI is possible in the 2020s, published by HunterJay on August 14, 2024 on LessWrong. Back in June 2023, Soroush Pour and I discussed AI timelines on his podcast, The AGI Show. The biggest difference between us was that I think "machines more intelligent than people are likely to be developed within a few years", and he thinks that it's unlikely to happen for at least a few decades.[1] We haven't really resolved our disagreement on this prediction in the year since, so I thought I would write up my main reasons for thinking we're so close to superintelligence, and why the various arguments made by Soroush (and separately by Arvind Narayanan and Sayash Kapoor) aren't persuasive to me. Part 1 - Why I Think We Are Close Empirically You can pick pretty much any trend relating to AI & computation, and it looks like this:[2] We keep coming up with new benchmarks, and they keep getting saturated. While there are still some notable holdouts such as ARC-AGI, SWE-Bench, and GPQA, previous holdouts like MATH also looked like this until they were solved by newer models. If these trends continue, it's hard to imagine things that AI won't be able to do in a few years time[3], unless they are bottlenecked by regulation (like being a doctor), or by robot hardware limitations (like being a professional football player)[4]. Practically The empirical trends are the result of several different factors; changes in network architecture, choice of hyperparameters, optimizers, training regimes, synthetic data creation, and data cleaning & selection. There are also many ideas in the space that have not been tried at scale yet. Hardware itself is also improving -- chips continue to double in price performance every 2-3 years, and training clusters are scaling up massively. It's entirely possible that some of these trends slow down -- we might not have another transformers-level architecture advance this decade, for instance -- but the fact that there are many different ways to continue improving AI for the foreseeable future makes me think that it is unlikely for progress to slow significantly. If we run out of text data, we can use videos. If we run out of that, we can generate synthetic data. If synthetic data doesn't generalise, we can get more efficient with what we have through better optimisers and training schedules. If that doesn't work, we can find architectures which learn the patterns more easily, and so on. In reality, all of these will be done at the same time and pretty soon (arguably already) the AIs themselves will be doing a significant share of the research and engineering needed to find and test new ideas[5]. This makes me think progress will accelerate rather than slow. Theoretically Humans are an existence proof of general intelligence, and since human cognition is itself just computation[6], there is no physical law stopping us from building another general intelligence (in silicon) given enough time and resources[7]. We can use the human brain as an upper bound for the amount of computation needed to get AGI (i.e. we know it can be done with the amount of computation done in the brain, but it might be possible with less)[8]. We think human brains do an equivalent of between 10^12 and 10^28 FLOP[9] per second [a hilariously wide range]. Supercomputers today can do 10^18. The physical, theoretical limit seems to be approximately 10^48 FLOP per second per kilogram. We can also reason that humans are a lower bound for the compute efficiency of the AGI (i.e. we know that with this amount of compute, we can get human-level intelligence, but it might be possible to do it with less)[10]. If humans are more efficient than current AI systems per unit of compute, then we know that more algorithmic progress must be possible as well. In other words, there seems to be...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ten arguments that AI is an existential risk, published by KatjaGrace on August 13, 2024 on LessWrong. This is a snapshot of a new page on the AI Impacts Wiki. We've made a list of arguments[1] that AI poses an existential risk to humanity. We'd love to hear how you feel about them in the comments and polls. Competent non-aligned agents Summary: 1. Humans will build AI systems that are 'agents', i.e. they will autonomously pursue goals 2. Humans won't figure out how to make systems with goals that are compatible with human welfare and realizing human values 3. Such systems will be built or selected to be highly competent, and so gain the power to achieve their goals 4. Thus the future will be primarily controlled by AIs, who will direct it in ways that are at odds with long-run human welfare or the realization of human values Selected counterarguments: It is unclear that AI will tend to have goals that are bad for humans There are many forms of power. It is unclear that a competence advantage will ultimately trump all others in time This argument also appears to apply to human groups such as corporations, so we need an explanation of why those are not an existential risk People who have favorably discussed[2] this argument (specific quotes here): Paul Christiano (2021), Ajeya Cotra (2023), Eliezer Yudkowsky (2024), Nick Bostrom (2014[3]). See also: Full wiki page on the competent non-aligned agents argument Second species argument Summary: 1. Human dominance over other animal species is primarily due to humans having superior cognitive and coordination abilities 2. Therefore if another 'species' appears with abilities superior to those of humans, that species will become dominant over humans in the same way 3. AI will essentially be a 'species' with superior abilities to humans 4. Therefore AI will dominate humans Selected counterarguments: Human dominance over other species is plausibly not due to the cognitive abilities of individual humans, but rather because of human ability to communicate and store information through culture and artifacts Intelligence in animals doesn't appear to generally relate to dominance. For instance, elephants are much more intelligent than beetles, and it is not clear that elephants have dominated beetles Differences in capabilities don't necessarily lead to extinction. In the modern world, more powerful countries arguably control less powerful countries, but they do not wipe them out and most colonized countries have eventually gained independence People who have favorably discussed this argument (specific quotes here): Joe Carlsmith (2024), Richard Ngo (2020), Stuart Russell (2020[4]), Nick Bostrom (2015). See also: Full wiki page on the second species argument Loss of control via inferiority Summary: 1. AI systems will become much more competent than humans at decision-making 2. Thus most decisions will probably be allocated to AI systems 3. If AI systems make most decisions, humans will lose control of the future 4. If humans have no control of the future, the future will probably be bad for humans Selected counterarguments: Humans do not generally seem to become disempowered by possession of software that is far superior to them, even if it makes many 'decisions' in the process of carrying out their will In the same way that humans avoid being overpowered by companies, even though companies are more competent than individual humans, humans can track AI trustworthiness and have AI systems compete for them as users. This might substantially mitigate untrustworthy AI behavior People who have favorably discussed this argument (specific quotes here): Paul Christiano (2014), Ajeya Cotra (2023), Richard Ngo (2024). See also: Full wiki page on loss of control via inferiority Loss of control via speed Summary: 1. Advances in AI will produce...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Debate: Get a college degree?, published by Ben Pace on August 13, 2024 on LessWrong. Epistemic Status: Soldier mindset. These are not our actual positions, these are positions we were randomly assigned by a coin toss, and for which we searched for the strongest arguments we could find, over the course of ~1hr 45mins. That said, this debate is a little messy between our performed positions and our personal ones. Sides: Ben is arguing against getting a college degree, and Saul is arguing for . (This is a decision Saul is currently making for himself!) Reading Order: Ben and Saul drafted each round of statements simultaneously. This means that each of Ben's statements you read were written without Ben having read Saul's statements that are immediately proceeding. (This does not apply to the back-and-forth interview.) Saul's Opening Statement first - i do think there's a qualitative difference between the position "getting an undergrad degree is good" vs "getting the typical undergrad experience is good." i think the second is in some ways more defensible than the first, but in most ways less so. For "getting the typical undergrad experience is good" This sort of thing is a strong Chesterton fence. People have been having the typical experience of an undergrad for a while (even while that typical experience changes). General upkeeping of norms/institutions is good. I think that - for a some ppl - their counterfactual is substantially worse. Even if this means college is functionally daycare, I'd rather they be in adult-day-care than otherwise being a drain on society (e.g. crime). It presents the option for automatic solutions to a lot of problems: Socializing high density of possible friends, romantic partners, etc you have to go to classes, talk to ppl, etc Exercise usually a free gym that's at-least functional you gotta walk to class, dining hall, etc Tons of ability to try slightly "weird" stuff you've never tried before - clubs, sports, events, greek life, sexual interactions, classes, etc I think a lot of these things get a lot more difficult when you haven't had the opportunity to experiment w them. A lot of ppl haven't experimented w much of anything before - college gives them an easy opportunity to do that w minimal friction before doing so becomes gated behind a ridiculous amount of friction. E.g. getting into a new hobby as an adult is a bit odd, in most social settings - but in college, it's literally as simple as joining that club. Again - while all of these sorts of things are possible outside of college, they become more difficult, outside of the usual norms, etc. For "getting an undergrad degree is good": This is a strong Chesterton fence. People have been getting undergrad degrees - or similar - for a wihle. It's an extremely legible symbol for a lot of society: Most ppl who get undergrad degrees aren't getting the sort of undergrad degree that ben or i sees - i think most are from huge state schools, followed by the gigantic tail of no-name schools. For those ppl, and for the jobs they typically seek, my guess is that for demonstrating the necessary things, like "i can listen to & follow directions, navigate general beaurocracies, learn things when needed, talk to people when needed, and am unlikely to be a extremely mentally ill, etc" - an undergrad degree is a pretty good signal. my guess is that a big part of the problem is that, despite this legible signal being good, ppl have indexed on it way too hard (& away from other signals of legibility, like a trade school, or a high school diploma with a high GPA or something). there are probably some instances where getting an undergrad degree isn't good, but those instances are strongly overrepresented to ben & saul, and the base rate is not that. also, it seems like society should give greater affordan...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Humanity isn't remotely longtermist, so arguments for AGI x-risk should focus on the near term, published by Seth Herd on August 13, 2024 on LessWrong. Toby Ord recently published a nice piece On the Value of Advancing Progress about mathematical projections of far-future outcomes given different rates of progress and risk levels. The problem with that and many arguments for caution is that people usually barely care about possibilities even twenty years out. We could talk about sharp discounting curves in decision-making studies, and how that makes sense given evolutionary pressures in tribal environments. But I think this is pretty obvious from talking to people and watching our political and economic practices. Utilitarianism is a nicely self-consistent value system. Utilitarianism pretty clearly implies longtermism. Most people don't care that much about logical consistency,[1] so they are happily non-utilitarian and non-longtermist in a variety of ways. Many arguments for AGI safety are longtermist, or at least long-term, so they're not going to work well for most of humanity. This is a fairly obvious, but worth-keeping-in-mind point. One non-obvious lemma of this observation is that much skepticism about AGI x-risk is probably based on skepticism about AGI happening soon. This doesn't explain all skepticism, but it's a significant factor worth addressing. When people dig into their logic, that's often a central point. They start out saying "AGI wouldn't kill humans" then over the course of a conversation it turns out that they feel that way primarily because they don't think real AGI will happen in their lifetimes. Any discussion of AGI x-risks isn't productive, because they just don't care about it. The obvious counterpoint is "You're pretty sure it won't happen soon? I didn't know you were an expert in AI or cognition!" Please don't say this - nothing convinces your opponents to cling to their positions beyond all logic like calling them stupid.[2] Something like "well, a lot of people with the most relevant expertise think it will happen pretty soon. A bunch more think it will take longer. So I just assume I don't know which is right, and it might very well happen pretty soon". It looks to me like discussing whether AGI might threaten humans is pretty pointless if the person is still assuming it's not going to happen for a long time. Once you're past that, it might make sense to actually talk about why you think AGI would be risky for humans.[3] 1. ^ This is an aside, but you'll probably find that utilitarianism isn't that much more logical than other value systems anyway. Preferring what your brain wants you to prefer, while avoiding drastic inconsistency, has practical advantages over values that are more consistent but that clash with your felt emotions. So let's not assume humanity isn't utilitarian just because it's stupid. 2. ^ Making sure any discussions you have about x-risk are pleasant for all involved is probably actually the most important strategy. I strongly suspect that personal affinity weighs more heavily than logic on average, even for fairly intellectual people. (Rationalists are a special case; I think we're resistant but not immune to motivated reasoning). So making a few points in a pleasant way, then moving on to other topics they like is probably way better than making the perfect logical argument while even slightly irritating them. 3. ^ From there you might be having the actual discussion on why AGI might threaten humans. Here are some things I've seen be convincing. People seem to often think "okay fine it might happen soon, but surely AI smarter than us still won't have free will and make its own goals". From there you could point out that it needs goals to be useful, and if it misunderstands those goals even slightly, it might be...