The Nonlinear Library

Peter Wildeford, author of Eight Fundamental Constraints for Growing a Research Organization, discusses the key constraints for organizational growth. These constraints include having sufficient important work, skill in prioritization, talented researchers, and fostering a positive organizational culture.

AI researcher Andrew Critch discusses the importance of incorporating social models in technical AI safety, dispelling the myth that it is inherently beneficial and highlighting the need for nuanced approaches in research.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing The Midas Project - and our first campaign!, published by Tyler Johnston on June 13, 2024 on The Effective Altruism Forum. Summary The Midas Project is a new AI safety organization. We use public advocacy to incentivize stronger self-governance from the companies developing and deploying high-risk AI products. This week, we're launching our first major campaign, targeting the AI company Cognition. Cognition is a rapidly growing startup [1] developing autonomous coding agents. Unfortunately, they've told the public virtually nothing about how, or even if, they will conduct risk evaluations to prevent misuse and other unintended outcomes. In fact, they've said virtually nothing about safety at all. We're calling on Cognition to release an industry-standard evaluation-based safety policy. We need your help to make this campaign a success. Here are five ways you can help, sorted by level of effort: 1. Keep in the loop about our campaigns by following us on Twitter and joining our mailing list. 2. Offer feedback and suggestions, by commenting on this post or by reaching out at info@themidasproject.com 3. Share our Cognition campaign on social media, sign the petition, or engage with our campaigns directly on our action hub. 4. Donate to support our future campaigns (tax-exempt status pending). 5. Sign up to volunteer, or express interest in joining our team full-time. Background The risks posed by AI are, at least partially, the result of a market failure. Tech companies are locked in an arms race that is forcing everyone (even the most safety-concerned) to move fast and cut corners. Meanwhile, consumers broadly agree that AI risks are serious and that the industry should move slower. However, this belief is disconnected from their everyday experience with AI products, and there isn't a clear Schelling point allowing consumers to express their preference via the market. Usually, the answer to a market failure like this is regulation. When it comes to AI safety, this is certainly the solution I find most promising. But such regulation isn't happening quickly enough. And even if governments were moving quicker, AI safety as a field is pre-paradigmatic. Nobody knows exactly what guardrails will be most useful, and new innovations are needed. So companies are largely being left to voluntarily implement safety measures. In an ideal world, AI companies would be in a race to the top, competing against each other to earn the trust of the public through comprehensive voluntary safety measures while minimally stifling innovation and the benefits of near-term applications. But the incentives aren't clearly pointing in that direction - at least not yet. However: EA-supported organizations have previously been successful at shifting corporate incentives in the past. Take the case of cage-free campaigns. By engaging in advocacy that threatens to expose specific food companies for falling short of customers' basic expectations regarding animal welfare, groups like The Humane League and Mercy For Animals have been able to create a race to the top for chicken welfare, leading virtually all US food companies to commit to going cage-free. [2] Creating this change was as simple as making the connection in the consumer's mind between their pre-existing disapproval of inhumane battery cages and the eggs being served at their local fast food chain. I believe this sort of public advocacy can be extremely effective. In fact, in the case of previous emerging technologies, I would go so far as to say it's been too effective. Public advocacy played a major role in preventing the widespread adoption of GM crops and nuclear power in the twentieth century, despite huge financial incentives to develop these technologies. [3] We haven't seen this sort of activism leveraged to demand meaningful...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Paper] AI Sandbagging: Language Models can Strategically Underperform on Evaluations, published by Teun van der Weij on June 13, 2024 on LessWrong. We have written a paper on sandbagging for which we present the abstract and brief results in this post. See the paper for more details. Tweet thread here. Abstract Trustworthy capability evaluations are crucial for ensuring the safety of AI systems, and are becoming a key component of AI regulation. However, the developers of an AI system, or the AI system itself, may have incentives for evaluations to understate the AI's actual capability. These conflicting interests lead to the problem of sandbagging - which we define as strategic underperformance on an evaluation. In this paper we assess sandbagging capabilities in contemporary language models (LMs). We prompt frontier LMs, like GPT-4 and Claude 3 Opus, to selectively underperform on dangerous capability evaluations, while maintaining performance on general (harmless) capability evaluations. Moreover, we find that models can be fine-tuned, on a synthetic dataset, to hide specific capabilities unless given a password. This behaviour generalizes to high-quality, held-out benchmarks such as WMDP. In addition, we show that both frontier and smaller models can be prompted, or password-locked, to target specific scores on a capability evaluation. Even more, we found that a capable password-locked model (Llama 3 70b) is reasonably able to emulate a less capable model (Llama 2 7b). Overall, our results suggest that capability evaluations are vulnerable to sandbagging. This vulnerability decreases the trustworthiness of evaluations, and thereby undermines important safety decisions regarding the development and deployment of advanced AI systems. We publish our code at https://github.com/TeunvdWeij/sandbagging Results Evaluators focus on dangerous capability evaluations, where high performance may trigger regulatory interventions. We demonstrate that LMs can be prompted or fine-tuned to selectively underperform on dangerous capability evaluations while maintaining overall performance. Even smaller models can do this using "password-locking", which is fine-tuning models to hide their actual capabilities unless given a password. We show that when Mistral-7b or Llama3-8b are password locked on synthetic data, they still hide capabilities on a high-quality held-out benchmark (WMDP). If a seemingly competent AI system scores surprisingly low on a capability benchmark, detecting sandbagging is easy. However, we find that models can accurately aim for specific scores on benchmarks. Similarly, inconsistent or random performance can be detected. More credibly, strong models could hide capabilities by mimicking weaker models. We demonstrate this capability by successfully fine-tuning Llama 3 70b to emulate Llama 2 7b only when the password is not present. Our work suggests that capability evaluations are vulnerable to sandbagging, which is bad news, but good to know. In our following project, we will work on how to mitigate this problem. Reach out if you are interested in working on this. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on the Purpose of EAGs, published by Tristan Williams on June 13, 2024 on The Effective Altruism Forum. And Other Recommendations for EAGs Based on The Art of The Gathering The Motivation: I read The Art of the Gathering awhile ago, and thought it had a lot of interesting insights for greater community building efforts. I realized that EAGs[1] might be a ripe place to try to apply some of these insights, and so I set out writing this piece. What was meant to be 10 similar-in-length suggestions naturally turned into a piece focused on the purpose of EAGs, but I hope you'll still read over the other points too in the original spirit of the post, I think they have something to offer too. Epistemic Status: All the principles themselves have strong grounding in the book and are things I feel quite confident Parker would endorse. I'll make concrete suggestions below for each, and also venture a long commentary on the potential purpose of EAGs, which are both attempts at applications of these principles and are more speculative and should be taken as "Tristan's best guess". A brief note: CEA and individual EAGx organizers are already doing many of these things to some degree, and as such I hope to not only introduce new ideas or frames, but also highlight and reinforce some of the great things they're already doing. 1. EAGs should have a clear purpose. I think this is the most important point, so I'll spend a great bit of time on it. To put it another way, every gathering needs a greater why that justifies the what. I don't think this is spelled out well for EAGs and has been the cause of some confusion, and I think it would be good to decide on, and then state, a clear why for EAGs. Below I'll mention four different whys I think are at play, and what prioritizing them (more) would mean. For Learning: Talks are consistently an important part of EAGs, and some have argued that they've been underrated[2] in the process of promoting 1-on-1s. Significant staff time is spent organizing the content, giving indication that even if 1-on-1s are considered to be more valuable, there's got to be something worthwhile about them. I think the ToC here is that the talks afford a particularly good opportunity for others to further learn about a topic they're interested in but maybe haven't fully had the chance to explore yet, something likely to be especially valuable for those newer to the EA space. But the talks do actually afford more than this[3]: they are also a sort of podium by which CEA can try to diffuse ideas throughout the community, whether that be promoting a new potential cause area, a better recognition of needs in the EA space, or desirable cultural norms. Focusing the content in 2017 on operations, for example, allowed the community to address a significant gap at the time. They're also a way to expand impact outside the conference when recorded, which could be important as CEA begins to focus more on EA branding and outwards content. Either way, I think many people only have a vague sense of the purpose Learning currently serves, and I think we're even further behind on figuring out how to measure the effectiveness of Learning specifically vs the general impact. Implications: Focus more content on connection. If talks aren't just informational, but also are used to mold the norms of the community both in and outside the conference, maybe more talks should be had in service of helping people make better/more connections. Yes, there's already some of this in the first timers meetup, and motivation in the introductory talks, but I still think far more content should go towards this end given 1-on-1s seems to be the greatest source of value. Be more explicit about the purpose of content at a given EAG. Perhaps there could be some short statement placed in the attende...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quantifying and prioritizing shrimp welfare threats, published by Hannah McKay on June 13, 2024 on The Effective Altruism Forum. Citation: McKay, H. and McAuliffe, W. (2024). Quantifying and prioritizing shrimp welfare threats. Rethink Priorities. https://doi.org/10.17605/OSF.IO/4QR8K The report is also available on the Rethink Priorities website and as a pdf here . Executive summary This is the fourth report in the Rethink Priorities Shrimp Welfare Sequence. In this report, we quantify the suffering caused to shrimp by 18 welfare threats to assess which welfare issues cause the most harm. See a complete description of the methodology here. We focused on penaeid shrimp in ongrowing farms and broodstock facilities. Incorporating uncertainty at each step of the model, we estimated the prevalence, intensity, and duration of pain caused by each welfare issue. The intensity was based on the Welfare Footprint Project's Pain-Track categories, and 'pain' refers to their definition of pain, encapsulating both physical and mental negative experiences ( Alonso & Schuck-Paim, 2024a). We collapse different pain type estimates into a single metric: 'Disabling-equivalent pain'. See the results in Figure 1. The average farmed shrimp spends 154 hours in disabling-equivalent pain (95% Subjective Credible Interval (SCI): [13, 378]). If we assume that 608 billion penaeid shrimp die on ongrowing farms annually (i.e., including those that die pre-slaughter; Waldhorn & Autric, 2023) then mean values imply that they experience 94 trillion hours of disabling-equivalent pain a year (95% SCI: [8 trillion, 230 trillion]). The highest-ranking threats are chronic issues that affect most farmed shrimp. The top three are high stocking density, high un-ionized ammonia, and low dissolved oxygen. Threats ranked lower are broadly acute, one-off events affecting only a subpopulation (e.g., eyestalk ablation, which affects only broodstock). However, the credible intervals are too wide to determine the rank order of most welfare issues confidently. Box 1: Shrimp aquaculture terminology The terms 'shrimp' and 'prawn' are often used interchangeably. The two terms do not reliably track any phylogenetic differences between species. Here, we use only the term "shrimp", covering both shrimp and prawns. Note that members of the family Artemiidae are commonly referred to as "brine shrimp" but are not decapods and so are beyond the present scope. We opt for the use of Penaues vannamei over Litopenaeus vannamei (to which this species is often referred), due to recognition of the former but not the latter nomenclature by ITIS, WorMS, and the Food and Agriculture Organization of the United Nations (FAO) ASFIS List of Species for Fishery Statistics Purposes. The shrimp farming industry uses many terms usually associated with agriculture - for example, 'crops' for a group of shrimp reared together, 'seed' for the first shrimp stocked into a pond, and 'harvest' for collecting and slaughtering shrimp. For clarity, we broadly conform to this terminology. Although we acknowledge animal welfare advocates may prefer terminology that does not euphemize or sanitize the experience of farmed shrimp, here we favor ensuring readability for a wide audience. Introduction We began the Shrimp Welfare Sequence by asking whether the Animal Sentience Precautionary Principle ( Birch, 2017, p. 3) justifies implementing reforms in shrimp aquaculture. The first three posts collectively provide an affirmative answer: More shrimp are alive on farms than any other farmed taxa Half of them die before slaughter, suggesting that some of the welfare threats they endure must be serious. The welfare threats shrimp experience are varied, ranging from poor water quality to environmental deprivation to inhumane slaughter. Unfortunately, it is probably n...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Paper] AI Sandbagging: Language Models can Strategically Underperform on Evaluations, published by Teun van der Weij on June 13, 2024 on The AI Alignment Forum. We have written a paper on sandbagging for which we present the abstract and brief results in this post. See the paper for more details. Tweet thread here. Abstract Trustworthy capability evaluations are crucial for ensuring the safety of AI systems, and are becoming a key component of AI regulation. However, the developers of an AI system, or the AI system itself, may have incentives for evaluations to understate the AI's actual capability. These conflicting interests lead to the problem of sandbagging - which we define as strategic underperformance on an evaluation. In this paper we assess sandbagging capabilities in contemporary language models (LMs). We prompt frontier LMs, like GPT-4 and Claude 3 Opus, to selectively underperform on dangerous capability evaluations, while maintaining performance on general (harmless) capability evaluations. Moreover, we find that models can be fine-tuned, on a synthetic dataset, to hide specific capabilities unless given a password. This behaviour generalizes to high-quality, held-out benchmarks such as WMDP. In addition, we show that both frontier and smaller models can be prompted, or password-locked, to target specific scores on a capability evaluation. Even more, we found that a capable password-locked model (Llama 3 70b) is reasonably able to emulate a less capable model (Llama 2 7b). Overall, our results suggest that capability evaluations are vulnerable to sandbagging. This vulnerability decreases the trustworthiness of evaluations, and thereby undermines important safety decisions regarding the development and deployment of advanced AI systems. We publish our code at https://github.com/TeunvdWeij/sandbagging Results Evaluators focus on dangerous capability evaluations, where high performance may trigger regulatory interventions. We demonstrate that LMs can be prompted or fine-tuned to selectively underperform on dangerous capability evaluations while maintaining overall performance. Even smaller models can do this using "password-locking", which is fine-tuning models to hide their actual capabilities unless given a password. We show that when Mistral-7b or Llama3-8b are password locked on synthetic data, they still hide capabilities on a high-quality held-out benchmark (WMDP). If a seemingly competent AI system scores surprisingly low on a capability benchmark, detecting sandbagging is easy. However, we find that models can accurately aim for specific scores on benchmarks. Similarly, inconsistent or random performance can be detected. More credibly, strong models could hide capabilities by mimicking weaker models. We demonstrate this capability by successfully fine-tuning Llama 3 70b to emulate Llama 2 7b only when the password is not present. Our work suggests that capability evaluations are vulnerable to sandbagging, which is bad news, but good to know. In our following project, we will work on how to mitigate this problem. Reach out if you are interested in working on this. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Three Preconditions for Helping Wild Animals at Scale, published by William McAuliffe on June 13, 2024 on The Effective Altruism Forum. Executive Summary 1. A theory of change specifies how a social movement will achieve the change it desires. The theory first posits preconditions that are necessary for meeting its goals. It then explains how the movement's activities help meet the preconditions. This report lays out the preconditions for the wild animal welfare movement to help wild animals at scale. 2. The movement's main goal is to promote the interests of individual nonhuman animals not under the direct control of humans as ends in themselves. 1. The movement's fundamental normative assumption is that speciesism is ethically unjustifiable. 2. The fundamental empirical assumption is that wild animals face a number of anthropogenic and non-anthropogenic threats. 3. Humans already have the ability to help some wild animals with some problems. But three preconditions must be developed in order to help a substantial fraction of wild animals with the conditions have the biggest negative impact on their welfare: 1. Valid measurement: Knowledge of (a) how to measure well-being among wild animals and (b) the causal relationships among the factors that influence it. 2. Technical Ability: Technology and skill to implement and evaluate interventions to help wild animals at scale, while minimizing unintended negative consequences. 3. Stakeholder Buy-In: Consent from stakeholders with veto power, and collaboration from stakeholders who can implement scalable interventions. Who Should Read This Report Newcomers to wild animal welfare who want a primer that is fairly comprehensive and up-to-date. Readers of A Landscape Analysis of Wild Animal Welfare who want a justification for the preconditions we use to contextualize the movement's ongoing activities. Introduction There is growing concern that wild animals do not receive the degree of moral consideration they deserve.[1] However, there is little common understanding of what minimal conditions must be met to improve wild animal welfare. The process of articulating a "theory of change" can help clarify assumptions about how a movement's activities will help it achieve its long-term goals ( Weiss, 1995). A theory of change is a "comprehensive description and illustration of how and why a desired change is expected to happen in a particular context" ( Center for Theory of Change, n.d.). In essence, one reasons backwards from the desired outcome to the preconditions required to obtain that outcome, and in turn to the activities required to bring about those preconditions. We claim that, at the broadest level of analysis, there are three preconditions for helping wild animals, which we label Valid Measurement, Technical Ability, and Stakeholder Buy-In. In our companion report (Elmore & McAuliffe, 2024), we describe ongoing activities to help meet these preconditions. A Primer on Wild Animal Welfare Defining Wild Animal Welfare We define wild animal welfare as a movement that aims to promote the interests of individual nonhuman animals not under the direct control of humans as ends in themselves. In particular, the goal is to help at scale so that all wild animal populations can benefit and all major threats can be addressed. The phrase "not under the direct control of humans" refers to both (a) "wild" animals, who live in habitats that were not intentionally constructed by humans, and (b) "liminal" animals, who are free-ranging within areas settled by humans ( Donaldson & Kymlicka, 2011, p. 210). The term "interests" is intentionally broad, in order to accommodate disagreement about what welfare consists of. The emphasis on "individuals" distinguishes wild animal welfare from those that focus on collectives. For example, although m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: microwave drilling is impractical, published by bhauth on June 13, 2024 on LessWrong. microwave drilling startups I've seen a bunch of articles about startups trying to do microwave drilling of rock for geothermal energy. Multiple people have asked me about Quaise Energy. (Here's a popular video.) I'm tired of hearing about them, so I'm writing this post to explain some of the reasons why their idea is impractical. vaporized rock condenses When rock is vaporized, that rock vapor doesn't just disappear. What happens to it? The answer is, it would quickly condense on the hole wall and pipe. Initially, a lot of people working on microwave drilling didn't even think about that. Once they did, they decided the solution was to use compressed air to condense the rock and blow the rock particles out. But as anyone familiar with drilling would know, that introduces new problems. air pressure Current drilling sometimes uses air to lift up rock particles, but "rotary air blast" (RAB) drilling has limited depth, because: Air velocity at the bottom of the hole needs to be high enough to lift up rock particles. That means the bottom part of the hole needs a certain pressure drop per distance. So, the deeper the hole is, the higher the air pressure needs to be. 1 km depth requires about 300 psi, and obviously deeper holes require even higher pressure. Higher pressure means more gas per volume, so energy usage increases faster than depth. That's why drilling of deeper holes uses liquid ("mud") instead of air to lift rock particles. But here's Quaise, saying they're going to do ultra-deep holes with air. At the depths they propose, there are even more problems: A pipe to contain 1000+ psi gas would be pretty thick and heavy. At some point, the gas itself starts becoming a significant weight, and then required pressure increases exponentially. I suppose the particle size of condensed rock could theoretically be smaller than RAB particles and thus require a lower pressure drop, but that's not necessarily the case. Hot rock particles would stick together. Also, particle size depends on the mixing rate at the bottom, and fast mixing requires fast flow requires a significant pressure drop rate at the bottom of the hole. energy payback energy usage Vaporizing rock takes ~25 kJ/cm^3, or ~7 MWh/m^3. That doesn't include heat loss to surrounding rock, and microwave sources and transmission have some inefficiency. In order to cool vaporized rock down to a reasonable temperature, you need a lot of air, perhaps 20x the mass of the rock. Supposing the air is 500 psi, the rock is granite, and compression has some inefficiency, that'd be another, say, 5 MWh per m^3 of rock. thermal conductivity Rock has fairly low thermal conductivity. Existing geothermal typically uses reservoirs of hot water that flows out the hole, so thermal conductivity of the rock isn't an issue because the water is already hot. (It's like drilling for oil, but oil is less common and contains much more energy than hot water.) Current "enhanced geothermal" approaches use fracking and pumps water through the cracks between 2 holes, which gives a lot of surface area for heat transfer. And then after a while the rock cools down. With a single hole, thermal conductivity is a limiting factor. The rock around the hole cools down before much power is produced. The area for heat transfer is linear with distance from the hole, so the temperature drop scales with ln(time). payback period The heat collected from the rock during operation would be converted to electricity at <40% net efficiency. The efficiency would be worse than ultra-supercritical coal plants because the efficiency would be lower and pumping losses would be much higher. Considering the efficiencies involved, and the thermal conductivity and thermal mass of rock, the roc...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AiPhone, published by Zvi on June 13, 2024 on LessWrong. Apple was for a while rumored to be planning launch for iPhone of AI assisted emails, texts, summaries and so on including via Siri, to be announced at WWDC 24. It's happening. Apple's keynote announced the anticipated partnership with OpenAI. The bottom line is that this is Siri as the AI assistant with full access to everything on your phone, with relatively strong privacy protections. Mostly it is done on device, the rest via 'private cloud compute.' The catch is that when they need the best they call out for OpenAI, but they do check with you first explicitly each time, OpenAI promises not to retain data and they hide your queries, unless you choose to link up your account. If the new AI is good enough and safe enough then this is pretty great. If Google doesn't get its act together reasonably soon to deliver on its I/O day promises, and Apple does deliver, this will become a major differentiator. AiPhone They call it Apple Intelligence, after first calling it Personal Intelligence. The pitch: Powerful, intuitive, integrated, personal, private, for iPhone, iPad and Mac. The closing pitch: AI for the rest of us. It will get data and act across apps. It will understand personal context. It is fully multimodal. The focus is making everything seamless, simple, easy. They give you examples: 1. Your iPhone can prioritize your notifications to prevent distractions, so you don't miss something important. Does that mean you will be able to teach it what counts as important? How will you do that and how reliable will that be? Or will you be asked to trust the AI? The good version here seems great, the bad version would only create paranoia of missing out. 2. Their second example is a writing aid, for summaries or reviews or to help you write. Pretty standard. Question is how much it will benefit from context and how good it is. I essentially never use AI writing tools aside from the short reply generators, because it is faster for me to write than to figure out how to get the AI to write. But even for me, if the interface is talk to your phone to have it properly format and compose an email, the quality bar goes way down. 3. Images to make interactions more fun. Create images of your contacts, the AI will know what they look like. Wait, what? The examples have to be sketches, animations or cartoons, so presumably they think they are safe from true deepfakes unless someone uses an outside app. Those styles might be all you get? The process does seem quick and easy to generate images in general and adjust to get it to do what you want, which is nice. Resolution and quality seems fine for texting, might be pretty lousy if you want better. Image wand, which can work off an existing image, might be more promising, but resolution still seems low. 4. The big game. Take actions across apps. Can access your photos, your emails, your podcasts, presumably your everything. Analyze the data across all your apps. Their example is using maps plus information from multiple places to see if you can make it from one thing to the next in time. Privacy Then at 1:11:40 they ask the big question. What about privacy? They say this all has 'powerful privacy.' The core idea is on-device processing. They claim this is 'only possible due to years of planning and investing in advanced silicon for on device intelligence.' The A17 and M1-4 can provide the compute for the language and diffusion models, which they specialized for this. An on-device semantic index assists with this. What about when you need more compute than that? Servers can misuse your data, they warn, and you wouldn't know. So they propose Private Cloud Compute. It runs on servers using Apple Silicon, use Swift for security (ha!) and are secure. If necessary, only the necessary d...