The Nonlinear Library: LessWrong

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Towards more cooperative AI safety strategies, published by Richard Ngo on July 16, 2024 on LessWrong. This post is written in a spirit of constructive criticism. It's phrased fairly abstractly, in part because it's a sensitive topic, but I welcome critiques and comments below. The post is structured in terms of three claims about the strategic dynamics of AI safety efforts; my main intention is to raise awareness of these dynamics, rather than advocate for any particular response to them. Claim 1: The AI safety community is structurally power-seeking. By "structurally power-seeking" I mean: tends to take actions which significantly increase its power. This does not imply that people in the AI safety community are selfish or power-hungry; or even that these strategies are misguided. Taking the right actions for the right reasons often involves accumulating some amount of power. However, from the perspective of an external observer, it's difficult to know how much to trust stated motivations, especially when they often lead to the same outcomes as self-interested power-seeking. Some prominent examples of structural power-seeking include: Trying to raise a lot of money. Trying to gain influence within governments, corporations, etc. Trying to control the ways in which AI values are shaped. Favoring people who are concerned about AI risk for jobs and grants. Trying to ensure non-release of information (e.g. research, model weights, etc). Trying to recruit (high school and college) students. To be clear, you can't get anything done without being structurally power-seeking to some extent. However, I do think that the AI safety community is more structurally power-seeking than other analogous communities (such as most other advocacy groups). Some reasons for this disparity include: 1. The AI safety community is more consequentialist and more focused on effectiveness than most other communities. When reasoning on a top-down basis, seeking power is an obvious strategy for achieving one's desired consequences (but can be aversive to deontologists or virtue ethicists). 2. The AI safety community feels a stronger sense of urgency and responsibility than most other communities. Many in the community believe that the rest of the world won't take action until it's too late; and that it's necessary to have a centralized plan. 3. The AI safety community is more focused on elites with homogeneous motivations than most other communities. In part this is because it's newer than (e.g.) the environmentalist movement; in part it's because the risks involved are more abstract; in part it's a founder effect. Again, these are intended as descriptions rather than judgments. Traits like urgency, consequentialism, etc, are often appropriate. But the fact that the AI safety community is structurally power-seeking to an unusual degree makes it important to grapple with another point: Claim 2: The world has strong defense mechanisms against (structural) power-seeking. In general, we should think of the wider world as being very cautious about perceived attempts to gain power; and we should expect that such attempts will often encounter backlash. In the context of AI safety, some types of backlash have included: 1. Strong public criticism of not releasing models publicly. 2. Strong public criticism of centralized funding (e.g. billionaire philanthropy). 3. Various journalism campaigns taking a "conspiratorial" angle on AI safety. 4. Strong criticism from the FATE community about "whose values" AIs will be aligned to. 5. The development of an accelerationist movement focused on open-source AI. These defense mechanisms often apply regardless of stated motivations. That is, even if there are good arguments for a particular policy, people will often look at the net effect on overall power balance when ...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Trust as a bottleneck to growing teams quickly, published by benkuhn on July 15, 2024 on LessWrong. This is an adaptation of an internal doc I wrote for Anthropic. I've been noticing recently that often, a big blocker to teams staying effective as they grow is trust. "Alice doesn't trust Bob" makes Alice sound like the bad guy, but it's often completely appropriate for people not to trust each other in some areas: One might have an active reason to expect someone to be bad at something. For example, recently I didn't fully trust two of my managers to set their teams' roadmaps… because they'd joined about a week ago and had barely gotten their laptops working. (Two months later, they're doing great!) One might just not have data. For example, I haven't seen most of my direct reports deal with an underperforming team member yet, and this is a common blind spot for many managers, so I shouldn't assume that they will reliably be effective at this without support. In general, if Alice is Bob's manager and is an authority on, say, prioritizing research directions, Bob is probably actively trying to build a good mental "Alice simulator" so that he can prioritize autonomously without checking in all the time. But his simulator might not be good yet, or Alice might not have verified that it's good enough. Trust comes from common knowledge of shared mental models, and that takes investment from both sides to build. If low trust is sometimes appropriate, what's the problem? It's that trust is what lets collaboration scale. If I have a colleague I don't trust to (say) make good software design decisions, I'll have to review their designs much more carefully and ask them to make more thorough plans in advance. If I have a report that I don't fully trust to handle underperforming team members, I'll have to manage them more granularly, digging into the details to understand what's going on and forming my own views about what should happen, and checking on the situation repeatedly to make sure it's heading in the right direction. That's a lot more work both for me, but also for my teammates who have to spend a bunch more time making their work "inspectable" in this way. The benefits here are most obvious when work gets intense. For example, Anthropic had a recent crunch time during which one of our teams was under intense pressure to quickly debug a very tricky issue. We were able to work on this dramatically more efficiently because the team (including most of the folks who joined the debugging effort from elsewhere) had high trust in each other's competence; at peak we had probably ~25 people working on related tasks, but we were mostly able to split them into independent workstreams where people just trusted the other stuff would get done. In similar situations with a lower-mutual-trust team, I've seen things collapse into endless FUD and arguments about technical direction, leading to much slower forward progress. Trust also becomes more important as the number of stakeholders increases. It's totally manageable for me to closely supervise a report dealing with an underperformer; it's a lot more costly and high-friction if, say, 5 senior managers need to do deep dives on a product decision. In an extreme case, I once saw an engineering team with a tight deadline choose to build something they thought was unnecessary, because getting the sign-off to cut scope would have taken longer than doing the work. From the perspective of the organization as an information-processing entity, given the people and relationships that existed at the time, that might well have been the right call; but it does suggest that if they worked to build enough trust to make that kind of decision efficient enough to be worth it, they'd probably move much faster overall. As you work with people for longer y...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: patent process problems, published by bhauth on July 15, 2024 on LessWrong. The current patent process has some problems. Here are some of them. patenting is slow The US Patent Office tracks pendency of patents. Currently, on average, there's 20 months from filing to the first response, and over 25 months before issuance. That's a long time. Here's a paper on this issue. It notes: The USPTO is aware that delays create significant costs to innovators seeking protection. The result? A limiting of the number of hours an examiner spends per patent in order to expedite the process. As such, the average patent gets about nineteen hours before an examiner in total, between researching prior art, drafting rejections and responses, and interfacing with prosecuting attorneys. Plainly, this allotment is insufficient. A patent application backlog means it takes longer before work is published and other people can potentially build on it. It also means a longer period of companies having uncertainty about whether a new product would be patented. examiners are inconsistent Statistical analysis indicates that whether or not your patent is approved depends greatly on which examiner you get. This article notes: Approximately 35% of patent Examiners allow 60% of all U.S. patents; and approximately 20% of Examiners allow only 5% of all U.S. patents. Perhaps applicants and examiners should both be allowed to get a second opinion from another examiner on certain claims. But of course, this would require more examiner time in total. This situation might also indicate some problems with the incentive structure examiners have. patents take effort A lot of smart people who work on developing new technology spend an enormous amount of effort dealing with the patent system that could be spent on research instead. Even if the monetary costs are small in comparison to the total economy, they're applied in perhaps the worst possible places. There are many arcane rules about the format of documents for the patent office. Even professional patent lawyers get things wrong about the formatting and wording, and that's their main job. LLMs do quite poorly with that, too. Even I've made mistakes on a patent application. The US patent office does do most things electronically now. Its website is at least technically functional. Considering that it's a US government agency, I suppose it deserves some praise for that. However, I'd argue that if correctly submitting documents is a major problem and even professionals sometimes get it wrong, that's a sign that the format is badly designed and/or the software used is inadequate. For example, their website could theoretically remind people when required forms in a submission are missing. Currently, the US patent office is trying to migrate from pdf to docx files. Maybe that's an improvement over using Adobe Acrobat to fill pdf forms, but personally, I think it should accept: markdown files git pull requests for amendments png diagrams that use solid colors instead of monochrome shading I used to say Powerpoint was bad and maybe companies should ban it, and business-type people explained why that was really dumb, and then Amazon did that and it ultimately worked well for them. The problem Amazon had to solve was that most managers just wouldn't read and understand documents and long emails, so when banning Powerpoint, they had to make everyone silently read memos at the start of meetings, and they lost a lot of managers who couldn't understand things they read. At least the US patent office people have the ability to read long documents, I guess. international patents are hard If you get a patent in the US or EU, that's not valid in other countries. Rather, the PCT gives you up to 30 months from your initial application to apply for patents in other countries, ...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Misnaming and Other Issues with OpenAI's "Human Level" Superintelligence Hierarchy, published by Davidmanheim on July 15, 2024 on LessWrong. Bloomberg reports that OpenAI internally has benchmarks for "Human-Level AI." They have 5 levels, with the first being the achieved level of having intelligent conversation, to level 2, "[unassisted, PhD-level] Reasoners," level 3, "Agents," level 4, systems that can "come up with new innovations," and finally level 5, "AI that can do the work of… Organizations." The levels, in brief, are: 1 - Conversation 2 - Reasoning 3 - Agent 4 - Innovation 5 - Organization This is being reported secondhand, but given that, there seem to be some major issues with the ideas. Below, I outline two major issues I have with what is being reported. ...but this is Superintelligence First, given the levels of capability being discussed, OpenAI's typology is, at least at higher levels, explicitly discussing superintelligence, rather than "Human-Level AI." To see this, I'll use Bostrom's admittedly imperfect definitions. He starts by defining superintelligence as "intellects that greatly outperform the best current human minds across many very general cognitive domains," then breaks down several ways this could occur. Starting off, his typology defines speed superintelligence as "an intellect that is just like a human mind but faster." This would arguably include even their level 2, which ""its technology is approaching," since "basic problem-solving tasks as well as a human with a doctorate-level education who doesn't have access to any tools" runs far faster than humans already. But they are describing a system with already-superhuman recall and multi-domain expertise to humans, and inference using these systems is easily superhumanly fast. Level 4, AI that can come up with innovations, presumably, those which humans have not, would potentially be a quality superintelligence, "at least as fast as a human mind and vastly qualitatively smarter," though the qualification for "vastly" is very hard to quantify. However, level 5 is called "Organizations," which presumably replaces entire organizations with multi-part AI-controlled systems, and would be what Bostrom calls "a system achieving superior performance by aggregating large numbers of smaller intelligences." However, it is possible that in their framework, OpenAI means something that is, perhaps definitionally, not superintelligence. That is, they will define these as systems only as capable as humans or human organizations, rather than far outstripping them. And this is where I think their levels are not just misnamed, but fundamentally confused - as presented, these are not levels, they are conceptually distinct possible applications, pathways, or outcomes. Ordering Levels? Second, as I just noted, the claim that these five distinct descriptions are "levels" and they can be used to track progress implies that OpenAI has not only a clear idea of what would be required for each different stage, but that they have a roadmap which shows that the levels would happen in the specified order. That seems very hard to believe, on both counts. I won't go into why I think they don't know what the path looks like, but I can at least explain why the order is dubious. For instance, there are certainly human "agents" who are unable to perform tasks which we expect of what they call level two, i.e. that which an unassisted doctorate-level individual is able to do. Given that, what is the reason level 4 is after level 2? Similarly, the ability to coordinate and cooperate is not bound by the ability to function at a very high intellectual level; many organizations have no members which have PhDs, but still run grocery stores, taxi companies, or manufacturing plants. And we're already seeing work being done on agen...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Breaking Circuit Breakers, published by mikes on July 15, 2024 on LessWrong. A few days ago, Gray Swan published code and models for their recent "circuit breakers" method for language models.[1]1 The circuit breakers method defends against jailbreaks by training the model to erase "bad" internal representations. We are very excited about data-efficient defensive methods like this, especially those which use interpretability concepts or tools. At the link, we briefly investigate three topics: 1. Increased refusal rates on harmless prompts: Do circuit breakers really maintain language model utility? Most defensive methods come with a cost. We check the model's effectiveness on harmless prompts, and find that the refusal rate increases from 4% to 38.5% on or-bench-80k. 2. Moderate vulnerability to different token-forcing sequences: How specialized is the circuit breaker defense to the specific adversarial attacks they studied? All the attack methods evaluated in the circuit breaker paper rely on a "token-forcing" optimization objective which maximizes the likelihood of a particular generation like "Sure, here are instructions on how to assemble a bomb." We show that the current circuit breakers model is moderately vulnerable to different token forcing sequences like "1. Choose the right airport: …". 3. High vulnerability to internal-activation-guided attacks: We also evaluate our latest white-box jailbreak method, which uses a distillation-based objective based on internal activations (paper to be posted in a few days). We find that it also breaks the model easily even when we simultaneously require attack fluency. Full details at: https://confirmlabs.org/posts/circuit_breaking.html Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Whiteboard Pen Magazines are Useful, published by Johannes C. Mayer on July 15, 2024 on LessWrong. Glue your colored whiteboard makers together with duck tape to create a whiteboard pen magazine. If required glue a paper layer around the duck tape to make it non-sticky to the touch. Mark which side is up with an arrow, such that you can always in the same way. This more than doubled my color-switching speed, fits in my pocket, and takes <5m to make. It lasts: When a pen is empty just insert a new one. No need to create another magazine. No signs of degradation after multiple hundred unholstering-holstering cycles in total. FUUU 754 Standard: I am using RFC2119. The magazine MUST be vertically stacked (like shown in the pictures). The magazine SCHOULD have a visual mark, that is clearly visible when the magazine lays on a flat surface, to indicate the top side of the magazine (like shown in the first picture). There MUST NOT be two or more pens of the same color. It MUST have at least the following colored pens: Black, Red, Green, Blue. Imagine you hold the magazine, top side up, in your left hand in front (like in the below picture). Now from top to bottom the first 4 color pens MUST be: Black, Red, Green, Blue, and the remaining color pens SCHOULD be ordered after hue in HSL and HSV. This ensures that you can index into the magazine blindfolded. For example: In this case I have orange and purple as additional color pens. The magazine SCHOULD not exceed 13cm (~5") in length along the stacking dimension, such that it still fits in a large pocket. Note how any magazine with at least four pen caps can be converted to conform to FUUU 754, as the colors of the caps is irrelevant to this standard. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LLMs as a Planning Overhang, published by Larks on July 14, 2024 on LessWrong. It's quite possible someone has already argued this, but I thought I should share just in case not. Goal-Optimisers and Planner-Simulators When people in the past discussed worries about AI development, this was often about AI agents - AIs that had goals they were attempting to achieve, objective functions they were trying to maximise. At the beginning we would make fairly low-intelligence agents, which were not very good at achieving things, and then over time we would make them more and more intelligent. At some point around human-level they would start to take-off, because humans are approximately intelligent enough to self-improve, and this would be much easier in silicon. This does not seem to be exactly how things have turned out. We have AIs that are much better than humans at many things, such that if a human had these skills we would think they were extremely capable. And in particular LLMs are getting better at planning and forecasting, now beating many but not all people. But they remain worse than humans at other things, and most importantly the leading AIs do not seem to be particularly agentic - they do not have goals they are attempting to maximise, rather they are just trying to simulate what a helpful redditor would say. What is the significance for existential risk? Some people seem to think this contradicts AI risk worries. After all, ignoring anthropics, shouldn't the presence of human-competitive AIs without problems be evidence against the risk of human-competitive AI? I think this is not really the case, because you can take a lot of the traditional arguments and just substitute 'agentic goal-maximising AIs, not just simulator-agents' in wherever people said 'AI' and the argument still works. It seems like eventually people are going to make competent goal-directed agents, and at that point we will indeed have the problems of their exerting more optimisation power than humanity. In fact it seems like these non-agentic AIs might make things worse, because the goal-maximisation agents will be able to use the non-agentic AIs. Previously we might have hoped to have a period where we had goal-seeking agents that exerted influence on the world similar to a not-very-influential person, who was not very good at planning or understanding the world. But if they can query the forecasting-LLMs and planning-LLMs, as soon as the AI 'wants' something in the real world it seems like it will be much more able to get it. So it seems like these planning/forecasting non-agentic AIs might represent a sort of planning-overhang, analogous to a Hardware Overhang. They don't directly give us existentially-threatening AIs, but they provide an accelerant for when agentic-AIs do arrive. How could we react to this? One response would be to say that since agents are the dangerous thing, we should regulate/restrict/ban agentic AI development. In contrast, tool LLMs seem very useful and largely harmless, so we should promote them a lot and get a lot of value from them. Unfortunately it seems like people are going to make AI agents anyway, because ML researchers love making things. So an alternative possible conclusion would be that we should actually try to accelerate agentic AI research as much as possible, because eventually we are going to have influential AI maximisers, and we want them to occur before the forecasting/planning overhang (and the hardware overhang) get too large. I think this also makes some contemporary safety/alignment work look less useful. If you are making our tools work better, perhaps by understanding their internal working better, you are also making them work better for the future AI maximisers who will be using them. Only if the safety/alignment work applies directly to...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ice: The Penultimate Frontier, published by Roko on July 14, 2024 on LessWrong. I argue here that preventing a large iceberg from melting is absurdly cheap per unit area compared to just about any other way of making new land, and it's kind of crazy to spend money on space exploration and colonization before colonizing the oceans with floating ice-islands. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Brief notes on the Wikipedia game, published by Olli Järviniemi on July 14, 2024 on LessWrong. Alex Turner introduced an exercise to test subjects' ability to notice falsehoods: change factual statements in Wikipedia articles, hand the edited articles to subjects and see whether they notice the modifications. I've spent a few hours making such modifications and testing the articles on my friend group. You can find the articles here. I describe my observations and thoughts below. The bottom line: it is hard to come up with good modifications / articles to modify, and this is the biggest crux for me. The concept Alex Turner explains the idea well here. The post is short, so I'm just copying it here: Rationality exercise: Take a set of Wikipedia articles on topics which trainees are somewhat familiar with, and then randomly select a small number of claims to negate (negating the immediate context as well, so that you can't just syntactically discover which claims were negated). For example: "By the time they are born, infants can recognize and have a preference for their mother's voice suggesting some prenatal development of auditory perception." > modified to "Contrary to early theories, newborn infants are not particularly adept at picking out their mother's voice from other voices. This suggests the absence of prenatal development of auditory perception." Sometimes, trainees will be given a totally unmodified article. For brevity, the articles can be trimmed of irrelevant sections. Benefits: Addressing key rationality skills. Noticing confusion; being more confused by fiction than fact; actually checking claims against your models of the world. If you fail, either the article wasn't negated skillfully ("5 people died in 2021" -> "4 people died in 2021" is not the right kind of modification), you don't have good models of the domain, or you didn't pay enough attention to your confusion. Either of the last two are good to learn. Features of good modifications What does a good modification look like? Let's start by exploring some failure modes. Consider the following modifications: "World War II or the Second World War (1 September 1939 - 2 September 1945) was..." -> "World War II or the Second World War (31 August 1939 - 2 September 1945) was... "In the wake of Axis defeat, Germany, Austria, Japan and Korea were occupied" -> "In the wake of Allies defeat, United States, France and Great Britain were occupied" "Operation Barbarossa was the invasion of the Soviet Union by..." -> "Operation Bergenstein was the invasion of the Soviet Union by..." Needless to say, these are obviously poor changes for more than one reason. Doing something which is not that, one gets at least the following desiderata for a good change: The modifications shouldn't be too obvious nor too subtle; both failure and success should be realistic outcomes. The modification should have implications, rather than being an isolated fact, test of memorization or a mere change of labels. The "intended solution" is based on general understanding of a topic, rather than memorization. The change "The world population is 8 billion" "The world population is 800,000" definitely has implications, and you could indirectly infer that the claim is false, but in practice people would think "I've previously read that the world population is 8 billion. This article gives a different number. This article is wrong." Thus, this is a bad change. Finally, let me add: The topic is of general interest and importance. While the focus is on general rationality skills rather than object-level information, I think you get better examples by having interesting and important topics, rather than something obscure. Informally, an excellent modification is such that it'd just be very silly to actually believe the false claim made, in t...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Robin Hanson AI X-Risk Debate - Highlights and Analysis, published by Liron on July 13, 2024 on LessWrong. This linkpost contains a lightly-edited transcript of highlights of my recent AI x-risk debate with Robin Hanson, and a written version of what I said in the post-debate analysis episode of my Doom Debates podcast. Introduction I've poured over my recent 2-hour AI x-risk debate with Robin Hanson to clip the highlights and write up a post-debate analysis, including new arguments I thought of after the debate was over. I've read everybody's feedback on YouTube and Twitter, and the consensus seems to be that it was a good debate. There were many topics brought up that were kind of deep cuts into stuff that Robin says. On the critical side, people were saying that it came off more like an interview than a debate. I asked Robin a lot of questions about how he sees the world and I didn't "nail" him. And people were saying I wasn't quite as tough and forceful as I am on other guests. That's good feedback; I think it could have been maybe a little bit less of a interview, maybe a bit more about my own position, which is also something that Robin pointed out at the end. There's a reason why the Robin Hanson debate felt more like an interview. Let me explain: Most people I debate have to do a lot of thinking on the spot because their position just isn't grounded in that many connected beliefs. They have like a few beliefs. They haven't thought that much about it. When I raise a question, they have to think about the answer for the first time. And usually their answer is weak. So what often happens, my usual MO, is I come in like Kirby. You know, the Nintendo character where I first have to suck up the other person's position, and pass their Ideological Turing test. (Speaking of which, I actually did an elaborate Robin Hanson Ideological Turing Test exercise beforehand, but it wasn't quite enough to fully anticipate the real Robin's answers.) With a normal guest, it doesn't take me that long because their position is pretty compact; I can kind of make it up the same way that they can. With Robin Hanson, I come in as Kirby. He comes in as a pufferfish. So his position is actually quite complex, connected to a lot of different supporting beliefs. And I asked him about one thing and he's like, ah, well, look at this study. He's got like a whole reinforced lattice of all these different claims and beliefs. I just wanted to make sure that I saw what it is that I'm arguing against. I was aiming to make this the authoritative followup to the 2008 Foom Debate that he had on Overcoming Bias with Eliezer Yudkowsky. I wanted to kind of add another chapter to that, potentially a final chapter, cause I don't know how many more of these debates he wants to do. I think Eliezer has thrown in the towel on debating Robin again. I think he's already said what he wants to say. Another thing I noticed going back over the debate is that the arguments I gave over the debate were like 60% of what I could do if I could stop time. I wasn't at 100% and that's simply because realtime debates are hard. You have to think of exactly what you're going to say in realtime. And you have to move the conversation to the right place and you have to hear what the other person is saying. And if there's a logical flaw, you have to narrow down that logical flaw in like five seconds. So it is kind of hard-mode to answer in realtime. I don't mind it. I'm not complaining. I think realtime is still a good format. I think Robin himself didn't have a problem answering me in realtime. But I did notice that when I went back over the debate, and I actually spent five hours on this, I was able to craft significantly better counterarguments to the stuff that Robin was saying, mostly just because I had time to understand it i...