The Nonlinear Library: LessWrong

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Humming is not a free $100 bill, published by Elizabeth on June 6, 2024 on LessWrong. Last month I posted about humming as a cheap and convenient way to flood your nose with nitric oxide (NO), a known antiviral. Alas, the economists were right, and the benefits were much smaller than I estimated. The post contained one obvious error and one complication. Both were caught by Thomas Kwa, for which he has my gratitude. When he initially pointed out the error I awarded him a $50 bounty; now that the implications are confirmed I've upped that to $250. In two weeks an additional $750 will go to either him or to whoever provides new evidence that causes me to retract my retraction. Humming produces much less nitric oxide than Enovid I found the dosage of NO in Enovid in a trial registration. Unfortunately I misread the dose- what I original read as "0.11ppm NO/hour" was in fact "0.11ppm NO*hour". I spent a while puzzling out what this meant, with the help of Thomas Kwa, some guy on twitter, and chatGPT (the first time it's been genuinely useful to me). My new interpretation is that this means "actual concentration upon application*1 hour/time at that concentration". Since NO is a transient molecule, this means my guess for the amount of NO in Enovid was off by 2-3 orders of magnitude. My estimates for the amount of NO released by humming may also be too high. I used this paper's numbers for baseline NO concentration. However the paper I used to estimate the increase gave its own baseline number, which was an order of magnitude lower than the first paper. This wasn't intentional cherrypicking- I'd seen "15-20x increase in concentration" cited widely and often without sources. I searched for and spotchecked that one source but mostly to look at the experimental design. When I was ready to do math I used its increase but separately looked up the baseline concentration, and found the paper I cited. I just asked google again and got an even higher estimate of baseline nasal concentration, so seems like there is a great deal of disagreement here. If this were the only error I'd spend the time to get a more accurate estimate. But it looks like even the highest estimate will be a fraction of Enovid's dose, so it's not worth the energy to track down. Using the new values, you'd need 28 minutes of humming to recreate the amount of NO in Enovid (spreadsheet here). That wouldn't be so bad spread out over 4-6 hours, except that multiple breaths of humming in a row face diminishing returns, with recovery to baseline taking 3 minutes. It is possible to achieve this in 6 hours, but only just. And while it's not consequential enough to bother to look it up, I think some of the papers applied Enovid more often than that. This leaves humming in search of a use case. People who care a lot about respiratory illnesses are better off using Enovid or another nasal spray. People who don't care very much are never going to carefully pace their humming; and the amount of humming they might do won't be very effective. The only use case I see is people who care a lot and are pushed into a high risk situation without notice, or who want a feeling of of Doing Something even if it is not doing very much at all. Reasons to not write off humming entirely The math above assumes the effect is linear with the amount of NO released, regardless of application time. My guess is that frequent lower doses are more effective than the same amount as a one off. Probably not one effective enough to give humming a good non-emergency use case though. Another possibility is that Enovid has more nitric oxide than necessary and most of it is wasted. But again, it would have to be a lot moreto make this viable. Conclusions Humming hasn't been disproven as an anti-viral intervention, but the primary reason I believed it worke...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SB 1047 Is Weakened, published by Zvi on June 6, 2024 on LessWrong. It looks like Scott Weiner's SB 1047 is now severely weakened. Some of the changes are good clarifications. One is a big very welcome fix. The one I call The Big Flip is something very different. It is mind boggling that we can have a political system where a bill can overwhelmingly pass the California senate, and then a bunch of industry lobbyists and hyperbolic false claims can make Scott Weiner feel bullied into making these changes. I will skip the introduction, since those changes are clarifications, and get on with it. In the interest of a clean reference point and speed, this post will not cover reactions. The Big Flip Then there is the big change that severely weakens SB 1047. 1. 22602 (f)(1): Definition of covered model changed from trained with at least 10^26 flops OR a model expecting to have similar capabilities to what 10^26 flops would have gotten you in 2024 "was trained using a quantity of computing power greater than 10^26 integer or floating-point operations, AND the cost of that quantity of computing power would exceed one hundred million dollars ($100,000,000) if calculated using average market prices of cloud compute as reasonably assessed by the developer at the time of training." 2. On and after January 1, 2026, the dollar amount in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment. 3. Later: They will also publish the annual inflation adjustments. Bolded text is exact, except I capitalized AND for clarity. The AND, rather than an OR, makes my heart sink. Effectively, the 10^26 requirement is dead. Long live the $100 million. Where the law previously strengthened over time, now it weakens further. It starts weakening this year. The cost for buying one-time use of 10^26 flops of compute seems likely to fall below $100 million this year. Consider this from Jack Clark, where he got napkin math of $70 million a few months ago, or $110 million if you rented A100s. Jack clarified on Twitter that he expects B100s to offer a large further cost reduction. The compute minimum to be a covered model will begin to rise. The strength of non-covered models then rises both with the fall in compute costs, and also with gains in algorithmic efficiency. The previous version of the bill did an excellent job of handling the potential for Type I (false positive) errors via the limited duty exemption. If your model was behind the non-hazardous capabilities frontier, all you had to do was point that out. You were good to go. Alas, people willfully misrepresented that clause over and over. In terms of the practical impact of this law, the hope is that this change does not much matter. No doubt the biggest models will soon be trained on far more compute than $100 million can buy. So if you train on what $100 million can buy in 2026, someone else already trained a bigger model, and you had a limited duty exemption available anyway, so you not being covered only saved you a minimum amount of paperwork, and provides peace of mind against people spreading hyperbolic claims. What this does do is very explicitly and clearly show that the bill only applies to a handful of big companies. Others will not be covered, at all. If you are spending over $100 million in 2024 dollars on compute, but you then claim you cannot comply with ordinary regulations because you are the 'little guy' that is being stomped on? If you say that such requirements are 'regulatory capture' on behalf of 'big tech'? Yeah. Obvious Nonsense. I have no intention of pretend...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book review: The Quincunx, published by cousin it on June 6, 2024 on LessWrong. The Quincunx is a 1989 novel by Charles Palliser, set in early 1800s England. I want to recommend it to everyone because it's really good, and it might be relevant to the AI transition. Let me try to explain. The surface level of the book is a kind of mishmash of Dickensian themes. The main character is caught in a complicated inheritance dispute involving multiple families, each having histories of murder, uncertain parentage, stolen and returned documents and so on. The plot contains numerous puzzles that are fun to solve, the amount of planning is really kind of amazing, there are tons of details and everyone lies or makes mistakes but it still connects logically. But the really interesting level of the book is the social level. The main character doesn't just progress through a bunch of plot puzzles; he also starts out as a child of minor nobility and then moves through society downward. His journey is a kind of descent into hell, ending up in the lowest levels of poverty existing in the early 1800s. The book is very well researched in that regard, borrowing a lot from the fantastic "London Labor and the London Poor". There are parallel plotlines involving rich and poor people, and the book paints a vivid picture of how the rich prey upon the poor. England at that time was conducting enclosures. Basically, rich people put up fences around common land to graze sheep on it. The poor were left with no land to grow food on, and had to go somewhere else. They ended up in cities, living in slums, trying to find scarce work and giving their last pennies to slumlords. In short, it was a story of mass impoverishment of the population, conducted by the state and upper levels of society, who all benefited from it. In the book we get a tour of all of it. From the countryside being hollowed out, to the city with the desperate search for work, the run-down lodgings, the drinking, prostitution, crime (we spend a bit of time with the protagonist living in a gang), the sometimes horrifying occupations that people are pushed into (like scrounging for coins in sewer tunnels under the city while avoiding tides). The injuries, disabilities, early deaths. Where Dickens called out specific social ills, like workhouses in Oliver Twist, in order to fix them, Palliser says society as a whole is unjust. His account is so historically detailed that it somehow transcends time, makes you feel that the same kind of events are happening now. How does your society treat the economically unfortunate? What if we come into another period where economic growth makes many people unfortunate to the point of homelessness? I think it's especially important to not forget about such stories because they give an analogy to what might happen with the rise of AI. If AI can do your job cheaper than you, and can outbid you for resources you need to survive (most importantly land) - and there are lots of other tools available to AI and AI companies, like crafting messages to make you exchange your savings for consumption, or spend money on lobbying for laws, and do it all superhumanly well - then we might be facing the same kind of future as the poor in The Quincunx. And the main reason I wanted to make this point, and write this review, is that AI alignment isn't enough to prevent this. All above things can be done legally. Can be done with endorsement of the state, as the state happily benefits from AI as it did from enclosures. And they can be done by AI which is "aligned" to people, because historically these things were done by people. There's nothing higher than people to align to. The regulator, the AI company boss and all these other nice people are no different in nature than the people back then. When given power, they'll ...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: rapid psychological growth, published by Chipmonk on June 6, 2024 on LessWrong. After a one-hour session with an exceptional counselor, I never suffered over that romantic incident again. Although, that's inaccurate, I also had 2x half-hour relapses in following month. After a few more sessions, I stopped doing depression. I brought the rest of my anxieties to that counselor over the following year, and… Radically effective and rapid psychological growth is possible with the right combination of counselor and method. But this is rare in 2024! Introspection that actually works It was while working with that counselor that, for the first time I could remember, I was able to actually do introspection. Before, whenever I had problems that seemed to be caused by my psychology, I would do the obvious thing and ask myself, "Why am I doing ? Why am I not doing ?" But that almost never worked. Usually I would get a response back like, "Because it's hard, I'm lazy, and it's just a bad habit." The same problems would come back again and again. Meditation didn't help me much either. But, for me, this counselor did. I would come to a session suffering from something, he would prompt me into feeling into my body about the issue - which is important because the body represents the unconscious - and then in the following Socratic conversation I would be able to make rapid and dramatic progress on my problem. Big anxieties gone in an hour. (For context, most of my problems then could be reduced to either "I feel anxious about X social situation." and/or "I am disliking myself and I'm suffering about that.") Learning to facilitate Later, I trained with that counselor and learned his method. As part of my training I facilitated for four volunteers, and they seemed to have similar results that I had: rapid and dramatic resolution of the issue they came with in one hour. (Caveat: I never spoke to these volunteers again, so I don't know if the effect lasted.) But the sixth time I facilitated for someone was different. I experimented: I let the conversation run as long as it needed to, and I proactively tried to target the deepest roots of his emotional insecurity using the full force of my psychological research. After our three-hour conversation, he said, This session was significantly more productive than the last 6 months of professional CBT and talk therapy I did combined. (For context, he was a CFAR alumni and also very experienced with Focusing.) We didn't do any other sessions, but I followed up after six months to ask how he was doing: I can't stress how much I appreciated that dialogue, it really made me feel better, and I think I have already expressed much of what it made me feel. […] The effectiveness of your presence defeated my incredulity, and then some. This seems not to be a fluke, either. I've facilitated for seven other people since then and four have had similarly large shifts, eg, Your communication style made it easy to identify and release limiting beliefs. I felt noticeably more secure after just a few hours. That said, the other three people I facilitated seemed to have smaller effects, though each claims it was positive. More information about my emotional security tune-ups is available on chrislakin.com/now Radically effective and rapid psychological growth is possible with the right combination of counselor and method! What does a session look like? Here's the closest example I could find of what rapid psychological growth looks like in practice. (Note: I don't completely agree with their method, and also I wonder if the client's progress could've been even quicker.) Bolding is mine. Coherence Therapy for Panic Attacks, 2007 Bruce Ecker & Laurel Hulley: Carmen, a stylish freelance writer, was 35 and happily married, but she experie...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Former OpenAI Superalignment Researcher: Superintelligence by 2030, published by Julian Bradshaw on June 5, 2024 on LessWrong. The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace many college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. In the link provided, Leopold Aschenbrenner explains why he believes AGI is likely to arrive within the decade, with superintelligence following soon after. He does so in some detail; the website is well-organized, but the raw pdf is over 150 pages. Leopold is a former member of OpenAI's Superalignment team; he was fired in April for allegedly leaking company secrets. However, he contests that portrayal of events in a recent interview with Dwarkesh Patel, saying he leaked nothing of significance and was fired for other reasons.[1] However, I am somewhat confused by the new business venture Leopold is now promoting, an "AGI Hedge Fund" aimed at generating strong returns based on his predictions of imminent AGI. In the Dwarkesh Patel interview, it sounds like his intention is to make sure financial resources are available to back AI alignment and any other moves necessary to help Humanity navigate a turbulent future. However, the discussion in the podcast mostly focuses on whether such a fund would truly generate useful financial returns. If you read this post, Leopold[2], could you please clarify your intentions in founding this fund? 1. ^ Specifically he brings up a memo he sent to the old OpenAI board claiming OpenAI wasn't taking security seriously enough. He was also one of very few OpenAI employees not to sign the letter asking for Sam Altman's reinstatement last November, and of course, the entire OpenAI superaligment team has collapsed for various reasons as well. 2. ^ Leopold does have a LessWrong account, but hasn't linked his new website here after some time. I hope he doesn't mind me posting in his stead. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Evidence of Learned Look-Ahead in a Chess-Playing Neural Network, published by Erik Jenner on June 5, 2024 on LessWrong. Paper authors: Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, Stuart Russell TL;DR: We released a paper with IMO clear evidence of learned look-ahead in a chess-playing network (i.e., the network considers future moves to decide on its current one). This post shows some of our results, and then I describe the original motivation for the project and reflect on how it went. I think the results are interesting from a scientific and perhaps an interpretability perspective, but only mildly useful for AI safety. Teaser for the results (This section is copied from our project website. You may want to read it there for animations and interactive elements, then come back here for my reflections.) Do neural networks learn to implement algorithms involving look-ahead or search in the wild? Or do they only ever learn simple heuristics? We investigate this question for Leela Chess Zero, arguably the strongest existing chess-playing network. We find intriguing evidence of learned look-ahead in a single forward pass. This section showcases some of our results, see our paper for much more. Setup We consider chess puzzles such as the following: We focus on the policy network of Leela, which takes in a board state and outputs a distribution over moves. With only a single forward pass per board state, it can solve puzzles like the above. (You can play against the network on Lichess to get a sense of how strong it is - its rating there is over 2600.) Humans and manually written chess engines rely on look-ahead to play chess this well; they consider future moves when making a decision. But is the same thing true for Leela? Activations associated with future moves are crucial One of our early experiments was to do activation patching. We patch a small part of Leela's activations from the forward pass of a corrupted version of a puzzle into the forward pass on the original puzzle board state. Measuring the effect on the final output tells us how important that part of Leela's activations was. Leela is a transformer that treats every square of the chess board like a token in a language model. One type of intervention we can thus do is to patch the activation on a single square in a single layer: Surprisingly, we found that the target square of the move two turns in the future (what we call the 3rd move target square) often stores very important information. This does not happen in every puzzle, but it does in a striking fraction, and the average effect is much bigger than that of patching on most other squares: The corrupted square(s) and the 1st move target square are also important (in early and late layers respectively), but we expected as much from Leela's architecture. In contrast, the 3rd move target square stands out in middle layers, and we were much more surprised by its importance. In the paper, we take early steps toward understanding how the information stored on the 3rd move target square is being used. For example, we find a single attention head that often moves information from this future target square backward in time to the 1st move target square. Probes can predict future moves If Leela uses look-ahead, can we explicitly predict future moves from its activations? We train simple, bilinear probes on parts of Leela's activations to predict the move two turns into the future (on a set of puzzles where Leela finds a single clearly best continuation). Our probe architecture is motivated by our earlier results - it predicts whether a given square is the target square of the 3rd move since, as we've seen, this seems to be where Leela stores important information. We find that this probe can predict the move 2 turns in the future quit...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Just admit that you've zoned out, published by joec on June 4, 2024 on LessWrong. Summary: Zoning out is difficult to avoid and common, zoning out without admitting it hurts your comprehension, therefore you should admit that you zoned out and ask people to repeat things. If you're anything like me, you've "zoned out" before. You've probably even zoned out when you're trying to learn something interesting. In fact, I'd bet that you've zoned out when listening to someone you respect teaching you something interesting, and that you didn't admit it, and that this left you with a gap in understanding that you may or may not have filled in later.[1] Perhaps I'm falling for the typical minds fallacy, but I don't think I am. This happens to me very often,[2] and I think it happens to others, and I think that any community focused on rationality or scholarship or understanding ought to account for this. I doubt we'll be able to prevent people from zoning out, but I know we can encourage people who are listening to admit when they've zoned out and we can encourage people who are speaking to patiently re-iterate the thing they just said without taking offense. One time I was explaining something to a friend of mine and she said the unthinkable. "Sorry, I zoned out. Could you repeat what you said after first bringing up mitochondria?" I was at first somewhat taken aback, but quickly realized that I've been in the same position as her. I repeated myself and took less than a minute to do so. I think her understanding was better than it would have been if she hadn't simply admitted she zoned out. I'm thankful she did it, since it brought the fact that I could do the same to my awareness. If you're in the right company, admitting that you've zoned out has barely any cost and real benefits. Zoning out when someone is talking to you is far more common if the things they're saying are boring or hard to comprehend or otherwise unpleasant. It's perfectly rational to, as a speaker, take "people are zoning out" as evidence of a poor job. However, if you were unpleasant to listen to, nobody would ask you to repeat yourself. If someone admits to you that they stopped paying attention and asks you to repeat yourself, it doesn't imply any fault of yours. The right thing to do in that situation is to resist the temptation to be offended or annoyed and just go along with it. Of course, there's always a limit. If someone admits to zoning out twenty times in than thirty minutes, perhaps you ought to suggest that they get some sleep. If someone admits to daydreaming for 20 minutes straight while you talked to them, then it's probably time to end the conversation.[3] Even so, most people don't admit to this even once per week, and most fatal zone-outs are quite short. Telling others that you lost focus is done far less than it should be. One of my favorite things about the rationality(-adjacent) community is that its members admit when they're wrong. We acknowledge that our knowledge is limited and that our intelligence is only human. We ask what unfamiliar words mean. We don't try to hide our confusion or ignorance. It's a basic extension of the underlying principle of understanding and compensating for our cognitive shortcomings to also admit that we lost focus while listening, or got distracted by some irresistible thought that floated to the surface, or just needed a moment to let the things we just heard sink in. Paying attention for an extended period of time is actually kinda hard. Honestly, given that we had to sit in beige boxes for several hours a day for most of the year from ages 5-18 while someone preached to us about subjects we already knew, I'm surprised that reflexively zoning out isn't radically more common. Or, perhaps it is and I'm just not aware of it because nobody admits to z...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: in defense of Linus Pauling, published by bhauth on June 4, 2024 on LessWrong. Linus Pauling was a chemist who won a Nobel Prize in Chemistry in 1954. He later became well-known for advocating large doses of Vitamin C. I've heard that advocacy referred to as a cautionary tale, but I've long had a bit of sympathy for Linus, and I'm writing this post to explain how and why. mainstream nutrition One reason for my sympathy for Linus is that I've heard him used as an example of why you shouldn't go off on your own instead of trusting mainstream views in a field, yet his advice, while not particularly helpful, caused much less harm than contemporary "mainstream nutritional advice", such as: the food pyramid "partially hydrogenated vegetable oil is healthier than butter" "a very-low-fat diet is healthy" "eating very little salt is healthy" I certainly wouldn't suggest trying to independently compete with the conceptual framework of, say, semiconductor physics or structural engineering, but when a field is rotten enough (nutrition, psychology, education, and economics come to mind) history indicates to me that someone smart from another field is often more correct than specialists on that topic, when they have an interest in it. my view of Vitamin C To be clear, I'm not advocating for the very high doses of Vitamin C that Linus did. I do think the World Health Organization's RDA (45 mg/day) is a bit low, but the RDA of Japan and the EU (~100 mg/day) seems entirely reasonable to me. Amounts above that generally don't have much effect on blood levels of Vitamin C, because it's absorbed less and the excess is expelled. Thus, some people have advocated administering it by IV, but one has to consider the possibility that there's a good reason for a specific level being naturally maintained. Research since Linus first advocated for Vitamin C megadoses has supported oxidative stress being a major cause of aging. It's associated with many problems (Alzheimer's comes to mind) and there's a good theoretical basis (DNA I-compounds) for its long-term harms. We've also seen that previous suggestions for Vitamin D doses were much too low, so there's also precedent for that. Where, then, did Linus go wrong? Vitamin C is an antioxidant, but it's also a pro-oxidant. It can reduce iron and copper, which can then react with hydrogen peroxide or oxygen to form hydroxyl radicals or peroxide ions. It can also form some complexes with metal ions that could conceivably have some harmful catalytic effects. (Its ability to interact with metal ions in certain ways is the main reason it's used in cells rather than some other compound: it's a cofactor.) The normal levels of free Fe and Cu ions are low, but my view is that the natural blood level of Vitamin C is a compromise set by pro-oxidant effects. When an infection happens that causes hypochlorite production by immune cells, it's logical that the optimal level of Vitamin C would be higher. And indeed, there's evidence that extra Vitamin C during infections (especially bacterial infections) helps somewhat. But the main antioxidant in mammals seems to be glutathione rather than Vitamin C, and it has to be used in combination with superoxide dismutase. So, Linus was one the right track. He was trying to solve the right problem, and he found a reasonable solution to it, but he overlooked some complicated side effects. That's a mistake I consider forgivable. He should have realized that there was a reason for homeostasis of Vitamin C levels, but the ideology of his time was that biological regulatory systems were so ineffective that any deliberate management by people would be better. There were, thus, people optimizing the balance of purified starch/fat/protein diets they fed rats, and being puzzled when the rats kept dying. Then, as soon as they discov...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: (Not) Derailing the LessOnline Puzzle Hunt, published by Error on June 4, 2024 on LessWrong. (spoiler alert: may meta-spoil future iterations of the LOPH, if you haven't already read other posts about it) I knew early on that I wouldn't be able to finish the LessOnline puzzle hunt. I contributed to solving two of the first six puzzles, each of which revealed the combination to a locked box. Each box contained wooden medallions for the solvers, plus a QR code. The QR codes led to the second layer of a larger meta-puzzle. On discovering the existence of the meta-puzzle, I knew I would have to drop out. It was a shame, because I have never done a puzzle hunt before and I could see that the puzzlemasters had produced something amazing. But the opportunity cost of playing "for real" was just too high. There were too many other things I wanted to do, and the integral of my attention over time is not infinite. So I stopped hunting. But, unexpectedly and hilariously, I ended up contributing to the game in a very different way. Suspicion During the Fooming Shoggoth concert I noticed (probably thanks to the song lyrics) that the parts of the puzzle I was aware of were all lockboxes. I mean, I knew that already, but now I noticed. And I knew from the contents of the two boxes I had opened that there was more to the puzzles than the obvious. The whole setup seemed like a suspiciously plausible metaphor for the AI Box Experiment. I suddenly, strongly suspected that the puzzle hunt had a hidden narrative -- one that would end with the release of a rogue AI. Sequence Breaker! So I did what any Less Wronger should have done: I tried to warn others. No unbinding of seals, no opening of gates! At first I just told nearby hunters directly, but quickly realized it wouldn't work; if nothing else, I didn't know who was working on the puzzle hunt and who wasn't. I abandoned that plan and asked the front desk to print out three notes for me. The notes outlined my suspicions and warned the reader not to open whatever final box might be found at the end of the puzzle chain. I taped Note 1 to my medallion, which I returned to its original box. In addition to the warning, Note 1 asked anyone opening the box to leave the medallion and the note itself for later hunters to see, and suggested they return their own medallion(s), just in case. I had no reason to believe it mattered whether the medallions stayed in the boxes, but I had no reason to believe it didn't, and it was an obvious thing to try. I taped Note 2 to the table by the lockboxes. Note 2 asked anyone who shared my suspicions to sign it, as social pressure on others to not open boxes that might contain Very Bad Things. Note 3 was a copy of note 2, and stayed in my backpack for contingencies. After placing the notes, I moved on to other things. I'd volunteered to run a talk the following day on a whim, and preparation funged against sleep. By the time I had the slide deck put together it was 4am. Before going to bed, I checked on my warning notes. Note 2 was gone. I'd thought of that possibility, which was why I had a contingency copy. Now I had to decide whether to use it. Crap. Decision Theory So far I'd been intentionally open about what I was doing. I told multiple hunters about both my suspicions and the notes. I showed Note 1 to a volunteer when placing it. Note 2 was in plain sight. I even had a staff member at the front desk print the notes for me. My in-character warnings to hunters doubled as an out-of-character warning to the puzzlemasters: "possible derailment in progress". I didn't want to actually ruin whatever awesomeness they had planned. I wanted to prevent hunters from opening the hypothetical final box if and only if that was the true win condition. The problem was that I didn't know -- and couldn't know -- whether th...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Standard Analogy, published by Zack M Davis on June 3, 2024 on LessWrong. [Scene: a suburban house, a minute after the conclusion of "And All the Shoggoths Merely Players". Doomimir returns with his package, which he places by the door, and turns his attention to Simplicia, who has been waiting for him.] Simplicia: Right. To recap for [coughs] no one in particular, when we left off [pointedly, to the audience] one minute ago, Doomimir Doomovitch, you were expressing confidence that approaches to aligning artificial general intelligence within the current paradigm were almost certain to fail. You don't think that the apparent tractability of getting contemporary generative AI techniques to do what humans want bears on that question. But you did say you have empirical evidence for your view, which I'm excited to hear about! Doomimir: Indeed, Simplicia Optimistovna. My empirical evidence is the example of the evolution of human intelligence. You see, humans were optimized for one thing only: inclusive genetic fitness [Simplicia turns to the audience and makes a face.] Doomimir: [annoyed] What? Simplicia: When you said you had empirical evidence, I thought you meant empirical evidence about AI, not the same analogy to an unrelated field that I've been hearing for the last fifteen years. I was hoping for, you know, ArXiv papers about SGD's inductive biases, or online regret bounds, or singular learning theory ... something, anything at all, from this century, that engages with what we've learned from the experience of actually building artificial minds. Doomimir: That's one of the many things you Earthlings refuse to understand. You didn't build that. Simplicia: What? Doomimir: The capabilities advances that your civilization's AI guys have been turning out these days haven't come from a deeper understanding of cognition, but by improvements to generic optimization methods, fueled with ever-increasing investments in compute. Deep learning not only isn't a science, it isn't even an engineering discipline in the traditional sense: the opacity of the artifacts it produces has no analogue among bridge or engine designs. In effect, all the object-level engineering work is being done by gradient descent. The autogenocidal maniac Richard Sutton calls this the bitter lesson, and attributes the field's slowness to embrace it to ego and recalcitrance on the part of practitioners. But in accordance with the dictum to feel fully the emotion that fits the facts, I think bitterness is appropriate. It makes sense to be bitter about the shortsighted adoption of a fundamentally unalignable paradigm on the basis of its immediate performance, when a saner world would notice the glaring foreseeable difficulties and coordinate on doing Something Else Which Is Not That. Simplicia: I don't think that's quite the correct reading of the bitter lesson. Sutton is advocating general methods that scale with compute, as contrasted to hand-coding human domain knowledge, but that doesn't mean that we're ignorant of what those general methods are doing. One of the examples Sutton gives is computer chess, where minimax search with optimizations like α-β pruning prevailed over trying to explicitly encode what human grandmasters know about the game. But that seems fine. Writing a program that thinks about tactics the way humans do rather than letting tactical play emerge from searching the game tree would be a lot more work for less than no benefit. A broadly similar moral could apply to using deep learning to approximate complicated functions between data distributions: we specify the training distribution, and the details of fitting it are delegated to a network architecture with the appropriate invariances: convolutional nets for processing image data, transformers for variable-length sequences. Ther...