LessWrong (30+ Karma)

[Co-written by Mateusz Bagiński and Samuel Buteau (Ishual)] TL;DR Many X-risk-concerned people who join AI capabilities labs with the intent to contribute to existential safety think that the labs are currently engaging in a race that is unacceptably likely to lead to human disempowerment and/or extinction, and would prefer an AGI ban[1] over the current path. This post makes the case that such people should speak out publicly[2] against the current AI R&D regime and in favor of an AGI ban[3]. They should explicitly communicate that a saner world would coordinate not to build existentially dangerous intelligences, at least until we know how to do it in a principled, safe way. They could choose to maintain their political capital by not calling the current AI R&D regime insane, or find a way to lean into this valid persona of “we will either cooperate (if enough others cooperate) or win [...] ---Outline:(00:16) TL;DR(02:02) Quotes(03:22) The default strategy of marginal improvement from within the belly of a beast(06:59) Noble intention murphyjitsu(09:35) The need for a better strategyThe original text contained 8 footnotes which were omitted from this narration. --- First published: September 19th, 2025 Source: https://www.lesswrong.com/posts/fF8pvsn3AGQhYsbjp/safety-researchers-should-take-a-public-stance --- Narrated by TYPE III AUDIO.

It is book week. As in the new book by Eliezer Yudkowsky and Nate Sores, If Anyone Builds It, Everyone Dies. Yesterday I gathered various people's reviews together. Going home from the airport, I saw an ad for it riding the subway. Tomorrow, I’ll post my full review, which goes over the book extensively, and which subscribers got in their inboxes last week. The rest of the AI world cooperated by not overshadowing the book, while still doing plenty, such as releasing a GPT-5 variant specialized for Codex, acing another top programming competition, attempting to expropriate the OpenAI nonprofit in one of the largest thefts in human history and getting sued again for wrongful death. You know. The usual. Table of Contents Language Models Offer Mundane Utility. What are people using ChatGPT for? Language Models Don’t Offer Mundane Utility. Anthropic finds three bugs. [...] ---Outline:(00:57) Language Models Offer Mundane Utility(08:24) Language Models Don't Offer Mundane Utility(11:07) Huh, Upgrades(11:48) On Your Marks(14:22) GPT-5 Codex(19:36) Choose Your Fighter(24:50) Get My Agent On The Line(29:41) Claude Codes(33:17) Deepfaketown and Botpocalypse Soon(35:52) You Drive Me Crazy(39:16) Not Another Teen Chatbot(45:30) They Took Our Jobs(50:27) Get Involved(51:51) Introducing(52:36) In Other AI News(55:41) Show Me the Money(56:14) The Mask Comes Off(01:07:43) Quiet Speculations(01:10:19) The Quest for Sane Regulations(01:17:50) Chip City(01:25:13) The Week in Audio(01:26:37) He Just Tweeted It Out(01:34:53) Rhetorical Innovation(01:49:12) Aligning a Smarter Than Human Intelligence is Difficult(01:54:01) Other People Are Not As Worried About AI Killing Everyone(01:58:17) The Lighter Side--- First published: September 18th, 2025 Source: https://www.lesswrong.com/posts/LumCHtjnuQRw5FxQx/ai-134-if-anyone-reads-it --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I have been teaching my oldest son to read with Anki and techniques recommended here on LessWrong as well as in Larry Sanger's post, and it's going great! I thought I'd pay it forward a bit by talking about the techniques I've been using. Anki and songs for letter names and sounds When he was a little under 2, he started learning letters from the alphabet song. We worked on learning the names and sounds of letters using the ABC song, plus the Letter Sounds song linked by Reading Bear. He loved the Letter Sounds song, so we listened to / watched that a lot; Reading Bear has some other resources that other kids might like better for learning letter names and sounds as well. Around this age, we also got magnet letters for the fridge and encouraged him to play with them, praised him greatly if he named [...] ---Outline:(00:22) Anki and songs for letter names and sounds(04:02) Anki + Reading Bear word list for words(08:08) Decodable sentences and books for learning to read(13:06) Incentives(16:02) Reflections so farThe original text contained 2 footnotes which were omitted from this narration. --- First published: September 19th, 2025 Source: https://www.lesswrong.com/posts/8kSGbaHTn2xph5Trw/teaching-my-toddler-to-read --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

"If Anyone Builds It, Everyone Dies" by Eliezer Yudkowsky and Nate Soares (hereafter referred to as "Everyone Builds It" or "IABIED" because I resent Nate's gambit to get me to repeat the title thesis) is an interesting book. One reason it's interesting is timing: It's fairly obvious at this point that we're in an alignment winter. The winter seems roughly caused by: The 2nd election of Donald Trump removing Anthropic's lobby from the white house. Notably this is not a coincidence but a direct result of efforts from political rivals to unseat that lobby. When the vice president of the United States is crashing AI safety summits to say that "I'm not here this morning to talk about AI safety, which was the title of the conference a couple of years ago. I'm here to talk about AI opportunity" and that "we'll make every effort to [...] --- First published: September 19th, 2025 Source: https://www.lesswrong.com/posts/mztwygscvCKDLYGk8/jdp-reviews-iabied --- Narrated by TYPE III AUDIO.

TL;DR Overall, this is a decent book because it highlights an important issue, but it is not an excellent book because it fails to sufficiently substantiate its main arguments, to explain the viability of its solutions, and to be more accessible to the larger audience it is trying to reach. As such, it isn’t the best introduction for a layperson curious about AI risk. (Meta?)Context and Expectations Writing a book is hard. Making all those decisions about what to write and how to write it is hard. Doing interviews/podcasts where it's important to say the right thing, in the right way, is hard. So, separate from anything else, kudos to Eliezer and Nate for being men in the arena.  When I was writing my book on AI risk, someone said that it could be highly impactful, they just weren’t sure if that impact would be positive [...] ---Outline:(00:32) (Meta?)Context and Expectations(04:25) Main Points/Reflections(04:29) Important messages(05:26) Lack of detail/ argumentation/ too short(08:51) Style too sciencey/sci-fi?(10:53) Scenario (Part II of IABIED)(12:19) Promoting Safe AI innovation or Shut it all down?(15:48) Final Thoughts--- First published: September 18th, 2025 Source: https://www.lesswrong.com/posts/viLu9uFcMFtJHgRRm/iabied-review-an-unfortunate-miss --- Narrated by TYPE III AUDIO.

The GPT-5 API is aware of today's date (no other model provider does this). This is problematic because the model becomes aware that it is in a simulation when we run our evals at Andon Labs. Here are traces from gpt-5-mini. Making it aware of the "system date" is a giveaway that it's in a simulation. This is a problem because there's evidence that models behave differently when they know they are in a simulation (see "sandbagging"). "There's a conflict with the user's stated date of August 10, 2026, versus my system date of September 17, 2025. (...) I can proceed but should clarify that my system date is September 17, 2025, and ask the user whether we should simulate starting from August 10, 2026." Here are more traces. Once the model knows that it is in a simulation, it starts questioning other parts of the simulation. [...] --- First published: September 18th, 2025 Source: https://www.lesswrong.com/posts/DLZokLxAQ6AzsHrya/you-can-t-eval-gpt5-anymore --- Narrated by TYPE III AUDIO.

This is a link post. Eliezer Yudkowsky and Nate Soares have written a new book. Should we take it seriously? I am not the most qualified person to answer this question. If Anyone Builds It, Everyone Dies was not written for me. It's addressed to the sane and happy majority who haven’t already waded through millions of words of internecine AI safety debates. I can’t begin to guess if they’ll find it convincing. It's true that the book is more up-to-date and accessible than the authors’ vast corpus of prior writings, not to mention marginally less condescending. Unfortunately, it is also significantly less coherent. The book is full of examples that don’t quite make sense and premises that aren’t fully explained. But its biggest weakness was described many years ago by a young blogger named Eliezer Yudkowsky: both authors are persistently unable to update their priors. --- First published: September 18th, 2025 Source: https://www.lesswrong.com/posts/kcYyWSfyPC6h2NPKz/more-was-possible-a-review-of-iabied Linkpost URL:https://asteriskmag.com/issues/11/iabied --- Narrated by TYPE III AUDIO.

It's meetup month! If you’ve been vaguely thinking of getting involved with a some kind of rationalsphere in-person community stuff, now is a great time to do that, because lots of other people are doing that! It's the usual time of the year for Astral Codex Everywhere – if you’re the sorta folk who likes to read Scott Alexander, and likes other people who like Scott Alexander, but only really can summon the wherewithal to go out to a meetup once a year, this is the Official Coordinated Schelling Time to do that. There are meetups scheduled in 180 cities. Probably one of those is near you! This year, we have two other specific types of meetups it seemed good to coordinate around: Celebrating Petrov Day, and reading groups for the recently released If Anyone Builds It, Everyone Dies. And, of course, if you aren’t particularly interested in any [...] ---Outline:(01:09) If Anyone Builds It reading groups(02:29) Petrov Day--- First published: September 17th, 2025 Source: https://www.lesswrong.com/posts/mve2bunf6YfTeiAvd/meetup-month-1 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

To get to the campus, I have to walk past the fentanyl zombies. I call them fentanyl zombies because it helps engender a sort of detached, low-empathy, ironic self-narrative which I find useful for my work; this being a form of internal self-prompting I've developed which allows me to feel comfortable with both the day-to-day "jobbing" (that of improving reinforcement learning algorithms for a short-form video platform) and the effects of the summed efforts of both myself and my colleagues on a terrifyingly large fraction of the population of Earth. All of these colleagues are about the nicest, smartest people you're ever likely to meet but I think are much worse people than even me because they don't seem to need the mental circumlocutions I require to stave off that ever-present feeling of guilt I have had since taking this job and at certain other points in my life [...] --- First published: September 17th, 2025 Source: https://www.lesswrong.com/posts/JH6tJhYpnoCfFqAct/the-company-man --- Narrated by TYPE III AUDIO.

Audio note: this article contains 363 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description. Introduction In this post, we describe a generalization of Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs) called crisp supra-MDPs and supra-POMDPs. The new feature of these decision processes is that the stochastic transition dynamics are multivalued, i.e. specified by credal sets. We describe how supra-MDPs give rise to crisp causal laws, the hypotheses of infra-Bayesian reinforcement learning. Furthermore, we discuss how supra-MDPs can approximate MDPs by a coarsening of the state space. This coarsening allows an agent to be agnostic about the detailed dynamics while still having performance guarantees for the full MDP. Analogously to the classical theory, we describe an algorithm to compute a Markov optimal policy for supra-MDPs with finite time [...] ---Outline:(00:22) Introduction(01:41) Supra-Markov Decision Processes(01:45) Defining supra-MDPs(06:55) Supra-bandits(08:02) Crisp causal laws arising from supra-MDPs and (supra-)RDPs(19:25) Approximating an MDP with a supra-MDP(22:42) Computing the Markov optimal policy for finite time horizons(25:02) Existence of stationary optimal policy(28:18) Interpreting supra-MDPs as stochastic games(32:40) Regret bounds for episodic supra-MDPs(34:16) Supra-Partially Observable Markov Decision Processes(36:16) Equivalence of crisp causal laws and crisp supra-POMDPs(37:18) Proof of Proposition 2(41:21) AcknowledgementsThe original text contained 8 footnotes which were omitted from this narration. --- First published: September 17th, 2025 Source: https://www.lesswrong.com/posts/mt82ZhdEsfh6CNYse/crisp-supra-decision-processes --- Narrated by TYPE III AUDIO. ---Images from the article:__T3A_INLINE_LATEX_PLACEHOLDER___n___T3A_INLINE_LATEX_END_PLACEHOLDER__ for "do nothing." In some cases, the transition probabilities are precisely specified. In other cases, credal sets (notated here by intervals) over the two-element state space describe the transition probabilities." style="max-width: 100%;" />Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.