Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We Don't Know Our Own Values, but Reward Bridges The Is-Ought Gap, published by johnswentworth on September 19, 2024 on LessWrong. Background: "Learning" vs "Learning About" Adaptive systems, reinforcement "learners", etc, "learn" in the sense that their behavior adapts to their environment. Bayesian reasoners, human scientists, etc, "learn" in the sense that they have some symbolic representation of the environment, and they update those symbols over time to (hopefully) better match the environment (i.e. make the map better match the territory). These two kinds of "learning" are not synonymous[1]. Adaptive systems "learn" things, but they don't necessarily "learn about" things; they don't necessarily have an internal map of the external territory. (Yes, the active inference folks will bullshit about how any adaptive system must have a map of the territory, but their math does not substantively support that interpretation.) The internal heuristics or behaviors "learned" by an adaptive system are not necessarily "about" any particular external thing, and don't necessarily represent any particular external thing[2]. We Humans Learn About Our Values "I thought I wanted X, but then I tried it and it was pretty meh." "For a long time I pursued Y, but now I think that was more a social script than my own values." "As a teenager, I endorsed the view that Z is the highest objective of human existence. … Yeah, it's a bit embarrassing in hindsight." The ubiquity of these sorts of sentiments is the simplest evidence that we do not typically know our own values[3]. Rather, people often (but not always) have some explicit best guess at their own values, and that guess updates over time - i.e. we can learn about our own values. Note the wording here: we're not just saying that human values are "learned" in the more general sense of reinforcement learning. We're saying that we humans have some internal representation of our own values, a "map" of our values, and we update that map in response to evidence. Look again at the examples at the beginning of this section: "I thought I wanted X, but then I tried it and it was pretty meh." "For a long time I pursued Y, but now I think that was more a social script than my own values." "As a teenager, I endorsed the view that Z is the highest objective of human existence. … Yeah, it's a bit embarrassing in hindsight." Notice that the wording of each example involves beliefs about values. They're not just saying "I used to feel urge X, but now I feel urge Y". They're saying "I thought I wanted X" - a belief about a value! Or "now I think that was more a social script than my own values" - again, a belief about my own values, and how those values relate to my (previous) behavior. Or "I endorsed the view that Z is the highest objective" - an explicit endorsement of a belief about values. That's how we normally, instinctively reason about our own values. And sure, we could reword everything to avoid talking about our beliefs about values - "learning" is more general than "learning about" - but the fact that it makes sense to us to talk about our beliefs about values is strong evidence that something in our heads in fact works like beliefs about values, not just reinforcement-style "learning". Two Puzzles Puzzle 1: Learning About Our Own Values vs The Is-Ought Gap Very roughly speaking, an agent could aim to pursue any values regardless of what the world outside it looks like; "how the external world is" does not tell us "how the external world should be". So when we "learn about" values, where does the evidence about values come from? How do we cross the is-ought gap? Puzzle 2: The Role of Reward/Reinforcement It does seem like humans have some kind of physiological "reward", in a hand-wavy reinforcement-learning-esque sense, which seems to at l...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #82: The Governor Ponders, published by Zvi on September 19, 2024 on LessWrong. The big news of the week was of course OpenAI releasing their new model o1. If you read one post this week, read that one. Everything else is a relative sideshow. Meanwhile, we await Newsom's decision on SB 1047. The smart money was always that Gavin Newsom would make us wait before offering his verdict on SB 1047. It's a big decision. Don't rush him. In the meantime, what hints he has offered suggest he's buying into some of the anti-1047 talking points. I'm offering a letter to him here based on his comments, if you have any way to help convince him now would be the time to use that. But mostly, it's up to him now. Table of Contents 1. Introduction. 2. Table of Contents. 3. Language Models Offer Mundane Utility. Apply for unemployment. 4. Language Models Don't Offer Mundane Utility. How to avoid the blame. 5. Deepfaketown and Botpocalypse Soon. A social network of you plus bots. 6. They Took Our Jobs. Not much impact yet, but software jobs still hard to find. 7. Get Involved. Lighthaven Eternal September, individual rooms for rent. 8. Introducing. Automated scientific literature review. 9. In Other AI News. OpenAI creates independent board to oversee safety. 10. Quiet Speculations. Who is preparing for the upside? Or appreciating it now? 11. Intelligent Design. Intelligence. It's a real thing. 12. SB 1047: The Governor Ponders. They got to him, but did they get to him enough? 13. Letter to Newsom. A final summary, based on Newsom's recent comments. 14. The Quest for Sane Regulations. How should we update based on o1? 15. Rhetorical Innovation. The warnings will continue, whether or not anyone listens. 16. Claude Writes Short Stories. It is pondering what you might expect it to ponder. 17. Questions of Sentience. Creating such things should not be taken lightly. 18. People Are Worried About AI Killing Everyone. The endgame is what matters. 19. The Lighter Side. You can never be sure. Language Models Offer Mundane Utility Arbitrate your Nevada unemployment benefits appeal, using Gemini. This should solve the backlog of 10k+ cases, and also I expect higher accuracy than the existing method, at least until we see attempts to game the system. Then it gets fun. That's also job retraining. o1 usage limit raised to 50 messages per day for o1-mini, 50 per week for o1-preview. o1 can do multiplication reliably up to about 46 digits, andabout 50% accurately up through about 810, a huge leap from gpt-4o, although Colin Fraser reports 4o can be made better tat this than one would expect. o1 is much better than 4o at evaluating medical insurance claims, and determining whether requests for care should be approved, especially in terms of executing existing guidelines, and automating administrative tasks. It seems like a clear step change in usefulness in practice. The claim is that being sassy and juicy and bitchy improves Claude Instant numerical reasoning. What I actually see here is that it breaks Claude Instant out of trick questions. Where Claude would previously fall into a trap, you have it fall back on what is effectively 'common sense,' and it starts getting actually easy questions right. Language Models Don't Offer Mundane Utility A key advantage of using an AI is that you can no longer be blamed for an outcome out of your control. However, humans often demand manual mode be available to them, allowing humans to override the AI, even when it doesn't make any practical sense to offer this. And then, if the human can in theory switch to manual mode and override the AI, blame to the human returns, even when the human exerting that control was clearly impractical in context. The top example here is self-driving cars, and blame for car crashes. The results suggest that the human thirst for ill...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Which LessWrong/Alignment topics would you like to be tutored in? [Poll], published by Ruby on September 19, 2024 on LessWrong. Would you like to be tutored in applied game theory, natural latents, CFAR-style rationality techniques, "general AI x-risk", Agent Foundations, anthropic s , or some other topics discussed on LessWrong? I'm thinking about prototyping some topic-specific LLM tutor bots, and would like to prioritize topics that multiple people are interested in. Topic-specific LLM tutors would be customized with things like pre-loaded relevant context, helpful system prompts, and more focused testing to ensure they work. Note: I'm interested in topics that are written about on LessWrong, e.g. infra-bayesianism, and not magnetohydrodynamics". I'm going to use the same poll infrastructure that Ben Pace pioneered recently. There is a thread below where you add and vote on topics/domains/areas where you might like tutoring. 1. Karma: upvote/downvote to express enthusiasm about there being tutoring for a topic. 2. Reacts: click on the agree react to indicate you personally would like tutoring on a topic. 3. New Poll Option. Add a new topic for people express interest in being tutored on. For the sake of this poll, I'm more interested in whether you'd like tutoring on a topic or not, separate from the question of whether you think a tutoring bot would be any good. I'll worry about that part. Background I've been playing around with LLMs a lot in the past couple of months and so far my favorite use case is tutoring. LLM-assistance is helpful via multiple routes such as providing background context with less effort than external search/reading, keeping me engaged via interactivity, generating examples, and breaking down complex sections into more digestible pieces. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Would You Ask The Archbishop of Canterbury?, published by JDBauman on September 19, 2024 on The Effective Altruism Forum. The head of the Church of England is the second most influential Christian alive today. [1] The current Archbishop, Justin Welby, is speaking at the EA-adjacent Christians for Impact conference with Rory Stewart about faith and poverty. What should we ask Archbishop Justin in the Q&A? Feel free to submit anonymous thoughts here. 1. ^ Source: ChatGPT Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Intuitive self-models] 1. Preliminaries, published by Steven Byrnes on September 19, 2024 on LessWrong. 1.1 Summary & Table of Contents This is the first of a series of eight blog posts, which I'll be serializing over the next month or two. (Or email or DM me if you want to read the whole thing right now.) Here's an overview of the whole series, and then we'll jump right into the first post! 1.1.1 Summary & Table of Contents - for the whole series This is a rather ambitious series of blog posts, in that I'll attempt to explain what's the deal with consciousness, free will, hypnotism, enlightenment, hallucinations, flow states, dissociation, akrasia, delusions, and more. The starting point for this whole journey is very simple: The brain has a predictive (a.k.a. self-supervised) learning algorithm. This algorithm builds generative models (a.k.a. "intuitive models") that can predict incoming data. It turns out that, in order to predict incoming data, the algorithm winds up not only building generative models capturing properties of trucks and shoes and birds, but also building generative models capturing properties of the brain algorithm itself. Those latter models, which I call "intuitive self-models", wind up including ingredients like conscious awareness, deliberate actions, and the sense of applying one's will. That's a simple idea, but exploring its consequences will take us to all kinds of strange places - plenty to fill up an eight-post series! Here's the outline: Post 1 (Preliminaries) gives some background on the brain's predictive learning algorithm, how to think about the "intuitive models" built by that algorithm, how intuitive self-models come about, and the relation of this whole series to Philosophy Of Mind. Post 2 ( Awareness ) proposes that our intuitive self-models include an ingredient called "conscious awareness", and that this ingredient is built by the predictive learning algorithm to represent a serial aspect of cortex computation. I'll discuss ways in which this model is veridical (faithful to the algorithmic phenomenon that it's modeling), and ways that it isn't. I'll also talk about how intentions and decisions fit into that framework. Post 3 ( The Homunculus ) focuses more specifically on the intuitive self-model that almost everyone reading this post is experiencing right now (as opposed to the other possibilities covered later in the series), which I call the Conventional Intuitive Self-Model. In particular, I propose that a key player in that model is a certain entity that's conceptualized as actively causing acts of free will. Following Dennett, I call this entity "the homunculus", and relate that to intuitions around free will and sense-of-self. Post 4 ( Trance ) builds a framework to systematize the various types of trance, from everyday "flow states", to intense possession rituals with amnesia. I try to explain why these states have the properties they do, and to reverse-engineer the various tricks that people use to induce trance in practice. Post 5 ( Dissociative Identity Disorder ) (a.k.a. "multiple personality disorder") is a brief opinionated tour of this controversial psychiatric diagnosis. Is it real? Is it iatrogenic? Why is it related to borderline personality disorder (BPD) and trauma? What do we make of the wild claim that each "alter" can't remember the lives of the other "alters"? Post 6 ( Awakening / Enlightenment / PNSE ) is a type of intuitive self-model, typically accessed via extensive meditation practice. It's quite different from the conventional intuitive self-model. I offer a hypothesis about what exactly the difference is, and why that difference has the various downstream effects that it has. Post 7 (Hearing Voices, and Other Hallucinations) talks about factors contributing to hallucinations - although I argue ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Organization Updates: September 2024, published by Toby Tremlett on September 19, 2024 on The Effective Altruism Forum. If you would like to see EA Organization Updates as soon as they come out, consider subscribing to this tag. Some of the opportunities and job listings we feature in this update have (very) pressing deadlines (see AI Alignment Teaching Fellow opportunities at BlueDot Impact, September 22, and Institutional Foodservice Fellow at the Good Food Institute, September 18). You can see previous updates on the "EA Organization Updates (monthly series)" topic page, or in our repository of past newsletters. Notice that there's also an "org update" tag, where you can find more news and updates that are not part of this consolidated series. These monthly posts originated as the "Updates" section of the monthly EA Newsletter. Organizations submit their own updates, which we edit for clarity. (If you'd like to share your updates and jobs via this series, please apply here.) Opportunities and jobs Opportunities Consider also checking opportunities listed on the EA Opportunity Board and the Opportunities to Take Action tag. ALLFED published a new database containing numerous research projects that prospective volunteers can assist with. Explore the database and apply here. Apply to the upcoming AI Safety Fundamentals: Alignment course by October 6 to learn about the risks from AI and how you can contribute to the field. The Animal Advocacy Careers Introduction to Animal Advocacy Course has been revamped. The course is for those wishing to kickstart a career in animal advocacy. Giv Effektivt (DK) needs ~110 EU citizens to become members before the new year in order to offer tax deductions of around 450.000DKK ($66.000) for 2024-25 donations. Become a member now for 50DKK ($7). An existing donor will give 100DKK for each new member until the organization reaches 300 members. Anima International's Animal Advocacy Training Center released a new online course - Fundraising Essentials. It's a free, self-paced resource with over two hours of video content for people new to the subject. Job listings Consider also exploring jobs listed on the Job listing (open) tag. For even more roles, check the 80,000 Hours Job Board. BlueDot Impact AI Alignment Teaching Fellow (Remote, £4.9K-£9.6K, apply by September 22nd) Centre for Effective Altruism Head of Operations (Remote, £107.4K / $179.9K, apply by October 7th) Cooperative AI Foundation Communications Officer (Remote, £35K-£40K, apply by September 29th) GiveWell Senior Researcher (Remote, $200K-$220.6K) Giving What We Can Global CEO (Remote, $130K+, apply by September 30th) Open Philanthropy Operations Coordinator/Associate (San Francisco, Washington, DC, $99.6K-$122.6K) If you're interested in working at Open Philanthropy but don't see an open role that matches your skillset, express your interest. Epoch AI Question Writer, Math Benchmark (Contractor Position) (Remote, $2K monthly + $100-$1K performance-based bonus) Senior Researcher, ML Distributed Systems (Remote, $150K-$180K) The Good Food Institute Managing Director, GFI India (Hybrid (Mumbai, Delhi, Hyderabad, or Bangalore), ₹4.5M, apply by October 2nd) Institutional Foodservice Fellow (Independent Contractor) (Remote in US, $3.6K biweekly, apply by September 18th) Organization updates The organization updates are in alphabetical order (0-A-Z). 80,000 Hours There is one month left to win $5,000 career grants by referring your friends or colleagues to 80,000 Hours' free career advising. Also, the organization released a blog post about the recent updates to their AI-related content, as well as a post about pandemic preparedness in relation to mpox and H5N1. On the 80,000 Hours Podcast, Rob interviewed: Nick Joseph on whether Anthropic's AI safety policy is up to the task...

Lauren Mee, an advocate in animal welfare and effective altruism, shares her journey with Animal Advocacy Careers. She discusses their impressive track record of filling 105 roles and supporting over 150 organizations in five years. Key insights include tackling recruitment challenges and the significance of role clarity within small teams. Lauren also highlights strategic advancements in hiring practices and future directions for enhancing their impact, while advocating for continuous improvement in programs and outreach to drive positive change for animals.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Obliqueness Thesis, published by Jessica Taylor on September 19, 2024 on The AI Alignment Forum. In my Xenosystems review, I discussed the Orthogonality Thesis, concluding that it was a bad metaphor. It's a long post, though, and the comments on orthogonality build on other Xenosystems content. Therefore, I think it may be helpful to present a more concentrated discussion on Orthogonality, contrasting Orthogonality with my own view, without introducing dependencies on Land's views. (Land gets credit for inspiring many of these thoughts, of course, but I'm presenting my views as my own here.) First, let's define the Orthogonality Thesis. Quoting Superintelligence for Bostrom's formulation: Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal. To me, the main ambiguity about what this is saying is the "could in principle" part; maybe, for any level of intelligence and any final goal, there exists (in the mathematical sense) an agent combining those, but some combinations are much more natural and statistically likely than others. Let's consider Yudkowsky's formulations as alternatives. Quoting Arbital: The Orthogonality Thesis asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal. The strong form of the Orthogonality Thesis says that there's no extra difficulty or complication in the existence of an intelligent agent that pursues a goal, above and beyond the computational tractability of that goal. As an example of the computational tractability consideration, sufficiently complex goals may only be well-represented by sufficiently intelligent agents. "Complication" may be reflected in, for example, code complexity; to my mind, the strong form implies that the code complexity of an agent with a given level of intelligence and goals is approximately the code complexity of the intelligence plus the code complexity of the goal specification, plus a constant. Code complexity would influence statistical likelihood for the usual Kolmogorov/Solomonoff reasons, of course. I think, overall, it is more productive to examine Yudkowsky's formulation than Bostrom's, as he has already helpfully factored the thesis into weak and strong forms. Therefore, by criticizing Yudkowsky's formulations, I am less likely to be criticizing a strawman. I will use "Weak Orthogonality" to refer to Yudkowsky's "Orthogonality Thesis" and "Strong Orthogonality" to refer to Yudkowsky's "strong form of the Orthogonality Thesis". Land, alternatively, describes a "diagonal" between intelligence and goals as an alternative to orthogonality, but I don't see a specific formulation of a "Diagonality Thesis" on his part. Here's a possible formulation: Diagonality Thesis: Final goals tend to converge to a point as intelligence increases. The main criticism of this thesis is that formulations of ideal agency, in the form of Bayesianism and VNM utility, leave open free parameters, e.g. priors over un-testable propositions, and the utility function. Since I expect few readers to accept the Diagonality Thesis, I will not concentrate on criticizing it. What about my own view? I like Tsvi's naming of it as an "obliqueness thesis". Obliqueness Thesis: The Diagonality Thesis and the Strong Orthogonality Thesis are false. Agents do not tend to factorize into an Orthogonal value-like component and a Diagonal belief-like component; rather, there are Oblique components that do not factorize neatly. (Here, by Orthogonal I mean basically independent of intelligence, and by Diagonal I mean converging to a point in the limit of intelligence.) While I will address Yudkowsky's arguments for the Orthogonality Thesis, I think arguing directly for my view first will be more helpful. In general, it seems ...

Cameron Berg, an author focused on AI alignment, presents a fresh perspective on advanced AI risks. He argues for the concept of a negative alignment tax, suggesting that investing in alignment could actually boost AI performance rather than hinder it. The conversation explores the vital need for adaptive alignment strategies, likening this to Monte Carlo Tree Search in AI. Berg emphasizes continuous reevaluation in alignment research to match the rapidly evolving capabilities of AI and considers the complexities of human motivation in these discussions.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Match funding opportunity to challenge the legality of Frankenchickens, published by Gavin Chappell-Bates on September 18, 2024 on The Effective Altruism Forum. We have a once-in-a-generation opportunity to improve the lives of millions of chickens raised for food in the UK. In October 2024 The Humane League UK (THL UK) will be heading to the High Court to challenge the legality of fast-growing breeds of chicken - Frankenchickens. We need to raise £55k to fund the hearing. The Jeremy Coller Foundation has pledged to match funding half of the costs up to £28k. We need to raise a further £12.5k to maximise the match funding pot and fully fund the hearing. Please contact me directly should you wish to donate and fight for 1 billion chickens. Frankenchickens ' Frankenchickens' are selectively bred to grow unnaturally big and fast to maximise profits. They are destined to suffer extremely short and painful lives, suffer heart attacks, are often unable to walk and succumb to open sores from laying in their own waste. They grow 400% faster than is natural for their bodies, creating the biggest animal welfare crisis of our time. In the UK alone, there are over 1 billion chickens raised for meat and over 90% are fast growing. THL UK's three-year legal battle In 2020, we saw an opportunity to challenge the legality of Frankenchickens and began building a legal case against the Department for Environment, Food & Rural Affairs (Defra). This culminated in a judicial review taking place at the High Court in May 2023. Getting to this point was a major success in itself as only 5% of cases are granted a full hearing. The judge stated that a full hearing of the facts regarding fast-growing chickens was in the public interest. Represented by Advocates for Animals, we argued that fast-growing chicken breeds, known as Frankenchickens, are illegal under current animal welfare laws, as they suffer as a direct result of their breeding. Our case was bolstered by evidence given by the RSPCA which shows that fast-growing breeds of chicken do suffer, no matter the environment they're raised in. This was despite Defra attempting to block the submission of the RSPCA's evidence. The fight continues In May 2023, the High Court ruled that Defra hadn't behaved unlawfully in their interpretation of the Welfare of Farmed Animals Regulation of 2007. Shortly after the ruling we decided to appeal the court's decision, and continue our three-year legal battle. There is overwhelming scientific consensus that chickens raised for meat suffer due to their breed. Defra itself has offered no evidence to contradict the RSPCA report and even accepted that there are welfare problems with fast-growing breeds of chicken. In October 2023, we found out that our appeal had been granted. In October 2024, we will be back in court, in front of a new judge, to take on Defra to end the cruel use of Frankenchickens in the UK. Our two-day court hearing is due to start on either Tuesday 22nd or Wednesday 23rd October. This is a once-in-a-generation opportunity to force the Government, with one decision from an appeals court judge, to transform one billion innocent lives per year. Our chances of success By virtue of being granted an appeal, our chances for a favourable final outcome have increased significantly. Being granted an appeal means that serious problems with the previous judge's findings have been uncovered, and the judge approving our appeal thinks our case still has merit that needs final and careful deliberation. A positive ruling would mean that the judge found Defra's interpretation of the Welfare of Farmed Animals Regulation of 2007 illegal, and would compel them to create a new policy on fast growing breeds of chicken, one that would invariably lead to farmers being disincentivized or even banned from keeping f...