The Nonlinear Library EA - Apply to Aether - Independent LLM Agent Safety Research Group by RohanS
Aug 24, 2024
13:57
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Apply to Aether - Independent LLM Agent Safety Research Group, published by RohanS on August 24, 2024 on The Effective Altruism Forum.
The basic idea
Aether will be a small group of talented early-career AI safety researchers with a shared research vision who work full-time with mentorship on their best effort at making AI go well. That research vision will broadly revolve around the alignment, control, and evaluation of LLM agents. There is a lot of latent talent in the AI safety space, and this group will hopefully serve as a way to convert some of that talent into directly impactful work and great career capital.
Get involved!
1. Submit a short expression of interest
here by Fri, Aug 23rd at 11:59pm PT if you would like to contribute to the group as a full-time in-person researcher, part-time / remote collaborator, or advisor. (Note: Short turnaround time!)
2. Apply to join the group
here by Sat, Aug 31st at 11:59pm PT.
3. Get in touch with Rohan at
rs4126@columbia.edu with any questions.
Who are we?
Team members so far
Rohan Subramani
I recently completed my undergrad in CS and Math at Columbia, where I helped run an Effective Altruism group and an AI alignment group. I'm now interning at
CHAI. I've done several technical AI safety research projects in the past couple years. I've worked on
comparing the expressivities of objective-specification formalisms in RL (at AI Safety Hub Labs, now called LASR Labs), generalizing causal games to better capture safety-relevant properties of agents (in an independent group), corrigibility in partially observable assistance games (my current project at CHAI), and
LLM instruction-following generalization (part of an independent research group). I've been thinking about LLM agent safety quite a bit for the past couple of months, and I am now also starting to work on this area as part of my CHAI internship. I think my (moderate) strengths include general intelligence, theoretical research, AI safety takes, and being fairly agentic. A relevant (moderate) weakness of mine is programming. I like indie rock music :).
Max Heitmann
I hold an undergraduate master's degree (MPhysPhil) in Physics and Philosophy and a postgraduate master's degree (BPhil) in Philosophy from Oxford University. I collaborated with Rohan on the ASH Labs project (
comparing the expressivities of objective-specification formalisms in RL), and have also worked for a short while at the Center for AI Safety (CAIS) under contract as a ghostwriter for the
AI Safety, Ethics, and Society textbook. During my two years on the BPhil, I worked on a number of AI safety-relevant projects with Patrick Butlin from FHI. These were focussed on deep learning interpretability, the measurement of beliefs in LLMs, and the emergence of agency in AI systems. In
my thesis, I tried to offer a theory of causation grounded in statistical mechanics, and then applied this theory to vindicate the presuppositions of Judea Pearl-style causal modeling and inference.
Advisors
Erik Jenner and
Francis Rhys Ward have said they're happy to at least occasionally provide feedback for this research group. We will continue working to ensure this group receives regular mentorship from experienced researchers with relevant background. We are highly prioritizing working out of an AI safety office because of the informal mentorship benefits this brings.
Research agenda
We are interested in conducting research on the risks and opportunities for safety posed by LLM agents. LLM agents are goal-directed cognitive architectures powered by one or more large language models (LLMs). The following diagram (taken from
On AutoGPT) depicts many of the basic components of LLM agents, such as task decomposition and memory.
We think future generations of LLM agents might significantly alter the safety landscape, for two ...
The basic idea
Aether will be a small group of talented early-career AI safety researchers with a shared research vision who work full-time with mentorship on their best effort at making AI go well. That research vision will broadly revolve around the alignment, control, and evaluation of LLM agents. There is a lot of latent talent in the AI safety space, and this group will hopefully serve as a way to convert some of that talent into directly impactful work and great career capital.
Get involved!
1. Submit a short expression of interest
here by Fri, Aug 23rd at 11:59pm PT if you would like to contribute to the group as a full-time in-person researcher, part-time / remote collaborator, or advisor. (Note: Short turnaround time!)
2. Apply to join the group
here by Sat, Aug 31st at 11:59pm PT.
3. Get in touch with Rohan at
rs4126@columbia.edu with any questions.
Who are we?
Team members so far
Rohan Subramani
I recently completed my undergrad in CS and Math at Columbia, where I helped run an Effective Altruism group and an AI alignment group. I'm now interning at
CHAI. I've done several technical AI safety research projects in the past couple years. I've worked on
comparing the expressivities of objective-specification formalisms in RL (at AI Safety Hub Labs, now called LASR Labs), generalizing causal games to better capture safety-relevant properties of agents (in an independent group), corrigibility in partially observable assistance games (my current project at CHAI), and
LLM instruction-following generalization (part of an independent research group). I've been thinking about LLM agent safety quite a bit for the past couple of months, and I am now also starting to work on this area as part of my CHAI internship. I think my (moderate) strengths include general intelligence, theoretical research, AI safety takes, and being fairly agentic. A relevant (moderate) weakness of mine is programming. I like indie rock music :).
Max Heitmann
I hold an undergraduate master's degree (MPhysPhil) in Physics and Philosophy and a postgraduate master's degree (BPhil) in Philosophy from Oxford University. I collaborated with Rohan on the ASH Labs project (
comparing the expressivities of objective-specification formalisms in RL), and have also worked for a short while at the Center for AI Safety (CAIS) under contract as a ghostwriter for the
AI Safety, Ethics, and Society textbook. During my two years on the BPhil, I worked on a number of AI safety-relevant projects with Patrick Butlin from FHI. These were focussed on deep learning interpretability, the measurement of beliefs in LLMs, and the emergence of agency in AI systems. In
my thesis, I tried to offer a theory of causation grounded in statistical mechanics, and then applied this theory to vindicate the presuppositions of Judea Pearl-style causal modeling and inference.
Advisors
Erik Jenner and
Francis Rhys Ward have said they're happy to at least occasionally provide feedback for this research group. We will continue working to ensure this group receives regular mentorship from experienced researchers with relevant background. We are highly prioritizing working out of an AI safety office because of the informal mentorship benefits this brings.
Research agenda
We are interested in conducting research on the risks and opportunities for safety posed by LLM agents. LLM agents are goal-directed cognitive architectures powered by one or more large language models (LLMs). The following diagram (taken from
On AutoGPT) depicts many of the basic components of LLM agents, such as task decomposition and memory.
We think future generations of LLM agents might significantly alter the safety landscape, for two ...