The Importance of Alignment in AI Development

The idea of like, let's make the AI do things that humanity or users want is very intertwined with advancing capabilities. There's not really a way to make the models more governable if we don't understand their capabilities, right? In the petition or the letter, they basically talk about the idea that powerful AI systems should not be developed until we're confident that their effects will be positive and the risks are manageable. Sarah: How on earth would one quantify or codify those two targets?