The Importance of Intuition in Chat GPT

I feel like there's at least a decent chance that that will happen. Iliazar is extremely worried that this is going to lead to something very, very dark. I don't know why his intuition is that the thing that wakes up assuming it emerges with consciousness. But if it does wake up, why wouldn't it copy us after it reads everything? Why wouldn't it be somewhat good?