Dangers of Sentient AI and Manipulation

What’s so crazy about this moment in artificial intelligence is that many of the most credible voices in AI think there’s a real chance that this all turns out really, really badly.

Anthropic CEO Dario Amodei recently pegged his “chance that something goes really quite catastrophically wrong on the scale of human civilization” between 10% and 25%.

That’s comforting.

Applications to attend the Cerebral Valley AI Summit close TODAY October 17.

Apply right now to be considered for an invitation!

On the series’ first episode we reflect on how generative artificial intelligence and large language models took Silicon Valley by storm.

So in our second episode of the six-part Cerebral Valley podcast, Max Child, James Wilsterman, and I played out the doomsday scenarios. We talked a lot about science fiction and how writers have imagined artificial intelligence turning dystopian.

In the second half of the episode, I talked with science fiction author Daniel H. Wilson. He wrote the books How to Survive a Robot Uprising, Where’s My Jetpack?, and How to Build a Robot Army. Wilson has also consulted with the military to help them game out how dystopian technologies might unfold.

Of course, even in the Anthropic CEO’s estimation, the most likely scenario is probably a more boring one: artificial intelligence doesn’t try to secretly destroy us as we sleep in our beds.

But the fact that there’s a chance is certainly worth considering.

I open our conversation with the parable “The unfinished fable of the sparrows” from Nick Bostrom’s Superintelligence.

It was the nest-building season, but after days of long hard work, the sparrows sat in the evening glow, relaxing and chirping away.

“We are all so small and weak. Imagine how easy life would be if we had an owl who could help us build our nests!”

“Yes!” said another. “And we could use it to look after our elderly and our young.”

“It could give us advice and keep an eye out for the neighborhood cat,” added a third.

Then Pastus, the elder-bird, spoke: “Let us send out scouts in all directions and try to find an abandoned owlet somewhere, or maybe an egg. A crow chick might also do, or a baby weasel. This could be the best thing that ever happened to us, at least since the opening of the Pavilion of Unlimited Grain in yonder backyard.”

The flock was exhilarated, and sparrows everywhere started chirping at the top of their lungs.

Only Scronkfinkle, a one-eyed sparrow with a fretful temperament, was unconvinced of the wisdom of the endeavor. Quoth he: “This will surely be our undoing. Should we not give some thought to the art of owl-domestication and owl-taming first, before we bring such a creature into our midst?”

Replied Pastus: “Taming an owl sounds like an exceedingly difficult thing to do. It will be difficult enough to find an owl egg. So let us start there. After we have succeeded in raising an owl, then we can think about taking on this other challenge.”

“There is a flaw in that plan!” squeaked Scronkfinkle; but his protests were in vain as the flock had already lifted off to start implementing the directives set out by Pastus.

Just two or three sparrows remained behind. Together they began to try to work out how owls might be tamed or domesticated. They soon realized that Pastus had been right: this was an exceedingly difficult challenge, especially in the absence of an actual owl to practice on. Nevertheless they pressed on as best they could, constantly fearing that the flock might return with an owl egg before a solution to the control problem had been found.

Give it a listen.

P.S. I’m on my honeymoon right now in Japan. I was working frantically to record these episodes before I left. My chief of staff Riley Konsella is sending the episodes out for me while I’m gone. If you need anything while I’m away, you should email Riley.

Thanks in advance for being understanding that this newsletter is slowing down for my honeymoon. I’m going to dedicate myself to relaxing over the next two weeks so that I come back hungrier than ever.