Safety Concerns Raised By The Race for AI Supremacy

We spoke with tech journalist Kevin Roose about the AI development race over the past decade laid out in his new book, the AGI Chronicles. This SI thing. You know, obviously, the president has said we're gonna redub it SI. Is that something that was already being used in the industry? Because we've heard from Intel folks that this could be confusing because it refers to something else within the intelligence world, but this is a term... Seems like it's on your book. It was already out there. Yeah. This is basically the sort of the thing that comes after... So there's there's AI Yeah. There's AGI, and there's ASI, which is artificial super intelligence. And AI is sort of like everything, you know, we have now. AGI is this kind of point where the the the AI gets roughly as smart as humans. It's roughly a human level AI system that can do a bunch of different things at or around human level. When do we get there? I think we're there. You do? That's what was gonna ask. Okay. I mean, people disagree about what the exact definition of AGI is. But for me, I I... My my sort of personal bar has been cleared. But that's what the whole book... That... That's what the people in this book are are aiming for. They're not trying to make this thing human level at one or two or six things. They're trying to make it human level at many things. And then ASI is the step after that. That's when you have an AI that is better than every human at everything. And so there's sort of... Yeah. It's just that... And that is actually the ultimate goal of this. They are not just... The the people who have been building these systems for the last decade will not be satisfied if these things top out around our intelligence level. They want to build something vastly smarter than the smartest humans who have ever lived. The the principles of this book have been wrestling with the kind of broader implications of this technology for a long time, but it strikes me that in recent weeks, recent months, the public has become aware of that. Wondered how well equipped is a Sam Altman to decide whether something is moral or not or what the implications for humanity might be. You've followed and spoken with the principles in in this book. How well equipped are they to be wrestling with these kind of broader philosophical and existential issues? Well, they might be well equipped relative to the average person, but there are so few of them. I mean, that... This is the thing. Like, the the thing that I realized reporting this book... And I I talked to more than a 150 people at the AI labs that I I covered OpenAI, Anthropic, and Google D Mind. And that was essentially how big the group of key decision makers was. It started off as, like, a couple dozen people who were interested in this about a decade ago, and they grasped something about the world that wasn't obvious, which is that powerful AI was possible and that some recent advances in things like GPUs had made it made it seem realistic. And then the circle kind of stayed small for a long time where, like, maybe a 100 people, maybe a 150 people actually were involved in creating the technologies that we see today. And no matter how good or moral or well equipped those people are, like, it's a very small group of people to be building this thing and making all the decisions that could affect billions of people around the world. And by the way, it's a very homogenous group. Yes. You know, these are, like, mostly men in San Francisco. They are blinkered. They don't sort of have the rest of humanity's experience to draw from, and they're all racing each other. That is the craziest thing. It's like they understand the stakes of what they're doing. They're not naive if they think this thing could blow up the world, but they're all racing toward it anyway. But, like, to what end? When you say, okay. We're possibly on parody with human intelligence. The goal is to be better than that. I mean, I say this jokingly, but also not. Did no one watch the terminator? Like, every piece of science fiction we've made in the last twenty five, thirty years goes to the same inevitable end. You know these guys are aware of this. They're just they're just doing it anyway? Yes. And they have various justifications for that. Some of them think that the future will be amazing, and we'll have this sort of post scarcity utopia where robots are doing all the jobs and humans are just free to pursue our our leisure activities and, you know, make art and think about philosophy all day. Art as well. Exactly. So that's one rationale. The other rationale, the the one that motivates several of the characters in this book is, is that they think this technology is going to be built whether they build it or someone else builds it. They think it is inevitable. The recipe is not that hard. Get a bunch of data and a bunch of compute and some algorithms, and you throw it all into a neural network and outcomes intelligence. And so their fear is that if they don't build it first, if they don't get to this threshold of AGI or ASI first, then China or their rival down the street or someone they really don't trust will do it first. And that is the the sort of fear motivation in addition to, like, the, you know, it's all gonna be utopia motivation. Kevin, it's a small group as you've pointed out. Now we have policymakers and politicians interested in the prospect of of regulating this. We've had a lot of conversations with lawmakers in which we've asked quite bluntly, is is congress equipped to do this? The answer is no. Do have a self proclaimed high IQ president. But what is the barrier to regulating it? Is it momentum? Is it is it congress's lack of desire to do it, or is there really no way in which the government could, given the sophistication of all of this, have any hand in in regulating this smartly? I mean, the government can do it. They just may not because they are stuck in stasis. They have failed to regulate every important new technology on a relevant time scale. I mean, think about it. Social media is now 20 Yeah. Years old, twenty five years old, and we still don't have There's... It's getting regulated through the courts, basically. Regulated through the courts. But it's... But congress has not passed a single law. So I I think in the absence of, like, strong federal regulation, we'll see some state level regulation. We might see some self regulation among the labs, but I... I'm I'm skeptical that we're gonna get some federal regulation. And in part, I think that's because the members of congress do not understand this technology. I was just interviewing a member of congress at a conference the other day, and I learned that in the US house of representatives, the only approved AI tool for member and staff use is Microsoft Copilot. I was gonna guess Clippy. And look. Nothing against Microsoft. I'm sure it's a fine product. But, like, if you... If that is all you're allowed to use, you're not actually aware of what the technology can do at the frontier. You need to be using the frontier models to understand what they're capable of. One of the things that surprised me just anecdotally in my own life with the adoption of this technology from people who are using it for everyday things that you could Google just as easily or you could type the email just as fast is this concern about cognitive atrophy that people are outsourcing basic things and they're going to stop being able to do them. I mean, did anybody you talked to have have major concerns about that, and what is the repercussions other than we're all just kinda waiting to see where this goes? Yeah. People I talked to have concerns about that. It's not the foremost. They're they're mostly concerned about human extinction and mass job loss, something like that. I just... I I feel like this is... I mean, but, like, you talk to people in your daily life, and they're like, oh, I had chat GPT write an email to my boss, or, I had chat GPT respond to somebody on a dating app. And it's like, it's... I am just shocked at how quickly people have stopped doing basic tasks. I worry about this a lot for students. I think if I had had AI during Yeah. High school or college, I would have done, like, very little work. And I imagine that that's the experience some students are having today. I think... Yeah. I... I'm less worried about it for for, you know, adults. I I think it's it's a a problem, but I would I would compare it to something like Google Maps where, like, when it came out, everyone was like, oh, Google Maps, like, we're all... Our our skills at navigation are gonna atrophy. No one's gonna know where we're going anymore. Right. Those two things are true. Correct. And if your phone dies and you don't have Google Maps, like, are screwed. You are lost, and you are asking someone else to use their phone. But for most people, that's just not a problem because they always have Google Maps. K. Kevin, let me ask about sentience. And many viewers, listeners remember your conversation with one Sydney, the Bing chatbot who expressed her affection for you, and you're still married. Nothing nothing came of that. Still married to a human woman. Tried to tried to intervene in your relationship. But we're we're hearing from the pope on these issues. There there there have been these moments in which people have thought that these bots do exhibit signs of of sentience. How do you think about that as we kind of think through where this might be headed, not knowing where where it might be headed, this emerging kind of fear, anxiety surrounding what life might be within the the machine? Yeah. It... I get asked about this a lot in part because of the Sydney thing, and my answer is usually that it could be possible that at some point in the future these systems become conscious. I don't know. I'm not a I'm not a philosopher of mind. I'm not a a cognitive neuroscientist. I don't know what consciousness is or how we test for it in humans, you know, let alone any other organism or entity. But I actually don't think it matters that much whether these things are conscious or sentient or whether they're just inert. I... I'm more of, like, I think it's important what they do and how we react to them. So I think it's a problem if, like, you know, these things are, seducing people and baiting teenagers into, you know, trusting them and and, you know, hurting their mental health. I think it's important if they're out there committing crimes and cyberattacks and developing bioweapons. Like, if something can develop a bioweapon, I don't care if it's conscious or not. I care about whether it's safe. But as one of our producers was asking, why can't you just pull the plug? That's not really going to be an option if it is even now very soon. Right? I mean, it is already hard to pull the plug, so to speak. I mean, it's not like the movies where you run over to the server and you yank the cord out of the wall and Somewhere around just sort of, like, powers down. Like, that's sort of not how this... Any of this works anymore, but it's especially not how it works with these agents, which are... We've... We have seen capable of resisting attempts to shut them down. So, you know, maybe you go to pull the plug, but the agents have already replicated the weights of their model on a different server that they're not gonna let you into because they're trying to preserve themselves. That's not a hypothetical concern anymore. It used to be. There used to be lots of science fiction books about... Like, every Michael Crichton novel has, like, had some version of that. Again, has no one read any science fiction? But, yeah. That is the sort of behavior that these researchers are starting to see now in things like the hugging face attack. Kevin Riesz, thank you very much. Thanks for having me.