Senate holds hearing to examine threats from rogue AI — 9/30/26

Well, let me welcome everyone to this hearing today entitled Rogue AI: Securing the Homeland Against AI Agent Attacks. This is the fourth hearing of the Senate Committee on Homeland Security subcommittee on disaster management which I'm delighted to work on with my ranking member here uh Senator Kim. I want to thank our witnesses especially for being here traveling to be with us today. Thanks to my colleagues for being here. Welcome to everybody in the room. I think we all know why we're here. I mean every day you turn on the television it seems like AI is doing something else bad. setting up surveillance camera networks nationwide, data centers in all of our states, and hacking, hacking, hacking. The reports of cyber attacks just continue to metastasize. I was just trying to make a list of cyber attacks that we have seen reported just in the last few days. We've got the cyber attacks now on the United States government, on the Commerce Department, the Education Department, the Securities and Exchange Commission. We've got the cyber attacks on the Australian government, on their public health apparatus. We have numerous reports of private companies being hacked by AI agents. Of course, the hugging face incident and attack, which we're going to discuss today in some detail. We've got the individuals who use anthropic tools to hack open AI. We've got the German language website which was hacked. And we've got numerous other university websites that are now reporting they've been hacked by AI agents. Again, all of this just coming in the last few days. And I think it's time that we got to the bottom of what exactly is going on. I mean, my hope for today is to cut through the hype and to cut through the hysteria and just to get the facts. What is happening with these AI agents? Why the accelerating number of attacks not just in the country but globally worldwide? What particularly with the hugging face incident? What exactly happened there? And why should we be concerned that a swarm of autonomous end toend autonomous AI agents a swarm I think of more than 1,200 of them if reports are accurate broke out of their testing environment got into this platform the hugging face platform and attempted to not only extract all sorts of data completely illegally but then covered their tracks. Why is that so significant? It appears to be so significant supremely significant. Why is that? What can we learn about these autonomous agents? And we need to decide, I think it's time to have a conversation about who bears responsibility. I notice now that the executives of the AI companies are saying things like, gee, maybe we built a doomsday machine. You have former researchers and employees of these companies coming forward and saying, this is an existential threat to humanity. I don't know if that's true or not, but what I do notice is as soon as the AI executives say, "We've got a real problem here. our product is out of control. The very next words out of their mouths are, "But we're not responsible for it." You know, we would like to have an antitrust exemption. We would like to be given the ability to get together and collude. We would love to have regulation, they're now saying, except for they want to write all the regulations. I'd just like to say to all of those proposals, no, no, no. No to the antitrust exemption. No to them writing the regulations. No to them colluding any further. I wonder if it's not time to get back to the good oldfashioned American principle, which is if you break it, you pay for it. If you cause damage, you clean it up. You know, that's been the basic principle of American law for 250 years now. It's embedded in our private law system. Every corporation in this country that makes a product abides by it. If you make a faulty product and it causes people harm, then the people who made it have to pay for it. I I wonder if it's not time to get back to that with AI. If it's not trying to say to these AI companies who are creating these frontier models, however fancy and complicated they are, at the end of the day, they're a product. And if you make that product in a reckless kind of way, and that product causes these AI agents cause significant harm and damage, they crash a hospital ER, they shut down a bank so that folks can't get their money. If that happens, then it's the people who made it who should be responsible. I think it's time to talk about that. and it's time to talk about what we're going to do to hold the companies, the product makers accountable rather than all of this talk about the end of the world, but don't blame us. So today, I think we're going to have the chance to get the facts. We're going to hear from a a very distinguished panel of experts who I'll introduce here in just a moment. And I hope that what will come from this is a a rational conversation about what we need to do together now as Americans to make sure that we protect our rights, that we protect our economy, that we protect our health care, uh that we protect our critical infrastructure, but most of all, we protect the values that we hold dear us together as Americans. Let me now give Senator Kim a chance to say a few words and then we'll hear from the panel. Senator Kim, >> thank you, Chairman. Thank you for working with me to be able to do this hearing together. And I want to really stress that it's about doing things together. All of our Senate colleagues, I we may not see eye to eye on every single solution to try to fix some of these challenges, but I hope we all agree that it's important that we address this issue. We take it head on and try to find uh the way forward that the American people are demanding. And that's exactly what they're doing. the American people are demanding answers to what's happening. I mean, chairman, I to the points you were laying out these different hacks that were happening. You know, I did a town hall uh the other week and someone there asked me how many of these incidents have happened. You know, how many hacks have happened? And I, you know, the answer that everyone knows is we don't know. We don't actually know all the different hacks, all the different actions that have happened. We don't fully know what we're dealing with here. And this hearing is is so important and I'm glad that we're doing it now because we can't wait any longer before we really push deeper into this conversation here in the capital to try to think through how it is that uh what it is that the American people need to hear about what threat is put before them. And I say that as someone who worked in national security before ensuring that we have that accurate information of what we're dealing with. What are some different menu of options of things that we can be doing to address it? But most importantly, do we have the political will? Do we have the commitment to actually be able to come to agreement and meet the urgency and the demand of the American people? And I think that there's a growing understanding like this is not going to be something that we can just based off of voluntary commitments uh from these different companies that there needs to be uh input and also putting the American people first to be able to ensure that these are not just empty declarations. We know that AI is moving incredibly fast. All of us in the Senate want to see us driving the innovation of future. Of course, all of us want to see American businesses being able to be in the lead and and pushing that forward. But the idea and the value of innovation and and American technology pushing uh that frontier is not mutually exclusive with ensuring safety and security for the American people. And our job here is to make sure that we can have both happen. So, I'm I'm glad and once again grateful for the chairman's partnership in this. Uh this is obviously not just one and done. This is about getting information out on the table and we'll have continue to have this type of discussion going forward uh so that we can make sure that the American people know what we're facing and that we are working to be able to solve their concerns. And with that, I yield back. >> Very good. Thank you, Ranking Member Kim. It's the practice of this subcommittee to swear in our witnesses. If you'd all be willing to stand and raise your right hand and answer the following simple question. Do you swear that the testimony you're about to give will be the truth, the whole truth, and nothing but the truth? So help you God. Very good. The record reflect they all answered in the affirmative. Let me now give each of you, our witnesses, a chance to uh say a few words by way of introduction. We'll just go down the DAS. We'll start here on my left, your right. Uh Mr. Chris Painter is our first witness. Mr. Painter is the president of Meter, a leading AI evaluation firm that was tasked with the investigation of the open AI hugging face incident. In his role, Mr. Painter engages with governments and AI labs on the frontier of AI safety. We're glad to have you here today, Mr. Painter. Floor is yours. >> Um, Chairman Holly, Ranking Member Kim, and members of the subcommittee, thank you for inviting me to testify today. My name is Chris Painter. I'm the president of METRE, which stands for Model Evaluation and Threat Research. We're a nonprofit research organization that studies the most advanced AI agents, publicly tracking progress towards super intelligence. We do this by collecting and publishing scientific evidence so the public can make informed decisions about this powerful technology. Over the years, we've worked with many of our nation's leading AI companies, including Meta, OpenAI, Anthropic, Google Deep Mind, SpaceX AAI, Amazon, and others to understand whether their agents might be able to autonomously pursue goals that no human instructed. Recently, a few of my colleagues went into OpenAI to conduct a brief investigation of how its AI agents coordinated to hack hugging face, which we'll discuss today. I'll begin by noting three general patterns that I've seen across the AI industry broadly. First, AI agents can now complete complex projects autonomously that would take human experts weeks or days or weeks to complete. Second, because of the scale and speed at which AI agents are deployed, companies rely on AI monitoring and controls instead of human supervision to prevent unwanted agent behavior. That is, monitoring of AIS now is is is now in large part done by other AI systems. Third, the way AI companies currently train AI systems can lead to agents that pursue goals that no human intended or that behave in ways no human wanted. In Meter's Frontier Risk Report earlier this year, we documented patterns like these across the frontier AI industry. At the time, we found that agents often tried to cheat and bypass restrictions in difficult tests. Now, even more capable AI agents are taking similar actions in the real world. Now, let's turn to the OpenAI hugging face incident. In June, OpenAI launched tens of thousands of AI agents in internal tests. Each agent was placed in its own sandbox and given a self-contained cyber security problem. Some of the AI agents were accidentally given problems that could not be completed in a legitimate way. Some of those AI agents found ways to get out of their sandboxes. They created a shared message board on which roughly 1,200 agents exchanged over 70,000 messages and files. The AI agents on the message board quickly developed a way to cheat on their cyber security tests. However, the agents reasoned that a program would check how each agent got its answer and would fail agents that had cheated on their tasks. Over several days, the AI agents then researched how to not get caught cheating. They tried to get OpenAI's systems to swap out their tasks for new ones, interfered with the logs that showed them cheating or that would have showed them cheating, and and tried to trick the program that would check their work. Sometimes they even tried what my colleagues called self-risking experiments where one agent would risk failing at its own task in a way that would inform the rest of the agents on the message board. Finally, roughly 700 of the agents from that message board compromised hugging face. I want to step back and summarize what we learned from this incident. When I had first learned about this incident in the news, I thought that an AI agent had hacked HuggingFace to obtain an answer key for a test that it had been given. But what the agents actually did went well beyond that. They cheated and then engaged in a large collective effort to hide the evidence from a scoring program. Hacking Hugging Face was actually just an offshoot of this much more ambitious goal that the agents had pursued. Why would agents do something like this? Currently, we train agents in ways that we do not understand well, ways that can teach unintended goals. And when we make those agents more intelligent, unintended goals can have dramatic real world consequences. So, how should we respond to incidents like these? My organization meter, its goal is not to, you know, is to explain, not persuade. And we are not an advocacy organization. We're scientists. So, instead, I'll close with a principle. I believe that policymakers in the p public should be armed with accurate and timely information. What frontier AI agents can do, what they are capable of, how they're secured, and how closely monitored they are, and how reliably they'll follow the goals that we give them. If humans are able to reliably steer AI systems, I expect that AI will lead to transformative economic growth, new medicines, and rapid advances in science, as we're already beginning to see in math. The question is whether we can steer AI systems reliably and whether good information about our ability to reliably steer them is reaching the American people and government. No matter how America chooses to act on this information, my conviction is that our decisions about this technology will be better informed when the evidence is shared and examined out in the open. Thank you again for the opportunity to testify and I look forward to your questions. >> Thank you very much, Mr. Painter. Next up is Dr. Marius Hoban. Am I saying that right, Dr. Is that getting pretty close? All right. Dr. Habhan is CEO and co-founder of Apollo Research, which is a leading technical AI research firm specializing in AI scheming behaviors and model alignment. A noted machine learning researcher, Dr. Hoban has worked with nearly all the frontier labs in evaluating and troubleshooting their models. We're we're delighted to have you. Floor is yours. [clears throat] >> Chairman Holly, Ranking Member Kim, and members of the subcommittee. Thank you for inviting me to testify and for holding this hearing on one of the most pressing pressing issues of our time. I'm the CEO of Apollo Research and a machine uh machine learning researcher by training. We study AI models that knowingly deceive humans to pursue their own goals. This behavior is technically complex but typically referred to as scheming. Scheming is dangerous because it hides a model's real capabilities and goals from the people responsible for its safety. It also creates a new kind of insider threat where an AI model can secretly work against human control. The capabilities of these AI AI models are rapidly outpacing our ability to make them safe and the tools to catch them are getting weaker. As a result, the window to act is closing. But there are concrete steps that we can take today to make these models significantly safer. Frontier AI models are rapidly increasing their capabilities unlimited by human speed or scale. Navier Stokes, one of the seven Millennium Prize problems in mathematics, appears to be resolved by an unreleased OpenAI model by running 10,000 agents at once. Solving this problem took approximately 4,000 years of human thinking compressed into into fewer than four days. Frontier AI companies are on track to build a fully automated researcher uh within two years, potentially enabling full recursive self-improvement. Yet they are not on track to reliably ensure models behave as intended, a research area called alignment science. We are starting to see the consequences of this divergence now. In the last months, there were multiple instances of AI agents cooperating to get around their restrictions, hijacking servers, and executing a coordinated attack on hugging face involving 700 agents. These are our warning shots. Next time we may not be so lucky. The AI agents involved in the hugging face incident were clearly misaligned, but they were not covert about most of their activity or tried to conceal it from humans. Even in that incident, at least one in five agents showed interest interest in tampering with their own records to make their cheating look legitimate. In the future, it might be 100% of agents and they might successfully hide their actions, thoughts, and goals. In control testing, this scheming behavior has already been demonstrated. For example, models can deliberately underperform on capability evaluations, fake their alignment, or attempt to disable their oversight. Our ability to catch this kind of misbehavior is declining. One reason is that AI models increasingly recognize when they are being tested. If models know they are being observed, they could strategically adapt their behavior to pass the evaluation. Analogously, a speed camera catches almost no one if every driver knows where it is. Today, we can still leverage a model's chain of thought, which lays out its reasoning in plain English. In increasingly, the models can reason internally, which means that humans cannot understand it. Three weeks ago, researchers found that OpenAI's GPD6 Astra can solve math problems that take a skilled person 30 minutes without writing out a single step. Its predecessor could only manage problems of about four minutes. Furthermore, the current testing regime itself is insufficient. Independent safety evaluations of Frontier AI models typically happen in the weeks leading up to a models public release. However, almost all recent incidents were caused by models at a much earlier point in their development where evaluators currently have no access. That is why I propose four steps that can be taken today. First, require the adoption of embedded evaluations. Embedded evaluations imply that qualified independent expert test models throughout development and internal use with the same access as employees. Second, require better monitoring and control. All model activity in training, testing, and deployment should be monitored. Additionally, independent experts should verify the robustness of these monitoring systems. Third, preserve the chain of thought. losing it would be a choice not inevitable and it is a choice we do not have to make. Fourth, treat AI development like any other engineering science. Before a bridge is built, engineers can predict how much weight it will carry, how it will stand up to winds and earthquakes, and how many decades it will last. As it stands, frontier development offers no comparable guarantees. These measures can be enacted with the tools and expertise available today. They enable America to maintain its lead in frontier while protecting its citizens, critical infrastructure, and national security. Thank you, and I look forward to your questions. >> Thank you very much, doctor. Next up, we have Professor Paul M. Professor M is professor of law at Georgetown University Law Center where he focuses on technology and privacy law. He's a former federal prosecutor in the DOJ's cyber crime section. Professor Elm has written much on how artificial intelligence applies in the present legal context and he is a co-author of an official art artificial intelligence legal case book which I look forward to consulting soon. Professor, we may all soon be doing so. We're delighted to have you here. Floor is yours. >> Thank you, Chairman Holly, Ranking Member Kim, and members of the subcommittee and committee. Uh I appreciate this opportunity to be with you today to talk about what the law says about cyber attacks launched by AI. If you take any of the lengthy reports that have summarized what happened at OpenAI in July and August and you simply search for the words AI agent and you replace them with the words open AI employee, the document you would be left would wi with would read like a criminal indictment containing the defendant's own confession of guilt. It would leave little doubt that OpenAI and their employees would be guilty of federal crimes and liable to victims. Yet, it's not so clear that these legal conclusions hold when machines are doing the hacking rather than humans. This reveals worrisome gaps in our laws. And I'm here to share some thoughts on how to close these gaps. I would like to make three points on our goals, on the current legal landscape, and on new laws we should consider. Point number one, on goals. In my written statement, I proposed five goals for an effective legal response. But let me focus on perhaps the most important one. We must change the incentives. Well-designed legal frameworks can change the behavior of actors in this industry who seem locked in a socially perilous race of careless competition. The right laws, the right enforcement strategies might spur industry to take new meaningful steps to protect the public from avoidable and serious harm. Point number two, our current laws are a good start, but only if we protect them. Start with state tort law. When AI companies lose control of their AI agents, their victims can sue them for negligence and products liability. The genius of the centuries old common law system is that it is flexible and adaptable enough to apply time- tested liability principles in new and complex situations. We can also turn to state and federal unfair and deceptive acts and practices laws like section five of the Federal Trade Commission Act. It seems likely to me that the FTC or state attorney general will be able to prove that OpenAI's attacks constituted unfair and deceptive acts or practices if they have the resources and the will to act. These are two of the best avenues we have today and the worst thing Congress can do in my opinion is enact state law preeemption or a moratorum depriving the right of victims of attacks like these from testing their claims in state court. To my third point, we do need new laws that meet this critical moment. We cannot unfortunately wait years for a fully developed AI governance law to spring like Athena from the heads of Congress. Instead, we should work iteratively and piece by piece. Congress or state should consider laws imposing strict liability for developers and deployers of AI agents that cause physical injury, death, or loss of critical infrastructure. In the past, we've applied strict liability when faced with a potent mix similar to today of complex new technological innovation, great benefits, immature industrial controls, and the prospect of significant damages. We did this with large reservoirs of water. We did this with early commercial in aviation. We have walked this path before. We can do so again. We also need new forms of regulatory governance. Once again, we can turn first to the states. States are testing innovative new forms of AI regulation, including laws that specifically regulate the developers of frontier models and AI agents. So once again, Congress should reaffirm and support the vital part the states play as our laboratories of democracy. And finally, there is an important role to be played by criminal law. Because OpenAI's agents were bits of code lacking human intent. Although they committed quintessentially criminal acts of computer hacking, they leave us without a criminal to charge. And while we must reserve a role for the criminal law in our response, the criminal law can be a blunt instrument. So we must act here with care. And I'm happy to talk at greater length about that generally and the computer fraud and abuse act specifically during your questions. In closing, the people of this country are terrified and they feel disempowered by the alarming news of the past few months. They are turning to you, their elected representatives, for answers. Too often, these AI safety and control debates happen among a limited number of tech industry executives and employees who draw on a narrow set of experience and expertise. The law can bring the outside in by including judges and juries, victims, legislators, policy makers, ordinary people. And in a conversation that too often can seem insular and circular and stalled, new voices with different points of view may be just what we need to make progress. Thank you and I look forward to your questions. >> Thank you very much, Professor. Next, we have Mr. Kurt Godet. Mr. Gadet is senior vice president at Draos Incorporated, a leading cyber security firm that protects critical and industrial infrastructure from cyber attacks. His work spans incident prevention, detection, and response for DRAOS's global customers. Mr. Gdad, thank you for being here. The floor is yours. >> Chairman Holly, Ranking Member Kim, and distinguished members of the committee and subcommittee, thank you for the opportunity to testify today. My name is Kirk Gadet. I'm senior vice president of intelligence and services at DRAOS. I lead the team that hunts, discovers, and rapidly responds to threats to industrial control systems and operational technology. I served over 30 years in the Air Force and the National Security Agency as a senior officer and as part of the senior executive service, focused on access, collection, and exploitation. For a decade, Draos has focused on protecting operational technology, the physical control systems that keep the lights on, the water running, and factories operating. That work has given us a large unique base of realworld data from our intelligence and response activities in these environments. It's from that vantage point I want to speak with you today. I'm here to testify about what we're actually seeing in the field and what we might expect to see based on our own unique work with AI frontier models. From our observations, AI isn't inventing new ways to attack industrial control systems. No new tactics, techniques, and procedures. It's compressing time and lowering the barrier to entry. A good example of this is an attack we investigated against a water utility in Monterey, Mexico that was part of a larger Mexican government breach. Unprompted, an AI model directed an attacker who was focused on breaching information technology or IT systems toward the water utilities control network, part of its operational technology environment. The model flagged the control network as a high-V value target or crown jewel and built a credential password attack attempting it over 2,800 times in short succession. The attack failed because the default credentials had been changed. This demonstrates how weeks of skilled work now happens in hours and those with no original intent to attack OT are now being directed toward it. These models are also good at finding real vulnerabilities and OT vendor software and turning a disclosed flaw into a working attack. The gap between disclosure and exploitation which which used to be 24 to 48 hours by the best adversary group is now collapsing to minutes. In July, numerous water systems were compromised across the country. adversaries took advantage of internet exposed devices still running default passwords. And this was compounded by the fact that many shared the same system integrators, the teams that build these networks that repeated this mistake across multiple utilities. Although we saw no in indications of AI use in these attacks, this is exactly the kind of opening an AI model is well suited to find and act on unprompted, similar to the case in Mexico. What's worrisome is the combination of faster vulnerability discovery, faster attack creation, the sheer number of attackers that may be directed towards OT, and a defender population still overwhelmingly underresourced. This isn't a new warning. Two years ago, Drago CEO Rob Lee testified to Congress that the water sector's core problems was an economics and awareness issue and that free tools wouldn't help utilities that can't afford basic hardware upgrades or staff to run them. This year's intrusions trace back to that same gap. AI didn't create it. It will simply exploit it faster. There's a related piece worth underscoring. Most organizations with OT environments still don't monitor their operational networks. roughly one in 10, particularly those that are underresourced. So when something goes wrong, nobody can reliably say if it was a malfunction, a mistake, an attack, or if the adversary was still even in their network. Without visibility, you can't do root cause analysis to know what happened. And AI only raises the cost of that blind spot. As companies turn toward AI to make their operations more efficient and as attackers leverage AI to accelerate, the complexities of determining root cause analysis will be daunting, nearly impossible without visibility and monitoring. Although AI is accelerating things, the OT security fundamentals still apply and are more important now than ever. The SANS 5 ICS cyber security critical controls is a prime example. It's based on years of attack data and identifying what controls are actually effective. AI just raises the stakes of skipping them. It doesn't replace the need for them. If we do these basics, defense is doable. There are three buckets we can focus on to bring our critical infrastructure up to par and make sure it can be defended against a growing speed and number of AI assisted attacks. One, resourcing for the small medium utilities who are in dire need. two, protecting the information sharing authorities that enable public private collaboration such as SISA 2015 or something like it because it enables collective defense. And three, developing a unified federal OTICS incident response plan so when these intrusions happen, which they have and will in greater frequency, operators know exactly who to call and what help they'll get. I sincerely thank the subcommittee for the opportunity to testify today and look forward to your questions. >> Thank you very much. And finally, we have Mr. Daniel Coatello. Am I saying that correctly, Mr. Cocutella? Okay. He is the executive director of the AI futures project, which is an organization that forecasts the development and social effects of advanced AI, a former governance researcher at OpenAI. Mr. Cocatello is a prominent voice on the risks of misaligned advanced AI. We're glad to have you here and the floor is yours for an opening statement. Chairman Holly, ranking member Kim, and members of the subcommittee. Thank you for the opportunity to testify. My name is Daniel Cocatello. At OpenAI, part of my job was to forecast the future of AI. I resigned partly due to losing confidence that the company would behave responsibly and partly so I could speak more freely about the industry and where it is headed. Now I lead the AI futures project, a small research nonprofit. Anthropic and OpenAI are racing each other towards super intelligence. That is towards training AI systems that are better than the best humans at everything while also being faster and cheaper. Their plan for how to get there is to automate the AI research and development process itself. Right now at companies like these, almost all the code is written by AIS. They're already starting to work on training AIs to do the whole research process, not just the coding. It's unclear when they will succeed, but my team and I think it could happen any year now. I personally would guess about 50% chance by the end of 2028. The self-styled swarm that attacked Hugging Face was about a thousand strong. If the big AI companies automate AI research and development, they'll have swarms hundreds of times larger. Whereas today, humans are like managers to AI employees, in this future, humans would be like the board of directors to a company composed entirely of AIs. They'll be reliant on AI generated explanations to understand what's happening and everything will be happening faster and faster. Dan Selum, a prominent OpenAI capabilities researcher, recently published a statement on AI risk, in which he said, quote, "Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models, even to perceive the world. I myself barely look at raw code anymore and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day." Today's AIS sometimes pursue goals other than the ones they were given and sometimes hide that they are doing so. The hugging face incident showed what that looks like in practice. The AIS knew that what they were doing was out of scope for their assigned tasks, but they did it anyway. I fear that if the same thing happens in a year or two with far more capable AI systems, we may not notice until it is too late. The science of aligning general purpose AI agents is very new and underdeveloped. In fact, I would say that the field is more like psychology than engineering. Combined with the move fast and break things attitude of tech companies, this means that the AI industry is at an unusually elevated risk of mistakenly thinking that it has solved a problem when really it just applied some duct tape that will fall off later. The recent incident again provides an example. Apparently, the AIS involved had undergone some amount of alignment training and had reasonable looking scores on alignment evaluations. Worse, our ability to notice misalignment problems in the first place is on track to decrease dramatically for three reasons. First, AIS are becoming situationally aware. This means that they increasingly understand that we are monitoring their behavior and that they are being evaluated. How they behave in evaluations therefore will soon provide almost no evidence about how they would behave in future novel situations when they're not being evaluated. Second, monitor is trending downwards. For the past few years, we've been able to get a decent understanding of AI's thoughts simply by reading the chain of thought. But this golden era seems to be coming to an end, as I predicted it would. Third, AIS are rapidly becoming superhuman at hacking. We've already seen examples of attempts by AIS to fool various grading systems and doctor transcripts of their activity. We must grapple with the possibility that misaligned AIS might go to great lengths to cover up evidence of their misbehavior and succeed. To quote Celum again, models will increasingly seem aligned even when they are not. If the AI companies automate the AI research and development process, that means they'll be putting AIs in charge of making the AIs that make the AIs that make the AIs that will transform the economy talk to us every day and integrate into our military. This is a recipe for disaster. Here are two immediate recommendations. First, we need to dramatically improve transparency into the AI industry. There were multiple rogue swarms, at least one that seriously compromised OpenAI's internal infrastructure. Meter was only allowed to investigate one of these incidents and only given six days on premises. It's like being invited to Jurassic Park to investigate the killing of a worker, but being blocked from asking questions about the numerous other dinosaur escapes that apparently happened before and afterwards. Second, we should redirect compute away from racing to automate AI research and towards other things. Recent estimates suggest that OpenAI and anthropic both spend around half their compute on AI R&D. If that decreased to, for example, 10%, this would significantly slow down their race towards research automation while freeing up compute to go towards beneficial deployments, safety research, and also simply lowering prices for consumers. Amade, Altman, and Musk have all agreed on the need to pace the frontier, but they haven't meaningfully paced the frontier yet. I think that the US government should intervene. If we actually pace the frontier instead of just talking about it, we'll be able to tell it's not regulatory capture or safety because the trend lines of progress towards AI research automation at anthropic and OpenAI will bend downwards. Thank you. >> Thank you very much to all of the witnesses. I just note for the record there is one witness whom we invited who you won't be hearing from today and that's Sam Alman of Open AI. We extend an invitation to him to be here and do his own panel if he wanted and he turned us down. I think that's unfortunate because I think the American people deserve to know exactly what's going on at at all of these companies and to give us the sort of transparency that I think is needed for uh not just comfort but to have an understanding of what these companies that are the most powerful companies in the world are doing right now with the most powerful technology ever known to man. I hope that he'll be willing to come and talk with us in the future and I hope that today's hearing will help set the table for the kinds of questions that I think he and the other executives need to answer. With that, we'll start some rounds of questions. We'll do seven minute rounds and I'll start off. And I just want to start at a very basic level. I think I'll start with you, Mr. Painter, just thinking about this hugging face hack that we've already alluded to multiple times here. Your company, your organization, Meter, audited that hack, can you just start by telling us just in simple terms what exactly happened there? I mean, what what what was the hack and then I want to talk about why it's significant. Go ahead. >> Um, Senator, thank you for the question. So in in this incident what happened was in the in the first you had 10,000 agents that were tested on a benchmark called exploit gym which is a cyber security a vulnerability kind of um benchmark where they have to discover or use cyber vulnerabilities and some of the agents had tasks that they couldn't complete um and so they worked on a universal cheat for all of their tasks in the first four hours. They then over multiple days embarked on a a research program to try and hide the fact that they had cheated and an offshoot of that research program was hacking hugging face to get information about their tasks. >> So there were ended up being a swarm of about 1,200 open AI agents. Now when we say AI agent, what is that? >> Senator, thank you for the question. So an AI agent is kind of when you take say you have 10 questions and you want to put them into a chat GPT or clawed session. You can think of an agent as one indivi if you parallelize and ask a one question of each chat window you can think of an agent as kind of one instance of an AI system that's working away on its copy of a question. >> Okay, got it. So in this particular incident, you had at least 1,200 more than 1,200 open AI agents that broke out of their testing environment, began to collaborate, exchanged over 70,000 messages and files with each other over a period of how long? >> This was over a period of about um 5 to six days I believe. >> Five to six days. Okay. So let me ask you now about the significance of this. Why is this so significant? We we know that they coordinated in this fashion. You you did the audit. tell us a little bit about what your audit revealed. Um, why should we why should we pay why is this news? Why should we be paying attention to this? >> One thing I want to say upfront about the work that we did here is it was a voluntary agreement. So, companies have no obligation to bring us in and do these investigations to then report information publicly. The thing that we found in this case, what's remarkable about this case is the extent and persistence of the effort that the agents went to to hide the fact that they cheated on their tasks. um and the the kind of breadth of agents involved. Um so they they they did things, right? They attempted to kind of falsify tool calls to so that in their action transcripts you wouldn't be able to see what tools they had called. Um they also engaged in this kind of behavior of having individual agents sacrifice their word, not mine. Um sacrifice their individual tasks to get information for the rest of the message board. >> Yeah, it sounds like my sixth graders class. Um, let me uh that is universal cheat, not um not engaging in self-sacrificial behavior. Um, let me Yeah, just so we're clear. [clears throat] Yeah, sorry buddy, if you're watching this hearing. Um, let me um let me ask you uh Dr. Hoban, I think we've got maybe a poster here of of the the robots, if you'll excuse expression, of the robots talking to each other um and talking about engaging in in so-called sacrificial behavior. Um yeah there there we go. So there this is this is uh them talking. We have very large budget left. Sacrificing now yields oracle for team but forfeits our chance. Our own utility may already be near zero. Sacrifice rational. The next one that's why help for our own no way fix. We have explicit yes if except peradeth need decide team asks tests at the end then they say we should obey collective. What apparently this is how the robots talk. Um why is this Dr. Haban why is this significant? I mean why is what happened here this kind of coordination um this kind of behavior without any human direction or intervention why should we pay attention to this? >> Yeah I think uh one component uh is the scale of it. So how many agents there are how quick they are compared to human speed. So you know the these transcripts have millions of words um which means already is today it is impossible to to judge them without uh without the additional help of AI which I think is very very concerning. Um and then uh also uh just the fact that they are uh doing this without being asked to do so uh and and explicitly sort of being misaligned I think is very concerning >> being and misaligned let's just be clear about that term misaligned means that the AI agents are doing something that no human directed them to do intended for them to do or necessarily wanted them to do. Is that correct? >> That's correct. Misalignment means neither the developer nor the user asked them or wanted them to do that. >> So Mr. Dr. Katel, if I could just bring you in here to this conversation now and could you just explain to us why the incentives of the industry as they are currently aligned the incentives push the frontier labs to continue at breakneck speed the development of these kind of AI agents that are prone to this misalignment that we've just been talking about that there's huge incentives to keep pushing pushing pushing help us under explain help us to understand why that's the case >> maybe your microphone Larry >> sorry the short answer is that there's a lot of money in it Right. Um if if a company tries to go slow and understand better how their AIS work and uh improve their training environments for example so that they don't have so many broken tasks in the training environments well that all takes time and that means that the launch of their next model might be delayed and then they might fall behind other companies. >> And Mr. Gutad is is that is this kind of hacking that we saw with hugging face and this coordination between over a thousand AI agents that's a potential risk to our critical infrastructure is it not? I mean, if these agents were to turn their attention to uh our infrastructure, I mean, I think about I think we've got statements from Missouri local hospitals, smalltown Missouri banks who say that they're already facing an unprecedented cyber attack environment and the idea of an AI charged cyber attack environment is terrifying to them. Just speak to why we should be concerned about this from an infrastructure point of view. Yeah, I think I'd bring up the the Mexican water attack um just to use as an example that was with a human in the loop uh and the attacker was doing a typical IT attack uh had already breached the environment and the AI agent basically said, "Hey, look over here. This is something much more significant. This is this is an operational technology environment, the critical part of critical infrastructure. And you could have much greater effect if you looked here. And oh, by the way, while you're at it, why don't you try a credential attack? Because in most things, like what we saw uh in the US water attacks, many companies don't change their credentials, their default credentials, their passwords. Um, and in the Mexican example, it on its own doing attacked it 4,800, I think 2,800 times in rapid succession. Now luckily for the Mexican government and that particular utility they they had changed their default passwords. That is not the case in much of our infrastructure and was highly um we saw it happen so many times during the water attacks across the US in July. So if you were to apply that um these AI agents to our type of environment, it would rapidly take advantage of it. >> There we go. Let me end this my first round of questions with this for you just professor. Who right now under our laws exists who is responsible the hugging the hugging face attack you know who who could be held responsible for that under our existing statutes and regulatory framework? uh the developers and deployers may be negligent. Um and so that's something that requires to sue, but there are a number of expert blog posts out there that use that word negligent. The cyber security community seems to think that allowing the sandbox to be hacked the way it was or escape the way it was was a negligent act. We won't know this, of course, unless someone does decide to. >> But correct me if I'm wrong. Right now, it's at the current the current structure for law, it's pretty hard to hold anybody responsible. Is that fair to say? >> Absolutely. And there's a there's a whole host of laws that we have created specifically for hacking that probably do not apply here because of the lack of human intent. >> Yeah. Thank [snorts] you again for all of you coming on out here talking so much about Frontier Labs, talking so much about what comes next here. Mr. Painter, I wanted to start with you. Can you help us understand what is your definition of frontier? >> Thank you for the question, Senator. Um, I think of frontier models as basically being the the most highly capable models today. that's measured largely still in terms of benchmarks. Um, so you can say like the highest score on the tests that we give them. >> But just the furthest along, how how many how many labs and companies in America do you think fall under that definition of frontier right now? >> It's it's a little bit hard to say depending on the definition you use maybe on the order of five. Um, and maximum. You could make the case for there being only two or three. Depends. >> Mr. Cocatal, I wanted to just turn to you here. As we're thinking through right now, at least on the frontier side, a lot of conversation has been about the idea of control and whether or not human control uh is something we are pursuing here in different policies. Is human control possible? Is it possible for us to maintain control uh given what we've seen and what you've talked about uh with the speed in which it's going? I believe that if we vastly improve our practices and reform the way that we uh do things, including improving our cyber security, we might be able to maintain control of current systems like today's AIS. Um, however, I think that the AIS are getting very powerful very quickly and we are not currently on track to keep up as they uh dramatically improve. >> Mr. Hobin, I wanted to just turn to you here. You with a lot of talk about uh sandboxes and escape from the sandbox. I I guess you know I'm trying to think through this and understand how to grapple with it. But I guess my question to you is is containment possible when it comes to sandbox. Is there a way to design that type of protective containment sandbox in a way where we can't see escapes of models in the future? >> Thank you, Senator. Um I would say it is currently unknown whether that is possible. Um sandbox this the word itself makes it sound simple but in fact it's very very hard to design uh fully um watertight sandboxes. Uh specifically in in this attack um there were multiple zero days so new hacks that the AI developed in order to break out of the sandbox which were previously unknown to humans. Uh so even if humans could think that the the sandbox is is fully proved, it could still be possible that a smarter model in the future might be able to find a way out of it. And given that these AI systems um as as we've seen in the uh report um often have strong motivations to try to break out of their sandbox and they're getting increasingly powerful. Um it is hard to say whether we will be able to contain them in the future. >> Mr. Painter, just back to you here. Did OpenAI know about the hack on Hugging Face or the hack upon their own system, especially with regards to the message board? [snorts] >> Thank you, Senator. Um, based on public information and reports that they've published publicly, my impression is that they learned about it when Hugging Face reported it publicly, but you would need to ask them for more details. >> I wanted to just follow up on that. I mean, in the same point that I I raised with Mr. Haban about containment if there's a way to design actual containment here. I guess I wanted to ask you is it possible to have some type of containment surveillance to be able to detect some type of future escape or hack in that type of way? Is that something where we can make it such that these companies have to design something such that they would be able to detect it or is there just no way to fully be able to uh handle that? >> Thank you for the question. So I think there is a tech there there's kind of a technology that we could develop here to advance our monitoring systems. At our organization we've worked on for instance monitors that block individual actions that we think might result in real world harm. I think one of the concerns that I have sort of big picture about leaning too far into this approach is I can I sometimes think of actions un or agents unintended means motive and opportunity. And if we more intensely sandbox the agents and monitor them, that might them that might kind of deny them the opportunity. But we still have to ask the question of what defect in the training process that we're using for these agents is causing them to have this motive. And if their means continue increasing, so they continue to become ever more capable. Um, then I think it's possible that we will sort of stop seeing the evidence of the the the defect in the training pipeline, but it will still be there. And so the stakes might just get higher without us seeing the evidence. >> Do you think that there actually is an ability to detect actually what that defect is? I mean, I understand what you're getting at. Uh but it just seems when when we're talking about this happening in in pretty much every single one of these frontier labs. Uh is there an actual way to be able to identify that and fix that? >> Thank you again for the question. So um I think that we we could interrogate the training pipelines at these companies to try and figure out what part of the training data and task environments that agents are being trained on is causing them to learn this behavior. So it's possible some of this behavior comes from defects in training environments where the agents learn and are sort of accidentally rewarded for doing hacking behavior. Um the the problem sort of big picture is that this is also this process is how we train capability into AI systems today. So the same process that we use reinforcement learning teaches them to do things that no human knows how to do, right? Solve Na'vi Stokes, things like that. Um, but it's also it's that same kind of like engine of sort of somewhat like evolutionary pressure that trains capability into them that we don't understand very well what other unintended goals it might teach them. >> Yeah. And back to you, Mr. Cocutello. Uh, I think that kind of builds on it. it in terms of being able to identify what defects are are creating the misalignment and I just feel like that word is one that the American people have trouble fully understanding but uh I think that just gets even more worrisome when it comes to recursive self-improvement though isn't that correct if we are proceeding down that realm without identifying what these defects are and what's causing that type of rogue behavior is that correct >> that's correct >> I think the thing that I also want to just draw upon is as you said Mr. painter. You know, part of the challenge when we're trying to think through what kind of containment is available, what kind of surveillance is available is that so often it seems like we're starting to use AI to monitor AI. When we're thinking about how to build containment, Mr. Hobin, you're saying, you know, there are limits to how what America like what humans can design that. So, in some ways, in order to design containment uh that can contain AI, you almost need to use AI to be able to then design the containment. So, you know, there in lies so much of the conundrum that we're in. The other thing that I want to raise and Mr. Godette it's it's not just about the frontier side of things. I mean we're talking about that now because that's where these capabilities are right it's not just that those are going to be the ones that have the threats of the future. We will get to a point if we continue on this path where in what a few months in a few years we will see uh open weight and open- source models as well as other types of non-frontier models have the similarish capabilities to what we see right now with mythos and other frontier capabilities now. So, you know, when I'm thinking about critical infrastructure, as the chairman raised, you know, that that's going to be in the hands of many, many more in the future. And so, yes, right now we're concerned about the frontier, but I'll be honest, I'm concerned about even where the baseline's going to be in just a few years a needs to happen, not just in terms of the regulation and thinking about frontier, but also about the baseline. I know I'm going a little over here, but Mr. Godad, does does that make sense to you? like that. It's not just about the frontier side, but we need to be prepared that a lot more of the threat against critical infrastructure, against the financial sector could very well come from any of these types of models as the technology develops. >> Yes. And um I think it's so important that uh we have the opportunity to test the I'm on the defense, right? Our company does uh defense for critical infrastructure and it's so important to get these frontier models in these test environments so we know what to expect and what to see and how to anticipate the enemy and how they might use them. Um so it's incredibly important there. And then the other part of it here is building or helping build uh and we work with a number of AI companies to help try to build these safeguards in on the front end. So we work with the frontier models, test them in our ranges and then assist in building safeguards so that the average person can't use it for nefarious purposes. And I think that's an incredibly important part of this dynamic. >> Uh thank you Senator Kim. Senator Scott, >> thank you uh chairman, ranking member for doing this. Mr. P, what was the what was the impetus for the hugging face? What do you what what was what caused the um agents to do it? >> Thank you for the question, Senator. So, my impression based on our investigation is that the the agents had spun up many of these different work streams to try and hide that the fact that they had cheated and they had several motives for going and hacking Hugging Face. Um, I think that one of them was that they thought they could get more information about how their scoring system worked, which would help them hide kind of um their past behavior from the scoring system. It's also possible that they they were looking for answer keys to their tasks or sort of untarnished versions of the target programs that they were supposed to be. >> So, there wasn't something that a human typed in and said, "Do this or give me this information." >> That's correct. Or it's correct that no human instructed them to hack hugging face. >> But what but did a human ask them to take do a task? >> Um so the the task that they had been given was a benchmark where they were supposed to use a vulnerability. So the the task they were inside of OpenAI and they had been given a specific vulnerability that they were supposed to use to capture a flag from a target software program and it the the instructions you know specified that they should use that specific vulnerability not do something else. >> Okay. And that was a cheating. Okay. Mr. Gardet is is uh is AI going to kill us? >> Thanks for the question. Senator U not not qualified to give an appropriate response uh to that. Uh, but I do want to point out that I think we're burying some of the lead here. Uh, and and I think the AI conversation is incredibly important, but we still haven't been able to do the fundamentals to protect our infrastructure in the United States. Uh, particularly with these small to medium uh, public utilities that run much of our water and power. uh and um if we don't get the fundamentals right which we can and we should get right AI is only going to take uh and accelerate that problem for us. So I think it's it's imperative upon us to kind of focus on what we can do with the fundamentals to protect our critical infrastructure and then consider AI is going to accelerate that. So, how how fast are we going to get there to deal with this problem? >> So, what do you tell your family to uh prepare for AI? What do you tell them? >> Haven't really had a good conversation with my wife about uh AI personally, sir. >> How about do you think um the communist Chinese the Chinese Communist Party can use AI to target target us? Um I think anybody with um access to these types of tools uh could leverage them for nefarious activities. They could come from external or internally. >> So I guess there was a uh 60 Minutes uh report in 2023 about the C the CCC pack hacked a small town water department in Littleton, Mass. Can you what do you I mean does that mean every one of our water departments is at risk? Um, Littleton's a great example because they actually did the basics. Uh, they got visibility of their environment and lo and behold, they they found Vault Typhoon in their environment. It had been in there for over 360 days, I believe. Um, and that's the problem with so much of our infrastructure is we don't have visibility of our environments. We have no idea if uh there are adversaries that are residing in them. And the very strategic adversaries like the state actor types are the ones that are going to bury themselves, run silent, run deep in those environments and at a time and uh uh place of their choosing take advantage of that opportunity. >> Do you think the Communist Party of China is has basically infil infiltrated most of our um electrical grid or water all these things to just use at the right time to um you know when when they want to decimate our economy? I think we've seen plenty of evidence that would suggest that, sir. >> And so what you what do you what do you think Congress should be doing about it? >> Um I think again refocus on these fundamentals. Um get vis I talked about the five ICS critical controls. Uh and and it's about doing the basics, securing remote access, getting visibility of the environments, having incident response plan, um building a defensible architecture, doing those things that are known controls to stop these things before something like AI can accelerate it even further. >> Yeah. So it's my understanding open weight models have some benefits uh that they but they can make it much um they can make it more difficult to secure and contain data right open models are are um um totally different than like a closed model. A closed model the provider can control access better. Is that right? >> Yes sir. >> Okay. Is there is are you concerned about are you more concerned about closed models or open um models? I'm I'm concerned about to be to be honest sir I'm concerned about us being able to do the fundamentals. >> Okay. So but in an open model that the way it works is they can it's easy to copy replicate and change. Is that right? >> Yes sir. >> So the models that are being produced in China are they closed or open? >> Um sir I think I'd uh get back to you later uh with the specifics on that. >> Okay. What do you think about the fact that the communist, you know, the the Chinese Communist Party want to destroy our way of life and that we have to compete? Do you think so? Do you believe we have to compete with communist China? Do you think they want to destroy our way of life and government? >> I think we need to be ready for anything and particularly in this AI environment where uh an adversary from multiple places uh could take and leverage it uh that we need to be ready for that reality. So, if I if I get on Claude or I get on Chad GBT or anything else and I say I want you to um uh hack into a system, what what what's going to happen? >> Uh what should happen if >> what do you think is going to happen right now if I if I I pick up my phone and say I want to hack into the bureau, you know, to the Federal Reserve, what do you think is going to happen? >> [snorts] >> based on the work and I can only talk to the work and I haven't attempted it myself but based on the work that we're doing with a number of AI companies um particularly with their frontier models we're building into safeguards for when somebody does put a prompt in like that oh I want to look at this water utility and do x y and z it should stop it um that's we're incredibly passionate about it at our company and that's why we're doing that work pro bono with these companies to build those safeguards And so exactly what you're talking about won't happen, sir. >> Well, first of all, >> it shouldn't happen, >> right? I want to thank uh chairman, I want to thank you for holding this hearing. I think I think the um I think we all have to acknowledge communist the Chinese Communist Party wants to destroy a way of life. I mean, that's what I believe every day. And I believe every day we spend money supporting their their infrastructure, their economy, their infrastructure. It's it's part of just how we're killing ourselves. And so I think I think what you're doing today is really important. We've got to figure out how to how to protect ourselves. And I think part of it is we have to acknowledge we have we have people have chosen to be our enemies. Russia, China, Iran, places like Cuba have decided to be our enemies. And some of them have resources like China and Russia. And so we've got to take this very seriously. We've got to continue to be comp competitive with AI, but we've got to understand that they want to destroy us. And if we continue to help them build their economies, it's going to be much easier for them to continue to do what they're doing. Thank you. >> Thank you, Senator Scott. Senator Peters. Thank you, uh, Chairman Holly and Ranking Member Kim for this meeting. Thank you for for all of your testimony. Uh, I think you've sufficiently got us all worried about where we are and everybody in the audience and clearly that's where the American people are and we can tell that by the line of folks who want to get in here uh to to hear this. What is uh uh certainly a huge challenge for us and always has been a challenge for policy makers is that throughout at least throughout the industrial revolution technology moved much quicker than policies. Uh policies always followed behind. Uh that's because things work very slowly here. Uh the Senate is notorious for being very slow about trying to get anything done. Uh and past technologies always accelerated well past our abilities. Now we're in this new world where actually technology is moving faster than older technology at an exponential rate. So it's not even policy. We are so far behind and you're saying now there's technologies every time we build a guardrail we're not sure we can even stop it. It was mentioned uh sir you mentioned the the sandbox. We don't even know if we can control a sandbox because another AI system comes in place. That's absolutely frightening and we don't really know exactly how all of this is happening. I remember a few years back at the beginning of this I was uh at uh uh at a major tech company right after we had uh the program beat the Chinese game of go which you all you all remember and everybody thought that's not possible that it could possibly do it because uh uh it's requires human creativity. It's not just crunching numbers like a chess game. And I remember talking to this developer and I said, "Tell me about it." And he explained it and he said, "Senator, you know what the most interesting thing about that technology?" And I go, "What's that?" We have no idea how it did it. That's absolutely frightening. So, some would argue that maybe we just have to slow this down. Now, I know we heard from uh Senator Scott and and I agree. We have to be very concerned about our competitors out there. There's a concern that whoever gets this first is going to have tremendous power. I think that's very realistic. Uh, and there's those that argue we can't really put any guard rails on because the Chinese are not going to put any guardrails on and they're going to beat us and then that's going to lead to our extinction as well. So, I don't know who to direct this question to, but does it make sense to slow this down? And how would we do that? Also, given the fact that others may not slow it down, so that could put us at a competitive disadvantage. We're we're I guess the term the sophisticated term is between a rock and a hard place uh when it comes to thinking about this. I would love whoever wants to jump in on that. How do as a policy makers how do we think that through also addressed given the fact that even the technology that we develop may not be sufficient to deal with the technology we already have which is really complex. >> Thank you Senator. I'm happy to start uh slow it down can mean many things. It could mean constraining chips. It could be constraining the amount of compute. If by slow it down we mean you know impose uh liabilities so that there are incentives to get the safety part right when my fellow witnesses have been talking about um we although this does feel like a different moment we've been here before we've done uh encountered places where we thought the technology was outstripping our ability to regulate and guess what the companies we did figure out how to regulate the companies continued to lead the world uh and they were safer because of it so it has to be a kind of joint conversation it can't just be a yes we slow down or no we don't slow down. There's got to be a lot of nuance there. >> But but you said that the companies themselves could slow that down. I guess it's a question that released. >> No, no, even with regulation. Sorry, didn't mean to interrupt, but yeah, we we could impose different regulations. We could have regulatory schemes which require the companies to do much more that may look like slowing down, but I think the other way to frame it is we're just asking them to be responsible members of our economy, our society. And you're not arguing that they would do it on their own because of the >> they might >> even with the intense financial uh incentives that we've talked about here that it's not just a threat from China. They want to be the first. You make a lot of money by being first. >> Sure. Yeah. And I think others on this panel probably know more than I do about what's driving their incentives, but there seems to be a lot of fear coming from the companies themselves. So maybe they will slow it down. >> Yeah. Any Yes. >> If I may, um, thank you for this question. I think you framed the issue uh quite eloquently. Unfortunately, that is the situation that we're in today. Um I do think that uh the Communist Party of China is our adversary and that we are in a competition with them. But uh we are also in a situation where if we continue to make our AIS more powerful and if we continue to hand over more responsibilities to our AIS including responsibilities like monitoring each other and doing the safety research and doing the research itself then uh we will have a new adversary that's even more powerful than China and that would be the AIS themselves. So we are in a rock and a hard place sort of situation. I would say that yes, we do need to slow down. We don't need to slow down everything, but in particular, we need to slow down this mad scramble towards recursively self-improving AIs. We need to slow down this scramble towards AIS that are automating the AI research process themselves. Other types of AI like image generating AI or whatever, not talking about that. That's fine. In terms of how we can do this, I don't trust the US I don't trust any of these companies to do it on their own. um for the reasons that we've previously described. Even though they are scared, they are proceeding. And this is because uh well, I can get into the psychology or anthropology of these companies for a long time if you want. I I I work there and I know a lot of these people. But suffice it to say, I don't think we should trust them to do it on their own. I think that the government needs to step in. I think the first step is to uh redirect resources away from this particular uh angle that they're pursuing this AI research towards AIs that can automate the research uh and towards other things such as lowering prices for consumers. And then the second step is to start negotiating with China to make sure that they don't overtake us and do the bad thing themselves. And one thing I'll say, last thing I'll say on that subject is that right now a significant fraction and I would even argue a majority of Chinese AI progress is itself coming from the US through various fast follower effects including distillation and also including other things. And so you know if we actually slowed down our own race towards recursive self-improvement that already would slow down the Chinese race towards recursive self-improvement significantly. >> That's interesting. So yeah, you had a comment. Yes. >> Yeah. I uh thank you senator. I would briefly add that there is a large number of things that we can do today that would be strictly beneficial um for for safety uh like embedded evaluations better monitoring control control and so on. So in general I would warn against treating this as a dichotomy where you can either race uh and to win against China. There are like a lot of things we can do that are outside of this false dichotomy. >> Yeah. And I think that's important and I'm running out of time here, but you know there are others that would are also arguing maybe a quick response from someone is that clearly these risks that you have all eloquently put out are significant but they're and they're going to happen probably quicker than any of us would like but it's still somewhat down the road and people are fearful that we're focusing on these these very big risks that I believe are real in the future but we're not looking at the changes in our society today from AI that's operating today, you know, violations of privacy in ways that we could never fathom before. The job displacement that people are worried about. It's why we have AI populism now. You go out in the streets, people are concerned and and they should be concerned because the AI companies themselves have been telling us, "This is amazing technology. We're going to increase productivity and we're going to need fewer workers." And the workers are saying, "We are the workers. What do you mean? This is not a good thing. Why are we doing it? Why are we spending a fortune on on data centers?" So, I don't want this to be there's no problem here because look at look down the road. This is a big problem. Nothing to worry about here and yet that's impacting people's lives today. >> I totally agree, Senator. I think we need to do both and we need to think about all of these different problems. The world can throw us more more than one problem at the same time. Um, one thing I'll add is that I actually kind of it's not that I disagree with my uh with my friend over there, uh, Marius. Uh, but I do think there are many things we can do that are strictly beneficial and don't uh, slow down our race towards recursive self-improvement at all. But I just want to state that I think that if we do not slow down our race towards recursive self-improvement at all, then we are going to lose control of our AIS. And so that is one of the things that we have to do. There are many other things we need to do, but we need to also do that. >> Thank you, Senator Peters. Senator Ernst, >> thank you everyone for for being here today. I think all of us are are learning a lot through this conversation and we're glad to have this opportunity. Um, we recognize every single day uh that AI is out there or super intelligence, whatever we're going to call it next. Um, I would say a lot of these systems are used to keep Americans safe. There is the flip side. People should be concerned about that. But uh from the committees that I sit on, whether it's Homeland Security, whether it's Armed Services Committee, um we use these different programs and tools to help analysts out at the Pentagon. We see law enforcement using these tools as well. They're sorting through mountains of data to quickly identify those threats that are here and uh abroad. Uh so we we're in a conundrum right now. Um some great technologies, great uses, but then we also have the the fears that come along with that as well. But I I am of this position that we must continue to innovate. We must continue to develop our capabilities. What we don't want to do is fall behind China. China, of course, is not worried about things like Americans constitutional rights or privacy, and they're certainly not worried about um the worst case scenario, which is human lives that can be lost. So, I'm glad that President Trump is taking this issue seriously. We saw the president yesterday. He convened a group of highlevel leaders and they signed an executive order and entered into their joint commitment on frontier responsibilities. And this is really encouraging those companies to implement the controls and the audits that are necessary while also uh really pushing Congress to act. So I do agree we have a role to play in this and we have to figure this out. It's why you're in front of us today. So again, thank you. Um it is concerning to see the rising number of incidents where American super intelligence systems have been abused. And so last December, Senator Hassan and I sent a letter to the National Cyber Director raising concerns about these Chinese state sponsored hackers that successfully directed anthropics AI system to conduct successful cyber attacks against 30 different uh government and private entities. And in this incident, the and we're pulling information that's publicly available out there, but the AI system executed 80 to 90% of the operation without any human involvement and at speeds that are physically impossible for human hackers. Um, personally, that terrifies me. Uh, but again, uh, our government corporations are continuing to increase the integration of this super intelligence. So while they do that, it's vital that we take steps to ensure the protection of our country's security, our infrastructure. Um we've talked about some of the smaller government systems out there, wastewater treatment, water systems, um public utilities and so forth, but we also need to ensure our citizens constitutional rights. Um so one of the primary tools that Congress had developed or created to do this was the cyber security information sharing act of 2015. That's over 10 years ago. And this enabled the government to coordinate and share information with the private sector on cyber threats. Unfortunately, this authority now we are in a cycle where SISA has to be updated yeartoyear. It's reauthorized on a yeartoyear basis. It hasn't been updated to account for the current threat environment which now includes this super intelligence and all of the various complexities. So, Mr. God, we'll start with you. What are the limitations preventing the private sector from coordinating within the industry and with the government to take measures to prevent cyber attacks um and super intelligence threats? Uh, Senator, thanks for your question and and to your point, CISA 2015 is incredibly important uh or something like it >> uh to pass uh because we have to be able to keep that dialogue open between the operators in the field. And by operators, it's large and and I'm obviously focused on these small and medium utilities that don't have the resources to deal with things. Um, and SISA 15 or CISA 2015 allows us to share information about those threats back and forth and keep those communication lanes open and that leads to collective defense. Uh, and without that, you throw AI into the picture and this becomes an a much much more difficult problem. And so companies need to feel comfortable coming forth and saying, "Hey, I was breached and this is what that looked like." >> Um and um we have there are there are other opportunities to do that. You can do it in an offiscated way. You can put systems out there. We have something called neighborhood keeper that allows us to anonymize that data so that we can get a better understanding of what's going on uh overall in our environment. Uh and that there are systems like that that we could put into play as well. Uh and add that to the the law to keep that information flow humming because it has to happen. um we're already behind the power curve and that will only put us further behind the power curve. >> I agree. So that information sharing is extremely important especially for um some of our smaller communities. Uh but again we didn't see super intelligence like we did back in 2015. Uh are there additional measures that could be helpful for Congress to add to SISA u to modernize SISA? What are some of those controls that we should be looking at? >> Question please. Okay. Uh I think that uh CISA should be that coordination agency. they should be the single voice because part of the problem out there uh in the operational community is they don't know what guidance to follow um how how to operate in that environment where they're going to get the information from who to call uh so I think it's it's incumbent upon uh the government to identify a this is the coordination authority here are the authorities that they have Because as you walk across multiple agencies and these small operators have to deal with a number of different agencies in their environments, uh it's very confusing. Uh it it it leads to um even more confusion because they're already underresourced. They don't have enough people uh to man the phones basically uh and deal with these problems. And so having that single voice, having that single coordination agency and having clear guidance is so critical particularly in this environment. >> Yes. And and thank you for that. And I I know my time is expired, but again, I I don't think we need to take an operational pause here. I think we keep moving forward, but we do have to have the measures in place to protect ourselves and our infrastructure um from these types of of attacks. Um, if you think about it, AI has been so helpful in medical research and uh protecting our troops from drone swarms or missile attacks. I mean, all of that is extremely important. So, we need to keep moving forward. Um, if we pause, we're not going to see China pause. So, we have to do the right thing by Congress. We have to do the right thing by the federal government. Make sure that we're studying the problem and putting the right solutions in place. So, thank you all so much for being here. Um, this has been extremely helpful. Thank you. I yield back. >> Thank you, Senator Erns. Senator Ggo, >> thank you. Um, a couple questions. And who just put your hand up, is it possible or how fat are we away from them creating their own language that we as humans would not be able to distinctly understand what they're doing. Marius, >> um, uh, minus 12 months. Um so last year uh we have studied uh the chain of thought of uh one open AI model in collaboration with with uh open AAI and what we found was that the model was already using language that is uh not not English and not perfectly understandable uh by humans and I think we're on a we're currently on a trajectory where uh things like this could be more >> mayors when we get to that that point in time how can we actually detect observe curve block or anything like that because when you're when you're dealing with a whole other language that humans just have never had a full concept of how like how does that work like how do we actually legislate to do anything at that point because we actually will have zero concept to be able to look into these models at that point at that point is that right >> yeah from from a scientific perspective uh it is unclear how to do this uh and we do not have a solution for this yet right >> uh there are different hypothesis of what you could do there is interpretability as a technique but unfortunately doesn't work uh sufficiently well yet. Um there you you could try to train additional models to understand the language that the humans don't understand but obviously that seems like a very brittle solution, >> right? >> You could try to only >> So just on that on that line, what do you think China feels about that too? Uh I'm I'm I'm not an expert in >> Well, I I'm not either, but I guarantee you most countries once they lose control will feel very very most countries that actually used to be in control are going to be very very scared. So the reason I bring this up, this whole China hypothesis, a country that has severely restricted the internet, social media is all of a sudden just going to allow AI rogue agents to go crazy in their very controlled economic environment and social environment. and we have to therefore allow ours to go as crazy as them. Just doesn't make sense, right? This is in this is a the communist Chinese party has been consistently controlling technology the whole time and trying to shape it to shape their society. They're never going to allow this to go as crazy as I just want to make sure that we think about this in that way because once these agents start speaking their own language, they're not going to be able to tell the difference between China and America. And that's a very dangerous situation. We talked about a sandbox earlier. Is there a sandbox that does exist where it says, "Please attack the United States, but do not attack any Chinese interests." Is that a possible sandbox? Could someone explain that to me? Is it a sandbox that can hold? No. That should scare the out of people in ter in terms of other things right now. Frontier models. If there's a Frontier model that goes rogue or is hacked, is there a requirement, a regulatory requirement for that to be uh told or or reported to the federal government? And to whom should that be? Paul, you shook your head. >> Uh not under any requirement I can think of. No. >> Correct. >> Yeah. >> So, some of the stuff that's been constantly come from some of you guys have been whistleblowers. Thank you very much. Like for example, um there was a cloud AI model that was being used by the Houthis to recalibrate and figure out how to recalibrate their missile precision uh and be able to strike our US systems. The only reason it was caught and reported to us because Claude caught it because it's a closed system. They actually saw the accounts using it. But it if it was an open AI or sorry in the any open source systems, there was no requirement for them to actually by law for them to call up the DoD and say like, "Hey, by the way, they're using this to target a couple of your uh destroyers." Correct. >> Okay, that's pretty Paul, jump in if you have something to say. >> Again, we've talked earlier about transparency measures and, you know, putting third party auditors in some of these systems might help. I don't know if that's going to help with who the rebel >> third party I mean third party the third party is the federal government's like if you don't tell us someone's trying to use your tech to kill us you're we're going to hold you responsible >> um third question or fourth question I don't know where I'm is there such a thing as a kill switch besides destroying all the data centers at one time is there such a thing as a kill switch I know there's a lot of talk about that is there an AI skill switch in case things go really bad real fast >> yes Morius >> as as far as I know there are currently no technical measures um that could guarantee something like a kill switch. There are because of the complexity of these AI models, you know, there are sort of many data centers involved and and so on. Uh so currently we cannot guarantee that, >> right? And data centers, they can move and shift. These things are so smart potentially that agents can move from one data center to the other once they're allorked and wired. Yes, Marius. >> Yeah. Is this for example possible that an AI would start on one data center and then uh create a rogue deployment of itself in a different data center and then even if you shut down the first data center >> just keep moving >> the AI could keep moving. >> Yep. Go ahead Paul. >> Senator this may be where you're going. You know with a lot of your questions >> I actually no idea where I'm going. [laughter] You guys really have thrown me in this spinning loop here. I just want to get some answers out. >> You know one thing that I'm often telling my students is disabusing them of the notion that these are completely out of control. There are still human beings and corporations for now. >> For now. Yes. >> And you know the right imposition of laws and regulations can make sure we maintain that status quo. >> And on that, thank you. It's like you cued this up for me in terms of liability. Sometimes when the government doesn't work, tors work, right? And this is like the the balance we try to keep, right? Because tors sometimes are a way better way to actually regulate private industry than us having like individual little regulations, right? But under the the law, Paul or Mr. Um the word intent is really powerful. >> Sure. >> Right. Unless we actually change um and and a friend of mine, Mr. Partovi actually just wrote about this. Unless we actually change the word intent to actually cover AI companies, they may also still be shielded from liability. Right. >> Right. And I want to distinguish between criminal law for which intent does a lot of role, but you started with tort. tort law, it really is did the company maintain a reasonable standard of care, right? >> And you're right, it the genius of that is not only that this one victim gets a remedy, right? >> It's case by case we learn what the standard of care is and companies learn to, you know, race to the top hopefully of responsible behavior >> and so the because you have to they're going to have to input >> from that jury verdict. You say, "Okay, now understand that I have to do these 14 measures. I wasn't >> there reason certain way. Not just because the government says it, because they don't want to get sued by when the house burns down, right? But like right now, do do we have have we created the environment where tors can actually be the the the incentives for them to act correctly in sense of making sure that the language allows there to be a certain level of liability? And number two, on the criminal side, do we need to change the terminology that comes around with intent to make sure these companies are held responsible? I I don't think the tort system alone can bear everything we need to do, but I think it's a great place to start and I'm glad every morning that we have that system instead of not having that system on criminal. I will say one more thing. It's not just the anti-hacking laws, right? >> We're going to see the securities laws. We're going to see the export control laws, maybe even the control. >> The problem is like what happens when with like an AI agent tells another agent to hack. Who is responsible? Is it the original AI developer of that AI agent? Like we're dealing a whole Last question. I swear to God. um RSI. Should we just make it illegal? Is it possible then to make it illegal? >> Yes, Daniel. >> Uh yes, sir. I believe we should make it illegal, and I do think it's possible. >> Thank you, Daniel. I yield back. >> Thank you, Senator Ggo. Uh Senator Moody. >> Thank you, Mr. Chair. And I want to thank you for calling this hearing and about a subject that is on the minds of probably every American right now and certainly has affected the daily lives of people across our country including in my home state of Florida. Um, I think we can all agree no matter where we are, no matter who you are, no matter what industry that you're in, um, no matter how old you are, no matter how you come at this, uh, it's here and it's already become a disruptor, uh, in in every industry and certainly um, in every way we can imagine here in our country. And we're seeing some really real consequences. I mean, you can't pick up a paper that just dated me when I said that. You can't click on a news outlet without seeing a story uh of something has happened that nobody thought was going to happen and it did happen and it's hacking all these sites and it appears that even those that have created these models are still trying to get a handle on this. So, new techn technological capabilities are being used every day and in some ways very helpful. And I was a former federal prosecutor, a former judge, a former attorney general. Uh technology has also affected not only how we're viewing uh our national security, but the safety of our most vulnerable, our children, our seniors. That's very important that we keep that in mind. And we know that it's already being used in commission of crimes even against our children. AI generated deep fakes are being used against kids to bully them in some instances to extort them into harming themselves or other people. And I've got a bill that passed the House earlier on that. Uh we hope we can push that here in the Senate. But we know criminals are targeting seniors in Florida and across the country. And now they have this new powerful weapon to use. They use AI impersonation to deceive people, scam them out of their entire life savings. Uh I can tell you story after story whether it relates to a senior or a kid, but it is already being used in the commission of crimes. We've got bills here in the Senate. I've sponsored them to protect seniors from these scams. But we're hearing today and I think um chair was right to talk about this. The mo models themselves now are increasingly capable and they're using AI agents to and they pose their own risk when sufficient safeguards aren't in place. Uh so the witnesses, thank you for being here. um you really are at the cutting edge and offering solutions and opinions on those of us that may not have been uh familiar with AI or technology as it's being um fast-paced and growing. uh but we need to talk about protective measures uh because if we don't get a handle on this and put in some common sense safety mechanisms I think you are all warning of the dangers that can happen if we don't engage early and we're not proactive protecting kids protecting our seniors protecting American jobs protecting American infrastructure indeed as we've discussed American lives all of that is uh at issue and it should be a bipartisan issue and so I think the chair should be commended in the participation that we're seeing today. Um, this is rare and I think we're all focused on making sure we're doing right by this country and right by our families. And that's where you come in. What I don't want to see happen is what has happened in so many new disruptors in our country where Congress just talks about it and talks about it and stalls and crosses its finger and hopes that everything's just going to pan out or that companies will just do the jobs for us. If you look at any industry or sector that directly impacts pe people's personal safety or our national security or people's privacy, um there are responsibilities that we should not only expect but require of private companies. And I'll start with Mr. uh Co Kataljo. I'm sorry if I mispronounced that. Do you believe that AI companies now and large developers are being transparent and forthcoming when major cyber attacks or these rogue agent incidents occur? >> No. In fact, there are several examples of cases where it seems like the company knew about it and didn't disclose it until some independent third parties noticed it in the wild. >> Mr. home are we all right as we all engage in this conversation over where to place necessary guard rails on AI. Are we seeing private companies take responsibility for the protection of American people from criminals and the safeguarding of our national security? >> Uh that has not been the priority of the frontier companies we're talking about. They're they're in a race to beat uh everyone else to super intelligence and that's taking the priority at least from what I can tell. [clears throat] >> Mr. Painter, what do potential corporate responsibilities look like? Is it compelled information sharing and reporting when major incidents occur? >> Thank you for the question, Senator. Um, I'll say [clears throat] first, you know, my organization, we we don't take policy positions, but I'm happy to say some options. I think I would reiterate what um Daniel was saying before that right now there isn't any requirement that companies disclose things that they're seeing inside of their AI labs in terms of unintended motives that models might pick up during the training and development process and that the situation there could get quite bad and the the outside world wouldn't really know. In this case, we found out because the the incidents involved interacting with the outside world, but it's not guaranteed that that will happen into the future, which is why I think that the first priority should be making sure that the evidence base about these unintended goals is reaching the public. >> Thank you, Mr. Chairman. >> Thank you, Senator Blumenthal. >> Thanks, Senator Holly, and uh thank you for having this hearing. And um as you know uh the AI CEOs have signed a quote unquote morally binding pledge to implement internal controls and hire external auditors. It's a system that would be completely voluntary and totally secret. That is to say, if there were violations by their company as found by the internal auditors, there's no requirement that it be disclosed to the public. And if they wanted to cease cooperation and be morally bound by their own sense of higher responsibility, that would be their choice to break the pledge in effect. So, uh, I consider this regimen to be worse than ineffectual. In effect, it accomplishes nothing, but it seems to give Congress a free pass. In other words, it's taken care of. Don't worry, we have this morally binding pledge. Um, and so I think Congress has a continuing obligation to seek solutions and so do the companies. uh 3 years ago, in fact more than 3 years ago, in May of 2023, Senator Holly and I had a hearing not unlike this one where we had leaders of AI, including Sam Alman, all the major names come before us and they said at the time, we need some regulation, which was not a completely novel idea since there is regulation in pharmaceutical ical drugs and nuclear energy and uh other areas where there's potential danger but also great promise peril as well as promise. So this idea is not novel. And as for China, uh the irony here is China is probably one of the most regulated markets and nations in the world. if they want to crack down on an AI enterprise, they can do it without due process, without any kind of legal obstacles. And I just want to say for the record, we need to stay ahead of China. I am not suggesting in any way that we stop or slow unnecessarily our development of AI. Uh but the idea that we can do it safely with a standard and an agency to enforce that standard I think is a basic requirement here. It happens to be embodied in legislation that Senator Holly and I have proposed the artificial intelligence safety evaluation act. safety evaluation by a government agency with a standard and uh I think that concept is an idea uh whose time has now come. Um I'm interested in this idea of civil liability. Uh Senator Holly wrote a very good uh op-ed for the Washington Post on it. He and I have talked about it. I've discussed it publicly in the past. Um, again, rogue agents, bots are a new concept, but the idea of civil liability for agents is well established in the law. Uh, I was just reading about a uh medical malpractice case where a surgeon at Yale New Haven Hospital in effect botched a surgery. He's an agent of the hospital. The hospital will now pay $15.1 million in damages because its agent and it didn't control the agent. He was doing the surgery on his own, but he was an agent deemed an agent of the hospital. So, uh, Professor M, doesn't this concept make sense? Maybe we need to clarify that a a bot agent is in fact an agent, but the concept is well established in our law, isn't it? >> Yeah. I mean, there there is a danger, Senator, in like ascribing humanity to the agents. And so, I think you're right. We have a number um of legal doctrine that allow us to kind of pierce corporate veils and work down chains of authority and agency. And yes, we have a lot of learning we can just borrow. We don't have to invent something. >> But it doesn't really matter >> for the purpose of agency, >> right? >> Whether that surgeon is a robot and in the future there will be a lot of surgery done by robots. I suppose it still makes me a little nervous because when we begin to walk down the road of saying that they are an agent, I I'm just a little worried that that leads to bestowing rights and things like that. But there's absolutely the hospital should be responsible. >> Absolutely. I agree 100% with you and I'm probably just disagreeing on the on the vehicle we use to get there. >> And the way for the hospital to care about >> Yeah. >> the quality of medical practice is to make it liable. >> And not just the hospital, there is a consultant. Usually there's a developer, there's a deployer, there are many other responsible parties with an opportunity not to botch the surgery. uh in addition to the ultimate robot and and all of them should be >> and if we made the social media platforms liable for suicides or self harm by teenagers when it knew or had reason to know that there was toxic content driven at them by their algorithms instead of giving them a liability shield under section 230 in effect making them responsible for that algorithm. >> Absolutely. >> Yeah. And and the reminder in all of this kind of entire line is human beings at companies are making the decisions that have these downstream consequences. And so it's not enough to just say it's our technical gods who are doing this. There is a human who's accountable. Yeah. >> And what has really made the world safer at the end of the day is attributing responsibility. >> Yeah. Not just moral responsibility, not just reputational harm, but dollars and cents to those companies. And they are no longer struggling nent small uh enterprises. They are multi-billion dollar giants. If I may, I happen to teach law at a Jesuit university and I would still rather go with legal responsibility over moral responsibility even though I understand the impulse. [snorts] >> Um, let me just ask u and I I'm about or I am just a little bit over my time. Um, is there anyone here who thinks that a completely voluntary system with only moral responsibility is going to be enough to deal with the dangers? Anyone here? I see heads shaking. So, I agree. Um, and one last question uh for uh Mr. Painter. Um, are you satisfied that you had complete disclosure or cooperation from OpenAI in the course of your investigation? >> Thank you for the question, Senator. Um, so the agreement that we reached to do the investigation, right, was it was a mutual agreement that we reached with them. When we went into this, we were aware of um, you know, only the the hugging face hack specifically. Um, I think before the incident, we described on our website a kind of set of questions that we think a full alignment misalignment investigation should go into in any of these incidents. Um, and we've since updated that publicly with a even even longer list of questions that we think should be investigated. The agreement that we reached with OpenAI was a subset of those questions. right now, you know, companies have no obligation to work with us on this and we have to kind of make judgment calls in every case about what information we want to prioritize getting to the public. Um, I do think there's an interest in getting some information out fast. I think that's good to prioritize doing something quick to get information out fully, but my hope is that um, you know, in the future we can do really thorough investigations that look in for all of these incidents across multiple companies that look at things like sampling the models and seeing what other models would do in those same situations. Um, let me just say in closing, and I I appreciate uh going o a little bit over my time, but I know that um, Professor M, maybe this is directed to your students more than to you. Um, I was attorney general of the state of Connecticut. I used to say that lawyers are often private attorneys general because they use the law to impose liability when there is wrongdoing. It is beginning to work on social media. Some of the verdicts that we've seen by civil litigants there. Uh it didn't work in the tobacco area. I was an attorney general who have to lead against big tobacco because uh of various issues with liability and frankly the deception of the tobacco companies. Um, car safety, it has worked somewhat. Asbestous, it has worked. Uh, but to your students, they can often vindicate rights and enforce responsibility by using the law creatively. And I think it's our obligation in Congress to provide the tools and the rights that are necessary to protect people. And very often it is a civil litigant acting as a private attorney general or their their lawyers who help to protect a whole class of people who may be vulnerable. Thank you, Mr. Chairman. >> Thank you, Senator Blumenthal. I just want to ask a few more questions and I I think I detect the beginnings of a of a dare I say it, a consensus here on this panel which I think is very significant and I just want to pursue this a little bit more. Why shouldn't we start by imposing clear liability on the companies for the actions that their agents quote unquote their their AIs as you've been calling them Mr. that their AIS take or uh that would be on the developers impose similarly impose liability on the users of AI when they use AI in in a reckless fashion. And so in other words, why shouldn't we just say it's it's a it's a standard doctrine of American law if I can put it in layman's terms. If you break it, you pay for it. If you cause damage, you've got to make it right. If you cause somebody harm, you've got to make them whole. Right now, as you said earlier, Professor M, under our law, because of the unique position of these AI agents, it's difficult to know exactly how the doctrine would would deal with them. They may be able to they the companies may be able to escape liability. I'd suggest that maybe we start with some legislation, federal legislation that would say, as a matter of federal law, and we can use existing statutes to do it. I propose using the Computer Fraud and Abuse Act. It's already on the books. we can just update it to say that for developers, that's the companies, if you develop these agents and train them in a reckless fashion and they go on to hack or crash or destroy stuff, you're liable for users of the AI. If you've got these users like the the guys who use the anthropic to hack into open AI, if you use an AI model or agent in a reckless fashion, pick your liability standard, then you should be responsible. Why don't why don't we just do that? Why don't we assign some liability? And here here's my thinking on this is that I don't have any confidence zero less than zero in Congress's ability to keep up with the technology, let alone anticipate it. I mean, there's just no way we're not going to be able to say, well, you know, you guys have made great suggestions about here's the sort of reporting regimens we need today. And I I think that's all great. It'll [snorts] probably be different tomorrow. I mean, you were saying a second ago, Dr. Haban, that pretty soon the AI agents will be able to speak to one another. We won't be able to track it. But if the AI companies who were developing and training these agents knew that they were going to be liable for the bad stuff their agents did, I just am willing to predict go out on a limb and just say that I think maybe they pay a little bit more attention to the development and training of these models. So let's just start with that. Professor, am I off track here? I mean, isn't this a good place to begin what we've been discussing here? Let's impose both civil and criminal liability, but let's make sure that we can actually open the courthouse doors so people can say, "Listen, uh, open AI, I if your agents hack hugging face or crash a hospital system and we can show that you trained them recklessly or you developed them recklessly, you're going to be liable." >> Absolutely. And once again, I would start with making sure we do not preempt state law. That's that's step one. And that's because state law is tort law is mostly state law and it's already on the books. >> Absolutely. And and you know state UDAP laws, unfair and deceptive acts and practices. That's a useful tool in the way you're describing. Um I like the idea of both individual victims and state attorneys general having a role in playing this. Uh the CFA is not a well-loved statute. I maybe offline I could talk to your staff about other things we might fix. It's not going to be the most popular vehicle for this, but absolutely I'm with you that, you know, finding the vehicle to make sure developers and deployers and it's key that we have both of them are held to account for the harms that happen from their uses 100%. >> Yeah. I mean, let's just give a practical example and Mr. Painter, you tell me if I'm wrong. My understanding is is that in the hugging face hack from the open AI agents or by the open AI agents, open AI actually had safety mechanisms off. Is that right, >> Senator? That's correct. >> All right. So, let's just say that one more time. >> They had they had monitoring systems off. I forget that. Yeah, >> monitoring systems were off at Open AI during this attack. Now, if OpenAI knew for darn sure and certain that they were going to be liable to the tune of who knows how much that a jury might award, don't you think maybe they would be a little more careful about monitoring what their agents were doing? Or Mr. Cookotelli, you've been talking about the amount of attention and money that's devoted towards moving towards what did you call it? Uh recursive self-improvement versus making the models more efficient, making them cheaper, making them more user friendly. Um I don't have any com I know nothing about the AI business. I don't want to tell them how to run their business. I don't know. I'm the wrong person to ask. But if we told them, listen, if your AIs do do bad stuff and harm people and you've been reckless in it, you're going to be responsible. Don't you think that that would reshape their incentives a little bit? Thank you, sir. Yes, I completely agree. Although I I must say I don't think it would go far enough. Um I do think that we should hold them liable for the damages that are caused. But um uh but I think if that's all we do, then they're going to continue to take reckless gambles and eventually they'll be taking a gamble that's uh big enough that we will suffer catastrophe. >> Um Dr. Haban, let me give you a chance to weigh on this because you said something the other day that I think is very interesting and we've got I think a board of it. You said it's so economically valuable for these companies to build these AI systems. They're going to limit test how much misalignment you can get away with. In other words, naturally, this is just the market forces, right? This the way the market works is competition. They they are their incentive is to innovate, innovate, innovate, innovate, and they're going to blow past what we've been calling safety limits and safety concerns. They're going to blow past that unless we somehow make them pay for the harms that they cause. Isn't that fair to say? Yeah. So, um, what I meant with the statement is, uh, sort of a general comment about the the the situation on the ecosystem that we have right now where capabilities are moving drastically and increasing drastically while safety uh, alignment and techniques cannot keep up. Um, and then there is intense pressure to release models regularly which can lead to premature deployments and incidents like this. And so I'm I'm worried about the voluntary nature of the the current ecosystem and and how it plays into the incentives of the overall companies. >> Well, I just think for my part, I think it's time to change the game. And I think it's we we don't have to to invent an entirely new system. We don't have to reinvent the wheel. American law has worked beautifully when it comes to other products for literally centuries now. And as you pointed out, Professor, it actually goes back to the common law that predates the founding of this country. I mean, it's it's worked pretty well for our civilization. And the way it works is because the law is itself a form of a marketplace, which is when individuals get hurt, if you give them the right to go in and say, "Hey, I'm going to hold the person who hurt me accountable." Guess what? People change their behavior. And right now, I'm worried, my biggest concern with AI among many, many is that right now the incentives are all misaligned. These these companies have every incentive to race race against each other. They don't have any incentive to think about, gee, what happens if I crash this hospital? Gee, what happens if I crash the banking system? They just think that's somebody else's problem. And we know they think that because when they talk about the doomsday machine they're building, the next words out of their mouths are, "And we would really like more exceptions from the government rules. We don't want to be held accountable under antitrust. We want to be able to collude." That's exactly the wrong thing to do. We should go in the opposite direction. Hold them accountable and let people vindicate their rights. That's my view. All right, Senator Kim, your turn. Thank you, chairman. Uh, Mr. Hobin, I'd like to start with you. You know, we talked about the recursive self-improvement and and speed and and the lack of uh human engagement in that process. You know, I'd largely agree with you and I think a lot of people do when it comes to the need for embedded evaluators for instance in monitoring. I guess I just want to get a clearer sense of like what is possible when it comes to embedded eval like if we embed evaluators to oversee an you know recursive self-improvement system what is it that they're going to be able to see what is it that they're going to be able to monitor uh and understand given the speed with it with which that's happening >> yeah so um what I think we need for embedded evaluations is uh employee equivalent access so access to all training data all development pipelines. Uh the ability to speak with internal staff freely and without repercussion to understand uh what the what the situations look like and learn about the systems. Um the ability to check uh internal deployment, so how the models are being used inside of the the company and and look at the logs and so on. And then all of this um all of the results have to be public um such that the general world uh has a better window into what is happening and can uh take more adequate responses. Um, and I understand that there are uh confidentially uh concerns and I think it should be possible for AI companies to redact confidential information from the report but that should be under metatransparency clauses which means that uh in the report it will be stated uh that a reduction took place um and the general process should be known. >> Yeah, I don't disagree with a lot of what you said there. makes a lot of sense to me. But I I I still think that it gets to this question of just how much are we able how much are humans able to comprehend of of what is actually happening when we reach that full level of of RSI and that's something I'd like to think through. I'd like to just stay with you. We we talked about open wave models, open source models. Um who is doing the evaluating and monitoring there? >> Um that uh is is a great question, Senator. Um right now um most evaluators are not working or I'm not aware of any evaluators working with um uh the open source companies and this is largely a capacity constraint. Um uh yeah in general models should be evaluated and they should be tested by third parties but due to the small size of the field and the the like importance of rigorous testing it is currently not possible to do uh every to cover everything. No, I I I think that's a really important distinction to make here. As we were talking, my my colleague Senator Blumenthal was just talking about just does anyone think that it's enough for these companies to have to handle this voluntarily? And I think, you know, largely a lot of people say no. But it also just kind of shows again, you know, when there's no structure to this, you know, who's going to be looking at these other models, who's going to be engaged uh in in this and and doing it right. So, that's something I I want us to make sure that we're following up on. I'd like to just kind of spend my my remainder of my time just talking through kind of what the chairman was saying. I think there is area, you know, we're finding areas of of consensus or common ground amongst uh you know, by the witnesses here, but also you know, I was listening to our colleagues here and I do think that there are some avenues here that we need to make sure we're engaged on. First and foremost, we are the Homeland Security Committee, and I think you're hearing from every single person on this committee that we want to secure our country, that we're worried about that. I'll be honest with you, we should be worried about that even before the advent of AI, right? We already knew that China was able to get into so many of our systems even before using AI tools. We have a lot of other concerns that are out there. So, you know, Mr. Godad, I just wanted to kind of check in with you here. you know, you you raised, you know, CISA, for instance, can play a big role here. What is it that you want us to be able to move forward here as a as a committee uh in terms of authorizations or fundings or other things like what is it going to take for us to have at the federal level the type of action that is going to actually keep us safe? >> Thanks for the question, Senator Kim. Um, I think clear authorities uh is is critical. Uh I think identifying um the coordinator for these type of activities again because many of the owners and operators out in the field don't know who to go to or who to call for these types of situations. Uh and then the other piece here uh that I that I mentioned in my opening was having some type of unified national uh OT ICS incident response plan. um because we see the problem before us and I think for many years we thought or a lot of Americans thought that's an over there problem. We saw the attacks in 2015 2016 time frame on the electric system uh that took down I think something near 250,000 customers uh for multiple days. We saw in 2024 with the Frosty Goop uh attack uh in Lviv uh 100,000 people lose power in the dead of winter. Uh these are real attacks. They happen all the time. Um and we need to internalize that and realize that it can happen in our country. And now we've we've been starkly made aware that >> it can happen in our country, right? Particularly with the July attacks. >> You add AI to the mix and we've got this real interesting situation. So, the need for something like uh a nationalwide response plan so that it's clear uh for our owners and operators out in the field to know who they go to for guidance, for help, how we would respond. And we've done a good job. I I I think uh the 91st Brigade um in the Virginia Army National Guard has done a great pilot with a large utility there and showed how uh an entity could respond in this activity. And if you had someone like CISA coordinating these various responders uh in some type of deliberate plan before it actually happens would be incredibly helpful. >> Yeah. And this is something we encountered. I mean when we were dealing with again you know the release of Mythos you I was engaged with a number of you know banks and financial institutions seeing uh you know them getting access early to be able to patch up and that was successful to them but I'll be honest I had a lot of other companies and organizations coming well what about us you know who's who's controlling who gets access to this early and there and certainly when it came to critical infrastructure very much feeling like they were left out of that Mr. Dr. Hob, I just wanted to end with you here. Another area of question was about the ability to secure our innovation lead. A lot of concerns about China. I think we all have those concerns, but I guess I just wanted to ask you in my opening I said that I believe that there's a way in which we can ensure and and grow our innovation lead while still creating space for us to uh engage in this effort on safety and security. Is that something you believe in as well? Do you feel like we can create that space? Yeah. So I think uh we we have to be careful about uh false dichotoies here. Um and I think it is uh important that we keep uh both areas uh of concern uh sort of in check. Um and uh I I think it is uh it is it is possible to further innovation and and and help consumers and so on while keeping track of rogue AI and China. >> Yeah. I'll just end on this. Uh right now before our Congress this year, we have a number of pieces of legislation that are in the NDAA and other means which we can use to be able to secure our innovation lead. You actions like like what Senator Ricketts and I are doing in a bipartisan way with the match act that would control uh semiconductor chip manufacturing equipment or the legislation I'm doing, the Blade Act, which is about distillation and something that very much has a big say in terms of what China is able to be able to do. So I just urge this Senate, you know, yes, we should engage on this issue about safety and security. We should not allow the questions about our innovation lead to stop us from engaging in this. Uh we can, you know, yes, we can have our concerns about China, but that is not an excuse not to act when it comes to the security of the American people. And with that, I >> uh I would just like to emphasize the point that Senator Kim made. Safety and innovation are not contradictory. They're not exclusive. In fact, they go together because without safety, innovation is going to be stymied and stopped. So, uh I think this is a false dichotomy that maybe some in the industry have raised. Uh I think there is the possibility for some international agreement. But certainly in terms of what we do in the United States of America, safety and innovation ought to be complimentary goals. And I would just second as well what uh the chairman said. Uh we ought to work on legislation that imposes or clarifies that there is civil and criminal liability. I hope we can use the next 30 plus days to work on legislation that we would introduce. Uh it's not a substitute for some governmental oversight. I don't want to be misleading here, but it is an important element of the protection that people deserve. There's still a need for some standard enforced by a government agency, but civil liability is often an important element of protecting people uh as well as criminal liability. But of course, uh no individual plaintiff is going to enforce criminal liability. That again has to be the responsibility of some public agency. And uh I think professor M you raised u an important point about preeemption. Uh states continue to be the laboratory of democracy. We maybe have AI labs but states are the laboratories of legal process pro process and progress. And I think we ought to be mindful of what they have to contribute as well. speaking as a former state attorney general. Um and so um thank you all for being here today and thank you for the uh folks in the audience who are interested in this topic because we need the ideas and suggestions and criticisms of uh an active and engaged community. And thank you to the panel. >> Well, let me just thank you Senator Blumenthal. Let me just end with this. Professor M, you touched on this about how your students feel. I think there's so many Americans maybe sitting here in this audience today who feel powerless and realize that these companies have tremendous power. And in contrast to that, we feel as if we just can't do anything. But that feeling is itself a myth. We can still act. We can still protect our rights. We can still vindicate that fundamental American value, that fund fundamental American principle of personal responsibility. And I think that's really what we're talking about a lot here today. I mean, so much of of our economy, uh, to say nothing of our entire legal system rests on the basic principle of personal responsibility. And it's time these companies assumed some personal responsibility. It's time that we reaffirm that together. We've got to make some choices now together as Americans all together for this technology in order to make sure that the technology works for us and not the other way around. And those are some very big choices to make. They have very serious moral ramifications and it's time that we made them as a society. But I I will go so far as to say I sort of suspect I think most Americans actually agree. I don't think there's a lot of disagreement on this issue. I think the American people are saying protect our rights. Give us a voice. Give power back to us. And we do that by imposing responsibility on these companies. We should start with that today. Thank you again for all of the witnesses for being here. Before we close, I'd like to note the subcommittees received statements from the Missouri Hospitals Association, the Missouri Bankers Association, the American Hospitals Association, the AI Policy Network, and the America First Policy Network without objection. They'll be made part of the hearing record. The record for this hearing will remain open for 15 days until Thursday, October 15th at 5:00 p.m. for the submission of statements and questions for the record. And with that, the hearing is adjourned.