Flipping AI’s kill switch.

Dave Bittner: Hello, everyone, and welcome to "Caveat," N2K CyberWire's privacy, surveillance, law, and policy podcast. I'm Dave Bittner, and joining me is my co-host, Ben Yelin, from the University of Maryland Center for Cyber Health and Hazard Strategies. Hey there, Ben.

Ben Yelin: Hello, Dave.

Dave Bittner: On today's show, Ben discusses Anthropic's win in a court case against the Pentagon. I've got the story of potential AI kill switch legislation. While this show covers legal topics and Ben is a lawyer, the views expressed do not constitute legal advice. For official legal advice on any of the topics we cover, please contact your attorney. [ Music ] All right, Ben. Let's jump into some stories here. Why don't you start things off for us?

Ben Yelin: So this is a big one. This is about a case that was decided in the Federal District Court in Northern California, and it was Anthropic challenging the Pentagon policy that, basically, blacklisted them. If you'll recall, the Pentagon and leaders from Anthropic were working together as part of just normal federal contracting, and the Pentagon wanted Anthropic to give the Pentagon certain authorities that Anthropic was not prepared to give them, so the use of autonomous weapons, mass surveillance. Obviously, the Pentagon has the right in that circumstance to terminate that contract. If you're not going to do the work we want you to do, then we're not going to use you. And sure enough, OpenAI came crawling to the Pentagon and assumed that work without --

Dave Bittner: Probably galloping in.

Ben Yelin: Exactly. It was quite lucrative.

Dave Bittner: Sure.

Ben Yelin: So they came in, and they swooped up that project, and they got all the money.

Dave Bittner: Okay.

Ben Yelin: But the legal controversy was about this blacklisting. The administration blacklisted Anthropic and basically said that it's a danger. The use of Anthropic and Claude AI is a danger to the federal government and to its systems. It's a national security risk, and therefore, no government agencies are allowed to do business, and that applies to both military matters -- so military contracts -- and just general contracts across the federal government. So it bars, you know, the Department of Treasury for using Claude AI to do coding to figure out something related to tax policy.

Dave Bittner: And that's a big deal.

Ben Yelin: It's a huge deal. You know, you might think, well, they can just go to the private sector and win a sufficient amount of business that they can forego this federal contract, but that's just not really how it works. I mean, the federal government is a big purchaser when it comes to these things. So there are actually several lawsuits that followed, but one of them was filed in this federal district court in Northern California. A judge by the name of Rita Lin, who was a Biden appointee, held in a four-page memo that this blacklist was illegal and baseless; basically, that all of the national security justifications were flawed and not supported by evidence, and that this was an overbroad administration policy. Basically, she said the Pentagon was free to pick and choose which contractors it wants to work with, but for the administration to take a sweeping action using this statute that's about national security, that's about threats to our national security, to ban the use of Anthropic in all federal government agencies and with all federal government contracts, was without any basis. And she points to some of the evidence, the contemporaneous evidence. This is actually funny to me, because this is how the Trump administration always gets in trouble, is they kind of narrate what they're doing out loud in interviews.

Dave Bittner: Right, right.

Ben Yelin: You know, you have to, like, in the legal world, sometimes you have to put on a show for the court system so that you're demonstrating that the reason you're doing something is the legally justifiable reason.

Dave Bittner: So it's at least plausible?

Ben Yelin: Exactly.

Dave Bittner: Yeah.

Ben Yelin: And, you know, judges can speculate privately about what your true motivations are, but you don't want to actually go out and say what you're trying to do. That's not what the administration did. They were going to media sources, on TV, to bloggers, and basically, they were saying that this entire policy was an effort to make an example out of Anthropic for its arrogance in criticizing the government, and that's just unacceptable. I mean, basically what the judge is saying is they're being unduly punished for their free speech, and that's not constitutionally permissible, and if you want to ban them from any government agency or contractor, you're going to have to find a better reason. There is still a separate case in the District of Columbia on this, so this isn't the final word, but it's a huge victory for Anthropic. Amazingly, they're still on the Pentagon servers, just because it's taken this long. We're now talking six months for the Pentagon to, kind of, unwind its connection to Anthropic's AI tools.

Dave Bittner: Well, isn't it sort of like the, I don't know, the behind-the-scenes understanding is that, you know, places like NSA and CISA are still using these tools because they're valuable?

Ben Yelin: They're very valuable.

Dave Bittner: Yeah.

Ben Yelin: I mean, there's a reason, especially when it comes to coding. I mean, I think a lot of different LLMs have their own positive use cases. You know, I'm certainly not the best person to give you insight about this, compared to some of your other guests, I'm sure, but I think coding is the main function where Claude has a competitive advantage, and the government benefits from doing a lot of vibe coding. I mean, I think this is something that the Trump administration has really pioneered over the past couple of years, is using the capabilities of AI to do labor-intensive work within federal agencies that allows them to "Doge" people, which is an informal way of saying fire a lot of federal employees.

Dave Bittner: Right, right, "gain efficiencies" --

Ben Yelin: Gain efficiencies.

Dave Bittner: -- "through automation."

Ben Yelin: That's the corporate speak.

Dave Bittner: Right.

Ben Yelin: You know, there had been, kind of, a bit of a softening in the relationship since the big blowup in February. So Anthropic announced that it developed Mythos, which has powerful new hacking capabilities -- some of which are kind of terrifying -- but basically, the government didn't want to have an enemy that had created this very powerful computer hacking tool, so they reached out to Anthropic's Chief Executive, Dario Amodei. There was a whole White House visit. You know, they kind of met and shook hands, but, you know, there's still this antagonistic relationship that spans from the government's actions in rejecting Anthropic in February.

Dave Bittner: So you mentioned there's another case in D.C. Why two cases, and what potential outcome could that have on all this?

Ben Yelin: A couple of different things. There can be different cases in different jurisdictions because there can be different plaintiffs. You have a lot of different options when you're shopping for venues when it comes to this case. Because Anthropic is domiciled as a corporation in California -- that's where they're headquartered -- you can sue in the Northern District of California. That's kind of a well-known venue for these cases, and there are judges who are more amenable to big-tech companies. So if you think that you're not going to get the, kind of, broad ruling that you might get from the Federal District Court in Northern California, you might try to diversify and try to bring a case in the District of Columbia. That's where, obviously, the federal government is domiciled, so the District of Columbia has jurisdiction. Sometimes you just bring as many cases as possible and try and get a favorable ruling in as many jurisdictions as possible. If there's no disagreement among district courts and circuit courts, then you can get a favorable result without the Supreme Court weighing in. There's been, I believe, discovery in the D.C. case, but we haven't gone through with the full legal proceeding, and so we don't have a final decision from D.C. Really, it just depends on whether the judge sees the issue the same way.

Dave Bittner: So in terms of how this would likely play out, if you're the Pentagon, do you wait to see how the D.C. court weighs in before trying to appeal the California decision?

Ben Yelin: I would file the appeal now. You know, it's timely. Obviously, we're not giving them actual legal advice here.

Dave Bittner: Yeah.

Ben Yelin: Then just on a separate track, wait to see what happens with the D.C. case, but you have an unfavorable outcome if you're the government. I think it depends on if they want to maintain this antagonistic relationship with Anthropic. It's possible that the thaw in their relationship has, kind of, mended the conflict that existed in February, and there might be a detente where the government doesn't appeal the case. There's an understanding that Anthropic will not be the Pentagon's preferred AI contractor, but there's not going to be this blanket ban within the federal government. I think that's a very distinct possibility, but, you know, there's also a tendency in this administration to maximize conflict. So if they do choose to appeal this, that will go to the Ninth Circuit Court of Appeals, and that's, you know, a pretty liberal court of appeals. It depends on kind of what judges you draw for how favorable a decision you're going to get, but I think there's a decent chance that this would get upheld in the Ninth Circuit. Then you see what happens in D.C., and like I said, if there's disagreement within jurisdictions -- which there frequently is -- then that could be something that the Supreme Court weighs in on.

Dave Bittner: Right.

Ben Yelin: You know, it would be confusing if there were conflicting decisions, because then it's like, well, where does the D.C. case apply? Where does the Northern California case apply? It gets very confusing. You know, you get those conflicts of laws. We're not there yet, so I kind of think we'll wait to see what the disposition is, but I would guess, if I had to, that they're probably going to appeal and do so shortly.

Dave Bittner: I can imagine, and I'm totally speculating here, but I can imagine the situation where, you know, initially when Anthropic said, no, we don't want to allow the use of our product for these things. And the Pentagon pushed back and said, you know, yes or else. And then, when the Pentagon put out this ban, I can imagine -- I can imagine the Pentagon putting out that ban without first checking with folks like the NSA.

Ben Yelin: I think that's accurate.

Dave Bittner: Then the NSA coming and saying, wait, what, hold on. We've already got a ton of time, effort, and money invested in these tools. You can't just pull the rug out from under us, and that's a national security issue.

Ben Yelin: I mean, you'd think they would have had that conversation, but part of it seemed like it was really based on, like, a visceral anger on the part of the Department of War, as I guess we're calling it now.

Dave Bittner: You're calling it that.

Ben Yelin: You know, Congress, as part of this bill that's being considered, the Defense Authorization Bill, is going to change the statute to name it to the Department of War.

Dave Bittner: Oh, is that right? I didn't know that.

Ben Yelin: We'll see if that goes through the legislative process.

Dave Bittner: Right.

Ben Yelin: This case is called Anthropic v. U.S. Department of War.

Dave Bittner: Okay.

Ben Yelin: So, you know, they're already using it in our court system, but I digress. [laughter] Yeah, I mean, I would have been surprised they wouldn't consult with one another, because as you said, I'm sure the NSA is doing very important work using these tools.

Dave Bittner: Right.

Ben Yelin: But I also have the impression that brass at the Pentagon and senior officials of the administration were willing to drown the baby with the bathwater, so to speak, because they were so angry at Anthropic's very public refusal to take on these responsibilities. The administration might have thought, incorrectly, that one of these other companies could come in and produce the exact same output.

Dave Bittner: Right, or that Anthropic would just fold and say, okay, you know, this business is worth too much to us.

Ben Yelin: Yeah, I mean, I think that's another common tack we see with this administration is -- I never am entirely sure if they're bluffing, and they just think they have more leverage than they actually do, and they just expect the other side to fold. It's, like, you look at what's happening in Iran. I mean, that's kind of what they thought would happen.

Dave Bittner: Right.

Ben Yelin: You look at what's happening with Canada and these trade negotiations, I think they expected that they'd just be belligerent, and Canada's not a powerful country, and so they'll just do whatever we want and fold.

Dave Bittner: Right, right, so maybe the success that they saw themselves having in Venezuela, how quickly that went down. They thought, how hard could it be in all these other things, and it just hasn't worked out the way they'd hoped.

Ben Yelin: We are the United States of America. We're powerful. You're not. You know, I think that might have been their expectation, but they're discovering in a lot of different realms that that's not entirely how it works. I think Anthropic was wise to try and use our court system to fight back against this, and they had very legitimate gripes here. Anthropic's own statements about AI safety count as protected speech, even as part of a government contracting dispute, and you can't discriminate against a company solely based on their speech. That type of retaliation is unconstitutional and is illegal, so they knew that they had this good cause of action, and they pursued it.

Dave Bittner: I think also, you know, looking from the judge's point of view, you know, the Pentagon was very eager to do business with Anthropic, right up until the moment Anthropic put some limitations on what they could do. And those limitations, you can imagine they wouldn't affect Anthropic's business with many of the other government agencies in D.C., right? You know, the -- I don't know. The IRS doesn't need access to, you know, automated drone things. You know, the military applications are different from many things the other departments would need, so to say that excluding those things should also exclude Anthropic's use at other agencies that would not need those things seems a stretch to me, and it sounds like the judge thought so too.

Ben Yelin: And the funny thing is, like, if they had gone about this in the correct way, like, if they had followed all the procedures and done an investigation -- even if it was a bogus investigation but, like, produced some findings to show that Anthropic was a national security risk --

Dave Bittner: Yeah.

Ben Yelin: -- and if they didn't have contemporaneous comments from Pete Hegseth and others that were, like, we're retaliating against you because of your speech --

Dave Bittner: Right.

Ben Yelin: -- they probably could have gotten away with it. They just broke the fourth wall. I mean --

Dave Bittner: "If it weren't for those meddling kids."

Ben Yelin: Exactly. I think they all have to be more careful. If they want to win these court cases, like, they have to be more careful about what they're saying publicly because judges can easily impute a bad motive to government agencies if those bad motives are being made extremely public.

Dave Bittner: Yeah, it's a ready, fire, aim in a lot of cases here, working against their self-interest, I guess.

Ben Yelin: Yeah, you know, and maybe it was just like a temporary fit of rage because they were told no by this powerful company and they're not used to it, and they did have leverage through their procurement authority, but I think a lot of it had to do with the way they went about, which was not doing the legwork and developing a record on the national security threats presented and depriving Anthropic of due process when they hastily went through with this decision before giving Anthropic the chance to even respond to the allegations. It's something where, perhaps, if the government had been more meticulous about this and could just shut their mouths for a little while, they might not be in the position that they're in now.

Dave Bittner: Be a little more disciplined? Right, right. All right. Well, we will have a link to the story from the Washington Post about this. I'll tell you what, Ben, let's take a quick break here. We'll be right back after these messages. [ Music ] And we are back. My story this week is actually -- it's an op-ed in CyberScoop, and this is from Luke O'Grady, who is a senior analyst at Venable and also at the Center for Cybersecurity Policy and Law, where he researches and advises on cybersecurity policy, regulation, and governance.

Ben Yelin: Kind of sounds like a competitor of ours.

Dave Bittner: [laughter] That's right.

Ben Yelin: We've got to meet up on the playground somewhere, Mr. O'Grady.

Dave Bittner: Yeah, so this op-ed is about some legislation that a couple of representatives have introduced, Representatives Ted Lieu and Nathaniel Moran.

Ben Yelin: Can I just stop you and say, very quickly, when you talk about bipartisanship, you know, sometimes these bipartisan bills are sponsored by, like, the most centrist Republican and centrist Democrat. These guys are partisan warriors in their respective parties.

Dave Bittner: [laughter] Right.

Ben Yelin: And the fact that they're teaming up on this, I think, shows how broad the support is --

Dave Bittner: Yeah.

Ben Yelin: -- but I'll leave it to you to describe the issue.

Dave Bittner: So they've introduced the AI Kill Switch Act, which would give CISA the power to require these frontier AI companies to build in mechanisms capable of throttling, suspending, or shutting down their systems. What Luke O'Grady is saying is that mandatory kill switches amount to deliberately engineered vulnerabilities and create their own security risks in the attempt to address AI safety concerns. He actually compares this to the Clipper chip, which was probably before your time, wasn't it?

Ben Yelin: Yeah, I was going to say it was before my time, but --

Dave Bittner: Well, I remember the Clipper chip.

Ben Yelin: Yeah, describe the Clipper chip for us.

Dave Bittner: So back in the '90s, NSA proposed the Clipper chip, which was, basically, a government backdoor. It would enable the government to have access to encrypted communications through a mechanism in chips that would be built into devices. Of course, researchers found out, as is often the case, that it could be exploited and it wasn't going to do what it was intended to do, and, in fact, would introduce vulnerabilities. Now, O'Grady says he acknowledges the analogy isn't exact. The Clipper chip threatened confidentiality, but AI kill switches would threaten system availability and be a mechanism for taking AI services offline. Before we dig any further into this, what's your initial take on this, Ben?

Ben Yelin: I mean, it's a conversation we've been having for a long time, even outside the context of AI, where if you build the kill switch and you think that only the good guys will ever have access to the kill switch, you are not thinking broadly enough, and you are not appreciating every single risk factor involved. I mean, it's the same thing we talked about with the mandatory backdoors for encrypted devices. If there's a backdoor, yes, the government might be able to get in, but nation-states who are sworn enemies of the United States might also be able to get in. So by creating this tool, even though it's intended to empower CISA here, it creates these types of new vulnerabilities. You can certainly see how this type of kill switch could be weaponized. For starters, I mean, this actually ties into the story we just did. Like, what if Anthropic did what they did with the Pentagon, refused to participate in mass surveillance and autonomous weapons, and out of retaliation, CISA pressed the kill switch? That would have incredible downstream effects on the economy, but that's, you know, just for starters. Like, it could impact our critical infrastructure. It could impact intelligence and defense capabilities, so it's a very powerful tool, even if we are just assuming that only CISA or other government agencies are going to be the ones to employ it. Then the broader argument here is the tool itself becomes a vulnerability, and I think that's compelling. I can certainly understand the argument against the kill switch. I think the idea is coming from a very valid perspective, which is that if these models get so far out of hand that we can't control them, I think we're seeing a lot of discussion around this after the Hugging Face incident. There has to be some type of mechanism for humans to put a stop to these models.

Dave Bittner: Well, and yeah, I would imagine that all of the frontier model companies have kill switches built in. This is just a matter of who gets to press the button.

Ben Yelin: Right. If it stays within the company, I think it's a much safer proposition. I mean, this is the same thing for backdoors into encrypted devices or encrypted networks, where the company, for its own business purposes, might have that capability, but to create it as a tool that you're supposed to give to the government just creates all these downstream additional vulnerabilities. Beyond what a rogue government might do, if you have CISA that's not motivated by cybersecurity, but that's motivated by revenge, they could wield this tool in a way that would hurt all of us. Then the fact that if this tool isn't properly secured, even AI agents themselves are becoming smart enough where couldn't you foresee an incident where a bunch of agents conspired to bring down a competitor through the use of this kill switch? All the things that agents, when they put their minds together, can hack into it's hard for me to believe that, eventually, they couldn't figure out how to breach this kill switch, and they could use it to bring down a competitor.

Dave Bittner: Or disable it, so as to not be disabled themselves, you know?

Ben Yelin: Yeah, I mean, they're as ruthless as we are because they're trained on --

Dave Bittner: [laughter] Right, right.

Ben Yelin: -- our jealousy and our will to power.

Dave Bittner: I have said it. I've said it before. I'll say it again. My perception is that these systems reflect who we actually are, not who we aspire to be, and --

Ben Yelin: Yeah, it's kind of like the base values of humanity, like survival, gaining any sort of competitive advantage. We are looking in a mirror, and that's kind of what we've imputed on our agents, which is sad in a lot of ways because I think a lot of the promise of AI was, well, the agents are going to have this type of super-intelligence that we don't have. And if you talk to some of the founder class in Silicon Valley, I mean, a lot of them were talking about how agents or AI would have more humanity than humans, just because the smart people could properly train our AI overlords with good values, and they wouldn't have the types of things that have befallen human civilizations. But now, we're seeing that that's just not really the case, right?

Dave Bittner: Right. [laughter]

Ben Yelin: They have, like --

Dave Bittner: Once again.

Ben Yelin: They will try to lie. They will cheat. They will cover their tracks. They're just like every other petty criminal out there, and we have to contend with that reality.

Dave Bittner: Yeah, I'm particularly thoughtful about this notion of the models being throttled. You know, like, if suddenly you find your LLM is stupid, or it's making bad decisions, or just imagine that capability of being able to dial back which model it uses, but also imagine our adversaries being able to take advantage of that capability. If, in some theater of war, we're using these AI systems to make decisions and one of our adversaries is able to just throttle back our LLM's decision-making process, that could be a threat itself.

Ben Yelin: Oh, totally. It's, like, which weapon should I use? How about the paper straw?

Dave Bittner: Right, it's a slingshot.

Ben Yelin: Yeah. We are the smartest LLM, and you've trusted us before and, you know, don't concern yourself with the prompt injections that caused us to give you a terrible answer.

Dave Bittner: Yeah.

Ben Yelin: Yeah, I mean, and I think that motivates a lot of the fear here. Even if it's not about kill switches, it's just about creating the power to intervene and manipulate the responses to these AI tools, and there are just better ways to protect AI security. You know, in this article, Mr. O'Grady advocates AI security standards that are developed at the federal level, things like mandatory red teaming, stronger mandatory security environments, clearer liability frameworks, which we, basically, don't have any clear liability frameworks in this country among states.

Dave Bittner: Right.

Ben Yelin: I mean, we're starting to see some experimentation with liability reform, but certainly, that's not something we've seen at the federal level. So those would all be preferable, in his view, than reacting to a security vulnerability by introducing another security vulnerability.

Dave Bittner: Yeah, I mean, you mentioned Hugging Face and, you know, with the Hugging Face incident where the AI broke into Hugging Face, escaped its sandbox, and broke into Hugging Face, I saw someone snarkily comment on that and say, "Oh, in the descriptions of what happened after the fact, it was really nice of these AI companies to so thoroughly document their crimes" -- because they're crimes.

Ben Yelin: They are crimes.

Dave Bittner: According to the Computer Fraud and Abuse Act, they're crimes. But in this moment, we seem to be willing to look the other way in this moment of "extreme innovation," let's call it.

Ben Yelin: I mean, who are we going to handcuff? Who are we going to read the Miranda Rights to?

Dave Bittner: Well, I mean, the delicious answer would be the CEO.

Ben Yelin: But then they say, "I had no control over what happened."

Dave Bittner: Right, so there's your clearer liability framework, right?

Ben Yelin: Absolutely. I mean, I think there are ways you could do this. I was reading about just ascribing, kind of, common law theories of liability where if you can't impute who was negligent or who committed some type of legal wrong, then it's the person who had the most control over the system or the situation. The human being who had the most control, that person is liable and can be held accountable in a court of law. I think if we apply that to AI, that becomes a really interesting theory because it's probably the developers through their CEOs. That would, hopefully, provide them enough incentive to, when they are designing these systems, not to just be, like, "Hey, this is really cool." It also might destroy the world, but, you know, we're trying to save humanity here, and we're trying to be super innovative, so just go with it. I mean, if they had that threat of liability or potential liability hanging over them, I think they might be more cautious in how they develop these tools, and that might accrue to our benefit.

Dave Bittner: It's so hard sometimes to talk about these things because it's so easy to exaggerate, and yet, hard to know what actually is exaggeration.

Ben Yelin: Yeah, I mean, part of it for me is I see people that I trust in tech media who, in previous incidents, have said things like, eh, this is nothing, like, people are overreacting to this. But with Hugging Face, they're like, no, you need to read this independent report and see for yourself exactly what happened. Now, when I read the independent report, a lot of it sounds completely foreign to me. Sometimes I have to put the independent reports in an LLM to have it explained to somebody --

Dave Bittner: To get its unbiased view?

Ben Yelin: Unbiased opinion, and then I realize what I'm doing.

Dave Bittner: You just need to make sure you load it into a competitor. It's not the actual one that committed the crime.

Ben Yelin: Exactly, exactly, "Do not use OpenAI." [laughter] Then I realize what I'm doing, and I'm, like, okay, this is silly.

Dave Bittner: Right.

Ben Yelin: I should really try and just understand this, but I mean, I think that's one of the reasons so much of this for non-technical people is just a complete black box, so to try to explain it, it just goes over the heads of 99% of potential stakeholders, voters, etc. It's really hard to understand exactly what happened, but people I trust are sounding the alarm bells here, and I think this is something that's rippled through the industry.

Dave Bittner: Well, the bottom line here is that O'Grady is saying that policymakers shouldn't address one security risk by intentionally creating another. Again, he looks at the history of the Clipper chip as a cautionary tale, so we'll have a link to that story from CyberScoop in the show notes. Again, we would love to hear from you. If there's something you'd like us to consider for the show, please email us. It's caveat@n2k.com. [ Music ] And that's our show, brought to you by N2K CyberWire. We'd love to know what you think of this podcast. Your feedback ensures we deliver the insights to keep you a step ahead in the rapidly changing world of cybersecurity. If you like our show, please share a rating and review in your favorite podcast app. Please also fill out the survey in the show notes or send an email to caveat@n2k.com. This episode is produced by Liz Stokes. Our Executive Producer is Jennifer Eiben. The show is mixed by Tré Hester. Peter Kilpe is our publisher. I'm Dave Bittner.

Ben Yelin: And I'm Ben Yelin.

Dave Bittner: Thanks for listening.