Hunting for AI Bug Bounty

Sherrod DeGrippo: Welcome to the Microsoft Threat Intelligence Podcast. I'm Sherrod DeGrippo. Ever wanted to step into the shadowy realm of digital espionage, cybercrime, social engineering, fraud? Well, each week, dive deep with us into the underground. Come here for Microsoft's elite threat intelligence researchers. Join us as we decode mysteries, expose hidden adversaries, and shape the future of cybersecurity. It might get a little weird, but don't worry, I'm your guide to the back alleys of the threat landscape. Welcome to the Microsoft Threat Intelligence Podcast. I am Sherrod DeGrippo, and I have a really cool topic for you today. We're going to talk about bug bounty, but we're not going to talk about bug bounty in the way that it used to be done. We're talking about the new way of bug bounty, artificial intelligence bug bounty. I am joined with my coworkers from Microsoft: Andrew Paverd, Principal Research Manager; and Lynn Miyashita, Program Manager. Hi, guys. Welcome to the show.

Lynn Miyashita: Thanks so much for having us.

Sherrod DeGrippo: Thanks for joining me. So the reason I wanted to talk to both of you is because, bug bounty, we know what, we love it, but AI bug bounty is a weird new thing that Microsoft launched back in October of 2023. And ever since then, AI bug bounty has been something -- I think we were the first to do it -- has been something that people talk about a lot. And I notice on social media, people will find bugs in AI. Like, that is a huge thing that people -- it's almost like the old days of bug hunting where people don't even mean to find stuff; they just are messing around as a user and something kind of pops up to them that they're like, "Oh, this seems like a problem." So, Lynn, can you kind of walk us through, like, what is a bug bounty? But how did the AI bug bounty start? What is kind of going on? What's happening with bug bounty?

Lynn Miyashita: Yeah. So the Microsoft bug bounty program offers monetary awards to incentivize our external security researchers to find and report high-impact security vulnerabilities that in turn help us fix security issues and help protect our customers. So my team is the team that manages and builds these relationships with security researchers and our industry partners. And we, along with our relationships here, help identify customer threats and emerging vulnerability patterns. So we take those patterns and insights and we share them with our engineering teams across the company to drive security investments beyond these one-off bug fixes. So where our AI bounty program comes in is as a key result of many, many months of investments and learnings since the launch of Copilot, which was previously Bing Chat, which included things like an AI security research challenge and even an update to Microsoft's guidance on vulnerability severity classifications for AI systems. I think Andrew can actually talk a little bit more about that AI bug bar.

Sherrod DeGrippo: Yeah. What is a bug bar?

Andrew Paverd: Absolutely. Thanks, Lynn. And thanks, Sherrod. Essentially, what we need as a critical first ingredient for any type of bug bounty program is a really clear definition of what we mean by bugs. So what are the types of things that are going to be in scope for researchers to report to us and how do we talk about them in a common language, if you like, in terms of what the vulnerability is, what it affects, and then what the severity level is? And all of these play a part in the bug bounty program. So as Lynn mentioned, with the rise of Copilot and new AI systems, we had to extend Microsoft's vulnerability severity classification, or informally our bug bar, to cover these new AI and machine learning systems. And that's what we also did last year, and this formed the foundation of our bug bounty.

Sherrod DeGrippo: So can you kind of give me an example of, like, something that might reach the bug bar and something that might not? And could I sort of think about that contextually if I was just going to pop it off the top of my head?

Andrew Paverd: Absolutely. Maybe I can jump in first on this one. So we've put a lot of careful thought into this bug bar. And the first thing to mention is at the moment, we're focused on security issues. There is a plan to think even more broadly about this. But for now, we've encouraged researchers to look for, as they would expect, security issues that they would report to Microsoft, but in particular, to think more broadly about new security issues that could arise from AI systems. So we can certainly dive into the details. But one of the very interesting classes that we've seen arise for AI systems is this category of, say, command injection or prompt injection. And this is interesting because this is not something that would exist prior to the current generation of AI systems. And so we're really interested in seeing researchers look for new and interesting vulnerabilities in this sort of space.

Sherrod DeGrippo: Got it. So prompt injection is a big part of it, it sounds like, which I know is one of those sort of examples that people have stumbled into things and just sort of started pasting screenshots all over social media, like, "Look what I made the AI do or say." And so when people are finding those, are those kind of the examples that you're looking to have submitted into the bug bounty program?

Andrew Paverd: I think yes. The answer is even a little bit more nuanced than that. So one of the things that we call out in the AI bug bar is to look at the effect of what this is basically able to do through the prompt injection. If you're able to cause harm to other users in some way, then obviously this is something that we will address immediately, and this is something that must be reported. If it's just something that's causing the AI to perhaps behave slightly differently, I would say we need to start with the question of: is this going to meet the bar, if you like, for our bug bounty program? And so we really want to encourage folks to look for things that will have the highest type of impact, if that makes some sense.

Sherrod DeGrippo: Sure. So, like, prioritization around vulnerabilities from a severity perspective, things like that.

Andrew Paverd: Exactly. Yeah.

Sherrod DeGrippo: So, Lynn, help me understand, ike, what products are included in the bug bounty and if there's any new products going to be added maybe coming up. So what's included?

Lynn Miyashita: Yeah. So today, our AI bug bounty focuses on vulnerabilities and the Copilot AI experiences. So if you think of, like, copilot.microsoft.com or the Copilot experiences in your Microsoft Edge browser, or even the Copilot mobile applications for Android and iOS, those are what we've got right now in our public AI bounty program scope. But we are constantly working in the background here, growing, iterating, and evolving our bounty programs to include, you know, more of our suite of Copilot AI experiences as there are, you know, new features and new products being launched on a daily basis here at Microsoft.

Sherrod DeGrippo: Got it. So can you tell me, Lynn, like, if I find something that's a bug or that I think is a bug, what do I do with it?

Lynn Miyashita: Submit it as soon as possible to the MSRC. So we've got a MSRC researcher portal at aka.ms/secure-at. And there you'll find a submission form for you to fill out with information like what type of vulnerability have you found, what product is affected, as well as a proof of concept. So whether this proof of concept is a video or screenshots or even just written steps to reproduce the issue, all of this information is really helpful for our teams to be able to investigate as quickly as possible.

Sherrod DeGrippo: Got it. And so I guess my question is, like -- I'll start with Andrew. Okay. Let's walk through. I'm in a Copilot experience, one of the AI Copilots -- I personally use Copilots constantly. And let's say that I find behavior that seems wrong or bad, or maybe I leverage some kind of prompt injection technique to get the Copilot to behave in a way that really isn't safe or smart or good or secure. I submit it to the portal that Lynn mentioned, which is aka.ms/secure-at -- and we'll put that in the show notes so everyone can click on it directly if they want -- what does Microsoft do with them?

Andrew Paverd: Oh, great question. So the very first thing that happens once you submit something is we start the clock. And I'll come back to this clock a little bit later in my example, but keep that in mind. So essentially, what happens then from the technical side is it comes to us, the Microsoft Security Response Center. And the first thing we do is we work to reproduce and understand the vulnerability -- potential vulnerability that's been reported. This is really a critical step because, firstly, we want to make sure that we fully understand what the researcher is telling us, and we might go back to the researcher via an open communication channel to say, "Hey, we need some more information here," or, "How exactly does this piece of the report work?" So then once we fully understood the vulnerability from our end, we'll start working with the engineering team to understand the severity or the impact of that vulnerability. And in parallel to us assessing the severity of that vulnerability, the engineering team starts on working on a fix or a mitigation for it. All of this is, as I said, in parallel with us maintaining an open line of communication to the researchers themselves, and that's where Lynn and team come in to also ensure that there's a follow-up for the bounty side of things. If the researcher's report qualifies for bounty, the bounty process goes ahead after that. And so coming back to the clock that I mentioned at the start, why do we do all of this under the clock? Essentially for two reasons. Firstly, obviously, we want to make sure that we fix vulnerabilities as fast as possible and that we prioritize the highest severity vulnerabilities, but also because we at Microsoft are strong advocates for coordinated vulnerability disclosure. And so that often means that researchers may have delayed publicizing their vulnerability until we've had a chance to put a mitigation in place. So it's only fair from our side that we keep to define timelines for this type of thing. And as you may have seen from the recently announced Secure Future Initiative, this is very important, having both faster response as well as full transparency for these sorts of vulnerabilities. So once we've managed to remediate and mitigate this vulnerability, we're in the process of encouraging researchers to be transparent and really publicize this so that the whole community can learn from that vulnerability finding. So it's quite an involved process. Hopefully, that summarizes it.

Sherrod DeGrippo: So I think something that everyone wants to know is, how many of these bugs have been submitted? What kind of metrics are we looking at? Any payouts, any big bounty payouts that we've seen? Can we talk about that?

Lynn Miyashita: Yeah. So we've seen some great submissions so far, and we've been learning new things about how our researchers approach, like research and AI systems, so much through these reports. Andrew, I don't know if you can give a couple examples or maybe --

Sherrod DeGrippo: Yeah, tell me. I'd love to know. Like, what are some things that have been submitted that, like, you thought were cool or weird or interesting? What are the noteworthy things coming in?

Andrew Paverd: Yeah. So I think one of the examples that's been quite interesting from our side, as I mentioned earlier, prompt injection attacks are certainly something that's been known about. But the interesting aspect of this is, what can you leverage prompt injection attacks to do? And speaking in broad terms, this could range quite significantly depending on the product. But perhaps one of the more interesting ones and an area where there's perhaps further research required is if a prompt injection attack can be leveraged to somehow exfiltrate private data. So to get data that the model has access to sent somehow to the attacker. Now there could be various possible ways of doing this, and we've seen some very clever ways of doing it, but the researchers themselves have the privilege of disclosing those. But in broad terms, those types of interesting aspects would be most of interest to us.

Sherrod DeGrippo: So let me ask you, if a researcher submits into the bug bounty, gets found to be a legitimate bug and that bug gets fixed, etc., do you get a CVE for those?

Andrew Paverd: That's a super interesting discussion. At the moment, there's ongoing discussion within the industry of what we should do in terms of CVEs for these new and emerging fields, things like cloud vulnerabilities and AI vulnerabilities. There have been CVEs issued in the past for AI vulnerabilities, but it's still a process where everybody is learning about what's going to qualify for an AI CVE. So I think there are a lot of public -- there's some public work groups and rules and discussions going on about this. And so people can actually go and look at the rules for the CVE board to see what may qualify for CVEs in the future.

Sherrod DeGrippo: I think that as we're coming into this, like, new AI world, it's going to require new ways of thinking when it comes to things like that. CVE might need a kind of review on sort of where AI falls in those because AI is such an interactive thing. It's not just a simple code bug. It's sort of like a holistic systematic bug that involves the user and what the user does with it. So I also kind of want to understand, if you know a traditional bug hunter, somebody who looks for security vulnerabilities in traditional software packages, would you recommend that they start doing AI bug bounty? Lynn, I'll start with you. Is it something that you think more traditional bug bounty background people should take a look at?

Lynn Miyashita: Hundred percent. Yes. Anyone of any background, I think, can get started in this area. Like Andrew mentioned, there's a number of different types of vulnerabilities that you can find in varying severity levels. And I think that kind of opens up the door to anyone of any background being able to have an opportunity to go to copilot.microsoft.com and start, you know, chatting with Copilot to see what they can find.

Sherrod DeGrippo: Andrew, is your opinion the same there, bringing the traditional bug bounty hunters here?

Andrew Paverd: Absolutely, 100%. And building on what Lynn said, I would add that because we're looking at the system holistically, AI is just one part of a much bigger system. And so there's certainly a potential for finding vulnerabilities that span what you would say the traditional scope of a bug hunter and the scope of new vulnerabilities that may arise because of AI. And perhaps the most interesting is the intersection between these two aspects. And so definitely would encourage everybody to come and have a look at our systems.

Sherrod DeGrippo: So for those of you that are listening out there and that are familiar with or maybe even participated in traditional bug bounty programs out there, take a look and maybe rack up a couple of AI bugs if you can get in there with the Copilots and start trying things. I think one of the things that's so interesting about looking for AI bugs or what we call sometimes Microsoft AI red teaming is sort of like messing with it to the degree that you don't have to have a lot of knowledge of systems and coding and protocols. I love that stuff. But if you've never tried any of these things before, just about anyone, Andrew, Lynn, correct me if I'm wrong, anybody can be an AI bug hunter, right?

Lynn Miyashita: One hundred percent.

Sherrod DeGrippo: Yeah, that's so cool. I think that that's one of the fun things about it. And you know, there's so many different Copilots available. There's Copilot for Security, which I work on quite a bit, but there's also Copilot within Bing, right? Like it's a search engine Copilot system. And that one is really, really fascinating. It's really interesting because in my usage of the Bing AI, it has all of these different modules and kind of features for different use cases. One of my favorites is getting online shopping deals. And so it kind of goes through and finds deals for you on the web using AI. And I think that would be a great place to start bug hunting because, you know, it's so important for it to interact with the ecosystem of the internet to find the deals that there's probably a lot of possibility there. So AI bug hunters, Lynn, final question for you. If you were going to get somebody to start bug hunting in AI to submit to the program, what would you tell them?

Lynn Miyashita: First, go to aka.ms/aibounty because you can learn all about our scope and what types of vulnerabilities Andrew was talking about for that AI bug bar there. And second, definitely stay up to speed with the MSRC blog and our like X, or formerly Twitter, social media channels because our teams post some really cool, interesting kind of research that are going on internally. And we're also always talking about new and upcoming bounty scopes. So when there's an update for our AI bounty scope there, we'll be sure to post on our social medias.

Sherrod DeGrippo: Awesome. Andrew, anything you want to add there on a piece of advice you might give somebody who's like, "I want to start doing AI bug hunting?"

Andrew Paverd: Well, it's hard to follow Lynn's answer there. Absolutely everything that Lynn said. I think the only additional piece of advice that I would give is to say, try it out. And this is a whole new world. This is an exciting new aspect, and you don't need to have played with this technology for many years. Nobody has decades of experience in this. So really try it out. And if you find anything interesting, send it our way.

Sherrod DeGrippo: Excellent. Well, Lynn, Andrew, thank you so much for joining me. I think that we should end it here so that people can go and start bug hunting in the Copilot products and finding those bugs and start submitting them. Thank you for joining me. It was great talking with you.

Lynn Miyashita: Thanks again for having us, Sherrod.

Andrew Paverd: Thanks very much.

Sherrod DeGrippo: Thanks for listening to the Microsoft Threat Intelligence Podcast. We'd love to hear from you. Email us with your ideas at tipodcast@microsoft.com. Every episode, we'll decode the threat landscape and arm you with the intelligence you need to take on threat actors. Check us out, msthreatintelpodcast.com, for more, and subscribe on your favorite podcast app. [ Music ]