Threat Vector 11.21.24
Ep 44 | 11.21.24

Bridging AI and Cybersecurity Gaps with Mileva Security Labs’ Harriet Farlow

Transcript

Harriet Farlow: The far more pressing issue is the potential for nation states or criminals or just road actors in general to hack the AI that we have now, which they can do. [ Music ]

David Moulton: Welcome to Threat Vector, the Palo Alto Networks Podcast, where we discuss pressing cybersecurity threats, resilience, and uncover insights into the latest industry trends. Now I'm your host, David Moulton, Director of Thought Leadership for Unit 42. [ Music ] For today's episode, I'm handing over the mic to Michael Heller, our Executive Producer, to share his conversation with Harriet Farlow, CEO of Mileva Security Labs and a PhD candidate in Adversarial Machine Learning. Harriet has a unique background that spans both data science and cybersecurity, having worked across defense projects in Australia, served in the Australian government, and gained critical experience in the Australian Signals Directorate, the nation's equivalent of the NSA. She's also a regular speaker at DEFCON and is focused on identifying novel security risks and AI models. Michael and Harriet were both at DEFCON this year and recorded at the event. They discussed AI security, machine learning, and how organizations often lack awareness of the threats they face. Harriet's work highlights not only the risk, but the ways that companies can proactively address AI security in a landscape that still lacks mandatory regulation. Here's their conversation.

Michael Heller: All right. Well, Harriet, welcome to Threat Vector and thanks for taking the time to talk.

Harriet Farlow: No, thank you. It's great to chat.

Michael Heller: So, starting off with the easy question. What was your journey to get into cybersecurity?

Harriet Farlow: Well, I didn't think that I would end up working in AI security, that's for sure. My bachelor was in physics and I didn't really know what to do with it. So I did what a lot of physicists do. And that is to end up in data science. I was a data scientist for a while in a consulting company in Australia, mostly on defense projects. I then moved to the United States to work at a startup. And then during COVID, I moved back to Australia and worked in the Australian government. And that's when I became acquainted with cybersecurity because I was a data scientist in cybersecurity teams.

Michelle Heller: Okay.

Harriet Farlow: And during that time I also started my PhD in Adversarial Machine Learning. And I- as a data scientist, I think I was just shocked that I'd never encountered sort of the adversarial side of building models before. And it was never something that I'd learned. It was not something that was talked about in my experience in any data science organizational team. And so I decided to start Mileva Security Labs to address that.

Michael Heller: Yeah. It doesn't seem like a great thing that Adversarial AI is never part of data science.

Harriet Farlow: Yeah. Well, it's fascinating because the idea of hacking an algorithm has existed for a long time. The idea that a machine learning model could represent an attack surface in its own right is a really new idea. And certainly for most data scientists, especially when I started, it was the kind of field where no one was trained in data science. You just happened to get into it because you did data processing in another field. And it wasn't usually taught. You just picked it up because it's what you needed to do to get the job done. So, you just always prioritize accuracy, and efficiency, and precision, and scores like that. And safety and bias was starting to be talked about, but certainly not security. Not until now though. It's been changing a lot recently.

Michael Heller: Yeah. So what kind of research have you been doing in AI security?

Harriet Farlow: So in my PhD, I've been looking at creating novel attacks in the computer vision and natural language processing space, but with a focus on being able to use that information to model and quantify risk, which is not the most sexy or glamorous topic, but I think it's really important because certainly in the work we've done with Mileva, we find the biggest problem is usually that, you know, an executive thinks that the IT team is handling AI related issues, the AI team thinks that like IT or the security team is handling the security issues and vice versa. And then it turns out that no one is addressing it and certainly no one really knows what it means to think about applying cybersecurity methodology into AI. And so understanding what your risk is so you know where to start is actually really important. So I find it really interesting, but not everyone does. And that's okay.

Michael Heller: No, that's fine. There's always been a lot of talk about, you know, security by design in all of software. And it seems like it's going to be even more important in AI.

Harriet Farlow: Yeah. Well, it's fascinating though because that's something we take for granted in software and cyber, but it's definitely not present in data science and never has been, but it is now. And that's why it's really great at this particular DEFCON a- as well, but in the security generally over the last few years, people are starting to take AI security seriously. But when I started in the field in 2021, I was often the only AI security person talking at any conference I went to. And it was a- it was also Australia though, which is, you know, a smaller market and many say maybe it's -- behind isn't quite the right word. We just don't have tech giants over there the same way you do here, but yeah. Going from trying to convince people that AI security is important and most of those people had never heard about it to, you know, coming here and everyone is fairly cognizant of it now and figuring out what we can do about it is really great to see.

Michael Heller: Yeah. And speaking of bringing awareness to it, one of your talks here at DEFCON has been about how you, I guess we could say, worked with the casino in Canberra.

Harriet Farlow: My talk was called on your Ocean's 11 team. I'm the AI guy or technically girl. And basically, I show a new attack and apply it to facial recognition in the casino context. And I say new attack instead of quotation marks, really, because all attacks are basically the same or similar in that they rely on optimization in different ways, at least definitely in the computer vision space of being able to recognize objects and faces and things like that, but I think that the point of the talk wasn't necessarily to show something really complicated or niche, but to highlight how most models are still really vulnerable to basic kinds of attacks. And that this hasn't really changed in the last few years. And that while there are lots of really good conversations happening about, you know, the importance of Frontier Labs, you know, OpenAI, Anthropic companies like that, and the security work that are happening in big tech organizations like that, most organizations don't have access to those kinds of resources or skills and are still relying on inherently insecure models.

Michael Heller: Yeah.

Harriet Farlow: Yeah.

Michael Heller: And part of the talk was that, you know, the casino had to rely on all these third parties. How can an organization like Vet, these third parties, to make sure that they have some sort of security in place or AI safety rules?

Harriet Farlow: Yeah. There are lots of questions that organizations should be asking of their third party providers, but at the end of the day, it's really on that provider to secure their AI. And some companies are really good about it. Like Microsoft does fantastic things. The Frontier AI Labs, they're really talking about it and doing things to actually make a tangible impact, but there's no requirement they do that. Like lots of other companies that produce AI, you know, we have no idea how robust or secure they are. And there's no mandate that requires them to have any sort of transparency over that, which I think is the main problem. And I guess kind of leads into the talk that I'm giving tomorrow, which is a nice segue maybe.

Michael Heller: Yeah. So you mentioned that you worked with the Australian government and tomorrow you're giving a talk more about policy. So what kind of policy changes would you like to see?

Harriet Farlow: Well, yeah. So my talk tomorrow is called Hacker Versus AI Perspectives from x-spy because I used to work at the Australian Signals Directorate in the Australian government, which is Australia's equivalent of the NSA. And so I was not like out there doing cool spy things. I was in, you know, doing cool cybersecurity things, which is still cool, but yeah. I think it was really interesting to take all of the technical skills that one has in AI and security and have to consider what that means in a national security context. And certainly one of the driving forces for starting Mileva was that there are a lot of companies in the AI safety space that are working on long-term risks of AI. Things like preventing misaligned models from being designed and preventing Skynet from happening basically. But from my point of view, the far more pressing issue is the potential for nation states or criminals or just rogue actors in general to hack the AI that we have now, which they can do. You know, as we see from all the different talks on AI security, it's very easy. And they just weren't as many like companies or organizations that were addressing the short-term risk or even policy, like people are starting to talk about AI safety policy. So again, things like, yeah, making sure that AI doesn't try and take over the world, but most AI policy in the world at the moment is really on increasing adoption. And it's more about ensuring the security of nation states in terms of their economic advantage and their innovation and their sort of competitiveness and a global landscape and not at all about the security of the AI systems themselves. And I think the challenge for policy makers is that like security is still framed in terms of the nation as the referent object, which is -- I don't know. Security studies speak that I sort of go into in the talk tomorrow, but I guess the point of the talk is to really think about, you know, what are we trying to secure and from what and how can we make good policy that addresses technical issues with enough flexibility to be future proof as well?

Michael Heller: And so with a lack of policies around AI security and AI companies responsible for doing it themselves, is there anything that organizations can do? Because obviously there's a lot of hype, there's a lot of rush to adopt AI and implement it in any way they can. So what can organizations do since we're pretty much still in the Wild West of this technology?

Harriet Farlow : I think the Wild West analogy is really good, because that's always applied to the early days of the internet and computer security. And we're sort of in a Wild West when it comes to AI, but not really because we can learn from all of the lessons in cybersecurity. So I think the things that organizations should be doing is applying the security mindset to AI now and maybe pressuring isn't the right word, but advocating for security because most companies who provide AI do care about it or at least they certainly do when their customers want that. And we've definitely seen that happen because companies like Microsoft, and Meta, and OpenAI, and Anthropic, like they do talk about security because there has been quite a lot of pressure from their users. We just need to make sure we apply that same pressure to all the other producers of AI as well and to the government to encourage those companies in whatever way is feasible based on the jurisdiction to do that as well.

Michael Heller: We described what your talk was about a bit. Maybe we should have a more- could you give a more concrete or a more real world example of the types of attacks that somebody can --

Harriet Farlow: Sure, yeah. The premise of the talk was basically that most of the ways we hack AI systems at the moment, whatever domain it is. So computer vision or natural language processing or signal data. It's usually by perturbing I guess the thing in question. In the context of my talk, it was a computer vision problem. So you are basically changing the thing that you're trying to identify. And so the techniques I was looking at basically where you're able to change other things about the data you have and still cause a deceptional disruption to the model.

Michael Heller: So specifically in your talk, you had like a picture of a navy boat and you just put in a few --

Harriet Farlow: Yeah, a few different adversarial regions.

Michael Heller: Yeah. You changed a few things in the image that were not on the boat itself.

Harriet Farlow: Yes, yes. And you can still cause a misclassification. Which is- I think it's kind of crazy. And most models were vulnerable to something like that. And then the specific use case in the talk was facial recognition technology and how that is also vulnerable, but I think it really speaks to the brutalness of a lot of models, especially the open-source models that are out there. And also, you know, probably the third party models that we can't really verify.

Michael Heller: During your talk, you were talking about the different attack surfaces for AI. And intuitively, it makes sense that you would attack a visual model differently than a large language model, but can you talk a little bit about the different types of attack surfaces? Because from the chart that you had up there, it looked like there were quite a few --

Harriet Farlow: Um hmm. Yes. So I guess the same way that you would consider any computer or cyber system as a, you know, collection of different datasets and movements between them at a like very basic level, you can think about the same way in machine learning systems and that they're very reliant on data and how it's processed, but the way that it's processed can be really different. And usually people think about an AI system as sort of this ephemeral kind of intelligence, but at the end of the day, there are machine learning models that process and ingest data in very different ways depending on what they're meant to do, whether it's computer vision, or natural language processing, or signal classification. And all of the different model architectures represent different kinds of attack surfaces with different, you know, vulnerabilities and exploits. And especially depending on how you implement it as well as part of the system and whether you have a single model or many models working together. You know, it's far more complicated than people realize. And it means that the attack surface is actually -- you know, it's a real thing and it's something that you need to consider.

Michael Heller: Attack surface in general are already getting bigger and more complex, but having the attack surface change depending on what models you're using and what combination of models, like that seems --

Harriet Farlow: Yeah. Yeah. It's something that people sort of forget and especially because most data scientists, you know, the people who are building machine learning models, don't come from a security background, something they don't think about. And then security professionals often don't know a lot about models. So it's challenging, but we're trying to bridge the two.

Michael Heller: And then also talking about adversarial attacks, you showed something called the 3D mode of. Can you walk us through that?

Harriet Farlow: This is something I'm very proud of. I'm probably too proud giving how boring it is, but this is something that we taught a lot in our workshops and is also the backbone of how we define risk in our product. So the 3D model stands for disrupt, deceive, and disclose. So basically, it represents all of the different AI security threats, but groups have been to three main axes in terms of the way that those threats can be conducted. And you could either disrupt the model so it basically doesn't do what it's intended. You can deceive it so it does something specific, you know, perhaps it chooses a predefined outcome, predefined to you as the attacker, of course, or disclose. So to disclose confidential or sensitive information about the model. And this is a way that we've been able to characterize like all of the 100 plus adversarial machine learning techniques into three axes. And I was basically teaching this to a defense official we were working with last year, and he said, "Oh, that's just like the CIA triad, but for adversarial machine learning and AI security." And I was like, oh, yes. That's what I would love it to be just because there are so many frameworks and taxonomies out there, but they're so complicated. And I think the reason that the CIA triad has worked so well is because it's simple. Like anyone can understand it at a basic level. And then you can become like complex from there. But at the moment, AI security doesn't really have anything simple at the top that people can refer to. So we've found that really helpful in talking to executives or non-technical people and communicating different AI security risks.

Michael Heller: So between that triad that you have and the taxonomies, it sounds almost like you have built like a MITRE ATTACKsurface for AI.

Harriet Farlow: I would go- MITRE already has its own for AI security. And we love it. We don't need to replace it, but I think it's really hard to communicate what's in MITRE ATLAS to people who don't already know a whole lot about AI security. And the problem that we find and many other companies out there are starting to work on AI security and they see it primarily as a technical problem. You know, they're building pen testing solutions for AI models, but our experience is really that it isn't a technical problem. The problem really is that cyber professionals are suddenly faced with dealing with AI and they don't have the time, they don't have the training. It's a people and a process problem, just as much as it is a technology problem. And that's where things like the 3D model really have a lot of value.

Michael Heller: When we are specifically talking about AI security and keeping models safe, is there anything organizations can do to make sure that's happening in their companies?

Harriet Farlow: Maybe it's a bit woo woo of an answer, but I think it's so much on education. I think we take it for granted at DEFCON specifically that the audience is pretty familiar with security speak and are usually pretty AI literate too. But most people I work with are definitely not. And it's really hard for them because they keep hearing about this AI thing and they want to get on board. They know it does cool things and then suddenly they hear terms like AI safety and AI security and they just feel so disempowered and they don't need to be, like none of it is really all that difficult if it's taught well. And I think the more people who feel like they can be part of the conversation, the more likely we are to have firstly people asking the questions at all. And then more people from different areas coming up with good solutions as well, but I think across the board, general AI and AI safety and security and literacy is fairly low. And if we bring that up, then we will have so many more people working on these problems and able to like ask really robust questions in their organizations about it more so than we see now.

Michael Heller: So looking at professionals who are looking to get into cybersecurity or maybe data scientists who never knew that this was a good option, what are the skills that they should have?

Harriet Farlow: The thing I really love about AI security is that it's so new that we're right at the ground floor. So people who really want to or are able to sort of upskill in cyber or AI in whatever way, usually have something valuable to add to the conversation based on whatever field they come from before. I've seen a much more diverse group of people in terms of their backgrounds and like how they identify in AI and AI security than any other field, which is really cool. I think everyone has something to contribute to the field and learning about AI and security is definitely not out of reach of most people. Most people I meet at DEFCON have come into security from other fields. And that's what's great about it.

Michael Heller: What do you think about the future of AI? I get the feeling you're generally an optimist in terms of like whether or not AI is going to take over the world.

Harriet Farlow: Yeah.

Michael Heller: Maybe.

Harriet Farlow: You know, I'm inclined to probably agree with that. I don't think- is it optimistic to say I think it's far more likely that people are going to hack our AI and cause massive impact before Skynet or the Terminator takes over? But no, I'm generally optimistic. I think the risk that the Terminator is going to happen anytime soon is fairly low.

Michael Heller: What's the big takeaway that you would like somebody to remember from this conversation?

Harriet Farlow: AI security is a real threat and it's different to cybersecurity, but everyone should play a part in ensuring that AI security does not supersede cybersecurity and becoming the next big threat. We can prevent that from happening. And yeah. I hope what people take away from the talk is that it still doesn't take a lot to hack a model. And there are a lot of things we can do, but there's no requirement that we do that yet. And that is what the problem is. [ Music ]

David Moulton: I hope you found today's conversation with Harriet Farlow insightful and that it deepened your understanding of the evolving challenges and opportunities in AI security. As always, our goal is to empower you with the knowledge to stay resilient against tomorrow's cybersecurity threats. That's it for today. If you like what you've heard, please subscribe wherever you listen and leave a review on Apple Podcast or Spotify. Your reviews and feedback really do help us understand what you want to hear about. I want to thank our Executive Producer, Michael Heller, our content and production teams, which include Kenny Miller, Joe Benco, and Virginia Tran. Elliot Peltzman edits our show and mixes the audio. We'll be back next week. Until then, stay secure, stay vigilant. Goodbye for now. [ Music ]