AI in the Wrong Hands

Assaf Keren: Being a CSO feels very lonely at times, and because you carry a very unique risk for the business that the-alot of times the business doesn't understand. And so, building the community, reaching out, listening to stuff like this, listening to other podcasts, but creating the understanding that this is an industry that is dealing with this that the industry is fighting, we're all fighting the same fight against the same people, we're not fighting each other. And so, reach out, ask for help. [ Music ]

David Moulton: I'm David Moulton and this is "Treat Vector." Today, I'm speaking with Assaf Keren about AI in the wrong hands. Assaf is SVP and Chief Security Officer at Qualtrics and the author of a new book, "Lessons from the Frontlines" out now from Wiley. Here's our conversation. [ Music ] Assaf, welcome to "Threat Vector." I'm really glad to have you here. I know there have been some scheduling nonsense, but we finally got it down. We're finally on the mic together, so let's have a good conversation.

Assaf Keren: Six missed scheduling to get to this point if I counted correctly, but let's go. I'm excited.

David Moulton: Yeah. Before we get into our topic, I'd actually like to hear a little bit about your journey. You know, I dug into it a bit and I'm sure our guests would actually find this interesting. You've had actually a fairly long career in cybersecurity from your early work in Israel, for PayPal, and now with Qualtrics. How do you think about the path that brought you here?

Assaf Keren: There were really hard points in my career. I had to step out of the startup, because I had a lack of clarity and lack of creation from my cofounders after a year.

David Moulton: Yeah.

Assaf Keren: And that was really, really hard. It was probably one of the hardest years in my life, but also probably one of the years was the most learning for me. And what I will say is that I didn't know that at the time. If you look at the different decisions that I've made, I didn't know that at the time, but there was always those search for experience not for title, that guided my career progression. So, when I went to run the startup, it was I wanted to do this thing, I want to try running a company. You know, I want to do the enough twice. I don't want to do that again by the way. Even leaving my second startup and going to work for PayPal, I took a role cut. I moved from being a CTO to being a manager of four people in EMEA. Ended up being the CISO at PayPal which was a great decision probably in hindsight, but-but it was a searching experience. It was searching the, what is the experience gap that I have to make me a more full professional. And that's what I suggest to people when they come talk to me, especially people that say hey I want to be a director or I want to be a senior director, I want to be VP. It's not a good pursuit. The pursuit is, I want to do something I enjoy and I want to learn new things, and this is the direction that I want to go to, and I think that in hindsight that's what's driven my career so far.

David Moulton: Yeah. Well, in your book you wrote about this danger of feeling like you know enough and how that confidence can become-- quietly become a liability. In field that's moving as fast as AI security, I think the trap feels easy to fall into. Where do you see that showing up now?

Assaf Keren: Specifically with AI, I think that I'm seeing a lot of security teams not understanding how pivotal this moment is and the using legacy thinking in making decisions. And maybe defaulting to the default of security teams, which is being "the department of no."

David Moulton: Um-hmm.

Assaf Keren: I think especially there is a gap of knowledge in security teams understanding AI machine learning. I think it has been there for a while, but the-but with the explosion happening right now, that fear is dangerous and I-that lack of curiosity that I'm seeing in a lot of places is bothering me, because I think that we're creating more impact than good when we're doing it.

David Moulton: How do you catch yourself from falling into that trap?

Assaf Keren: Sometimes successful, sometimes I'm not by the way. I don't want to make it sound like I'm always curious, but I do curiosity checkups. I sit down and then generally say to myself, "What did I miss?" There is a friend of mine, Lea who is the CISO of LinkedIn and they wrote on LinkedIn that something I agree with completely, that there is a superpower in willing to look like you don't know the answer.

David Moulton: Um-hmm.

Assaf Keren: Or willing to look like you're stupid and ask questions like you're stupid.

David Moulton: Um-hmm.

Assaf Keren: And-and sometimes I'm successful, sometimes I'm not. In the day-to-day, like the accelerated day-to-day phase that we're in, a lot of times it's just easy to come in and say, "Hey, this is the answer move on."

David Moulton: Right.

Assaf Keren: And I do have a good team around me that knows to also challenge me when I'm that way, and tell me "Hey, Assaf, you're wrong here. Let's have a conversation."

David Moulton: Yeah.

Assaf Keren: And that's really, really humbling and it's great to have that support structure.

David Moulton: Yeah. I have concluded that there's a difference between being dumb and being stupid. And I think being dumb is acceptable, it is a natural state. It's all of us are dumb. And when we refuse to learn or refuse to learn the lessons that whatever the situation is tries to teach us, that's being stupid. So, you touch the stove the first time. Sorry kid. Kind of dumb, now you know better. You touch it the third, fourth time, and now it's just getting kind of stupid. So, I think that it's okay to look dumb. In fact, if you never look dumb you're not really walking into situations that are going to challenge you. I think it's when you look stupid later one when you had that opportunity to go learn or to understand or to be curious and dig in, that's when we end up looking stupid and I think that's what we try to avoid, not realizing that there is a-there's-I'm grateful when I'm in a room and I'm the dumbest one. It means about to learn some things and maybe have a, you know, a lot of time to reflect and think and ask good questions. And I like the idea that you have a team that's around you that can push you too. I think that's a sign of a strong leader when your team can push back on you and say, "We got to rethink this."

Assaf Keren: Yeah.

David Moulton: Alright. A quick question for you before we get into the deeper topic, you've had this really long career, obviously you've got tons of stories to tell, why this book, why now?

Assaf Keren: So, I've been wanting to write a book just for the experience if writing a book for about 5-6 years.

David Moulton: Okay.

Assaf Keren: And I kept hitting the wall of what do you have to add to literature that wasn't already written. And that's valuable for other people. And then it got to a part where like, okay I'll start writing something and then stop and then start writing something and then stop. And I said, "Well, you know what? My stories are my stories and I like to tell them. Maybe I write a book based on my stories." So, I sat down and wrote a lot of different stories that I have and then frame-start framing frameworks in my mind, then like curiosity, grit, and optimism, and the diplomacy, business-business acument, change management, and execution. Like the frameworks that I use when I talk to people telling them how to build things, or to build teams, or how to behave. And I said, Okay, now I have stories. Now I have frameworks, it comes together into a book that is based on my personal experience. And none of this groundbreaking, but it is I think, the first time it was written in the sense of a security leadership book and.

David Moulton: Um-hmm.

Assaf Keren: And with some grounding in the life-to-life that we deal with which is a bit different. And the peak part I'm proud of the most is actually the last part of the book that is a lot about psychological safety and taking care of yourself, and acknowledging the mental challenge that is working in security that is very different to other roles that other people play. And that came together really, really nicely into a book. And so, the actual act of writing the book, that was a few months. It was pretty easy once I had the structure in place, but getting there took me years. I'm hoping that people find it useful. I don't know, we'll see.

David Moulton: I recently interviewed Allie Mellen about her new book "Code War" and her big lesson was be aware of time management when she put the book together. And she said she learned the lesson that she didn't have her time management as tight as she wanted. And it sounds like yours was finding that truth that you wanted and then it flowed out of you. Well, Assaf let's get concrete on this one, I appreciate you letting us go behind the scenes and get to know a little bit about you, but let's talk about what happens when AI gets in the wrong hands and what that actually looks like today. Maybe [multiple speakers] like a future scenario, but what are you seeing attackers doing with these tools right now?

Assaf Keren: Great timing. I just published a blog post about the whole Mythos thing and I said in that blog post, "It's not a future conversation. The fire started in 2023 when GPT was unveiled." Okay, initially what we saw was the basic things; phishing, deep fakes, those kinds of discussions. I think a month ago, Amazon has published that they've seen an attacker go in and do it and takeover, go it and utilize AI agents to do discovery within the network or within the customer environment. We're definitely seeing conversation of the timeframe from vulnerability to execution of the vulnerability.

David Moulton: Um-hmm.

Assaf Keren: AI red teaming is a real thing and if AI red teaming is a real thing, we're-AI attacking is also a real thing. And all of these things are reality right now, and the interesting piece about what happened with the Anthropic Mythos publication is that everybody say, "Oh, this is going to be bad." "No, it's bad already." And I think a great example of how this shifts things drastically; I had a conversation with RSA, it already a month ago, my god. I had a conversation with RSA with a founder of this email security company and I asked him how it's going, because they raised a seed and they were going, and usually when you raise a seed you go to the U.S. market and you start there. And he said, "We're actually big in Japan." I told him, "Why? Like why Japan out of all places?" And he said, "Look, one, the Japanese culture is very trusting. It's one of the safest places on the planet. You can leave your wallet on the desk and with lunch and nobody will touch it." And so, inherently it's a very trusting culture. And that's great, but they have had a language and culture moat around phishing all these years and now they don't have that anymore, because GPT and other models are able to mimic Japanese well enough. So, phishing now has become an epidemic and the government is very focused on that. So, we're getting a lot of traction in Japan. And we're going to see a lot of these shifts in which the assumptions that we've made of the things that will keep us secure are going to be null and void. And we will need to change the way we think to building better cultures and better systems, and that's the reality right now. Now, will it get worse in the future? Yes, it will get worse in the future as we improve models and I like to say when I talk to people, "Attackers don't have security teams telling them not to use AI." Corporates do. And we're the security teams who are making this bad for our companies, but they will continue using it. [ Music ]

David Moulton: So, when you think about like AI-generated phishing, you mentioned that with the Japanese market losing that sort of natural or just that defense they had because there was the language barrier; LLM-assisted, recon, deep fakes, right, like there's all these different tactics and things that AI is helping an attacker with, is there a capability that really stands out to you more than the others?

Assaf Keren: I think that we are already seeing semiautonomous if not fully autonomous agentic attacks.

David Moulton: Um-hmm.

Assaf Keren: And that means that the-what it means is that the scale and scope, the economic pressure on people that attackers have is going to reduce the scale and scope of what they can try is going to be, or already is accelerated and they're going to get to vulnerable endpoints quicker.

David Moulton: Um-hmm.

Assaf Keren: So, it's about the how fast they can move, which is scary because we can say as much as we want to say security by obscurity is not a thing, but security by obscurity is a thing and unless we take a really strong stance against it, then we're going to be bit in the ass by these attackers that now don't have people constraints in doing fully autonomous recon. The other thing that I'm worried about is dedicated crafted malware that does not have signatures.

David Moulton: Okay, I want to get into both of those, but I first.

Assaf Keren: Yeah.

David Moulton: My fist question for you would be what does that AI-assisted or agentic attack look like at scale? You know, I've tried to look back at some of the big attacks in the past and imagine a world where there weren't human capacity constrain and it's unsettling for me to think about that. But like, walk through that for me.

Assaf Keren: I think one is once you're in the crosshair of an attacker, then the enumeration discovery of the endpoints that allow entry into your environment is going to be very fast and very thorough, and then the attempts to hijack those endpoints is going to be very fast and very thorough. Probably noisy to start which is where we have some level of ferocity maybe?

David Moulton: Okay.

Assaf Keren: Ferocity of them going and accessing data and taking the data away and exfiltrating, is something that we've never seen before. And you have seen like living off the land type of attacks where people were trying to install OpenClaw on devices after they've breached them.

David Moulton: Um-hmm.

Assaf Keren: OpenClaw that they managed, so getting to persist through AI which is also very, very interesting.

David Moulton: So, Assaf, one of the things that you may have noticed and I certainly have and it's counterintuitive to think this way I think, is that there's a lot of focus on AI. And I think that that is warranted. On the other hand, have we pulled so much of our focus away from some of the basics that seem like we need to be able to go in and deal with the discipline and grit work that isn't all that sexy and new, but needs to be done such that the attack that you're talking about isn't so damned easy?

Assaf Keren: Yes. Yes, thank you for that. We need to say this more. The best solution-two good solutions for AI attacks: One is, minimization.

David Moulton: Um-hmm.

Assaf Keren: If it doesn't need to be on the Internet, it shouldn't be on the Internet. If it doesn't need to be on the endpoint, it doesn't need to be on the endpoint. If it doesn't need to be in a package in the source repo, they shouldn't be there. And we have been in a world where we're maximizing things. We need to minimize. We need to reduce the attack surface to a point where the attack is not possible and not get to the point where we're trying to defend a growing attack surface.

David Moulton: Um-hmm.

Assaf Keren: And the other is baseline boring architecture. We need to do identity right. We need to do data right. We need to do scoping right. We need to do network segmentation right. We need to do recovery, BCP right. And these are hard things and we've been glossing, as an industry, we've been glossing over them with mitigating controls and good enough, and all of the-not-there is no good enough anymore.

David Moulton: Um-hmm.

Assaf Keren: Because what we're doing is even worse than attackers using AI. We're putting AI on top of broken mechanisms. And so, we're putting a nondeterministic engine on top of a broken deterministic architecture that can go and do whatever it wants, and our ability to control a nondeterministic engine is very, very low right now. Until we get into the world where there is runtime security for the AI solutions that we provide to our customers, there has to be very strong architectural guardrails on the bottom. And if we put on an AI agent on that identity infrastructure, it will find a way through prompt injection, through other means, through-I don't know, to get to the data that it wants to get to or the attacker wants to get to using our own bad infrastructure. So, completely agree with you. There is in my mind, a whole resurgence of being brilliant with the basics.

David Moulton: Yeah. I mean, sometimes this idea of if everyone is going to zig, it's time to zag. And a lot of-a lot of oxygen is used up worrying about a version of a problem that we see coming and then we're distracted from the problem that we have, you know, this security debt, technical debt, whatever you want to call it, where that's just sitting there. And I think you said publically that when you bring an AI tool into your environment, you have less slack. I think that is what you were just describing.

Assaf Keren: Yes.

David Moulton: And that you can't skip the steps. And I know you mentioned some of them, but I want to hammer home on this, what steps do most organizations skip and which one of those exposures do you think is going to end up being the one that haunts organizations the most?

Assaf Keren: Identity. Identity is probably the hardest especially in product, with your SAS company or if you're, even a consumer company; identity is probably the hardest piece. In a lot of places identity was homegrown years ago, customer identity was homegrown years ago. And there are best practices there and there is not a lot of people that know how to build it right.

David Moulton: Um-hmm.

Assaf Keren: And if people miss identity, that's the baseline structure for everything else.

David Moulton: Years ago, a CISO told me that there are three rails and the third rail is identity in any CISO's job. And I want to say data was one piece and.

Assaf Keren: Data, yeah.

David Moulton: Network was another, but like those were not the ones that if you touched them, the business would zap you. It was identity, because you had three, four different identity systems. Some of them worked for the executive owner, but it didn't all work together and they certainly didn't work well for security. And it seems like now we're at a point where that being the third rail as a mental model for a security leader, has to flip around. It has to be the first thing that you're looking at and getting right and getting right really quickly, or you.

Assaf Keren: Yeah.

David Moulton: Remain exposed.

Assaf Keren: The second is data, by the way, like.

David Moulton: Yeah.

Assaf Keren: A very close second.

David Moulton: Yeah.

Assaf Keren: But identity. I would go after identity first.

David Moulton: So, I know a lot of security leaders are being asked to make decisions about AI risk faster than guidelines can be issued or updated as things change. What does good judgment actually look like in this environment where, you know, the threat intelligence on AI is drafted, maybe it's being written.

Assaf Keren: Yeah.

David Moulton: Alight? It's come in over the weekend out of a user group who felt the need to put something together, right? This stuff is not tried, true, tested, public comments are done, right? It's really fresh. How do you operate that environment?

Assaf Keren: It's also changing very quickly.

David Moulton: Yeah.

Assaf Keren: It's also changing very quickly on an ongoing basis. So, what you've done a week ago can change next week, because new model, new capability, new thinking. I think that going back to basic principles is important. What are we trying to solve? Where are we trying to solve it? I think that being realistic about the risk is important and understanding-because we as a community, we have a tendency to over exaggerate risk, because we don't understand it, because it's changing so fast, because it's this new thing and there is hair on fire, people running around to an industry that's deep in thought.

David Moulton: Um-hmm.

Assaf Keren: And wait to fight it. Also, we need to understand that it's not going away. I know security leaders that in 2023 said, "Yeah, yeah this will be a fad, it will go away." No, no it's not going away. Like this is part of the future. We need to lean into it and not the other way around. I think when we try to block the business from using AI, we're creating more risk than value. We need to sit and create mechanisms in place to allow the business to use AI in a secure and reliable manner knowing that we're taking risks, but we need to enable the business to use AI. We need to build guardrails around that. Now, there isn't a lot of enterprise software that is there yet that is doing all the things that we need to do. So, we're going to need to do a mix of vendors, or a mix of internally build stuff, and mix of externally build stuff, and open source and stuff like that. But, building the guardrails to make us feel good about, or better, not good, about where we are from risk perspective is important. So, in a lot of places what I'm hearing from peers is that use AI to use AI, because the board said use AI which is a wrong framing for that conversation. You use AI to get to an outcome.

David Moulton: Um-hmm.

Assaf Keren: That is a better outcome with AI. And so, I think what we've managed to do internally is say, "Hey, we want to do these things. We want to automatically triage all of our SOCs with AI, or we want to do vulnerability triage with AI, or we want to do questionnaires, customer questionnaires for AI, to free up people so that our people can do bigger and better things." Those are really important outcomes. But I don't feel the push on "Oh, just use AI for AI sake."

David Moulton: Um-hmm. It's kind of like you said [multiple speakers].

Assaf Keren: Which is unfortunate about, yeah.

David Moulton: You know, don't go for the next job title, because it's the next job title, right? Like it doesn't make sense to.

Assaf Keren: Yeah.

David Moulton: You know, apply that logic on using AI especially when it is a tool for an outcome, not the outcome itself. Let's just say that a security leader is listening right now, and they are not sure how exposed their organization really is, maybe they heard you say that we are over-indexing on the risks, and hopefully that's true for them, but they're trying to figure out where to start. What's the first thing they need to do?

Assaf Keren: So, two lenses to this.

David Moulton: Um-hmm.

Assaf Keren: This is the-the internal AI exposure. People are using AI and/or products using AI within the constraints of their organization and then the attackers. When we talk about the internal piece is, get an understanding of usage. Because you're going to trust some vendors and you're not going to trust other vendors. And this is very similar to what we've had when the cloud came in.

David Moulton: Um-hmm.

Assaf Keren: You're going to need to make decisions on which vendors or which hyperscalers, or AI vendors you're going to trust, or which SaaS companies you're going to trust and which SaaS companies you're not going to trust, or which Opscales you're not going to trust, because you don't think they have the right controls in place or they have the right structures in place, or their responsible enough, not responsible enough. There are a lot of different ways to skin that cat.

David Moulton: Um-hmm

Assaf Keren: But, understanding usage is extremely important. And starting to build guardrails on that usage, and if you're building your own models, if nothing else, you can go do ISO 4201, but if nothing else, look at the NIST AI Risk Management Framework and start looking how you build your model inventory and how you build your model risk scorecard, which is extremely important and try to publish-at least publish it internally.

David Moulton: Um-hmm

Assaf Keren: So, the people understand the different risks in using different models, bias, ethics, operational risks, not operational risks; these are baseline things. So, that's what I would say for the internal risk. For the external risk, attackers using AI to attack companies, I would ask where are the places where you can be much faster if you utilized automation? Go and automate with AI or without AI, I don't care, but go and automate. Where are the places where you can be better if you are reducing attack surface and you can do it fast? Go and do that. Start building both speed and reduction of the attack surface as soon as you can, because those are the things that are going to save you. The other things are.

David Moulton: Yeah. Attack surface diet.

Assaf Keren: Long-term. Attack surface diet, I like that. I'm going to use that.

David Moulton: Yeah. Yeah, you got to.

Assaf Keren: You do.

David Moulton: I'm stealing brilliant on the basics from you.

Assaf Keren: Yeah. Yeah. Well, I stole it from a guy called Sri Shivananda that was my boss at PayPal, so let's go.

David Moulton: Yeah.

Assaf Keren: I think he stole it from a guy called [foreign name spoken] Head of RSA. I do a weekly post to my external leadership team, so "attack surface diet" is going the next one.

David Moulton: I mean, it's good practice and it's maybe now required practice, you know?

Assaf Keren: Yeah.

David Moulton: Get your attack surface on a healthy diet, you know, shrink down man.

Assaf Keren: Attack surface calorie counting.

David Moulton: Yeah, get that beach body attack surface.

Assaf Keren: Yeah.

David Moulton: Before summer.

Assaf Keren: Yeah. Summer is coming. It's.

David Moulton: Yeah. Yeah, there you go. AI summer is coming quick. I want to end on hopefully a positive note, right? You've written this book about what it takes to lead in this field long-term. You're watching everything that's going on with AI right now. Is there anything that gives you confidence that defenders may come out ahead in this era?

Assaf Keren: Yeah. To steal a quote from Phil Venables, "I'm a short-term pessimist, long-term optimist." I think that the next couple of years are going to be either hilarious or daunting depending on who you are. But we, I think, in the end, this technology is so exciting that we're going to be able to do something that we've been trying to do for years and years and years unsuccessfully, which is to free up people to do people work and not do manual labor tasks. And we're going to have-we're already at the deficiency of the amount of people in the profession. We're going to-- - and people are burning out, because they need to handle incidents of a day-by-day basis, so copy/paste, answers into questionnaires, or do third-party risk management things that don't bring value but are a part of the process. And we're going to be able to automate a lot of these processes and reduce the amount of time people are actually doing stuff like phone triage, or incident triage, and have them work on the larger picture that it's going to be much easier-not easier, it's going to be much more exciting to be a security professional in two years than it is right now, because you're going to work on big picture stuff more than you are today. And I think that.

David Moulton: Yeah.

Assaf Keren: That's exciting and I think we will get ahead of the curve. We need to adopt the technology as fast as attackers; that will not happen, so that's why we have two years of catching up and I think we'll catch up in the end.

David Moulton: So, I've been trying to think about the future and what it might look like, and I found this image of a 1920s potato farm and there were laborers digging and working in the field. And then I contrasted this with a vertical hydro farm, and they are as far apart world-wise as jobs go; they're both farms. But I do wonder, are we in a moment where we are laboring and digging and trying to keep that potato farm going? And we're going to transform into one where it's a controlled environment. We have incredible productivity. Some level of, you know, small team being able to handle that vertical farm of the future for security. And I'm hopeful for that, right? And I look at the potential, but I think that the first thing that has to happen as we think differently about those basics as fundamentals, we go on that attack surface diet, we put together, you know, a different model that allows us to control the environment and flourish rather than try to work harder and longer and not have much effect. So, I'm hopeful and I like to hear that you think that it's going to be two years and we're through it given the time lately. You know, maybe it's two, maybe it's two months, maybe it's two years?

Assaf Keren: Every-yeah, we'll see.

David Moulton: We'll see. Assaf, thanks for the great conversation today. I really appreciate you sharing your perspective on AI in the wrong hands, but also letting me get a glimpse of your path, your art, your wife's art. Folks, Assaf's written a new book "Lessons on the Frontline: Insights from a Cybersecurity Career." It's published by Wiley. It's out and available. We'll have a link the show notes, along with the blog that you mentioned earlier, and I appreciate you coming on "Threat Vector" today and having this conversation with me.

Assaf Keren: Thank you very much. Appreciate it. It was a lot of fun. [ Music ]

David Moulton: That's it for today. If you like what you've heard, please subscribe wherever you listen and leave us a review on Apple Podcast or Spotify. Those reviews and your feedback really do help me understand what you want to hear about. If you want to reach out to me directly about the show, email me at threatvector @paloaltonetworks.com. I want to thank our executive producer Michael Heller, original mix and music by Elliott Peltzman. We'll be back next week. Until then, stay secure, stay vigilant. Goodbye for now. [ Music ]