The phishing kit that thinks like a human.

Dave Bittner: Hello, everyone, and welcome to the CyberWire's "Research Saturday." I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems, and protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us. [ Music ]

Piotr Wojtyla: We've been seeing a number of attacks that was leveraging accounts or were originating from Gmail accounts, from legitimate infrastructure, and had a very nice and polished AI-generated content. And we've been on the lookout for, is there any particular tool, is there any particular capability, is there any particular new thing that the attackers developed that might have-- support that particular types of, you know, flavor of attacks. And that ultimately is how we arrived at this particular tool is through research and continuous monitoring of what the attackers are doing.

Dave Bittner: That's Piotr Wojtyla, Head of Threat Intelligence and Platform at Abnormal AI. The research we're discussing today is titled InboxPrime AI, new phishing kit fueling scalable AI-powered cybercrime. [ Music ] Well, for listeners who haven't seen your research, how would you describe InboxPrime AI?

Piotr Wojtyla: Well, ultimately, you can think of it as a tool that allows you to send email attacks or craft email attacks. And it has-- it's a very point-and-clicky. So it has a very friendly user interface, a very intuitive interface. It pretty much looks like legitimate marketing software. But the kicker, or the big selling point, is that it is AI-enabled. So it allows you to customize the content of the email, whether it's subjects, whether it's the content of the messages themselves, with AI, which makes those emails look extremely professional. It makes them look-- obviously, they're polished, they are grammatically correct. So, everything that AI has to offer. And on top of that, it has a number of different templates and parameters. So you can adjust the tone, you can adjust the language, you can adjust the industry, you can adjust the theme. So it has pre-built themes within the tool itself. And most importantly, it also operates as pretty much as a legitimate user, or as if it was a legitimate user coming from a Google infrastructure. So to send those email attacks out, it leverages the Google Gmail infrastructure and the accounts that are Gmail accounts. So ultimately, it's a mix of the legitimate infrastructure and AI content that is pre-generated in this very intuitive interface that makes pretty much crafting attacks extremely, extremely easy.

Dave Bittner: Yeah, one of the striking aspects of this is, as you say, it operates through Gmail's web interface. Why is that such a meaningful design choice here?

Piotr Wojtyla: Well, when you think of email security, or one of the ways how different companies and organizations try to combat the problem of email attacks, it's pretty much to ensure that those emails are not spoofed, that they're coming from the sender who claimed-- who actually said that it's going to send that email, or from the person who is actually associated with the organization that they claim to be associated with. So, for that, we have pretty much three particular records, so SPF, DKIM, DMARC, that allow you to pretty much verify that the email is coming from legitimate infrastructure, from a specific organization, from a specific sender. So if you leverage Google, if you leverage trusted infrastructure such as Google, those headers or those checks will pass. And that's one of the ways how vendors and security solutions check for the security of the incoming email. So it's one way to kind of bypass those very simple checks, let's call it that way. And then the other thing is that obviously, that creates a lot more trust and a lot more legitimacy with the recipient. Sending an email from Gmail account is probably better than, hey, I want a ponu.com account.

Dave Bittner: Right. Well, let's talk about the AI component. So what role does AI play in generating these phishing emails themselves?

Piotr Wojtyla: Yeah, so ultimately what we have here is we pretty much have a pre-built, you can think of it, not even templates, but pre-built parameters that AI takes and then crafts specific emails based on your need. So if you want to send an email that is coming from an HR person, if you want to send an email that is focused to a specific recipient, who might be a payroll analyst because you want to attempt a payroll fraud, you can specifically call out what type of email you want to create, what tone of that email should be, whether you're an expert, whether you're a beginner. You can-- you know, you can create urgency, you can be very specific with your needs, and ultimately, what the AI component of it will create that entire content of the email for you. So, pretty much within a number of clicks, you have the entire content of an email ready. And also, there's an ability for you to templatize that. So if you want to create different variants of the template, if you want to create different types of that email because you want to send to different recipients, there's an ability for you to include certain parts of that email as templates, and then AI will take care of the rest of the generation of that content and really make sure that it's exactly fit to your needs and what you want it to be before that's being sent out. So that entire concept of creating the content is AI-generated. And then the intuitive interface and the Gmail infrastructure that is responsible for sending, that's pretty much just the orchestration of that tool.

Dave Bittner: So does this make it much easier for the attackers to create a high level of polish compared to older phishing operations?

Piotr Wojtyla: Oh, 100%. I think one of the key indicators, especially in the world of business email compromise back in the day, was really looking for typos, you know, grammar errors, or even things that just don't sound right. Like, I'm not a native English speaker, so when I speak, you can probably pick up on little things when I say that just don't sound right. Like, an English speaker might not say in a specific way, or the way I put words together might just-- it just doesn't make sense. So, it's very-- it's a similar concept applies to those emails. Back in the day, you would have some of those emails that would just not sound right, or they would have some errors, they would have some mistakes. This completely removes that layer of ability for users to spot, hey, there's something off about this email. But not only that, it creates the polish, it creates the professionalism, it creates pretty much the perfect lure for what the attackers are after. Because you can really adjust that hook, you can really adjust that email to whatever your need is, and make AI do its magic, and really polish it in a way that those emails are a lot more polished, are more slick, and ultimately can gain a lot more trust from the recipients.

Dave Bittner: I have to say that InboxPrime looks more like a commercial SaaS product than a crime tool.

Piotr Wojtyla: It does, it's quite interesting how some of those tools really focus on the user experience. But at the same time, like when you think about it, it really lowers the bar of entry for anyone. So the price point is not that scary; you can get that pretty much for $1000. But also, back in the day, a lot of-- before the age of AI, having the knowledge and the know-how to create the infrastructure, the underlying infrastructure, both to be able to send the emails, to be able to orchestrate the framework that would actually be able to send emails from different accounts through different servers, and then having templates and content, and then having the landing infrastructure. There's so many different components to a successful phishing operation, and that requires a certain amount of skill, or access, in terms of you buying those tools from someone else. And this is a perfect example of that bar being so low now that you can literally be someone who has no idea how any of this works, and then open a tool, click a few buttons, and pretty much you're able to deliver a phishing campaign and attack most organizations around the world. [ Music ]

Dave Bittner: We'll be right back. [ Music ] Can we touch on the scalability here? I mean, this has bulk management tools and things like proxies and templates, as you mentioned. This is designed to be able to fit the needs of a lot of different types of operators.

Piotr Wojtyla: That is correct. Well, one thing that is, you know, worth keeping in mind is that when you're a cybercriminal, you want to obviously perform your operations probably in a way that won't land you a nice, you know, place in jail somewhere. So you want to make sure that your operational security is up to a certain standard. And you want to make sure that you don't expose your real IP address, your real location. So having those basic capabilities in place that allow you to proxy your access to certain Gmail accounts for additional hops and layers, that is one of those ways how you can obscure and hide yourself. Not to mention that obviously you can run that tool from some sort of VPS somewhere in the internet and probably sit behind some sort of Tor browser, etc., etc. So there's a number of ways where you already have pre-built capabilities in the tool itself that can allow you to obscure some of your origin and some of the origin of your attacks. But also, to your point, there's additional things in place that allow you to also adjust the quality of your emails. So there's like a quality assurance capability that pretty much looks at the message and ensures that like, hey, this could actually be flagged by a spam filter or this could actually be flagged by a potential security filter. So let's adjust this wording, let's adjust this sentence, let's adjust this even more so it doesn't hit on those very obvious, you know, static signatures that some of the email security vendors might have. And that also allows for scale. And not only the scale of the attack, but also allowing you to adjust it specifically to your needs. And I think I also mentioned that the final piece, when it comes to the scale, is the template variation. So you can pretty much adjust the templates with specific tags and pretty much specify that like, hey, within that tag, here's a number of different variations I want you to iterate on when you create the emails. So each email kind of comes out with a custom take on it, a custom twist on it. And we already touched on the Gmail aspect of it as well. So you can pretty much-- you have, you know, pre-configured email addresses, and that also allows for scale because it doesn't come from one specific address, it originates from multiple different addresses. So ultimately, you can send as many operations as you want and they'll be as custom as you want them to be.

Dave Bittner: Now, one of the things you pointed out was a shift in their business model here, that this was originally sold as a subscription service, but they made some changes. What's going on there?

Piotr Wojtyla: Yeah. So originally, like you mentioned, it was sold as a subscription, so there was an ongoing monthly payment versus a one-off payment. Also, in subscription-based services, you usually don't get access to the underlying code, and you only get access as a user to the platform or to the tool itself. So obviously, you know, we don't sit in the mind of the attacker, so we can only speculate about what might be the reasons for that. But often, what that might suggest is a certain level of market maturity where there's enough confidence in the market and in the followers and in the customers that you ultimately-- you know, there's a certain level of your brand and your tool being established that allows you to make that shift. Also, another thing that, you know, might be a reason for that change is pretty much, you know, democratizing that access and lowering that bar and pretty much reaching other customers that were previously not available. Because some of the threat actors don't want to leverage tools that they don't control, and by giving access to that source code, you pretty much allow them to control fully the tool itself, having insight into the code, so they can-- there's a level of transparency to that. And finally, there's also an ability for those threat actors to then customize that code even further. So you can then take that tool in whatever direction you would like and, you know, create new modules, create new capabilities. So, ultimately, that can also-- one of the big reasons for that change might be to drive more revenue from the type of buyers who would not be the typical buyers for the subscription kind of models-- type of models.

Dave Bittner: Do you think that adding AI in this way and increasing the sophistication of this tool, has that changed the economics of running phishing campaigns?

Piotr Wojtyla: What I would say is that we're at a very interesting tipping point, I think, when it comes to phishing and email attacks, because we've already seen a number of use cases with many different AI tools. Like, we've seen certain, you know, dark LLMs and dark chatbots that were built specifically for cybercrime. We've seen some use cases where you can hijack or manipulate the legitimate LLMs, the chat GPTs, the Claudes of this world, and pretty much tell them to give you or create you a phishing page or create you a phishing email. And you can bypass those safeguards that those models have already in place. Ultimately, what I'm kind of going on a little bit of a tangent here is that we see attackers adopting AI, and it should not come as a shock to anyone who's been in security because attackers are extremely innovative, they're extremely creative, and they will leverage whatever the next best thing is that will help them monetize on their campaigns. So I am not surprised that AI and the usage of AI, whether it's this tool or whether it's other tools that the attackers are using, is present and it's only going to get worse from here because it creates so much more efficiency, it creates a much higher quality, and it just makes pretty much something that was previously-- might have been separated between many different groups or many different people within one team that were operating under a specific threat group, you can pretty much perform a lot of actions in conjunction with AI as a single individual today. You can create tools, you can sell those tools. AI allows you to monetize in a completely different way that I don't think was previously available to a lot of people. So right now, your imagination and your creativity is really your own limitation. Because you have a companion that is pretty much going to do whatever you're going to ask it to do if you have a good understanding of what you're trying to create. So, to kind of answer, you know, this was a long tangent, but to ultimately answer your question, I do think that like leveraging AI, whether it's in this particular tool or in general by threat actors, really enables them to monetize and make attacks even more efficient.

Dave Bittner: Yeah, it really strikes me that, you know, for the past couple of years, we've been talking about how these tools would be coming, you know, and everyone needed to prepare and brace themselves that these types of tools were inevitable. And now we're at the next step, where these tools are readily available, they're easily available, they're affordable. And so that's the world we're in now.

Piotr Wojtyla: Exactly. And we're, you know, just waiting for the next year's RSA, where we're going to see that every company is selling some form of AI agent. And usually, that is just a certain level of abstraction and representation of where we are as an industry. So we're inching towards, you know, more autonomous capabilities. We still have an assisted Copilot-type of attack tools right now, but we're inching towards the world where those workflows will be fully automated, will be autonomous. And that is going to be also an extremely interesting point in that phishing evolution where we're going to start seeing attacks that might be fully automated, that might be actually done through a fully automated workflow and through AI agents. Or some flavor of AI agent implementation by the threat actors. So that is also going to be extremely interesting to see.

Dave Bittner: So what are your recommendations then for folks who are tasked with defending their organizations? What's the best way to defend yourself against these sorts of things?

Piotr Wojtyla: Yeah, so honestly, like one of the things that I've been thinking about coming into this conversation, but also, you know, being part of many different conversations, is just seeing some of the attack trends and some of the changes in what attackers do. I feel like we're really at the stage where you cannot trust the email content or the email that you're receiving. And ultimately, what you can and should trust and kind of like what that shift should happen in your mental model is from not trusting the email and then focusing and trusting on the verification process. So, really thinking of like, okay, what is the way in which I can verify that the information presented to me is actually what it's claimed that I see in front of me. So when I think about, you know, business email compromise attacks, if you're talking to the same vendor every day, and let's say the threat actor takes over the account on the vendor's behalf and then pretty much tells you, "Hey, I just changed my bank details. Can you please update that in your system?" If you see that email and you just go and action that, that is likely going to expose you to an attack at some point. However, if your mind shifts from this, okay, I cannot trust the information that is in front of me, but I can trust the verification process, that can be as easy as you picking up the phone and calling your vendor and saying, "Hey, I just got this, you know, what do you need me to update?" Or any form of, there's a process component, there's a human component, and there's a technological component. Like with every other problem, it's exactly the same. So what defenders can do is they can focus on pretty much every aspect of those three components. And that starts with process, that starts with people, and obviously, like there's a technology that can also support you. So I can speak to each and every one of those individually. But that concept of not trusting the emails and not trusting the email content and really trusting the verification process, allowing yourself to have a process in place, whether it's your training, your ability to look at the email through the lens of like, if this was an attack, what should I look for? Do I have a technology that allows me to verify if this is an attack? Do I have a process in place that allows me to verify that this is an attack? That is something where we, as organizations and as defenders, can really focus our attention and hopefully make ourselves and others more secure. [ Music ]

Dave Bittner: That's Piotr Wojtyla from Abnormal AI. The research is titled "InboxPrime AI, New Phishing Kit Fueling Scalable AI-Powered Cybercrime." We'll have a link in the show notes. And that's "Research Saturday" brought to you by N2K's CyberWire. We'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly-changing world of cybersecurity. If you like our show, please share a rating and review in your favorite podcast app. Please also fill out the survey in the show notes or send an email to cyberwire@n2k.com. This episode was produced by Liz Stokes. We're mixed by Elliott Peltzman and Tré Hester. Our executive producer is Jennifer Eiben. Peter Kilpe is our publisher. And I'm Dave Bittner. Thanks for listening. We'll see you back here next time. [ Music ]