Your AI sidekick might be a spy.

Dave Bittner: Hello everyone and welcome to the CyberWire's "Research Saturday". I'm Dave Bittner and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems, and protecting ourselves in a rapidly-evolving cyberspace. Thanks for joining us. [ Music ]

Or Eshed: I think what was interesting here is the scope and motivation of the attacker behind this, which is a very well-coordinated and orchestrated campaign, that's only purpose is actually to steal ChatGPT accounts.

Dave Bittner: That's Or Eshed, Co-Founder and CEO at LayerX Security. The research we're discussing is titled, "How We Discovered a Campaign of 16 Malicious Extensions Built to Steal ChatGPT Accounts". [ Music ]

Or Eshed: Within LayerX, we're a browser security company. We have millions of browsers we secure, but also a collaboration with Google. So actually we're one of the sandboxes Google is using. So I have a pretty good database, probably the largest in the world. One of the things that we do on our database is conducting threat hunt campaigns. So we're taking TTPs or do attributions. So basically there are all kinds of extensions out there. Malicious extensions are not behaving the same way malware works. So there are different ways to analyze them. We've built our own platform for that. We call it the LayerX Malware Lab, in which we find clusters of extensions that seem similar to one another. Within that scope, we've detected a first malicious extension within this campaign. Afterwards, the attribution is happening automatically. What's interesting was to see a very coordinated campaign that's aimed at stealing ChatGPT accounts. So unlike other methods to discover, we are trying to get things when the blast radius is relatively low. So if an extension is in infancy, what it means is once we detect some sort of a mechanism that the malicious extension is using, Since we have visibility into the entire marketplace through Google, we can catch extensions as they come to the marketplace and not once they infect the user browser. So you basically have to get to the marketplace then to do the infection. I think what was interesting here is the scope and motivation of the attacker behind this, which is a very well-coordinated and orchestrated campaign. That's all its purpose is actually to steal ChatGPT accounts.

Dave Bittner: Well, let's start off there. At a high level, what was it that these extensions claimed to do, and what were they actually doing instead?

Or Eshed: So, they claim to be productivity tools for AI, and that makes sense because of, A, they want to make sure that they hit users with extensive ChatGPT usage. Secondly, they inject a lot of code into ChatGPT, so that also provides evasion within the ChatGPT, within the marketplace sandboxing capabilities. So you just want to make sure that the fact that they inject a ton of code to ChatGPT goes across as genuine, credible. And it's not really clear what was the benefit, but they managed to get significant distribution. Once they are there, they are stealing tokens used for authentication. So they claim to be something that used to export data or images, providing timestamp displays, all kinds of very basic functionalities you don't actually need an extension for. But eventually they do advertise themselves as something that automate them. And behind the scenes, stealing tokens used for authentication to ChatGPT.

Dave Bittner: How did you realize that this wasn't just a single bad extension, but that this was actually a coordinated campaign?

Or Eshed: Actually, that's the easiest part. The hard part is the catching the first one. Once you catch the first one, the next ones to follow are pretty easy. So, we look at a couple of things. We look at code behavior and code repeatability. Think about yourself. What's the most expensive thing you have in the world is time. So once you've developed something that works, you try to replicate that. You try to automate it. So basically they were copying and pasting their own code into a bunch of different extensions. Aside from that, they use the same visuals, the same fav icons on the extension, and even the same domain to register them. So, there were a couple of connections between all those extensions on the ownership level, on the visual level, and on the code level, which is really a smoking gun. And all of them are attributed to the same attacker.

Dave Bittner: Well, let's go through this together. I mean, what actually happens when someone installs one of these extensions? What does it do inside the browser?

Or Eshed: So an extension has visibility to a lot of things that happen within the context of a web session. So for anyone that's hearing, once you go into ChatGPT, you're already signed in. How does ChatGPT know it's you and it's not, let's say, Dave? At the same time, the ChatGPT app is doing that based on a cookie stored in your browser or some sort of a token that's being cached in the browser memory space. All of those items are visible to any extension. So any extension with visibility to the ChatGPT domain is able to see those data types. So the extension is basically copying all the different attributes that are used by ChatGPT to recognize the user, the cookie, the tokens used by the browser, the screen resolution, and even the browser version, everything to create basically a replica, an identical twin of that browser owned by the attacker. So the attacker can just log in into the app. Actually, the attacker doesn't have to log in because they are instantly validated by ChatGPT. They don't even have to log in. They just go in and then they can just steal conversation history and fetch data.

Dave Bittner: So as far as ChatGPT is concerned, this, it thinks that it is the user of the stolen token.

Or Eshed: It just mimics everything of the victim in a way that the attacker owns. And the attacker can just sign in and have visibility and access to everything owned by the user.

Dave Bittner: Well, help me understand here, because my understanding is this doesn't exploit a vulnerability in ChatGPT itself. It's this token vulnerability. Why does that make this harder to detect?

Or Eshed: Have you ever seen some sort of a '90s action movie in which the thieves create some sort of a replica of a housekey? It's pretty much the same thing. They just create the replica of the key that you use, they get in, identifying as you, and then they can steal any data you uploaded. In reality, it can be done on any site. What the attacker has to do is really know ChatGPT and where ChatGPT hides his secrets. That's not really hard to do. From that point on, it's becoming a very easy task. Actually, the complex part is getting the infections. An extension has visibility into everything identity-related within the browser.

Dave Bittner: So my understanding is that right now the download numbers for these are relatively small. Can you give us a sense of the scale of this problem?

Or Eshed: Well, that's a good question. The way LayerX works, we don't wait for large distribution to do the takedown. We try to do the takedown as early as possible. So this campaign was blocked in relatively low numbers, but with high motivation. I said it historically campaigns of that sort managed to get to thousands or tens of thousands of infections per extension. What attackers typically do, they use rogue advertisements to get installations and all kinds of evasion techniques. So sometimes they will actually add some sort of a legitimate functionality to the extension, or they will buy an extension on the marketplace that already has infections. Interestingly, an extension owner has visibility into who owns the browser. So once I install an extension, let's say I'm using an extension using my work device. My work device is creating some sort of flagging to my browser that says that this browser is managed and is attributed to a domain owned by my business. So actually, an extension owner can see who is owning the extension and actually understand whether this is data that's owned by a consumer, and then it's really hard to monetize on that, or whether it's owned by a business, and then they can actually do some sort of a ransomware or something else. I'd say that the holy grail from an attacker standpoint, or I'd say the knockout, will be companies that actually have a ChatGPT corporate account. So they have some sort of an on-prem or internal ChatGPT. And by getting access to one account, they can actually steal the data of all the organization. If that makes sense, creating some sort of an intrusion, that's, you know, a game changer for the attacker. [ Music ]

Dave Bittner: We'll be right back. [ Music ] What are your recommendations, then? I mean, when we're talking about browser extensions, how can organizations vet them to make sure that they're not going to have these sorts of problems?

Or Eshed: I'll use the cliche and say it starts with visibility. If you don't know what exists in your environment, which browsers are there, which extensions they have, you're probably in a bad spot. It's one of the most effective attack techniques. A couple of years ago, according to Mandiant, the third reason in terms of scope for account takeovers and intrusions on the identity level. It's also very low-hanging fruit for an attacker. So we need to have visibility, but the visibility has to be continuous because attackers are changing extensions on the fly. An extension can be born benign and become malicious over a while. So I'll call it a Shawshank Redemption, you know, process of taking, you know, it's kind of like, you know, digging a tunnel day by day. The attacker is building an extension, adding a little bit of malicious code daily, until they get to a good enough distribution, and then they monetize. So they're really aware of the limitations of allow-list, block-list approach. Eventually, you need to know which browsers you have, which extensions are there, and also to understand which identities are exposed to them. So not all identities are at the same risk. I said that the low-hanging fruit is understand how users use browsers in the organization. Users are able actually to import via agentless sign-in their personal browser setting into the work device, including all the extensions they have. So you can actually import a bunch of malware instantly into your work device. I think once you understand that and you have a basic inventory, you define what's a reasonable use, what's not a reasonable use. You can get to a pretty sweet balance between risk and productivity. I think one of the challenges is that historically, who used browser extensions? So historically, the browser extensions that were really corporate legit or ad blockers, password managers, Grammarly, things of that sort. But today you have, like, a million AI extensions out there, and every user says they must use them. And it's really becoming a headache for IT teams to approve or vet extensions over time. I think visibility, context, continuous risk analysis can get you to blocking something that's probably more common in your environment than actual traditional malware.

Dave Bittner: Are we looking at behavioral detections here of trying to keep an eye on what these extensions are trying to do?

Or Eshed: So you can't actually do that without being deployed in the browser, unfortunately, but to the very, very least, to the very least, understanding what's there. So I'll give you a point. So let's say I'm, as a CEO, I have the Salesforce extension on my browser. You know how many extensions on the Chrome marketplace are Salesforce something? They have Salesforce in their name? Hundreds. And ChatGPT extensions, thousands. So no one really says what's a real one, what's not a real one. You need to actually check that. Is this a real Salesforce? Is this a real ChatGPT extension? Those are very, very basic hygiene things you need to do on your environment. So you need to have visibility into everything about those extensions and be able to block them based on risk, context, reasonable usage, things of that sort.

Dave Bittner: Yeah, it seems like an uphill battle here. As you say, the numbers are not in the defenders' favor, it seems to me.

Or Eshed: Well, unfortunately it is, but it's a brave new world. Eventually the traditional operating system is not as interesting as it used to be, even though we're going back to a device-centric world. But what's really interesting is what's happening on top of the device. AI applications, browsers, IDs, this is where employees spend most of their time. Historically, I remember myself as a junior security analyst with more hair on my head, and everything was around files. Is this a good file? Is this a bad file? Is this data-rich file? Is this, you know, whatever. And now everything is applicative. Everything is dynamic. Basically, extension is agentless. In order to understand what it is, you need more context and you need to really change the way you think about security. That actually agentless is more powerful and more risky to your organization. Actually, it's agentless malware.

Dave Bittner: So how do you recommend that security professionals strike that balance? I mean, we can see that some of these extensions have utility and they do help people do their jobs better, and yet we have this risk here.

Or Eshed: So assuming that the question is how to do it, you know, avoiding to buy a tool on the browser level, the DIY method would be to restrict which browsers are approved in your organization. And then you need visibility into the different plugins. Chrome, Edge, and soon Firefox have enterprise flavors, so they have management capabilities. Other browsers don't have management capabilities. You need to build it yourself using MVM or some sort of a security tool that you may have to buy. Once you do that, you need routinely, at least once a week, to audit all the different extensions, understand what's happening with their permissions, code, sandbox them, and to apply risk-based classification. Eventually in real life, the road bump you'll hit will be that not every extension will say, Hey, I'm malware. It will say something like, "Amazon coupon code". And then a security architect would not want to get in a face-to-face battle with some sort of an employee, whether they should or shouldn't have that. So understanding what's reasonable usage on your environment, on your device's fleet, that's key. Because if you decide that you don't waste time on things that are not work-related, just avoid having all that other stuff, all that crap. And if your culture says that everything is allowed, the user is the champion of the organization. You need to really scan nonstop everything in your environment to understand what's risky.

Dave Bittner: Yeah, it really is striking a careful balance.

Or Eshed: And what you could do, actually, at LayerX we have a free extension media. So about a big chunk of our database is actually exposed to the broad audience. If you want to invest and scan and get for some sort of a risk scoring, you actually do it on the LayerX site. Search for LayerX Extensionpedia and that database is a combined database of Google and ourselves. So, you know, it's two startups. One of them is 30 years old and over a trillion dollars in worth and LayerX, but we have the largest database in the world for browser extensions with risk scoring. You can do that and then you can understand what's, you know, what's going on in your environment.

Dave Bittner: Now hold on a second, or are you suggesting that people install one of your extensions?

Or Eshed: I mean, everyone is welcome to be a LayerX customer but I think, I was on the other side. I think it's my, I need it as an entrepreneur in cyber security, I need to always talk about what's the basics. Because, you know, it's a part of the community to be able to share. And it was important for us to provide the basics of extension security for free for the entire world. And eventually we built great relationships with customers, so I'm not, you know, shy to say that. We're happy to give away some for free. Eventually, we understand that that's our way to prove credibility. And many of those organizations then are interested to move on with us. And, you know, sometimes I go to conferences and I meet people saying, you know, they tell me, you know, I built an extension security framework for free using your Extensionpedia. And then I'll tell them, you know, it's great. How much time did you spend on that? And apparently they spend a lot of time. And then eventually, you know, they do try to automate and they do reach out and they do engage with us. So I feel very comfortable with where we are.

Dave Bittner: Yeah. All right, well, Or, I think I have everything I need for our story here. Is there anything I missed, anything I haven't asked you that you think it's important to share?

Or Eshed: I think one thing is the timing, the why now with AI. Historically, companies were using all kinds of tools, but, you know, every risk level has kind of like a long tail and a big mass. So the big mass is always really managed. So think about SaaS security or identity security. It's always said that the big mass is already secure by design, then you have a long tail. With AI, I think something really, really changes that perspective. One, it's really web-based. It's really hard to catch that. It's really interactive. But users are very, very not model loyal. So everyone is aware of what's the hottest new AI tool, and everyone is experimenting. Those tools are not very cheap. So think about how you gain security. You gain security by controlling the configurations, the backend controls. You tie them to an identity provider. You put them behind some sort of a reverse proxy. You have all kinds of tools you can use. But that's good for traditional SaaS. Within AI, things change really fast. And when I think about the cost of those licenses, paying about $400 a month for getting all of them for all your employees, that's a lot of money. It's really, really a lot of money. So most organizations actually buy only one or at most two AI platforms. But the users use everything. And sometimes they even use their personal webmail to sign in to Claude or something of that sort. And eventually it means that the long tail is actually bigger than the main body of that risk. So AI just really fuels malicious extensions as a mechanism to create a very, very powerful intrusion by attackers. That creates urgency. [ Music ]

Dave Bittner: Our thanks to Or Eshed from LayerX Security for joining us. The research is titled, "How We Discovered a Campaign of 16 Malicious Extensions Built to Steal ChatGPT Accounts". We'll have a link in the show notes. And that's "Research Saturday" brought to you by N2K CyberWire. We'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity. If you like our show, please share a rating and review in your favorite podcast app. Please also fill out the survey in the show notes or send an email to cyberwire@N2K.com. This episode was produced by Liz Stokes. We're mixed by Elliott Peltzman and Tre Hester. Our executive producer is Jennifer Eiben. Peter Kilpe is our publisher. And I'm Dave Bittner. Thanks for listening. We'll see you back here next time. [ Music ]