Why Threat Actors Love Your RMM

Elliot Volkman: Hello, and welcome back to the "Microsoft Threat Intelligence Podcast." I'm your sometime host Elliot Volkman, the director for Microsoft Threat Intelligence. And I have the pleasure of having one of our colleagues and peers over at Huntress. I believe you might be going by Spike today, but you also have a regular name called Andrew Grant. Who is the principal threat intelligence and incident commander over at Huntress. Did I get all that right or how much did I mess up?

Andrew "Spike" Grant: Yeah. No. That's all correct. It's the longest title by character count in cybersecurity. >> Elliot Volkman. I love it. Was that intentional? It was not my choice. It was decided by whoever was hiring me.

Elliot Volkman: So I cannot give you cramp about the commander part in the title.

Andrew "Spike" Grant: Oh no. Absolutely you can give me all the crap.

Elliot Volkman: Okay. I was just checking. Yeah. That is a new one for me. I don't know that that is -- well, we don't ever have standard titles and language in our field so to speak. Actually that is a problem in itself. But yeah. That's a unique one.

Andrew "Spike" Grant: It's so I think there is our CEO has a military background and so we have a lot of titles that are about -- like incident commander is actually a title specific to working with rallying the troops within the company to help organize a response. And in my case the role is about when there is a large cybersecurity incident, whether it's affecting our customers or just, you know, the whole internet as a whole, helping us to rally incident responders and threat intelligence people and PR and communications folks and being able to work the incident in a very coordinated way.

Elliot Volkman: Interesting. So a holistic approach or sort of organizing the chaos. And it very much is the case.

Andrew "Spike" Grant: Oh. It is all chaos all the time. Yeah.

Elliot Volkman: That is great context. So I appreciate you helping maybe elevate your background a little bit. So as we jump in to this topic it will hopefully lend itself to your role in this equation which is why threat actors love your remote management or RMM technology and I think we're saying other strange stories which I think is going to be a default answer of AI. Maybe. We'll find out. But let me read off my little bit of notes and we'll kind of jump from that and then I promise I will not be reading and hopefully pay attention because I'm terrible at this. So actually listen to this. Do you want to be -- are we Spike or are we Andrew?

Andrew "Spike" Grant: Yeah. Call me Spike.

Elliot Volkman: Okay. Just making sure. Yeah. That's good. Spike, you spend a lot of your time pretty close to actual incidents obviously. Before we get in to the RMs specifically, what is happening out there right now that makes you say security teams really need to know this? AI otherwise quantum.

Andrew "Spike" Grant: I mean there's a -- if you walk around the show floor here at Black Hat AI is on every other booth in this place and it is striking to me how dependent the industry is on using AI for marketing. Whether or not it's actually benefiting any of us is kind of an open question. So think -- but the things that I see involve the actual threat actors doing specific kinds of targeted attacks, fishing attacks, you know, click fix attacks, things that are the, you know, sort of the root cause of breaches.

Elliot Volkman: Yeah. That absolutely makes sense to me. So maybe I'll put this one out there for you to totally go off on a tangent which is very easy to do with this topic.

Andrew "Spike" Grant: Yeah.

Elliot Volkman: There's the headlines which are to certain companies doing certain things, getting a lot of attention. And then there's what we actually see threat actors doing across the attack chain, cochain, however you want it. Right? Do you feel like there's any specific thing that's emerging in totality or do we actually see agents going rogue at the discretion of certain companies?

Andrew "Spike" Grant: Well, I mean I wouldn't know about the agents going rogue, but I do know for a fact we can pretty much be sure that threat actors are using LLMs and other AI tools to improve the lures that they're using in their attacks. So we see much better crafted spam, much better built like fake websites, and they're learning from the mistakes that they've made in the past and using AI to sort of improve those things. And they're still not perfect and the AI doesn't necessarily always help with their grammatical issues if they're not native English speakers, but there is -- AI is definitely improving their lures and the components of the attack that they're using.

Elliot Volkman: Perfect. All right. So that is definitely grounding us in reality. So I like to terrorize people about what they see and how AI's used because it helps detail their, you know, philosophy of are we like pie in the sky talking about crazy stuff or not. But I kind of figure that's the case. So maybe we'll bring us back to reality to talk about RRMs. Why do threat actors in particular love them? Why are they abusing them? What is -- is there an underlying component that they, you know -- they've been doing this for ages.

Andrew "Spike" Grant: Yeah. Well, I mean I don't know. They haven't been doing it for ages. So I've been in this space for a long time doing investigations in to, you know, social engineering driven malware attacks.

Elliot Volkman: Of course.

Andrew "Spike" Grant: And this, you know, typically when you see these, you know, very common, almost they're like tropes of spam where, you know, they're pretending to be the IRS. They're pretending to be your bank. They're pretending to be your university or a service that you use. And they send you a thing that says "Oh. We're changing our billing. You have to click here." Take you to a web page and then the web page convinces you to either give them some information or download a file, sometimes both. You know, all of that is it's enhanced by AI stuff, but the RMMs is where we're seeing where it used to be, you know, specific families of malware that were doing these kinds of attacks the end result would have been "Oh. You'll get tricked by." Or you'll get Dyreza or one of these other like well known families of malware. Now we're seeing at the low end of the threat actor scale they're turning to these -- RMM is a term of art that was coined at Huntress, I believe. It stands for remote management and monitoring. And it's an -- what it means is technically it's a commercial remote access tool for it's designed for IT managers so that they can keep an eye on all their servers. And the end points as well. Right? And especially these became very popular when the pandemic lock down happened. Suddenly everyone was a distributed working from home workforce. And the IT managers needed to still be able to fix things. And they need to be able to access those things and so, you know, they tried a little bit of that with remote desktop. It kind of turned out to be not such a great thing to just open up, you know, the ports from a desktop to the internet. So now these RMM tools gave them a slightly more secure way to do that. The threat actors realized that, you know, why spend the time, the trouble, the money, the effort on building a better malware when there are countless companies that are trying to support the legitimate needs of IT departments to give remote access to them. And so the threat actors are like, "Hey. These are cheap. They work. They're signed with legitimate certificates. They get installed no problem. Anti virus won't necessarily detect them." Or input security, whatever it is, you know, your term of art for like the thing that protects your Windows computer. And so they're just turning to these more and more. And it is actually honestly shocking to me as someone who's done this for such a long time that so much of what I'm seeing now is the end result of these complex attack chains is that you get an RMM instead of what used to be malware.

Elliot Volkman: Do you feel like is there -- maybe I can clarify. When I say ages anything over six months is ages in my world. It just changes too quick. But do you feel like maybe from a less technical perspective it's just easier for them to abuse trusted mechanisms because it's less visible? Maybe appears benign unless you have, you know, better intelligence, better signal.

Andrew "Spike" Grant: Yeah. I think there's two pieces to this. So there's a lot of companies here who do end point security. They have gotten very very good at building end point security that can detect these, you know, various malware families and so it's harder for a malware developer to create malware that evades an end point. On the other hand yes it's cheap. It's easy. And most end point security products are not going to detect, you know, Screen Connect because it's a real company. They have a business plan. They have a website. Like anybody can sign up for it and use it. And they have millions of customers, legitimate customers that are using them for managing their networks. So yeah. It's that's where the difference is is that all of a sudden those things can evade end point detection. They evade Microsoft Defender. Right? Which is the, you know -- every Windows machine has it. It's the primary end point protection that every Windows computer has. If you're evading Defender that's a real problem.

Elliot Volkman: Just a little bit.

Andrew "Spike" Grant: Yeah.

Elliot Volkman: So that begs the question of like if we're not doing it through the standard tech stack, security stack, how are you pinpointing when something is obviously off kilter versus expected traffic?

Andrew "Spike" Grant: Yeah. So I mean that's, well, when it's network traffic it sometimes has to do with the every -- so in the research that I've done, and the work that Huntress does in general, we monitor more than 50 different RMM, you know, companies and their products. And we see them being used in circumstances where the company who is our customer maybe they at one point used it and then they stopped, but then all of a sudden this RMM just fires up and starts communicating over a network. Sometimes it's that the, you know -- maybe they're doing some exploration. And they're trying to find a good RMM that they want to use so they install it on a trial basis and then they just leave it. And we'll find out that like the -- you know, the account for that RMM got compromised. And then, you know, threat actor doesn't even have to do the complex attack chain to get their RMM on the machine. They just get the password from somewhere. Maybe a stealer took it. Maybe they've got it from an initial access broker. Maybe there's a phishing attack that, you know, they were able to get someone's admin credentials. And then they just try those credentials on everything that they can find and they can break in and they don't even have to do the -- you know, the dropping the RMM on the machine.

Elliot Volkman: Interesting. So I think assuming I got the right research correctly one of the things that stood out to me is that there have been cases where a threat actor in the process has dropped in a second or third RMM. Is that a thing?

Andrew "Spike" Grant: Yeah. No. I mean it's actually kind of nuts. So -- so in the -- in my lab where I work I have a bunch of these spam honey traps that, you know, bring in spam. And then I am the guy who clicks all the bad links, opens all the bad files so you don't have to. So I run these things in my environment and one of the examples that I've been talking about we published the research, you know, a couple months ago. The attacks started with a -- it looked like it was telling you that there was an invoice that needed to be paid. And you end up downloading this file that's an MSI. Right? Microsoft installer. Well, you know, it's an executable file format, but you can also tear it apart pretty easily. So we opened it up and took a look at what was inside and there was this RMM that nobody at Huntress had heard of before. It's from a small company in Brazil. The RMM's called Tiflux. And the installer had, you know, all of the components that were embedded with it. So it installed itself which, you know, it had its own remote access capabilities. It also installed a VNC which is another remote access tool. It's an open source remote access tool. And then it had installed a what we call a BYOVD driver, like a bring your own vulnerability driver. And it's a kernel level driver from something, from a piece of hardware that's now deprecated. It had been signed. It's now expired, but it's so old that it's not really invalidated. There's no one to invalidate the certificate. So it gets installed along with this, you know -- this Tiflux RMM. And there was some scripting within the installer that once it installs, you know, itself and it installs VNC and it puts this driver on the machine it then runs a script and the script was to go out and pull down a copy of Splashtop which is another RMM. And it would pull that off of the threat actor, an IP address where the threat actor was hosting it, just parking it there as a dead drop. And then the fun thing with that was several hours after. Leaving it running, set for several hours, it then pulled down Screen Connect. And it ran a fifth. So we got Tiflux, VNC, and this weird driver, and then Screen Connect and Splashtop. So four RMMs and a vulnerable kernel driver that can elevate the privileges of anything else that the threat actor chooses to do. So we did a little bit of looking. We figured out that this, you know -- this company Tiflux they're a real company. They're based in Brazil. It's $5 a seat. And there's no validation on, you know, when somebody installs this it doesn't go -- it doesn't route through Tiflux's servers. It's got its own basically ability to set up a command and control server. So there was no way for the Tiflux folks to stop it. And it left three levels of backups beyond itself so that if Tiflux gets removed they have fallbacks as an initial access to get in to the network.

Elliot Volkman: That's wild.

Andrew "Spike" Grant: Yeah.

Elliot Volkman: So let's maybe pull that apart a little bit more. So obviously if it is falling in to one of your traps they're an opportunistic actor. Do you have any kind of sense of like are they just spray and pray? Like do you have any sense of --

Andrew "Spike" Grant: Yeah. It's actually interesting. So I operate, you know, a lot of different sort of services that I've had for a long time. And I've been coming to Black Hat for a long time and I use the spam trap domains with a catch all email to be able to see what comes in.

Elliot Volkman: Right.

Andrew "Spike" Grant: The initial email that this came to, that the attack was targeted at, was one that I used to register at Black Hat in 2016. So some -- so at some point I walked around. 10 years ago I was here. And someone was scanning my badge and it gets the, you know -- gives the contact information to some vendor.

Elliot Volkman: Right.

Andrew "Spike" Grant: That vendor's list was sold, lost, leaked, breached, stolen. And now that email address that I used to register for Black Hat in 2016 gets some of the best malicious spam that I've ever seen and it seems like this person's targeting someone who works in the security industry. Right? Because why else would you target this? And that address was never used for any other purpose. So I know for a fact that it was some vendor who was here in 2016 who scanned my badge. I'm pretty sure it wasn't Black Hat themselves that got breached, but someone lost control of their list, sold it, or it was stolen. And now that list is being used to target people in the industry.

Elliot Volkman: Yeah. A marketing team would never lose sight of --

Andrew "Spike" Grant: I mean yeah. No. Everybody here is always 100% on their game, myself included. Right? Of course. Yeah.

Elliot Volkman: I'm not terrorizing the people in the audience who are [inaudible 00:17:02].

Andrew "Spike" Grant: No. But -- no, but the reality is is that like yeah. There -- the threat actors are out there. They are looking for a specific -- I think in this case they were looking for a specific audience of people who have valuable data.

Elliot Volkman: Okay. So it would be a data -- not an access. Or maybe potentially both, but --

Andrew "Spike" Grant: I mean what these remote access tools allow you to do is log in to a machine as though you were remote desktoping in to it. So they see the whole desktop. They can. And then in addition to installing all these RMMs there were all these additional profiling scripts that were running after everything had installed. It wanted to know everything about the machine it was running on. It created a, you know, list of all the directories. And so, you know, they were able to then poke around before they decided to log in. They get this sort of like list of things that might be interesting. So this machine I installed this thing on is sitting right next to me and I'm working on my corporate laptop right next to it. And I'm taking notes and writing the report about like what I found on this thing and the box suddenly the screen blanks and it looks like it says "Windows updates are installing. Don't shut down your computer." But it doesn't have the little spinning balls. So at that moment I realize oh they're using the screen blanking feature within this RMM to hide what they were doing. And the blank screen that they were using was a screenshot of the Windows update, you know, is installing. Don't shut your computer down right now. Because it did that. It then went back to the desktop. Didn't reboot. Then blanked again, then went back to the desktop. So after a few times of watching it do this and realizing okay they're logging in to this machine I opened up Notepad and I wrote a message and I said, you know, "Look. This is getting kind of stale. You guys need to get some new material." And just left it sitting there with Notepad open on the screen because I knew that they could see the desktop. Right? And I'm watching them connect in and out. Then the screen blanks and two minutes later unblanks and in that same Notepad window under where I had written the message they just wrote no.

Elliot Volkman: Good lord. Well, there's our clip for the episode. It's always fun when you can interact directly with the threat actor.

Andrew "Spike" Grant: Yeah. I've had a couple of experiences where a threat actor installed a RAT or Trojan of some kind and I could see that they were connected to it at the same time. And it's actually, you know, it's interesting because for a long time I've had -- I've worked in this industry for a very long time and for most of that time a very monolithic view of people who run malware campaigns and install rats on people's machines and target individuals as just kind of scummy jerky criminals. Right? I had this one experience one time where someone installed a RAT. I installed the RAT intentionally. Right? Just to see what it did. I could see that it was doing -- it was connecting to this IP address in Tunisia. And I don't know what they were doing. They were just kind of -- you saw the traffic going back and forth to this IP address. But I started looking at the network traffic that was going through and I realized that this RAT is capable of -- it was reading off the text that was in the title bar of every open application that was running. So I had like process explorer running and a few other programs running. And so it was I looking in the network traffic and I saw the names of all these things. I'm like oh it's reading the title bar. So I in that case opened Notepad, left it blank, but did save as. And I saved it as "Hey I want to talk to you. Can you send me a message?" And just did save and left it sitting on the desktop. And about 10 minutes later something that looked like an old instant messaging program popped open on the desktop. I did not have this installed in my test bed. It was the RAT. And it basically was a little chat box and it said "What?" And I, you know, then -- you know, the bottom of it you could send a message back so I said "Hey, I'm a security researcher. I'm just curious. I'd like to know more about why do you do this. What is your -- you know, what are you trying to do here?" And that was the start of a conversation that went for two and a half days over -- at the time I was using -- the person wanted me to use ICQ which is an instant messaging program. So I created an ICQ account and chatted with this person and it turned out that the guy who was running this, I assume it's a guy, don't know for sure, he was in Tunis, the capital of Tunisia. It was during one of the color revolutions that was happening in the middle east. He said -- he told me "I run a small IT shop here in Tunisia. I usually do, you know -- I'm like an MSP for small businesses. And there's a revolution going on. I can't pay the bills to feed my family so I started running bot nets." And he told me he had gotten so many bots installed on so many machines that someone from the government came to him and told him, "You better knock this off because we can see what you're doing." And he had to like kill about a third of his, you know -- his Trojans. But he was basically committing crime because he was in a situation of financial and social desperation. He had nothing left except to just rely -- you know, go to like petty crime to survive.

Elliot Volkman: Do you feel like that was true or was he just adding social engineering trying to manipulate you on top of it?

Andrew "Spike" Grant: I, you know -- it's hard to tell. It was a text message chain and an AIM client. But it did give me reason to think about like, yeah, you know, there are social aspects to cyber crime that are sort of outside our purview when he talk about the fact that like people are stealing stuff to make a lot of money. Yeah. Sometimes it is just it's desperation. Right? People do turn to petty crime to feed themselves. I mean I don't know that it was like a Jean Valjean situation where he needed that loaf of bread right then, but it did kind of like change my perspective a little bit and make it a little bit more nuanced about like the motivations that people can have to commit crime like this. Now I'm not saying that with these guys who are doing the RMM stuff that that is their motive. Clearly they're, you know, looking for a quick payday. What happens with these RMMs is often the people who are doing the attacks where they deliver the payload they then sell that on to initial access brokers who then sell that to the real cyber criminals who want to conduct ransomware attacks and breaches and steal data and ransom your data back to you. It isn't always, you know, necessarily that Jean Valjean situation. Most of the time it's not. But sometimes it can be. Right?

Elliot Volkman: Yep. So thank you for sharing those stories. That is fantastic. But I want to make sure that we always end on some guidance.

Andrew "Spike" Grant: Yes.

Elliot Volkman: What are we doing about this? How are we telling our world of analyst researchers and hunters?

Andrew "Spike" Grant: Well, there's a few things that we can do. So, you know, at Huntress we when we're doing monitoring for a customer the way that we detect that these things are happening is we have these very complex rules and alerts that we've created and they trigger alerts in our SOC team that then, you know, the SOC team has to like jump on those. And we will notice when a customer says like, you know, "We use these RMMs." And then all of a sudden we see a different RMM firing off. Like that's an alert for us. One thing we tell -- we try to encourage customers of all sizes to do is to, you know, audit the materials and tools that they use so that they know what RMM is expected and what's outside of the norm for your organization. When there's like 50 plus RMMs out there that could be being run legitimately it's really important to know what's right and what's out of band for what is normal for you. The other thing that we tell people to do is there's this concept of application allow listing where you can, you know -- it's a little bit more narrowly focused for certain kinds of people who work in certain fields, but okay. You're allowed to use, you know, Outlook and Teams and Zoom and, you know, various browsers. And maybe you get to use Screen Connect, but not Splashtop. That allow listing prevents a ton of this stuff from happening without you getting to the point where we have to like dive in and do real, you know, remediation and fighting back. Am I --

Elliot Volkman: The audio just went. I think we're having to whisper. I think on that note we're being booted off the stage so thank you for joining the "Threat Intelligence Podcast." No, Spike, we really appreciate you joining, providing those stories and the guidance. We will definitely provide a little bit more connection points and some of your research so folks can dig more in to that.

Andrew "Spike" Grant: Thank you, Elliot. It's been a pleasure to be here. I appreciate you. Appreciate it. Take care.