Bulletproof hosting (BPH) and how it powers cybercrime.
Dave Bittner: Hello everyone, and welcome to the CyberWire Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.
Jason Passwaters: I guess in its kind of rawest definition it's really just infrastructure that is used for malicious purposes.
Dave Bittner: That's Jason Passwaters. He's COO at Intel 471. We're discussing their research on bulletproof hosting.
Jason Passwaters: But the folks that run the service are doing things to help defend against takedown and make it more resilient so they can hide the infrastructure, they can keep it up and running longer, and then they can make adjustments if there's any kind of takedown attempts. But bulletproof hosting as a service is kind of broader in the sense that they provide anything from domain registration, server administration, stuff like that, help desk, as well as most services now are providing not only, like, backend infrastructure, but also a front-end kind of reverse proxy net that serves as a protection layer. So it's kind of more of a broader or a larger business model.
Dave Bittner: Hmm. So, in the past, if someone wanted to do this sort of thing, they would tend to roll their own and find someone who'd be willing to let them host their own server up to the Internet?
Jason Passwaters: Yeah, I mean, bulletproof hosting has been around since cybercrime has been around. If you go back to the mid 2000s, there's some of the same actors in the game today were back then doing the same. And they would just have their infrastructure in hard-to-reach places and hard to identify different things they were doing to kind of hide things. And over time, with takedowns, with things that have exposed how they operate, they have evolved and made changes. And that's why you see that, like I said, that two-sided setup where they've got the backend of a structure and they'll have a reverse proxy net or some kind of botnet that sits in front as well.
Dave Bittner: Well, let's go through some of the things that you've covered here in the research. One of the highlights here is you talk about the power of "fast flux." Can you take us through, what exactly does that mean?
Jason Passwaters: Yeah, so the fast flux has been around for quite some time, and essentially what it's doing is there's two types, main types. The first is kind of VPS based – basically servers that are set up with different providers. And the bad guys will, you know, leverage automation to rotate IP addresses across – sorry, the domains across that pool of IP addresses. They've got another type – it's more bot-based. So, malware-infected machines are acting as kind of proxies, and they'll rotate the domain names that are associated across all of those IPs at some high frequency to help add some resilience to identifying the infrastructure and takedown and stuff like that.
Dave Bittner: Help me understand, because they're rotating these quickly, I mean, in a matter of minutes, right?
Jason Passwaters: Yeah, in some cases it could be minutes. But in the case of, you know, when you have somebody that's doing more – like, one provider specifically abuses cloud providers, and he maintains a huge pool of IP addresses and he'll rotate the domains at different frequencies and they won't necessarily be super fast. And it might be as long as that instance stays alive until the cloud provider identifies it and takes it down and then another one takes its place. In the case of fast flux, when there's bots involved, they're actually having a small time to live on the DNS record side so that it automatically changes. You have a pool of IP addresses that are, you know, within seconds – it could be 90 seconds, it could be a little bit longer – changing across the entire botnet.
Dave Bittner: Well, help me understand – wouldn't they have issues with propagation, with the domains, you know, the propagation of the alignment of the domain name with the IP address, wouldn't there be a lag with that, or is that not really an issue these days?
Jason Passwaters: No, I don't think it's an issue these days. And, you know, that time to live helps with the propagation, you know, as far as how DNS works and stuff. So, it's been pretty resilient. It's been around for a while. They really have two versions. And the bot side, there's double flux and then there's just regular fast flux. The double flux is even, I guess, more resilient because the name servers themselves that you're asking basically for resolution also are on the botnet as well. So you've got not only the bots that are kind of dealing with the A-records, which is domain-to-IP mapping, but it's also they NS-records, which is the name servers use to kind of resolve it. So they're also kind of fluxing as well. So it makes it twice as hard sometimes.
Dave Bittner: Hmm. And this is a tough one to stop, yes?
Jason Passwaters: Yeah, very much. Especially when, you know, right now I think there's only one actual double flux hosting kind of framework that's out there for bulletproof hosting, that we're aware of. And they leverage kind of hard-to-reach places as far as, you know, core infrastructure, and even the bots that are on the front end, if you will, you know, you don't see them as much in the United States. You don't see them in Canada and different countries. You'll see it a lot in – you'll see it in Ukraine and Romania. And the purpose is just to kind of proxy back to the back end.
Dave Bittner: Well, and I mean, that transitions into sort of the physical location of the data centers themselves, which is a big part of that. I mean, they're setting these things up in places that have a certain amount of flexibility when it comes to law enforcement, I suppose?
Jason Passwaters: Yeah, for sure. I mean, there's kind of a range. You know, you'll have some providers that are – just don't have the capabilities to respond and, you know, they're not necessarily bad or partaking in the activity, they just don't know and they don't know how to respond if they do figure it out. And then the full other side would be more providers that are actively kind of, you know, they have their, quote, "data centers." And we've seen a couple of those where they have their own, you know, their own core infrastructure, even down to having their own AS – autonomous system – number, their own prefixes, and companies that kind of associated with this infrastructure.
Dave Bittner: And so, what can the rest of the online community do to try to tamp this down?
Jason Passwaters: I think it's really, you know, tracking this kind of activity, you know, the threat actors associated with it, the way it can be beneficial is if you can map out the infrastructure and tie it back is, as you kind of go back from a single domain or a single IP address and you can go up the chain to a net blocker prefix, a whole group of IP addresses. You know, if you associated that with, say, a bulletproof hosting service, you know, you've got kind of swaths of space that you can you can block or alert to once you have a confidence in that kind of linking back to a malicious service.
Jason Passwaters: And then to be proactive, I mean, you could you could look at the AS-level and proactively monitor BGP messages to identify new prefixes or new blocks of IP that come up under that infrastructure when it happens and then proactively kind of alert to that if you see it on your network, instead of just kind of whack-a-mole with a single IP address sometimes. The proxy, the reverse proxy that or the fast flux stuff, it's a little bit harder. It is going to be single ISPs. And, you know, if you have more, if you have vendors, you know, obviously they can provide that kind of stuff. It would be more advanced type tracking and research to kind of see that.
Dave Bittner: Can you give us a sense of the kind of spectrum offerings that are out there? I mean, are there varying degrees of bulletproof-ness depending on what people need?
Jason Passwaters: Yep, it depends on basically what people need as far as the activity they're doing. You know, if they've got some very noisy stuff, that is going to be, you know, maybe high-bandwidth needs, you know, they have solutions for that. If they're going to need just, like I said, that kind of protection layer, you know, let's take a ransomware blog, for example. You know, they might hide it behind one of these these kind of fast-flux setups. Now, if you need core infrastructure – say, C2 infrastructure that, you know, needs to sit on an actual server somewhere, they'll have backend hosting for that as well. And that'll typically be in hard to reach places, you know, that we've seen, you know, obviously Russia, you know, parts of Ukraine, or eastern Ukraine is a big one sometimes because it's a bit of a contested area right now and hard to reach. I think it's Transnistria between Moldova and Ukraine. That's a popular place as well.
Dave Bittner: Hmm. Yeah. Folks have other things on their minds, I suppose, than chasing down these servers.
Jason Passwaters: Yep.
Dave Bittner: Yeah. So, to what degree do these have the attention of law enforcement? I mean, you know, there was this story not long ago about the cyber bunker, which was a very dramatic kind of thing. I mean, are there takedowns of these or is this like so many things, one of those games of whack-a-mole?
Jason Passwaters: You know, I think it definitely is on the radar of law enforcement. You know, every time you see the same infrastructure pop up or the same kind of core infrastructure pop up with different stuff, it's tough to, I think, quantify the significance that, say, something like a bulletproof hoster might play across cybercrime in general. So that's tough. But I do think that there is a tension, there are takedowns. I mean, there's a number of takedowns that have happened. You know, Avalanche botnet was one, one of the bulletproof hosting services in Ukraine, Sosweet, that was another one.
Jason Passwaters: And it's interesting because when you take down a core enabler like this or you impact a core enabler, it has a reverberating effect across different aspects of cybercrime. And what happens is, even in the kind of marketplace that we watch and track, you'll see that kind of manifest itself with the service themselves doing damage control. You'll see it with other hosting providers, you know, malicious hosting providers, providing, you know, specials, and basically saying, hey, sorry for my competition's situation, we feel for him, but we're offering a special. And then you see it with actors that are complaining, because when infrastructure suddenly goes down, you know, everybody starts to complain about why I can't access my servers or why all my ops are pretty much paused. So it's interesting to see that. So you see that kind of wider effect, and I think the point is that when you affect a core enabler like this, it has deeper, probably more lasting impact.
Dave Bittner: Do the different providers have varying degrees of reputation? Like, are some known for their for their uptime? You know, the things that you would see in a normal hosting, do all of those apply to these folks as well?
Jason Passwaters: Yeah, absolutely. You know, reputations are maintained in the marketplace. If you don't have support personnel that are interacting with the with the market and the clients, you're going to get dinged for that. If uptime is bad, you're gonna get dinged for that. So it's just like any other kind of hosting service. It has all the same challenges, you know, from dealing with customers and clients to scaling the business, you name it.
Dave Bittner: Our thanks to Jason Passwaters from Intel 471 for joining us. We'll have links to their research on bulletproof hosting in the show notes.
Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Elliott Peltzman, Puru Prakash, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.