Like anything these days, you have to disinfect it first.

Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday, presented by Juniper Networks, I'm Dave Bittner and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Johannes Ullrich: CyberBunker is one of these really interesting things in security that's sort of a real event made for a movie.

Dave Bittner: My guests today are Karim Lalji and Johannes Ullrich. Karim is a security consultant and master's candidate at the SANS Technology Institute. Johannes Ullrich is the dean of research at the SANS Technology Institute, and a regular contributor to the CyberWire. The research we're discussing today is titled, "CyberBunker 2.0: Analysis of the Remnants of a Bulletproof Hosting Provider."

Johannes Ullrich: People set up bulletproof hosting facility that – well, usually when you talk bulletproof hosting, you talk about hosting that's hard to take down with abuse complaints. They actually were bulletproof, in a Cold War bunker that was actually originally designed to house the German government in case of a nuclear attack. Now, with the end of the Cold War, that bunker became redundant, and this group purchased it. And they had about a million square feet of space in this bunker until it got raided last year.

Dave Bittner: Wow. I mean, a million square feet is nothing to sniff about, and I suppose we have to give them something for style points.

Johannes Ullrich: Yeah, and they had it fully equipped, including a lounge area and everything, if you look at the pictures. So, it was a real fancy operation. Now, this is known as CyberBunker 2.0, because the same group actually did the same thing with a bunker in the Netherlands that just happened to burn out when one of their drug cooking operations kind of went bad.

Dave Bittner: Hmm. Now, in terms of bulletproof hosting, I suppose I would have expected those sorts of operations to run in places like Russia, you know, those sort of Eastern Bloc countries. How unusual is it for something like this to run in Germany, for example?

Johannes Ullrich: It's very unusual. And that sort of was their downfall, in part because, of course, eventually law enforcement became aware of what they were doing and that's why they got raided.

Dave Bittner: Well, Karim, why don't you jump in here and explain to us, where did your part in all of this start when you all took notice of what was going on here?

Karim Lalji: Yeah, great. It was a really interesting project to work on, actually very enlightening. So, as Johannes mentioned, there was this CyberBunker group that was offering bulletproof hosting. They were hosting, you know, darknet sites illegal pornography, drug markets, all sorts of things. And in the fall of 2019, the police raided that facility and arrested the individuals that were involved. And they're still actively undergoing trials in Germany. And one of the methods they used to liquidate some assets to help pay for their defense was to sell the IP address space that belonged to this hosting facility. Now, we're talking about a fairly large IPv4 address space. So, it's two 0/22 networks and one 0/24 network, which is about 2,300 IPs. So, when they sold this, they sold it to a company that had a relationship with Johannes and were able to redirect some of that traffic to the SANS Internet Storm Center honeypots, so that we could take a peek behind the lid and see what was going on.

Dave Bittner: Hmm. Well, I mean, let's dig into some of the details here. For folks who might not be deeply familiar with what exactly goes into setting up a honeypot, can you give us a little bit of the background there, and then tell us how you applied it to this particular situation?

Karim Lalji: Yeah, absolutely. So, the honeypot was just a host that was set up, Internet-bound. It had a lot of the common services that you would expect – so, things like a web server listening on Port 80 and 443, an FTP server as well. So, when the traffic got redirected – for example, if it made a request for a website – that would then get trapped by our honeypot and logged by whatever application layer protocol was waiting at that end – for example, an Apache web server. So, we had permission to look at any of the traffic aside from email. So, we didn't have anything listening on port 25 because we weren't able to inspect email. I presume – and Johannes can correct me if I'm wrong – that that had something to do with some of the investigations by law enforcement.

Dave Bittner: Hmm.

Johannes Ullrich: Yeah, and partly they didn't really want us to look at email because that may have been useful later for law enforcement and they didn't want us to tamper with any of that evidence, essentially. 

Dave Bittner: I see. Well, so, take me through – what sort of traffic were you seeing here? I mean, what were folks who were – the type of folks who'd be purchasing this sort of bulletproof hosting – what were some of the use cases that you all detected with the honeypot?

Karim Lalji: Yeah, so, we looked specifically for the things that were contained inside of the press release by the attorney general in Germany. And that was things like botnet traffic, and illegal pornography, and malicious ads. And we did find evidence of some of that. So, we definitely found a fair amount of residual botnet communication. And one thing that's important to note is that the analysis that was undertaken here was about nine months after the police raid. So, these servers had been taken apart, they've been removed from the facility, and then they're being resold to another company. So, we're seeing this back-scatter traffic nine months after, and there's still so much of it. So, there was a lot of C2 communication using techniques like IRC, as well as encrypted botnet communication, which was tied to specific, known malware families.

Dave Bittner: What other types of things were coming in?

Karim Lalji: We saw a lot of residual traffic from malicious ad networks. So, just like regular organizations, criminal organizations also leverage ads. And what was interesting about this ad network is the volume. I mean, every few minutes you see a host resolving this ad network with URLs and query strings inside of those URLs that were definitely on the questionable side. I mean, these were adult content. Some of them looked like they might have been involving child sexual abuse images as well. And on top of that, we had a lot of phishing attacks that seemed like they were still happening, targeting things like PayPal, or Apple, and even a couple of instances of Royal Bank of Canada.

Dave Bittner: Interesting. So, when this traffic comes in, what were you all doing with it? Were you merely logging it, or were you trying to set a hook to see where things could lead next? What was the spectrum of responses that you all set up here?

Karim Lalji: So, the honeypot was passively capturing PCAP data. For things like web traffic, our intent was to get the traffic that was coming in, but no responses would be sent back out. So, for example, if a request came in from a compromised bot to the honeypot, we'd see the request and then drop the traffic. And that was just to make sure that we're doing our due diligence and not interacting with what traffic was coming in. But it allowed us to see everything coming inbound, which painted a really interesting picture.

Johannes Ullrich: In part, we wanted to be careful with what we're sending back to these requests because, of course, many of these requests came from victims, and we didn't want to cause them any more harm than them already being infected with this bot. As far as the phishing sites go, we knew what kind of companies they were looking for based on the host names being used, but of course, then putting up a phishing site of our own and asking for the credentials would have been probably more than we should have done, so we just sent an empty page back.

Karim Lalji: Yeah.

Dave Bittner: Yeah, that's a really interesting insight, because I suppose you would have – if you were responding to anything, you'd have the risk of inadvertently triggering something, of setting something into motion without really knowing what you're doing, right?

Karim Lalji: Yeah. And one trick we did have, though, without actually sending a response is, when we were looking at the log files. We could see these hosts making request for a specific IP over and over and over again for what looked like a phishing landing page. So, when you scan those IPs through a sandboxing service, you can actually see other sites that are being hosted on that same IP. And when I did that, I ended up finding phishing landing pages for, like, Chase Bank and Apple ID. And it was some strange looking URL, but it had a perfectly designed credential-harvesting page for Chase Bank. So, even though once we got that request coming in, we knew the IP that the victim was looking for, then we scanned that particular IP and looked at a page that was offered back in time. It was not active, but, you know, it's a cached page that was available at some point back in time.

Dave Bittner: Now, you also saw some – you were able to put together some information that you thought some of these botnets were doing cryptomining.

Karim Lalji: Well, we didn't specifically look at cryptomining. It was one of the accusations or one of the things that CyberBunker is on trial for...

Dave Bittner: I see.

Karim Lalji: ...What I ended up seeing specifically was IRC botnets, where it was using Internet Relay Chat to communicate with a command-and-control server. And the reason that assumption was made is partly the volume of data from these random victim hosts all over the Internet, communicating with just a handful of IP addresses in the CyberBunker space. But what was interesting about it is the payload that was being sent had these computer names. So, it would be like "Linda PC" and "Lenovo123" and "HP Admin Office." So, that kind of hostname would indicate that there was some intent to compromise, like a home victim computer, which is then calling home.

Dave Bittner: Yeah, I mean, it must have been fascinating to see, like, you know, one piece of this larger machine that's been disabled, and so many other machines around the world calling back to it, trying to continue the communications. It must have been interesting to be able to gain insights from that.

Karim Lalji: It really was. And one thing I also should mention is that the traffic volumes we were getting was quite large. So, my analysis was a seven-day period and an approximately four-hour chunk of traffic on each day. And just that portion of time was about forty or fifty gigabytes just of packet capture data. So, there had to be some kind of limitation on the amount of data that we were analyzing. And just from that IRC alone, we had about seven-thousand unique source IPs and over two-thousand unique computer names – presuming they are computer names, which we feel that they are. So, just in that small analysis window, you have seven-thousand unique IPs still talking home to their C2 channel, which is enlightening.

Johannes Ullrich: Yeah, it's enlightening, but it's also frustrating. And doing this for a while, this is really one of the frustrations in this business – that there are all of these infected systems out there and it's really hard to clean them up. Now, given the short time we had, we didn't do sort of any effort of reaching out to these victims, but having done it in the past, usually the success rate is very bad on any kind of outreach like this. So, once a PC is infected, it often stays infected for months or years.

Dave Bittner: Hmm. How do you go about deciding what you're going to spend your time on? When you're vacuuming up that much data, where do you begin? How do you set your priorities?

Karim Lalji: And that's always an interesting one. I think that's one of the skills you have to develop in this industry, as you're given so much data and you've got to figure out, well, where do I spend my time? And really, for me, it was just getting a baseline of the traffic and looking for odd deviations or things that stood out. You know, for example, I found the IRC botnet by simply looking at a smaller chunk of data and seeing the largest connection streams based on statistics in that sample data, and that kind of led me down the path. But you do have to make that determination of, what do you actually look at? And there's a very strong possibility that things have been missed, simply because it was a time-boxed exercise.

Johannes Ullrich: And the other problem, of course, is just like any IP on the Internet, we also saw a lot of just random attacks. Like, Karim in his paper talked about, like, Mirai scans and things like that. Of course, it takes a little bit of experience there to be able to figure out this is just something that anybody connected to the Internet will see, versus this IRC traffic that's different and special and really related to some of the alleged activity the CyberBunker was involved in.

Dave Bittner: Was there anything particularly surprising – anything unexpected in terms of the traffic you were analyzing? Things that caught your eye, made you raise an eyebrow?

Karim Lalji: I think for me personally, it was the volume. Because we have to keep in mind that we're looking at this network almost a year after it had been taken down, and the amount of traffic we're seeing is still so great. And I knew I expected to see some, but I didn't quite expect this much. Especially since it's been reclaimed by another Internet provider, let's say, those hosts or whatever, you know, phishing landing pages, they're going nowhere. But yet it's still being actively prodded. So in a phishing campaign, you would expect that the malicious email's being sent out and somebody's clicking on it. So for this, considering it's been so many months and you're still seeing those phishing pages being hit, it's quite enlightening.

Dave Bittner: What happens to this range of IP addresses now? Do they just get turned over to someone else, or do they stay dark for a certain amount of time? Where do they go?

Johannes Ullrich: So, the company that owns the IP address space now – they're actively involved in trading IP addresses. That's their business. Of course, with IPv4 address space being so scarce, they often end up with IP address space that's sort of had a history, like we have here... 

Dave Bittner: (Laughs)

Johannes Ullrich: ...And I guess, like anything these days, you have to disinfect it first. (Laughs)

Dave Bittner: Well, I was thinking, you know, it's kind of like if you get a new phone and it turns out that your, you know, the phone number they give you used to belong to someone who either had a lot of friends or lived an interesting life or something like that. And you're getting all these phone calls and texts or whatever. To what degree do people have to worry that an IP address that they've been assigned has some sort of dark history behind it?

Johannes Ullrich: That's very common, and yes, you have to worry about this. So, not only will you receive all this traffic that you're not interested in, that you're paying for – you're paying for this bandwidth that you're receiving here – but also because this IP address range has a history, it's known all kinds of blocklists and such. And in part, the company that uses or owns the IP address space now – one of their specialties is also to essentially clean that IP address space and prep it for resale.

Dave Bittner: I see. Interesting, interesting. So, what are the take-homes for you? When all was said and done and you were able to gather up the information, what were the main lessons that you all learned here?

Karim Lalji: You know, for what Johannes just mentioned, that was one of the big ones for me as well, is when you're getting access to an IP address space, it's important to at least take a cursory look at who had this before me? Is it on any known blacklists? Should I be checking it? I mean, for the average individual, it's going to be abstracted to the Internet provider that's purchasing these blocks of IPs. But it's still an important factor to consider, because in this situation, you know, for example, we didn't see this, but if there was still credit card data being exfiltrated, it might be encrypted, but even if there was things like that being exfiltrated out of the environment and you're now purchasing an IP that ends up being housed at someone's bank and then it's getting credit card information from across the world, that could be a big implication for that organization. So, important to at least take a quick peek to make sure that those IPs are safe and sanitized before you start using them.

Dave Bittner: Yeah, I was thinking, you know, what if I'm suddenly receiving a stream of unsolicited child pornography or something like that... 

Karim Lalji: Exactly. 

Dave Bittner: ...Because, you know – and could there be some liability? Is it a danger for me? How do I turn off that firehose if I inadvertently find myself in a bad neighborhood of the Internet?

Karim Lalji: And that's why it's important to look at that before it comes in. And I mean, there would be a traceback activity to see where it came from – we know that this IP address space belonged to CyberBunker in the past. But that's why it's important even just to do a quick sanity check, you know, look at a blocklist that's already available on the Internet, maybe run a basic packet capture to see the data that's coming in. Because that's what we did on the honeypot – we just had packet captures running, we were able to do some analysis. And of course, the skill level is needed to do that, but presumably someone buying a block of Internet-facing IPs would do a quick sanity check.

Dave Bittner: Yeah. What sort of insights did this give you all to other bulletproof hosting sites? Were there any information you gained or insights from that?

Karim Lalji: Yeah, well, it's definitely – with CyberBunker, their motto was, we will provide you with hosting without asking any questions. And it's important to realize that these types of organizations do exist. And when someone is wanting to engage in a cybercrime of some kind, they're going to need an infrastructure, and they're going to seek out organizations like this to help them with that. Whether it's a distributed denial-of-service attack, illegal hosting, they're going to try and use a service like this. So, I don't think this will be the last one we'll see for the foreseeable future. It's definitely a good training exercise.

Johannes Ullrich: And I think the breadth of activity is also a little bit surprising – or not surprising, depending on how long you've looked at these kind of companies. They essentially engage in whatever criminals need to do business. So, you have the entire range of cybercrime hitting an address space like this.

Dave Bittner: Mm-hmm. All right, well, I mean, are there any – I'm trying to think if there's any lessons to be learned for folks in the general defensive community. I mean, is there any tips or advice based on the traffic that you saw here for folks who are out there defending their own networks? Any insights there?

Karim Lalji: You know, when we're taught to do incident response – and, you know, a lot of organizations, even at a much smaller scale, get hit with some kind of cyberattack – we always talk about making sure that we do a good and thorough job of cleaning up the host, and not just, you know, pulling out the power cord and hoping for the best. But this is really a great example of that, because if your hosts are infected and you don't go through your eradication and containment phases properly, you risk these hosts continuing to engage in malicious activity long after things are unplugged. And I mean, this is a much larger scale. You know, most organizations that will go yank the cord out and hope that everything goes away. Well, here, not only have you yanked the cord out, you've taken the servers apart, you've sold it to somebody else, the IP address space is gone, yet you're still seeing traffic. So I think that's an important takeaway where you can scale it down to a smaller organization trying to clean up their environment.

Dave Bittner: Yeah, and how many of these devices out there around the world that were sort of phoning home into this IP address space were still performing their primary functions the way they should be. You know, this secondary activity going undetected.

Johannes Ullrich: And I think one lesson for the defender here is also with these hosts being still active nine months later, as a defender, you have to check these blocklists. You have to make sure that you're doing very simple indicators of compromise that you're pulling in. Yes, there is often a lot of garbage in the sense that you get false positives, that you get the indicators that are really of interest to you. But if your network is communicating with CyberBunker IP address space, you should know that. And I think that's really something that administrators have to be aware of – what are these bad IP address spaces and what data I am sending to them?

Dave Bittner: Our thanks to Karim Lalji and Johannes Ullrich for joining us. The research is titled, "CyberBunker 2.0: Analysis of the Remnants of a Bulletproof Hosting Provider." We'll have a link in the show notes. 

Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Elliott Peltzman, Puru Prakash, Stefan Vaziri, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.