Unpacking the Malvertising Ecosystem

Dave Bittner: [00:00:03] Hello everyone, and welcome to the CyberWire's Research Saturday, presented by Juniper Networks. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, and solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Dave Bittner: [00:00:26] And now a word about our sponsor, Juniper Networks. Organizations are constantly evolving and increasingly turning to multicloud to transform IT. Juniper's connected security gives organizations the ability to safeguard users, applications, and infrastructure by extending security to all points of connection across the network. Helping defend you against advanced threats, Juniper's connected security is also open, so you can build on the security solutions and infrastructure you already have. Secure your entire business, from your endpoints to your edge, and every cloud in between, with Juniper's connected security. Connect with Juniper on Twitter or Facebook. And we thank Juniper for making it possible to bring you Research Saturday.

Dave Bittner: [00:01:13] And thanks also to our sponsor, Enveil, whose revolutionary ZeroReveal solution closes the last gap in data security: protecting data in use. It's the industry's first and only scalable commercial solution enabling data to remain encrypted throughout the entire processing lifecycle. Imagine being able to analyze, search, and perform calculations on sensitive data, all without ever decrypting anything - all without the risks of theft or inadvertent exposure. What was once only theoretical is now possible with Enveil. Learn more at enveil.com.

Craig Williams: [00:01:53] Our work in malvertising goes back to really the advent of Talos.

Dave Bittner: [00:01:57] That's Craig Williams. He's the head of Talos Outreach at Cisco. The research we're discussing today is titled, "Malvertising: Online Advertising's Darker Side."

Craig Williams: [00:02:06] Back when Talos was first formed, there was really one malvertising campaign and exploit kit that ruled them all, and that was the Angler exploit kit. We estimated it was making I think it was something like $60 million a year. You know, we plotted out how we arrived at that number. And I think it was a real eye opener to the security community about how effective these campaigns were. Now, you know, as a result of that campaign - sorry, that research, and some other research, the industry started cracking down on. Groups are put in jail, and it kind of disappeared a little bit for a while. And so, the reason we wanted to write this up is because we wanted to talk about what they're doing with the infrastructure, what we're still seeing from a malvertising standpoint, and some of the newer things that they're doing that I think users need to be aware of.

Dave Bittner: [00:02:53] What I love about this research that you've published is how there's something in here for everybody. No matter what level you consider yourself to be at when it comes to understanding this stuff, this is a great place to start when it comes to understanding how the online advertising world works, and these threats against it and how they get to us and do the things they do. So let's start with that together. Let's start with the very beginning with some basic stuff. Can you walk us through what happens when someone starts doing online advertising? How does it work?

Craig Williams: [00:03:27] So basically, a user will go to a website, and that website will need an ad, right? And that ad request will go to a publisher, and then basically that goes to what's called an ad exchange. Now, here's where it gets weird. There is a real-time bidding system that'll basically go back and forth between the publisher and the exchange, and that will figure out whose ad gets displayed. Now, this is the problem. Now, let's say you're a very reputable website...

Dave Bittner: [00:03:50] Mm-hmm.

Craig Williams: [00:03:50] ...And let's say you want to make sure that the ads you show are non-intrusive ads. Maybe you, like, make sure they're not for anything questionable or morally or ethically sketchy. Right? You just wanted to be like maybe an insurance company or something.

Dave Bittner: [00:04:04] Right. Right. 

Craig Williams: [00:04:05] Something middle of the road.

Dave Bittner: [00:04:07] Yeah.

Craig Williams: [00:04:07] The problem is with this system, that becomes difficult. Right? You may sign up for something like that, you may think you're getting something like that, but then at the end of the day, the reality is you get an ad that certainly may look like that. But in the very bottom corner of the ad is a hidden redirection link that basically hits a series of sites that all do a very sophisticated system of checking to make sure you're not a security researcher, that will end up directing you to a site that's either hosting malware or even potentially exploiting your browser to install malware directly.

Dave Bittner: [00:04:39] Hmm. All right, well, let's back up and walk through this just really, really step-by-step, because there's a lot of nuance here to how it works. When you say that this bidding process happens, I mean, this is happening in a fraction of a second, right?

Craig Williams: [00:04:56] Absolutely. It's all automated.

Dave Bittner: [00:04:57] And it's based on the information they've gathered about me?

Craig Williams: [00:05:02] Yes. So have you ever been, you know, like, surfing a website and all of a sudden it pops up and says, hey, are you interested in computer security? Go take a class of local university or become an expert. And you're like, what the hell?

Dave Bittner: [00:05:14] (Laughs) Every day, Craig. Every. Day. Yeah.

Craig Williams: [00:05:18] (Laughs) That's how that kind of thing happens, right? Your browser is tracking what you're looking at and providing that information to advertisers so that they can target you with ads. Now, to make it even more insidious, I'm sure all of you have some sort of ad blocker, or let's hope. And you probably noticed a little button in there saying "allow non-intrusive ads."

Dave Bittner: [00:05:35] Hmm.

Craig Williams: [00:05:35] So there's actually a specification on advertisements that basically are - I forget the exact wording, but effectively, you know, ads you want to allow. And they have a unique identifier and you have to provide that identifier before your ad and that will allow it to walk through your ad blocker.

Dave Bittner: [00:05:52] Hmm. And I imagine that advertisers being good, upstanding online citizens totally respect that tag.

Craig Williams: [00:05:59] (Laughs) Well, the interesting part is the malware that we found actually is using one of the - they're called adblock keys, in order to bypass that type of detection. So the malware is taking advantage of that to bypass ad blockers to still compromise the host. And to give you an idea of how we found this one, what kind of got us back into this search, is we were looking at a piece of sporting equipment - I can't remember exactly what it was. It was going to be made, and then the company basically realized that it was too far of an out-there idea and it wasn't going to be feasible. And so they killed the project and shut down the website.

Dave Bittner: [00:06:34] Hmm.

Craig Williams: [00:06:35] Well, when you have something out there that's like a, you know, cutting-edge piece of technology combined with sports, you know, people may go click. And so what happened was the advertisers picked up the domain, and they parked it, and parked all their ads on it. And so what was happening was anytime anybody Googled this or looked it up, you would hit the site, you'd see the adblock key bypass your blocking system, and then if and only if you were using Safari - so this affected Mac users specifically - it would serve up what we call a "potentially unwanted program." And that's a very nice way of saying garbage software.

Dave Bittner: [00:07:09] (Laughs) That's quite a euphemism.

Craig Williams: [00:07:12] Yeah, well, in this particular case, it actually took it the extra mile and it was just flat-out malware. But you didn't know that right away. It was actually completely unnecessarily sophisticated. It would serve you an individually encrypted payload. That individually encrypted payload would have its guts double-encrypted using that same individual private key. When you extracted that, it would actually look like a - I think it was a fake flash update at the time. And that would actually install this piece of OS X malware, which would basically intercept the web browser and shoot ads all over the screen and do all kinds of other uncool stuff.

Dave Bittner: [00:07:48] Now, this is partially a result of the way that the ecosystem has developed for placing ads on websites. Right? Because I mean, it's impractical for, you know, if I'm the website for my local newspaper or my regional newspaper or even, I suppose, the New York Times or the Washington Post, it's impractical for me to be manually placing these ads myself. That doesn't give me the returns that I'd get if I turn it over to someone else.

Craig Williams: [00:08:18] Absolutely. Unfortunately, we've looked at a lot - we've looked at large advertising sites, we've looked at small ad providers. We have not found any ad provider that is 100 percent clean of malware. Even the really, really good ones, they still occasionally serve up malware. A lot of the time we have these systems set up - you know, I think probably the most well-known one would be our Threat Grid system, where people can go submit links. Right? Submit malicious links.

Dave Bittner: [00:08:44] Right.

Craig Williams: [00:08:45] So that's the kind of system that you can automatically run these in sometimes, because the way they work, like, let's say you go to a site, you go through a series of redirections, and then you end up getting compromised. Well, you may take the last website and send it to your friend and say, hey, is this malware? Well, what will happen is the website will look at that and we'll check the referral link, and the referral link won't be what it's supposed to be, and so then the website won't serve you the malware.

Dave Bittner: [00:09:09] Hmm.

Craig Williams: [00:09:10] And so what you have to do is find that original page, the source page with the ad link on there. And keep in mind, as we just discussed, because ads aren't predictable and because they rotate, you might have to hit it a hundred times, a thousand times, ten thousand times, before you get that magical compromised ad. So automated systems really help find these. And because of the way that they're designed, it can be very frustrating to try and track these down manually, particularly if you got compromised and weren't capturing traffic.

Dave Bittner: [00:09:39] So, walk me through the various ways that websites and the people who run them are monetizing these ads.

Craig Williams: [00:09:47] Well, the main one is they just do it through an ad exchange. Right? If you have a large website, you can go to an ad exchange, and basically, you know, you'll have ads pop up on your site and for each ad, you'll get a - I don't know, one one-trillionth of a penny. (Laughs) I'm not sure what the conversion rate is exactly.

Dave Bittner: [00:10:02] Right. So you sign a deal with this site and you say, in exchange for space on my site, I'm turning over the control of of placing ads to you, and these are the list of things that I'm requesting. You know, you're not going to put any ads for things that I find objectionable on my site.

Craig Williams: [00:10:21] Well, I think that kind of tuning probably really depends on the provider, but at a high level, yes.

Dave Bittner: [00:10:25] OK.

Craig Williams: [00:10:25] You basically pick an ad provider, you set it up on your site and then hopefully it all goes well. But from what we've seen - and you know, I don't want to knock the ad providers entirely, because a lot of this - I don't want to say it's not their fault, but it's basically someone abusing the system, right? You know, an ad provider has, you know, what, millions of ads a day they serve on a variety of sites.

Dave Bittner: [00:10:49] Right.

Craig Williams: [00:10:49] Of that million, how are you supposed to find the one one-tenth of one percent that has a link hidden in there that goes through a series of, say, thirty websites that redirect that then may serve malware, if your browser responds with the right things to the malvertising site.

Dave Bittner: [00:11:05] Hmm.

Craig Williams: [00:11:05] So it can be very difficult. Unfortunately, that's why I think most security-conscious people have opted to just block ads, because there's not really a bulletproof solution here.

Dave Bittner: [00:11:16] Yeah. And that's a big stick, I mean, it's sort of an on or off. It's it can be frustrating, I find, because it's not that I don't want to support the websites that I read through allowing them to put ads in front of me, but it's all this other stuff, all this tracking and all of the possibility for malware. I feel like it's not proportional.

Craig Williams: [00:11:40] Absolutely. And it's unfortunate now, because more and more news sites are saying if you don't turn on ads, we're not going to allow you to view our site. And so there's a lot of different ways to deal with it. You know, one of the most effective is doing it through your DNS system. So if you have something like OpenDNS, right? You can go take all your ad servers and say, I don't want those to work. And that will fix a lot of the problem. But even then, that can cause you issues. So there's not really a great way to do it. That's why it's usually not on by default. You know, if you go to work, chances are they're not blocking ads because they want the Web pages to work so that you can do your job. But at home, on the other hand, I run a very aggressive ad blocking system, you know, because I don't trust my children. (Laughs)

Dave Bittner: [00:12:25] (Laughs) All right. I can relate to that.

Craig Williams: [00:12:28] (Laughs) You know, and I know that if they do need to do something on a website and it's not working because of the restrictions I put in place, I'll happily go fix it. Now, unfortunately, that doesn't really scale to the enterprise environment... 

Dave Bittner: [00:12:40] Right.

Craig Williams: [00:12:39] ...And that's where it's very difficult. And that's why from an enterprise perspective, I think you've really got to rely on that layered defense. Right? Maybe run some sort of ad blocker, block the really bad stuff, you know, run some DNS security, block the known bad domains, and do what you can to block as much of it as possible while not impacting known good sites.

Dave Bittner: [00:13:02] Mm-hmm. Well, let's walk through this together. On the research that you published here, you have an example of a malvertising campaign and you sort of take us through step-by-step, what's going on, how it works, and how they get away with doing what they're doing. Can we do that together?

Craig Williams: [00:13:19] So this was the one where we had the sports website that, you know, basically the company had abandoned. My boss went there and said it was down and I went there and I was like, well, it doesn't appear to be down. Oh, look, it's offering me a flash update. I'm reasonably certain that's not cool. (Laughs) 

Dave Bittner: [00:13:35] (Laughs)

Craig Williams: [00:13:36] So we started taking it apart and that was the one that had the encoded blob inside of it. Right? And so we started decrypting it, taking it apart. And it turned out it was a really well-known piece of OS X malware, basically a piece of - I don't want to say just adware, because that doesn't do it justice. I'm drawing a blank on the family name, but basically it would install itself into the system so that it would intercept calls to the browser and inject ads in the background. I think it's really important for people to realize that ten years ago OS X didn't have this type of problem.

Dave Bittner: [00:14:06] Right.

Craig Williams: [00:14:07] Well, these days OS X is as popular as Windows. Right? So all the problems that we have with Windows are going to be in OS X.

Dave Bittner: [00:14:16] When you say "as popular," you mean popular with users, not necessarily with the bad guys yet, but they're heading in that direction?

Craig Williams: [00:14:23] I want to say they're already headed in that direction.

Dave Bittner: [00:14:25] OK. They've arrived. They've established a beachhead.

Craig Williams: [00:14:28] Yes. I think they've established a beachhead, and we're not really good at seeing it because most Mac users don't have any sort of antivirus.

Dave Bittner: [00:14:35] Oh, interesting.

Craig Williams: [00:14:36] Yeah. You know, I know Apple does a really great job of looking for malicious DMGs, but one of the very first things that this malware does is it went in and disabled the system that looks for signed binaries. Right? And so, by doing things like that, it basically allows it to take full advantage of the system. And so, if you look at the blog, you'll notice there's a chart, a sequence of 1 to 9. And so, this is the redirection system that I mentioned. And so, I wanted to be very clear to anyone looking at the blog, while this particular chain only had a sequence of nine different sites that are kind of ground through in order to get to the actual malware, as I was knocking these down, right, as Matt was knocking these down, we would watch it change. So it was a redundant system. I want to say we ended up blocking probably dozens to hundreds of different redirection stops. We ended up scripting it and automating it because it was very clear that the system that was being used was not one that was basically made by a human. It was something that somebody scripted up the design, and so it was enormous. And so, that's really what blew me away. It was that for this adware - and it's "adware" with quotes, because I would qualify it as malware, but it's a piece of malware designed to show ads - basically had an enormous redirection system that we previously really had only seen with things like malvertising in order to distribute the software.

Dave Bittner: [00:15:58] And they're making money how?

Craig Williams: [00:16:00] So, historically, when we see things like this, they make money through the ads. They make money by installing third-party software. One of the very first things we looked at from a cross-platform malware perspective was one called Kyle and Stan. And the reason it reminds me of this, when you bring that up, is it would actually pass the dollar value encoded back to the server. And so, if the malware installed, you know, somebody's piece of malware, well that would get called back as, like, you owe them a dime or a penny or whatever. 

Dave Bittner: [00:16:32] Mm-hmm.

Craig Williams: [00:16:32] So they do get paid by the software, they do get paid by the ad, generally. And so that's really how these situations work. And, you know, think about it. We're comparing, like, ransomware and cryptomining. Right? Well, if they had installed typical malware, maybe they'd have gotten some accounts, maybe that would be worth a little bit of money. However, much like cryptomining, if instead you're injecting ads into the system constantly, and have a very small yet very consistent revenue stream, if you can do that in a large enough scale and if you can do that regular enough, well, number one, it's not high enough profile for most law enforcement to bother with.

Dave Bittner: [00:17:08] Hmm.

Craig Williams: [00:17:09] Number two, are there really any significant damages? You're just injecting ads and making the user experience unpleasant, but you're not damaging data, you're not damaging the computer. And number three, chances are the user is not going to fix it and you're going to continue to have income for a while. So, you know, I think there's advantages to this, and I think that's why bad guys are looking at it. And I think that's what we kind of wanted to put these two out there together, to show people the problem with some of these potentially unwanted programs.

Craig Williams: [00:17:36] And that kind of gets us to the last part I wanted to talk about today, and it's not necessarily to do directly with the blog post, but it's one of the things that I see constantly. People advertise apps in app stores, you know, like, hey, would you like a free VPN.

Dave Bittner: [00:17:51] Right.

Craig Williams: [00:17:52] Or hey, would you like free antivirus done in the wire? And you know, if you see that, you should run in terror. (Laughs) 

Dave Bittner: [00:17:59] (Laughs)

Craig Williams: [00:17:59] There is no free VPN, right? You're taking your secure traffic and you're just giving it to some guy in some other country, or some girl in some other country, and maybe she has nefarious ideas for it. You really don't know. So I think when it comes down to programs like that or programs like this or fake flash updates, users need to be terrified. They need to realize that that's a bad idea. No one offers that for free.

Dave Bittner: [00:18:28] So, in terms of defending against this malicious advertising, from an enterprise level, like you mentioned before, you know, defense in depth, what sort of tips do you have? Do you have any specific tips?

Craig Williams: [00:18:42] Well, I think the main one is to make sure that you're using a DNS provider that provides some level of security. Right? And there's lots of good free ones out there, right? Personally, I love OpenDNS because we own it, and I get telemetry from it if people use it. Come on, guys, use it. (Laughs) 

Dave Bittner: [00:18:57] (Laughs)

Dave Bittner: [00:18:57] But, you know, Google provides it. There's some other ones out there and they provide varying degrees of security. You know, I think that's one good layer. You know, another layer is making sure you have some sort of security client on the endpoint. Right? And that could be antivirus. That can be something more advanced like AMP. It's just got to be something that you have on that endpoint in case something silly happens, you click the wrong thing and the file comes across, you need something to intercept it and fix it. Right? And I think, you know, the third thing is obvious, right? Patch. You know, you never know when you might be directed to a malicious site. So patch. You know, and if you can't patch maybe the built-in browser, well, install a secondary one you can patch and use that for your primary browser.

Dave Bittner: [00:19:38] You know, I think we've all been through this experience, particularly on our mobile devices, where you're minding your own business, browsing from site to site, you visit a legitimate site and suddenly you're your device gets taken over with that message that says, congratulation, you're today's, you know, five-hundredth visitor. You're gonna get a, you know, a free iPhone...

Craig Williams: [00:20:01] (Laughs)

Dave Bittner: [00:20:01] ...Or a free iPad or a free car or something. And obviously, that's frustrating. Can you give us some insights, first of all, what is what is likely to have happened when we experience that?

Craig Williams: [00:20:15] Well, a lot of times that's just an ad, right? And that ad may link you to a site trying to get your personal information or, you know, to even install malicious software or potentially unwanted program. I think that's very, very common.

Dave Bittner: [00:20:27] Right.

Craig Williams: [00:20:27] The one that I worry more about is when I go to a site that looks legitimate, the page pulls up, and then all of a sudden I'm being redirected through dozens of sites. Right? That will never happen from a benign perspective. It just doesn't.

Dave Bittner: [00:20:41] And can you - when that redirection, that bouncing from site to site happens, can you see that happening? Is that happening in plain view?

Craig Williams: [00:20:49] Yes. Usually you can see it happening. You'll notice your URL is changing very, very rapidly. And you'll notice that it's usually got some sense of randomness in it, like at the back of the URL or something. And you'll wonder, why on earth am I going to this site? Well, the reality is you're going to a site that the attacker doesn't want people to know about, and they know that if you do end up on the last site, the site with the landing page, and it gets blocked, well, they have a redirection chain of, you know, a dozen sites to get there. They can simply point that last link or one of the other links to somewhere else and still compromised users.

Dave Bittner: [00:21:21] From the website that's hosting the ads, from their point of view, is there anything that they're doing on their end to try to prevent this sort of stuff? Are they doing any analyzing or filtering of their own?

Craig Williams: [00:21:35] I don't want to say they're not, because I know there's a lot of attempts to do something good.

Dave Bittner: [00:21:38] Yeah.

Craig Williams: [00:21:38] What I can say is I haven't seen anything super effective.

Dave Bittner: [00:21:42] OK.

Craig Williams: [00:21:43] Right? Now, you've gotta remember from their perspective, they may not even see what's happening. Right? You basically go to their site, you see their ad, and then you get linked off to another site from a, you know, a hidden frame or a link hidden somewhere in the ad. They're not really going to see that.

Dave Bittner: [00:21:57] Yeah.

Craig Williams: [00:21:58] So, they're not even going to necessarily know what happened. Which is why it's so difficult to be put in a position where you're hosting a site with ads, because if you are compromising your user base, you may not notice. And at Talos, we have reached out hundreds of times to these sites that unknowingly are hosting these ads. I mean, we're talking anything from like, a major news site to utilities and everything in between, you know, government sites. Even some of the more sketchy businesses were more than happy to help so they don't compromise their users.

Dave Bittner: [00:22:27] Right. But I think that so, you know, getting back to that thing about the pop-ups on the mobile device, I think that's one of the really frustrating things about it, is that for folks who want to try to do the right thing and report this, to feel as though that's really not going to be effective. There's really no good way to report this to someone who's really going to be in a position to do anything about it.

Craig Williams: [00:22:51] Well, I mean, you know, there's always the good folks at Cisco.

Dave Bittner: [00:22:53] Tell us, do you really want to open yourself up to all those emails, Craig? I suppose it is your job. (Laughs)

Craig Williams: [00:22:59] Yeah, we actually have a system in place. You can go to Cisco Talos and go to our Reputation Center. It's at the top of the page. 

Dave Bittner: [00:23:07] Uh-huh.

Craig Williams: [00:23:06] It's where you file disputes for sites that should be blocked or sites that are blocked that shouldn't be. So, by all means, if you have information, we'd love to have it. Now, the reality is a lot of these sites, they get compromised. It's not even necessarily an ad sometimes. Sometimes they'll use an exploit and inject it into the main page of the site. 

Dave Bittner: [00:23:25] Mm-hmm.

Craig Williams: [00:23:24] Those typically get cleaned up pretty quickly, so sometimes by the time we see it, it's already gone. But luckily, due to our telemetry systems and our sandbox and all our automatic stuff, we do catch a lot of these very, very quickly.

Dave Bittner: [00:23:38] Yeah, that's an interesting point. I mean, by their nature, I suppose a lot of these campaigns are fleeting?

Craig Williams: [00:23:43] It depends on how it's implanted. Right? If it's on an advertisement site, then it's going to be popping up randomly all over the Internet. Right? If, on the other hand, maybe the victim had a WordPress site for their recruiting portal. Well, somebody could, you know, use a WordPress exploit. There's like a new one, what, every 138 days?

Dave Bittner: [00:24:03] (Laughs) 

Craig Williams: [00:24:03] You know, they can use that to actually edit one of the pages and put it in there, in which case you'll see it until the person who owns the website notices it. We report a lot of those. And so there's a lot of different ways to go about it. But from a user's perspective, it's all the same. Right? You're going to see a malicious link embedded in a website, and that's where you rely on either, you know, your endpoint security system, your DNS security system, or maybe even something like Firepower in between to take care of that and mitigate it for you. And you know, when some people say defense in depth, it's not a marketing term. I mean, that's what it means, is have overlapping security so that if one product doesn't see it, because maybe it's not an exploit on the page, right? So that means Firepower is not going to block it.

Dave Bittner: [00:24:44] Mm-hmm.

Craig Williams: [00:24:45] Maybe it's a domain known to be associated with nefarious activity, and so that means instead, you know, a DNS security system like Umbrella is going to say, oh, you want to look up supermalware dot com, I'm not going to let you do that, you're making a mistake, bud.

Dave Bittner: [00:24:57] Right.

Craig Williams: [00:24:57] And so that extra layer can protect you.

Dave Bittner: [00:25:03] Our thanks to Craig Williams from Cisco Talos for joining us. The research is titled, "Malvertising: Online Advertising's Darker Side." We'll have a link in the show notes.

Dave Bittner: [00:25:13] Thanks to Juniper Networks for sponsoring our show. You can learn more at juniper.net/security, or connect with them on Twitter or Facebook.

Dave Bittner: [00:25:23] And thanks to Enveil for their sponsorship. You can find out how they're closing the last gap in data security enveil.com.

Dave Bittner: [00:25:32] The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technology. The coordinating producer is Jennifer Eiben. Our amazing CyberWire team is Stefan Vaziri, Tamika Smith, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Nick Veliky, Bennett Moe, Chris Russell, John Petrik, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.