SolarWinds, SUNBURST, and supply chain security.

Dave Bittner: Hello everyone, and welcome to CyberWire X, a series of specials where we highlight important security topics affecting organizations around the world. I'm Dave Bittner. Today's episode is titled SolarWinds SUNBURST and supply chain security.

Dave Bittner: The SolarWinds Orion SUNBURST exploit forced organizations to determine whether, and to what extent they'd been compromised. It's not enough to eject the intruders and their malware from the networks. Affected organizations also need to know what systems and data had been breached, and for how long. The adversary behind sunburst has advanced, quietly breaching the perimeter and moving freely to access steal or destroy business critical data and to disrupt operations.

Dave Bittner: We begin the show with my conversation with Ryan OlsonVice President of Threat Intelligence, of Palo Alto's Unit 42. Later in the show we're joined by Bill Yurek, President and Founder of Inspired Hacking Solutions, and we'll conclude our discussion with our show's sponsor, ExtraHop's Matt Cauthorn to discuss the challenges of detecting these advanced threats, and to share insights from behavioral analysis on what the new breed of threat actor is doing inside our networks.

Dave Bittner: A program note, each CyberWire X special features two segments. In the first part of the show, we'll hear from industry experts on the topic at hand, and in the second part we'll hear from our show's sponsor for their point of view. And speaking of sponsors, here's a word from our sponsor, ExtraHop.

Dave Bittner: Ryan Olson leads the Palo Alto Networks Global Threat Intelligence team known as Unit 42. His responsibilities include identification and tracking of threats around the world and across all major industries.

Ryan Olson: Initially for us, this started on December 8th when FireEyecame out and said that their red team tools, the tools used bytheir red team to attack their customers, to test their defenses had been stolen. And at the time they didn't say much about how the tools had been stolen, just that they had and they were releasing signatures so that security vendors like us and other defenders could go and make sure that we could detect those tools and protect against them.

Ryan Olson: And it wasn't until five days later on December 13th that they revealed how they were compromised, that it was SolarWinds software itself had been compromised. That had enabled the attackers to have a initial foothold inside the FireEye network and then to eventually steal these tools. And it was really that Sunday night when all of us realized that this is a big deal. SolarWinds at the time wasn't extremely broadly known outside the technical community, the network Orion is a network monitoring solution. The scale of how many customers they had when we started digging into, 30 something thousand customers, they said 18,000 or so used Orion and had active licenses at that time. That's massive because it's not 18,000 laptops. It's not 18,000 individual people's computers. This was 18,000 network monitoring servers that are placed at the core of the network so that they have access to databases and systems and routers and networking equipment, so that they can monitor and manage them.

Ryan Olson: And at that point, that was when we all started thinking this was going to be a big deal and everyone is going to spend the next few months trying to understand whether they've been impacted, and if they have, what was the extent of the impact?

Dave Bittner: You know, we talk about defense in depth and I think a lot of folks have an aspect of their defensive measures that include behavioral things. So we're looking for unusual activity within their network. Was this able to go undetected to even those sorts of things?

Ryan Olson: Well it certainly could have made a lot of tools. But in our case, Palo Alto Networks was a SolarWinds Orion customer, and we actually had a security incident at the beginning of October where our Cortex XDR, our end point agent which does behavioral monitoring, detected the attempted execution of Cobalt Strike on that system and prevented it. It stopped execution from running on our SolarWinds Orion server.

Ryan Olson: So that behavioral monitoring prevented that attack from having a further impact for us. At the time we didn't connect that to a SolarWinds Orion supply chain compromise because we didn't have that data. We knew that our SolarWinds server had Cobalt Strike running, or attempted to run on it. We had blocked it at that time. We investigated how did Cobalt Strike get on this system, but Cobalt Strike is used really, really broadly. We've got 25,000 samples of Cobalt Strike used from actors from the very top to the very bottom. Red teamers and everything else. So we weren't able to determine at that time that there had been this supply chain compromise, and we investigated it because we thought it was significant and serious. But it was only an attack on us at that point from an unknown actor that we had blocked.

Ryan Olson: And you couldn't stop this. You certainly could. We couldn't stop SUNBURST from being downloaded. That was downloaded legitimately from SolarWinds, but when they went to take additional action, that's when our behavior monitoring solution said, "Hey this is out of the ordinary. It looks like Cobalt Strike. Let's stop it and alert the SOCs so that they can go and perform their investigation." So when FireEye released their blog saying that on the 13th they'd detected this as coming from a SolarWinds Orion supply chain compromise, that's when we connected the dots on those things and said "Oh that attack that we saw at the beginning of October was probably through the same channel." We then confirmed that SUNBURST had been installed on our Orion server that we'd shut down at that time and was able to say this was the same attack that we had blocked. We just didn't experience impact from it, so there wasn't an additional investigation that went to the point of going and really figuring out where it came from. We just know that it got on this system through SolarWinds somehow.

Dave Bittner: And was that kind of an aha moment that rippled through the industry as people took a look at their logs? Like exactly what happened to you all? You know, you had an incident but you went through its normal range of checks and perhaps you thought that was that?

Ryan Olson: Yes. I think everybody in the wake of the disclosure that SolarWinds themselves had been compromised and the software had been modified, everyone had to go through a couple of processes. One was determine whether or not they have SolarWinds Orion in their network. That's not something everyone could quickly determine. Generally we've had lots of conversations about software inventory, people knowing if they have a huge network with thousands of devices, what piece of software is running on every single one of them.

Ryan Olson: Not everybody can go and do that quickly, and so that was the first step. Do you have Orion running or not? Second was, do you have a version of Orion that was running that got the SUNBURST update? Because you might have had an old unlicensed version, a version that wasn't getting automatic updates. In some cases, people were, found that they were safe simply because they were running Orion, but they weren't getting the automatic updates which is problematic for other reasons.

Ryan Olson: And then beyond that, once you discover you had SUNBURST on your system, now you've got a compromise in the network at a certain date, probably months ago, and what you're trying to determine is, did they take any action with that? Because that was the big chasm that existed in SUNBURST intrusions. Approximately 18,000 organizations probably got that update. They downloaded the DLL, at that point the DLL makes some DNS requests, basically to tell the attacker, "Hey it compromised this network" and encode the name of the actor directory domain that the system is partnered into a DNS request basically to check in and say "Hey, this network is compromised."

Ryan Olson: The attacker then has options. They can say "Ignore that, don't give them any additional commands." Or "I've selected them for additional intrusion," and at that point they were telling it, "Go and download Cobalt Strike and execute that." And it was a bespoke version of Cobalt Strike for each individual intrusion. They all had their own unique commandercontrol domain and server and they got to choose, let's go for these government agencies. Let's go for FireEye, let's go for Palo Alto Networks. All these organizations were then chosen to be targeted later on.

Ryan Olson: But that's a pretty small number. It's not 18,000. I think it's probably somewhere around 100, at least that's what I've seen from the US Government around what they think organizations who saw that further impact was. But that opens a big door. If you're in that 18,000, discovering whether or not you are part of that select few who was further targeted, was not straightforward.

Dave Bittner: Yes, that brings up the question of how do you suppose this changes things going forward? Does it? Is this a deflection point where people are going to have to recalibrate some of the ways that they approach their security?

Ryan Olson: I expect it will cause people to shift a little bit. But it shouldn't necessarily. So this is not the first offer supply chain attack. There have been quite a few of them in the past. Some of them really impactful, NotPetya in particular in 2017, where you saw a Ukrainian software company compromised, malware deployed to their update channel, the same way with SolarWinds and then a worm that was built into that, which shut down networks all over Europe, and caused a huge amount of damage.

Ryan Olson: That should have been a big wake up call for people to say "Oh, software supply chain security is important." And I think there's different directions that deflection that you described it could go. People could say let's focus entirely on trying to make sure our vendors are doing things right and they are securing their software properly which I think would be certainly a useful thing to do. But it only solves one little component of the overall threat landscape.

Ryan Olson: Whereas the alternative is, let's make sure that we're building networks and monitoring our end points in a way that would help us detect this and ideally prevent it the next time it happens, because it will happen again. If someone had deployed a zero trust architecture where that SolarWinds Orion server can only talk to the systems it absolutely has to, it would have been able to talk to the update server, get the SUNBURST Trojan, but it wouldn't have been able to go and download Cobalt Strike. It would have been blocked at that point.

Ryan Olson: Same with an end point behavior monitoring solution. If you were using something that could automatically detect when the system deviates from its normal pattern and stop it from doing that, or at least alert your SOCs so that they can respond, you'd also be in a secure environment. And those things aren't about software supply chains. They're about all security. They would stop an insider threat the same way. They would stop and security internet that came in through another type of exploit. Those are just sort of good practices which are now accessible, they're not necessarily easy to deploy, getting perfect zero trust is not simple for a network to have.

Ryan Olson: But, moving in that direction helps solve all of these challenges, rather than just software supply chain which is important, but if we over rotate and focus on it too much here in 2021, then we ignore the fact that organizations are getting hit with ransomware and business email compromise every single day, we may end up losing the forest for the trees.

Dave Bittner: Bill Yurek is a certified cyber crime investigator, cyber attorney and critical infrastructure protection specialist and is owner and president of Inspired Hacking Solutions, a cyber security consulting company. His experience spans over 35 years including time as both a federal agent, federal prosecutor and congressional investigator.

Bill Yurek: You know it's one of those things, I don't think I'm alone in this, a lot of people said "Well I'm not surprised it happened because it was inevitable." But really for me I guess is sort of a mixed silver lining slash painful lesson learned was, the idea that even those companies, those entities that are in the security business, that do this for a living, that are established, can in fact be compromised. There's no such thing as too big to fail in the cyber war environment, the cyber crime environment.

Bill Yurek: And for me that's the first thing that struck me. When I would teach for a small business administration, one of the first things we taught, like day one, was, there's no silver bullet. You're not going to find a company that's going to help you whether it be monitoring, whether it be with instant response. That's going be the silver bullet. They can just say "Hey look, I've got the A team and that's OK, I don't really need to worry any more." It just doesn't work that way and to me I think that was my first reaction to this was that it's sad to see it but it wasn't inevitable that some of the very things, processes or technology that we counted on to believe this is part of our defensive system, can in fact be compromised. Can in fact be defeated. And that is a harsh reality.

Dave Bittner: Talking about small businesses, would you say it's accurate that a lot of small business owners think they're too small to be noticed. They don't have anything. What do they have that they're interested in?

Bill Yurek: That's such a common thing, and as a matter of fact that's one of my things that I try and get out there right off the bat, is "Why me? Why would they care about me?" And there's a couple of things I try to point out. Number one, the bad guys a lot of times, they may or may not be specifically targeting you, but some are targets opportunity. You're weak or something. And if they compromise you they're not going to say "Oh OK they're just this." They're going to try and find a way to benefit from compromising you.

Bill Yurek: And so you might say, "Well what would I have to offer Bill?" A number of things, and you know the example I like to use is let's say you had a carpet cleaning business. You have three vans, you have 15 people that work for you. You have a carpet cleaning business. "Bill, why would they care about me?" Well you know what if all I got was your customer list and I could sell that off as a mailing list. I could find people who are competitors to you or just want to get a list of 300 people in a particular region that are known to spend upwards of $10,000 on carpet cleaning. That are known to have a valid email address and mailing address and phone number. That's worth money.

Bill Yurek: Now, it might not be worth a lot, but it's worth it. If you keep your clients information, you keep how they pay, their credit card information, well now you're a little bit juicier right? And now I can get that as well and now maybe between those I can do a number of things. I could of course credit card theft and probably going to try and bundle those and sell them. But now I can wrap those things together right, I can do identity theft movement. I can try and get a loan in your client's name. I can try and file in your name somebody's trying to use your access to try and put a debit towards that and maybe they won't notice it. Maybe I can get a mortgage in their name.

Bill Yurek: And then what about your employees? What about those 15 people? Do you have medical information? Do you have their contracts? And now I get a treasure trove there right? I can get your insurance information. So if you're a small business, you're inherently vulnerable because I hate to say it, the pure nature of the average small business starting out, they don't have the resources or they don't want to put the resources towards cyber security. And I'm not saying that's right or wrong, that's a decision to make. But you're vulnerable. You're one of the ripest juiciest targets out there, and you're not worthless, it's not like you're not worth compromising.

Bill Yurek: Maybe you won't be the first thing on someone's list, but if I'm able to compromise you easily, why not? Why not exploit everything I can out of you? And then there's always the third party aspect. Who are you contracting to? Who are your business partners? Who do you have trusted relationships with? What cloud environments are you connected with? And how can I exploit that? So you can't sit back these days and say "I'm just little old me, who would care about me?" It's a pretty hard thing to do these days when you're in an era of electronic commerce, electronic funds transfers, where so little cash these days, where everything seems to go electronic.

Dave Bittner: You know, it strikes me that every time we have a major event like this, you see folks saying "Well this is a wake up call." And I think there's something to that. But also it happens every time with an event like this. To what degree do you think that's true in this case? I mean is this going to change how we approach things going forward?

Bill Yurek: Whether it will I don't know. Whether it should, yes it probably should. I mean there's a number of lessons learned to come out of this, in other words just because it's a big name doesn't mean it's faultless, in other words you can count on them to protect you. Whilst it doesn't mean they're faultless in terms of protecting themselves, I can never protect myself like name the big company. Well maybe you don't want to. I mean some of them don't do that good a job.

Bill Yurek: And there's a number of things that come out of this. Digital signing of updates for example. It was I think a great lesson to be learned here because it just isn't working the way we're doing it now. So there's always lessons to be learned. Do I think they will be learned by those entities or organizations that A, have the staff, whether it be contracted out or support or their own indigenous capability? To look at this and really apply it to themselves?

Bill Yurek: And B, those who have that and want to, and take the effort to apply it to themselves, absolutely. There are some great lessons learned to it. But for those who just say it's just one more sign, and you kind of touched on this with the idea of persistence, over time and you've heard this many times probably in cyber security, a sophisticated bad guy given enough time, will in fact succeed. The basic things are still there. The basic rules are still there. Access control. Lease privilege. Those are still there. But now you have to apply them to a different environment, a different set of data, different data flows, different storage, different levels of security. Shared responsibility model.

Bill Yurek: And even if you have the lessons learned from the last time, they're not perfectly matched with this time. It doesn't always click just right. Oh yes, it's the same. It's everything taken a step further. Do I think this was a...? I think as I said earlier on, I think this was a sort of shot across the bow bigger than many others, because of the nature of the very entities that were compromised.

Dave Bittner: Matt Cauthorn is vice president of Cloud and Security Field Engineering at ExtraHop, our sponsors for this CyberWire X episode.

Matt Cauthorn: The first thing that we did is we went back in time. So we very quickly reviewed when the disclosure came out we reviewed the indicators, and in particular the DNS and IP based indicators as well as with an eye towards subsequent artifacts which we'll probably get to. So when we had the domain, the stage one and stage two C2 indicators, to suspicious domains and IPs, there was quite a long list, we compiled that list and we put a script out on GitHub for our customers to download and ask questions retroactively.

Matt Cauthorn: And so one of the interesting things about this particular one is to me, it really blurred the lines between retroactive detection and threat hunting and incident response, because usually those things are sort of cast in these time series buckets. One happens first, you detect and then you respond and then retroactively sometimes you speculatively interact with your data to find indicators of compromise. Well this was like a mash up of all three.

Matt Cauthorn: And the big condition here was that you actually had network based data, because a lot of this stuff wasn't logged and in this particular case end point was so actively evaded, and by the way that's not a deficiency on their part. It's just an artifact of the way these solutions instrument themselves, or are installed. In the sophistication of SUNBURST itself, they actively evaded the end point and they were very low and slow and very patient, especially in the early stages.

Matt Cauthorn: So what we did is we provided this list to the customers in the form of a script that they could run and they could automatically query the system for these indicators. And unfortunately many found indicators.

Dave Bittner: Help us understand here, looking for your insights on why this particular breach set everyone back on their heels so much? It seems as though this one really caught everyone's attention and set our imaginations running.

Matt Cauthorn: So the thing, at least for me, and I think I can speak comfortably for several of the people out there, but at least for me having come from operations in security myself in my past, is that this one basically is analogous to an Amazon package coming to your door, that had a specific in Amazon branding delivered by the truck associated with an Amazon order. Very deterministic.

Matt Cauthorn: But when it's opened, now you're in trouble. And so what it did is, it drop shipped you for lack of a better way to put it, maybe to brutalize the metaphor, but it drop shipped you right into the east west corridor and it happened to do so inside of a software system that had privileges. Because in most cases, I don't know about every case, but in most cases, systems like that that do monitoring, they often ask for database credentials, domain credentials and many many other things in order to do synthetic transactions and other monitoring tasks.

Matt Cauthorn: And so A, it's in the east west corridor, in other words it's next to critical assets. B, you're used to lots and lots of transactional noise coming from this thing and then C, it's running with privilege. The coalescence of those three factors is what really landed this thing so hard on the industry in my opinion.

Dave Bittner: Well let's dig into this notion of behavioral analysis. First of all before we look into how it applies to something like SUNBURST can you give us a little bit of how you define it? How exactly it works?

Matt Cauthorn: Yes, so it's a worthwhile exercise to think about how you define behavioral analysis, and it's very easy to conceptually get some intuitive understanding. But building out that intuition into a set of concrete expectations is really much more important for a security practice. So for us at least, behavioral analysis means understanding behavioral patterns with transactional, at the end of the day on the wire for data in flight, we're a covert analysis engine from the network's perspective. And we analyze transactional behaviors because transactions is where a lot of the rubber meets the road from a behavioral analysis perspective, and so it's understanding the nature, the assets in question, the sort of participants on the network, and the nature of the transactions that they're both serving and consuming.

Matt Cauthorn: Because in heterogeneous environments, and in large environments, you're seeing lots and lots of server and client activity regardless of the formal role of a system and so it's really really important to understand the disposition, the protocol mix, the behavioral mix and any behavioral changes over time. So that's, for us, where we really land hard on network based behavioral analysis. Now part of behavioral analysis, and this is really worth commenting on. Part of behavioral analysis might mean a simple pattern that you can match in real time and flag some event or fire off a detection or an alert.

Matt Cauthorn: That too is a form of behavioral analysis, but I would encourage the listeners to think about network behavioral analysis as a much much more comprehensive art in the modern era, because it used to be pattern matching, without long term look back and historical trends and feature extraction and the game has completely changed now and it's all of those things and more actually.

Dave Bittner: Is there the ability to detect for example, if someone's trying to boil the frog, making slow changes over time to try and fly under your radar to mix metaphors?

Matt Cauthorn: Part of the implication of behavioral analysis, and I just danced around this, but we have a very declarative term for what I'm about to describe. I talked about matching suspicious patterns, which is like a rules based detection methodology, and then I talked about historical trends and feature extraction. And that taken together, represents a spectrum of detection capabilities and behavioral detection capabilities.

Matt Cauthorn: And one of those behavioral categories is the low and slow attack or the first time observed attack, or an unusual end point has presented itself to the wire, but it's actually acting different than its peers even though it's trying to emulate them. Like all of these are behavioral categories that you need to be mindful of when you're in the business of network detection and response.

Dave Bittner: Well let's pivot back to SolarWinds and the SUNBURST exploit. How specifically would behavioral analysis apply?

Matt Cauthorn: So, when the disclosure first lands this was zero day style, like boom here it is, FireEye wrote it up, they did a great job doing so. And all of a sudden the entire industry was like in retroactive analysis mode. Sort of step one is like, do I have artifacts in my environment? And therefore I need to look back in time, regardless of the data source, whether it's logs or end point or network. You need to be able to go back in time and have captured that analysis to answer those questions.

Matt Cauthorn: So that's stage one. Then stage two is the subsequent, in my opinion it was a very sophisticated attack from the stage one and stage two command and control. Some of the intelligent decisions it made at the time of initial compromise. From there south of that, once they decided "OK I've got this environment now and it's game on." Then you're falling back to very standard TTPs. That many many solutions out there, including ours, are able to detect.

Matt Cauthorn: And some of them are network based so you need network and others are more end point based, but your end points have been evaded, and so it was a really interesting cocktail of problems. But after the initial compromise, and that initial two week period, because it was quite good at, as I said, being very patient, then it's using very standard stuff. One can think of DC Syncspotentially or Golden Tickets or other privilege escalation mechanisms, whatever. And from there it's lateral movement in the east west corridor and trying to get actions on your objectives.

Matt Cauthorn: So it flipped from retroactive analysis mode to continuous behavioral analysis mode. And interestingly, we were able to go back. We've got a blog post on this from our chief data scientist. We were able to go back and in the time period in question, we saw 150 percent increase in our behavioral detections in the lateral, and the east west corridor, in the server to server corridor. During the time of SUNBURST. And it wasn't the traditional indicators. It wasn't the sort of IP based or DNS based. These were behavioral lateral movement and actions on objective privilege escalation, things like that.

Matt Cauthorn: So we saw this correlation that was pretty unfortunate actually, but it was there.

Dave Bittner: How do you recommend that folks get started with this? When someone is beginning the journey and they want to integrate behavioral analysis into their defensive measures, what's the best way to begin?

Matt Cauthorn: First of all, and this might sound flippant or whatever. I honestly don't mean it that way. Understand threat modeling is really important and threat modeling doesn't have to be this advanced sort of thing. You don't need a sophisticated team of consultants to come in if you don't have inhouse folks, but there's great resources out there. One of them's called the threat modeling manifesto. Start with that. Ask yourself a set of high quality questions.

Matt Cauthorn: That starts to encapsulate your risk and starts to prioritize the technical and physical and personnel level controls around your critical assets. So that's step one and two. Then you're going to start to back into your choice of solutions and for that I would say from a behavioral analysis perspective, don't think that machine learning is some fancy panacea. Machine learning is a pragmatic and very required strategic asset for the SOC nowadays. Just given the asymmetry of the problem which heavily favors the adversary.

Matt Cauthorn: So first of all accept the idea of machine learning, but also have a balanced perspective. It's not a panacea, it's not a be-all end all. And then take advantage of solutions that can both invoke machine learning and time series intelligence with extracting behavioral features and artifacts. Then bringing them to bear, especially in concert with one another. And this is part three.

Matt Cauthorn: We talk about three pillars, but I added a fourth based on another friend of mine in the analyst community who finally shook me hard enough by the shoulders to convince me he was right. You know, there's end point which is mandatory. There's a SIM which is mandatory. There's covert network based analysis which is mandatory. So those three data sources are the operational points of leverage, and they all nowadays can and should work in concert with one another because each one of us have our own constraints.

Matt Cauthorn: We work on the network, we don't have a concept of the individual process that's running in memory that we see the behaviors as that process presents itself on to the network. But we don't have the call stack say of that process. And similarly, with logs, they've got this system;s sense of awareness of itself and the process is running on it and that's very very useful as well. And taken together, collectively, it gives you levels of telemetry that are really really hard to get especially if you treat them again as a one plus one equals three kind of thing.

Matt Cauthorn: Then lastly, threat intelligence. That's the fourth that I now agree. Threat intelligence, look SUNBURST was exhibit A for the power of the commons and the collective especially now. And so you can't ignore the power of the commons. Get yourself some good threat intel. And so now taken together, you've got the sort of springboard for a SOC strategy that doesn't have to be complex. You don't have to be a level four out of five to implement this stuff. You need a couple of capable analysts and some good SISAdmin talent to help run the stuff if needed. And now you've got a strategy that is going to really really set you up and stack the deck in your favor, such as you can.

Matt Cauthorn: Look supply chain attacks, and you know this better than I do David right? You've been talking about supply chain for a very long time. In fact I was on about a year ago and I think I talked about it then. So this one was really really different and I'm afraid we're going to end up with the sort of copycat killer model for some of these more advanced supply chain attacks, because this one was just so devastating.

Matt Cauthorn: And I think that as a vector, if I'm an adversary and I don't have to go through all the trouble of convincing you to click on the link and then elevating my privileges and doing all that sort of scaffolding work ahead of time, if you can just fast track into a doc or containers supply chain, you know via NPM or whatever, that's got me pretty concerned and it's a very difficult class of detection problem to solve frankly. It really is.

Dave Bittner: Our thanks to Bill Yurek of Inspired Hacking Solutions and Ryan Olson of Palo Alto's Unit 42 for sharing their expertise and for ExtraHop's Matt Cauthorn for providing his insights and for sponsoring this program.

Dave Bittner: CyberWire X is a production of the CyberWire and is proudly produced in Maryland, at the start up studios of DataTribe where they're co-building the next generation of cyber security teams and technologies. Our coordinating producer is Jennifer Eiben, our executive editor is Peter Kilpe, I'm Dave Bittner, thanks for listening.