Following DOJ indictment, a look back on NotPetya and Olympic Destroyer research.

Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Craig Williams: The amount of time between horrible campaigns and ones that aren't as bad is shrinking because bad guys are learning from each other, and if there's one method that works better than others to get either the reaction you want or the profit that you want, that's the avenue being pursued.

Dave Bittner: Joining us this week are Craig Williams and Matt Olney from Cisco Talos. We're discussing their NotPetya and Olympic Destroyer research.

Craig Williams: You know, I was glad that we're taking the steps to hold people accountable. I mean, obviously charging someone with a crime is not going to suddenly stop these type of actions...

Dave Bittner: That's Craig Williams.

Craig Williams: ...But I think if we don't start holding folks responsible, if we don't start making sure that we're drawing those lines in the sand, that when crossed, there will be repercussions, it's going to get even more out of hand. So I'm glad that we took these actions. I hope that we take more of these in the future. And I'm really happy Talos could play a part of it.

Dave Bittner: Matt, what are your thoughts?

Matt Olney: I was taken by the scope of the charges, the number of incidents – they referenced seven different incidents. And the lack of US focus, I thought was also interesting, in that these were – while certainly there were some American victims in the NotPetya event – when you look at Olympic Destroyer, that's definitely an outside-the-US sort of event. So it's very interesting that they went in that direction. And I was happy to see that we had actually, like, actively investigated three of those seven events, not just the NotPetya one.

Dave Bittner: Well, let's rewind the clock here and dig into some of the research you all have done here, the part that you all played with these folks, can we sort of go back in time? What's your first recollection of these folks popping up on your radar?

Matt Olney: Well... so, yeah-.

Dave Bittner: You can share. (Laughter)

Matt Olney: Yeah, this is a safe space – just you, me, and the CyberWire. Yeah, I would – and Craig's gonna probably panic – but, like, there was an interview with an interview candidate that we did at RSA a number of years ago who was trying to decide, you know, whether they go on Craig's team or my team, and we were talking about what their capabilities were. And the candidate asked me, you know, if I came to your team, what would you do? And my response was, you're going to go to Ukraine and you're going to assist them with the difficulties that they're having there. And so, we made a determination in the immediate aftermath of the BlackEnergy attacks that we were going to invest a lot of time and resources in kind of assisting the Ukraine government in dealing with the events that they were having and trying to kind of help them build an efficient and effective defensive strategy in the face of some fairly advanced and persistent actors.

Matt Olney: So, on that list of seven, the first event that we were involved in was actually the Ukraine Treasury and Finance Ministries. And what was really interesting in those – and I think I'm not mixing it up, there's been a lot that's happened in Ukraine. 

Dave Bittner: Yeah.

Matt Olney: One of the interesting things is that was the first time that we saw them – they were using disk wiper at that point to just corrupt the disks entirely. And we were able to deploy an effective strategy to shield the computers from that using our firewall software. And they were actually – that was the first time we saw them pivot off of the disk wiper stuff to using ransomware as their destructive capability. And if I'm remembering correctly, they were actually using normal Petya at that point, in those events.

Matt Olney: And so, it was definitely interesting to kind of see and kind of recognize that, hey, we're dealing with a human adversary here, because we're defeating them here and then they're countering with this, and we're having to consciously go back and forth with them. So that was kind of the earliest pre – you know, the kind of set up, the building of trust between Talos and Cisco and the Ukraine cyber police and other Ukrainian government entities, so that when NotPetya happened, I got a phone call while I was standing in line for Starbucks asking for help, instead of me finding out about it in some other way.

Dave Bittner: Yeah. That international collaboration, I mean, these are not skills that Ukraine had in-house, and I think Ukraine is very good at international collaboration, if that's the question. No, I'm thinking of – in other words, they reach out to your team, to Cisco and to Talos – Ukraine did not have their own threat intelligence – your capabilities exceeded their own

Matt Olney: I'm not gonna say they exceeded, but they definitely augmented and assisted them. You have to understand that Ukraine is a country that's been embroiled in conflict, and it's also embroiled in, like, a decision about its own future, because it's only very recently that it has come out from under Russia's control. And so – and the further you go back, the closer you kind of get to that kind of Maidan Square event where they kind of threw off that Russian control. It was only a few years before we arrived. And so, they are still in the process of solidifying their capabilities in the cyberspace, and what I would say is they're very capable and have more experience than just about anybody. But in terms of scope and scale, when you're operating at the kind of scale that we're talking about, it is always useful to have a partner when you're working on things. So I don't think that I was ever in a – I'll say I was never in a room with anyone working on Ukrainian cyber issues in Ukraine where I thought these guys were idiots.

Dave Bittner: Right.

Matt Olney: They were they were keenly aware of what was going on. They knew what they were facing. And what they were trying to do is assemble all the available tools and capabilities that they had so that they could best serve the people of Ukraine.

Craig Williams: I mean, if you look at that same idea, right, this is why groups like the Cyber Threat Alliance exist. You know, even large commercial companies like all of our peers in the industry – we want to work together. And it's not for lack of knowledge or ability on the part of any one company, it's just that we're stronger together

Dave Bittner: Right, and I mean, it's a really good point that Matt brings up, which is, you know, when you're Ukraine and your next door neighbor is Russia, you are going to have good capabilities. You must have good capabilities.

Matt Olney: Yeah. I mean, the fact that they have a functioning society at all in the face of what's gone after them is a testament to their skill.

Dave Bittner: Well, I mean, let's go through the timeline then. Walk me down the path – after that initial activity with Petya, where does it lead to next?

Matt Olney: So, I don't remember all seven, but in terms of us, the next thing that occurred – and as Craig rightly points out, in very rapid succession – was, first, WannaCry, and then not NotPetya, and then Olympic Destroyer. And so, the next case was WannaCry, and I'll only mention that real briefly to kind of set up the discussion about NotPetya, because WannaCry was like a crazy man on a rampage. (Laughter) Like, there was no sense to what was happening. It was just released and it went bonkers.

Matt Olney: And it was also between – of only WannaCry and not NotPetya, really the only two major international, globally impacting – everyone experienced it at the same time – sort of events that I can remember, where the timescale was in terms of hours instead of weeks or months, that occurred. And so, we actually had – I think our response was very good, but in terms of, like, the sanity of our response, we were sort of crazy in the background trying to, like, handle all the inbound information, and everybody wanted to help, and all the sales people wanted information, and all our customers wanted information, and we were trading information with our partners and standing up calls. And Craig was telling people, you know, it's not email, everybody settle down. And, you know, it was kind of bonkers. And so we kind of put into place this incident response system called Tasers (ph) that we've only used twice since then, but one of them was in NotPetya.

Matt Olney: And so, I got a phone call, like I said, and standing in line where our Ukrainian sales staff was like, hey, Ukraine cyber police are experiencing this and they'd like help. We agreed. Very shortly after there was a tweet from the Ukraine cyber police saying we're working with Cisco on this malware event. And also maybe the funniest tweet that I've ever seen from a country, where they had the dog with, like, fire everywhere, and they're like, this is fine. 

Dave Bittner: (Laughs)

Matt Olney: So, like, it's kind of like very gallows humor, sort of.

Dave Bittner: Right. (Laughs)

Matt Olney: So we activated our incident response thing and – which was great because what it allowed us to do is we essentially completely reorganized how Talos is set up. People that are on Craig's team ended up working under me. Some of our capability went over to work under Chris. And we kind of like – if you were doing reverse engineering of any kind, you were under this guy. If you're doing intelligence analysis of any kind, you're under this guy. And so, temporarily, we kind of re-architected, and then had a whole tracking mechanism so that when we came time to communicate with our customers, we had a really cohesive and very accurate, this is what has happened and this is what you need to do. And then that was before we had the opportunity to get an IR team actually into Kiev and on the premises of MeDoc to actually do a forensic analysis of what happened at MeDoc, which was the epicenter of NotPetya.

Dave Bittner: And can you give us some insights? I mean, when you get that team over there, when you get, you know, boots on the ground, as it were, I mean, what sort of things take place? What is that process like?

Matt Olney: Well, one, I would say – I would point out that NotPetya is thankfully not the norm. The way that all of these kind of things went down is they were very much in-and-of the moment, in kind of, like, phone calls, offers of assistance, except in we went to Ukraine.

Dave Bittner: Mm-hmm.

Matt Olney: Like, there were no salespeople, there was no – you know, it was very much like, hey, we're gonna be there tomorrow morning. We'll meet you there. 

Matt Olney: So, it was crazy in that sense, but it also allowed us to really have the most rapid understanding of what was going on. So, it took most of the day for them to do the forensic pulls off of the servers that were affected and kind of interview the MeDoc staff and get an understanding of how everything was built, what the inside was. The on-site team did a great job, but it was well into the evening by the time they had the drives. And so, they actually hosted those drives for us in the US. And then kind of about mid-afternoon our time, we started the forensic analysis and it was primarily me and a guy who's no longer at Cisco but who was fundamental to this investigation named Dave Maynor.

Matt Olney: So, me and Dave did the forensic analysis and determined how the Russians had breached the site and had gone into the web servers and had redirected all update traffic to this server in OVH. And that server in OVH was then redirecting back updates that would then deploy the NotPetya malware. And so, we figured that out – I think we figured that out at about 3:00 a.m. our time, and then we just stayed up overnight waiting for the sun to come up in Kiev and had a 7:00 a.m. Kiev-time phone call, but we were like, OK, this is what happened. And then the Ukraine cyber police were free to go forward and do what they needed to do.

Dave Bittner: Can we touch on the human side of this? I mean, you know, you mentioned you're pulling all-nighters, that sort of thing. I mean, is it fair to say that you guys are running on adrenaline? Probably a fair amount of caffeine as well, but are there concerns of not being at your best when you're running at that pace?

Matt Olney: Oh, a hundred – like, yeah, a thousand percent. And Craig – I mean, Craig always brings up the balance between speed and accuracy, right? And so, in what we were doing here, we had to be completely correct. And so, I essentially – the way it happened to go down is, I was actually who had the server that was kind of at the center of it. So I kind of found these error messages that kind of indicated this stuff. I looked up the manual of NGINX to kind of figure out what the error messages mean, and they implied this. And so I said, all right, Dave, here's my theory. And Dave was like, yep, that all checks out. And then, so we then to a fresher set of eyes in Ukraine, and said this is what we think happened and here is the evidence. So it was very much – and we did this multiple times – it was very much, this is where we started, here are the pieces of evidence, here's how we tie the evidence together, and this is our conclusion. And that conclusion is held up remarkably well over time.

Dave Bittner: I mean, it's fascinating in a way that, I suppose – I mean did time zones play to your advantage? That while they were sleeping, you were able to work, and vice versa?

Matt Olney: (Laughs) I don't think everything we've ever said times those have played to our advantage.

Dave Bittner: (Laughs) Perhaps I'm overstating it.

Craig Williams: So I do understand what you're asking, Dave. My team does make use of time-zone handoffs pretty frequently.

Dave Bittner: Yeah.

Craig Williams: It's one of those things that can help and can hurt. And when we were doing the events Matt mentioned – NotPetya, the ones before – my team did work out a system. We would have what's called a "hot handoff." And I think Matt's team probably does the same thing with a different name, where it's not an email, it's not just a doc you send somebody. It's, you get on the phone, you walk them through everything you found, why you believe what you believe, and then they basically go to try and prove your conclusions or not. Because, you know, one of the most important things to Talos is that the information that we provide our customers needs to be accurate so that they can ensure that they're defended properly.

Craig Williams: And, you know, as Matt pointed out, it really bothers me when I see people rush out incorrect assertions because we've seen so many defensive strategies that didn't help, right? Like, when the NHS shut down their email server with, you know, NotPetya, there was no reason for that. It put customers at risk, it hampered communications, and it didn't do anything because one company wanted to get a notification out quickly. And so that's something that we have strategies in place to prevent and something we take super seriously. And yeah, in those situations, having a global team is definitely useful because it gives you that second string to check your work, to make sure you're right, and to help get those communications written so that everyone else can be informed.

Dave Bittner: I had another sort of basic question here. I mean, do you have – is there an element where you're dealing with language barriers?

Craig Williams: I mean, most of the people on my team speak more than one language. I think the Americans are probably the weaker set because we only usually speak one or two...

Dave Bittner: Uh-huh.

Craig Williams: ...But everyone in Europe on my team probably speaks more than four. 

Matt Olney: More than four? 

Craig Williams: Yeah, we have a lot of people who cross a lot of country lines regularly.

Matt Olney: That's fantastic.

Dave Bittner: (Laughs)

Matt Olney: So, I mean, in terms of the Ukraine stuff, we definitely had the benefit of having Azim Khodjibaev on my team. And he is, you know, the child of immigrants, worked at DHS in the Office of Bombing Prevention, and came to us with that kind of national-security-focused background. But he's a fluent Russian speaker. Like, you can't tell the difference between Azim and someone off the streets of Moscow. And so, while Ukrainians don't always prefer to converse in Russian, they're all fluent in Russian. So, frequently, we had Ukrainians would speak Russian, he would then translated into English and then back, as their English failed them and I have no Ukrainian to speak of.

Dave Bittner: Well, let's move on and sort of wrap up our conversation today, talking about Olympic Destroyer and Sandworm. What was your involvement with those?

Craig Williams: Well, Olympic Destroyer is one of the ones that my team found. So, after NotPetya, obviously we suspected there would be an increase in similar attacks. And so, we went up and set up certain indicators in various systems to look for these attacks. And that's literally how we found Olympic Destroyer, was just preventative planning and having the detection technology deployed on our internal systems.

Craig Williams: Once we found the samples – I think we actually ended up finding them in VirusTotal – we knew it was something new. We had a good idea what it was doing. And, you know, we started our investigation, and we named it, and we put our write-up out there. And I think we were not only one of the first ones out there, but one of the first ones out there with information that stood up. This was another example of one where there was a ton of bad information. And I think that's one of the reasons this was the most notable. You know, Olympic Destroyer is without a doubt probably the best example of false flags planted in malware. And I would even go to this step of saying these weren't necessarily designed to fool. I mean, they do initially, like at first glance with automated systems. But the deeper you dive into it, it's almost there to make a statement as well as fool, right? Like, to point out the fact that we're planting a false flag – it's super brazen and it's obviously false.

Dave Bittner: Hmm. Yeah, that's fascinating. Well, take us through then, I mean, what were the false flags, how did they work, and why were they important?

Craig Williams: Well, so, the initial set that I think jumped out at everyone – and this is one of the sources of the bad intel – were some of the embedded credentials in the sample. You know, if you look through just the strings in the file, it makes it look like the network was penetrated previously and that credentials were embedded in the malware. The malware was actually gathering them as it went and then compiling them into the binary – or inserting them is probably the more correct word. Which is pretty unusual. And that's something just, again, to mislead people who were doing IR responses. You know, I think without a doubt, though, the biggest one was the malware's basic grafting of APT code into the guts of the malware. So, like, literally, it had vestigial, non-functional pieces of other malware's code embedded in the body. And really, the only reason this is in there that we could come up with is that it's fooling automatic detection systems and sending the message that we did this on purpose. 

Craig Williams: And so, to give you a concrete example, you know, everyone's, I think, familiar with EternalBlue at this point, right? It was a Windows exploit that was stolen from the National Security Organization (sic) and they had embedded the code from EternalBlue from the – that actual set of attack into the malware, and it actually did nothing. It wasn't active. There wasn't enough stuff in there to do anything other than just fool some binary comparison tools. But it was enough for Microsoft to actually initially tweet that they saw it in there. And, you know, of course, we reached out to them. We work very closely with Microsoft. They're one of our good buddies in the threat intel space, and once we shared our information with them, they corrected that message immediately. But the fact that it was in there well enough to fool people at first glance is interesting. And I think that's why this is in there.

Craig Williams: I mean, this is a really important thing to consider, right? Because a lot of companies get hung up on attribution. The reality, though, is computer code isn't really like a fingerprint, right? Computer code is out there for everyone to see. Everyone can get a literal exact copy. So, it's – you know, you're trying to base the uniqueness of something off of something that you can literally make an exact copy of and put anywhere you want.

Craig Williams: And, you know, Matt and I have written, I don't know, two or three posts on this, right? We had Matt's great post on – what was it – on conveying confidence, and then we had one on "Attribution: A puzzle," by Warren and Paul. And the overall theme of these is that if you only have network or malware data on a threat, it's really not enough to confidently do attribution. You know, you need that backing of a traditional intelligence apparatus. 

Craig Williams: And so, one of the exercises that we did in the "Attribution: A puzzle" post was that we took the assertion from, I believe it was NSA and GCHQ on a malware sample. And then we went back like a year later and looked at all the available public information and could we reach that assertion? And we couldn't. And so, our overall conclusion was that, look, while attribution is important for a variety of reasons, it's important that folks realize that you're probably not going to be able to get there with just Internet-based intelligence. You're going to need the support of law enforcement and that traditional intelligence apparatus to get there or, you know, your conclusions should be looked at pretty closely.

Dave Bittner: Can we just touch on some of the incentives here? I mean, Craig, you mentioned and Matt pointed out, you know, how for you it's very important that things be correct rather than necessarily fast. It seems to me like there are powerful incentives to be first to get information out there, to be fast, that organizations get rewarded for that even if they have to go make corrections later. I mean, what are your thoughts on that? Am I – first of all, is my thinking along the right lines?

Craig Williams: Oh, and to be clear, we want to be correct and first. (Laughter)

Dave Bittner: I see. (Laughs)

Craig Williams: That's our goal.

Dave Bittner: All right.

Craig Williams: I think for our customers, that is what they need to look for. And, you know, I would love to say, oh, they should keep a literal scorecard and check off boxes, but that's obviously not something that people are going to do. So I think you just need to realize who are the reliable sources of intel and what conclusions are they reaching. And when there's a situation like this where one company does make that first statement, just read it carefully and read it from a critical standpoint and see if it makes sense, see if the information is supported by other sources, and if there are conclusions that are not supported by other sources, you need to start looking for them and you need to maybe consider that before you take action. You know, they're going to be people who have the information first. That always happens, right? But if you're making a critical decision based off information that only one person says is true, you need to consider that while you take that action and make sure that you're not potentially hampering your response.

Dave Bittner: Yeah, good to have a reputation to be a voice of reason, I suppose.

Matt Olney: I mean, we try.

Craig Williams: Matt's always the responsible kid in the room. 

Dave Bittner: (Laughs)

Matt Olney: Always is pretty strong language. (Laughter)

Dave Bittner: Well, gents, I mean, let's wrap it up here. I mean, in terms of, you know, looking back, again, you know, using these indictments of these Russian operators as sort of an excuse to look back, to look through things on that lens. I mean, what are some of the overarching lessons here? As you look back on these campaigns and the research that you did with them, how do they inform what you guys are doing moving forward?

Craig Williams: Oh, boy. Well, I will go with my easy answer, then let Matt have the hard one. You know, to me, the takeaway from this and prior campaigns is that malware actors learn from each other, right? We knew when we saw the SamSam campaign years before this that a wiper-malware-based worm was possible and coming. We warned people for years before WannaCry happened that this was coming. I think it was like two years, literally. We knew it was happening. It was obvious it was going to happen, and then it happened. And then people had another month, and then they still didn't patch, and then NotPetya happened. So, I think my point with that statement is that the time folks had to address vulnerabilities is shrinking. The amount of time between horrible campaigns and ones that aren't as bad is shrinking because bad guys are learning from each other, and if there's one method that works better than others to get either the reaction you want or the profit that you want, that's the avenue being pursued.

Dave Bittner: Matt, what are your thoughts?

Matt Olney: I think people should take the opportunity to look at Sandworm and understand that that's what we mean when we're talking about an APT actor. It's also kind of a great example of the risks of supply chain attacks. It's also a great example of actors living off the land or using previously known vulnerabilities. with NotPetya, you need to remember that Sandworm's working for the Russian government – the Russian government is telling Sandworm, these are your objectives.

Matt Olney: And our assessment is that NotPetya, the directive was, we want you to punish Ukraine and those people that choose to do business with Ukraine. And to solve that ask, they discovered that there was a tax software that most people who do business with Ukraine use, that they were able to breach that software, that that software had automated updates, that that software could be modified without being detected and then distributed. So essentially, they were using MeDoc as a malware distribution center for months before this came. They generated a list of every entity doing business with Ukraine using the tax ID numbers, and they were able to cross-reference those tax ID numbers with strings that said, this is who they are. So they had an absolute list of who would be affected. And then they chose to execute NotPetya, and designed it in a way that would limit it to the affected parties, but would spread incredibly rapidly. So they were able to do exactly what they were tasked to do. They knew exactly what would happen when they executed on it, and they executed on it, even though they knew what the outcome was going to be.

Matt Olney: And that's – when I talk about APTs, and I have a pretty high bar, that's what I'm talking about. Most of what we see on a day-to-day basis, even the really serious ransomware stuff we see, is not APT-level work. This is what I'm calling APT-level work.

Dave Bittner: Our thanks to Craig Williams and Matt Olney from Cisco Talos for joining us. You can find more about their NotPetya and Olympic Destroyer research on their blog. It's blog.talosintelligence.com

Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Elliott Peltzman, Puru Prakash, Stefan Vaziri, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.