Security Unlocked 12.8.21
Ep 53 | 12.8.21



Nic Fillingham: Hello. And welcome to "Security Unlocked," a podcast from Microsoft where we unlock insights from the latest in news and research from across Microsoft's securities, engineering and operations teams. I'm Nic Fillingham.

Natalia Godyla: And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft security, deep dive into the newest threat intel, research and data science. 

Nic Fillingham: And profile some of the fascinating people working on artificial intelligence in Microsoft security. 

Natalia Godyla: And now let's unlock the pod. 

Nic Fillingham: Hello, the internet. Hello, listeners. Welcome to Episode 53 of "Security Unlocked." My name is Nic Fillingham. I'm joined, as always, by Natalia Godyla. Natalia, welcome to you. Welcome to this special episode of "Security Unlocked." Are you well? 

Natalia Godyla: I'm well. I'm very excited for us to be bringing an episode centered on NOBELIUM. As many of you might know, NOBELIUM was the threat actor responsible for the SolarWinds compromise in 2020. We've talked about it in so many different ways both as Microsoft and across different media outlets. And I think it's a really good moment for us to sit and reflect about what happened. 

Nic Fillingham: Absolutely. And instead of having a sort of new conversation here on the podcast, what we've decided to do is take the "Decoding NOBELIUM" docuseries, which is a four-part video series. We'll put the link in the show notes. You can go and watch. Each episode is about 10 to 12 minutes. There's four episodes. You're going to hear from the actual front line defenders, the folks that were sitting in the SOC that got the alerts and were sort of there from the very, very beginning. You're going to hear from others across the industry that really talked about why this was an unprecedented moment in cybersecurity. It's really a fascinating set of conversations and insight. And I think you'll really, really enjoy it. 

Natalia Godyla: I think what really stuck with me is just the human element of the stories. I mean, you're hearing the folks who, like you said, were the frontline defenders. You're hearing them talk about what this looked like in their lives. I mean, some of them had to spend the holidays trying to deal with this NOBELIUM attack. Folks were getting the first texts and having to understand quickly what this really meant for us as an industry. And I think that all of those human elements really play out nicely in the series and also just speak to how impactful NOBELIUM was. And we definitely get to that in the docuseries, ruminating on what this means for us moving forward, what types of security protections we should consider with the rising state of nation-state attacks. 

Nic Fillingham: Absolutely. So you won't hear from Natalia and I in this episode beyond this little introduction here. The "Decoding NOBELIUM" docuseries was a standalone project that was produced by colleagues of ours in the security team here at Microsoft. But you're not going to miss our voices. 

Natalia Godyla: (Laughter). 

Nic Fillingham: It's a fantastic piece of content - lots of insight, lots of analysis. And you'll hear from us again on the next episode of "Security Unlocked." And so I think, with that, on with the pod? 

Natalia Godyla: On with pod. 


John Lambert: The reality is we live in a geopolitical world. And there are lines of competition and contest across the globe. And there is a consequence for all those things in the world of cyber. 

Unidentified Narrator: Geopolitics in the digital age has led to a proliferation of state-sponsored cyberattacks. Let's go over the facts. Nation-state attacks are malicious cyberactivity that originates from a particular country to further national interests. These actors are focused and have the means to develop and deploy novel techniques and tactics, representing some of the most advanced and persistent threat activity Microsoft tracks. Historically, foreign actors have focused on governments, think tanks and infrastructure. Today, the enterprise is now the most common target, representing 35% of all attacks. Nation states might compromise a business together high-value intellectual property or as a way to access their ultimate victim. And the risks to enterprises continue to grow. Microsoft has tracked an upsurge in the sophistication and frequency of nation-state attacks, delivering over 13,000 nation-state notifications to customers in the past two years. Nation-state threat actors like NOBELIUM are increasingly targeting supply chains, with a 78% increase in attacks on vendors. 

John Lambert: Hi. I'm John Lambert. I run the Microsoft Threat Intelligence Center. The Microsoft Threat Intelligence Center has a mission to defend Microsoft and its customers against adversary-based threats. One thing that's different about that class of attacker is they have a durable interest in their victims. They want to be there to stay. And so as a result, they're not just attacks of opportunity. They're professionals. They pride themselves on their tradecraft. They improve it. They constantly have to respond to detection and still be successful in their mission because it's of national importance for them. 

Dave Kennedy: They focus on weaponizing and tooling for a number of different reasons. It could be for intellectual property theft and espionage, to give your country a better foothold in the global economy versus another by stealing research and development - or intelligence-related missions. 

Dave Kennedy: Hi there. My name's Dave Kennedy. I'm a cybersecurity consultant for TrustedSec and Binary Defense, founder and CEO of both companies; been in the industry for over 21 years focusing on cybersecurity. It doesn't matter the size of your organization or company. Unfortunately, everybody is a target. 

Roberto Bamberger: Nation-state actors are hard because they effectively have kind of infinite funding - right? - they're above the law - at least in their country - and, you know, they have very good technical resources. So it's not like they're going to go - they're not going to give up. That's one of the reasons we put in the 80-hour days or 80-hour weeks. It felt like 80-hour days. 

Dave Kennedy: Nation-states really formed and changed how cyberwarfare is being conducted on a day-to-day basis not just from a country perspective, but also what we see from an organized crime standpoint as well. 

Rob Lefferts: The first moment that I heard about it, it was about a particular security partner, FireEye, that was impacted. And then it was - that was instantly sort of eye-catching or mind-catching on like, oh, that's a big deal because they're good. Like, they're a great security company. And for them to have been hit by this, we know that something deeper is going on. And it sort of just rapidly peeled the onion from there. 

Rob Lefferts: Rob Lefferts - corporate vice president for M365 security products and technologies. 

Rob Lefferts: There's a couple things about it that make it really significant. The first - it was this moment where we saw a nation-state attacker really laying the groundwork on a significant operation over a number of years in a way that we always feared that they would. The issues around supply chain are something that we'd been hypothesizing, and to see them really put it into action was, you know, dismaying. 

Rob Lefferts: And then the other part of it that was significant was just the number of organizations that got hit. It was the span of it and world-class organizations with great security teams really falling prey to this style of attack. 

Unidentified Narrator: During a software supply chain attack, an adversary compromises a technology vendor, slips malicious code into a trusted piece of software and uses their distribution system to spread malware to customers. In the case of the SolarWinds compromise, a malicious code was inserted deep into client organizations' networks through legitimate software updates, bypassing multiple layers of security and providing NOBELIUM with immediate high-level permissions. Ultimately, hacking just one organization can result in thousands of additional victims. 

Dave Kennedy: But, you know, when you go after a company like SolarWinds for example, which is a third-party, you know, private company that does work for a number of organizations - you know, with the SolarWinds Orion platform, they had over 35,000 customers it went to. I mean, to go after and hijack an update mechanism isn't easy. You have to understand the build process, how they push code out, you know, where you can insert your code in, the structure and code of the actual applications itself. You know, you have to be very careful on code quality because you don't want to push updates out that all the sudden now break the functionality and all the sudden you're busted. So this operation has to go flawlessly. 

Joanne: So when they got a toehold in there, they basically got a route into the most important parts of everybody's computer network. Everybody uses SolarWinds. Well, most people - many people use SolarWinds. So they just got a ticket - a free ticket into all these organizations in a place that nobody would even look twice. 

John Lambert: This attacker was sophisticated and stealthy in the sense that they took a lot of effort to silo how their operations looked and presented to a victim to be very different victim to victim. 

Elia: They were very methodic, and this attack was carefully planned over a year time frame. So nothing was accidental. Nothing was left unplanned or was improvising. You know, everything that they were doing was carefully studied. And when they compromised the supply chain company, they first tested the attack for a couple months. So they - instead of introducing the backdoor right away, they actually introduced just one line of code modification to make sure that this line was not detected. 

Cristin Goodwin: So when you look at what the attacker was doing, the attacker had to make a lot of choices about the victims they wanted to pursue and how they wanted to go about it. And so what emerges is that it became an attack looking for information to be able to use for its own advantage or potentially for future operations. 

Ramin Nafisi: This was that spy versus spy game. That's really no different than the same schematics that you've heard about or seen only in movies during the Cold War. This is just the next level. This is just the next iteration of the battle space here. 

Dave Kennedy: While we are dealing with a lot of different areas that threaten our security programs, you know, a lot of the fundamentals and basics still can make a big difference in protecting your organization. You know, first and foremost some the basics. Vulnerability management - huge. You know, you look at a lot of the breach statistics, and most of the breaches that occur, over 80% of them occur from a CD that's six months or older. Keeping up with your patches on your operating system, your workstations, you know, your middleware tier, your web applications - all of those things are really important to ensure that you're maintaining a base level of security because those are already known issues that, you know, hackers are going to exploit and specific things to that effect. 

Dave Kennedy: Even basics, like, around multi-factor authentication - it is one of those basic security principles that you have to have enabled. I can't emphasize multi-factor authentication enough. 

Dave Kennedy: There's a concept of zero trust, and I want to break down zero trust for a second. It is not a product or a tool, it's a way of thinking. Zero trust is essentially, you know, minimizing your attack surface as much as possible and only giving users in your environment access to what they need. And that goes for servers too. So for example, instead of using full VPNs into your network, they now have access to your entire infrastructure. Why not just provide them a web portal that they can do all of their services on that they need for their specific job role or functionality? So compartmentalizing data, compartmentalizing how information is accessed and the roles and responsibilities for users. And that's really going to make the big difference in these types of attack. One breach of an individual user should not lead to the entire compromise of an entire organization. These are all things that we can build very specifically to make things a lot better. 

Roberto Bamberger: This entire investigation happened because a security guy at FireEye noticed that there was a sign-in event for a user using a different registered device. And they called that user and said, did you register a different device? And he said, no. Somebody had the intuition that something looked weird. Something looked a little bit off. 

Charles Carmakal: In the early days, we didn't know exactly who they were or how advanced they were. But as we continued to dig into it and continued to learn new things that the attacker was doing, I mean, it became very clear to us that we were dealing with a highly capable, highly clandestine and advanced adversary. My name is Charles Carmakal, and I'm a senior vice president and CTO at FireEye Mandiant. Yeah, so I lead a team of instant responders and security consultants that both help organizations respond to breaches as well as help them become more resilient to attacks. We decided to get in touch with Microsoft because we knew that we needed some additional expertise to help us with this. We didn't want to do it alone because we knew that we'd be able to gain more speed by pulling in an organization that is highly capable - has smart people, has really strong intelligence and could help us investigate it. And so I called the leader of the Microsoft Detection and Response team. It was roughly 9 o'clock at night. I told Dan Taylor that, you know, we had a security incident at FireEye. He initially thought I was joking. I told him I was serious, and I said, Dan, we'd really appreciate your help. 

Roberto Bamberger: It was late November, and I got a text from my boss. I was out walking my dog, right? And, you know, I get this message. It's from Dan. I better answer it. He's like, well, one of our partners is asking us a question about this weird error code, and maybe you've seen it before. And so I said, well, what is it? And so he brings me into the conversation with, you know, it's this third-party IR company that we work with. And it actually wasn't an error code. And I was like - well, I think I might have uttered something that is not suitable for the workplace at that time because I had seen that before during a nation-state-related investigation for another customer about a year earlier. 

John Lambert: Early on in November, we started working with FireEye and jointly collaborating with them to get initial insight into this new stealthy actor. And then from information we learned in collaborating with them, we have Microsoft's ability of visibility into the threat actors around the world. And we took those leads, and we started turning those over in our data sets. And we understood the scope started to grow very quickly from that. 

John Lambert: Hi, I'm John Lambert. I run the Microsoft Threat Intelligence Center. The Microsoft Threat Intelligence Center has a mission to defend Microsoft and its customers against adversary-based threats. And so it's very important that defenders in multiple organizations and across industry lines of competition collaborate to see the threat as best we can. When you defend so many customers around the globe and you have that responsibility, the parts of Microsoft know how to come together in a response and bring what they have to bear on the problem. And the first thing that we needed to do was understand what was the scope and scale of what was going on in this attack. 

Elia: So there was basically a huge collaboration from everyone. Everybody really wanted to do the right thing and, you know, helping Microsoft and the customers. Every meeting, you end up having now - here now 50 new people. We are now 10 new reverse engineer or, like, we have 10 new analysts coming in. And I will say, like, really we're not a thousand people. We are only close to that number because you really needed to try to tackle in two months an attacker that's been working for over a year. 

Peter Bryan: In November last year, I had decided to travel back to the U.K. from where I am based in Seattle. And primarily, that was to go and see my fiancee at the time because we hadn't seen each other for 10 months, awfully, due to the pandemic. So I was actually doing my two-week required quarantine on arrival in the U.K. when my colleague Christopher sent me a message saying, hey, we've got this interesting activity that we've been made aware of. 

Joanne: Saturday morning, I was getting ready to go out for a hike, had all my stuff on. I was literally getting ready to go out the door, and I got a text message. Who's texting me? It's my boss. And I start reading it, and he sent it to the whole team. He says, all right, this stuff's blown up. We need everybody. All hands on deck. 

Franklin: We immediately started to look deeply at the logs for anything that looked abnormal. And over the course of about three days, we were able to pick out just a few markers - very subtle markers - that then we looked across the environment for those markers and saw other instances. 

Christopher Glyer: You know, they didn't do the same playbook at every company, and this was, you know, one of the biggest challenges I've certainly ever had to investigate. Let's just take, like - if you knew the Nobelium was using a particular IP address at victim A, you couldn't look for that same IP address and go look at Victim B, right? They wouldn't even reuse simple things like that. If they were harvesting mail from, let's say, 20 users, they would leverage 20 different IP addresses to harvest mail for different users or access their mail or log in as them. So brittle indicators, whether it was hashes or file names or IP addresses, weren't really useful. And then we worked with various teams to help them, you know, understand the tradecraft. And then they would then try to emulate or look for it in different environments and access the data they had access to. 

Charles Carmakal: As we learned about things, we would share it with Microsoft. As Microsoft learned about new tradecraft from the attackers, they would share it with us. And I think we both recognized that, you know, collectively, we're able to see more of the - what the attacker's doing and find ways to disrupt them by sharing knowledge of what the attackers are doing. 

Roberto Bamberger: So there were a lot of theories about what was going on here. Theory No. 1 was, well, this actor - this threat actor was able to go steal credentials from lots of places and, in particular, compromise an on-premise network, move through that network without being detected and get to an Active Directory Federation server, ADFS server, and steal this secret off of it called the token-signing certificate. And we're like, that's really hard to do in one organization. 

Franklin: Inspecting the actual token that was presented to Azure Active Directory and looking for any anomalies with it. The thing we noted about those adversary tokens was that more than one of them had a very specific time to live, which was 40 hours. And that became a marker for us. And it was through that that we identified, then, follow-on customers, which we were able to notify. But also, with each additional organization that we recognized was compromised, we saw other markers of the adversary. 

Roberto Bamberger: We knew to go look at those particular servers - right? - those Active Directory Federation servers to go look for suspicious activity. And as we started pulling that thread of, did anything interesting happen on those, that's where it started pointing back to, yeah, something happened on premise. There was some very stealthy malware there. There were weird activities that we couldn't explain. And eventually, it led us to their SolarWinds servers. 

Charles Carmakal: As we started really digging in to the SolarWinds system, there was one thing that we noticed - that a specific DLL had spawned a interesting and, you know, suspect process. So we started to dig in to that specific DLL file, and we had folks from our malware reverse engineering team take a look at it. So they decompiled what amounted to about 50,000-plus lines of code. 

Ramin Nafisi: So we quickly worked with reverse engineers here at Microsoft and over at FireEye to break out and share the differences. Some of the things that we saw with the malware was the way that they had done their homework when it comes to malware development, and they had made sure that they wrapped the malware in additional layers of code that would basically hide it from antivirus products and security solutions out there so that way they can operate without getting detected by some of these security solutions. So it took us some time to basically remove some of these layers to get to the actual malicious code. 

John Lambert: We worked directly with FireEye. They make the crucial breakthrough that the SolarWinds software they were running had a supply chain vector. Once we understood a supply chain attack against SolarWinds and how critical that device was used in very privileged parts of customer networks, you got a sense that this attacker could start in hundreds of customer networks, very deep into them, skip many phases of the kill chain and where they would start already with elevated rights. And so at that point, when you realize how many enterprise customers, government departments use this, you knew that this attacker had achieved a place to really have major impact across the globe. 

Peter Bryan: When we found that scope, it was a combination of exciting and scary. 

Ramin Nafisi: So back when the intrusion started, it was one being the actor versus many being the victims. But when the incident was discovered, it turned into this sort of many-versus-one collective effort to investigate and mitigate the attack. In this case, many different partners from across the industry came together with a common goal. 

Peter Bryan: So in terms of guidance for customers going forward, I think one of the big things is making sure you're collecting data ahead of time. This incident showed that attackers will leverage very different parts of an environment both in the cloud and on prem to achieve what they want. 

Joanne: One thing is if you RDP into a box, Remote Desktop Protocol into a box, don't leave the session open when you leave. Close the session 'cause then they can't just grab your session and start using your login if they do get into that system. Using - for your secure systems, you want to use a secure networking device. You don't want to use your everyday workstation or everyday desktop to do administrative tasks on sensitive systems. 

Ramin Nafisi: Given some of our findings and some of our takeaways from this attack, investing in penetration testing, investing in putting together TI teams and TI practices and building relationships and partnerships with other companies, other partners. Everybody has their own unique visibility. So at the end of the day, it comes down to joining forces to be able to go against a sophisticated actor like this. 

Rob Lefferts: As soon as something like this happens, we need to put a number of trains in motion in parallel. Like, one train obviously is understanding, just like gathering as much data as we possibly can about what's going on. The second is about notifying people (laughter). And that's a deep and difficult set of conversations. The third one - and this is where a lot of the folks on my side of the world get deeply involved - is about, what are we going to do immediately for impacted customers? 

Elizabeth Stephens: I spent 19 years in the United States military - three combat deployments, aviation, logistics, supply and combat engineering. All of the teams came together in a way that very much reminded me of the way that my Marine Corps came together. And so the way that we respond is like - very much like first responders. We pride ourselves on being able to come together regardless of what our areas of specialty are and our areas of expertise and fill in the gaps between each other very quickly on a team that can actually get a mission completed - just selflessness and the sense of, like, if we weren't defending, then who else was going to? 

Elia: We ended up doing - like, you know, trying to really reach out to customer. And we were very fast to try to notify them and really help them with recovery. That actually enabled us to really limit the possibility of the attacker to become more dangerous. They have to fight a little more, and usually when they end up facing unplanned actions, that's where the adversary makes mistakes, right? And mistakes, they all pass through the deck. So it's kind of a cycle, so we end up eventually, you know, finding little mistakes or eventually gathering more evidence that allow us to detect more, right? 

Cristin Goodwin: We had one customer that refused to believe that we were Microsoft and, in fact, hung up on our colleague that was making the actual phone call - just wouldn't even pick up the phone. One of the challenges, of course, when you call a customer to tell them that there's been a problem is that they've been conditioned for years to believe Microsoft's never going to call you. You really have to meet the customer where they are because the attack is so significant that they're all going to need help in different sorts of ways. And so that's the cool thing about the size and scope and breadth of Microsoft Response is that we can pivot, and we have the skills that help us meet the customer and the needs that they have. 

Franklin: To see the looks on people's faces as they - as the gravity of that settled in was certainly sobering for me and my team, but also, it was a tremendous incentive for us to keep going until we could get to the very bottom of it. 

Sarah Fender: As a product team, we were really invested in helping our customers and hunting for and investigating, looking for indicators of the NOBELIUM attack within their own environments. And so while we weren't front lines at defending Microsoft, we were there to provide the support and the tooling that they needed to do that very hard job. 

Unidentified Narrator: Threat hunters analyzed telemetry to identify indicators, including tactics, techniques and procedures, or TTPs, the markers of NOBELIUM activity. 

Michael Shalev: So our researchers look at identity. They look at email and collaboration. They look at end point. They even look at cloud activity and cloud application security. And by taking the holistic view, we're able to track the attackers who move from domain to domain, and that's usually where they get lost in the noise - in the transitions. 

John Lambert: And we had identified over 70 TTPs that we thought - these are the body of techniques that you need to understand in order to see this attacker. And we worked to go publish those on our blogs and embody them in the products to speed detection there. 

Unidentified Narrator: These TTPs are then used to build automated detections for security products to rapidly find and respond to threats. 

Peter Bryan: By pivoting towards taking what we are finding and turning into these customer-facing detections so that more people can be protected from them. And really, the key aspect of that is taking the patterns of activity we find, determining how generic they are - as in, are specific to a particular customer or are they a attribute of this threat actor, generally, or could it be more broadly scaled than that? And then developing that into logic that fits into our tools. 

Corina: We were continuously collecting more and more information about how this attack evolves, what are the different pieces, what's new and unique about it. And then working with the different teams of - in our products to create new detections and capabilities. 

Unidentified Narrator: After release into products, these detections help customers automatically find and respond to new threats across their environments. 

Michael Shalev: So what our researchers and we are trying to do with our product is - and lay it out for our customers, so they can see and understand what it was that the attacker did and also help them remediate and to return their network and their assets to a healthy state. 

Peter Bryan: We've had instance before where we needed to rapidly release detection activity or detection material but never on the scale that we did with Nobelium. I mean, we were we were releasing multiple detections a day into the product. And that continued for the best part of three weeks. 

Sarah Fender: It wasn't just, you know, one detection. Like, oh, if we just could find this one thing, that would solve all of our problems. The reality is that, you know, there are collection of activities that a threat actor takes as they execute against the kill chain. And it is that sort of collection of detections at those various stages using those different tactics and techniques that were linked to Nobelium. I think that really helped organizations to get that end-to-end view of the attack and the scope and the impacted resources, so they could respond. But, you know, seconds count when responding to an attack like this. And so we were able to kind of put things into high gear and expedite that process of getting this content out. And our Microsoft Threat Intelligence Team published workbooks and Azure Sentinel notebooks to help our customers to do the same work themselves. So security analysts were able to use Azure Sentinel to hunt for specific TTPs across a broad set of data. 

Peter Bryan: Because the threat actor had compromised several customers' on-premise AD FS servers and stolen key material, people were very keen to protect that. And we got a lot of feedback saying, this is - how do we detect this aspect of it? 

Michael Shalev: All traffic, all work across an enterprise is really based upon our identities and what access those identities have. And so trying to pick out which signal or which behavior is malicious from something that is benign is really difficult in order to create a detection of the attack. We leverage signal from our endpoint sensor on the AD FS server itself with signal from identity token traffic to and from the AD FS server. If we take a signal of an activity that happens on an endpoint that by itself might even be benign some of the time, we detect an identity, which in its own might also be benign. But we know that those two, when they occur together, are a clear indication of malicious activity. By correlating signals from different sources, we're able to increase the confidence of our detection and actually create an incident in Microsoft 365 Defender that describes the entire attack, kill chain stage by kill chain stage, from the AD SF server to the cloud, step by step. 

Peter Bryan: So once we managed to create that and publish it, I think that was one that was kind of very impactful for customers but also quite kind of personally rewarding. 

Joanne: Another thing that they did was they would turn off logging and/or antivirus on the systems. And, of course, that essentially - they attempted to blind the security people so that what they did wouldn't even show up in our logs. But in larger organizations, people often have reasons for turning off logging, whether they're good or bad reasons. The Microsoft Defender for Endpoint team did actually come up with a bunch of new ways and techniques of making sure that their defender tool doesn't get turned off. Or if it does, it does throw an alert that's noticed somewhere. So they were able to grow their tool based off of the lessons we learned from this. 

Roberto Bamberger: We're helping customers on their worst days. So we're really respecting the fact that our customers are having a hard time. They're overwhelmed. They are tired, right? They don't know what to do. Having empathy for that, I think, is really important. 

Cristin Goodwin: Turning on multifactor authentication, making sure patches are installed - those are the basics. They're the diet and exercise of the information technology sector. And we really owe it to ourselves and to our customers to bring that back into the conversation, to make it cool again to talk about hygiene, cyber hygiene and the basics because it's too often that nation-states don't need advanced, sophisticated tactics like we saw. 

Elizabeth Stephens: Identity is the No. 1 entry and access point for the majority of all of these attacks. And if you can get a handle on identity first, then your journey towards being secure is going to be immensely faster and more efficient. 

Rob Lefferts: You have to think about a modern distributed estate. You have to think about how you're going to deploy zero trust. And so what you saw is the attacker traversing a lot of different parts of the estate and bypassing the firewalls altogether. So if you want to protect against that, you actually need a monitoring system that's highly distributed. And you have to have tools that really give you advanced behavioral modeling to help your defenders have more insight. And then the third thing is you have to practice. You have to exercise the whole system, including the humans, including the security teams, including the end users. 

Brian: You know, when we do our job right, we get to feed information to the customer. We get to feed information back into our products. Our products get to take advantage of that. And then the next time, our products do a little bit better. We do a little bit better. We find a new attack. We rinse and repeat. And this is a cycle that continually goes on every single day. 

Cristin Goodwin: In the first two, three weeks of the incident, the focus had to be on helping the customer see where the incident was and understand its scope inside their environment. And so - you know, later on, we started to learn differences about how the attackers behaved and how they leveraged different environments and products or tools that were in it. They didn't need to exploit a vulnerability. They abused the rules of the product. 

Unidentified Narrator: NOBELIUM is a well-resourced, state-sponsored actor with an extensive arsenal of tactics and techniques used to conduct cyberattacks - perhaps best-known for the widespread SolarWinds supply chain breach. Publicly disclosed in late 2020, it was actually part of an even larger and more sophisticated campaign that had been quietly underway for over a year. Now, let's review the key findings from the Microsoft Threat Intelligence Center's investigation of the NOBELIUM attack. 

John Lambert: Hi, I'm John Lambert. I run the Microsoft Threat Intelligence Center. The Microsoft Threat Intelligence Center has a mission to defend Microsoft and its customers against adversary-based threats. 

John Lambert: NOBELIUM had a rich set of techniques that they used across the kill chain. If you start, you can start with how they achieve penetration into their victim environments. They - while a lot of discussion on this incident really focused on SolarWinds as a method of penetration, they would do password spraying. They would use vulnerabilities against edge devices that were just unpatched against those vulnerabilities. So they had a number of different techniques for getting inside an internal environment, or they would get access to an organization, use their credentials - and their credentials already had access to several, say, subtenants of that organization, their customers and so on. And so they had multiple ways to get into an environment. 

John Lambert: Once they're in, they would take a typical pattern of doing internal reconnaissance - find out the elevated accounts, find out machines that were there, take that data out so that they could, you know, understand the map of what they were looking for and then expand to gain additional elevated rights in an environment and expand their persistence. As they worked within an environment, they took a lot of effort to really silo that. And as they moved laterally within a network from machine to machine, they took great pains to clean up after each step. And so they try to preserve their stealth as they enter the victim. 

John Lambert: And then armed with that information, they ultimately would then go do whatever they were interested in in a network - and some of our customer was gaining access to their production secrets or their systems, exfiltrating source code. And then we saw a different set of TTPs, but for the same goals against some of our customers from our cloud services. And so once they again had achieved elevated rights within side of a cloud tenant. They would use those rights to start to access email within those tenants or otherwise gain persistence. 

Franklin: Attackers' operational security was better than we've ever seen in any other campaign. And so they would be very discreet in how they used that authentication. And then they would just do it once from a specific piece of actor infrastructure and then move on to different infrastructure and different techniques after that. So they would get their foot in the door, but then not continue pushing on that door. 

Charles Carmakal: They would do things like disable EDR solutions from being launched upon system startup. So they would wait a month for Patch Tuesday to come around for the computers to get rebooted. And then when the computer got rebooted, the configuration was changed or the registry key was changed on a Windows system such that the EDR solution wouldn't start up. And so if an EDR solution didn't start up, the attackers more or less had free reign on the machine. 

Joanne: They were so deliberate and careful about what they did. It wasn't like a smash and grab where they came in and just vacuumed up everything noisily, ran Mimikatz to grab everybody's passwords and fled. It wasn't a situation sometimes where you see - like the NotPetya in Ukraine a number of years back. They were very sneaky. They planned on staying. They had every intention of coming back. They just didn't know when because they had lots of stuff on their plate. 

Charles Carmakal: Most threat actors don't have the patience to wait a month to do something, but some will. And I think, you know, when threat actors adopt that tradecraft, it will become harder and harder to find them. 

Peter Bryan: This was a very sophisticated, persistent and stealthy threat actor. But the fact that once we picked up on the patterns of activity, that we could go and look across every kind of customer within the Azure cloud to pick out where that threat had been was really powerful. And if we had been operating 10, 15 years ago when everything was on prem, it would have taken us years to even get a vague idea of the scope of this incident, and we would probably never have a full picture of it. But the fact that we have this power and data scale in the cloud means that we can much more effectively kind of identify where the threat actors have been and what the scope of their operations are. And that's only going to increase over time as more people move to cloud services, and then as that grows, we can start to leverage more advanced machine learning models to make this detection of this sort of activity in the future - again, that's only going to improve and increase as we go forward. 

Sarah Fender: In order to respond to an attack no like NOBELIUM, with its scope and breadth and sophistication, you really need to have visibility into various entities across your entire digital estate. So you need to have visibility into security, data and events relating to users and endpoints and infrastructure, on prem and in the cloud. And so that breadth of security data and the ability to quickly analyze and search that data is just very, very clear with NOBELIUM. 

Roberto Bamberger: The attacks of the future, you know, a lot of them are going to be identity based. Once I can authenticate into your environment, I don't need malware anymore. So that means that monitoring behaviors, building a profile for - when Roberto is using his machine, he accesses these 25 resources, and he does these kinds of things from - you know, and he's never been in these four countries. And if I ever see something that doesn't fit that pattern, I need to alert on it. 

John Lambert: One of the things when I think about what does this incident mean going forward, again, it certainly reinforces the need of the world to work together on these threats. No one company sees it all. And it is very important, especially with sophisticated threats, to be able to work very quickly with lines of trust established. This is not just about companies working together; it's also about individuals trusting each other with insight, impacted companies, fellow security industry companies, government institutions. Early parts of this can be chaotic, but the sooner you can come to a crystal-clear picture about what actions people can take to respond, you're going to be able to focus and mobilize organizations around the world to deal with this more effectively. While there are, of course, things that we do from a technology perspective, from a product and service perspective, to go improve that and make things simpler and make getting insight clearer and easier, working together as from Microsoft with all of our customers and partner organizations and industry - across industry and government is just super important for threats of this caliber. 

John Lambert: Yeah, I think some things that customers can focus on right away in dealing with this that will help them against many different threats that they face are along the lines of zero-trust, managing supply chain security concerns that they have and also being resilient in response. Zero-trust principles around identity are really about ensuring you have strong identity, so you know who is accessing something from what device or endpoint and that is strongly authenticated against what service and where you have areas of risk because you're not able to get the strength of the identity or authentication as you want, you limit or have conditional access to what they're doing so you can manage your risk proportional to the situation. Supply chain threats really reinforce how important it is to know what's in your environment and be able to manage it and then, critically, have a backup plan. It's that - it's not a matter of if; it's when. And you want to have responders that are well-practiced at these incidents and able to respond. 

John Lambert: Some things that help them in response are, one, multiple points of perspective on your environment so that you're not overly dependent on one sensor because you can only respond to what you know about, and so it's important to have multiple points of perspective across identity, endpoint, network. You have to realize at the same time defenders - defender fatigue is a real thing. And so you have to be able to invest in those defenders so that they can surge when they need to. And security, like other professions, it's not just a job; it's also a calling in their way, and people - security gets in people's blood, and they want to do it. But it also leads to fatigue and exhaustion if the incident drumbeat is too strong. So you have to have reserves and plan for that so that you can support your defenders and rest them in between incidents. 

Rob Lefferts: We can't get our head space into a world where we think the machines are going to fix this problem. Technology is force amplifiers for the insight from our researchers who are watching the landscape across MSTIC and Defender Research and DSRE. And it is connected directly into the people that we are trying to help and turn into heroes, which are the defenders in the companies and partners that we work with. And so we really have to keep that front and center, rather than - look how shiny our technology is. 

Natalia Godyla: Well, we had a great time unlocking insights into security, from research to artificial intelligence. Keep an eye out for our next episode. 

Nic Fillingham: And don't forget to tweet us at @msftsecurity or email us at with topics you'd like to hear on a future episode. Until then, stay safe. 

Natalia Godyla: Stay secure.