Tracking Nation State Actors
Nic Fillingham: Hello, and welcome to Security Unlocked, a new podcast from Microsoft, where we unlock insights from the latest in news and research from across Microsoft security engineering and operations teams. I'm Nic Fillingham.
Natalia Godyla: And I'm Natalia Godyla. In each episode, we'll discuss the latest stories from Microsoft security, deep dive into the newest threat intel, research and data science.
Nic Fillingham: And profile some of the fascinating people working on artificial intelligence in Microsoft security. If you enjoy the podcast, have a request for a topic you'd like covered or have some feedback on how we can make the podcast better.
Natalia Godyla: Please contact us at securityunlockedatmicrosoft.com or via Microsoft Security on Twitter. We'd love to hear from you. Hey Nic, how's it going?
Nic Fillingham: Hello, Natalia. It's going well, thank you. Welcome to episode 10 double digits. It feels like a milestone. That's a milestone, right?
Natalia Godyla: Heck, yes. I think we were proud of ourselves after episode two. So I feel like this feels a little bit more legitimate, a good start to 2021.
Nic Fillingham: Great start to 2021. But we were talking, just before we started recording and there is some sad news.
Natalia Godyla: Okay. So to listeners that had heard and loved our story about the Somerville Turkey, of course. The Somerville Turkey is no longer, so the Somerville residents had fed the turkey and the turkey became aggressive as a result. And it is no longer a hallmark of our city.
Nic Fillingham: The problem was they fed the turkey pure creatin, that was the issue and Red Bull.
Natalia Godyla: They didn't publish that in the news story, they're trying to keep that hash, hash.
Nic Fillingham: That's why it got aggressive. But no, if you have no idea what we're talking about on our Thanksgiving episode, Natalia told us about a famous turkey in Boston that has a name and it's got an Instagram page or something like that, but unfortunately it's no more, it's pretty sad.
Natalia Godyla: Now that the turkey is no longer, maybe we should memorialize it.
Nic Fillingham: Ooh, so you're thinking we could potentially adopt the Somerville Turkey as our Security Unlocked mascot. Maybe we could create some kind of small statues, some kind of plush toy, is that where you're going?
Natalia Godyla: For some reason, my immediate image was a butter sculpting contest, in which we sculpted butter sculptures of the turkey.
Nic Fillingham: Hang on, what? So, I had said as a mascot and something, I think I said the word swag, at least it was in my brain. So something we could send to listeners, and so I just immediately jumped to the logistics of how do you send butter through the US Postal Service in an intricate shape, like that of a turkey?
Natalia Godyla: Yeah. I don't think you should be taking my suggestions quite so realistically, I mean-
Nic Fillingham: If we had to choose though, between memorializing the Somerville Turkey and our previous plan which was the mighty alpaca as our animal mascot, where are you leaning?
Natalia Godyla: Alpaca.
Nic Fillingham: Can we justify that from a security perspective? Is there any security link whatsoever from either a turkey, Somerville Turkey or an alpaca? What are you looking up? You're looking up something right now.
Natalia Godyla: I'm looking up facts about alpacas because I have to be honest, this is purely on level of cuteness for me.
Nic Fillingham: Okay. So our Executive Producer, Bruce Bracken has just chimed in saying that god llamas and god alpacas are a thing. So it says here that a god llama, alpaca or hybrid can be used in farming to protect sheep, goats, hens, or other livestock from coyotes, dogs, foxes, and other predators. Ladies and gentlemen, we have a winner. We now have a solid link from the alpaca to security. Well done everybody, congratulations, mission accomplished, we can go home now. All right, beautiful.
Natalia Godyla: On a minimum, we can talk about our next episode.
Nic Fillingham: Absolutely. All right, so let's table that. We've decided it is going to be the alpaca because the alpaca can be employed as a rudimentary guardian of livestock. But speaking of the podcast, on today's episode, first up we have Jeremy Dallman joining us from the MSTIC Group. I'm not going to explain what MSTIC stands for because Jeremy will talk about it. And it's a great start to the conversation. Jeremy is coming on to talk to us about the nation-state section or chapter in the Microsoft Digital Defense Report, the MDDR, this is the third of five conversations that we're going to be having on Security Unlocked, where we deep dive into some of the topics covered in that report.
Nic Fillingham: This is also I think, the first time that the MSTIC team have compiled a lot of their nation-state tracking activity over a sort of 12 month period into a single report. So first of all, it's a great read, make sure you download the report, aka.ms/digitaldefense. And then, it's a great conversation with Jeremy who really helps us sort of understand some of the core principles and ideas around sort of why is Microsoft in this space, and then sort of what does Microsoft do with tracking nation-state actors. And then after Jeremy, we talk to-
Natalia Godyla: Randy Treit, a Principal Security Researcher at Microsoft, a long time employee at Microsoft who has seen a lot of different groups and brings that expertise to his security team today. So we're talking to him about his path to security and he is another security professional who doesn't have a formal or standard path to security. So he doesn't have a formal education. And I think it's a good testament to the fact that so many security folks are autodidactic and just have a love of technology and find themselves continuously passionate and interested in it, and eventually get to do their passion for a job.
Nic Fillingham: On with the pod?
Natalia Godyla: On with the pod.
Nic Fillingham: Jeremy Dallman, welcome to the Security Unlocked podcast. How are you doing?
Jeremy Dallman: I'm doing great guys. Thanks for having me.
Nic Fillingham: Thank you so much for coming on the podcast. This is one of several conversations we're going to have with folks that have contributed to the Microsoft Digital Defense Report that was released in September of 2020. Jeremy, thanks for coming on. You're going to talk to us about chapter two, which is the chapter that talks about nation-state threats. This is going to be a fascinating conversation. I'm really, really interested and excited to hear what you've got to tell us. But can we just start a little bit with, who are you? What's your job? What do you do at Microsoft? What does your day-to-day look like?
Jeremy Dallman: Sure. So let's see, in Microsoft terms, I'm a Principal Program Manager, in the Microsoft Threat Intelligence Center. We call ourselves MSTIC. So I'll probably use that term off and on throughout the conversation, it's much easier to say it than Microsoft Threat Intelligence Center. As a Program Manager in MSTIC, I am responsible for, let's see, directing a large number of projects that kind of span incubation and driving threat intelligence initiatives, both in MSTIC and across Microsoft.
Jeremy Dallman: I do things around building and creating strong collaboration partnerships across the security industry, because malicious actors, like nation-state actors, don't just target Microsoft. I also work on sourcing the best possible tooling for our analyst and managing all of our public facing messaging around MSTIC and the threats that we track. So I guess in general, my role is always looking for ways to improve how MSTIC protects our customers, making sure that the analysts are successful and effective at hunting. And making sure that MSTIC knowledge outside the company is communicated effectively to protect our customers and enable better protections across the ecosystem.
Nic Fillingham: I have ask, is MSTIC a backronym? Did you guys get in a room and say, "How can we come up with the coolest acronym in the company, and then make it work for what we do?"
Jeremy Dallman: There's actually a couple of others I think, that are cooler, as well though. Nonetheless, no, our GM is notorious for let's just say, obscure acronyms that translate into words. So it took a little bit of effort, it took a little bit of time, but we came up with Microsoft Threat Intelligence Center and M-S-T-I-C pronounced as MSTIC. So we worked through a few other variations, but I think this was the best one that came out and it seems to have stuck.
Nic Fillingham: I think there needs to be an offshoot team for analytics and learning at the end. Does anyone get that-
Jeremy Dallman: Yes, Nic. Yes, yeah.
Nic Fillingham: Okay, good.
Jeremy Dallman: I know a couple of people on the analytical side that might actually run with that, I might have to jot a note down.
Nic Fillingham: There you go, you can have that one for free, no royalties from me, that's fine.
Natalia Godyla: The next one's charged, though.
Nic Fillingham: The next one's not free, this first one's free. So Jeremy, you're going to walk us through chapter two, the nation-state threats, it's a pretty lengthy section of the MDDR. It's also, I think, correct me here, this is the first time that we've done sort of an annual wrap-up of what Microsoft has seen on the nation-state space. I think obviously, we've had lots of blog posts and activity over the many years on the activity, that we've seen and sort of how we've contributed to it. But previous sort of security intelligence reports didn't really include a lot of nation-state activity. I mean, correct me if I'm wrong here, but is this sort of the first time that we've done an annual look back at what happened in the nation-state space?
Jeremy Dallman: Historically, our team hasn't been very publicly outspoken and we haven't really, historically didn't spend a lot of time talking about what we've done externally. So this is definitely unprecedented and something that's brand new for our team. It's kind of along the lines of what we've been doing over the last couple of years, talking a little bit more publicly about threat actors and such. So I think this is a fantastic roll up in view of what we do. I think it goes along with our expansion of MSTIC as an organization and kind of what we've been trying to do, informing our products and customers more broadly.
Natalia Godyla: So Jeremy, why does Microsoft do this work? Why do we partner with the industry to identify nation-state actors?
Jeremy Dallman: Sure. I think the short version is that Microsoft customers using our products are often the target of nation-state actors. And those customers expect Microsoft security products and Microsoft to help protect them from those threats. So MSTIC tracks nation-state activities to protect our platforms, to protect our services and protect our customers from those more sophisticated threats.
Nic Fillingham: So, Jeremy, I've got the report open here in front of me and for those playing along at home, you can download the report. It's the Microsoft Digital Defense Report @ aka.ms/digitaldefense. And if you scroll down to page 44, there is a really interesting sort of graphic here. It says, "The sample of nation-state actors and their activities." And there's a bunch of what look like sort of chemistry symbols from sort of the periodic table of elements with a lot of chemistry names and symbols. And then there's some sort of other things as well. Can you sort of walk us through, what are we looking at here? Is this actual sort of nation-state actors and sort of how they're referred to? And the names that are being used to refer to them?
Jeremy Dallman: Across the security industry, a number of different security vendors use different code names to refer to sets of activity that are tied to certain actors or sets of activity groups. So we use code names because we can't always necessarily tie that to a specific country, or we may want to do attribution. Other security vendors will use kittens and tigers and bears, some use numbers and a variety of different code names. And at Microsoft and in MSTIC, when we were trying to figure out how we were going to do code names, we tried a bunch of different things. I think initially, there was some use of dinosaur names, that got fairly complicated and hard to pronounce fairly quickly. I think we played around with a bunch of other things. At one point, I recall we were looking at flavors on the beer flavor wheel, I'm not sure there was enough of them.
Jeremy Dallman: So we played around with this a little bit and we ended up basically at periodic table of elements because there's not really a licensing violation there, so we didn't need to worry about that. And there was plenty of them and they were fairly unique. So we code name our actors by elements in the periodic table. And we will name an actor, an element, once we understand that actor has a unique set of activity. But on that page 44 in the report is a summary of a few of our key activity groups via their element names. And largely focusing on the four regional sets of actors that we, and most threat intelligence teams will focus on, Iran, China, North Korea, Russia.
Nic Fillingham: And is there any sort of logic to the particular element that's chosen? I mean, I noticed that there's no hydrogen, there's no oxygen. Well, they seem to be up towards the top end of the periodic table. I've never even heard of-
Jeremy Dallman: Yttrium?
Nic Fillingham: Yttrium? Did Kanye West come up with that one? What's that?
Jeremy Dallman: No, it's kind of funny because we actually have an individual on our team over in our UK office. She's responsible, she's our librarian, is kind of the role that she plays and she is responsible for naming. So I don't think there's any specific logic or pattern to who gets what name. I don't even know if our analysts have a say in picking any of the names, but our librarian is the person who basically gives these names out. And I don't think she has any set structure or method for picking the names.
Nic Fillingham: I was really hoping you were going to say there was a periodic table of elements stapled to the wall, and then you had to start with dots.
Natalia Godyla: Somehow, I knew I was going to be dots.
Jeremy Dallman: You know what? I honestly would not be surprised if that was actually the case, but I can't verify that.
Nic Fillingham: All right. Well, that's for another episode of the podcast for us to follow up on.
Natalia Godyla: So can you provide a little bit more context on the players? What do we know about them? Their motivations? Infrastructure?
Jeremy Dallman: Sure. So a number of these actors are pretty well known. When you talk about kind of the more popular, more widely discussed actors, it's kind of hard to not fairly rapidly, get to Strontium, which others refer to as APT28 or Fancy Bear. And this is an actor set that we believe originates in Russia.
Jeremy Dallman: This is someone that we've... an activity set that we've talked about fairly extensively over the years of public discussion around these actors. Whether targeting individuals or campaigns or entities involved with politics. So they're probably the more well known out of Russia. I'll just kind of hit a couple in each one of these here.
Jeremy Dallman: Phosphorus, which is an actor set that we believe is originating from Iran, also known as APT 35 and Charming Kitten. They're well known for targeting government defense industrial, especially in the region, in the Middle Eastern region. Especially fond of targeting personal email accounts and going after personal email accounts as a way to gain access to systems that they're targeting or individuals and surveil individuals. A lot of activity there tied to sanctions and research around policy, that sort of thing.
Jeremy Dallman: In China, we have actors that more broadly, I would say are more likely to use more sophisticated technical solutions. Trying to bypass or using more sophisticated malware, but technology, supply chain targeting, targeting education and medical research. Actors like Barium known as APT 41. Manganese, which will often target communication infrastructure. They'll even go after things like satellite or defense industry or GPS navigation.
Jeremy Dallman: And then North Korea actors like Thallium and Zinc. We'll see them targeting human rights organizations and surveilling human rights organizations that might be involved in their region geographically. But we'll also see them often targeting think tanks and governments that are involved in sanctions or policy decision-making that might be tied to the Korean peninsula.
Nic Fillingham: Why is Strontium a nation state actor and not simply just a sort of independent group of baddies?
Jeremy Dallman: No, that's a great question. I think the simple definition of a nation state activity group is we defined it as cyber threat activity that originates in a particular country with an intent to further national interests. So because that activity fits that parameter, there's an assumption that it's more well-funded, potentially more sophisticated. And they'll more likely going to be using what we call advanced persistent threats which is an adversary that possesses a sophisticated level of expertise and significant resources that allow it to achieve its objectives using a lot of different attack vectors. It's a combination of expertise and significant resources, adequate funding to achieve specific objectives in a particular country with intent to further the national interests.
Natalia Godyla: And what about attack techniques? So you hinted at that in your definition. So what are some commonalities or patterns that you can identify across nation state actors that differentiate them from other threat actors?
Jeremy Dallman: So when you think about nation state actors, and I would say in most of our threats even outside of nation state actors, you're going to see most threats start with email. I think there was a blog post we put out not too long ago that said 95% of threats start with email. Start with an email lure. From a nation state actor perspective, that's largely a technique to achieve reconnaissance. To find out or identify who the people are that they need to target to achieve the objective that they're trying to achieve. So they will do things like password spray techniques to attempt to guess log in passwords for a number of accounts tied to a specific organization that they're trying to target. They will do brute force login attempts, trying to guess the passwords and try to brute force their way into an organization. That early reconnaissance technique allows them to establish an initial foothold into an organization and also then harvest credentials.
Jeremy Dallman: So if they can start guessing passwords and they can understand what those passwords might be, they can harvest those credentials, store those credentials and then use those in future operations to come back into that network and execute whatever operation or mission they might be trying to achieve. Once they've actually established in there, and often as a way to get a foothold into a network, they'll use malware. Malware is a very common method by nation state actors. And I would say some actors on the nation state side, because of the excessive funding that they have at their disposal, they will go above and beyond in building up particularly sophisticated malware techniques to bypass common detections by security vendors and some networks. So that's constantly a game that we're playing to understand these malware techniques. We'll also see nation state actor using very sophisticated and personalized lures.
Jeremy Dallman: They will spend a significant amount of time. And this is something we just blogged about a couple of weeks ago, an actor named Phosphorus, which originates in Iran. We're actually using building rapport and building relationships with individuals that are tied to international policy. And by building that rapport with those people, they were actually able to send them invitations masquerading as the Munich security conference, which is a prominent international policy conference. Masquerading as the conference and trying to lure that person to their fake invitation so that they could steal their credentials. A little bit of social engineering happening there. But a nation state actor is going to have the resources and funding at their disposal to be able to build out those more sophisticated techniques. And then finally, I would say there's a lot of nation state actors that spend a significant amount of time building out capabilities, relying on common weaknesses.
Jeremy Dallman: So when a new patch goes out, patching a security flaw within a Microsoft product, for example. A lot of actors will reverse engineer that flaw. Better understand it then use it to weaponize a new exploit. Which is why it's exceptionally important for customers to patch as quickly as they can to avoid that weakness that Microsoft is attempting to patch. That weakness becoming an entry point for a malicious actor because nation state actors will move rapidly to take advantage of that and then attempt to exploit those weaknesses where they can. So that's a couple of techniques that I would say, like I said, we dive a little bit more into in the report. But there's more in there, especially things like web shell based attacks, which we see increasing, but I'll let you go read that into the report.
Natalia Godyla: Yes. Nice teaser for our audience. One interesting point made in the nation state section of the MDDR was the downstream effect. So if I understand it correctly, the nation states will pursue these techniques and then eventually other actors will pick them up. So how does that happen if they are these sophisticated groups that are leveraging, like you said, more complicated malware? Is it that the other attackers use simplified versions of it, or as it's in the wild they get more exposure and are educated on that strain of malware and then are able to use it? So what does the process look like from nation state actor using these attack types to another attacker in the wild?
Jeremy Dallman: Yeah, I think you nailed it there with the second example you gave. Because that's typically what happens is once this exploit gets out in the wild it's not just Microsoft watching for these more sophisticated threats. All of the other actors out there, whether they're criminal organizations or individual hackers, whoever it might be. There's a whole ecosystem of people out there that are watching for these threats to evolve and looking for new techniques. So when a nation state actor might have a particularly sophisticated attack that goes out, there's any number of people who will pick up and discover that through various security researchers in the ecosystem. And then they will immediately go do exactly what we do, which is reverse engineer that, understand how it works. And then you'll see variants come out. You look at things like the VPN exploits that came out in mid 2019.
Jeremy Dallman: Those VPN exploits were picked up and used by an actor that we call Manganese to steal credentials and gain access to victim networks, using VPN infrastructure and holes in unpatched systems on VPN networks. So when you think about a world, the world we live in right now, where everybody's working remote. And global enterprise IT departments are relying on VPNs to improve connectivity and security for their systems. If that VPN infrastructure is not updated in its patching, actors like Manganese were taking advantage of that patch, reverse engineering it, and then going out to find VPN infrastructure that hadn't been patched and then exploiting it to gain access to those networks. Well, what we've seen subsequently is everybody else saw the technique and realized, hey, VPN, everybody's using those right now. And they started taking that and tweaking the same technique. And now those exploits have become, unfortunately become fairly commonplace.
Nic Fillingham: Jeremy, you said that one of the characteristics of a nation state group is the sophistication in their techniques. And so I sort of have to ask, do we know if many of these groups, any of these groups are utilizing AI machine learning? If so, how?
Jeremy Dallman: We don't have conclusive evidence I don't think. I mean, short of us walking into their infrastructure and taking pictures of systems, which isn't something we do. But I think there's enough-
Nic Fillingham: Why not?
Jeremy Dallman: ... indicators.
Nic Fillingham: That sounds like a great idea. I'd make that a priority.
Jeremy Dallman: That would definitely make our jobs a lot more interesting. I would say that we've seen indication of nation state actors starting to take advantage of whether it's machine learning or AI. It's unclear. They're starting to take advantage of more sophisticated techniques in those directions. When you think about a password spray campaign, where you are trying to attempt to guess the passwords for a number of different accounts across one organization, that takes a certain amount of compute, a certain amount of effort and a certain amount of automation that can be enabled. But if you take that and you expand it into something like we blogged about from Strontium in September, for example. We saw Strontium attempting to password spray a number of organizations, and they were spearfishing hundreds of organizations with thousands of password guesses in very short periods of time.
Jeremy Dallman: And then on top of that they were using thousands of IP addresses and anonymization platforms to obfuscate their activity. So when you think about the complexity of that operation and the speed at which they were able to execute it, it would make sense that actors like that are starting to take advantage of machine learning or some automation capabilities on the backend to increase the speed, the effectiveness and the scope of their operations.
Natalia Godyla: I think all of this is leading up to what is Microsoft doing? So how are we disrupting nation state threats today?
Jeremy Dallman: So we do a number of different things. I would say probably the best and most effective way is using Microsoft's voice to raise awareness of these activities. And that comes in a number of different ways. We have the blog posts that we put out. The Microsoft On The Issues blog puts on a lot of interesting content that's derived from MSTIC research. And what that does is it kind of helps drive that broad discussion around what can be done to combat malicious nation state activity against governments, academia, social organizations, individuals. A lot of nation states like to target your personal email accounts, but we still defend those private email accounts because whether it's Outlook or a personal email account, that's something that we also have to protect our customers who might be getting attacked through that type of a vector. I would say probably one of the more interesting ways has been on the legal side.
Jeremy Dallman: So one of our unique ways to target nation state actors has been partnering with our colleagues in the Digital Crimes Unit here at Microsoft. And the Digital Crimes Unit is responsible for pulling together a lot of the evidentiary information and understanding the threats for a legal perspective. And then they take that to courts and use litigation to seize domains and other assets that are being used by these nation state actors. And then actually through legal action shutting down those attack vectors. And then from time to time, we'll also, if we have sufficient information to warrant one time action to delete or shut down infrastructure or assets that are associated with the nation state actor. We'll also take those proactive measures against that infrastructure to basically eliminate visibility or capability on an actor and forcing them to go rebuild that infrastructure. They will typically rollover infrastructure and start rebuilding and come back later.
Jeremy Dallman: So that's not necessarily a whack-a-mole game we want to get into in a lot of cases, but if it's for the protection of our customers, or if we feel it's particularly effective, that is something that we'll do as well. So that's a variety of a few ways. Obviously the one that I didn't touch on is probably the most obvious one, is leveraging our own technology and using all the knowledge that mystic collects about these threats, these actors, their tactics, their techniques and translating those into detections. Transforming and putting those into blocks and protections that show up in our security products and protect our customers in their environments. And the whole objective there has always been to make sure that we're implementing relevant, accurate and actionable threat intelligence for our customers.
Nic Fillingham: Where can folks go apart from reading the MDDR? Where can they go for more information on how to protect themselves against a nation state attacks if they find themselves in one of these targeted industries?
Jeremy Dallman: So we don't have a MSTIC page. I would say in the MDDR,
Jeremy Dallman: We definitely have a section at the end of the Nation-States Reference called comprehensive protections required and it walks through to defensive positions that you can take, the strategies that you can enable there. And then at the end of the digital defense report, we have what are called actionable learnings. And I would recommend you go there and dive into that section as well. And every time MSTIC puts out a blog post, we will always have something at the bottom that are generalized recommendations also. So if we put out a technical blog posts that walks through the techniques of gadolinium or strontium, we will always have at the bottom the specific techniques for that threat that would help you mitigate or protect yourself from that threat. So always watch for those blog posts and then probably for the digital defense report. Go out and look at the actionable learnings. That's probably the best place to start.
Nic Fillingham: Hey, Jeremy. Thank you so much for your time. This has been a fascinating conversation. We've really only scratched the surface of that nation-state threat section of the MDDR report. So if you enjoyed this conversation, would like to learn more head to aka.ms/digitaldefense and download the report, and there's lots more detail and lots more articles linked too, that you can read to learn more about this space. Jeremy Dallman, thank you so much.
Jeremy Dallman: This was fun. Thanks for having me guys.
Natalia Godyla: And now let's meet an expert in the Microsoft security team to learn more about the diverse backgrounds and experiences of the humans creating AI and tech at Microsoft. Today we are joined by Randy Treit. Thank you, Randy, for being here.
Randy Treit: I'm happy to be here. Thanks for having me.
Natalia Godyla: Great. Well, let's kick things off by chatting a little bit about what you do. So what's your role at Microsoft? What does your day to day look like?
Randy Treit: My title is principal security researcher. I'm on the Defender endpoint team. So focused mainly on detecting new threats that we haven't seen before. Protecting patient zero is a big focus of mine. Recently I've started looking into some new kinds of attacks using OAuth phishing. So that's sort of my current main focus area, but I've done a lot in the cloud protection. I've been on the team forever. So I've worn a lot of hats and done a lot of roles.
Natalia Godyla: So what were some of the other roles that you've been at at Microsoft? What was the first one that brought you to Microsoft?
Randy Treit: I've been at Microsoft 20 years. I started in the exchange team and worked on some mobility stuff. But pretty quickly... So I started in 2000. In 2003, I joined the antivirus team, which was brand new at Microsoft. Really Microsoft's first foray into trying to get serious about the antivirus space. And I joined as a program manager, actually. So security research is a fairly new role for me, but was basically worked on the backend infrastructure for the antivirus platform in the early days. And that was the days of worms running rampant everywhere you had SQL Slammer, MSBlast, Sasser worm, Code Red, Nimda. All the greatest hits of when security was a very dark, dark time at Microsoft. And that's when I started and then have done a ton of stuff since then. So I worked on the antivirus engine as a PM and from the engineering side. Eventually moved on to do a lot of work with our cloud protection system in the last period. And then, about two years ago, I guess I moved from engineering side into security research.
Natalia Godyla: So were you sold on security after being part of the AV team? Was that what did it for you?
Randy Treit: Our customers, Microsoft's reputation, friends and family, everybody was just getting hammered by security threats at the time. And I really wanted to do something about that. Working on exchange was fascinating from a technical perspective, but getting into the security space where there was a real opportunity to go to battle against the bad guys and try and really protect. I'm sure we all, back in those days, this is mid-2000s, early 2000s, had friends and family who got hit by a worm or a virus or a scam. And so it was very motivating for me to get into a place where I could do something about that. And that's sort of driven me ever since. And I've done a few other forays into some stuff, like I took a break from security for about two years. Around 2012, went and worked on Xbox for the Xbox One when that was getting released and learned a ton about services. And that was a good break, but I couldn't stay away from the security space.
Nic Fillingham: Randy, I'd love to come back to that first gig of yours working in the anti-malware space. So for whatever reason, I actually went down a rabbit hole recently trying to better understand the history of Defender. It sounds like you were there at its sort of inception. My understanding is that the first anti-malware, antivirus client, first of all, it wasn't built into the OS. It was a download. And was it something that we built in-house or was it an acquisition? Was it a combination? Do you know the history? Were you there for that?
Randy Treit: Yeah. So I was the third PM hired into the antivirus team and it was right after the decision to acquire RAV from a Romanian company called Gecad. And so I started on a Monday and on Wednesday all of the Romanian developers showed up, many who are still on the team today. Marty Marinescu, who was the lead developer of the engine, is still the lead architect on the antivirus engine. And I remember, it was an interesting cultural experience, because they all came in and the custom in Romania was that you would, every morning, go to everybody's office and shake their hand and greet them in the morning. And so that was-
Nic Fillingham: That's awesome!
Randy Treit: Yeah, it was great. Unfortunately they, I think, became acclimatized to the not as polite American way of doing things. That sort of died out after a few weeks. But yeah, it was an acquisition and we didn't actually know what we were going to do with it at the time. So there was always a desire bring the protection capability into the operating system, but that's a big rock to lift and eventually we got there with Defender in the interim. It started out as, like you said, a download. So the initial... For years we've had the malicious software removal tool that comes out every patch Tuesday and runs on everybody's machine to clean up the ecosystem of malware.
Randy Treit: But before that it was actually the very first release of the same engine that runs in Defender today, was something called Blast Clean. It was a Blaster removal tool to remove the Blaster worm. And we released that in late 2004. I have some stories about testing it out on my home machine and actually infecting it. And my kids not being able to play Magic School Bus the next day, and getting a call at the office. So those were fun times.
Nic Fillingham: Can you elaborate on that? Is that the story? Is there more to it?
Randy Treit: So what happened was the Blaster worm, there was a particular patch that if you weren't patched, it would infect your computer within a few seconds of being online. And so we had the early builds... This was December, heading into Christmas season in 2004. And I decided, well, I've got my computer at home. I'll just uninstall the patch and let it get infected. And then I will run our removal tool and make sure that it works. It was not the brightest thing to do. Don't do this at home kind of thing. I was younger and more eager to just do crazy stuff that I would probably be a little more careful these days, but I did it. I uninstalled the patch. The machine got infected. Rebooted, which was part of the infection.
Randy Treit: And then it came up and I ran our removal tool and it worked great and then I decided to try it again. So for those who may remember the Blaster worm, there was another worm called Nachi that somebody else had written and released, exploiting the same vulnerability. And Nachi tried to remove Blaster and then patch your computer. And so our tool removed both of those. And what happened, in my case, was the machine got infected with Nachi, but it was a copy of the Nachi worm, that had itself been infected with a file infecting virus, which infected all the executables is on my machine and then basically bricked it and made it so it wouldn't boot.
Nic Fillingham: I know that I got infected with Blaster worm. I couldn't remember that because I got in big trouble from my dad.
Randy Treit: Oh, yeah.
Nic Fillingham: But I sort of can't remember what it did. I know that it stopped... No one could use the computer. It just completely... The computer was unusable, but can you just kind of bring us down memory lane? If you were infected by Blaster worm, what actually happened?
Randy Treit: It was not a worm that was exfiltrating data off your machine. Now it's all about money and these crime groups trying to exploit the ecosystem with Ransomware and that kind of thing. It was really just designed to spread. So it was purely, as I recall and if I'm remembering correctly, but it would just try and infect... It would infect your machine and your machine should actually be able to run with the infection. Although like in my case, and maybe in yours, if it got infected with a version that was itself infected with something else, it would just brick the machine. Like if there was a file infector, which is what I experienced with the Nachi worm. But essentially it would just try and spread to other machines that were unpatched, randomly spraying IP addresses trying to find another machine that had the vulnerability.
Natalia Godyla: So you mentioned that, right now, part of your role is to focus on protecting patient zero. So how is that different than some of the work you've done in the past? And what's different about focusing on patient zero in specific?
Randy Treit: The attackers could guarantee that they could release something into the wild that wasn't detected because it wasn't detected by current signatures. So before we had cloud protection, you just had the heuristics and signatures that were on disc in these virus definition updates that computers would download periodically. Typically, a few times a day. So you couldn't really protect patient zero because the attackers would always be able to tweak their malware until they saw from scanning with, say, the virus signatures that you weren't going to be able to detect it. And then they would release it. And then the clock starts ticking at that point. And you have a certain amount of time before, say, a customer reports that to Microsoft, or we discover that a sample from some sort of honeypot, or whatever.
Randy Treit: And then now you have, okay, we need to quickly add a signature and ship that out to the customers. So the cloud has been a real game changer because it gives us an opportunity to run all these machine learning models in real time, in milliseconds to make an evaluation of a file that we've never seen before and decide that it's malicious and then block it. That has been a huge game changer in terms of protection capability and really shrinking that time to protection to milliseconds from where it used to take days and hours to get a signature out.
Nic Fillingham: And how do you measure the false positive rate? If there is one, in that sort of protecting patient zero. How do you measure and then how do you find that balance between a couple of false positives, which would be, probably, annoying. But do you allow yourself a few of those to slip through in order to genuinely protect patient zero? Or are the models so good now that the false positives are extremely rare?
Randy Treit: Oh, well, we're always going to have some false positives. ML is not perfect and human expert rules and human logic is not perfect. So there always will be false positives. We have certain thresholds that we try and keep our rules under, or that are basically lines in the sand that, hey, in order to release a new, say, detection rule in our cloud protection infrastructure, it has to run in an experimental period for a certain amount of time. Typically, even a few weeks while we gather all the data on what it would have blocked on, and then we can evaluate, is it having a nice, low, false, positive rate? So there are certain thresholds that we need to make sure all those rules are running under.
Randy Treit: And then we have guard rails to make sure that if all of a sudden a rule or an ML model starts... Something changes under the hood and it starts having too high of a false positive rate, then we have systems to alert and automatically disable things until somebody goes and investigates and that kind of thing. So we're definitely very cognizant of trying to find that balance between blocking the bad stuff, but not causing too many false positives and causing pain and headache for our customers.
Nic Fillingham: And does your team monitor those metrics? Is that what your team, as part of looking after patient zero, is that one of the things that you track day to day, or is that another part of the org?
Randy Treit: Yeah, it's definitely our team. There are other kind of data science focused people who will do a lot of the infrastructure work to support running those metrics. But our team looks... That's creating the cloud rules and some of that capability. We'll work on writing watchdogs and guardrails and alerts and things like that. Just as part of the end to end pipeline of creating that protection.
Nic Fillingham: What are some of those tools that you use day in, day out, Randy? When you start your day, where are you going to? Do have some sort of team dashboard, or are you going into some kind of Azure ML service? Yeah, what's in your toolbox?
Randy Treit: So we definitely have our dashboards and tools that are the sort of go-to place for, oh, you want to see the trend of detections over time, and these kinds of things and monitor your rules and whatnot. I tend to go a lot deeper into the actual data. So I'm a big fan of Jupyter notebooks and pandas on Python. I've done a bunch of stuff in R, in the last couple of years. Lately I've been using Databricks notebooks, which are fantastic because it basically lets you do big data. Sorry. I don't know if you're familiar with the notebook type environment, but it's essentially a combination of marked down notes and graphs and visualizations. Nick, I know you've seen some of my heat maps that I like to generate, showing where we're seeing particular attacks happening globally.
Randy Treit: That's all done in this notebook environment where you have this data under the hood. You can write Python code or R or Scala, and then, to process the data, and then not the other, it'll spit out a beautiful global heat map or graphs or data. And you can just sort of have instant querying at your fingertips. So typically, my day starts with usually firing up some kind of a notebook, pulling in some data.
Randy Treit: I'm often looking for gaps, so where are we not doing well. So what did we see over the last... Let me find files that we're now blocking in the cloud, because our cloud learned that these are malicious, but maybe we miss patient zero and maybe we missed the first 25 encounters. Now, then we started blocking. Oh, let me figure out what happened there. Why didn't we block? How do we close that gap?
Randy Treit: My day job, I would say, is really trying to find protection gaps where we're not doing a good job and figure out how we close them. They go actually implement something to close those gaps. I tend to work with Python mostly day-to-day in a Jupyter Notebook or more recently, these Databricks Notebook type environment. I love it. Compared to the old days of you're running just SQL queries against a small set of SQL data, the things you can do with these, I would say, data scientist type tools like Jupyter Notebooks is very freeing. I guess that's how I would put it.
Nic Fillingham: Randy, what's flagging those gaps? So, you said you look for gaps. Is that just your experience, your expertise, you know what you're looking at when you see data, when you see dashboards, when you see reports; or are there a combination of processes that are specifically looking for a detection that picks something up and then went backwards in time and realized that "Oh, here are some historical detections that we actually miss"? How do you find gaps? I think that's the question.
Randy Treit: It's a combination of manual spelunking on into the data and going off intuition or things I've done before, but we do have automation that will flag certain events. We have watchdogs and other rules that researchers write. In my mail inbox in the morning, often, I will have a list of these potential misses where maybe we missed detection on the first patient 0 through 10, and then we started blocking. So, I might go and look at, "Oh, let me dig into that a little bit and find out what happened there." So, in some cases, it might be that we have a malware probability threshold that we were looking forward to say from an ML model that says, "Oh, block if the probability is 0.95. So, 95% probability that this file is malware."
Randy Treit: Going into the data in telemetry, I might see that we didn't block because the probability was 0.93. So, one of the things I would look into then, oh, can we reduce that probability that we're looking forward to block from that 0.95 threshold to 0.93? Maybe code up something to model that or to run for a few days in experimental audit mode and see, "Does that lower threshold still meet our false positive targets?" If that's looking good, we can turn that on live, something like that.
Natalia Godyla: This is a bit of a deviation, but it would be great to understand, "What kind of context do you bring to this role from previous jobs? What were you studying in school? What did you intend to do? What were your jobs prior to Microsoft, and how do you use them in your day-to-day?"
Randy Treit: Yeah, that's a great question. So, I was actually studying Philosophy in Pacific Lutheran University down in Tacoma. I'm a native Washingtonian. So, Microsoft was right in my backyard. It was basically the height of the dotcom boom and the end of the '90s. I had finished up the Philosophy Program at PLU and was planning to become a philosophy professor but needed to get a job. In the interim, I was married. We had a young child, another one on the way. So, I decided to take a break from school, get a job. I started as a technical writer actually at Microsoft on the Exchange Team. I think you talked to Emily Hacker. I listened to the interview and learned that she was also started as a technical writer. So, that was pretty cool.
Randy Treit: And then worked in exchange for a few years before I got asked about joining this newly formed antivirus team. I made the jump there. I actually never finished my four-year degree. So, I made a plan with my advisor. I finished the philosophy program, but still had some general university stuff to finish up. But once I started at Microsoft, I was just off and running and never looked back. So, it's been an interesting journey. Sometimes I definitely suffer from, I would say, imposter syndrome here and there, where I spent a lot of time writing code day-to-day, but I've never been formally trained in computer science. It's all been self-taught or picked up on the job thing.
Randy Treit: When I moved from a program management and the engineering side into research, I came without the deep reverse engineering background that a lot of my colleagues had. So, that was something that I felt like, "Oh, this is going to be hard for me to pick up." Sometimes that lack of a formal academic background, I feel like it was a bit of a chip on my shoulder, but I just try and do the best I can and go from there.
Nic Fillingham: Talk a bit about philosophy, and then I'd love for you to talk about how and if you use it in your job today.
Randy Treit: Yeah. So, I was not a good student in high school. So, I barely graduated high school with a very low GPA. So, when I decided to finally get my act together and go back to school, I started at a community college. I needed to take English 101 just as part of every college requirement. So, the English 101 class I took was a combined English 101 and Philosophy taught by two professors who were husband and wife. Debbie Kuder, the wife taught the English portion, and then her husband, John taught the Philosophy portion. It was basically an amazing class. My identical twin brother, who also works at Microsoft by the way, was in the same class with me. We both just fell in love with philosophy.
Randy Treit: I think, I just love the idea of open-ended questions that had no answers. So, philosophy, I think differentiated from the sciences, it's dealing with questions that will never actually be answered, like what is beauty and what is a good argument? There's always going to be different opinions. Just the idea of these big open-ended unsolvable questions, but the people will keep getting closer and closer to the truth hopefully over time, I just fell in love with that. In terms of applying philosophy at work, I think the biggest thing that I got out of studying philosophy in undergraduate school at PLU was the rigorous approach to problem solving. So, even though you have these big open-ended problems, like I said, there probably are never going to get answered.
Randy Treit: The approach of philosophical approach is very rigorous and requires incredibly good communication skills to be able to communicate your ideas effectively and, in your arguments, cogently. That, I think, has stood me in extremely good stead in my career. I think that's one of the things that I bring to the table. I think someone like Emily, you mentioned with the journalism background, it's just that ability to communicate. There's so many brilliant people who work in the technical field, but who are unfortunately not great communicators. Often, they need someone to help translate what their brilliant ideas into something that other people can actually understand what they're aiming at.
Randy Treit: That's something that I think I've been able to do fairly successfully. Just that ability to really rigorously attack a problem and break it down into small components, which I think comes from some of that training I think has also done a great job or has stood being a good stead with malware analysis and threat analysis and that kind of thing.
Natalia Godyla: So, I know Nic is dying for me to ask this, but you said you had an identical twin, you just dropped it in there casually that works at Microsoft. Do you guys pull pranks together. Have you done it as kids? Do you do it at Microsoft?
Randy Treit: You have no idea. So, Mike actually worked on the antivirus team at the same time as I did. So, he joined Microsoft before me and has worked on NT 5, which became Windows 2000 and is a brilliant dev, but he was actually one of my devs and I was his PM working on the antivirus. This is probably mid-2000s. For a number of years, we were on the same team. And then he went off to Intune. But I mean, the amount of confusion we caused when people would walk into meetings or even just down the hall, it was quite fun. I'm sure we played some pranks. It's been great.
Randy Treit: There was one time very early on, we weren't on the same team at that point, where he was in my office over an exchange. He had come over to grab a coffee. He was across the street. I had gone down to get a refill or use the restroom or something. This guy, David came in and started talking to Mike, like he was me, "Hey, Randy, I've got some questions about this thing." Mike was like, "Oh, I'm not Randy." David looked at him and just shook his head and said, "So, anyway, I've got questions. Do you know about this?" Mike's like, "No, I'm not Randy." He looked at him and he said, "wait, are you serious?" So, we've had those kinds of incidents.
Randy Treit: Mike is my go-to person whenever I get stuck on a programming problem, because he's a brilliant programmer. So, I'm constantly sending him my code and saying, "Hey, I'm struggling with this." He usually responds with something like, "What is this monstrosity?", and things like that since I'm not nearly the coder that he is.
Natalia Godyla: Subtle.
Nic Fillingham: Who's the older twin by a fraction of a second or a minute?
Randy Treit: Mike's four minutes older than I am.
Nic Fillingham: I love it that your prank was actually a wholesome misunderstanding, an unintentional wholesome misunderstanding. I was typing frantically with Natalia, trying to see if there was some example, where you each went to the other's annual review and just tried to just say ludicrous things to the manager to see when they caught on, but no.
Randy Treit: No, I haven't done too much of that at work. Although, I mean, in high school, he would skip class and I would go to his art class, because I had a girlfriend who was in the same class. One day, I got called up to make a presentation, the person they thought I was Mike. I was completely unprepared and I just fumbled my way through it. I learned that, "Oh, that didn't work out the way I was hoping it would." I'll throw this out there. My younger brother also works at Microsoft. He is a producer on Xbox video stuff. So, there's a bunch of us running around.
Nic Fillingham: How many other Treits are there?
Randy Treit: My sister, Tammy worked on Exchange at the same time I did back in the day. There are six of us Treit siblings. I guess four of us have worked at Microsoft. My younger sister is a doctor in Seattle, and my older sister is a teacher in Germany.
Natalia Godyla: Thank you, Randy. We're happy to have you at Microsoft. Happy to have two-thirds of your family at Microsoft here, and we'll definitely love to have you back.
Randy Treit: That was a lot of fun. I really enjoyed the conversation.
Natalia Godyla: Well, we had a great time unlocking insights into security from research to artificial intelligence. Keep an eye out for our next episode.
Nic Fillingham: Don't forget to tweet us, @msftsecurity, or email us at firstname.lastname@example.org with topics you'd like to hear on a future episode. Until then, stay safe.
Natalia Godyla: Stay secure.