Supply Chain Attacks: Open Source or Open Door?

Sherrod DeGrippo: Welcome to the Microsoft Threat Intelligence Podcast. I'm Sherrod DeGrippo. Ever wanted to step into the shadowy realm of digital espionage, cybercrime, social engineering, fraud? Well, each week, dive deep with us into the underground. Come here for Microsoft's elite threat intelligence researchers. Join us as we decode mysteries, expose hidden adversaries, and shape the future of cybersecurity. It might get a little weird. But don't worry, I'm your guide to the back alleys of the threat landscape. Today we're talking about what modern incident response actually looks like when open source ecosystems, CI/CD pipelines, cloud identity, and supply chain compromises all collide at once. Welcome to the Microsoft Threat Intelligence Podcast. I'm Sherrod DeGrippo. From the outside, people usually see something like a headline. "Malicious package discovered, GitHub compromise, suspicious NPM library." But what they don't see is the operational reality behind those headlines. People figuring out whether something was actually exploited, separating the telemetry from an active compromise, and dealing with things like infrastructure that disappears, while you're still investigating it. We love incident response on this podcast just as much as we love threat intelligence. And joining me today are two people who live in that world every day, Allie Luhrs and Mario Samolis from Microsoft Security. So we're going to talk about investigations, reverse engineering, and what was presented at this year's BlueHat USA by the two of you. Allie and Mario, thank you for joining me.

Allie Luhrs: Thank you so much for having us. We're really excited to be here.

Mario Samolis: Likewise, thank you for having me.

Sherrod DeGrippo: So I'm really excited to talk to you guys. I'm so glad you're here. Tell me a little bit, Allie, I'll start with you about sort of what your day-to-day at work looks like and what you're typically doing.

Allie Luhrs: Yeah, that's a great question. So our team in particular kind of spans the gambit in terms of what we're dealing with every day. It's always something new. We're generally not dealing with the standard cases, so we get to dig into the stuff that is really interesting. And this is kind of where we landed with the NPM stuff. So a lot of it involves incident response. We'll do static analysis, reverse engineering, and just generally fielding and investigating a lot of like the larger incidents that might occur in the Microsoft ecosystem.

Sherrod DeGrippo: Okay, that sounds pretty intense. Mario, how about you? You're taking it easy all day, huh?

Mario Samolis: Oh, I wish. Never enough time to do everything we need, as we say. For myself, a lot of similarities with Allie. I do a lot of threat hunting as well, so proactive and reactive threat hunting across the Microsoft environment, as well as a lot of reverse engineering. So I'm a malware analyst by trade and I live and breathe malware particularly. So I have a propensity to find something interesting and start hunting off it and see what threats I can pull and where does the iceberg go from there.

Sherrod DeGrippo: I think that a lot of our listeners relate to this because we've got a lot of incident responders in the audience who want to know what it's like out there. And Mario, I'll kind of start with you. You've worked on incidents where the infrastructure was live and then it was gone before your investigation was completed. Tell me a little bit about how fast threat actors move and change.

Mario Samolis: That's a great question. It's going to largely vary by threat actor. Some of them within a matter of minutes, they're burning infrastructure rotating through. Some of them are more long-lived. From my experience, commonly we see the initial infrastructure, so the initial delivery mechanisms, might rotate quite often, but once you get behind that and actually find the second or third stage C2 environment, that's going to be more long-lived. So the challenge quite often is how fast can I move through to figure out what the adversary is initially trying to do? And then start trying to figure out their TTPs and tradecraft to get to that more enduring infrastructure.

Sherrod DeGrippo: Allie, what about you? How are you seeing threat actors adapt?

Allie Luhrs: Yeah, that's a great question. In particular, like going back to kind of what we talked about at BlueHat Redmond, I mean, the TTPs and the IOCs, the C2 servers, they're all so incredibly dynamic. And luckily, we were able to find some of the C2s and the IOCs that persisted really across a lot of the campaigns that we were looking at. But generally, like Mario said, those are the second stage that we're seeing. And we're seeing a lot of the first stage B so incredibly dynamic where we're seeing within an hour rotation of the C2 domains of different hashes. It can be really difficult to keep up and that's why we have to dig a little bit deeper to find the things that persist a little bit longer.

Sherrod DeGrippo: So let's start off then about the BlueHat presentation. So for those of you listening, BlueHat is Microsoft's opportunity to bring researchers, vulnerability specialists, security practitioners, all to the same place. We do it in Redmond on the Microsoft campus. It's an incredible event. It is invite only. So if you've ever been there, consider yourself in the exclusive club. And we see lots of presentations both from Microsoft employees and from external speakers. Mario, I'll start with you. What was the basis of your talk this year at BlueHat? And then we'll get some color from Allie on it as well.

Mario Samolis: Absolutely. Our basis for our talk was, you know, looking at the recent events in NPM ecosystem as well as other registries, determining the root cause behind it. Why is it continuous, you know, attacks that we're seeing, why malicious campaigns that are occurring, and what is the overarching theme, overlapping theme, behind it? Essentially, it was a revelation in that this is a factory-style operation. It's not a one-off for a group of individual actors, and it's a very templatized, very methodical campaign that the adversary is conducting.

Sherrod DeGrippo: Allie, what was your point of view on it?

Allie Luhrs: Yeah, so this is really interesting. And to add a little bit of fun to it, this all really spurred on when the first Shai-Hulud campaign kind of came out. So Mario and I were actually on call together as incident responders, and we were fielding just a swath of these NPM incidents come in. We were wondering why these were back to back, why we were fielding so many at once, what, I guess, spurred this as kind of a threat actor focus? And so we were interested in that. We got really good at responding to those. And then about four months later when we were on call again, we hadn't heard anything in the NPM ecosystem until we were on call, and we started fielding these again. That kind of spurred us to take a look a little bit deeper at, is this more of a pattern that we're looking at? Can we correlate this, these campaigns, all together and dig a little bit deeper to see some more, I guess, common aspects of these attacks that we can use to potentially move a little bit more proactive in our response? Because every single time we're caught flat-footed, where, you know, responding is taking a lot of in-depth work for us to go back and identify were we compromised, what assets might have been compromised, what exactly was exposed during this attack, and moving towards really tightening up those security controls with the findings that we found through this research.

Sherrod DeGrippo: So Mario, tell me, for those listening, if they don't know what NPM compromise is, how could you kind of walk us through what that means?

Mario Samolis: Well, that's a very broad topic. But NPM, just like PyPI, NuGet, or any of the other packaged ecosystems, we're talking about open source software that millions upon millions of tooling products, developers across the globe depend on every single day. Certain packages, such as React, for example, have 100 million downloads a week. So you think about all your modern web applications, things of that nature, they use these open source packages that enables them to conduct their daily operations, right? So you don't have to reuse library code, recreate code. It's all there, created for you by the community. So these compromises are essentially targeting the ecosystem, the software ecosystems, and the various different ways they can subvert the controls or poison the ecosystem.

Sherrod DeGrippo: So essentially, a legitimate package can get hijacked. Things like a malicious package can impersonate a real one, or a maintainer account on an open source piece of software gets stolen or compromised. Dependencies in a tree become malicious, something around the infrastructure is compromised for the build or publishing. This is big, dangerous security implications for the software supply chain. And I think for me, what I've been really watching lately is, as various AI tools become better and better at finding vulnerabilities, there also is this software supply chain risk that it's been escalating for a while. But Allie, I'll ask you, is it accelerated right now?

Allie Luhrs: Absolutely. I mean, one of the biggest issues that we're seeing is that the use of AI and kind of templatizing these attacks makes it so that the cost for the attackers is almost zero. So they're able to churn out these packages, whether they're kind of typo squatted impersonated packages or being able to make attacks on objectives on compromising either maintainer accounts or taking over packages in general. We're able to see that at a scale that we haven't fielded before. And so while their cost is zero, our cost is not as incident responders. And that's definitely kind of the crux of the situation that we we're trying to drive down.

Sherrod DeGrippo: I think for those listening, too, what's scary to me to think about is that so much of the open source ecosystem runs on things like volunteer labor, trust, automation, maintainers getting tired or going on vacation, and it being sort of like a load-bearing individual in that ecosystem, where if one person kind of gets a little tired and is unable to keep up with things, that can be really bad. Defenders can't manually validate every single thing. It's unfortunately a broad responsibility amongst all of us.

Allie Luhrs: Yeah, definitely. I think even most recently, we saw a dormant co-maintainer that had their account taken over, and that ended up in a kind of a cascade of compromises in that ecosystem.

Sherrod DeGrippo: What I think is going to be interesting -- and again, I say this all the time and I'll continue to say it, I am not a software developer. I am not a coder. I am a hack it together until it kind of works person. But now the reality is we can generate so much code even without a software engineering background, even without that capability. And it's interesting watching AI coding assistants create code because sometimes they do and sometimes they don't do library dependencies. Sometimes they follow the standard, well, we'll tap into this particular package, and we'll use that as part of your application that you're writing. And sometimes the AI coding assistants just create it themselves and don't use the libraries. So I guess something I'd like to ask both of you is, from the open source software supply chain ecosystem reality, would you rather see people coding with dependencies on these tried and true libraries, or would you rather see them creating them themselves?

Mario Samolis: For myself, coming from the government and then working in the civilian space, I think it's dependent on your usage and what you need. Some environments you do need to create it all self-contained, no external dependencies for many reasons, air gap networks, for example, and so forth. So it really depends on your usage. The big thing that I would say is whoever's conducting the engineering, they have to have an engineering plan, have to have an ability to validate the code that the AI is producing. It's not enough to say, go create this, and then have no means to actually validate it and just take it on its word. There are several issues and dependencies that could be present that you aren't aware of, almost like a ticking time bomb that's waiting to be exploited by adversaries just because you're operating at a very rapid publishing pace. But you have no context or the skill set to properly validate and make adjustments as you need to in a plan.

Sherrod DeGrippo: Allie, do you have any opinion on that? What's the best way to architect software for a defender?

Allie Luhrs: That's a great point. And I'd like to start off kind of by saying we shouldn't be afraid of dependencies as a whole. I think that there are ways to include dependencies to kind of build on the work of others in a much safer way where we're protecting ourselves and we're also kind of auditing the ecosystem that we've pulled into our code and that we've presented in either in our products or whatever we are maintaining at the time. Things like pinning our dependencies to specific versions or even better hash pinning, things like making sure we have full, complete SBOMs of all of our dependencies, tertiary dependencies, and all the packages that are getting pulled into, you know, a coding project or product or whatever it may be. So while it might feel like the safest thing to do would be to code it all yourself -- and obviously that can be hit or miss with AI right now, and we're still dealing with a lot of like hallucinations, so to speak. I think that it's important to understand that we shouldn't shy away from that collaboration that we've really built on for decades as kind of a internet community and a coding community.

Sherrod DeGrippo: I think what's interesting, too, is that, you know, for a long time we thought about hosts, we thought about endpoints, we thought about threat actors getting on the endpoint, then we started thinking about threat actors getting in the cloud. And now it's really like threat actors getting into the software that makes the software. Like this isn't just like end-user stuff that's being compromised. These are the dependencies, the libraries, the foundational pieces that make up the software that gets onto everyone's box.

Mario Samolis: Yeah, and unfortunately, a lot of the tooling that we have, whether it be AI or otherwise, they have permissions because you have to use them to debug. They have the context that they need to run your operational environment. So now, as you mentioned, the calculus has changed where attacker compromises an AI or any other tooling. Now they have almost unfettered access in some cases to the environment and the entire build or publish pipelines by virtue of the permissions that it needed to in order to run for you.

Sherrod DeGrippo: It's such an interesting and really smart on the threat actor's part way to go about an attack, right? It's why target a single organization when you can target this software that builds the software that they use, that your target organizations use? So Allie, I'll ask you, what was something really interesting that in your BlueHat research for putting this presentation together, what was something that you were really excited to talk about here?

Allie Luhrs: Oh, that's hard to pick just one.

Sherrod DeGrippo: You can have more than just one. We have a whole podcast.

Allie Luhrs: [Laughter] Okay. I think that, gosh, there was a lot of really cool things that we uncovered in our research. I think that some of the most exciting things -- exciting is such a double-edged sword there -- but exciting things to uncover was just seeing kind of the operational patterns we saw from the threat actors where we could use all of these clues that we picked up together, correlate them into a nice little package and understand a little bit better about when the operators are operating. What does their workday, quote unquote, look like when they're publishing these packages? What are some patterns that we're uncovering in terms of how they might collaborate with each other? Are there multiple operators? Are they working together or is there like a tiered operational system? In our research in particular, looking at kind of the main operating group that we were looking at, starting to see that clear hierarchical infrastructure come out was really, really interesting for me. So give me the answer to some of those. What does it look like? What are the operating hours? Do they ever sleep? [Laughter] Mario's shaking his head like no, they don't ever sleep.

Mario Samolis: No, so that feeds into the question and I'll kind of, I mean, we could piggyback back and forth, Allie, because there's a lot of things we found. The temporal analysis, right? So seeing the actual distribution, seeing when they publish and what their time zone, what it looked like, it was very evident that there's a professional, almost military government operation and that they had standardized hours. Kind of similar if you worked in a skiff, for example, you know, a nine to five, nine to three type operation, you're out of work on a day and you can't bring anything home. And it was a very clear Monday through Friday distribution on certain hours that align with career standard time hours. So things like non-publishing during certain federal holidays, their own federal holidays, and where you see a big burst that happens prior to the holiday and then a complete silence for multiple days and then it picks back up. Seeing that interval where every two to three months, there was a repeated pattern, almost like a wave, where you see a spike increase of publishing after rounds of production style testing, and then the campaigns operationalize and it drops back off. The things like that started, as you put them together, started painting a very clear picture of what their operational schedule appears to be like.

Sherrod DeGrippo: And are they working hard or hardly working?

Mario Samolis: I would say a combination of both, in that some of the research we found, there was a clear templatized model that was happening where one operator decoupled the delivery and payload mechanisms, right? And once they finalized that, it pushed it out and then they started using automation. As Allie said earlier, AI to mass produce these packages and then just keep pumping them out at a significant pace. So they're working a lot on their cover stories, but some of the other aspects they're not working so hard on.

Sherrod DeGrippo: That's interesting. I wonder if they're -- do you think they're working 40 hours a week?

Mario Samolis: Yeah, I think they're working more than 40 hours a week, to be honest with you. What they're doing essentially is working probably 60 hours, maybe more than that. But a lot of it is attempts to catch developers in other countries and line up with their development cycles, right? So as things start happening, you hear news of ramp ups of different technology. They ramp up their production and they're developing, try to match that and catch somebody off guard and then they ramp back down.

Sherrod DeGrippo: I see. Allie, any comments you want to make on threat actor operational functionalities?

Allie Luhrs: Yeah, so I guess to add on to what we've all talked about is it's been really interesting to see -- to kind of pull in the AI component as well, it's been interesting to see the ramp up to the operations as they've started, say like 15, 16 months ago to now. I think as NPM compromises and open source security being kind of at the forefront of everybody's mind right now, we're seeing their operations ramp like significantly. And I think that that's a combination of a few things. One is the increased reliability of AI tooling and their ability to pump out the packages faster. But also as this becomes more visible to developers and responders alike, they're having to churn out more of these typo squatted or fake packages to catch the same number of people they were catching before. So it's definitely like spraying it as far as they can go to catch the stragglers here and there.

Sherrod DeGrippo: That's fascinating. And I think when it comes to software supply chain, this stuff is just going to get worse and worse, because they've got the operational capability to make the things happen that they need to have happen.

Allie Luhrs: Yes, absolutely. It is getting scarier by the day, but we do have the tools and the knowledge as we kind of dive deeper to combat it better. And that's kind of the silver lining of all of that, even though it is definitely at the forefront in the fears of, I think, responders and executives alike in the security space.

Mario Samolis: Can I add one quick thing to the audience? And as we said in BlueHat, even with the advent of new technology, you know, using AI, for example, to code and create the packages, that's also a discriminating factor in identifying an enemy. And what I mean by that is if you look at somebody's typical usage and how they write code versus the shifts, you can start profiling those changes in that dynamic. And that's very evident and you start creating markers for identification as well. So while AI is helping in the economics of war being pushed to insignificant levels for the adversary, that is also something that you can use against them to profile them and identify them even further.

Sherrod DeGrippo: So let's talk a little bit about responding to these incidents. Mario, I'll start with you. How did you come across this material that you ended up building the BlueHat talk around? This was an incident that you responded to. What was that experience like?

Mario Samolis: So it started off, as Allie said, with Shai-Hulud. We were, you know, on call together. I had just come off a different engagement, and she tagged me in and said, "Hey, I got some malware. Do you want to look at it real quick?" Being the malware analyst, of course, I'm like, yes, I'll go look at it. And as we started working through this engagement, looked at it, we dealt with it, and we're just kind of responding from one engagement to another. So kind of in my head was like the military background that I come from is, okay, instead of going from one firefighter, as we say in the military, to another, what is the overall picture? What pictures are pertaining? And I started digging into the payloads themselves and started pulling more research. And as I pulled more threads, investigative threads, then I started seeing a pattern start emerging, right? And I started forming my investigative hypothesis. And that led us to the KM-SEC feed data that we saw the initial stages of DPRK activity that was being published and being validated. So we took that data set and then we started looking at all the major ecosystems in NPM, things like Vites and Tailwind CSS and so forth. And I did a full decomposition on those, full reverse engineering the kill chain, in order to identify if this is valid data set and what the effects would be if the adversary had exploited it, right? And from there, Allie and myself started forming a hypothesis on how we could further correlate that data. And then just started, as we found more information, we started feeding it more into our hypothesis and just kept pulling more and more threats from there, to be honest with you. From there, the last aspect I'll kick over to Allie is, you know, as we started pulling each piece of thread and we started looking at our individual hypothesis, then we looked at data set and it's like, okay, individually it tells us one thing, but when we start laying, overlaying all the information, all the results together, what is the composite, what does the mosaic tell us? And that's where it started leading to our evidence in the BlueHat talk.

Sherrod DeGrippo: So when you're responding to this incident, Allie, Mario kind of walked us through the beginning of it. What is it like being in the heat of the moment in that and turning more and more things over? And then if you could also kind of help me understand, from an incident response perspective, how do you prove where the impact stops?

Allie Luhrs: That's a super great question. So at least coming from the incident responder perspective, blast radius can vary on these so much. It really just depends on maybe what package they're targeting and/or compromised at the time and how that might widely be used either as dependencies or just as the parent package as a whole. And so we really struggled to, at the start, get developers on board with this. I think that that was one of the main things that we had to focus quite heavily on when we first started responding to these is working as a team together to understand, okay, we have these compromised packages, but it doesn't just stop at, say, a developer doing NPM install of the package. We have to look deeper to understand the full blast radius, and that's where it gets so huge. You're not just talking about the main dependencies, you're talking about the tertiary dependencies. And this is all on a web of trust. So when you have a parent package, you have to trust that that parent package is safely treating its dependencies properly. And if they aren't, that significantly increases that blast radius. We're adding packages that aren't necessarily compromised to something that we have to dig deeper and keep kind of at the forefront of our mind.

Sherrod DeGrippo: And so when you say working with developers, what does that look like?

Allie Luhrs: So that's a really great question. Obviously, there is a lot of telemetry that defenders have to kind of dig through to understand and kind of dissect developers' work to point out where we're seeing those packages and the dependencies and whatnot. And when you're working with developers who are really focused on putting out a great product and doing it at the pace that their customers need, sometimes that might not be done in the most secure way, simply because that's just not at the forefront of their mind. And so working with them as a team to say, hey, let's figure this out together. Let's take a look at your dependencies, your SBOM, and let's go through it together so that you're not just a consumer of these blast radius findings that I'm throwing at you and saying, hey, you need to go check this pipeline or check this package, or you need to look at what secrets your pipeline might have access to. I want to partner together so we can investigate it together. You feel invested in our investigation, and you feel like you have ownership of, hey, I also found this and I understand why I have to do these response actions that you're asking me to do.

Sherrod DeGrippo: And how did you find that response when you said, hey, I need you to go do this stuff? Were developers like, yes, absolutely right away? Were developers like, what? Who are you? How did that work?

Allie Luhrs: Obviously, that really depended on the team that you were working with. Some were fantastic and asked no questions and just wanted to go do it, though I don't dissuade questions by any mean. There were others that maybe their product had a much higher impact and it was a lot harder to do these remediation actions because it had a lot of downstream impact. And that's generally where you got a lot of pushback. But honestly, it was down to developers just not understanding why they're being asked to do these things. It has gotten so much better, I would say, over the past six months now that unfortunately these are becoming much more common. The developers are used to us bothering them about these things. But, you know, I've noticed that shifting my mindset really too from just incident responder sending findings to developer to that team holistic mindset has really helped with the response and then their attitudes towards, or our attitudes in general towards taking these findings and then actioning them however it needs to be.

Sherrod DeGrippo: Mario, what was your experience? Did you get any developers, you know, rolling on the floor and crying and having tantrums or were they cool or what?

Mario Samolis: My experience working with developers at first, typical incident response, you're reporting from one thing to another. It feels like you're frantic. You're just trying to organize. And each team is very chaotic. Developers are not understanding why you're asking them, who are you? They're more concerned about their pipeline just broke or the product just broke and the second order effects behind that. And what I realized is kind of similar approach is if we -- for myself, if I looked at it as a, I'm here to help you, collaborative environment, right, I'm here to explain to you and work with you, almost like if we were side saddling on a project together, working together, they were more responsive, more willing to meet me in the middle, right? So coming to them and saying, hey, you know, I think you might have a compromise. Here's the reasons behind it. Here's what I see second order effects. Can you validate this? Because I don't have access to your pipeline. Can you look at the other side and see? And we kind of take both of our findings together and create that mosaic and see what the reality, the ground truth is, then they're more willing to work with us. So it's more of like a team dynamic, not I'm here, you messed up, let's go fix this type thing.

Sherrod DeGrippo: And so what you're both essentially asking these developers and maintainers to do is to go back through their code and potentially find a threat actor in that code. And do they feel equipped to be able to do that? Are they like, no, this is crazy, we could never do that large of a code audit? We'll never find it. Do they seem like they understand what you're asking for?

Allie Luhrs: I think that generally we have enough information to give them where we can kind of pinpoint where they might need to look. But definitely there are lots of times where we're getting on calls with them and, as Mario said, riding kind of side saddle to investigate it together. So we're trying to make it as seamless as possible for them to get back up and running and doing the things that they want to be doing rather than having to work with us on maybe the worst day of their week or their month or something like that because their ecosystem was compromised. Some of them are very excited to help us. They want to kind of dig in and they find it interesting, even though it's a little scary for them. And others just want us to give them the data, tell them what to do, and they go and do it and then there are no questions, and we don't talk to each other again, you know, type thing [laughter].

Mario Samolis: Yeah, it varies drastically by team. And in some cases, you know, sometimes the developers just aren't, they're not aware of what's going on or what the compromise is, right? The ground truth of what they believe based on their ecosystem, their pipeline, is different than what we're seeing as the responders. So we've had developers before tell us, no, I am not compromised. Here's the reasons why. And then we go over and we're like, no, you are because of these, and we have to trace it through with them and show them. And in that case, for us, it's more of an analytical process, right? Almost like a scientific experiment, kind of showing them why, not necessarily, you know, blaming them, but just trying to get them to get buy-in as to why they should let us help them. So a lot of time it's getting that initial buy-in and then that opens the doors for us to operate and help them.

Sherrod DeGrippo: I talk a lot about the differences between kind of like the InfoSec persona and software developer persona. It's something that has been fascinating to me for a long time. But coming to Microsoft and working so closely with so many developers, you might not know it, but you might be able to imagine that Microsoft is a very software developer heavy company. We have a lot of people writing a lot of code all the time. And there really is a whole cultural identity here at Microsoft around being a Microsoft developer. And I've spent a lot of time with our developers, and I try to meet with them pretty frequently. I have big workshops that I do around threat intelligence and security for developers. And it's been really fascinating how different they are from me, from security people. They're happier, I think. They enjoy their work in a way that I don't know that I ever have [laughter]. You know, software developers, they live to release code. They live to put features out. They live to have something in somebody's hands that someone is using and loving. And they're makers. And I feel like security people are very much breakers. We are looking for the problem. We're looking for the issue. We're looking for the bad thing. We're looking for the vuln. We're looking for how anything in front of us can be bad or wrong or dangerous. And it's a very, very different mindset. And it's one that I think obviously we have to really partner with developers. Because ultimately developers are creating a battlefield that we're going to defend on. And we have to ask them to secure the battlefield as best they can. But at the very least, give us the higher ground so that we have the tools that we need to fight to get threat actors out of environments when they do get into them. And I think that kind of what Allie and Mario put together for BlueHat is a good example of that. There was varying responses across the different developers and maintainers and everything, but ultimately everyone had to get it done. And so I'll ask you both, now that, you know, this side of the engagement has kind of wound down, how do you feel about how that went? What would you do differently? Mario, I'll start with you. You seemed really to light up when I said, what would you do differently? How did it go and how would you change things?

Mario Samolis: As far as BlueHat as a whole, amazing experience. To be quite honest, I think the reoccurring theme when me and Allie were talking about it was, we wish we could have talked more and more in depth about things. And the biggest limitation was time. There was so much amazing research that we had to come up with and it's just trying to distill down the big nuggets. But then as you start talking about the big rocks, as I say, am I giving enough context? Am I giving enough information where those large analytical conclusions make sense? And am I, you know, painting the picture correctly? Yeah, I would love to have spent more time, been able to dissect it more and give more of the foreground of what I'm seeing in the code base and so forth. For next time, just continue it. We have a lot of amazing research, and it'd be amazing to keep evangelizing what's going on in the ecosystem and how we can better defend it as an organization as a whole, as defenders as a whole, not just Microsoft specific.

Sherrod DeGrippo: Absolutely. I love that point of view. Allie, how about you? How did this experience kind of impact you and, you know, what were you thinking you'd want to do differently next time?

Allie Luhrs: That's a great question. This was, just to echo Mario, just an amazing experience. It was great to be able to get on stage and kind of nerd out, so to speak, with all the other security folks that were in the room. We had some great post-conversations with lots of different people that came up to us for Q&A. They just stopped us in the hall for a quote unquote hallway con. And just spurring on that chatter within the security community was so fun. I think that the only thing I would love to have for next time, and again, echoing Mario's comment, would love just to have more time to talk about this with folks, but also to maybe spend a little bit more time tying it back to the so-what. You know, we had so much great research that was so much fun presenting. And of course, we're the researchers that are compiling it. So of course, we're interested in it and we're passionate about it. But when we're presenting to the wider security community, they want to know the so-what. Why is this important for me? And we did touch on that a little bit in our talk, but I would have loved to bring it home a few separate times in the talk, just to drive home, hey, this is something that needs to be higher on your radar than just I'm consuming these blog posts and taking a look at things every so often.

Sherrod DeGrippo: I think something to really take away is that, in today's reality, incident response and finding threat actors in your environment, in your code, really requires a lot of validation, competence in your operational capabilities, being able to make the decisions when you don't have all the information. And then what you were talking about, pulling in all of those owners and stakeholders of those package ecosystems that can actually make the difference when you need them to move at that moment. And I think that that's really huge. So a lot of this stuff is now happening. As I was saying, it's not the endpoint. It's not the cloud. It's now happening in places that defenders really didn't think of as part of the attack surface, as part of the perimeter. And these ecosystems, these workflows, these code repositories, these open source platforms, these are the places now that threat actors are seeing an opportunity that they may not have seen before. And it is ramping up exponentially. It's always been a vulnerable space, but now it really is a target. Allie, Mario, thank you both so much for joining me today. And thank all of you for listening to the Microsoft Threat Intelligence Podcast. Be sure to follow us at msthreatintelpodcast.com or wherever you get your favorite podcasts. Thanks for joining me.

Allie Luhrs: Thank you so much. This was so fun.

Mario Samolis: Thank you.

Sherrod DeGrippo: Thanks for listening to the Microsoft Threat Intelligence Podcast. We'd love to hear from you. Email us with your ideas at tipodcast@microsoft.com. Every episode, we'll decode the threat landscape and arm you with the intelligence you need to take on threat actors. Check us out, msthreatintelpodcast.com for more and subscribe on your favorite podcast app.