Securing the Cloud With Mark Russinovich

Announcer: This is a Cyberwire Podcast.

Bret Arsenault: Hi, I'm Brett Arsenault, Chief Information Security Officer in a little company called Microsoft. Recently, I was approached by some customers who were really struggling with the complexities of the security threat landscape, in particular, just looking for practical advice. With the increase in threats, with the changing landscape and visual transformation that's going on, people are really trying to understand, from experts, what could they do, practically, that would actually help them in this new threat landscape we're living in today, and I realized how fortunate I am to have met with some of the sharpest minds on this topic. Whether it's competitors, vendors, internal Microsoft people, government people, who all share a vision for a mission on how to better protect ourselves. This created an opportunity to take some of those learnings and share them in this podcast series. Hopefully you'll find this interesting, I know I'll learn a lot from it.

Bret Arsenault: I'd like to introduce you to Mark Russinovich, Chief Technology Officer and technical fellow from Microsoft Azure, Microsoft's global cloud platform. I've had the pleasure of working side by side with this gentleman for the past 12 years, and our paths have crossed for almost 22 years. Mark, talk to me a little bit about what you do in the Azure team?

Mark Russinovich: So, my role is Leading Techno-Strategy in Architecture for the Azure platform, which takes me all over the place, from our data center designs to our servers, infrastructures, software, platform as a service, cloud native computing, Azure resource manager, that's the bulk of my time is spent working with engineers on that. But I also do a lot of stuff like working with customers, internally and externally, on what they need out of Azure Platform and making sure it meets their needs.

Bret Arsenault: Well, yes, and then, if that's not enough, you occasionally write a few books here and there, fiction and non-fiction books both, right?

Mark Russinovich: Yeah, less of that these days than in the past.

Bret Arsenault: I'm sure you're super busy. I think in addition though, I think, from crossing paths, obviously we're working on some hard problems, like we worked on Saw and high risk environments and some of the enclaves that we've built, but you're not just a technology leader, you've been a great person for us on security, and not just at Microsoft, but around the industry, and so I'd really like to focus a little bit on that today if that's alright with you?

Mark Russinovich: Sure, yes, very passionate about security and always have been.

Bret Arsenault: Yeah, I know, it's great, we've had a few wonderful late night conversations on the topic, and I think in a context perspective, telling people here, I think conversations, for me, at this company to start with and then the Earth called, and I came here and we didn't even have a routable protocol, and I was working on getting TCPIP as our new standard protocol, and I remember this little place that had created this NT internals, and I need an inert 5C chaining sequence, I couldn't get the right answer to, so, I'm looking for it, and I end up using some of your tools to help solve a lot of problems. So, one, I owe you a great debt of gratitude all the way back into the mid-90s, so, I appreciate that.

Mark Russinovich: Sure, well I'll take you up on that vino.

Bret Arsenault: That sounds like a great deal. Just to set expectations for the audience, and sort of from a data perspective, your first book was Zero Day, and I always liked the concept of Zero Day, but, sadly, when we looked even last year, over 60% of the breaches that we saw were from unpatched vulns, like vulns that were known that had patches that people didn't do, and the other 40% weren't even zero data, they were other things. I think that's an interesting backdrop. I'd love to hear your perspective on what you've learned, you know, we go through a lot of incident reviews, and just some of your perspective about what we've learned recently?

Mark Russinovich: I mean, well, you've hit on it, the fact is that bread and butter security is the place to start, and if you manage the basics you'll cover a tremendous amount of ground. And the fact is that most breaches are failures to just cover the basics.

Bret Arsenault: Yes, I think that's true, and I think it's the pedestrian part of the job, like, I love all the amazing AI and everything else, but we do have to help people do more in that space, and, you know, what do you think we can do? And what would you suggest people do to make sure they can get to that work and, and prioritize that work?

Mark Russinovich: Well, when you say what we can do, I think what you're speaking what Microsoft can do.

Bret Arsenault: Absolutely.

Mark Russinovich: And a big part of it is education, which I think we do a lot of all over Microsoft, documentation and our conferences talking about security best practices. And even in our field, working directly with customers, on getting them up to a good security base lines, we also build security into our tools, and really, fundamentally, it's like in Azure, close to my turf, Azure's security center with the secure score is a way to help people focus on the basics, because you're gonna raise your secure score by covering the basics. And then the other part is just the support in the products for the security basics, including automated patch management, which we support in Azure, in fact, with Azure virtual machines, automated patch management, with MFA support built into Azure actor directory, the authenticator app. So, just from that perspective it kind of spans from education to giving you insights into how you can improve your security posture, whether it's using our products or somebody else's products, as well as ensuring that our products are a way for you to be able to meet those baselines.

Bret Arsenault: I think you've raised a good point. I think when you first came here, I think it was in 06, I was a CTO for E-Commerce, and we were patching all of our own servers, we were patching all of our exchange servers, and like, today, all of that's dealt with by the various cloud providers. So, I think there's a question about, you know, if you look at the evolution even over the last ten years, there's people that get convergence, do we think about and how do you think about cloud security as a first choice, as opposed to it wasn't the first choice ten years ago. Maybe some of the examples there.

Mark Russinovich: Well, that's been an interesting evolution, because when I started in Azure there was a bunch of concerns around moving to cloud, a big one was obviously unknown and lack of skills and, well, how's it going to transform how my organization works in a basic way, but then it was like security was considered a big risk, if I move to the cloud am I going to be less secure. And then, starting about four or five years ago, as the cloud capabilities got more mature, people started to change to, actually, moving to the cloud's going to get me into a better security posture. And I think that that is hard to argue the case that that's not true today, especially with all of the capabilities now that have come online for you to be able to create a deployment in a cloud that is completely isolated off the internet that has security monitoring automatically built into it, and alerting, and the insights that you get out of security center, like I mentioned, that it's become easier than ever.

I mean, in fact, one of the things that I pointed out to customers, and I've always believed that would make the cloud a better place for security is the fact that the cloud is built on consistent APIs across all resources, and when you take a look at an on prem environment, the way that they've evolved pretty organically, they're extremely heterogeneous, with respect to the servers, the network typologies and the policies that are in place, and the software that's running in different areas, and my challenge to CISOs, coming from an on-prem world, was if I walked into your colo and pointed at a server would you be able to tell me what that server is doing and what it's running, and would you be able to recreate it if something happened to it? And in many cases the answer would be no, no, no, or I don't know.

But, in the cloud, being API driven, every resource you can go and get an inventory of, and because of the automation that's in place with modern cloud development it's easier to rebuild something if it fails, in fact, that's the core philosophy. But, even if you're lefting and shifting and you've got those enterprise workloads that are bespoke, still, you know what its talking to on your network using standard cloud APIs, you know what other resources it's talking to. So, I think that the argument has changed a lot in the last ten years when it comes to cloud and security.

Bret Arsenault: Yes, it's interesting, and, for folks who don't know, I'm Mark's customer in this scenario, because we run all the services as well as the sort of tautological part of the conversation. But, if you remember, I think you were putting on the four to five years ago, as we were moving Microsoft Digital Transformation and all the tools, security was a blocker for a number of things, and then we got to where we had all the security capabilities and more that we needed, and yet, the migration stopped, I don't know if you remember when that happened? And we were like, okay, well, security's not the blocker, so what's happening, do you remember? I know you won't because I probably never shared it. FFUUEE, this is the FFUUEE, it's one of the most amazing things we learned when that happened, which is why, when we met all the security controllers, did the migration stop, and FFUUEE is the six reasons, FFUUEE, the six reasons why anybody, boss, subordinate, spouse, children, grandparents, don't do something, they fear it, they don't think it's fair, they don't understand it, there's not a sense of urgency, they're entitled or they're exhausted.

And so, when we saw that pitch, we thought people didn't understand, so we gave them more training internally, and, to your point, they were fearful of a job or, in many cases, they were just exhausted. And we had to take a completely different approach to now we're at 95% of all of our apps have moved onto the cloud. So, that's a really good point about the capabilities versus moving, you know, helping people continue to move to that life cycle.

Mark Russinovich: I don't remember that acronym, but I've written it down, because that's a great one.

Bret Arsenault: All due credit goes to Dick Butterfield, but it's super helpful, I use it at home, not always successfully. But I try. I'd be really curious though, because you mentioned a really good point about the API component of the cloud, and the resilience piece, right? Like all this stuff we used to have to with your off site provider, with dark fiber and spare computer, now in path services you just click a few buttons and you have this resilience capability. But that data, maybe some examples about how we really think about customers who really use that data to create that continuous feedback loop and continue to create the most security experience possible.

Mark Russinovich: Yes, well, I mean part of the premise of cloud has been agility, agility for customers because of the on demand self service nature of it, but it's also agility in terms of getting new capabilities, and the agility that we get from the continuous deployment systems that we have in place that are based off of the telemetry we get, the telemetry that comes from just the direct observability of how customers are using the product, which gives us insights into which features are tough to use, which ones are easy to use, which ones is nobody paying attention to, where they're having issues with performance or scalability, go right back into a feedback loop that helps us improve the product. And then those improvements can show up, I mean, you mentioned earlier, like the patching and exchange, I mean, on the back end of Office 365, new features roll out in Office 365 basically on a weekly cadence, and that is customer just get it.

I mean, you remember, as anybody running enterprise IT does, a new version of software comes out and it's a many months long process, or even years, of first validating it and then doing a pilot of it, and then starting to roll that out broadly across the infrastructure, that's been basically eliminated in the cloud. And so, the pace of innovation and the pace of improvements in areas like reliability, scalability and security, the basics, has accelerated to levels that make the old era look really slow moving.

Bret Arsenault: We started this reflective view on 2020, and why I do this podcasts in 2021, and obviously, with the pandemic, we had massive digital transformation around the world, and I obviously think about if we had been doing all the things we were doing on prem it would have been a very different outcome in terms of technology capability and what people needed to do. Just the scale unit problem would have been really devastating for companies, even our own company would have really struggled without that. 3

Mark Russinovich: Yeah, totally. Basically the services weren't to the level of maturity needed to support work from home learn from home. Just even between five and ten years ago. And then the scale of the systems underneath them weren't where they needed to be either to support it. I mean, if Covid had happened ten years ago, or worse, 20 years ago, multiply the problem that we've faced with Covid by a couple of orders of magnitude, like, the world just would not have been able to function.

Bret Arsenault: Yes, I know, I think it had a far more devastating impact both from a physical standpoint as well as on an economic standpoint, and I'm not belittling it, it had a huge impact, and many industries were more impacted than others, but I would say it has been amazing to see what's happened in that. Which pushes me to a question, you think about what's happened in 2020 in the end, people realize they can work remotely, how do you think about that now? Two things, one, hybrid workforce, right, so now people realize they can be productive in more remote locations, and the cloud's been a big part of that, the whole intelligent edge and client to cloud and all that has been super helpful, but, how do you, as a leading engineer, think about that, with your people? And then I'm going to come back and do the workforce issue in a minute, but I'd love to hear your thoughts on that?

Mark Russinovich: So I've got just, for me and my team in the office of the CTO, I've grown probably 50% since Covid lockdowns happened, so that means 50% of the people, many of them from outside the company, have not physically met anybody else on the team, and yet, have come in and gotten very productive very quickly to the level of productivity that's no different than pre-Covid times of somebody on boarding into a team like ours. So, it's just been really amazing to see the fact that from a perspective of remote work, nothing slowed down. Now, I think that there's going to be big challenges as we go to hybrid in that human nature is human nature, and the bandwidth of communication when you're in person with somebody, especially in a group meeting, right, one on ones is a little bit different because you can focus directly, but if you're in a meeting with a bunch of people and you're in a conference room, you can instantly get signals from all over the room.

Bret Arsenault: Right.

Mark Russinovich: But, if you're on a Zoom call or a teams call, those signals get lost, and you don't have the opportunity of walking into the conference room of, "Hey Bret, what's going on?" Kind of side conversations. So, the advantage of being in person is, I think, real, and it's gonna be tough to not make the people that are remote feel the way they did pre-Covid, which was, hey, everybody in the room is ignoring me.

Bret Arsenault: Right. On that note, talking about your team growing and all the other things, customers continue to tell me how they really struggle about finding qualified talent, finding security talent, and obviously, one thing is having your security team and mine work for customers is awesome, but, how do you think about that just in general? About getting talent and driving through talent in the hybrid work environment? You mentioned a little bit about the teams you've hired so far, but, any other thoughts you have on helping people and how they think about attracting talent?

Mark Russinovich: I mean, a few things that Microsoft's done to try to attract talent is just the culture that we've adopted, and, of course, driven by Satya, of diversity inclusion as I think a key part of that, that corporate culture I think matters a lot to people these days, whereas in the past it was something that wasn't really even explicitly considered as a draw for especially people coming out of college. But, when it comes to skilling, the world has changed a lot, like we've talked about, especially for anybody that's in IT over the last decade, a tremendous amount, and I think this is the challenge that I see, talking to customers that are doing cloud migrations, many of them doing it as fast as they can, and what's slowing them down is skills. And this is where platforms like Microsoft learn, which have guided learning paths for different areas of specialty, like, on cloud admin, or on the cloud security architect, to get skills.

I mean, this is part of Microsoft's broader goals to skill the workforce. I think Brad Smith just published, about a month ago, an update on our commitments for Acrom 2020, which was to skill 25 million people, and we'd actually gotten to 30 million, and there's a scaling platform being integrated on the Linked-In to help skill the modern IT workforce for this new landscape that everybody's having to face.

Bret Arsenault: I was thinking, you know, if you were to give advice to a security practitioner, some actionable things, skilling is obviously important, and we touched a little bit at the beginning, but are there three things you can tell people go do it and go do it now, what would it be?

Mark Russinovich: Basically the three things, I think, would be the two that we already talked about, one is MFA, actually, we've talked about all three, MFA, for sure, getting into a non-fishable posture, and MFA doesn't completely necessary completely get either, but it gets you past the stupid fishing. And then the second one would be patching, and this is a tough one, especially with on-prem environments and fragile IT that exists a lot, but, it is, like you said, you're forcing yourself into this "We need to patch," and have systems in place to do pilot tests of updates, kind of a CICD pipeline for patching, so that you can quickly roll that out, and every time there's patches it's not a unique kind of situation to deal with, but it's just part of a process, like we do in the cloud when we roll out new software, it's just part of a process, it's not a special once every three years kind of situation.

Bret Arsenault: Right.

Mark Russinovich: And then the final one is get visibility into your environment, with logging, and not just logging do Devnull, but actually logging into some place where you can actually run analytics on the logs to get inside into what's happening. Something like Sentinel really resonated with customers.

Bret Arsenault: Can you just remind people what Sentinel is?

Mark Russinovich: Yes, so Azure Sentinel is basically I think of it as the realization of a security data lake, it is a place where you can bring in and fuse data from all your different sources of security monitoring, whether it's your cloud services, including, of course, Azure and Office 365, and other Microsoft services, but also you can use leverage connectors, I think we've got 50 or 70 or something connectors, to bring in data from your own on premises systems and services, and put it in that lake and then get a whole ton of capabilities right off of that where now you're not siloed between data that's from one service with data from another service, but actually it's part of a lake where you can see activity and correlations across the different services, because in the modern world, threat actors are moving across your services, they're not standing in one.

Bret Arsenault: Yeah.

Mark Russinovich: Yeah, so I think it's really key to view it that way.

Bret Arsenault: Yeah, that's a great point. So you said MFA and obviously the patching component, and the pervasive telemetry I think is super not really possible the way we could do it today, like, the scaling that you can do that at, and into getting the accretive value of the disparate data, like a diverse workforce is really important, but, it turns out, diverse data is as important, and so being able to correlate across that is super helpful, it's a really good way to look at it. On that note though, if we look at the MFA adoption, it's still low, right?

Mark Russinovich: Yes.

Bret Arsenault: I'm curious, like, from your perspective, for people who are out there, how do we help people? What should they do and what can we do to help them continue to drive that up? Because there's awareness, someone said this to me once about training, they said, people don't care, and I said, well, if they don't know they can't care, so you have to make sure they know, like, Secure Score makes you know, right? But then you go from knowing to being able to do something about it. So, obviously we have a lot to do, we can help people really move that adoption up, do you have thoughts on how we move to an option of 2FA up? And not just for a Microsoft platform, other things?

Mark Russinovich: Yeah, have it be the default. Make it so that people have to opt out of it. Because, right now, I mean, I think even still, it's like opt into MFA rather than opt out. And when you have to opt into something it's like, okay, so, got to convince somebody, here's why you need to opt in, why it's better for you, and it's going to take a little more work, and then it's just easier just to not bother, and to look at that as friction. But, opting out, at least that's in your face as, hey, this is the best practice, and if you want to opt out of it you're kind of signing up for that risk, explicitly.

Bret Arsenault: Yeah, and that's a good thing to drive that awareness on what people are doing in that space. One other thing, I was just curious, and this is, you know, as we prognosticate going forward, we've seen the way threats and the way they evolve to be faster blended on software, and we're seeing supply chain, obviously, in the last year having a similar model, and then we think hard about a lot of the things that we do for users and end points, there's a lot we're going to probably see changing in the way we think about develop our pipeline. You mentioned on the get up stuff, there's some pretty amazing things we can do, but any thoughts on, as we have citizen developer and other folks, how we help them fall into the pit of success of secure coding?

Mark Russinovich: Well, I think that there's a few things, and I've been involved across Microsoft's initiatives, that have even gone out in to become cross-industry initiatives to help secure the supply chain, where developers are a key part of that supply chain. And developers are sitting there producing code, which is consumed downstream by some enterprises and cloud providers, and they're also consuming things from upstream too, to build their software. So, this is an interconnected graph of dependencies that flow from some dev that might be sitting in part time on the weekends contributing to some project, into critical infrastructure way downstream. So, I think that there's a few things that we're doing there, one is education, and this is part of the cross-industry stuff that we started last year, which is the formation of the Open Source Security Foundation, OSSF, that's part of the Linux Foundation. Though it's called Open Source Security Foundation, it applies to close or software development as well. So it just happens to be called that because the focus is on the open source ecosystem, which so many companies, including Microsoft, have deep dependencies on.

But, the education, so if you go to OSSF, there's education security best practices aimed at open source developers, the people that are contributing to GitHub projects on the weekend that end up becoming critical infrastructure. There is also efforts on standardizing software build materials, and I think that this is an area to watch, especially given solar winds, solar agate focus that this has brought onto it, and there's been many incidents related to supply chain and bad actors infiltrating software supply chain over time, is having some record of providence, and ultimately reproducible build evidence, so that you know how did this thing get produced, did it get produced in a trustworthy way? Did the people that contributed code to it, did they have multi-factor authentication enabled, which gives us some assurance that they're following security best practices.

Bret Arsenault: From a healthy device.

Mark Russinovich: From a healthy device, yes, and from a healthy device. So, all of these things I think that this is what you're going to see a lot of investment across the industry over the next five to ten years to get us into a place where we have better supply chain tracking, and, like I said, it's not jut for Open Source Security, but closed source as well.

Bret Arsenault: Yes, and I think, to your point, I think people should go and look at that, because, you know, many of the people who might think are our competitors, we all have a view of rising tide lifts all boats and are contributing to that model, so, I think you get the best of both worlds in that scenario for sure.

Mark Russinovich: Yes.

Bret Arsenault: I love your thoughts on virtualization and the evolution of virtualization and what it can do for us. As we started thinking about end point, virtualization and more like Windows virtualization or the Citrix has come into its own, and this idea of that model of if you're going to have something that you trust as an end point you might put your trusted end point in the cloud.

Mark Russinovich: Yes, we're getting to the point where the network computer idea from the 90s, that actually, the infrastructure is there now to support that idea. So you've been leading pilots on getting some of the Microsoft workforce onto virtual desktops, built onto Windows Virtual Desktop service, it's running on Azure, but we've also been talking about in the context even of envelopers, enveloper work stations as a service. I mean, all the benefits you can get out of that from the fact that you can have a provision device instantly to it's not jut a provision but it's with everything you need, but it's constantly upgraded as well, accessible from anywhere in the world at th same time.

So, there's a lot of benefits from a security, reliability, productivity perspective, and, finally, the infrastructure in terms of the cloud services to support the scalability of it as well as the internet connections now that we've got, to be able to access it basically from anywhere, have made it so that its time is finally showing up here.

Bret Arsenault: Yes, it'll be fascinating to see how it plays, and that goes back to culture, you think about your beginnings of some of the stuff we talked about on Win32, and just being so on-prem focused, to now we're virtualizing the cloud, virtualizing on end point, and you're right, but you needed the global infrastructure to make it as secure and reliable as we'll need it to be for people to be productive, so, it'll be a pretty fascinating future. Any closing comments, Mark? It's obviously so awesome to have you on board, and I appreciate everything you do, and for the company and our customers, and me as your customer, but any other comments you have?

Mark Russinovich: Closing thoughts is, this is really a fun time to be in technology, this is what I tell people as they're coming into Microsoft as well, enterprise IT was actually pretty much defined before I even showed up on the scene, it was a matter of incremental improvement to the existing architecture and systems and the way of delivering software, and what cloud brought was a big disruption that we're still not done defining what that disruption is, what it looks like. And so, like a service like Azure Sentinel, which just showed up two years ago, and it still hasn't realized, I think the full potential of the vision that we have for it. If you come into technology now you're a part of this disruption, you're helping to define what the future's going to look like. I don't know if at some point the weight of all of the systems that we're putting in place ends up becoming like a foundation that's in cement, kind of the way that Enterprise systems ended up becoming, but now's the time to have fun defining it.

Bret Arsenault: I think you're spot on. Hey, a fun question, any new hobbies in the last year? Or any chance you're writing another book? You can't answer best read book as being any of the ones you've written. So anything you have read that you would say has been super-exciting and you're recommend people read?

Mark Russinovich: Let's see, well, new hobbies, I drew a lot when I was growing up, and I didn't draw for 20 years, and Covid got me revisiting my drawing again, in fact, if you go to my Twitter feed I posted some of the drawings that I did over lockdown.

Bret Arsenault: Well, from a guy who can't draw a straight line with a rule I'm impressed. So, I think that's awesome. Well, thanks for your time.

Mark Russinovich: Alright, thanks, Bret.

Mark Russinovich: Bye-bye.

Bret Arsenault: Thanks for listening, I look forward to our next episode. Remember, stay safe and stay secure.