Software supply chain management: Lessons learned from SolarWinds.

Rick Howard: Hey, everyone. Welcome to "CyberWire-X," a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the chief security officer of N2K and the chief analyst and senior fellow at the CyberWire. And today's episode is called "Software Supply Chain Management: Lessons Learned from SolarWinds." A program note - each "CyberWire-X" special features two segments. In the first part, we'll hear from an industry expert on the topic at hand. And in the second part, we'll hear from our show's sponsor for their point of view. After a word from our sponsor, some of our regular subject matter experts will visit me at the CyberWire's Hash Table to tell us how they think about software supply chain risk. Come right back.

Rick Howard: Rick Doten is the CISO for Healthcare Enterprises and Centene, a regular guest at the CyberWire Hash Table and an old friend of mine. And we were talking about the fact that software supply chain attacks aren't new. The first one that everybody remembers is the 2013 hacks against Target. But since then, in 2017, we've had NotPetya, SolarWinds in 2020, we got the Log4j stuff in 2021 and 2022 and still ongoing. And every time we get one of these big attacks, you hear pundits say, well, this is the wake-up call. Everything is going to change now. But that never happens, at least that's my sense. I asked Rick if he felt the same way.

Rick Doten: Yeah, I thought you were going to say, when we see those things, we're going to be like, yep, not surprised. 

Rick Howard: (Laughter). 

Rick Doten: It's like, I don't know why everyone's upset because, like, how did you not see this coming? 

Rick Howard: Yeah. 

Rick Doten: But it should be an opportunity to be able to say, see? Let's not be that. But it's really short lived. And, you know, we've struggled with third-party risk management for decades. I mean, I remember, as a consultant in the early 2000s, going out to companies and, you know, walking around data centers and call centers and print shops, you know, that, you know, were - you know, where banks mostly, or other companies, were sending information that was shared to go out to their customers. And so - but we keep asking the same questions, but we're not really jigging down into the root of where these came from, which is not checking for just the presence and appropriateness of controls, but the effectiveness of them. 

Rick Howard: Steve Winterfeld is the Akamai Advisory CISO, another regular member of the CyberWire Hash Table and just happens to be my best friend. He said that in order to cover third-party risk management, you should focus on two areas. Here's Steve. 

Steve Winterfeld: So, Rick, I agree. You know, supply chain is a really weird threat. We don't systemically have an approach across the cybersecurity industry. And it's one - kind of, like, insider threat, you know? There's a major incident with insider threat or supply chain every year or two, and we really put a lot of effort into it in the short term. But systemically, it's not a high enough risk that we're doing long-term and systemic investment. When we do think about how to solve it, for me, there's two approaches. There's vendor management, and then there's technical controls. 

Steve Winterfeld: On the vendor management, it comes down to the kind of contract you write. What are the technical controls that they're required to operate? What audit rights do you have? If they have a pen test or they have a third party come in and do an audit, are you allowed to see the results, and are you included in the remediation plan? And finally, what are your notification rights? If they start to do an investigation and determine that they're breached and then make an announcement, at which one of those stages are you notified? Ideally, you want to be notified before the public is notified, and you want to be part of the investigation so you know if there's any chance that there's actually penetration of your network, as well. 

Steve Winterfeld: Then on the technical side, for me, it's all about situational awareness, that visibility. A great example nowadays is, you know, that microsegmentation, a software agent-based thing that's allowing you to see data flows. And now I can say, why has this application got a steady communication outside my network, or why is this application touching multiple other servers in my network? So I can see those anomalies. And this might be, you know, not only a tool like that, but there are teams. I want my threat intelligence team involved. I want my threat hunting teams involved, looking for those odd behaviors for both insider threat, supply chain, anything that's in my network and inherently trusted, with odd or suspicious behaviors. And I say my environment - in this multi-environment, you know, we have now, I've got stuff in a data center. I've got stuff in maybe two or three different cloud providers. I may have stuff in SAS providers or third-party applications and all of this - you know, how do you get a common risk portfolio across all of that, I think, is a big challenge. I want to go back to the original question. Because of these big software supply chain hacks of the last decade, is anybody managing this any differently these days? 

Rick Doten: I don't think so because, it's like you said, it's - to get to that level of granularity to find those things, that - you know, where those examples come from - you know, it'd take, you know, real scrutiny to say, how does this process actually work? You know, the SolarWinds is my favorite one to talk about because when we look at - oh, SolarWinds got bought by an investment firm, they decided to move some of the development to, you know, to Eastern Europe. You know, a lot more Eastern Europeans closer to Russia are doing the development. 

Rick Doten: Of course there's going to be - you know, it's like - of - you know, even if you just said that to me, I'm like, oh, yeah, then there's going to be some coercion and people are going to get some - try to sneak some code in because SolarWinds is everywhere. And it's also one of those - just, you know, easy to ignore 'cause security people really don't use it. The network people love it. It's SNMP-based. It's like whatever, but it's always there, and it's just quietly, like, sitting there collecting stuff. So it was the perfect compromise and the perfect way of doing it that you're - you know, like I said, when I saw it, I'm like, of course. We didn't see that? It's on us for not having somebody scrutinize it enough to notice that. 

Rick Howard: Well, I agree. And then - and even in - let's just talk about Log4j for a second. I read a news item this morning that said the - a third of the instances that are deployed on the internet for Log4j are the old version that was compromised, you know, a year ago, right? A third, right? So... 

Rick Doten: Yeah. I mean, configuration management's still very hard for a lot of organizations. You know, I think I've said before that, you know, in the last few years of doing all these CISO roundtables, I'm really feeling this big gap between the haves and have-nots. And you have Fortune 500 companies that have people and budget and things, and you have 600,000 other companies who don't. 

Rick Howard: Yeah. 

Rick Doten: And so we have just, you know, this little sliver of companies who can do these things and others are like, I don't even know how to update it. You know, and I look at that as being responsible for my subsidiaries who are not connected and are smaller, that they don't have security people, they have IT people, they're focused on the business. You know, and so even when we, as the parent company, say, hey, have you fixed all this stuff? They're like, where is it? I don't even know what to do with that. 

Rick Howard: Well, you're right about that. And, you know, on my own podcast here at the CyberWire, "CSO Perspectives," I've been spending the last two years talking about first principles. And what I've come to the conclusion is that for most of them, most organizations don't have a way to do them. If you're talking about zero trust or intrusion kill chain prevention, that's mostly in the box for midsize to Fortune 500 companies. 'Cause, you know, I work at the Cyberwire. It's a startup. I don't have a staff to do those things. So for the small companies, they have to do something different, like resilience, right? Just survive it and keep delivering their servers. So it's definitely different for how big you are and how you deal with these supply chain attacks. You talk to a lot of CISOs. You're in so many forums and things that I can't keep up with you. What are they talking about? The things that - is zero trust coming up in those conversations? 

Rick Doten: Yeah, I mean, it comes up just because it's a very popular term. And, you know, and we all roll our eyes, when the first person who brings it up. And, you know, because it's different for everybody. You know, it's not a thing, it's not a product. And there's no, like, we are - there's no finite way that's - you're finished with it. It's just an ever - it's a journey, not a destination. And it's also - if you look at - all the tenants do it. It's something that we should have been doing for 20 years. And it's like, what do you mean? You're not doing that already? 

Rick Doten: So - but I think that we come back to the basics. You know, back to the CIS critical security controls and just, like, let's go 1-5 - asset management, configuration management, vulnerability management, patch management. You know, I mean, just get the basic out of the way, and you'll take care of a lot of these things. And then we lay, on top of that, application security, which is what this is about. So, you know, obviously I've been saying more about, like, third-party risk management in general. But I know the topic being, like, application security supply chain, you know, there is a - you know, we take that gap of maturity and then apply it to software, and it gets even bigger - of people who actually understand software. You know, unless you're a small company, that that's what you do is create software. 

Rick Doten: And you know, back in the day when I ran pen test teams 20 years ago, we would just do code reviews or we would do - you know, because usually, it's libraries or segments or pieces, not, like, whole applications. We do pen tests on whole applications. So how does somebody put that in? And I think that the cloud is actually giving us opportunity because in the CI/CD pipeline, in its - you know, this application development that is made up of so many different pieces, most of it being third-party libraries and other borrowed thing or shared services and APIs. We're starting to get back to it. I've been really thrilled to see that the industry is now coming back to threat modeling, you know, that we've kind of ignored for 15 years. And then now cloud is making it viable again. And so I think that the tools are there. It's just kind of getting the awareness back out to say, we need to look at all the pieces of it, not just the stuff that's in the check boxes and some of the frameworks. 

Rick Howard: So with your interactions with all these CISOs, zero trust gets a hand wave. What about SBOMs? Anybody thinking, oh, we got to do SBOMs to fix this? Is anyone jumping on that bandwagon? 

Rick Doten: Yes, that one is because there's kind of a mandate for it... 

(LAUGHTER) 

Rick Doten: ...You know? So, yes. But fortunately, the tools are better now - are good enough now, to where they can, like, create it. It's not like, you know, some poor developer has to sit there and document everything. You know, the tools now can look at it and create them and then even run vulnerability management against - vulnerability assessments against them. So that's been very good. 

Rick Howard: Dawn Cappelli is the director of the OT CERT at Dragos, a regular guest here at the CyberWire Hash Table and also author of the Cybersecurity Canon Hall of Fame book, "The CERT Guide to Insider Threats: How to Prevent, Detect and Respond to Information Technology Crimes." I asked her if she thought the infosec community was making progress with their SBOM deployments, and her answer completely surprised me. She said that because of a set of security standards developed by the industrial process sector years ago, called IEC 62443, that mandated SBOMs, industrial organizations are way ahead of all the other sectors, which is unusual because, typically, those organizations lag behind the other sectors when it comes to defensive architecture. Here's Dawn. 

Dawn Cappelli: I'm a big proponent of SBOMs. I think that industrial organizations are way ahead of the rest of the community because, in the industrial control system arena, IEC 62443 is the security certification that the OEMs have been marching to for many, many years. And it takes many, many years to get IEC 62443 certifications. One of the things that's required from 62443 is a secure development lifecycle. And part of that secure development lifecycle requires software bill of materials. Since companies like Rockwell Automation and Schneider and Siemens and Honeywell have had these certifications for years, this means that they have SBOMs for their products, at least their products that are certified. So, like, at Rockwell Automation, where I was CISO previously, part of our SDLC required an SBOM. So before you could release your product, you had to submit the software bill of materials for that version of the product. Now, years ago, that was basically Excel spreadsheets. So here's my source code and here is my SBOM in this Excel spreadsheet. Then, as time went on, technology improved so that now there are tools that will create your SBOM for you. 

Dawn Cappelli: So when Log4j hit, for example, it was discovered and made public, disclosed in the middle of the night, on Friday morning. By Sunday night, Rockwell Automation was the first vendor in the industrial control system arena that disclosed all of our Log4j vulnerabilities in our products. So by Sunday night, we had the complete list of all of our products that had the Log4j in it, and we never had to update that list. So it was 100% accurate when we put it out on Sunday night. And that was because we had those SBOMs going back many, many years. So I'm a big proponent. And by the way, as you get higher and higher maturity level certifications in IEC 62443, it is required that you also impose the SBOM requirement on your software suppliers. So again, in the industrial control system arena, this is one place where SBOMs are much more mature, not only for the OEMs themselves, but also for their software supply chain. 

Rick Howard: The obvious strategy to me to reduce the risk of software supply chain risk is zero trust. Unfortunately, because of the way security vendors glom on to popular ideas, many marketing claims insist that their product solves zero trust for their customers, even when, at best, whatever their product does is a mere piece of the zero trust equation, so much so that CISOs and other security practitioners just roll their eyes whenever a vendor makes another claim. And as a community, we tend to throw the baby out with the bath water. Here's Rick. 

Rick Doten: And I'll go back to, like, mature CISOs eyeroll at zero trust. You know, CISOs, who - you know, have been successful in using that as getting budget, you know, because it's made the mainstream media people saying - and their executives - like, hey, what's this zero trust thing? Are we doing that? You know, it's like, did we buy that yet? So they are getting traction with it. So as much as I eyeroll to it, it is effective. And if you're not doing it, then it's the right path to take. 

Rick Howard: Well, I'm going to push back on that. We're just in the Gartner trough of disillusionment with the idea of... 

Rick Doten: Yes. 

Rick Howard: ...A zero trust, but the idea of it's right - limit access, that's all it is. 

Rick Doten: Right. It took, like, five different things... 

Rick Howard: Yeah. 

Rick Doten: ...To identity, to resources, to data, to systems to, you know, just, you know, blah, blah, blah. Yes, of course. That's how you're supposed to do it. If you haven't been doing that and you're too mature, then, you know, you can label it and then get money because it's now got a label. 

Rick Howard: So let's get back to what you said before. I want to bring you back to this because it seems like that's the most practical thing to do, is really a formalized, third-party risk management process. 

Rick Doten: This will be my third semester. I've had friends who teach master's courses and we do capstone projects on third-party risk management, to have the students kind of develop what a program would be. And so it's interesting, kind of, what they come up with. But - and foundationally, it's kind of like you identify all of the people who you share data systems or rely on for systems, you know, from a resiliency standpoint. And then you prioritize them based on impact to the business. If something goes wrong, whatever those things may go wrong, you kind of identify those, whether it's a stalling of data or a lack of that resource being available. 

Rick Doten: And then you kind of define what is it - do I need to know about them to make me feel comfortable that they're protecting it to the level I need to be protected, whether it's, again, resilience standpoint, protection or privacy, et cetera? Then, we come back to you, like, the old-school questionnaire thing. You know, or there might be, you know, yes, there's things like security scorecard - BitSight, they're doing this real-time. passive scanning - you know, those are, you know, kind of like a credit score thing, which doesn't really mean a lot. But some people use it 'cause it's, you know, at least it's some piece of - a piece of the puzzle. 

Rick Doten: But then sometimes it might take, you know, being able to say, all right, you're so important to me. I'm going to send somebody to come sit and talk with you. And I did that 20 years ago. I'm like, just make sure you're cool. You know, do you have someone responsible? Do you have - but what's been interesting in this particular space is the insurance companies because the insurance companies are kind of stepping in, since they've been losing their shirts the last couple years. And they were getting very, very granular with what they're asking for. I have peers of mine saying, oh, my gosh, I had, like, 15 questions about how I manage my service accounts. Or another one is saying they were really drilling into me about how my backups are and whether it's connected and offsite and how I'm managing the passwords to backups. 

Rick Doten: You know, so you can tell which people are getting - you know, having payouts for things like, OK, your service account's misused or backups not - blah, blah, blah - you know, from a ransomware standpoint. And I have a feeling that that - you know, and I don't know if it's the - a good thing or not, that that might be our standard, is, like, hey, did you go through the insurance process, and which one was it? And do they think you're cool? Because they're actually starting to get some real metrics of, like, what's - you know, what's eating their lunch. 

Rick Howard: So that's, like, a shorthand. So you either do the third-party investigation yourself or, hey, do you have cyber insurance? And if it's through Aetna, let's say, oh, that's good enough. I don't have to do anything else. Is that what you're saying? 

Rick Doten: Right. You know, you might get - you know, you can - you know, depending on how big you are. 'Cause also, like, it depends on size. Like, I'm not going to be able to get Microsoft or Google or Amazon to, like, give me their - I mean, they'll have, on their website, their SAS 70 - I mean, I'm sorry. What year is this? Anyway... 

Rick Howard: (Laughter). 

Rick Doten: Their SOC 2 and their ISO certification and you know... 

Rick Howard: (Laughter) That's Rick going through a senior moment, everybody. OK? 

Rick Doten: Yeah. 

(LAUGHTER) 

Rick Doten: And so they have those. And they say, listen, you're going to accept these because we don't - you know, we're bigger than all of you. So just trust us. We're doing more than you are. But if you're smaller, you can request those things. And going back to the insurance is, like, we will often request insurance, but some of them can't afford it for the reasons that I said. I mean, one thing that all my friends have said is, like, their insurance premiums have gone up like 300%, and they're getting where 2 1/2 years ago, it was a two-page, like, hey, do you suck or not? Answer these five questions, and that's it - to now, you know, you, pretty much, are going through, you know, an IRS audit, you know, of, like, how are you doing this thing? And I want evidence of that and et cetera, et cetera. 

Rick Doten: So, you know, I don't think people are necessarily using them yet. You know, so we - you know, so it might require also, for instance, we're in health care. So if there's someone who's sharing, you know, PHI with, then there is a business associate agreement, which has all the contractual stuff and everything and requirements of what they're supposed to do, that they - that we, you know, we'll negotiate whether, like, well, we don't really use the old PHI passwords and you're out. Is that good enough? You know, it's - you know. But we won't say you can't - you're not - you - we won't accept, like, they don't use MFA, as example. 

Rick Howard: So is there a takeaway here, Rick? Is there a one-liner that says if you're going to do anything to mitigate the risk - software supply chain risk - is there one thing you could focus people on and say, do this and you're in the right ballpark? 

Rick Doten: Well, certainly the - you know, examining any code you get from somebody. If you're getting libraries and other things or if you're having third party develop software for you in your software development, then, you know, as we said for 20 years, you know, scrutinize that code that you get and validate that it's effective. If we're talking about the examples you said, which are, you know, full COTS products that are, like, OK, well, does this thing have some backdoor into it? - then you might want to look at the providence and, you know, kind of see, is there anything changing in the relationship, you know, such as the SolarWinds example. You know, or, you know, even in the - even going back, you know, 15 years now - has it been 15 years since Target? You know, we've - we just talked about that like it's yesterday 'cause everybody's... 

Rick Howard: It feels like it. 

Rick Doten: ...Like, you know, the Target breach. And I'm like, what year is it? 

Rick Howard: Yeah, yeah. There's youngsters out there that don't even know what the Target breach is, right? So... 

Rick Doten: So that - it's kind of like, again, when I saw that, I'm like, well, of course the HVAC had access to it. Why wouldn't they? Because then no one would have thought to have not given them a service account with these access, you know - the access controls, as an entity. 

Rick Howard: That's why I think that there's a zero trust for Solar - for - if you're a SolarWinds customer, zero trust is your best bet because you shouldn't be allowing the SolarWinds platform inside your network to have access to anything besides the things that it needs access to, right? And that's what those bad guys did. They moved laterally once they were inside, right? So that's why I like... 

Rick Doten: Yep. so I would agree with that. And, yeah, so I agree with that concept of, you know, isolate it, you know, to least privilege, to only those things that it needs to do, that should do, whether it's talking internally or externally, you know? It should only be able to talk out to - if it needs to talk out to anything, then it only talks out to the thing that it needs to talk to and nothing else. And, you know, we - it's kind of a whole security architecture approach to it. So, yeah, so I guess it's the - the thing I guess I will say about zero trust is it's kind of like the - expect that everything is compromised... 

Rick Howard: Exactly. 

Rick Doten: Which I don't think that - you know, unfortunately, a lot of my peer group just expect all the technology to work. I was in a - I was at... 

Rick Howard: (Laughter). 

Rick Doten: ...An event one time and we were talking about ransomware. And one of my CISO peers said, like, oh, don't worry (inaudible) - I don't worry about ransomware. I have a good EDR. 

Rick Howard: OK. 

Rick Doten: And I'm like (laughter). And I'm like, wait... 

Rick Howard: Yeah. 

Rick Doten: ...You're serious. 

Rick Howard: Yeah, OK. 

Rick Doten: You know? So I think that there is a mentality of, hey, I bought all these tools and they work 100% of the time, so I'm OK. But if you're prudent, then you realize, like, OK, what if it fails? What's my way of catching it? I think it was one of your peers. It was one of my West Point friends, you know, had said - it's like - what - I asked him, you know, like, 20 years ago - I was like, what did they teach you in War 101? And he was like, well, if you have something of value, you put up a barrier and you monitor that barrier with firepower (ph) because at some point - minutes, hours, days, weeks or years - it's going to be breached and you got to be ready for it to happen. 

Rick Howard: What I love about bringing subject matter experts to the CyberWire Hash Table is that we definitely get a wide number of viewpoints about what is and what isn't working in the InfoSec community today. And speaking of the Hash Table, next up is my conversation with Tim Brown, the CISO for SolarWinds - yes, that SolarWinds. Tim Brown is currently the CISO for SolarWinds. He joined the company in 2017 as the VP of security about three years before the SolarWinds attacks. After the incident, he got the CISO title. But he's had a long 30-year career wearing many hats. He's been an IT and security engineer, a strategy guy, a security architect, a cloud security CTO, a journalist. He's even been a Dell fellow. I started out by asking him, what was the thing that attracted him to InfoSec? 

Tim Brown: Yeah. Security is always changing, right? And it's always something new. And, you know, kind of I love the challenge of something new happening every day. And, you know, I sometimes get more than I ask for. But I really do like the challenge of everything, everything new and changing. It's never boring, right? We have plenty to do. 

Rick Howard: I love that, too. It's a reason I love this field. But you know what I hate about it? It changes all the time. 

Tim Brown: (Laughter) It is kind of a love-hate relationship. 

Rick Howard: So on December 12, 2020, FireEye announced to the world that it had been compromised, but what has become the poster child for supply chain attacks and the nightmare for all CISOs - responding to a breach in our internal networks. So, Tim, can you - in broad terms, just can you describe what the hackers behind the Nobelium attack campaign did? What was their process? 

Tim Brown: Yeah, absolutely. You're one of the few people that had known the term Nobelium. So that was what Microsoft called the attackers, Nobelium. We also know them as Russian SVR. We also know them as APT29. So a number of different names for them. So, yeah. So FireEye ended up discovering that SolarWinds had shipped (inaudible) panic code and contacted us on December 12. December 13, everything went public and, you know, culminated with us doing a, you know, issuing a notice of 10-Q, a 10-K on early Monday morning. 

Tim Brown: So the attack was, you know, just as we said, right? We ended up - the threat actor essentially compromised email first, did reconnaissance for a good period of time, over a year of doing reconnaissance, then essentially did a test run in October where they inserted no code, came back in February and inserted 2,500 lines of code, which was really Sunburst code. That stayed in releases from March to June. And then they left. So attack against us was very specific, very targeted, very mission centric, very quiet. It wasn't like they were in our environment, making noise all the time. Their whole motto was, how do we do it stealthily? And then the code that they dropped did things that made it hard to detect, right? So it wouldn't run inside of SolarWinds domains, wouldn't run inside of Microsoft domains, wouldn't run in a number of domains, waited 14 days before it started. So very thoughtful - right? - from an attack perspective. 

Tim Brown: You know, I don't say it's innovative. I don't think it's like machine learning type stuff, but very, very thoughtful in the way they attacked us and others. So supply chain, the first time we called it a supply chain attack was that first day because it didn't affect - attack our source code system where they would have been discovered. What they attacked was one of our transient virtual machines that's part of our build process. That's where the insertion was. We couldn't find the source code anywhere, wasn't in any of our systems. So it happened in our supply chain. What came to be known very quickly, though, is where SolarWinds participated in the larger supply chains around the world. And where we sat at our customers became very important. And that's what really sparked the big supply chain conversations now, simply because we were in the middle of supply chains, and people didn't even know we were there. So having them discover that. So as I'd say, you know, I had my Christmas ruined, but a lot of IT folks around the world, you know, also had their Christmas ruined trying to figure out, hey, is SolarWinds here? Where is it? What's it doing? What versions are we on? So quite a journey. 

Rick Howard: Well, I agree with you. I - you know, it wasn't like we'd never heard of supply chain attacks before. They've been in the public sphere many times. But this seemed to be prominent notice because of how successful SolarWinds is, all the customers. 

Tim Brown: Yeah, And it's a good thing, right? When you look at some good that comes out of it, it's the discussions around supply chain. So discussions of us, you know, how do we get vendors more resilient, but also, how do we figure out what's in the supply chain at our customers? And so those conversations starting as, you know, one of the kind of the good things that has come out of it, legislation starting to come out of it, executive orders coming out of it. So the progress is, yeah, we hate to see something bad happen, but it's good that we see something good come out of it. 

Rick Howard: So let's talk about that because you guys came out with some new strategy, all right, to handle security going forward. You call it secure by design, right? And you've come up with a number of tactics to implement it at SolarWinds. It's triple build software development, zero trust, red teams and security awareness training. Let's start with triple build software development process. What is that? 

Tim Brown: You know, one of the most important things we had to do was, yeah, re-instill confidence in our customer base, right? Part of doing that is saying, hey, we're not just as good as everybody. We're better, right? We took six months off from development of new features. And the full engineering staff focused on the secure by design efforts. So part of those secure by design efforts were build processes. So with 12 products essentially making up Orion, we had a multiple repos - right? - for information. We consolidated down to one repo. That was part of stage one. Did a two-way build to start with. So we go from source code to product. We then take that product that's built, we install the product, we decompile it and get back to source code. So we can do source code checks against the decompiled code - C#. 

Tim Brown: So by being able to do that, we get assurance that source code matched what was in the release, right? So second part of it was to take our build environment and move to AWS and make it all ephemeral. So nothing exists - everything's in code, so nothing to attack. Third part was multiple build pipelines, and multiple build pipelines needed to be deterministic, right? And you know that, you know, if you build something one time, you build it again in a minute later that you're not deterministic anymore because time changed. So we were able to make certain languages deterministic - Java, C#. Having them deterministic means that I could do multiple builds and then compare them at the end and only ship when those builds are compared. Why that's important is because my production build has two people having access to it. My staging build has about 30. My development build probably in the hundreds, right? 

Tim Brown: So by comparing them, I end up saying, well, you would need collusion among multiple people. So those two people that are in charge of production don't have access to dev or staging. So in order to affect builds now, you'd have to have collusion among multiple people. Really taking an assumed breach process across the board and saying, OK, if Tim Brown has been breached, what would happen now? This - our instrument didn't show insider but very well could be possible in the future, possible for somebody else. So it's important to think about that when you're doing builds and everywhere along your line, whether it's your SRE team, whether it's you build team, you know, how do you take that assumed breach process and then add protections over the top? 

Rick Howard: So the triple build software development process is really a protection against the insider threat to protect against outsiders inserting code like what happened in your incident. 

Tim Brown: So it would protect against that because they inserted into the build process. So GitHub did not match what the product was that was produced. So that insertion would get checked, too. We posted things on GitHub, so we now - we always have peer reviews. Now we have peer reviews plus architecture reviews. So additional reviews on anything that's checked in, additional red teaming of the environments as well. 

Rick Howard: So the typical software development process protects against insiders and protects against outside breach infecting your source code because it's going to check the beginning stages of the code against what happens at the end and make sure they're the same. 

Tim Brown: Correct. So much more resilient from any build process. And we wrote white papers on it. And we've done things like that to try to bring it to the community. So it's been in place really since July, August of '21. So that just adds resilience. It adds assumed breach, which says a single individual won't be able to affect the build, take collusion. Those types of things just really hardens that whole process. 

Rick Howard: So let's talk about one of the other tactics - and it comes to mind immediately after you say all that - your red team tactic to go after this process, I'm assuming, right? You're sending your red teams to break that process. So how does that work? 

Tim Brown: We had a part-time red team before, right? Post-incident, we have a full-time red team. Now, the first object of my red team was both infrastructure testing, but then every component that is associated with the build system. All right. So I trust certain things that they probably do well, right? Salesforce is fine. Palo Alto is fine, you know, as you know, right? They're fine. But what about my configuration of them? Right? Is my configuration where it needs to be? So my red team's focus, can I take advantage of the configuration that you have in place to circumvent security controls? 

Tim Brown: So as opposed to always looking for domain admin, these guys are looking to say, OK, you know, when I look at my firewall, all right, who has access? What's change control look like? How are my checks and balances there? What's my configuration there? GitHub, same idea. How many admins do I have? What if I take over one admin? Can he affect something else? So do I have protections in place from that? So my - on the SaaS services, it's often looking at, you know, OK, I've got a well-configured SaaS servicer, well-configured product. And is that configuration resilient to attack? So we've done that for the components of essentially the build system and always looking to tighten things down a little bit, right? So red team hasn't gotten through, but they've tightened stuff down. They said, OK, if this was here, we could have done this. So it's a double track. So the red team's very important to us. Now we're starting on mission and business critical assets on those applications. And we'll spin back around and redo essentially build system. 

Rick Howard: So is the red team - in this new strategy, Tim, is that purely for the software development process or do you turn them loose on the network configuration also? 

Tim Brown: Yeah. I turn them loose on the network. I've turned them loose on the infrastructure as well. 

Rick Howard: Are they dipping their toe into adversary emulation? Are they trying to duplicate what Nobelium did? 

Tim Brown: Yeah. In some ways that, the eastern Nobelium guys were coming into our environment absolutely, right? Secondary attacks, not so much, right? Secondary attacks were very specific. Orion had to be public facing. You had actual command and control server. You had to be able to be somebody that folks cared about. So that's one of the reasons why we saw our number of truly affected customers. Now, everybody was affected by looking and saying, am I running one of the versions which is tainted? But when we looked at the number of affected version, it's actually not 18,000 which downloaded a piece of software. It's under a hundred that went to a second stage. Used to say nine agencies, report came out and said four agencies went to a secondary target. So had to be very specific. So we don't necessarily test that scenario, but we do test for scenarios of them potentially entering our environment, making sure that the environment itself is as resilient to attack from the outside. The other benefit of the red team that touched my (inaudible), tests my backend systems and my recovery and my playbooks. All of those things get tested. 

Rick Howard: So the third tactic you guys have launched in your secure by design strategy is zero trust, something on everybody's mind these days. 

Tim Brown: So on a journey, right? I think zero trust is a journey, right? So part of our journey is to move the authentication authorization out to the edges, right? 

Rick Howard: In a software-defined perimeter kind of idea or... 

Tim Brown: You consider that we've got so much, so many cloud properties, so many things that we've been moving towards from a cloud perspective. Office 365, Salesforce, you know, you name it, are moving to essentially the cloud-based solution. So the more centralized we get authentication authorization, that's been a big push to use Azure SSO conditional access, now YubiKey for authentication authorization for, you know, all admins, all dev folks and moving YubiKey out to the world just because of the MFA overload that we've seen, you know, that's got people concerned - right? - that, you know, are you going to get them? 

Tim Brown: I'm worried about YubiKeys too, because we use them here at the CyberWire. But, you know, if you're like me, I'm going to lose that YubiKey. 

Tim Brown: Right. So if you just keep it in your - with the little ones, we keep it in the machine. It works pretty well. 

Rick Howard: Everybody I talk to on this show, Tim, is talking about how they're doing zero trust. You're a long way down that journey. Is there any one lesson learned that you wish you knew before you started that would have helped you in this, that you can part to all of our listeners here? 

Tim Brown: Yeah. I think just, you know, definitely start, right? Start your process, right? And get from one app to two apps to three apps to four apps. I think we're up to - I can't remember what my numbers are, but it's in the - above 50, right? So, but start, right? Start down the process because the benefit of - the benefits of having kind of central auth, the benefits of central auditing, the benefits of having central control are worth their weight in gold. So, you know, start the process. Don't be afraid that you have to have it all right or, you know, all done because, you know, get to 50%. Get to 20%. It's better than you were yesterday. 

Rick Howard: Yeah. Like you said at the top, it's a journey, not a - we're not going to get there - we're not going to get to the (inaudible) any time soon. Yeah. So the last tactic in your secure by design process is security awareness training. And it's a little different than most people think about it, I think because of the software pipeline that we've been talking about. So what's involved in that part of that tactic? 

Tim Brown: Yeah. Really, when you look at kind of culture of security - right? - how do we keep them by being a culture of security across the company? So some of those are having additional places to communicate security topics to people. So you get your annual training. That's normal. So we'll do newsletters. We'll do topical things out to people. We'll also do - we've been doing a - phishing campaigns pretty consistently for, I think, the last six, seven quarters, right? And when somebody clicks on something, they get remedial training. 

Rick Howard: So we're getting towards the end of this, Tim. Any last words for the audience about your experience after the incident? And what can you recommend for everybody? 

Tim Brown: Yeah. So what worked for us is really making a focus on our customers, right? Now it's so our key focus. It was, how do we understand and help the customers get through it? And it took time. It was hard, took a lot of effort. But the customers responded, right? We helped them move forward. We helped them get into the right place. And then be exemplary. So where can you be exemplary? Where can you do things that are not just OK but really exemplary in process? 

Rick Howard: We'd like to thank our interview guest, Tim Brown, the CISO for SolarWinds, and Rick Doten, the CISO for health care enterprises in Centene. And special thanks to Dawn Cappelli from Dragos and Steve Winterfield from Akamai for helping us get some clarity about where the InfoSec community stands with software supply chain management issues. And finally, we'd like to thank SolarWinds for sponsoring the show. "CyberWire-X" is a production of the CyberWire and is proudly produced in Maryland at the startup studios of DataTribe, where they are co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Eiben. Our executive editor is Peter Kilpe. And I'm Rick Howard. Thanks for listening.