Pt 2 – Supply chains around the Hash Table.

Rick Howard: As you all know, the entire infosec community responded over the 2021 holiday break to the Log4j vulnerability. In this year, 2022, I know all of us will be reviewing how we did during that crisis and how we can improve in the future.

Rick Howard: Errol Weiss, the Health-ISAC CSO and regular here at the Hash Table, reached out to me about his initial thoughts on how his organization did during the crisis. 

Errol Weiss: Like you, we were all hoping for a quiet end of the year, but it became pretty clear by December 9 that we were going to be dealing with a widespread critical remote exploitable issue. At Health-ISAC, we published the first version of our Log4j vulnerability bulletin on Friday, December 10, including crowdsourced executive-level communications. We also sent out a survey to our members to assess the impact to the sector. The response also included hosted member calls or something we call a fireside chat for lots of different reasons. 

Errol Weiss: So what did we do well and how can we improve for the next one of these? I thought sharing communications that I saw happening between the ISACs was pretty good during the Log4j incident, but it can be improved. So I challenged my fellow ISAC leaders to work together to improve our intra-ISAC sharing and join in on some of the pilots that we're literally testing today. It's all about helping our respective members and protecting the critical infrastructure. 

Rick Howard: Like Errol said, most of us plan to be off for the holiday break. What's that old Yiddish proverb? We plan, God laughs. Exactly. I love the life of 24-by-7 security operations, but our response to that incident highlights not just the seriousness of this particular vulnerability but also reminds us all that we should be thinking hard right now about this digital supply chain attack vector. 

Rick Howard: Last week, I covered the history of supply chains from the physical world all the way to the digital world, the zero-trust strategy that we should use to reduce the risk and a potential tactic, SBOMs or software bill of materials, that's promising but at least five to 10 years away from being useful. 

Rick Howard: But as you all know, it's not good for me to be by myself in my own thought bubbles. I needed to bounce some of those crazy ideas off of somebody way smarter than me. So I reserved the CyberWire conference room, where we store the Hash Table, and invited a brand-new guest. 

Rick Howard: My name is Rick Howard, broadcasting from the CyberWire's Secret Sanctum Sanctorum Studios, located underwater somewhere along the Patapsco River near Baltimore Harbor. 

Rick Howard: One side note - one of our alert CyberWire Pro listeners, Peter Nicolaides (ph), called me out on mispronouncing Secret Sanctum Sanctorum Studios in the last episode. Apparently, I added the letter I to Sanctorum, saying instead, Sanctorium (ph). 

(SOUNDBITE OF ARCHIVED RECORDING) 

Unidentified Person: Oh, no. 

Rick Howard: Good catch, Peter. The Marvel Cinematic Universe police has just suspended my nerd card for the next 90 days, preventing me from binge-watching "Agent Carter" on the Disney Channel, and rightfully so. I'm so ashamed. 

Rick Howard: And just in case you all forgot what you're listening to, this is "CSO Perspectives," my podcast about the ideas, strategies and technologies that senior security executives wrestle with on a daily basis. 

Rick Howard: On this show, I'm joined by Amanda Fennell. 

Amanda Fennell: Fennell. It's Fennell, yeah. 

Rick Howard: The CIO and CSO for Relativity... 

Amanda Fennell: Yay. 

Rick Howard: ...A company that makes SaaS products to aid their customers in monitoring and managing their legal and compliance obligations. Amanda, thanks for coming on the show. 

Amanda Fennell: Thanks for having me. You sound better at what Relativity does than I do. 

Rick Howard: (Laughter) That's - my job's done here, then. I've managed to meet my expectations. 

Rick Howard: Before we get started, though, I would be remiss if we didn't mention that your podcast, called "Security Sandbox," is officially joining the CyberWire Network this week. That's fantastic news, and welcome to the CyberWire family. 

Amanda Fennell: Thank you so much. It's been a long journey through Season 1 - learned a lot. So really excited to join. 

Rick Howard: I'm a relatively new podcast person, too, so I know what you're talking about. Boy, I listen to some of my earlier shows and go, ooh, OK (laughter). I totally get it. 

Amanda Fennell: Yup. 

Rick Howard: So - but tell me about - what is "Security Sandbox" about? 

Amanda Fennell: It's a - I think I was listening to - you know, during COVID, took a lot of walks and stuff and running. And I just was listening to different podcasts. And when I would listen to a lot of security ones, there's a lot of the same material... 

Rick Howard: Yup. 

Amanda Fennell: ...'Cause I think when we've been in the field for 20 years, we kind of all think the same, pretty similarly. 

Rick Howard: (Laughter) It's true. 

Amanda Fennell: And, you know, we know that - we know what's good. We know what's bad. But it's where diversity can come in and actually be really helpful. So I started thinking, why don't we use all the things that we're oddly passionate about externally and bring it into security? 

Amanda Fennell: And so Season 1 was about a lot of different curious areas, like archaeology, coffee, neurosurgery, whatever - didn't matter - astrology - didn't matter - but how you could take some of the cool things from those different areas and bring them into security and make us stronger. And we found that while that was super cool and unique, it really was the people side that was the most intriguing. And that really kicks us off into Season 2 and where we're going to focus there. 

Rick Howard: Well, I'm a huge believer in, you know, getting out of our comfort zones. Most of us read as much as we can, but I find that if you just only read infosec stuff, you really are limiting yourself. So you need to get out and read some other things and maybe jump-start how we might solve some of these harder problems that we're going to talk about today. 

Amanda Fennell: Yeah. 

Rick Howard: So I'm very excited to have you guys on board. That's fantastic. 

Amanda Fennell: Thanks. 

Rick Howard: So this week we're talking about strategies and tactics to secure the digital supply chain, both for commercial vendors, like companies that got into the spotlight last year with some high-profile breaches - SolarWinds and Accellion, to name two; they've been in the news a lot, right? - and, you know, from open-source libraries, like the Log4j vulnerability that we all had to deal with over the holiday break. 

Rick Howard: So let's start there, Amanda. What did you do as the CIO and CSO of Relativity? How did you guys respond to the Log4j crisis last year? 

Amanda Fennell: Yeah, it's a similar process between that, as well as what it was for SolarWinds. It's the same thing that everybody has to always ask first is, does this affect us, No. 1? 

Rick Howard: Right. 

Amanda Fennell: And so the only way to know that is if you have a good handle on your assets and your data flows and things like that and your software. So once you've had the work done ahead of time to make sure you weren't exposed - that's kind of like the choose-your-own-adventure best option; we don't have any exposure or risk - then you have to immediately say, but do anybody that we work with have that, like, large, you know, perspective situations and customers and peers and things like that. 

Amanda Fennell: So we're pretty active in the threat intelligence community. And so we got to just make sure we're helping in some way. So we try not to - candidly, it didn't affect us. And so I hate to say we just sat back on our laurels 'cause we didn't. We definitely leaned forward and said, OK, well, let me help you with the threat intelligence we have for people who were exposed. 

Amanda Fennell: So step one - do we have any risk that we have to worry about or exposure or exploitation? Step two - increased monitoring takes place, put in signatures and so on and everything - which you only get because you have access to really good threat intel normally. 

Rick Howard: Right. 

Amanda Fennell: And then step three is making sure you're doing something to help, even if it's not your backyard. 

Rick Howard: Well, I've talked to a lot of CSOs about this. Seems like the first couple weeks after hearing about this everybody was scrambling just to find out if they were exposed, if they were using that software component in anything that they had developed or if any of the vendors that they, you know, purchased stuff from - if they were using any of that. So did you find talking to your peers that people had that information at hand, or were we all scrambling, trying to figure out if it was - applied to us? 

Amanda Fennell: You know, I'd hate to judge based upon the median time to resolution on that one, like how long it took when you made a request versus how long you got the answer back. 

Amanda Fennell: But I think it's something to be said that, you know, the government - we're FedRAMP in our government environment and stuff. And the government sends out these things and says the FedRAMP ATO - hey, were you exposed with this vulnerability? We require something in writing back in 48 hours or something. I think that's a pretty good SLA to keep in mind - that you really should have some expectation of being able to tell people, were you or any of the suppliers you use exposed with this absolutely horrific thing? You should know within 48 hours. 

Amanda Fennell: There was a lot of scrambling. It is difficult. I don't think a lot of people were prepared for it, even after going through it with SolarWinds. But those are the best tabletops - OK, no exposure, no risk. How did we do great this time, and how do we make sure we do better next time when the next one hits, which it will? 

Rick Howard: Most of us use software from a handful of commercial vendors, everything from operating systems like Windows and macOS to business software like Microsoft Office and Google Workspace to the specialty software like SolarWinds. And they all do software updates over the internet. And it isn't like we've never seen the supply chain attack vector before. You know, bad guys... 

Amanda Fennell: Target. Target. Target, right? 

Rick Howard: Exactly. The Target breach of 2014, when the bad guys compromised Target's HVAC contractors' network and used legitimate credentials stolen there to get into the Target network proper. And let's not forget NotPetya. In 2017, those bad guys installed backdoor into the software produced by a European accounting company called M.E.Doc. 

Rick Howard: But it wasn't till just recently that I would say that most of us have developed a robust strategy to deal with this kind of thing. Like you were talking about, we didn't have this information at hand. We had to kind of scramble for it. 

Rick Howard: After we've gone through all these things, the SolarWinds and the Log4j stuff, how do you frame it now after we've been through it all? How do you tell your team how to think about it? And what are you telling your boss about this? 

Amanda Fennell: Well, you know, this is where I think having done some education in this area and having gotten a master's in high-tech crime rears its ugly head. And it is risk equal to probability times impact. And so we constantly are focusing and training, making sure that we understand, is there a probability - how high, how low and so on - versus the impact it might have. 

Amanda Fennell: And so while we might have, like, one area that could be exposed, if there's no customer data, there's no personal data, there's no, you know, my own employees' data and so on, you know, this impact goes down, down, down, down. All of a sudden, we find out it's a virtual machine we only use for a test environment. Cool. That is a very low impact, has no potential for lateral movement. Yada, yada, yada. 

Amanda Fennell: So we talk through these things, but it's really on the role of the security people to be the teachers for the rest of the executives, as well as the company, about how we measure risk - so probability times impact; got to make sure that everyone follows that one - and that you also have buy-in before something bad happens. 

Amanda Fennell: And that was really the biggest news for a lot of people. You know that everyone loves to say they never let a good incident go to waste, right? 

Rick Howard: Yeah. 

Amanda Fennell: We've got maybe five to 20 days to get budget after something hits, so keep your wish list ready. I like to go in ahead of time. You know, my Christmas list is ahead of time for the next three years. Like, if something really bad happens, we're going to need this. So I think it's just thinking - normally in security, we just think about, like, a year out because it's such a changing threat landscape. It does help just to push to three to five years for your strategy and where you're going down the line because I think that's the only way we're going to get ahead of a lot of our adversaries who are becoming so prolific with these attacks in the digital supply chain. 

Amanda Fennell: So long story, big answer. Probability, you know, times impact is risk. It's our job to teach people that. It's also our job to constantly be aware of what that probability is. 

Rick Howard: Well, I totally agree with that, and we've been pursuing that idea here on this show for the last couple years - the idea that not everything is material to the business. All of us, we don't have unlimited resources to spend on everything. So focus on the things that have high impact and can really affect the business and move on. So I totally agree with that. 

Rick Howard: One of the ideas that's been kicking around the industry, though, for the past decade are these SBOMs, these software bill of materials. They're kind of like - you know this, but they're kind of like food labels for commercial software that we're running in any of the open-source code libraries that we're using in-house. And if we had - all had some version of SBOMs in our environments when the Log4j vulnerability popped up, there wouldn't have been a mad scramble to see if we were affected. The SBOM would tell us if we were running Log4j anywhere. 

Rick Howard: So the U.S. government is pushing for SBOM capability sometime in the spring of 2022, but most of us are nowhere near that capability. And I'm guessing that in the best case, it's at least five to 10 years out for anything being universally useful. Have you seen anybody using SBOMs in any kind of productive way in their environments? 

Amanda Fennell: Wow. I mean, I feel like that's a setup to say productive. No, not productive. 

Rick Howard: (Laughter). 

Amanda Fennell: And I think it's because, you know, the - look. Asset management - this is what we already try to do, even in a digital arena. So we need to know, where are we supposed to protect? For anyone in security, the first thing you do is try to figure out how many egresses, what am I doing, how many doors do I have to, you know, whatever? The first thing we try to do is put our hands around and figure out how big is the bread box. 

Amanda Fennell: And these SBOMs, the bills of material - the problem with them is just changes every month. It's just a monthly basis. Your subscriptions to something might go up, might go down, might not be used anymore. Software's being, you know, taken out for whatever reason. It's just it changes all the time. And I don't know that everyone's - anyone's done a great job of having that planned ahead of time. I know I'm probably setting someone up to be a vendor to do this, by the way. Before we know it, there's going to be... 

Rick Howard: I think that could be our next adventure. 

Amanda Fennell: ...Startup (ph). 

Rick Howard: We can be a SBOM, you know, vendor (laughter). 

Amanda Fennell: I can't wait for all of the ways I could use this one. But, yeah, it definitely - it seems like it's an empty space. And we saw different movements over time, you know, like GRC, right? Everyone knew they had to have some kind of way to gather evidence to prove they were doing a control. So a lot of different companies came out as a software that got in front of it and said, well, we can catalog all that for you. We can keep you updated. We can automate it. I think we are looking at a space right now that doesn't have a lot of maturity. And so it just - it changes every month. And a lot of people aren't ready for something like that. 

Rick Howard: Yeah, I was looking through this. There's a whole space, and it's kind of a corner that no one talks about. And it's vendors who actually scan open-source software to look for vulnerabilities. And it's been a backwater. No one has really been there that much. 

Rick Howard: But with President Biden mandating that the U.S. government will demand SBOMs for the vendors they're using, it may have a chance to speed this up a little bit. But like I said, it's five to 10 years away. And it has to be automated. It can't be people reading letters and spreadsheets and things. It has to be automatically updated or just is not going to be useful at all. So... 

Amanda Fennell: Yeah, but they don't tend to wait for that. They tend to force the mandate first and make us do everything... 

Rick Howard: Yeah. 

Amanda Fennell: ...Manually. And then eventually there'll be a tool we can buy for it. But in the end, I feel like the best digital supply chain defense is going to remain that close monitoring of your release channel. Ensure that content's not being modified by someone. The scanning that you're referring to and keeping in front of that - I feel like it's about being healthy preventative, but it's going to be a while until we've got this one automated. 

Rick Howard: Well, let's talk about that 'cause I think that's the only viable option here - right? - some sort of zero-trust strategy. And I know we were talking about before we started recording that zero trust has a bad name for some reason. Everybody hates the idea of zero trust. But the idea of zero trust is limiting access to applications and people and devices to only the things that they need and nothing else. So if you can apply that idea to software, I think that's our best strategy. Or what do you think? 

Amanda Fennell: Yeah, it's - this is what you and I refer to offline, but I think we agree with this one. It's interesting security that something can seem not interesting for a couple of years and everyone shies away from the word. Let's just say disruption. That was a thing, like, 10 years ago, right? 

Rick Howard: Right. 

Amanda Fennell: Everybody used the word disruption. Nobody in security liked it. Then five years ago, it became a thing. Everybody loved it. And then five years after that, everybody hates it because it's overutilized. OK, another word that's overutilized - OK? - world peace is overutilized. 

Rick Howard: (Laughter). 

Amanda Fennell: It's lost its value because people use it so much. 

Rick Howard: Very true. 

Amanda Fennell: So my imploration is do not let zero trust lose the value of what it's trying to perpetuate. And let's not get stuck in the semantics of the word choice because I saw something on a post today on LinkedIn this morning. And it was like, it's not zero trust. It's explicit trust. Like, sure. OK. Whatever. Roll (ph) these privileges. OK. 

Rick Howard: I could - I responded to that one. Yeah, I saw that same post today. 

Amanda Fennell: Did you? 

Rick Howard: Yeah. 

Amanda Fennell: I was like, OK, sure. And I didn't bother to engage 'cause I was like, sometimes you don't feel like somebody's going to hear what you have to say, so why bother? But I will say, yes, zero trust is a thing. You can call it blank trust - whatever you want to call it - implicit trust, explicit. I don't really care. But the point is that your org has got to do something to reduce that increased surface risk area that you've created with the hybrid work environment. You have to do something. 

Amanda Fennell: And this arena is where you reduce that application or web application risk surface. There's no silver bullet, but this is our way of saying, hey, we know that our space we have to protect is this big, but we just found a way to limit it so now it's only this space we have to worry about. And for everyone who can't see, I'm using my hands to make big circle and a small circle. So... 

Rick Howard: (Laughter) Well, but you look at our bag of tricks, you know, how we can reduce risk in our own environments, nothing else applies here. Pick your list of things you like to do. The only thing that has any hope of having success here is some sort of zero-trust strategy, I think. 

Amanda Fennell: I think there's that, but, you know, not letting go of all of the other parts that we think are helpful... 

Rick Howard: Oh, true. 

Amanda Fennell: ...Human beings not being the weakest link, extending that we have constant training - those micro trainings, reminders, testing and so on - pen testing, vulnerability scanning. All those things, like, are so much part of - you know, just like a human being, we have so many aspects of us holistically to keep us healthy. So does our security program. It's a holistic perspective, and it's all of those things moving together that makes something really strong. And getting caught up on something like the semantics of wording, it will lose its power. It's not about that. It's about the intent of trying to make things more secure. 

Rick Howard: Well, it's all good stuff, Amanda, and unfortunately, we're going to have to leave it there. So for all of our listeners, that's Amanda Fennell, the CIO and CSO of Relativity. And she's also the newest member of the CyberWire's Hash Table crew. And her podcast, the "Security Sandbox," is the latest show to join the CyberWire network. And like I said at the top of the show, Amanda, welcome to the family. 

Amanda Fennell: Thanks. 

Rick Howard: And that's a wrap. Next week, the CyberWire is celebrating George Washington's birthday here in the States. In other words, we have a three-day holiday, so we won't have a show next week. But the week after, I'm breaking out the cyber sand table again, this time to talk about the Chinese government's compromise of the U.S. Office of Personnel Management, or OPM. You don't want to miss that. 

Rick Howard: And as always, if you agree or disagree with anything I have said or have any suggestions about what you would like us to cover on the show, hit me up on LinkedIn or Twitter, and we can continue the conversation there. Or if you prefer email, drop a line to csop@thecyberwire.com. That's csop@thecyberwire.com. We would love to hear from you. 

Rick Howard: The CyberWire's "CSO Perspectives" is edited by John Petrik and executive produced by Peter Kilpe. Our theme song is by Blue Dot Sessions, remixed by the insanely talented Elliott Peltzman, who also does the show's mixing, sound design and original score. And I'm Rick Howard, in the penalty box, not binge-watching "Agent Carter" for the next 90 days. Thanks for listening.