What Log4Shell has taught us.
Rick Howard: Hi, everyone. And welcome to "CyberWire-X," a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the chief security officer, chief analyst and senior fellow at the CyberWire. And today's episode is titled What Log4Shell Has Taught Us. Over the holiday break in 2021, the entire infosec community reacted to an international incident response around the newly announced Log4j zero-day exploit. And if 2021 taught us anything, it's that our supply chain, especially our technical supply chain, hangs in the balance on a very fragile system. In this episode, my colleague David Bittner and I invited two guests to the CyberWire Hash Table - Tom Quinn, the CISO at T. Rowe Price, and Ted Driggs, the director of product management at ExtraHop, to discuss what the Log4j vulnerability tells us about the state of cyber defense going into 2022 and what enterprises can do to prepare.
Rick Howard: A program note - each "CyberWire-X" special features two segments. In the first part of the show, we will hear from industry experts on the topic at hand. And in the second part, we will hear from our show's sponsor for their point of view. And since I brought it up, here's a word from today's sponsor, ExtraHop.
Rick Howard: To start things off, I'm joined by Tom Quinn, the CISO of T. Rowe Price and a regular here at the CyberWire Hash Table. Hey, Tom. Thanks for coming on the show.
Tom Quinn: Great to have you invite me again. Thank you.
Rick Howard: We love having you on, Tom. So today we're talking about the industry's great crisis over the 2021 holiday break, our collective reaction to the Log4j vulnerability and any lessons learned that we can use for future problems of this sort that are going to come up to us down the line. Now, the Log4j vulnerability is a form of a digital supply chain attack, and these third-party attack vectors seem to be having a moment right now. We had some pretty high-profile attacks last year like SolarWinds and Acellion. And then we got hit by this open-source software library vulnerability, Log4j, at the end of the year. Can you walk us through how and when your organization first heard about the problem and what you did to reduce the risk?
Tom Quinn: So we have a variety of threat intelligence capabilities at the firm, and we received notice that this Log4j vulnerability was something that required attention. Our threat intelligence team, as well as our vulnerability management team, assembled to evaluate the issue based on our own risk plans. And then we recognized that it was something that we needed to do. And then we engaged our technology operations teams for T. Rowe Price.
Tom Quinn: Our technology operations teams manage and operate the vulnerability management execution capability at the firm. We focus on the risk, and then we work with them to execute. So they did their part of this, which was verify and validate the inventories that we have in place. And we have a variety of inventories of software, both commercial, open source and others. Our inventory was verified. And then we went about the work of testing and then certainly deploying the capability. You mentioned something about risk mitigation. I think it's important. The timing of the year when this occurred - right? - was right around the holidays. And one of my jobs at the firm is to make these risk calls where we actually had to call back people from their vacations.
Rick Howard: Oh, that's horrible. I know. Everybody was doing it, though. It was horrible.
Tom Quinn: And you have to. I also think it's fortunate that we had some remote access practice over the past two years with COVID, being full-time remote. So people quickly assembled, thankfully, onto the remote access capabilities that we had and then did the work that they needed to do and then in very short order, you know, were working on the things that needed to be done. The other side of this - and this is also part of the threat intel and part of the vulnerability management side of the program that my team does as well - is now you've got to start measuring. You need to measure to make sure and validate that the inventories are correct. Validate and measure - right? - that the fix that was deployed actually had the intended effect.
Tom Quinn: So my team was on this rinse-and-repeat cycle of validating the work that needed to be done. And believe it or not - right? - we even noticed situations where conflicting work was occurring, where a patch was deployed, it was tested, things seemed to be fine. And then we ran a scan day or two later and noticed that what was accomplished was actually undone. So it's critical to check and verify.
Rick Howard: Well, let's unpack some of that stuff because you threw a lot of detail out there. So first, I want to know, Tom, was there a moment that you and your team decided the problem went from, oh, this is serious to, oh, my God, this is serious? Was there one of those kind of wake-up moments for you?
Tom Quinn: There was. And, like, you know, we evaluate significant vulnerabilities on a regular basis, right? You know, that's the nature of the work that we do. But I think as we started to unpack Log4j and some of the details for it, you know, we recognized pretty quickly the ubiquitous nature of it. And I had mentioned before, you know, about open-source inventories that we maintain at the firm for software that we have and commercial inventories and applications that we build. As the data started coming in, we recognized how impactful it was to us, not just the industry. And then I think the Part B to that, which was - we realized it was like, oh, my, this is a big deal when we started recognizing and realizing that many of our colleagues, many of my peers, our vendors had the same issues that had to be done. This really expanded this into a much bigger potential.
Rick Howard: You mentioned that you had on hand already developed libraries of all the software you were using. Walk me through that. Is that something you guys built in-house? Is it in spreadsheets or using some software to do that? Or how are you guys managing your software inventory, both commercial and open-source stuff?
Tom Quinn: There's a multifold effort that we have. So for - I'll call them traditional operating systems, we have a view on the inventory that's there. And we have a very robust and well-known third-party tool that helps us keep the inventory. In addition, we have vulnerability scanners that acts as a check in the balance to that, a reconciliation capability. So that that keeps the - I'll call it traditional operating system inventories well-understood.
Tom Quinn: And then for our development community, we have a build train capability where you not only store your code and the associated libraries that are there. And by the way, we also do static and dynamic testing within this build train that we have. Some people may call that the CI/CD pipeline. Others may call it something else.
Tom Quinn: But we also have a gate capability for libraries coming in. So there is a product that we have and a capability that we have so that we limit where you can get open-source libraries and other libraries that we use for so that licensing reviews can be done, version control, version management can be done and a whole host of, you know, administrative and management actions can be taken. That really accounts for, you know, the vast majority of the inventories that we have. And, again, the scanning technologies that we have, whether vulnerability scanning or just scanning in general, fill in some of the blanks.
Tom Quinn: I'll also point out that the other way that we do reconciliation for inventories is we actually look at, you know, our network logs that may be generated from our firewall telemetry points, our intrusion detection prevention and other kinds of telemetry points that we get because when we see scans occur and we see traffic occurring to destinations that we didn't expect to have Log4j there, it's another indicator that something may be going on. So we triangulate between those. And, again, it's a couple of enterprise-grade capabilities for inventory and reconciliation and then, I think, just lots of good judgment being applied as well. Look in the corners of our network for other things that may be going on.
Rick Howard: So the Log4j software is a bit of open-source software written for the Linux distribution. When you were checking your inventories, was it as simple as - can we just search for this component in the library? Was it as simple as that for the tools that you were developing? Or was it more complicated than that?
Tom Quinn: So that's certainly one of the things to do. The other thing was to actually look at some of the code bases that we had and look for those calls, you know, when, you know, Log4j was actually being called as a function. So there's a couple of places that we did that with. And even from a network traffic perspective that I mentioned, too - not the inventory side, Rick. And I know you asked about inventories, but, again, doing triangulation and looking at other areas where you can make inference that Log4j may be there because, you know, again, you may see indicators in network traffic - right? - that that's going on as well.
Rick Howard: There's two problems here, right? One is code that, you know, T. Rowe Price is writing themselves. And they may have downloaded and used the Log4j component, right? And so you should be able to look it up in your inventory for that. But for the network traffic, that's for all the other software that you're running, both commercial and open source, that is running that module also that you didn't know about, that you weren't really paying attention to. So that's how you would discover it.
Tom Quinn: It's certainly one of the ways. And I think the other thing that we relied on - right? - was our vendors or certainly our significant vendors to be doing the same thing. There were bulletins that were published by significant software vendors saying, you know, listen. You know, we have a Log4j dependency. This is what you need to do. Here's the version that you need to download, because in some cases, what a company may do with an open-source library - it's not as easy as, even if we wanted to, like, to pluck it out...
Rick Howard: Yeah.
Tom Quinn: ...And put our version in because you could damage the product - right? - the appliance or whatever else it may be. So you're right. In some cases - right? - our inventory may have seen something. But in some cases, we needed to wait for the vendors as well to come back so. It's critical to make sure that you have very robust inventories because that speeds up the cycle for action, the cycle for knowledge - right? - that you need to do. And that's key - knowing that a few things are going to be missed. And that trade-off is something that we recognize and why we invest so heavily in the asset inventories that we have so that we really can make quicker decisions. And it's less for us to be looking around for - right? - in other corners of the network.
Rick Howard: In the early days of the crisis, you know, we were all saying, oh, my God, the world's coming to an end. But pretty quickly, the community discovered that if they just blocked X data coming from Log4j traffic going out to the internet that you could reduce the risk almost to nothing. Is that what you guys decided to do, too?
Tom Quinn: Yeah, we took the same approach, you know, one doing the inventory work - and you have to. You just have to maintain the code base, your software base and your inventories at a high level and then look at mitigating, you know, controls, as well. And we did. So there was a variety of rules that we incorporated, and there were some rule modifications that we also made based on the traffic analysis that we were seeing.
Rick Howard: Oh, that you found on your own. Oh, I get it, yeah.
Tom Quinn: Yeah, so - because what we found in some cases was some of the rules were imperfect. And that's another reason why it's critical to make sure that you've got your own capability to evaluate what's happening on in your network because believe it or not, the bad actors are smart, too.
Rick Howard: What? What? Tell me.
(LAUGHTER)
Tom Quinn: They recognize what the rules are that are being given out right to everyone else. And if they can modify in a small way and have a successful outcome because of that, you'll see that kind of thing occur. So I like the 80-20 rule in general, right? If we have high-fidelity, acid inventories, we can deal with the 20% looking around. And I think it's the same thing for these rule sets. If they can account for 80% or 90% or whatever it may be and be effective, it allows us to focus in on areas where they may be less effective for a whole variety of reasons.
Rick Howard: Well, you keep mentioning asset inventories. The idea of an asset inventory has been around for decades. I mean, even when we were young fellows doing this, Tom, we were talking about that. But that's not the same thing as kind of a newer idea that's getting some traction here recently, which is software bill of materials. Am I right that those two things are different? Or are they pretty much the same idea with different names?
Tom Quinn: I think the concepts are related, and I think that they're similar. But I liken it to in the old days when people used to say, hey, you should only have access to the minimum amount of stuff that you need. And now that's morphing into things, concepts like zero trust. So I think, Rick, you know, my view on this is it is similar. It's not the same. And it's the level of granularity, you know, I think that's being asked of things is different. And I also think it's a recognition in that the way that software is being consumed, being used, being manufactured and alike - it's not just in your data center. It's not just on your server, per se, right? There's a lot of dependencies - right? - supply chain dependencies. So I think that phrase makes sense. And I think what that phrase allows one to do is really to have a much more holistic view about what - you know, where your boundary of control thought needs to be so that you can have a view on the work that's needed to round that out.
Rick Howard: That's my understanding of it, too - is, like, where asset inventories are kind of big blocks of software - like you said, we know about Microsoft's operating system. We know about that Linux operating system. Where the SBOMs will help us down the line - because they're not here yet - but where they're going to help us is when big, commercial vendors used open-source software, and all those nested components that we don't see just on the normal day-to-day operations. We don't know that Microsoft used Log4j unless we see the traffic. So I think SBOMs will give us, like you said, that granularity.
Tom Quinn: And I think, too, in some cases, there may not be an understanding that a vendor is actually - has a dependency on some of these - call it small companies, maybe. Or it may be a big company. They're just pulling down components - right? - that they may need or they may think they may need to get something done. So - but anyway, that concept of bill of materials, I think, is important. The amount of rigor that's going to be needed around that I think is quite a bit. But the tools that are out there, I think, have improved, right? I won't talk about specific companies, but there are companies that are providing - I'll call it, dependency analysis capabilities - that are pretty darn mature, leveraging cloud compute capabilities, cloud storage capabilities, analytics and the like. The modernization of thought and the modernization of capability is allowing people to have much more view about what's going on. And I even think, too, for some of the cloud-native companies, they've built a lot of their systems around high-fidelity view of components that they're using. So I think the industry's ability to do that or the desire to do that is catching up with some of the engineering and architecture and technology improvements over the past 20 years.
Rick Howard: So make a prediction for me. How long before SBOMs are best practice and common practice for just general purpose network defenders?
Tom Quinn: I think ubiquitous capabilities of that place or expectations is a long way off. My guess is that, say, the next three to five years, you'll have certain parts of either the stack or certain applications may have very high-fidelity capabilities. I think for high-risk transactions and high-risk systems and high-assurance needs, I think you'll see that first. And then I think that will just become - that will trickle down. And then over time, I think you're going to see that in more and more - more and more in use.
Rick Howard: Good stuff, Tom. But we're going to have to leave it there. This has been Tom Quinn, the CISO for T. Rowe Price and subject matter expert here at the CyberWire Hash Table. Tom, once again, thanks for coming on the show.
Tom Quinn: You're welcome. Thank you for having me.
Rick Howard: Next up is Dave Bittner's conversation with Ted Driggs, the director of product management at ExtraHop, our show's sponsor. And before ExtraHop, Ted was a product manager for Windows at Microsoft. And he is a regular on tech security podcasts like "Risky Business," "Security Weekly" and "DM Radio." In his free time, you can expect to find Ted on the side of a mountain, zipping through powder or hiking up a rock. Here's Dave.
Dave Bittner: I'd love to start out just by getting some insights from you. You know, I recall wrapping up the end of last year, I think for a lot of us as we were getting ready to hopefully take a break going into the holidays, and we started to hear word that there was this thing Log4j, and indeed it was real, and indeed it seemed serious. You know, we went through that sort of cycle of that sort of news coming through the ecosystem here. I'm curious, what was it like for you as the reality of Log4j set in for you and your team?
Ted Driggs: Great question. So first, there was a little bit of here we go again, right? It's another December. But this time, instead of Russian state-aligned actors, we're dealing with kids playing Minecraft who have broken the internet. If we needed any further proof that children are the most destructive force in the world, we have it.
Dave Bittner: (Laughter).
Ted Driggs: But it was followed by the realization that unlike Sunburst, which was a concerted effort with a particular objective, a particular point of ingress, this was the other end of the worst-case scenario, right? This was a vulnerability that was incredibly widely deployed library. So you were no longer looking for a particular program that was behaving erratically. You were looking for any program that could be behaving erratically. And the nature of the vulnerability being passed layer through layer through layer made it very difficult to even predict how deep past the perimeter you would need to go to really protect yourself. This wasn't something where you would send a malicious request and immediately compromise the targeted server. That string might be passed through multiple tiers before it eventually was logged by Log4j and triggered that - the look-up or the remote code execution vulnerability. So from an NDR perspective, we started by asking, are we ourselves vulnerable - and then asking what challenges our customers would be facing with detection. And after getting repros (ph) up, we pretty quickly began to push out detections and to provide as much guidance as we could to our customers on the subject.
Dave Bittner: Well, and where we stand today, you know, several weeks out from the initial information, where do we find ourselves? Where do things stand at this moment?
Ted Driggs: Exploit attempts remain rampant and probably will forever. The customers that we've worked with have begun to patch or to attempt to disable the impacted features. I mean, we heard multiple - like, this and VaR (ph) is going to work. No, wait, that doesn't. There's a new vulnerability with the threat context. The initial shock wave seems to have passed, but the long tail on this is going to be a problem for years to come.
Dave Bittner: And so how does that inform the things that we do going forward. And I'm thinking of this from two perspectives. I mean, we have the, you know, famous unknown unknowns. You know, we don't know what other vulnerabilities like Log4j may be out there, but we certainly want to protect ourselves as much as possible against those possibilities, but then also going forward with the information we have here against that specific problem.
Ted Driggs: Right. So at the beginning of COVID, a lot of organizations went, quote, unquote, "boundaryless," which was a nice way of saying that they hastily finished a VPN deployment and hoped that everything would be fine and shipped it because their entire work force was coming from home. And then at the end of 2020, supply chain attacks like Sunburst blew another hole in this idea of the boundary of the perimeter, right? And Log4j is the third leg of that, where components and applications that you yourself might have built are now to now be supported to trigger a remote code execution, to trigger C2 behaviors and to establish that foothold inside your environment.
Ted Driggs: We really need to continue to think that initial compromise is an inevitability now, right? There are too many ways, whether it is this or whether it's the next vulnerability in some widely used library, that will enable an attacker to get a beachhead somewhere in your environment, and it may or may not be one hop from the perimeter. So there are now multiple avenues of ingress into the environment. In addition to your traditional phishing and credential theft, supply chain attacks, attacks against network infrastructure that's now exposed to the internet, especially after the shift to remote work - all of these are things that are making the possibility of initial compromise almost a certainty for every organization. So we shouldn't be defeatist. We shouldn't give up and just wait for attackers to get that foothold. But we do need to be thinking beyond that initial point into how we identify, contain and eradicate post compromise, right?
Ted Driggs: We've seen a lot of good efforts here from multiple parts of the industry. We've seen the work on SBOM, CycloneDX working to make a lot of SCA - commercial SCA stuff available in the public domain. We're seeing better and better tools around detecting post-compromised behavior. You know, more and more vendors are recognizing that perimeter or endpoint protections by themselves are insufficient and are building alliances, partnerships or products to go after that. And we're continuing to see the SaaS providers, who are running a growing portion of email that's used for phishing and credential theft, helping to lock that down. There's a lot of people who seem to sort of dogmatically believe that one of these is going to be the path out of this, and that's incorrect. We're going to have to keep doing all of these forever.
Dave Bittner: You know, it strikes me that when we talk about a lot of these defenses, I mean, even the, you know, what people refer to as basic hygiene, we talk about it in kind of an aspirational way where you need to do - you need to take care of the basics. And yet so many organizations have trouble keeping up with the basics and for good reason. You know, patching is hard. It's hard to - all these things are hard to do while you're in business. You know, a friend who jokes about it, it's like trying to change the oil while the engine is running. And how do you advise organizations to do the best that they can, given the reality that this stuff is not as easy as some people make it sound?
Ted Driggs: That is a great question. So this is one place where ExtraHop comes at this from a different angle than some might because we are a network detection and response tool. What happens inside the endpoint is actually less relevant to us. So whether you patched it or didn't, we're going to still monitor it and look for evidence of exploit attempts and exploit success or anomalous post-compromised behaviors. So that's where we feel like these tools complement each other well. Having tools in the endpoint, having vulnerability management tools, having SCA tools, those help you reduce the risk that you will need to do a more expensive containment and eradication after an attacker has gained a foothold. But you can't rely on those tools being perfect and expect to be successful.
Dave Bittner: So what is the best practice then? You know, how do people dial it in in terms of the variety of tools that they have, you know, using that limited amount of time, resources, money, all those things, turning those dials?
Ted Driggs: So what's important to come back to is that security is ultimately everyone's job, right? The SoC team is not going to be able to make patches to critical applications during the Q4 change freeze and hope to be successful at having zero business impact. So they need the participation of app owners to make those updates. And we did see a lot of vendors very quickly rally to distribute updates of their products to remove vulnerable versions of Log4j.
Ted Driggs: The second piece is that the SoC needs to be integrated into an organization's overall cybersecurity strategy, right? It's not possible to treat security as this bolt-on sidecar that, you know, there's this CISO and there are some people who show up from time to time and tell you that you can't do something. Security is an ally in maintaining the availability and reliability of the business, and it needs to be integrated into those conversations from the outset.
Ted Driggs: You mitigate the time crunch by spreading the load of this across multiple parties, right? App owners who have that understanding know that they need to update their applications as soon as is practical, while SoC teams have the tools available to them to protect those applications until such a patch can be made without jeopardizing the business. So that's the best answer, but it's not often not the easy answer and it's often not the realistic answer. In those situations, it's a matter of combining defenses, you know, at the perimeter on the endpoint and east-west on the network and trying to identify using management which systems are exposed to this and limiting their access to limit the damage that they could do if they are attacked while still unpatched.
Dave Bittner: Looking ahead towards 2022 and beyond, are you optimistic that we're in a good place to take on these challenges?
Ted Driggs: No, but we're getting better. So the work being done by the NTIA on SBOM is going to be hugely important for setting a standard for the entire industry around distribution of what's in this software, right? Part of the Log4j nightmare was that initial response of I don't even know which things I have in my environment that use this. I'm going to give a shout-out here to the work being done by CycloneDX in particular in automating that SBOM build process, so that for a variety of common languages and platforms, it is able to recursively figure out those dependencies and auto generate a build artifact to ensure that the thing that is distributed matches the actually built software.
Ted Driggs: One piece that ExtraHop feels is a necessary addition to SBOM is something we've been calling behavior transparency. So knowing what's in a piece of software is crucial for addressing vulnerabilities. However, knowing what behaviors a piece of software might perform in an environment is crucial to identifying when it has been the victim of a conscious supply chain attack, right?
Ted Driggs: We've been talking a bunch about Log4j as evidence of supply chain vulnerability, which is certainly true. But the counterpart to supply chain vulnerability is a deliberate supply chain attack. In that situation, it's likely that the bill of materials won't show anything amiss because the attacker likely concealed their involvement in there or they've hidden themselves in some .dll that appears benign to an outside observer. So we think that both parts of that are going to be necessary. But the fact that the industry is recognizing and moving to address supply chain problems through collective action rather than through trying to have a number of unilaterally imposed solutions is really promising.
Rick Howard: And that's a wrap. We'd like to thank Tom Quinn, the CISO of T. Rowe Price, and ExtraHop's Ted Driggs for joining us. "CyberWire-X" is a production of the CyberWire and has proudly produced in Maryland at the startup studios of DataTribe, where they are co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Eiben. Our executive editor is Peter Kilpe. And on behalf of my colleague Dave Bittner, I'm Rick Howard. Thanks for listening.
