Behavioral transparency – the patterns within.
Dave Bittner: Hello everyone and welcome to Cyberwire-X, a series of specials where we highlight security topics affecting organizations worldwide. I'm Dave Bittner. Today's episode is titled Behavioral Transparency, The Patterns Within. President Biden's Cyber Executive Order includes provisions for a Software Bill of Materials in government contracts. It's a critical and necessary first measure for protecting the software supply chain to defend against cyber attacks against the ones that affected SolarWinds and Colonial Pipeline, organizations also need transparency about the way the software in their supply chain behaves, how and with whom that software engages, in and outside of their networks. A program note, each CyberWire-X Special features two segments. In the first part of the show, we'll hear from industry experts on the topic at hand, and in the second part we'll hear from our show sponsor for their point of view and, speaking of sponsors, here's a word from our sponsor, ExtraHop.
Dave Bittner: To start things off, my CyberWire colleague, Rick Howard, speaks with Caleb Barlow, CyberWire Partner and former CEO at cyber resiliency firm, CynergisTek. The second part of our program features my conversation with Benjamin Higgins and Ted Driggs, both from ExtraHop, to explore how behavioral transparency can give organizations an advantage by distinguishing between expected noise and indications of compromise. Here's Rick Howard.
Rick Howard: I'm joined by Caleb Barlow, longtime cybersecurity practitioner and former Vice President of the IBM X-Force Threat Intelligence team. He knows a thing or two about how cyber adversaries attack their victims. So, Caleb, President Biden signed his Executive Order on improving the nation's cybersecurity back in May. One of the requirements was for software vendors to provide Software Bill of Materials or SBOMs to their customers. What is an SBOM?
Caleb Barlow: Well, if you think about it, it's kind of like how when you buy a box of cornflakes, right, the ingredients are on the back of the box of what is in this, and we've all tried to get healthier. We all look at the back of the box and go, "Alright, what am I eating? Is this corn syrup or is this sugar?" Well, a Software Bill of Materials is kind of like that. You know, the reality, building software nowadays is that you don't start from scratch and build everything, you borrow tools and APIs and capabilities both from code repositories, from open source but also from various cloud tools and vendors. So, when you look at a software Bill of Materials, it's really about understanding what's inside the box so I can then decide, "Hey, do I trust this and is it good for me?" Not any different than the ingredients label on a box of cereal.
Rick Howard: Well, I got to tell you Caleb that, you know, having the ingredients on my Cap'n Crunch hasn't made me any more healthy.
Caleb Barlow: Well, we probably need to talk about that if you're still eating Cap'n Crunch, but seriously, right? If you think about it, we have an entire generation now that pays attention to what they eat and my kids that are teenagers, they don't always eat the healthiest things, but they do know enough to look on the back of a label and go, "Oh my, that's just chock full of sugar, I'm gonna go get something else." In a lot of ways, security professionals have got to start doing the same thing and look at something and go, "Wait a second, this is chock full of open source. Do I know the pedigree of what's in here and who built it and where it was built?"
Rick Howard: So, okay, I have a list and I agree that's a good idea to know where all that's coming from, but how does a security practitioner on the back end use that information? What can they do with that?
Caleb Barlow: Well, I think the reality is they're probably not doing it themselves directly. They're probably using a service, either from a services firm or, you know, there's a whole cottage industry that's been built up to help people understand what's inside their software and, most importantly, what is the vulnerability inside a piece of software or of hardware that they may have deployed in their environment. So, you know, a very common tool set is to buy a set of tools that will scan your environment, take an inventory of what you've got, hardware and software, and then alert you of when there's a vulnerability or something to be concerned about in those ingredients. So, again, this isn't probably so much about a CISO sitting down and looking at every software package and trying to decide for themselves if it's good or not. It's more likely enabling a service or another piece of software to monitor that for them, because the reality is what's inside, and if that is vulnerable, could change all the time. I mean, look at how often we update our Microsoft technology to the point at which we've reserved one day a week just for doing that.
Rick Howard: So, you see this as being an opportunity for young cybersecurity entrepreneurs that could provide this as a service? They would be the go-to vendor for their customer, they would be the experts at all the software that's out there and then advise their customers about what they should do about it, or maybe even be given the permission to upgrade their software packages if they were found wanting?
Caleb Barlow: I think that industry already exists. The challenge is the ingredients label doesn't exist on a lot of enterprise software packages. So, although the industry is already out there today to say, "Oh hey, you know, there's a vulnerability in SolarWinds in this particular version," or, "There's a vulnerability in this particular version of Microsoft Exchange, and you need to get it patched," and oftentimes those tools will also help you with the patching. "Hey, you're using this particular version, it has this particular issue, here are the IOCs and TTPs associated with that," if someone's actively attacking it in the wild. And then also, "Here's the patch," or point you to the patch or, in some cases, actually facilitate the patching. The difference is the industry today often doesn't know what's inside an enterprise software package, and that's where this is the first step to starting to fix that.
Caleb Barlow: But I think there's a bigger thing underneath all of this, Rick, which is that, at the end of the day, the US government is doing something very interesting to influence the private sector. So, rather than passing legislation to say, "Hey, private sector, you need to do the following things to get your act together on cybersecurity," what they're doing in the light of, you know, SolarWinds and all the recent major attacks is they're leveraging their procurement power, they're leveraging Executive Orders to influence the private sector through the purchasing power of the US government. And I actually think that's not only a novel way to approach this, but it'll probably be pretty effective, because the US government purchases an awful lot of software and hardware.
Rick Howard: Well, it's amazing to me that this is a pretty technical thing, and it's a relatively new idea compared to some of the ideas that government has tried to do in the past. The fact that it's coming out from as Presidential decision directive, that a President is giving us this order on this technical thing, does that mean that an SBOM is a silver bullet? Is it going to solve all of our problems, or is it just one more thing we've got to do?
Caleb Barlow: Of course not. It's one more thing we've got to do, but you hit a key point here, Rick, which is this is now a Presidential directive, right? I mean, just think about that for a second. The President of the United States is giving us effectively a technical order, right? But, if anything, that just helps to articulate how big of a problem this is for our nation.
Rick Howard: So, what you're saying is that it's a different way to approach the problem. As opposed to passing legislation, this says, "You know, you all should be better at cybersecurity," because that's what basically most legislation has been over the last two decades. This is something directed at procurement, that you have to this or contractors aren't going to get the contract they want. So, it's a different angle, and kind of innovative for the government to do this.
Caleb Barlow: Absolutely! And you know what? This has been going on for years in the private sector, right? If you're a manufacturer of a product or good, who influences you on quality, on cost, on availability? It's your retailer, right? So, this dynamic has been going on in supply chains for years. I think what we're actually seeing as we're talking about supply chain today, is supply chains doing exactly what they're designed to do, coming back around and saying, "Hey, what we're doing today is not acceptable. So, if I'm going to keep buying from you, if I'm going to rely on you as my supplier, you're going to need to do some things differently," and just as we see the US government flexing their muscles, I think we're also going to see banks, manufacturers, energy companies start to leverage their muscles in a similar way, to say, "Hey, I need to know what's inside this software or hardware. I need to know who you're working with and where you built it, and I need you to demonstrate to me, not just tell me that you have good cybersecurity, I need you to prove it to me by having a third party either assess what you're doing or, better yet, validate what you're doing."
Rick Howard: So, I totally agree this is a fantastic idea, and we all should be doing this as best practice, but if I was going to ask you to place this on the Gartner Hype Cycle, where would you say it is in the journey? Is that the beginning, or are we further on down the path that it's almost going to be useful to us out of the box? Where would you put it?
Caleb Barlow: Oh, that's an awesome question, Rick! I think we're right at that inclination where you start to see the formation of the hockey stick, right? There are lots of companies out there that are kind of in this space. These capabilities aren't necessarily mainstream yet, and some of them are fare more advanced,
Rick Howard: But to bring it back to the Hype Cycle idea for a second, I agree with you that it's at the beginning. Everybody that I've talked to about it says that this SBOM idea is right at the beginning and we're all very excited about it, but for it to become useful somebody is going to have to figure out how to automate it all, because right now it's just another manual requirement that we have to collect all this information and do something about it. So, until it can plug in to our existing software deployment stacks and then give me some automated reports and then, in my fantasy world, be able to offer suggestions to patch it or to take it offline or whatever the recommendations are going to be, I don't see this is as being too useful anytime soon. I'm thinking, you know, five, ten years. What's your estimate there?
Caleb Barlow: Well, here's the thing. I remember, and I'm going way back here, right, the early days of Lotus Notes and being kind of part of the team that would work on that stuff, there was something like--
Rick Howard: Lotus Notes! Man, oh man, let's go back a while!
Caleb Barlow: I know, I'm old!
Rick Howard: I've read about that, I'm not old enough.
Caleb Barlow: Oh, please, you were around then! But here's the thing, right? 30 million lines of code, and I can only imagine what it's at now, right? But there was code in there that Ray Ozzie wrote back when he was probably in his 20s that was still in this thing.
Rick Howard: That's true. That's very true.
Caleb Barlow: Oh, it is totally true! And there's nothing wrong with that. We have to kind of acknowledge that these things exist and say, "Hey, I bet you some companies don't even know what's in their Bill of Materials, because this thing's been too long."
Rick Howard: The problem I have with all that, and I don't disagree that there are issues like that in almost every piece of software that we run, but what do you do about it? Look, you brought up the SolarWinds attacks. A lot of people had that platform running in their organization, because there isn't a lot of competition out there. You got SolarWinds and maybe two or three other vendors. So, the option is you run it at risk if you found out about it, or you rip it all out and bring in somebody else, who probably has similar issues. So I don't know how that solves the problem too much, except it gives you something else to worry about. Am I off the base, there?
Caleb Barlow: Well, you're not totally off the base, but I think to a certain degree we have to hold all of us accountable for the cybersecurity posture of our stuff, right? So, we all have a fiduciary duty to look at the code that we're producing and say, "Look, can I say, with a reasonable level of attestation, that this is secure? Have I done all of the things that I need to do to secure that?" The first step, as we saw in that Executive Order is we better know what's in it! We at least need to know the ingredients. Now, you know, you get into all kinds of issues around trade secrets and everything else, but we're going to have to do this so we've got to start somewhere. I think the wrong answer is to throw our hands up and say, "This is an unsolvable problem."
Rick Howard: Yeah, I totally agree. I would be glad to have it as an automated capability somewhere down the road. But let me ask you this, and let's turn on a dime. This SBOM, the main thing it would work against is supply chain vulnerabilities, right? Is there something else we can be doing in the meantime? What else should security practitioners be thinking about as we figure out how to build this SBOM capability?
Caleb Barlow: I think the next we really need to be thinking about is actually advancing how are we looking at our suppliers. So, kind of the start of the art, if you will, is you hire a services firm to go out and evaluate your suppliers, which usually involves asking them a bunch of questions. So, you ask them 40 or 50 questions or what do they use and how do they secure it, and everybody learns how to fill them out in a way that will pass. That's really not securing anything. All that's doing is creating a whole lot of work for a whole lot of consultants, by the way, including my company, to go out and do that work! But is it really changing the dynamics on the security posture? No.
Rick Howard: Let me come at it from a different angle, an angle that I think would have more impact sooner than all of this SBOM stuff which, by the way, I'm not saying it's a bad thing, I want to reiterate that. It's a great idea, I can't wait for it to come to fruition. But here is what I propose that we do today. I really think that we should institute a zero trust strategy for all of our supply vendors. And I know zero trust, it's become such a bad word because nobody really understands what it is, but what I mean by that is, if I'm running SolarWinds, let's say, and I have the SolarWinds platform in my environment, I better know exactly what that software, what the administrators, what those accounts on that platform can do inside my environment, and watch it like a hawk and give them no permission to do anything of consequence inside my organization. And if I have to let them do something of consequence, I'm going to watch those transactions and make sure and triple check them and triple authorize them. I think that would have a much bigger impact than the SBOM will have, you know, in say, five to then years. What do you think about that?
Caleb Barlow: I think you're spot on, and I think these things actually go together, because one of the things you can do is collide these two ideas to say, "Hey, here's the deal. We want to see your SBOM, and ideally you produced that, but anybody that can't produce that or won't produce that, then we'd better treat them in a zero trust environment." Because you know, let's face it, there are certain situations where you need to bring in a vendor and maybe there's some legacy reason or some issue of why you need to move beyond zero trust.
Rick Howard: There's always that reason.
Caleb Barlow: There's always that reason, but here's a great way to think about this. To your point, Rick, if I've got one of those exceptions where I've got to trust this vendor then I think, again, it's completely reasonable to go and say, "I need your SBOM. I need a full third-party assessment from a reputable firm. I need to know everything about what you're doing, because I have to trust you." If, for whatever reason, I can't get that. They're a startup, it's too early; it's an intellectual property problem, whatever, or it's built in a country that maybe I don't completely trust, well, in any of those reasons, absolutely has to be zero trust. I think at the end of the day, what we're learning is you have to look at all these directives, whether it's the work that's happening with CMMC, the Presidential directive, it's all about getting a more robust supply chain.
Caleb Barlow: I'll give you a little idea of why this is so important, coming from an industry in which we operate. We go out and do a ton of work in the healthcare space. From our own data, 76% of America's healthcare systems fail in securing the supply chain as measured on an s-score, so just think about that for a second.
Rick Howard: Yikes! Yeah.
Caleb Barlow: And this is critical infrastructure, and it's not that they're not trying, they're just not investing fast enough to keep up with the adversary.
Rick Howard: Good stuff, Caleb. Thanks for coming on the show, we appreciate it.
Caleb Barlow: Thank you.
Dave Bittner: Next up is my conversation with Benjamin Higgins and Ted Driggs from ExtraHop, our show sponsor.
Ted Driggs: We've seen modern organizations need to move faster and faster to be successful in delighting their customers or their stakeholders and to survive competition.
Dave Bittner: That's Ted Driggs, Head of Product at ExtraHop.
Ted Driggs: As they've done that, the budget for security, especially up front, has not kept pace, so instead we now trust our vendors implicitly more than ever, and our vendors change their product's behavior faster than ever to meet our needs. So, that's created a situation where it becomes very difficult for analysts to understand what's in their environment, what behaviors it should exhibit, the consequences for interdicting those behaviors and, therefore, it's very hard for them to block anything or take action without a lot of fear and uncertainty involved.
Benjamin Higgins: So, just to key off of that...
Dave Bittner: That's Benjamin Higgins, Software Architect at ExtraHop.
Benjamin Higgins: ...say you have a server, say it's running Windows and it's running an app, perhaps, like SolarWinds Orion, and maybe you've also got an agent or some other admin software installed on that server, and perhaps you get an alert from one of your security products, or maybe you're just doing a threat hunt and you want to explore the DNS queries or the network connections for that server. The problem is, how do you quickly known good network activity? If you've ever looked at DNS logs or netstat, which lists active network connections on a machine, you're probably familiar with this feeling of wondering what all of this activity is. So, today, what you have to do, it's often not hard but it can be very time-consuming to track down and investigate each behavior one by one. The interesting thing, when it comes to at least enterprise software, vendors already publish lists of some of their behaviors, in particular, external network behavior, for the sake of their customers that are running that software in a lockdown environment, where they have to configure their firewall to allow these known good connections to occur.
Benjamin Higgins: So, some of what we're going after with behavior transparency, it's really just a small step beyond existing practice, where we want to add two things. One is in addition to behaviors being listed on a web page or in a PDF, we want to have a simple machine-readable format that can be consumed by various tools and solutions, including firewalls, for that use case I've just mentioned, and also SIEMs and EDR and NDR products, or whatever custom tooling that companies might have. The second is to have all of this information in one place instead of being scattered around the web. So, with these relatively minor additions to existing practice, we ultimately think that security analysts would be able to answer questions much faster about behaviors they see, and potentially help catch supply chain or similar attacks when they do occur faster, especially when they target servers and server software that tends to have a more tractable network behavior profile. So, that's behavior transparency in a nutshell.
Dave Bittner: And what specifically is the challenge here? You mentioned that there's just so much information traversing the network. Is a lot of this separating the signal from the noise?
Benjamin Higgins: Yeah, I think that's a big piece. Like I mentioned, I've had this experience many times when my own devices. When I'm looking at netstat or similar output, you see all of these destinations, all of these domains and IP addresses. You sort wonder, like, "What is all this stuff?" And the truth is, like Ted mentioned, software's changing, software changes rapidly these days. Even at the OS level, whether it be Windows or Linux or something else, there's often a lot of different OS components that are reaching out, in addition to the server software that you're running, as well as whatever supporting tooling or agents that you have. So, that ends up being a large list and, you know, most of the time it's a completely benign list. But let's say you have a list of 100 domains, 100 DNS queries, which is probably a conservative number, and then one day you have something new that shows up on that list, well, if you haven't already done the work to categorize all those other 100 items, your first step is establishing that those are known good, that those have been documented by the respective vendors, and that work can be time-consuming.
Benjamin Higgins: So, the hope is that, once you eliminate all that known good stuff, you're left with this sort of much more tractable list where you can do that manual investigative work, and you can find these potentially malicious behaviors much faster as a result.
Ted Driggs: This is not as simple as a WHOIS, it's worth mentioning, so as we see software vendors themselves using third party components, for example for telemetry or analytics, if I see acme.appanalytics.com being invoked with a bunch of API requests, the WHOIS information for that might not say that it's Acme Corp, but appanalytics.com may or may not have any ownership requirements over sub-domains. Maybe it was someone who just signed up for this fledgling appanalytics service and chose a name to disguise themselves as something that's much more widespread. So, one of the challenges that we run into is that traditional tools for assigning ownership to things don't work well in these more complex ownership or partnership scenarios and, as an analyst, I'm now sitting there kind of puzzling over, okay, this is a probably signed request, but it's by this company Appanalytics. Do I need to go ask Acme if they use Appanalytics legitimately? Who do I contact for that? Do I call my support person, will they even know? How many hours am I signing up for myself just on hold or in emails to get this answer? And then if I get this answer, great. That's not going to be a particularly satisfying return on investment, to spend two hours trying to run this down, to find out, "Yes, that's a thing that we added a release ago, and it's standard for our thing now."
Dave Bittner: Give me some insight. What is it like on the other side, for the folks that you work with, in other words the folks who are implementing behavior transparency and are seeing the fruits of that effort? What are the benefits that they're seeing there?
Ted Driggs: So that's something that we're going to talk through at Black Hat in quite a bit more detail. We're going to look at four scenarios where the behavior transparency can help here. Some of these aren't built yet, frankly. We were still working on building out an initial catalog, and we've been vetting requirements for it and scenarios with some partners, but the broad strokes are that this can be used in multiple places in the SOC lifetime. One great example is with a SOAR partner. If, say, a scanning detection was fired, somebody could use the behavior transparency database to look up scanning on that particular port number, and then report back, "Here are the softwares that that could be," and then the analyst doesn't have to go look for themselves, they now have software that ideally even links to documentation for each of those pieces of software, explaining what it's doing that for.
Ted Driggs: Another example would be the, "Hey, I've got a device in my environment, and I'm trying to make sure I know all the software that's on it." Let's take those connections that it's making, run them through the behavior transparency database and see what software is on that device. So now I can use this to make my CMDB even more accurate, more up to date, and then once I have that profile, I can now, to Ben's point, come back again and say, "Okay, knowing that this is the software that should be on there, tell me what connections it's made that are not explained by that piece of software."
Dave Bittner: How is this enhancing my relationship with my suppliers up and down that chain? You know, who are supplying me and the people I'm supplying things to. Is this making it easier for us to keep tabs on each other and make that more of a trusted collaboration? Is that part of this, Ben?
Benjamin Higgins: Yes, I think so. So, when, you're in the situation where you're running software in a lockdown environment, you've got to publish this list and enable your customers to actually have it work in that environment, but I think, when it comes to the completeness or the accuracy of that list, there's absolutely room for error, and that comes from a couple of sources. One is, as you're updating software, that list of external destinations might change, or it could also be because there is maybe a more obscure feature that it's only when you enable it do you talk to a new external destination. The hope is that, instead of having just these one on one conversations with vendors, that potentially when you have this information centralized and you have more eyeballs on it, that there can be more collaboration and more back and forth as to the accuracy and completeness of the list.
Ted Driggs: It also, for my downstream partners, participating in behavior transparency makes me a less exciting target for a supply chain attack, because, in order to get that behavior into the behavior transparency manifest, to avoid it sticking out as an obvious oversight, somebody is going to need to make a public amendment to its repository. So, that creates a lot more opportunities for someone to identify the bad behavior and to then burn whoever it was that admitted that update in addition to disavowing or otherwise retracting that false domain claim. So, you could imagine, for example, in the SolarWinds Orion case, let's say that the compromise at SolarWinds had been complete enough to push something as SolarWinds into the behavior transparency database, at that point, the expectation is that our intel providers are going to be very interested in looking at new additions to this database and who they came form. That domain is going to very clearly stick out as not being anything SolarWinds related, and the expectation is that there's kind of a security in numbers that comes from a bunch of people looked at that domain and went, "Huh, what is this? Like, this doesn't match anything else that SolarWinds has done historically, it's not reflected in their website, we should reach out to them and ask what's going on here."
Benjamin Higgins: To make Ted's point in a different way, any vendor that's already publishing network behavior information would be benefited to publish that in this new format in a centralized place for a couple of reasons. As Ted mentioned, it makes that vendor less of a target for supply chain attacks, because a potential supply chain attacker is going to see that this vendor seems to care more about security, it seems to care more about having people have eyes on their software's network behavior. The other reason is, even if they were successful with a supply chain attack, having this information published in this way improves the chances of someone else catching it faster. So, it would still be bad news, but the harm would be reduced or minimized by being caught early on.
Dave Bittner: So, it really helps take you off of that list of potential low hanging fruit?
Ted Driggs: Yes.
Benjamin Higgins: Exactly.
Dave Bittner: Yes. Well, you mentioned earlier that you all are presenting at Black Hat on this, can you give us a little overview or preview of what we can expect there? You're going to have a little bit more time than we have today to dig into some of the details, yes?
Ted Driggs: Yes, so we're going to be going more in depth into where behavior transparency in partnership with multiple security vendors or directly consumed by SOC teams would enable better decision-making or faster decision-making with better information at various points during the work of the SOC and of INFOSEC teams. So, everything from during procurement decisions, CMDB building, threat hunting and responding to alerts, where behavior transparency can be used by different stakeholders at each point, as well as an example of what a manifest might look like.
Dave Bittner: So, for folks who want to learn more about this, what's the best place for them to go for that?
Benjamin Higgins: So, today the best place is to go to extrahop.com/behaviortransparency. We've got a brief description on there and a form that you can fill out, that we're using just for this purpose, for people who want to get involved.
Dave Bittner: On behalf of my colleague, Rick Howard, our thanks to Caleb Barlow from CynergisTek for sharing his expertise, and to ExtraHop's Benjamin Higgins and Ted Driggs for joining us. CyberWire-X is a production of The Cyberwire, and is proudly produced in Maryland at the startup studios of DataTribe, where they're co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Eiben, our executive editor is Peter Kilpe, I'm Dave Bittner. Thanks for listening.
