Defining the intruder’s dilemma.
Rick Howard: Hey everyone. Welcome to CyberWire-X, a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the chief security officer, chief analyst and senior fellow at the CyberWire. And today's episode is titled "Defining the Intruders Dilemma." And what we're talking about here is the failure of perimeter defense, an architecture that we invented back in the 1990s but never really worked that well. The idea was that we would build giant electronic fences around all of our digital assets, designed to keep the bad guys out. But we had to poke holes in the perimeter to allow employees, contractors and partners to do legitimate business with us. Those same holes could be exploited by those same bad guys we were trying to keep out. The question then is, what should we be doing instead? What are the strategies and tactics that are more secure than perimeter defense? One idea that is relatively new but catching on is called software-defined perimeter. And another idea has been around for a decade called the intrusion kill chain prevention. In this episode, my colleague Dave Bittner and I invited two guests to the CyberWire Hash Table to discuss the issues - Jerry Archer, the Sallie Mae chief security officer, and Mike Ernst, ExtraHop's VP of Sales Engineering.
Rick Howard: A programming note - each CyberWire-X special features two segments. In the first part of the show, we'll hear from industry experts on the topic at hand. And in the second part, we will hear from our show's sponsor for their point of view. And since I brought it up, here's a word from today's sponsor, ExtraHop.
Rick Howard: I'm joined by Jerry Archer, the chief security officer at Sallie Mae. He's also the founder of the Security Advisor Alliance, a nonprofit group of CISOs focused on finding and encouraging the next generation of cybersecurity talent. And you've been coming to the CyberWire Hash Table since we started the conversation some two years ago. So, Jerry, thanks for coming back to the show.
Jerry Archer: Wow, I'm getting to be an old hat, huh?
Rick Howard: Yeah. When your bio is longer than the show, you've been around for too long, I think.
(LAUGHTER)
Jerry Archer: Time to retire, Rick.
(LAUGHTER)
Rick Howard: So in a survey done by the Cloud Security Alliance back in 2020, only a quarter of the respondents even had heard about software-defined perimeter. And I know I didn't learn about it until I interviewed you last May for my own podcast, "CSO Perspectives," when we were talking about identity management. So for our audience members who are not familiar, can you take a swing at defining what software-defined perimeter is?
Jerry Archer: Software-defined perimeter essentially sets up what amounts to a software perimeter around your entire environment. And what that means is, is that you need to be pre-authenticated in order for the environment to recognize you. So basically, the only thing that's exposed to the internet, if you will, is a single controller port that you have to present your credentials to. If your credentials are accepted by the controller, then the controller will notify the environment of what you have access to and allow you to come inside. Other than that, the inside, if you will, will not even acknowledge that it exists. So you can't even ping it. It won't even respond to a ping.
Rick Howard: So I don't even like the name software-defined perimeter because with this model, we've completely abolished perimeter defense as we knew it back in the old days. It's completely gone. Instead of rocking up to the workload that you're trying to get access to and entering your user ID and password, like we've been doing for 20, 30 years, this goes to another place, a different place, not even associated with the workload. You log in there with - in the, what you called it - the security controller verifies your identity and verifies the authorization of where you can get to. And if it likes all that, then it establishes the secure connection to the workload. And like you said, that hides everything else. The bad guys don't see anything. I really like that architecture.
Jerry Archer: It's hard to attack what you can't see.
Rick Howard: (Laughter) That's very true. That's very true.
Rick Howard: On a side note, I was talking to my old buddy, Steve Winterfeld - he's the Akamai advisory CISO - about how the phrase software-defined perimeter is a bit misleading. Here's Steve.
Steve Winterfeld: So it's interesting you ask around a software-defined perimeter because we go back to the concept of a perimeter. And we don't have a perimeter around our data because our data isn't inside a castle or any of those things we like to think about. It is in so many different places and so many third parties and different kinds of infrastructure. I really am not sure we should be using a location-based word. I want it to become more location-agnostic, move to something more like Identity-Aware Proxy, where it doesn't matter where the person is. We're not talking about branches. We're not talking about remote office work. We're simply talking about access.
Rick Howard: And Gartner places software-defined perimeter on their 2020 Cloud Security Hype chart about halfway down the trough of disillusionment and gave it about 2 to 5 years before it's ready for prime time. And when I talk to security leaders about these new ideas that come down the pipe, most are thinking about how to deploy them. And some even have pilot projects. But you, shall we say, were an early adopter at Sallie Mae. You have it fully deployed and have had it fully deployed for years. So can you walk us through that process and how you came to decide that that was the way to go?
Jerry Archer: When we first started at Sallie Mae the move to the cloud, one of our guiding principles was to try and not use rocket science technology. We wanted to use standard sorts of things that could be employed, for sake of argument here, to create a rocket science solution without the rocket science technology because that's hard to maintain. The software-defined perimeter had been around for quite some time. It had been pioneered in the intelligence community as a way of mining open source data with anonymity. And it had been then taken out of the community by a group called Vidder, and Vidder pioneered it into commercial space. So when we purchased it, it was Vidder as a company. And then it moved over to Verizon. And strangely enough, now Verizon is going to discontinue it. So now there's more companies out there than Vidder who are now offering essentially the software-defined perimeter in various products that they now have. I mean, we were very surprised when Verizon decided to eliminate it. I guess Verizon eliminated product from its security portfolio and went primarily to services. So...
Rick Howard: I did not know that. That's really interesting. Well, you're right that it came out of the government channels earlier. I mean, the DOD worked on a project called the Jericho Forum - kind of outlined some of these best ideas - but they never implemented it, and they just kind of let it die off. And then when Google got hit by the Chinese government back in 2010 in Operation Aurora, they redesigned their internal network from the ground up, using first principles of zero trust and software-defined perimeter for their own internal stuff. And then later on, they eventually released a commercial product called BeyondCorp. When did you first install your software-defined perimeter for Sallie Mae?
Jerry Archer: Now it's almost five years ago. And we did it in the initial security stack. So that was part of the architectural design of our initial security stack, was to have the software-defined perimeter - and actually, that's the first step to get inside the front door is to be pre-authenticated. So one of the things that we recognized as part of that was that identity and access management became paramount. And the amount of identity and access management you have to do is significantly greater because now you have to go out and define every user that wants into your environment.
Rick Howard: I totally agree with this architecture. To me, it's so far superior than the way we used to do it. Tell me if I got this wrong, though. It's basically a SaaS application that does all your identity and access management for you and then arranges the connections to the workloads that your employees and devices need to get access to, right? It's basically a SaaS app.
Jerry Archer: Well, we still use Active Directory. So once you get inside, you still have to go through Active Directory, and Active Directory decides what you - ultimately, the resources that you can get access to. So you still have to have that capability, right? You have to be identified within AD in order to get access to resources.
Rick Howard: So what's the big lesson learned when you guys deployed it? What did you run into that you weren't expecting, that you had to figure out?
Jerry Archer: Yeah, I don't think we had much of a problem at all. Now, we, again - I can tout Vidder because they no longer exist as a company. So I'm not...
(LAUGHTER)
Jerry Archer: ...Pushing a venue that anybody can go out and engage with. So - but we had Vidder that helped us out significantly in deployment. And because they had a lot of experience doing it, we really didn't have any trouble at all. I mean, we were very clean and deep, very, very fast to get it up and going. And once it's up and running, now it's just a matter of identity and access management.
Rick Howard: So let me change gears a bit. One of the cybersecurity myths that started to emerge in the mid-2000s was that attackers only needed to be right once, where defenders need to be right 100% of the time. And we know now that cyber-adversaries have to string a series of actions across the intrusion kill chain to - in order to be successful, that if we can break a link anywhere in that attack chain, we can defeat the entire attack campaign. In fact, we've completely reversed that old myth. The attacker has to be 100% successful across his attack sequence. We only have to be successful once in defeating him. So, Jerry, I know you guys subscribe to the intrusion kill chain strategy at Sallie Mae. Are you guys a big user of the MITRE ATT&CK framework to get your intelligence, or do you get your intelligence about attack sequences from other intelligence sources? Or is it kind of a mix?
Jerry Archer: It's a mix. I mean, MITRE's ATT&CK kill chain is there very much. A lot of the stuff we use as software is a service model. So we use a lot of tools or services that have their own kill chain models that we use. But I would say one of the primary kill chains is also the MITRE ATT&CK model.
Rick Howard: On another side note, I asked Steve Winterfeld about how easy it is to use the MITRE ATT&CK framework.
Steve Winterfeld: I really love the fact that you can walk all the way through the process. So if you go out to a CISA or FBI alert and you go down into the reference material, you'll see MITRE ATT&CK APT link. And if you follow that link, it goes over to the MITRE page on APT 41. And then if you click on the Attack Navigator layers, it just pulls you right into that attack framework and highlights which ones - if you're looking at initial access, it tells you the two they're using. If you look at execution, it tells you the five different capabilities they use. And it's just so easy then to turn to your red team or your pen test team and say, we think this group is going to attack us. We want you to follow this kind of run book to do an attack against us to validate whether or not we're secure against it. I think it's just a great resource.
Jerry Archer: But we've now very much focused on mean time to remediate or repair versus mean time between failing. We've basically acquiesced to the notion that there's no such thing as perfect intrusion prevention. You can't stop people from getting in. And honestly, phishing now represents - I think I read a statistic - about 97% of all the attacks are now phishing. I hate to say this out loud, but I can't fix stupid. Somebody's going to click on one, right? What we've now done is we've really focused on EDR and trying to stop malware right at its inception.
Rick Howard: Put me firmly in the stupid box because, you know, we have one of those services at the CyberWire where they send fake spam messages to see if you're smart enough to avoid them. And I've only been at the CyberWire for a couple years, but I've been caught at least three times, and I know better. If we can't help me not get caught, I'm sure we can't help the grandmas out there. So I agree that you're going to be penetrated at some point. What we're trying to do with the intrusion kill chain defense strategy, though, is prevent the adversaries from being successful. Do you task your intelligence team specifically to go after known adversary group sequences? Like, do you say, hey, go see if we see anybody from Panda Bear running around your networks? Is that how you guys do it?
Jerry Archer: (Laughter). Yeah, we do. We do a lot of purple teaming, and we use automated tools to do that. The purple teaming is a big way to put in the attack scenarios and look for the indicators of compromise to make sure that we can detect the bad guys as they're coming in the door and trying to execute. And so we do that on a continuous basis. So that's the way we look at the problem. And then we array our defenses based upon the current situation in the world. The other side of the coin is you want to reduce mean time to repair so that the impact is less and less and less. So you want to get mean time to repair to zero. Take it into the cyber world - it's exactly the same thing, right? All of our defenses, perimeter defenses, firewalls and everything like that are designed to put MTBF longer and longer and longer. You stop people from getting in the front door. But inevitably when somebody gets in the front door, now what you want to do is you want to stop them as quickly as possible, stop lateral movement, stop any ability to move forward. So EDR, XDR are sort of your paramounts in terms of intercepting that before anything can happen.
Rick Howard: So when you're looking at an adversary group like Sandworm or Panda Bear or any of the 150 to 200 adversary groups that are out there and they have a number of things they have to do in their attack sequence - let's say it's a hundred. So that means your team has to know there's a threshold. Like, if you see one or two of these things pop up in your network out of the hundred, it might or might not be Panda Bear. But if you see 50 or 60 of them associated with Panda Bear, you got Panda Bear in your network, and then your teams can go and remediate that, right?
Jerry Archer: Oh, no. We're much earlier in the kill chain. We're right at the inception when they first go after command and control. Our goal is the moment that malware executes in the environment and goes out and looks for a payload or looks for any kind of command and control, we want to stop it right there. So we're very much focused on an endpoint kind of detection capability. Obviously, the higher up the stack we go, the more things we're looking for in the environment, on the networks and all that sort of thing. But I mean, I would tell you that we're really focused on the idea of endpoint detection right away.
Rick Howard: In order to follow the intrusion kill chain strategy, you need much more than firewalls and intrusion detection in your security stack. You need endpoint detection and response, maybe even network detection and response or combining them into extended detection and response tools like XDR. That's how you're able to track these adversaries, right?
Jerry Archer: Exactly. We start early. We have multiple agents sitting on every single endpoint these days, and they're looking for all kinds of abnormal behaviors and indicators of compromise. We take 2 1/2 billion events a day and shove it into a data lake. That's our XDR data lake. So we're looking for any indicators of compromise across the entire enterprise.
Rick Howard: So Jerry, as always, you're well ahead of most of the security practitioners that I get to talk to on this show, especially in terms of software-defined perimeter and the intrusion kill chain prevention strategy. Any last words about lessons learned that you figured out while you were doing this for the past 14 years that you could pass on to our listeners?
Jerry Archer: We started our cloud journey five years ago, and I would tell you that the biggest single thing that one can do is nibble your way to success. People who sit down and define megaprojects almost always fail. When we started, we put out a cloud environment that did almost nothing, but it demonstrated we could go to the cloud. And we added a little bit more and a little bit more and a little bit more, and that's how we got to where we are.
Rick Howard: So take small bites. Don't try to boil the ocean is what you're saying. Take small bites, demonstrate success and keep moving forward. Good stuff, Jerry. Unfortunately, we're going to have to leave it there. That's Jerry Archer, the chief security officer at Sallie Mae. Jerry, thanks for coming on the show.
Jerry Archer: Anytime, Rick. Thanks a lot.
Rick Howard: Next up is Dave's conversation with Mike Ernst, the VP of sales engineering at ExtraHop.
Dave Bittner: So today, we are talking about this notion of the intruder's dilemma. Let's start off with some high-level stuff here. Can you give us a little bit of the lay of the land, like, where, in your estimation, we find ourselves when it comes to this sort of thing?
Mike Ernst: Sure. So I feel like we got to start with a bit of background, which is, you know, the defender's dilemma is super common, you know, all over the press, industry analysts. And that's the adage that, you know, the attackers only need to be right once. You know, defenders need to be right 100% of the time. And therefore, you know, you need to invest in perimeter security defenses - walls, castles, moats, however you want to describe - you know, to keep the bad guy out. And even though that's your objective, it's - you're kind of inherently screwed because of this asymmetry.
Dave Bittner: Do you think that that is an accurate statement, the whole notion of - that the defenders need to be right 100% of the time?
Mike Ernst: Yes, well, with a caveat. If your goal is to prevent any aspect of your environment from ever being compromised, yes, you know. If that's the objective is, we must prevent all compromises, then you do face this asymmetry. And you do have to invest an inordinate amount of money, time and resources into preventing that compromise.
Dave Bittner: Well, let's flip it on its head then. I mean, we're talking about the potential for there to be an intruder's dilemma as well. Can you walk us through that idea?
Mike Ernst: Sure. So you know, this is the post-compromise reality, right? Like, we're taking the perspective that someone is going to get in somewhere, somehow, you know? Someone clicked a link they shouldn't. They opened an email they shouldn't, you know? They plugged in a USB drive that they shouldn't. Something is going to happen somewhere where somebody is going to slip through your defenses. And then you have an intruder. That's the starting point for the intruder's dilemma.
Dave Bittner: So the intruder has made their way in. And now they have some decisions ahead of them. How does that usually play out?
Mike Ernst: So what does the intruder do? So he's got your laptop or my laptop or a phone - whatever resource he's obtained access to, that's his foothold, his toehold. He's got one. Everybody starts with one, right, you know? You got in somewhere. So the first thing you need to do, you need to do reconnaissance of some sort, you know? What else is around here? What's adjacent to me, you know, what other systems, devices? What's out there? You're going to do some form of scanning, reconnaissance, you know? Network reconnaissance is usually the most common step, you know, practitioners are going to be familiar with. NMAP, you know, that's probably the scanning tool of choice for blue teams and red teams alike. So you'll, from your beachhead, you know, survey the environment, get a lay for the land and then plan your next step.
Dave Bittner: And how do they go about doing that while also keeping their head down and, you know, not drawing attention to themselves?
Mike Ernst: I mean, in the end, you still need to scan the environment over the network, you know? There's no avoiding that. But you could be a bit more crafty or stealthy, you know? You could do it very slowly, very carefully, you know? You can run an NMAP scan slow enough that it's almost impossible to detect the scanning activity, you know? Let's say, I'm only going to look at one IP an hour. Well, that might take you months, so unlikely. But, you know, there's various knobs and dials that you have to kind of control the - how obvious is your scanning activity.
Dave Bittner: And so once you have established what's going on around you, you've kind of mapped out that environment, what's the next step?
Mike Ernst: Well, the odds are almost certain that you want to gain access to more machines, you know? It's very unlikely that the first footprint that you got a hold of is your end game and has all the - whether you're looking for data, whether you're looking to do ransomware or whether you're just looking to be destructive, it's highly unlikely that that first entry point has everything you need. So you need to move laterally, which means, you know, moving to another machine, different area - maybe get to a server, maybe get to, you know, a device that is more stealthy, like a printer. Or there's all sorts of smart devices out there. Maybe there's some older network security devices that you can get. I mean, usually, the - you know, people think the hardest thing is to get in. And that's why we spend so much money to keep people out. But then once you're in, there's not as much defensive capability to prevent this lateral movement.
Dave Bittner: Is it fair to say that with each bit of movement, with each bit of activity, that the intruder has provided the defenders with an opportunity for detection?
Mike Ernst: That's the intruder's dilemma that we want to get to, which is each of these steps that you're going through from initial access to, you know, end game success, the attacker needs to be right 100% of the time. So you've kind of flipped the dilemma on its head, you know? Now it's - the defender just needs to see one of those things. So let's say they didn't see your reconnaissance. They didn't see your lateral movement. They didn't see you establish persistence. But they did see you try to exfiltrate data. Like, that's it - you lost, assuming they can then go back and piece together the threads and see what else you did. But, you know, now the burden is on the intruder to not get caught.
Dave Bittner: So is part of the notion that, perhaps, we're paying too much attention to the perimeter defenses?
Mike Ernst: I would say, yes, because, I mean, while it's not something you can ignore, because the more effective defenses you have - I mean, you know, there's different layers. I mean, there's people out there that are running scripts all the time. There's the not-serious actor. There's the cybercriminal. And then there's, you know, the advanced nation-state threat. So you need a perimeter because you need to make it harder for an intruder to get in. But I don't think enough thought or investment goes into the, well, what happens if I was compromised? Like, what do I do next? And what tooling and capabilities do I have to prevent that initial intrusion from becoming, you know, a complete breach?
Dave Bittner: So what are your recommendations in terms of the types of things that are available to folks to be able to monitor inside the proverbial castle walls, to have as many opportunities as possible for detection, but at the same time not providing too much friction for the users who are inside?
Mike Ernst: Thanks for that softball setup there. I mean, we're kind of biased...
Dave Bittner: (Laughter).
Mike Ernst: ...In this sense. But, you know, we think the network is the perfect resource to pick up these post-compromise activities because it is everywhere. It is passive. It is, you know, convenient to monitor. And it does provide you the opportunity to detect all of these post-compromise activities. The reconnaissance we talked about uses the network, you know? Making your next move laterally uses the network. If you need to establish persistence, some kind of beaconing command-and-control, uses the network, you know? Moving data around the environment uses the network, you know? Trying to get into the, you know, active directory or some of the, you know, key systems, you know, it all uses the network, like, literally every step of an intruder's post-compromise playbook.
Dave Bittner: Help me understand how an organization goes about dialing this in, you know, to not be hit with a lot of false alerts, to not be chasing their tails? What is that initial process like?
Mike Ernst: So this is where, you know, network security has gotten a lot better. And, you know, a bit of - I mean, people have been monitoring the network for security purposes since time immemorial. But the technology hasn't been there to do much beyond - you know, there's a lot of store the PCAPs, capture packets, warehouse them, you know, use them from a forensic perspective or investigative perspective. But that's not real-time monitoring. And it's been hard, you know? Then we've got NGFWs out there, which have better data and better visibility into what's going on. But it's still just scratching the surface at how much intelligence you can actually get off the network. And this is where, you know, network detection and response comes into play. You know, advanced technology that can extract all of the Layer 7 payload, all of the transaction activity, feed that into sophisticated machine learning engines to give you better, more actionable alerts that are reasonable. You're not, you know - and I guess I missed a step there. The IDS was one step in that, you know, evolution of network security. But that was a signature-based matching engine that quickly became an alert canon and was not super useful for security teams.
Dave Bittner: Well, help me understand. I mean, to what degree is this examining the data itself - you know, actually going and looking at packets versus flagging activities? You know, as you mentioned, exfiltration. Is it a blend of both or is - how is all that dialed in?
Mike Ernst: I feel like different vendors will give you different answers. You know, from our perspective, the best data is in the packets themself. I mean, that has everything. So if you can analyze all of the packets in real time and scale and extract the relevant transactions and information on those transactions, like, that should be the basis of your network analysis because that has the full context. You know, there are other kind of higher-level network statistic analyses - you know, like NetFlow or firewall logs. And, you know, if we use an example, like, that would say, you know, Dave access to database. Dave pulled this much gigabytes from that database. That's helpful. But it would be way more helpful to see the exact select statement that you ran on that database. That tells us exactly what you did.
Mike Ernst: And there's a lot of other activity that has been hard to see because it's encrypted, and it's not accessible outside of the network - things like MSRPC calls, you know, Kerberos and LDAP traffic, you know? So we can see - I mean, MSRPC is basically everything you can do remotely to a Windows machine. There's a lot of anodyne, run-of-the-mill activities there, but there's also the ability to create a new service or to schedule a task. And we can pick that up off the wire. But you have to be decrypting MSRPC traffic, and then you have to understand the protocol sufficiently to know, like, aha, this was a task-creation activity that could be potentially malicious versus a lot of other MSRPC calls that are always going to be of no risk or concern. That's new.
Dave Bittner: So how do folks best start down this path? I mean, if this is something - I'm interested to know if this is a good fit for my organization. What's the best way to start?
Mike Ernst: Regardless of where you want to go in the NDR space, you have to understand - to do NDR, we - you need network packets and you need to find key chokepoints in your environment, either in the data center or the cloud, where you're going to be able to collect that traffic. And there's actually a whole industry - I think Gartner calls them the network packet broker space - dedicated to getting traffic off the network and to products like ExtraHop for this kind of analysis. And you'll need to have a - you know, an idea for where in your environment is optimal to obtain that traffic.
Dave Bittner: Yeah, yeah. What is it usually like on the other side for folks? You know, after they have started down this pathway and they see it's up and running, they're starting to realize some of the benefits. What do you hear from them?
Mike Ernst: I mean, we enjoy that first conversation because there's almost always something insightful or actionable that shows up almost immediately, like, just once after we've been hooked up. It could be something small, something from a security hygiene perspective that they thought had been decommissioned or they thought had been fixed. But, you know, we're just reporting on what's actually taking place on the network. So it's a pretty undisputable - indisputable, I think, is the word. I don't - now you got me thinking - I think indisputable source of truth. You know, if it took - if it crossed the network and it showed up in ExtraHop, like, it happened. You know, there's no real debate on the subject.
Mike Ernst: And, you know, big companies have complex environments and, you know, frequently will leave a test system there or a system that was scheduled to be decommissioned or upgraded and it didn't happen. And it just - nobody caught it until, you know, we saw in the wire that it's still there and still exhibiting something from a hygiene perspective that should have been cleaned up a long time ago. You know, there's usually a quick win out of the gate and then it's a wow. I mean, think about this. This is all of the traffic in your network - up to hundreds of gigabits per second, all fed into appliance, metadata extracted, built into our model of the environment. And then the machine learning starts to kick in. You know, it learns your environment. So, you know, gone are the old days where you had to manually configure all these alerts and tune all these knobs and thresholds. You just let it do its thing. And then it starts to tell you, you know, hey, I saw something that this particular device or endpoint has never done - looks suspicious. You know, this warrants a further look.
Mike Ernst: You know, down to some - you know, I mean, it's rare that we find a smoking gun, you know, attack in progress at the moment we do it. But we find a lot of things that look like suspicious activity, unusual behaviors that customers get a kick out of investigating. Even if it does turn out to be totally acceptable, like, they get a feel for how this works. And this is the - that post-compromise posture that we talked about where, you know, now they're looking at things that are taking place on the inside. Should they be taking place? It's a different vantage point than the perimeter than they're used to.
Rick Howard: And that's a wrap. We'd like to thank Jerry Archer, the chief security officer at Sallie Mae, and Mike Ernst, ExtraHop's VP of Sales Engineering, for joining us. CyberWire-X is a production of the CyberWire and is proudly produced in Maryland at the startup Studios of DataTribe, where they are co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Eiben. Our executive editor is Peter Kilpe. And on behalf of my colleague Dave Bittner, I am Rick Howard. Thanks for listening.
