“Shift Left”: A case for threat-informed pentesting.
Rick Howard: Hey, everyone. Welcome to "CyberWire-X," a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, N2K's chief security officer and the CyberWire's chief analyst and senior fellow. Today, Dave Bittner, the senior producer and host of many of the CyberWire's podcasts, will be joining me at the CyberWire Hash Table to discuss the art and science of pen testing. After the break, you'll first hear my conversation with Bob Turner, the field CSO for education at Fortinet, and Etay Maor, the senior director of security strategy at Cato Networks. And then Dave will talk with Dan DeCloss, the founder and CEO of PlexTrac. Come right back.
Rick Howard: Penetration testing is a vital part of any robust security program. But some security experts feel like the traditional pen testing model is in a rut. Assessments happen infrequently. The scope is often very broad. And the report is usually overwhelming. The question is, what if you can increase the overall ROI of your pen testing program and avoid these limitations? Every penetration test should have specific goals. So I asked Etay Maor, the senior director of security strategy at Cato Networks, to describe the various ways security practitioners could use a penetration test. And he came up with a unique metaphor to describe the differences.
Etay Maor: I want to make the distinction between three different functions within a security group, because sometimes I see these returns being used interchangeably, both in discussions as well as even in job openings. And those are vulnerability scanning, red teams and threat hunting. So what are these three groups, and what do they do? And two years ago, roughly, I had to deal with a flooding in my house, so I'm going to use that as my metaphor for today.
Etay Maor: So vulnerability scanning is like somebody standing outside your house and trying to spray it with water and see if anything leaks in. So that's what vulnerability testers do. They come from the outside trying to get in, using known vulnerabilities - Log4j, for example - and trying to get into the house. Red team - those usually are the people who are within the organization internally, and they try to break something from the inside and see if anybody notices. So, for example, it'd be like if we take our metaphor of somebody breaking a pipe in the house and starts a leak, let's see if anybody can identify that. A lot of the red teaming we see, again, it's very broad, but you'll see, like, types of phishing attacks or people within the organization trying to utilize an insider or social engineering and so on.
Etay Maor: And threat hunting is something a little bit different than that, when - well, if we go back to the house, you'll see something, a wet spot on the wall, and you'll say, OK, let's find out what that is and try to trace it back. So threat hunters will deal with something that already happened on the network and try to trace it back and see where it started from. Again, this is a pretty high-level approach to it, but there are differences between vulnerability scanning, red team and threat hunting.
Rick Howard: I would clarify Etay's three ways here and maybe add a fourth, to emulate adversary campaigns and a blue-team-red-team exercise - some call those purple team exercises. It's the idea that a pen test team playing as the red team would emulate the attack campaign of some known attack sequence, let's say Wicked Panda, out of the MITRE ATT&CK Framework. The blue team, the internal security team, would respond to the red team attack sequence, and the two teams would compare notes at every stage. In this way, the pen test is a training exercise for your internal security teams and a way to check the internal defensive posture against a known threat campaign.
Rick Howard: Bob Turner is the field CSO for education at Fortinet. His previous CISO gig was at the University of Wisconsin at Madison. I asked him about what he did for adversary emulation at the university.
Bob Turner: Let's talk about how you weight the reason for testing. If you're weighting that on simply a part of your build process, then yeah, that's OK. But I think that you should be thinking about the systems you're testing. And these are just kind of made up percentages, but really sort of a proportion of the way I think of the order of battle, for lack of a better term, I think that half of your penetration testing, if you're doing it, should be targeted at the high-value systems and infrastructure as a whole. And this is particularly important in manufacturing, as well as banking, as well as retail. You really need to be able to reveal the weaknesses in your security architecture. So if you're looking at the high-value targets at least half of the time you're doing pin testing, then that's - you're in the right place. I think that 30% of that testing should be aimed at the systems with vulnerabilities discovered during routine cyber ops, which could simply just be going over the list of known vulnerabilities, you know? Microsoft is famous for putting theirs out. And a lot of the other technology manufacturers are available to do that. So go in, make sure the systems are patched. And retest those systems. And I call that routine cyber operations because we should be doing that anyway. The last 20% is for those major systems where the testing is being used to satisfy some kind of a compliance program, whether it's the 800-171, you know, privacy rules for education, institutions that are handling federal data, HIPAA, FERPA, GLBA and the rest of the alphabet soup. And then, of course, the privacy-focused regulations, GDPR, CCRA and all of the other requirements that are out there. And I want to include privacy because when you do the penetration tests, you're trying to make sure that the data in that system is going to be handled correctly and remain private. So part of that testing is, try to manually go in and see if you can get access to data. And that's a skill that penetration testers have, in my experience, managed very, very well. But guess what? That's also a hacker's skill (laughter). So...
Rick Howard: Yeah, exactly.
Bob Turner: If the hacker can get into it and your pen testing team can get into it, you probably need to change your access requirements and do something different. I think that it is, really, more the diagnostic and prevention than it is the emulating hacker skills. Every penetration test that our team ran had a script. And we always made sure that we involved the system owner and let them know what was going on and what the possible outcomes would be. The difference between that and the penetration tester is finding out before the fact what could go wrong or finding out after the fact what did go wrong. That's the difference between the two.
Rick Howard: Yeah, I think it's more of - it's a more advanced idea, because it's one thing to go in and let a pen test find things, right? It's another entirely, I think, that says, you know, when we see APT15 hit our networks, we should see this kind of thing happen by our defensive teams and systems. And if they don't happen, that's where you need to fix some stuff, I believe.
Bob Turner: It ups the realism. It ups the realism of your program when you have an opportunity to script out what you think might happen, you know? And you really have to take the entire attack surface and the MITRE chain in effect when you're doing that. No. 1, it's because that's how the enemy's getting in. But it's also how to build confidence in your processes and procedures around how to deal with it when it is the bad guy and not your pen tester.
Rick Howard: What's the ROI on these things, Bob? How do you convince your boss that you want to spend X amount of dollars in terms of the people, process and technology triad to conduct a test? What does he or she get out of this investment?
Bob Turner: There is no value you can put on the confidence factor in your team. If your team has the ability to do this and prove that the system is now more secure than it was before they did the pen test, then that's invaluable. I can't put a number on that. But I think the return on investment comes with finding out where the vulnerabilities are and making sure that that system can remain online. In the CIA triangle, I harp on the A. Availability of systems is paramount. You can't get business done without them. What's the cost of losing your business for a day or two?
Rick Howard: Well, exactly - right? - if it's material to the organization, meaning that you have to devise resilient systems. Then it can stay up regardless of what happens, either some horrible failure in the IT system or power or APT15 comes in and causes you to have a bad day. All of our systems have to handle that kind of thing. You have to plan these things, right? You just don't do these willy-nilly. You have to coordinate not only within your own team, but with the organization, too, who doesn't really know what the heck you guys are doing.
Bob Turner: And this is not just an education-specific thing. It happens in a lot of companies where understanding that somebody is going to be looking over your shoulder at the system that you have hand-built from parts around the shop there, now they're going to penetrate and poke in and find out where your vulnerabilities are.
Rick Howard: And nobody wants you to call their baby ugly, right? So - (laughter) right, so you have to kind of...
Bob Turner: Well, yeah. And that's a big thing that, you know, if you think about it, it's really something that needs to become the norm. It just - we need to desensitize system owners from anything other than what we're here to do is help them. System owners think of their application, their network, their tool as their attack surface. They don't necessarily take it in the context of the entire organization's attack surface. I venture to say that if we're lucky, we might see in an education environment probably 60 to 75% of the attack surface, the total attack surface, because there's so much that happens behind layered firewalls. And although I see that changing, I think it's really important to understand that the reason why we need to do penetration testing is to find out what is going on in those environments.
Rick Howard: So that's one reason you might do it, right? It's a discovery process, because, you know, people are installing stuff all the time, taking stuff offline, replacing it with stuff. So that's one of the benefits you get by just going in an area you're not sure of.
Bob Turner: And it's also the - think about the pace of updates and changes that happen to operating systems, cloud applications. And I can - this - another survey I was reading is something about 44 to 50% of education cybersecurity teams lack confidence that all systems are updated and patched. And I would say that that number is probably skewed a little bit low.
Rick Howard: So you're going in, releasing the dogs to see if they can find anything you didn't know about.
Bob Turner: I characterize it as, you know, every time you perform a penetration test, it actually becomes a platform for learning more about your technology stack and your campus networks. It's really perspective-oriented. When a system's first placed in service, pen testing provides that final check before live data is applied. I call this an extension of whatever your GRC-related testing and documentation contains. I think the second one is periodically testing those high-value information systems. You're actively pursuing any adverse findings you find out of that first round of testing, then you're promoting confidence in using the system, and penetration testing more is a gateway to providing that mythological continual cybersecurity assurance.
Rick Howard: Well, you mentioned GRC systems. That is essentially checking to make sure you're in compliance with the various laws that your organization has to follow, laws and regulations.
Bob Turner: I think the R is the important part of that particular acronym, and that's just the risk management. If you're doing that pre-incident, then if you ever get into the post-incident phase, returning an information system back to service with a clean bill of health is the ultimate power in doing that penetration testing. You really will know when it's ready to go and you really will know by doing that penetration test whether you fixed all the problems.
Rick Howard: So, Bob, we've been yakking for a few minutes now. Any last words of wisdom about pen testing that you've gleaned in your historic career? What can you tell newbies about this that they should know before they launch into this kind of thing?
Bob Turner: Penetration testing is intrusive. It requires participation not only from the team and the testers, but also from the system owners. And again, the planning of the test, conducting the test and reviewing and analyzing and acting on the results is great.
Rick Howard: This is not a gotcha test. This is a we're working together to see if we can make the system better.
Bob Turner: Absolutely. Think that we have to be careful because the skills that are necessary to mint a top-notch pen tester are also the skills that could attract a higher price tag outside of education, and I think education needs to worry about that. You have to be able to think of the tester as a complete human who is very smart, and you need to make sure you treat them well and give them the experiences that are going to help them grow or you're probably going to pay significantly to replace them at some point.
Rick Howard: Next up is Dave Bittner's conversation with Dan DeCloss, the founder and CEO of PlexTrac, our show's sponsor.
Dave Bittner: Today, we're talking about this notion of shifting left and around the idea of threat-informed pen testing. Can we start off with some definitions here? I'd love to get your take on how do you and your colleagues there define the whole notion of shifting left?
Dan DeCloss: Yeah. So how we define it, and how we speak about it to our customers and in conversations in the industry, really is are we doing everything that we can in a proactive measure to prevent as much as possible or as deep of an impact from a breach or a, you know, an event that causes downtime from a malicious activity, you know, related to a cyberattack or something like that? So it's, are we shifting left in how we approach our responsive measures, our security controls that we have put in place and the technical capabilities that our team has to be able to prevent or detect as quickly as possible?
Dan DeCloss: So in the biggest context that we refer to that in is related to our proactive security testing, you know, namely penetration testing and tabletop exercises, all of the security controls and the technical expertise that it takes to try and identify where your gaps are in your environment or your applications. Are you taking as proactive of a measure - of measures as possible to prevent any of those nefarious activities from happening in the first place? Clearly, that is a - you know, that's a - what we would call, like, a BHAG, right? It's always going to be a matter of when, not if, related to getting breached in some fashion. But, yeah, so shifting left is really, have you done all of the proactive measures from a testing perspective, from an identification perspective of what are your gaps to identify these key areas that an attacker could get in and preventing as limited blast radius as possible when and if that event does occur?
Dave Bittner: And how does an organization kind of calibrate how far left to shift? You know, I can imagine you want to be in that sort of butter zone - not too far, not far enough.
Dan DeCloss: Yeah, exactly. I mean, yeah, like - yeah, the butter or the Goldilocks zone.
Dave Bittner: Right.
Dan DeCloss: I mean, I think in terms of - it really - that's really going to depend on the maturity of the organization, right? I mean, we're all aware of the fact that security itself is a marathon. It's a journey. It's not a sprint. You know, it's not something that can be solved overnight. And so there's always a progression, so that's where the notion of trending really plays an important factor. Start with what you can do today, and then, you know, compare yourself to where you were this time last year or this time last week, and are you making progress? And I think that's what's most important. And that's even, you know, how we as professionals can stay - you know, stay healthy and sane - right? - is that, hey, we know we're working on the right things, and we're making progress. So I don't think that there's a true litmus test for, like, are we doing enough or not? It's more have we been able to make progress in what we were doing last week or last year or last month, right?
Dave Bittner: You know, I think most folks who are listening to us are certainly familiar with penetration testing, but I'm curious for your take on where do we stand right now when it comes to pen testing? You know, what's the state of the industry when it comes to that? Are we where we need to be?
Dan DeCloss: I would say, not yet, right? And I think what the awareness is really growing is that there is a capability in a way to do this in a more continuous fashion and to have a true program around it and that it is a vital and robust part of your security program. And so, you know, the - I would say kind of the traditional or maybe even archaic way to view penetration testing is we do this once a year. And we just give carte blanche to the testers and let them go at it for a few weeks or even a month, or maybe more, and then they finish that up. And then they deliver us this report that's really, really long and has a lot of findings or a lot of, you know, security holes that we need to fix, and we have to figure out how to prioritize them.
Dan DeCloss: We do the best that we can, but then we kind of move on with our day, and then next - the next penetration test a year later rolls around. And they may find some additional things. They may find some things that didn't get fixed - or, I mean, sorry - they might find some things that did get fixed. But at the end of the day, there's not a whole lot of progress there. And I think that's the old kind of way of thinking about penetration testing.
Dan DeCloss: Where the industry is continuing to shift is a more continuous mindset, knowing that these are very valuable assessments, but they can be expensive if we don't come at them with a more programmatic approach. So the ways that people can do that is, one, being more specific in what we're testing for on a regular basis and either bringing some of that capability in-house or working out, you know, arrangements from a continuous fashion with your service providers to say, hey, every month we're going to do something like this, and we're going to test these things. We want to know how we can incorporate these types of threats that come in, you know, periodically throughout the year and really make sure that we're honing in on the true gaps and fixing those first and making progress.
Dan DeCloss: So in terms of the way that the notion around penetration testing is transforming is it's becoming much more continuous and much more accessible. We have platforms that can do automated testing. We have much more training opportunities for individuals to do some of this work on their own. And then more and more companies are bringing a lot of these capabilities in-house so that there is a notion of we have our internal team that's doing the continuous approach, and then we have our external team that is going to be a fresh set of eyes. They're going to have that more periodic and global view of our environment. And, you know, and then really checking the box on the compliance respects as well.
Dave Bittner: You all frame this in terms of threat-informed pen testing. Help me understand, what exactly does that mean? What's the nuance there?
Dan DeCloss: Yeah. So I think in the industry today, you know, we use threat intelligence a lot from a reactive and a response capability perspective, and we're really trying to take the notion of threat-informed defense to a threat-informed offense, right? So then threat-informed penetration testing is really saying, how can I identify what's going on in the wild, how it applies to our environment and be able to actually test against those capabilities with those techniques? And it also invigorates the team to know, hey, when, you know, a big threat comes out or, you know, one of those notable, you know, breaches like, you know, the Uber breach or something happens, like, you know, Log4j, right? - and you get that, you know, dreaded question from your executive staff like, hey, how are we doing against that? You least have now an answer to that question because you're now incorporating that into your testing program. You're saying, hey, these are the things that we've tested for with respect to Log4j. Here's how we're identifying it. Here's how we're identifying the gaps. Something like that is really incorporating that notion of taking threat intelligence, building it into your testing plans and being proactive about it, rather than waiting for the response of capabilities and only using threat intelligence for indicators of compromise and plugging in threat intelligence into more of your response capability. So it's really an augmentation of threat intelligence to add to your testing capabilities, your proactive measures, to identify any of the gaps that you have in your environment related to that threat.
Dave Bittner: And what about ROI? I mean, how does an organization measure that they're getting everything they need to out of this sort of investment?
Dan DeCloss: Yeah, exactly. So in terms of being able to measure, you have to have some way to track it - right? - and have a way to understand what you've been doing over time. And that's where that trending capability really comes into play - is like, how are we doing compared to last year? And so being able to incorporate these test plans show the coverage that you're getting from not only the threats that are being identified but also the general tactics and techniques that attackers use. And we have a lot more resources around that. So having a capability of tracking how you're doing and getting better really starts to show the progress that you're making and then being able to help translate that to the business of, like, what the impact, you know, has been, right? And so you can't really show an ROI without that trending data and the ability to show, like, these are where our efforts are going on a daily basis and what we're testing against and how we're making progress in resolving or remediating the risk related to those.
Dave Bittner: What are your recommendations for an organization who's looking to start down this path, to begin this journey of using threat-informed pen testing? How do they begin?
Dan DeCloss: Yeah, exactly. So I would say, you know, we have a lot of resources that are on our website, plextrac.com. But also, MITRE ATT&CK framework is a great place to start or from the OWASP Top 10 are great places to start just from, like, an idea of, like, these are the common techniques and tactics that attackers are using and starting to build out your own test plans. You can grab test plans from these organizations, as well. And specifically related to individual threats, particularly on the MITRE ATT&CK framework, you can highlight, these are the threats that I want to test for. And they have their threat emulation plans. And then, you know, staying in touch with - in tune with kind of the free threat intelligence feeds that exist out there, you can start to say, like, OK, I'm learning a little bit more about what the attackers were doing in this type of a breach. And so I'm going to take some of those techniques and then incorporate them into my testing in my environment.
Dan DeCloss: Now, the biggest thing is, how do I do that? How do I actually do the testing? And that's the beauty of things like the MITRE ATT&CK framework - is that you really can start small, right? It can be, hey, I don't have to test for every single possible procedure that this threat actor is - has the capability of doing in my environment. But I know that I should probably start with these important techniques or these important tactics and just test those and start there and then just getting those on a continuous basis. And then you can continue to expand your reach as you get more mature, as you get better at understanding how to do the testing, how to measure it, how to track it. And that's the beauty of, you know, something like PlexTrac - is that you really can utilize a platform to have the ongoing capability in a tracking mechanism and have the analytics around it.
Rick Howard: We'd like to thank Dan DeCloss, the founder and CEO of PlexTrac, Bob Turner, the field CSO for education at Fortinet, and Etay Maor, the senior director for security strategy at Cato Networks, for helping us get some clarity about pen testing and making it work for us. And we'd like to thank PlexTrac for sponsoring the show. "CyberWire-X" is a production of the CyberWire and is proudly produced in Maryland at the startup studios of DataTribe, where they are co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Eiben. Our executive editor is Peter Kilpe. And on behalf of my colleague Dave Bittner, this is Rick Howard signing off. Thanks for listening.
