The future of security validation – what next?
Rick Howard: Hey everyone. Welcome to CyberWire-X, a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the chief security officer, chief analyst and senior fellow at the CyberWire. And today's episode is titled "The Future of Security Validation: What's Next?" It's clear that security executives need visibility into their actual cyber risk in real time. But with the flood of security alerts, it's tough for organizations to pinpoint impactful security gaps. To meet this challenge, security teams are shifting to an exploit-centric approach to security validation. In other words, they are looking towards adversary group emulation - the process of imitating the tactics, techniques and procedures of a specific adversary in order to assess and improve how resilient an organization is against known adversary attack sequences; in other words - red teaming.
Rick Howard: In this episode, I've invited two subject matter experts to the CyberWire Hash Table to discuss the current state of red teaming, blue teaming, purple teaming and penetration testing. A program note - each CyberWire-X special features two segments. In the first part, we'll hear from an industry expert on the topic at hand. And in the second part, we'll hear from our show sponsor for their point of view.
Rick Howard: I'm joined by Rick Doten, the Carolina Complete Health CISO. He's an old friend of mine and a regular here at the CyberWire Hash Table. Rick, thanks for coming on the show.
Rick Doten: Thanks, Rick, always happy to be here.
Rick Howard: So we're talking about red teaming today. And I want to distinguish the idea as being something different than penetration testing. So do you want to take a crack at explaining the difference between the two activities?
Rick Doten: Yeah, absolutely. And it's funny because two years ago when we did a Hash Table talk about this, I really didn't understand red teaming, and I ran penetration testing teams for 10 years...
Rick Howard: Yeah.
Rick Doten: ...In the late '90s to late 2000s. And so I'm like - and it also answered a question that I had from 20 years ago because I had hired people out of the NSA red team, and when they did their first pen test and they were like, well, what can I do? What tools can I use? I'm like, you can do whatever you want. It was like, really? It's like, well, what's the scope? It's like, hey, it's this range. It's like, really, I can do anything? Like, yeah. And I'm like, wow, they're real excited. They must be really limited. And then it occurred to me now, 20 years later, oh, that's because they were red teaming. And so penetration testing is - here's a target, do whatever you can to get in and exploit it to get as far as you can. Red teaming is mimicking a very specific adversary using their tools and their tactics and their techniques to be able to do it.
Rick Doten: So if you're like, hey, I'm trying to imitate this particular campaign, they leverage this kind of tool, they go this way, this is their entry point, then, you know, we're seeing how well we're going to defend against that particular adversary. And so that's the difference between red teaming and pen testing, which, honestly, two years ago, if you had asked me, I said, it's just a different word for the same thing.
Rick Howard: Well, I don't know about you, but I don't find a lot of value in pen testing. Pay some NSA red teamers to come in and find a hole in my network - yeah, they're going to find a hole because, you know, that's what happens. But if you want to go after something specific, like here, I improved this area of my network, can you guys get into that, you know, limit the scope? Or, like you said, what a real red team does is emulate somebody like Fancy Bear and see if their tactics, techniques and procedures will actually work against my network. That is something useful that I can use. So I don't know - do you agree with that philosophy?
Rick Doten: Yeah, I agree - I agree with the exception of compliance. We're a health care payer, and so some of our Medicaid state contracts require us to do this yearly. And so there are things that you have to check the box to do that. And it's also kind of good just to get a feel as you go around the network. But general pen testing - yes, it is better as a validation step because we have no problem finding things - our challenge is fixing things. So, like, let's work on the process to fix things, and then let's validate it by then trying to find if there's anything we missed.
Rick Howard: I agree with you, right. I don't need someone else to find more things for me to fix. You know, my list of to-dos is endless. I already know a bunch of stuff I got to fix and to pay somebody - some group - an awful lot of money to come in and tell me some more things - I don't find a lot of value in that. And I don't find a lot of value putting my own folks on it too because, like you said, we should spend those resources fixing things, not finding new things to go do. I mean, there's value in finding new things, but that's not the first thing I want to go do.
Rick Doten: Yeah, I mean, I remember, you know, also, like, maybe 15 years ago, companies stopped doing internal pen tests because we do an external pen test, which was, like, a certain range - an internal - and we get as far as we can. And the internal pen test reports were hundreds of pages of, like, real things. And they're like, listen, I don't have time to fix all this anyway, so let's just stop doing that. And I bring that up only because just recently this came up in another CISO roundtable I was on that they were, like, hey, should we be doing more of this? But, you know, again, focus more on the asset configuration management, vulnerability management stuff you have. And then maybe as a validation step, you know, do that. But that's also like the variation between a vulnerability assessment and a penetration test, right? You know, vulnerability assessment - you identify vulnerabilities. Penetration tests - you exploit them and see how far you get. Oftentimes, I just - I'm fine just knowing that it's there. I trust that it can be exploited.
Rick Howard: Yeah. I don't want to spend a lot of time figuring that out. So you mentioned earlier that you ran a pen testing team - a commercial pen testing team. What's your best story about those guys? What's the craziest thing they ever discovered?
Rick Doten: One of my favorites, I guess, was that the team - this was about 2005, and we had the team in a cube farm. And when they're all huddled around somebody, you know something's going on. So I kind of looked down there like, hey, what you guys got? And so I go down. It was on Friday evening. And I go down, and they're like, hey, they're testing this online bank mortgage application. And they had already, I knew, like, did a snail mail DDoS. You know, we had, like, 500 rejection letters show up at our office.
Rick Howard: (Laughter).
Rick Doten: You know, letter letters - like, OK, that's not good.
Rick Howard: Letter letters - those things still exist? Yeah (laughter).
Rick Doten: Yeah. I mean, this was 2005.
Rick Howard: OK. Yeah.
Rick Doten: Right.
Rick Howard: That's right.
Rick Doten: Yeah. It was a long time ago. And so they were saying, yeah, what we have is - OK, when you fill out the form, then it says, OK, here's everything. Check and make sure it's good, and then you hit submit. But when that one place where they say, here's your form, up on the top, far right, is - they had - the URL had the database record number. And so all you did is very easily, like, back off a few numbers, and then, poof, there's somebody else's record, you know? And just, you know - and you can then script to, like, go through and harvest ones that, you know, give you a good feedback and not a 404. And so they were, like, getting this - like, hey, we can pretty much get anybody's application that had - by just going to the database just because of this. And it was - in 2005, this was not an uncommon thing to be able to find.
Rick Doten: So at the time, I'm like, OK, we need to call the customer and tell them they need to take this down, and we need to maybe do forensics and see, has this already been exploited? And they may have to have a, you know, reportable event and blah, blah, blah. Being the manager, you know, I'm saying these things. And so I called my point of contact who was a friend of mine. And he was like, wow, that's a good one. He goes, well, it's Friday, and it's probably been there for a year or so. So we're going to just...
Rick Howard: Yeah.
Rick Doten: We'll address it on Monday. It can wait a whole weekend. And I'm like, I strongly suggest not doing that. And the reason why I use this example is, like, technical people don't make business decisions.
Rick Howard: Exactly right.
Rick Doten: It is not the CISO's decision to make this decision. It was like, you need to let the business know, and they will decide based on risk. And so that's, like, one of my favorite ones because it's also a lesson (laughter).
Rick Howard: Well, I agree with that, and it goes back to what we were saying before - right? - 'cause I already have a list of high-priority things I got to fix, and I'm just going to throw this one in the pile and put it - and adjust it, you know, appropriately. But yeah, I'm totally on board with what that guy decided to do (laughter). I don't know. But let's take it back to the red teaming itself. What I really like about red teaming, the idea of it, is that we pretty much know a lot of what's going on in cyberspace in terms of adversary activity. If you just look at a minor attack framework, they're tracking some 250 adversary group campaigns and very detailed knowledge across the intrusion kill chain on their tactics, techniques and procedures. So why would they go find something new when I need to go make sure I'm protected against everything we know is already out there?
Rick Doten: So new - I guess what you're saying is, like, if a human pen tester is, like, coming up with a new exploit to get somewhere where they use one to exploit and then queue traverse and live off the land and kind of do things, I think that the concepts are still the same as what's defining the attack framework. You know, whether you're doing a buffer overflow or whether you're taking advantage of some misconfiguration, whether you can see certain things, those are all, like, you're still doing it. But that was the thing I always loved about pen testing. I actually was a pen tester for a couple of years, and then people that were better with me - and I'm more of a manager anyway, so I moved into management. Is that...
Rick Howard: Me too.
Rick Doten: It's one of those things where you're doing something that no one's ever done before every day, and there's a certain personality type for that. And that's great. And the reason why I'm going down that thread is that keeping pen testers happy is giving them work to do that's interesting that they can still, like, try to push themselves every day. And when we were doing, like, commodity online banking applications that we've been testing for five or six years that are pretty solid and we're not going to find anything new, then they just get frustrated, you know? So I guess that there is something to be said for the baseline of, like, make sure you don't suck, and then there is the, all right, is there something that's interesting? And so that's why I had some customers who only wanted, like, a three-day pen test, like, 'cause they know that if they find something, it's going to be right away. And then the level of effort - and it's always talked about cost of attack, right? So a, you know, top-tier pen tester, after two days, would find, like, anything that was severe. And then maybe in a week, they'll find something else, but that's enough for the risk level that they're willing to accept.
Rick Howard: I agree that the personality type of a pen tester versus a red team are slightly different. But the thing I like about red teaming is the idea of combining it with the blue team...
Rick Doten: Right.
Rick Howard: ...These things called purple operations where...
Rick Doten: Yep - purple team stuff. Yep.
Rick Howard: Yeah. So the red team emulates Fancy Bear. The blue team doesn't know they're operating and hopefully is alerted to their presence. And then there's a push me, pull you about how they react and all that. And then once all that's done, then they can compare notes and say, well, we did this here. What did you guys do? Well, we did this other thing. So it's a huge training exercise on testing your incident response process. So do you see a lot of people having the resources to get that done?
Rick Doten: No (laughter).
Rick Howard: Yeah. That's it.
Rick Doten: I was going to be like, yes. I was going to - I thought you were saying, what do you think about that? Yes, I love them. Purple teams are great.
Rick Howard: Yeah.
Rick Doten: Everyone should do them. Very few people do...
Rick Howard: Yeah.
Rick Doten: ...For that reason. It's like, you know, you can get some outside pen test team to be the red team, and then you test your purple team. And sometimes people do it in testing their managed service provider and those kinds of things. But I think it's something that's not used as much as it can be, or oftentimes, it's used more just to make them look bad. You know, sometimes it's still like, you know, hey, our defense is really bad. We're going to do a blue - a purple team and show that, or our defense is really good in the purple team. And so I think they're great.
Rick Howard: So purple teams can be done internally, like you said, right? Your own - you can hire out a purple team exercise. And I'm seeing discussions now of firms using, you know - becoming SAS services that you can just automate a Fancy Bear attack where there's no humans on the other end. It's just a simulation. Are you seeing any of that as you talk to your customers and your peers out in the industry?
Rick Doten: Oh, yeah. I talk to a lot of vendors, as you know, and a lot of startups. And this attack simulation - continuous attack simulation is something that's becoming more popular, and they're doing these specific - like right from the matter take - right from the MITRE ATT&CK framework like you mentioned, and let's just automate those things, and then we'll just do different segments, and that's an easier way without having to hire expensive pen testers to be able to kind of test what you're doing. Like, alright, well, let me run these modules for these types of things against this area of this type of the matter, tech framework and kind of see how I go.
Rick Doten: But then it goes back to exactly what you said at the beginning about, like, the value of red team. It's like just doing it for the sake of doing it may not be very helpful and knowing that you're going to fail may not be helpful unless it's to really improve. It was like, wow, that was bad. OK, well, now we know, like, are we going to do anything about it? Nope, we're just going to now know, which then could also be a liability, because that's why some pen test teams go through third party council so that pen test report is published - is covered under client privilege. So it can't be disclosable because, like, you know about it, that's kind of bad.
Rick Howard: So red team, in my mind, has two benefits, right? One is it tests your defensive controls against known adversary activity. I like that. So if we run an emulation or simulation or even use humans to do it, we'll know if the things we put in place specifically for Fancy Bear, let's say, they actually work. So I like that. The second thing though, it tests your internal teams that - how come you didn't pick these guys up? You've been watching for them for a year. How come you didn't pick up any Fancy Bear activity? So those are two positives. But what you said before is important. This is not something done by startups and people with no resources, right? You have to have done a lot of other things to make your defensive posture mature, and then I would take on red teaming.
Rick Doten: Red team exercise and purple team exercise is a higher up on the maturity scale. If you were just still doing your security programs based on tools - I put in EDR, I have this thing, I'm watching this thing, I have this other thing, and I'm using encryption and multifactor - then you're not doing security operations. You are just kind of managing tools and keeping them up to date. When you then kind of elevate to - all right, now I have security operations. I'm watching, I'm responding, I'm recovering. I have teams to be able to do these things. And now I'm looking for stuff and then getting more proactive into, like, threat intelligence and threat hunting.
Rick Doten: Then we're starting to get to the point where - all right, now we're ready to do this. So you're right. I always kind of use it as a pyramid. And there's fewer people as you get to the top. And this level of maturity is, like, in the smaller part of the pyramid. It's not just some - any - when I was a CISO nine years ago for a 2,500-person company, pen test, red teaming was really not that useful. We're just trying to keep our head above water and make sure we're covering all the doors and windows.
Rick Howard: Good stuff, Rick, but we're going to have to leave it there. That's Rick Doten, the Carolina Complete Health CISO. Thanks for coming on the show, Rick.
Rick Howard: Next up is Dave Bittner's conversation with Jay Mar-Tang, the sales engineering manager for the Americas at Pentera.
Dave Bittner: So today we're going to be digging into the topic of security validation. Can we start off with just some high-level stuff here? Can you give us a little bit of the background? You know, how did this start to become a thing and, what led us to the place where we find ourselves today?
Jay Mar-Tang: Yeah, I think security validation has always been an initiative in organizations. What we've seen is, though, and the challenge is that the way this was typically done before was it was all done through penetration testing and manual third-party efforts - whether that is third parties coming in because you don't have the expertise in-house or you're starting to and have to build a team and rely on this expertise internally, and then those offensive exercises can partake. But the challenge is, is how often are you doing them? How much time is it taking? And again, are you able to acquire the talent necessary to test all those controls?
Jay Mar-Tang: And that's really what it is, using offensive exercises and the offensive security approach to understand all the investments, the layered defense that you've put in place. Are those controls working? And not only are the controls working, but are the processes and procedures that surround these controls - are they working as you expect them to? And what you don't want to do is assume and just wait for a breach to say, oh, are we ready or are we not ready?
Dave Bittner: Is it fair to say - my perception is that, you know, if you're bringing in someone from outside, that you're really sort of getting a snapshot of that point in time as opposed to an ongoing real time sort of evaluation of where you stand.
Jay Mar-Tang: That is 100% correct. So - and there are - there's many talented people out there, right? So that's not a knock on the talent as much as it's a knock on how often you can do it. You're exactly right. You're taking a snapshot in time. Maybe you do it once a year, but you do a penetration test, and then next week you might make a change in the environment. I mean, the environment in itself is dynamic. You're adding new infrastructure. You might be adding identities, you might be adding technologies or you might be adding policy. And all of that can fundamentally change the way an organization is positioned from a risk posture perspective.
Dave Bittner: So describe to me what you're advocating here. When we talk about automating our security validation, what exactly does that entail?
Jay Mar-Tang: Right, we're talking now about automating the offensive exercises that were once done manually enabling someone to hit a button through automation, be able to take those motions, those steps, those procedures, and do it at any point in time and not necessarily have to be the expert anymore. So imagine being able to do this on the weekly - hit a button, run offensive exercises, understand - or invalidate - are there any gaps in the posture? Yes or no? And validate that the controls are working as expected, and then be able to go back to the business - whether that's an auditor or just upper-level management or just in general, be able to answer the question, are we ready? Yeah, well, we did this a week ago, or maybe we're not ready, but we understand the steps and what the gaps are in order to get to the point where we are going to be in a better place and not having to wait once a year to do that is extremely powerful. And that's what we're advocating - is to always know, are you ready or not through automation.
Dave Bittner: Can you walk us through how the setup of something like this works? I mean, how do you get it configured?
Jay Mar-Tang: Yeah, I mean, what we notice is that - and I think it's a trend overall in the industry is no one wants to deploy agents anymore, right? And then there's already a lot of security agents for different purposes. And again, they have their place, but if we can do this agentlessly (ph), that's an advantage. And it's also more true to life because - to be quite honest, attackers aren't using agents. They're not going to say, oh, well, we need to put this agent on this machine, and then we're going to have system level access. No, attackers need to acquire that system level access. So what we're looking to do is start with that - the assume breach mindset as well as come from the outside.
Jay Mar-Tang: So take two angles - it's almost like a jab and a right cross, right? We need to understand from both angles, are there gaps in what controls are going to stop and work and work well? From the outside, right, it's really using open-source intelligence and seeing what is exposed externally, to see what - where a potential foothold could be. Where could - where can attackers essentially breach the perimeter? On the inside, you can use something - just software, right? - with that assume breach mindset. And say, all right, well, if we're at this part of the network or in this VLAN or this area, typically you like to start where users are because as we know, there really is no - the perimeter isn't that wall that we thought it was, right, many, many, many years ago. The perimeter is now really porous. There's holes all over the place. And you want to know what those holes are? It's our people. And that's not to say that they're bad people. It's just that, you know, you get a phishing email or you go to a website that you typically go to that is now compromised.
Jay Mar-Tang: And all of a sudden, there's an exploit kit and there's malware on that machine. It's very, very easy to breach that perimeter. Again, whether that's coming in from the outside or users are just clicking links, something's going to happen. So taking it from there, from the - where users sit, especially privileged users and because attackers are going to - they're going to do reconnaissance or anything - they're going to say, well, who's working for the organization? Oh, well, let's just go on LinkedIn. And we have, oh, we have some database people, we have some developers and oh, look at all these C-level executives and understanding from those - from that perspective with the compromised identity, what would an attacker be able to do? Would they be able to listen on the wire and find credentials? Or would they be able to with a - with again, with a compromised identity or a certain level of privilege, now move laterally or be able to get around?
Jay Mar-Tang: That's where we like to start with the goal, of course, being seeing if we can easily compromise critical assets. And that is going to do change fundamentally based on the industry vertical. Every industry has critical assets, is based on the vertical. What are those assets? And those assets could even be, you know, something that is relevant to businesses - or companies that you do business with. You might just be part of a supply chain, which we often see as well.
Dave Bittner: You know, you mentioned that an organization is a dynamic thing, that there is always change. And it strikes me that the adversaries are dynamic as well. They're adjusting, you know, their techniques. How does the security validation - when it's automated, how does it adjust in a dynamic way as well?
Jay Mar-Tang: Well, I think the dynamic portion of it - whether it's organizations or attackers, what we need to focus on generally are TTPs - tools, techniques and procedures. And by that, I mean we're not looking for the newest hash of or signature of a specific piece of code, right? I mean, when we think about and when we talk about vulnerabilities, vulnerabilities are ones that we just are aware of and vulnerabilities that are just - when I say aware of - the ones that are disclosed. So if SolarWinds doesn't disclose - right? - or maybe even RayFire (ph) doesn't disclose or whatever and if you - anyone who's been or had experience with incident response probably agree in the fact that there are a lot of incidents that are happening that are not being disclosed to the public, which means those vulnerabilities aren't out there.
Jay Mar-Tang: So I think it's important - when we talk about dynamic, whether that's the organization or the attackers, we need to focus on those TTP's. And what are those TTP's? It's a combination - all different attack vectors and ways of getting around the environment, which is very challenging. I'm not saying it's easy. But I think we need to take a holistic approach in that regards to figure out, well, it's not just a point in time. It's not just one thing. It's a numerous amount of techniques. Think of it this way - right? - when a good fighter doesn't rely on just a strong right hand. It's a combination of jab, jab, cross, right hook, uppercuts, dodging, head movement. All of that you can consider the TTPs in the environment. That's what we really need to focus on because everything - it's not black and white. There's a lot of grey.
Dave Bittner: You know, I think there's a tendency, perhaps even a stereotype. When people hear of automation, you know, they fear that they're going to get a firehose of information shot at them. How do you separate the signal from the noise here?
Jay Mar-Tang: I think the - what's important to focus on and what we need to realize is that we're not looking to exploit vulnerabilities in general, right? There are thousands and thousands and thousands of vulnerabilities, and vulnerabilities are just a means to an end. You can't exploit something unless the vulnerability exists and if the vulnerability is exploitable. So I think focusing on exploitable vulnerabilities and not only the exploitable vulnerabilities but the exploitable vulnerabilities in the environment itself helps eliminate all the noise because at that point, when we're looking at what has transpired in the network, we know with certainty that this happened.
Jay Mar-Tang: And then if we can prioritize based on impact which one should we focus on - if one domino - right? - maybe we have many dominoes here. Like, think of a table. We have multiple - a lot of dominoes. And if one domino caused thousands of others to knock over, we need to prioritize that first. So the fear of having all this noise doesn't really apply. It's not as applicable when we're talking about things that just apply from an exploitability perspective and very - and that's very unique to the environment. And then when we focus on the ones that matter based on the impact, we can get the biggest bang for our buck, the lowest-hanging fruit - right? - all those fancy terms - right? - that really mean we're going to really make our return on investment, from a time perspective, matter.
Dave Bittner: What are your recommendations for an organization who's looking to set down this path, you know, someone who doesn't have a whole lot of experience with this? What's the best way to get started?
Jay Mar-Tang: Well, I think any organization - the best way to get started is to have a solid understanding of their business. And by that, I mean we need to have an understanding of, what are the critical points in the environment? Now, everybody - if they're still using Windows, that's going to be domain controller. So we should focus very much on the Windows domain controllers as well as the privileged identities inside that environment. We need to understand then after that, what are the critical points of that business? What if, you know - if certain assets were to go down, does that mean certain doom for our business and our organization? So we need to focus on those next. And then the same way that we've built defense in depth around those critical assets, we now need to start validating those controls in depth with that same mindset. Start with critical, and then move our way out.
Rick Howard: That was Jay Mar-Tang, the sales engineering manager for the Americas at Pentera. And we'd like to thank Jay and Rick Doden, the Carolina Complete Health CISO, in helping us gain a bit more clarity about the function of security validation. And finally, a special thanks to Pentera for sponsoring the show. CyberWire-X is a production of the CyberWire and is proudly produced in Maryland at the startup studios of DataTribe, where they are co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Eiben. Our executive editor is Peter Kilpe. And on behalf of my colleague, David Bittner, I'm Rick Howard signing off. Thanks for listening.
