You can add new features, just secure the old stuff first.

Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Lee Christensen: A lot of times people focus more on adding new features rather than, you know, securing and modernizing their older stuff, or just completely removing older things that shouldn't be used anymore.

Dave Bittner: That's Lee Christensen from SpecterOps. He and his colleague Will Schroeder are joining us today to discuss their research, "Abusing Active Directory Certificate Services."

Lee Christensen: Active Directory, first of all, is a Microsoft technology, and it's used primarily to set up, configure, and administer users and computers inside of a network. So, for example, users log in with their username and password, a computer somewhere has to store that, and in a lot of networks that's an Active Directory. I'd say like 90 percent of, you know, probably more than 90 percent of businesses use Active Directory underneath. And so, it manages authentication and also controls access to things as well, and then allows administrators to configure computers.

Lee Christensen: And Active Directory Certificate Services is Microsoft's public key infrastructure. And that means a lot of things. That means that it does several different things. But for us, what's really interesting is it adds a new form of authentication. This is kind of like our our area of research, is authentication. And so rather than just using, like, a password to log in, you can use something that's called a certificate. And it's just like a special kind of file that the Active Directory Certificate Services can issue to a user and then using – well, to a user or a computer. And if you have that certificate, you can then log in to Active Directory using that certificate file.

Dave Bittner: I see. So basically, it's a way to establish machines on a network within an organization to say, we know who you are, you've verified you have this legitimate certificate, and so, you know, we're good here. Let's go on with our communications.

Will Schroeder: Exactly.

Dave Bittner: Yeah. So, in your research, you point out that there was a particular sentence that stood out to you in the Active Directory technical specification. I'll just read it here real quick. It says, "In the case of DCs, the external authentication information that is used to validate the identity of the client making the bind request comes from the client certificate." How did that start you off down on this pathway? Will, can you fill us in there?

Will Schroeder: It's actually – I'll kind of toss it to Lee, just because Lee was the one reading the specification and then came to me in early January and said, hey Will, did you know that you can use this certificate stuff? And, you know, we just kind of started diving into it. So he piqued my interest because we've both been researching Active Directory for a while, and I was like, oh, yeah, I've been meaning to look into that certificate thing or that PKI thing in Active Directory. So that was kind of the spark in Lee's mind that drove both of us down this very large rabbit hole about Active Directory Certificate Services and everything that it entails.

Dave Bittner: And what was it about that made you raise your eyebrows, that grabbed your attention?

Lee Christensen: I think in particular, it was, I mean, you've heard about logging on with a username and password. We've done a lot of research previously with Kerberos. We've heard about NTLM. But I'd never played around with this thing, you know, that says you can log on with certificates before. And that was – as I was learning about, I was reading the section and the specification about LDAP authentication, and I just never actually tried that before. And so I wrote up a quick PoC of authenticating to LDAP using a certificate, and I just shot that over to Will, and I was like, hey, like, did you know you could do this? And that kind of got the ball rolling, and here we are now.

Will Schroeder: With a lot of the research that we end up doing, we don't set out ahead of time thinking we are going to exploit this thing or find these security issues necessarily. It starts from kind of a point of curiosity, kind of like you mentioned, of, oh, this is kind of interesting, let's start diving into this. And then usually out of that process, we end up finding something interesting from a security context.

Dave Bittner: Well, let's go down that path together. Can you walk me through – what did your exploration look like, and at what point did you start to realize that perhaps there were some security ramifications here?

Lee Christensen: Yeah, I would say, so, as we started looking into this, we definitely built on top of, you know, a lot of other people's work. So Benjamin Delpy, the creator of Mimikatz, he's done a lot of research into this previously. And as we looked – as we started diving into this before, too, we noticed, scattered throughout the Internet, people had touched on Active Directory Certificate Services here and there, talking about different types of abuse, different types of escalation techniques. And it was just like – it was kind of hidden in different corners of the Internet. And so, part of our research was just kind of looking at what people had already published and aggregating that, but also kind of distilling it all in a way that makes sense from an offensive perspective or from like a penetration testing and red-teaming perspective. So kind of consolidating that all together.

Lee Christensen: And as we did that, we kind of applied this to, you know, we found several different categories of attacks and tradecraft that we could do. So, different forms of privilege escalation inside of Active Directory, as well as ways you can persist in on a machine or on a computer, or ways that you can, you know, different ways to steal credentials or maintain access to credentials inside of a network.

Will Schroeder: As well as kind of finishing off with – that was kind of near the beginning of the research, and like Lee said, you know, we would start – we always try to do our due diligence and just try to research on what existing work is kind of out there and build upon that and aggregate everything and kind of iterate on it. Near the end of the research, we also realized, well, we had this kind of pipe dream for a long time of saying, if we're able to steal the private key for a certificate authority, could we then forge our own kind of golden certificates as well? So that was kind of the final cherry on top, I think, where we were able to do this domain persistence and actually build the tooling and prove that it was possible to forge these on unrevokable certificates as a new user in the domain.

Dave Bittner: So, is this a matter of – when you're going through that aggregation part, when you're seeing the work that others have have done? I mean, what was it a matter of lots of people had found, you know, little bits and pieces here and there and had said, oh, this is interesting, but as far as you all knew, no one had really rolled it all together to get the, you know, sort of that old saying of the whole being greater than the sum of the parts.

Will Schroeder: That's true. There's, specifically, there is a lot of stuff that we cover in the paper from attack or tradecraft standpoint. Like Lee had mentioned, we have user credential theft without touching LSASS, we have machine persistence, domain escalation paths, and domain persistence. So most of the stuff that we had found online tended to focus on one or two specific domain escalation scenarios. We cover eight in the paper, and one of them is completely novel. The most severe one is definitely novel that Lee had discovered. But most of those have been covered to some degree, but people would just maybe touch on one of them, like one issue, one misconfiguration. What we started to realize is that if you don't bring all these pieces together into one place, if people run across, you know, this one specific misconfiguration – we title those ESC 1 through 8 in the paper just for shorthand – that if they fix that one thing, they might think they're secure. But there's so many of these different just escalation scenarios that the vast majority of environments that we've run across have had at least one of them. But there's a lot of value in bringing them all together so people can holistically say, you know, is my Active Directory Certificate Service installation misconfigured or not?

Lee Christensen: I'd also add on to that saying, you know, as we saw some of this existing prior work and as we started looking at real networks – so we're consultants. Our company is a consulting company, and we perform penetration testing and red teaming. And in our role, we're helping expand – as technical architects, our role is to help expand our capabilities. And so, as we were performing assessments, we were trying out this new technique to try and kind of formalize our process around looking for it. And we realized, too, that there's a lot of – like, it's a very complex system and there's a lot of gotchas that come along with that. So, like, you know, maybe the public stuff had documented a scenario where it's vulnerable, but if these other three configurations are in place, it's not going to work. Or if it's configured this way, you have to do, like, another thing. And so there's some added complexities that, as we started applying this to real networks, we realized, oh, this isn't necessarily as simple as a lot of people portray it. And so, for a lot of these techniques, we state several requirements that have to be in place beforehand, before these are actually vulnerable.

Dave Bittner: Can you take us through – you mentioned that one of the things you discovered was novel and also severe. Can you walk us through that one as an example of the sorts of things that you all are describing here in your research?

Lee Christensen: Yeah, so this is escalation vector eight in the white paper. And what it is, is – so, Active Directory Certificate Services has a function called "enrollment," and so a client machine – so, like, I'm a user – I can basically say to Active Directory Certificate Services, hey, I would like a certificate,  and AD CS – Active Directory Certificate Services – will send me a certificate. And then I can use that certificate to authenticate. 

Lee Christensen: While that's fine, usually the protocol that where that occurs, it runs over DCOM, so it's an RPC-based protocol, basically. And what we found is there's an additional server role that many environments have enabled – it's called the web enrollment web interface or something like that. But when that's installed, there's an HTTP endpoint that gets enabled on the AD CS server, and this endpoint uses NTLM by default and doesn't use HTTPS. And why that's interesting to us is now we can perform NTLM relay to that endpoint and request a certificate.

Lee Christensen: And when you combine this with some other tradecraft that's out there – so, three years ago, I discovered a technique called – we created a tool called Spool Sample. It's a great name, but the whole point of the tool is it allows me to coerce a machine to authenticate to me. So I can basically say to, like, a domain controller, hey domain controller, please authenticate to my machine. And when that domain controller authenticates to me, I can use NTLM relay and impersonate the domain controller, and access the AD CS HTTP endpoint and request a certificate as the domain controller. And this has since been expanded on more recently. So if you've seen in the news recently, there's an attack called "PetitPotam," and it's basically another coerced authentication technique. So you can use that – instead of using Spool Sample, you can use that tool instead, and that'll coerce authentication, and then you can relay it to the AD CS HTTP endpoint, impersonate a domain controller, and get a certificate as the domain controller. And once you have that, you can basically log on as a domain controller and do things like steal users' password hashes and then log on as those users.

Dave Bittner: Wow. How serious do you consider these findings to be? In terms of, you know, when you balance it against the number of organizations who are using this and the ease at which people could take advantage of these vulnerabilities you all are describing here, you know, how big of a flare are we sending up here?

Lee Christensen: I'd say – so, for escalation eight, I feel like that's a pretty – I personally feel it's a fairly severe vulnerability and I don't think it's that trivial to exploit. There are some preconditions around it, but it by default, if you install Active Directory certificate's services with that HTTP endpoint, like, your your environment's vulnerable. And every environment that we've been into so far that has that HTTP endpoint enabled has been vulnerable. So, I personally feel it's pretty severe. 

Lee Christensen: In regard to a lot of the other tradecraft and, like, the credential theft and the persistence techniques that we talk about in the paper, those are important to be aware of. I wouldn't say they're quite as severe, because it's the intended functionality of AD CS. So, like, this is a legitimate technology that, you know, it's there for a purpose. And like a lot of useful technologies, they can be used for good, but they can also be used by attackers to achieve their own purposes. So I wouldn't necessarily state everything is, like, a severe vulnerability, but it is things that network defenders need to be aware of, because they need to have detections in place for certificate abuse

Lee Christensen: And then they also need to add in steps to their incident response process to account for certificate-based tradecraft. So, a good example of this is, certificates are valid by default for – it's either one to two years. And computers and user accounts, you know, typically will have a certificate on a machine. And so, if an attacker compromises that machine for thirty seconds, the attacker could steal those certificates and then basically be able to log on as that user for up to one or two years after the compromise. And that survives a password reset, that will survive a system wipe, because it's a completely separate form of authentication. So, it's not a password, and because it's not a password, even if the password gets reset, the certificate is still valid. And so, incident responders and defenders need to take steps so that when a machine's compromised, they also look at Active Directory Certificate Services to see if – to revoke issued certificates to that compromised machine

Dave Bittner: I see. So what are your recommendations there? I mean, there's a lot in this research, and I recommend people check it out. There's really – you guys have covered a lot of ground here. But from a basic point of view, you know, what are the best steps to protect yourself if you're running these services?

Will Schroeder: Oh, there's a lot. So, in the white paper – and we actually have a we have a short link for the white paper for any listeners that want to check it out. It's cutt.ly/preowned, all one word. It's also on the SpecterOps website under the research and development section. We have forty pages of defensive and preventive guidance, like in the white paper. So, the issue with this or the one of the problems is that because there's so many aspects of the attacker tradecraft, that there's not just one simple fix for one thing. There's prevention guidance that we cover, of ways to kind of secure and lock down some of these things. For example, locking down the NTLM relayable endpoint – that HTTP enrollment endpoint that Lee had mentioned. Or, you know, things like treating certificate authority servers as, like, tier-zero assets and things like that. 

Will Schroeder: There's also detective guidance. So, for things that you can't completely stop, how do you, for example, investigate if somebody or a compromised user had requested a certificate maliciously, you know, someone had taken over their account and stolen their credentials by doing a malicious enrollment. It's so expansive for the different things that we cover, that we have, you know, prevent one through eight and detect one through eight, or something like that as well, as well as architectural guidance for how to structure Active Directory Certificate Services.

Will Schroeder: So, it's pretty expansive with the things that we touch on. So there's – I think just at a super tl;dr high level, it's, you know, treat these servers like they are tier-zero assets, and then come up with playbooks for your organization to investigate certificate enrollments and things like this, as well as actively auditing your Active Directory Certificate Services environment to see if any of these misconfigurations are there for the escalation scenarios. And for that case, we have a public tool already called PSPKIAudit. It is on the github/ghostpack repository, that organization. So that is a defensive-oriented tool that we released along with the research about forty days ago or, you know, a little over a month ago. And those will enumerate this escalation one-through-eight scenarios. And then there's also advice in the GitHub repo and the white paper and how to mitigate each scenario.

 

Will Schroeder: And we are also releasing an offensive tool at Black Hat here coming up pretty soon called "Certify." So we kind of did a self embargo for the offensive tooling to make sure that defenders could try to audit and try to tune everything up and get everything mitigated as best they could.

Dave Bittner: So, if you're running Active Directory Certificate Services, it's worth another look. Don't assume that what you've got is properly configured or secure.

Will Schroeder: Yeah, we definitely agree with that statement, because many of the environments that we've looked at, they're run by very smart people and by very good Active Directory architecture teams, but this is almost like – Active Director Certificate Services and PKI is almost entirely separate domain of knowledge. And some of these misconfiguration can be very subtle. It can be one bit flipped, literally one bit flipped on one certificate template that can cause, you know, compromise of the entire environment or something like that. So, and again, a lot of these pieces can kind of stack together to where maybe even an Active Directory Certificate Services administrator wasn't even aware of all the possible ways it could be misconfigured. So if you are running AD CS we highly recommend auditing for the escalation scenarios at a minimum. But definitely check out the white paper for the other attacker, tradecraft, like credential theft and things like that.

Lee Christensen: I always like to compare Active Directory Certificate Services to like the old big boiler that you have running in the basement that like, it's been working fine, but it's been there for probably, you know, twenty years. And as long as it's working, like the network is going to keep running. But you should probably get it inspected because there's probably some things wrong with it. And, you know, just given that it's so old, it's very likely that there's been misconfiguration over the last twenty years, like Will had mentioned, in like 80 to 90 percent of the networks we've looked at since starting this research, they've been vulnerable, like, from domain users to low-privileged to domain admin, like a complete takeover of Active Directory. So, for us, like, we know, based off of real world experience, that this is misconfigured. So using PSPKIAudit to help audit for these things will definitely – organizations definitely should focus on doing that.

Will Schroeder: I will say that we one common question we get is, did you report this to Microsoft? We did. So, we reported everything, all the tradecraft and everything, which we assumed probably wasn't going to be a fixable situation or wasn't going to be a scenario they really made changes, because most of the escalations and other types of things for this tradecraft tend to be post-install misconfigurations. So it would be administrators, like, accidentally misconfiguration something in an environment. So those scenarios tend to not be fixed by Microsoft. But that one escalation that we mentioned with the relay to the web enrollment endpoint is pretty much out of the box, and we do believe it should have been fixed. We did report it to Microsoft, you know, several months ago, and they came back with a "this finding is valid, but we are not going to fix it" type of determination. So, in case anyone was wondering, we did try to do our due diligence and we reported everything before we even published anything. And because of that and because we believe these misconfiguration are so widespread, that's why we kind of did the self embargo of releasing all the information and defensive between first and waiting forty-five to sixty days to release the offensive tooling.

Lee Christensen: I'd add on to that too, kind of a more general problem the industry as a whole has is just either deprecating old technologies or modernizing older technologies. And, you know, this specific escalation technique that the relay to the HTTP endpoint – it abuses NTLM authentication. And that's like an old protocol, and our industry knows it's insecure, but despite that, it's been over twenty years now since we knew it was – since we've been abusing it, it still hasn't been deprecated. And there's been very little, I guess, coercion by Microsoft to get people off of using NTLM. So like, they actually issued a KB article in response to the Active Directory Certificate Services abuse that's going on in response to our paper and the PetitPotam attack that's going around right now. And they said, like, we've talked about, the KB article basically said, you know, we've mentioned this previously, like you need to work on disabling NTLM, and like that's been said repeatedly over time. But I think once Microsoft kind of takes that stance of, they're going to disable NTLM, that's when people are actually going to be protected because they kind of need it. That's how you're going to achieve maximum coverage across the entire industry. And once they make the change, rather than telling customers, hey, it's like you have the burden of knowing that you're vulnerable and then also configuring it so that you're not vulnerable. We think it should be fixed out of the box, and more generally NTLM itself should be deprecated in networks.

Dave Bittner: It's kind of like your analogy that you used earlier about the, you know, the boiler in the basement, and it's as if the building were built around the boiler and it's, you know, it's almost – to get it out and replace it, you have to shut it down, cut it up into pieces haul it out, you know, get a new one in there. And boy, that's a lot of work when everything right now is just running fine, right?

Will Schroeder: Exactly.

Lee Christensen: Yeah. And that's kind of going back to the whole deprecating and modernizing technology. Like, a lot of times, product vendors – it's not specific to Microsoft – but in general, people focus more on adding new features rather than securing and modernizing their older staff are just completely removing older things that shouldn't be used anymore. So, and the same thing goes with, you know, organizations themselves, like when they're coming to secure a network, like, network defenders, you know, they may try and get the latest and greatest product to help secure their network, but doing, quote unquote, "the basics" of disabling and NTLM and, like, doing network segmentation isn't always prioritized as much. And so, and those will have a lot bigger bang for the buck than, you know, maybe purchasing the latest and greatest security product.

Dave Bittner: Our thanks to Lee Christensen and Will Schroeder from SpecterOps. The research is titled, "Abusing Active Directory Certificate Services."

Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Tre Hester, Elliott Peltzman, Puru Prakash, Justin Sabie, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.