CSO Perspectives (Pro) 6.17.24
Ep 112 | 6.17.24

The current state of XDR: A Rick-the-toolman episode.


Rick Howard: In the early days of this podcast back in 2021, we published a Rick the Toolman Love Letter to this newfangled security tool called XDR. [ Soundbite of TV Show, "Home Improvement" ] You might have heard about it. The acronym stands for Extended Detection and Response, and I was gushing about how this tool might transform the modern-day security architecture. [ Laughing ] Back then, Gartner placed XDR at the beginning of the journey on its famous hype chart, just starting to climb the peak of inflated expectations, and I was jumping on the bandwagon to help inflate the hype. Two years later, July 2023, Gartner placed XDR on the back end of the peak, just starting the steep rollercoaster ride down toward the trough of disillusionment and forecasted five to 10 years before it reaches the plateau of productivity. Since this is the time typically when security pros start to lose faith in a product idea because the hype surrounding it hasn't matched existing products, I thought it was time to revisit the current state of XDR because I still believe that it represents the future security architecture that we all need. I don't want the InfoSec profession to lose sight of this potentially transformational tool just because it's not quite ready for prime time. So hold onto your butts.

Samuel L. Jackson: Hold onto your butts.

Rick Howard: In this Rick the Toolman episode, we're going to explore the current state of XDR. [ Soundbite of TV Show, "Home Improvement" ] [ Music ] My name is Rick Howard, and I'm broadcasting from M2K Cyber's Secret Sanctum Sanctorum Studios located underwater somewhere along the Patapsco River near Baltimore Harbor, Maryland, in the good old US of A. And you're listening to "CSO Perspectives," my podcast about the ideas, strategies, and technologies that senior security executives wrestle with on a daily basis. [ Music ] I can understand why the idea of XDR is sprinting towards the trough of disillusionment though. Most of the security platform vendors have a product that they call XDR like SentinelOne, Splunk, Microsoft, IBM, CrowdStrike, Cisco, Palo Alto Networks, just to name a few. But none of their explanations about what XDR is and what it does matches exactly. Gartner says that XDR is a, "unified security incident detection and response platform that automatically collects and correlates data from multiple proprietary security components." [ Soundbite of TV Show, "Home Improvement" ] That's accurate, but you could also say the same thing about SIEM tools, security information and event management tools. [ Soundbite of TV Show, "Home Improvement" ] I'm looking for something a little more descriptive. What makes XDR special? The subtle difference between a SIEM tool and an XDR tool is how the two technologies collect the data. With SIEM tools, the monitored system, let's say a Fortinet firewall, generates logs as part of its normal operation. The firewall administrator configures the system to automatically send the log data to the SIEM tool for storage and processing. The XDR tool is different. XDR administrators configure the tool to directly connect to the Fortinet firewall via an API, an application programming interface. The API allows XDR administrators to interrogate the firewall for the specific data they need, not just general-purpose log data, but any information on the system and transports the data to the vendor-provided XDR data lake for storage and future processing. Both methods allow, as Gartner says, a platform to collect data from varied sources, log data in the case of the SIEM tool, and any kind of data in the case of the XDR tool, but the evolutionary step of using APIs to collect the data is what makes XDR tools so transformational. It gives us some options. [ Soundbite of TV Show, "Home Improvement" ] [ Music ] Rick Doulton is an old friend of mine, the Security VP at Centene, and a regular contributor here at the N2K Cyberwire Hashtable. This is how he describes it.

Rick Doten: Like zero trust, it's not a thing. It is an approach, and so when someone says there is an XDR tool, then it's like, well, that's how everything works now. I mean, to me, it's about the difference between waiting for logs to be written and then consuming logs and reading logs and deriving things from those logs, as opposed to connecting directly with the API and having instant access into things that are happening and then sending alerts and be able to do responses based on that. I mean, that's the fundamental to it, and, you know, I talk to a lot of vendors and a lot of startups and all of the posture management tools, whether it's cloud posture management, data posture management, identity batch management, asset management, you know, runtime, all of them, this is how it works. It's like everything is just everything is API based, so let's just plug into the APIs and pull the stuff we want to pull and be able to set rules around it. [ Music ]

Rick Howard: In order to understand what I mean by this, it might help to understand that XDR arrived on the scene in 2018 by merging two different security tool sets, logging and antivirus. These were the prequels to XDR, you might say. So let's start with logging. Raphael Marty over at the Venture Beach website says that you can trace the origin of the logging piece all the way back to the original email SendMail program on BSD Unix in the 1980s. Eric Allman was building SendMail to be one of the first to implement the simple mail transfer protocol. He needed a way to log what was happening as the various pieces and parts of the SendMail system banged against each other. When he wrote the first syslogd program for BSD Unix to do that, he birthed the first logging system that we all know and use today. For the uninitiated, syslog stands for systems logging and the d stands for daemon. In the Unix world, daemons are little standalone programs that start up, do a task, and then disappear again until needed. In this case, syslogd receives a log message from a monitored system like the Fortinet firewall and stores it somewhere. As an aside, I did a Word Notes podcast on the word daemon back in 2020. For the nerd reference in the show, I highlighted one of my favorite sci-fi novels. It's called Daemon and was self-published by Daniel Suarez in 2006. Here is Suarez describing the book at a Google Tech Talk in 2009.

Daniel Suarez: So for those of you who haven't read it, I'll give you the high concept that I gave the Hollywood folks that seemed to work okay. It is the story of a highly successful online game designer who creates a program that monitors the web for the appearance of his own obituary, and when that appears, this program activates and cascades in activating other programs that begin to tear apart the systems supporting the modern world. [ Music ]

Rick Howard: As the years went by though, we started collecting logs on everything. The amount of stored data started to become unmanageable. In the late 1990s and early 2000s, SIEM tools emerged to help us corral the volume of messages. Instead of collecting logs separately for each application and trying to manually correlate the information with homemade databases, administrators could dump all the logs to this centralized SIEM system and use some of the vendor-provided functionality to scrub the data, but these seam systems were expensive. You had to provide local storage, hard disk space, to accommodate the volume of data. I remember it was a constant struggle to keep ahead of the demand. Every time we added more disk space, we filled them up with data quickly. The vendors, of course, made their money by selling more disk space, so they were only too accommodating to help us upgrade. But like I said, upgrades were expensive. InfoSec professionals were making trade-off decisions about what not to save to disk or how long we would store things before we would overwrite them. That was counter to what we were trying to do with a logging project in the first place. You wanted to use the logs to trace bad-guy activity over time. If your logs only went back three weeks or if your analysts needed log data on systems you weren't watching, that was a problem. It was also a major task to manage the storage system. Unless you were a Fortune 500 company or your vertical had strict compliance and reporting requirements, most of us couldn't afford to buy and maintain them. That all started to change when Amazon rolled out AWS in 2006. AWS made it possible to store all kinds of data relatively cheaply and they handled all the administration. Bonus. [ Soundbite of TV Show, "Home Improvement" ] There was another big problem though. All vendors used their own proprietary logging format. If security professionals tried to correlate their Cisco firewall logs with their Symantec antivirus logs, that represented a ton of low-level grunt work normalizing the data so that the SOC analysts could make sense of it all. By normalizing, I mean, they had to match the fields of the Cisco firewall data set to the fields of the Symantec antivirus logs. That normalizing task was, and is, an intermediate step that provides no value. Google site reliability engineers call that toil. We needed to do normalization to get to the thing that was valuable, but the normalization thing itself wasn't. The vendor community took a swing at addressing that issue back in the mid-2000s. They started working on something called the Common Event Format, CEF. According to Splunk's Steven Watts, it's a standardized logging format designed to simplify the process of logging security-related events and making it easier to interrogate logs from different sources into a single system. Today, many vendors use the CEF format, but other competing standards have emerged too, like JSON, JavaScript Object Notation, Windows Event Logs, the NCSA Common Log Format, or CLF, the Extended Log Format, ELF, the W3C Extended Log File Format, and the Microsoft IIS Internet Information Server. The logging landscape is still a bit of a Tower of Babel, if you get my drift. [ Soundbite of TV Show, "Home Improvement" ] The vendors can't seem to agree on what log files should look like, and so SOC analysts still execute a lot of toil to normalize the data. It's all in one spot, and the administrative burden is lower than it was back in the 1990s, but SOC analysts are still sifting through multiple piles of data haystacks looking for needles, and they spend a lot of time making the haystacks look the same. So, why is logging a prequel to XDR, you might ask? Well, SOC analysts sifting through reams of machine-generated log files looking for bad guys has been the standard operating measure since the 2000s. When XDR tools hit the market in 2018, the tool gave the InfoSec profession a chance to upgrade that process. [ Music ] The other prequel to XDR is the evolution of antivirus software and EDR, endpoint detection and response. In 1987, a German hacker and computer security expert named Bernd Fix wrote software he designed to remove the infamous Vienna virus from his system, thus becoming the first documented author of antivirus software ever written. Soon after, the notorious John McAfee created the first antivirus commercial product called VirusScan, and the InfoSec profession gained a must-have tool for the security stack. By the late 1990s, if you had any budget at all, your security stack had a firewall and an intrusion detection system at the network level, and at least one antivirus system deployed on every endpoint. When I was working in the Pentagon in the early 2000s, we had two deployed on each endpoint because we didn't trust just one to get the job done. The idea behind antivirus systems was that the vendors would write signatures for known viruses and malware designed to detect their deployment. Once detected, the engine could remove it or render it benign. It was a constant battle to get the latest signatures deployed in a timely manner, but in the late 2000s, a new technology emerged that looked at endpoint behavior to detect malicious code. Instead of just using signatures of known malware behavior, the engine watched the entire operating system looking for anomalies. If the endpoint started communicating with servers in Tajikistan when it previously never did before, that might be an indicator that something was amiss. This model allowed the system to detect previously unknown malicious code, a big benefit over signature-based antivirus. Anton Chuvakin was working for Gartner in 2013, and he gave the new technology its name, Endpoint Threat Detection and Response, ETDR. Now we all just call it EDR. According to CrowdStrike, EDR acts like your old TV's DVR, recording relevant activity to catch incidents that evaded prevention. While EDR was an innovative and disruptive technology, it was limited because it only dealt with the endpoint on the adversary attack campaign. It didn't see the entire picture. The Lockheed Martin research team had just published their now famous intrusion kill chain paper in 2010, and the InfoSec profession was just starting to get their head around the idea that bad guys had to navigate the entire kill chain undetected and unstopped in order to be successful. [ Music ] EDR was just one-piece InfoSec professionals could use on the kill chain. To have control and visibility on the entire kill chain, SOC analysts dumped the alerts from their EDR engines, as well as all the other tools in their security stack, into their SIEM tools. With that configuration, the SIEM, potentially, held all the data across the intrusion kill chain for any adversary attack campaign. The EDR only saw a piece of it. Let me pause for a second and remind everybody that the primary technology working in the background for all of this InfoSec activity for the past 30 years is logging. I say that because the next innovation, XDR, is going to introduce a better way. In 2018, I was sitting in the big keynote room at the annual Palo Alto Networks Customer Conference when Nir Zuk, the Palo Alto Networks founder and CTO, took the stage and introduced a brand-new product, XDR. He explained that in the past seven years, the company had demonstrated success using machine learning algorithms to detect previously unknown malware to a high degree of accuracy and precision. He suggested that it was possible to extend that idea to not only files stored on the endpoint, but also to behavioral data from the EDR systems and from the network data stored on security stack tools, or really, any device on the network. The idea was to collect all the relevant data intelligence into a giant data lake, run machine learning algorithms on it, and potentially discover previously undetected bad guys inside the network, but it was inefficient to collect that intelligence via general purpose log files, configuring each system to send everything it has in terms of log files to the data lake. Instead, he suggested it would be much better if a new tool, XDR, connected to each security tool in the security stack via an API and collected the exact intelligence it needs. Brilliant, but here's the best part. Since XDR uses APIs to connect to systems to collect data, it can just as easily send information the other direction. InfoSec professionals could use XDR to send updated configuration information back to the security stack. For example, let's say that the XDR tool has detected elements of the attack campaign run by the hacker group Magic Hound. According to Tidal Cyber, a startup that I advise, Magic Hound is an Iranian-sponsored threat group that conducts long-term resource-intensive cyber espionage operations, likely on behalf of the Islamic Revolutionary Guard Corps. The Tidal Cyber Intelligence Team says that the attack campaign uses 75 different techniques across the intrusion kill chain. Our SOC analysts can use the XDR engine to query all the tools in the security stack to determine if we have any prevention or detection rules already in place for those 75 techniques. If not, we can easily push an update through the XDR engine to do so, and that's the power of an API. You can't do that with a SIEM tool. Regular listeners to this podcast know that I'm a big believer in first principle thinking. When we published our book back in 2023, Cybersecurity First Principles, we said that if you assume I got the absolute cybersecurity first principle right, that is, reduce the probability of material impact due to a cyber event in the next three to five years, that there are logical follow-on strategies that you might pursue to achieve it. One of them is automation, something the InfoSec profession is not that good at today. But with XDR, that might begin to change. Using what we did for Magic Hound as a template, we could duplicate those efforts for every known attack campaign in the MITRE Attack wiki, some 150 the last time I counted. And if we could do that, that gets us a long way down the journey for one of the other potential follow-on strategies, intrusion kill chain prevention. [ Music ] There are a few reasons that Gartner has XDR careening down the peak of inflated expectations towards the trough of disillusionment. First, current vendor solutions mostly only work with their own products. The promise of using APIs hasn't met Nir's vision yet of connecting to everything. Second, remember the problem that SIEMs have with vendors not agreeing on a standard logging format? Well, that hasn't gone away. Every commercial security stack tool has their own unique format to store and transmit data. It's probably the main reason that current XDR tools don't easily connect to other vendor products, but in 2022, AWS and Splunk co-founded the Open Cybersecurity Schema Framework Project, the OCSF project, and most of the vendors that have an XDR product have signed on. [ Music ] I was talking to Milad Aslaner last year. He's the SentinelOne product manager for XDR, SIEM, and Threat Intelligence, aAnd I asked him if he agreed with my assessment that the XDR vendors are struggling to talk to each other.

Milad Aslaner: I think like part of the reason is -- goes back to the data schema challenges. I think the idea is great. We're going to bring in all these different data into a central location, and that will enable the teams to do best detection, response, and so on. What was challenging for many organizations like how do we first make sense of that data? It always goes back to the data problem. I think that's the reason why a year plus ago, I think it's now, where OCSF came into picture like with AWS, Splunk, and a number of others saying, hey, we need to come together as an industry to define a common data schema for security data. So today, OCSF is our primary data schema, which allows us to essentially focus on detections. But I think, as always, we can always, collectively as an industry, can do better to make life easier for our customers and also help them protect better against cybercrime.

Rick Howard: So let me double down on what you were talking about. That standard allows us to take telemetry from all the different, say, firewall vendors, you know, how about the networks, checkpoints, Cisco, you name them, and they all have their own little nuances about how the data comes across. The standard allows us to get it all in the same format. So analysts in XDR engines can process it more efficiently and make some decisions.

Milad Aslaner: That's spot on, right? So as part of OCSF is an open framework. Everything is outlined on GitHub, full transparency. There's an open community for it. There's a number of vendors in the security space that are active members, including SentinelOne, Splunk, AWS, et cetera, right? But that's exactly the goal, right? We want to provide a common data schema that makes life easier for security professionals.

Rick Howard: My third reason why XDR solutions are heading down toward the trough of disillusionment is that the way that XDR solutions have evolved, they are not offering permanent data lake storage like a SIEM tool would. The idea is to collect the data, find bad guys, and then discard the data. If you're looking to perform long-term analysis over a couple of years, current XDRs can't help you there. Today, you're still going to need a SIEM to do that, but that's a minor tweak to some future version of XDR. I think it could still happen. Lastly, machine learning algorithms running against a giant lake of XDR data hasn't really performed as I described above. It's been five years since Nir's announcement of XDR, and as far as I know, none of the current vendor offerings have found the Magic Hound attack campaign or any others for that matter. They currently just find anomalies, more things for SOC analysts to run down, more hay for them to sift through to find the needle. Today, if your SOC analysts are already overwhelmed, this is probably not that useful to you. [ Music ] So, what is the future of XDR? I agree with Gartner's forecast that it's probably five to 10 years until XDR reaches the plateau of productivity. Whether product managers fix the limitations described in the show is anybody's guess, but if you're like me, you see the potential of this new paradigm of using APIs to collect data from the security stack versus sending log files from the security stack to a SIEM somewhere. The collection part is not the single-most important reason to adopt the technology, but because APIs allow the ability to send configuration updates back to the security stack, that is a huge reason to adopt it, and we can help push the vendors in the right direction. First, encourage all your security stack vendors to join the Open Cybersecurity Schema Framework Project, the OCSF Project. In fact, make it a requirement for the next contract renewal. This just helps everybody. Second, encourage your XDR vendors to upscale what they're looking to detect. We don't need another alert about something that may or may not be important. We don't need more hay. We need them looking for attack campaigns like Magic Hound, suggesting prevention controls designed to prevent Magic Hound's success across the kill chain, and providing an easy button to send those prevention controls to the security stack. That's what I want in an XDR tool, and I think you should, too. [ Music ] And that's a wrap. I'd like to thank my colleagues Rick Doten, the Security Vice-President at Centene, and Milad Aslaner, head of product management at SentinelOne, for coming on the show and helping me understand how to think about XDR as the tool worked its way through the Gartner hype cycle. CSO Perspectives is brought to you by N2K CyberWire. Visit thecyberwire.com for additional resources that accompany this episode. I've added some helpful links in the show notes to help you do more of a deep dive if that strikes your fancy, and check out our book, Cybersecurity First Principles, A Reboot of Strategy and Tactics, for another deep dive on a lot of the topics covered in this podcast. And by the way, we'd love to know what you think of this show. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity. If you like the show, please share a rating and review in your podcast app, and you can also fill out the survey in the show notes or send an email to csop@n2k.com. We are privileged that N2K CyberWire is part of the daily routine of the most influential leaders and operators in the public and private sector, from the Fortune 500 to many of the world's preeminent intelligence and law enforcement agencies. N2K makes it easy for companies to optimize your biggest investment, your people. We make you smarter about your teams while making your team smarter. Learn how at n2k.com. Now here at N2K, we have a wonderful team of talented people doing insanely great things to make me sound good. I think it's only appropriate that you know who they are.

Liz Stokes: I'm Liz Stokes. I'm N2K CyberWire's Associate Producer.

Tre Hester: I'm Tre Hester, Audio Editor and Sound Engineer.

Elliott Peltzman: I'm Elliott Peltzman, Executive Director of Sound and Vision.

Jennifer Eiben: I'm Jennifer Eibin, Executive Producer.

Brandon Karpf: I'm Brandon Karpf, Executive Editor.

Simone Petrella: I'm Simone Petrella, the President of N2K.

Peter Kilpe: I'm Peter Kilpe, the CEO and Publisher at N2K.

Rick Howard: And I'm Rick Howard. Thanks for your support everybody.

N2K Team: And thanks for listening. [ Music ]