SOAR around the Hash Table.
Rick Howard: At my last CSO gig, our SOC director moved out early on the entire SOAR idea - this Security Orchestration, Automation and Response concept, this infrastructure-as-code theory that within nine months of the project's start, his team was able to reduce the number of alerts coming into the SOC from 1 billion every quarter to just under 500. In the process, he was able to repurpose some of his Tier 1 analysts from being console jockeys doing scut work, swiping left on meaningless alerts, to learning how to be threat hunters. Talk about a forced injection of training around detecting cyber-adversary campaigns. And that is the power of automation, the force behind why the IT folks have been pursuing this DevOps philosophy for 10 years and the reason that network defenders should get on board now.
Rick Howard: My name is Rick Howard. You are listening to CSO Perspectives, my podcast about the ideas, strategies and technologies that senior security executives wrestle with on a daily basis. In this episode, I talk to three CISOs at the CyberWire's Hash Table about the two problems a SOAR philosophy adoption might solve for you - the collapsing of the SOAR market into the SIEM market and vice versa and the fear that some CIOs have of letting their CISOs into the infrastructure-as-code game.
Rick Howard: The main reason that SOAR tools started to gain traction in the network defender world sometime around 2018 was the fact that the number of alerts coming in from the security stack that a typical SOC analyst had to process had exponentially exploded compared to what they were doing just 10 years ago. Like I said at the top of the show, at my last job, my internal SOC was processing some 1 billion alerts every quarter, and that was overwhelming. And we were not even that big compared to some of the internet giants like Microsoft, Amazon or Google or some of the big government outfits like the Department of Defense.
Rick Howard: I was talking to Kevin Magee about this. He is currently Microsoft's CSO for Canada. He and I worked together at Palo Alto Networks for a few years, not too long ago. And this is his first appearance at the Hash Table.
Kevin Magee: Microsoft sees 8 trillion threat signals a day. And every time I get a new update of the PowerPoint, the number grows. So we have to do this, or there's no way we can possibly keep up with the sheer amount of threat signals we're dealing with.
Rick Howard: Eight trillion messages coming into the SOC a day. That makes my 1 billion coming into the SOC every quarter seem almost trivial. To put that into perspective, a billion seconds adds up to roughly 32 years, while a trillion seconds adds up to 32,000 years. That's a lot of alerts. If you think of it in those terms, it's amazing that SOAR tools are not standard practice for every SOC on the planet.
Kevin Magee: The concept of SOAR basically exists to address the problem of finite resources, finite number of humans to review, analyze and really respond to infinite and growing number of potential threats. You know, how do we address these concerns around skill gap, around burnout in our industry and whatnot? And then how do we really just compete at cloud speed with our adversaries who are using those tools against us? So I think that's, you know, the way I sort of think about SOAR.
Rick Howard: But not everybody agrees. Rick Doten is the CISO for Carolina Complete Health and a regular at the Hash Table. He thinks that the SOAR tools are a waste of resources for security teams who haven't taken the time to configure the tools that they already have. In other words, they have installed so many tools incompletely that they need another tool to help parse out the noise.
Rick Doten: Go back to the source instead of adding something to add more complexity because I also want, you know, better visibility. By focusing on my instrumentation, I'm tuning it and adjusting it, and I'm using it because that's the other - my other kind of point against it is that it lets you not effectively use the tools you have. It kind of covers up for the fact that, well, I put in this email gateway. And I just, you know, left the default settings on. And it does - it blocks spam, and it helps, you know, find, you know, bad links and malware and stuff. You know, but I get all these extra things to it. So I'll add a SOAR tool that will kind of clean it up instead of looking at it and - how can I use it to its potential? So you have a lot of tools. You're using 20% of their potential because you don't want to dig into it and having something kind of pick up all the slack to kind of, like, normalize it, so a human doesn't get bombarded.
Rick Howard: My own view is that I prefer the centralization that a SOAR tool gives me. Yes, I may not have completely configured all the tools in my security stack that, if you recall from last episode, range in size from 10 tools for small organizations to over 200 tools for very large organizations. But to spend time on each individual tool, configuring the noise down to something meaningful, seems so inefficient when I can use a SOAR tool to automatically do it with software in a centralized repository where all the data flows into. To me, it is a matter of convenience. So let's agree to disagree on this one.
Rick Howard: But for the sake of argument, let's assume you desire a SOAR capability. Do you need a SOAR tool, like from a vendor, or can you just roll your own? Rick Doten says that if you have good people trained well, they are already doing that.
Rick Doten: We have all these tools because we don't have enough and enough good people in the industry because if you have, like, really good people, then they're not chasing tools. They're figuring out how to automate stuff and scripting things and leveraging the things they already have.
Rick Howard: Kevin Magee disagrees.
Kevin Magee: I think our industry has always had a group that wants to build everything themselves. And that's great because that's what, you know, spawned the hacker culture and made us we are. But at some point, you know, it doesn't make sense to really spend time, you know, building your own. You know, I think about something as simple as racks and data centers. You know, back in the day, it was very expensive to buy pre-configured racks, or you could build your own. But is that the best use of your time, really, when someone else has already got it figured out? Contributing to increasing the security posture of your organization, reducing threats, you know, looking at new ways to gain efficiencies out of the tools, I think, is a much better use of our people's time.
Rick Howard: Which brings us to the SIEM, or Security Information and Event Management systems. These tools showed up in the marketplace circa 2006 as on-prem analysis engines that, according to Stephen Gailey over at the Cybersecurity Magazine, quote, "combine security event management with security information management," end quote. In other words, security stack alerts plus intelligence.
Rick Howard: But even back in those early days, long before we had the Microsofts of the world collecting a trillion messages a day, we couldn't stuff enough information into them to make them that useful. Since they were on-prem, they never had enough hard drive space. People like me kept having to make decisions about what not to collect in the SIEM. And for the stuff we did collect, we had to decide how long we would keep it. Like I said, not that useful. That situation got better as some vendors started using the cloud to store everything sometime in 2017. Suddenly, network defenders had infinite hard drive space in the cloud at a relatively cheap price. In the cloud, they could store everything they wanted. The result is that most SOCs have SIEM tools of some sort. The decision that comes to mind, then, is a data flow decision. Do you send security stack telemetry to the saw tool first to pass out the noise and then dump everything into the SIEM? Or do you dump all that telemetry and intelligence straight into the SIEM and let the SOAR tool parse the SIEM to reduce the noise? I was talking to Kevin Ford about this. He's the North Dakota State CISO and a regular Hash Table contributor. He says that it's a little bit of both.
Kevin Ford: It does both for state traffic, right? It does both. It'll go into the SIEM, and it'll go into the SOAR at the same time. And then if the SIEM turns and does its thing and if it finds anything interesting, it'll pop - also pop a warning into the SOAR. But for the K-12, the counties and the cities and the - and higher education, we're getting information directly from the asset, and that's being popped right into the SOAR, right into our data, like, that the SOAR runs on.
Rick Howard: The interesting question is, why do you need a SOAR tool at all? Why can't the SIEM do everything? After all, SIEMs are just big log collector databases anyway. Network defenders have been able to write scripts for them from almost Day 1 of them hitting the marketplace. And like the other tools in the security stack, most of us are only using a small percentage of this capability. Why then do I need an additional SOAR tool to automate the process when I can leverage a tool that I already have? Here's Rick Doten again, the Carolina Health CISO.
Rick Doten: All right, well, you have a SIEM. You have stuff going into the SIEM. You know, you can script - and also, the other toolings that you have - you know, I can do stuff like an email response. You know, we have things to be able to script those out. People use 10% of what a SIEM can do.
Rick Howard: The truth of the matter is that SIEM tools have always been hard to automate. You could receive telemetry from many kinds of security tools, but the internal scripting language that came with the SIEM has been notoriously obtuse.
Rick Howard: In one of my previous CISO gigs, I hired a full-time guy to be the SIEM programmer. After a year of work, we had little to show for the effort - frustrating. That situation left an opening for the disruptor SOAR tools to come in and fill the need, which they did. It left the SIEM vendors scrambling to stay relevant. The result is that the two capabilities are collapsing into each other. SIEM vendors are way better today at SOAR stuff now, and SOAR vendors work more and more seamlessly with the SIEM vendors. Here's Kevin Magee again.
Kevin Magee: Integrating SOAR, integrating other tools really to make the tool sentinel more than sum of its parts. I'm not sure if there's a term coined for next-generation SIEM or whatnot, but I'm sure it's coming at some point. But I think that's where we're headed. And cloud scale's really allowing us to do that, something we'd never, ever done before.
Rick Howard: So far, all we have talked about in terms of SOAR capability is the process of automatically separating the signal from the noise. For me, just doing that is probably worth the investment. But SOAR tools have the potential to do so much more today. What I'm talking about is SOAR's built-in ability to interface with the tools within your security stack. With a SOAR tool, you not only collect and process telemetry from your security stack. You also have the ability to update your security stack with new configurations - with code. This is essentially DevOps, and since we are updating the security stack, it's really DevSecOps. Instead of logging into each tool on the stack individually to complete the day's configuration changes, you can configure all the changes in the SOAR tool and then push the button to execute. By flipping this process, a whole world of opportunities open up to the infosec team. Here's Kevin Magee again from Microsoft.
Kevin Magee: How do we start to rethink and redesign our security posture to think about the threat actor? And we're seeing a lot more human-operated ransomware campaigns. There's a lot more thought going into each step of the attacks. So they're taking a different approach. Protecting against the threat actors most likely to attack you is probably a better way to set up your security than to focus on specific tools. Yet our industry seems to be fixated on those specific tools. So I see SOAR as a chance to rethink that and to go back to first principles on how we want to design our security posture.
Rick Howard: What Kevin is talking about has been the entire premise to the "CSO Perspectives" podcast. I've been talking about establishing some first principle strategies in cybersecurity and thinking about how to pursue them.
Kevin Magee: More importantly, from my perspective, is SOAR is an opportunity to really rethink how you build your security posture because we design for threats around - based on tools now. And I think we really need to think about redesigning our security posture based on threat actors. When you hear someone describe a cyberattack, they usually use the tool's name. So it was a Riak attack. And I think that perpetuates a challenge in our industry because now we're always focused on the tools. And I equate that to focusing on the arrows when we should really be thinking about the archer. The archer is the real threat.
Rick Howard: In season one, episode six, I go into depth about what a first principle is and why we need to use the idea to develop our cybersecurity strategies. Also in season one, episode eight this time, I talk about intrusion kill chain prevention as a foundational strategy. If you haven't listened to those, I recommend that you go back and catch up. And Kevin Ford, the North Dakota state CISO, thinks you can use our tools to automate those processes.
Kevin Ford: Who would understand the kill chain? And then you prebuild - you know, prebuild the responses. I mean, that's the dream, right? That should, I think, be the dream for every everyone using SOAR - is making sure that you can - you've got a framework. Now, again, I like MITRE, and I like the ATT&CK Framework and the understanding of the kill chain. But you have a framework. And you can disrupt the attack kill chain through meaningful interactions between your SOAR tool and the information systems that are potentially victims to those sorts of attacks. You know, that's the beauty of SOAR, but that's also the beauty of, you know, this - the understanding of the attack kill chain and having an established framework.
Rick Howard: The one pushback you get from CISOs on this idea is that the CIOs of the world are generally reluctant to turn the management of the security infrastructure completely over to code, especially since they have ultimate responsibility for keeping the network up and running. And they don't want to run the risk of a bunch of security nerds screwing something up.
(SOUNDBITE OF MUSIC)
Rick Howard: Kevin Magee thinks that when they hear automation, they conjure up Skynet from the "Terminator" movies that will somehow wake up and take over their operation. But there is a difference between automation and fully automatic. We are talking about automating processes here, not building autonomous systems.
Kevin Magee: We have jokes about Skynet, you know, taking over, whatnot when we have SOAR discussions. And when comments are made like that, it sort of brings out the concerns that the customer may have around, you know, I'm not interested in fully automating. And that's maybe what they think SOAR is - is fully automatic. When you talk about fully automatic, it makes people uneasy about sort of writing code and then having it execute. So there's a balance there versus automated and automatic. I think that we need to establish comfort levels with people. And maybe that's a human, you know, gate process, where a portion kicks off automated, then a human looks at it and then makes a decision what to do next or whatnot. Doesn't have to be fully automated. But removing a lot of that redundancy makes the job of analysts much better.
Rick Howard: One thing to consider about SOAR tools and just the general philosophy of DevSecOps is that the network defender community has been slow to the game of automating their processes and infrastructure. At this point, we are at least 10 years behind the IT community, who started working towards the DevOps concept way before 2010. SOAR tools, whether homegrown or vendor provided, gives us an opportunity to dip our toes into the DevSecOps waters inside our own SOC world without too much risk of affecting the overall performance of our internal networks. We can begin to automate our intrusion kill chain prevention strategy, which, by the way, is the only choice we have if we want to get ahead of our cyber-adversaries, who have already automated all of their processes. But it doesn't stop there. SOAR tools provide the opportunity to help automate your incident response processes, your cyberthreat intelligence sharing agreements and could even be used to facilitate the monitoring and maintenance of your compliance infrastructure. And those are just the obvious things we could automate. Once we get comfortable in these environments, I expect that the innovation we get from our infosec teams will astonish us. But we need to get going, so let's take the first step.
Rick Howard: And before I leave you, let me recommend a couple of books from the Cybersecurity Canon project. These are both Hall of Famers. The first one is "The Phoenix Project: A Novel About IT, DevOps, And Helping Your Business Win" by Gene Kim, Kevin Behr and George Spafford. And "Site Reliability Engineering: How Google Runs Production Systems" by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy. "The Phoenix Project" is really about the philosophy of DevOps, and the "Site Reliability Engineering" book is about how Google does it.
Rick Howard: And that's a wrap. If you agreed or disagreed with anything I have said about SOAR technology or really anything, hit me up on LinkedIn, and we can continue the conversation there. Next week, we will be talking about, where in the organization should the office of the CSO report? You don't want to miss that. The CyberWire's "CSO Perspectives" is edited by John Petrik and executive produced by Peter Kilpe. Our theme song is by Blue Dot Sessions, and the mix of the episode and the remix of the theme song was done by the insanely talented Elliott Peltzman. And I am Rick Howard. Thanks for listening.
