CSO Perspectives (Pro) 8.16.21
Ep 55 | 8.16.21

Orchestrating the security stack and cybersecurity first principles.


Rick Howard: In March of 2003, I was the commander of the Army's computer emergency response team or ACERT. The internet was just really taking off then. Wikipedia just launched a couple of years before. Apple had launched iTunes that year, but we were still four years away from seeing our first iPhone. In the military, we were still trying to figure out what cyber operations meant and every organization that could spell cyber correctly three times out of five, thought that they should own it. 

Rick Howard: One of my ACERT responsibilities was to coordinate offensive and defensive cyber operations for all of the army cyber stakeholders, like intelligence, networking, law enforcement, legal, information operations, and many others with our sisters and brothers in the joint world. You know, the Air Force, the Navy and the Marines. 

Rick Howard: These were the Title 10 forces, as they say. And my job was to make sure that whatever they were doing, didn't step all over what the Title 50 cyber forces at the National Security Agency and the Central Intelligence Agency were doing. Title 10 and Title 50 referred to the chapters in the United States Code that provides, among other things, the laws governing the armed forces and their use, Title 10, and things like spying, covert operations, and espionage, Title 50. Many people probably don't know that spying and espionage, Title 50, are things primarily reserved to the American spy organizations. Title 10 forces mainly fight the nation's wars. There are some exceptions, but on the whole, this is the general division of labor. In theory, what this means is that the army doesn't do espionage missions unless it's working directly for the NSA, and the intelligence community doesn't fight wars, unless they are directly supporting the military. 

Rick Howard: I mentioned all of this because during this time, early 2003, the United States and some of its allies were about ready to launch the invasion of Iraq. In preparation for that event, the Army's cyber stakeholders realized that we were caught flat-footed. Previously, we had divided operational control of the Army's cyber assets into various regional CERTS, or RCERTS, North America, South America, Europe, Pacific and South Korea. But we had no presence in Southwest Asia, or SWA. 

Rick Howard: And we needed one. So we built one lickety split, recalled a bunch of reservists to manage and shipped them all out to the sandbox in time to support the invasion. Immediately, the RCERT team noticed several continuous, low and slow probes of the RCERT SWA electronic perimeter coming from multiple locations and countries in the Middle East. That couldn't be good. We began to worry that whomever, those bad guys were, might be gearing up to degrade or dismantle this fledgling network designed to support the tanks and the infantry when they crossed the line of departure on H-Hour. We needed a plan to counter that contingency. We basically went into stealth mode. 

Rick Howard: We orchestrated a plan across all Title 10 interested parties, where at the push of a button, we switched the entire RCERT SWA infrastructure to new domains and IP addresses. Essentially when H-Hour arrived, the RCERT SWA infrastructure went dark from the perspective of any outside entity trying to keep tabs on us. Internally, we were fully functional, but to the outside world, RCERT SWA disappeared off the board. Just like a Klingon Bird of Prey using his cloaking device. It didn't last long, maybe a day before they found us again. But we knew that going in. Our goal was to cause confusion and disorientation to whomever might want to cause the Army harm at the beginning of the war. 

Rick Howard: Now I love this story because it highlights a capability that all network defender organizations need, and most don't have, orchestrating the security stack. In other words, deploying the policy and strategy to the operational equipment on the ground in real time. 

Rick Howard: My name is Rick Howard. You are listening to CSO Perspectives, my podcast about the ideas, strategies, and technologies that senior security executives wrestle with on a daily basis.  

Rick Howard: You may be asking yourselves, just why do we need orchestration again? Let me lay out some Readers Digest history for you. I've mentioned this past history in previous shows, but in the early internet days, say the late 1990s, orchestration wasn't a problem. We only had three tools in the security stack: firewalls, intrusion detection systems, and antivirus systems. When we wanted to make a change to the policy. We manually logged into each tool and made the change. Fast forward to 2021, and our environments have morphed into enormously complex systems of systems deployed across the multiple data islands, hybrid cloud, SaaS services, internal data centers, and mobile devices. Orchestrating the security stack for our first principal strategies like zero trust, intrusion kill chain prevention, resilience, and risk forecasting, across all those data islands in some consistent manner with the velocity is exponentially hard to do compared to those early days. And truth be told most of us don't do it very well.  

Rick Howard: So what are my orchestration options? The first to consider is DevOps and DevSecOps. Back in 2003 when Google was still nothing but a search engine, they decided to give the task of network management to the developers. Remember the industry, didn't get the DevOps name for what they were doing until seven years later in 2010, but Google in 2003 pioneered the concept of infrastructure as code. Instead of technicians, manually logging into network devices to update configurations, Google's site reliability engineers automated the low-level task or toil as they call it. 20 years later, they are one of a handful of internet giants that dominate electronic commerce. And by the way, the other is like Netflix, Microsoft and Amazon also adopted this DevOps model early. Now, perhaps there's a repeatable pattern there, but what do I know? I'm just a lowly CISO.  

Rick Howard: Innovative startup companies, the ones that came up with the DevOps name in 2010, realized that the way they could distinguish themselves in the marketplace was to deliver their services from a SaaS model using infrastructure as code. Now two Cybersecurity Canon Hall of Fame books talk about this history and how to think about the philosophy. "Site Reliability Engineering" from the team at Google, in "The Phoenix Project" by Gene Kim.  

Rick Howard: With this approach as part of the app development process practitioners build into the system the way to manage the security stack at scale and velocity. If you want an example, check out Netflix's "Chaos Monkey Research". If you want to get a lesson on how to think about a hardcore infrastructure as code resiliency strategy, it'll blow your mind.  

Rick Howard: A second approach is to deploy a commercial tool that does the bulk of the work for you. Security pundits like John Oltsik, the principal analyst at Enterprise Security Group, started talking about this concept as early as 2015. They were describing the need for the security vendors themselves to develop services that automated the collection of security tool telemetry, made policy decisions based on that telemetry, and delivered those new policies back to the security stack. These all-in-one orchestration platforms started appearing in the market a couple of years later from the big firewall vendors like Checkpoint, Cisco, Fortinet, Juniper, and Palo Alto Networks. 

Rick Howard: These platforms still did traditional firewall type things, but they also started adding subscription service add-ons to help with zero trust, intrusion kill chain prevention, and resiliency. Instead of the practitioner managing the integration of multiple standalone security tools, anything from 5 to 300, depending on an organization's size. They deployed instead one orchestration platform in various form factors to each data island. That platform performed many of the same tasks as the individual tools, but it was all controlled under one coherent policy. Where it was possible, each subscription service integrated with the others automatically. The downside was at this collection of services probably didn't represent the best of breed for any particular security tool category. The upside though, was that they were likely good enough had the added benefit of being fully integrated with the other subscription services where possible, and were automatically updated with the latest prevention controls discovered by the vendor.  

Rick Howard: Since these firewall vendors had multiple customers scattered around the world. They saw a lot of bad guy telemetry in real time. If they develop new prevention controls because of something they saw in customer A's network, all of their customers benefited from that process. But the idea that you could trust one single vendor to do the bulk of the security work, that was the tough sell. 

Rick Howard: Most security practitioners wanted to hedge their bets with multiple vendors. The platforms were expensive, too. Small and medium sized organizations couldn't afford them. And these same small and medium-sized companies were likely not doing DevOps either despite the disruptive success of startup companies in the early 20 teens.  

Rick Howard: And by the way, we're having a running debate here at the CyberWire about how to refer to the decade represented by the years 2010 to 2020, you know, we have the 1980s, the 1990s, the 2000s, but the 2010s, that just doesn't sound right. What about 2000 teens? Our editor, John Petrik, says that 2000 teens means something else entirely like 2000 teenagers running around, going through puberty, and generally causing trouble. So let's not use that. If any of you have any thoughts on the matter, please let me know. And in the meantime, I'm sticking with 20 teens, which is a smaller subset of 2000 teens, but I digress.  

Rick Howard: Which brings us to a third hybrid approach SOAR, or security orchestration, automation, and response. Gartner coined the term in 2017 about a new kind of security operations center, or SOC, tool that in general terms knows how to communicate with every device in the security stack and provides basic automation capability to handle repetitive data patterns. For example, if a newbie, SOC analyst swipes left on the same intrusion detection system alert a thousand times during her shift, the SOAR tool facilitates the automation of that swipe. The automation piece made SOAR tools unique compared to their sister tools called SIEM tools, or security information and event management tools, that just kind of collected the telemetry for the most part.  

Rick Howard: But I expect at some point, these two capabilities we'll start to merge. SOAR companies will add SIEM functionality and SIEM tools will add SOAR functions, but the SOAR tools are pretty great at reducing the noise inside the SOC. At my last CSO gig, we went from 1 billion alerts coming into the SOC every quarter that a human had to look at down to just 500. That's amazing.  

Rick Howard: If SOC analysts did just that, their life would be so much easier, but there is this untapped capability with SOAR/SIEM platforms. We don't have to limit ourselves to only receiving telemetry from the security stack. The information can flow both ways. The SOAR/SIEM platforms already know how to talk to all the devices in the security stack. What if we use these tools as our DevOps bridge? We could build zero trust, intrusion kill chain prevention, resiliency, and risk forecasting frameworks within the SOAR/SIEM platforms that might be able to give us push button capability to update our security stacks similar to what we did when I was in the Army with our Klingon cloaking device. But I haven't seen anybody doing that in the real world. 

Rick Howard: One last relatively new option is to use a SASE vendor. SASE stands for secure access service edge, and flips the InfoSec security architecture model on its head by using a cloud provider as the first hop destination for any network traffic leaving the local site. Local sites could be anything like headquarters buildings, sales offices, data centers, cloud workloads, and remote employees working from home or at the local store. Gartner again, coined the term SASE back in 2019 and defined three elements that would distinguish a SASE vendor from say a standard MSSP or managed security service provider.  

Rick Howard: Number one: the security stack. In a shared responsibility model, the SASE vendor keeps the blinky lights working on whatever security stack tools they provide. The customer sets the policy. The range of options for the security stack are wide. So buyer beware. If you're doing this, make sure the SASE vendor security stack can handle all of the first principle strategies that we've been talking about. 

Rick Howard: Number two: SDWAN. The SASE vendor plugs into your SDWAN meta layer to ensure that all traffic goes through the security stack and routing is as efficient as it can be. That's the good news. The bad news is that you have to have an SDWAN meta layer. I'm not saying that SDWAN is bad. It's actually quite good when used in conjunction with a SASE model. I'm just saying that it's just another element in your security stack that adds complexity.  

Rick Howard: And lastly, number three: peering. The only way this SASE model works is that if it doesn't slow down normal internet traffic. If your SASE vendor only has a handful of cloud locations around the world that could impose a serious bandwidth limitation. The fix for that is for your SASE vendor to establish peering connections in their data centers with some of the big content provider networks like Google, Amazon, and Netflix.  

Rick Howard: For example, if your SASE vendor has the proper peering connections, your employees in Singapore could ride the vast fiber network of Google to get all the way back to the SASE vendor security stack. That's way better. So when you're talking to SASE vendors, make them describe their peering connection roadmap.  

Rick Howard: SASE is essentially a modified version of using a single vendor's orchestration platform. The good news here is that this model is even less complex than deploying and maintaining the orchestration platform yourself on all of your data islands. The bad news here is that it's not clear how expensive these SASE services will be in the future. It's early days for them, but I'm assuming they will reach some economies of scale as their customer base grows.  

Rick Howard: Of the four options, using a SASE vendor is probably the easiest in terms of complexity, followed closely by deploying a single orchestration platform. But today both tend to be the most expensive. If the SASE vendors can keep the cost down, I think that the SASE architecture is the future, especially for small and medium sized organizations.  

Rick Howard: On the other hand, adopting a DevSecOps mentality is probably the way to go if your organization is trying to be the next internet giant in the wake of the Googles, the Netflixes and the Amazons. But if you are starting DevOps, now, you are years away from having something useful. I expect that most organizations are in the middle somewhere with the SOAR/SIEM model, but they most likely are only using it as a SOC noise reducer and not as an orchestration platform. 

Rick Howard: After all of that, where does orchestration sit on our InfoSec first principle barbecue pit? Here's the thing, if you're we're on the same wavelength as me, in terms of first principles, then you have bought into the idea that the most important thing that we are all trying to do is reduce the probability of material impact to our organization in the next few years due to some cyber event. That's the foundation of the pit. In order to do that, though, we have to pursue four strategies in parallel: zero trust, intrusion kill chain prevention, resilience, and risk forecasting. Those barbecue bricks sit on top of the foundation and give it strength. But what comes next? In this podcast series, we've talked about intelligence operations, SOC operations, incident response, data loss protection, identity management, and purple team operations 

Rick Howard: I want to make an argument here that security orchestration is so important that it should be placed as a big slab of stone that straddles the four parallel strategies. Before we add any of these other bricks on top. My reasoning is that for every other brick you add to the pit, you're adding complexity. The more complexity you have and the more manual your operation is, the more chances you have of leaving something open, or misconfiguring something that will completely unravel all the work you've done to establish each individual brick.  

Rick Howard: Orchestration has to be a key part of your InfoSec program. That's something that you do eventually or do only in certain areas, and that's why we frame problems like this in terms of first principles. You identify the atomic thing that you were trying to do, and then create the more important principles that you will need to support it. When I started this series, the concept of orchestration was mentioned all the time, but never called out as its own thing. It's obvious to me now that orchestration has to be the thing that we are all good at, and that binds the entire program together.  

Rick Howard: And that's a wrap. Next week, we are inviting experts to the Hash Table to see what they say about my crazy idea of the importance of orchestration. You don't want to miss that. But as always, if you agree or disagree with anything I've said, hit me up on LinkedIn or Twitter and we can continue the conversation there. 

Rick Howard: The CyberWire's CSO Perspectives is edited by John Petrik and executive produced by Peter Kilpe. Our theme song is by Blue Dot Sessions, remixed by the insanely talented Elliott Peltzman, who also does the show's mixing, sound design, and original score. And, I am Rick Howard. Thanks for listening.