Security operations centers and cybersecurity first principles.

Rick Howard: Hello, hello - Rick here. Welcome to Season 2 of "CSO Perspectives." We took a couple of weeks off at the end of Season 1 to build some new elements into the show. In addition to the individual podcasts that we did last season - or, as my executive editor Peter Kilpe likes to call them, Rick's brain dump episodes - in this season, we are also introducing you to the CyberWire's hash table of experts. I have browbeat some of my friends and other key leaders in the network defender space to have a seat with me at the CyberWire hash table to discuss important topics to the network defender community. So one episode will be me talking about what I think regarding a particular topic, and the very next episode will be me talking with one or more experts from the hash table about it. So strap in. I think it's going to be fun.

Rick Howard: My name is Rick Howard. You are listening to "CSO Perspectives," my podcast about the ideas, strategies and technologies that senior security executives wrestle with on a daily basis. In Season 2, we are going to continue our discussion about cybersecurity first-principle thinking by adding more bricks to our metaphorical infosec wall. In this episode, we are talking about security operations centers. 

Rick Howard: The idea of operations centers has been around seemingly forever. Friedrich Klemm, in his "A History Of Western Technology," suggests that the concept goes back as far as 5000 B.C. - amazing. Any time an organization grows big enough - either in terms of people or function - where one small team can't do everything, leaders have built these centers to manage the workflow and status of the various groups and to coordinate actions among them. If you fast-forward to the early days of the technological revolution, we started seeing organizations that began looking like a modern-day SOC but weren't quite there yet. 

Rick Howard: The classic example is how NASA managed its space missions starting way back in 1958. Now, for those who don't know me, I'm a space geek. Specifically, I love everything about the space race between the Russians and the Americans during the 1960s. In fact, as a side note, The Washington Post's Lillian Cunningham produced a 13-episode podcast about that very thing last year. It is called "Moonrise," and I highly recommend it. But did you know that when Neil Armstrong and Buzz Aldrin landed on the moon in 1969 that the Russians had a remote-controlled spacecraft up there at the same time? I didn't know that until I listened to the "Moonrise" podcast. The Russians crashed it into a moon mountain as Armstrong and Aldrin were flying back to the lunar module, so maybe that is why the Russians don't advertise it that much. But I digress. 

Rick Howard: One of my favorite space movies is "Apollo 13" directed by Ron Howard, and one of the things I love about that movie is how it depicts the energy and sense of purpose of an operations center. Here is actor Ed Harris playing the famous Gene Kranz, the NASA flight director in the mission control operations center getting ready to launch Apollo 13. 

(SOUNDBITE OF FILM, "APOLLO 13") 

Ed Harris: (As Gene Kranz) In the meantime, you're going to have a frozen command module up there. In a couple days, we're going to have it power it up using nothing but the reentry batteries. 

Gabriel Jarret: (As GNC White) Never been tried before. 

Googy Gress: (As RETRO White) Hell, we've never even simulated it before, Gene. 

Ed Harris: (As Gene Kranz) Well, we're going to have to figure it out. I want people in our simulators working reentry scenarios. I want you guys to find every engineer who designed every switch, every circuit, every transistor and every lightbulb that's up there. Then I want you to talk to the guy on the assembly line who actually built the thing. Find out how to squeeze every amp out of both of these goddamn machines. I want this mark all the way back to Earth with time to spare. We never lost an American in space. We're sure as hell not going to lose one on my watch. Failure is not an option. 

Rick Howard: When telephone networks started appearing in the early 1920s, phone companies like AT&T built traffic control bureaus to handle long-distance traffic issues. By the early 1960s, AT&T handled most telephone switching through mechanical devices and built a network control center, or NOC, to manage it. AT&T historians consider this to be the first NOC ever built. By 1977, Bell Systems had built the first national NOC in Bedminster, N.J., which looked a lot like modern NOCs today. There wasn't much security yet, but if there was any, NOC operators were doing it. 

Rick Howard: In the U.S. intelligence community, the 1960s were fraught with international incidents like the Cuban Missile Crisis of 1962, the Arab-Israeli Six-Day War in 1967, the USS Pueblo capture in 1968, the Prague Spring crisis in Czechoslovakia also in 1968 and the EC-121 shootdown crisis in 1969. The NSA decided that they needed an operations center to manage their efforts across a wide swatch of international activity. 

Rick Howard: Based on a Freedom of Information request, the NSA released a document in 2007 that described the formation of the first National SIGINT Operations Center, or INSOC, in 1973. And according to Charles Berlin - I hit him up on LinkedIn, and he answered me; he's a former INSOC director - the NSA kept adding more responsibility to it over time. He said that its secret sauce was when the NSA decided to pair offense, or SIGINT, with defense, or COMSEC, in the same place. Eventually, they replaced the word SIGINT in the title with security. In other words, it became the National Security Operations Center. Berlin said that when cyber came along years later, the total mission became too big to keep in the NSOC. And the NSA created the National Cyber Threat Operations Center, or the NTOC, to deal with it. But with the addition of the COMSEC mission, these operations centers started to lean toward defensive security. 

Rick Howard: On the government side and in the aftermath of the Morris Worm, which was the first destructive internet worm, DARPA - a science and technology organization of the U.S. Department of Defense - sponsored Carnegie Mellon University to establish the first CERT/CC, or Computer Emergency Response Team Coordination Center, back in 1988. According to Rich Pethia, the first CERT/CC director, one of his missions was to help the military services build their own CERTs, which they did. The Air Force established the Air Force CERT in 1993. The other services followed suit soon thereafter. 

Rick Howard: Geoff Hancock, a former special operations soldier, said that he helped build the first special operations SOC at MacDill Air Force Base around the same time. He said that the work done at MacDill and in the military CERTs contributed to the eventual stand-up of the Joint Task Force Computer Network Defense, or JTF-CND, in 1998. With the creation of these military CERTs, the requirement for a SOC to coordinate defensive actions and intelligence operations within an organization began establishing itself as a general purpose best practice for all network defenders. 

Rick Howard: On the commercial side, we started to see the first managed security service providers - or MSSPs - in the late 1990s and early 2000s. And, also, President Clinton established the ISAC system - the Information Sharing and Analysis Center framework - when he signed Presidential Decision Directive 63 on May 22, 1998. Both kinds of organizations provided SOC-type services for those that couldn't do it themselves in the MSSP case or provided a threat intelligence sharing capability for peer verticals like finance, IT, health, electricity and others in the ISAC case. At some point, between 2002 and 2012, the idea that network defenders in the commercial space should build and operate their own in-house SOCs started to gain traction. 

Rick Howard: Today, many medium-to-large organizations either have their own internal SOCs or contract the function - or at least part of the function - out to a third-party MSSP. Smaller organizations usually haven't grown large enough to warrant a centralized point to coordinate the activities of multiple groups. IT and security are done by the same small team. The latest development in the commercial space is SOC services delivered as SaaS applications. However, the functionality of any specific SOC compared to another varies widely. Based on the history and evolution of the operations center concepts, you would think that the SOCs would be the one point in the organization that coordinates all security issues, but that just isn't the case. The functions range from simply monitoring certain pieces of the network with no ability to make changes to the security policy, on one end of the spectrum, to having complete control of the security stack across all deployments of data on the other. 

Rick Howard: Which brings us to the question - why do we want to have a SOC in the first place? As we build our infosec wall, layering bricks in order to strengthen the foundation, we have placed six bricks so far - zero trust, intrusion kill chains, resilience, DevSecOps, risk and cyberthreat intelligence. No single team inside most organizations owns all of these functions nor, more importantly, does it run the business units that will be most impacted by them. That's the very characteristic that leadership has used to justify the building of operations centers for centuries, but especially in the modern era. 

Rick Howard: When a task gets so big in scope that it requires multiple teams to complete it, an operations center is needed to coordinate those efforts. In order for the infosec wall to be effective, network defenders must have a space where they bring all relevant information in from all of the bricks on the wall. Analysts review the information and make recommendations to leadership. Leadership makes decisions, and then they send actions out to the individual brick teams to execute or, even better, use automation to deploy those decisions, without having to put people in the loop. And with all the other bricks on the infosec wall, there is a direct line of support from the SOC brick all the way back to the original pillar. What makes the SOC special is that its line of support travels through all the other bricks first. The first principle of security operations then is this - coordinate and execute security tasks across the organization in order to reduce the probability of material impact due to a cyber event. At a certain maturity level for any organization, you can't build an infosec wall without a SOC. 

Rick Howard: Let me be clear - there are a handful of organizations in the world that designed their SOCs to operate in this manner. Most come nowhere near it. They monitor. They collect data. Their SOC analysts grind their way through billions of alerts, trying to find the one item that may mean they've been compromised. Most don't try to defeat adversary campaigns across the intrusion kill chain. Instead, they focus on blocking access to technical vulnerabilities that an adversary might use to be successful. Most SOC analysts aren't involved in the resilience plan at all. Many can't even spell DevSecOps, let alone contribute to the infrastructure as code philosophy. 

Rick Howard: The SOC may have some control over the zero-trust deployment and policy, but not the entire plan. Most SOC analysts have no idea how to calculate the risk probability of a material cyber event in some future time frame, not because they can't but because no leader has asked them to do it. The one security brick that many do have control of is the intelligence brick, but even that effort isn't completely focused on defeating the adversary across the intrusive kill chain. And leadership most likely doesn't use the intel team to calculate the risk probability of an organization. 

Rick Howard: That is the reason to rethink your security program through a first-principle lens. It isn't enough to have an organization called the SOC. The SOC you build must absolutely support the infosec wall. Or why bother? And I realize that this is hard to do. This refocusing of the SOC along first principles goes against 20 years of established infosec best practice. And even if you agree with me that a change is required, the chances aren't good that you can convince senior management to centralize security decisions in one place, to break across many swim lanes of established bureaucracy over decades of not doing that. But it has to be done, and it can be done. 

Rick Howard: Go back to the inaugural first principle podcast. There is a reason that the Elon Musks of the world has succeeded. Elon Musk had the strength of his convictions that incrementally improving something without considering the ultimate purpose of what he was trying to do was a recipe for failure. By using first principles, he was able to build a Mars rocket, a luxury electric car and affordable solar power when most thought it was impossible to build even one of those things. I'm not saying that first principle thinking is the only thing that made Mr. Musk successful; I'm just saying that it was an essential part, that without it, none of it would have happened. The concept of a modern-day SOC has a winding, interesting and evolutionary backstory that started in the early 1900s and continues today. But we are not done. We have a long way to go to create a SOC in the future that can support every brick in our infosec wall and add strength to its foundation. 

Rick Howard: And that's a wrap. If you agree or disagree with anything I have said, hit me up on LinkedIn or Twitter, and we can continue the conversation there. The CyberWire's "CSO Perspectives" is edited by John Petrik and executive produced by Peter Kilpe. Mix, sound design and original music by the insanely talented Elliott Peltzman. And I am Rick Howard. Thanks for listening.