CSO Perspectives is a weekly column and podcast where Rick Howard discusses the ideas, strategies and technologies that senior cybersecurity executives wrestle with on a daily basis.
Security operations centers: a first principle idea.
Note: This is the eighth essay in a planned series that discusses the development of a general purpose cybersecurity strategy using the concept of first principles to build a strong and robust infosec program. The other essays are listed here:
Consider your infosec program as a metaphorical wall. I made the case in the first essay, called “First principles,” that the foundation of that wall is this:
Reduce the probability of material impact to my organization due to a cyber event.
That’s it. Nothing else matters. This simple statement is the pillar on which we can build an entire infosec program, a wall of interconnected concepts that relies on the subsequent bricks to give it even more strength. And that is the main characteristic of first-principles thinking. Once you find the atomic idea in the problem domain, the essential element of purpose, everything you design into the wall directly underpins it by giving it more energy and structural support.
I am reminded of one of my favorite science fiction novels, “Dune,” written by Frank Herbert back in 1965. Paul Atreides is the hero and follows the classic hero’s journey to become the leader of the Fremen on the planet Arrakis. His people name him Usul, which, in Fremen, means “the strength of the base of the pillar.” He is the backbone of the entire planet and the Fremen enhance his strength by their support. And that describes perfectly the importance of the ultimate first principle idea. For each brick I laid in that wall, each of the essay topics, I could draw a straight line of support back to the ultimate first principle. Sometimes that line went through other bricks, but always it ended up at the foundation.
As season two of my CSO Perspectives podcast begins, I will continue to build the wall. The next brick to consider is the security operations center, or SOC.
History of security operations centers.
The idea of operations centers has been around seemingly forever. Friedrich Klemm, in his “A History of Western Technology,” suggests that the concept goes as far back as 5,000 B.C. Any time an organization grows big enough, either in terms of people or function, where one small team can’t do everything, leaders have built these centers to manage the workflow and status of the various groups and to coordinate actions among them. If you fast forward to the early days of the technological revolution, we started seeing organizations that began looking like a modern-day SOC but weren’t quite there yet.
When telephone networks started appearing in the early 1920s, phone companies like AT&T built traffic control bureaus to handle long distance traffic issues. By the early 1960s, AT&T handled most telephone switching through mechanical devices and built a network control center, or NOC, to manage it. AT&T historians consider this to be the first NOC ever built. By 1977, Bell Systems had built the first national NOC in Bedminster, New Jersey, which looked a lot like modern NOCs today. There wasn’t much security yet, but if there was any, NOC operators were doing it.
Meanwhile, in the U.S. intelligence community, the 1960s were fraught with international incidents like the Cuban Missile Crisis of 1962, the Arab-Israeli Six Day War in 1967, the U.S.S Pueblo capture in 1968, the Prague Spring Crisis in Czechoslovakia in 1968, and the EC-121 shootdown crisis in 1969. The NSA (National Security Agency) decided that they needed an operations center to manage their efforts across a wide swatch of international activity. Based on a Freedom of Information Request (FOIA,) the NSA released a document in 2007 that described the formation of the first National Sigint Operations Center (NSOC) in 1973. According to Charles Berlin, a former NSOC Director, NSA kept adding more responsibility to it over time. He said that its secret sauce was when NSA decided to pair offense (SIGINT) and defense (COMSEC) in the same place. Eventually, they replaced the word “SIGINT,” in the title with “Security.” In other words, it became the National Security Operations Center. Berlin said that when cyber came along years later, the total mission became too big to keep in the NSOC, and the NSA created the National Cyber Threat Operations Center (NCTOC) to deal with it. But, with the addition of the COMSEC mission, these operations centers started to lean toward defensive security.
On the government side, in the aftermath of the Morris Worm—the first destructive Internet worm— DARPA (Defense Advanced Research Projects Agency, a science and technology organization of the US Department of Defense) sponsored Carnegie Mellon University to establish the first CERT/CC (Computer Emergency Response Team/Coordination Center) in 1988. According to Rich Pethia, the first CERT/CC director, one of his missions was to help the military services build their own CERTS, which they did. The Air Force established the AFCERT (Air Force Computer Emergency Response Team) in 1993. The other services followed suit soon thereafter. Geoff Hancock, a former special operations soldier, said that he helped build the first special operations SOC at McDill Air Force base around the same time. He said the work done at McDill and in the military CERTS contributed to the eventual stand-up of the Joint Task Force - Computer Network Defense (JTF-CND) in 1998. With the creation of these military CERTS, the requirement for a SOC—coordination of defensive actions and intelligence within an organization—began establishing itself as a general purpose best practice for all network defenders.
On the commercial side, we started to see the first managed security service providers (MSSPs) in the late 1990s and early 2000s. President Clinton established the ISAC system, the information sharing and analysis center framework, when he signed Presidential Decision Directive-63 (PDD-63) on May 22, 1998. Both kinds of organizations provided SOC-type services for those that couldn’t do it themselves in the MSSP case or provided a threat intelligence sharing capability for peer verticals—finance, IT, health, electricity, etc.—in the ISAC case. At some point between 2002 and 2012, the idea that network defenders in the commercial space should build and operate their own in-house SOCs started to gain traction.
Current state of security operations centers.
Today, many medium-to-large organizations either have their own internal SOCs or contract the function (or at least a part of the function) out to a third-party MSSP. Small organizations usually haven’t grown large enough to warrant a centralized point to coordinate the activities of multiple groups. IT and security are done by the same small team.
The latest development in the commercial space is SOC services delivered as SaaS applications. However, the functionality of any specific SOC compared to another varies wildly. Based on the history and evolution of the operations concepts, you would think that the SOCs would be the one point in the organization that coordinates all security issues, but that just isn't the case. The functions range from simply monitoring certain pieces of the network, with no ability to make changes to the security policy, on one end of the spectrum to having complete control of the security stack across all deployments of data.
The first principle of security operations centers.
Which brings us to why we want to have a SOC in the first place. As we build our infosec wall, layering bricks in order to strengthen the foundation, we have placed six bricks so far on top:
- Zero trust
- Intrusion kill chains
- Resilience
- DevSecOps
- Risk
- Cyber threat intelligence
No single team inside the organization owns all of them, nor, more importantly, does it run the business units that will be most impacted by them. That’s the very characteristic that leadership has used to justify the building of operations centers for centuries, but especially in the modern era. When a task gets so big in scope that it requires multiple teams to complete it, an operations center is needed to coordinate those efforts. In order for the infosec wall to be effective, network defenders must have a space where they bring all relevant information in from all of the bricks on the wall, analysts review the information and make recommendations to leadership, leadership makes decisions, and then they send actions out to the individual brick teams to execute or, even better, use automation to deploy those decisions without having to put people in the loop.
As with all the other bricks on the infosec wall, there is a direct line of support from the SOC brick all the way back to the original pillar. What makes the SOC special is that its line of support travels through all the other bricks first. The first principle of security operations then is this:
Coordinate and execute security tasks across the organization in order to reduce the probability of material impact due to a cyber event.
At a certain maturity level for any organization, you can’t build an infosec wall without a SOC.
Aspirations for security operations centers.
Let me be clear. There are a handful of organizations in the world that design their SOCs to operate in this manner. Most come nowhere near it. They monitor. They collect data. Their SOC analysts grind their way through billions of alerts trying to find the one item that may mean they’ve been compromised. Most don’t try to defeat adversary campaigns across the intrusion kill chain. Instead, they focus on blocking access to technical vulnerabilities that an adversary might use to be successful. Most SOC analysts aren’t involved in the resilience plan at all. Many can’t even spell DevSecOps let alone contribute to the infrastructure as code philosophy. The SOC may have some control over the zero trust deployment and policy, but not the entire plan. Most SOC analysts have no idea how to calculate the risk probability of a material cyber event in some future time frame, not because they can’t, but because no leader has asked them to do it. The one security block that many do have control of is the intelligence brick. But even that effort isn't completely focused on defeating the adversary across the intrusion kill chain and leadership most likely doesn’t use the intel team to calculate the risk probability of the organization.
That is the reason to rethink your security program through a first principle lens. It isn't enough to have an organization called the SOC. The SOC you build must absolutely support the infosec wall, or why bother? And I realize that this is hard to do. This refocusing of the SOC along first principles goes against twenty years of established infosec best practice. Even if you agree with me that a change is required, the chances aren’t good that you can convince senior management to centralize security decisions in one place, to break across many swim lanes of established bureaucracy over decades of not doing that. But it has to be done and it can be done.
Go back to the inaugural essay in this series. There is a reason that the Elon Musks of the world have succeeded. He had the strength of his convictions that incrementally improving something without considering the ultimate purpose of what he was trying to do was a recipe for failure. By using first principles, he was able to build a Mars rocket, a luxury electric car, and affordable solar power when most thought it was impossible to build even one of those things. I’m not saying that first principle thinking is the only thing that made Mr. Musk successful. I’m just saying that it was an essential part, that without it, none of it would have happened.
The concept of a modern-day SOC has a winding, interesting, and evolutionary backstory that started in the early 1900s and continues today—but we are not done. We have a long way to go to create a SOC in the future that can support every brick in our infosec wall and add strength to its foundation.
Recommended reading.
"5G/SOC: SOC Generations,” by HP ESP Security Intelligence and Operations Consulting Services, May 2013.
“ABOUT ISACs,” by The National Council of ISACs.
“A History of Western Technology,” by Friedrich Klemm, published by Iowa State Press, 1 July 1991.
“A tour of AT&T's Network Operations Center (1979) - AT&T Archives,” by AT&T Tech Channel, 19 November 2012.
Linked-In Conversation with Charles Berlin, 8 July 2020.
Linked-In Conversation with Rich Pethia, 10 July 2020.
“Phenomenati's Taxonomy of a SOC™ for Cyber Security Operations,” by Phenomenati.
Phone conversation with Geoff Hancock, 10 July 2020.
“Richard Pethia,” by the Software Engineering Institute, Carnegie Mellon University.
“Testimony of Richard Pethia, Manager, Trustworthy Systems Program and CERT Coordination Center Software Engineering Institute, Carnegie Mellon University, Before the Permanent Subcommittee on Investigations U.S. Senate Committee on Governmental Affairs,” Federation of American Scientists (FAS), 5 June 1996.
“The CERT Division,” by the Software Engineering Institute, Carnegie Mellon University.
"The Exabeam 2020 State of the SOC Report,” by Exabeam, 2020.
“The Morris Worm: 30 Years Since First Major Attack on the Internet,” FBI, 2 Novemebr 2018.
"The National Sigint Operations Center,” NSA FOIA Release, 4 May 2007, Wayback Machine.
“U.S. Cyber Command History,” by U.S. Cyber Command.