Cybersecurity first principles: DevSecOps.

By Rick Howard

Jun 8, 2020

CSO Perspectives is a weekly column and podcast where Rick Howard discusses the ideas, strategies and technologies that senior cybersecurity executives wrestle with on a daily basis.

Cybersecurity first principles: DevSecOps.

Listen to the podcast episode.

Note: This is the fifth essay in a planned series that discusses the development of a general purpose cybersecurity strategy for all network defender practitioners-- be they from the commercial sector, government enterprise, or academic institutions-- using the concept of first principles.

We are building a strategy wall, brick by brick, for a cyber security infosec program based on first principles. The foundation of that wall is the ultimate and atomic first principle:

Reduce the probability of material impact to my organization due to a cyber event.

That’s it. Nothing else matters. This simple statement is the pillar on which we can build an entire infosec program.

The first three bricks we put on that pillar were zero trust, intrusion kill chains, and resilience. It is now time to think about a different kind of strategy, a different kind of brick. The first three bricks set a sturdy direction for our infosec teams; a vision if you will about what the perfect endstate might be if we had infinite resources to pursue them. Those bricks constitute a solid base to our metaphorical infosec wall. This new brick is more about how we are going to get there. We are still going to place it on the same pillar as the other three, because it is that important, that fundamental, but it has a different purpose. This brick represents the mechanism for how we are going to lay the other bricks. We are talking about DevOps (development operations) or DevSecOps (Development Security Operations) or, if you want a more descriptive phrase, infrastructure as code.

DevOps: history and concept.

The idea of DevOps really started at Google in 2004, about five years before the IT community even had a name for it. In its earliest days, the Google leadership team gave the responsibility of network management not to the traditional IT team but to the development team. That decision began the idea that we wouldn’t use humans to log into consoles to make configuration changes on our network devices any more. Instead, we could write programs to do it.

DevOps began to emerge as an industry best practice out of three converging ideas sometime in late 2009: The Agile development method, a 2009 Velocity Conference talk called “10+ Deploys per Day” by John Allspaw and Paul Hammond, and Eric Ries’ book, “Lean Startup” which influenced many Silicon Valley companies between 2007 and 2010. It is the idea that there needs to be a much tighter integration between software developers and information technology operations (IT Ops); that once the developers, the quality assurance teams, and the security analysts pass any new code or maintenance updates to IT Ops for deployment, their jobs aren’t done.

Instead of creating artificial black boxes within each team where updates come in, get worked on, and then are passed to the next black box, DevOps is the recognition that update creation, deployment, and maintenance is one big system of systems that needs to be managed that way. It is the idea that organizations would use the same Agile methodology they use today with their software development teams but expand it across all organizations in the deployment cycle: product managers, marketing professionals, developers, quality assurance practitioners, systems engineers, system administrators, operations staff, database administrators, network engineers, and security professionals. In other words, DevOps uses the Agile philosophy across the entire lifecycle of deployed systems from design, to development, to testing, to deployment, to maintenance, and finally to end-of-life.

By 2014 or so, the big internet giants like Google, Amazon, Netflix, and Facebook had become who they were, stand alone leaders in their industry, due in no small part to their adoption of the DevOps philosophy. Their competitors who traditionally used the old waterfall software development method might take years to deploy a new service for their customers. The DevOps companies were doing 10 deployments a day.

The Infosec team says to the DevOps team: don’t leave us behind.

In many organizations before DevOps emerged, the security teams were never in the driver’s seat when it came to IT priorities. In the best cases, they were mostly added to the development pipeline at the end. Once the new thing was ready to go, somebody would ask the security team to take a quick look. In the worst cases, they weren't even consulted. Both situations caused the security team to bolt on expensive and complex security tools and processes if they had any hope of providing a modicum of security for the new service. This wasn’t the case across the board, but it was in far too many organizations. When you hear the security community yelling at the IT community to “Shift Left,” that is what they are talking about. If they were consulted at the beginning of the project and not at the end, the security for the new thing might be less expensive and less cumbersome.

Cybersecurity nirvana and chickens.

When DevOps started to emerge for the IT community, network defenders not only had a chance to shift left in the manual IT process of building new things, they could potentially instantiate a consistency layer of security integration into every new thing developed in the future. Think of deploying your own infosec suite, similar to the Netflix Simian army that I mentioned in the resilience podcast, that automates the common tasks of your infosec program. Netflix used monkeys as the theme in their suite. Why don’t we use chickens?

Coding chicken: An application that checks the software that programmers deposit into the development library for common security errors and for errors made in previous versions (regression testing.)
Zero trust chicken: An application that ensures that only authorized developers can check in and check out modules to and from the development library and only authorized applications and users can install code into production.
Kill chain chicken: An intrusion kill chain application that checks for known adversary techniques against the code base.
Kill chain rooster: Another intrusion kill chain application that deploys security controls at every phase of the intrusion kill to the deployed security stack for all adversary campaigns like Fancy Bear and Oil Rig.
Resilience chicken: An application that ensures that all newly deployed code to the production systems is also deployed to the redundant systems to the same level.

I could go on and on, mostly because I really like this chicken theme.

In 2016, I finally read Gene Kim’s now famous novel called The Phoenix Project which explains the philosophy of DevOps. And then I read the Google book on “Site Reliability Engineering.” This was the DevOps how-to manual written by Google’s SREs and my mind was blown. I was convinced that this was the way the network defender community was moving. It made so much sense to me that we would join forces with the new DevOps teams and form this more robust hybrid team called the DevSecOps. I was convinced that it was inevitable that we would reach this state of chicken nirvana. I was telling newbies and veterans alike that in the near future, say five years, our infosec teams wouldn’t consist of security experts who dabbled in coding. They would consist of coders who knew a lot about security. Our world was about to change and I was looking forward to this bright future.

DevOps and infosec - oil and water.

That was 2016. It is practically five years later and I can truthfully say that I was completely wrong. We are no more closer to realizing this vision than we were when I first read Kim’s book. Much to my chagrin, the network defender community didn’t embrace the concept. It turns out that there are obstacles for us to pursue this strategy with any vigor. I believe the biggest one is that most of us in the network defender community aren’t coders and it is difficult to build infrastructure as code if you don’t have anybody on the team with that skill set. Who knew?

Don’t get me wrong. There is absolutely a small subset of the network defender community who are brilliant coders. I am just saying that the bulk of the community is not. And I will be the first to say that when I started out, I didn't have to know how to code to become a security practitioner either. It was a handy skill to have but it wasn’t essential. But if we are ever to have any hope of approaching my chicken nirvana, we have to reverse that trend. In order to build infrastructure as code, it follows that most of us have to be coders.

One of the industry developments that has exacerbated this lack of skill set is our desperate desire to fill cybersecurity jobs. Estimates vary widely about how many open international cybersecurity jobs exist but the range is somewhere between 1.8 million and 4 million. Even at the lower end of that range, that is still a big number. Hiring managers, to fill the gap, have taken up the mantra that cybersecurity is a great second career for transitioning government employees to the civilian sector and just for folks looking for a change. And I support those initiatives. The downside is that the typical second-career employees haven’t been knee deep into their laptops for the past 20 years. They probably don’t know how to code. They might have taken some night school classes to get their CISSP certification or studied for their second bachelor’s degree in information security, but those programs don’t feature coding as a prerequisite. If they have courses at all, they are electives. The bottom line is that we are trying to fill the cybersecurity job shortage with skill sets that we needed 20 years ago, not for the skills we need today and in the future. If we continue down this path, we will get further from our goal of shifting left and automating those processes once we get there.

The situation is even further aggravated because the DevOps teams use coding tools that the infosec folks aren’t familiar with. They have strange names like Jenkins, Bamboo, Docker, Kubernetes, Puppet, and Ansible that don’t sound anything like the typical cybersecurity languages the infosec teams prefer like Python, JavaScript, C, PHP, and Powershell. Even if we were inclined to learn how to code, which most of us aren’t, would we spend our time learning Jenkins or Python? I think most would pick Python. The two teams, DevOps and infosec, are living in worlds that don’t readily mix like oil and water.

The last obstacle that is causing friction in our efforts to achieve chicken nirvana is that, for the most part, the security teams are still on the outside looking in at the IT teams. The IT teams don’t want to slow down to implement the Chicken suite and the security teams don’t want to change how they have always done things. That is unfortunate because DevSecOps is absolutely the linchpin to the entire metaphorical infosec wall that I have been talking about.

We are trying to reduce the probability of a material cyber event for our organization. It is one thing to use strategies like zero trust, intrusion kill chains, and resiliency to bring that probability down, but if you can’t pursue those strategies with any speed or with any consistency, you are likely to fail. The adversaries automate their attack sequences. The network defenders have no hope to keep up with them if they are trying to respond manually. The infosec teams can’t go fast enough and when they try, they make mistakes that the adversaries take advantage of.

SOAR is DevSecOps.

The good news is that the infosec teams have their own infrastructure that they run and don’t have to interact with the IT teams to make work. At first blush, that seems to go against the idea of chicken nirvana and the building of a hybrid DevSecOps team, but it might be a useful first step. The systems in question are what the SOC (Security Operations Center) teams run on a daily basis. It is the place where we can practice the DevSecOps philosophy without having to worry about breaking some essential business process that the IT team is responsible for because we don’t know what we are doing yet in this new DevSecOps world. And the network defender community has made some progress in automating key tasks within that environment.

The big problem that SOC teams face is the extremely high volume of alerts that they have to review. It is not uncommon to have over a billion alerts show up in the SOC over a period of 90 days. If you are a Fortune 500 size company, it’s even bigger than that.

SOAR (Security Orchestration, Automation and Response) tools started to emerge for SOC use in 2018. These tools purported to solve the telemetry volume problem. Through APIs (Application interfaces through software), the SOAR tool can connect to practically any kind of networking equipment and security stack tool deployed on your network, automatically collect telemetry from that device in terms of health and welfare and security alerts, and provide the SOC team with the ability to easily write code that handles that telemetry in a prescribed way; things like, save for the future, delete, or escalate to the a human for review. This has enormous implications.

In a typical SOC environment before SOAR, there are generally three tiers of analysts. Tier 1 analysts are typically the newbies, less than a year or two of experience. These are the console jockeys. Their job is to review each and every alert that comes into the SOC and decide what to do about it. They spend a lot of their time “swiping left” not because the alert isn't important, but because the volume is so high that they can’t possibly look at all of them in any meaningful way. If they discover something they are not sure about, they bump it up to the Tier 2 analysts. The Tier 2 analysts typically have 3-5 years of hard-won experience and know how to handle most of the events that come into the SOC. If they can’t, they bump it up to the Tier 3 team. When you think of Tier 3 analysts, think of the stereotypical Unix greybeards. They don’t all look like that but most everybody has one or two of these people in their organization who have all been there, done that, and wear the scars on their back because of it. They’re the go-to people for all issues out of the ordinary. This essential design of the SOC team hasn’t changed that much since we invented it back in the late 1970s.

With SOAR tools in place, we can essentially eliminate the need for humans to do tier 1 and tier 2 functions by replacing them with automation. We can reduce the billion alerts that come into the SOC every quarter to something more manageable that humans have to look at; something in the 500 range. That reduces to about 5 events a day that the Tier 3 analyst has to spend time on and that seems much more manageable. Of course, we don’t fire the old Tier 1 and Tier 2 analysts. We change their job. For some of them, we turn them into DevSecOps developers and put them on the task of automating the other things we need to automate like kill chain rooster and resilience chicken. For others, we put them on the threat hunting team.

I will discuss threat hunting teams later in this first principle series but the point is that tier 1 analysts don’t learn a thing manning a desk and ”swiping left” eight hours a day in the old SOC design. You could make a strong argument that the tier 2 analysts, once they settle in, are not learning a lot of new things either. You take those two sets of employees and put them in the DevSecOps team and the threat hunting team and they become cybersecurity experts in six months. And that is the power of automation and one of the reasons that DevSecOps is an essential brick on our first principles infosec wall.

Take this first step in SOAR, but don’t stop there.

Many SOCs are doing this SOAR thing now and many others are anticipating that they will soon. They most likely don’t consider the use of SOAR technologies as a DevSecOps function, but it absolutely is. And that is the first step; to get the idea of the DevSecOps philosophy into the minds of the infosec teams, that automation will drive everything, that coding is an essential skill for everybody. Once there, it will become time to reach across the aisle to the IT teams and learn what DevOps tools they are using to deploy infrastructure as code. If they are using Jenkins, the SOAR team should use it too to build coding chicken. If they are using Ansible, the SOAR team should use that to build zero trust chicken. When the infosec team stumbles or hits a roadblock, reach out to the DevOps team for help. Start building the bridges so that you can ultimately shift left in the infrastructure as code process.

DevSecOps as an essential brick in the first principle infosec wall.

We have chosen the task of reducing the probability of material impact to our organization due to a cyber event as the most important thing we have to do as security practitioners. That is the base pillar for our metaphorical infosec wall. The next three bricks we stacked on that pillar to give it strength were zero trust, intrusion kill chains, and resilience. But we will lose most of that intended strength if we don’t treat each brick as a system of systems and maintain each at the speed of code. We can’t respond to an automated adversary with manual processes. I had an old boss of mine, Mark McLaughlin, who used to say, “You can’t bring people to a software fight.” And he was so right. DevSecOps is the brick that will get us there. Use SOAR as a starting point. Automate the processing of those alerts coming into the SOC and use that as the first step of your DevSecOps program. Reach out to your DevOps teams and get their advice and help using the DevOps tools they prefer while you are building your version of the chicken suite.