DevSecOps: a first principle of cybersecurity.

Rick Howard: In 2016, three years after it came out, I finally read Gene Kim's now-famous novel called "The Phoenix Project," which explains the philosophy of DevOps. Now, you know, he wrote a novel because he and his co-authors knew that the audience for a nonfiction book about DevOps was pretty small. He wanted to reach a wider group, and he was right. That same year, I read the Google book "Site Reliability Engineering." Now, if "The Phoenix Project" was the DevOps philosophy, "Site Reliability Engineering" was the how-to manual written by Google's site reliability engineers, or SREs. And I have to tell you my mind was blown. 

Rick Howard: I was convinced that this was the way the network defender community was moving. It made so much sense to me that we would join forces with the new DevOps teams and form this more robust hybrid team called DevSecOps. I was positive that this was inevitable, that we would reach this new state of nirvana. I was telling newbies and veterans alike that in the near future, say five years, our infosec teams wouldn't consist of security experts who dabbled in coding. It would consist of coders who knew a lot about security. Our world was about to change, and I was looking forward to this bright future. That was 2016. It is practically five years later, and I can truthfully say I was completely wrong. We are no more closer to realizing this vision than we were when I first read Kim's book. Much to my chagrin, the network defender community didn't embrace the concept. 

Rick Howard: My name is Rick Howard. You are listening to "CSO Perspectives," my podcast about the ideas, strategies and technologies that senior security executives wrestle with on a daily basis. This is the fifth show in a planned series that discusses the development of a general purpose cybersecurity strategy using the concept of first principles. So far, I've explained what first principles are and made an argument about what the very first principles should be. Since then, I've covered zero trust, intrusion kill chains and resilience. In this show, we are talking about DevSecOps. 

Rick Howard: It turns out that there are obstacles for the network defender community to pursue this strategy of DevSecOps with any vigor. The biggest obstacle is that most of us aren't coders, and it is difficult to build infrastructure as code if you don't have anybody on the team with that skillset. Who knew? Don't get me wrong. There is a small subset of us who are really talented coders. But for the most part, the rest of us defer any coding chores to that small subset. 

Rick Howard: As I covered in the resilience episode, the idea of DevOps really started at Google in 2004, about five years before the IT community even had a name for it. In the earliest days, the Google leadership team gave the responsibility of network management not to the traditional IT team but to the development team. That decision began the idea that we wouldn't use humans to log into consoles to make configuration changes on our network devices anymore. Instead, we would write programs to do it. 

Rick Howard: DevOps began to emerge as an industry-best practice out of three converging ideas sometime in late 2009. First was the Agile development method. Second was a 2009 Velocity Conference talk called 10+ Deploys per Day by John Allspaw and Paul Hammond. And third, it was Eric Ries' book "Lean Startup," which influenced many Silicon Valley companies between 2007 and 2010. DevOps is the idea that there needs to be a much tighter integration between software developers and information technology operations, or IT Ops, so much so that once the developers and the quality assurance teams and the security analysts pass any new code or maintenance updates to IT Ops for deployment, their jobs aren't done. 

Rick Howard: Instead of creating artificial black boxes within each team where updates come in, get work done and then are passed to the next black box, DevOps is the recognition that update creation, deployment and maintenance is one big system of systems that needs to be managed that way. It is the idea that organizations would use the same Agile methodology that they use today with their software development teams but expand it across all organizations in the deployment cycle - product managers, marketing professionals, developers, quality assurance practitioners, systems engineers, system administrators, operations staff, database administrators, network engineers and security professionals. In other words, DevOps uses the Agile philosophy across the entire life cycle of deployed systems from design to development to testing to deployment to maintenance and finally to end of life. 

Rick Howard: By 2014 or so, the big internet giants like Google, Amazon, Netflix and Facebook had become who they were - stand-alone leaders in their industry - due in no small part to their adoption of the DevOps philosophy. Their competitors who traditionally used the old waterfall software development method, they might take years to deploy a new service to their customers. The DevOps companies, they were doing 10 deployments a day. In many organizations before DevOps emerged, the security teams were never in the driver's seat when it came to IT priorities. In the best cases, they were mostly added to the development pipeline at the end. Once the new thing was ready to go, somebody would ask the security team to take a quick look. In the worst cases, they weren't even consulted. 

Rick Howard: Both situations caused the security team to bolt on expensive and complex security tools and processes if they had any hope of providing a modicum of security for the new service. That wasn't the case across the board, but it was in far too many organizations. When you hear the security community yelling at the IT community to shift left, that is what they are talking about. If they were consulted at the beginning of the project and not at the end, the security for the new thing might be less expensive and less cumbersome. 

Rick Howard: When DevOps started to emerge for the IT community, network defenders not only had a chance to shift left in the manual IT process of building new things. They could potentially instantiate a consistency layer of security integration into every new thing developed in the future. Think of deploying your own infosec suite similar to the Netflix simian army that I mentioned in the resilience podcast that automates the common tasks of your infosec program. Netflix used monkeys as the theme to their suite. Why don't we use the theme to one of my favorite superhero shows from the 1970s, "Chickenman"? 

(SOUNDBITE OF ARCHIVED RECORDING) 

Jim Runyon: (As narrator) Now another exciting episode in the life of the most fantastic crime fighter the world has ever known... 

Dick Orkin: (As character, imitating chicken). 

Jim Runyon: (As narrator) ...Chickenman. 

Unidentified Group: He's everywhere. He's everywhere. 

Rick Howard: We could build an application called Coding Chicken that checks the software that programmers deposit into the development library for common security errors and for errors made in previous versions or regression testing. We could build an app called Zero Trust Chicken that ensures that only authorized developers can check in and check out modules to and from the development library and only authorized applications and users can install code into production. We could build an application called Kill Chain Chicken that checks for known adversary exploitation techniques against the code base. We can build an application called Kill Chain Rooster, another intrusion kill chain application, but this one deploys security controls at every phase of the intrusion kill chain to the deployed security stack for all known adversary campaigns like Fancy Bear and OilRig. And we can build an application called Resilience Chicken that ensures that all newly deployed code to the production systems is also deployed to the redundant systems to the same level. 

Rick Howard: I could go on and on, mostly because I really like this chicken theme. Let me just say that when I started out, I didn't have to know how to code to become a security practitioner. It was a handy skill to have, but it wasn't essential. Still, if we're ever to have any hope of approaching my chicken nirvana, we have to reverse that trend. In order to build infrastructure as code, it follows that most of us have to be coders. 

Rick Howard: One of the industry developments that has exacerbated this lack of programming experience is our desperate desire to fill cybersecurity jobs. Estimates vary widely about how many open international cybersecurity jobs exist, but the range is somewhere between 2 million and 4 million. Even at the lower end of that range, that is still a big number. Hiring managers, to fill the gap, have taken up the mantra that cybersecurity should be a great second career for, like, transitioning government employees into the civilian sector and just for folks looking for a career change, and I support all of those initiatives. The downside, though, is that the typical second-career employees haven't been knee-deep into their laptops for the past 20 years. They probably don't know how to code. They might have taken some night school classes to get their CISSP certification or study for their second bachelor's degree in information security, but those programs don't feature coding as a prerequisite. If they have courses at all, they are electives. 

Rick Howard: The bottom line is that we are trying to fill the cybersecurity job shortage with skillsets that we needed 20 years ago, not for the skills we need today and in the future. If we continue down this path, we will get further from our goal of shifting left and automating those processes once we get there. The situation is even further aggravated because the DevOps teams use coding tools that the infosec folks aren't familiar with. They have strange names like Jenkins and Bamboo and Docker and Kubernetes and Puppet and Ansible that don't sound anything like the typical cybersecurity languages that infosec teams prefer like Python and JavaScript and C and PHP and Powershell. Even if we were inclined to learn how to code, which most of us aren't, would we spend our time learning Jenkins or Python? I think most of us would pick Python. The two teams - DevOps and infosec - are living in worlds that don't readily mix, kind of like oil and water. 

Rick Howard: The last obstacle that is causing friction in our efforts to achieve our chicken nirvana is that for the most part, the security teams are still on the outside looking in at the IT teams. The IT teams don't want to slow down to implement the chicken suite, and the security teams don't want to change how they have always done things. That is unfortunate because DevSecOps is absolutely the linchpin to the entire metaphorical infosec wall that I've been talking about for the past few weeks. We are trying to reduce the probability of a material cyber event for our organization. It is one thing to use strategies like zero trust, intrusion kill chains and resiliency to bring that probability down, but if you can't pursue those strategies with any speed or with any consistency, you are likely to fail. The adversaries automate their attack sequences. The network defenders have no hope to keep up with them if they are trying to respond manually. The infosec teams can't go fast enough, and when they try, they make mistakes that the adversaries take advantage of. 

Rick Howard: The good news is that the infosec teams have their own infrastructure that they run and don't have to interact with the IT teams to make work. And at first blush, that seems to go against the idea of chicken nirvana and the building of a hybrid DevSecOps team, but it might be a useful first step. The systems in question are what the SOC team, or security operations center, run on a daily basis. It is the place where we can practice the DevSecOps philosophy without having to worry about breaking some essential business process that the IT team is responsible for because we don't know what we are doing yet in this new DevSecOps world. And the network defender community has made some progress in automating key tasks within that environment. 

Rick Howard: The big problem that SOC teams face is the extreme high volume of alerts that they have to review. It is not uncommon to have over a billion alerts show up in the SOC over a period of 90 days. If you are a Fortune 500-size company, it's even bigger than that. SOAR tools, or security orchestration automation and response tools, started to emerge for SOC use sometime in 2018. These tools purported to solve the telemetry volume problem. Through APIs, or application interfaces through software, the SOAR tool can connect to practically any kind of networking equipment and security tool deployed on your network, automatically collect telemetry from that device in terms of health and welfare and security alerts and provide the SOC team with the ability to easily write code that handles that telemetry in a prescribed way - things like, you know, save for the future or deleted or escalate it to a human for review. This has enormous implications. 

Rick Howard: In a typical SOC environment before SOAR, there are generally three tiers of analysts. Tier 1 analysts are the newbies. They have less than a year or two of experience. These are the console jockeys. Their job is to review each and every alert that comes into the SOC and decide what to do about it. They spend a lot of their time swiping left, not because the alert isn't important but because the volume is so high that they can't possibly look at all of them in any meaningful way. But if they discover something they are not sure about, they bump it up to the Tier 2 analysts. 

Rick Howard: The Tier 2 analysts typically have three to five years of hard-won experience and know how to handle most of the events that come into the SOC. If they can't, they bump it up to the Tier 3 team. When you think of the Tier 3 analysts, think of the stereotypical Unix graybeards. Now, they don't all look like that for sure, but most everybody has one or two of these people in their organization who've all been there, done that and wear the scars on their back because of it. They're the go-to people for all issues out of the ordinary. Now, this essential design of the SOC team hasn't changed that much since we invented it back in the 1970s. 

Rick Howard: With SOAR tools in place, we can essentially eliminate the need for humans to do Tier 1 and Tier 2 functions by replacing them with automation. We can reduce a billion alerts that come into the SOC every quarter to something more manageable that humans can look at, something in the range of 500. That reduces to about five events a day that the Tier 3 analyst has to spend time on. And that seems much more manageable. 

Rick Howard: Of course, we don't fire the Tier 1 and Tier 2 analysts. We change their job. For some of them, we turn them into DevSecOps developers and put them on the task of automating the other things we need to automate, like Kill Chain Rooster and Resilience Chicken. For others, we put them on the threat hunting team. Now, I'm going to discuss threat hunting teams later in this first principle series, but the point is that Tier 1 analysts don't learn a thing manning a desk and swiping left eight hours a day in the old SOC design. You can make a pretty strong argument, too, that the Tier 2 analysts, once they settle in, are not learning a lot of new things either. You take these two sets of employees and put them in the DevSecOps team and the threat hunting team and they become cybersecurity experts in six months. And that is the power of automation and one of the reasons that DevSecOps is an essential brick to our first principles infosec wall. 

Rick Howard: Many SOCs are doing this sort of thing now, and many others are anticipating that they will soon. They most likely don't consider the use of SOAR technologies as a DevSecOps function, but it absolutely is. And that is the first step - to get the idea of DevSecOps philosophy into the minds of the infosec teams that automation will drive everything, that coding is an essential skill for everybody. Once there, it will become time to reach across the aisle to the IT teams and learn what DevOps tools they are using to deploy infrastructure as code. 

Rick Howard: If they are using Jenkins, the SOAR team should use it to build coding chicken. If they are using Ansible, the SOAR teams should use that to build zero trust chicken. When the infosec team stumbles - and they will 'cause they're learning something new - or if they hit some sort of roadblock, they should reach out to the DevOps team for help. Start building the bridges so that you can ultimately shift left in the infrastructure as code process. 

Rick Howard: We have chosen the task of reducing the probability of material impact to our organization due to a cyberevent as the most important thing we have to do as security practitioners. That is the base pillar for our metaphorical infosec wall. The next three bricks we stack on that pillar to give us strength for zero trust, intrusion kill chains and resilience. But we will lose most of that intended strength if we don't treat each brick as a system of systems and maintain each at the speed of code. We can't respond to an automated adversary with manual processes. I had an old boss of mine, Mark McLaughlin, who used to steal a line from his favorite movie "The Untouchables" to illustrate the point. 

(SOUNDBITE OF FILM, "THE UNTOUCHABLES") 

Sean Connery: (As Jim Malone) ...Brings a knife to a gunfight. 

Rick Howard: McLaughlin always said you can't bring people to a software fight, and he was so right. DevSecOps is the brick that will get us there. Use SOAR as a starting point. Automate the processing of those alerts coming into the SOC, and use that as the first step of your DevSecOps program. And don't forget to reach out to your DevOps teams and get their advice and help using the DevOps tools they prefer while you're building your version of the chicken suite. 

Rick Howard: And that's a wrap. If you agree or disagree with anything I've said, hit me up on LinkedIn or Twitter and we can continue the conversation there. The CyberWire "CSO Perspectives" is edited by John Petrik and executive produced by Peter Kilpe; mix, sound design and original music by the insanely talented Elliott Peltzman. And I am Rick Howard. Thanks for listening.