Security infrastructure as code.

Rick Howard: Before we get started today, I know at the end of our last episode, I promised I would be talking about risk forecasting today. Well, best laid plans and all of that. We're going to push that episode to the next season for lots of reasons that you don't care about. But rest assured we'll get to it - just not today. And now on with the show.

Rick Howard: When I was a young captain back in the early 1990s, the U.S. Army assigned me as an assistant professor at the United States Military Academy to teach computer science in the Electrical Engineering and Computer Science Department, or EE and CS for short, affectionately referred to by the cadets as EECS. One of my additional duties as assigned was to manage the department's internal network for the faculty and the students. All the teachers had Windows NT systems on their desktops and in their classrooms, and we had a number of labs scattered about running Sun Solaris, the cool version of Unix back in the day.

Rick Howard: The cadets also had desktop computers in their barracks, too. Practically, what that meant for me and Marie, the GS-7 government employee system administrator who worked for me, was that every summer as the new class of cadets came in - about 1,500 plus or minus, depending on the year - and as the graduating seniors left to become second lieutenants in the Army, we needed to create an entire bucket of accounts with the proper permissions on both the Windows side and the Solaris side. And we had to delete a bunch of accounts, you know, because of security. 

Rick Howard: That first summer, July, Marie and I did that manually because I was the new guy and didn't know any better, and we had to get it done before August when all the cadets returned to the academy from the four corners of the world to begin the fall semester. That was a lot of brainless, repetitive toil in a very short amount of time. And because we did it manually, we made a lot of mistakes that we had to fix later when the cadets tried their first logins. OK, OK, I made a lot of mistakes. Marie was spot on. But the point is that I vowed not to ever be in that situation again. 

Rick Howard: So to prepare for the next summer, I automated it. Fresh out of grad school with a copy of the books "Unix Power Tools" and "Advanced Windows NT: the Developer's Guide to the Win32 Application Programming Interface" on my desk, I broke out the Perl programming language for Solaris and whatever programming language I was using for Windows NT at the time and automated the entire thing. When Marie and I got the file with all the cadet names that needed to be added and deleted, I pulled the trigger on the scripts. It took 10 minutes to run, and there were no mistakes. We reduced the toil that normally took Marie and me 30 days to accomplish down to 10 minutes. Now, of course, it took me 30 days of work to get the program running without any errors, but once it was done, it was done. I never had to touch it again if I didn't want to. 

Rick Howard: That was a significant emotional event for a young automator like myself. I mean, I started computer science in grad school and learned enough about programming to be dangerous. But I really didn't understand the power of automation until that very moment. Yes, automation was hard to do upfront, but once done, it took an entire class of busybody work off my plate so that I could focus on other, more important matters. And if that isn't the very definition of why automation is important, I don't know what is. 

Rick Howard: My name is Rick Howard, and I am broadcasting from the CyberWire's secret sanctum sanctorum studios located underwater somewhere along the Patapsco River near Baltimore Harbor. And you are listening to "CSO Perspectives," my podcast about the ideas, strategies and technologies that senior security executives wrestle with on a daily basis. 

Rick Howard: When I was in college - late 1970s and early 1980s - my future mother-in-law and I sat riveted through a 10-episode documentary hosted by British broadcaster, science historian, author and television producer James Burke. The show was called "Connections," and you can still watch it on YouTube for free if you want. It's about how the technology in our world that we all take for granted is connected in the most bizarre and unknown ways. And this is before the internet boom. If you grab James Burke today, he can make another 10 episodes on the evolution of software development alone, and it would be just as compelling. 

(SOUNDBITE OF TV SHOW, "THE ROCKY AND BULLWINKLE SHOW") 

Bill Scott: (As Mr. Peabody) Hello there. Peabody here. Once again, it is time to take another revealing peek back into history. 

Walter Tetley: (As Sherman) What famous date shall I set it to today, Mr. Peabody? 

Rick Howard: To understand the power of automation, we have to set the Wayback Machine to 1960. And I can hear my good friend Steve Winterfeld, the Akamai Advisory CISO and regular here at the CyberWire Hash Table, rolling his eyes. Oh, no. 

(SOUNDBITE OF TV SHOW, "HOME IMPROVEMENT") 

Tim Allen: (As Tim Taylor) Oh, no. 

(LAUGHTER) 

Rick Howard: Not another history lesson. Yes, Steve, at this point, I don't need to do the history lessons. I'm basically doing it just to annoy you. So strap in. Back in the dinosaur days, 1960s, when computers were bigger than houses, large software development projects didn't have a standard methodology yet. We were still figuring out what to do with these things called mainframes. Computer scientists leaned on established general purpose project management theory to do software development, like the original Gantt Charts from the 1910s and the Critical Path method made popular in the 1950s. 

Rick Howard: In 1956, Herbert Benington invented the first version of the Waterfall software development model. Interestingly, Benington didn't get credit for his work early on. Another guy, Dr. Winston Royce in 1970, got the credit when he published a criticism of the model that didn't even mention the model by name. But his paper had nice diagrams that showed the process, requirements, analysis, design, implementation, testing and operations, all flowing from top to bottom just like a waterfall. In 1976, Bell and Thayer referred to the Royce diagrams as the Waterfall model, and the name stuck to Dr. Royce. But when the personal computing revolution began in the 1980s, things began to spin out of control. Computer science founding father Edsger Dijkstra said this about the problem. Quote, "the major cause of the software crisis is that the machines have become several orders of magnitude more powerful. As long as there were no machines, programming was no problem at all. And now we have gigantic computers, programming has become an equally gigantic problem," end quote. 

Rick Howard: In 1985, to address the issue, the U.S. Department of Defense, or DOD, adopted the waterfall model as a requirement for all contractors, despite Royce's criticism, and started a period of ponderous, iceberg-like progress in producing software. According to Alexey Krutikov, quote, "although Royce himself believed that the waterfall method should be iterative and advocated pilots and micromodels, waterfall was and continues to be wrongly considered as a sequential methodology," end quote. The impact was that many programming projects took years to finish, and the team spent as much time documenting the requirements as they did writing the code. 

Rick Howard: In the 1990s, some rebel developers started experimenting with ways to improve the process. They began toying with the rational unified process in 1994, the Scrum in 1995 and extreme programming in 1996. But in February 2001, 17 men - and, yes, they were all men. We can have a longer discussion later about the misogyny of the IT community in the early 2000s - these guys traveled to Utah for a long weekend of skiing and discussions about building software. The result was the Agile Manifesto, a complete rejection of the waterfall model. And you can check out the show notes to read the manifesto's 12 points if you like. But in essence, it embraced the idea of producing real, working code as milestones of progress. 

Rick Howard: Up to this point, software development was mainly concerned with general-purpose coding. In other words, software projects built applications to solve specific problems in business, the commercial space and in academia. There wasn't a lot of talk about using software to run the IT infrastructure, and there wasn't much discussion about how to write secure software. This is the point where it all started to change. As a community, we started to see parallel development in improving security in all software as well as deploying code as infrastructure. 

Rick Howard: In May of 2000, hackers released the ILOVEYOU worm that began a string of global, impactful and malicious computer worms that targeted Microsoft Windows products - you know, their operating systems and their browsers. Just in 2001 alone, we got the Code Red worm in July, the Code Red II worm in August, the Nimda worm in September and the Klez worm in October. By February 2002 that next year, Bill Gates turned Microsoft on a dime to implement trustworthy computing. He shut down future deployments of the Windows operating system to redirect development focus on security. The result was the creation of the first-ever Microsoft Security Development Lifecycle, or SDLC. 

Rick Howard: In 2003, Dave Wickers and Jeff Williams, working for Aspect Security, a software consultancy company, published an education piece on the top software security coding issues of the day. That eventually turned into the OWASP Top 10, the Open Web Application Security Project, a reference document describing the most critical security concerns for web applications. 

Rick Howard: In 2004, when Google was nothing more than a search engine and not the internet giant that it is today, company leaders made an extraordinary decision. Instead of assigning the responsibility of network management to an IT team, as was the standard best practice of the day, they handed the task off to the development team. This group of site reliability engineers, or SREs, got busy automating all the infrastructure tasks that were repetitive, error prone and provided little value to the future for the company. They called those tasks toil and started the movement that wouldn't have a name for another six years - DevOps. 

Rick Howard: In March of 2008, Dr. Gary McGraw published the first Building Security in Maturity Model Report, or BSIMM for short, a survey of some 30-plus companies that collated initiatives and activities around software security. The next year, 2009, Pravir Chandra published the first SAMM - Software Assurance Maturity Model. This is a prescriptive security best practice security model. With these two models, the security community started to have a way to measure its progress against its peers in the industry - BSIMM - and against what other security experts recommended - SAMM. 

Rick Howard: Amazon started the cloud revolution when it rolled out AWS in 2006, Microsoft followed suit with a competing service in 2010 with Azure, and Google started to compete in the space with Google Cloud Platform, or GCP, in 2012. And there are other smaller cloud service providers out there that some of you are using. But when AWS rolled out, the cloud became the impetus for everyone in the IT community to consider infrastructure as code. But even as Agile replaced the waterfall model as the standard software development framework, it was still agonizingly slow. Startups born in the cloud realized that they could do better by using software to create a competitive edge against their brick-and-mortar competitors. With software, they could upgrade their products and services over the internet. They could run circles around their competition who still had to ship hardware. They could beat their competition to market for software applications if they could just streamline the process. 

Rick Howard: By 2009, DevOps began to emerge as an industry-disruptive idea out of three converging concepts - one, a 2009 Velocity Conference talk called "10+ Deploys per Day" by John Allspaw and Paul Hammond; two, the previously described Agile development method; and, No. 3, Eric Ries' book "Lean Startup," which influenced many Silicon Valley companies between 2007 and 2010. 

Rick Howard: DevOps is the idea that there needs to be a much tighter integration between software developers and information technology operations - that once the developers, the quality assurance teams and the security analysts pass any new code or maintenance updates to ITOps for deployment, their jobs aren't done. Instead of creating artificial black boxes within each team where updates come in, get worked on and then are passed to the next black box, DevOps is the recognition that update creation, deployment and maintenance is one big system of systems and needs to be managed that way. It's the idea that organizations would use the same Agile methodology they use today with their software development teams. But it expanded across all organizations in the deployment cycle - product managers, marketing professionals, developers, quality assurance practitioners, systems engineers, system administrators, operations staff, database administrators, network engineers and security professionals. In other words, DevOps uses the Agile philosophy across the entire lifecycle of deployed systems from design to development, to testing, to deployment, to maintenance and finally, to end of life. In 2013, Gene Kim, Kevin Behr and George Spafford published the Cybersecurity Canon Hall of Fame book "The Phoenix Project (A Novel About IT, DevOps, and Helping Your Business Win)." It captures the essence of the DevOps movement in a novel because the authors wanted it to be accessible to more people, not just the tech nerds, but specifically to general purpose business leaders. Like many who have read this book, I found that the way the novel portrays the CISO of the fictitious company hits pretty close to home. 

Rick Howard: By 2014 or so, the big internet giants - like Google, Amazon, Netflix and Facebook - had become who they were, stand-alone leaders in their industry, due in no small part to their adoption of the DevOps philosophy. Their competitors, who traditionally used the old Waterfall software development method, might take years to deploy a new service for their customers. The DevOps companies were doing 10 deployments a day. At this point, you might be asking yourself - this is all well and good. But hasn't security fallen off the map? With OWASP, BSIMM and SAMM, we were at least in the discussion. But in the years between 2008 and, say, 2017, it seemed that the IT community and their new and fancy DevOps model sprinted away from the security community. Even in "The Phoenix Project" novel, the security leaders weren't part of the DevOps movement. They were outsiders, not convinced about the new direction. They eventually came around. But it took the entire novel. 

Rick Howard: John Willis, one of the authors of "The DevOps Handbook," said in an interview in March 2021 that everybody involved in the DevOps movement was patting themselves on the back by creating this great thing. But they almost completely forgot about security for eight years or so. People were talking about DevOps and security, but not with any detail. And then around 2017, Shannon Lietz, then working for Intuit, staked a claim for the DevSecOps phrase. She created a foundation and a website dedicated to the purpose of putting security into DevOps. Now, there was some controversy there because many in the DevOps movement thought that they had invented DevSecOps. But according to Willis, none of that matters. By creating the foundation, she got the idea front and center again in both the IT and security communities. And DevSecOps started to gain traction. 

(SOUNDBITE OF ARCHIVED RECORDING) 

John Willis: You know, there's always been, obviously, discussion around security and DevOps, right? And I get pushback from friends, actually - say, well, John, don't you remember 2013 devopsdays Austin? And I talked about security and operations and dev. But Shannon Lietz, who's over at Intuit, she coined the term, right? Like, it's no big deal. But it is a big deal. 

Dan Lines: Yeah. 

John Willis: Like, she literally created a website, said this is this thing, DevSecOps. Everybody got all upset and mad and, you know - because, you know, she just put a stake in the ground on it. And a lot of us fought it, right? And then all of a sudden, this DevSecOps come out. And I'm like, here we go again, somebody trying to rename to own the name. But when you got to meet Shannon and you read the sort of literature and then you went back and looked at the problem is, we went almost, like, maybe eight, nine years of sort of ignoring security. So all of us, like, DevOps people patting ourselves on the back and like, we're so awesome. I wrote this book. No, I wrote that book. I used to do a presentation where I'd say - I'd take the mic. And I'd take my hat. And I'd throw it on the floor. And I'd go, God, darn it, we forgot security, you know? And a lot of people argue that we didn't. 

Rick Howard: In 2019, Gartner placed DevSecOps on their hype chart as just coming out of the trough of disillusionment, to begin the climb up the slope of enlightenment, with about 5 to 10 years away from reaching the plateau of productivity. And that seems about right, as vendors, like Palo Alto Networks, my old alma mater, are starting to roll out infrastructure-as-code services to support their cloud offerings. And the same year that Gartner released their hype chart, the U.S. Department of Defense formalized their own process by publishing Version 1 of their DevSecOps Reference Design document. The bottom line is that if your organization has adopted some form of the DevOps model, then you most likely have a version of the continuous integration/continuous delivery pipeline - or CI/CD for short. 

Rick Howard: The CI pipeline is the DevOps best practice in which, according to synopsis, quote, "incremental code changes are made frequently and reliably. Automated build-and-test steps triggered by CI ensure that code changes being merged into the repository are reliable. The code is then delivered quickly and seamlessly as part of the CD process," end quote. These pipelines are complex infrastructure-as-code software projects that, according to Teri Radichel, an AWS DevSecOps expert, quote, "require appropriate architecture and design by the right people, not just technology. The CI/CD pipeline is part of a larger security architecture that must be well thought-out. Otherwise, your security strategy will either be eternally talking about the cloud but never getting there, or akin to herding cats," end quote. Well said, Teri. But herding cats is not a new thing in the security community. So let's saddle up and get on with it. 

(SOUNDBITE OF ARCHIVED RECORDING) 

Unidentified Actor #1: (As character) This man right here is my great-grandfather. He's the first cat herder in our family. 

Unidentified Actor #2: (As character) Herding cats, don't let anybody tell you it's easy. 

Unidentified Actor #3: (As character) Anybody can herd cattle. Holding together 10,000 half-wild shorthairs? Well, that's another thing altogether. 

Rick Howard: In order to reduce the probability of material impact, I have advocated for several first principle strategies - zero trust, intrusion kill chain prevention, resilience and risk forecasting. Underneath each of those strategies are a number of tactics that support it. For example, for the zero-trust strategy, one tactic required is a robust identity and authorization program. For the intrusion kill chain prevention strategy, you need the ability to build adversary playbooks. In order to reduce the probability of material impact, I have advocated for several first-principle strategies - zero trust, intrusion kill chain prevention, resilience and risk forecasting. Underneath each of those strategies are a number of tactics that support it. For example, for the zero-trust strategy, one tactic required is a robust identity and authorization program. For the intrusion kill chain prevention strategy, you need the ability to build adversary playbooks. The truth of the matter is that if the security community has any hope of making progress in these four big strategies, we have to automate the toil, the manual and repetitive security work that has to be done in order to build and maintain the tactics that support these strategies, just like I did back in the day when I automated the account creation process for the West Point cadets. Consider this automation effort as the glue that binds everything together and makes the entire effort a system of systems that has feedback loops into the various pieces and parts. Automation has to be the fifth first-principle strategy that we are all pursuing. DevOps or DevSecOps has to be the tactic we implement. 

Rick Howard: But security teams can't or shouldn't do this in a vacuum or in parallel to what the IT side of the house is already doing. Why reinvent the wheel? After all, the entire mantra of the DevOps movement is to move development and operations onto the same team, not keep them in black-box silos that never talk to each other. The security community has to attach ourselves to the existing CI/CD pipeline process. In other words, we have to become part of the internal DevOps program, not resist it or build our own. Specifically, we have to find ways of inserting code into the pipeline that supports each of our strategies. For zero trust and code coming into the CI pipeline, one, build separate regression test code to ensure this new code follows the best practice recommendations from OWASP, BSIMMs and SAMM, and flag for upgrade if necessary before sending to the delivery pipeline. Two - for software asset inventory, or SBOMs, software bill of materials, collect names of new software assets, open-source code libraries and version information. Three - for vulnerability management, check incoming code against a known vulnerabilities list, and flag for upgrade if necessary before sending to the delivery pipeline. And finally, four - check that the permissions granted to each software module is in line with the stated zero-trust policy or need-to-know. 

Rick Howard: For intrusion kill chain prevention, this is where we can monitor each system in our existing security stack toolset. One - collect telemetry of the security controls actually deploying for the approximately 250 known adversary playbooks in existence. And two - send new prevention and detection configuration updates as we discover new intelligence or find more innovative ways to mitigate. For resilience on all the material data operated on by all software, one, ensure that it's encrypted before sending the code to the delivery pipeline. Two - ensure that it's configured to be part of the organization's backup scheme. And three - ensure that it's configured to be part of the organization's recovery scheme. 

Rick Howard: And finally, for risk forecasting, this is where we can collect evidence from each of the first principal strategy's CI modules to feed into the risk forecasting model. One - what percentage of the CD pipeline code is in compliance with our zero-trust policies? Two - what percentage of the known adversary playbooks do we have prevention and detection controls in the CD pipeline? And finally, No. 3 - have we seen evidence of any known adversary playback activity in our known environments? 

Rick Howard: Before I wrap this up, there is another discussion entirely around SOAR, or security orchestration automation and response, SIEM, security information and event management, and XDR, extended detection and response, via APIs and SASE, secure access service edge, that contribute to this automation first-principle idea. I've done separate shows for each of those topics, and you can find links in the show notes for each. But all of it is a big lift for the security executives to get their hands around and likely foreign to how most of us have done business for the past 20 years. But it's time to make the change, to shift left, as they say, to get this done. 

Rick Howard: The immediate impact to the security executive is that some portion of your team, maybe the biggest portion, will become part of the internal DevOps movement, maybe as developers but most definitely as product managers for each of these elements of your first-principle strategy. That's something to consider when budgeting season comes around and you are considering the skillsets of your team. We've been wrestling with the idea of software development methodologies like Waterfall and Agile, infrastructure as code, like cloud deployments and DevOps and DevSecOps and coding best practices from OWASP, BSIMMs and SAMM going on now for two decades. These are not independent systems; they overlap and interact. Up to this point, at least for the security side, they have been manual tasks, toil, that are prone to mistakes. We all know that automation can reduce the impact, at least be consistent with the mistakes we make and can offer a uniform fix across the enterprise once we've decided what to do. Automation is the key first-principle strategy to get this done. And DevOps - or DevSecOps, if you will - is the tactic we will all use to get there. 

Rick Howard: And that's a wrap. In the show notes, you can find the 12 points for the Agile manifesto, or you can just Google it. They're easy to find. Also in the show notes is a software development timeline from the 1950s to about 2019. That you can't Google. I had to put that together myself. And that's it for Season 8 of the "CSO Perspectives" podcast. We will be on a short sabbatical for the next few weeks as we put the finishing touches on the Season 9 episodes, which we will start publishing at the end of April. I hope that you've enjoyed this season. I know I have. And I certainly learned a lot. And as always, if you agree or disagree with anything I have said, hit me up on LinkedIn or Twitter, and we can continue the conversation there. Or if you prefer email, drop a line to csop@thecyberwire.com That's C-S-O-P, the @ sign, thecyberwire - all one word - dot com. And if you have any questions you would like us to answer here at "CSO Perspectives," send a note to the same email address, and we will try to address them on the show. 

Rick Howard: The CyberWire "CSO Perspectives" is edited by John Petrik and executive produced by Peter Kilpe. Our theme song is by Blue Dot Sessions, remixed by the insanely talented Elliott Peltzman, who also does the show's mixing, sound design and original score. And I am Rick Howard. Thanks for listening.