CSO Perspectives is a weekly column and podcast where Rick Howard discusses the ideas, strategies and technologies that senior cybersecurity executives wrestle with on a daily basis.
Software Bill of Materials (SBOMs): A Rick the Toolman episode.
This weekend, my daughter called seeking advice about the red warning light flashing in her Subaru Crosstrek.
You may be surprised to learn that I'm not much of a car guy (well, maybe not). I mean, any old white guy like me who spends a lot of time obsessing about superheroes probably doesn’t spend a lot of time in overalls underneath cars worrying about brake lines, carburetor timing, or compression loss. I mean, if you put a gun to my head, I could probably manage changing the oil and fixing a flat tire, but it wouldn’t be pretty. And if I'm lifting the car hood, toolbox by my side, with the intent to fix some mechanical thing, my best advice to any standers-by is to run in the opposite direction as fast as possible for fear of becoming some kind of collateral damage when the engine explodes.
I'm just saying.
But since I have the toolbox right there, it sounds to me that we are getting ready for another Rick-the-toolman essay.
Automobile manufacturing is similar to DevOps.
So, when I tell you that my advice to the daughter consisted of recommending a call to the local auto repair guy, you won’t be shocked.
But that got me thinking about repair parts she might need; which got me wondering about where, exactly, did car manufacturers get their parts to build the cars in the first place; which led to me to think about the automobile assembly line process; which made me remember the Cybersecurity Canon Hall of Fame book, The Phoenix Project, where the authors explained that how we build software today should be similar to how Toyota reimagined the car manufacturing process after World War II into the famous Toyota Production System (TPS).
I'm sorry you had to hear all of that. That was a small glimpse into how my mirror maze of a mind works. Trust me, you don’t want to go in there without a guide. Just take your children by the hand and walk, don’t run, to the exits. You’ll be fine.
Going through that thought process made me realize something. Comparing software development to an automobile production line is all true at a macro level, but it breaks down at the micro level. What I mean is that the Toyota Production System applied to software development is the DevOps movement, and I'm a big believer in adopting its philosophy. It gets you away from the old waterfall software development method of producing one new working piece of code every couple of years, if ever, to ten new deploys in a day. That’s the macro level and that works.
At the micro level though, the day-to-day operational level, the Toyota Production System metaphor breaks down. Each part (and I'm talking about hardware parts here, not parts that have a software component) either comes from the same trusted contractor or you build it yourself. Once built, they don’t change that much. There are exceptions to this of course, but that’s the norm. The point is, these hardware parts come from the same place every time you make an order. You know for the most part who those people are. That’s not true in software development.
Commercial code is open source code.
According to a report by Synopsys in 2022, 97% of commercial code has an open source component. And that’s not all. Within that 97%, the bulk of that code base consists of open source software (78%.) Let me restate that. Almost all commercial software is over three-quarters open source. That means, unlike the hardware parts that feed the Toyota Production System, we have no idea where our software parts are coming from, who built them, and whether or not the people who built them are maintaining them. The problem is even more insidious than you might think. If most software developers are using open source software, there is a good chance that whoever built a specific open source component also used a different open source module to create it. The whole idea spirals exponentially in a fractal kind of way. And that's not even mentioning that hardware parts manufacturers don’t have to contend with armies of researchers that routinely find ways to exploit those software parts for criminal, espionage, and continuous-low-level-cyber-conflict purposes. The use of open source software provides these adversary groups a wide-open attack pathway that we all call the software supply chain. Compared to car manufacturers, DevOps people have to operate at an entirely different level.
Software supply chain and cybersecurity first principles.
We’ve seen a handful of these so-called software supply chain attacks in the past 20 years, but recently, it feels like the bad guys have re-discovered this attack strategy and doubled down on it. Just in the last five years, we have observed highly impactful attacks against third parties like MEDoc, Solarwinds, Asus, CCleaner, Kaseya, Accellion, and Codecov, just to name a few. These kinds of attacks are even more machiavellian than a straight-up frontal assault against our own cyber defenses that we designed, deployed and spent considerable resources on. With a supply chain attack, it doesn’t matter how mature those programs are. The hacker’s initial compromise in ƒtheir attack sequence comes right in through a wide open front door to our data island defenses, propped open with a chair to make entry even easier. Cyber adversary groups know this and, as Dmitry Raidman (the CTO at Cybeats) says, "The bad guys are on the hunt for vulnerable open source software (OSS) supply chain components so that they can trojanize other legit commercial and open source products.”
As an aside, to break the chain of this attack sequence, even if the initial breach was successful, robust deployments of our first principle strategies will work. Even if the adversary group is successful in establishing a beachhead somewhere in our environment, those strategies have a high probability of preventing any kind of material impact.
But if we are pursuing zero trust as one of our first principle strategies, it’s ludicrous to think that we, as a community, don’t have a handle on supply chain defenses yet. We have been busily running around trying to identify all of our employees, contractors, and devices, and then prescriptively authorizing the workloads they have access to. In parallel, we have mostly turned a blind eye to new software coming in through the front door to update our production systems. The cybersecurity leaders who do monitor and manage this situation rely on manual, homegrown, and incomplete tooling to get this done.
The solution that will solve this problem for us all is a concept called a software bill of materials, or SBOMs.
SBOM concept and history.
We don’t really have a standard SBOM platform yet. What we do have is a bunch of developing standards and requirements for tools that will help us reduce the risk of software supply chain exposure. Although vendors are starting to sell SBOM platforms, the idea of an SBOM is, at this point, still more of a concept than a reality.
In its simplest form, an SBOM is a formal record containing the details and supply chain relationships of various components used in building software. They are lists of nested software components designed to enable supply chain transparency. So, if my business runs a software application called Fortnite (because I’ve been trying to convince my boss that playing Fortnite builds team camaraderie. So far, he’s not buying it, but I'm still trying), an SBOM will list all of the component pieces to the software; all of the original code written by the developers at Epic (Fortnite’s owner), plus all the open source components they used to make it run, plus, all the fractal sub-components built by other open source developers. According to Dmitry, when thinking about this, it's good to distinguish between “SBOM Producers (upstream, focused on their products) and SBOM Consumers (downstream, focused on the upstream vendor’s products).” Both sides would use tools that follow a formal specification of a standard.
Pertinent SBOM standards.
Spearheaded by the Linux Foundation back in 2010, the Software Package Data Exchange® (SPDX®), also known as ISO/IEC 5962, became the international open standard for security, license compliance, and other software supply chain artifacts last year (September 2021). In other words, they became the official SBOM standards body. Despite only being internationally recognized for a short while, companies like Intel, Microsoft, Sony, and VMware are already using the SPDX® standards to communicate SBOM information. SPDX® was not an overnight sensation though. It was the result of ten years of collaboration from vendors across the Software Composition Analysis (SCA) space; tools that assess open source software, code libraries, and containers, to provide a unified view of risks and remediations and offer strategies to keep this kind of software up to date.
In 2015, the International Organization for Standardization (ISO) released ISO/IEC 19770-2 for Software identification tags (SWID). This format creates a template in XML format to identify and describe software components and relevant patches. Then, in 2017, the Open Web Application Security Project Foundation (OWASP), designed CycloneDX, a lightweight standard with features of both SPDX® and SWID.
As I said, the SBOM today is a collection of emerging standards that are not quite there yet, but the community is very close to having something usable.
Presidential directive.
The SBOM concept got a big push in 2021 when President Joe Biden signed an executive order on cybersecurity, E.O. 14028, that mandates that all Federal Civilian Executive Branch Agencies (FCEBs) and Key Players meet or exceed specific cybersecurity requirements including the development of an SBOM program. FCEBs are
- Administrator of General Services
- Assistant to the President and National Security Advisor (APNSA).
- CSPs (Cloud Service Providers)
- Director of the Cybersecurity and Infrastructure Security Agency (CISA).
- Director of the Office of Management and Budget (OMB)
- Federal Acquisition Regulation (FAR) Council.
- Secretary of Homeland Security.
- Secretary of Defense.
I was talking to Dr. Georgianna Shea this week about SBOMs in general and specifically about President Biden’s directive. She is an old friend of mine, a regular at the Cyberwire Hash Table, and one of the world’s experts on the SBOM concept. She’s been tracking the progress of all the FCEBs in relation to the Presidential Directive and says that, for the most part, the government is meeting its deadlines. But she caveats that a bit by saying, “the requirements weren’t to be operationally executable. So the actual requirements were easy to meet. There were 2 specific SBOM requirements in the EO: 1 - identify the minimum elements and 2 - provide guidance addressing an SBOM.” The government met both of those. So, all the FCEB’s are on track but there is still a lot of work to do.
3 Tools for supply chain risk reduction.
It turns out that an SBOM is not the only tool that we need. Once we have a handle on all of our software components with an SBOM, it would be nice if there was a place that was the authoritative source for automatically discovering vulnerabilities and exploits in component software. This would be an upgrade to the Common Vulnerability Scoring System (CVSS). Instead of me reading reports on software vulnerabilities and trying to determine if they apply to the data in my SBOM, the upgraded CVSS would be machine readable and allow another system, an asset management system, to review my SBOM information and compare it to this upgraded CVSS. We don’t really have that yet either.
But what we do have is a standard way to articulate vulnerability information in component software. It’s called a VEX document (Vulnerability Exploitability EXchange). According to Dmitry, “VEX is a result of the work by the continuation of the NTIA Working group that created the SBOM standard” and is managed by CISA. The NTIA is the National Telecommunications and Information Administration and their working group created this format that we can use to store vulnerability and exploit information about software components. Dmitry says that OASIS has published a draft standard in the Common Security Advisory Framework (CSAF) to describe VEX in machine readable format similar to CycloneDX. Currently, both CSAF and CycloneDX both support the encapsulation of VEX information in their object structure.
What VEX isn’t is a system to store that information in any automated way. The bottom line is that if we are going to reduce the risk of software supply chains, we need an SBOM system (1), an asset management system (2) that is the go-between to the upgraded CVSS system that stores information in the VEX format.
Future of SBOMs is bright.
In fairness, the concept of an SBOM has been around for many years, but there wasn’t a lot of incentive out there for a massive push to get it done. So, quietly in the background, volunteers organized by the NTIA, the Linux foundation, OWASP, and others, began work to establish some standards. When supply chain attacks started to rise these past few years coupled with several hugely impactful ones (Solarwinds, MeDoc, and Accellion), somebody convinced President Biden to include SBOMs as part of a presidential directive. That’s the good news.
Progress on SBOM development is happening. It helps that the U.S. government, at some point, will mandate that all software suppliers provide SBOM information as part of their contract. I believe that will tip the dominos and the rest of the industry will follow that lead. But I'm guessing that we are still five years away from the SBOM transitioning from a concept to a real thing. Dmitry disagrees and is more of an optimist. He points to the fact that he has witnessed extensive efforts from organizations “ranging from startups to fortune 500 companies and surprisingly most of them have already figured out how to generate SBOMs.” He says they fundamentally understand the return on investment at managing SBOMs at scale. He thinks that by the end of 2022, SBOMS will become a real thing.
If you are looking for more detail, Dr. Shea has written several papers on the subject and she knows who all the key players are in the government and in the commercial space. Also, the National Telecommunications and Information Administration has published a small fact sheet called “SBOM myths and Facts.” Finally, Dmitry was deeply involved in the NTIA standards process and has written an essay on the DevOps.com website. Links are below and I highly recommend them.
References.
11 MAY 2020:
CSOP S1E6:: Cybersecurity First Principles
- Hash Table Guests: None
- Link: Pro Podcast (6)
- Link: Pro Transcript
- Link: Pro Essay
- Link: Public Podcast
18 MAY 2020
CSOP S1E7:: Cybersecurity first principles: zero trust
- Hash Table Guests: None
- Link: Podcast (7)
- Link: Transcript
- Link: Essay
08 JUN 2020:
CSOP S1E10:: Cybersecurity first principles - DevSecOps
- Hash Table Guests: None
- Link: Podcast (10)
- Link: Transcript
- Link: Essay
15 JUN 2020:
CSOP S1E11:: Cybersecurity first principles - risk
- Hash Table Guests: None
- Link: Podcast (11)
- Link: Transcript
- Link: Essay
11 JAN 2021:
CSOP S4E1: SolarWinds through a first principle lens.
18 JAN 2021:
CSOP S4E2: Solarwinds vs First Principles Hashtable Interviews
- Hash Table Guests:
- Gary McAlum, USAA CSO (3)
- Don Welch, Penn State CIO (2)
- Link: Podcast (34)
- Link: Transcript
- No Essay
14 MAR 2021
CWX: SolarWinds, SUNBURST, and supply chain security.
- Guests:
- Ryan Olson, Palo Alto Networks (Unit 42) Threat Intelligence VP (2)
- Bill Yurek, Inspired Hacking Solutions President
- Matt Cauthorn, Cyber Security & Cloud Evangelist at ExtraHop (Sponsor)
- Link: Podcast
- Link: Transcript
- No Essay
7 FEB 2022:
CSOP S8E3: Pt 1 – Supply chains.
14 FEB 2022:
CSOP S8E4: Pt 2 – Supply chains around the Hash Table.
- Hash Table Guests
- Amanda Fennell: the Relativity CIO and CSO (1)
- Link: Podcast (83)
- Link: Transcript
- No Essay
7 MAR 2022:
CSOP S8E6: Vulnerability Management: An essential tactic for zero trust from the Rick the Toolman Series..
“2022 Open Source Security and Analysis Report,” by Synopsys,” 2022.
“A Software Bill of Materials Is Critical for Comprehensive Risk Management,” by DR. GEORGIANNA SHEA, The Foundation for Defense of Democracies (FDD), 29 September 2021.
“A Software Bill of Materials Is Critical for Comprehensive Risk Management: TCIL Technical Note,” By Dr. Georgianna Shea, The Foundation for Defense of Democracies (FDD), 29 September 2021.
“SBOMs: Securing the Software Supply Chain, “ by Sam Ingalls, ESecurity Planet, 26 October 2021.
“Supply Chain Attacks in 2021: It Takes a Village,” by Sean Nikkel, Digital Shadows, 4 August 2021.
“The Complete Guide to SBOM (Software Bill of Materials),” by Eran Orzel, Argon Security, 27 January 27 2022.
"The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win,"
by Gene Kim, Kevin Behr, George Spafford, Published by IT Revolution Press, 10 January 2013.
"The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win
Book Review," by Rick Howard, The Cybersecurity Canon Project, Ohio State University, 11 May 2017.
“Understanding Vulnerability Exploitability eXchange (VEX),” by Derek Kruszewski, aDolus Technology, September, 2021
“SBOM Myths vs. Facts,” by NTIA Multistakeholder Process on Software Component Transparency, National Telecommunications and Information Administration, 1 November 2021.
“Thousands of Npm Accounts Use Email Addresses with Expired Domains,” by Catalin Cimpanu, The Record, Recorded Future, 11 February 2022.
“What Is VEX and What Does It Have to Do with SBOMs?” by Derek Kruszewski, Adolus.com, 2021.
“What Is VEX? It’s the Vulnerability Exploitability EXchange!,” by Frederick Kautz, zt.dev, 2021.
“What Needs to Be in an SBOM?” by Zachary Comeau, My TechDecisions, 15 March 2022.
“Why It’s Time for Cybersecurity to Go Mainstream,” Hosts Dave Bittner and Rick Howard, Guests: Dr. Georgianna Shea and Chris Hallenbeck, Cyberwire-X, The CyberWire, 26 September 2021.
“Why We Need a Software Bill of Materials Industry Standard,” by Dmitry Raidman, DevOps.com, 20 August 2020.