CSO Perspectives (Pro) 5.24.21
Ep 48 | 5.24.21

New CISO responsibilities: supply chain.


Rick Howard: In his 2014 book, "No Place to Hide: Edward Snowden,  the NSA,  and the  U.S. Surveillance State," Glenn Greenwald published photos of three TAO personnel intercepting the shipment of new Cisco routers en route to paying customers in Syria, inserting backdoor electronics into the device, and then shipping the modified routers to the intended destination. 

Rick Howard: TAO stands for Tailored Access Unit, now called Computer Network Operations, and is the NSA's hacker group charged with penetrating the computers of foreign governments and other targets overseas for the purpose of cyber espionage. They are the yin to the Russian SVRs yang. The customer in this case was the Syrian Telecommunications Establishment, or STE, and they wanted to use the Cisco equipment to support the country's internet and wireless backbone.

Rick Howard: Greenwald got the photos from the Edward Snowden data dump. Snowden, for those of you that have lived under a rock for the past decade, is the former NSA contractor hired to administer the NSA's high side networks, but he believed that the NSA was overstepping its legal authority to collect foreign intelligence.  As the poster child for why we all need a zero trust architecture, he logged into the NSA's high side network, ran a web crawler that he purchased on the dark web for a hundred dollars, and managed to abscond with some 1.5 million classified documents and then proceeded to release them to the public.

Rick Howard: The NSA used this particular photo as part of training material slides labeled TOP SECRET Communications Intelligence, don't share with foreign entities. On the slide, the NSA stated the purpose of this operation, "Such operations involving supply chain interdiction are some of the most productive operations in TAO, because they pre-position access points into hard target networks around the world." 

Rick Howard: Hey, no judgments here. If I'm running my country's hacker group charged with penetrating our frenemy's' networks, this is exactly what I would do. The point is that if the U.S. Is doing it, so is every major foreign power worth their salt in the hacking department. And I'm looking at you, China, Russia.

Rick Howard: When John Kindervag, the man who formalized the zero trust concept back in 2010, said to assume that your networks are already compromised, this is one example of what he was talking about. I was the Palo Alto Networks Chief Security Officer when all of the Snowden stuff hit the fan, and we had international customers who, quite frankly, demanded to know what we were doing to prevent this kind of en route tampering operation for our own equipment. We had an entire team dedicated to putting up roadblocks that made this way more difficult to accomplish, everything from manufacturing our own products, meaning we didn't farm it out to cheaper foreign entities, to using special colorized seals on all of the locking screws that secured the hardware boxes and made it obvious if somebody tampered with them, to using colorized electronic components that you couldn't just buy out at Radio Shack. 

Rick Howard: But if I can generalize here, there are really three types of supply chain attacks security executives need to consider. The first is just inbound widgets, every organization, commercial, academic, and government, purchases, supplies from outside entities. In the world of Internet of Things, that is one avenue a TAO-like organization might try to penetrate like the Cisco operation. 

Rick Howard: The second is software supply chain. Everybody is running software today that we have agreed can be automatically updated with the latest patches and features, think SolarWinds or the MeDoc accounting software that the Russians leveraged against Ukraine.

Rick Howard: And finally, for those organizations that make products, either software or hardware, what do you do to protect the manufacturing and content delivery pipeline so that your customers trust you to deliver uncorrupted products? Meaning if you're Microsoft regularly updating your operating system software or Siemens building hardware programmable logic devices, or PLDCs, how do you protect your customers from the Russians, the Chinese, and the U.S. inserting malicious code into your software and hardware products?

Rick Howard: These are all big jobs. Any one of them could probably consume an entire team full time, but since all of them directly impact the probability that the organization might be materially impacted due to one of these attack vectors, does that mean that the CSO should have overall responsibility to protect against them?

Rick Howard: Let's find out. 

Rick Howard: My name is Rick Howard. You are listening to CSO Perspectives, my podcast about the ideas, strategies, and technologies that senior security executives wrestle with on a daily basis. 

Rick Howard: I've invited two security executives who are quite familiar with all three supply chain attack categories. The first is an old friend of mine and a regular at the Hash Table Ann Johnson, Microsoft's Corporate VP on Security, Compliance and Identity. And the second is an old Army buddy of mine, Ted Wagner. He and I worked together when I ran the Army CERT back in the early 2000s, and he was my deputy several years later when I was the CSO for TASC, a small beltway bandit company here in DC. Today, he's a big and important CISO in his own right at SAP National Security Services.

Rick Howard: I started out by asking Ann to describe just how different the CISO role is today compared to what it was just 10 years ago. 

Ann Johnson: CISOs today are so different than CISOs 20-25 years ago when CISOs were largely focused on policy, then you had IT that actually ran the systems. Maybe the CISO would be able to stand up a SOC. Maybe they'd have some joint accountability for the network admins, but really, really different type of environment that we're in today. I do see more centralization of both systems and control and security architecture, and security engineering existing within the CISO function. 

Ann Johnson: As a matter of fact, if anything, we're being the, we're asking them to do way too much today, right? We're asking them to be policy experts, privacy experts, to understand compliance, and also to understand security. And by the way, that security is on premises, that security is in the cloud. That security as hybrid, it might have multiple cloud providers. So we're asking an awful lot of the CISO function as it exists today. 

Ann Johnson: And the final thing I would say is they need to be very, very articulate in communicating with the board, right? In telling the board of directors and very senior executives, the risk profile of the business, communicating with them about what they need, communicating about, about things that could fundamentally cause harm to the business, and it takes a really unique and special person today to fulfill that CISO role.

Rick Howard: Ted and I I commiserated about how easy we had it just seven years ago when we were working as the CISO and Deputy CISO at TASC. 

Ted Wagner: I'm the corporate CISO,  but, I also have supervision of  the customer facing cloud. A CISO back in the day when we were just worried about the corporate network. Now I'm worried about development. I'm worried about a very large cloud footprint that federal customers are in. If it was just the corporate network, it would be easy.

Rick Howard: When you look at the SolarStorm campaign in terms of the intrusion kill chain model, where the Russian foreign intelligence service, or SVR, compromised the SolarWinds company and inserted malicious code into the software update mechanism, you realize that the supply chain attack wasn't the kill chain step that caused the damage. The damage came after lateral movement and the compromise of administrator credentials needed for the authorization of access tokens to cloud resources, which by the way, nobody was watching. 

Rick Howard: Which begs the question, even if you were a victim of the SolarStorm campaign, if you had a robust zero trust strategy deployed, would you have been okay? Isn't it true that the subversion of the SolarWinds software update mechanism, while scary, is not the major problem here.? The real problem is that we didn't protect our critical assets, namely their authorization credentials for cloud resources.

Rick Howard: I asked Ted if he thought zero trust was a good strategy here.

Ted Wagner: I'm a big supporter of zero trust. I think it's combination of things. How does a CISO sleep at night? It's not because of one thing that the team does or two things that the team does it's in aggregate all the, the things that you put in place. 

Ted Wagner: So 802.1x implementation to protect ports and accessing your network. Requiring a certificate to authenticate the device. Multi-factor authentication as part of your identity management. Restricting access to anything that does not have a valid reason for accessing your environment. And then having a lot detection capability around your perimeter at the end point. 

Ted Wagner: It's an aggregate that you gain security. It's not one thing. It's not a great analyst looking for the adversary. It's a lot of things programmatically in place that compliment each other, and zero trust is a good model and I, I think he can only expound upon that.

Rick Howard: I asked Ann about some of the things she is recommending to CISOs about improving.

Ann Johnson: That's a really hard question. I think if you see any anomalous behavior on your computer immediately post update.  Some of the things are not obvious. You would never see it. So if you have good malware detection everywhere, if you're actually doing zero trust, if you're actually looking, interrogating every transaction. Let's say you do get a bad update from somebody. There's something happening in your network environment. 

Ann Johnson: One of the security researchers, I can't remember who it was, but he mentioned that the attackers have your security stack and they're testing against it. So one of the things that we saw in SolarWinds across a large customer base, is that they figured out how to get some of the endpoint detection solutions to throw off low fire informational events. They flew under the radar. 

Ann Johnson: One thing we're telling folks is if you're seeing hundreds of low fire, informational events across a set range of IP is in a certain period of time, that should signal something to your SOC and your machine learning engine, whatever you're using aggregates signal needs to be smart enough, not just to take medium and high priority events.

Ann Johnson: So they understand that the bad actors are doing all their opsec against your current security tools. And they know how to defeat them to a certain extent, right? Everybody has anti-tampering. Everyone improves our anti-tampering all the time, but they know how to get the right signal sent cause they know how your SOC works too.

Ann Johnson: That's the problem. They've done so much reconnaissance, they understand us really well. So you need to change the playing field to the extent of it's like the Sun Tzu of a cyber war. You need to change the playing field so that you're looking for things that they don't expect you to be looking for. And automation is the one thing that will help you in doing that because it should be able to aggregate that kind of signal. And if you're getting that type of anomalous detection, even if it's low fi in a certain period of time in your environment, that should be a flag.

Ann Johnson: Globally people are consolidating vendors. Not just for financial reasons, but for security and control and even risk reasons. One of the customers I was talking to and I really thought she, she put it in good perspective. She said, you know, we do a lot of talk about controls and our vendor surveys. Are you using MFA? We'll just keep using MFA as an example, but, and she said, we need to actually get a lot more precise in what we're asking the vendors. It's not a, are you using MFA? Is, is the person that's accessing my environment using MFA 100% of the time? She said, and then I need to work with my systems people to make sure we're requiring it. It's not just to be enough to ask the vendors, are they doing it? She said, we're not closing the gap to see if our systems are even requiring some of the things that the vendor survey says they're doing. And I thought that was really important.

Rick Howard: Both Ann and Ted work for commercial companies who provide software updates to their customers across the internet. Ted has the direct responsibility of securing that delivery pipeline.

Ted Wagner: We sell and support and deliver SAP software in the cloud. We leveraged the SAP software development program, but here recently, we've started developing our own software in-house to deliver it to customers. We have a segment which is called NS2 Mission which is done directly for customers. But now we are making it more commercially available, and as part of developing that process to deliver software to customers, we mapped it all out and I was integral in assessing and approving for release based off security criteria, new software. 

Ted Wagner: We recently made available a product called Cloud Mixer that I oversaw the security assessment, and make some determination out of our delivery model. That's in conjunction with other stakeholders, as you can imagine, but very much integral in the assessing and securing of its distribution.

Ted Wagner: The model that we're looking at is to deliver as a cloud-based solution, so software-as-a-service. But we also do a web application, web application scans, and then we do review of how it is delivered within the infrastructure that's presented. So access controls, identity, you mentioned identity, uh, are all those things proper and meeting our security posture requirements.

Rick Howard: If you have been in the industry for more than two seconds, you're completely familiar with the Microsoft Windows software update system. Ann says that Microsoft leadership has rightfully identified their software update program as one of their crown jewels and has provided a lot of resources towards it to ensure their customers can trust it. She refers to that pivotal moment in Microsoft history when in January, 2002, Bill Gates, the then Microsoft chairman and chief software architect sent a memo to all employees that turned Microsoft around on a dime to focus on security. He called it trustworthy computing. And by the way, wouldn't it be great almost 20 years later, if all board chairman's would do something similar? Here's Ann. 

Ann Johnson: It's because we have one of the largest software update supply chains in the world.  With the start of trustworthy computing, which was, I think 2000, 2001, is when Bill wrote his memo. We rallied around it to make sure that people had confidence. I don't have the stats in front of me, but the automatic update rate is quite high.

Ann Johnson: The word the Windows Update team uses, I love it. They say we are maniacal about it. We are absolutely and we have been. We recognize the threats there pretty early. And we have actually been maniacal about how we produce our updates, how we do the final build for updates, how we publish the updates, how we make sure the updates are signed, how the updates are downloaded, because we can just imagine that type of wholesale attack.

Ann Johnson: We talk about our software development life cycle and our secure software development life cycle. We talk about the different ways that we test the code, how we protect the process. And it's not just protecting the development process, but it's protecting the process from when it goes from dev to when it goes for to production. And then the moment we click that update. We push it out. There's a whole security process that's built around that, checking it every step of the way. 

Ann Johnson: That supply chain aspect is real. Organizations need to be realistic about it. You don't want to be in a position though, where you're not updating because updates, give you security patches, and they fix real problems.

Rick Howard: I asked Ted about what he told customers concerning the extra lengths that SAP goes through to protect its software update pipeline.

Ted Wagner: We have about half a dozen products that have fed ramp authorizations so that we're delivering to Department of Defense customers and federal and civilian agencies. Products like Success Factors and SAP Analytics Cloud. The way we deliver is through a cloud delivery model software-as-a-service. We do support on-prem customers, but they leverage the SAP software distribution model, which is somewhat similar to the Microsoft model, but you connect to a support portal and you are able to download software updates through a authentication mechanism. 

Rick Howard: Ted points out a relatively new change in our community. Usually the common notion in the cybersecurity industry is that unless you're selling cybersecurity tools, the idea of security doesn't sell. In other words, if you are selling a widget that provides contract services let's say, the sales team usually doesn't lead with how secure the product is. Ted says that is absolutely changing for cloud delivered services.

Ted Wagner: In the line of business that I'm in, there's a huge demand for security. Our customers are screaming for it. We're seeing a lot of activity in the sales arena, especially around our cloud because we profess a higher security posture than commercially available software-as-a-service. 

Ted Wagner: Security does sell. The sales staff loves it because they walk in the door and the first thing out of their mouth is security. The hard part is to follow up on those commitments. Security doesn't get any easier just because you put it on the organization's name. SAP National Security Services, that's nice, but there's a lot of work on the backend that's not always glamorous to make sure that happens.

Rick Howard: Ann points out that securing the supply chain responsibility has not traditionally been given to the CISO in the past, but because of recent supply chain attacks like SolarStorm that hit 18,000 potential victims, like the Codecov attack that led to the compromise of security vendor, Rapid7, and like the Russian Sandworm campaign against MEDoc that led to many Ukraine critical infrastructure compromises, senior leadership is starting to turn their heads towards the CISO to take the ownership.

Ann Johnson: I have seen people who are responsible for supply chain have not historically sat within the CISO function. I think you you've seen that. They, they tend to sit somewhere in vendor management and they tend to just get a checklist from the security office and they tend not to be technical people. Right? And they're just, they're going through their checklist. 

Ann Johnson: Now we're seeing, as you can imagine, right? Now we're seeing this wave of there must be technical supply chain folks that are actually completely aligned with the CISO function that are actually driving what the supply chain has to adhere to. And the, and the CISO is getting, and I think it's a good thing, Rick, the CISO is getting, I never want to use the word power because people think that's a negative word. But they're getting more responsibility. They're getting more accountability for that supply chain and here's what that's going to drive. That's going to mean that those checklists, that vendors have to sign off on, I think they're going to be more grounded in risk as opposed to grounded in, here's some compliance standards we need to meet. Now, it's like, okay, where are actually the threats? Where's the attack surface? Let's make sure those checklists are actually and those out of stations are actually grounded in where the real risk is.

Rick Howard: Ted says that he has been given more responsibility for at least the software supply chain, but he can see where other organizations might not want to consolidate everything under the CISO. 

Ted Wagner: We had a working group to develop the process, to develop and release software within NS2. We came to the recognition that I had to lead the security assessment of that and the approval of that. So that wasn't an arbitrary decision. If you broaden out supply chain in total, I think there's still a conversation going on about that. And I think it's just because of the complexity of it. The CISO in general is properly positioned and, assuming properly resourced in and, endorsed can do it. 

Ted Wagner: But I also know that, organizations can be very different culturally and the way that organized and based off the kind of work that they do. So I don't want to make a global statement, because I think people do it differently. And I think that it's perfectly good in what they do. It may be different from organization to organization. 

Ted Wagner: If you're building, hammers and shovels, Yeah, that's one type of supply chain, versus if you're a software company where everything is in the ether and in software, and it's a lot more dynamic, that's a different kind of culture, a different kind of organization and different requirements. So I think it does vary from organization to organization.

Rick Howard: I agree with both Ann and Ted, this transition of supply chain responsibilities is just starting and may not be the right choice for every organization. But in general, what I believe should be the case is that if the thing that your company is buying or the thing that your company is producing is designed to connect to your network or your customer's network in some fashion, then the security of the thing should be the CSO's responsibility.

Rick Howard: And, that's a wrap. As always, if you agree or disagree with anything I've said or anything our guests have said, hit me up on LinkedIn or Twitter and we can continue the conversation there. Next week is Memorial Day week here in the United States. And the CSO Perspectives team is taking the week off to honor the U.S. Soldiers who have died in American wars, but don't be disappointed, we have a special treat for you instead. The Cybersecurity Canon project has announced the author selectees for the 2021 Hall of Fame Awards. And you all know that I'm a huge advocate for reading in general, all kinds, really. I particularly like horror and fantasy, but specifically we all need to read more good cybersecurity books and I emphasize the good there because there are a lot of published, bad cybersecurity books in circulation. And I had been involved in the Cybersecurity Canon project since the beginning in an attempt to find the books that all of us should have read.

Rick Howard: The result of all of that is that I will be interviewing the winning authors during breaks in the CSO Perspectives schedule, and the first break is next week. Perry Carpenter is coming to the Hash Table to talk about his Cybersecurity Canon Hall of Fame book, "Transformational Security Awareness: What Neuroscientists, Storytellers, and Marketers Can Tell Us About Driving Secure Behaviors". You won't want to miss that. 

Rick Howard: The CyberWire's CSO Perspectives is edited by John Petrik and executive produced by Peter Kilpe. Our theme song is by Blue Dot Sessions, remixed by the insanely talented Elliott Peltzman who also does the show's mixing, sound design, and original score. And I am Rick Howard. Thanks for listening.