Cybersecurity crisis planning: An essential tactic for resilience.

Rick Howard: Hey everybody. Rick here. Back in 2011, I was working for a company called VeriSign as the intelligence director for a commercial cyber intelligence business unit called iDefense. But the CEO had just tapped me on the shoulder to come work for him as his chief of staff. I was in his office talking about what he wanted me to do for him - I mean, we hadn't even announced the job change yet - when he got a call from security informing him that we might have an active shooter situation in one of the office buildings. At the time, VeriSign shared a three-building complex with other companies situated around a manmade pond. The VeriSign headquarters building where we were located was on one end. The iDefense building was located on the opposite side with one building in between. The alleged shooter was in the iDefense building. That turned out not to be true. There was no shooter, just an angry employee from one of the other companies yelling at his boss. But we didn't know that at the time.

Rick Howard: For about 30 minutes, we were scared - really scared - that somebody might get hurt. The boss' main concern was to get the employees out of harm's way. He directed security to evacuate the two buildings not immediately affected away from the shooter's location, away from the iDefense building, and directed the iDefense people to lock their doors and hunker down under their desk. That was a hell of a first day as chief of staff, and it made an impression on me about leaders having a clear vision about what needs to be done in a crisis - what are the desired outcomes and how to prepare for any future crisis somewhere down the line. And that's what we're talking about today - cybersecurity crisis planning. So hold on to your butts. 

(SOUNDBITE OF FILM, "JURASSIC PARK") 

Samuel L Jackson: (As Arnold) Hold on to your butts, butts, butts, butts. 

Rick Howard: My name is Rick Howard, and I am broadcasting from the CyberWire's secret sanctum sanctorum studios, located underwater, somewhere along the Patapsco River near Baltimore Harbor, Md., in the good, old U.S. of A. And you're listening to "CSO Perspectives," my podcast about the ideas, strategies and technologies that senior security executives wrestle with on a daily basis. 

Rick Howard: First things first, the senior cybersecurity practitioner for most organizations is probably not in charge of the overall general purpose crisis management plan. It turns out that there are many kinds of potential crises that a commercial company, a government agency or academic institution might encounter that don't involve some kind of ransomware attack from the likes of BlackByte or Pandora or some kind of cyber espionage operations from nation-state actors like Hurricane Panda or Helix Kitten. That means that to prepare the organization for a cyber crisis, security leaders have to plug themselves into the existing crisis management apparatus as one of its key players. How big the overall organization is and how well-resourced the crisis management team is will dictate the level of formality of the crisis plan. What may not be obvious is that the size of the organization and the maturity of the crisis management team aren't as important as simply having a plan - any plan - that the leadership team is comfortable with. 

Rick Howard: And when I say plan, I don't mean a hundred-page memo that nobody has ever read. It's a plan that has been lived with, played with, tweaked, bent, crushed, stomped on, straightened out, ripped up again, thrown out, redone and iterated on so many times that it's second nature. Because when the plan goes south during an actual crisis, as it will inevitably do, the important thing is that the senior leadership team members are so familiar with each other and the desired outcome is so well understood that any audibles or improvisations during the event have a decent chance of leading to the preferred result, the desired outcome. When I was in the Army, I worked for a colonel who understood this. He always said that it was great to have a plan that we could deviate from. As Mike Tyson, the famous heavyweight boxer, more eloquently said... 

(SOUNDBITE OF ARCHIVED RECORDING) 

Mike Tyson: A wise man once said, everyone has a plan until you're punched in the mouth. 

Rick Howard: Here's what I mean by that. The difference between a group of planners and a group of survivors after a crisis is that the survivors are crystal clear before the instigating event happens about what the desired outcome should be. It doesn't matter if the plan is a hundred pages, neatly organized in colored binders, coded to the role that each senior leadership member plays or if the plan is a hastily drawn stick diagram on a whiteboard. The survivors are so comfortable with each other and what they all want to get done that the improvised plan after we get hit in the mouth, based on outcomes, saves the day. Let's take a look at two case studies to highlight the point - two approaches that demonstrate each end of the spectrum of what to do correctly and what not to do - the RSA Security breach of 2011 and the Equifax breach of 2017. 

Rick Howard: In the spring of 2011, intelligence analysts working for RSA Security an EMC company at the time, noticed that something was amiss on the permissions and behavior for one of their Australian employee accounts. The subsequent investigation revealed a massive cyber espionage operation conducted by the yet-unnamed Chinese adversary group APT1, the People's Liberation Army Unit 61398. Mandiant wouldn't anoint the night them their name until two years later. APT1 hackers had managed to phish that Australian employee, use his account as a beachhead, and then proceeded to move laterally through the RSA Security network, escalating privilege and looking for the data they wanted to steal. In this case, according to Andy Greenberg at WIRED, the seed values for the RSA SecurID token product, the two-factor authentication device used by, quote, "tens of millions of users in government and military agencies, defense contractors, banks and countless corporations around the world," end quote. With those seed values, APT1 could bypass the two-factor authentication system in all of them. 

(SOUNDBITE OF TV SHOW, "THE SIMPSONS") 

Dan Castellaneta: (As Homer Simpson) D'oh. 

Rick Howard: Let that sink in for a second. APT1, through this bold cyber espionage campaign, rendered inert this security device that 760 customers around the world had purchased, distributed, installed and maintained to reduce their attack surface for government secrets, financial data and other sensitive information. The mind boggles. If I were one of those customers, I would have been angry. And I would have been actively seeking RSA Security's biggest competitor, so that I could kick the SecurID token product to the curb and install a new system that I could trust. When you're in the business of selling security specifically designed to protect secrets, your own systems where you keep your secrets had better be airtight. I imagine that's what a lot of RSA Security customers were thinking at the time. And according to The New York Times, some big-ticket customers said publicly that they plan to switch vendors as soon as possible - Bank of America, JPMorgan Chase, Wells Fargo and Citigroup. 

Rick Howard: But then the RSA Security leadership team executed a crisis communication plan to save the company. Within a week of discovery, according to Greenberg, quote, one person in legal suggested they didn't actually need to tell their customers, end quote. The CEO at the time, Art Coviello, wasn't having any of that. Greenfield says that, quote, "he slammed a fist on the table. They would not only admit to the breach, he insisted, but get on the phone with every single customer to discuss how those companies could protect themselves," end quote. When somebody on the staff suggested they codename the crisis plan as Project Phoenix, Coviello rejected it. Quote, "we're not rising from the ashes. We're going to call this project Apollo 13. We're going to land this ship without injury," end quote. 

(SOUNDBITE OF FILM, "APOLLO 13") 

Ed Harris: (As Gene Kranz) With all due respect, sir, I believe this is going to be our finest hour. And that's what they did. They immediately filed a Form 8-K with the Securities and Exchange Commission, a report of unscheduled material event. The next day, according to Greenberg, Coviello published an open letter to RSA customers on the company's website and created a group of 90 staffers who began arranging one-on-one calls with all other customers. Coviello and his senior staff attended hundreds of these calls personally. In the end, it worked. In the second quarter earnings call of 2011, EMC reported that their internal incident response cost was about $66 million. By the end of the third quarter, though, according to CSO Online, EMC reported record earnings. 

Rick Howard: So much for the fear of reputation lost due to a cyber event. But I can make a strong case here to attribute that quick recovery, that resilience to the crisis communications plan led by the CEO, Art Coviello. After all, resilience is, quote, "the ability to continuously deliver the intended outcome despite adverse cyber events," end quote. That definition comes from a paper published by two Stockholm University researchers, Stirna and Zdravkovic. In March of 2013, RSA Security, EMC, experienced a black swan event, a phrase made famous by Nassim Taleb and his 2007 book, "The Black Swan: The Impact of the Highly improbable." Black swan events are so unlikely that you never expect to be affected by one, like a meteor hitting the earth, but when they do happen, the impact is catastrophic. This was EMC's black swan event, and by all rights, the company shouldn't have recovered from it. Customers should have left the company in droves. But that's not what happened. Because of Coviello's stated support of his customers and his laser focus, most customers stayed with the company after the crisis, when they had plenty of reasons to leave. So consider the opposite end of the spectrum - the Equifax breach of 2017. On 10 March, 2017, Chinese hackers, members of the 54th Research Institute, another component of China's People's Liberation Army, or PLA, established a beachhead within the Equifax networks. The Equifax internal security team didn't discover the intrusion until over three months later, at the end of July. Immediately, they hired Mandiant, and that's what they did. They immediately filed a Form 8-K with the Securities and Exchange Commission, a report of unscheduled material event. The next day, according to Greenberg, Coviello published an open letter to RSA customers on the company's website and created a group of 90 staffers who began arranging one-on-one calls with all of their customers. Coviello and his senior staff attended hundreds of these calls personally. And in the end, it worked. In the second quarter earnings call of 2011, EMC reported that their internal incident response cost was about $66 million. By the end of the third quarter, though according to CSO Online, EMC reported record earnings. So much for the fear of reputation lost due to a cyber event. 

Rick Howard: But I can make a strong case here to attribute that quick recovery, that resilience, to the crisis communications plan led by the CEO, Art Coviello. After all, resilience is, quote, "the ability to continuously deliver the intended outcome despite adverse cyber events," end quote. That definition comes from a paper published by two Stockholm University researchers, Stirna and Zdravkovic. In March of 2013, RSA Security, EMC, experienced a black swan event, a phrase made famous by Nassim Taleb in his 2007 book, "The Black Swan: The Impact of the highly Improbable." Black swan events are so unlikely that you'd never expect to be affected by one, like a meteor hitting the Earth. But when they do happen, the impact is catastrophic. This was EMC's black swan event, and by all rights, the company shouldn't have recovered from it. Customers should have left the company in droves. But that's not what happened. Because of Coviello's stated (ph) support of his customers and his laser focus, most customers stayed with the company after the crisis when they had plenty of reasons to leave. 

Rick Howard: So consider the opposite end of the spectrum - the Equifax breach of 2017. On 10 March 2017, Chinese hackers, members of the 54th Research Institute, another component of China's People's Liberation Army or PLA, established a beachhead within the Equifax networks. The Equifax internal security team didn't discover the intrusion until over three months later, at the end of July. Immediately, they hired Mandiant as an outside incident response team. Mandiant eventually discovered that Equifax had lost the PII, or personally identifiable information, to some 60% of all Americans. That's 143 million U.S. consumers. And to put that number into context, that's 43 million more Americans than voted in the last presidential election - 43 million. Yikes. 

(SOUNDBITE OF TV SHOW, "HOME IMPROVEMENT") 

Tim Allen: (As Tim Taylor) Oh, no. 

Rick Howard: The Equifax CEO, Rick Smith, decided to sit on that information for over a month but eventually went public on 7 September. He announced what has become the traditional hand wave of support to his customers for public breach announcements - free credit monitoring, a website for information and a call center for customer questions you know, thoughts and prayers but nothing of value. And he kept piecemealing out the information to the public in dribs and drabs over weeks. It felt as if from the start that Smith was making it up as he was going along. The message was, at best, confusing and, at worst, opaque and misleading on purpose. It felt like amateur hour. Three days later, customers discovered that they could get the coveted free credit monitoring service but only if they agreed they couldn't sue the company later. By 15 September, Smith fired the CIO, Susan Mauldin, and the CSO, David Webb. On 21 September, the breach information website was still not ready, and so the company started directing customers and journalists to a white hat phishing site specifically intended to test the company's security response. By 26 September, the Equifax board fired the CEO. In March of the following year, the Securities and Exchange Commission secured the indictment of Jun Ying, the replacement CIO, for using the not-as-yet public breach information to sell his vested Equifax stock options. 

Rick Howard: It had come to seem that the entire Equifax culture was made up of opacity and used car salesmen chicanery. During that period between the breach and Smith's firing, most pundits agreed that Smith bungled the communications plan. He waited six weeks before he announced. He chose not to reach out to customers specifically, instead setting up a website and a website that wasn't ready for days after the announcement. He offered free credit monitoring but required enrollees to waive their rights to sue. He changed his mind later, but customers had to send Equifax written notice of their decision within 30 days. But then the written opt-out language for his general terms of service was wrong. He initially charged customers impacted by the breach for freezing credit. Now, freezing credit is one of the Equifax offered services and is a go-to move if you personally suspect fraud, if you think your identity has been stolen. Equifax caused the potential fraud and then charged you for the mitigation service. 

(SOUNDBITE OF TV SHOW, "THE SIMPSONS") 

Dan Castellaneta: (As Homer Simpson) D'oh. 

Rick Howard: And then to add insult to injury, Equifax assigned easy-to-guess pins to the people who froze their credit. In the end, at least four executives lost their jobs. The U.S. House Digital Commerce and Consumer Protection Subcommittee hauled Rick Smith, the CEO, in to explain himself. And finally, in May of 2019, two years later, Equifax reported that the incident response cost was roughly $1.4 billion, plus legal fees. Yikes. 

Rick Howard: At the top of the show, I mentioned that for cybersecurity crisis planning, having a clear vision of your desired outcomes is key. But that idea leads us back to the overall cybersecurity first principle strategy - for any kind of planning, but especially for cybersecurity crisis planning, what is the ultimate task that we are all trying to do? For the past two years on this podcast, I've made the case that the ultimate first principle is to reduce the probability of material impact. And I've outlined six substrategies to consider that might help - zero trust, intrusion kill chain prevention, risk forecasting, automation, compliance and resilience. Here's the thing - during a real cyber crisis, your black swan event, the only substrategy that matters now is resilience. If you're in a cybersecurity crisis, it means your other first principle strategies failed. None of them prevented the crisis from happening. 

Rick Howard: So now what? We can talk about what went wrong with those strategies after the crisis is over. But in the meantime, what should leaders be focusing on? Going back to the Stirna and Zdravkovic resilience definition, continuously deliver the intended outcome despite adverse cyber events. For EMC in 2011, that meant keeping their customers and meeting their quarterly numbers. Check. For Equifax in 2017, I'm not sure what they were trying to do. Reviewing the literature on both attacks, it's not clear to me that either of the companies had a formal crisis plan before the black swan event. The difference in outcomes stems from Coviello's leadership setting the desired outcome from the start. We're going to land this ship without injury. In contrast, Equifax's Smith was all over the map within inconsistency. 

Rick Howard: So how do you get the leadership team on the same page in terms of desired outcomes before a cyber crisis occurs? As violinist Mischa Elman said when two New York City tourists asked him how they could get to Carnegie Hall, practice, he said. And if you don't believe me, let me give you the authoritative source from "The Muppet Show." 

(SOUNDBITE OF TV SHOW, "THE MUPPET SHOW") 

Jim Henson: (As Slim Wilson) Hey. Do you know how to get to Carnegie Hall? 

Jerry Nelson: (As Floyd Pepper) Practice, man. Practice. 

(LAUGHTER) 

Rick Howard: Now, I'm not trying to be flippant about this. Regardless, if you have a hundred-page strategic plan or a whiteboard stick figure plan, walking the senior leadership through various scenarios to get the reactions and to reaffirm the desired outcomes is key. It's my experience that large organizations execute at least one formal scenario exercise a year. Some do several, where they dust off the plan, bounce a scenario off of it, like ransomware or cyberespionage or cyber hacktivism and get the senior leadership teams' reaction to it. The first priority is to make them aware of the various resilience tactical measures that you already have in place that might mitigate the event, like incident response, backups and encryption. But during the exercise, you'll discover gaps in your tactics that you hadn't thought of before, and that is totally acceptable and desired - in fact, maybe the reason to do them in the first place. More importantly, though, you'll get your senior leadership teams' reactions to those gaps and their desire to close them. 

Rick Howard: In every one of these exercises I've done in my career, I have always learned something new. Either the plan was not clear enough, or the plan was wrong about how to handle some detail, or some senior executive objected to what we were trying to do with the plan. The point of these exercises, however, is not to run the leadership team through every possible scenario; the point is to give them practice in making decisions that will support the desired outcome, regardless of the given scenario and regardless if the stated plan is tossed out as soon as we get hit in the mouth. In other words, practice not the scenario but the outcome. These scenario exercises don't have to be that formal, either. The senior leadership team is busy. Getting them all to commit to an afternoon of exercise play once a year is a tremendous act of schedule deconfliction, convincing them that this is a good use of their time and making do when some have to drop out at the last second because some fire pops up that requires immediate attention. Even if the CEO is totally committed to the exercise, which is not always a given, things happen. 

Rick Howard: But there are simpler approaches. One that I've used with some success in the past is an extended lunch, maybe 90 minutes, on a regular basis with the senior leadership team. The purpose is to drop a scenario on the table during the meal, remind everybody what the desired outcomes are based on the current plan and previous scenario lunches and get their reaction. As they discuss what they would be doing during each phase of the scenario, the crisis team leader would be interjecting what the rest of the company would be doing based on the current plan. The beauty of this approach is that even senior executives like a free meal, and this is not a huge time commitment for them. And it's informal. People are more likely to throw ideas around when you're all sharing the same salad. Further, this might be a better approach for small- and medium-sized organizations, too, who may barely have the resources to keep the payroll system working, let alone to spend a day on an exercise scenario. So that's something to be considered. 

Rick Howard: In order to have any hope of successfully executing our resilience substrategy, practice makes perfect. Give your senior executives a lot of chances to make decisions that further a desired outcome before the actual black swan event happens. As the saying goes, you don't want them to be thinking about this stuff for the first time during a real crisis. You want them comfortable making the right calls in these kinds of crisis situations. And that's what cybersecurity crisis planning gives you. 

Rick Howard: And that's a wrap. For most of these episodes, I also write a companion essay for those of you who prefer to read the material, as opposed to listening to me drone on in this podcast, punctuated by Tim the Toolman sound effects and references to obscure 1970s nerd TV shows. In this companion essay, though, I've included the historic timelines for both the RSA security breach and the Equifax breach. Much to the chagrin of my friend Steve Winterfeld, the Akamai advisory CISO who hates all things regarding InfoSec history, I find that I can't really understand some big event unless I see the sequential timeline of it. If you're like me, check out the essay. 

Rick Howard: And as always, if you agree or disagree with anything I have said, hit me up on LinkedIn or Twitter, and we can continue the conversation there. Or if you prefer email, drop a line to csop@thecyberwire.com. That's C-S-O-P, the @ sign, thecyberwire - all one word - dot com. And if you have any questions you would like us to answer here at "CSO Perspectives," send a note to the same email address, and we will try to address them in the show. 

Rick Howard: For next week's show, I'm excited to say that I'm finally going to address how internal security teams can do risk forecasting. This topic has been on my radar since we started the podcast, and I can't wait to discuss it with you. 

Rick Howard: The CyberWire's "CSO Perspectives" is edited by John Petrik and executive produced by Peter Kilpe. Our theme song is by Blue Dot Sessions, remixed by the insanely talented Elliott Peltzman, who also does the show's mixing, sound design and original score. And I'm Rick Howard. Thanks for listening.