Cybersecurity crisis planning: An essential tactic for resilience.

First things first: the senior cybersecurity practitioner for most organizations is probably not in charge of the overall, general-purpose crisis management plan. It turns out that there are many kinds of potential crises that a commercial company, a government agency, or academic institution might encounter that don’t involve some kind of ransomware attack by the likes of BlackByte or Pandora, or some kind of cyberespionage operation from nation state operators like Hurricane Panda or Helix Kitten. That means that to prepare the organization for a cyber crisis, security leaders have to plug themselves into the existing crisis management apparatus as one of its key players. How big the overall organization is and how well resourced the crisis management team is will dictate the level of formality of the crisis plan. What may not be obvious is that the size of the organization and the maturity of its crisis management team are not as important as simply having a plan, any plan, that the leadership team is comfortable with.

And when I say “plan,” I don’t mean a hundred page memo that nobody has ever read. It’s a plan that has been lived with, played with, tweaked, bent, crushed, stomped on, straightened out, ripped up again, thrown out, redone, and iterated on so many times that it’s second nature. Because, when the plan goes south during an actual crisis, as it will inevitably do, the important thing is that the team members are so familiar with each other, and the desired outcome is so well understood, that any leadership team audibles or improvisations during the event have a decent chance of leading to the desired result.

When I was in the Army, I worked for a colonel who understood this. He always said that it was great to have a plan so that we could deviate from it. As Mike Tyson, the famous heavyweight boxer, so eloquently said, "Everybody has a plan until they get punched in the mouth." What I mean by that is the difference between a group of planners and a group of survivors after a crisis is that the survivors are crystal clear before the instigating event happens about what the desired outcome should be. It doesn’t matter if the plan is a hundred pages neatly organized in colored binders coded to the role that each senior leadership member plays, or if the plan is a hastily drawn stick diagram on a whiteboard. The survivors are so comfortable with each other and what they all want to get done, that the improvised plan based on outcomes saves the day.

Let’s take a look at two case studies to highlight the point; two approaches that demonstrate each end of the spectrum of what to do correctly and what not to do: The RSA Security breach of 2011 and the Equifax breach of 2017.

RSA Security: a case study in crisis communications.

In the spring of 2011, intelligence analysts working for RSA Security (an EMC company at the time), noticed that something was amiss on the permissions and behavior for one of their Australian employee accounts. The subsequent investigation revealed a massive cyber espionage operation conducted by the as yet unnamed Chinese adversary group, APT1 (The People’s Liberation Army Unit 61398). Mandiant wouldn’t anoint them their name until two years later.

APT1 hackers had managed to phish that Australian employee, used his account as a beachhead, and then proceeded to move laterally through the RSA Security network, escalating privilege, and looking for the data they wanted to steal. In this case, according to Andy Greenberg at Wired, the seed values for the RSA SecurID token product, the two-factor authentication device used by “tens of millions of users in government and military agencies, defense contractors, banks, and countless corporations around the world.” With those seed values, APT1 could bypass the two factor authentication system in all of them.

Let that sink in for a second. APT1, through this bold cyber espionage campaign, rendered inert this security device that 760 customers around the world had purchased, distributed, installed, and maintained to reduce their attack surface for government secrets, financial data, and other sensitive information. The mind boggles. 

If I were one of those customers, I would’ve been angry, and I would have been actively seeking RSA Security’s biggest competitor so that I could kick the SecurID token product to the curb and install a new system that I could trust. When you’re in the business of selling security specifically designed to protect secrets, your own systems where you keep your secrets had better be airtight. I imagine that's what a lot of RSA Security customers were thinking at the time. According to the New York Times, some big ticket customers said publicly that they planned to switch vendors as soon as possible: Bank of America, JPMorgan Chase, Wells Fargo and Citigroup. 

But then, the RSA Security leadership team executed a crisis communication plan to save the company. Within a week of discovery, according to Greenberg, “One person in legal suggested they didn’t actually need to tell their customers.” The CEO at the time, Art Coviello, wasn’t having any of that. “He slammed a fist on the table: They would not only admit to the breach, he insisted, but get on the phone with every single customer to discuss how those companies could protect themselves.” When somebody on the staff suggested they codename the crisis plan as “Project Phoenix,” Coviello rejected it. “We're not rising from the ashes. We're going to call this project Apollo 13. We're going to land the ship without injury.”

And that’s what they did.

They immediately filed a form 8-K with the Securities and Exchange Commission, a report of an unscheduled material event. The next day, according to Greenberg, “Coviello published an open letter to RSA’s customers on the company’s website” and created a group of 90 staffers who began arranging one-on-one calls with all of their customers. Coviello and his senior staff attended hundreds of these calls personally.

In the end, it worked. In the second quarter earnings call of 2011, EMC reported that their internal incident response cost was about $66 million. By the end of the third quarter though, according to CSO Online, EMC reported record earnings; so much for the fear of reputation loss due to a cyber event. But, I can make a strong case here to attribute that quick recovery, that resilience, to the crisis communications plan led by the CEO, Art Coviello.

After all, resilience is “… the ability to continuously deliver the intended outcome despite adverse cyber events.” That definition comes from a paper published by two Stockholm University researchers: Stirna and Zdravkovic. In March of 2013, RSA Security (EMC) experienced a black swan event, a phrase made famous by Nassim Taleb in his 2007 book, “The Black Swan: The Impact of the Highly Improbable.” Black Swan events are so unlikely that you never expect to be affected by one (like a meteor hitting the earth) but when they do happen, the impact is catastrophic. This was EMC’s black swan event and by all rights, the company shouldn’t have recovered from it. Customers should have left the company in droves. But that’s not what happened. Because of Coviello’s stated support of his customers and laser focus, most customers stayed with the company after the crisis when they had plenty of reasons to leave.

Consider the opposite end of the spectrum: the Equifax Breach of 2017. 

Equifax: a case study in crisis communications.

On 10 March of that year, Chinese hackers (members of the 54th Research Institute, a component of China’s People’s Liberation Army or the PLA), established a beachhead within the Equifax networks. The Equifax internal security team didn’t discover the intrusion until over three months later at the end of July. Immediately, they hired Mandiant as an outside incident response team. Mandiant eventually discovered that Equifax had lost the PII (Personally Identifiable Information) to some 60% of all Americans (143 million U.S. consumers).

The Equifax CEO, Rick Smith, decided to sit on that information for over a month, but eventually went public on 7 September. He announced what has become the traditional handwave of support to his customers for public breach announcements: free credit monitoring, a website for information, and a call center for customer questions. You know, thoughts and prayers but nothing of value. And he kept piecemealing the information out to the public in dribs and drabs over weeks. It felt as if, from the start, that Smith was making it up as he was going along. The message was at best confusing and at worst opaque and misleading on purpose. It felt like amatuer hour.

Three days later, customers discovered that they could get the “coveted” free credit monitoring service, but only if they agreed they wouldn’t sue the company later. By 15 September, Smith fired the CIO (Susan Mauldin) and the CSO (David Webb). On 21 September, the breach information website was still not ready, and so the company started directing customers and journalists to a white-hat phishing site specifically intended to test the company’s security response. By 26 September, the Equifax board fired the CEO. In March of the following year, the Securities and Exchange Commission secured the indictment of Jun Ying, the replacement CIO, for using the not-as-yet public breach information to sell his vested Equifax stock options. It had come to seem that the entire Eqiufax culture was made up of opacity and used-car-salesman chicanery. 

During that period between the breach and Smith’s firing, most pundits agreed that Smith bungled the communications plan.

• He waited six weeks before he announced. 
• He chose not to reach out to customers specifically, instead setting up a website, and a website that wasn't ready for days after the announcement. 
• He offered free credit monitoring, but required enrollees to waive their right to sue.
• He changed his mind later, but customers had to send Equifax written notice of their decision within 30 days. But the written opt-out language from its general terms of service was wrong. 
• He initially charged customers impacted by the beach for freezing credit.
• Equifax assigned easy-to-guess PINs to people who froze their credit.

In the end, at least four executives lost their jobs. The U.S. House Digital Commerce and Consumer Protection subcommittee hauled Rick Smith in to explain himself. In May of 2019, Equifax reported that the incident response cost was roughly $1.4 billion plus legal fees.

Desired outcomes.

The desired-outcomes idea leads us back to the overall cybersecurity first principle strategy: reduce the probability of material impact. In these essays and podcasts, I have outlined six sub strategies to consider that might help: zero trust, intrusion kill chain prevention, risk forecasting, automation, compliance, and resilience. 

Here’s the thing. During a cyber crisis, your black swan event, the only sub strategy that matters now is resilience. If you’re in a cybersecurity crisis, it means your other first principle strategies failed. None of them prevented the crisis from happening. So, now what? We can talk about what went wrong with these strategies after the crisis is over but in the meantime, what should leaders be focusing on? 

Going back to the Stirna and Zdravkovic resilience definition: “continuously deliver the intended outcome despite adverse cyber events.” For EMC in 2011, that meant keeping their customers and meeting their quarterly numbers. Check! For Equifax in 2017, I'm not sure what they were trying to do.

Reviewing the literature on both attacks, it’s not clear to me that either of the companies had a formal crisis plan before their black swan event. The difference in outcomes stems from Coviello’s leadership setting the desired outcome from the start: “We're going to land the ship without injury.” In contrast, Equifax’s Smith was all over the map with inconsistency.

Executives are busy - exercise them efficiently.

So, how do you get the leadership team on the same page in terms of desired outcomes before a cyber crisis occurs? As violinist Mischa Elman said, when two New York City tourists asked him how to get to Carnegie Hall, “Practice.” I'm not trying to be flippant about this. Regardless if you have a hundred page strategic plan or a whiteboard stick figure plan, walking the senior leadership team through various scenarios to get their reactions and to reaffirm the desired outcomes is key. 

It’s my experience that large organizations execute at least one formal scenario exercise a year. Some do several where they dust off the plan, bounce a scenario off of it (like ransomware, cyber espionage, or cyber hacktivism), and get the senior leadership team’s reaction to it. The first priority is to make them aware of the various resilience tactical measures that you already have in place that might mitigate the event, like incident response, backups, and encryption. During the exercise, gaps will be found in your tactics that you hadn’t thought of before and that is totally acceptable and desired. More importantly though, you will get your senior leadership team's reactions to those gaps and their desire to close them.

In every one of these exercises in my career, I have always learned something new. Either the plan was not clear enough, or the plan was wrong about how to handle some detail, or some senior executive objected to what we were trying to do with the plan. The point of these exercises however is not to run the leadership team through every possible scenario. The point is to get them all making decisions that will support the desired outcome regardless of the given scenario and regardless if the stated plan is tossed out as soon as we get hit in the mouth. In other words, practice not the scenario, but the outcome.

These scenario exercises don’t have to be that formal either. The senior leadership team is busy. Getting them all to commit to an afternoon of exercise play once a year is a tremendous act of scheduling deconfliction, convincing them that this is a good use of their time, and making do when some have to drop out at the last second because some fire pops up that requires immediate attention. Even if the CEO is totally committed to the exercise, which is not always a given, things happen. But there are simpler approaches.

One that I have used with some success in the past is an extended lunch (maybe 90 minutes) on a regular basis with the senior leadership team. The purpose is to drop a scenario on the table during the meal, remind everybody what the desired outcomes are based on the current plan and previous scenario lunches, and get their reaction. As they discuss what they would be doing during each phase of the scenario, the crisis team leader would be interjecting what the rest of the company would be doing based on the current plan.

The beauty of this approach is that even senior executives like a free meal, and this is not a huge time commitment for them. And it’s informal. People are more likely to throw ideas around when you are all sharing the same salad. Further, this might be a better approach for small to medium sized organizations too who may barely have the resources to keep the payroll system working, let alone to spend a day on an exercise scenario.

Take-aways.

In order to have any hope of successfully executing our resilience sub strategy, practice makes perfect. Give your senior executives a lot of chances to make decisions that further the desired outcome before the actual black swan event happens. As the saying goes, you don’t want them to be thinking about this stuff for the first time during a real crisis. You want them comfortable making the right calls in these crisis situations. And that’s what cybersecurity crisis planning gives you.

Equifax timeline:

2017

10 March

Mandiant says the first evidence of hacker “interaction” occurred on March 10th, considerably earlier than May 29thas Equifax originally stated.4 7

Between May 13th and late July

Intruders 

• Accessed sensitive information “stored in databases in an Equifax legacy environment”. 7
• Compromised two systems that support Equifax’s online dispute application.7
• Set up “about 30 web shells”that were accessed from around 35 “distinct public IP addresses”. 7

According to Mandiant, the attackers methods and tools do not match any “threat actor group” it tracks, and does not “overlap with those seen in previous investigations by the firm”. 7

29 July

The in-house security team discovered and blocked the assault, and then took the website offline the following day after observing additional questionable activity. A followup investigation determined that hackers gained access to Equifax’s archives through a known security flaw in its database framework.5

2 August

Equifax hired Mandiant, an independent cybersecurity firm, to investigate the breach. The inquiry concluded that the hack may have involved the theft of personal data including credit card, social security and, in some cases, drivers’ license numbers, birth dates, and addresses of about 143 million U.S. consumers – roughly 60 percent of American adults.5

7 September  

Equifax publicly acknowledged the breach and took steps to provide consumers with information and assistance to find out if their personal data had been compromised. 5

• Creating a website specifically for consumers to find out if they had been impacted, to learn more information about the hack, what they might be able to do about it, and what they can do to protect themselves from potential future cyberattacks.
• Offering free credit file monitoring and identity theft protection to U.S. consumers whether they were affected by the attack or not.
• Establishing a call center to answer consumer questions concerning the breach and to encourage consumers to sign up for the company’s monitoring and theft protection service

8 September

Equifax shares plunge 13.7% in first day of trading after breach announced.4

10 September

Customers who signed up post data breach to Equifaxs' credit monitoring program learn that in the Equifax terms of service, they are barred from participating in any class-action lawsuits that may arise from the incident.6

11 September

Sen. Orrin Hatch, R-Utah, who chairs the Senate Committee on Finance, and Sen. Ron Wyden, D-Oregon, the panel's ranking minority member, ask the credit-reporting giant for a timeline of the breach, along with details of Equifax's efforts to quantify the scope of the intrusion and limit consumer harm.4

12 September  

Equifax CEO apologizes in USA TODAY op-ed.4

Equifax announced both its chief information officer and chief security officer would retire, effective immediately. 5

13 September

BBC News: Equifax had 'admin' as login and password in Argentina. 5

15 September

Equifax announces its chief information officer, Susan Mauldin, and chief security officer, David Webb are retiring "effective immediately." 4

21 September

The company is so inept, it’s been directing people to a white hat phishing site specifically intended to test the company’s security response.6

26 September 

Equifax announces its CEO, Richard Smith, retires. Paulino do Rego Barros, Jr., a seven-year veteran of the company, is appointed interim Chief Executive Officer.4

Its chief executive office stepped down on September 26, but is scheduled appearance to testify before Congress in early October.5

2 October

Equifax releases information from a report by forensic computer security company Mandiant which identified an additional 2.5 million people whose information was stolen. 4 

3 October

Former Equifax CEO Richard Smith testifies before the House Digital Commerce and Consumer Protection subcommittee in which Smith says "mistakes were made." 4

12 October

Equifax Inc. took part of its website offline Thursday after code on the site redirected users to a malicious URL urging them to download malware.6

2018

14 March

Jun Ying, who was to become the company’s next chief information officer, was indicted for using confidential information to exercise his vested Equifax stock options and then sell the shares before the company publicly reported a breach. 6 

2019

12 May

Equifax revealed in its earnings release that the incident has cost about $1.4 billion plus legal fees.6

RSA Security timeline:

2011

3 March

Establishing the beachhead: RSA analysts eventually traced the origin of the breach to a single malicious file that they believed had landed on an RSA employee’s PC. A staffer in Australia had received an email with the subject line “2011 Recruitment plan” and an Excel spreadsheet attached to it. He'd opened it. Inside the file was a script that exploited a zero-day vulnerability—a secret, unpatched security flaw—in Adobe Flash, planting a common piece of malicious software called Poison Ivy on the victim’s machine.1

8 March

First Indication something was amiss. The admin had noticed that one user had accessed a server from a PC that the user didn’t typically work on, and that the permissions setting on the account seemed unusual.1 

16 March

RSA’s executives debated how to go public. One person in legal suggested they didn’t actually need to tell their customers, Sam Curry remembers Coviello slammed a fist on the table: They would not only admit to the breach, he insisted, but get on the phone with every single customer to discuss how those companies could protect themselves. As the recovery effort got under way, one executive suggested they call it Project Phoenix. Coviello immediately nixed the name. “Bullshit,” he remembers saying. “We're not rising from the ashes. We're going to call this project Apollo 13. We're going to land the ship without injury.” 1 

17 March

RSA (EMC) files form 8-K, a report of unscheduled material events or corporate changes at a company that could be of importance to the shareholders or the Securities and Exchange Commission ( SEC ).2 

18 March

Coviello published an open letter to RSA’s customers on the company’s website. “Recently, our security systems identified an extremely sophisticated cyberattack in progress,” the letter read. “While at this time we are confident that the information extracted does not enable a successful direct attack on any of our RSA SecurID customers, this information could potentially be used to reduce the effectiveness of a current two-factor authentication implementation as part of a broader attack,” the letter continued—somewhat downplaying the crisis. 1 

25 May

Breach goes public. A post appeared on the influential tech blogger Robert X. Cringely’s website, titled “InsecureID: No More Secrets?” 1 

27 May

Reuters revealed thatLockheed Martin had been hacked using the stolen tokens. 1 

June

In another open letter to customers, RSA’s Art Coviello admitted, “We were able to confirm that information taken from RSA in March had been used as an element of an attempted broader attack on Lockheed Martin, a major US government defense contractor.” 1 

In the second quarter earnings call, EMC reported that their internal incident response cost was $66 million.3

2013

February

The New York Times and the security firm Mandiant attribute the attacks to a Chinese state hacker group that named APT1, People’s Liberation Army Unit 61398. Among its dozens of targets over the previous five years: the governments of the United States, Canada, South Korea, Taiwan, Vietnam; and the United Nations—and RSA.1

References.

“CEO Who Led Equifax during Data Breach Gets Huge Raise,” by Kevin Dugan, New York Post, 3 April 2018.

“Cyber Crisis Management: The Practical Handbook on Crisis Management and Crisis Communication,” by Holger Kaschner, Published by Springer, 5 January 2022.

"Cyber Resilience – Fundamentals for a Definition,” by Fredrik Björck, Martin Henkel, Stockholm University, Janis Stirna, Jelena Zdravkovic, Stockholm University, Article in Advances in Intelligent Systems and Computing, January 2015.

“Cybersecurity Crisis-Planning Checklist: Tips for Planning and Ensuring Business Continuity.” by Zscaler, 2020.

“Equifax: An Epic Fail in Crisis Communications,” by Strategic Vision PR Group, 21 September 2017. 

“Equifax Engages in Almost Wholly Reactive Crisis Communications,” by Thom Weidlich, Prcg. 14 September 2017.

“Equifax Turned Its Hack into a Public Relations Catastrophe,” by Danielle Wiener-Bronner, CNNMoney, 12 September 2017. 

“‘Everybody Has a Plan until They Get Punched in the Mouth.’ - How Did the Famous Mike Tyson Quote Originate?” by Anwesha Nag, Sportskeeda, 5 January 2021.

“EXPLORING THE EVOLUTION OF BUSINESS CONTINUITY MANAGEMENT,” by DENOVO Blog, 31 May 31 2018.

“Five Steps to Developing a Cyber Crisis Communications Plan,” by Neil Stinchcombe, Forbes, 25 August 2021.

“Lessons from the RSA Breach,” Stephen Bell, CSO Online, 4 October 2011. 

“How to Create a Cybersecurity Crisis Management Plan in 5 Steps,” by Maria Rahul, GetApp, 27 May 2020. 

“Organizational Resilience: Security, Preparedness, and Continuity Management Systems -- Requirements with Guidance for Use, ASIS SPC.1-2009” by ASIS International, 2009.

"Partnering for Cyber Resilience,” by The World Economic Forum, 2012.

"Presidential Policy Directive 21: Critical Infrastructure Security and Resilience,” President Obama, 12 February 2013.

“Report on Cyber Crisis Cooperation and Management,” by Panagiotis Trimintzios, Roger Holfeldt, Mats Koraeus, Baris Uckan, Razvan Gavrila and Georgios Makrodimitris, ENISA (European Union Agency for Network and Information Security), November 2014. 

“Rick Smith, Chairman and CEO of Equifax, on Cybersecurity Incident Involving Consumer Data,” by Rick Smith, Equifax, YouTube, 7 September 2017.

“Risk, Crisis and Security Management,” by Edward Borodzicz, Published by Wiley, 1 July 2005.

“RSA Faces Angry Users after Breach,” by By Nelson D. Schwartz and Christopher Drew, The New York Times, 7 June 2011.

“Security and resilience — Organizational resilience — Principles and attributes

: ISO 22316:2017(en),” by ISO, 2017.

“SecurID Data Breach Cost RSA $66 Million-so How Much Did It Cost You?” by SecurEnvoy, 23 August 2011.

Source 1: “The Full Story of the Stunning RSA Hack Can Finally Be Told,” by Andy Greenberg, Wired, 20 May 2021.

Source 2: “EMC CORPORATION, Form 8-K,” UNITED STATES SECURITIES AND EXCHANGE COMMISSION, 17 March 2011.

Source 3: “What Did the RSA Breach End up Costing EMC?” by Help Net Security, 28 July 2011.

Source 4: “A Timeline of Events Surrounding the Equifax Data Breach,” by Elizabeth Weise, USA TODAY, 26 September 2017.

Source 5: “The Equifax Credit Breach Timeline: What Happened?” by the Eichholz Law Firm, 28 September 2017.

Source 6: “Timeline of Equifax Databreach,” by Alan Mac Kenna, Serve IT, 29 March 2018. 

Source 7: “Equifax Breach: Updated Timeline, Phishing, Regulation, and a Roundup,” by Risk Based Security, 26 September 2017. 

Source 8: “Chinese Hackers Charged in Equifax Breach,” Federal Bureau of Investigation,” 10 February 2020

“The Black Swan: The Impact of the Highly Improbable,” by Nassim Nicholas Taleb, Published by Random House, 17 April 2007. 

“The disaster recovery handbook : a step-by-step plan to ensure business continuity and protect vital operations, facilities, and assets,” by Michael Wallace and Lawrence Webber, Published by AMACOM, 1 January 2004. 

“The File That Hacked RSA: How We Found It,” by Timo Hirvonenm, F-Secure Weblog, 26 August 201.1

“The Joke,” by By Matt Carlson, Carnegie Hall, 10 April 2020.