Backups as a cybersecurity first principle.

About 15 years ago, I set up the backup scheme for the Howard family data set. Digital was just starting to become mainstream, and we had all of these electronic artifacts scattered across mobile phones, digital cameras (remember those?) and the family home computer. Between five people (the wife, the two daughters, and one son), it was starting to get out of hand. I realized that some of these items, like precious videos of my daughters leaping across the stage in their dance studio production of the Lion King and all of our Turbotax files for the past 20 years, just to name two, might be worth spending the time to get organized in one place and then backing the data up so that one catastrophe didn’t wipe everything out.

Just as an aside, by the time the family wrapped the production of the Lion King, we had spent enormous resources in terms of the purchase of costumes, dance rehearsal time, and backstage prep time. And make no mistake, it was a full court press on family participation. The two daughters were in something like a thousand numbers combined, Mom was the backstage coordinator, and my son and I were security (which meant we spent a lot of time directing traffic in the local high school parking lot). At the end of the production, we treated the entire family to a Disneyworld trip for a job well done. And there we were in the middle of Disney’s Animal Kingdom, at the intersection of the Pangani Forest Exploration trail and the Wildlife Express Train, when a bunch of Disney street performers began singing and dancing to the Lion King soundtrack and asking patrons to join. And oh my god, this was going to be the perfect Kodak moment. Right? My two daughters had just spent the last six months perfecting the aforementioned thousand Lion King dance numbers. They were going to kill this. Video camera in hand, I started recording but all I got were two embarrassed teenagers swaying back and forth as awkward as if they just learned to walk and chew gum last Tuesday. They weren’t even swaying in time with music.

I think that’s my all time favorite video of my two daughters and their dancing career. Clearly, I needed to make sure that no computer catastrophe would cause me to lose that video and all the other digital detritus that we had collected over the years. I went to work.

Not only did I build a scheme that would automatically upload a copy of all of our files to one of the early cloud providers, I also built a local RAID Array for my home system so that if any one disk in the array failed, I could just remove it and stick a brand new one in and nobody would be the wiser. This system was foolproof. I had backups of backups.

About a year later, the inevitable catastrophe happened. The hard drive for my home computer failed and I could not getit to come back online. My wife gave me that panicked, “What about all of my files” look. Smugly, I just looked back at her saying, “Not to worry. I have backups.” After building a new computer, I went to my cloud provider first to restore the data. Much to my horror, none of the data were there. I couldn't believe it. There wasn’t a single video, picture, or Turbo Tax file anywhere in the cloud. And that’s when my own panic started to creep in. You know the feeling, that sense where you know you might have screwed up royally in some way that you don’t comprehend yet. Ya, that’s the feeling I had. But then I remembered the RAID Array. That was my backup. I could restore from there.

All I can say is that I had a great plan and failed completely in execution. Oh, I had a cloud backup system in place and routinely checked that the system was saving all of my files there. And I had a RAID Array where I made sure to make a backup copy of the backup files. My failure stemmed from where I told the two systems to back up. Apparently, I configured it so that every day, my backup system was copying files from an empty directory and not the directory where everything was stored.Every week or so I would check to make sure the system was working and every week I would get the green light, everything A-OK.

I’m embarrassed to admit that in order to get my files back, I had to pay the Geek Squad down at the local Best Buy to recover the corrupted files on the home computer. The experience was, shall we say, humbling and fifteen years later, that’s the one story my wife loves to tell to family and friends when they start asking questions about my storied cybersecurity career. It goes something like this. “Ya, let me tell you about my husband and his big fancy pants cybersecurity career when he lost all of the family data for the past 20 years.

So, in immortal words of Bill Murray in one of my favorite movies, “Caddy Shack:” “I got that going for me.”

Which is a long way around the horn for me to emphasize that, for this essay, we are talking about the extremely sexy topic of enterprise backup schemes as a key strategy to improve our infosec first principle of resiliency. 

Backups as strategy against ransomware

Ransomware seems to be having a moment right now. It’s interesting to see the evolution of cyber crime and ransomware, specifically over the last decade. When ransomware first started, the target victim was the home user. Cyber criminals would compromise Grandma’s computer and tell her that if she wanted her cherished pictures of her cats and grandkids back, that she would have to pay $500 in bitcoin. The backend business systems that these ransomware groups developed to make this model work was, and is, astonishing.They had entire call centers dedicated to walking Grandma through a Bitcoin transaction. How amazing is that?

The preferred target victim changed sometime in 2017. I heard Nicole Perlroth, the New York Times reporter, talk about this on a podcast somewhere. She said that after the North Koreans launched WannaCry and one month later, the Russians launched NotPetya, the ransomware gangs realized that there was a much more lucrative revenue stream to tap into: the corporate world. Instead of working really hard for a $500 payout, they could now ask upwards of $10Million in corporate extortion money.

It turns out that ransomware criminals have at least four ways to make money from their victims:

1. Extortion to unlock data.
2. Extortion to not make the data public.
3. Extortion to not sell the data to competitors.
4. Extortions 1-3, but sell the data anyway regardless of payment.

In the past, I have recommended an enterprise encryption strategy to counter ransomware revenue streams #2-4. If your material data is encrypted, it's not worth anything to outside parties because they won’t be able to read it. But that leaves us with revenue stream #1.Encryption doesn’t work here because the ransomware criminals will just encrypt your already encrypted data. They don’t have to read it to make it unusable to the victim organization. 

And by the way, unofficially, I have been tracking ransomware groups and campaigns in the news for the past year. By my unofficial count, there are some 50 unique ransomware groups that run one or more ransomware campaigns. The FBI said they were tracking at least 100. That’s not a lot, but the price tag to you if your organization gets caught in the crosshairs is high. According to a study that Sophos did in 2020, “the average ransomware remediation cost in the United States is $622,596.18.” But we have seen the costs to recover from a ransomware attack go very high. According to Andy Greenberg in his excellent Cybersecurity Canon Hall of Fame book, “Sandworm,” the total recovery costs for the 2017 NotPetya attacks for all the victims combined topped out at $10 Billion.

The only way to protect against revenue stream #1, then, is to back it up somehow and be able to restore it at the drop of a hat when the ransomware criminals come calling. Like encryption, that’s a lot easier to say than it is to do. For most of us, our data is scattered across multiple data islands (mobile devices, SaaS Applications, data centers, and hybrid cloud environments). There is no easy button anywhere that will backup your material data on all of these islands and magically restore it all if some catastrophe happens.

Backups as part of the resiliency strategy

As I said in the previous essay, resiliency is “ ... the ability to continuously deliver the intended outcome despite adverse cyber events.” Ransomware is one of these adverse cyber events but the spectrum of potential catastrophes is wide; anything from cyber attacks on one side to natural disasters on the other. 

In this essay and podcast series, I have used the backyard BBQ pit as a metaphor for our first principle infosec program. Each brick for our BBQ pit gives strength to the brick underneath it. The foundation for the BBQ pit consists of four pillars and resiliency is as important as our other three: Intrusion kill chain prevention, zero trust, and risk assessment.For now, I have two BBQ pit bricks that sit on top of the resiliency brick. I covered the first one, encryption, in the last essay. In this essay, I’m going to do some first principle thinking when it comes to backups.

Backup only material data.

One note of caution: you don’t have to have a complete solution for backing up and restoring all of your data, just the data that’s material to your business. Depending on your organization, the complexity of this first principle task could range from slight to chaotic. Compare what we do here at the CyberWire to Amazon.Those two companies are good indicators of each end of the spectrum. Still, material data is a subset of all of the data. Let’s not waste resources on things that we don’t need.

Centralized system for all data islands.

According to the June 2021 Gartner quad chart on enterprise backup and recovery platforms, there are six leaders in the space:

• Veeam
• Commvault
• Rubrik
• Cohesity
• Dell
• Veritas

Perusing each of the respective websites, you get a sense about what these centralized platforms try to do. They all claim the capability of backing up and restoring virtual workloads (like VMware and Hyper-V), hybrid cloud environments (like Google, Amazon, and Microsoft), specific SaaS applications (like SAP and Exchange), and storage devices (like NetApp and Nutanix). You can install them in the cloud or run them from your own data centers. With this kind of model, one organization within your business would be responsible for maintaining the system. In other words one business unit, say the IT shop,would keep the blinky lights blinking. Other business units would provide input into the specific policies.

Decentralized systems for each data island.

By decentralization I mean that you might consider backup and restoration solutions designed for the specific data islands you are worried about. If you are in an Amazon Cloud for example, you might consider using their EBS Snapshot service that can, according to Amazon, “enable disaster recovery, migrate data across regions and accounts, and improve backup compliance.” For your data centers, if you are using the Nutanix storage system, you might consider their “Disaster Recovery Solutions for Business Continuity” service. The point is that instead of having just one backup and restore platform that handles the tasks for all data islands, you would run specific backup and restore solutions for each data island.

DevSecOps for each application.

The DevSecOps plan would be to include backup and restore capability whenever you roll out a new application as infrastructure as code. This is how the big boys (like Google, Netflix, and Salesforce) do it. From the Cybersecurity Canon Hall of Fame book, “Site Reliability Engineering: How Google Runs Production Systems,” deploying backup systems is part of the task. Their Site Reliability Engineers (SREs) apply computer science and engineering to “the design and development of computing systems.” In other words, they are looking to build reliable solutions and backups and restore operations are a key part of it. “Traditionally, companies protect data against loss by investing in backup strategies. However, the real focus of such backup efforts should be data recovery, which distinguishes real backups from archives. As is sometimes observed: no one really wants to make backups; what people really want are restores.” 

How do you get to Carnegie Hall? Practice.

Bringing this conversation full circle to my personal Lion King data recovery fiasco, the lesson I learned back then is as important today as ever. Whatever backup and recovery tactic you choose to support the resiliency strategy, you are not done before you have actually practiced the restoration process and you’re sure that you can deliver the intended outcome with the new, reinstated data. This is the thing that you have to iterate on. You have to be so good at this that it becomes second nature. Don’t make the mistake I made and end up humbling yourself to the teenage member of the Best Buy Geek Squad. Trust me, that is not a good feeling.

Reading list.

S1E6: 11 MAY: Cybersecurity First Principles

S1E9: 01 JUN: Cybersecurity first principles - resilience

S2E5: 17 AUG: Data loss protection: a first principle idea.

S2E6: 24 AUG: Data loss protection: around the Hash Table.

Sources

“Amazon EBS Snapshots- Backup and Data Protection Service - Amazon Web Services.” 2020. Amazon Web Services, Inc. 2020. 

"Cyber Resilience – Fundamentals for a Definition,” by Fredrik Björck, Martin Henkel, Stockholm University, Janis Stirna, Jelena Zdravkovic, Stockholm University, Article in Advances in Intelligent Systems and Computing, January 2015, last visited 30 April 2020.

“FBI Tracking More than 100 Active Ransomware Groups.” by Kevin Collier, NBC News, 27 July 2021. 

“Gartner Dumps IBM from 2021 Enterprise Backup’n’recovery MQ Leader Corner.” by Chris Mellor, Theregister.com, 20 July 2021. 

“NEW Veeam Backup & Replication V11.” 2021. Veeam Software. 2021. 

“Nasuni.” 2018. Gartner.com. 2018. 

“Nutanix Backup: Disaster Recovery Solutions for Business Continuity.” 2021. Nutanix. 2021. 

“Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin's Most Dangerous Hackers,” by Andy Greenberg, Published by Doubleday, 2019.

“‌Site Reliability Engineering: How Google Runs Production Systems,” by Betsy Beyer (Editor), Chris Jones (Editor), Jennifer Petoff (Editor), and Niall Richard Murphy (Editor), Published by O'Reilly Media, 16 April 2016.

“The Joke.” 2021. Carnegiehall.org. 2021. 

“THE STATE OF RANSOMWARE 2020,” by Sophos, 2020.

“Veritas | the Leader in Enterprise Data Protection.” 2021. Veritas.com. 2021.