Enterprise backups and cybersecurity first principles.

Rick Howard: About 15 years ago, I set up the backup scheme for the Howard family dataset. Digital was just starting to become mainstream and we had all of these electronic artifacts scattered across mobile phones, digital cameras (geez, remember those?), and the family home computer. Between five people, the wife, two daughters, and one son. It was starting to get out of hand. I realized that some of these items like precious videos of my daughters leaping across the stage to their dance studio production of The Lion King and all of our TurboTax files for the past 20 years, just to name two might be worth spending the time to get it organized in one place and then backing the data up so that one catastrophe didn't wipe everything out.

Rick Howard: And just as an aside, by the time the family wrapped the production of The Lion King, we had spent enormous amounts of resources in terms of the purchase of costumes, dance rehearsal time and backstage prep time. And make no mistake, it was a full court press on family participation. The two daughters were in something like a thousand numbers combined. Mom was the backstage coordinator and my son and I, we were security, which meant we spent a lot of time directing traffic in the local high school parking lot. At the end of the production, we treated the entire family to a Disney World trip for a job well done. 

Rick Howard: And there we were in the middle of Disney's Animal Kingdom at the intersection of the Pangani Forest Exploration Trail and the Wildlife Express Train, when a bunch of Disney street performers began singing and dancing to The Lion King soundtrack and asking patrons to join in. 

Music from The Lion King

Rick Howard: And oh my God, this is going to be the perfect Kodak moment, right? My two daughters had just spent the last six months perfecting the aforementioned thousand Lion King dance numbers. They were going to kill this. Video camera in hand, I hit the record button, but all I got were too embarrassed, teenagers, swaying, back and forth as awkward as if they learn to walk and chew gum last Tuesday. They weren't even swaying in time to the music. I think that's my all time, favorite video of my two daughters and their dancing career. 

Rick Howard: Clearly I needed to make sure that no computer catastrophe, what caused me to lose that video and all of the other digital detritus that we had collected over the years. I went to work. Not only did I build a scheme that would automatically upload a copy of all of our files to one of the early cloud providers, I also built a local raid array for my home system, so that if any one disc in the array failed, I could just remove it and stick a brand new one in, and nobody would be the wiser.

Rick Howard: The system was foolproof. I had backups of backups. So about a year later, the inevitable catastrophe happened. The hard drive on my home computer failed and I couldn't get it to come back online. My wife gave me that panicked, "what about all my files?" look. Smugly, I looked back at her and say, "Don't worry. I got backups." After building a new computer, I went to my cloud provider first to restore the data. Much to my horror, none of the data was there. I couldn't believe it. There wasn't a single video, picture, or TurboTax file anywhere in the cloud. And that's when my own panic started to creep in, you know the feeling, that sense where you might have screwed up royally in some way that you don't yet comprehend. Yeah. That's the feeling I had, but then I remember the raid array.

Rick Howard: That was my backup of backups. I can restore from there. All I can say is that I had a great plan, and failed completely in execution. Oh, I had a cloud backup system in place, and routinely checked that the system was saving all of my files there. And, I had a raid array where I made sure to make a backup copy of the backup files.

Rick Howard: My failure stemmed from where I told the two systems to backup from. Apparently I configured it so that every day my backup system was copying files from an empty directory and not the directory where everything was stored. Every week or so, I would check to make sure the system was working, and every week I would get the green light, everything a-okay.

Rick Howard: And I'm embarrassed to admit that in order to get my files back, I had to pay the Geek Squad down at the local Best Buy to recover the corrupted files on my home computer hard drive. The experience was, shall we say, humbling, and 15 years later, that's the one story my wife loves to tell to family and friends when they start asking questions about my storied cybersecurity career. It goes something like this. Yeah, let me tell you about my husband and his big fancy pants cybersecurity career when he lost all of the family data for the past 20 years. So in the immortal words of Bill Murray and one of my favorite movies, Caddyshack, "I got that going for me."

Caddyshack clip

Rick Howard: Which is a long way around the horn for me to emphasize that for this episode, we are talking about the extremely sexy topic of enterprise backup schemes as a key strategy to improve our InfoSec first principle of resiliency. 

Rick Howard: My name is Rick Howard. You are listening to CSO Perspectives, my podcast about the ideas, strategies, and technologies that senior security executives wrestle with on a daily basis. 

Rick Howard: Ransomware seems to be having a moment right now. 

Rick Howard: It's interesting to see the evolution of cybercrime, and ransomware specifically, over the last decade. When ransomware first started, the target victim was the home user. Cybercriminals would compromise grandma's computer and tell her that if she wanted her cherished pictures of her cats and grandkids back, that she would have to pay $500 in Bitcoin. 

Rick Howard: The backend business systems that these ransomware groups developed to make this model work was and is astonishing. They had entire call centers dedicated to walking grandma through a Bitcoin transaction. How amazing is that? 

Rick Howard: I'm not sure I know how to do a Bitcoin transaction. But explaining one to grandma? I'm sure there's a special place in hell for that activity that I don't want to visit. 

Rick Howard: The preferred target victim changed sometime in 2017. I heard Nicole Perlroth, the New York Times reporter, talk about this on a podcast somewhere. She said that after the North Koreans launched WannaCry, and one month later the Russians launched notPetya, the ransomware gangs realized that there was a much more lucrative revenue stream to tap into, the corporate world. Instead of working really hard for a $500 payout, they can now ask upwards of $10 million in corporate extortion money.

Rick Howard: And it turns out that ransomware criminals have at least four ways to make money from their victims. Number 1: extortion to unlock the data that they have encrypted. Number 2: extortion not to make the stolen data. Number 3: extortion to not sell the data to competitors. And finally, number 4: all three extortions, 1 - 3, but sell the data anyway, regardless of payment. 

Rick Howard: In the past, I have recommended an enterprise encryption strategy to counter ransomware revenue streams 2 – 4. If your material data is encrypted, it's not worth anything to outside parties because they won't be able to read it. But that leaves us with revenue stream number 1. Encryption doesn't work here because the ransomware criminals will just encrypt your already encrypted data. They don't have to read it to make it unusable to the victim organization. 

Rick Howard: And by the way, unofficially I've been tracking ransomware groups and campaigns in the news for the past year. By my own unofficial count, there are some 50 unique ransomware groups that run one or more ransomware campaigns on any given day. The FBI said in July, 2021, that they were tracking at least a hundred. Now that's not a lot, but the price tag to you if your organization gets caught in the crosshairs, is high. According to a study that Sophos did in 2020, the average ransomware remediation costs in the United States is just over $600,000.

Rick Howard: But we have seen reports from some victims where the recovery costs get very high. According to Andy Greenberg in his excellent Cybersecurity Canon Hall of Fame book "Sandworm", the total recovery costs for the 2017 notPetya attacks for all the victims combined topped out at $10 billion. That's billion with a B.

Rick Howard: The only way to protect against revenue stream number 1, extortion to unlock data, is to back it up somehow and be able to restore it at the drop of a hat when the ransomware criminals come calling. Like encryption, that's a lot easier to say than it is to do. For most of us, our data is scattered across multiple data islands, mobile devices, SaaS applications, data centers, and hybrid cloud environments. There is no easy button anywhere that will back up your material on all these islands and magically restore it all if some catastrophe happens. 

Rick Howard: In the last episode, I said that resiliency is the ability to continuously deliver the intended outcome despite adverse cyber events. Ransomware is just one example of an adverse cyber event, but the spectrum of potential catastrophes is wide, anything from cyber attacks on one side to natural disasters on the other. 

Rick Howard: For this entire podcast series, I've used the backyard barbecue pit as a metaphor for our first principal InfoSec program. Each brick for our barbecue pit gives strength to the brick underneath it. The foundation for the barbecue pit consists of four pillars and resiliency is as important as the other three: intrusion kill chain prevention, zero trust, and risk assessment. 

Rick Howard: For now, I have two barbecue pit bricks that sit on top of the resiliency brick. I covered the first one, encryption, last episode. For this one, I'm going to do some first principle thinking when it comes to backups. One note of caution, you don't have to have a complete solution for backing up and restoring all of your data, just the data that's material to your business. 

Rick Howard: Depending on your organization, the complexity of this first principle task could range from slight to chaotic. Compare what we do here at the CyberWire to Amazon for example. Those two companies are good indicators of each end of the spectrum. Still material data is a subset of all the data. Let's not waste resources on things that we don't need. 

Rick Howard: Now, if we start with the premise that we are all managing several data islands, mobile devices, data centers, SaaS applications, cloud environments, and maybe even hybrid cloud environments, it occurs to me that there are perhaps four ways to conduct backup and restore operations. 

Rick Howard: The first is a single platform that works on all data islands. According to the June, 2021 Gartner Quad Chart on Enterprise Backup and Recovery Platforms, there are at least six leaders in the space. Veeam, Commvault, Rubrik, Cohesity, Dell, and Veritas. Perusing each of these respective websites, you get a sense about what these centralized platforms try to do. They all claim the capability of backing up and restoring virtual workloads (like like VMware  and Hyper-V), hybrid cloud environments (like Google, Amazon, and Microsoft), specific SaaS applications (like SAP and Exchange), and storage devices (like NetApp and Nutanix). 

Rick Howard: You can install them in the cloud or run them from your own data centers. With this kind of model, one organization within your business would be responsible for maintaining the system. In other words, one business unit, say the IT shop, would keep the blinky lights blinking. Other business units would provide input into the specific policies. This is perhaps the least complex of the four options, but it's difficult to find the one solution that covers all of your needs. 

Rick Howard: The second is to deploy distinct backup solutions for each data island. Where option one was a centralized approach, this option is de-centralized for each data island. If you are in an Amazon cloud, for example, you might consider using their EBS Snapshot servers that can, according to Amazon, enable disaster recovery, migrate data across regions and accounts, and improve backup compliance. For your data centers, if you are using the Nutanix storage system, you might consider their Disaster Recovery Solutions for Business Continuity service. 

Rick Howard: The point is that instead of having just one backup and restore platform that handles the task for all data islands, you would run specific backup and restore solutions for each data island. This is way more complex than option number one, but you'll get better coverage. 

Rick Howard: The third option is to embrace DevSecOps where we design the backup and recovery operations for each application, as part of our infrastructure is code movement. This is how the big dogs like Google, Netflix, and Salesforce does it. From the Cybersecurity t of the task. Their site reliability engineers, or SREs, apply computer science and engineering to the design and development of computing systems. They are looking to build reliable solutions and backups and restore operations are a key part. 

Rick Howard: Here's a quote from the book, "Traditionally, companies protect data against loss by investing in backup strategies. However, the real focus of such backup efforts should be data recovery, which distinguishes real backups from archives. As is sometimes observed, no one really wants to make backups. What people really want are restores."  This DevSecOps or SRE option is probably more efficient than the first two options, centralized and decentralized, but most of us have no expertise in the area. Don't get me wrong. I totally believe that this DevSecOps option is the best path to pursue in the long run, but you're not going to achieve this overnight. It will probably take years to get this established. 

Rick Howard: Finally, what I think is the most realistic solution is some combination of all three: centralized, decentralized, and DevSecOps. This is a really complex way, but it's probably what most people are doing right now.

Muppet clip

Rick Howard: Bringing this conversation full circle to my personal Lion King data recovery fiasco, the lesson I learned back then is as important today as ever, whatever backup and recovery tactic you choose to support the resiliency strategy, you are not done before you have actually practiced the restoration process, and you're sure that you can deliver the intended outcome with the new reinstated data.

Rick Howard: This is the thing that you have to iterate on. You have to be so good at this that it becomes second nature. Don't make the mistake I made and ended up humbling yourself to the teenage member of the Best Buy Geek Squad. Trust me, that's not a good feeling. 

Rick Howard: And that's a wrap. Next week, we are inviting our subject matter experts to the Hash Table to see how they do backup and restore operations for their organizations. You don't want to miss that, but as always, if you agree or disagree with anything I've said, hit me up on LinkedIn or Twitter and we can continue the conversation there.

Rick Howard: The CyberWire's CSO Perspectives is edited by John Petrik and executive produced by Peter Kilpe. Our theme song is by Blue Dot Sessions, remixed by the insanely talented Elliott Peltzman, who also does the show's mixing, sound design, and original score. And, I am Rick Howard. Thanks for listening.