Resilience Case Study: Colonial Pipeline attacks of 2021 (Cyber Sandtable).

Rick Howard: This past weekend in Northern Virginia, we had glorious weather. My wife Kathy (ph) and I took the opportunity to do a little antiquing. Actually, she did the antiquing, and I mostly drove and napped in the car. As I parked near the Roaches in the Attic antique shop - one of Kathy's favorites - I just happened to notice on the GPS that we were very close to something called the Colonial Pipeline Dulles Junction. Yes, that Colonial Pipeline, the energy conduit that gave us all so much trouble back in 2021 when cybercriminals extorted Colonial Pipeline, the company, with ransomware and the company leadership shut down their gas distribution pipeline just to be safe. I gave Kathy my go-to look with over 35 years of marriage practice that said, we just got to see it.

(SOUNDBITE OF FILM, "INDEPENDENCE DAY") 

Will Smith: (As Captain Steven Hiller) I have got to get me one of these. 

Rick Howard: She returned the favor with her own steely gaze, complete with an eye roll that signified that I could do whatever I wanted as long as I was back by noon to take her to lunch. With permission in hand, I set off to see if I could find anything interesting. I don't know what I was expecting, but there wasn't much to look at. It's pretty small, a tad tinier than your typical house plot, situated between two neighborhoods near the Dulles International Airport and sitting alongside the Horsepin Run Stream Valley Park. The space is flat and surrounded by a six-foot-high fence. But inside the perimeter on the left side is a silver gas pipe, 32 inches in diameter, that protrudes from the ground for maybe 15 horizontal feet and then sinks back into the dirt to continue its journey to Baltimore. This line - line four - moves roughly 7,000 barrels of gas per day from Greensboro, N.C., to Baltimore, Md. It's one of the many connecting points for the Colonial Pipeline system, and it's part of the largest pipeline in the United States. The entire system can carry roughly 3 million barrels of fuel a day over 5,500 miles from Houston to New York. It connects directly to several major airports, including Atlanta, Nashville, Charlotte, Greensboro, Raleigh-Durham, Dulles and Baltimore/Washington. 

Rick Howard: In other words, this is how your airports on the East Coast get their jet fuel. When you think of the pipeline in those terms, the scale of it, you quickly realize the significance of the system to the national economy. One small glitch in any of those pipeline junction points from Houston to New York could send rippling waves of energy shortages across the country. The Colonial Pipeline ransomware attack of 2021 did just that, and the attack sequence didn't even touch the pipeline's operational technology - or OT - and industrial control systems, or ICS. The attackers went after the traditional business IT systems. They caused the shortage for being in the general vicinity of the pipeline. And that's just kind of scary. 

Rick Howard: When we talk about cybersecurity first principle strategies, the sexy ones today are zero trust and intrusion kill chains. There have been a lot of interesting developments in the 2010s from vendors and security researchers that will contribute to making it easier to deploy those ideas in the future. There isn't a lot of innovative discussion about resilience and risk forecasting, though. We don't talk about risk forecasting, you know, because it's hard. We don't talk about resilience because it's hard and not sexy. For resilience specifically, I think a lot of us feel like it's kind of the cybersecurity equivalent to eating your vegetables. And to pile on, resilience is much bigger than just cybersecurity alone. As my favorite Swedes, Bjorck, Henkel, Stirna and Zdravkovic said in a paper published in 2020, resilience is, quote, "the ability to continuously deliver the intended outcome, despite adverse cyber events," end quote. That includes cybersecurity attacks but also an entire slate of other business continuity issues that security leaders don't own and probably don't want. Truth be told, the leaders that do own those programs don't want the CISO's help anyway. 

Rick Howard: Still, for the Colonial Pipeline attacks, I think there are some lessons to be learned here about resilience by reviewing how the hackers orchestrated their attacks and how the Colonial Pipeline leadership responded. Were there resilience things that could have been improved that would have prevented the crisis? That sounds to me like a pretty decent cyber sand table exercise. Let's get started. 

Rick Howard: My name is Rick Howard, broadcasting from the CyberWire's secret sanctum sanctorum studios, located underwater somewhere along the Patapsco River near Baltimore Harbor. And you're listening to "CSO Perspectives," my podcast about the ideas, strategies and technologies that senior security executives wrestle with on a daily basis. 

Rick Howard: Let's set up the sand table and we will start with Colonial Pipeline. According to Clifford Krauss at The New York Times, the Colonial Pipeline started as a consortium of big oil companies - Phillips, Sinclair and Continental Oil back in 1961. Today, it's owned by Royal Dutch Shell, Koch Industries and many other international investment companies. The reason it's so vital to many Eastern Seaboard airports is that they mostly only have a supply of three to five days of fuel stored locally. Claudia Piccirilli, reporting for the WTW website, said that at the time of the attack, Colonial Pipeline didn't have a chief information security officer. And so the security responsibility for the company's infosec program fell to the CIO who had been in the seat since 2016. Let that sink in for a second. In a critical infrastructure company, a company that's responsible for providing the fuel for many of the airports on the Eastern Seaboard of the United States, didn't have an executive dedicated to security. The mind boggles. 

(SOUNDBITE OF TV SHOW, "HOME IMPROVEMENT") 

Tim Allen: (As Tim Taylor) Oh, no. 

(LAUGHTER) 

Rick Howard: Christopher Burgess from CSO Online cited an AP interview with a consulting firm, IMERGE, that said in a 2018 audit report that Colonial Pipeline's network security was severely deficient. This is three years before the ransomware attack. That said, the CIO did take some steps to improve the situation by hiring a senior director of technology solutions and increasing the IT budget by 50% - not the security budget, but the IT budget. And presumably, some of that money was spent on improving Colonial's security environment. 

Rick Howard: So let's look at the sand table from the criminal side, DarkSide. The Cybersecurity and Infrastructure Security Agency, CISA, and the Federal Bureau of Investigation, the FBI, almost immediately attributed the Colonial Pipeline cyberattack to a ransomware-as-a-service offering called DarkSide, with headquarters based in Russia but with no known ties to the Russian government. My editor, John Petrik, says that, yes, of course, quote, "no known direct ties but longstanding protection, toleration and enablement by the Russian government. DarkSide was, in all probability, a privateer," end-quote. In other words, DarkSide probably had unofficial state approval to harass and disrupt foreign enemies with cybercrime. 

Rick Howard: Researchers over at the Votiro website say they started seeing the first instances of the DarkSide servers back in November of 2020, but there is some evidence that hackers behind the servers were experimenting with the tools as far back as April of 2019. Their typical attack pattern was to compromise the victim and then go dark for a while before launching the ransomware. In the past, their one-two punch consisted of the exfiltration of data for blackmail purposes and then encryption of the data for ransom purposes. John Petrik says that they were early adopters of this double extortion idea. Researchers observed them gaining initial access through phishing attacks. They preferred to target remotely accessible accounts and virtual desktop infrastructure, or VDI, and they maintained persistence with a remote desktop protocol, or RDP. According to Snir Ben Shimol at Varonis, the DarkSide malware checks device language settings to avoid Russian victims, like looking for a Cyrillic keyboard, a mark of a privateer. And it works on both Windows and Linux systems. On their deep-web advertising site, DarkSide's marketing people - and yes, the gangs do have marketing crews - profess to have an honor code of not attacking hospitals and schools, and they claim to have on several occasions donated some of their proceeds to charities. 

Rick Howard: CSO Online's Cynthia Brumfield interviewed CrowdStrike's Josh Reynolds and Eric Lou back in 2021. Those two attributed the DarkSide attack pattern to another adversary playbook, Carbon Spider, aka FIN7, aka G0046 and aka JokerStash. According to Cynthia, Carbon Spider started in 2013 using the Carbanak malware to target financial institutions. The CrowdStrike researchers said that the group split into two in 2016. Cobalt Spider targeted credit card data theft, and Carbon Spider, the group behind the Colonial Pipeline attacks, stayed with financial entities. 

Rick Howard: So now that we have set up the sand table, let's begin the game with turn one, Carbon Spider - DarkSide, the red team, from 29 April to 6 May 2021. On 29 April DarkSide hackers started their initial journey across the Colonial Pipeline kill chain. According to Shimol, they gained the initial beachhead into the Colonial Pipeline's IT infrastructure, not their OT and ICS infrastructure, through a phishing email targeting a contractor who used a VDI system put in place during the pandemic. Before this, they performed stealthy reconnaissance, and when they did attack, quote, "they took steps to ensure that their attack tools and techniques would evade detection on monitored devices and endpoints," end-quote. 

Rick Howard: Shimol says that DarkSide hackers installed the Tor browser on victim machines to establish their main command and control, or C2, channel. They configured them to run as a persistent service and redirected traffic through the Tor network. He said that his team found evidence of Tor clients on many of the Colonial Pipeline servers and collected telemetry on several active Tor connections. As a secondary C2 channel, the DarkSide hackers used Cobalt Strike. The Varonis team found evidence of dozens of Cobalt Strike stagers that DarkSide deployed using WinRM, or Windows Remote Management. They configured every stager differently and connected each to its own unique and remote C2 server, which indicates a relatively large C2 infrastructure. Shimol said that the hackers stored Tor browser executables on Colonial Pipeline file shares but avoided systems with deployed EDR clients, or endpoint detection and response clients. 

Rick Howard: For lateral movement, the Varonis team said the DarkSide hackers logged into many VDI accounts, sometimes several at once, and created dot-link files back to the compromised victim's home folders. They used these shortcuts to keep track of successful machine breaches and the associated accounts used. They collected credentials using the Mimikatz DCSync attack tool to steal credentials from the entire domain and an active directory reconnaissance tool called ADRecon.ps1 that siphoned out information about users, groups and privilege. They also retrieved user profile credentials from the browsers of Microsoft, Firefox and Google and took care to delete each attack tool after use. Shimol says that the DarkSide hackers mined data from hundreds of victims' machines simultaneously using a batch routine and then compressed them into zip files. He said that even though they had gained the elevated privileges, they chose instead to reduce restrictions on the various file shares, so that normal users without any privilege could access them. Quote, "the batch file, target data and the archives were deleted by the attackers within hours of collection" end-quote. According to Varonis, DarkSide delivered the ransomware code - the data encryption piece and the accept payment piece - through the already established C2 infrastructure. On 6 May, the hackers behind the DarkSide attack completed their exfiltration of 100 gigabytes of data, the equivalent of a stack of paper taller than the tower over Burj Khalifa, one of the tallest skyscrapers in the world, encrypted everything and then demanded payment, not only to decrypt the data but to prevent the release of it to the public. The actors then threatened to publicly release the data if the ransom wasn't paid. 

Rick Howard: So that's turn one for DarkSide. Let's see what Colonial Pipeline, the blue team, did for turn one, 7 through 15 May, 2021. According to Joseph Blount, the Colonial Pipeline CEO, on 7 May, Friday, just before 5 a.m., a control room employee saw the ransom demand appear on his computer screen. Once notified, the employee's supervisor began shutting down the pipeline as a precautionary step and completed the task by 6:10 in the morning. The same day the Colonial Pipeline leadership team decided to inform the FBI about the attack, they also authorized and delivered the payment of the 75 bitcoin ransom, nearly $5 million. 

(SOUNDBITE OF TV SHOW, "HOME IMPROVEMENT") 

Tim Allen: (As Tim Taylor, grunting). 

Rick Howard: Let's pause on that little factoid for a second. Within hours of the ransom request, Colonial Pipeline leadership was able to put its hands on $5 million and execute a bitcoin transaction. I mean, I knew that the oil industry was rich and connected, but if I want to extract just $500 out of my own bank account, it usually takes three days. They were able to call the bank, grab a cool $5 million and transfer it out to some shady ransomware players all in one day. I'm just saying. I have been unable to find this out for sure, but the execution speed for this three-step action list - ransomware notification, notify the FBI and execute a bitcoin payment - was so swift that it leads me to believe that this was part of a planned crisis action response, something that the leadership team had practiced and were prepared to execute. I could be wrong about that. I asked the Colonial Pipeline CIO to come on the show to discuss, but she didn't respond. I don't blame her, either, but I thought I would ask. 

Rick Howard: The next day, 8 May, Saturday, Colonial Pipeline announced to the public that they had been hit by a ransomware crew and what they were doing about it. Again, this is another piece of evidence that this was part of the crisis action playbook. Announce early, and keep the public informed. They continued the public broadcast of information throughout the crisis. But according to Derek Johnson at SC Media, quote, "the company had a plan in place for undergoing a controlled shutdown of pipeline operations in the event of a loss of SCADA or voice communications control, but federal regulations specifically require companies to have and test a plan for resuming operations manually in those conditions. Colonial didn't do that," end-quote. Incident responder contractors, with help from the FBI, CISA and the NSA, identified the Colonial Pipeline internal staging servers that DarkSide was using to exfiltrate data and took them offline. This was too late to stop the 100 gigabytes of data already exfiltrated but prevented even more damage down the road. A computer security company specializing in cryptocurrency, Elliptic, announced that it had identified the bitcoin wallet used by DarkSide to collect the Colonial Pipeline ransom payment. 

Rick Howard: The next day, 9 May, Sunday, U.S. President Joe Biden declared a state of emergency and removed restrictions concerning fuel transportation by road. On Monday, 10 May, Georgia Governor Brian Kemp declared a state of emergency and waived collection of the state's taxes on diesel and gasoline. President Biden also announced that the attacks were of Russian origin but not sponsored by the Russian government, and the FBI confirmed that DarkSide was behind the attack. Colonial Pipeline leadership announced it had manually opened a piece of the pipeline temporarily - line four from North Carolina to Maryland - for a short period to get the existing oil it had on hand down the line. On 11 May, Tuesday, CISA and the FBI issued a cybersecurity advisory that described how the DarkSide ransomware worked and provided suggested risk mitigation strategies. Colonial Pipeline outlined their alternative fuel shipping strategies that they now had in place to lessen the impact of the crisis. The very next day, Wednesday, 12 May, Colonial Pipeline resumed basic operations. It had taken the previous five days to verify that the pipeline OT and ICS systems weren't infected by the DarkSide ransomware. Still, more than 1,000 gas stations didn't have any gas, and U.S. citizens were in the middle of a panic-buying spree across the Southeastern United States. Remember the picture of the guy filling plastic bags at the pump with gasoline, the same kind of plastic bags that would dissolve in minutes because of its reaction to the fuel? 

(SOUNDBITE OF TV SHOW, "HOME IMPROVEMENT") 

Tim Allen: (As Tim Taylor) Oh, no. 

Rick Howard: Yeah - that kind of panic-buying. On 13 May, Thursday, the FBI announced it had hacked the DarkSide bitcoin wallet and moved the digital currency to a wallet that they controlled. Essentially, they stole the money back from DarkSide. How cool is that? Law enforcement officials wouldn't elaborate on how they did it, but according to Mathew Schwartz, writing for Euro Infosec, clues exist. He quotes Pamela Clegg, the director of education investigations for blockchain analytics at CipherTrace, who claimed that the FBI got the DarkSide wallet key from some other international law enforcement agency who had penetrated the DarkSide cryptocurrency infrastructure prior to the pipeline attacks. According to Schwartz, reading from an affidavit in support of a search warrant filed with the Northern District Court of California, quote, "the cryptocurrency was moved through at least six other bitcoin wallets," end-quote. The FBI followed the flow of funds until they ended up in a wallet for which they had a private key. By 15 May, Saturday, Colonial Pipeline had everything turned on again, but at this point, there were over 10,000 gas stations still out of fuel. It still took several days to get back to normal. 

Rick Howard: In this case, turn two is a wrap-up for both sides on the sand table, DarkSide and Colonial Pipeline. On 14 May, DarkSide told its affiliates that because of pressure from U.S. law enforcement, it was closing shop. But many intel analysts were skeptical and suspected that this announcement was just another rebranding exercise, similar to what other ransom groups in the past had undertaken, like BitPaymer changing to DoppelPaymer and ultimately to Grief or Hermes rebranding from Ryuk to eventually Conti. According to Cynthia Brumfield at CSO Online, a new ransomware-as-a-service emerged on 21 July called BlackMatter. CrowdStrike said that there was enough overlap in the tools in the attack sequence that they were fairly certain the service was just DarkSide operating under another name. 

Rick Howard: For Colonial Pipeline, almost to the day a year after the Darkside attacks, the U.S. Department of Transportation announced it was seeking to levy nearly $1 million in fines against Colonial Pipeline for a series of safety violations that they say contributed to the pipeline's decision to temporarily shut down gas operations on the first day of the attack. According to SC Media, Colonial Pipeline leadership welcomes the investigation and wants everybody to know that this was, quote, "the first step in a multistep regulatory process, and we look forward to engaging with the process to resolve the matters," end-quote. They also defended the contingency planning in the wake of the ransomware attack, saying it was necessary and tailored to the company's operating environment. 

Rick Howard: In this podcast, we talk about cybersecurity first principle strategies. Resilience is one of them, and it's of the same importance as the other three - zero trust, intrusion kill chain prevention and risk forecasting. As I said at the top of the show, resilience is the ability to continuously deliver the intended outcome despite adverse cyber events. Clearly, the Colonial Pipeline response to the DarkSide ransomware attacks did not meet that standard. Not only did the leadership not continuously deliver fuel to their customers during the crisis, but there was an Eastern Seaboard shortage of over a week. 

Rick Howard: To have a well-deployed resilience strategy, though, you have to be pretty good at several resilience tactics - crisis planning, backup and encryption of material data and incident response. For crisis planning, it appears they had a plan to deal with the ransomware and had at least talked about how they would execute it before the crisis happened. The way Colonial Pipeline came out of the gate swinging immediately, shutting down the pipeline, notifying the FBI and paying the ransom all on the first day of the attacks shows some prior planning. As I said, I haven't confirmed that with anybody at Colonial Pipeline, and the U.S. Department of Transportation has some concerns with the plan they executed, but it looks like they had a plan. 

Rick Howard: Colonial Pipeline didn't encrypt their material data - and probably any data for that matter, especially the 100 gigabytes of data Darkside exfiltrated to their own servers. And it's unclear if Colonial Pipeline had a decent backup of their material data. That didn't seem to factor into their incident response plan, though. They saw the ransom demand and immediately shut down the pipeline. The one bigger error in their plan was that they couldn't determine whether the ransomware attack was isolated to the IT side of the house or had contaminated the OT and ICS side. They just assumed that everything was contaminated and shut it all down. That was the safe call for sure, but it didn't meet the resilience standard of continuously delivering the intended outcome. And I'm not even really talking about how Colonial Pipeline failed to implement the other three first principle strategies either. A simple zero-trust tactic like two-factor authentication would have prevented the DarkSide initial entry point. But we're focusing on resilience here. 

Rick Howard: As I said when I did the OPM sand table exercise, it's easy to Monday morning quarterback massive failures in preventive cybersecurity. But for all network defenders, during the heat of battle, it's tough to take a beat and reflect on what could be done better next time. This is why cybersecurity sand table exercises are important. When there isn't a crisis afoot, you can learn quite a bit by taking a few moments to analyze what happened on both sides. I highly recommend that you insert them into your first principle programs. 

Rick Howard: And that's a wrap. As always, if you agree or disagree with anything I have said, hit me up on LinkedIn or Twitter, and we can continue the conversation there. Or if you prefer email, drop a line to csop@thecyberwire.com That's csop@thecyberwire.com. And if you have any questions you would like us to answer here at CSO Perspectives, send a note to the same email address, and we will try to address them in the show. 

Rick Howard: Next week is a three-day weekend here at the CyberWire. It's Juneteenth National Independence Day. So CSO Perspectives is off next week. Besides, we're all still recovering a little bit from attending the big RSA conference last week. With the time change between San Francisco and Washington, D.C., and all the interviews we did and all the evening's festivities, we're all just a little worn out. Let me just say that I'm not as young as I used to be. But the week after, we're going to do a case study on resilience, specifically on the Netflix Chaos Monkey system. You don't want to miss that. 

Rick Howard: The CyberWire "CSO Perspectives" is edited by John Petrik and executive produced by Peter Kilpe. Our theme song is by Blue Dot Sessions, remixed by the insanely talented Elliott Peltzman, who also does the show's mixing, sound design and original score. And I am Rick Howard. Thanks for listening.