Introducing the cyberspace sand table series: The DNC compromise.

Rick Howard: When I was a young captain in the U.S. Army, one of my most memorable tours was being assigned to a field artillery battalion as their signal officer at Fort Polk, La. That's the same Fort Polk that the Army created back in the early 1940s to train its famous commanders - like Eisenhower, Clark, Bradley and Patton - and the soldiers that served them to get ready for World War II. Camp Polk, as it was known back then, was a real-live, physical training environment where military units could actually maneuver on the ground, make mistakes and make adjustments to correct those mistakes before the bullets started flying for real in Europe.

Rick Howard: As a field artillery signal officer in the 1980s, I was in charge of all the communication systems among the five battery commands, the battalion staff elements like admin, logistics, operations and intelligence and the higher commands at the brigade and division levels. By then, the Army had moved the training environment to Fort Irwin, Calif., at a place called the National Training Center, or NTC, and the Army routinely ran all of its units through there to get evaluated. Your career was made there. If you did well at NTC, regardless of your position, you were likely to get promoted; if you didn't, well, there was plenty of insurance and car sales positions open just outside the base. As my editor, John Petrik, says, the Carthaginians used to crucify unsuccessful generals. Today, we just make them sell real estate. 

Rick Howard: On our trip to the NTC, after two weeks of hard training, we were on our last task. We were supporting an infantry brigade on a night mission defending the back end of a valley with two mountain ridges in front of us on both the right and left flanks. We knew exactly where the enemy was going to attack us from, right down the valley's middle, and we deployed many devious surprises designed to convince them to turn around and run away. We were ready. And then we were overrun. At zero dark thirty on the second night of the defense, I was startled awake from my sleeping cot as enemy tanks and personnel carriers drove around and through our position. All of our MILES equipment - and that stands for multiple integrated laser engagement system - think, you know, laser tag for Army people - lit up like a Christmas tree. We were all dead, and we had no idea how the bad guys got through our defenses. 

Rick Howard: Hours later, bleary-eyed at the NTC evaluators' hot wash, all the battalion leadership gathered to get our collective butts chewed and to discover what we did wrong. The briefing officials directed our attention to a sand table stood up in front of the tent. It was a three-dimensional model of the valley we were supposed to have defended, complete with all of our weapons systems placements and those devious surprises that we were so proud of. And then we all saw it collectively at the same time. There was this tiny skinny dirt trail that ran along the outside of the valley's left mountain ridge that we all had completely ignored. The commander of the NTC's world-class opposing forces - or OPFOR, as the cool kids say it - drove a brigade of tanks and infantry single file down that trail. By going down the outside, the OPFOR commander totally screened his noise and movement. He popped out of the trail on our brigade's left flank without opposition or early warning and rolled up our side. The fight was over in 20 minutes. For me, the good news was despite everybody being dead, all the radios continued to work. So, you know, I had that going for me. 

(SOUNDBITE OF FILM, "CADDYSHACK") 

Bill Murray: (As Carl Spackler) So I got that going for me. 

Rick Howard: All of that is a long story that has absolutely nothing to do with cybersecurity - except for the sand table idea, which brings me to my point. One thing we all should be doing is adapting that technique to improve our own cyberdefenses. Now, it doesn't have to be anything fancy like a big physical model inside a giant and dusty tent in the middle of the California desert. It can be done as a pencil-and-paper exercise where we look at famously known cyberattacks - like, you know, OPM and maybe NotPetya - across the intrusion kill chain and compare our current defenses against how the adversary actually succeeded. Military planners do this all the time with famous battles like the Battle of Waterloo and the Battle of Gettysburg. They look at what actually happened on the ground to discover better ways to succeed in future battles. 

Rick Howard: In my NTC example, the sand table crystallized essential defensive problems that we didn't learn from conducting our own physical recon of the area and by revealing the associated contour maps. Maybe we at least should have been watching the left side of the ridge for early warning. Perhaps we should have had a contingency plan in the unlikely case that the crazy OPFOR commander would send his forces single file down a treacherous dirt path at night to get on our flank. I believe we can use cyberspace sand tables in a similar manner, and we can use the intrusion kill chain model to help us move the pieces around the board. 

Rick Howard: My name is Rick Howard, and I am broadcasting from the CyberWire secret sanctum sanctorum studios located underwater somewhere along the Patapsco River near Baltimore Harbor. And you are listening to "CSO Perspectives," my podcast about the ideas, strategies and technologies that senior security executives wrestle with on a daily basis. 

Rick Howard: Cyberspace sand tables is what this new series is about. Every once in a while, I'm going to review an infamous cyberattack that has been in the news to see how the adversary pulled it off, and then I'm going to look at our first principle strategies to see if they would have defeated the adversary's playbook. If not, we might have to make some adjustments or maybe even invent new ones. We'll cross that bridge when we come to it. 

Rick Howard: But let's start with one of my personal favorite cyberattacks - the Democratic National Committee hack of 2016. Let's begin by setting up the cyber sand table. What I mean by that is let's put the pieces on the board before everything got started so that we can visualize the movements later. And let's start with the blue team, the Democratic National Committee, or DNC. The U.S. Democratic Party's operating arm has been the DNC since 1848. Who knew? And the DNC established the Democratic Congressional Campaign Committee, or DCCC, to help elect Democrats to the U.S. House of Representatives roughly 20 years later. 

Rick Howard: In the spring of 2016, though, Secretary Hillary Clinton was the presumptive Democratic candidate of the office of the United States president, and John Podesta was her campaign chairman. Sadly, the security of the DNC network wasn't so good. From David Sanger's interview of Richard Clarke, the former special adviser to the president for cyberspace, in the Cybersecurity Canon Hall of Fame book "The Perfect Weapon," despite the past history of the Watergate break-ins in 1972 and the Chinese and Russian cyber intrusions into President Obama's campaigns in 2008 and 2012, Clarke said that the DNC was, and I quote here, "securing its data with the kind of minimal techniques that you might expect to find in a chain of dry cleaners," end quote. Ouch. That hurts. 

Rick Howard: According to Sanger, they did have an anti-spam service, but it wasn't a very good one. He said that the DNC, quote, "lacked any capability for anticipating attacks or detecting suspicious activity in the network. It was 2015, and the committee was still thinking like it was 1972. The network was a baling-wire-and-duct-tape organization held together largely by the labors of recent college graduates working on shoestring budgets," end quote. It goes without saying they didn't have a CISO. 

Rick Howard: So that's the blue team, the DNC. What about the red team, the Russian GRU? Since 2013, the Russians had been experimenting with something loosely called the Gerasimov doctrine. General Valery Gerasimov, the Russian Federation's chief of the general staff at the time, outlined the strategy in a speech. No. 1 - the quick reduction of the enemy's military and economic potential by destroying critical military and civilian infrastructure. No. 2 - simultaneous warfare on the ground and in cyberspace. And No. 3 - indirect operations or influence operations to confuse and bewilder the enemy's military and civilian populations. In February 2014, Russia invaded Ukraine and used that country as a learning lab to perfect the techniques of the Gerasimov doctrine. Here's Andy Greenberg, author of another Cybersecurity Canon Hall of Fame book called "Sandworm." 

Andy Greenberg: Sandworm is a group of Russian hackers that, since late 2015 or so, have carried out what I think is the first full-blown cyberwar. Starting in Ukraine, they attacked pretty much every part of Ukrainian society with these data-destructive attacks that hit media and the private sector and government agencies and then, ultimately, the electric utilities, causing the first-ever blackouts triggered by cyberattacks. Sandworm hit Ukraine's power grid not once but twice - in late 2015 and then again in late 2016. 

Rick Howard: Despite all the colorful adversary names that the commercial cybersecurity industry has used to describe Russian cyber-adversaries for well over a decade - names like Sandworm, Cozy Bear, Fancy Bear, the Shadow Brokers, CyberBerkut - one of my favorites - Unit 26165, Unit 7455 and Guccifer 2.0 - Greenberg followed the trails from all of that collective activity back to one singular organization, the Russian GRU or Main Intelligence Directorate. Prior to 2016, the GRU turned its attention to the U.S. presidential election. 

(SOUNDBITE OF ARCHIVED RECORDING) 

Unidentified People: (Singing in non-English language). 

Rick Howard: With the cyber sand table board set, let's start the game with the red team, the Russians, their actions from the summer of 2015 to June 2016. To fund their operation, the GRU mined bitcoin and laundered some $95,000 through a web of transactions using what they thought was the complete secrecy of cryptocurrencies. They also used bitcoin to pay for their command-and-control infrastructure and to build their social media personas, like DCLeaks and Guccifer 2.0. According to CrowdStrike, an incident response vendor, the GRU may have gained access to the DNC network as early as the summer of 2015. By 15 March 2016, nearly a year later, they began working their way across the intrusion kill chain, and they started with recon. 

Rick Howard: Four days later, 19 March, they delivered the first phishing message to John Podesta, crafted to look like it came from Google claiming that someone had stolen his password and he should change it immediately. After verifying with his IT guy that the message was legitimate - and yes, it was a guy, one of those overworked and underpaid new college graduates - Podesta clicked the link that took him to a decoy login page, where he entered his credentials. From the kill chain model, this is exploitation. Two days after that, the GRU exfiltrated 50,000 email messages from Podesta's account. Again, from the kill chain model, this is command and control and actions on the objective. 

Rick Howard: By the end of March, the GRU had widened the aperture on the phishing campaign to other DNC staffers. On 6 April, they began delivering phishing email with a malicious Excel attachment called Hillary-Clinton-favorable-rating.xlsx to some employees that worked for both the DNC and the DCCC. Once opened, the staffers were transported to the GRU-controlled website, where they entered their DNC credentials. Some employees that were both DNC and DCCC staffers had adopted the bad practice of using the same credentials to log into their assets on both sides. This gave the GRU legitimate credentials to log into the DCCC network. The very next day, 7 April, they began to recon. 

Rick Howard: By 12 April, they had gained access and moved laterally, compromising as many machines as they could. By 22 April, they had exfiltrated several gigabytes of opposition research material. Between April and June, the GRU reconned for keywords like Trump, Benghazi investigation and opposition research, exploited DCCC computers with malware installation and exfiltrated data to servers in Arizona and Illinois. To obscure their exfiltration activity, the GRU bounced their data through international servers first. Again, from the kill chain model, this is command and control. 

Rick Howard: Let's turn our attention to the Americans, their actions from the summer of 2015 to May 2016. Eight months before the DNC noticed that they were under attack, the National Security Agency, the NSA, notified the FBI of Russian intrusions into DNC networks. An overworked FBI agent tried to call the DNC computer security team but discovered that they didn't have one. He ended up talking to the DNC computer help desk. The help desk guy - and yes, it was a guy - took the information but didn't really trust it. He thought it might be a spoof. He sent a memo to senior leadership anyway, but nobody did anything about it. By November, the FBI had more hard evidence of GRU-controlled command and control traffic originating from the DNC network. The DNC tech people still didn't trust the information and didn't tell the DNC leadership. Meanwhile, the FBI never told the White House. 

Rick Howard: By April 2016, four months later, the FBI finally established a face-to-face meeting with the DNC tech people and convinced them that the intelligence was legitimate. They also convinced them to deploy some security detection technology. The DNC installed CrowdStrike's Falcon project, an endpoint detection and response product, or EDR. The next month, May 2016, the DNC's EDR platform identified indicators of compromise from both Cozy Bear and Fancy Bear, and the CrowdStrike incident response team began the work of ejecting that presence from the DNC network. 

Rick Howard: Meanwhile, back in Russia, between April and June 2016, the GRU gained access to the DNC's Microsoft Exchange server and stole thousands of emails. They compromised 33-plus DNC hosts, including the email server and the DCCC website. The DCCC website redirected visitors to a look-alike site called actblues.com, plural, meant to mimic the popular fundraising site ActBlue, singular. They began reconning for information about state boards of election, political parties and in the process compromised at least 10 DCCC computers. 

Rick Howard: By July, they had compromised the Board of Elections website for the state of Illinois and stole personal information for some 500,000 voters. They conducted a supply chain cyberattack by compromising a software vendor called VR Systems, responsible for verifying voter registration in several U.S. elections. By October, they had reconned the networks of election officials for specific counties in Georgia, Iowa and Florida. They also gained access to the DNC's cloud presence, created backups using the cloud provider's technology and then copied the backups over to the GRU's own instance in the same cloud provider. I have to hand it to them. That was a nice touch. 

Rick Howard: Before the election in November, the GRU spear phished potential victims from spoofed accounts that looked like they originated from the voter registration software company VR Systems. The email contained Word documents that presented the software vendor's logo but was also infected with malware. 

Rick Howard: Back in the American camp from May 2016 to November 2016, the CrowdStrike incident response team started collecting intelligence on the DNC intrusions on 1 May and began formulating the ejection plan. They executed a plan on 10 June and completed their work just three days later. They were ready to go much earlier, but the DNC leadership wouldn't let them because they were worried about ongoing campaign operations. Yikes. That sounds vaguely similar to how the OPM leadership, the U.S. Office of Personnel Management, reacted to discovering that Chinese hackers were in their network. They decided to watch them to see what they would do instead of kicking them out. Double yikes. But I digress. We'll get to them in a show down the road. 

Rick Howard: The DNC leadership delayed executing the ejection plan for an entire month while the GRU forces ran around in their networks. Eventually, they had all the DNC employees turn in their computers and mobile devices. When the DNC gave them back, the incident response team had wiped all the data clean. It had installed brand-new versions of software. From that time on, with permission of the DNC, CrowdStrike shared its intelligence with the FBI. That said, the DNC leadership still didn't fully trust the FBI and didn't allow them access to their physical servers. According to David Sanger, the only intelligence the FBI was getting was secondhand via CrowdStrike. On June 14, the DNC went public with the information. And just over a month later, 29 July, the DCCC did the same. Subsequently, the DNC chair, Debbie Wasserman Schultz, and the CEO, Amy Dacey, resigned. The new interim chair, Donna Brazile, formed a cybersecurity advisory board to provide advice to senior leadership. Members included Shawn Henry, the CrowdStrike President, Aneesh Chopra, President Obama's former chief technology officer, and Michael Sussmann, a former federal prosecutor and a former partner at the law firm Perkins Coie, who focused on privacy and cybersecurity law - just to name three. Brazile also brought in cybersecurity expert volunteers from Facebook, Google and Coinbase. The DNC implemented new security measures, too. They required employees to log off when they left their machines. And they implemented two-factor authentication. They abandoned the DNC email system and replaced phone calls for sensitive subjects with Apple's FaceTime audio. CrowdStrike did miss a Linux-based version of malware called X-Agent in their June sweep. They didn't inject that from the DNC networks until October. X-Agent, also known as Sofacy, collects keystrokes on the infected computer. 

Rick Howard: In the end, the American voters chose candidate Trump to be the 45th president of the United States. It's unclear if the GRU's tactical execution of the Gerasimov Doctrine with the goal to influence the U.S. presidential election changed the outcome. Expert opinion on the matter differs. On the one hand, measuring causal effectiveness in marketing campaigns or influence operations is not an exact science, especially when there are so many competing efforts trying to sway opinion. We have everything from official candidate campaign messaging to less official political action committee operations to Uncle Kevin's rants on Facebook. Combine all of that with Russian influence operations, and it's a mess. On the other hand, the election was so close. President Trump received 304 electoral votes, compared to 227 that Secretary Clinton collected. Still, Secretary Clinton earned more than 2 million popular votes over what President Trump gathered. The election really came down to a handful of states that could have gone either way right up to Election Day. An influence operation wouldn't have had to do much to change the outcome. Regardless, there is one question the DNC leadership should have been asking themselves some two years before the election. What data are material to our campaign efforts? What data, if lost, destroyed or leaked, will materially damage our efforts? Because they didn't choose to protect their material data - or any data, for that matter - during the final days of the election cycle, they had to redirect their campaign messaging resources to rebut the negative press from the leaked Podesta email cache. I wasn't there, so take this with a grain of salt, but I assume they would have preferred to stay on their positive messaging about Clinton's platform. Because of the GRU cyberattack, the Democratic National Committee chair, Debbie Wasserman Schultz, and the DNC CEO, Amy Dacey, resigned after the hacks went public. Donna Brazile replaced Schultz and in her book "Hacks: The Inside Story of the Break-ins and Breakdowns That Put Donald Trump in the White House," said that the DNC had already spent $300,000 on remediation but expected to pay out a total of $4 million when it was all said and done. That's how it played out. From the summer of 2015 to the presidential election in November 2016, both sides - the DNC, the blue team, and the Russian GRU, the red team. If this was an NTC exercise, like the one I told you about at the top of the show, and I was an NTC evaluator, I would have gathered all of the DNC leadership around my cyber sand table to chew their butts collectively and to show them what they did wrong so that they could fix it the next time. I would've walked them through that analysis we just did and then asked them what they could have done better to prevent it in the future. 

Rick Howard: But let me play devil's advocate for a second. In 2016, we should consider the DNC and the DCCC as comparable to start up organizations in the commercial world. They had limited resources in a short amount of time to get their product noticed and successful. For them, their product was a Democratic candidate for the U.S. president. And with most startup leadership, they weren't inclined to spend resources on side projects that didn't directly contribute to the bottom line. And according to Brazile, the interim DNC chairman, their burn rate was almost $4 million a month during that last year. And they were already swimming in debt. Let's face it. Cybersecurity was not high on anybody's priority list. That said, before 2015, if you would've asked me to estimate the probability of material cyberattack before the election in 2016, my first guest would have been north of 75%. With the Democratic Party's history of physical and electronic break-ins - Watergate in '72 and President Obama campaigns in 2008 and 2012 - and the fact that it didn't have a CISO nor any kind of detection and prevention technology in place to stop bad guys, there were an obvious and vulnerable target. The fact that nobody in the Democratic leadership, from the DNC all the way to the White House, didn't know that is scary. I mean, I could've hacked the DNC with my crack team with 7-year-old "Fortnite" players. I'm just saying. Even with the DNC's resource constraints, some simple things could've been done early on in terms of cybersecurity first principles that could have prevented the disaster - maybe not the GRU breach but perhaps the prevention of the Podesta email exfiltration. The first simple task that comes to mind falls under the resiliency first-principle strategy and the need for crisis planning. The very least they could've done at no cost would be do establish an open channel with the FBI. As the saying goes, you don't want to be exchanging business cards with the FBI for the first time during an actual crisis. But that's exactly what they did. And the thing is that the FBI is really good at this kind of advisory role. If the DNC had an open communication channel with the FBI, the two parties wouldn't have waited months from the summer of 2015 through April 2016 learning how to trust each other. They might have been able to put measures in place in 2015 that would've prevented the success of the GRU in 2016. The next first-principle task falls under the zero trust strategy and the tactic of using two-factor authentication for employee logins. That simple preventative measure would have stopped the exploitation of the Podesta emails and the compromises of the DNC and DCCC hosts. Even if the DNC and the DCCC employee victims did unwittingly give their login credentials to the GRU, the Russian hacking team wouldn't be able to use them easily because they wouldn't have had access to the two-factor devices. And finally, the last first-principle task falls under the intrusion kill change strategy. Before the DNC deployed the CrowdStrike Falcon product, they couldn't detect the GRU and their networks even with the FBI showing them the way. After the installation, their product immediately discovered indicators of compromise for Cozy Bear and Fancy Bear, CrowdStrike's code words for Russian cyber campaigns. Clearly, having the capability to look for known adversary behavior is an essential capability, especially for quasi-government organizations like the DNC that have known enemies around the world. 

Rick Howard: The bottom line to all of this is that deploying these three first-principle tactics - one, talking to the FBI for resiliency and crisis planning; two, installing two factor authentication Brazil trust and identity; and three, looking for known adversary activity across the intrusion kill chain for GRU attack campaigns - would have reduced the probability of material impact. How much is debatable. But without those measures, I forecast that the probability would be above 75%. With them, I put it at below 20%. The GRU may have found a way around those measures eventually, but they would have had to work for it. And that's with a minimal deployment of our first-principle strategies. Consider a more mature deployment and how low they could have reduced the probability even more. The proof is in what the DNC did next for the 2018 and 2020 election cycles. They hired Bob Lord, former Yahoo CISO, to run the infosec program. And by all accounts, because of the changes he made to the infosec program and other help he got from the Department of Homeland Security, the Russians didn't penetrate the DNC during that time. Military commanders have used some version of a physical sand table since the world was young. They did so because it works. And I know that some network defenders don't like using military metaphors for cyberdefense. Fine. If you don't like the military metaphor, use a sports metaphor. Cyber sand tables are no different than your high school basketball coach drawing up plays on the whiteboard at halftime to illustrate why the other team was kicking your backside on the court. They are no different from Tom Brady, the current NFL Tampa Bay quarterback, watching hours of film on his opponents to get ready for the next contest. And who can argue with that success? He's won seven Super Bowls out of 10 tries while playing on two different teams. If he can spend the time at the sand table, I think we can, too. 

Rick Howard: And that's a wrap. As always, if you agree or disagree with anything I've said, hit me up on LinkedIn or Twitter, and we can continue the conversation there. Or if you prefer email, drop a line to see csop@thecyberwire.com. That's csop@thecyberwire - all one word - .com. And if you have any questions you would like us to answer here at the "CSO Perspectives" podcast, send a note to the same email address, and we will try to address them in the show. The CyberWire's "CSO Perspectives" is edited by John Petrik and executive produced by Peter Kilpe. Our theme song is by Blue Dot Sessions, remixed by the insanely talented Elliott Peltzman, who also does the show's mixing, sound design and original score. And I am Rick Howard. Thanks for listening.