Cyber sand table series: OPM.

Last year, I wrote an essay called, “Introducing the cyberspace sand table series: The DNC compromise.” I got the idea from my old military days when, after my unit completed an on-the-ground field exercise, the leaders would all gather around a map board for a “hot-wash” and replay the exercise to see what we could learn. In the future, we would want to repeat all the good things we did and forget all the things that didn’t work. If we were really fancy, we would use an honest to goodness physical contour map complete with sand to represent the terrain (thus the phrase, “sand table”) and plastic army soldiers to represent the units on the ground. 

Since some network defenders don’t like using the military metaphor in conjunction with infosec, I made the point that hot-washes were no different from when Tom Brady, the recently retired and perhaps most successful NFL quarterback of all time, studied hours of game film each week to prepare for his next contest. I made the case that we, as network defenders, might learn a lot by adopting some version of these sandtable exercises—or if you will, game film reviews—to learn how to improve our own digital defenses. I started by walking us through the infamous Russian compromise of the Democratic National Committee in 2016. In this essay, I'm going to dust off the sandtable and reset it for 2013: the Chinese compromise of the U.S. Government’s Office of Personnel Management (OPM). 

Setting up the sand table: OPM history.

According to the U.S. Congressional Research Service (CRS), the OPM origin story begins with the 1883 Pendleton Act. This act tried to transform the hiring of federal workers from “The Spoils” system to a system of choosing employees based on merit. 

50 years before, President Andrew Jackson (1828 - 1837)—the famous victorious commander against the British in the Battle of New Orleans and the infamous instigator of the Trail of Tears for the Cherokee nation—began the spoils system. He operationalized, at the federal level, the famous saying, “to the victor belong the spoils.” He fired about 10% of the federal workforce and replaced them with people who supported him in his presidential campaign. From then on, each incoming president would do the same, dismissing thousands of government workers and replacing them with members from their own party. 

Fast forward through forty odd years of “The Spoils” system to July 2, 1881, when a mentally disturbed man, Charles J. Guiteau, who thought he deserved a federal job because of his qualifications but didn’t get it, assassinated President James Garfield at the Baltimore & Potomac Railroad Station in Washington, D.C. According to my editor, John Petrick, “Even local postmasters were Presidential appointees. Mr. Guiteau's nose was out of joint because he didn't land what today would probably be described as a GS-11 job.” When Garfield finally died from infection several weeks later, he had only been in office for 200 days. But the public outcry to reform “The Spoils” system grew louder because of it and Congress passed the Pendleton Act in 1883, two years later.

A diversion: “Tod: ” the German word for death.

One side note apropo to nothing about OPM—I just think this is fascinating—is that there have been exactly four U.S. presidential assassinations: 

• President Lincoln (1865)
• President Garfield (1881)
• President McKinley (1901)
• President Kennedy (1963)

The jaw dropping trivia that I guarantee you’ll be sharing at dinner parties from now on is that President Lincoln’s son, Robert Todd Lincoln, was in the general vicinity for three of them:

• Lincoln: He attended his father's deathbed.
• Garfield: While Secretary of War, he witnessed President Garfield's attack in the Washington D.C. train station.
• McKinley: He arrived at the Buffalo train station to attend the Pan-American Exposition, at President McKinley's invitation, and learned of the President’s death just moments before.

Because of all of that, Sarah Vowel, the author who told this story in her book “Assassination Vacation” gave Robert Todd Lincoln the nicknames, “Assassination Cameo three-peat” and “Tod (the German word for death.)” 

And if that wasn’t enough, hold onto your hat, this will frost your hair. You probably know that John Wilkes Booth assassinated President Lincoln. But you probably don’t know that Booth’s brother, Edwin Booth, one of the best Shakespearean actors of the day by the way, saved Robert Todd Lincoln’s life in Jersey City during the Civil War when Todd fell onto a train track in front of an approaching train. How weird is that?

Fun fact--John Wilkes Booth is buried in Balitmore's Greenmount Cemetery, not far from Napoleon's grand nephew Charles Bonaparte, who served as Teddy Roosevelt's Secretary of the Navy and Attorney General.

But I digress.

Setting up the sand table: OPM in the modern day.

According to the CRS, nothing significantly changed for the federal workforce until President Jimmy Carter signed the 1978 Civil Service Reform Act. Among other things, the act created OPM and gave the new organization responsibility for

• Human Capital Management. 
• Benefits. 
• Vetting.

Essentially, OPM became the HR department for federal civilian employees. 

In 1996, OPM contracted the vetting piece of their mission to a commercial company, the US Investigation Services (USIS), in an effort to save costs. In 2009, they added two more contractors (Keypoint and CACI). This is important because the Chinese hackers, most likely the adversary group called Deep Panda, used USIS and Keypoint as a third party digital supply chain attack vector. In other words, Deep Panda compromised the USIS and Keypoint networks first and then used the credentials they stole there to legitimately log into the OPM network. More on that in a bit. But in 2005, OPM took on the additional vetting mission of the U.S military which made it responsible for over ninety percent of all background investigations for the federal government.

The difference between stored commercial HR data compared to what OPM stores is massive in-depth history. A commercial HR department might store your name, address, education and past four or five jobs. OPM stores all of that plus every place you’ve lived and all of your friends and acquaintance names and contact info for the past ten years. They also keep track of all of your family members. Compound that information with any legal trouble you might have had or any subversive or illegal habits (like drug and alcohol use, DWIs, adultery, etc) plus organizational affiliations you had during that time ranging anywhere from your daughter’s travel soccer team to that one time, when you were in college, and donated money to the communist party as a joke (Allegedly).

That’s a lot of information.

In 2007, an investment firm called Providence Equity Partners bought USIS and implemented extreme cost cutting measures to increase profits. USIS began approving clearances without actually doing the required investigations for a large number of cases. At the end of every month, they would “flush” unfinished investigations to meet their profit quota. In other words, they approved candidates without doing the investigation and got paid for each. Unfortunately, OPM didn't discover the fraud for another five years.

In 2011, USIS became the subject of a whistle-blower lawsuit where an insider claimed that USIS used a proprietary computer software program to automatically release OPM background investigations that had not gone through the full review process and thus were not complete.

OPM estimates a total federal workforce of 2.1 million civilian workers. The Department of Veterans Affairs estimates about 19 million U.S. veterans, and the Council on Foreign Relations estimates about about 1.3 million active-duty personnel. OPM stores vetting material for all those groups.

That’s a lot of people.

In terms of espionage and counterespionage operations, any foreign entity that could get their hands on that data set would have a gold mine. They could use it to blackmail federal workers and military personnel to reveal secrets or disrupt important projects or operations. Since the data collected lists all foreign contacts for the past decade, they would have a rich source of potential double agents to track down and neutralize, setting back intelligence collection for years. Not to put too fine a point on it, but the data set will be useful for the next 50 years since it will take that long for the current set of employees and military to age out of the system. In the 2016 report on the OPM breach published by Congress, the title of the report sums up the impact: “The OPM Data Breach: How the Government Jeopardized Our National Security for More than a Generation.”

Readers of these essays know that I’m a big believer in first-principle thinking. As security professionals, our first-principle task is to reduce the probability of material impact due to a cyber event. OPM is a giant bureaucratic U.S. Federal Government organization. Leaders of that institution have many things on their plate. But, OPM leadership had known for years that their internal security posture was garbage. Between 2005 and 2014, OPM had four directors. Their own internal inspector general told them all, year after year, that the data they hosted was extremely sensitive and valuable and that the security measures they had in place had material weakness, significant deficiencies and concerns, and was getting worse. None of the four directors thought that the data they stored was material enough to their organization, and indeed to the entire U.S. government, to actually spend resources to improve the situation. Before the breach, they didn’t even know where the copies of all the data were and they didn’t have a security team.

Setting up the sand table: The Chinese.

The origins of the Chinese cyber attack capability can be traced to a white paper (later turned into a book) published by two People's Liberation Army (PLA) colonels, Qiao Liang and Wang Xiangsui, in 1999. They developed their thesis after watching the U.S. military’s utter dominance of the Iraqi army in the 1991 Persian Gulf War. They concluded that any nation, but especially China, going toe-to-toe with the U.S. in a conventional military fight could only result in, at best, a standoff with massive damage on both sides and, at worst, utter destruction to the American opponent. They proposed instead something called unrestricted warfare. In an interview about the book, COL Qiao said, "the first rule of unrestricted warfare is that there are no rules, with nothing forbidden." 

And if that sounds eerily similar to the line from the famous movie “Fight Club” released in 1999, it’s not. I thought so too but I checked. The line from the movie is:

“The first rule of fight club is, you do not talk about fight club.”

Qiao and Wang had a different view altogether. Unrestricted warfare means that you don’t fight based on the rules set by the superior opponent. Tyler Tidwell, in reviewing the book, says that the two authors proposed to radically expand the conventional military battle space into financial markets, television, cyberspace, and outer space. In fact, physical tank-on-tank violence is the last refuge. You never want to get there.

Interestingly, the Chinese “Unrestricted Warfare” doctrine is pretty close to the Russian Gerasimov doctrine established in 2014, sixteen years later. Policy wonks will argue with me on whether these ideas constitute an actual national doctrine. I'm confident I would lose those debates. But, the one thing I'm sure of is that the big five cyber nations (China, Russia, North Korea, Iran, and the U.S.), and lots of other nations who are dabbing their feet into the cyber pool (India, Pakistan, Palestine, Israel, and Vietnam) have determined in the last two decades that they can get a whole lot more done in terms of national objectives by pursuing, as David Sanger says in his book “The Perfect Weapon,” a continuous low level cyber conflict. It’s low level because it rarely crosses the line that might start an actual tank-on-tank battle but the effects can be devastating. And China was one of the first nations to jump into the cyber pool and have a major impact.

My editor, John Petrik, will point out that “Unrestricted Warfare” is really an extension of asymmetric warfare and he would be right. According to John,  the granddaddy of asymmetric warfare was the French naval theory of the Jeune Ecole, who thought the inexpensive torpedo boat rendered the battleship obsolete. Britanica online defines asymmetric warfare as “unconventional strategies and tactics adopted by a force when the military capabilities of belligerent powers are not simply unequal but are so significantly different that they cannot make the same sorts of attacks on each other.” This is Guerrilla warfare and weaker forces have used the strategy against stronger forces since the sixth century. In more modern times, guerrilla warfare defeated the U.S. in Vietnam and in Afghanistan. The genius of Qiao and Wang though is extending the idea to cyberspace. 

In 2003, U.S. military network defenders discovered that Chinese cyber operators had inserted themselves into large swatches of military networks around the world. I was the commander of the U.S. Army Computer Emergency Response Team (ACERT) back then and military leadership classified that activity with the cool code name “TITAN RAIN.” Before “TITAN RAIN,” the most serious hacker I tracked at the ACERT was a British citizen, Gary McKinnon, who was confident that if he poked around enough military networks, he was sure to find the evidence that aliens existed. We were all secretly hoping that he’d be successful. After “TITAN RAIN,” all the military cyber defenders started operating at a different level. Instead of defeating low-level cyber criminals, we were now laser-focused on nation-state cyber espionage activity.

By 2006, the general public started to learn that the U.S. wasn’t the only target. Chinese cyber operators compromised Taiwan’s Ministry of National Defense (MND) and the American Institute in Taiwan (AIT). 

In 2007, the Chinese found their way into the U.S. Office of the Secretary of Defense (OSD) and German government entities that included the Federal Chancellery, the Ministry of Economics and Technology and the Federal Ministry for Education and Research. At the same time, the public started to learn that the Chinese weren’t just interested in government secrets. They were after business intelligence too. That same year, the British domestic intelligence service, MI5, alerted 300 business leaders warning that the PLA targets confidential business information. A decade later, in 2015, the former National Security Agency Director and Commander of the U.S. Cyber Command, Keith Alexander—my senior rater when I was at the ACERT by the way—told Congress, “It’s intellectual property, it’s our future. I think it’s the greatest transfer of wealth in history.”

Around 2008 (probably as early as 2006), the Chinese penetrated the Lockheed Martin classified network and stole the plans related to the American F-35 fighter jet. That same year, they compromised the election campaigns of both Senator Obama and Senator McCain as well as the White House information system and NASA’s Kennedy Space Center and Goddard Space Flight Center.

A Canadian research team in 2009 published intelligence on the GhostNet cyber espionage campaign that targeted government embassies from Germany, the Philippines, India, Pakistan, Portugal, and the Tibetan Government in Exile. 

And then, in early 2010. Google sent out shockwaves when it announced that it had been hacked by the Chinese government. When all was said and done, two different Chinese government entities in separate and uncoordinated missions had established a persistent presence within the Google networks: the PLA (who stole intellectual property, specifically source code from tech companies) and the Ministry of State Security or MSS (who targeted political dissidents like the Dalai Lama, Uighur and Tibetan ethnic minorities). 

The announcement was a shockwave because, before, no commercial company would ever admit in public that it had been compromised by some cyber adversary for fear of risking its reputation on the stock market. Google opened the door for everybody. Today, nobody even blinks twice when we hear about another cyber breach.

In 2011, the Chinese government stole the key material that was the essential secret sauce used in the RSA Secure Token two-factor authentication product and later used that intelligence to compromise Lockheed Martin. And the irony isn’t lost on me that the famous Beltway bandit U.S. contracting company, Lockheed Martin, the company responsible for arguably the greatest innovation in cybersecurity strategy (inventing the intrusion kill chain prevention model in 2010), was the victim of two major Chinese cyber espionage campaigns around the same time: the F-35 fighter jet espionage operation and the compromise of the RSA Secure Token two-factor authentication product.

All of this activity, from about 2003 to 2012, sets the stage for what can arguably be classified as the most impactful cyber espionage campaign that is known to the public: The OPM breach of 2013. 

Turn one: Deep Panda, the red team, July 2012 - March 2014.

Using the Lockheed Martin intrusion kill chain model as a guide, it’s unclear how Deep Panda gained initial access to the OPM, USIS, and KeyPoint networks. With all the analysis, after the fact, from OPM’s IT team, two security vendors (Cylance and Cytech), and a Congressional committee, how Deep Panda conducted early recon and exploited the first victim machines is unknown. This is mostly because OPM wasn’t logging anything useful in terms of threat intelligence. 

What is known is that another Chinese adversary group (commonly referred to as Axiom but also attributed to China’s 2nd Bureau of the People’s Liberation army - Unit 61398) managed to sneak a piece of malware (Hikit) onto the OPM network as early as July 2012, but probably much sooner. The U.S. CERT noticed it was beaconing to a Chinese-owned command-and-control server. Axiom and Deep Panda are not the same adversary group if you compare the steps each takes across the intrusion kill chain; but it's not a big stretch of the imagination to speculate that the Chinese would use one group to establish initial access and another group to perform lateral movement and exfiltration operations. Today’s cyber criminals do it all the time.

Probably sometime in 2013, Deep Panda established a command-and-control server called opmsecurity.org. The owner of the domain was listed as Steve Rogers. Marvel Cinematic Universe fans will recognize the name as the alter ego of Captain America thus proving again that hackers are science fiction and fantasy fans, too. Recall that the code developed by the Russians in the notPetya attacks against Ukraine was riddled with references from a famous and beloved science fiction book entitled “Dune” written by Frank Herbert back in 1965. Andy Greenberg’s book on notPetya is even called “Sandworm.” 

By April 2013, Deep Panda had established a beachhead on the USIS network. They stole legitimate credentials from USIS employees and used them to log onto the OPM network. Once there, they began reconning for useful information. By November, they had exfiltrated manuals of IT system architecture giving them the blueprints of how to navigate the OPM networks. By December, Deep Panda had exploited the first victim machine on the Keypoint network.

Turn one: OPM, the blue team, January 2014 - April 2014.

In January, on the backend of the 2011 whistleblower complaint, the U.S. Justice Department sued USIS in a 25-page complaint filed in United States District Court in Montgomery, Alabama, claiming that, from 2008 to 2012, about 40 percent of the company’s investigations were fraudulently submitted. Newspaper headlines highlighted the many fraudulent cases, but two were particularly memorable. USIS approved the background checks for the government insider, Edward Snowden, who leaked classified documents to the public in June 2013 and for the mentaly unstable Navy Yard shooter, who killed 12 people in Washington D.C. in September 2013. 

Through no fault of their own, the OPM IT team had not a single security prevention tool deployed on their networks. They tried to convince their leadership to do something over the years, but OPM directors rejected all requests. Instead, leadership relied on outside entities to monitor their networks for them, like the U.S. CERT and the U.S. government-developed Einstein Intrusion Detection System. Throughout the entire attack, the Einstein system didn’t detect any Deep Panda activity at all. 

On 20 March, the U.S. CERT notified OPM again of more data exfiltration (the last time was July 2012). OPM investigators determined that since the stolen data didn’t contain PII (Personal Identifiable Information) and that the hacker was confined to a certain part of the network, that OPM leadership didn’t have to go public with the incident. They decided that the best course was to monitor the threat in order to gain counterintelligence and to start planning what they called the "big bang"—a system reset that would purge the attackers from the system. The OPM CIO, Donna Seymour, approved the plan five days later.

Let me just point out here that an IT team, with no security experience, no security tools deployed, and no logging telemetry to speak of, decided to “monitor” the hacker to gain intelligence. Incredible.

On 21 April, an OPM contractor from SRA found another piece of malware that communicated with the command-and-control server, opmsecurity.org, that Deep Panda had established in 2013. 

Turn two: Deep Panda, the red team, May 2014 - March 2015.

On 7 May, using credentials stolen from the KeyPoint network, Deep Panda legitimately logged into the OPM network and installed a Remote Access Trojan (RAT), called PlugX, on roughly ten machines for command and control. PlugX had been around as far back as 2008 and has been used by several Chinese adversary groups: APT1, APT3, APT27, APT41, DragonOK, GALLIUM, Mustang Panda, and TA459.

By 27 May, twenty days later, Deep Panda began installing keylogger software on database administrators’ workstations, and on 5 June, they installed malware on a Keypoint web server. 

On 20 June, they began extended remote sessions with OPM servers containing sensitive data, and they probably gained access to the OPM mainframe on 23 June.

By July, Deep Panda had discovered the OPM Jump Server; a kind of toll booth that stands between day-to-day networking resources and sensitive data resources (in this case, the OPM background check data on all federal employees and military personnel.) If you want access to the crown jewels, you have to go through the jump server. Deep Panda installed a PlugX variant on the OPM jump server.

According to the Congressional report, “During the long Fourth of July weekend, when staffing was sure to be light, the hackers began to run a series of commands meant to prepare data for exfiltration.” Deep Panda collected batches of personnel information, wrapped each in .zip or .rar files, and stored them on hard drives for later exfiltration. 

On 21 July, OPM Director Katherine Archuleta downplayed the 20 March breach in an ABC interview: “We did not have a breach in security. There was no information that was lost. We were confident as we worked through this that we would be able to protect the data.”

At the end of July, Deep Panda established another command and control server, opmlearning.org, this time registered to Tony Stark, the alter ego of Iron Man in the Marvel Cinematic Universe.

On 16 August, the malware installed by Deep Panda on the Keypoint web server on 5 June 2014 stopped functioning. But, by the end of August, Deep Panda had exfiltrated all of the data they had collected in .zip and .rar files; detailed information on some 20 million federal civilian employees and military personnel. By 20 October, Deep Panda used OPM credentials to bridge over to the U.S. Department of the Interior, which held another 4.2 million OPM records.

Deep Panda’s mission almost complete at OPM, they turned their attention to Anthem, a U.S. based insurance company. They already gained access to the Anthem network sometime in April 2014. In December, the group exfiltrated some 80 million customer records. In February 2015, a commercial intelligence firm, ThreatConnect, discovered that Deep Panda used the same Tony Stark registered command-and-control server that they used in the OPM attacks.

The last nail in the coffin, on 26 March 2015, Deep Panda exfiltrated just over 5.5 million fingerprint records of federal employees. 

Turn two: OPM, the blue team, May 2014 - December 2014.

On 27 May 2014, OPM executed it’s Big Bang strategy and thought they were successful. Unfortunately, they didn’t know about Deep Panda’s compromise of Keypoint and how they stole OPM credentials. OPM technicians had no visibility of Deep Panda’s keylogger installs onto OPM systems, remote sessions with OPM sensitive servers, access to the OPM mainframe, and the collection and storage of background investigation data.

On 10 June, OPM CIO Donna Seymour testified before the Senate Homeland Security and Government Affairs Subcommittee on her strategic information technology plan. She didn’t mention any of this because she didn’t know about it. But, she also didn't mention Deep Panda’s activity from March through May, which she did know about.

On 12 June, the OPM tech team deployed an evaluation copy of one of Cylance’s security products; OPM’s first security detection tool. They hadn’t purchased it yet. The product was just a feature-reduced demo.

On 22 June, the Department of Homeland Security (DHS) released an incident report for OPM’s first breach discovered on 20 March. In an interview with the New York Times on 9 July, an OPM spokesperson acknowledged the March 2014 breach but emphasized that they lost no PII; just manuals and technical documents. This is absolutely true because, as I said, they didn’t know about the Keypoint compromise yet. They did neglect to say that some of the technical documents were blueprints of the OPM networking architecture.

In August, USIS notified OPM regarding their breach back in April 2013, over a year after the fact. Personal data of some 25,000 government employees was probably compromised in this operational phase. OPM responded by issuing a stop work order on 6 August until everything could be sorted out. 

The USIS delay in notifying OPM about their breach coupled with the Justice Department’s whistleblower lawsuit on fraudulent work practice convinced OPM to decline to renew the USIS contract in September. But the damage had been done.

That left CACI and KeyPoint as the remaining contractors conducting background investigations. Later that year, in December, KeyPoint announced that it had been breached to the tune of 4.2 million records exfiltrated.

Turn three: OPM, the blue team, Spring 2015 - June 2015.

Sometime in the spring of 2015, OPM discovered evidence of the remote sessions between Deep Panda and OPM’s sensitive servers, over a year after the sessions happened. On 9 March, they thought they shut down communication between the OPM networks and the Deep Panda command-and-control server (opmseccurity.org registered to Steve Rogers). But, on 15 April, OPM’s investigators discovered more Deep Panda command-and-control beaconing to opm­security.org with the Cylance evaluation product. As they notified the U.S. CERT about the traffic, they realized that Deep Panda still had a foothold in their system.

The next day (16 April), almost a year to the day since Deep Panda’s first initial access, OPM leadership assigned a person, Curtis Mejeur, to eradicate the group out of the OPM network. His first move was to ask for Cylance’s help using the demo product to diagnose forensic images of OPM servers. The demo product wasn’t built for that. OPM needed a tool with more capabilities. Cylance agreed to let OPM use their upgraded product, Cylance Protect, in a free trial mode on 2000 devices and according to the Congressional Report, the product "‘lit up like a Christmas tree’ with widespread infections.”

On 19 April, a Cylance technician discovered a rare Deep Panda mistake. When Deep Panda finished the exfiltration of all the data back in December, they forgot to delete at least one .rar file. That gave the Cylance technician the lead to discover the massive data exfiltration. In an email to his CEO, Stuart McClure, explaining the discovery, he said, “They are F%&Ked btw.”

According to the Congressional Report, by 21 April, besides all the Deep Panda evidence discovered by the Cylance Protect, it also discovered some “2,000 individual pieces of malware that were unrelated to the attack in question (everything from routine adware to dormant viruses).” More importantly, the product also discovered the PlugX variant. It was only present on about 10 OPM machines but they were key machines including the jump server. 

On the same day, 21 April, another commercial vendor, CyTech, arrived at OPM for a long-scheduled appointment to demonstrate their CyFIR product. According to the Congressional Report, “The breach was not public knowledge at this point, and OPM staff did not share any information about it with company founder Ben Cotton, who was there to lead the demo. CyFIR also detected the malware, and Cotton immediately agreed to help with the response.” According to CyTech, at the end of the day, OPM owed them over $800,000 but because no contract was in place, CyTech was never paid.

On 23 April, OPM finally decided that it had a major breach on its hands involving the loss of PII. That triggered a requirement to notify Congress. The next day, 24 April, in conjunction with a scheduled power outage as part of a Washington D.C. power grid modernization program, OPM eradicated all of the malware that had been previously discovered.

By 26 April, Cylance engineers identified the command-and-control sessions from important and sensitive servers in June 2014 which triggered another notification to Congress. They discovered another large-scale exfiltration in May and that triggered another requirement to notify Congress.

Finally, 4 June, OPM briefed the media about the breach.

The Committee on Oversight and Government Reform in the U.S. House of Representatives held the first of two hearings on 16 June. On 19 June, the commercial security vendor, FireEye, attributed Deep Panda (first noticed by another security vendor Crowdstrike) as the adversary Group that conducted attacks against OPM, at least they used some of the same tactics.

OPM Director Katherine Archuleta repeatedly told the House Oversight and Government Reform Committee that she couldn’t say if any PII was lost in the 2014 hack but didn’t mention any of the most recent developments. According to the National Review, “Her answers under oath in front of the Oversight Committee … left Republicans and even some Democrats convinced she either knows exceptionally little about the state of her agency’s cyber-security or she’s comfortable lying about it, insisting that breaches aren’t really breaches and that obviously insecure systems are secure.”

OPM CIO Donna Seymour, in her testimony, only mentioned the initial exfiltration of tech manuals in March 2014 and nothing about the latest discoveries. She also lied under oath about the OPM response. She told the committee that OPM had purchased the CyFir tool and that they were running it in a quarantine environment (they were running it on the production network). She also testified that the manuals stolen in March 2014 were merely "outdated security documents” when in fact they were mostly current security architecture documents.

At the end of the month (30 June) OPM finally purchased the Cylance Protect product, a day before the trial period was set to expire (They were in trial mode the entire time) . Cylance didn’t actually receive payment for months. OPM deployed the Cylance Protect on 10,250 devices and found nearly one piece of malware for every five devices.

Impact.

On 9 July 2015, OPM issued a press release confirming 21.5 million records were compromised. The very next day, Director Archuleta resigned. In February 2016, Donna Seymour resigned.

According to the Washington Post’s Ellen Nakashima, “The vast majority of those affected [by the Deep Panda attack]— 21.5 million people — were included in an OPM repository of security clearance files … At least 4.2 million people were affected by the breach of a separate database containing personnel records including Social Security numbers, job assignments and performance evaluations.”

James Comey, the FBI Director at the time said, “It is a very big deal from a national security perspective and from a counterintelligence perspective. It’s a treasure trove of information about everybody who has worked for, tried to work for, or works for the United States government.” 

The NSA Senior Counsel, Joel Brenner, said, “This is crown jewels material .. . a gold mine for a foreign intelligence service. This is not the end of American human intelligence, but it's a significant blow."

And finally, a former Director of the CIA, Michael Hayden, said, “[OPM data] remains a treasure trove of information that is available to the Chinese until the people represented by the information age off. There's no fixing it.” 

The Deep Panda compromise of OPM’s clearance information, coupled with the Anthem attacks that took place immediately after, might be the largest and longest lasting one-two cyber espionage punch known to the public against any known country. The vast amounts of data collected plus the longevity of it (over 50 years since that’s how long it will take for all individuals caught in the net to age out of government service) will be useful for many years to come.

Hotwash: Things that OPM could have done.

It’s easy to Monday-morning-quarterback what the OPM leadership should have done over the years to prevent the success of the Deep Panda espionage operation. Before I pile on, I just want to say that OPM’s leadership—Director Katherine Archuleta, CIO Donna Seymour, and many others—made a risk assessment. They looked at what their own inspector general told them about the state of OPM’s security posture and made a call. They decided that the risk was acceptable compared to all the other risks that they were dealing with. This is what leaders do. They make risk calls. It’s why we pay them the big bucks. In this case, they got it wrong, but I understand the thought process.

I have said many times in these essays that in my younger days, when the leadership didn’t approve some grand plan of mine, I blamed the leadership for being naive and uninformed or incapable of understanding the complexity of my security world. In hindsight, that was just immaturity and hubris talking. What really happened is that I didn’t convince them that the risk I was talking about was more important than some of those other business risks that they thought had a higher priority. When I think back to those discussions, I have to admit that at least some of those times, the business leaders were right. Back then, I didn’t have the tools to communicate, with any authority or precision, what the exact risk to the business was. This is why in these essays, when I talk about cybersecurity first principles, risk forecasting as a strategy is as important as zero trust, intrusion kill chain prevention, and resilience. So, let’s first talk about forecasting risk for OPM prior to the first Deep Panda breach in March of 2014.

This one’s pretty easy. OPM had no security team and no prevention or detection tools in place before the breach. There is no way that they could have noticed activity from any known adversary groups across the intrusion kill chain at the time, let alone the Chinese. They had nobody watching. And even if they did, they weren’t collecting any telemetry from their endpoints and networking nodes. And even if they were, they had no tools in place to detect cyber bad guy behavior across the kill chain. After all the public intelligence about Chinese cyber operations from the early 2000s to the OPM breach, OPM leadership had refused all upgrades. They were essentially blind. When they finally installed their first tool after the fact, the Cylance product, as the Congressional report said, “It lit up like a Christmas tree.”

In terms of zero trust, they were architecturally in a better position. The OPM IT team had placed a jump server between the day-to-day OPM operations and the crown jewel data for employee and military background checks. That’s the good news. The bad news is that they weren’t watching it. They didn’t know that Deep Panda installed software on it (PlugX) and that somebody was storing .zip and .rar files somewhere on the sensitive network. They definitely didn’t notice 21 million records going through the jump server and out to the back door to the internet. 

Even if all of that were rock solid, OPM had no zero trust controls in place for their contractors (USIS, KeyPoint, and CACI) to restrict their permissions. In hindsight, they should be allowed to write to the OPM database but not read any records that they didn’t create. If they did need to read other records or change records once submitted, some secondary OPM control should have been in place to check for legitimacy. The contractors definitely shouldn’t have been allowed to copy data out of the OPM network and they also shouldn’t have been allowed to store data in their own networks for any length of time.

For resilience, OPM had no incident response plan, no crisis plan at all. From reading through the congressional record, you get the sense that they were making it up as they were going. Their decision to “monitor” the adversary when they first noticed them and not to eradicate them (when they had no real way to do that) was a huge blunder. If they had reacted immediately, they would have had an opportunity window where they might have prevented the exfiltration of 20 million records. The first step should have been to call in the FBI. OPM had no resources to reject a nation-state threat like this, but the FBI did. Before the OPM breach, after years and years of negative cybersecurity report cards from her own IG, Director Archuleta should have at least had the FBI on speed dial. In response to the sensitive data she was protecting, she should have collaborated with the FBI on the crisis plan before anything happened.

Deep Panda’s intent was not to destroy. They were stealing. A robust backup plan wouldn’t have helped here. That said, OPM didn’t have one. In fact, when the Deep Panda attacks started, OPM technicians weren’t even sure where all the copies of the data resided on the OPM networks and in their contractor networks. In terms of encryption, it was non-existent. How OPM decided to store this kind of sensitive data in a non-encrypted form is a mystery and a major failure in resiliency and zero trust planning. If I were doing this analysis before March 2013 and before Deep Panda had any success, I would have told Director Archuleta that OPM was a prime candidate for a ransomware attack, let alone a target for a major cyber espionage operation.

Before the breach, if I were providing a risk forecast to Director Archuleta, I would have told her that there is a 100% chance that some bad guy would penetrate the OPM networks in the next three years. But as I have said many times in these essays, not all hacks are material to the business. In fact most of them aren’t. In terms of OPM, the material data—the data that if stolen, destroyed, or manipulated would have a major impact on not just OPM but on the entire U.S. government— is the background check data that OPM was responsible for. From the congressional hearings on the matter, it was clear that Director Archuleta didn’t know that. In my risk forecast to her before the breach, I would have told her that there is a 95% chance that some foreign actor (probably Russia, China, North Korea, or Iran) would gain access to that material data in the next three years. The only thing stopping them is their own bureaucracy and priorities. Nothing OPM is doing will deter them in the slightest.

The cybersecurity sandbox.

As I said, it’s easy to Monday-morning-quarterback massive failures in cybersecurity prevention. The OPM case is like shooting fish in a barrel. And admittedly, I'm still a bit angry that this happened. I'm an old military retiree at this point and my records got scooped up with the rest of the 20 million records. The credit check service offered by OPM to all impacted personnel after the incident as an appeasement doesn’t seem quite adequate for the enormity of the failure. So, if you noticed a little harshness in my presentation of this material, I'm not going to deny it.

But, for all network defenders, during the heat of the battle, it’s tough to take a beat and reflect on what could be done better next time. This is why I'm advocating for cybersecurity sand table exercises to become a staple for network defender best practice. When there isn’t a crisis afoot and you can take a few moments to analyze what happened on both sides, you can learn quite a bit. Just like I did while I was still serving in the active Army participating in hot washes after a field exercise, and just like Tom Brady did to prepare for every future football game. Replaying the exercise or watching game film can solidify in your mind what needs to be done before the next crisis.

References.

“After 40 Years, a Look Back at the Unlikely Passage of Civil Service Reform,” by Charles Clark, Government Executive, 3 July 2018. 

“Anthem Hack,” BY: RASHA ALTAMIMI, NIHARIKA ARORA, and AMAL KADI, Boston University, 

“A Short History of the Office of Personnel Management (OPM).” by Benefits Ben, Federal Agency News, 22 April 2021.

"Assassination Vacation,” by Sarah Vowell, Published by Simon Schuster, 29 March 2005.

“Asymmetrical Warfare,” Encyclopædia Britannica, 2022. 

“AUDIT OF THE QUALITY ASSURANCE PROCESS OVER BACKGROUND INVESTIGATIONS (Report No. 4A-IS-OO-09-060),” by OPM’s OFFICE OF THE INSPECTOR GENERAL, OFFICE OF AUDITS, 22 June 22, 2010.

“Background Investigations Moving from OPM to DoD, What Comes next Will Make or Break Them,” by Jeff Neal, Federal News Network, 2 July 2019. 

"Capability of the People’s Republic of China to Conduct Cyber Warfare and Computer Network Exploitation," by Bryan Krekel, George Bakos, Christopher Barnett, Northrop Grumman Corporation Information Systems Sector, The US-China Economic and Security Review Commission, October 2009.

“Completing your Investigation: Request in e‐QIP: Guide for the Standard Form (SF) 86,” by OPM, July 2018.

“Computer Spies Breach Fighter-Jet Project.” Siobhan Gorman, August Cole, and Yochi Dreazen, The Wall Street Journal, 21 April 2009. 

“DEF CON 25 Packet Hacking Village -Demystifying the OPM Breach, WTF Really Happened,” By Ron Taylor, DEFCON, YouTube, 6 November 2017. 

“Demographics of the U.S. Military,” Council on Foreign Relations, 2020. 

“Ex-NSA Head: Chinese Hacking Is ‘the Greatest Transfer of Wealth in History’,” by Giuseppe Macri, InsideSources, 4 November 2015. 

“FACT SHEET: Cybersecurity National Action Plan.” Whitehouse, 9 February 9 2016..

“Federal Workforce Statistics Sources: OPM and OMB (R43590),” Congressional Research Service, 24 June 2021.

“FireEye Identifies Chinese Group behind Federal Hack,” by Arik Hesseldahl, Vox, 19 June 2015. 

‌“Four Charts That Reveal Tom Brady’s Greatness,” by Josh Katz, Kevin Quealy and Alanis Thames, The New York Times, 2 February 2022.

“Gary McKinnon Extradition to US Blocked by Theresa May,” BBC News, 16 October 2012. 

"Google Aurora Hack Was Chinese Counterespionage Operation," by Mathew J. Schwartz, Information Week Security, 21 May 2013.

“Hackers Took Fingerprints of 5.6 Million U.S. Workers, Government Says,” by David E. Sanger, The New York Times, 23 September 2015.

"Hackers Who Breached Google in 2010 Accessed Company’s Surveillance Database," BY KIM ZETTER, Wired, 20 May 2013.

“Hacks of OPM Databases Compromised 22.1 Million People, Federal Authorities Say,” by Ellen Nakashima, The Washington Post, 10 July 2015.

“How China Stole the Designs for the F-35 Stealth Fighter,” by Eli Fuhrman, 19FortyFive, 15 July 2021. 

“How Government Contractors Harm the Public in Pursuit of Profit,” In the Public Interest (ITPI), April 2016.

"INSIDE THE CYBERATTACK THAT SHOCKED THE US GOVERNMENT," by BRENDAN I. KOERNER, Wired Magazine, 23 October 2016.

"Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill Chains,” by Eric Hutchins, Michael Cloppert, Rohan Amin, Lockheed Martin Corporation, 2010. 

“Life After the OPM Hack,” by Peter W Singer and Sara Sorcher, The Cybersecurity Podcast.

“Man Who Sold F-35 Secrets to China Pleads Guilty” By Justin Ling, Vice.com, 24 March 2016.

“Number of Governmental Employees in the U.S. 2020,” by Statista, 2020.

“OPM Prepares for Transfer of the Department of Defense’s Background Investigations Services Division,” by Brendan LaCivita, U.S. Office of Personnel Management, 2019. 

“Pendleton Civil Service Act: An Effort to End the Spoils System,” by the Smithsonian National Postal Museum, 2022.

“RECOGNIZING AND ADAPTING TO UNRESTRICTED WARFARE PRACTICES BY CHINA,” by COL Bryan K. Luke, Air War College, 15 February 2012.

"Report: Breach Hit 25,000 Govt. Workers Background Check Firm Breach Exposed U.S. Agency Staff Info," by Jeffrey Roman, Data Breach Today, 25 August 2014.

“Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin's Most Dangerous Hackers,”by Andy Greenberg, Published by Doubleday, 2019. 

“Second OPM Contractor Breached: Records of 48,439 Federal Employees, Contractors Exposed," by Eric Chabrow, GovInfoSecurity), 19 December 2014.

“Security Check Firm Said to Have Defrauded U.S.,” By Matt Apuzzo, The New York Times, 23 January 2014.

“Special Report: In Cyberspy vs. Cyberspy, China Has the Edge,” by Brian Grow and Mark Hosenball, Reuters, 14 April 2011. 

“The Changing Face of America’s Veteran Population,” by Katherine Schaeffer, Pew Research Center, 5 April 2021. 

“‘The Gerasimov Doctrine’ - Berlin Policy Journal - Blog.” Berlin Policy Journal - Blog, April 28, 2020./

“The Office of Personnel Management (OPM): An Overview,” by Taylor N. Riccard, the Congressional Research Service, 20 July 2021. 

"The OPM Data Breach: How the Government Jeopardized Our National Security for More than a Generation," by Committee on Oversight and Government Reform U.S. House of Representatives, 114th Congress, 7 September 2016.

"The OPM Data Breach: Lessons Learned," by Michael Wheat, Undergraduate Honors CapstoneProjects, 589, Utah State University, 2016.

“The OPM Hack and Obama’s Politicization of the Federal Bureacracy,” by Jum Geraghty, National Review, 29 June 2015. 

”The Pendleton Act (1883),” by Digital History, ID 1098, Uh.edu.

“The Perfect Weapon: How the Cyber Arms Race Set the World Afire,” by David E. Sanger, Published by Crown, 19 June 2018. 

“Timeline of the Persian Gulf War,” In Encyclopædia Britannica, 2022. 

"This Is How They Tell Me the World Ends: The Cyberweapons Arms Race," by Nicole Perlroth, Read By, Allyson Ryan, Published by Bloomsbury Publishing, 9 February 2021 

“Unrestricted Warfare : China’s Master Plan to Destroy America,” by Qiao Liang and Wang Xiangsui, Published by Pan American Publishing Company, February 1999.

"U.S. charges Chinese spies and their recruited hackers in conspiracy to steal trade secrets," By Ellen Nakashima, The Washington Post, 30 October 2018.

“US Government Intervenes in False Claims Lawsuit against United States Investigations Services for Failing to Perform Required Quality Reviews of Background Investigations.” Justice.gov, 30 October 2013. 

“USIS That Vetted Snowden under Investigation; Booz Allen Hamilton Overlooked Snowden Resume Discrepancies,” by Amrutha Gayathri, International Business Times, 21 June 2013.

“Why the OPM Breach Is Such a Security and Privacy Debacle,” by Kim Zetter and Andy Greenberg. Wired, 12 June 2015. 

“Why the OPM Hack Is Far Worse than You Imagine” by Michael Adams, Lawfare, 11 March 2016.

CSO Perspectives episodes that pertain

11 MAY 2020:

CSOP S1E6:: Cybersecurity First Principles.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay

18 MAY 2020

CSOP S1E7:: Cybersecurity first principles: zero trust.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay

26 MAY 2020:

CSOP S1E8:: Cybersecurity first principles: intrusion kill chains.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay

01 JUN 2020:

CSOP S1E9:: Cybersecurity first principles - resilience.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay

15 JUN 2020:

CSOP S1E11:: Cybersecurity first principles - risk.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay

03 AUG 2020:

CSOP S2E3: Incident response: a first principle idea.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay

10 AUG 2020:

CSOP S2E4: Incident response: around the Hash Table. 

• Hash Table Guests:
• Jerry Archer - Sallie Mae CSO
• Ted Wagner - SAP National Security Services CISO
• Steve Winterfeld - Akamai Advisory CISO
• Rick Doten - Carolina Complete Health CISO
• Link: Podcast
• Link: Transcript
• No Essay

31 AUG 2020:

CSOP S2E7:: Identity Management: a first principle idea.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay

07 SEP 2020:

CSOP S2E8: Identity Management: around the Hash Table.

• Hash Table Guests:
• Helen Patton - CISO - Ohio State University
• Suzie Smibert - CISO - Finning
• Rick Doten - CISO - Carolina Complete Health
• Link: Podcast
• Link: Transcript
• No Essay

14 SEP 2020:

CSOP S2E9: Red team, blue team operations: a first principle idea.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay

21 SEP 2020:

CSOP S2E10: Red team blue team operations: around the Hash Table.

• Hash Table Guests:
• Tom Quinn: CISO - T. Rowe Price
• Rick Doten: CISO - Carolina Complete Health
• Link: Podcast
• Link: Transcript
• No Essay

16 MAY 2021

CWX: Zeroing in on zero trust.

• Guests:
• John Kindervag, Cybersecurity Strategy Group Fellow at ON2IT 
• Tom Clavel, Global marketing director at ExtraHop (sponsor)
• Link: Podcast
• Link: Transcript
• No Essay

17 MAY 2021

CSOP S5E5: New CISO Responsibilities: Identity.

• Hash Table Guests:
• Jerry Archer, Sallie Mae's CSO
• Greg Notch, the National Hockey League's CISO
• Link: Podcast
• Link: Transcript
• Essay: None

30 AUG 2021

CSOP S6E7: Pt 1 - Cybersecurity first principles - adversary playbooks.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay and Podcast

13 SEP 2021

CSOP S2E8: Pt 2 - Cybersecurity first principles - adversary playbooks.

• Hash Table Guests: None
• Ryan Olson, the Palo Alto Networks (Unit 42) Threat Intelligence VP
• Link: Podcast
• Link: Transcript
• No Essay

01 NOV 2021:

CSOP S7E3: Introducing the “Sandtable Series:” the DNC Compromise.

• Hash Table Guests: None
• Link: Podcast
• Link: Transcript
• Link: Essay

OPM Timeline & Notes

1996

USIS established as a result of the privatization of the investigative branch of OPM.

2009

OPM's Federal Investigative Services (FIS) contracted with three background investigative contractors: US Investigations Services (USIS), KeyPoint Government Solutions (KGS), formerly Kroll, and CACI International (CACI).

2011

USIS became the subject of a whistle-blower lawsuit claiming that USIS used a proprietary computer software program to automatically release to OPM background investigations that had not gone through the full review process and thus were not complete.

2012

May

Members of the hacktivist Anonymous (AKA @k0detec) stole 37 USER ID / Password records from OPM.

July

US CERT discovers Command and control activity from OPM via malware called Hikit, associated with a Chinese cyber adversary group (the 2nd Bureau of the People’s Liberation army - Unit 61398) commonly referred as Axiom.

2013

April

Deep Panda Initially compromise of USIS.

November

Deep Panda, called X1 by the Congressional OPM data breach report, exfiltrated manuals and IT system architecture information.

December

Deep Panda compromises KeyPoint.

2014

The OPM IT team pushed leadership to purchase Cylance's Protect, a higher-end product from Cylance. OPM IT rejected it because of office politics.

23 January

The U.S. Justice Department sues USIS in a 25-page complaint filed in United States District Court in Montgomery, Ala., claiming that, from 2008 to 2012, about 40 percent of the company’s investigations were fraudulently submitted.

20 March

US-CERT notifies OPM of data exfiltration.

OPM doesn't publicize the breach.

OPM Investigators thought that Deep Panda was confined to a part of the network that didn't have any personnel data.

OPM officials chose to allow the attackers to remain so they could monitor them and gain counterintelligence.

OPM started planning for what they called the "big bang"—a system reset that would purge the attackers from the system.

25 March

OPM CIO Donna Seymour briefed.

April

Deep Panda (presumably) gained initial access to Anthem.

21 April

OPM contractor (SRA) discovers malware.

25 April

opmsecurity.org registered, some sort of command and control mechanism that implied that the discovered malware had been installed as far back as 2013.

The domain’s owner was listed as “Steve Rogers”—the scrawny patriot who, according to Marvel Comics lore, used a vial of Super-Soldier Serum to transform himself into Captain America, a member of the Avengers.

7 May

Deep Panda, dubbed X2 by the Congressional report, logged in as a KeyPoint contractor using OPM credentials and installed PlugX

This breach went undetected and the "big bang" didn't remove X2's access.

In July and August of 2014, these attackers exfiltrated the background investigation data from OPM's systems.

27 May

Big Bang Strategy. OPM Shuts down infected systems but they didn't know about Deep Panda (X2).

Deep Panda began loading keyoggers onto database administrators' workstations.

5 June

Deep Panda install malware on a KeyPoint web server.

10 June

OPM CIO Donna Seymour testified before the Senate Homeland Security and Government Affairs Subcommittee on her strategic information technology plan; does not disclose the hacks.

12 June

OPM executes a Cylance Product Eval.

20 June

Deep Panda conduct a remote (RDP) session with important and sensitive servers supporting the background process; not discovered until Spring 2015.

22 June

DHS releases incident report for first breach discovered on 20 March 2014.

23 June

Deep Panda likely first had access to OPM's mainframe.

July - Aug

Once established on the agency’s network, they used trial and error to find the credentials necessary to seed the jumpbox with their PlugX variant.

During the long Fourth of July weekend, when staffing was sure to be light, the hackers began to run a series of commands meant to prepare data for exfiltration.

Deep Panda (X2) copied bundles of records onto drives from which they could be snatched, chopped them up into .zip or .rar files to avoid causing suspicious traffic spikes.

In July and August of 2014, these attackers exfiltrated the background investigation data from OPM's systems.

9 July

OPM acknowledges March 2014 breach to the NYTs. No PII Lost (Just manuals and technical documents which is true.

21 July

OPM Director Katherine Archuleta downplayed the breach in an ABC interview: “We did not have a breach in security. There was no information that was lost. We were confident as we worked through this that we would be able to protect the data.”

29 July 

opmlearning.org registered to Tony Stark; C&C node.

August

USIS notified OPM and acknowledged the breach of its system.

6 August

OPM issued a stop work order after a breach in USIS's computer network.

16 August

KeyPoint malware installed on 5 June 2014 ceases operational capabilities.

25 August

Personal data of 25,000 government employees was likely compromised in the cyber-attack against USIS,

USIS also worked on the security clearances of National Security Agency leaker Edward Snowden and Aaron Alexis, who fatally shot 12 people and three others at the Naval Sea Systems Command at the Washington Navy Yard in September 2013. 

USIS also is being sued by the Justice Department. A whistleblower accused the company of speeding through a mountain of investigations as the wars in Iraq and Afghanistan fueled a heightened demand for cleared workers, according to the Washington Post.

The Justice Department joined the whistleblower civil suit, accusing the company of submitting 665,000 background checks that were incomplete. 

12 September

OPM declined to exercise its option to continue using USIS services.

October

Deep Panda bridge from OPM network to the Department of the Interior which holds OPM Personnel Records (4.2 million).

December

KeyPoint had announced a breach of its own, 4.2 million records exfiltrated.

Deep Panda (Presumably) exfiltrated 80 million records from Anthem.

2015

January

Anthem discovers Deep Panda in their networks.

February

Threat­Connect's analysis of the Anthem hack discover a suspicious domain registered to “Tony Stark”—the alter ego of Iron Man. Previously, that domain was named opm-learning.org.

Anthem discloses their breach to the public.

Spring

OPM discovers a Deep Panda 20 June 2014 remote session with important and sensitive servers.

3 March

idc-news-post.com registered. Used for C2 and data exfiltration. 

9 March

OPM shuts down beaconing activity from opmseccurity.org (Steve Rogers).

26 March

Deep Panda exfiltrates fingerprint data.

March

Deep Panda's last beaconing activity to opmsecurity.org.

1 April

Curtis Mejeur started work as one of OPM's senior IT strategists.

15 April

Brendan Saulsbury discovers command and control beaconing from OPM's network to opm­security.org with Cylance's Cylance V product.

They realized that Deep Panda still had a foothold in their system.

OPM notifies US-CERT about suspicious network traffic related to

opmsecurity.org. This domain was registered to Steve Rogers, a.k.a. “Captain

America” in April 2014 and the last beaconing activity occurred in March 2015.

16 April

Curtis Mejeur assigned the job to eradicate Deep Panda out of the OPM network.

OPM staff requested Cylance's help fusing Cylance V to diagnose forensic images of OPM servers. Since this was a task more suited to Cylance Protect, they rolled out that tool in a free trial mode, installed on 2000 devices, and it "lit up like a Christmas tree" with widespread infections.

19 April

Cylance technician, after discovery of a Deep Panda mistake, an undeleted .rar file, told his CEO, Stuart McClure, “They are fucked btw.”

21 April

Investigators completed due diligence and had identified over 2,000 individual pieces of malware that were unrelated to the attack in question (everything from routine adware to dormant viruses).

The PlugX variant they were seeking to annihilate was present on fewer than 10 OPM machines; unfortunately, some of those machines were pivotal to the entire network, including the jumpbox, the administrative server that’s used to log in to all the other servers.

CyTech arrived at OPM for a long-scheduled appointment to demonstrate their CyFIR product. The breach was not public knowledge at this point, and OPM staff did not share any information about it with company founder Ben Cotton, who was there to lead the demo. CyFIR also detected the malware, and Cotton immediately agreed to help with the response. Realizing that the crisis was grave enough to demand immediate action, Cotton began providing software and services based on a handshake agreement.

OPM racked up more than $800,000 in bills from CyTech—but no contract was executed and CyTech was not paid.

22 April

OPM CIO Donna Seymour testifies before the committee and discloses the "manual" breach. She made a series of false and misleading statements under oath regarding the agency's response. She testified that OPM purchased CyTech licenses, but OPM did not make any purchases from CyTech. She also testified that CyTech's CyFIR tool was installed in a quarantine environment for the demonstration, but this tool was running on a live environment at OPM when it identified malware on April 22, 2015 .

She testified on April 22, 2015 that "our antiquated technologies may have helped us a little. Two months later, on June 24, 2015, she testified that the stolen manuals that were a roadmap to OPM's systems were merely "outdated security documents."

OPM IG first notified (accidental chance encounter in the hall); advices no need for a public announcement.

23 April

OPM determines they have a major incident involving the exfiltration of personnel records which triggers a requirement to notify Congress.

24 April 

As part of a grid modernization program in Washington, OPM's building was scheduled to have its power cut for several hours.

OPM decided that, even though it would mostly be just a psychological triumph, they would dump the malware just minutes before the blackout. If Deep Panda was monitoring the network, they wouldn’t realize their access had been cut until everything finished booting up at least 12 hours later.

By the time power was restored on the 25th, the Deep Panda no longer had the means to roam OPM's network—or at least that’s what everyone hoped.

OPM orders global Quarantine.

26 April

Cylance engineers identify command and control session from important and sensitive servers on June 2014.

30 April

OPM Notifies Congress.

20 May

OPM notices another large scale exfiltration which triggers a requirement to notify Congress.

27 May

OPM notifies Congress.

June

Tony Scott, US federal CIO, orders a 30 day sprint to improve basic hygiene throughout the government. “Don’t waste a good crisis,”

4 June 

OPM briefs the media about 4.2 million records stolen.

16 June

the Committee holds first of two hearings on the OPM data breaches.

OPM Director Katherine Archuleta repeatedly told the House Oversight and Government Reform Committee two weeks ago that she couldn’t say if any non-personnel information was lost in the 2014 hack.

19 June 

FireEye attributes Deep Panda (first noticed by Crowdstrike) as the adversary Group that conducted attacks against OPM, at least they used some of the same tactics.

24 June

OPM CIO Donna Seymour testifies before the committee and minimizes the importance of the 2014 manuals breach.

29 June

The American Federation of Government Employees (AFGE - Union for KeyPoint employees) files a class action lawsuit alleging that "OPM violated our constitutional right to informational privacy by recklessly disregarding its Inspector General’s warnings over many years about its IT security deficiencies."

A judge threw the suit out in 2017 because the Privacy Act, the law that the suit was based on, used the word "disclosed" in relation to data and that didn't apply in cases where data was stolen but not publicly revealed.

30 June

OPM finally decides to buy Cylance Protect a day before the trial period was set to elapse (The were in trial mode all this time) . Cylance did not actually receive payment for months.

Cylance reports that on the 10,250 devices they are deployed on, they found nearly one piece of malware for every five devices.

9 July 

OPM issues a press release confirming 21.5 million records compromised.

10 July

OPM Director Katherine Archuleta resigns.

23 September

OPM updates fingerprint record loss estimate from 1.1 million fingerprints to 5.6 million.

2016

22 February

OPM CIO Donna Seymour resigns.