Comcast's Xfinity service responds to a major data breach.

Comcast has begun alerting customers of a major data breach affecting its Xfinity Internet and television service.

CitrixBleed (CVE-2023-4966) exploited to access customer data.

In a media release issued Monday afternoon, Comcast's Xfinity unit issued a "Notice To Customers of Data Security Incident." The announcement opens with a quick history of the incident: "On October 10, 2023, Citrix announced a vulnerability in software used by Xfinity and thousands of other companies worldwide. Citrix issued additional mitigation guidance on October 23, 2023. Xfinity promptly patched and mitigated the Citrix vulnerability within its systems. However, during a routine cybersecurity exercise on October 25, Xfinity discovered suspicious activity and subsequently determined that between October 16 and October 19, 2023, there was unauthorized access to its internal systems that was concluded to be a result of this vulnerability."

CitrixBleed is a software supply chain vulnerability, now patched, that has been exploited in attacks against Boeing, the Industrial and Commercial Bank of China, Toyota, and other targets.

More than 35 million individuals' data were exposed.

In a notification filed with the Maine Attorney General Comcast put the number of affected individuals at 35,879,455. That would be, the Verdict observes, effectively all of Xfinity's customers, but "individual" in the context of the disclosure is a term of art: it really means "user IDs," a Comcast spokesman told the Register, and a single customer might well have several user IDs. Still, nearly thirty-six million is a lot by any reckoning.

Not all Xfinity's customers, we note, have received the notifications Comcast says it's making, which suggests either that notification is lagging or that not all customers were affected. BleepingComputer reports that some Comcast customers began receiving notifications as early as last week, although those were simple directions to change passwords without providing background about the compromise.

The compromised data for those customers who are affected include usernames and hashed passwords. Some of those customers may also have suffered exposure of names, contact information, the last four digits of Social Security Numbers, dates of birth, "secret questions" and the answers thereto. The notifications advise the recipients to to reset their passwords and enable multifactor authentication.

Comcast hasn't received a ransom demand, nor has it seen any evidence of stolen data being exploited. “We are not aware of any customer data being leaked anywhere, nor of any attacks on our customers,” a company spokesman told TechCrunch. Comcast continues to investigate, and is working with appropriate law enforcement agencies.

Background to CitrixBleed.

Kiran Chinnagangannagari, CTO, CPO and co-founder of Securin, explained the vulnerability exploited at Xfinity:

“CVE-2023-4966, or more commonly known as “CitrixBleed,” is a vulnerability within the Citrix NetScaler ADC and Gateway software that could allow a cyber bad actor to take control of an affected system. 

"At the time of patch release, Citrix had no evidence of the vulnerability being exploited in the wild. However, Securin has observed exploitation just a week later, including ransomware groups LockBit and Medusa leveraging this vulnerability. Securin also observed mentions of this vulnerability in deep, dark web and hacker forums.

"Vulnerabilities within commonly used software are extremely dangerous because they can be replicated across other companies that might not have patched it either, which we have seen in the case of Citrix Bleed, as it is being linked to many incidents in 2023, including Boeing, ICBC, DP World, Allen & Overy, and thousands of other organizations. These big-name victims emphasize ransomware gangs’ ongoing commitment to crippling and disrupting operations that could affect the security of everyday people and even U.S. critical infrastructures.

"While large-scale companies have been facing ever evolving and continuous threats to their cybersecurity, it’s important to remember that these vulnerabilities are all too common and risk exploiting data like names, contact information, the last four digits of social security numbers, dates of birth, and answers to secret questions on the site. This particular vulnerability leaks the content of system memory to the attacker, allowing the attacker to impersonate a different authenticated user. This exploit poses a grave threat to system security and user integrity, emphasizing the critical need for immediate attention and remediation. CWE-119 is the weakness associated with this vulnerability and Securin is tracking 14,231 additional vulnerabilities associated with this weakness with quite a few of them being exploited by ransomware and APT groups."

Mike McGuire, Senior Software Solutions Manager, Synopsys Software Integrity Group, points out the widespread exposure to the vulnerability. “CitrixBleed represents another vulnerability that impacts a broad set of victims, and offers a simple exploit, which leads to its heightened severity. Vulnerabilities like these are why we're seeing emerging requirements regarding software bills of materials (SBOMs). Bugs and vulnerabilities will always be an artifact of software development, made clear by the thousands of vulnerabilities publicly disclosed every year, and SBOMs offer software supply chain visibility that enables software builders and buyers alike to streamline resolution processes. While most of these requirements impact software providers, the consumers of commercial software should be taking on the responsibility to protect themselves from upstream software risk as well by validating third-party SBOMs and monitoring them for newly disclosed vulnerabilities.”

Thomas Richards, Principal Consultant also at Synopsys Software Integrity Group, sees the authentication bypass risk the vulnerability presents as particularly dangerous. “The 'CitrixBleed' vulnerability is particularly concerning because it allows unauthenticated remote attackers to gain sensitive information from the servers, such as session authentication tokens," Richards wrote. "Once an attacker gains access to the session tokens, they can impersonate the authenticated user and perform actions as that user. In the instance of Comcast, the attackers were able to hijack a session of an employee and gain access to the same systems that employee has access to. Buffer overflow vulnerabilities such as this are less common nowadays due to better secure design practices, however, when they occur they are always damaging. Organizations can protect themselves from these threats by installing critical patches by vendors as soon as they are released and monitoring critical systems for malicious traffic.”

This kind of exploitation puts attackers inside the defenders' OODA loop.

CitrixBleed was quickly exploited over a short period of time. Immersive Labs’ Director of Cyber Threat Research, Kev Breen, thinks that this compressed tempo makes vulnerabilities of this kind especially risky. “The time to exploitation is what makes this type of CVE vulnerability so dangerous. In 2022, the median time to exploitation was one day from exploitation, while timing of public patches were on average 7 days. This year we’ve consistently seen recently disclosed vulnerabilities and zero days actively exploited in the wild by threat actors at scale," Breen wrote in emailed comments.

He added that defenders haven't been able to keep pace. "Despite government intervention to try and strengthen transparency and guidance around cybersecurity practices, many standard implementations still haven’t kept pace. For example, FedRAMP guidelines say organizations have 30 days to remediate high-risk threats — yet attackers just need one day to discover a vulnerability and take advantage to wreak havoc on systems and cause costly damage to organizations. Cybercriminals will likely continue to have first mover advantage, so it is security teams' responsibility to assume compromise and remain cyber resilient as it is unlikely that guidelines such as FedRAMP will be updated to meet the standards of today's threat landscape."

And you can't achieve resilience just by being fast to patch. "It’s important for organizations to understand that resilience is not just a matter of patching quickly," Breen thinks. "You can never be as fast as the attacker, so they’ll always have first mover advantage, with defenders always playing catchup. If you assume compromise by a zero day, as you are not able to patch quickly enough, your teams should all be ready to respond quickly and efficiently to get operations restored and understand the full scale of impact to customer or user data. Being cyber resilient means having teams with knowledge, skills and judgment with safe access to hands-on cyber drills covering the latest exploits.This approach enables proactive threat hunting and mitigations by defensive teams while network teams push changes through CABS and prioritize updates.”

(Added, 5:15 PM ET, December 21st, 2023.) Third-party exploitation can be challenging for organizations to handle. Jeffrey Wheatman, SVP and Cyber Risk Evangelist at Black Kite, explained:

"Third-party attacks are booming and are showing no signs of slowing down. Over the last 36 hours alone, we have seen two major supply chain attacks --- ESO Solutions and the Comcast/Citrix breaches --- together impacting nearly 40 million people (over 10% of the US population). In both situations, sensitive data including usernames, passwords, social security information and dates of birth were obtained. And in the case of ESO, more personal medical information was acquired from patients associated with its hospital and clinics in the US.

"Losing this type of critical data to hackers is catastrophic, both for companies and its stakeholders -- in these cases consumers and patients. Because ecosystem risk (the convergence of third-party risk management, vendor risk management and supply chain risk management) is abundant, here are key questions organizations should ask themselves to see how secure they are:

• "Who in your ecosystem has access to your data and your customers' data?
• "What are your partners and suppliers doing to protect their systems?
• "Are you continuously monitoring your digital supply chain? Or are you using tools that provide more than a point-in-time snapshot of security?
• "Are you relying on your partners to tell you about their software and patches they are performing?

"Considering these questions will inform your team and help create a security strategy to better protect your organization --- and avoid the time, money and reputation hit your organization could take in the event of a major attack."

What affected users should be doing now.

Securin's Chinnagangannagari added a quick general recommendation to his background comments on CitrixBleed. "Companies must look at leveraging a framework like Continuous Threat Exposure Management (CTEM) framework to prioritize and mitigate risks. In addition to multi-factor authentication (MFA), cybersecurity teams must implement and update basic security practices with routine scans of their attack surface, consolidating third-party applications, updating access controls, systems, and routine updates to complex passwords.”

Paul Bischoff, Consumer Privacy Advocate at Comparitech, thinks that "Xfinity customers should change their passwords and keep an eye on their credit reports and financial accounts for signs of fraud. The stolen passwords were hashed, meaning hackers don't have access to the plain text, and accounts aren't under immediate threat. But it's better to be safe than sorry." What if you find you can no longer get into your account? "If you find that your password no longer works, that could be a sign of compromise. Comcast has stated some secret questions and answers were compromised in the breach, which are used for account recovery. Attackers could use them to hijack accounts and change their passwords."

Chris Hauk, Consumer Privacy Champion at Pixel Privacy, agrees that "Xfinity customers should immediately change their passwords, as well as their secret questions and answers before they are locked out of their accounts." And he recommends sound practices when they do make the change. "Make sure the password is secure and unique. Use a password manager to create the passwords if you have trouble coming up with a unique password." If you're still reusing credentials across different services, let this be a warning to change that practice (and those credentials). "Customers should also check to make sure they haven't reused their Xfinity login/password combination for any other sites or apps and change them to secure unique passwords as soon as possible. They should also keep a close eye on their credit and banking accounts and take advantage of any credit monitoring services that will likely be offered by Xfinity."

The practice of reusing credentials also concerned Darren James, senior product manager at Specops Software, an Outpost24 company. James commented, “This breach is particularly alarming as the type of data that has been declared stolen indicates that passwords and answers to identifying secret questions and answers have been lost. We have seen that many people re-use the same password and security questions across many platforms, so if this data has been exposed, then it’s not just the Xfinity account, it’s potentially many other services as well. Even though the passwords may have been hashed, depending on the hashing algorithm used and the length of the password, it is still relatively easy to brute force these hashes back to clear text very quickly using relatively inexpensive hardware. It does not appear that the secret questions and answers were hashed at all."

And he urges some self-examination in the interest of security. "It is of extreme importance that anyone affected by this breach change not only their Xfinity password immediately, but also take time to think about where else they may have provided the same answers to security questions, whether for personal or business use. We all have many passwords to remember, but it’s important to make sure that the ones we use are not already breached and that the security questions we might use to identify ourselves are not the same across all platforms. Businesses need to employ strong MFA services for their users and customers and move away from weaker ID services such as security questions.”

Lessons for patching practices.

Dr. Darren Williams, CEO and Founder of BlackFog, wrote about the implications of a third-party vulnerability like CitrixBleed. “Third-party vulnerabilities can often result in delayed patching for the company which holds customer data. While Comcast is insisting that no customers have been directly affected or ‘attacked’, this is unlikely as customer data was actively exfiltrated. This breach highlights that any company – small, medium or large – can quickly become a victim of a cyber incident. The key consideration is not 'if' a breach will happen, but 'when.' This makes it essential to have the proper safeguards in place to prevent data exfiltration. Stopping cybercriminals in their tracks and preventing them from obtaining the one thing they are there for: your data, is the only way to prevent a breach. Additionally, this attack brings emphasis on the need for businesses to strongly consider and assess the security measures of the vendors they work with. With this mindset, companies can better prepare for the inevitable attack.”

Etay Maor, Senior Director of Security Strategy at Cato Networks, draws some general lessons on patching from the incident. "Reportedly, hackers were able to leverage the vulnerability before Xfinity was able to implement the necessary patch," he wrote. "This is why virtual patching, a security policy enforcement layer that prevents the exploitation of a known vulnerability, is critical for enterprises and SMBs alike to protect their data. This is an integral part of a single vendor Secure Access Service Edge (SASE) solution. (For example, Cato's virtual patching process protects thousands of customers, usually within hours or days from the release of new vulnerabilities). Traditional patching often has to be done manually for each different system and has several challenges: finding all affected systems, patching them all, continuously updating and more, all while still being vulnerable to the exploit. Virtual patching ensures constant security across the entire network as you update your systems accordingly.”

(Added, 3:00 PM ET, December 21st, 2023.) Paul Laudanski, Director of Security Research at Onapsis, offered some perspective on the incident. The vulnerability was exploited rapidly, but, he thinks, we should note that the response was quick too, and limited the damage.  

“Comcast, the parent company of Xfinity, experienced security breaches due to delayed patching,” Laudanski wrote. “Reports indicate that Comcast's comprehension of the breach's magnitude took time, and fortunately, the incident did not escalate to involve ransomware. In this case, it's essential to empathize with security personnel and to acknowledge the relentless efforts of security teams at Comcast and similar organizations. Their tasks often involve a myriad of challenges, including navigating blocked periods where production changes are prohibited, or grappling with incomplete asset and environment understanding, crucial for risk assessment and patching vulnerabilities. The focus should be on allowing these professionals to perform their duties effectively, particularly in incident management and root cause analysis. Such analyses are vital for identifying lapses and implementing solutions to prevent future breaches.

“Regarding the rapid exploitation of vulnerabilities, evidence suggests that new threats are identified and acted upon swiftly by malicious actors. This underlines the importance of timely patch assessment and application by companies to deter cybercriminals, who often operate within narrow time frames to breach targets. Additionally, adversaries are continually scanning and cataloging potential exploitation targets, utilizing the same databases and systems as security professionals to enhance their offensive strategies. Their goal is to infiltrate systems, steal critical data, deploy ransomware, or disrupt essential services. Because of this, while empathizing with security teams in crisis situations is important, it is equally crucial to ensure they receive adequate support from management and board members. This support should not only be reactive but also proactive, emphasizing security as a fundamental aspect of organizational strategy.

“Ultimately, the responsibility to safeguard customer data rests with the businesses. This is not only a business imperative but also a fiduciary duty to the customers. In the end, as consumers, we depend on these companies to protect our data, emphasizing the necessity of prompt and effective patching practices for overall data security.”

Effect of the incident on Comcast.

(Added, 3:45 PM ET, December 21st, 2023.) Neil Begley, Senior Vice President for Moody’s Investors Service, has offered an assessment of Comcast's breach. “Comcast’s announced cybersecurity breach is credit negative. Though cyber incidents have become more common, there remains risk as it could adversely impact customer behavior, cause churn to spike, and/or attract the scrutiny of the FCC and other regulators. Cyber incidents in the telecoms industry have been rising, raising questions about the industry’s cyber risk governance and defenses, as well as the overall exposure profile.”