Log4j and other issues in open-source software.

The Log4j vulnerabilities represent an international challenge. Open source software supply chains run freely across national borders; it's unsurprising that their vulnerabilities are no respecters of sovereignty.

CISA's Log4shell update.

The US Cybersecurity and Infrastructure Security Agency's (CISA) Known Exploited Vulnerabilities Catalog now includes Log4shell, and that's consistent with the agency's aspiration, expressed clearly during yesterday's media call, of serving as a single authoritative source for information on risk and remediation. The agency's leaders, while emphasizing the seriousness of the vulnerability, nonetheless offered a broadly optimistic view of the response.

It's noteworthy that Director Easterly said, more than once, that CISA had so far observed no confirmed nation-state exploitation, and no confirmed instances of Log4shell being exploited to distribute ransomware. As a number of the media outlets who participated in the call have pointed out, Easterly said it was entirely possible that some nation-state activity has simply gone undetected. This is simple prudence, not fear-mongering. Of course it's possible nation-state activity has gone undetected, and CISA will be more in any case more reticent about attribution than are its private sector partners (Microsoft and CrowdStrike among them) who have reported attempts at exploitation by Chinese, Russian, Iranian, and even Turkish state actors. Easterly put it this way:

"At this time, we have not seen the use of Log4Shell resulting in significant intrusions. This may be the case because sophisticated adversaries have already used this vulnerability to exploit targets and are just waiting to leverage their new access until network defenders are on a lower alert. Everybody remembers the Equifax breach that was revealed in September of 2017 was a result of an open-source software vulnerability discovered in March of that year. It may also be due in part to the urgent actions taken by defenders and many organizations to rapidly mitigate the most easily exploitable devices, such as those accessible directly from the internet, We do expect Log4Shell to be used in intrusions well into the future." 

NSA's Greg Bednarski explained in a Twitter thread the five possible explanations he sees for how a vulnerability that seems ripe for mass exploitation should so far have been used for petty larceny:

"1. Victims could be keeping their status as quiet as possible. They're out there in large numbers, just keeping it very confidential

"2. Victims don't know they've been exploited. Exploitation moved so fast and stealthily that attackers succeeded and pivoted to alternate, unrelated means of access

"3. We've under-estimated potential targets' competence, and services using log4j in their environment are locked in containers, chroot jails, or run under constrained accounts

"4. We've over-estimated the vulnerability itself, and most attackers find themselves constrained by the access they've gained, due to the nature of the services they've landed in

"5. Something else I’ve not mentioned"

To be sure, Possibility #5 is a waffle, but the list as a whole is an interesting one. Bednarski analogizes the situation to the one the Fermi Paradox about alien life called out, which in brief asks, "If there are aliens, well, where are they?" He draws two lessons from Log4shell. First, we need "a better method for estimating the potential threat of a vulnerability; we have to improve our forecasting ability so we can all optimize for the best response. Remote + public facing + easy is a relevant, but clearly insufficient." Second, we should be looking for "a scalable way to collect accurate victimization data representative of the population after the fact; this is the feedback loop to help us refine the forecasting, amongst many other things."

The Log4shell response and lessons learned.

CISA's Easterly emphasized how gratifyingly successful she thinks cooperation with public and private sector partners has been so far in addressing Log4shell:

“I just want to footstomp the unprecedented nature of how we've been handling this because there are so many products and being able to track all of this in one place, really with the help of crowdsourcing from vendors, from the research community, has helped to create some order in the chaos. We're learning a lot of lessons about how to deal with something that is as widespread as this particular vulnerability.”

The incident is requiring organizations to manage an extraordinarily complicated supply chain with thousands of vendors, all of whom, Easterly emphasized, must prepare and deliver their own patches.

The Senate Homeland Security Committee Chair, Senator Gary Peters (Democrat of Michigan), convened meetings last week to receive briefings from senior Administration officials on the response to Log4shell. Homeland Preparedness News quotes Senator Peters as saying, "I was pleased to hear how our government has swiftly mobilized to respond to this threat – including by requiring federal agencies to secure their systems and by offering support to impacted organizations. However, I remain concerned that we will likely never know the full scope and impacts of this widespread vulnerability or the risk posed to critical infrastructure.”

Concern about the difficulty of the Government gaining adequate insight into the scope of any particular risk has lent renewed urgency to lawmakers, Senator Peters among them, who are sponsoring legislation that would require organizations to report major cyber incidents. Roll Call notes that House and Senate measures failed to make it through conference during the last session of Congress, but that efforts to arrive at a bill that could pass both houses would resume as Congress reconvenes.

Australian authorities also believe they're containing Log4shell.

Australian authorities have been running through the same Log4shell issues--detection, remediation, improvement--other governments are dealing with, and they've been doing so in close partnership with their Five Eyes allies. The Australian Signals Directorate and the Australian Cyber Security Centre have been full participants in getting out guidance to the many stakeholders affected by the vulnerabilities in the open source library.

The Australian Associated Press today published an account of where the country stands with respect to remediation. The piece focuses on the healthcare sector, with particular attention to the Australian Immunisation Register and the Medicare and Pharmaceutical Benefits Scheme, all of whose portals required updates over the Christmas holidays. Services Australia general manager Hank Jongen told AAP that, “We’re not aware of any data being exposed by third party vendors and we continue to actively work with developers to transition." The assessment in the AAP is thus similar to the one CISA leaders shared yesterday: Log4shell represents a major challenge, but so far it's a challenge that organizations have for the most part risen to.

The pervasiveness of Log4j vulnerabilities.

Log4j vulnerabilities will not be amenable to any quick or easy fix, and in this respect they represent, in heightened form, a problem that's common to open source software generally. As Wired points out, not even the big stick the US Federal Trade Commission has waved at businesses will be able to effect an overnight cure for the flaws. Remediation will have, as CISA's Executive Assistant Director for Cybersecurity Eric Goldstein said yesterday, "a very long tail."

Pravin Madhani, CEO and Co-Founder of K2 Cyber Security, sees the Log4j episode and its sequlae as instances of the larger challenge securing third-party code in the software supply chain represents:

"Log4j is a good reminder of how vulnerable today's organizations are to attacks on the software supply chain. Third party software purchased through the supply chain should have just as much security review as internal applications, and how seriously a vendor implements security in their product should become a standard part of the buying process.

"The challenge with the Log4j flaw is that new variants of the original Log4j vulnerability are being discovered and each one of them requires a new patch. Also, organizations may not be able to take down all the servers at once for patching. Ideally, organizations should consider an application runtime security solution which eliminates the urgent need for patching against new vulnerabilities like Log4j, and gives organizations time to methodically schedule patches."

The open source software supply chain and the free-rider problem.

BleepingComputer reported an infinite-loop a developer inserted into two widely used open-source libraries. It was a gesture of protest. The developer, whom BleepingComputer identified as Marak Squires, is thought to regard himself as having been exploited by the many organizations who've used his software without either adequate compensation or support. Sonatype describes what the two libraries do for developers: "The immense download rate of these two components can be attributed to the basic, but essential, functionality they provide to JavaScript developers: ‘Colors’ lets you print colorful text messages on the console, whereas ‘faker’ helps devs generate fake data for their applications, for testing or staging purposes." The company adds that there seems to be little doubt that the developer pushed the changes deliberately. "Moreover, the commit history of ‘colors’ and ‘faker’ seen by us also indicates these changes are intentionally pushed by the developer to push people to either fork the projects, resolve the bugs themselves, or support the open source developers that are largely volunteers, sustaining critical projects in their free time." Earlier versions may be free of the bug; sometimes latest isn't always best.

The Apache Foundation addressed this sentiment and its causes in a position paper that it's shared with, among others, the White House. SecurityWeek headlines its account of the paper "Apache Foundation Calls Out Open-Source Leechers." That's a strong way of putting it (and we couldn't find any variant of "leech," still less "freeloader" or "parasite") in the Apache text, but it's not too far off the mark. The position paper frames a central difficulty of open source software as free-rider problem: people who benefit from an activity but don't contribute to that activity tend to exhaust and weaken it. "We can't fix open source supply chain issues by focusing exclusively on the upstream producer," is how Apache puts it in their first "take-away."

The downstream users of open source software should, Apache argues, "contribute back." That is, "Help fix bugs. Conduct security audits and feed back the results. Cash, while welcome and useful, isn't sufficient.  We eagerly welcome audits and fixes from any source. We have a process defined for doing so: https://www.apache.org/security/."

Uriel Maimon, senior director of emerging technologies at cybersecurity company PerimeterX, also commented on the "colors" and "faker" incident:

“This case shows how problematic the security controls around modern web applications are. Modern web applications use a complex supply chain of third-party libraries and code, which use other third-party libraries that nest a long way down. The operators and developers of modern websites have no visibility into what those libraries do and what changes occur to them.

"In this attack, the changes were made by the disgruntled maintainer of a couple of libraries like this. This underscores how one actor with malicious intent can affect hundreds of thousands of websites with just one action. That said, it could have also been someone who contributes code to open source or community projects or that took over the maintainer’s GitHub account. Though the malicious intent was obviously visible due to the defacement and malfunctioning of the libraries, what if instead of breaking the websites, the attacker used the library to steal PII or credit card information? How long would that have continued with no obvious change to the website before it became known? Months? Years?

"This underscores just how critical it is for modern websites to run real-time detection on their web applications in production to find out how third parties and the supply chain are affecting the integrity and security of their business.”

PerimeterX's Osterman Report describes the challenges of securing open-source software and third-party code. 99.6% of respondents to their survey said their organization's website "uses at least one third-party script," a total so close to everyone as to make no difference. And the criticality of such third-party code is difficult to determine, especially by the upstream provider. Criticality is highly dependent upon context and purpose. To return to Apache's position paper:

"As our software is without cost and can be downloaded by anybody, we have no way of knowing how widely our projects are used. This means that, by necessity, determining if a component is critical can only either be determined by automated means which is always going to be incomplete and imperfect, or more manually such as by identifying critical products first, and then getting an inventory of what components those products embed. 

"This problem is not unique to the ASF [Apache Software Foundation], so it makes sense for there to be industry standards and best practices. We are prepared to participate in these activities. 

• "As noted above, vulnerabilities can be a result of a how a number of components are combined.
• "The owner of a system is best placed to determine the criticality of that system and the degree to which individual components contribute to that criticality.
• "Criticality depends on usage. We need to avoid solutions that create burdens for all system owners using a software component when the component is only critical for some systems."