Log4shell: risk and reaction.

At the end of last week a vulnerability in the Java Log4j library was disclosed. Now generally being called "Log4shell," formally CVE-2021-44228, the effects of the vulnerability are serious, widespread, and difficult to mitigate. NIST describes the problem as "An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled." The problem lies in the lookup function, Sophos explains, (Apache describes the function and how it might be exploited in its Logging Services blog.) The vulnerability, if exploited, could give attackers a means of controlling a server, executing whatever code they might choose, and doing so by exploiting a flaw in an ordinary, mundane, logging function. Cygenta has a useful overview of how exploitation works, and credits researchers at Alibaba with discovering the flaw back in November, and responsibly disclosing it to Apache, which is why upgrades to Log4j were out by the time the vulnerability was disclosed last week. The Wall Street Journal compares Log4shell in scope and risk to 2014's Heartbleed vulnerability.

Governments issue alerts on Log4shell.

Cybersecurity and Infrastructure Security Agency (CISA) Director Jen Easterly released the following statement Saturday, December 11th, 2021, on the “log4j” vulnerability, which we quote in full: 

“CISA is working closely with our public and private sector partners to proactively address a critical vulnerability affecting products containing the log4j software library. This vulnerability, which is being widely exploited by a growing set of threat actors, presents an urgent challenge to network defenders given its broad use. End users will be reliant on their vendors, and the vendor community must immediately identify, mitigate, and patch the wide array of products using this software.  Vendors should also be communicating with their customers to ensure end users know that their product contains this vulnerability and should prioritize software updates. 

“We are taking urgent action to drive mitigation of this vulnerability and detect any associated threat activity. We have added this vulnerability to our catalog of known exploited vulnerabilities, which compels federal civilian agencies -- and signals to non-federal partners -- to urgently patch or remediate this vulnerability. We are proactively reaching out to entities whose networks may be vulnerable and are leveraging our scanning and intrusion detection tools to help government and industry partners identify exposure to or exploitation of the vulnerability.  

“The Joint Cyber Defense Collaborative is designed to manage this kind of risk. We have established a JCDC senior leadership group to coordinate collective action and ensure shared visibility into both the prevalence of this vulnerability and threat activity. By bringing together key government and private sector partners via the JCDC, including our partners at the FBI and NSA, we will ensure that our country’s strongest capabilities are brought to bear in an integrated manner against this risk. To ensure the broadest possible dissemination of key information, we are also convening a national call with critical infrastructure stakeholders on Monday afternoon where CISA’s experts provide further insight and address questions.  

“We continue to urge all organizations to review the latest CISA current activity alert and upgrade to log4j version 2.15.0, or apply their appropriate vendor recommended mitigations immediately.

“To be clear, this vulnerability poses a severe risk. We will only minimize potential impacts through collaborative efforts between government and the private sector. We urge all organizations to join us in this essential effort and take action.” 

CISA's not alone. Similar warnings have been issued by other governments, including all the Five Eyes. Britain's National Cyber Security Centre (NCSC) warns that it's detecting active scanning for the vulnerability, and singles out five Apache frameworks as particularly at risk: Apache Struts2, Apache Solr, Apache Druid, Apache Flink, and Apache Swift. The Australian Cyber Security Centre tells affected organizations that it's standing by and available to render assistance. The Canadian Centre for Cyber Security urges immediate patching as a number of Canadian government sites are taken offline. The reaction was especially quick and thorough in Québec, where the province's Ministry for Government Digital Transformation has, CBC reports, shut down almost 4000 website as a precautionary measure. The responsible Minister, Éric Caire, explained the decision, saying, "We were facing a threat with a critical level of 10 out of 10. According to the new protocols by the head of government information security, [that rating] automatically calls for the closure of the targeted systems." CERT-NZ, in New Zealand is also urging users to protect themselves.

Germany's Bundesamt für Sicherheit in der Informationstechnik (BSI) in its alert emphasizes both the severity of the risk and prospect of remote code execution. The BSI rates the risk "red," that is, of the highest severity. France's CERT-FR warns that the issue is already undergoing exploitation in the wild, and urges users to upgrade to the latest version of Log4j as soon as possible.

Apache SwiftExploitation in the wild.

Log4shell is being exploited in the wild. Widespread exploitation appears to have begun only after the vulnerability was publicly disclosed, but Cloudflare and Cisco Talos both say they saw signs of an exploit in the wild some nine days before that disclosure.

Recommendations for immediate action.

CISA urges "asset owners" to take three actions, immediately:

1. Identify their exposure to the Log4j risk. "Enumerate any external facing devices that have log4j installed."
2. Take alerts on devices seriously. "Make sure that your security operations center is actioning every single alert on the devices that fall into the category above." 
3. Use automated tools to help security staff concentrate on those alerts. "Install a web application firewall (WAF) with rules that automatically update so that your SOC is able to concentrate on fewer alerts."

The NCSC emphasizes the importance of using the latest versions of affected software. Beyond that, Britain's cyber agency advises:

• "If you are using the Log4j 2 library as a dependency within an application you have developed, ensure you update to version 2.15.0 or later
• "If you are using an affected third-party application, ensure you keep the product updated to the latest version
• "The flaw can also be mitigated in previous releases (2.10 and later) by setting system property "log4j2.formatMsgNoLookups" to "true" or removing the JndiLookup class from the classpath."

And vendors should talk to their customers about the problems they're likely to face.

CERT-NZ has similar recommendations, and they too advice upgrading Log4j to Log4j-2.15.0. In addition to bringing software up-to-date, CERT-NZ advises "Setting log4j2.formatMsgNoLookups to true by adding: '‐Dlog4j2.formatMsgNoLookups=True' to the JVM command for starting your application." This works only for versions 2.10 and later, however, so if you're running an earlier release, the agency recommends, again, updating to version 2.15.0. And CERT-NZ cautions that these mitigations may come with penalties: "This mitigation may impact the behaviour of your system’s logging if it relies on Lookups for message formatting."

The Netherlands NCSC has posted a comprehensive list of affected software. And the Swiss Government Computer Emergency Response Team, like the NCSC, offers advice on what to do when patching is impossible or impractical. It adds a list of indicators of compromise, and it also has a clear description of the exploitation kill chain that defenders will find useful.

Recommendations for policy, going forward.

CISA thinks the discovery of this vulnerability should motivate action toward widespread adoption of software bills of materials: "This effort also underscores the urgency of building software securely from the start and more widespread use of Software Bill of Materials (SBOM), both of which were directed by President Biden in his Executive Order issued in May 2021.  A SBOM would provide end users will the transparency they require to know if their products rely on vulnerable software libraries."

Industry assessments.

Arshan Dabirsiaghi, Co-founder and Chief Scientist at Contrast Security thinks that organizations must take a proactive approach to securing their Java applications: 

“Any Java application that logs data uses Log4j [which is] the most popular logging framework in the Java ecosystem and is used by millions of applications. This zero day exploit impacts any application using Log4j and allows attackers to run malicious code and commands on other systems. Make no mistake, this is the largest Java vulnerability we have seen in years. It’s absolutely brutal. There are three main questions that teams should answer now—where does this impact me, how can I mitigate the impact right now to prevent exploitation, and how can I locate this and similar issues to prevent future exploitation?”

Quentin Rhoads-Herrera, Director of Professional Services at CRITICALSTART wrote us with an overview of what's at stake with this vulnerability. The risk is serious because Log4j technology is so widely used, and is embedded in so many different products:

“The biggest issue with this vulnerability is the widespread adoption of the Apache Log4j technology. Combine that with the extreme simplicity of the exploit and the lack of any authentication in order to abuse, this makes this one of the worst zero-days in recent years.

"This vulnerability will give attackers access to companies internal networks. Even if the internal network is locked down for the company the attacker at least has access to the application running Apache Log4j so any visitor to that site could also be impacted as attackers could leverage this foothold to do things like crypto mining, spreading malware and ransomware, and data theft of not only data on the victim companies server but also that companies website visitors who may be passing data to it like passwords, credit card data, or other sensitive information.

"The level of access an attacker might have is greatly dependent on the level of access that web server has. However, even during red team engagements we take an initial access of a web server and quite often get elevated access to all of the companies data and servers within hours.

"Organizations using the open source logging utility should first start with identifying internet facing assets that might be using the vulnerable version of Log4J and quickly patch and implement the recommended work arounds. The reason to do both is we are already seeing researchers attempting to bypass the patch so with both the patch and mitigation in place the company should be able to mitigate the risk. From that point on the company should work on doing the same steps for any other application leveraging Log4j. They should also identify vendor software that might be impacted and confirm with those vendors when a upgrade will be available. Finally, all companies should be updating their defensive technologies such as Web Application Firewalls in an effort to identify and block potential malicious actors trying to abuse the Log4j vulnerability.

"Similar to any other breach they should first identify the source of the breach in an effort to close off the issue, identify where the attackers have gone and remove them from the environment, and assess the damage caused. The only difference in this bring than any other is the number of potential victims as Apache Log4j is a major library used by thousands of companies.

"Companies leveraging open source components should be tracking issues and vulnerabilities within those components actively. As soon as a component has a known issue with a patch available they should test and deploy it. This is not a small task though. To help there are several security solutions that can help companies identify known third party components in their software and any issues that might arise such as GitHub’s Dependabot which checks for dependencies you may be using and issues with those dependencies.

"This vulnerability is extremely dangerous and further highlights the risk third party software brings to it’s users. Along with being able to detect a potential breach, companies need to focus efforts around identifying known vulnerabilities within their environments and act quickly on patching and remediating to mitigate the chances those issues are abused by criminals.”

How were threat actors able to begin exploitation so quickly? Apparently it wasn't that difficult to come up with an exploit. Dr. Richard Ford, CTO, Praetorian, says the company's researchers were able to develop a weaponized, proof-of-concept exploit within hours of the issue's public disclosure:

“On December 9th, a 0-day vulnerability was reported in a popular component used in many Java applications that help run the Internet, Apache Log4j. This code is widely used and, because exploiting the vulnerability often does not require authentication or special access, it has exposed an incredible array of systems - there are even unconfirmed reports that simply changing your phone’s name to a particular string can exploit some online systems. 

"Praetorian researchers weaponized the vulnerability within hours and have a fully working exploit that we can use in the field. As background, Praetorian is an Austin-based cybersecurity solutions company that helps solve complex cybersecurity problems across critical enterprise assets and product portfolios. Their combination of software and security expertise puts them at the forefront of vulnerabilities such as this. Earlier this year, Praetorian was at the forefront of another critical vulnerability, proxylogon. The company says, as critical as proxylogon was to resolve, it had a much smaller potential impact than Log4j.

"The company’s engineers and researchers have been working since last night in a war room to scan its customers and are finding vulnerabilities in the field. Worse yet, we’re also inadvertently discovering the vulnerability in 3rd parties who are on adjacent or integrated systems. Naturally, we are following responsible disclosure policies so cannot call out these systems by name, but it is one of the largest exposures we have seen at Internet scale. All vulnerabilities are typically scored by how dangerous they are: this vulnerability has pracrically the highest score possible, and it seems likely that even some professionals are unaware of its potential impact. The situation is rapidly evolving, and we are learning a great deal about the scope and impact of this vulnerability as we quickly work with customers to help mitigate the risk in the short term while they work on a long term solution, which will require patching all instances of the vulnerable code - a process which could take months."