Log4Shell lessons: Software is less vulnerable – but threats remain.

In Fall 2021, the term “Log4j” was not widely known outside of the software development community, nor did it play any significant role in my day-to-day work as a security researcher. Log4j is a remarkably popular open-source library used by developers to log user input and additional data for Java applications. It’s used by organizations large and small, including Apple, Twitter, Amazon, Tesla, Google and countless others.

On December 9, 2021, everything changed. The Apache Software Foundation – which maintains the library – disclosed the existence of a vulnerability that would be dubbed “Log4Shell”. This vulnerability allows unauthenticated threat actors to take over applications running vulnerable versions of Log4j with the input of a single line of code. With this, they could conduct remote code execution (RCE), enabling them to run the application as if they were an authorized user.

When the news broke, many companies went into “all hands on deck” mode. Response teams urgently worked to identify instances of Log4j in their custom applications as well as third-party tools in their environments.  

Teams around the world were slammed, and for good reason: Apache gave Log4Shell the highest possible Common Vulnerability Scoring System (CVSS) severity rating of 10. News reports revealed that criminal syndicates and associates of adversarial nation-states were leveraging it to target government agencies, critical infrastructure operators, and other organizations.

The vulnerability was impactful and memorable because it was relatively simple to exploit, and targets were seemingly everywhere across the Internet. Some experts claimed it was the most critical and potentially damaging vulnerability ever – and the jarring prospect of mass, global threats sent response teams everywhere into a patching frenzy.  

By December 2022, I was curious: What does the threat landscape look like one year later in regard to Log4Shell? Many pieces of research have looked at the mechanics and impact of Log4Shell over this time. However, I was more intrigued by how the collective Internet community has responded to it since the December 2021 disclosure. 

Our team initially investigated the global response to Log4j in our State of the Internet Report in September 2022. To continue down a path of exploration and discovery this past December, we examined hosts running Metabase, Neo4j, Solr, Rundeck, and UniFi software to identify what is still likely vulnerable to Log4Shell and what is not. Here is what we found:

• A swift response emerged immediately after the disclosure, with a 34 percent decline in Log4Shell vulnerable devices from December 2021 to January 2022.
• The decline continued from December 2021 to December 2022, with a 78 percent decrease in hosts which appear to be running software vulnerable to Log4Shell within the year-long timeframe.
• Despite the positive progress, our team found more than 23,000 hosts, which still appear to be vulnerable. Attempts to exploit vulnerable Log4j versions are not limited to state-sponsored actors, although Chinese and Iranian threat actors have targeted them, according to news reports. Most recently, GreyNoise identified several hundred hosts that appear to be actively scanning for the vulnerability. Previously, at the peak of activity, adversaries overwhelmed the GreyNoise sensor network with nearly 1 million Log4Shell requests in a single week.
• Amazon tops the list with respect to owning the most vulnerable hosts, with our team observing more than 3,000 instances for autonomous system (AS) Amazon-02 and nearly 2,000 for AS Amazon-AES. This comes as no surprise since Amazon owns some of the Internet’s largest autonomous systems. DigitalOcean, Microsoft, OVHcloud, Google and Alibaba also rank high on the list.
• In identifying which countries have the most vulnerable services, the U.S. ranks #1 with more than 6,500 instances, followed by Brazil (1500), Germany (1200) and the U.K. (800). These findings are unique to Log4j: If the distribution of vulnerable Log4j versions mirrored the distribution of services globally overall, we wouldn’t see Brazil so high on the list.

I have mixed sentiments about our findings: It’s encouraging to see security teams reacting quickly and effectively to Log4Shell. This is an indicator that the security community can come together and respond in a tangible and meaningful way to major threats. But it’s concerning to see that this reaction apparently was not entirely universal, and many hosts still remain vulnerable.

There’s likely no single explanation for this, but rather a multitude of factors that play into how many vulnerable hosts are still exposed on the Internet. Perhaps the most obvious is that many organizations may not have the proper response and patching processes in place. It’s also possible that organizations unknowingly have Log4j in their tech stacks due to third-party tools that may use it. Thus, they may not be aware of the issue or may not have a way to remediate if the vendor hasn’t released updates or patches. 

If you haven’t done so already, it’s worth confirming vendors’ responses to Log4Shell–were they vulnerable? Did they release a patched version of their tool or software? Beyond this, ensuring your organization has a patch and vulnerability management process is key.

After all, the best time to remediate Log4Shell was a year ago, but the second-best time is now.