Log4shell remains a tough problem, as vendors and users work to foreclose the possibility of exploitation of the vulnerability disclosed in Apache's Log4j. For a guide to all our coverage of Log4shell, click here.
Log4shell: exploitation and remediation.
Die Zeit, in a long and glum piece on the implications of the Log4shell vulnerability, points out that the term "affected" can be ambiguous, particularly when it appears in phrases like "not affected." ("Was 'betroffen' in den konkreten Fällen heißt, ist nicht immer klar.") What counts as "affected?" It's not necessarily synonymous with "attacked," "breached," or even "vulnerable." If you've had to devote time and resources to inventorying your software for a specific vulnerability, there's a sense in which you've "been affected," even if at the end of it all you've found nothing. In any case, remediation and defense remain a long and complicated slog.
Spies, crooks, and (maybe) saboteurs.
The saboteurs so far represent an a priori possibility, but the crooks and spies have been up and at-em this week, in real life. Haaretz reports, citing sources at Check Point, that Iranian operators had by yesterday sought to compromise seven Israeli governmental and commercial targets using Log4shell exploits. Both Microsoft and Mandiant have warned of Chinese, and Iranian exploitation of the vulnerability, the Wall Street Journal sums up, adding that Microsoft also reports seeing North Korean and Turkish attempts to take advantage of Log4j. (The Chinese embassy in Washington told the Journal that they're opposed to “cyberattacks of any kind.” The embassy also pointed out that it was a Chinese company that first discovered the issue and disclosed it to Apache. In fairness to Beijing, they're right about that second point: Alibaba's Cloud Security Team found and reported the problem on November 24th.)
In some respects, however, nation-state exploitation seems almost a case of a dog not barking. The Journal quotes CrowdStrike's senior vice president of intelligence, Adam Meyers to that effect: “It’s a surprise it’s not more widespread. The question that everyone is asking is, ‘What aren’t we seeing?’” Mandiant also expects to see more nation-state exploitation: "We expect threat actors from additional countries will exploit it shortly, if they haven’t already. In some cases, state sponsored threat actors will work from a list of prioritized targets that existed long before this vulnerability was known. In other cases, they may conduct broad exploitation and then conduct further post-exploitation activities of targets as they are tasked to do so."
And among the dogs not obviously barking? Well, not dogs, but in this case the dogs' ursine cousins, the Bears, like Cozy and Fancy. Russian state actors, BGR observes, are noticeably not being mentioned in dispatches.
Taking action on Log4shell.
The US FBI's advisory yesterday on Log4shell is brief enough to be worth quoting in full:
If you feel your systems have been compromised as a result of the Log4j vulnerability or are seeking remediation, we encourage you to employ all recommended mitigations and follow guidance from CISA.
If you think your organization has been compromised as a result of the Log4j vulnerability, visit fbi.gov/log4j to report to the FBI. Please include as much information as possible to assist the FBI and CISA in determining prioritization for victim outreach.
Due to the potential scale of this incident, the FBI and CISA may be unable to respond to each victim individually, but all information we receive will be useful in countering this threat. As always, we stand ready to assist any impacted entities.
A number of organizations have issue guidance on how to defend against Log4shell exploitation. Mandiant yesterday issued some clear and actionable steps organizations can take to protect themselves. They begin with the tough problem of identifying vulnerable software. "The first step an organization must consider is to determine the scope of applications and dependent services (organization managed and third-party integrated technologies) that leverage the Log4j library," Mandiant writes, adding. "This can be a very challenging and time-consuming process, as the Log4j library could be integrated with many third-party vendor applications and products, in addition to being installed locally on servers and endpoints within an environment."
Comment from industry on the challenges of addressing this kind of software supply chain issue.
We heard from several industry sources about the challenges remediation and defense present in this case.
Shiran Grinberg, Director of Research & Cyber Operations at Cynet, points out the complexities of addressing vulnerabilities in third-party software:
“Since Log4shell is a vulnerability in a third-party software, attackers will first need to understand how it is being implemented and used in specific environment and only then locate and control the data logged with log4j functionality, so the range of methods to exploit the vulnerability is wide and has dependency on the specific use of the functionality. At Cynet, we’ve observed several techniques used to exploit the vulnerability, all of them allowing malicious threat actors to execute remote code on a vulnerable server. For example, a simple python script can be used to trigger RCE on a vulnerable server by exploiting additional user-controlled data which is commonly logged as the User-Agent field within HTTP requests.”
Grinberg offers some additional advice for protecting an organization against exploitation of CVE-2021-44228 that goes beyond the not-so-simple patching and updating everyone should be working on:
"Disabling lookups using by setting the parameter 'formatMsgNoLookups' to 'True' value: Log4j2.formatMsgNoLookups=true. Setting an environment variable for disabling lookups: LOG4J_FORMt_msg_no_lookups=true. Another option would be to remove Jndilookup class from the classpath." This can be done by the following command – Zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
Lou Steinberg, now a venture capitalist funding cybersecurity companies at CTM Insights, wrote:
Log4j is one of the most popular open source logging programs around, used by thousands of popular programs and hosted services. It saves messages about the status of programs for monitoring and debugging. For example, in minecraft, user chats are logged with log4j.
It was found to have a serious "remote code execution" vulnerability. What that means is if I can get log4j to store (log) a message that includes code, I can get it to run that code for me. In the minecraft example, all I have to do is enter the specially formatted code in a chat window and I can run software on their server.
Vulnerable servers can be told to run code that lets an attacker completely take them over. Patches have been issued and people are scrambling to deploy them before servers are compromised. We are in a race against attackers to patch before they can exploit. Likely millions of servers are at risk.
The good news is there are what's called "compensating controls". If attackers cant directly hand you messages to log, you are far less exposed. Also, the vulnerability uses the "java" programming language and one of the most common versions of java (from Oracle) has a default setting that appears to prevent the exploit. Of course, not everyone uses Oracle's java, and some may have changed the defaults. Those servers need to be checked first.
Industrial control systems and the operational technology (OT) network presents distinctive challenges with respect to remediation and defense. Dennis Hackney, of the ABS Group, notes that availability is a central concern in industrial control systems in a way that it's not for, say IT systems. That's not to say that outages are trivial matters for IT systems. Witness, for example, last week's disruption of AWS services. It was serious and inconvenient, but, on the other hand, life went on, both for businesses and natural persons. An outage in an industrial system can be more serious, both financially and in terms of safety. Dr. Hackney commented:
“In my experience, managing security on human machine interface (HMI) applications is extremely difficult for organizations with operational technology (OT) environments that could include large turbine generators, pumps, machinery controls, and more. For example, it’s difficult for them to restrict user accounts and access to an application that must be running 24/7. In most cases, an account that allows for the highest level of access is logged into to allow for an organization’s operations to perform their duties, uninhibited.
"The Log4j API primarily affects the debugging and logging capabilities within very common historian and logging applications in the OT environment. This includes the loggers that are installed locally on the HMIs, and not just on the actual historian servers. That means that these systems can be exploited completely over the wire. Log4Shell allows for unlimited remote code execution (RCE). What a lot of companies don’t realize is that supervisory control and data acquisition (SCADA) and HMI applications typically include open-source technologies like Java and Apache as found in the Log4j 2.0 vulnerability, to provide the most cost-effective and operational functionality as possible.
"The Log4j API is used in very common SCADA systems and historians in the industry. Think GE Cimplicity, OSI Pi, Emerson Progea, and SIMATIC WinCC. We actually witnessed one example where the engineer was unable to start the runtime environment for his IO servers. These are the servers that control the object linking and embedding for process control (OPC) communications between the HMIs (SCADA) and the controllers, or other SCADA and between controllers. This means the operators were running completely blind and without control; one of the most dangerous way to operate.
"As this is being actively and universally exploited, the only saving grace for some OT environments is that they are behind firewalls or DMZs limiting access from the web. However, many of today’s OT environments are still not adequately protected by firewalls or DMZs.
"In the coming days, industrial organizations should receive notifications from original equipment manufacturer (OEM) providers or should proactively reach out to these providers. For instance, Siemens (T3000) specifically utilizes Java for its programming and HMI interfaces. The security alert should provide details about whether they are affected and any patches or solutions available to mitigate or resolve the vulnerability. In the near term, they should not apply any fixes that do not originate with the OEM providers as it may cause unintended problems or failures that may be difficult to resolve or roll-back.
So watch for the fixes.