The web is an collaborative ecosystem. Web standards exist to ensure that participants of the network behave in a predictable way. If network participants deviate from the established standards then there can be unintended consequences. This blog post is about one of these unintended consequences.
A group of researchers recently published a paper "Forwarding Loop Attacks in the Content Delivery Networks" describing what can happen when web services interact in a non-compliant way. They describe an attack where a malicious user can force multiple service providers to send each other an unending stream of requests in a loop. This request loop can result in resource exhaustion and denial of service at the service provider. This paper also demonstrated that the attack is practical, and can be performed using a large list of service providers.
CloudFlare's service has been modified to be standards-compliant with respect to HTTP proxying. However, fixing the vulnerability that enables this attack requires all proxy services to conform to the same standards. If even one service provider is non-compliant, the attack can still be carried out against compliant services. In this post, we will describe the attack and explain how a proxy services can go from being part of the problem to part of the solution.
Reverse Proxies
CloudFlare works as a reverse proxy. When an HTTP(S) request comes into CloudFlare’s network, one of two things happen; either CloudFlare returns a cached response for the request or CloudFlare makes a request to the website’s origin server and returns that response. The fact that CloudFlare can inspect and modify requests going through its network enables powerful features like caching, WAF, RocketLoader and more.
It’s also not uncommon to put multiple reverse proxies in front of your site. This practice, called stacking proxies, is often used to enable multiple service providers that provide different features. For example, you could use another service for WAF and CloudFlare for caching. Although we would prefer that customers use CloudFlare's state-of-the-art WAF this is a reasonable configuration and one that several CloudFlare customers use. Proxies can be stacked in front of an origin server, but what happens if the last proxy in the stack points to the first? You get a proxy loop.
A proxy loop of two is easy to imagine. Configure a first reverse proxy service to use a second reverse proxy service as it’s origin, and then configure the second service to use the first as its origin.
In theory, any request to this site would end up being sent back and forth between the two proxies. Each iteration of this loop will cause a request to be sent and use up resources. Luckily, most reverse proxy services have protections in place to avoid basic attacks like this.
Preventing Naïve Loops
The authors of HTTP 1.1 were aware of the potential of request loops and built in protection as part of the standard. This loop prevention comes in the form of the "Via" header. From section 5.7.1 of [RFC 7230](https://tools.ietf.org/html/rfc7230#section-5.7.1](https://tools.ietf.org/html/rfc7230#section-5.7.1):
The "Via" header field indicates the presence of intermediate
protocols and recipients between the user agent and the server (on
requests) or between the origin server and the client (on responses),
similar to the "Received" header field in email (Section 3.6.7 of
[RFC5322]). Via can be used for tracking message forwards, avoiding
request loops, and identifying the protocol capabilities of senders
along the request/response chain.…
Multiple Via field values represent each proxy or gateway that has
forwarded the message. Each intermediary appends its own information
about how the message was received, such that the end result is
ordered according to the sequence of forwarding recipients.
A proxy MUST send an appropriate Via header field, as described
below, in each message that it forwards. An HTTP-to-HTTP gateway
MUST send an appropriate Via header field in each inbound request
message and MAY send a Via header field in forwarded response
messages.…
For example, a request message could be sent from an HTTP/1.0 user
agent to an internal proxy code-named "fred", which uses HTTP/1.1 to
forward the request to a public proxy at p.example.net, which
completes the request by forwarding it to the origin server at
www.example.com. The request received by www.example.com would then
have the following Via header field:
Via: 1.0 fred, 1.1 p.example.net
A sender SHOULD NOT combine multiple entries unless they are all
under the same organizational control and the hosts have already been
replaced by pseudonyms. A sender MUST NOT combine entries that have
different received-protocol values.
CloudFlare currently uses this mechanism to prevent request loops. When a request comes through the CloudFlare network that isn’t in cache, CloudFlare creates a new request for the origin server. Outgoing requests get a "Via" header containing the HTTP protocol version of the incoming request and a CloudFlare-specific pseudonym, e.g.:
Via: 1.1 cloudflare
If a request comes into the network with "cloudflare" in the Via header, an error is returned. This short circuits any request that loops through CloudFlare.
Implementing this protection means CloudFlare is safe from HTTP request loops, right? Not so soon.
The Bad News
Not all reverse proxy services are RFC 7230 compliant. Some services give customers the ability to strip or modify headers, including the Via header. This is expressly not allowed by the RFC: proxies are obligated to leave existing tags from other proxies as part of incoming Via headers. A proxy is only allowed to modify the parts of the header that were created by its own organization. Allowing proxies to change this header leads to bad things.
Let’s take the example above of a two proxy loop. Suppose both proxies are configured to add their tag to the Via header on outgoing requests, and to fail requests that contain their own Via tag. Suppose also that both proxies are set up to remove the "Via" header from incoming requests. When the request comes back from the second proxy, the first one is unable to identify that it has already proxied the request. This request is then treated like a regular request, causing the first proxy to make request to the second proxy, which strips the header and sends it back, and so on.
This attack can do a lot of damage, using up resources from both service providers. It can also cause unexpected behavior in automated attack prevention systems, resulting in one service provider being blocked by another. Needless to say, this is not ideal.
For CloudFlare Customers
By default, some web servers do not compress outgoing response if the Via header is present in the request. This can result in your origin using unnecessary bandwidth to send data to CloudFlare. Consult the CloudFlare Knowledge Base to learn how to test if your server has this behavior and how to disable it. If you do not have the ability to modify your server's configuration, please contact support.
A Call to Action
The authors of the paper contacted the affected parties before publishing this paper, but not all of them may have taken action. If your organization provides reverse proxy services to the general public, we recommend you implement the following logic:
-
Under no circumstance allow customers to remove or modify Via headers for requests to their site.
-
Append an RFC 7230-compliant Via header when proxying traffic.
-
If a request comes in with your Via header, return an appropriate error.
One non-compliant reverse proxy service can cause negative consequences for everyone, let’s work together to avoid request loops.
I would like to thank Tao Wan, Jianjun Chen, Jian Jiang, Jinjin Liang for working with us on a fix. I would also like to give a special thanks to Pasha Kravtsov for helping investigate this issue and Rajeev Sharma for our Via implementation.