Cloudflare and HashiCorp have been technology partners since 2018, and in that time Cloudflare’s integration with HashiCorp’s technology has deepened, especially with Terraform, HashiCorp’s infrastructure-as-code product. Today we are announcing a major update to our Terraform bootstrapping tool, cf-terraforming. In this blog, I recap the history of our partnership, the HashiCorp Terraform Verified Provider for Cloudflare, and how getting started with Terraform for Cloudflare developers is easier than ever before with the new version of cf-terraforming.

Members of the open source community wrote and supported the first version of Cloudflare's Terraform provider. Eventually our customers began to bring up Terraform in conversations more often. Because of customer demand, we started supporting and developing the Terraform provider ourselves. You can read the initial v1.0 announcement for the provider here. Soon after, Cloudflare’s Terraform provider became ‘verified’ and we began working with HashiCorp to provide a high quality experience for developers.

HashiCorp Terraform allows developers to control their infrastructure-as-code through a standard configuration language, HashiCorp Configuration Language (HCL). It works across a myriad of different types of infrastructure including cloud service providers, containers, virtual machines, bare metal, etc. Terraform makes it easy for developers to follow best practices when interacting with SaaS, PaaS, and other service provider APIs that set up infrastructure. Like developers already do with software code, they can store infrastructure configuration as code in git, manage changes through code reviews, and track versions and commit history over time. Terraform also makes it easier to roll back changes if developers discover issues after a deployment.

Our developers love the power of Cloudflare + Terraform for infrastructure provisioning. IT operations teams want a platform that provides complete visibility and control. IT teams want a platform that is easy to use, does not have a steep learning curve, and provides levers to control resources much better. Cloudflare + Terraform platform provides just that.– Dmitry Zadorojnii, Chief Technology Officer, Autodoc GmbH

Since the 1.0 release of Cloudflare’s Terraform provider, Cloudflare has continued to build out the capabilities exposed in the provider while HashiCorp has expanded its ecosystem by developing additional features like the Terraform Registry. This helps developers find documentation on Cloudflare’s Terraform provider or any others. Terraform itself has also matured greatly–Terraform’s v0.12 release had many changes, outlined here.

“Leveraging the power of Terraform, you can codify your Cloudflare configuration. Codification enables version control and automation, increasing productivity and reducing human error. We are pleased to have Cloudflare as a technology partner and look forward to our ongoing collaboration.”– Asvin Ramesh, Director, Technology Alliances, HashiCorp

Here are some great resources for developers looking to better understand how to get started using Terraform with Cloudflare:

• Cloudflare’s Developer Docs for Terraform

• HashiCorp Tutorial: Host a Static Website with S3 and Cloudflare

• Cloudflare Terraform Provider Documentation

• Github repo for Cloudflare’s Terraform Provider

We released the first version of cf-terraforming in early 2019 in this blog post. Since then we learned a few lessons about building and maintaining such a tool. In this section I’ll recap why we built such a tool in the first place, lessons learned over the last two years, and what is new and improved with the version we are launching today.

The name for the cf-terraforming library comes from another tool created by dtan4 on github: https://github.com/dtan4/terraforming. The original tool allowed users to generate tfconfig and tfstate files for existing AWS resources. This made it much easier for existing AWS customers to begin using Terraform. Terraform generally expects to be authoritative about the configuration it manages. Effectively, it expects that you only make changes to that config through Terraform and not anywhere else, like an API or a dashboard. If you were an existing customer of AWS (or Cloudflare) and already had everything configured via API or a UI, this posed a challenge: How do I quickly and correctly get all of my existing config into Terraform so that it can be the authoritative source of truth? For AWS resources, before dtan4’s terraforming you had to manually write the tfconfig for every object and the run import commands to generate the corresponding tfstate. For sizable deployments, this could be nigh impossible.

cf-terraforming served to solve the same problem for Cloudflare services. I had many conversations with customers who had been using Cloudflare for years and who were interested in migrating control of their Cloudflare configuration to Terraform. Cf-terraforming gave them a way to quickly convert all of their existing Cloudflare usage into tfconfig and tfstate.

cf-terraforming was one of the enabling technologies that we used to bootstrap Cloudflare into our Terraform Git-ops workflow. We had thousands of records to port, import, and manage, and doing this by hand would have been an arduous and error-prone task. Using cf-terraforming to generate our initial set of resources allowed our engineers to submit Cloudflare changes, enabling our product engineers to be infrastructure operators.– Sean Chittenden, Infrastructure, DoorDash

After having cf-terraforming available for some time, we’ve learned quite a bit about the challenges in managing such a tool.

When Cloudflare releases new services or features today, that typically means new or incremental changes to Cloudflare’s APIs. This in turns means updates to our Terraform provider. Since Terraform is a golang program, before we can update our provider we have to first update the cloudflare-go library. Depending on the change, this can be a couple lines in each repo or extensive changes to both. Once we launched cf-terraforming, we now had a third library that needed synchronous changes alongside the provider and go library. Missing a change meant that if someone tried to use cf-terraforming, they may have incomplete config or state, which would not work.

Not only did our own API changes create additional work, but changes to Terraform itself could mean changes for cf-terraforming. The Terraform 0.12 update was a massive update that required a lot of careful testing and coordination with our provider. It also meant changes in HCL and in provider interactions that cf-terraforming had to account for. Such a massive one-time hit was very difficult to account for, and we’ve struggled to ensure compatibility.

The ability to have cf-terraforming generate a tfstate file was both incredibly important and also experimental. In general a developer never really needs to concern themselves with what is in the tfstate file but needs to know it contains the actual state of those resources such that references in the config can be resolved and managed correctly. We opened up that black box, which meant that we were involved in state file implementation details that we ultimately shouldn’t be.

Given these lessons, we looked at how we could update cf-terraforming to alleviate these problems and provide a better tool for both customers and ourselves. After some prototyping to validate our ideas, we came up with a new model that has been productized and is now available for customers.

Today we are launching a new version of cf-terraforming that improves upon our previous work. This new version makes it easier for us to support new resources or changes to existing resources and simplifies the workflow for developers looking to bootstrap their Cloudflare Terraform configuration.

Instead of hand crafting how to generate both the tfconfig and tfstate for each of the 48 or so resources supported in the Terraform provider, we now leverage Terraform’s capabilities to do more auto generation of what’s needed for similar resource types. HashiCorp has a great CLI tool called terraform-exec that provides powerful out-of-the-box capabilities we can take advantage of. Using terraform-exec we get access to `terraform providers schema -json`, which gives us the json schema of our provider. We use this to auto generate the fields we need to populate from the API. In many cases the API response fields map one to one with the json schema, which allows us to automatically populate the tfconfig. In other cases some small tweaks are necessary, which still saves a lot of time to initially support the resource and lowers the burden for future changes. Through this method, if the terraform provider changes for any reason, we can build new versions of cf-terraforming that will fetch the new schema from terraform-exec versus us having to make a lot of manual code changes to the config generation.

For tfstate, we simplify our approach by outputting the full set of terraform import calls that would need to be run for those resources instead of attempting to generate the tfstate definition itself. This removes virtually any need for future library changes since the import commands do not change if Cloudflare’s API or provider changes.

With that, let’s look at the new cf-terraforming in action. For this walkthrough let’s assume we have an existing zone on Cloudflare with DNS records and firewall rules configured. We want to start managing this zone in Terraform, but we don’t want to have to define all of our configuration by hand.

Our goal is to have a ".tf" file with the DNS records resources and firewall rules along with filter resources AND for Terraform to be aware of the equivalent state for those resources. Our inputs are the zone we already have created in Cloudflare, and our tool is the cf-terraforming library. If you are following along at home, you will need terraform installed and at least Go v1.12.x installed.

Before we can use cf-terraforming or the provider, we need an API token. I’ll briefly go through the steps here, but for a more in-depth walkthrough see the API developer docs. On the Cloudflare dashboard we generate an API token here with the following setup:

PermissionsZone:DNS:ReadZone:Firewall Services:Read

Zone Resources:garrettgalow.party (my zone, but this should be your own)

TTLValid until: 2021-03-30 00:00:00Z

Note: I set an expiration date on the token so that when I inevitably forget about this token, it will expire and reduce the risk of exposure in the future. This is optional, but it’s a good practice when creating tokens you only need for a short period of time especially if they have edit access.

API Token summary from the Cloudflare Dashboard

Now we set the API Token we created as an environment variable so that both Terraform and cf-terraforming can access it for any commands (and so I don’t have to remove it from code examples).

$export CLOUDFLARE_API_TOKEN=<token_secret>

Terraform requires us to have a folder to hold our Terraform configuration and state. For that we create a folder for our use case and create a cloudflare.tf config file with a provider definition for Cloudflare so Terraform knows we will be using the Cloudflare provider.

mkdir terraforming_test
cd terraforming_test
cat > cloudflare.tf <<'EOF'
terraform {
    required_providers {
        cloudflare = {
            source = "cloudflare/cloudflare"
        }	
    }
}

provider "cloudflare" {
# api_token  = ""  ## Commented out as we are using an environment var
}
EOF

Here is the content of our cloudflare.tf file if you would rather copy and paste it into your text editor of choice:

terraform {
    required_providers {
        cloudflare = {
            source = "cloudflare/cloudflare"
        }	
    }
}

provider "cloudflare" {
# api_token  = ""  ## Commented out as we are using an environment var
}

We call terraform init to ensure Terraform is fully initialized and has the Cloudflare provider installed. At the time of writing this blog post, this is what terraform -v gives me for version info. We recommend that you use the latest versions of both Terraform and the Cloudflare provider.

$ terraform -v
Terraform v0.14.10
+ provider registry.terraform.io/cloudflare/cloudflare v2.19.2

And finally we install cf-terraforming with the following command:

$ GO111MODULE=on go get -u github.com/cloudflare/cf-terraforming/...

If you’re using Homebrew on MacOS, this can be simplified to:

$ brew tap cloudflare/cloudflare
$ brew install --cask cloudflare/cloudflare/cf-terraforming

We are now ready to start generating a Terraform config. To begin, we run cf-terraforming to generate the first blocks of config for the DNS record resources and append it to the cloudflare.tf file we previously created.

cf-terraforming generate --resource-type cloudflare_record --zone <zone_id> >> cloudflare.tf

Breaking this command down:

generate is the command that will produce a valid HCL config of resources

--resource-type specifies the Terraform resource name that we want to generate an HCL config for. You can only generate configuration for one resource at a time. In this example we are using cloudflare_record

--zone specifies the Cloudflare zone ID we wish to fetch all the DNS records for so cf-terraforming can create the appropriate API calls

Example:

$ cf-terraforming generate --resource-type cloudflare_record --zone 9c2f972575d986b99fa03c7bbfaab414 >> cloudflare.tf
$

Success will return with no output to console. If you want to see the output before adding it to the config file, run the command without >> cloudflare.tf and it will output to console.

Here is the partial output in my case, if it is not appended to the config file:

$ cf-terraforming generate --resource-type cloudflare_record --zone 9c2f972575d986b99fa03c7bbfaab414
resource "cloudflare_record" "terraform_managed_resource_db185030f44e358e1c2162a9ecda7253" {
name = "api"
proxied = true
ttl = 1
type = "A"
value = "x.x.x.x"
zone_id = "9c2f972575d986b99fa03c7bbfaab414"
}
resource "cloudflare_record" "terraform_managed_resource_e908d014ebef5011d5981b3ba961a011" {
...

The output resources are given standardized names of “terraform_managed_resource_<resource_id>”. Because the resource id is included in the name, the object names between the config we just exported and the state we will import will always be consistent. This is necessary to ensure Terraform knows which config belongs to which state.

After generating the DNS record resources, we now do the same for both firewall rules and filters.

cf-terraforming generate --resource-type cloudflare_firewall_rule --zone <zone_id> >> cloudflare.tf
cf-terraforming generate --resource-type cloudflare_filter --zone <zone_id> >> cloudflare.tf

Example:

$ cf-terraforming generate --resource-type cloudflare_firewall_rule --zone 9c2f972575d986b99fa03c7bbfaab414 >> cloudflare.tf
$ cf-terraforming generate --resource-type cloudflare_filter --zone 9c2f972575d986b99fa03c7bbfaab414 >> cloudflare.tf
$

Before we can ask Terraform to verify the config, we need to import the state so that Terraform does not attempt to create new objects but instead reuses the existing objects we already have in Cloudflare.

Similar to what we did with the generate command, we use the import command to generate terraform import commands.

cf-terraforming import --resource-type cloudflare_record --zone <zone_id>

Breaking this command down:

import is the command that will produce a valid terraform import command that we can then run--resource-type (same as the generate command) specifies the Terraform resource name that we want to create import commands for. You can only use one resource at a time. In this example we are using cloudflare_record--zone (same as the generate command) specifies the Cloudflare zone ID we wish to fetch all the DNS records for so cf-terraforming can populate the commands with the appropriate API calls

And an example with output:

$ cf-terraforming import --resource-type cloudflare_record --zone 9c2f972575d986b99fa03c7bbfaab414
terraform import cloudflare_record.terraform_managed_resource_db185030f44e358e1c2162a9ecda7253 9c2f972575d986b99fa03c7bbfaab414/db185030f44e358e1c2162a9ecda7253
terraform import cloudflare_record.terraform_managed_resource_e908d014ebef5011d5981b3ba961a011 9c2f972575d986b99fa03c7bbfaab414/e908d014ebef5011d5981b3ba961a011
terraform import cloudflare_record.terraform_managed_resource_3f62e6950a5e0889a14cf5b913e87699 9c2f972575d986b99fa03c7bbfaab414/3f62e6950a5e0889a14cf5b913e87699
terraform import cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d 9c2f972575d986b99fa03c7bbfaab414/47581f47852ad2ba61df90b15933903d$

The output of this will be ready to use terraform import commands. Running the generated terraform import command will leverage existing Cloudflare Terraform provider functionality to import the resource state into Terraform’s terraform.tfstate file. This removes the tedium of pulling all the appropriate resource IDs from Cloudflare’s API and then formatting these commands one by one. The order of operations of the config then state is important as Terraform expects there to be configuration in the .tf file for these resources before importing the state.

Note: Be careful when you actually import these resources, though, as from that point on any subsequent Terraform actions like plan or apply will expect this resource to be there. Removing the state is possible but requires manually editing the terraform.tfstate file. Terraform does keep a backup locally in case you make a mistake though.

Now we actually run these terraform import commands to import the state. Below shows what that looks like for a single resource.

$ terraform import cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d 9c2f972575d986b99fa03c7bbfaab414/47581f47852ad2ba61df90b15933903d
cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d: Importing from ID "9c2f972575d986b99fa03c7bbfaab414/47581f47852ad2ba61df90b15933903d"...
cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d: Import prepared!
Prepared cloudflare_record for import
cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d: Refreshing state... [id=47581f47852ad2ba61df90b15933903d]
Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

With cloudflare_record imported, now we do the same for the firewall_rules and filters.

cf-terraforming import --resource-type cloudflare_firewall_rule --zone <zone_id>
cf-terraforming import --resource-type cloudflare_filter --zone <zone_id>

Shown with output:

$ cf-terraforming import --resource-type cloudflare_firewall_rule --zone 9c2f972575d986b99fa03c7bbfaab414
terraform import cloudflare_firewall_rule.terraform_managed_resource_0de909f3229341a2b8214737903f2caf 9c2f972575d986b99fa03c7bbfaab414/0de909f3229341a2b8214737903f2caf
terraform import cloudflare_firewall_rule.terraform_managed_resource_0c722eb85e1c47dcac83b5824bad4a7c 9c2f972575d986b99fa03c7bbfaab414/0c722eb85e1c47dcac83b5824bad4a7c
$ cf-terraforming import --resource-type cloudflare_filter --zone 9c2f972575d986b99fa03c7bbfaab414
terraform import cloudflare_filter.terraform_managed_resource_ee048570bb874972bbb6557f7529e094 9c2f972575d986b99fa03c7bbfaab414/ee048570bb874972bbb6557f7529e094
terraform import cloudflare_filter.terraform_managed_resource_1bb6cd50e2534a64a9ec698fd841ffc5 9c2f972575d986b99fa03c7bbfaab414/1bb6cd50e2534a64a9ec698fd841ffc5
$

As with cloudflare_record, we run these terraform import commands to ensure all the state is successfully imported.

Now that we have both the configuration and state in place, we call terraform plan to see if Terraform can verify everything is in place. If all goes well then you will be greeted with the following “nothing to do” message:

No changes. Infrastructure is up-to-date.
This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

You now can begin managing these resources in Terraform. If you want to add more resources into Terraform, follow these steps for other resources. You can find which resources are supported in the README. We will add additional resources over time, but if there are specific ones you are looking for, please create GitHub issues or upvote any existing ones.

Whether you are an existing Cloudflare customer and have been curious about Terraform or you are looking to expand your infrastructure-as-code to include Cloudflare’s services, you have everything you need to get building with Terraform, the Cloudflare provider, and cf-terraforming. For questions, comments, or feature requests for either the provider or cf-terraforming, see the respective github repos.

For this blog post, we will distinguish between two different categories of services running in each data center:

• Customer-facing services: all of our stack of products that our customers use, such as caching, WAF, DDoS protection, rate-limiting, load-balancing, etc.

• Management services: software required to operate the data center, that is not in the direct request path of customer traffic.

The reliability model of our customer-facing services is to run them on all machines in each data center. This works well as it allows each data center’s capacity to scale dynamically by adding more machines.

Scaling is especially made easy thanks to our dynamic load balancing system, Unimog, which runs on each machine. Its role is to continuously re-balance traffic based on current resource usage and to check the health of services. This helps provide resiliency to individual machine failures and ensures resource usage is close to identical on all machines.

As an example, here is the CPU usage over a day in one of our data centers where each time series represents one machine and the different colors represent different generations of hardware. Unimog keeps all machines processing traffic and at roughly the same CPU utilization.

Some of our larger data centers have a substantial number of machines, but sometimes we need to reliably run just a single or a few instances of a management service in each location.

There are currently a couple of options to do this, each have their own pros and cons:

1. Deploying the service to all machines in the data center:

   • Pro: it ensures the service’s reliability

   • Con: it unnecessarily uses resources which could have been used to serve customer traffic and is not cost-effective

2. Deploying the service to a static handful of machines in each data center:

   • Pro: it is less wasteful of resources and more cost-effective

   • Con: it runs the risk of service unavailability when those handful of machines unexpectedly fail

A third, more viable option, is to use dynamic task scheduling so that only the right amount of resources are used while ensuring reliability.

Having to pick between two suboptimal reliability model options for management services we want running in each data center was not ideal.

Indeed, some of those services, even though they are not in the request path, are required to continue operating the data center. If the machines running those services become unavailable, in some cases we have to temporarily disable the data center while recovering them. Doing so automatically re-routes users to the next available data center and doesn’t cause disruption. In fact, the entire Cloudflare network is designed to operate with data centers being disabled and brought back automatically. But it’s optimal to route end users to a data center near them so we want to minimize any data center level downtime.

This led us to realize we needed a system to ensure a certain number of instances of a service were running in each data center, regardless of which physical machine ends up running it.

Customer-facing services run on all machines in each data center and do not need to be onboarded to that new system. On the other hand, services currently running on a fixed subset of machines with sub-optimal reliability guarantees and services which don’t need to run on all machines are good candidates for onboarding.

Armed with our set of requirements, we conducted some research on candidate solutions.

While Kubernetes was another option, we decided to use HashiCorp’s Nomad for the following reasons:

• Satisfies our initial requirement, which was reliably running a single instance of a binary with resource isolation in each data center.

• Has few dependencies and a straightforward integration with Consul. Consul is another piece of HashiCorp software we had already deployed in each datacenter. It provides distributed key-value storage and service discovery capabilities.

• Is lightweight (single Go binary), easy to deploy and provision new clusters which is a plus when deploying as many clusters as we have data centers.

• Has a modular task driver (part responsible for executing tasks and providing resource isolation) architecture to support not only containers but also binaries and any custom task driver.

• Is open source and written in Go. We have Go language experience within the team, and Nomad has a responsive community of maintainers on GitHub.

Nomad is split in two different pieces:

1. Nomad Server: instances forming the cluster responsible for scheduling, five per data center to provide sufficient failure tolerance

2. Nomad Client: instances executing the actual tasks, running on all machines in every data center

To guarantee Nomad Server cluster reliability, we deployed instances on machines which are part of different failure domains:

• In different inter-connected physical data centers forming a single location

• In different racks, connected to different switches

• In different multi-node chassis (most of our edge hardware comes in the form of multi-node chassis, one chassis contains four individual servers)

We also added logic to our configuration management tool to ensure we always keep a consistent number of Nomad Server instances regardless of the expansions and decommissions of servers happening on a close to daily basis.

The logic is rather simple, as server expansions and decommissions happen, the Nomad Server role gets redistributed to a new list of machines. Our configuration management tool then ensures that Nomad Server runs on the new machines before turning it off on the old ones.

Additionally, because server expansions and decommissions affect a subset of racks at a time and the Nomad Server role assignment logic provides rack-diversity guarantees, the cluster stays healthy as quorum is kept at all times.

Nomad job files are templated and checked into a git repository. Our configuration management tool then ensures the jobs are scheduled in every data center. From there, Nomad takes over and ensures the jobs are running at all times in each data center.

By exposing rack metadata to each Nomad Client, we are able to make sure each instance of a particular service runs in a different rack and is tied to a different failure domain. This way we make sure that the failure of one rack of servers won’t impact the service health as the service is also running in a different rack, unaffected by the failure.

We achieve this with the following job file constraint:

constraint {
  attribute = "${meta.rack}"
  operator  = "distinct_property"
}

We leveraged Nomad integration with Consul to get Nomad jobs dynamically added to the Consul Service Catalog. This allows us to discover where a particular service is currently running in each data center by querying Consul. Additionally, with the Consul DNS Interface enabled, we can also use DNS-based lookups to target services running on Nomad.

To be able to properly operate as many Nomad clusters as we have data centers, good observability on Nomad clusters and services running on those clusters was essential.

We use Prometheus to scrape Nomad Server and Client instances running in each data center and Alertmanager to alert on key metrics. Using Prometheus metrics, we built a Grafana dashboard to provide visibility on each cluster.

We set up our Prometheus instances to discover services running on Nomad by querying the Consul Service Directory and scraping their metrics periodically using the following Prometheus configuration:

- consul_sd_configs:
  - server: localhost:8500
  job_name: management_service_via_consul
  relabel_configs:
  - action: keep
    regex: management-service
    source_labels:
    - __meta_consul_service

We then use those metrics to create Grafana dashboards and set up alerts for services running on Nomad.

To restrict access to Nomad API endpoints, we enabled mutual TLS authentication and are generating client certificates for each entity interacting with Nomad. This way, only entities with a valid client certificate can interact with Nomad API endpoints in order to schedule jobs or perform any CLI operation.

Deploying a new component always comes with its set of challenges; here is a list of a few hurdles we have had to overcome along the way.

When starting to use the exec driver to run binaries isolated in a chroot environment, we noticed our stateless root partition running on initramfs was not supported as the task would not start and we got this error message in our logs:

Feb 12 19:49:03 machine nomad-client[258433]: 2020-02-12T19:49:03.332Z [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=fa202-63b-33f-924-42cbd5 task=server error="failed to launch command with executor: rpc error: code = Unknown desc = container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:109: jailing process inside rootfs caused \\\"pivot_root invalid argument\\\"\"""

We filed a GitHub issue and submitted a workaround pull request which was promptly reviewed and merged upstream.

In parallel, for maximum isolation security, we worked on enabling pivot_root in our setup by modifying our boot process and other team members developed and proposed a patch to the kernel mailing list to make it easier in the future.

One very important aspect was to make sure the resource usage of tasks running on Nomad would not disrupt other services colocated on the same machine.

Disk space is a shared resource on every machine and being able to set a quota for Nomad was a must. We achieved this by isolating the Nomad data directory to a dedicated fixed-size mount point on each machine. Limiting disk bandwidth and IOPS, however, is not currently supported out of the box by Nomad.

Nomad job files have a resources section where memory and CPU usage can be limited (memory is in MB, cpu is in MHz):

resources {
  memory = 2000
  cpu = 500
}

This uses cgroups under the hood and our testing showed that while memory limits are enforced as one would expect, the CPU limits are soft limits and not enforced as long as there is available CPU on the host machine.

As mentioned above, all machines currently run the same customer-facing workload. Scheduling individual jobs dynamically with Nomad to run on single machines challenges that assumption.

While our dynamic load balancing system, Unimog, balances requests based on resource usage to ensure it is close to identical on all machines, batch type jobs with spiky resource usage can pose a challenge.

We will be paying attention to this as we onboard more services and:

• attempt to limit resource usage spikiness of Nomad jobs with constraints aforementioned

• ensure Unimog adjusts to this batch type workload and does not end up in a positive feedback loop

Now Nomad has been deployed in every data center, we are able to improve the reliability of management services essential to operations by gradually onboarding them. We took a first step by onboarding our reboot and maintenance management service.

In each data center, we run a service which facilitates online unattended rolling reboots and maintenance of machines. This service used to run on a single well-known machine in each data center. This made it vulnerable to single machine failures and when down prevented machines from enabling automatically after a reboot. Therefore, it was a great first service to be onboarded to Nomad to improve its reliability.

We now have a guarantee this service is always running in each data center regardless of individual machine failures. Instead of other machines relying on a well-known address to target this service, they now query Consul DNS and dynamically figure out where the service is running to interact with it.

This is a big improvement in terms of reliability for this service, therefore many more management services are expected to follow in the upcoming months and we are very excited for this to happen.

We covered installing Terraform, provider initialization, storing configuration in git, applying zone settings, and managing rate limits. This post continues the Cloudflare Terraform provider walkthrough with examples of load balancing, page rules, reviewing and rolling back configuration, and importing state.

Before we build on Part 1, let's quickly review what we configured in that post. Because our configuration is in git, we can easily view the current configuration and change history that got us to this point.

$ git log
commit e1c38cf6f4230a48114ce7b747b77d6435d4646c
Author: Me
Date:   Mon Apr 9 12:34:44 2018 -0700

    Step 4 - Update /login rate limit rule from 'simulate' to 'ban'.

commit 0f7e499c70bf5994b5d89120e0449b8545ffdd24
Author: Me
Date:   Mon Apr 9 12:22:43 2018 -0700

    Step 4 - Add rate limiting rule to protect /login.

commit d540600b942cbd89d03db52211698d331f7bd6d7
Author: Me
Date:   Sun Apr 8 22:21:27 2018 -0700

    Step 3 - Enable TLS 1.3, Always Use HTTPS, and SSL Strict mode.

commit 494c6d61b918fce337ca4c0725c9bbc01e00f0b7
Author: Me
Date:   Sun Apr 8 19:58:56 2018 -0700

    Step 2 - Ignore terraform plugin directory and state file.

commit 5acea176050463418f6ac1029674c152e3056bc6
Author: Me
Date:   Sun Apr 8 19:52:13 2018 -0700

    Step 2 - Initial commit with webserver definition.

We'll get into more detail about reviewing and rolling back to prior versions of configuration later in this post, but for now let's review the current version.

In lines 1-4 below, we configured the Cloudflare Terraform provider. Initially we stored our email address and API key in the cloudflare.tf file, but for security reasons we removed them before committing to a git repository.

In lines 6-8, we define a variable that can be interpolated into resources definitions. Terraform can be used to mass configure multiple zones through the use of variables, as we'll explore in a future post.

Lines 10-16 tell Cloudflare to create a DNS A record for www.${var.domain} using IP address 203.0.113.10. Later in this post, we'll explore adding a second web server and load balancing between the two origins.

Lines 18-26 apply zone-wide settings and lines 28-54 define a rate limiting rule to protect against credential stuffing and other brute force attacks.

$ cat -n cloudflare.tf 
     1	provider "cloudflare" {
     2	  # email pulled from $CLOUDFLARE_EMAIL
     3	  # token pulled from $CLOUDFLARE_TOKEN
     4	}
     5	
     6	variable "domain" {
     7	  default = "example.com"
     8	}
     9	
    10	resource "cloudflare_record" "www" {
    11	  domain  = "${var.domain}"
    12	  name    = "www"
    13	  value   = "203.0.113.10"
    14	  type    = "A"
    15	  proxied = true
    16	}
    17	
    18	resource "cloudflare_zone_settings_override" "example-com-settings" {
    19	  name = "${var.domain}"
    20	
    21	  settings {
    22	    tls_1_3 = "on"
    23	    automatic_https_rewrites = "on"
    24	    ssl = "strict"
    25	  }
    26	}
    27	
    28	resource "cloudflare_rate_limit" "login-limit" {
    29	  zone = "${var.domain}"
    30	
    31	  threshold = 5
    32	  period = 60
    33	  match {
    34	    request {
    35	      url_pattern = "${var.domain}/login"
    36	      schemes = ["HTTP", "HTTPS"]
    37	      methods = ["POST"]
    38	    }
    39	    response {
    40	      statuses = [401, 403]
    41	      origin_traffic = true
    42	    }
    43	  }
    44	  action {
    45	    mode = "ban"
    46	    timeout = 300
    47	    response {
    48	      content_type = "text/plain"
    49	      body = "You have failed to login 5 times in a 60 second period and will be blocked from attempting to login again for the next 5 minutes."
    50	    }
    51	  }
    52	  disabled = false
    53	  description = "Block failed login attempts (5 in 1 min) for 5 minutes."
    54	}

Thanks to the rate limiting set up in part 1, our login page is protected against credential brute force attacks. Now it's time to focus on performance and reliability. Imagine organic traffic has grown for your web server, and this traffic is increasingly global. It’s time to spread these requests to your origin over multiple data centers.

Below we'll add a second origin for some basic round robining, and then use the Cloudflare Load Balancing product to fail traffic over as needed. We'll then enhance our load balancing configuration through the use of "geo steering" to serve results from an origin server that is geographically closest to your end users.

To get started, we'll add a DNS record for a second web server, which is located in Asia. The IP address for this server is 198.51.100.15.

$ git checkout -b step5-loadbalance
Switched to a new branch 'step5-loadbalance'

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_record" "www-asia" {
  domain  = "${var.domain}"
  name    = "www"
  value   = "198.51.100.15"
  type    = "A"
  proxied = true
}
EOF

Note that while the name of the resource is different as Terraform resources of the same type must be uniquely named, the DNS name, i.e., what your customers will type in their browser, is the same: "www".

Below we'll check the terraform plan, merge and apply the changes.

$ terraform plan | grep -v "<computed>"
...
Terraform will perform the following actions:

  + cloudflare_record.www-asia
      domain:      "example.com"
      name:        "www"
      proxied:     "true"
      type:        "A"
      value:       "198.51.100.15"


Plan: 1 to add, 0 to change, 0 to destroy.

$ git add cloudflare.tf
$ git commit -m "Step 5 - Add additional 'www' DNS record for Asia data center."
[step5-loadbalance 6761a4f] Step 5 - Add additional 'www' DNS record for Asia data center.
 1 file changed, 7 insertions(+)

$ git checkout master
Switched to branch 'master'

$ git merge step5-loadbalance 
Updating e1c38cf..6761a4f
Fast-forward
 cloudflare.tf | 7 +++++++
 1 file changed, 7 insertions(+)

Let's add the second DNS record for www.example.com:

$ terraform apply --auto-approve
...
cloudflare_record.www-asia: Creating...
  created_on:  "" => "<computed>"
  domain:      "" => "example.com"
  hostname:    "" => "<computed>"
  metadata.%:  "" => "<computed>"
  modified_on: "" => "<computed>"
  name:        "" => "www"
  proxiable:   "" => "<computed>"
  proxied:     "" => "true"
  ttl:         "" => "<computed>"
  type:        "" => "A"
  value:       "" => "198.51.100.15"
  zone_id:     "" => "<computed>"
cloudflare_record.www-asia: Creation complete after 1s (ID: fda39d8c9bf909132e82a36bab992864)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

With the second DNS record in place, let's try making some requests to see where the traffic is served from:

$ curl https://www.example.com
Hello, this is 203.0.113.10!

$ curl https://www.example.com
Hello, this is 203.0.113.10!

$ curl https://www.example.com
Hello, this is 198.51.100.15!

$ curl https://www.example.com
Hello, this is 203.0.113.10!

As you can see above, there is no discernible pattern for which origin receives the request. When Cloudflare connects to an origin with multiple DNS records, one of the IP addresses is selected at random. If both of these IPs are in the same data center and sessions can be shared (i.e., it doesn't matter if the same user hops between origin servers), this may work fine. However, for anything more complicated such as origins in different geographies or active health checks, you're going to want to use Cloudflare's Load Balancing product.

Before proceeding, make sure that your account is enabled for Load Balancing. If you're on an Enterprise plan, you should ask your Customer Success Manager to do this; otherwise, you can subscribe to Load Balancing within the Cloudflare Dashboard.

As described in the load balancing tutorial on the Cloudflare Support site, you will need to do three things:

i. Create a monitor to run health checks against your origin servers. ii. Create a pool of one or more origin servers that will receive load balanced traffic. iii. Create a load balancer with an external hostname, e.g., www.example.com, and one or more pools. iv. Preview and merge the changes. v. Test the changes.

To monitor our origins we're going to create a basic health check that makes a GET request to each origin on the URL https://www.example.com. If the origin returns the 200/OK status code within 5 seconds, we'll consider it healthy. If it fails to do so three (3) times in a row, we'll consider it unhealthy. This health check will be run once per minute from several regions, and send an email notification to you@example.com if any failures are detected.

$ git checkout step5-loadbalance
Switched to branch 'step5-loadbalance'

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_load_balancer_monitor" "get-root-https" {
  expected_body = "alive"
  expected_codes = "200"
  method = "GET"
  timeout = 5
  path = "/"
  interval = 60
  retries = 2
  check_regions = ["WNAM", "ENAM", "WEU", "EEU", "SEAS", "NEAS"]
  description = "GET / over HTTPS - expect 200"
}
EOF

We will call our pool "www-servers” and add two origins to it: www-us (203.0.113.10) and www-asia (198.51.100.15). For now, we'll skip any sort of geo routing.

Note that we reference the monitor we added in the last step. When applying this confirmation, Terraform will figure out that it first needs to create the monitor so that it can look up the ID and provide to the pool we wish to create.

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_load_balancer_pool" "www-servers" {
  name = "www-servers"
  monitor = "${cloudflare_load_balancer_monitor.get-root-https.id}"
  origins {
    name = "www-us"
    address = "203.0.113.10"
  }
  origins {
    name = "www-asia"
    address = "198.51.100.15"
  }
  description = "www origins"
  enabled = true
  minimum_origins = 1
  notification_email = "you@example.com"
}
EOF

Note that when you create a load balancer (LB), it will replace any existing DNS records with the same name. For example, when we create the "www.example.com" LB below, it will supersede the two www DNS records that you have previously defined. One benefit of leaving these DNS records in place is that if you temporarily disable load balancing, connections to this hostname will still be possible as shown above.

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_load_balancer" "www-lb" {
  zone = "example.com"
  name = "www-lb"
  default_pool_ids = ["${cloudflare_load_balancer_pool.www-servers.id}"]
  fallback_pool_id = "${cloudflare_load_balancer_pool.www-servers.id}"
  description = "example load balancer"
  proxied = true
}
EOF

As usual, we take a look at the proposed plan before we apply any changes:

$ terraform plan
...
Terraform will perform the following actions:

  + cloudflare_load_balancer.www-lb
      id:                         <computed>
      created_on:                 <computed>
      default_pool_ids.#:         <computed>
      description:                "example load balancer"
      fallback_pool_id:           "${cloudflare_load_balancer_pool.www-servers.id}"
      modified_on:                <computed>
      name:                       "www-lb"
      pop_pools.#:                <computed>
      proxied:                    "true"
      region_pools.#:             <computed>
      ttl:                        <computed>
      zone:                       "example.com"
      zone_id:                    <computed>

  + cloudflare_load_balancer_monitor.get-root-https
      id:                         <computed>
      created_on:                 <computed>
      description:                "GET / over HTTPS - expect 200"
      expected_body:              "alive"
      expected_codes:             "200"
      interval:                   "60"
      method:                     "GET"
      modified_on:                <computed>
      path:                       "/"
      retries:                    "2"
      timeout:                    "5"
      type:                       "http"

  + cloudflare_load_balancer_pool.www-servers
      id:                         <computed>
      check_regions.#:            "6"
      check_regions.1151265357:   "SEAS"
      check_regions.1997072153:   "WEU"
      check_regions.2367191053:   "EEU"
      check_regions.2826842289:   "ENAM"
      check_regions.2992567379:   "WNAM"
      check_regions.3706632574:   "NEAS"
      created_on:                 <computed>
      description:                "www origins"
      enabled:                    "true"
      minimum_origins:            "1"
      modified_on:                <computed>
      monitor:                    "${cloudflare_load_balancer_monitor.get-root-https.id}"
      name:                       "www-servers"
      notification_email:         "you@example.com"
      origins.#:                  "2"
      origins.3039426352.address: "198.51.100.15"
      origins.3039426352.enabled: "true"
      origins.3039426352.name:    "www-asia"
      origins.4241861547.address: "203.0.113.10"
      origins.4241861547.enabled: "true"
      origins.4241861547.name:    "www-us"


Plan: 3 to add, 0 to change, 0 to destroy.

The plan looks good so let's go ahead, merge it in, and apply it.

$ git add cloudflare.tf
$ git commit -m "Step 5 - Create load balancer (LB) monitor, LB pool, and LB."
[step5-loadbalance bc9aa9a] Step 5 - Create load balancer (LB) monitor, LB pool, and LB.
 1 file changed, 35 insertions(+)

$ terraform apply --auto-approve
...
cloudflare_load_balancer_monitor.get-root-https: Creating...
  created_on:     "" => "<computed>"
  description:    "" => "GET / over HTTPS - expect 200"
  expected_body:  "" => "alive"
  expected_codes: "" => "200"
  interval:       "" => "60"
  method:         "" => "GET"
  modified_on:    "" => "<computed>"
  path:           "" => "/"
  retries:        "" => "2"
  timeout:        "" => "5"
  type:           "" => "http"
cloudflare_load_balancer_monitor.get-root-https: Creation complete after 1s (ID: 4238142473fcd48e89ef1964be72e3e0)
cloudflare_load_balancer_pool.www-servers: Creating...
  check_regions.#:            "" => "6"
  check_regions.1151265357:   "" => "SEAS"
  check_regions.1997072153:   "" => "WEU"
  check_regions.2367191053:   "" => "EEU"
  check_regions.2826842289:   "" => "ENAM"
  check_regions.2992567379:   "" => "WNAM"
  check_regions.3706632574:   "" => "NEAS"
  created_on:                 "" => "<computed>"
  description:                "" => "www origins"
  enabled:                    "" => "true"
  minimum_origins:            "" => "1"
  modified_on:                "" => "<computed>"
  monitor:                    "" => "4238142473fcd48e89ef1964be72e3e0"
  name:                       "" => "www-servers"
  notification_email:         "" => "you@example.com"
  origins.#:                  "" => "2"
  origins.3039426352.address: "" => "198.51.100.15"
  origins.3039426352.enabled: "" => "true"
  origins.3039426352.name:    "" => "www-asia"
  origins.4241861547.address: "" => "203.0.113.10"
  origins.4241861547.enabled: "" => "true"
  origins.4241861547.name:    "" => "www-us"
cloudflare_load_balancer_pool.www-servers: Creation complete after 0s (ID: 906d2a7521634783f4a96c062eeecc6d)
cloudflare_load_balancer.www-lb: Creating...
  created_on:         "" => "<computed>"
  default_pool_ids.#: "" => "1"
  default_pool_ids.0: "" => "906d2a7521634783f4a96c062eeecc6d"
  description:        "" => "example load balancer"
  fallback_pool_id:   "" => "906d2a7521634783f4a96c062eeecc6d"
  modified_on:        "" => "<computed>"
  name:               "" => "www-lb"
  pop_pools.#:        "" => "<computed>"
  proxied:            "" => "true"
  region_pools.#:     "" => "<computed>"
  ttl:                "" => "<computed>"
  zone:               "" => "example.com"
  zone_id:            "" => "<computed>"
cloudflare_load_balancer.www-lb: Creation complete after 1s (ID: cb94f53f150e5c1a65a07e43c5d4cac4)

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

With load balancing in place, let's run those curl requests again to see where the traffic is served from:

$ for i in {1..4}; do curl https://www.example.com && sleep 5; done
Hello, this is 198.51.100.15!

Hello, this is 203.0.113.10!

Hello, this is 198.51.100.15!

Hello, this is 203.0.113.10!

Great, we're now seeing each request load balanced evenly across the two origins we defined.

Earlier we configured zone settings that apply to all of example.com. Now we're going to add an exception to these settings by using Page Rules.

Specifically, we're going to turn increase the security level for a URL we know is expensive to render (and cannot be cached): https://www.example.com/expensive-db-call. Additionally, we're going to add a redirect from the previous URL we used to host this page.

As usual, we'll create a new branch and append our configuration.

$ git checkout -b step6-pagerule
Switched to a new branch 'step6-pagerule'

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_page_rule" "increase-security-on-expensive-page" {
  zone = "${var.domain}"
  target = "www.${var.domain}/expensive-db-call"
  priority = 10

  actions = {
    security_level = "under_attack",
  }
}

resource "cloudflare_page_rule" "redirect-to-new-db-page" {
  zone = "${var.domain}"
  target = "www.${var.domain}/old-location.php"
  priority = 10

  actions = {
    forwarding_url {
      url = "https://www.${var.domain}/expensive-db-call"
      status_code = 301
    }
  }
}
EOF

You know the drill: preview the changes Terraform is going to make and then merge them into the master branch.

$ terraform plan
...
Terraform will perform the following actions:

  + cloudflare_page_rule.increase-security-on-expensive-page
      id:                                     <computed>
      actions.#:                              "1"
      actions.0.always_use_https:             "false"
      actions.0.disable_apps:                 "false"
      actions.0.disable_performance:          "false"
      actions.0.disable_security:             "false"
      actions.0.security_level:               "under_attack"
      priority:                               "10"
      status:                                 "active"
      target:                                 "www.example.com/expensive-db-call"
      zone:                                   "example.com"
      zone_id:                                <computed>

  + cloudflare_page_rule.redirect-to-new-db-page
      id:                                     <computed>
      actions.#:                              "1"
      actions.0.always_use_https:             "false"
      actions.0.disable_apps:                 "false"
      actions.0.disable_performance:          "false"
      actions.0.disable_security:             "false"
      actions.0.forwarding_url.#:             "1"
      actions.0.forwarding_url.0.status_code: "301"
      actions.0.forwarding_url.0.url:         "https://www.example.com/expensive-db-call"
      priority:                               "10"
      status:                                 "active"
      target:                                 "www.example.com/old-location.php"
      zone:                                   "example.com"
      zone_id:                                <computed>


Plan: 2 to add, 0 to change, 0 to destroy.

$ git add cloudflare.tf

$ git commit -m "Step 6 - Add two Page Rules."
[step6-pagerule d4fec16] Step 6 - Add two Page Rules.
 1 file changed, 23 insertions(+)

$ git checkout master
Switched to branch 'master'

$ git merge step6-pagerule 
Updating 7a2ac34..d4fec16
Fast-forward
 cloudflare.tf | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
3. Apply And Verify The Changes
First we'll test requesting the (now missing) old location of the expensive-to-render page.

$ curl -vso /dev/null https://www.example.com/old-location.php 2>&1 | grep "< HTTP\|Location"
< HTTP/1.1 404 Not Found

As expected, it can't be found. Let's apply the Page Rules, including the redirect that should fix this error.

$ terraform apply --auto-approve
...
cloudflare_page_rule.redirect-to-new-db-page: Creating...
  actions.#:                              "0" => "1"
  actions.0.always_use_https:             "" => "false"
  actions.0.disable_apps:                 "" => "false"
  actions.0.disable_performance:          "" => "false"
  actions.0.disable_security:             "" => "false"
  actions.0.forwarding_url.#:             "0" => "1"
  actions.0.forwarding_url.0.status_code: "" => "301"
  actions.0.forwarding_url.0.url:         "" => "https://www.example.com/expensive-db-call"
  priority:                               "" => "10"
  status:                                 "" => "active"
  target:                                 "" => "www.example.com/old-location.php"
  zone:                                   "" => "example.com"
  zone_id:                                "" => "<computed>"
cloudflare_page_rule.increase-security-on-expensive-page: Creating...
  actions.#:                     "0" => "1"
  actions.0.always_use_https:    "" => "false"
  actions.0.disable_apps:        "" => "false"
  actions.0.disable_performance: "" => "false"
  actions.0.disable_security:    "" => "false"
  actions.0.security_level:      "" => "under_attack"
  priority:                      "" => "10"
  status:                        "" => "active"
  target:                        "" => "www.example.com/expensive-db-call"
  zone:                          "" => "example.com"
  zone_id:                       "" => "<computed>"
cloudflare_page_rule.redirect-to-new-db-page: Creation complete after 3s (ID: c5c40ff2dc12416b5fe4d0541980c591)
cloudflare_page_rule.increase-security-on-expensive-page: Creation complete after 6s (ID: 1c13fdb84710c4cc8b11daf7ffcca449)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

With the Page Rules in place, let's try that call again, along with the I'm Under Attack Mode test:

$ curl -vso /dev/null https://www.example.com/old-location.php 2>&1 | grep "< HTTP\|Location"
< HTTP/1.1 301 Moved Permanently
< Location: https://www.upinatoms.com/expensive-db-call

$ curl -vso /dev/null https://www.upinatoms.com/expensive-db-call 2>&1 | grep "< HTTP"
< HTTP/1.1 503 Service Temporarily Unavailable

Great, they work as expected! In the first case, the Cloudflare edge responds with a 301 redirecting the browser to the new location. In the second case it initially responds with a 503 (as it is consistent with the "I Am Under Attack” mode).

We've come a long way! Now it's time to tear it all down. Well, maybe just part of it.

Sometimes when you deploy configuration changes you later determine that they need to be rolled back. You could be performance testing a new configuration and want to revert to your previous configuration when done testing. Or maybe you fat-fingered an IP address and brought your entire site down (#hugops).

Either way, if you've determined you want to revert your configuration, all you need to do is check the desired branch out and ask Terraform to move your Cloudflare settings back in time. And note that if you accidentally brought your site down you should consider establishing a good strategy for peer reviewing pull requests (rather than merging directly to primary, as I do here for brevity)!

Before we figure out how far back in time we want to roll back, let's take a look at our (git) versioned history.

$ git log
commit d4fec164581bec44684a4d59bb80aec1f1da5a6e
Author: Me
Date:   Wed Apr 18 22:04:52 2018 -0700

    Step 6 - Add two Page Rules.

commit bc9aa9a465a4c8d6deeaa0491814c9f364e9aa8a
Author: Me
Date:   Sun Apr 15 23:58:35 2018 -0700

    Step 5 - Create load balancer (LB) monitor, LB pool, and LB.

commit 6761a4f754e77322629ba4e90a90a3defa1fd4b6
Author: Me
Date:   Wed Apr 11 11:20:25 2018 -0700

    Step 5 - Add additional 'www' DNS record for Asia data center.

commit e1c38cf6f4230a48114ce7b747b77d6435d4646c
Author: Me
Date:   Mon Apr 9 12:34:44 2018 -0700

    Step 4 - Update /login rate limit rule from 'simulate' to 'ban'.

commit 0f7e499c70bf5994b5d89120e0449b8545ffdd24
Author: Me
Date:   Mon Apr 9 12:22:43 2018 -0700

    Step 4 - Add rate limiting rule to protect /login.

commit d540600b942cbd89d03db52211698d331f7bd6d7
Author: Me
Date:   Sun Apr 8 22:21:27 2018 -0700

    Step 3 - Enable TLS 1.3, Always Use HTTPS, and SSL Strict mode.

commit 494c6d61b918fce337ca4c0725c9bbc01e00f0b7
Author: Me
Date:   Sun Apr 8 19:58:56 2018 -0700

    Step 2 - Ignore terraform plugin directory and state file.

commit 5acea176050463418f6ac1029674c152e3056bc6
Author: Me
Date:   Sun Apr 8 19:52:13 2018 -0700

    Step 2 - Initial commit with webserver definition.

Another nice benefit of storing your Cloudflare configuration in git is that you can see who made the change, as well as who reviewed and approved the change (assuming you're peer reviewing pull requests).

To begin with, let's see what the last change we made was.

$ git show
commit d4fec164581bec44684a4d59bb80aec1f1da5a6e
Author: Me
Date:   Wed Apr 18 22:04:52 2018 -0700

    Step 6 - Add two Page Rules.

diff --git a/cloudflare.tf b/cloudflare.tf
index 0b39450..ef11d8a 100644
--- a/cloudflare.tf
+++ b/cloudflare.tf
@@ -94,3 +94,26 @@ resource "cloudflare_load_balancer" "www-lb" {
   description = "example load balancer"
   proxied = true
 }
+
+resource "cloudflare_page_rule" "increase-security-on-expensive-page" {
+  zone = "${var.domain}"
+  target = "www.${var.domain}/expensive-db-call"
+  priority = 10
+
+  actions = {
+    security_level = "under_attack",
+  }
+}
+
+resource "cloudflare_page_rule" "redirect-to-new-db-page" {
+  zone = "${var.domain}"
+  target = "www.${var.domain}/old-location.php"
+  priority = 10
+
+  actions = {
+    forwarding_url {
+      url = "https://${var.domain}/expensive-db-call"
+      status_code = 301
+    }
+  }
+}

Now let's look at the past few changes:

$ git log -p -3

... 
// page rule config from above
...

commit bc9aa9a465a4c8d6deeaa0491814c9f364e9aa8a
Author: Me
Date:   Sun Apr 15 23:58:35 2018 -0700

    Step 5 - Create load balancer (LB) monitor, LB pool, and LB.

diff --git a/cloudflare.tf b/cloudflare.tf
index b92cb6f..195b646 100644
--- a/cloudflare.tf
+++ b/cloudflare.tf
@@ -59,3 +59,38 @@ resource "cloudflare_record" "www-asia" {
   type    = "A"
   proxied = true
 }
+resource "cloudflare_load_balancer_monitor" "get-root-https" {
+  expected_body = "alive"
+  expected_codes = "200"
+  method = "GET"
+  timeout = 5
+  path = "/"
+  interval = 60
+  retries = 2
+  description = "GET / over HTTPS - expect 200"
+}
+resource "cloudflare_load_balancer_pool" "www-servers" {
+  name = "www-servers"
+  monitor = "${cloudflare_load_balancer_monitor.get-root-https.id}"
+  check_regions = ["WNAM", "ENAM", "WEU", "EEU", "SEAS", "NEAS"]
+  origins {
+    name = "www-us"
+    address = "203.0.113.10"
+  }
+  origins {
+    name = "www-asia"
+    address = "198.51.100.15"
+  }
+  description = "www origins"
+  enabled = true
+  minimum_origins = 1
+  notification_email = "you@example.com"
+}
+resource "cloudflare_load_balancer" "www-lb" {
+  zone = "${var.domain}"
+  name = "www-lb"
+  default_pool_ids = ["${cloudflare_load_balancer_pool.www-servers.id}"]
+  fallback_pool_id = "${cloudflare_load_balancer_pool.www-servers.id}"
+  description = "example load balancer"
+  proxied = true
+}

commit 6761a4f754e77322629ba4e90a90a3defa1fd4b6
Author: Me
Date:   Wed Apr 11 11:20:25 2018 -0700

    Step 5 - Add additional 'www' DNS record for Asia data center.

diff --git a/cloudflare.tf b/cloudflare.tf
index 9f25a0c..b92cb6f 100644
--- a/cloudflare.tf
+++ b/cloudflare.tf
@@ -52,3 +52,10 @@ resource "cloudflare_rate_limit" "login-limit" {
   disabled = false
   description = "Block failed login attempts (5 in 1 min) for 5 minutes."
 }
+resource "cloudflare_record" "www-asia" {
+  domain  = "${var.domain}"
+  name    = "www"
+  value   = "198.51.100.15"
+  type    = "A"
+  proxied = true
+}

Imagine that shortly after we deployed the Page Rules, we got a call from the Product team that manages this page: "The URL was only being used by one customer and is no longer needed, let's drop the security setting and redirect.”

While you could always edit the config file directly and delete those entries, it's easier to let git do it for us. To begin with, let's ask git to revert the last commit (without rewriting history).

$ git revert HEAD~1..HEAD
[master f9a6f7d] Revert "Step 6 - Bug fix."
 1 file changed, 1 insertion(+), 1 deletion(-)

$ git log -2
commit f9a6f7db72ea1437e146050a5e7556052ecc9a1a
Author: Me
Date:   Wed Apr 18 23:28:09 2018 -0700

    Revert "Step 6 - Add two Page Rules."
    
    This reverts commit d4fec164581bec44684a4d59bb80aec1f1da5a6e.

commit d4fec164581bec44684a4d59bb80aec1f1da5a6e
Author: Me
Date:   Wed Apr 18 22:04:52 2018 -0700

    Step 6 - Add two Page Rules.

As expected, Terraform is indicating it will remove the two Page Rules we just created.

$ terraform plan
...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - cloudflare_page_rule.increase-security-on-expensive-page

  - cloudflare_page_rule.redirect-to-new-db-page


Plan: 0 to add, 0 to change, 2 to destroy.

The changes look good, so let's ask Terraform to roll our Cloudflare configuration back.

$ terraform apply --auto-approve
...
cloudflare_page_rule.redirect-to-new-db-page: Destroying... (ID: c5c40ff2dc12416b5fe4d0541980c591)
cloudflare_page_rule.increase-security-on-expensive-page: Destroying... (ID: 1c13fdb84710c4cc8b11daf7ffcca449)
cloudflare_page_rule.increase-security-on-expensive-page: Destruction complete after 0s
cloudflare_page_rule.redirect-to-new-db-page: Destruction complete after 1s

Apply complete! Resources: 0 added, 0 changed, 2 destroyed.

Two resources destroyed, as expected. We've rolled back to the previous version.

An important point to understand about Terraform is that it is only able to manage configuration that it created, or was explicitly told about after the fact. The reason for this limitation is that Terraform relies on a local state file that maps the resource names defined in your configuration file, e.g., cloudflare_load_balancer.www-lb to the IDs generated by Cloudflare's API.

When Terraform makes calls to Cloudflare's API to create new resources, it persists those IDs to a state file; by default, the terraform.tfstate file in your directory is used, but this can be a remote location as will be discussed in a future blog post. These IDs are later looked up and refreshed when you call terraform plan and terraform apply.

If you've configured Cloudflare through other means, e.g., by logging into the Cloudflare Dashboard or making curl calls to api.cloudflare.com, Terraform does not (yet) have these resource IDs in the state file. To manage this preexisting configuration you will need to first i) reproduce the configuration in your config file and; ii) import resources one-by-one by providing their IDs and resource names.

Before importing resources created by other means, let's take a look at how an existing DNS records is represented in the state file.

$ cat terraform.tfstate | jq '.modules[].resources["cloudflare_record.www"]'
{
  "type": "cloudflare_record",
  "depends_on": [],
  "primary": {
    "id": "c38d3103767284e7cd14d5dad3ab8669",
    "attributes": {
      "created_on": "2018-04-08T00:37:33.76321Z",
      "data.%": "0",
      "domain": "example.com",
      "hostname": "www.example.com",
      "id": "c38d3103767284e7cd14d5dad3ab8669",
      "metadata.%": "2",
      "metadata.auto_added": "false",
      "metadata.managed_by_apps": "false",
      "modified_on": "2018-04-08T00:37:33.76321Z",
      "name": "www",
      "priority": "0",
      "proxiable": "true",
      "proxied": "true",
      "ttl": "1",
      "type": "A",
      "value": "203.0.113.10",
      "zone_id": "e2e6491340be87a3726f91fc4148b126"
    },
    "meta": {
      "schema_version": "1"
    },
    "tainted": false
  },
  "deposed": [],
  "provider": "provider.cloudflare"
}

As shown in the above JSON, the cloudflare_record resource named "www" has a unique ID of c38d3103767284e7cd14d5dad3ab8669. This ID is what gets interpolated into the API call that Terraform makes to Cloudflare to pull the latest configuration during the plan stage, e.g.,

GET https://api.cloudflare.com/client/v4/zones/:zone_id/dns_records/c38d3103767284e7cd14d5dad3ab8669

To import an existing record, e.g., another DNS record, you need two things:

1. The unique identifier that Cloudflare uses to identify the record

2. The resource name to which you wish to map this identifier

We start by making an API call to Cloudflare to enumerate the DNS records in our account. The output below has been filtered to show only MX records, as these are what we'll be importing.

$ curl https://api.cloudflare.com/client/v4/zones/$EXAMPLE_COM_ZONEID \
       -H "X-Auth-Email: you@example.com" -H "X-Auth-Key: $CF_API_KEY" \
       -H "Content-Type: application/json" | jq .

{
  "result": [
    {
      "id": "8ea8c36c8530ee01068c65c0ddc4379b",
      "type": "MX",
      "name": "example.com",
      "content": "alt1.aspmx.l.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 15,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:11:50.163221Z",
      "created_on": "2016-11-06T01:11:50.163221Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },
    {
      "id": "ad0e9ff2479b13c5fbde77a02ea6fa2c",
      "type": "MX",
      "name": "example.com",
      "content": "alt2.aspmx.l.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 15,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:12:00.915649Z",
      "created_on": "2016-11-06T01:12:00.915649Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },
    {
      "id": "ad6ee69519cd02a0155a56b6d64c278a",
      "type": "MX",
      "name": "example.com",
      "content": "alt3.aspmx.l.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 20,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:12:12.899684Z",
      "created_on": "2016-11-06T01:12:12.899684Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },
    {
      "id": "baf6655f33738b7fd902630858878206",
      "type": "MX",
      "name": "example.com",
      "content": "alt4.aspmx.l.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 20,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:12:22.599272Z",
      "created_on": "2016-11-06T01:12:22.599272Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },
    {
      "id": "a96d72b3c6afe3077f9e9c677fb0a556",
      "type": "MX",
      "name": "example.com",
      "content": "aspmx.lo.google.com",
      "proxiable": false,
      "proxied": false,
      "ttl": 1,
      "priority": 10,
      "locked": false,
      "zone_id": "e2e6491340be87a3726f91fc4148b126",
      "zone_name": "example.com",
      "modified_on": "2016-11-06T01:11:27.700386Z",
      "created_on": "2016-11-06T01:11:27.700386Z",
      "meta": {
        "auto_added": false,
        "managed_by_apps": false
      }
    },

    ...
  ]
}

In the previous step, we found 5 MX records that we wish to add.

Before importing, we need to create Terraform configuration and give each record a unique name that can be referenced during the import.

$ cat >> cloudflare.tf <<'EOF'
resource "cloudflare_record" "mx-10" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "aspmx.lo.google.com"
  type    = "MX"
  priority = "10"
}
resource "cloudflare_record" "mx-15-1" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "alt1.aspmx.l.google.com"
  type    = "MX"
  priority = "15"
}
resource "cloudflare_record" "mx-15-2" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "alt2.aspmx.l.google.com"
  type    = "MX"
  priority = "15"
}
resource "cloudflare_record" "mx-20-1" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "alt3.aspmx.l.google.com"
  type    = "MX"
  priority = "20"
}
resource "cloudflare_record" "mx-20-2" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "alt3.aspmx.l.google.com"
  type    = "MX"
  priority = "20"
}
EOF

Before we import the records, let's look at what would happen if we ran a terraform apply.

$ terraform plan | grep Plan
Plan: 5 to add, 0 to change, 0 to destroy.

Terraform does not know that these records already exist on Cloudflare, so until the import completes it will attempt to create them as new records. Below we import them one-by-one, specifying the name of the resource and the zoneName/resourceID returned by api.cloudflare.com.

$ terraform import cloudflare_record.mx-10 example.com/a96d72b3c6afe3077f9e9c677fb0a556
cloudflare_record.mx-10: Importing from ID "example.com/a96d72b3c6afe3077f9e9c677fb0a556"...
cloudflare_record.mx-10: Import complete!
  Imported cloudflare_record (ID: a96d72b3c6afe3077f9e9c677fb0a556)
cloudflare_record.mx-10: Refreshing state... (ID: a96d72b3c6afe3077f9e9c677fb0a556)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ terraform import cloudflare_record.mx-15-1 example.com/8ea8c36c8530ee01068c65c0ddc4379b
cloudflare_record.mx-15-1: Importing from ID "example.com/8ea8c36c8530ee01068c65c0ddc4379b"...
cloudflare_record.mx-15-1: Import complete!
  Imported cloudflare_record (ID: 8ea8c36c8530ee01068c65c0ddc4379b)
cloudflare_record.mx-15-1: Refreshing state... (ID: 8ea8c36c8530ee01068c65c0ddc4379b)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ terraform import cloudflare_record.mx-15-2 example.com/ad0e9ff2479b13c5fbde77a02ea6fa2c
cloudflare_record.mx-15-2: Importing from ID "example.com/ad0e9ff2479b13c5fbde77a02ea6fa2c"...
cloudflare_record.mx-15-2: Import complete!
  Imported cloudflare_record (ID: ad0e9ff2479b13c5fbde77a02ea6fa2c)
cloudflare_record.mx-15-2: Refreshing state... (ID: ad0e9ff2479b13c5fbde77a02ea6fa2c)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ terraform import cloudflare_record.mx-20-1 example.com/ad6ee69519cd02a0155a56b6d64c278a
cloudflare_record.mx-20-1: Importing from ID "example.com/ad6ee69519cd02a0155a56b6d64c278a"...
cloudflare_record.mx-20-1: Import complete!
  Imported cloudflare_record (ID: ad6ee69519cd02a0155a56b6d64c278a)
cloudflare_record.mx-20-1: Refreshing state... (ID: ad6ee69519cd02a0155a56b6d64c278a)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ terraform import cloudflare_record.mx-20-2 example.com/baf6655f33738b7fd902630858878206
cloudflare_record.mx-20-2: Importing from ID "example.com/baf6655f33738b7fd902630858878206"...
cloudflare_record.mx-20-2: Import complete!
  Imported cloudflare_record (ID: baf6655f33738b7fd902630858878206)
cloudflare_record.mx-20-2: Refreshing state... (ID: baf6655f33738b7fd902630858878206)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Now when we run terraform plan it no longer wants to (re-)create the above records.

$ terraform plan | grep changes
No changes. Infrastructure is up-to-date.

That's it for today! We covered the Load Balancing and Page Rules resources, as well as demonstrated how to review and roll back configuration changes, and import state.

Stay tuned for future Terraform blog posts, where we plan to show how to manage state effectively in a group setting, go multi-cloud with Terraform, and much more.

As a Product Manager at Cloudflare, I spend quite a bit of my time talking to customers. One of the most common topics I'm asked about is configuration management. Developers want to know how they can write code to manage their Cloudflare config, without interacting with our APIs or UI directly.

Following best practices in software development, they want to store configuration in their own source code repository (be it GitHub or otherwise), institute a change management process that includes code review, and be able to track their configuration versions and history over time. Additionally, they want the ability to quickly and easily roll back changes when required.

When I first spoke with our engineering teams about these requirements, they gave me the best answer a Product Manager could hope to hear: there's already an open source tool out there that does all of that (and more), with a strong community and plugin system to boot—it's called Terraform.

This blog post is about getting started using Terraform with Cloudflare and the new version 1.0 of our Terraform provider. A "provider" is simply a plugin that knows how to talk to a specific set of APIs—in this case, Cloudflare, but there are also providers available for AWS, Azure, Google Cloud, Kubernetes, VMware, and many more services. Today's release extends our existing provider that previously only supported DNS records with support for Zone Settings, Rate Limiting, Load Balancing, and Page Rules.

Before we jump into some real-world examples of using Terraform with Cloudflare, here is a set of diagrams that depicts the paradigm shift.

Before Terraform, you needed to learn how to use the configuration interfaces or APIs of each cloud and edge provider, e.g., Google Cloud and Cloudflare below. Additionally, your ability to store your configuration in your own source code control system depends on vendor-specific configuration export mechanisms (which may or may not exist).

With Terraform, you can store and version your configuration in GitHub (or your source code control system of choice). Once you learn Terraform's configuration syntax, you don't need to bother learning how to use providers' UIs or APIs—you just tell Terraform what you want and it figures out the rest.

The installation process for Terraform is extremely simple as it ships as a single binary file. Official instructions for installing Terraform can be found here, and for purposes of this example we'll show to do so on a macOS using Homebrew:

$ brew install terraform
==> Downloading https://homebrew.bintray.com/bottles/terraform-0.11.7.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring terraform-0.11.7.sierra.bottle.tar.gz
?  /usr/local/Cellar/terraform/0.11.7: 6 files, 80.2MB

$ terraform version
Terraform v0.11.7

The following instructions are adapted from the Cloudflare Developers - Terraform documentation site, which includes a full tutorial and coverage of advanced topics.

If you're interested in seeing how to use a specific Terraform resource or technique, click on one of the following anchor links:

• Installing Terraform

• Hello, world!

• Tracking Change History

• Applying Zone Settings

• Managing Rate Limits

• Load Balancing Resource (next post!)

• Page Rules Resource (next post!)

• Reviewing and Rolling Back Configuration (next post!)

• Importing Existing State and Configuration (next post!)

Now that Terraform is installed, it's time to start using it. Let's assume you have a web server for your domain that's accessible on 203.0.113.10. You just signed up your domain, example.com, on Cloudflare and want to manage everything with Terraform.

First we'll create a initial Terraform config file. Any files ending in .tf will be processed by Terraform. As you configuration gets more complex you'll want to split the config into separate files and modules, but for now we'll proceed with a single file:

$ cat > cloudflare.tf <<'EOF'
provider "cloudflare" {
  email = "you@example.com"
  token = "your-api-key"
}

variable "domain" {
  default = "example.com"
}

resource "cloudflare_record" "www" {
  domain  = "${var.domain}"
  name    = "www"
  value   = "203.0.113.10"
  type    = "A"
  proxied = true
}
EOF

Now that you've created your basic configuration in HCL let's initialize Terraform and ask it to apply the configuration to Cloudflare. HCL stands for HashiCorp Configuration Lanaguage, and is named after the maker of Terraform.

$ terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "cloudflare" (1.0.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.cloudflare: version = "~> 1.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
When you run terraform init, any plugins required, such as the Cloudflare Terraform provider, are automatically downloaded and saved locally to a .terraform directory:

$ find .terraform/
.terraform/
.terraform//plugins
.terraform//plugins/darwin_amd64
.terraform//plugins/darwin_amd64/lock.json
.terraform//plugins/darwin_amd64/terraform-provider-cloudflare_v1.0.0_x4

With the Cloudflare provider installed, let's ask Terraform to show the changes it's planning to make to your Cloudflare account so you can confirm it matches the configuration you intended:

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + cloudflare_record.www
      id:          <computed>
      created_on:  <computed>
      domain:      "example.com"
      hostname:    <computed>
      metadata.%:  <computed>
      modified_on: <computed>
      name:        "www"
      proxiable:   <computed>
      proxied:     "true"
      ttl:         <computed>
      type:        "A"
      value:       "203.0.113.10"
      zone_id:     <computed>


Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

As you can see in the above "execution plan”, Terraform is going to create a new DNS record, as requested. Values that you've explicitly specified are displayed, e.g., the value of the A record—203.0.113.10—while values that are derived based on other API calls, e.g., looking up the zone_id, or returned after the object is created, are displayed as <computed>.

The plan command is important, as it allows you to preview the changes for accuracy before actually making them. Once you're comfortable with the execution plan, it's time to apply it:

$ terraform apply --auto-approve
cloudflare_record.www: Creating...
  created_on:  "" => "<computed>"
  domain:      "" => "example.com"
  hostname:    "" => "<computed>"
  metadata.%:  "" => "<computed>"
  modified_on: "" => "<computed>"
  name:        "" => "www"
  proxiable:   "" => "<computed>"
  proxied:     "" => "true"
  ttl:         "" => "<computed>"
  type:        "" => "A"
  value:       "" => "203.0.113.10"
  zone_id:     "" => "<computed>"
cloudflare_record.www: Creation complete after 1s (ID: c38d3103767284e7cd14d5dad3ab8668)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Note that I specified –auto-approve on the command line for briefer output; without this flag, Terraform will show you the output of terraform plan and then ask for confirmation before applying it.

Logging back into the Cloudflare Dashboard and selecting the DNS tab, I can see the record that was created by Terraform:

If you'd like to see the full results returned from the API call (including the default values that you didn't specify but let Terraform compute), you can run terraform show:

$ terraform show
cloudflare_record.www:
  id = c38d3103767284e7cd14d5dad3ab8668
  created_on = 2018-04-08T00:37:33.76321Z
  data.% = 0
  domain = example.com
  hostname = www.example.com
  metadata.% = 2
  metadata.auto_added = false
  metadata.managed_by_apps = false
  modified_on = 2018-04-08T00:37:33.76321Z
  name = www
  priority = 0
  proxiable = true
  proxied = true
  ttl = 1
  type = A
  value = 203.0.113.10
  zone_id = e2e6391340be87a3726f91fc4148b122

$ curl https://www.example.com
Hello, this is 203.0.113.10!

In the terraform apply step above, you created and applied some basic Cloudflare configuration. Terraform was able to apply this configuration to your account because you provided your email address and API token at the top of the cloudflare.tf file:

$ head -n4 cloudflare.tf 
provider "cloudflare" {
  email = "you@example.com"
  token = "your-api-key"
}

We're now going to store your configuration in GitHub where it can be tracked, peer-reviewed, and rolled back to as needed. But before we do so, we're going to remove your credentials from the Terraform config file so it doesn't get committed to a repository.

As a good security practice we need to remove your Cloudflare credentials from anything that will be committed to a repository. The Cloudflare Terraform provider supports reading these values from the CLOUDFLARE_EMAIL and CLOUDFLARE_TOKEN environment variables, so all we need to do is:

$ sed -ie 's/^.*email =.*$/  # email pulled from $CLOUDFLARE_EMAIL/' cloudflare.tf
$ sed -ie 's/^.*token =.*$/  # token pulled from $CLOUDFLARE_TOKEN/' cloudflare.tf

$ head -n4 cloudflare.tf 
provider "cloudflare" {
  # email pulled from $CLOUDFLARE_EMAIL
  # token pulled from $CLOUDFLARE_TOKEN
}

$ export CLOUDFLARE_EMAIL=you@example.com
$ export CLOUDFLARE_TOKEN=your-api-key

Note that you need to leave the empty provider definition in the file, so that Terraform knows to install the Cloudflare plugin.

After completing the above step, it's a good idea to make sure that you can still authenticate to Cloudflare. By running terraform plan we can get Terraform to pull the current state (which requires a valid email and API key):

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

cloudflare_record.www: Refreshing state... (ID: c38d3102767284e7ca14d5dad3ab8b69)

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

1. Store your configuration in GitHubNow that credentials have been removed, it's time to initialize a git repository with your Cloudflare configuration and then push it to GitHub.

First we'll create the GitHub repository to store the config. This can be done via the GitHub UI or with a simple API call:

$ export GITHUB_USER=your-github-user
$ export GITHUB_TOKEN=your-github-token

$ export GITHUB_URL=$(curl -sSXPOST https://api.github.com/user/repos?access_token=$GITHUB_TOKEN -H 'Content-Type: application/json' \
-d '{"name": "cf-config", "private":"true"}' 2>/dev/null | jq -r .ssh_url)

$ echo $GITHUB_URL
git@github.com:$GITHUB_USER/cf-config.git

Now we'll initialize a git repository and make our first commit:

$ git init
Initialized empty Git repository in $HOME/cf-config/.git/

$ git remote add origin $GITHUB_URL
$ git add cloudflare.tf

$ git commit -m "Step 2 - Initial commit with webserver definition."
[master (root-commit) 5acea17] Step 2 - Initial commit with webserver definition.
 1 file changed, 16 insertions(+)
 create mode 100644 cloudflare.tf

An astute reader may have noticed that we did not commit the .terraform directory nor did we commit the terraform.tfstate file. The former was not committed because this repository may be used on a different architecture, and the plugins contained in this directory are built for the system on which terraform init was run. The latter was not committed as i) it may eventually contain sensitive strings and ii) it is not a good way to keep state in sync, as HashiCorp [explains].

To prevent git from bugging us about these files, let's add them to a new .gitignore file, commit it, and push everything to GitHub:

$ cat > .gitignore <<'EOF'
.terraform/
terraform.tfstate*
EOF

$ git add .gitignore

$ git commit -m "Step 2 - Ignore terraform plugin directory and state file."
[master 494c6d6] Step 2 - Ignore terraform plugin directory and state file.
 1 file changed, 2 insertions(+)
 create mode 100644 .gitignore

$ git push
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 762 bytes | 0 bytes/s, done.
Total 6 (delta 0), reused 0 (delta 0)
To git@github.com:$GITHUB_USER/cf-config.git
 * [new branch]      master -> master

Now that you've got a basic website proxied through Cloudflare, it's time to use Terraform to adjust some additional settings on your zone. Below we'll configure some optional HTTPS settings, and then push the updated configuration to GitHub for posterity.

We'll use a new git branch for the changes, and then merge it into master before applying. On a team, you might consider using this step as an opportunity for others to review your change before merging and deploying it. Or you may integrate Terraform into your CI/CD system to perform tests automatically using another Cloudflare domain.

Here, we modify the Terraform configuration to enable the following settings: TLS 1.3, Always Use HTTPS, Strict SSL mode, and the Cloudflare WAF. Strict mode requires a valid SSL certificate on your origin, so be sure to use the Cloudflare Origin CA to generate one.

$ git checkout -b step3-https
Switched to a new branch 'step3-https'

$ cat >> cloudflare.tf <<'EOF'

resource "cloudflare_zone_settings_override" "example-com-settings" {
  name = "${var.domain}"

  settings {
    tls_1_3 = "on"
    automatic_https_rewrites = "on"
    ssl = "strict"
    waf = "on"
  }
}
EOF

Let's take a look at what Terraform is proposing before we apply it. We filter the terraform plan output to ignore those values that will be "computed”—in this case, settings that will be left at their default values. For brevity from here on out, we'll omit some extranneous Terraform output; if you'd like to see the output exactly as run, please see the full tutorial.

$ terraform plan | grep -v "<computed>"
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

cloudflare_record.www: Refreshing state... (ID: c38d3103767284e7cd14d5dad3ab8668)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + cloudflare_zone_settings_override.example-com-settings
      name:                                   "example.com"
      settings.#:                             "1"
      settings.0.automatic_https_rewrites:    "on"
      settings.0.ssl:                         "strict"
      settings.0.tls_1_3:                     "on"
      settings.0.waf:                         "on"


Plan: 1 to add, 0 to change, 0 to destroy.

The proposed changes look good, so we'll merge them into primary and then apply them with terraform apply. When working on a team, you may want to require pull requests and use this opportunity to peer review any proposed configuration changes.

$ git add cloudflare.tf
$ git commit -m "Step 3 - Enable TLS 1.3, Always Use HTTPS, and SSL Strict mode."
[step3-https d540600] Step 3 - Enable TLS 1.3, Always Use HTTPS, and SSL Strict mode.
 1 file changed, 11 insertions(+)

$ git checkout master
Switched to branch 'master'

$ git merge step3-https
Updating d26f40b..d540600
Fast-forward
 cloudflare.tf | 11 +++++++++++
 1 file changed, 11 insertions(+)

$ git push
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 501 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To git@github.com:$GITHUB_USER/cf-config.git
   d26f40b..d540600  master -> master

Before applying the changes, let's see if we can connect with TLS 1.3. Hint: we should not be able to with default settings. If you want to follow along with this test, you'll need to [compile curl against BoringSSL].

$ curl -v --tlsv1.3 https://www.upinatoms.com 2>&1 | grep "SSL connection\|error"
* error:1000042e:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
curl: (35) error:1000042e:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION

As shown above, we receive an error as TLS 1.3 is not yet enabled on your zone. Let's enable it by running terraform apply and try again:

$ terraform apply --auto-approve
cloudflare_record.www: Refreshing state... (ID: c38d3103767284e7cd14d5dad3ab8668)
cloudflare_zone_settings_override.example-com-settings: Creating...
  initial_settings.#:                     "" => "<computed>"
  initial_settings_read_at:               "" => "<computed>"
  name:                                   "" => "example.com"
  readonly_settings.#:                    "" => "<computed>"
  settings.#:                             "" => "1"
  settings.0.advanced_ddos:               "" => "<computed>"
  settings.0.always_online:               "" => "<computed>"
  settings.0.always_use_https:            "" => "<computed>"
  settings.0.automatic_https_rewrites:    "" => "on"
  settings.0.brotli:                      "" => "<computed>"
  settings.0.browser_cache_ttl:           "" => "<computed>"
  settings.0.browser_check:               "" => "<computed>"
  settings.0.cache_level:                 "" => "<computed>"
  settings.0.challenge_ttl:               "" => "<computed>"
  settings.0.cname_flattening:            "" => "<computed>"
  settings.0.development_mode:            "" => "<computed>"
  settings.0.edge_cache_ttl:              "" => "<computed>"
  settings.0.email_obfuscation:           "" => "<computed>"
  settings.0.hotlink_protection:          "" => "<computed>"
  settings.0.http2:                       "" => "<computed>"
  settings.0.ip_geolocation:              "" => "<computed>"
  settings.0.ipv6:                        "" => "<computed>"
  settings.0.max_upload:                  "" => "<computed>"
  settings.0.minify.#:                    "" => "<computed>"
  settings.0.mirage:                      "" => "<computed>"
  settings.0.mobile_redirect.#:           "" => "<computed>"
  settings.0.opportunistic_encryption:    "" => "<computed>"
  settings.0.origin_error_page_pass_thru: "" => "<computed>"
  settings.0.polish:                      "" => "<computed>"
  settings.0.prefetch_preload:            "" => "<computed>"
  settings.0.privacy_pass:                "" => "<computed>"
  settings.0.pseudo_ipv4:                 "" => "<computed>"
  settings.0.response_buffering:          "" => "<computed>"
  settings.0.rocket_loader:               "" => "<computed>"
  settings.0.security_header.#:           "" => "<computed>"
  settings.0.security_level:              "" => "<computed>"
  settings.0.server_side_exclude:         "" => "<computed>"
  settings.0.sha1_support:                "" => "<computed>"
  settings.0.sort_query_string_for_cache: "" => "<computed>"
  settings.0.ssl:                         "" => "strict"
  settings.0.tls_1_2_only:                "" => "<computed>"
  settings.0.tls_1_3:                     "" => "on"
  settings.0.tls_client_auth:             "" => "<computed>"
  settings.0.true_client_ip_header:       "" => "<computed>"
  settings.0.waf:                         "" => "on"
  settings.0.webp:                        "" => "<computed>"
  settings.0.websockets:                  "" => "<computed>"
  zone_status:                            "" => "<computed>"
  zone_type:                              "" => "<computed>"
cloudflare_zone_settings_override.example-com-settings: Creation complete after 2s (ID: e2e6491340be87a3726f91fc4148b125)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Now we can try the same command as above, and see that it succeeds. Niiice, TLS 1.3!

$ curl -v --tlsv1.3 https://www.example.com 2>&1 | grep "SSL connection\|error"
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256

Before proceeding, make sure that your account is enabled for Rate Limiting. If you’re on an Enterprise plan, you should ask your Customer Success Manager to do this; otherwise, you can subscribe to Rate Limiting within the Cloudflare Dashboard.

With our zone settings locked down, and our site starting to get some more attention, it's unfortunately begun attracting some of the less scrupulous characters on the internet. Our server access logs show attempts to brute force our login page at https://www.example.com/login. Let's see what we can do with Cloudflare's rate limiting product) to put a stop to these efforts.

After creating a new branch we specify the rate limiting rule:

$ git checkout -b step4-ratelimit
Switched to a new branch 'step4-ratelimit'

$ cat >> cloudflare.cf <<'EOF'
resource "cloudflare_rate_limit" "login-limit" {
  zone = "${var.domain}"

  threshold = 5
  period = 60
  match {
    request {
      url_pattern = "${var.domain}/login"
      schemes = ["HTTP", "HTTPS"]
      methods = ["POST"]
    }
    response {
      statuses = [401, 403]
      origin_traffic = true
    }
  }
  action {
    mode = "simulate"
    timeout = 300
    response {
      content_type = "text/plain"
      body = "You have failed to login 5 times in a 60 second period and will be blocked from attempting to login again for the next 5 minutes."
    }
  }
  disabled = false
  description = "Block failed login attempts (5 in 1 min) for 5 minutes."
}
EOF

This rule is a bit more complex than the zone settings rule, so let's break it down:

00: resource "cloudflare_rate_limit" "login-limit" {
01:   zone = "${var.domain}"
02:
03:   threshold = 5
04:   period = 60

The threshold is an integer count of how many times an event (defined by the match block below) has to be detected in the period before the rule takes action. The period is measured in seconds, so the above rule says to take action if the match fires 5 times in 60 seconds.

05:   match {
06:     request {
07:       url_pattern = "${var.domain}/login"
08:       schemes = ["HTTP", "HTTPS"]
09:       methods = ["POST"]
10:     }
11:     response {
12:       statuses = [401, 403]
13:     }
14:   }

The match block tells the Cloudflare edge what to be on the lookout for, i.e., HTTP or HTTPS POST requests to https://www.example.com/login. We further restrict the match to HTTP 401/Unauthorized or 403/Forbidden response codes returned from the origin.

15:   action {
16:     mode = "simulate"
17:     timeout = 300
18:     response {
19:       content_type = "text/plain"
20:       body = "You have failed to login 5 times in a 60 second period and will be blocked from attempting to login again for the next 5 minutes."
21:     }
22:   }
23:   disabled = false
24:   description = "Block failed login attempts (5 in 1 min) for 5 minutes."
25: }

After matching traffic, we set the action for our edge to take. When testing, it's a good idea to set the mode to simulate and review logs before taking enforcement action (see below). The timeout field here indicates that we want to enforce this action for 300 seconds (5 minutes) and the response block indicates what should be sent back to the caller that tripped the rate limit.

As usual, we take a look at the proposed plan before we apply any changes:

$ terraform plan
...
Terraform will perform the following actions:

  + cloudflare_rate_limit.login-limit
      id:                                     <computed>
      action.#:                               "1"
      action.0.mode:                          "simulate"
      action.0.response.#:                    "1"
      action.0.response.0.body:               "You have failed to login 5 times in a 60 second period and will be blocked from attempting to login again for the next 5 minutes."
      action.0.response.0.content_type:       "text/plain"
      action.0.timeout:                       "300"
      description:                            "Block failed login attempts (5 in 1 min) for 5 minutes."
      disabled:                               "false"
      match.#:                                "1"
      match.0.request.#:                      "1"
      match.0.request.0.methods.#:            "1"
      match.0.request.0.methods.1012961568:   "POST"
      match.0.request.0.schemes.#:            "2"
      match.0.request.0.schemes.2328579708:   "HTTP"
      match.0.request.0.schemes.2534674783:   "HTTPS"
      match.0.request.0.url_pattern:          "www.example.com/login"
      match.0.response.#:                     "1"
      match.0.response.0.origin_traffic:      "true"
      match.0.response.0.statuses.#:          "2"
      match.0.response.0.statuses.1057413486: "403"
      match.0.response.0.statuses.221297644:  "401"
      period:                                 "60"
      threshold:                              "5"
      zone:                                   "example.com"
      zone_id:                                <computed>


Plan: 1 to add, 0 to change, 0 to destroy.

The plan looks good so let's go ahead, merge it in, and apply it.

$ git add cloudflare.tf
$ git commit -m "Step 4 - Add rate limiting rule to protect /login."
[step4-ratelimit 0f7e499] Step 4 - Add rate limiting rule to protect /login.
 1 file changed, 28 insertions(+)

$ git checkout master
Switched to branch 'master'

$ git merge step4-ratelimit
Updating 321c2bd..0f7e499
Fast-forward
 cloudflare.tf | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

$ terraform apply --auto-approve
cloudflare_record.www: Refreshing state... (ID: c38d3103767284e7cd14d5dad3ab8668)
cloudflare_zone_settings_override.example-com-settings: Refreshing state... (ID: e2e6491340be87a3726f91fc4148b125)
cloudflare_rate_limit.login-limit: Creating...
  action.#:                               "" => "1"
  action.0.mode:                          "" => "simulate"
  action.0.response.#:                    "" => "1"
  action.0.response.0.body:               "" => "You have failed to login 5 times in a 60 second period and will be blocked from attempting to login again for the next 5 minutes."
  action.0.response.0.content_type:       "" => "text/plain"
  action.0.timeout:                       "" => "300"
  description:                            "" => "Block failed login attempts (5 in 1 min) for 5 minutes."
  disabled:                               "" => "false"
  match.#:                                "" => "1"
  match.0.request.#:                      "" => "1"
  match.0.request.0.methods.#:            "" => "1"
  match.0.request.0.methods.1012961568:   "" => "POST"
  match.0.request.0.schemes.#:            "" => "2"
  match.0.request.0.schemes.2328579708:   "" => "HTTP"
  match.0.request.0.schemes.2534674783:   "" => "HTTPS"
  match.0.request.0.url_pattern:          "" => "www.example.com/login"
  match.0.response.#:                     "" => "1"
  match.0.response.0.origin_traffic:      "" => "true"
  match.0.response.0.statuses.#:          "" => "2"
  match.0.response.0.statuses.1057413486: "" => "403"
  match.0.response.0.statuses.221297644:  "" => "401"
  period:                                 "" => "60"
  threshold:                              "" => "5"
  zone:                                   "" => "example.com"
  zone_id:                                "" => "<computed>"
cloudflare_rate_limit.login-limit: Creation complete after 1s (ID: 8d518c5d6e63406a9466d83cb8675bb6)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Note that if you haven't purchased rate limiting yet, you will see the following error when attempting to apply the new rule:

Error: Error applying plan:

1 error(s) occurred:

* cloudflare_rate_limit.login-limit: 1 error(s) occurred:

* cloudflare_rate_limit.login-limit: error creating rate limit for zone: error from makeRequest: HTTP status 400: content "{\n  \"result\": null,\n  \"success\": false,\n  \"errors\": [\n    {\n      \"code\": 10021,\n      \"message\": \"ratelimit.api.not_entitled.account\"\n    }\n  ],\n  \"messages\": []\n}\n"

After confirming that the rule is triggering as planned in logs (but not yet enforcing), it's time to switch from simulate to ban:

$ git checkout step4-ratelimit
$ sed -i.bak -e 's/simulate/ban/' cloudflare.tf

$ git diff
diff --git a/cloudflare.tf b/cloudflare.tf
index ed5157c..9f25a0c 100644
--- a/cloudflare.tf
+++ b/cloudflare.tf
@@ -42,7 +42,7 @@ resource "cloudflare_rate_limit" "login-limit" {
     }
   }
   action {
-    mode = "simulate"
+    mode = "ban"
     timeout = 300
     response {
       content_type = "text/plain"

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

cloudflare_zone_settings_override.example-com-settings: Refreshing state... (ID: e2e6491340be87a3726f91fc4148b126)
cloudflare_rate_limit.login-limit: Refreshing state... (ID: 8d518c5d6e63406a9466d83cb8675bb6)
cloudflare_record.www: Refreshing state... (ID: c38d3103767284e7cd14d5dad3ab8669)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ cloudflare_rate_limit.login-limit
      action.0.mode: "simulate" => "ban"

Plan: 0 to add, 1 to change, 0 to destroy.

$ git add cloudflare.tf

$ git commit -m "Step 4 - Update /login rate limit rule from 'simulate' to 'ban'."
[step4-ratelimit e1c38cf] Step 4 - Update /login rate limit rule from 'simulate' to 'ban'.
 1 file changed, 1 insertion(+), 1 deletion(-)

$ git checkout master && git merge step4-ratelimit && git push
Switched to branch 'master'
Updating 0f7e499..e1c38cf
Fast-forward
 cloudflare.tf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 361 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To git@github.com:$GITHUB_USER/cf-config.git
   0f7e499..e1c38cf  master -> master

$ terraform apply --auto-approve
cloudflare_rate_limit.login-limit: Refreshing state... (ID: 8d518c5d6e63406a9466d83cb8675bb6)
cloudflare_record.www: Refreshing state... (ID: c38d3103767284e7cd14d5dad3ab8669)
cloudflare_zone_settings_override.example-com-settings: Refreshing state... (ID: e2e6491340be87a3726f91fc4148b126)
cloudflare_rate_limit.login-limit: Modifying... (ID: 8d518c5d6e63406a9466d83cb8675bb6)
  action.0.mode: "simulate" => "ban"
cloudflare_rate_limit.login-limit: Modifications complete after 0s (ID: 8d518c5d6e63406a9466d83cb8675bb6)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

This step is optional, but it's a good way to demonstrate that the rule is working as expected (note the final 429 response):

$ for i in {1..6}; do curl -XPOST -d '{"username": "foo", "password": "bar"}' -vso /dev/null https://www.example.com/login 2>&1 | grep "< HTTP"; sleep 1; done
< HTTP/1.1 401 Unauthorized
< HTTP/1.1 401 Unauthorized
< HTTP/1.1 401 Unauthorized
< HTTP/1.1 401 Unauthorized
< HTTP/1.1 401 Unauthorized
< HTTP/1.1 429 Too Many Requests

That's it for today! Stay tuned next week for part 2 of this post, where we continue the tour through the following resources and techniques:

• Load Balancing Resource

• Page Rules Resource

• Reviewing and Rolling Back Changes

• Importing Existing State and Configuration