Cloudflare’s Partnership with HashiCorp and Bootstrapping Terraform with Cf-Terraforming

Cloudflare and HashiCorp have been technology partners since 2018, and in that time Cloudflare’s integration with HashiCorp’s technology has deepened, especially with Terraform, HashiCorp’s infrastructure-as-code product. Today we are announcing a major update to our Terraform bootstrapping tool, cf-terraforming. In this blog, I recap the history of our partnership, the HashiCorp Terraform Verified Provider for Cloudflare, and how getting started with Terraform for Cloudflare developers is easier than ever before with the new version of cf-terraforming.

Members of the open source community wrote and supported the first version of Cloudflare's Terraform provider. Eventually our customers began to bring up Terraform in conversations more often. Because of customer demand, we started supporting and developing the Terraform provider ourselves. You can read the initial v1.0 announcement for the provider here. Soon after, Cloudflare’s Terraform provider became ‘verified’ and we began working with HashiCorp to provide a high quality experience for developers.

HashiCorp Terraform allows developers to control their infrastructure-as-code through a standard configuration language, HashiCorp Configuration Language (HCL). It works across a myriad of different types of infrastructure including cloud service providers, containers, virtual machines, bare metal, etc. Terraform makes it easy for developers to follow best practices when interacting with SaaS, PaaS, and other service provider APIs that set up infrastructure. Like developers already do with software code, they can store infrastructure configuration as code in git, manage changes through code reviews, and track versions and commit history over time. Terraform also makes it easier to roll back changes if developers discover issues after a deployment.

Our developers love the power of Cloudflare + Terraform for infrastructure provisioning. IT operations teams want a platform that provides complete visibility and control. IT teams want a platform that is easy to use, does not have a steep learning curve, and provides levers to control resources much better. Cloudflare + Terraform platform provides just that.– Dmitry Zadorojnii, Chief Technology Officer, Autodoc GmbH

Since the 1.0 release of Cloudflare’s Terraform provider, Cloudflare has continued to build out the capabilities exposed in the provider while HashiCorp has expanded its ecosystem by developing additional features like the Terraform Registry. This helps developers find documentation on Cloudflare’s Terraform provider or any others. Terraform itself has also matured greatly–Terraform’s v0.12 release had many changes, outlined here.

“Leveraging the power of Terraform, you can codify your Cloudflare configuration. Codification enables version control and automation, increasing productivity and reducing human error. We are pleased to have Cloudflare as a technology partner and look forward to our ongoing collaboration.”– Asvin Ramesh, Director, Technology Alliances, HashiCorp

Here are some great resources for developers looking to better understand how to get started using Terraform with Cloudflare:

• Cloudflare’s Developer Docs for Terraform

• HashiCorp Tutorial: Host a Static Website with S3 and Cloudflare

• Cloudflare Terraform Provider Documentation

• Github repo for Cloudflare’s Terraform Provider

We released the first version of cf-terraforming in early 2019 in this blog post. Since then we learned a few lessons about building and maintaining such a tool. In this section I’ll recap why we built such a tool in the first place, lessons learned over the last two years, and what is new and improved with the version we are launching today.

The name for the cf-terraforming library comes from another tool created by dtan4 on github: https://github.com/dtan4/terraforming. The original tool allowed users to generate tfconfig and tfstate files for existing AWS resources. This made it much easier for existing AWS customers to begin using Terraform. Terraform generally expects to be authoritative about the configuration it manages. Effectively, it expects that you only make changes to that config through Terraform and not anywhere else, like an API or a dashboard. If you were an existing customer of AWS (or Cloudflare) and already had everything configured via API or a UI, this posed a challenge: How do I quickly and correctly get all of my existing config into Terraform so that it can be the authoritative source of truth? For AWS resources, before dtan4’s terraforming you had to manually write the tfconfig for every object and the run import commands to generate the corresponding tfstate. For sizable deployments, this could be nigh impossible.

cf-terraforming served to solve the same problem for Cloudflare services. I had many conversations with customers who had been using Cloudflare for years and who were interested in migrating control of their Cloudflare configuration to Terraform. Cf-terraforming gave them a way to quickly convert all of their existing Cloudflare usage into tfconfig and tfstate.

cf-terraforming was one of the enabling technologies that we used to bootstrap Cloudflare into our Terraform Git-ops workflow. We had thousands of records to port, import, and manage, and doing this by hand would have been an arduous and error-prone task. Using cf-terraforming to generate our initial set of resources allowed our engineers to submit Cloudflare changes, enabling our product engineers to be infrastructure operators.– Sean Chittenden, Infrastructure, DoorDash

After having cf-terraforming available for some time, we’ve learned quite a bit about the challenges in managing such a tool.

Duplication of effort when resources change

When Cloudflare releases new services or features today, that typically means new or incremental changes to Cloudflare’s APIs. This in turns means updates to our Terraform provider. Since Terraform is a golang program, before we can update our provider we have to first update the cloudflare-go library. Depending on the change, this can be a couple lines in each repo or extensive changes to both. Once we launched cf-terraforming, we now had a third library that needed synchronous changes alongside the provider and go library. Missing a change meant that if someone tried to use cf-terraforming, they may have incomplete config or state, which would not work.

Impact of changes to Terraform for the tool

Not only did our own API changes create additional work, but changes to Terraform itself could mean changes for cf-terraforming. The Terraform 0.12 update was a massive update that required a lot of careful testing and coordination with our provider. It also meant changes in HCL and in provider interactions that cf-terraforming had to account for. Such a massive one-time hit was very difficult to account for, and we’ve struggled to ensure compatibility.

TF State management

The ability to have cf-terraforming generate a tfstate file was both incredibly important and also experimental. In general a developer never really needs to concern themselves with what is in the tfstate file but needs to know it contains the actual state of those resources such that references in the config can be resolved and managed correctly. We opened up that black box, which meant that we were involved in state file implementation details that we ultimately shouldn’t be.

Given these lessons, we looked at how we could update cf-terraforming to alleviate these problems and provide a better tool for both customers and ourselves. After some prototyping to validate our ideas, we came up with a new model that has been productized and is now available for customers.

Today we are launching a new version of cf-terraforming that improves upon our previous work. This new version makes it easier for us to support new resources or changes to existing resources and simplifies the workflow for developers looking to bootstrap their Cloudflare Terraform configuration.

Instead of hand crafting how to generate both the tfconfig and tfstate for each of the 48 or so resources supported in the Terraform provider, we now leverage Terraform’s capabilities to do more auto generation of what’s needed for similar resource types. HashiCorp has a great CLI tool called terraform-exec that provides powerful out-of-the-box capabilities we can take advantage of. Using terraform-exec we get access to `terraform providers schema -json`, which gives us the json schema of our provider. We use this to auto generate the fields we need to populate from the API. In many cases the API response fields map one to one with the json schema, which allows us to automatically populate the tfconfig. In other cases some small tweaks are necessary, which still saves a lot of time to initially support the resource and lowers the burden for future changes. Through this method, if the terraform provider changes for any reason, we can build new versions of cf-terraforming that will fetch the new schema from terraform-exec versus us having to make a lot of manual code changes to the config generation.

For tfstate, we simplify our approach by outputting the full set of terraform import calls that would need to be run for those resources instead of attempting to generate the tfstate definition itself. This removes virtually any need for future library changes since the import commands do not change if Cloudflare’s API or provider changes.

With that, let’s look at the new cf-terraforming in action. For this walkthrough let’s assume we have an existing zone on Cloudflare with DNS records and firewall rules configured. We want to start managing this zone in Terraform, but we don’t want to have to define all of our configuration by hand.

Our goal is to have a ".tf" file with the DNS records resources and firewall rules along with filter resources AND for Terraform to be aware of the equivalent state for those resources. Our inputs are the zone we already have created in Cloudflare, and our tool is the cf-terraforming library. If you are following along at home, you will need terraform installed and at least Go v1.12.x installed.

Getting the environment setup

Before we can use cf-terraforming or the provider, we need an API token. I’ll briefly go through the steps here, but for a more in-depth walkthrough see the API developer docs. On the Cloudflare dashboard we generate an API token here with the following setup:

PermissionsZone:DNS:ReadZone:Firewall Services:Read

Zone Resources:garrettgalow.party (my zone, but this should be your own)

TTLValid until: 2021-03-30 00:00:00Z

Note: I set an expiration date on the token so that when I inevitably forget about this token, it will expire and reduce the risk of exposure in the future. This is optional, but it’s a good practice when creating tokens you only need for a short period of time especially if they have edit access.

API Token summary from the Cloudflare Dashboard

Now we set the API Token we created as an environment variable so that both Terraform and cf-terraforming can access it for any commands (and so I don’t have to remove it from code examples).

$export CLOUDFLARE_API_TOKEN=<token_secret>

Terraform requires us to have a folder to hold our Terraform configuration and state. For that we create a folder for our use case and create a cloudflare.tf config file with a provider definition for Cloudflare so Terraform knows we will be using the Cloudflare provider.

mkdir terraforming_test
cd terraforming_test
cat > cloudflare.tf <<'EOF'
terraform {
    required_providers {
        cloudflare = {
            source = "cloudflare/cloudflare"
        }	
    }
}

provider "cloudflare" {
# api_token  = ""  ## Commented out as we are using an environment var
}
EOF

Here is the content of our cloudflare.tf file if you would rather copy and paste it into your text editor of choice:

terraform {
    required_providers {
        cloudflare = {
            source = "cloudflare/cloudflare"
        }	
    }
}

provider "cloudflare" {
# api_token  = ""  ## Commented out as we are using an environment var
}

We call terraform init to ensure Terraform is fully initialized and has the Cloudflare provider installed. At the time of writing this blog post, this is what terraform -v gives me for version info. We recommend that you use the latest versions of both Terraform and the Cloudflare provider.

$ terraform -v
Terraform v0.14.10
+ provider registry.terraform.io/cloudflare/cloudflare v2.19.2

And finally we install cf-terraforming with the following command:

$ GO111MODULE=on go get -u github.com/cloudflare/cf-terraforming/...

If you’re using Homebrew on MacOS, this can be simplified to:

$ brew tap cloudflare/cloudflare
$ brew install --cask cloudflare/cloudflare/cf-terraforming

Using cf-terraforming to generate Terraform configuration

We are now ready to start generating a Terraform config. To begin, we run cf-terraforming to generate the first blocks of config for the DNS record resources and append it to the cloudflare.tf file we previously created.

cf-terraforming generate --resource-type cloudflare_record --zone <zone_id> >> cloudflare.tf

Breaking this command down:

generate is the command that will produce a valid HCL config of resources

--resource-type specifies the Terraform resource name that we want to generate an HCL config for. You can only generate configuration for one resource at a time. In this example we are using cloudflare_record

--zone specifies the Cloudflare zone ID we wish to fetch all the DNS records for so cf-terraforming can create the appropriate API calls

Example:

$ cf-terraforming generate --resource-type cloudflare_record --zone 9c2f972575d986b99fa03c7bbfaab414 >> cloudflare.tf
$

Success will return with no output to console. If you want to see the output before adding it to the config file, run the command without >> cloudflare.tf and it will output to console.

Here is the partial output in my case, if it is not appended to the config file:

$ cf-terraforming generate --resource-type cloudflare_record --zone 9c2f972575d986b99fa03c7bbfaab414
resource "cloudflare_record" "terraform_managed_resource_db185030f44e358e1c2162a9ecda7253" {
name = "api"
proxied = true
ttl = 1
type = "A"
value = "x.x.x.x"
zone_id = "9c2f972575d986b99fa03c7bbfaab414"
}
resource "cloudflare_record" "terraform_managed_resource_e908d014ebef5011d5981b3ba961a011" {
...

The output resources are given standardized names of “terraform_managed_resource_<resource_id>”. Because the resource id is included in the name, the object names between the config we just exported and the state we will import will always be consistent. This is necessary to ensure Terraform knows which config belongs to which state.

After generating the DNS record resources, we now do the same for both firewall rules and filters.

cf-terraforming generate --resource-type cloudflare_firewall_rule --zone <zone_id> >> cloudflare.tf
cf-terraforming generate --resource-type cloudflare_filter --zone <zone_id> >> cloudflare.tf

Example:

$ cf-terraforming generate --resource-type cloudflare_firewall_rule --zone 9c2f972575d986b99fa03c7bbfaab414 >> cloudflare.tf
$ cf-terraforming generate --resource-type cloudflare_filter --zone 9c2f972575d986b99fa03c7bbfaab414 >> cloudflare.tf
$

Using cf-terraforming to import Terraform state

Before we can ask Terraform to verify the config, we need to import the state so that Terraform does not attempt to create new objects but instead reuses the existing objects we already have in Cloudflare.

Similar to what we did with the generate command, we use the import command to generate terraform import commands.

cf-terraforming import --resource-type cloudflare_record --zone <zone_id>

Breaking this command down:

import is the command that will produce a valid terraform import command that we can then run--resource-type (same as the generate command) specifies the Terraform resource name that we want to create import commands for. You can only use one resource at a time. In this example we are using cloudflare_record--zone (same as the generate command) specifies the Cloudflare zone ID we wish to fetch all the DNS records for so cf-terraforming can populate the commands with the appropriate API calls

And an example with output:

$ cf-terraforming import --resource-type cloudflare_record --zone 9c2f972575d986b99fa03c7bbfaab414
terraform import cloudflare_record.terraform_managed_resource_db185030f44e358e1c2162a9ecda7253 9c2f972575d986b99fa03c7bbfaab414/db185030f44e358e1c2162a9ecda7253
terraform import cloudflare_record.terraform_managed_resource_e908d014ebef5011d5981b3ba961a011 9c2f972575d986b99fa03c7bbfaab414/e908d014ebef5011d5981b3ba961a011
terraform import cloudflare_record.terraform_managed_resource_3f62e6950a5e0889a14cf5b913e87699 9c2f972575d986b99fa03c7bbfaab414/3f62e6950a5e0889a14cf5b913e87699
terraform import cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d 9c2f972575d986b99fa03c7bbfaab414/47581f47852ad2ba61df90b15933903d$

The output of this will be ready to use terraform import commands. Running the generated terraform import command will leverage existing Cloudflare Terraform provider functionality to import the resource state into Terraform’s terraform.tfstate file. This removes the tedium of pulling all the appropriate resource IDs from Cloudflare’s API and then formatting these commands one by one. The order of operations of the config then state is important as Terraform expects there to be configuration in the .tf file for these resources before importing the state.

Note: Be careful when you actually import these resources, though, as from that point on any subsequent Terraform actions like plan or apply will expect this resource to be there. Removing the state is possible but requires manually editing the terraform.tfstate file. Terraform does keep a backup locally in case you make a mistake though.

Now we actually run these terraform import commands to import the state. Below shows what that looks like for a single resource.

$ terraform import cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d 9c2f972575d986b99fa03c7bbfaab414/47581f47852ad2ba61df90b15933903d
cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d: Importing from ID "9c2f972575d986b99fa03c7bbfaab414/47581f47852ad2ba61df90b15933903d"...
cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d: Import prepared!
Prepared cloudflare_record for import
cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d: Refreshing state... [id=47581f47852ad2ba61df90b15933903d]
Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

With cloudflare_record imported, now we do the same for the firewall_rules and filters.

cf-terraforming import --resource-type cloudflare_firewall_rule --zone <zone_id>
cf-terraforming import --resource-type cloudflare_filter --zone <zone_id>

Shown with output:

$ cf-terraforming import --resource-type cloudflare_firewall_rule --zone 9c2f972575d986b99fa03c7bbfaab414
terraform import cloudflare_firewall_rule.terraform_managed_resource_0de909f3229341a2b8214737903f2caf 9c2f972575d986b99fa03c7bbfaab414/0de909f3229341a2b8214737903f2caf
terraform import cloudflare_firewall_rule.terraform_managed_resource_0c722eb85e1c47dcac83b5824bad4a7c 9c2f972575d986b99fa03c7bbfaab414/0c722eb85e1c47dcac83b5824bad4a7c
$ cf-terraforming import --resource-type cloudflare_filter --zone 9c2f972575d986b99fa03c7bbfaab414
terraform import cloudflare_filter.terraform_managed_resource_ee048570bb874972bbb6557f7529e094 9c2f972575d986b99fa03c7bbfaab414/ee048570bb874972bbb6557f7529e094
terraform import cloudflare_filter.terraform_managed_resource_1bb6cd50e2534a64a9ec698fd841ffc5 9c2f972575d986b99fa03c7bbfaab414/1bb6cd50e2534a64a9ec698fd841ffc5
$

As with cloudflare_record, we run these terraform import commands to ensure all the state is successfully imported.

Verifying everything is correct

Now that we have both the configuration and state in place, we call terraform plan to see if Terraform can verify everything is in place. If all goes well then you will be greeted with the following “nothing to do” message:

No changes. Infrastructure is up-to-date.
This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

You now can begin managing these resources in Terraform. If you want to add more resources into Terraform, follow these steps for other resources. You can find which resources are supported in the README. We will add additional resources over time, but if there are specific ones you are looking for, please create GitHub issues or upvote any existing ones.

Whether you are an existing Cloudflare customer and have been curious about Terraform or you are looking to expand your infrastructure-as-code to include Cloudflare’s services, you have everything you need to get building with Terraform, the Cloudflare provider, and cf-terraforming. For questions, comments, or feature requests for either the provider or cf-terraforming, see the respective github repos.