Keyless SSL now supports FIPS 140-2 L3 hardware security module (HSM) offerings from all major cloud providers

This post is also available in 简体中文, 繁體中文, 日本語, Bahasa Indonesia, ไทย.

Private encryption keys stored in hardware security module offerings from all major cloud providers can now be used to secure HTTPS connections at Cloudflare’s global edge.

Cloudflare generates, protects, and manages more SSL/TLS private keys than perhaps any organization in the world. Private keys must be carefully protected, as an attacker in possession of one can impersonate legitimate sites and decrypt HTTPS requests. To mitigate this risk, Cloudflare has strict key handling procedures and layers of isolation at the edge that are designed to safeguard keys at all costs. But for a small minority of customers with information security policies dictating where they can (or cannot) custody their keys, these protections do not meet their requirements.

It was for these customers that we first released Keyless SSL in 2014, a protocol we use extensively inside our network: all of the TLS handshakes per day established at the Cloudflare edge that take place in a process that has no access to our customers’ private keys. The data required to establish the session is instead sent to a separate system, where the necessary cryptographic signing operation is performed. For keys uploaded to or generated by Cloudflare, we manage this other system, but some customers wish to manage it themselves.

Historically the keys were placed on the server running the open source gokeyless daemon we provide to process the handshake, or secured in an on-prem hardware security module (HSM) that gokeyless interfaces with using a standard protocol known as PKCS#11. However, as financial services, healthcare, cryptocurrency, and other highly regulated or security-focused companies have moved to the cloud, they cannot simply ship these expensive boxes and ask Amazon, Google, IBM, or Microsoft to rack and stack them.

For these customers, especially those whose information security policies mandate FIPS 140-2 Level 3 validated HSMs, we are announcing that Keyless SSL now supports the following cloud-hosted HSMs: Amazon Cloud HSM; Google Cloud HSM; IBM Cloud HSM; Microsoft Azure Dedicated HSM and Managed HSM. We also support any other HSM that implements the PKCS#11 standard, including series such as the nCipher nShield Connect and Thales Luna.

HSM overview

HSMs are purpose-built machines that are tamper-resistant, hardened against weaknesses such as side-channel attacks, and optimized to perform cryptographic operations such as signing and decryption. They can be deployed as stand-alone boxes, as expansion cards inserted into servers, or, most recently, as cloud services.

Rather than generate private keys on a server and upload them to the HSM, vendors and security experts typically recommend that keys are generated on (and never leave) the device itself. HSMs have better randomness guarantees than general-purpose servers, and take special precautions to protect keys in memory before synchronizing them to disk in an encrypted state. When operations that require the private key are required, services make authenticated API calls into the device using libraries provided by the HSM vendor.

HSMs and FIPS 140-2 level 3

HSMs are typically validated against the Federal Information Process Standard (FIPS) publication 140-2: Security Requirements for Cryptographic Modules. There are four levels — 1 through 4 — that specify increasingly stringent requirements around approved algorithms, security functions, role-based access control, and tamper evident/resistant protections.

The National Institute of Standards and Technology (NIST) publishes these guidelines, administers the Cryptographic Module Validation Program, and publishes a searchable database of validated modules, which includes the offerings listed below. We have provided instructions on how to use them with Cloudflare.

Getting started with cloud offerings

All existing Keyless SSL customers can immediately make use of this technology, and you can read instructions for doing so at https://developers.cloudflare.com/ssl/keyless-ssl/hardware-security-modules. Source code is available on GitHub: https://github.com/cloudflare/gokeyless.

End-to-end example: Microsoft Azure Managed HSM

Microsoft’s Azure Key Vault team released Managed HSM. The offering is FIPS 140-2 Level 3 validated and is integrated with Azure services such as Azure Storage, Azure SQL, and Azure Information Protection. Managed HSM is available in the following regions: East US 2, South Central US, North Europe, and West Europe.

The instructions below are taken from the Quickstart: Provision and activate a managed HSM using Azure CLI guide, followed by instructions for using the Managed HSM with Cloudflare. The commands were run on a Ubuntu VM created in the same region (South Central US) as the HSM; this is also where we will deploy the Cloudflare Keyless SSL daemon.

Provision and activate the HSM

First we log in via the CLI and create a resource group for the Managed HSM in one of the supported regions. Note that you may get warnings from various commands based on the preview status of the offering.

$ LOCATION=southcentralus; GROUPNAME=”HSMgroup”; HSMNAME=”KeylessHSM”
$ az login
$ az group create --name $GROUPNAME --location $LOCATION

Next, we provision the HSM resource and activate it by downloading the security domain. The example below grants administrative access to the signed-in user, along with another administrator whose OID can be retrieved by executing the same oid=$(...) command from the CLI where that user is logged in.

$ MYOID=$(az ad signed-in-user show --query objectId -o tsv)
$ OTHERADMIN_OID=...

$ az keyvault create --hsm-name $HSMNAME --resource-group $GROUPNAME --location $LOCATION --administrators $MYOID $OTHERADMIN_OID

Argument '--hsm-name' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Argument '--administrators' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
{- Finished ..
  "id": "/subscriptions/.../resourceGroups/HSMgroup/providers/Microsoft.KeyVault/managedHSMs/Keyles
sHSM",
  "location": "southcentralus",
  "name": "KeylessHSM",
  "properties": {
    "createMode": null,
    "enablePurgeProtection": false,
    "enableSoftDelete": true,
    "hsmUri": "https://keylesshsm.managedhsm.azure.net/",
    "initialAdminObjectIds": [
      "$MYOID",
      "$OTHERADMIN_OID"
    ],
    "provisioningState": "Succeeded",
    "softDeleteRetentionInDays": 90,
    "statusMessage": "The Managed HSM is provisioned and ready to use.",
    "tenantId": "..."
  },
  "resourceGroup": "$GROUPNAME",
  "sku": {
    "family": "B",
    "name": "Standard_B1"
  },
  "tags": {},
  "type": "Microsoft.KeyVault/managedHSMs"
}

Record the hsmUri property that is returned from the command above. You will need this shortly when configuring Keyless SSL on your VM.

$ HSMURI="https://keylesshsm.managedhsm.azure.net/"

Now that the HSM is provisioned, you must provide it with at least 3 RSA public keys. The HSM will encrypt the security domain with these keys and send it back to you, after which the HSM is ready to use.

$ openssl req -newkey rsa:2048 -nodes -keyout cert_0.key -x509 -days 365 -out cert_0.cer
$ openssl req -newkey rsa:2048 -nodes -keyout cert_1.key -x509 -days 365 -out cert_1.cer
$ openssl req -newkey rsa:2048 -nodes -keyout cert_2.key -x509 -days 365 -out cert_2.cer

$ az keyvault security-domain download --hsm-name $HSMNAME --sd-wrapping-keys ./cert_0.cer ./cert_1.cer ./cert_2.cer --sd-quorum 2 --security-domain-file $HSMNAME-SD.json

If you get a “Failed to connect to MSI” error, and you are using a cloud shell from the Azure Portal, run az login again as this is a known issue.

Once you have your HSM provisioned, add your private key to the keyvault

$ az keyvault key import --KeylessHSM

This will return a URI that you will later add to the Keyless YAML file to indicate where your private key is stored.

Now that you have your HSM provisioned and activated, you need to create a VM where you will deploy the Keyless daemon. For this example, we will create an Ubuntu Xenial VM in Azure. In the portal, go to the Virtual machines page and Add a VM. There, you can use the resource group that you created earlier for the HSM. For best results, choose the same region as the one for the HSM. Note the public IP of the VM, you will use this to remotely connect to the server.

Next, configure your VM as a key server. First, you need to add the Cloudflare Package Repository. Then, you will update your OS’ package listings and install the gokeyless server.

$ apt-get update
$ echo 'deb http://pkg.cloudflare.com/ Xenial main' |
sudo tee /etc/apt/sources.list.d/cloudflare-main.list
$ curl -C - https://pkg.cloudflare.com/pubkey.gpg | sudo apt-key
$ apt-get update
$ apt-get install gokeyless

Then, update the gokeyless YAML file. There, you will add the hostname of your keyserver — this hostname should have a DNS record that points to your VM, the zoneID, and Origin CA API key. The zoneID and Origin CA API key can be found in the Cloudflare dashboard. In addition to that, indicate the URI that to points to your private key’s directory under private_key_stores.

$ vim /etc/keyless/gokeyless.yaml

Lastly, start the keyless server.

$ service gokeyless start

Go back to the Azure portal and open the required TCP ports for the Azure VM. Go to your VM → Networking → Add inbound port rule. Make sure you allow traffic on any source port and indicate Port 2407 for the destination port.

Save the change, then go to the Cloudflare dashboard to upload your SSL certificate. You should see “Upload Keyless SSL Certificate” in the Edge Certificates section of the SSL/TLS tab. There, enter the fields with the key server label, the hostname in the YAML file, the key server port-- 2407 is the default port, and paste the SSL certificate.

Next you’re ready to test! Run curl -v https://zone.com and check that the TLS handshake is successfully completed.

Microsoft Azure Dedicated HSM

In addition to the Managed HSM offering that is now in public preview, Azure customers can configure Cloudflare’s edge to utilize keys stored in Microsoft’s Dedicated HSM offering, based on the SafeNet Luna Network HSM 7 Model A790 series.

Azure Dedicated HSM is validated against both FIPS 140-2 Level 3 and eIDAS Common Criteria EAL4+. After following the instructions to deploy the HSM, customers should follow the Azure specific Keyless SSL instructions here.

Amazon Web Services (AWS) Cloud HSM

AWS CloudHSM also provides FIPS 140-2 Level 3 validated HSMs to store your private keys.

The official AWS Terraform Provider now includes support for the aws_cloudhsm_v2_cluster, which is the version that Cloudflare supports. After provisioning the AWS CloudHSM cluster, customers should follow the AWS specific Keyless SSL instructions here.

Google Cloud HSM

Google Cloud HSM uses GCP’s Cloud KMS as its front end, and allows hosting of keys in FIPS 140-2 Level 3 validated HSMs. Additionally, Google offers the ability to host your own HSM in Google provided space; it is recommended that you contact your GCP account representative for additional information about this option.

Once the key ring and key have been created within the HSM, customers should follow the Google Cloud specific Keyless SSL instructions here. Additional details on using asymmetric keys with GCP KMS can be found here.

IBM Cloud

IBM Cloud HSM 7.0 provides FIPS 140-2 Level 3 validated HSM capabilities. The offering is based on the SafeNet Luna A750 series.

After provisioning the HSM, customers should refer to the IBM specific Keyless SSL instructions here.

Getting help and providing feedback

HSMs offer strong key protection capabilities, but can be complicated to set up and deploy. If you need assistance deploying the HSM on your cloud provider, we suggest that you start with their support channels.

However, if you need assistance configuring the HSM to work with Cloudfare’s edge, or would like to provide feedback on the process, you should reach out directly to your Solutions Engineering team who can put you in touch with Cloudflare’s Keyless SSL specialists.