Better InternetHTTP3Open SourcePrivacyQUICRust

We’re open sourcing our privacy proxy CLI

Hannah Wang, Ben Yang, and Fisher Darling

8 minute read

Debugging privacy-preserving protocols is hard. Oblivious HTTP has several different steps across four different parties, not to mention binary HTTP encoding and details spread across many draft RFCs. We've taken what we've learned operating protocols like Oblivious HTTP at the scale of millions of requests per second, and wrapped it up in a nice, clean CLI tool — that we are open sourcing today.

We call it our privacy-client, or pvcli. We’re releasing it under the Apache-2.0 License, and it is open for contributions.

Here’s a single line of code that executes a full Oblivious HTTP request with a relay, gateway and origin. Don't worry if you don't know what that means, we'll cover it below.

pvcli --ohttp \
   --first-hop https://relay-cloudflare.ohttp.info \
   --proxy https://gateway.ohttp.info \
   -X POST \
   --header "content-type: application/json" \
   --data '{"test":1}' \
   https://target.ohttp.info/anything

We’ll explain why we built this tool, and show just how handy it can be.

Why privacy protocols can be hard to debug

Let’s take a closer look at our motivation for creating pvcli. Over time, the Privacy team’s product suite and customer base grew. We added products like Privacy Proxy and Privacy Gateway, which power Apple’s Private Relay, Microsoft’s Edge Secure Network VPN, Flo Health’s Anonymous Mode, and more. With it came an increasing amount of special customer requirements, domain knowledge, and complexity. As a result, we saw increased friction in development and incident response.

To see this in action, let’s look at how one of our products implements Oblivious HTTP, also known as OHTTP. First, a quick primer. OHTTP provides users with a privacy guarantee: no one can know both who made a request and what they’re requesting. To achieve this, OHTTP requires two servers, a relay and a gateway, operated by two non-colluding parties. 

Below is a sequence diagram of OHTTP, where our customer owns the relay and Cloudflare owns the gateway. At a high level, OHTTP can be broken down into these steps:

  1. Client gets public key from the gateway.
  2. Client encrypts the request and sends it to the relay.
  3. Relay removes “who” the client is from the encrypted request, and sends it to the gateway.
  4. Gateway decrypts the request, and sends it to the target.
  5. Target processes the request, and sends a response to the gateway.
  6. Gateway encrypts the response, and sends it to the relay.
  7. Relay sends the encrypted response to the client.
  8. Client decrypts, and gets the plaintext response.

It involves quite a bit of back-and-forth, as you can see: 

BLOG-3291-diagram.png

Each step is a potential point of failure that we have to consider while debugging!

In particular, we saw certain kinds of problems when debugging OHTTP.

  • Customers asked for ways to test the live system from their end, and we often wrote one-off, custom clients for our customers specific deployments.
  • Figuring out which step caused an issue was time-consuming. Was the root cause a bug in our system or our customer’s system?
  • Examining raw bits was tedious and highly prone to human error. OHTTP builds on binary HTTP, which is a binary encoded HTTP request. Anytime we needed to check the binary encoding, we were painstakingly going through raw bits.

As a result, we decided to place all of our privacy protocols in one tool. It has a clean interface that’s already familiar, displays every single step of the protocol in order, and is flexible enough to support new protocols and architectures.

To see the difference this makes, let’s see an OHTTP debugging scenario — before and after pvcli.

Debugging without pvcli

Say we operate an OHTTP relay that sits in front of a customer's gateway. The customer has asked us to do an end-to-end test with a request:

POST /anything HTTP/1.1
Host: target.ohttp.info
Content-Type: application/json

{"test":1}

Recall the OHTTP steps from earlier. The first step is to fetch the public key from the gateway. We use curl to fetch it from the customer gateway and get this back:

0029550020b9bb667e2230dc01c6d6cc047f94a1083beb185c63e50ec09f7692a5a083254000040001000104a9280041c8321a49d6196fc82c05f48dc7a8c6fc24b15b60ae2685ca17810fc3b5424ee70265149236f254b832bf1f333fb899aac62709839a5aae3363f3133c35213323db18a9a490edb0a9f76728af0206e5754bdae82270b7978fc3226c015344f20b60dba799d6c9695a60ca16bf675625b6071f6cac844e08968a0970ec63cc8c6a900af2ca2eb279cc5248c3f09ed5140f9e3aa0a7b055adfa862e1552c5656e2691c2378442d2131ed5b863e9183fb161821ee0c923ec51b7864df068ca2b5470590736a6e01ae2194e84c46165b25d082b9df2d15ac1da785081c899166bb93a8c0865938380aa2012a26d745320b4c7e5a941eb8976a4534ac2cc9c9c99ccb5403dbe480d032ba4ca8a116564...[SNIP]

That’s a big binary string in hex. To make sense of it, we look at OHTTP RFC 9458 §3 and parse it manually:

  • 0029 is 41 in decimal, telling us this public key entry has 41 bytes associated with it.
  • 55 is the public key ID.
  • 0020 identifies the asymmetric encryption method we can use. In this case, DHKEM(X25519, HKDF-SHA256).
  • b9bb667e2230dc01c6d6cc047f94a1083beb185c63e50ec09f7692a5a0832540 is the public key.
  • 0004 tells us there are 4 bytes of symmetric cryptographic IDs that follow.
  • 0001 and 0001 identify the symmetric encryption methods we can use: HKDF-SHA256 and AES-128-GCM.

We repeat this process for however many public keys are in the binary string.

Next, we convert our original HTTP request into binary HTTP, referencing RFC 9292. We manually craft the binary with the help of some bespoke scripts:

0204504f5354056874747073117461726765742e6f687474702e696e666f092f616e797468696e670c636f6e74656e742d74797065106170706c69636174696f6e2f6a736f6e0a757365722d6167656e740c6665727265742f302e312e30000a7b2274657374223a317d2000

We verify each field:

  • 02 means it's an indeterminate-length request
  • 04504f5354 is POST
  • 056874747073 is https
  • 117461726765742e6f687474702e696e666f is target.ohttp.info
  • and so on

Finally, we form a wrapper HTTP request that will hold our OHTTP request. To do so, we spend some more time writing another makeshift script that encrypts the binary HTTP request in the manner OHTTP specifies, using the public key from earlier. We create a header, which is the concatenation of public key ID, asymmetric encryption method ID, and symmetric encryption method IDs. Then, we concatenate header and encrypted binary HTTP request, resulting in:

5500200001000164f03cef18f625f1dcd9fb26aa802081196bd6d7ba225bf60d3ba65f4f669d59f22d4d37dfd1e45ac9d5097bd7d8e186576ebff850509b5708fe1dbe4c88a7b93b95d153ee203382979063603fb1759bcc41049ac40950f80564de9d45fe8aa4e24259e313da4768b469bbb03037b9b0c34ae5f1b65dea09b8d25f2d5bb4833b68d516a3b19f0265841d5f23a11e74a8d17a04b616e888dd60f147

We put those bytes into the body of our wrapper HTTP request, and send it to our relay. We get back a response.

malformed request

What does that mean? We reach out to the customer to ask if they can share logs from their gateway. In the meantime, we double-check the bits we've crafted. The decoded public keys look fine. The binary HTTP request... Oh! We see:

...0a 7b2274657374223a317d 20 00 <- spaces added for human readability

BHTTP is length prefixed. That means we specify a length (0x0a is 10 in decimal), and then 10 bytes follow. But here, 11 bytes follow. There is an extra 20 before the 00. 20 represents a space character, so we must have accidentally added that when building the body. We remove the extra character, resend, and it works!

Debugging with pvcli

With pvcli, all of that is now a single command:

pvcli -vvv --ohttp \
    --first-hop https://relay-cloudflare.ohttp.info \
    --proxy https://gateway.ohttp.info \
    -X POST \
    --header "content-type: application/json" \
    --data '{"test":1}' \
    https://target.ohttp.info/anything

It handles all the binary parsing and encrypting for us, and prints logs in case we want to dive deeper:

TRCE Full decoded client config: ClientConfig { key_configs: [KeyConfig {
  key_id: 85,
  key: PublicKey { kem_id: X25519HkdfSha256, bytes: b9bb667e...0832540 },
  symmetric_algorithms: [(HkdfSha256, AesGcm128)] },
  ...
] }
...
DEBG Creating POST request to https://target.ohttp.info/anything
DEBG Parsed URI: scheme: Some("https"), host: Some("target.ohttp.info"), port: None, path: /anything
TRCE Building request with headers: ["content-type: application/json", "User-Agent:pvcli/0.1.0"], body: Some(b"{\"test\":1}")
...
TRCE BHTTP encoded request bytes, hex: 0204504f5354056874747073117461726765742e6f687474702e696e666f092f616e797468696e670c636f6e74656e742d74797065106170706c69636174696f6e2f6a736f6e0a757365722d6167656e740b7076636c692f302e312e30000a7b2274657374223a317d00
...
TRCE Encrypted request bytes, hex: 5500200001000164f03cef18f625f1dcd9fb26aa802081196bd6d7ba225bf60d3ba65f4f669d59f22d4d37dfd1e45ac9d5097bd7d8e186576ebff850509b5708fe1dbe4c88a7b93b95d153ee203382979063603fb1759bcc41049ac40950f80564de9d45fe8aa4e24259e313da4768b469bbb03037b9b0c34ae5f1b65dea09b8d25f2d5bb4833b68d516a3b19f0265841d5f23a11e74a8d17a04b616e888dd60f147

What used to be a fragile process — involving manipulating bits, gluing together scripts, and referencing long RFCs — is now one command.

What pvcli can do

To install:

# install rust if not already installed
curl https://sh.rustup.rs -sSf | sh
# install pvcli
cargo install --git https://github.com/cloudflareresearch/pvcli

pvcli takes a lot of inspiration from curl. We designed it with the “principle of least surprise” in mind. As a result, a lot of the arguments are the same as curl’s! Try a quick GET request to our cdn-cgi endpoint:

pvcli https://cloudflare.com/cdn-cgi/trace
pvcli --http3 https://cloudflare.com/cdn-cgi/trace

If you’re curious about what is happening under the hood, you can use -v to get detailed logs:

# more logs
pvcli -v --http3 https://cloudflare.com/cdn-cgi/trace
# even more logs
pvcli -vvv --http3 https://cloudflare.com/cdn-cgi/trace

Now, about that OHTTP command from earlier: you use --ohttp to tell pvcli to construct an OHTTP request. You pass in the relay as the --first-hop and the gateway as the --proxy. The target will be an echo server, so you can see what the target would see. In this command, we filled in the arguments with a relay, gateway, and target from ohttp.info.

pvcli -vvv --ohttp \
    --first-hop https://relay-cloudflare.ohttp.info \
    --proxy https://gateway.ohttp.info \
    -X POST \
    --header "content-type: application/json" \
    --data '{"test":1}' \
    https://target.ohttp.info/anything
# abridged output below
...
{
  "request": {
    "method": "POST",
    "url": "https://target.ohttp.info/anything",
    "headers": {
      "content-length": "10",
      "content-type": "application/json",
      "user-agent": "pvcli/0.1.0",
      "x-ohttp-gateway": "true"
    },
    "body": {
      "test": 1
    }
  },
  "cf": {
    "ip": "Unknown",
    "country": "Unknown",
    "colo": "Unknown",
    "asn": "Unknown",
    "asOrganization": "Unknown"
  },
  "message": "Hello from Lilo's OHTTP target!",
...

Try running the command yourself!

We’ve encountered many cases where we wanted to pass headers to the relay, rather than the target. You are able to do that with --first-hop-header:

pvcli -vvv --ohttp \
    --first-hop https://relay-cloudflare.ohttp.info \
    --first-hop-header "authorization: Bearer relay-token" \
    --proxy https://gateway.ohttp.info \
    -X POST \
    --header "content-type: application/json" \
    --data '{"test":1}' \
    https://target.ohttp.info/anything

Similarly, we’ve also had cases where we wanted to authenticate to the relay with mTLS, to ensure that the correct client is talking with the correct relay. To do that, you can use --first-hop-client and --first-hop-key.

pvcli -vvv --ohttp \
    --first-hop https://relay-cloudflare.ohttp.info \
    --first-hop-client ./relay-client.pem \
    --first-hop-key ./relay-client.key \
    --proxy https://gateway.ohttp.info \
    -X POST \
    --header "content-type: application/json" \
    --data '{"test":1}' \
    https://target.ohttp.info/anything

And it just works. Need to test a full Oblivious HTTP request with a relay, a gateway, arbitrary headers, and mTLS? Or perhaps only request through a gateway? Or maybe you just want to see the OHTTP key configuration? pvcli can do it with a single command, debugging included.

Why build our own tool?

There are some great tools for OHTTP that already exist. Martin Thomson’s Rust implementation and Chris Wood’s Go implementation were incredibly helpful when we built out our original OHTTP implementation a few years ago. But pvcli is not only focused on OHTTP. We’re looking to add as many privacy-preserving protocols as we can to the tool. So while there are other OSS tools out there for debugging OHTTP, nothing combines OHTTP, CONNECT proxying, MASQUE and Privacy Pass (coming soon) all in one place.

Contribute to pvcli

Oblivious HTTP is an amazing protocol, and we would love to see you use it. We hope that this tool helps people debug OHTTP and write their own OHTTP implementations. 

We are accepting contributions! To get started, clone the repo at https://github.com/cloudflareresearch/pvcli, and submit a pull request. 

If you're looking for ways to contribute, here are some things on our to-do list. For MASQUE, we plan to add support for proxying TCP over HTTP/3, and UDP and/or IP over HTTP/2 and HTTP/3. For OHTTP, we plan to support post-quantum cryptography, add timing/latency information, support Chunked OHTTP, and improve logging.

Contact us if you are interested in using Cloudflare’s OHTTP Relays and Gateways.