• When: a cold San Francisco summer afternoon

• Where: Room 305, Cloudflare

• Who: 2 from Cloudflare + 9 from the Tor Project

What could go wrong?

Two years ago this week Cloudflare introduced Opportunistic Encryption, a feature that provided additional security and performance benefits to websites that had not yet moved to HTTPS. Indeed, back in the old days some websites only used HTTP --- weird, right? “Opportunistic” here meant that the server advertised support for HTTP/2 via an HTTP Alternative Service header in the hopes that any browser that recognized the protocol could take advantage of those benefits in subsequent requests to that domain.

Around the same time, CEO Matthew Prince wrote about the importance and challenges of privacy on the Internet and tasked us to find a solution that provides convenience, security, and anonymity.

From neutralizing fingerprinting vectors and everyday browser trackers that Privacy Badger feeds on, all the way to mitigating correlation attacks that only big actors are capable of, guaranteeing privacy is a complicated challenge. Fortunately, the Tor Project addresses this extensive adversary model in Tor Browser.

However, the Internet is full of bad actors, and distinguishing legitimate traffic from malicious traffic, which is one of Cloudflare’s core features, becomes much more difficult when the traffic is anonymous. In particular, many features that make Tor a great tool for privacy also make it a tool for hiding the source of malicious traffic. That is why many resort to using CAPTCHA challenges to make it more expensive to be a bot on the Tor network. There is, however, a collateral damage associated with using CAPTCHA challenges to stop bots: humans eyes also have to deal with them.

One way to minimize this is using privacy-preserving cryptographic signatures, aka blinded tokens, such as those that power Privacy Pass.

The other way is to use onions.

Today’s edition of the Crypto Week introduces an “opportunistic” solution to this problem, so that under suitable conditions, anyone using Tor Browser 8.0 will benefit from improved security and performance when visiting Cloudflare websites without having to face a CAPTCHA. At the same time, this feature enables more fine-grained rate-limiting to prevent malicious traffic, and since the mechanics of the idea described here are not specific to Cloudflare, anyone can reuse this method on their own website.

Before we continue, if you need a refresher on what Tor is or why we are talking about onions, check out the Tor Project website or our own blog post on the DNS resolver onion from June.

As Matthew mentioned in his blog post, one way to sift through Tor traffic is to use the onion service protocol. Onion services are Tor nodes that advertise their public key, encoded as an address with .onion TLD, and use “rendezvous points” to establish connections entirely within the Tor network:

While onion services are designed to provide anonymity for content providers, media organizations use them to allow whistleblowers to communicate securely with them and Facebook uses one to tell Tor users from bots.

The technical reason why this works is that from an onion service’s point of view each individual Tor connection, or circuit, has a unique but ephemeral number associated to it, while from a normal server’s point of view all Tor requests made via one exit node share the same IP address. Using this circuit number, onion services can distinguish individual circuits and terminate those that seem to behave maliciously. To clarify, this does not mean that onion services can identify or track Tor users.

While bad actors can still establish a fresh circuit by repeating the rendezvous protocol, doing so involves a cryptographic key exchange that costs time and computation. Think of this like a cryptographic dial-up sequence. Spammers can dial our onion service over and over, but every time they have to repeat the key exchange.

Alternatively, finishing the rendezvous protocol can be thought of as a small proof of work required in order to use the Cloudflare Onion Service. This increases the cost of using our onion service for performing denial of service attacks.

Not quite. As discussed when we introduced the hidden resolver, the problem of ensuring that a seemingly random .onion address is correct is a barrier to usable security. In that case, our solution was to purchase an Extended Validation (EV) certificate, which costs considerably more. Needless to say, this limits who can buy an HTTPS certificate for their onion service to a privileged few.

Some people disagree. In particular, the new generation of onion services resolves the weakness that Matthew pointed to as a possible reason why the CA/B Forum only permits EV certificates for onion services. This could mean that getting Domain Validation (DV) certificates for onion services could be possible soon. We certainly hope that’s the case.

Still, DV certificates lack the organization name (e.g. “Cloudflare, Inc.”) that appears in the address bar, and cryptographically relevant numbers are nearly impossible to remember or distinguish for humans. This brings us back to the problem of usable security, so we came up with a different idea.

Forget for a moment that we’re discussing anonymity. When you type “cloudflare.com” in a browser and press enter, your device first resolves that domain name into an IP address, then your browser asks the server for a certificate valid for “cloudflare.com” and attempts to establish an encrypted connection with the host. As long as the certificate is trusted by a certificate authority, there’s no reason to mind the IP address.

Roughly speaking, the idea here is to simply switch the IP address in the scenario above with an .onion address. As long as the certificate is valid, the .onion address itself need not be manually entered by a user or even be memorable. Indeed, the fact that the certificate was valid indicates that the .onion address was correct.

In particular, in the same way that a single IP address can serve millions of domains, a single .onion address should be able to serve any number of domains.

Except, DNS doesn’t work this way.

Just as with Opportunistic Encryption, we can point users to the Cloudflare Onion Service using HTTP Alternative Services, a mechanism that allows servers to tell clients that the service they are accessing is available at another network location or over another protocol. For instance, when Tor Browser makes a request to “cloudflare.com,” Cloudflare adds an Alternative Service header to indicate that the site is available to access over HTTP/2 via our onion services.

In the same sense that Cloudflare owns the IP addresses that serve our customers’ websites, we run 10 .onion addresses. Think of them as 10 Cloudflare points of presence (or PoPs) within the Tor network. The exact header looks something like this, except with all 10 .onion addresses included, each starting with the prefix “cflare”:

alt-svc: h2="cflare2nge4h4yqr3574crrd7k66lil3torzbisz6uciyuzqc2h2ykyd.onion:443"; ma=86400; persist=1

This simply indicates that the “cloudflare.com” can be authoritatively accessed using HTTP/2 (“h2”) via the onion service “cflare2n[...].onion”, over virtual port 443. The field “ma” (max-age) indicates how long in seconds the client should remember the existence of the alternative service and “persist” indicates whether alternative service cache should be cleared when the network is interrupted.

Once the browser receives this header, it attempts to make a new Tor circuit to the onion service advertised in the alt-svc header and confirm that the server listening on virtual port 443 can present a valid certificate for “cloudflare.com” — that is, the original hostname, not the .onion address.

The onion service then relays the Client Hello packet to a local server which can serve a certificate for “cloudflare.com.” This way the Tor daemon itself can be very minimal. Here is a sample configuration file:

SocksPort 0
HiddenServiceNonAnonymousMode 1
HiddenServiceSingleHopMode 1
HiddenServiceVersion 3
HiddenServicePort 443
SafeLogging 1
Log notice stdout

Be careful with using the configuration above, as it enables a non-anonymous setting for onion services that do not require anonymity for themselves. To clarify, this does not sacrifice privacy or anonymity of Tor users, just the server. Plus, it improves latency of the circuits.

If the certificate is signed by a trusted certificate authority, for any subsequent requests to “cloudflare.com” the browser will connect using HTTP/2 via the onion service, sidestepping the need for going through an exit node.

Here are the steps summarized one more time:

1. A new Tor circuit is established;

2. The browser sends a Client Hello to the onion service with SNI=cloudflare.com;

3. The onion service relays the packet to a local server;

4. The server replies with Server Hello for SNI=cloudflare.com;

5. The onion service relays the packet to the browser;

6. The browser verifies that the certificate is valid.

To reiterate, the certificate presented by the onion service only needs to be valid for the original hostname, meaning that the onion address need not be mentioned anywhere on the certificate. This is a huge benefit, because it allows you to, for instance, present a free Let’s Encrypt certificate for your .org domain rather than an expensive EV certificate.

Convenience, ✓

Remember that while one exit node can serve many many different clients, from Cloudflare’s point of view all of that traffic comes from one IP address. This pooling helps cover the malicious traffic among legitimate traffic, but isn’t essential in the security or privacy of Tor. In fact, it can potentially hurt users by exposing their traffic to bad exit nodes.

Remember that Tor circuits to onion services carry a circuit number which we can use to rate-limit the circuit. Now, the question is how to inform a server such as nginx of this number with minimal effort. As it turns out, with only a small tweak in the Tor binary, we can insert a Proxy Protocol header in the beginning of each packet that is forwarded to the server. This protocol is designed to help TCP proxies pass on parameters that can be lost in translation, such as source and destination IP addresses, and is already supported by nginx, Apache, Caddy, etc.

Luckily for us, the IPv6 space is so vast that we can encode the Tor circuit number as an IP address in an unused range and use the Proxy Protocol to send it to the server. Here is an example of the header that our Tor daemon would insert in the connection:

PROXY TCP6 2405:8100:8000:6366:1234:ABCD ::1 43981 443\r\n

In this case, 0x1234ABCD encodes the circuit number in the last 32 bits of the source IP address. The local Cloudflare server can then transparently use that IP to assign reputation, show CAPTCHAs, or block requests when needed.

Note that even though requests relayed by an onion service don’t carry an IP address, you will see an IP address like the one above with country code “T1” in your logs. This IP only specifies the circuit number seen by the onion service, not the actual user IP address. In fact, 2405:8100:8000::/48 is an unused subnet allocated to Cloudflare that we are not routing globally for this purpose.

This enables customers to continue detecting bots using IP reputation while sparing humans the trouble of clicking on CAPTCHA street signs over and over again.

Security, ✓

You don’t need to. The Cloudflare Onion Service presents the exact same certificate that we would have used for direct requests to our servers, so you could audit this service using Certificate Transparency (which includes Nimbus, our certificate transparency log), to reveal any potential cheating.

Additionally, since Tor Browser 8.0 makes a new circuit for each hostname when connecting via an .onion alternative service, the circuit number cannot be used to link connections to two different sites together.

Note that all of this works without running any entry, relay, or exit nodes. Therefore the only requests that we see as a result of this feature are the requests that were headed for us anyway. In particular, since no new traffic is introduced, Cloudflare does not gain any more information about what people do on the internet.

Anonymity, ✓

Tor isn’t known for being fast. One reason for that is the physical cost of having packets bounce around in a decentralized network. Connections made through the Cloudflare Onion Service don’t add to this cost because the number of hops is no more than usual.

Another reason is the bandwidth costs of exit node operators. This is an area that we hope this service can offer relief since it shifts traffic from exit nodes to our own servers, reducing exit node operation costs along with it.

BONUS: Performance, ✓

Onion Routing is now available to all Cloudflare customers, enabled by default for Free and Pro plans. The option is available in the Crypto tab of the Cloudflare dashboard.

We recommend using Tor Browser 8.0, which is the first stable release based on Firefox 60 ESR, and supports .onion Alt-Svc headers as well as HTTP/2. The new Tor Browser for Android (alpha) also supports this feature. You can check whether your connection is routed through an onion service or not in the Developer Tools window under the Network tab. If you're using the Tor Browser and you don't see the Alt-Svc in the response headers, that means you're already using the .onion route. In future versions of Tor Browser you'll be able to see this in the UI.

We've got BIG NEWS. We gave Tor Browser a UX overhaul.

Tor Browser 8.0 has a new user onboarding experience, an updated landing page, additional language support, and new behaviors for bridge fetching, displaying a circuit, and visiting .onion sites.https://t.co/fpCpSTXT2L pic.twitter.com/xbj9lKTApP

— The Tor Project (@torproject) September 5, 2018

There is also interest from other privacy-conscious browser vendors. Tom Lowenthal, Product Manager for Privacy & Security at Brave said:

Automatic upgrades to `.onion` sites will provide another layer of safety to Brave’s Private Browsing with Tor. We’re excited to implement this emerging standard.

Similar to Opportunistic Encryption, Opportunistic Onions do not fully protect against attackers who can simply remove the alternative service header. Therefore it is important to use HTTPS Everywhere to secure the first request. Once a Tor circuit is established, subsequent requests should stay in the Tor network from source to destination.

As we maintain and improve this service we will share what we learn. In the meanwhile, feel free to try out this idea on Caddy and reach out to us with any comments or suggestions that you might have.

Patrick McManus of Mozilla for enabling support for .onion alternative services in Firefox; Arthur Edelstein of the Tor Project for reviewing and enabling HTTP/2 and HTTP Alternative Services in Tor Browser 8.0; Alexander Færøy and George Kadianakis of the Tor Project for adding support for Proxy Protocol in onion services; the entire Tor Project team for their invaluable assistance and discussions; and last, but not least, many folks at Cloudflare who helped with this project.

cflarexljc3rw355ysrkrzwapozws6nre6xsy3n4yrj7taye3uiby3ad.onion
cflarenuttlfuyn7imozr4atzvfbiw3ezgbdjdldmdx7srterayaozid.onion
cflares35lvdlczhy3r6qbza5jjxbcplzvdveabhf7bsp7y4nzmn67yd.onion
cflareusni3s7vwhq2f7gc4opsik7aa4t2ajedhzr42ez6uajaywh3qd.onion
cflareki4v3lh674hq55k3n7xd4ibkwx3pnw67rr3gkpsonjmxbktxyd.onion
cflarejlah424meosswvaeqzb54rtdetr4xva6mq2bm2hfcx5isaglid.onion
cflaresuje2rb7w2u3w43pn4luxdi6o7oatv6r2zrfb5xvsugj35d2qd.onion
cflareer7qekzp3zeyqvcfktxfrmncse4ilc7trbf6bp6yzdabxuload.onion
cflareub6dtu7nvs3kqmoigcjdwap2azrkx5zohb2yk7gqjkwoyotwqd.onion
cflare2nge4h4yqr3574crrd7k66lil3torzbisz6uciyuzqc2h2ykyd.onion

Subscribe to the blog for daily updates on our announcements.

In case you haven’t heard yet, Cloudflare launched a privacy-first DNS resolver service on April 1st. It was no joke! The service, which was our first consumer-focused service, supports emerging DNS standards such as DNS over HTTPS:443 and TLS:853 in addition to traditional protocols over UDP:53 and TCP:53, all in one easy to remember address: 1.1.1.1.

As it was mentioned in the original blog post, our policy is to never, ever write client IP addresses to disk and wipe all logs within 24 hours. Still, the exceptionally privacy-conscious folks might not want to reveal their IP address to the resolver at all, and we respect that. This is why we are launching a Tor onion service for our resolver at dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion and accessible via tor.cloudflare-dns.com.

NOTE: the hidden resolver is still an experimental service and should not be used in production or for other critical uses until it is more tested.

Imagine an alternative Internet where, in order to connect to www.cloudflare.com, instead of delegating the task of finding a path to our servers to your internet provider, you had to go through the following steps to reach Cloudflare:

1. You calculate a path to your destination, like this:

    You -> Your ISP -> X -> Y -> Z -> www.cloudflare.com.

2. You encrypt your packet with Z’s public key, then with Y’s, and finally with X’s.

3. You submit the result to X, who decrypts with their private key;

4. X submits the result to Y, who decrypts with their private key;

5. Y submits the result to Z, who decrypts with their private key to get the original packet;

6. Z submits the packet to www.cloudflare.com.

If everyone plays their roles correctly, it is possible to ensure only the entry relay X knows your IP address and only the exit relay Z knows the website you’re connecting you, thereby providing you with privacy and anonymity. This is a simplified version of Tor: a collection of volunteer-run computers and servers around the world acting as relays for a huge network built on top of the Internet where every hop from one relay to the next peels one layer of encryption, hence its name: the onion router.

Keeping internet users anonymous is not the only function of the Tor network. In particular, one caveat of the procedure above is that the connection is still accessible by the exit relay and anyone sitting between there and the destination, including network providers. To solve this problem, and to also provide anonymity for content publishers, Tor allows for onion services. Onion services are Tor nodes that advertise their public key, encoded as an address with .onion TLD, and establish connections entirely within the Tor network:

The process of returning an IP address given a domain name is called DNS resolution. Since Tor still uses IP addresses, you still need to do DNS resolution to browse the web over Tor. There are two common methods to resolve a domain name when using Tor:

1. Resolve the name directly, then talk to the IP address through Tor;

2. Ask a Tor exit relay to resolve the name publicly and connect to the IP.

Clearly, the first option leaks your IP to your DNS resolver and, unless your client uses DNS-over-HTTPS or DNS-over-TLS, it leaks your destination name to your ISP. What is less obvious is that the second option can open you to manipulation attacks such as DNS poisoning or sslstrip by bad relays. This is where our new service comes in:

1. Ask a .onion-based resolver service!

In a few words, our .onion-based resolver service is a Tor onion service which forwards all communication on DNS ports to the corresponding ports on 1.1.1.1, hence the apparent client IP is an internal IP rather than yours. There is, however, more than meets the eye.

One glaring difference between using 1.1.1.1 and this service is that the .onion address is "dns4tor" plus 49 seemingly random alphanumeric characters. This 56 character long string, in fact, contains a full Ed25519 public key which is used to secure communication with the onion service. This poses a number of challenges towards usable security:

1. How can the users make sure that that the address is correct?

We simply bought a certificate with tor.cloudflare-dns.com as subject name and the .onion address as a subject alternative name. This way, if you’re in the right place, you should see this:

1. How can the users remember this address?

We don’t think you should need to remember this address. Ideally, all you would need to do is go to https://tor.cloudflare-dns.com and have the browser route your request to the .onion address. This is possible using the "Alt-Svc" HTTP header which is an optional header notifying the browser that the resources can be accessed from an alternative network location, possibly using a different protocol. Thanks to Mozilla, using .onion addresses as alternative services is now possible in Firefox Nightly.

Think of this feature like opportunistic encryption: once your browser receives an Alt-Svc header indicating that a .onion address is available for tor.cloudflare-dns.com, if it knows that .onion addresses can be accessed (for instance through a SOCKS proxy), it attempts to check that the alternative service has the same or a higher level of security. This includes making sure that it is possible to connect to the onion service using the same certificate and Server Name. If that is the case, the browser uses the alternative service instead, therefore ensuring that your future requests do not leave the Tor network.

Here is a thought experiment: suppose between each two points on Earth there is a fiber-optic cable, capable of lossless transmission of packets at the speed of light.

Using a back-of-the-envelope calculation it’s easy to see that, on average, each packet traverses a distance equivalent to a quarter of the circumference of the Earth in about 33ms, while each Tor packet takes about 200ms to go one and a half turns around the Earth before reaching an onion service; that’s three turns for a round trip that ensures anonymity of both parties.

Cloudflare, however, does not require anonymity for its servers, which is why we can reduce the number of relays to just three by enabling an optional setting for onion services that prioritize lower latency over location anonymity of the service. To emphasize, this does not impact client privacy or anonymity whatsoever. Indeed, as you may have noticed, in the first onion service image the origin is three hops away from the rendezvous point whereas our onion service is only one hop away.

We are actively working on developing ways to make this service faster and ensure it has as little downtime as possible.

First and foremost, resolving DNS queries through the Tor network, for instance by connecting to Google’s 8.8.8.8 resolver, guarantees a significantly higher level of anonymity than making the requests directly. Not only does doing so prevent the resolver from ever seeing your IP address, even your ISP won’t know that you’ve attempted to resolve a domain name.

Still, unless the destination is an onion service, passive attackers can capture packets exiting the Tor network and malicious Exit Nodes can poison DNS queries or downgrade encryption through sslstripping. Even if you limit your browsing to only HTTPS sites, passive attackers can find out which addresses you’ve connected to. Even worse, actors capable of comparing traffic both before it enters the Tor network and after it leaves the network can potentially use the metadata (size, time, etc.) to deanonymize the client. The only solution, then, is to eliminate the need for Exit Nodes by using onion services instead. That is what our .onion-based resolver offers.

Moreover, if your client does not support encrypted DNS queries, using a .onion-based resolver can secure the connection from on-path attacks, including BGP hijacking attacks. This means having the same level of security for DNS-over-UDP and DNS-over-TCP as DNS-over-HTTPS and DNS-over-TLS provides.

Your personal anonymity, however, is not the only reason why you should use this service. The power of Tor in ensuring everyone’s anonymity rests on the number of people who use it. If only whistleblowers, for instance, were to use the Tor network, then anyone connecting to the Tor network would automatically be suspected of being a whistleblower. Therefore the more people use Tor to browse memes or to watch cat videos on the Internet, the easier it will be for those who truly need anonymity to blend in with the traffic.

One barrier to using Tor for many users is that it is simply slow, so I can try to sympathize with those who wouldn’t sacrifice quick website load times to help keep activists and dissidents anonymous. That said, DNS requests are small in size and since most browsers and operating systems cache DNS results the total traffic is not significant. As a result, using the .onion-based resolver will only slightly slow down your initial DNS request without slowing down anything else, while still contributing to the overall anonymity of the Tor network and its users.

Using a .onion-based resolver ensures that your ISP never finds out that you’re resolving a domain, the Exit Nodes don’t get a chance to manipulate DNS replies, and the resolver never finds out your IP address. However, the unique benefit of using the Cloudflare .onion-based resolver is combining the power of Tor with all privacy-preserving features of the 1.1.1.1 resolver, such as query name minimization, as well as a team of engineers working on improving it at every level, including standards like DNS-over-HTTPS and DNS-over-TLS.

As CEO Matthew Prince said about two years ago, anonymity online is a cause we value at Cloudflare. In addition, when we announced the 1.1.1.1 resolver we committed to taking every technical step to ensure we can’t know what you do on the internet. Providing a way to use the resolver through the Tor network and making it as fast as possible is a big step in that direction.

The .onion-based resolver supports every DNS protocol that 1.1.1.1 supports, only over the Tor network. However, since not every DNS client is capable of connecting to the Tor network, some hacking is required to get it to work. Here we will explain how to set up DNS-over-HTTPS provided from the .onion-based resolver, but for all other scenarios head to our developers page to get the details of how to use the .onion-based resolver.

Here is how you can set up cloudflared to start a DNS client that uses DNS over HTTPS, routed through the Tor network:

1. First, start with downloading cloudflared by following the regular guide for Running a DNS over HTTPS Client.

2. Start a Tor SOCKS proxy and use socat to forward port TCP:443 to localhost:

    socat TCP4-LISTEN:443,reuseaddr,fork SOCKS4A:127.0.0.1:dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion:443,socksport=9150

3. Instruct your machine to treat the .onion address as localhost:

    cat << EOF >> /etc/hosts
    127.0.0.1 dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion
    EOF

4. Finally, start a local DNS over UDP daemon:

    cloudflared proxy-dns --upstream "https://dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion/dns-query"
    INFO[0000] Adding DNS upstream                           url="https://dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion/dns-query"
    INFO[0000] Starting DNS over HTTPS proxy server          addr="dns://localhost:53"
    INFO[0000] Starting metrics server                       addr="127.0.0.1:35659"

5. Profit!