• When: a cold San Francisco summer afternoon

• Where: Room 305, Cloudflare

• Who: 2 from Cloudflare + 9 from the Tor Project

What could go wrong?

Two years ago this week Cloudflare introduced Opportunistic Encryption, a feature that provided additional security and performance benefits to websites that had not yet moved to HTTPS. Indeed, back in the old days some websites only used HTTP --- weird, right? “Opportunistic” here meant that the server advertised support for HTTP/2 via an HTTP Alternative Service header in the hopes that any browser that recognized the protocol could take advantage of those benefits in subsequent requests to that domain.

Around the same time, CEO Matthew Prince wrote about the importance and challenges of privacy on the Internet and tasked us to find a solution that provides convenience, security, and anonymity.

From neutralizing fingerprinting vectors and everyday browser trackers that Privacy Badger feeds on, all the way to mitigating correlation attacks that only big actors are capable of, guaranteeing privacy is a complicated challenge. Fortunately, the Tor Project addresses this extensive adversary model in Tor Browser.

However, the Internet is full of bad actors, and distinguishing legitimate traffic from malicious traffic, which is one of Cloudflare’s core features, becomes much more difficult when the traffic is anonymous. In particular, many features that make Tor a great tool for privacy also make it a tool for hiding the source of malicious traffic. That is why many resort to using CAPTCHA challenges to make it more expensive to be a bot on the Tor network. There is, however, a collateral damage associated with using CAPTCHA challenges to stop bots: humans eyes also have to deal with them.

One way to minimize this is using privacy-preserving cryptographic signatures, aka blinded tokens, such as those that power Privacy Pass.

The other way is to use onions.

Today’s edition of the Crypto Week introduces an “opportunistic” solution to this problem, so that under suitable conditions, anyone using Tor Browser 8.0 will benefit from improved security and performance when visiting Cloudflare websites without having to face a CAPTCHA. At the same time, this feature enables more fine-grained rate-limiting to prevent malicious traffic, and since the mechanics of the idea described here are not specific to Cloudflare, anyone can reuse this method on their own website.

Before we continue, if you need a refresher on what Tor is or why we are talking about onions, check out the Tor Project website or our own blog post on the DNS resolver onion from June.

As Matthew mentioned in his blog post, one way to sift through Tor traffic is to use the onion service protocol. Onion services are Tor nodes that advertise their public key, encoded as an address with .onion TLD, and use “rendezvous points” to establish connections entirely within the Tor network:

While onion services are designed to provide anonymity for content providers, media organizations use them to allow whistleblowers to communicate securely with them and Facebook uses one to tell Tor users from bots.

The technical reason why this works is that from an onion service’s point of view each individual Tor connection, or circuit, has a unique but ephemeral number associated to it, while from a normal server’s point of view all Tor requests made via one exit node share the same IP address. Using this circuit number, onion services can distinguish individual circuits and terminate those that seem to behave maliciously. To clarify, this does not mean that onion services can identify or track Tor users.

While bad actors can still establish a fresh circuit by repeating the rendezvous protocol, doing so involves a cryptographic key exchange that costs time and computation. Think of this like a cryptographic dial-up sequence. Spammers can dial our onion service over and over, but every time they have to repeat the key exchange.

Alternatively, finishing the rendezvous protocol can be thought of as a small proof of work required in order to use the Cloudflare Onion Service. This increases the cost of using our onion service for performing denial of service attacks.

Not quite. As discussed when we introduced the hidden resolver, the problem of ensuring that a seemingly random .onion address is correct is a barrier to usable security. In that case, our solution was to purchase an Extended Validation (EV) certificate, which costs considerably more. Needless to say, this limits who can buy an HTTPS certificate for their onion service to a privileged few.

Some people disagree. In particular, the new generation of onion services resolves the weakness that Matthew pointed to as a possible reason why the CA/B Forum only permits EV certificates for onion services. This could mean that getting Domain Validation (DV) certificates for onion services could be possible soon. We certainly hope that’s the case.

Still, DV certificates lack the organization name (e.g. “Cloudflare, Inc.”) that appears in the address bar, and cryptographically relevant numbers are nearly impossible to remember or distinguish for humans. This brings us back to the problem of usable security, so we came up with a different idea.

Forget for a moment that we’re discussing anonymity. When you type “cloudflare.com” in a browser and press enter, your device first resolves that domain name into an IP address, then your browser asks the server for a certificate valid for “cloudflare.com” and attempts to establish an encrypted connection with the host. As long as the certificate is trusted by a certificate authority, there’s no reason to mind the IP address.

Roughly speaking, the idea here is to simply switch the IP address in the scenario above with an .onion address. As long as the certificate is valid, the .onion address itself need not be manually entered by a user or even be memorable. Indeed, the fact that the certificate was valid indicates that the .onion address was correct.

In particular, in the same way that a single IP address can serve millions of domains, a single .onion address should be able to serve any number of domains.

Except, DNS doesn’t work this way.

Just as with Opportunistic Encryption, we can point users to the Cloudflare Onion Service using HTTP Alternative Services, a mechanism that allows servers to tell clients that the service they are accessing is available at another network location or over another protocol. For instance, when Tor Browser makes a request to “cloudflare.com,” Cloudflare adds an Alternative Service header to indicate that the site is available to access over HTTP/2 via our onion services.

In the same sense that Cloudflare owns the IP addresses that serve our customers’ websites, we run 10 .onion addresses. Think of them as 10 Cloudflare points of presence (or PoPs) within the Tor network. The exact header looks something like this, except with all 10 .onion addresses included, each starting with the prefix “cflare”:

alt-svc: h2="cflare2nge4h4yqr3574crrd7k66lil3torzbisz6uciyuzqc2h2ykyd.onion:443"; ma=86400; persist=1

This simply indicates that the “cloudflare.com” can be authoritatively accessed using HTTP/2 (“h2”) via the onion service “cflare2n[...].onion”, over virtual port 443. The field “ma” (max-age) indicates how long in seconds the client should remember the existence of the alternative service and “persist” indicates whether alternative service cache should be cleared when the network is interrupted.

Once the browser receives this header, it attempts to make a new Tor circuit to the onion service advertised in the alt-svc header and confirm that the server listening on virtual port 443 can present a valid certificate for “cloudflare.com” — that is, the original hostname, not the .onion address.

The onion service then relays the Client Hello packet to a local server which can serve a certificate for “cloudflare.com.” This way the Tor daemon itself can be very minimal. Here is a sample configuration file:

SocksPort 0
HiddenServiceNonAnonymousMode 1
HiddenServiceSingleHopMode 1
HiddenServiceVersion 3
HiddenServicePort 443
SafeLogging 1
Log notice stdout

Be careful with using the configuration above, as it enables a non-anonymous setting for onion services that do not require anonymity for themselves. To clarify, this does not sacrifice privacy or anonymity of Tor users, just the server. Plus, it improves latency of the circuits.

If the certificate is signed by a trusted certificate authority, for any subsequent requests to “cloudflare.com” the browser will connect using HTTP/2 via the onion service, sidestepping the need for going through an exit node.

Here are the steps summarized one more time:

1. A new Tor circuit is established;

2. The browser sends a Client Hello to the onion service with SNI=cloudflare.com;

3. The onion service relays the packet to a local server;

4. The server replies with Server Hello for SNI=cloudflare.com;

5. The onion service relays the packet to the browser;

6. The browser verifies that the certificate is valid.

To reiterate, the certificate presented by the onion service only needs to be valid for the original hostname, meaning that the onion address need not be mentioned anywhere on the certificate. This is a huge benefit, because it allows you to, for instance, present a free Let’s Encrypt certificate for your .org domain rather than an expensive EV certificate.

Convenience, ✓

Remember that while one exit node can serve many many different clients, from Cloudflare’s point of view all of that traffic comes from one IP address. This pooling helps cover the malicious traffic among legitimate traffic, but isn’t essential in the security or privacy of Tor. In fact, it can potentially hurt users by exposing their traffic to bad exit nodes.

Remember that Tor circuits to onion services carry a circuit number which we can use to rate-limit the circuit. Now, the question is how to inform a server such as nginx of this number with minimal effort. As it turns out, with only a small tweak in the Tor binary, we can insert a Proxy Protocol header in the beginning of each packet that is forwarded to the server. This protocol is designed to help TCP proxies pass on parameters that can be lost in translation, such as source and destination IP addresses, and is already supported by nginx, Apache, Caddy, etc.

Luckily for us, the IPv6 space is so vast that we can encode the Tor circuit number as an IP address in an unused range and use the Proxy Protocol to send it to the server. Here is an example of the header that our Tor daemon would insert in the connection:

PROXY TCP6 2405:8100:8000:6366:1234:ABCD ::1 43981 443\r\n

In this case, 0x1234ABCD encodes the circuit number in the last 32 bits of the source IP address. The local Cloudflare server can then transparently use that IP to assign reputation, show CAPTCHAs, or block requests when needed.

Note that even though requests relayed by an onion service don’t carry an IP address, you will see an IP address like the one above with country code “T1” in your logs. This IP only specifies the circuit number seen by the onion service, not the actual user IP address. In fact, 2405:8100:8000::/48 is an unused subnet allocated to Cloudflare that we are not routing globally for this purpose.

This enables customers to continue detecting bots using IP reputation while sparing humans the trouble of clicking on CAPTCHA street signs over and over again.

Security, ✓

You don’t need to. The Cloudflare Onion Service presents the exact same certificate that we would have used for direct requests to our servers, so you could audit this service using Certificate Transparency (which includes Nimbus, our certificate transparency log), to reveal any potential cheating.

Additionally, since Tor Browser 8.0 makes a new circuit for each hostname when connecting via an .onion alternative service, the circuit number cannot be used to link connections to two different sites together.

Note that all of this works without running any entry, relay, or exit nodes. Therefore the only requests that we see as a result of this feature are the requests that were headed for us anyway. In particular, since no new traffic is introduced, Cloudflare does not gain any more information about what people do on the internet.

Anonymity, ✓

Tor isn’t known for being fast. One reason for that is the physical cost of having packets bounce around in a decentralized network. Connections made through the Cloudflare Onion Service don’t add to this cost because the number of hops is no more than usual.

Another reason is the bandwidth costs of exit node operators. This is an area that we hope this service can offer relief since it shifts traffic from exit nodes to our own servers, reducing exit node operation costs along with it.

BONUS: Performance, ✓

Onion Routing is now available to all Cloudflare customers, enabled by default for Free and Pro plans. The option is available in the Crypto tab of the Cloudflare dashboard.

We recommend using Tor Browser 8.0, which is the first stable release based on Firefox 60 ESR, and supports .onion Alt-Svc headers as well as HTTP/2. The new Tor Browser for Android (alpha) also supports this feature. You can check whether your connection is routed through an onion service or not in the Developer Tools window under the Network tab. If you're using the Tor Browser and you don't see the Alt-Svc in the response headers, that means you're already using the .onion route. In future versions of Tor Browser you'll be able to see this in the UI.

We've got BIG NEWS. We gave Tor Browser a UX overhaul.

Tor Browser 8.0 has a new user onboarding experience, an updated landing page, additional language support, and new behaviors for bridge fetching, displaying a circuit, and visiting .onion sites.https://t.co/fpCpSTXT2L pic.twitter.com/xbj9lKTApP

— The Tor Project (@torproject) September 5, 2018

There is also interest from other privacy-conscious browser vendors. Tom Lowenthal, Product Manager for Privacy & Security at Brave said:

Automatic upgrades to `.onion` sites will provide another layer of safety to Brave’s Private Browsing with Tor. We’re excited to implement this emerging standard.

Similar to Opportunistic Encryption, Opportunistic Onions do not fully protect against attackers who can simply remove the alternative service header. Therefore it is important to use HTTPS Everywhere to secure the first request. Once a Tor circuit is established, subsequent requests should stay in the Tor network from source to destination.

As we maintain and improve this service we will share what we learn. In the meanwhile, feel free to try out this idea on Caddy and reach out to us with any comments or suggestions that you might have.

Patrick McManus of Mozilla for enabling support for .onion alternative services in Firefox; Arthur Edelstein of the Tor Project for reviewing and enabling HTTP/2 and HTTP Alternative Services in Tor Browser 8.0; Alexander Færøy and George Kadianakis of the Tor Project for adding support for Proxy Protocol in onion services; the entire Tor Project team for their invaluable assistance and discussions; and last, but not least, many folks at Cloudflare who helped with this project.

cflarexljc3rw355ysrkrzwapozws6nre6xsy3n4yrj7taye3uiby3ad.onion
cflarenuttlfuyn7imozr4atzvfbiw3ezgbdjdldmdx7srterayaozid.onion
cflares35lvdlczhy3r6qbza5jjxbcplzvdveabhf7bsp7y4nzmn67yd.onion
cflareusni3s7vwhq2f7gc4opsik7aa4t2ajedhzr42ez6uajaywh3qd.onion
cflareki4v3lh674hq55k3n7xd4ibkwx3pnw67rr3gkpsonjmxbktxyd.onion
cflarejlah424meosswvaeqzb54rtdetr4xva6mq2bm2hfcx5isaglid.onion
cflaresuje2rb7w2u3w43pn4luxdi6o7oatv6r2zrfb5xvsugj35d2qd.onion
cflareer7qekzp3zeyqvcfktxfrmncse4ilc7trbf6bp6yzdabxuload.onion
cflareub6dtu7nvs3kqmoigcjdwap2azrkx5zohb2yk7gqjkwoyotwqd.onion
cflare2nge4h4yqr3574crrd7k66lil3torzbisz6uciyuzqc2h2ykyd.onion

Subscribe to the blog for daily updates on our announcements.

The Internet is an amazing invention. We marvel at how it connects people, connects ideas, and makes the world smaller. But the Internet isn’t perfect. It was put together piecemeal through publicly funded research, private investment, and organic growth that has left us with an imperfect tapestry. It’s also evolving. People are constantly developing creative applications and finding new uses for existing Internet technology. Issues like privacy and security that were afterthoughts in the early days of the Internet are now supremely important. People are being tracked and monetized, websites and web services are being attacked in interesting new ways, and the fundamental system of trust the Internet is built on is showing signs of age. The Internet needs an upgrade, and one of the tools that can make things better is cryptography.

Every day this week, Cloudflare will be announcing support for a new technology that uses cryptography to make the Internet better (hint: subscribe to the blog to make sure you don't miss any of the news). Everything we are announcing this week is free to use and provides a meaningful step towards supporting a new capability or structural reinforcement. So why are we doing this? Because it’s good for the users and good for the Internet. Welcome to Crypto Week!

• Cloudflare goes InterPlanetary - Introducing Cloudflare’s IPFS Gateway

• End-to-End Integrity with IPFS

• Expanding DNSSEC Adoption

• RPKI - The required cryptographic upgrade to BGP routing

• RPKI and BGP: our path to securing Internet Routing

• Introducing the Cloudflare Onion Service

• Roughtime: Securing Time with Digital Signatures

Everything we do online depends on a relationship between users, services, and networks that is supported by some sort of trust mechanism. These relationships can be physical (I plug my router into yours), contractual (I paid a registrar for this domain name), or reliant on a trusted third party (I sent a message to my friend on iMessage via Apple). The simple act of visiting a website involves hundreds of trust relationships, some explicit and some implicit. The sheer size of the Internet and number of parties involved make trust online incredibly complex. Cryptography is a tool that can be used to encode and enforce, and most importantly scale these trust relationships.

To illustrate this, let’s break down what happens when you visit a website. But before we can do this, we need to know the jargon.

• Autonomous Systems (100 thousand or so active): An AS corresponds to a network provider connected to the Internet. Each has a unique Autonomous System Number (ASN).

• IP ranges (1 million or so): Each AS is assigned a set of numbers called IP addresses. Each of these IP addresses can be used by the AS to identify a computer on its network when connecting to other networks on the Internet. These addresses are assigned by the Regional Internet Registries (RIR), of which there are 5. Data sent from one IP address to another hops from one AS to another based on a “route” that is determined by a protocol called BGP.

• Domain names (>1 billion): Domain names are the human-readable names that correspond to Internet services (like “cloudflare.com” or “mail.google.com”). These Internet services are accessed via the Internet by connecting to their IP address, which can be obtained from their domain name via the Domain Name System (DNS).

• Content (infinite): The main use case of the Internet is to enable the transfer of specific pieces of data from one point on the network to another. This data can be of any form or type.

When you type a website such as blog.cloudflare.com into your browser, a number of things happen. First, a (recursive) DNS service is contacted to get the IP address of the site. This DNS server configured by your ISP when you connect to the Internet, or it can be a public service such as 1.1.1.1 or 8.8.8.8. A query to the DNS service travels from network to network along a path determined by BGP announcements. If the recursive DNS server does not know the answer to the query, then it contacts the appropriate authoritative DNS services, starting with a root DNS server, down to a top level domain server (such as com or org), down to the DNS server that is authoritative for the domain. Once the DNS query has been answered, the browser sends an HTTP request to its IP address (traversing a sequence of networks), and in response, the server sends the content of the website.

So what’s the problem with this picture? For one, every DNS query and every network hop needs to be trusted in order to trust the content of the site. Any DNS query could be modified, a network could advertise an IP that belongs to another network, and any machine along the path could modify the content. When the Internet was small, there were mechanisms to combat this sort of subterfuge. Network operators had a personal relationship with each other and could punish bad behavior, but given the number of networks in existence almost 400,000 as of this week this is becoming difficult to scale.

Cryptography is a tool that can encode these trust relationships and make the whole system reliant on hard math rather than physical handshakes and hopes.

Attribution 2.0 Generic (CC BY 2.0)

The two main tools that cryptography provides to help solve this problem are cryptographic hashes and digital signatures.

A hash function is a way to take any piece of data and transform it into a fixed-length string of data, called a digest or hash. A hash function is considered cryptographically strong if it is computationally infeasible (read: very hard) to find two inputs that result in the same digest, and that changing even one bit of the input results in a completely different digest. The most popular hash function that is considered secure is SHA-256, which has 256-bit outputs. For example, the SHA-256 hash of the word “crypto” is

DA2F073E06F78938166F247273729DFE465BF7E46105C13CE7CC651047BF0CA4

And the SHA-256 hash of “crypt0” is

7BA359D3742595F38347A0409331FF3C8F3C91FF855CA277CB8F1A3A0C0829C4

The other main tool is digital signatures. A digital signature is a value that can only be computed by someone with a private key, but can be verified by anyone with the corresponding public key. Digital signatures are way for a private key holder to “sign,” or attest to the authenticity of a specific message in a way that anyone can validate it.

These two tools can be combined to solidify the trust relationships on the Internet. By giving private keys to the trusted parties who are responsible for defining the relationships between ASs, IPs, domain names and content, you can create chains of trust that can be publicly verified. Rather than hope and pray, these relationships can be validated in real time at scale.

Let’s take our webpage loading example and see where digital signatures can be applied.

Routing. Time-bound delegation of trust is defined through a system called the RPKI. RPKI defines an object called a Resource Certificate, an attestation that states that a given IP range belongs to a specific ASN for this period of time, digitally signed by the RIR responsible for assigning the IP range. Networks share routes via BGP, and if a route is advertised for an IP that does not conform the the Resource Certificate, the network can choose not to accept it.

DNS. Adding cryptographic assurance to routing is powerful, but if a network adversary can change the content of the data (such as the DNS responses), then the system is still at risk. DNSSEC is a system built to provide a trusted link between names and IP addresses. The root of trust in DNSSEC is the DNS root key, which is managed with an elaborate signing ceremony.

HTTPS. When you connect to a site, not only do you want it to be coming from the right host, you also want the content to be private. The Web PKI is a system that issues certificates to sites, allowing you to bind the domain name to a time-bounded private key. Because there are many CAs, additional accountability systems like certificate transparency need to be involved to help keep the system in check.

This cryptographic scaffolding turns the Internet into an encoded system of trust. With these systems in place, Internet users no longer need to trust every network and party involved in this diagram, they only need to trust the RIRs, DNSSEC and the CAs (and know the correct time).

This week we’ll be making some announcements that help strengthen this system of accountability.

The Internet is great because it connects us to each other, but the details of how it connects us are important. The technical choices made when Internet was designed come with some interesting human implications.

One implication is trackability. Your IP address is contained on every packet you send over the Internet. This acts as a unique identifier for anyone (corporations, governments, etc.) to track what you do online. Furthermore, if you connect to a server, that server’s identity is sent in plaintext on the request even over HTTPS, revealing your browsing patterns to any intermediary who cares to look.

Another implication is malleability. Resources on the Internet are defined by where they are, not what they are. If you want to go to CNN or BBC, then you connect to the server for cnn.com or bbc.co.uk and validate the certificate to make sure it’s the right site. But once you’ve made the connection, there’s no good way to know that the actual content is what you expect it to be. If the server is hacked, it could send you anything, including dangerous malicious code. HTTPS is a secure pipe, but there’s no inherent way to make sure what gets sent through the pipe is what you expect.

Trackability and malleability are not inherent features of interconnectedness. It is possible to design networks that don’t have these downsides. It is also possible to build new networks with better characteristic on top of the existing Internet. The key ingredient is cryptography.

One of the networks built on top of the Internet that provides good privacy properties is Tor. The Tor network is run by a group of users who allow their computers to be used to route traffic for other members of the network. Using cryptography, it is possible to route traffic from one place to another without points along the path knowing both the source and the destination at the same time. This is called onion routing because it involves multiple layers of encryption, like an onion. Traffic coming out of the Tor network is “anonymous” because it could have come from anyone connected to the network. Everyone just blends in, making it hard to track individuals.

Similarly, web services can use onion routing to serve content inside the Tor network without revealing their location to visitors. Instead of using a hostname to identify their network location, so-called onion services use a cryptographic public key as their address. There are hundreds of onion services in use, including the one we use for 1.1.1.1 or the one in use by Facebook.

Troubles occur at the boundary between Tor network and the rest of the Internet. This is especially true for user is attempting to access services that rely on abuse prevention mechanisms based on reputation. Since Tor is used by both privacy-conscious users and malicious bots, connections from both get lumped together and as the expression goes, one bad apple ruins the bunch. This unfortunately exposes legitimate visitors to anti-abuse mechanisms like CAPTCHAs. Tools like Privacy Pass help reduce this burden but don’t eliminate it completely. This week we’ll be announcing a new way to improve this situation.

Let’s revisit the issue of malleability: the fact that you can’t always trust the other side of a connection to send you the content you expect. There are technologies that allow users to insure the integrity of content without trusting the server. One such technology is a feature of HTML called Subresource Integrity (SRI). SRI allows a webpage with sub-resources (like a script or stylesheet) to embed a unique cryptographic hash into the page so that when the sub-resource is loaded, it is checked to see that is matches the expected value. This protects the site from loading unexpected scripts from third parties, a known attack vector.

Another idea is to flip this on its head: what if instead of fetching a piece of content from a specific location on the network, you asked the network to find piece of content that matches a given hash? By assigning resources based on their actual content rather than by location it’s possible to create a network in which you can fetch content from anywhere on the network and still know it’s authentic. This idea is called content addressing and there are networks built on top of the Internet that use it. These content addressed networks, based on protocols such IPFS and DAT, are blazing a trail new trend in Internet applications called the Distributed Web. With the Distributed Web applications, malleability is no longer an issue, opening up a new set of possibilities.

Networks based on cryptographic principles, like Tor and IPFS, have one major downside compared to networks based on names: usability. Humans are exceptionally bad at remembering or distinguishing between cryptographically-relevant numbers. Take, for instance, the New York Times’ onion address:

https://www.nytimes3xbfgragh.onion/

This is would easily confused with similar-looking onion addresses, such as

https://www.nytimes3xfkdbgfg.onion/

which may be controlled by a malicious actor.

Content addressed networks are even worse from the perspective of regular people. For example, there is a snapshot of the Turkish version of Wikipedia on IPFS with the hash:

QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX

Try typing this hash into your browser without making a mistake.

These naming issues are things Cloudflare is perfectly positioned to help solve.First, by putting the hash address of an IPFS site in the DNS (and adding DNSSEC for trust) you can give your site a traditional hostname while maintaining a chain of trust.

Second, by enabling browsers to use a traditional DNS name to access the web through onion services, you can provide safer access to your site for Tor user with the added benefit of being better able to distinguish between bots and humans.With Cloudflare as the glue, is is possible to connect both standard internet and tor users to web sites and services on both the traditional web with the distributed web.

This is the promise of Crypto Week: using cryptographic guarantees to make a stronger, more trustworthy and more private internet without sacrificing usability.

In conclusion, we’re working on many cutting-edge technologies based on cryptography and applying them to make the Internet better. The first announcement today is the launch of Cloudflare's Distributed Web Gateway and browser extension. Keep tabs on the Cloudflare blog for exciting updates as the week progresses.

I’m very proud of the team’s work on Crypto Week, which was made possible by the work of a dedicated team, including several brilliant interns. If this type of work is interesting to you, Cloudflare is hiring for the crypto team and others!

In case you haven’t heard yet, Cloudflare launched a privacy-first DNS resolver service on April 1st. It was no joke! The service, which was our first consumer-focused service, supports emerging DNS standards such as DNS over HTTPS:443 and TLS:853 in addition to traditional protocols over UDP:53 and TCP:53, all in one easy to remember address: 1.1.1.1.

As it was mentioned in the original blog post, our policy is to never, ever write client IP addresses to disk and wipe all logs within 24 hours. Still, the exceptionally privacy-conscious folks might not want to reveal their IP address to the resolver at all, and we respect that. This is why we are launching a Tor onion service for our resolver at dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion and accessible via tor.cloudflare-dns.com.

NOTE: the hidden resolver is still an experimental service and should not be used in production or for other critical uses until it is more tested.

Imagine an alternative Internet where, in order to connect to www.cloudflare.com, instead of delegating the task of finding a path to our servers to your internet provider, you had to go through the following steps to reach Cloudflare:

1. You calculate a path to your destination, like this:

    You -> Your ISP -> X -> Y -> Z -> www.cloudflare.com.

2. You encrypt your packet with Z’s public key, then with Y’s, and finally with X’s.

3. You submit the result to X, who decrypts with their private key;

4. X submits the result to Y, who decrypts with their private key;

5. Y submits the result to Z, who decrypts with their private key to get the original packet;

6. Z submits the packet to www.cloudflare.com.

If everyone plays their roles correctly, it is possible to ensure only the entry relay X knows your IP address and only the exit relay Z knows the website you’re connecting you, thereby providing you with privacy and anonymity. This is a simplified version of Tor: a collection of volunteer-run computers and servers around the world acting as relays for a huge network built on top of the Internet where every hop from one relay to the next peels one layer of encryption, hence its name: the onion router.

Keeping internet users anonymous is not the only function of the Tor network. In particular, one caveat of the procedure above is that the connection is still accessible by the exit relay and anyone sitting between there and the destination, including network providers. To solve this problem, and to also provide anonymity for content publishers, Tor allows for onion services. Onion services are Tor nodes that advertise their public key, encoded as an address with .onion TLD, and establish connections entirely within the Tor network:

The process of returning an IP address given a domain name is called DNS resolution. Since Tor still uses IP addresses, you still need to do DNS resolution to browse the web over Tor. There are two common methods to resolve a domain name when using Tor:

1. Resolve the name directly, then talk to the IP address through Tor;

2. Ask a Tor exit relay to resolve the name publicly and connect to the IP.

Clearly, the first option leaks your IP to your DNS resolver and, unless your client uses DNS-over-HTTPS or DNS-over-TLS, it leaks your destination name to your ISP. What is less obvious is that the second option can open you to manipulation attacks such as DNS poisoning or sslstrip by bad relays. This is where our new service comes in:

1. Ask a .onion-based resolver service!

In a few words, our .onion-based resolver service is a Tor onion service which forwards all communication on DNS ports to the corresponding ports on 1.1.1.1, hence the apparent client IP is an internal IP rather than yours. There is, however, more than meets the eye.

One glaring difference between using 1.1.1.1 and this service is that the .onion address is "dns4tor" plus 49 seemingly random alphanumeric characters. This 56 character long string, in fact, contains a full Ed25519 public key which is used to secure communication with the onion service. This poses a number of challenges towards usable security:

1. How can the users make sure that that the address is correct?

We simply bought a certificate with tor.cloudflare-dns.com as subject name and the .onion address as a subject alternative name. This way, if you’re in the right place, you should see this:

1. How can the users remember this address?

We don’t think you should need to remember this address. Ideally, all you would need to do is go to https://tor.cloudflare-dns.com and have the browser route your request to the .onion address. This is possible using the "Alt-Svc" HTTP header which is an optional header notifying the browser that the resources can be accessed from an alternative network location, possibly using a different protocol. Thanks to Mozilla, using .onion addresses as alternative services is now possible in Firefox Nightly.

Think of this feature like opportunistic encryption: once your browser receives an Alt-Svc header indicating that a .onion address is available for tor.cloudflare-dns.com, if it knows that .onion addresses can be accessed (for instance through a SOCKS proxy), it attempts to check that the alternative service has the same or a higher level of security. This includes making sure that it is possible to connect to the onion service using the same certificate and Server Name. If that is the case, the browser uses the alternative service instead, therefore ensuring that your future requests do not leave the Tor network.

Here is a thought experiment: suppose between each two points on Earth there is a fiber-optic cable, capable of lossless transmission of packets at the speed of light.

Using a back-of-the-envelope calculation it’s easy to see that, on average, each packet traverses a distance equivalent to a quarter of the circumference of the Earth in about 33ms, while each Tor packet takes about 200ms to go one and a half turns around the Earth before reaching an onion service; that’s three turns for a round trip that ensures anonymity of both parties.

Cloudflare, however, does not require anonymity for its servers, which is why we can reduce the number of relays to just three by enabling an optional setting for onion services that prioritize lower latency over location anonymity of the service. To emphasize, this does not impact client privacy or anonymity whatsoever. Indeed, as you may have noticed, in the first onion service image the origin is three hops away from the rendezvous point whereas our onion service is only one hop away.

We are actively working on developing ways to make this service faster and ensure it has as little downtime as possible.

First and foremost, resolving DNS queries through the Tor network, for instance by connecting to Google’s 8.8.8.8 resolver, guarantees a significantly higher level of anonymity than making the requests directly. Not only does doing so prevent the resolver from ever seeing your IP address, even your ISP won’t know that you’ve attempted to resolve a domain name.

Still, unless the destination is an onion service, passive attackers can capture packets exiting the Tor network and malicious Exit Nodes can poison DNS queries or downgrade encryption through sslstripping. Even if you limit your browsing to only HTTPS sites, passive attackers can find out which addresses you’ve connected to. Even worse, actors capable of comparing traffic both before it enters the Tor network and after it leaves the network can potentially use the metadata (size, time, etc.) to deanonymize the client. The only solution, then, is to eliminate the need for Exit Nodes by using onion services instead. That is what our .onion-based resolver offers.

Moreover, if your client does not support encrypted DNS queries, using a .onion-based resolver can secure the connection from on-path attacks, including BGP hijacking attacks. This means having the same level of security for DNS-over-UDP and DNS-over-TCP as DNS-over-HTTPS and DNS-over-TLS provides.

Your personal anonymity, however, is not the only reason why you should use this service. The power of Tor in ensuring everyone’s anonymity rests on the number of people who use it. If only whistleblowers, for instance, were to use the Tor network, then anyone connecting to the Tor network would automatically be suspected of being a whistleblower. Therefore the more people use Tor to browse memes or to watch cat videos on the Internet, the easier it will be for those who truly need anonymity to blend in with the traffic.

One barrier to using Tor for many users is that it is simply slow, so I can try to sympathize with those who wouldn’t sacrifice quick website load times to help keep activists and dissidents anonymous. That said, DNS requests are small in size and since most browsers and operating systems cache DNS results the total traffic is not significant. As a result, using the .onion-based resolver will only slightly slow down your initial DNS request without slowing down anything else, while still contributing to the overall anonymity of the Tor network and its users.

Using a .onion-based resolver ensures that your ISP never finds out that you’re resolving a domain, the Exit Nodes don’t get a chance to manipulate DNS replies, and the resolver never finds out your IP address. However, the unique benefit of using the Cloudflare .onion-based resolver is combining the power of Tor with all privacy-preserving features of the 1.1.1.1 resolver, such as query name minimization, as well as a team of engineers working on improving it at every level, including standards like DNS-over-HTTPS and DNS-over-TLS.

As CEO Matthew Prince said about two years ago, anonymity online is a cause we value at Cloudflare. In addition, when we announced the 1.1.1.1 resolver we committed to taking every technical step to ensure we can’t know what you do on the internet. Providing a way to use the resolver through the Tor network and making it as fast as possible is a big step in that direction.

The .onion-based resolver supports every DNS protocol that 1.1.1.1 supports, only over the Tor network. However, since not every DNS client is capable of connecting to the Tor network, some hacking is required to get it to work. Here we will explain how to set up DNS-over-HTTPS provided from the .onion-based resolver, but for all other scenarios head to our developers page to get the details of how to use the .onion-based resolver.

Here is how you can set up cloudflared to start a DNS client that uses DNS over HTTPS, routed through the Tor network:

1. First, start with downloading cloudflared by following the regular guide for Running a DNS over HTTPS Client.

2. Start a Tor SOCKS proxy and use socat to forward port TCP:443 to localhost:

    socat TCP4-LISTEN:443,reuseaddr,fork SOCKS4A:127.0.0.1:dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion:443,socksport=9150

3. Instruct your machine to treat the .onion address as localhost:

    cat << EOF >> /etc/hosts
    127.0.0.1 dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion
    EOF

4. Finally, start a local DNS over UDP daemon:

    cloudflared proxy-dns --upstream "https://dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion/dns-query"
    INFO[0000] Adding DNS upstream                           url="https://dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion/dns-query"
    INFO[0000] Starting DNS over HTTPS proxy server          addr="dns://localhost:53"
    INFO[0000] Starting metrics server                       addr="127.0.0.1:35659"

5. Profit!

CC BY 2.0 image by Ben Salter

Think of it like a black box: traffic goes into the box, is bounced around between a random set of relays, and ultimately comes out to connect to the requested site. Anonymity is assured because anyone monitoring the network would have a difficult time tying the individuals making the requests going into the black box with the requests coming out.

Anonymity online is important for a number of reasons we at CloudFlare believe in. For instance, Tor is instrumental in ensuring that individuals living in repressive regimes can access information that may otherwise be blocked or illegal. We believe this is so important that we offer our service for free through Project Galileo to protect politically and artistically important organizations and journalists against attacks that would otherwise censor their work.

On the other hand, anonymity is also something that provides value to online attackers. Based on data across the CloudFlare network, 94% of requests that we see across the Tor network are per se malicious. That doesn’t mean they are visiting controversial content, but instead that they are automated requests designed to harm our customers. A large percentage of the comment spam, vulnerability scanning, ad click fraud, content scraping, and login scanning comes via the Tor network. To give you some sense, based on data from Project Honey Pot, 18% of global email spam, or approximately 6.5 trillion unwanted messages per year, begin with an automated bot harvesting email addresses via the Tor network.

CC BY 3.0 logo from the Tor Project

At CloudFlare we've not explicitly treated traffic from Tor any differently, however users of the Tor browser have been more likely to have their browsing experience interrupted by CAPTCHAs or other restrictions. This is because, like all IP addresses that connect to our network, we check the requests that they make and assign a threat score to the IP. Unfortunately, since such a high percentage of requests that are coming from the Tor network are malicious, the IPs of the Tor exit nodes often have a very high threat score.

With most browsers, we can use the reputation of the browser from other requests it’s made across our network to override the bad reputation of the IP address connecting to our network. For instance, if you visit a coffee shop that is only used by hackers, the IP of the coffee shop's WiFi may have a bad reputation. But, if we've seen your browser behave elsewhere on the Internet acting like a regular web surfer and not a hacker, then we can use your browser’s good reputation to override the bad reputation of the hacker coffee shop's IP.

The design of the Tor browser intentionally makes building a reputation for an individual browser very difficult. And that's a good thing. The promise of Tor is anonymity. Tracking a browser's behavior across requests would sacrifice that anonymity. So, while we could probably do things using super cookies or other techniques to try to get around Tor's anonymity protections, we think that would be creepy and choose not to because we believe that anonymity online is important. Unfortunately, that then means all we can rely on when a request connects to our network is the reputation of the IP and the contents of the request itself.

The situation reminds me of those signs you see in diners: "fast, good, cheap (pick any two)." In our case, the three competing interests are security, anonymity, and convenience. Unfortunately, you can’t provide all three so the question is: what do you sacrifice?

Our customers sign up for CloudFlare to protect them from online attacks, so we can't sacrifice security. We also believe anonymity is critical, having witnessed first hand how repressive regimes use control of the network to restrict access to content. So that leaves sacrificing a bit of convenience for users of the Tor browser.

Fundamentally, the challenge we have is telling automated malicious traffic sent via Tor from legitimate human users. To do that, when a visitor is coming from a Tor exit node with a poor reputation, we will often use some sort of CAPTCHA. CAPTCHA stands for "Completely Automated Public Turing test to tell Computers and Humans Apart" — which is exactly what we're trying to do.

CAPTCHAs have become known for squiggly letters that you type into a box, but they could be anything that is relatively easy for real humans but hard for automated systems. Building a CAPTCHA is an extremely difficult computer science problem because there's an increasingly small number of things humans are better at than machines. We currently use Google's reCAPTCHA and recently switched from the older ‘squiggly letters’ version to the latest revision to make life easier for humans and harder for bots. We're not particularly happy with reCAPTCHA, but we haven't found an alternative that is better and can operate at our scale.

To make sure our team understood what a pain CAPTCHAs could be, I blocklisted all the IP addresses used in Cloudflare's office so our employees would need to pass a CAPTCHA every time they wanted to visit any of our customers' sites. This dogfooding experience caused us to make some changes to make the CAPTCHAs more convenient, including convincing Google to allow us to use their new version of reCAPTCHA. It's better, but it's still less convenient than using a non-Tor browser.

We also made a change based on the experience of having to pass CAPTCHAs ourselves that treated all Tor exit IPs as part of a cluster, so if you passed a CAPTCHA for one you wouldn’t have to pass one again if your circuit changed. Over the last twelve months we’ve made incremental progress toward our goal of finding some way to provide a CAPTCHA that distinguishes automated and human traffic without being too inconvenient for the humans — but we’re not there yet.

Since the list of Tor exit nodes is public we’ve open sourced our tools used to examine the reputation and life of exit nodes. torhoney matches Tor exit nodes and information from Project Honey Pot to see which nodes are most misused and torexit can be used to determine the longevity of Tor exit nodes. We used it to tune our download of the Tor exit node list to ensure that we were up to date.

Using torhoney we produced the following chart showing the percentage of Tor exit nodes that were listed by Project Honey Pot as a comment spammer over the last year.

As you can see the percentage is progressing steadily upwards.

Using torexit we produced the following diagram. Each column represents an individual Tor exit node and each row is a 15 minute interval. White means that the exit node was not in the exit node list during that 15 minute interval.

As you can see most exit nodes are stable and work continuously. On the right are nodes that disappeared at some point and the slanting block of nodes on the left appear for 24 hours and then disappear (perhaps because of DHCP leases).

There are a number of problems with our current implementation and we are not satisfied with it. One is that Google themselves see significant malicious traffic from Tor so the reCAPTCHA challenges they present to Tor users are among the hardest. We're talking with Google about how we can overcome that. Another problem is that you have to pass a CAPTCHA for each site you visit. Unfortunately, to solve that, we'd need to track Tor users across sites which would sacrifice Tor’s anonymity so we’ve deemed it unacceptable.

So what are potential solutions? One suggestion has been that we treat GET requests for static content differently than we do more risky requests like POSTs. We actually already do treat more dangerous requests differently than less risky requests. The problem is Tor exit nodes often have very bad reputations due to all the malicious requests they send, and you can do a lot of harm just with GETs. Content scraping, ad click fraud, and vulnerability scanning are all threats our customers ask us to protect them from and all only take GET requests.

Another suggestion is that we allow our customers to allow Tor exit nodes. We resisted this for quite some time, but perhaps not for the reason you'd expect. If we provide a way to treat Tor differently by applying a rule to allow the network's IPs we couldn't think of a justifiable reason to not also provide a way to blocklist the network as well. And, while Tor users think it's a no-brainer that sites would allow their traffic, if you talk actually with site owners the majority would prefer to just block Tor traffic entirely. In fact, when we looked at our customer base, we found that far more had manually entered Tor exit node IPs to block them than to allow them. We didn’t want to make blocklisting easier because, again, we believe there’s value in the anonymous web surfing that Tor offers.

We relented a few weeks ago and allowed our customers to specify rules that apply to traffic from the Tor network, but we came up with a compromise to prevent the damage from full blocklisting. We now allow our customers to treat Tor the same way as we do traffic from a country (country code “T1” to be specific). Just like with countries, traffic can be allowed by anyone, but we don't allow our self-service customers to fully block traffic. Customers can force traffic to see a CAPTCHA, but they can't block traffic entirely. However, the choice of how to handle Tor is now in the hands of individual site owners.

The long term solution has to be something that allows automated, malicious traffic to be distinguished from non-automated traffic coming through the Tor network. We see two viable ways of doing that, but we need help from the Tor Project to implement either of them.

The first would be to make it easy, or maybe even automatic, for CloudFlare customers to create a .onion version of their sites. These .onion sites are only accessible via the Tor network and therefore less likely to be targeted by automated attacks. This was Facebook’s solution when faced with the same problem. We think it’s elegant.

The problem is generating SSL certificates to encrypt traffic to the .onion sites. Tor uses hashes generated with the weak SHA-1 algorithm to generate .onion addresses and then only uses 80 bits of the 160 bits from the hash to generate the address, making them even weaker. This creates a significant security risk if you automatically generate SSL certificates. An attacker could generate a site that collides with the 80-bit .onion address and get a certificate that could be used to intercept encrypted traffic.

Because of this risk, the CA/B Forum, which regulates the issuance of SSL certificates, requires certificates for .onion addresses to be EV certificates. EV certificates require extended validation procedures which limit the risk they can be issued to a malicious party. Unfortunately, those same procedures prevent EV certificates from being issued automatically and make them prohibitively expensive for us to automatically create for all our customers.

The solution is for the Tor Project to support a stronger hashing algorithm, such as SHA256, for .onion addresses. With that in place, we believe the CA/B Forum would be open to discussing the automatic issuance of certificates. That would allow us to create .onion sites for our customers, allow Tor traffic to the .onion sites, and continue to protect our customers from automated attacks sent via the Tor network targeting their non-.onion sites.

Some members of CloudFlare’s team have proposed a solution to the Tor Project that moves part of the process of distinguishing between automated and human traffic to the Tor browser itself. The Tor browser could allow users to do a sort of proof-of-work problem and then send a cryptographically secure but anonymous token to services like CloudFlare in order to verify that the request is not coming from an automated system.

By moving the proof-of-work test to the client side, the Tor browser could send confirmation to every site visited so that users wouldn’t be asked to prove they are human repeatedly. Providing a way to distinguish a human using Tor from an automated bot would not only benefit traffic to CloudFlare but, if it became a standard, could also make it easier for other organizations that have restricted Tor traffic out of concern for abuse to start allowing verified-human Tor users back on their sites.

This blinded token solution is outlined in this draft.

CloudFlare is working to reduce the impact of CAPTCHAs on Tor users without in any way compromising their anonymity and without exposing our customers to additional risk. Over the coming weeks and months we will roll out changes designed to make the lives of legitimate Tor Browser users easier while keeping our customers safe.

Our mission is to help build a better Internet and that means protecting web sites from harm and ensuring that web users who wish to remain anonymous are able to do so.

We believe that the Internet will be better off if we do so as sites will not find themselves wanting to ban Tor users completely just because of abuse.

As we've done for DDoS attacks we are working to put in place technology that will filter out the bad coming from Tor while allowing the good through and we are happy to work with the Tor Project to make that a reality.