This issue was occuring now because the ARPANET was on the verge of switching from its original NCP protocol, to the TCP/IP protocol which powers what we now call the Internet. With that switch suddenly there would be a multitude of interconnected networks (an ‘Inter... net’) requiring a more ‘hierarchical’ domain system where ARPANET could resolve its own domains while the other networks resolved theirs.

Other networks at the time had great names like “COMSAT”, “CHAOSNET”, “UCLNET” and “INTELPOSTNET” and were maintained by groups of universities and companies all around the US who wanted to be able to communicate, and could afford to lease 56k lines from the phone company and buy the requisite PDP-11s to handle routing.

In the original ARPANET design, a central Network Information Center (NIC) was responsible for maintaining a file listing every host on the network. The file was known as the HOSTS.TXT file, similar to the /etc/hosts file on a Linux or OS X system today. Every network change would require the NIC to FTP (a protocol invented in 1971) to every host on the network, a significant load on their infrastructure.

Having a single file list every host on the Internet would, of course, not scale indefinitely. The priority was email, however, as it was the predominant addressing challenge of the day. Their ultimate conclusion was to create a hierarchical system in which you could query an external system for just the domain or set of domains you needed. In their words: “The conclusion in this area was that the current ‘user@host’ mailbox identifier should be extended to ‘user@host.domain’ where ‘domain’ could be a hierarchy of domains.” And the domain was born.

It’s important to dispel any illusion that these decisions were made with prescience for the future the domain name would have. In fact, their elected solution was primarily decided because it was the “one causing least difficulty for existing systems.” For example, one proposal was for email addresses to be of the form <user>.<host>@<domain>. If email usernames of the day hadn’t already had ‘.’ characters you might be emailing me at ‘zack.cloudflare@com’ today.

Cloudflare allows you to move caching, load balancing, rate limiting, and even network firewall and code execution out of your infrastructure to our points of presence within milliseconds of virtually every Internet user.Read A Case Study Contact Sales

It has been said that the principal function of an operating system is to define a number of different names for the same object, so that it can busy itself keeping track of the relationship between all of the different names. Network protocols seem to have somewhat the same characteristic.

— David D. Clark, 1982

Another failed proposal involved separating domain components with the exclamation mark (!). For example, to connect to the ISIA host on ARPANET, you would connect to !ARPA!ISIA. You could then query for hosts using wildcards, so !ARPA!* would return to you every ARPANET host.

This method of addressing wasn’t a crazy divergence from the standard, it was an attempt to maintain it. The system of exclamation separated hosts dates to a data transfer tool called UUCP created in 1976. If you’re reading this on an OS X or Linux computer, uucp is likely still installed and available at the terminal.

ARPANET was introduced in 1969, and quickly became a powerful communication tool... among the handful of universities and government institutions which had access to it. The Internet as we know it wouldn’t become publically available outside of research institutions until 1991, twenty one years later. But that didn’t mean computer users weren’t communicating.

In the era before the Internet, the general method of communication between computers was with a direct point-to-point dial up connection. For example, if you wanted to send me a file, you would have your modem call my modem, and we would transfer the file. To craft this into a network of sorts, UUCP was born.

In this system, each computer has a file which lists the hosts its aware of, their phone number, and a username and password on that host. You then craft a ‘path’, from your current machine to your destination, through hosts which each know how to connect to the next:

sw-hosts!digital-lobby!zack

This address would form not just a method of sending me files or connecting with my computer directly, but also would be my email address. In this era before ‘mail servers’, if my computer was off you weren’t sending me an email.

While use of ARPANET was restricted to top-tier universities, UUCP created a bootleg Internet for the rest of us. It formed the basis for both Usenet and the BBS system.

Ultimately, the DNS system we still use today would be proposed in 1983. If you run a DNS query today, for example using the dig tool, you’ll likely see a response which looks like this:

;; ANSWER SECTION:
google.com.   299 IN  A 172.217.4.206

This is informing us that google.com is reachable at 172.217.4.206. As you might know, the A is informing us that this is an ‘address’ record, mapping a domain to an IPv4 address. The 299 is the ‘time to live’, letting us know how many more seconds this value will be valid for, before it should be queried again. But what does the IN mean?

IN stands for ‘Internet’. Like so much of this, the field dates back to an era when there were several competing computer networks which needed to interoperate. Other potential values were CH for the CHAOSNET or HS for Hesiod which was the name service of the Athena system. CHAOSNET is long dead, but a much evolved version of Athena is still used by students at MIT to this day. You can find the list of DNS classes on the IANA website, but it’s no surprise only one potential value is in common use today.

It is extremely unlikely that any other TLDs will be created.

— Jon Postel, 1994

Once it was decided that domain names should be arranged hierarchically, it became necessary to decide what sits at the root of that hierarchy. That root is traditionally signified with a single ‘.’. In fact, ending all of your domain names with a ‘.’ is semantically correct, and will absolutely work in your web browser: google.com.

The first TLD was .arpa. It allowed users to address their old traditional ARPANET hostnames during the transition. For example, if my machine was previously registered as hfnet, my new address would be hfnet.arpa. That was only temporary, during the transition, server administrators had a very important choice to make: which of the five TLDs would they assume? “.com”, “.gov”, “.org”, “.edu” or “.mil”.

When we say DNS is hierarchical, what we mean is there is a set of root DNS servers which are responsible for, for example, turning .com into the .com nameservers, who will in turn answer how to get to google.com. The root DNS zone of the internet is composed of thirteen DNS server clusters. There are only 13 server clusters, because that’s all we can fit in a single UDP packet. Historically, DNS has operated through UDP packets, meaning the response to a request can never be more than 512 bytes.

;       This file holds the information on root name servers needed to
;       initialize cache of Internet domain name servers
;       (e.g. reference this file in the "cache  .  "
;       configuration file of BIND domain name servers).
;
;       This file is made available by InterNIC 
;       under anonymous FTP as
;           file                /domain/named.cache
;           on server           FTP.INTERNIC.NET
;       -OR-                    RS.INTERNIC.NET
;
;       last update:    March 23, 2016
;       related version of root zone:   2016032301
;
; formerly NS.INTERNIC.NET
;
.                        3600000      NS    A.ROOT-SERVERS.NET.
A.ROOT-SERVERS.NET.      3600000      A     198.41.0.4
A.ROOT-SERVERS.NET.      3600000      AAAA  2001:503:ba3e::2:30
;
; FORMERLY NS1.ISI.EDU
;
.                        3600000      NS    B.ROOT-SERVERS.NET.
B.ROOT-SERVERS.NET.      3600000      A     192.228.79.201
B.ROOT-SERVERS.NET.      3600000      AAAA  2001:500:84::b
;
; FORMERLY C.PSI.NET
;
.                        3600000      NS    C.ROOT-SERVERS.NET.
C.ROOT-SERVERS.NET.      3600000      A     192.33.4.12
C.ROOT-SERVERS.NET.      3600000      AAAA  2001:500:2::c
;
; FORMERLY TERP.UMD.EDU
;
.                        3600000      NS    D.ROOT-SERVERS.NET.
D.ROOT-SERVERS.NET.      3600000      A     199.7.91.13
D.ROOT-SERVERS.NET.      3600000      AAAA  2001:500:2d::d
;
; FORMERLY NS.NASA.GOV
;
.                        3600000      NS    E.ROOT-SERVERS.NET.
E.ROOT-SERVERS.NET.      3600000      A     192.203.230.10
;
; FORMERLY NS.ISC.ORG
;
.                        3600000      NS    F.ROOT-SERVERS.NET.
F.ROOT-SERVERS.NET.      3600000      A     192.5.5.241
F.ROOT-SERVERS.NET.      3600000      AAAA  2001:500:2f::f
;
; FORMERLY NS.NIC.DDN.MIL
;
.                        3600000      NS    G.ROOT-SERVERS.NET.
G.ROOT-SERVERS.NET.      3600000      A     192.112.36.4
;
; FORMERLY AOS.ARL.ARMY.MIL
;
.                        3600000      NS    H.ROOT-SERVERS.NET.
H.ROOT-SERVERS.NET.      3600000      A     198.97.190.53
H.ROOT-SERVERS.NET.      3600000      AAAA  2001:500:1::53
;
; FORMERLY NIC.NORDU.NET
;
.                        3600000      NS    I.ROOT-SERVERS.NET.
I.ROOT-SERVERS.NET.      3600000      A     192.36.148.17
I.ROOT-SERVERS.NET.      3600000      AAAA  2001:7fe::53
;
; OPERATED BY VERISIGN, INC.
;
.                        3600000      NS    J.ROOT-SERVERS.NET.
J.ROOT-SERVERS.NET.      3600000      A     192.58.128.30
J.ROOT-SERVERS.NET.      3600000      AAAA  2001:503:c27::2:30
;
; OPERATED BY RIPE NCC
;
.                        3600000      NS    K.ROOT-SERVERS.NET.
K.ROOT-SERVERS.NET.      3600000      A     193.0.14.129
K.ROOT-SERVERS.NET.      3600000      AAAA  2001:7fd::1
;
; OPERATED BY ICANN
;
.                        3600000      NS    L.ROOT-SERVERS.NET.
L.ROOT-SERVERS.NET.      3600000      A     199.7.83.42
L.ROOT-SERVERS.NET.      3600000      AAAA  2001:500:9f::42
;
; OPERATED BY WIDE
;
.                        3600000      NS    M.ROOT-SERVERS.NET.
M.ROOT-SERVERS.NET.      3600000      A     202.12.27.33
M.ROOT-SERVERS.NET.      3600000      AAAA  2001:dc3::35
; End of file

Root DNS servers operate in safes, inside locked cages. A clock sits on the safe to ensure the camera feed hasn’t been looped. Particularly given how slow DNSSEC implementation has been, an attack on one of those servers could allow an attacker to redirect all of the Internet traffic for a portion of Internet users. This, of course, makes for the most fantastic heist movie to have never been made.

Unsurprisingly, the nameservers for top-level TLDs don’t actually change all that often. 98% of the requests root DNS servers receive are in error, most often because of broken and toy clients which don’t properly cache their results. This became such a problem that several root DNS operators had to spin up special servers just to return ‘go away’ to all the people asking for reverse DNS lookups on their local IP addresses.

The TLD nameservers are administered by different companies and governments all around the world (Verisign manages .com). When you purchase a .com domain, about 0.18goestotheICANN,and7.85 goes to Verisign.

It is rare in this world that the silly name us developers think up for a new project makes it into the final, public, product. We might name the company database Delaware (because that’s where all the companies are registered), but you can be sure by the time it hits production it will be CompanyMetadataDatastore. But rarely, when all the stars align and the boss is on vacation, one slips through the cracks.

Punycode is the system we use to encode unicode into domain names. The problem it is solving is simple, how do you write 比薩.com when the entire internet system was built around using the ASCII alphabet whose most foreign character is the tilde?

It’s not a simple matter of switching domains to use unicode. The original documents which govern domains specify they are to be encoded in ASCII. Every piece of internet hardware from the last forty years, including the Cisco and Juniper routers used to deliver this page to you make that assumption.

The web itself was never ASCII-only. It was actually originally concieved to speak ISO 8859-1 which includes all of the ASCII characters, but adds an additional set of special characters like ¼ and letters with special marks like ä. It does not, however, contain any non-Latin characters.

This restriction on HTML was ultimately removed in 2007 and that same year Unicode became the most popular character set on the web. But domains were still confined to ASCII.

As you might guess, Punycode was not the first proposal to solve this problem. You most likely have heard of UTF-8, which is a popular way of encoding Unicode into bytes (the 8 is for the eight bits in a byte). In the year 2000 several members of the Internet Engineering Task Force came up with UTF-5. The idea was to encode Unicode into five bit chunks. You could then map each five bits into a character allowed (A-V & 0-9) in domain names. So if I had a website for Japanese language learning, my site 日本語.com would become the cryptic M5E5M72COA9E.com.

This encoding method has several disadvantages. For one, A-V and 0-9 are used in the output encoding, meaning if you wanted to actually include one of those characters in your doman, it had to be encoded like everything else. This made for some very long domains, which is a serious problem when each segment of a domain is restricted to 63 characters. A domain in the Myanmar language would be restricted to no more than 15 characters. The proposal does make the very interesting suggestion of using UTF-5 to allow Unicode to be transmitted by Morse code and telegram though.

There was also the question of how to let clients know that this domain was encoded so they could display them in the appropriate Unicode characters, rather than showing M5E5M72COA9E.com in my address bar. There were several suggestions, one of which was to use an unused bit in the DNS response. It was the “last unused bit in the header”, and the DNS folks were “very hesitant to give it up” however.

Another suggestion was to start every domain using this encoding method with ra--. At the time (mid-April 2000), there were no domains which happened to start with those particular characters. If I know anything about the Internet, someone registered an ra-- domain out of spite immediately after the proposal was published.

The ultimate conclusion, reached in 2003, was to adopt a format called Punycode which included a form of delta compression which could dramatically shorten encoded domain names. Delta compression is a particularily good idea because the odds are all of the characters in your domain are in the same general area within Unicode. For example, two characters in Farsi are going to be much closer together than a Farsi character and another in Hindi. To give an example of how this works, if we take the nonsense phrase:

يذؽ

In an uncompressed format, that would be stored as the three characters [1620, 1584, 1597] (based on their Unicode code points). To compress this we first sort it numerically (keeping track of where the original characters were): [1584, 1597, 1620]. Then we can store the lowest value (1584), and the delta between that value and the next character (13), and again for the following character (23), which is significantly less to transmit and store.

Punycode then (very) efficiently encodes those integers into characters allowed in domain names, and inserts an xn-- at the beginning to let consumers know this is an encoded domain. You’ll notice that all the Unicode characters end up together at the end of the domain. They don’t just encode their value, they also encode where they should be inserted into the ASCII portion of the domain. To provide an example, the website 熱狗sales.com becomes xn--sales-r65lm0e.com. Anytime you type a Unicode-based domain name into your browser’s address bar, it is encoded in this way.

This transformation could be transparent, but that introduces a major security problem. All sorts of Unicode characters print identically to existing ASCII characters. For example, you likely can’t see the difference between Cyrillic small letter a (“а”) and Latin small letter a (“a”). If I register Cyrillic аmazon.com (xn--mazon-3ve.com), and manage to trick you into visiting it, it’s gonna be hard to know you’re on the wrong site. For that reason, when you visit ??.ws, your browser somewhat lamely shows you xn--vi8hiv.ws in the address bar.

The first portion of the URL is the protocol which should be used to access it. The most common protocol is http, which is the simple document transfer protocol Tim Berners-Lee invented specifically to power the web. It was not the only option. Some people believed we should just use Gopher. Rather than being general-purpose, Gopher is specifically designed to send structured data similar to how a file tree is structured.

For example, if you request the /Cars endpoint, it might return:

1Chevy Camaro             /Archives/cars/cc     gopher.cars.com     70
iThe Camaro is a classic  fake                  (NULL)              0
iAmerican Muscle car      fake                  (NULL)              0
1Ferrari 451              /Factbook/ferrari/451  gopher.ferrari.net 70

which identifies two cars, along with some metadata about them and where you can connect to for more information. The understanding was your client would parse this information into a usable form which linked the entries with the destination pages.

The first popular protocol was FTP, which was created in 1971, as a way of listing and downloading files on remote computers. Gopher was a logical extension of this, in that it provided a similar listing, but included facilities for also reading the metadata about entries. This meant it could be used for more liberal purposes like a news feed or a simple database. It did not have, however, the freedom and simplicity which characterizes HTTP and HTML.

HTTP is a very simple protocol, particularily when compared to alternatives like FTP or even the HTTP/3 protocol which is rising in popularity today. First off, HTTP is entirely text based, rather than being composed of bespoke binary incantations (which would have made it significantly more efficient). Tim Berners-Lee correctly intuited that using a text-based format would make it easier for generations of programmers to develop and debug HTTP-based applications.

HTTP also makes almost no assumptions about what you’re transmitting. Despite the fact that it was invented expliticly to accompany the HTML language, it allows you to specify that your content is of any type (using the MIME Content-Type, which was a new invention at the time). The protocol itself is rather simple:

A request:

GET /index.html HTTP/1.1
Host: www.example.com

Might respond:

HTTP/1.1 200 OK
Date: Mon, 23 May 2005 22:38:34 GMT
Content-Type: text/html; charset=UTF-8
Content-Encoding: UTF-8
Content-Length: 138
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux)
ETag: "3f80f-1b6-3e1cb03b"
Accept-Ranges: bytes
Connection: close

<html>
    <head>
        <title>An Example Page</title>
    </head>
    <body>
        Hello World, this is a very simple HTML document.
    </body>
</html>

To put this in context, you can think of the networking system the Internet uses as starting with IP, the Internet Protocol. IP is responsible for getting a small packet of data (around 1500 bytes) from one computer to another. On top of that we have TCP, which is responsible for taking larger blocks of data like entire documents and files and sending them via many IP packets reliably. On top of that, we then implement a protocol like HTTP or FTP, which specifies what format should be used to make the data we send via TCP (or UDP, etc.) understandable and meaningful.

In other words, TCP/IP sends a whole bunch of bytes to another computer, the protocol says what those bytes should be and what they mean.

You can make your own protocol if you like, assemblying the bytes in your TCP messages however you like. The only requirement is that whoever you are talking to speaks the same language. For this reason, it’s common to standardize these protocols.

There are, of course, many less important protocols to play with. For example there is a Quote of The Day protocol (port 17), and a Random Characters protocol (port 19). They may seem silly today, but they also showcase just how important that a general-purpose document transmission format like HTTP was.

The timeline of Gopher and HTTP can be evidenced by their default port numbers. Gopher is 70, HTTP 80. The HTTP port was assigned (likely by Jon Postel at the IANA) at the request of Tim Berners-Lee sometime between 1990 and 1992.

This concept, of registering ‘port numbers’ predates even the Internet. In the original NCP protocol which powered the ARPANET remote addresses were identified by 40 bits. The first 32 identified the remote host, similar to how an IP address works today. The last eight were known as the AEN (it stood for “Another Eight-bit Number”), and were used by the remote machine in the way we use a port number, to separate messages destined for different processes. In other words, the address specifies which machine the message should go to, and the AEN (or port number) tells that remote machine which application should get the message.

They quickly requested that users register these ‘socket numbers’ to limit potential collisions. When port numbers were expanded to 16 bits by TCP/IP, that registration process was continued.

While protocols have a default port, it makes sense to allow ports to also be specified manually to allow for local development and the hosting of multiple services on the same machine. That same logic was the basis for prefixing websites with www.. At the time, it was unlikely anyone was getting access to the root of their domain, just for hosting an ‘experimental’ website. But if you give users the hostname of your specific machine (dx3.cern.ch), you’re in trouble when you need to replace that machine. By using a common subdomain (www.cern.ch) you can change what it points to as needed.

As you probably know, the URL syntax places a double slash (//) between the protocol and the rest of the URL:

http://cloudflare.com

That double slash was inherited from the Apollo computer system which was one of the first networked workstations. The Apollo team had a similar problem to Tim Berners-Lee: they needed a way to separate a path from the machine that path is on. Their solution was to create a special path format:

//computername/file/path/as/usual

And TBL copied that scheme. Incidentally, he now regrets that decision, wishing the domain (in this case example.com) was the first portion of the path:

http:com/example/foo/bar/baz

URLs were never intended to be what they’ve become: an arcane way for a user to identify a site on the Web. Unfortunately, we’ve never been able to standardize URNs, which would give us a more useful naming system. Arguing that the current URL system is sufficient is like praising the DOS command line, and stating that most people should simply learn to use command line syntax. The reason we have windowing systems is to make computers easier to use, and more widely used. The same thinking should lead us to a superior way of locating specific sites on the Web.

— Dale Dougherty 1996

There are several different ways to understand the ‘Internet’. One is as a system of computers connected using a computer network. That version of the Internet came into being in 1969 with the creation of the ARPANET. Mail, files and chat all moved over that network before the creation of HTTP, HTML, or the ‘web browser’.

In 1992 Tim Berners-Lee created three things, giving birth to what we consider the Internet. The HTTP protocol, HTML, and the URL. His goal was to bring ‘Hypertext’ to life. Hypertext at its simplest is the ability to create documents which link to one another. At the time it was viewed more as a science fiction panacea, to be complimented by Hypermedia, and any other word you could add ‘Hyper’ in front of.

The key requirement of Hypertext was the ability to link from one document to another. In TBL’s time though, these documents were hosted in a multitude of formats and accessed through protocols like Gopher and FTP. He needed a consistent way to refer to a file which encoded its protocol, its host on the Internet, and where it existed on that host.

At the original World-Wide Web presentation in March of 1992 TBL described it as a ‘Universal Document Identifier’ (UDI). Many different formats were considered for this identifier:

protocol: aftp host: xxx.yyy.edu path: /pub/doc/README
 
PR=aftp; H=xx.yy.edu; PA=/pub/doc/README;
 
PR:aftp/xx.yy.edu/pub/doc/README
 
/aftp/xx.yy.edu/pub/doc/README

This document also explains why spaces must be encoded in URLs (%20):

> The use of white space characters has been avoided in UDIs: spaces > are not legal characters. This was done because of the frequent > introduction of extraneous white space when lines are wrapped by > systems such as mail, or sheer necessity of narrow column width, and > because of the inter-conversion of various forms of white space > which occurs during character code conversion and the transfer of > text between applications.

What’s most important to understand is that the URL was fundamentally just an abbreviated way of refering to the combination of scheme, domain, port, credentials and path which previously had to be understood contextually for each different communication system.

It was first officially defined in an RFC published in 1994.

scheme:[//[user:password@]host[:port]][/]path[?query][#fragment]

This system made it possible to refer to different systems from within Hypertext, but now that virtually all content is hosted over HTTP, may not be as necessary anymore. As early as 1996 browsers were already inserting the http:// and www. for users automatically (rendering any advertisement which still contains them truly ridiculous).

I do not think the question is whether people can learn the meaning of the URL, I just find it it morally abhorrent to force grandma or grandpa to understand what, in the end, are UNIX file system conventions.

— Israel del Rio 1996

The slash separated path component of a URL should be familiar to any user of any computer built in the last fifty years. The hierarchal filesystem itself was introduced by the MULTICS system. Its creator, in turn, attributes it to a two hour conversation with Albert Einstein he had in 1952.

MULTICS used the greater than symbol (>) to separated file path components. For example:

>usr>bin>local>awk

That was perfectly logical, but unfortunately the Unix folks decided to use > to represent redirection, delegating path separation to the forward slash (/).

Wrong. We are I now see clearly *disagreeing*. You and I.

...

As a person I reserve the right to use different criteria for different purposes. I want to be able to give names to generic works, AND to particular translations AND to particular versions. I want a richer world than you propose. I don’t want to be constrained by your two-level system of “documents” and “variants”.

— Tim Berners-Lee 1993

One half of the URLs referenced by US Supreme Court opinions point to pages which no longer exist. If you were reading an academic paper in 2011, written in 2001, you have better than even odds that any given URL won’t be valid.

There was a fervent belief in 1993 that the URL would die, in favor of the ‘URN’. The Uniform Resource Name is a permanent reference to a given piece of content which, unlike a URL, will never change or break. Tim Berners-Lee first described the “urgent need” for them as early as 1991.

The simplest way to craft a URN might be to simply use a cryptographic hash of the contents of the page, for example: urn:791f0de3cfffc6ec7a0aacda2b147839. This method doesn’t meet the criteria of the web community though, as it wasn’t really possible to figure out who to ask to turn that hash into a piece of real content. It also didn’t account for the format changes which often happen to files (compressed vs uncompressed for example) which nevertheless represent the same content.

In 1996 Keith Shafer and several others proposed a solution to the problem of broken URLs. The link to this solution is now broken. Roy Fielding posted an implementation suggestion in July of 1995. That link is now broken.

I was able to find these pages through Google, which has functionally made page titles the URN of today. The URN format was ultimately finalized in 1997, and has essentially never been used since. The implementation is itself interesting. Each URN is composed of two components, an authority who can resolve a given type of URN, and the specific ID of this document in whichever format the authority understands. For example, urn:isbn:0131103628 will identify a book, forming a permanent link which can (hopefully) be turned into a set of URLs by your local isbn resolver.

Given the power of search engines, it’s possible the best URN format today would be a simple way for files to point to their former URLs. We could allow the search engines to index this information, and link us as appropriate:

<!-- On http://zack.is/history -->
<link rel="past-url" href="http://zackbloom.com/history.html">
<link rel="past-url" href="http://zack.is/history.html">

The "application/x-www-form-urlencoded" format is in many ways an aberrant monstrosity, the result of many years of implementation accidents and compromises leading to a set of requirements necessary for interoperability, but in no way representing good design practices.

— WhatWG URL Spec

If you’ve used the web for any period of time, you are familiar with query parameters. They follow the path portion of the URL, and encode options like ?name=zack&state=mi. It may seem odd to you that queries use the ampersand character (&) which is the same character used in HTML to encode special characters. In fact, if you’ve used HTML for any period of time, you likely have had to encode ampersands in URLs, turning http://host/?x=1&y=2 into http://host/?x=1&y=2 or http://host?x=1&y=2 (that particular confusion has always existed).

You may have also noticed that cookies follow a similar, but different format: x=1;y=2 which doesn’t actually conflict with HTML character encoding at all. This idea was not lost on the W3C, who encouraged implementers to support ; as well as & in query parameters as early as 1995.

Originally, this section of the URL was strictly used for searching ‘indexes’. The Web was originally created (and its funding was based on it creating) a method of collaboration for high energy physicists. This is not to say Tim Berners-Lee didn’t know he was really creating a general-purpose communication tool. He didn’t add support for tables for years, which is probably something physicists would have needed.

In any case, these ‘physicists’ needed a way of encoding and linking to information, and a way of searching that information. To provide that, Tim Berners-Lee created the <ISINDEX> tag. If <ISINDEX> appeared on a page, it would inform the browser that this is a page which can be searched. The browser should show a search field, and allow the user to send a query to the server.

That query was formatted as keywords separated by plus characters (+):

http://cernvm/FIND/?sgml+cms

In fantastic Internet fashion, this tag was quickly abused to do all manner of things including providing an input to calculate square roots. It was quickly proposed that perhaps this was too specific, and we really needed a general purpose <input> tag.

That particular proposal actually uses plus signs to separate the components of what otherwise looks like a modern GET query:

http://somehost.somewhere/some/path?x=xxxx+y=yyyy+z=zzzz

This was far from universally acclaimed. Some believed we needed a way of saying that the content on the other side of links should be searchable:

<a HREF="wais://quake.think.com/INFO" INDEX=1>search</a>

Tim Berners-Lee thought we should have a way of defining strongly-typed queries:

<ISINDEX TYPE="iana:/www/classes/query/personalinfo">

I can be somewhat confident in saying, in retrospect, I am glad the more generic solution won out.

The real work on <INPUT> began in January of 1993 based on an older SGML type. It was (perhaps unfortunately), decided that <SELECT> inputs needed a separate, richer, structure:

<select name=FIELDNAME type=CHOICETYPE [value=VALUE] [help=HELPUDI]> 
    <choice>item 1
    <choice>item 2
    <choice>item 3
</select>

If you’re curious, reusing <li>, rather than introducing the <option> element was absolutely considered. There were, of course, alternative form proposals. One included some variable substituion evocative of what Angular might do today:

<ENTRYBLANK TYPE=int LENGTH=length DEFAULT=default VAR=lval>Prompt</ENTRYBLANK>
<QUESTION TYPE=float DEFAULT=default VAR=lval>Prompt</QUESTION>
<CHOICE DEFAULT=default VAR=lval>
    <ALTERNATIVE VAL=value1>Prompt1 ...
    <ALTERNATIVE VAL=valuen>Promptn
</CHOICE>

In this example the inputs are checked against the type specified in type, and the VAR values are available on the page for use in string substitution in URLs, à la:

http://cloudflare.com/apps/$appId

Additional proposals actually used @, rather than =, to separate query components:

name@value+name@(value&value)

It was Marc Andreessen who suggested our current method based on what he had already implemented in Mosaic:

name=value&name=value&name=value

Just two months later Mosaic would add support for method=POST forms, and ‘modern’ HTML forms were born.

Of course, it was also Marc Andreessen’s company Netscape who would create the cookie format (using a different separator). Their proposal was itself painfully shortsighted, led to the attempt to introduce a Set-Cookie2 header, and introduced fundamental structural issues we still deal with at Cloudflare to this day.

The portion of the URL following the ‘#’ is known as the fragment. Fragments were a part of URLs since their initial specification, used to link to a specific location on the page being loaded. For example, if I have an anchor on my site:

<a name="bio"></a>

I can link to it:

http://zack.is/#bio

This concept was gradually extended to any element (rather than just anchors), and moved to the id attribute rather than name:

<h1 id="bio">Bio</h1>

Tim Berners-Lee decided to use this character based on its connection to addresses in the United States (despite the fact that he’s British by birth). In his words:

In a snail mail address in the US at least, it is common to use the number sign for an apartment number or suite number within a building. So 12 Acacia Av #12 means “The building at 12 Acacia Av, and then within that the unit known numbered 12”. It seemed to be a natural character for the task. Now, http://www.example.com/foo#bar means “Within resource http://www.example.com/foo, the particular view of it known as bar”.

It turns out that the original Hypertext system, created by Douglas Engelbart, also used the ‘#’ character for the same purpose. This may be coincidental or it could be a case of accidental “idea borrowing”.

Fragments are explicitly not included in HTTP requests, meaning they only live inside the browser. This concept proved very valuable when it came time to implement client-side navigation (before pushState was introduced). Fragments were also very valuable when it came time to think about how we can store state in URLs without actually sending it to the server. What could that mean? Let’s explore:

There is a whole standard, as yukky as SGML, on Electronic data Intercahnge [sic], meaning forms and form submission. I know no more except it looks like fortran backwards with no spaces.

— Tim Berners-Lee 1993

There is a popular perception that the internet standards bodies didn’t do much from the finalization of HTTP 1.1 and HTML 4.01 in 2002 to when HTML 5 really got on track. This period is also known (only by me) as the Dark Age of XHTML. The truth is though, the standardization folks were fantastically busy. They were just doing things which ultimately didn’t prove all that valuable.

One such effort was the Semantic Web. The dream was to create a Resource Description Framework (editorial note: run away from any team which seeks to create a framework), which would allow metadata about content to be universally expressed. For example, rather than creating a nice web page about my Corvette Stingray, I could make an RDF document describing its size, color, and the number of speeding tickets I had gotten while driving it.

This is, of course, in no way a bad idea. But the format was XML based, and there was a big chicken-and-egg problem between having the entire world documented, and having the browsers do anything useful with that documentation.

It did however provide a powerful environment for philosophical argument. One of the best such arguments lasted at least ten years, and was known by the masterful codename ‘httpRange-14’.

httpRange-14 sought to answer the fundamental question of what a URL is. Does a URL always refer to a document, or can it refer to anything? Can I have a URL which points to my car?

They didn’t attempt to answer that question in any satisfying manner. Instead they focused on how and when we can use 303 redirects to point users from links which aren’t documents to ones which are, and when we can use URL fragments (the bit after the ‘#’) to point users to linked data.

To the pragmatic mind of today, this might seem like a silly question. To many of us, you can use a URL for whatever you manage to use it for, and people will use your thing or they won’t. But the Semantic Web cares for nothing more than semantics, so it was on.

This particular topic was discussed on July 1st 2002, July 15th 2002, July 22nd 2002, July 29th 2002, September 16th 2002, and at least 20 other occasions through 2005. It was resolved by the great ‘httpRange-14 resolution’ of 2005, then reopened by complaints in 2007 and 2011 and a call for new solutions in 2012. The question was heavily discussed by the pedantic web group, which is very aptly named. The one thing which didn’t happen is all that much semantic data getting put on the web behind any sort of URL.

As you may know, you can include a username and password in URLs:

http://zack:shhhhhh@zack.is

The browser then encodes this authentication data into Base64, and sends it as a header:

Authentication: Basic emFjazpzaGhoaGho

The only reason for the Base64 encoding is to allow characters which might not be valid in a header, it provides no obscurity to the username and password values.

Particularily over the pre-SSL internet, this was very problematic. Anyone who could snoop on your connection could easily see your password. Many alternatives were proposed including Kerberos which is a widely used security protocol both then and now.

As with so many of these examples though, the simple basic auth proposal was easiest for browser manufacturers (Mosaic) to implement. This made it the first, and ultimately the only, solution until developers were given the tools to build their own authentication systems.

In the world of web applications, it can be a little odd to think of the basis for the web being the hyperlink. It is a method of linking one document to another, which was gradually augmented with styling, code execution, sessions, authentication, and ultimately became the social shared computing experience so many 70s researchers were trying (and failing) to create. Ultimately, the conclusion is just as true for any project or startup today as it was then: all that matters is adoption. If you can get people to use it, however slipshod it might be, they will help you craft it into what they need. The corollary is, of course, no one is using it, it doesn’t matter how technically sound it might be. There are countless tools which millions of hours of work went into which precisely no one uses today.

This was adapted from a post which originally appeared on the Eager blog. In 2016 Eager become Cloudflare Apps.

Let’s consider jQuery, the most popular JavaScript library on Earth. This chart shows the number of requests made for a selected list of jQuery versions over the past 12 months:

Spikes in the CDNJS data as you see with version 3.3.1 are not uncommon as very large sites add and remove CDNJS script tags.

We see a steady rise of version 3.4.1 following its release on May 2nd, 2019. What we don’t see is a substantial decline of old versions. Version 3.2.1 shows an average popularity of 36M requests at the beginning of our sample, and 29M at the end, a decline of approximately 20%. This aligns with a corpus of research which shows the average website lasts somewhere between two and four years. What we don’t see is a decline in our old versions which come close to the volume of growth of new versions when they’re released. In fact the release of 3.4.1, as popular as it quickly becomes, doesn’t change the trend of old version deprecation at all.

If you’re curious, the oldest version of jQuery CDNJS includes is 1.10.0, released on May 25, 2013. The project still gets an average of 100k requests per day, and the sites which use it are growing in popularity:

To confirm our theory, let’s consider another project, TweenMax:

As this package isn’t as popular as jQuery, the data has been smoothed with a one week trailing average to make it easier to identify trends.

Version 1.20.4 begins the year with 18M requests, and ends it with 14M, a decline of about 23%, again in alignment with the loss of websites on the Internet. The growth of 2.1.3 shows clear evidence that the release of a new version has almost no bearing on the popularity of old versions, the trend line for those older versions doesn’t change even as 2.1.3 grows to 29M requests per day.

One conclusion is whatever libraries you publish will exist on websites forever. The underlying web platform consequently must support aged conventions indefinitely if it is to continue supporting the full breadth of the web.

Cloudflare is very interested in how we can contribute to a web which is kept up-to-date. Please make suggestions in the comments below.

Virtually every site is composed of not just the code written by its developers, but also dozens or hundreds of libraries. These libraries make it possible for websites to extend what a web browser can do on its own. For example, libraries can allow a site to include powerful data visualizations, respond to user input, or even get more performant.

These libraries created wondrous and magical new capabilities for web browsers, but they can also cause the size of a site to explode. Particularly a decade ago, connections were not always fast enough to permit the use of many libraries while maintaining performance. But if so many websites are all including the same libraries, why was it necessary for each of them to load their own copy?

If we all load jQuery from the same place the browser can do a much better job of not actually needing to download it for every site. When the user visits the first jQuery-powered site it will have to be downloaded, but it will already be cached on the user's computer for any subsequent jQuery-powered site they might visit.

The first visit might take time to load:

But any future visit to any website pointing to this common URL would already be cached:

<!-- Loaded only on my site, will need to be downloaded by every user -->
<script src="./jquery.js"></script>

<!-- Loaded from a common location across many sites -->
<script src="https://cdnjs.cloudflare.com/jquery.js"></script>

Beyond the performance advantage, including files this way also made it very easy for users to experiment and create. When using a web browser as a creation tool users often didn't have elaborate build systems (this was also before npm), so being able to include a simple script tag was a boon. It's worth noting that it's not clear a massive performance advantage was ever actually provided by this scheme. It is becoming even less of a performance advantage now that browser vendors are beginning to use separate cache's for each website you visit, but with millions of sites using CDNJS there's no doubt it is a critical part of the web.

My first Pull Request into the CDNJS project was in 2013. Back then if you created a JavaScript project it wasn't possible to have it included in the jQuery CDN, or the ones provided by large companies like Google and Microsoft. They were only for big, important, projects. Of course, however, even the biggest project starts small. The community needed a CDN which would agree to host nearly all JavaScript projects, even the ones which weren't world-changing (yet). In 2011, that project was launched by Ryan Kirkman and Thomas Davis as CDNJS.

The project was quickly wildly successful, far beyond their expectations. Their CDN bill quickly began to skyrocket (it would now be over a million dollars a year on AWS). Under the threat of having to shut down the service, Cloudflare was approached by the CDNJS team to see if we could help. We agreed to support their efforts and created cdnjs.cloudflare.com which serves the CDNJS project free of charge.

CDNJS has been astonishingly successful. The project is currently installed on over eighteen million websites (10% of the Internet!), offers files totaling over 1.5 billion lines of code, and serves over 173 billion requests a month. CDNJS only gets more popular as sites get larger, with 34% of the top 10k websites using the service. Each month we serve almost three petabytes of JavaScript, CSS, and other resources which power the web via cdnjs.cloudflare.com.

Spikes can happen when a very large or popular site installs CDNJS, or when a misbehaving web crawler discovers a CDNJS link.

The future value of CDNJS is now in doubt, as web browsers are beginning to use a separate cache for every website you visit. It is currently used on such a wide swath of the web, however, it is unlikely it will be disappearing any time soon.

CDNJS starts with a GitHub repo. That project contains every file served by CDNJS, at every version which it has ever offered. That’s 182 GB without the commit history, over five million files, and over 1.5 billion lines of code.

Given that it stores and delivers versioned code files, in many ways it was the Internet’s first JavaScript package manager. Unlike other package managers and even other CDNs everything CDNJS serves is publicly versioned. All 67,724 commits! This means you as a user can verify that you are being served files which haven’t been tampered with.

To make changes to CDNJS a commit has to be made. For new projects being added to CDNJS, or when projects change significantly, these commits are made by humans, and get reviewed by other humans. When projects just release new versions there is a bot made by Peter and maintained by Sven which sucks up changes from npm and automatically creates commits.

Within Cloudflare’s infrastructure there is a set of machines which are responsible for pulling the latest version of the repo periodically. Those machines then become the origin for cdnjs.cloudflare.com, with Cloudflare’s Global Load Balancer automatically handling failures. Cloudflare’s cache automatically stores copies of many of the projects making it possible for us to deliver them quickly from all 195 of our data centers.

The CDNJS project has always been administered independently of Cloudflare. In addition to the founders, the project has additionally been maintained by exceptionally hard-working caretakers like Peter and Matt Cowley. Maintaining a single repo of nearly every frontend project on Earth is no small task, and it has required a substantial amount of both manual work and bot development.

Unfortunately approximately thirty days ago one of those bots stopped working, preventing updated projects from appearing in CDNJS. The bot's open-source maintainer was not able to invest the time necessary to keep the bot running. After several weeks we were asked by the community and the CDNJS founders to take over maintenance of the CDNJS repo itself. This means the Cloudflare engineering team is taking responsibility for keeping the contents of github.com/cdnjs/cdnjs up to date, in addition to ensuring it is correctly served on cdnjs.cloudflare.com.

We agreed to do this because we were, frankly, scared. Like so many open-source projects CDNJS was a critical part of our world, but wasn’t getting the attention it needed to survive. The Internet relies on CDNJS as much as on any other single project, losing it or allowing it to be commandeered would be catastrophic to millions of websites and their visitors. If it began to fail, some sites would adapt and update, others would be broken forever.

CDNJS has always been, and remains, a project for and by the community. We are invested in making all decisions in a transparent and inclusive manner. If you are interested in contributing to CDNJS or in the topics we're currently discussing please visit the CDNJS Github Issues page.

One example of an area where we could use your help is in charting a path towards a CDNJS which requires less manual moderation. Nothing can replace the intelligence and creativity of a human (yet), but for a task like managing what resources go into a CDN, it is error-prone and time-consuming. At present a human has to review every new project to be included, and often has to take additional steps to include new versions of a project.

As a part of our analysis of the project we examined a snapshot of the still-open PRs made against CDNJS for several months:

The vast majority of these PRs were changes which ultimately passed the automated review but nevertheless couldn't be merged without manual review.

There is consensus that we should move to a model which does not require human involvement in most cases. We would love your input and collaboration on the best way for that to be solved. If this is something you are passionate about, please contribute here.

Our plan is to support the CDNJS project in whichever ways it requires for as long as the Internet relies upon it. We invite you to use CDNJS in your next project with the full assurance that it is backed by the same network and team who protect and accelerate over twenty million of your favorite websites across the Internet. We are also planning more posts diving further into the CDNJS data, subscribe to this blog if you would like to be notified upon their release.

To understand all these problems and how we solved them we first need to give you some background on how the Cloudflare network works:

The Cloudflare network is composed of data centers located in 194 cities and in more than 90 countries. Every Cloudflare data center is composed of many servers that receive a continual flood of requests and has to distribute those requests between the servers that handle them. We use a set of routers to perform that operation:

Our routers listen on Anycast IP addresses which are advertised over the public Internet. If you have a site on Cloudflare, your site is available via two of these addresses. In this case, I am doing a DNS query for “workers.dev”, a site which is powered by Cloudflare:

➜ dig workers.dev

;; QUESTION SECTION:
;workers.dev.      IN  A

;; ANSWER SECTION:
workers.dev.    161  IN  A  198.41.215.162
workers.dev.    161  IN  A  198.41.214.162

;; SERVER: 1.1.1.1#53(1.1.1.1)

workers.dev is available at two addresses 198.41.215.162 and 198.41.214.162 (along with two IPv6 addresses available via the AAAA DNS query). Those two addresses are advertised from every one of our data centers around the world. When someone connects to any Internet property on Cloudflare, each networking device their packets pass through will choose the shortest path to the nearest Cloudflare data center from their computer or phone.

Once the packets hit our data center, we send them to one of the many servers which operate there. Traditionally, one might use a load balancer to do that type of traffic distribution across multiple machines. Unfortunately putting a set of load balancers capable of handling our volume of traffic in every data center would be exceptionally expensive, and wouldn’t scale as easily as our servers do. Instead, we use devices built for operating on exceptional volumes of traffic: network routers.

Once a packet hits our data center it is processed by a router. That router sends the traffic to one of a set of servers responsible for handling that address using a routing strategy called ECMP (Equal-Cost Multi-Path). ECMP refers to the situation where the router doesn’t have a clear ‘winner’ between multiple routes, it has multiple good next hops, all to the same ultimate destination. In our case we hack that concept a bit, rather than using ECMP to balance across multiple intermediary links, we make the intermediary link addresses the final destination of our traffic: our servers.

Here's the configuration of a Juniper-brand router of the type which might be in one of our data centers, and which is configured to balance traffic across three destinations:

user@host# show routing-options

static {
  route 172.16.1.0/24 next-hop [ 172.16.2.1 172.16.2.2 172.16.2.3 ];
}
forwarding-table {
  export load-balancing-policy;
}

Since the ‘next-hop’ is our server, traffic will be split across multiple machines very efficiently.

IP is responsible for sending packets of data from addresses like 93.184.216.34 to 208.80.153.224 (or [2606:2800:220:1:248:1893:25c8:1946] to [2620:0:860:ed1a::1] in the case of IPv6) across the Internet. It's the "Internet Protocol".

TCP (Transmission Control Protocol) operates on top of a protocol like IP which can send a packet from one place to another, and makes data transmission reliable and useful for more than one process at a time. It is responsible for taking the unreliable and misordered packets that might arrive over a protocol like IP and delivering them reliably, in the correct order. It also introduces the concept of a ‘port’, a number from 1-65535 which help route traffic on a computer or phone to a specific service (such as the web or email). Each TCP connection has a source and destination port which is included in the header TCP adds to the beginning of each packet. Without the idea of ports it would not be easy to figure out which messages were destined for which program. For example, both Google Chrome and Mail might wish to send messages over your WiFi connection at the same time, so they will each use their own port.

Here's an example of making a request for https://cloudflare.com/ at 198.41.215.162, on the default port for HTTPS: 443. My computer has randomly assigned me the port 51602 which it will listen on it for a response, which will (hopefully) receive the contents of the site:

Internet Protocol Version 4, Src: 19.5.7.21, Dst: 198.41.215.162
    Protocol: TCP (6)
    Source: 19.5.7.21
    Destination: 198.41.215.162
Transmission Control Protocol, Src Port: 51602, Dst Port: 443, Seq: 0, Len: 0
    Source Port: 51602
    Destination Port: 443

Looking at the same request from the Cloudflare side will be a mirror image, a request from my public IP address originating at my source port, destined for port 443 (I’m ignoring NAT for the moment, more on that later):

Internet Protocol Version 4, Src: 198.41.215.16, Dst: 19.5.7.21
    Protocol: TCP (6)
    Source: 198.41.215.162
    Destination: 19.5.7.21
Transmission Control Protocol, Src Port: 443, Dst Port: 51602, Seq: 0, Len: 0
    Source Port: 443
    Destination Port: 51602

We can now return to ECMP! It could be theoretically possible to use ECMP to balance packets between servers randomly, but you would almost never want to do that. A message over the Internet is generally composed of multiple TCP packets. If each packet were sent to a different server it would be impossible to reconstruct the original message in any one place and act on it. Even beyond that, it would be terrible for performance: we rely on being able to maintain long-lived TCP and TLS sessions which require a persistent connection to a single server. To provide that persistence, our routers don’t balance traffic randomly, they use a combination of four values: the source address, the source port, the destination address, and the destination port. Traffic with the same combination of those four values will always make it to the same server. In the case of my example above, all of my messages destined to cloudflare.com will make it to a single server which can reconstruct the TCP packets into my request and return packets in a response.

For a conventional request it is very important that our ECMP routing sends all of your packets to the same server for the duration of your request. Over the web a request commonly lasts less than ten seconds and the system works well. Unfortunately we quickly ran into issues with WARP.

WARP uses a session key negotiated with public-key encryption to secure packets. For a successful connection, both sides must negotiate a connection which is then only valid for that particular client and the specific server they are talking to. This negotiation takes time and has to be completed any time a client talks to a new server. Even worse, if packets get sent which expect one server, and end up at another, they can’t be decrypted, breaking the connection. Detecting those failed packets and restarting the connection from scratch takes so much time that our alpha testers experienced it as a complete loss of their Internet connection. As you can imagine, testers don’t leave WARP on very long when it prevents them from using the Internet.

WARP was experiencing so many failures because devices were switching servers much more often than we expected. If you recall, our ECMP router configuration uses a combination of (Source IP, Source Port, Destination IP, Destination Port) to match a packet to a server. Destination IP doesn’t generally change, WARP clients are always connecting to the same Anycast addresses. Similarly, Destination Port doesn’t change, we always listen on the same port for WARP traffic. The other two values, Source IP and Source Port, were changing much more frequently than we had planned.

One source of these changes was expected. WARP runs on cell phones, and cell phones commonly switch from Cellular to Wi-Fi connections. When you make that switch you suddenly go from communicating over the Internet via your cellular carrier’s (like AT&T or Verizon) IP address space to that of the Internet Service Provider your Wi-Fi connection uses (like Comcast or Google Fiber). It’s essentially impossible that your IP address won’t change when you move between connections.

The port changes occurred even more frequently than could be explained by network switches however. For an understanding of why we need to introduce one more component of Internet lore: Network Address Translation.

An IPv4 address is composed of 32 bits (often written as four eight-bit numbers). If you exclude the reserved addresses which can’t be used, you are left with 3,706,452,992 possible addresses. This number has remained constant since IPv4 was deployed on the ARPANET in 1983, even as the number of devices has exploded (although it might go up a bit soon if the 0.0.0.0/8 becomes available). This data is based on Gartner Research predictions and estimates:

IPv6 is the definitive solution to this problem. It expands the length of an address from 32 to 128 bits, with 125 available in a valid Internet address at the moment (all public IPv6 addresses have the first three bits set to 001, the remaining 87.5% of the IPv6 address space is not considered necessary yet). 2^125 is an impossibly large number and would be more than enough for every device on Earth to have its own address. Unfortunately, 21 years after it was published, IPv6 still remains unsupported on many networks. Much of the Internet still relies on IPv4, and as seen above, there aren’t enough IPv4 addresses for every device to have their own.

To solve this problem many devices are commonly put behind a single Internet-addressable IP address. A router is used to do Network Address Translation; to take messages which arrive on that single public IP and forward them to the appropriate device on their local network. In effect it’s as if everyone in your apartment building had the same street address, and the postal worker was responsible for sorting out what mail was meant for which person.

When your devices send a packet destined for the Internet your router intercepts it. The router then rewrites the source address to the single public Internet address allocated for you, and the source port to a port which is unique for all the messages being sent across all the Internet-connected devices on your network. Just as your computer chooses a random source port for your messages which was unique between all the different processes on your computer, your router chooses a random source port which is unique for all the Internet connections across your entire network. It remembers the port it is selecting for you as belonging to your connection, and allows the message to continue over the Internet.

When a response arrives destined for the port it has allocated to you, it matches it to your connection and again rewrites it, this time replacing the destination address with your address on the local network, and the destination port with the original source port you specified. It has transparently allowed all the devices on your network to act as if they were one big computer with a single Internet-connected IP address.

This process works very well for the duration of a common request over the Internet. Your router only has so much space however, so it will helpfully delete old port assignments, freeing up space for new ones. It generally waits for the connection to not have any messages for thirty seconds or more before deleting an assignment, making it unlikely a response will arrive which it can no longer direct to the appropriate source. Unfortunately, WARP sessions need to last much longer than thirty seconds.

When you next send a message after your NAT session has expired, you are given a new source port. That new port causes your ECMP mapping (based on source IP, source port, destination IP, destination port) to change, causing us to route your requests to a new machine within the Cloudflare data center your messages are arriving at. This breaks your WARP session, and your Internet connection.

We experimented extensively with methods of keeping your NAT session fresh by periodically sending keep-alive messages which would prevent routers and mobile carriers from evicting mappings. Unfortunately waking the radio of your device every thirty seconds has unfortunate consequences for your battery life, and it was not entirely successful at preventing port and address changes. We needed a way to always map sessions to the same machine, even as their source port (and even source address) changed.

Fortunately, we had a solution which came from elsewhere at Cloudflare. We don’t use dedicated load balancers, but we do have many of the same problems load balancers solve. We have long needed to map traffic to Cloudflare servers with more control than ECMP allows alone. Rather than deploying an entire tier of load balancers, we use every server in our network as a load balancer, forwarding packets first to an arbitrary machine and then relying on that machine to forward the packet to the appropriate host. This consumes minimal resources and allows us to scale our load balancing infrastructure with each new machine we add. We have a lot more to share on how this infrastructure works and what makes it unique, subscribe to this blog to be notified when that post is released.

To make our load balancing technique work though we needed a way to identify which client a WARP packet was associated with before it could be decrypted. To understand how we did that it’s helpful to understand how WARP encrypts your messages. The industry standard way of connecting a device to a remote network is a VPN. VPNs use a protocol like IPsec to allow your device to send messages securely to a remote network. Unfortunately, VPNs are generally rather disliked. They slow down connections, eat battery life, and their complexity makes them frequently the source of security vulnerabilities. Users of corporate networks which mandate VPNs often hate them, and the idea that we would convince millions of consumers to install one voluntarily seemed ridiculous.

After considering and testing several more modern options, we landed on WireGuard®. WireGuard is a modern, high performance, and most importantly, simple, protocol created by Jason Donenfeld to solve the same problem. Its original code-base is less than 1% the size of a popular IPsec implementation, making it easy for us to understand and secure. We chose Rust as the language most likely to give us the performance and safety we needed and implemented WireGuard while optimizing the code heavily to run quickly on the platforms we were targeting. Then we open sourced the project.

WireGuard changes two very relevant things about the traffic you send over the Internet. The first is it uses UDP not TCP. The second is it uses a session key negotiated with public-key encryption to secure the contents of that UDP packet.

TCP is the conventional protocol used for loading a website over the Internet. It combines the ability to address ports (which we talked about previously) with reliable delivery and flow control. Reliable delivery ensures that if a message is dropped, TCP will eventually resend the missing data. Flow control gives TCP the tools it needs to handle many clients all sharing the same link who exceed its capacity. UDP is a much simpler protocol which trades these capabilities for simplicity, it makes a best-effort attempt to send a message, and if the message is missing or there is too much data for the links, messages are simply never heard of again.

UDP’s lack of reliability would normally be a problem while browsing the Internet, but we are not simply sending UDP, we are sending a complete TCP packet _inside_ our UDP packets.

Inside the payload encrypted by WireGuard we have a complete TCP header which contains all the information necessary to ensure reliable delivery. We then wrap it with WireGuard’s encryption and use UDP to (less-than-reliably) send it over the Internet. Should it be dropped TCP will do its job just as if a network link lost the message and resend it. If we instead wrapped our inner, encrypted, TCP session in another TCP packet as some other protocols do we would dramatically increase the number of network messages required, destroying performance.

The second interesting component of WireGuard relevant to our discussion is public-key encryption. WireGuard allows you to secure each message you send such that only the specific destination you are sending it to can decrypt it. That is a powerful way of ensuring your security as you browse the Internet, but it means it is impossible to read anything inside the encrypted payload until the message has reached the server which is responsible for your session.

Returning to our load balancing issue, you can see that only three things are accessible to us before we can decrypt the message: The IP Header, the UDP Header, and the WireGuard header. Neither the IP Header or UDP Header include the information we need, as we have already failed with the four pieces of information they contain (source IP, source port, destination IP, destination port). That leaves the WireGuard header as the one location where we can find an identifier which can be used to keep track of who the client was before decrypting the message. Unfortunately, there isn’t one. This is the format of the message used to initiate a connection:

sender looks temptingly like a client id, but it’s randomly assigned every handshake. Handshakes have to be performed every two minutes to rotate keys making them insufficiently persistent. We could have forked the protocol to add any number of additional fields, but it is important to us to remain wire-compatible with other WireGuard clients. Fortunately, WireGuard has a three byte block in its header which is not currently used by other clients. We decided to put our identifier in this region and still support messages from other WireGuard clients (albeit with less reliable routing than we can offer). If this reserved section is used for other purposes we can ignore those bits or work with the WireGuard team to extend the protocol in another suitable way.

When we begin a WireGuard session we include our clientid field which is provided by our authentication server which has to be communicated with to begin a WARP session:

Data messages similarly include the same field:

It's important to note that the clientid is only 24 bits long. That means there are less possible clientid values than the current number of users waiting to use WARP. This suits us well as we don't need or want the ability to track individual WARP users. clientid is only necessary for load balancing, once it serves its purpose we get it expunged from our systems as quickly as we can.

The load balancing system now uses a hash of the clientid to identify which machine a packet should be routed to, meaning  WARP messages always arrive at the same machine even as you change networks or move from Wi-Fi to cellular, and the problem was eliminated.

Cloudflare has never developed client software before. We take pride in selling a service anyone can use without needing to buy hardware or provision infrastructure. To make WARP work, however, we needed to deploy our code onto one of the most ubiquitous hardware platforms on Earth: smartphones.

While developing software on mobile devices has gotten steadily easier over the past decade, unfortunately developing low-level networking software remains rather difficult. To consider one example: we began the project using the latest iOS connection API called Network, introduced in iOS 12. Apple strongly recommends the use of Network, in their words “Your customers are going to appreciate how much better your connections, how much more reliable your connections are established, and they'll appreciate the longer battery life from the better performance.”

The Network framework provides a pleasantly high-level API which, as they say, integrates well with the native performance features built into iOS. Creating a UDP connection (connection is a bit of a misnomer, there are no connections in UDP, just packets) is as simple as:

self.connection = NWConnection(host: hostUDP, port: portUDP, using: .udp)

And sending a message can be as easy as:

self.connection?.send(content: content)

Unfortunately, at a certain point code actually gets deployed, and bug reports begin flowing in. The first issue was the simplicity of the API made it impossible for us to process more than a single UDP packet at a time. We commonly use packets of up to 1500 bytes, running a speed test on my Google Fiber connection currently results in a speed of 370 Mbps, or almost thirty-one thousand packets per second. Attempting to process each packet individually was slowing down connections by as much as 40%. According to Apple, the best solution to get the performance we needed was to fallback to the older NWUDPSession API, introduced in iOS 9.

If we compare the code required to create a NWUDPSession to the example above you will notice that we suddenly care which protocol, IPv4 or IPv6, we are using:

let v4Session = NWUDPSession(upgradeFor: self.ipv4Session)
v4Session.setReadHandler(self.filteringReadHandler, maxDatagrams: 32)

In fact, NWUDPSession does not handle many of the more tricky elements of creating connections over the Internet. For example, the Network framework will automatically determine whether a connection should be made over IPv4 or 6:

NWUDPSession does not do this for you, so we began creating our own logic to determine which type of connection should be used. Once we began to experiment, it quickly became clear that they are not created equal. It’s fairly common for a route to the same destination to have very different performance based on whether you use its IPv4 or IPv6 address. Often this is because there are simply fewer IPv4 addresses which have been around for longer, making it possible for those routes to be better optimized by the Internet’s infrastructure.

Every Cloudflare product has to support IPv6 as a rule. In 2016, we enabled IPv6 for over 98% of our network, over four million sites, and made a pretty big dent in IPv6 adoption on the web:

We couldn’t release WARP without IPv6 support. We needed to ensure that we were always using the fastest possible connection while still supporting both protocols with equal measure. To solve that we turned to a technology we have used with DNS for years: Happy Eyeballs. As codified in RFC 6555 Happy Eyeballs is the idea that you should try to look for both an IPv4 and IPv6 address when doing a DNS lookup. Whichever returns first, wins. That way you can allow IPv6 websites to load quickly even in a world which does not fully support it.

As an example, I am loading the website http://zack.is/. My web browser makes a DNS request for both the IPv4 address (an “A” record) and the IPv6 address (an “AAAA” record) at the same time:

Internet Protocol Version 4, Src: 192.168.7.21, Dst: 1.1.1.1
User Datagram Protocol, Src Port: 47447, Dst Port: 53
Domain Name System (query)
    Queries
        zack.is: type A, class IN

Internet Protocol Version 4, Src: 192.168.7.21, Dst: 1.1.1.1
User Datagram Protocol, Src Port: 49946, Dst Port: 53
Domain Name System (query)
    Queries
        zack.is: type AAAA, class IN

In this case the response to the A query returned more quickly, and the connection is begun using that protocol:

Internet Protocol Version 4, Src: 1.1.1.1, Dst: 192.168.7.21
User Datagram Protocol, Src Port: 53, Dst Port: 47447
Domain Name System (response)
    Queries
        zack.is: type A, class IN
    Answers
        zack.is: type A, class IN, addr 104.24.101.191
       
Internet Protocol Version 4, Src: 192.168.7.21, Dst: 104.24.101.191
Transmission Control Protocol, Src Port: 55244, Dst Port: 80, Seq: 0, Len: 0
    Source Port: 55244
    Destination Port: 80
    Flags: 0x002 (SYN)

We don’t need to do DNS queries to make WARP connections, we know the IP addresses of our data centers already, but we do want to know which of the IPv4 and IPv6 addresses will lead to a faster route over the Internet. To accomplish that we perform the same technique but at the network level: we send a packet over each protocol and use the protocol which returns first for subsequent messages. With some error handling and logging removed for brevity, it appears as:

let raceFinished = Atomic<Bool>(false)

let happyEyeballsRacer: (NWUDPSession, NWUDPSession, String) -> Void = {
    (session, otherSession, name) in
    // Session is the session the racer runs for, otherSession is a session we race against

    let handleMessage: ([Data]) -> Void = { datagrams in
        // This handler will be executed twice, once for the winner, again for the loser.
        // It does not matter what reply we received. Any reply means this connection is working.

        if raceFinished.swap(true) {
            // This racer lost
            return self.filteringReadHandler(data: datagrams, error: nil)
        }

        // The winner becomes the current session
        self.wireguardServerUDPSession = session

        session.setReadHandler(self.readHandler, maxDatagrams: 32)
        otherSession.setReadHandler(self.filteringReadHandler, maxDatagrams: 32)
    }

    session.setReadHandler({ (datagrams) in
        handleMessage(datagrams)
    }, maxDatagrams: 1)

    if !raceFinished.value {
        // Send a handshake message
        session.writeDatagram(onViable())
    }
}

This technique successfully allows us to support IPv6 addressing. In fact, every device which uses WARP instantly supports IPv6 addressing even on networks which don’t have support. Using WARP takes the 34% of Comcast’s network which doesn’t support IPv6 or the 69% of Charter’s network which doesn’t (as of 2018), and allows those users to communicate to IPv6 servers successfully.

This test shows my phone’s IPv6 support before and after enabling WARP:

Nothing is simple however, with iOS 12.2 NWUDPSession began to trigger errors which terminated connections. These errors were only identified with a code ‘55’. After some research it appears 55 has referred to the same error since the early foundations of the FreeBSD operating system OS X was originally built upon. In FreeBSD it’s commonly referred to as ENOBUFS, and it’s returned when the operating system does not have sufficient BUFfer Space to handle the operation being completed. For example, looking at the source of a FreeBSD today, you see this code in its IPv6 implementation:

In this example, if enough memory cannot be allocated to accommodate the size of an IPv6 and ICMP6 header, the error ENOBUFS (which is mapped to the number 55) will be returned. Unfortunately, Apple’s take on FreeBSD is not open source however: how, when, and why they might be returning the error is a mystery. This error has been experienced by other UDP-based projects, but a resolution is not forthcoming.

What is clear is once an error 55 begins occurring, the connection is no longer usable. To handle this case we need to reconnect, but doing the same Happy Eyeballs mechanic we do on initial connection is both unnecessary (as we were already talking over the fastest connection), and will consume valuable time. Instead we add a second connection method which is only used to recreate an already working session:

/**
Create a new UDP connection to the server using a Happy Eyeballs like heuristic.

This function should be called when first establishing a connection to the edge server.

It will initiate a new connection over IPv4 and IPv6 in parallel, keeping the connection that receives the first response.
*/

func connect(onViable: @escaping () -> Data, onReply: @escaping () -> Void, onFailure: @escaping () -> Void, onDisconnect: @escaping () -> Void)

/**
Recreate the current connections.

This function should be called as a response to error code 55, when a quick connection is required.

Unlike `happyEyeballs`, this function will use viability as its only success criteria.
*/

func reconnect(onViable: @escaping () -> Void, onFailure: @escaping () -> Void, onDisconnect: @escaping () -> Void)

Using reconnect we are able to recreate sessions broken by code 55 errors, but it still adds a latency hit which is not ideal. As with all client software development on a closed-source platform however, we are dependent on the platform to identify and fix platform-level bugs.

Truthfully, this is just one of a long list of platform-specific bugs we ran into building WARP. We hope to continue working with device vendors to get them fixed. There are an unimaginable number of device and connection combinations, and each connection doesn’t just exist at one moment in time, they are always changing, entering and leaving broken states almost faster than we can track. Even now, getting WARP to work on every device and connection on Earth is not a solved problem, we still get daily bug reports which we work to triage and resolve.

WARP is meant to be a place where we can apply optimizations which make the Internet better. We have a lot of experience making websites more performant, WARP is our opportunity to experiment with doing the same for all Internet traffic.

At Cloudflare we have a product called Argo. Argo makes websites' time to first byte more than 30% faster on average by continually monitoring thousands of routes over the Internet between our data centers. That data builds a database which maps every IP address range with the fastest possible route to every destination. When a packet arrives it first reaches the closest data center to the client, then that data center uses data from our tests to discover the route which will get the packet to its destination with the lowest possible latency. You can think of it like a traffic-aware GPS for the Internet.

Argo has historically only operated on HTTP packets. HTTP is the protocol which powers the web, sending messages which load websites on top of TCP and IP. For example, if I load http://zack.is/, an HTTP message is sent inside a TCP packet:

Internet Protocol Version 4, Src: 192.168.7.21, Dst: 104.24.101.191
Transmission Control Protocol, Src Port: 55244, Dst Port: 80
    Source Port: 55244
    Destination Port: 80
    TCP payload (414 bytes)
Hypertext Transfer Protocol
    GET / HTTP/1.1\r\n
    Host: zack.is\r\n
    Connection: keep-alive\r\n
    Accept-Encoding: gzip, deflate\r\n
    Accept-Language: en-US,en;q=0.9\r\n
    \r\n

The modern and secure web presents a problem for us however: When I make the same request over HTTPS (https://zack.is) rather than just HTTP (http://zack.is), I see a very different result over the wire:

Internet Protocol Version 4, Src: 192.168.7.21, Dst: 104.25.151.102
Transmission Control Protocol, Src Port: 55983, Dst Port: 443
    Source Port: 55983
    Destination Port: 443
    Transport Layer Security
    TCP payload (54 bytes)
Transport Layer Security
    TLSv1.2 Record Layer: Application Data Protocol: http-over-tls
        Encrypted Application Data: 82b6dd7be8c5758ad012649fae4f469c2d9e68fe15c17297…

My request has been encrypted! It’s no longer possible for WARP (or anyone but the destination) to tell what is in the payload. It might be HTTP, but it also might be any other protocol. If my site is one of the twenty-million which use Cloudflare already, we can decrypt the traffic and accelerate it (along with a long list of other optimizations). But for encrypted traffic destined for another source existing HTTP-only Argo technology was not going to work.

Fortunately we now have a good amount of experience working with non-HTTP traffic through our Spectrum and Magic Transit products. To solve our problem the Argo team turned to the CONNECT protocol.

As we now know, when a WARP request is made it first communicates over the WireGuard protocol to a server running in one of our 194 data centers around the world. Once the WireGuard message has been decrypted, we examine the destination IP address to see if it is an HTTP request destined for a Cloudflare-powered site, or a request destined elsewhere. If it’s destined for us it enters our standard HTTP serving path; often we can reply to the request directly from our cache in the very same data center.

If it’s not destined for a Cloudflare-powered site we instead forward the packet to a proxy process which runs on each machine. This proxy is responsible for loading the fastest path from our Argo database and beginning an HTTP session with a machine in the data center this traffic should be forwarded to. It uses the CONNECT command to both transmit metadata (as headers) and turn the HTTP session into a connection which can transmit the raw bytes of the payload:

CONNECT 8.54.232.11:5564 HTTP/1.1\r\n
Exit-Tcp-Keepalive-Duration: 15\r\n
Application: warp\r\n
\r\n
<data to send to origin>

Once the message arrives at the destination data center it is either forwarded to another data center (if that is best for performance), or directed directly to the origin which is awaiting the traffic.

Smart routing is just the beginning of WARP+; We have a long list of projects and plans which are all aimed at making your Internet faster, and couldn’t be more thrilled to finally have a platform to test them with.

Today, after well over a year of development, WARP is available to you and to your friends and family. For us though, this is just the beginning. With the ability to improve full network connection for all traffic, we unlock a whole new world of optimizations and security improvements which were simply impossible before. We couldn't be more excited to experiment, play, and eventually release, all sorts of new WARP and WARP+ features.

Cloudflare’s mission is to help build a better Internet. If we are willing to experiment and solve hard technical problems together we believe we can help make the future of the Internet better than the Internet of today, and we are all grateful to play a part in that. Thank you for trusting us with your Internet connection.

WARP was built by Oli Yu, Vlad Krasnov, Chris Branch, Dane Knecht, Naga Tripirineni, Andrew Plunk, Adam Schwartz, Irtefa, and intern Michelle Chen with support from members of our Austin, San Francisco, Champaign, London, Warsaw, and Lisbon offices.

Cloudflare has countless servers located in 180 data centers around the world. Each one is capable of acting as a Layer 7 load balancer, directing incoming traffic between origins wherever they may be. You could, for example, add load balancing between a set of machines you have in AWS' EC2, and another set you keep in Google Cloud.

This load balancing isn't just round-robining traffic. It supports weighting to allow you to control how much traffic goes to each cluster. It supports latency-based routing to automatically route traffic to the cluster which is closer (so adding geographic distribution can be as simple as spinning up machines). It even supports health checks, allowing it to automatically direct traffic to the cloud which is currently healthy.

Most importantly, it doesn't run in any of the provider's clouds and isn't dependent on them to function properly. Even better, since the load balancing runs near virtually every Internet user around the world it doesn't come at any performance cost. (Using our Argo technology performance often gets better!).

One of the hardest components to managing a multi-cloud deployment is networking. Each provider has their own method of defining networks and firewalls, and even tools which can deploy clusters across multiple clouds often can't quite manage to get the networking configuration to work in the same way. The task of setting it up can often be a trial-and-error game where the final config is best never touched again, leaving 'going multi-cloud' as a failed experiment within organizations.

At Cloudflare we have a technology called Argo Tunnel which flips networking on its head. Rather than opening ports and directing incoming traffic, each of your virtual machines (or k8s pods) makes outbound tunnels to the nearest Cloudflare PoPs. All of your Internet traffic then flows over those tunnels. You keep all your ports closed to inbound traffic, and never have to think about Internet networking again.

What's so powerful about this configuration is is makes it trivial to spin up machines in new locations. Want a dozen machines in Australia? As long as they start the Argo Tunnel daemon they will start receiving traffic. Don't need them any more? Shut them down and the traffic will be routed elsewhere. And, of course, none of this relies on any one public cloud provider, making it reliable even if they should have issues.

Argo Tunnel makes it trivial to add machines in new clouds, or to keep machines on-prem even as you start shifting workloads into the Cloud.

One thing you'll realize about using Argo Tunnel is you now have secure tunnels which connect your infrastructure with Cloudflare's network. Once traffic reaches that network, it doesn't necessarily have to flow directly to your machines. It could, for example, have access control applied where we use your Identity Provider (like Okta or Active Directory) to decide who should be able to access what. Rather than wrestling with VPCs and VPN configurations, you can move to a zero-trust model where you use policies to decide exactly who can access what on a per-request basis.

In fact, you can now do this with SSH as well! You can manage all your user accounts in a single place and control with precision who can access which piece of infrastructure, irrespective of which cloud it's in.

No computer system is perfect, and ours is no exception. We make mistakes, have bugs in our code, and deal with the pain of operating at the scale of the Internet every day. One great innovation in the recent history of computers, however, is the idea that it is possible to build a reliable system on top of many individually unreliable components.

Each of Cloudflare's PoPs is designed to function without communication or support from others, or from a central data center. That alone greatly increases our tolerance for network partitions and moves us from maintaining a single system to be closer to maintaining 180 independent clouds, any of which can serve all traffic.

We are also a system built on anycast which allows us to tap into the fundamental reliability of the Internet. The Internet uses a protocol called BGP which asks each system who would like to receive traffic for a particular IP address to 'advertise' it. Each router then will decide to forward traffic based on which person advertising an address is the closest. We advertise all of our IP addresses in every one of our data centers. If a data centers goes down, it stops advertising BGP routes, and the very same packets which would have been destined for it arrive in another PoP seamlessly.

Ultimately we are trying to help build a better Internet. We don't believe that Internet is built on the back of a single provider. Many of the services provided by these cloud providers are simply too complex to be as reliable as the Internet demands.

True reliability and cost control both require existing on multiple clouds. It is clear that the tools which the Internet of the 80s and 90s gave us may be insufficient to move into that future. With a smarter network we can do more, better.

Power is the precursor to all modern technology. James Watt’s steam engine energized the factory, Edison and Tesla’s inventions powered street lamps, and now both fossil fuels and renewable resources power the trillions of transistors in computers and phones. In the words of anthropologist Leslie White: “Other things being equal, the degree of cultural development varies directly as the amount of energy per capita per year harnessed and put to work.”

Unfortunately, most of the traditional ways to generate power are simply not sustainable. Burning coal or natural gas releases carbon dioxide which directly leads to global warming, and threatens the habitats of global ecosystems, and by extension humans. If we can’t minimize the impact, our world will be dangerously destabilized -- mass extinctions will grow more likely, and mass famines, draughts, migration, and conflict will only be possible to triage rather than avoid.

Is the Internet the primary source of this grave threat? No: all data centers globally accounted for 2-3% of total global power use in recent years, and power consumption isn’t the only contributor to human carbon emissions. Transportation (mostly oil use in cars, trucks, ships, trains, and airplanes) and industrial processing (steel, chemicals, heavy manufacturing, etc.) also account for similar volumes of carbon emissions. Within power use though, some internet industry analysts estimate that total data center energy (in kilowatt-hours, not percentage of global power consumption) may double every four years for the foreseeable future -- making internet energy use more than just rearranging deck chairs...

How does internet infrastructure like Cloudflare’s contribute to power consumption? Computing power resources are split into end users (like your phone or computer displaying this page) and network infrastructure. That infrastructure likewise splits into “network services” like content delivery and “compute services” like database queries. Cloudflare offers both types of services, and has a sustainability impact in both -- this post describes how we think about it.

The Cloudflare Network has one huge advantage when power is considered. We run a homogeneous network of nearly identical machines around the world, all running the same code on similar hardware. The same servers respond to CDN requests, block massive DDoS attacks, execute customer code in the form of Workers, and even serve DNS requests to 1.1.1.1. When it is necessary to bring more capacity to a problem we are able to do it by adjusting our traffic’s routing through the Internet, not by requiring wasteful levels of capacity overhead in 175 locations around the world. Those factors combine to dramatically reduce the amount of waste, as they mean we don’t have large amounts of hardware sitting idle consuming energy without doing meaningful work. According to one study servers within one public cloud average 4.15% to 16.6% CPU utilization, while Cloudflare’s edge operates significantly higher than that top-end.

One of the functions Cloudflare performs for our customers is caching, where we remember previous responses our customers have given to requests. This allows the edge location closest to the visitor to respond to requests instantly, saving the request a trip through the Internet to the customer’s origin server. This immediately saves energy, as sending data through the Internet requires switches and routers to make decisions which consumes power.

Serving a response from cache is as close to the lowest power requirement you can imagine to serve a web request; we are reading data from memory or disk and immediately returning it. In contrast, when a customer’s origin has to serve a request, there are two additional costs Cloudflare avoids: first, even getting the request to arrive at the origin often requires many hops over the Internet, each requiring CPU cycles and the energy they consume. Second, the request often requires large amounts of code to be executed and even database queries to be run. The savings are so great that we often have customers enable our caching to keep their servers running even when their request volume would overwhelm their capacity; if our caching were disabled they would almost immediately fail. This means we are not only saving CPU cycles on our customer’s origin, we are preventing them from having to buy and run multiple-X more servers with the proportionally greater energy use & environmental impact that entails.

Our breadth on the Internet also means the performance optimizations we are able to perform have a disproportionate impact. When we speed up TLS or fix CPU stalls we are shaving off milliseconds of CPU from requests traveling to 13 million different websites. It would be virtually impossible to get all of these performance improvements integrated into every one of those origins, but with Cloudflare they simply see fewer requests and energy is saved.

The energy efficiency of using a wax or tallow candle to create light is on the order of 0.01%. A modern power plant burning gas to power an LED light bulb is nearly 10% efficient, an improvement of 1,000x. One of the most powerful things we can do to lower energy consumption, therefore, is to give people ways of performing the same work with less energy.

Our connection to this concept lives not just in our network, but in the serverless computing platform we offer atop it, Cloudflare Workers. Many of the conventions that govern how modern servers and services operate descend directly from the mainframe era of computing, where a single large machine would run a single job. Unlike other platforms which are based on that legacy, we don’t sell customers servers, virtual machines, or containers; instead we use a technology called isolates. Isolates represent a lightweight way to run a piece of code which provides much of the same security guarantees with less overhead, allowing many thousands of different customer’s code to be executed on a small number of machines efficiently. A traditional computer system might be just as efficient running a single program, but as our world shifts into serverless computing with thousands of code files running on a single machine, isolates shine.

In a conventional computer system the complex security dance between the operating system and the code being executed by a user can consume as much as 30% of the CPU power used. This has only gotten worse with the recent patches required to prevent speculative execution vulnerabilities. Isolates share a single runtime which can manage the security isolation required to run many thousands of customer scripts, without falling back to the operating system. We are able to simply eliminate much of that 30% overhead, using that capacity to execute useful code instead.

Additionally, by being able to start our isolates using just a few milliseconds of CPU time rather than the hundreds required by conventional processes we are able to dynamically scale rapidly, more efficiently using the hardware we do have. Isolates allow us to spend CPU cycles on only the code customers actually wish to execute, not wasteful overhead. These effects are in fact so dramatic that we have begun to rebuild parts of our own internal infrastructure as isolate-powered Cloudflare Workers in part to save energy for ourselves and our customers.

All that means that the energy we ultimately do use for our operations is only a fraction of what it would otherwise take to accomplish the same tasks.

Last year, we took our first major step toward neutralizing the remaining carbon footprint from our operations by purchasing Renewable Energy Certificates (RECs) to match all of our electricity use in North America. This year, we have expanded our scope to include all of our operations around the world.

We currently have 175 data centers in more than 75 countries around the world, as well as 11 offices in San Francisco (our global HQ) London, Singapore, New York, Austin, San Jose, Champaign, Washington, D.C. Beijing, Sydney, and Munich. In order to reduce our carbon footprint, we have purchased RECs to match 100% of the power used in all those data centers and offices around the world as well.

As our colleague Jess Bailey wrote about last year, one REC is created for every Megawatt-hour (MWh) of electricity generated from a renewable power source, like a wind turbine or solar panel. Renewable energy is dispersed into electricity transmission systems similar to how water flows in water distribution systems — each of them is mixed inextricably in its respective “pipes” and it’s not possible to track where any particular electron you use, or drop of water you drink, originally came from. RECs are a way to track the volume (and source) of renewable energy contributed to the grid, and act like a receipt for each MWh contributed.

As we noted last year, this action is an important part of our sustainability plan, joining our efforts to work with data centers that have superior Power Usage Effectiveness (PUE), and adding to the waste diversion and energy efficiency efforts we already employ in all of our offices.

When combined with our ability to dramatically reduce the amount of data which has to flow through the Internet and the number of requests which have to reach our customer’s origins we hope to not just be considered neutral, but to have a large-scale and long-term positive effect on the sustainability of the Internet itself.

The promise of the AMP (Accelerated Mobile Pages) project was that it would make the web, and, in particular, the mobile web, much more pleasant to surf. The AMP HTML framework was designed to make web pages load quickly, and not distract the user with extraneous content that took them away from focusing on the web page’s content.

It was particularly aimed at publishers (such as news organizations) that wanted to provide the best, fastest web experience for readers catching up on news stories and in depth articles while on the move. It later became valuable for any site which values their mobile performance including e-commerce stores, job boards, and media sites.

As well as the AMP HTML framework, AMP also made use of caches that store copies of AMP content close to end users so that they load as quickly as possible. Although this cache make loading web pages much, much faster they introduce a problem: An AMP page served from Google’s cache has a URL starting with https://google.com/amp/. This can be incredibly confusing for end users.

Users have become used to looking at the navigation bar in a web browser to see what web site they are visiting. The AMP cache breaks that experience. For example, here’s a news story from the BBC website viewed through Google’s AMP cache:

Notice how the browser says the page is from google.com. That’s because of the AMP cache. This made the page load very quickly, but can be confusing. To help “fix” that problem Google shows that actual site at the top of the AMP page. There you can see that it was bbc.co.uk. Clicking on bbc.co.uk brings you to the same page served by the BBC’s web servers with bbc.co.uk in the web browser’s navigation bar:

But the problems with the AMP cache approach are deeper than just some confusion on the part of the user. By serving the page from Google’s cache there’s no way for the reader to check the authenticity of the page; when it’s served directly from, say, the BBC the user has the assurance of the domain name, a green lock indicating that the SSL certificate is valid and can even click on the lock to get details of the certificate.

Last November we announced a technical solution to these problems that would allow AMP pages to be served from a cache while retaining the original page URL and all its benefits. The in depth technical blog post by Gabbi Fisher and Avery Harnish gives the full details. The solution makes use of Web Packaging (which incorporates some clever use of cryptography) to allow the cache (run by Google, Cloudflare or others) to keep a copy of an AMP page and serve it quickly to the end user, but to also contain cryptographic proof of where the page originally came from.

In cooperation with a browser that understands Web Packaging this means that a page can be stored in an AMP cache and served quickly from it while showing the original site URL in the browser’s navigation bar. A major win all round!

We’re calling this “AMP Real URL” and it’s free to all of our customers starting today.

Google’s AMP Crawler downloads the content of your website and stores it in the AMP Cache many times a day. If your site has AMP Real URL enabled Cloudflare will digitally sign the content we provide to that crawler, cryptographically proving it was generated by you. That signature is all a modern browser (currently just Chrome on Android) needs to show the correct URL in the address bar when a visitor arrives to your AMP content from Google’s search results.

Gone is the hated grey bar, all your visitors see is your proper URL:

Importantly your site is still being served from Google’s AMP cache just as before; all of this comes without any cost to your SEO or web performance.

Since our original announcement we’ve had a chance to engage with dozens of members of the publishing and e-commerce community and would like to share what we’ve learned.

The Google-initiated AMP Project drives a huge percentage of mobile traffic and has greatly improved the experience of browsing the Internet on a phone. Many of the sites we have spoken to get as much as 50% of their web traffic through AMP, and the speed benefit it provides directly translates to better conversion rates.

AMP Real URL provides some serious benefits to sites which use AMP:

• Brand Protection: Web users have been trained that the URL in the address bar has significance. Having google.com at the top of a page of content hurts the publisher’s ability to maintain a unique presence on the Internet.

• Easier Analytics: AMP Real URL greatly simplifies web analytics for its users by allowing all visitors, AMP or otherwise, to coexist on the same tracking domain.

• Increased Screen Space: Historically when AMP was used room would be taken for a “grey bar” at the top of your site to show the real URL. With AMP Real URL that’s simply not necessary.

• Reduced Bounce Rate: We believe website visitors are less likely to bounce back to Google or another site when the publisher’s actual domain is in the address bar, but we will gather more data about this as AMP Real URL is rolled out.

• Content Signing: By relying on cryptographic techniques, AMP Real URL ensures that the content delivered to visitors has not been manipulated protecting the sites and brands it is used on. It’s now not possible for any external party to add, remove, or modify the content of a site.

We also spoke to Internet users of AMP, and there certainly are frustrations. There are some users who struggle with its complexity, or sites simply fail to load for them. Others are annoyed and confused with the “grey bar” at the top of the page and the gymnastics it requires to get a page’s original URL. Finally, there are folks who would like to ensure that Google is not modifying the content of pages as they travel through the AMP cache.

AMP Real URL happily fixes all of these issues. It ensures that sites are cryptographically signed which protects them from being modified by Google or anyone else, even when physically delivered from a domain you do not control. If the site is changed in any way the browser ensures the site’s real URL will no longer appear. It also greatly simplifies AMP, fixing many of the reliability issues people experience: AMP Real URL-powered links aren’t opened using the complex iframe mechanics used by AMP traditionally, instead they are loaded as any other website (Google uses rel=”prefetch” to get much of the same performance benefit). Finally, the “grey bar” is unnecessary, as the correct URL is right in the address bar at the top of the page, and copying the URL of a site to save or share works just as it does for non-AMP websites.

We are also taking this opportunity to sunset the other AMP products and experiments we have built over the years like Ampersand and Firebolt. Those products were innovative but we have learned that publishers value AMP products which pair well with Google’s search results, not which live outside it. Users of those older products were informed several weeks ago that they will be gradually shut down to focus our attention on AMP Real URL.

Google is rolling out support for AMP Real URL (referred to as Signed Exchanges outside Cloudflare) today, beginning with the primary Google search results. Over time, the hope is they will expand it to other areas of the search results page including the “Top Stories” news area at the top of the page. This makes AMP Real URL most valuable today for sites which get most of their AMP traffic from the primary search results like e-commerce, job boards, and ad-supported sites. News publishers can and should enable AMP Real URL, but the benefit they experience now will be from search results which live outside the “Top Stories” box. AMP Real URL is only supported in the Chrome browser at this time, but we are optimistic it will be supported more widely as its benefit to Internet users becomes clear.

After speaking with publishers and with Internet users, we have decided not to charge for AMP Real URL. This is not because our customers haven’t been excited or willing to pay for it, AMP makes up a huge component of many site’s traffic. Our motivation is the same as for offering CDN or SSL services to millions of customers free of charge, we are here to help build a better Internet and improving AMP is a huge step in that direction. We believe AMP Real URL is a technology which will fundamentally change the mobile web for the better and should be adopted by every AMP-supporting site. We do have another motive: we are hoping that this will motivate potential customers who value AMP to choose Cloudflare.

Beginning today you will find a new section on the Speed tab of your Cloudflare dashboard:

We will be rolling out support for AMP Real URL in stages over the next few weeks. Enable it now and we will notify you when it is activated on your domain. If you are an interested enterprise customer please reach out so we can expedite your activation.

We'll leave you with some perspectives from the early users of AMP Real URL:

"The performance benefits of AMP deliver value to our business and we are excited to see how AMP Real URL is able to take that even further"

— Solomon Moskalenko, Director of Interactive, US Xpress Trucking, The Johnson Group

"AMP is a crucial part of helping our business to grow and reach consumers everywhere. With AMP Real URL, we now have more control over our brand and can run analytics on our business site."

— Sumantro Da, Sr Director, Product Innovations & Growth Brands GM, 1-800-FLOWERS.COM

“AMP has played a key role in helping us to more effectively reach our audience and develop our online community, we’re keen to use AMP Real URL to better manage our online presence and keep our users engaged on the site.”

— Andrew Warner, CTO of Genius

The talk below was originally presented and recorded at Serverless Computing London in November 2018. If you’d like to join us in person to talk about serverless, we’ll be announcing 2019 event locations throughout the year on the docs page.

Cloudflare's Zack Bloom spoke at Serverless Computing London Conference

Many of the technical challenges of serverless (cold-start time, memory overhead, and CPU context switching) are solved by a new architecture which translates technology developed for web browsers onto the server. Learn about how serverless platforms built using isolates are helping to expand the kinds of applications built using serverless.

Zack Bloom helps build the future of the Internet as the Director of Product for Product Strategy at Cloudflare. He was a co-founder of Eager, an app platform for non-technical website owners, which was acquired by Cloudflare in 2016. He is also the creator of open-source projects which total over fifty-thousand stars on Github.

Check out our Workers recipes we have listed on our docs here »

Two years ago we had a problem. We were limited in how many features and options we could build in-house, we needed a way for customers to be able to build for themselves. We set out to find a way to let people write code on our servers deployed around the world (we had a little over a hundred data centers then, 155 as of this writing). Our system needed to run untrusted code securely, with low overhead. We sit in front of ten million sites and process millions and millions of requests per second, it also had to run very very quickly.

The Lua we had used previously didn’t run in a sandbox; customers couldn’t write their own code without our supervision. Traditional virtualization and container technologies like Kubernetes would have been exceptionally expensive for everyone involved. Running thousands of Kubernetes pods in a single location would be resource intensive, doing it in 155 locations would be even worse. Scaling them would be easier than without a management system, but far from trivial.

What we ended up settling on was a technology built by the Google Chrome team to power the Javascript engine in that browser, V8: Isolates.

Isolates are lightweight contexts that group variables with the code allowed to mutate them. Most importantly, a single process can run hundreds or thousands of Isolates, seamlessly switching between them. They make it possible to run untrusted code from many different customers within a single operating system process. They’re designed to start very quickly (several had to start in your web browser just for you to load this web page), and to not allow one Isolate to access the memory of another.

We pay the overhead of a Javascript runtime once, and then are able to run essentially limitless scripts with almost no individual overhead. Any given Isolate can start around a hundred times faster than I can get a Node process to start on my machine. Even more importantly, they consume an order of magnitude less memory than that process.

They have all the lovely function-as-a-service ergonomics of getting to just write code and not worry how it runs or scales. Simultaneously, they don’t use a virtual machine or a container, which means you are actually running closer to the metal than any other form of cloud computing I’m aware of. I believe it’s possible with this model to get close to the economics of running code on bare metal, but in an entirely Serverless environment.

This is not meant to be an ad for Workers, but I do want to show you a chart to reflect just how stark the difference is, to showcase why I believe this is not iterative improvement but an actual paradigm shift:

This data reflects actual requests (including network latency) made from a data center near where all the functions were deployed, performing a CPU intensive workload. Source

Not everyone fully understands how a traditional Serverless platform like Lambda works. It spins up a containerized process for your code. It isn’t running your code in any environment more lightweight than running Node on your own machines. What it does do is auto-scale those processes (somewhat clumsily). That auto-scaling creates cold-starts.

A cold-start is what happens when a new copy of your code has to be started on a machine. In the Lambda world this amounts to spinning up a new containerized process which can take between 500 milliseconds and 10 seconds. Any requests you get will be left hanging for as much as ten seconds, a terrible user experience. As a Lambda can only process a single request at a time, a new Lambda must be cold-started every time you get an additional concurrent request. This means that laggy request can happen over and over. If your Lambda doesn’t get a request soon enough, it will be shut down and it all starts again. Whenever you deploy new code it all happens again as every Lambda has to be redeployed. This has been correctly cited as a reason Serverless is not all it’s cracked up to be.

Because Workers don’t have to start a process, Isolates start in 5 milliseconds, a duration which is imperceptible. Isolates similarly scale and deploy just as quickly, entirely eliminating this issue with existing Serverless technologies.

One key feature of an operating system is it gives you the ability to run many processes at once. It transparently switches between the various processes which would like to run code at any given time. To accomplish that it does what’s called a ‘context switch’: moving all of the memory required for one process out, and the memory required for the next in.

That context switch can take as much as 100 microseconds. When multiplied by all the Node, Python or Go processes running on your average Lambda server this creates a heavy overhead which means not all of the CPUs power can actually be devoted to running the customer’s code; it’s spent switching between them.

An Isolate-based system runs all of the code in a single process and uses its own mechanisms to ensure safe memory access. This means there are no expensive context switches, the machine spends virtually all of its time running your code.

The Node or Python runtimes were meant to be run by individual people on their own servers. They were never intended to be run in a multi-tenant environment with thousands of other people’s code and strict memory requirements. A basic Node Lambda running no real code consumes 35 MB of memory. When you can share the runtime between all of the Isolates as we do, that drops to around 3 MB.

Memory is often the highest cost of running a customer’s code (even higher than the CPU), lowering it by an order of magnitude dramatically changes the economics.

Fundamentally V8 was designed to be multi-tenant. It was designed to run the code from the many tabs in your browser in isolated environments within a single process. Node and similar runtimes were not, and it shows in the multi-tenant systems which are built atop it.

Running multiple customers' code within the same process obviously requires paying careful attention to security. It wouldn’t have been productive or efficient for Cloudflare to build that isolation layer ourselves. It takes an astronomical amount of testing, fuzzing, penetration testing, and bounties required to build a truly secure system of that complexity.

The only reason this was possible at all is the open-source nature of V8, and its standing as perhaps the most well security tested piece of software on earth. We also have a few layers of security built on our end, including various protections against timing attacks, but V8 is the real wonder that makes this compute model possible.

This is not meant to be a referendum on AWS billing, but it’s worth a quick mention as the economics are interesting. Lambdas are billed based on how long they run for. That billing is rounded up to the nearest 100 milliseconds, meaning people are overpaying for an average of 50 milliseconds every execution. Worse, they bill you for the entire time the Lambda is running, even if it’s just waiting for an external request to complete. As external requests can take hundreds or thousands of ms, you can end up paying ridiculous amounts at scale.

Isolates have such a small memory footprint that we, at least, can afford to only bill you while your code is actually executing.

In our case, due to the lower overhead, Workers end up being around 3x cheaper per CPU-cycle. A Worker offering 50 milliseconds of CPU is \$0.50 per million requests, the equivalent Lambda is $1.84 per million. I believe lowering costs by 3x is a strong enough motivator that it alone will motivate companies to make the switch to Isolate-based providers.

Amazon has a product called Lambda@Edge which is deployed to their CDN data centers. Unfortunately, it’s three times more expensive than traditional Lambda, and it takes 30 minutes to deploy initially. It also doesn't allow arbitrary requests, limiting its usefulness to CDN-like-purposes.

Conversely, as I mentioned, with Isolates we are able to deploy every source file to 155 data centers at better economics than Amazon can do it to one. It might actually be cheaper to run 155 Isolates than a single container, or perhaps Amazon is charging what the market will bear and it’s much higher than their costs. I don’t know Amazon’s economics, I do know we’re very comfortable with ours.

Long-ago it became clear that to have a truly reliable system it must be deployed to more than one place on earth. A Lambda runs in a single availability zone, in a single region, in a single data center.

No technology is magical, every transition comes with disadvantages. An Isolate-based system can’t run arbitrary compiled code. Process-level isolation allows your Lambda to spin up any binary it might need. In an Isolate universe you have to either write your code in Javascript (we use a lot of TypeScript), or a language which targets WebAssembly like Go or Rust.

If you can’t recompile your processes, you can’t run them in an Isolate. This might mean Isolate-based Serverless is only for newer, more modern, applications in the immediate future. It also might mean legacy applications get only their most latency-sensitive components moved into an Isolate initially. The community may also find new and better ways to transpile existing applications into WebAssembly, rendering the issue moot.

I would love for you to try Workers and let us and the community know about your experience. There is still a lot for us to build, we could use your feedback.

We also need engineers and product managers who think this is interesting and want to take it in new directions. If you’re in San Francisco, Austin, or London, please reach out.

Interested in deploying a Cloudflare Worker without setting up a domain on Cloudflare? We’re making it easier to get started building serverless applications with custom subdomains on workers.dev. If you’re already a Cloudflare customer, you can add Workers to your existing website here.

Reserve a workers.dev subdomain

Your Workers now have access to a low-latency key-value data store which lives inside our network all around the world!

For those who don’t know, Cloudflare Workers is a new type of compute platform, built on top of our global network of 152+ data centers around the world. It allows you to write serverless code which runs in the fabric of the Internet itself, allowing you to engage with your users faster than other platforms can even get a packet to where your code is running. It’s built on a new architecture which eliminates cold starts and dramatically reduces the memory overhead of keeping your code running when compared to a platform like Amazon Lambda.

As powerful as this is, compute is just one component of what you need to build an application, you also need the ability to store data. We evaluated many of the available open source data stores on the market, but ultimately nothing was designed for a world with quite as many distributed nodes as our network. Instead, we have begun releasing our own vision for distributed storage, beginning today.

The Workers KV is a highly distributed, eventually-consistent, key value store. It will allow you to store up to a billion keys and values, and read them with ultra low latency anywhere in the world. It makes it possible to build entire applications with the performance traditionally associated with static content cached by a CDN.

First and foremost, you can build the same types of applications you build today, but in a more fault tolerant and performant way. Reading values from Workers KV is designed to have the same reliability as reading static files, making it much less likely to become unavailable than a traditional database. It’s designed to have the same performance as reading a file cached within our network, close to your users, giving it the speed of serving a static file as well.

That said, we do have some applications which we commonly see as a good entry point into writing code on the network itself. These use-cases combine a serious need for speed with a clean separation from the legacy components of your application which are stuck in a central location (for now!).

Here are just a few of those examples:

An API Gateway sits between your visitors and your API. It commonly handles tasks which would be redundant, time consuming, or slow to implement in each and every service in your system. This includes tasks like rate limiting, access token validation, and routing. They work together to deliver only authenticated requests directly to the appropriate components of your system. It’s also the perfect entry point to developing code which runs everywhere. When you use Cloudflare Workers as an API Gateway, your access tokens get validated at the Cloudflare data center closest to the customer before the request is securely forwarded to your origin.

In this example your authentication system will store a token when a user logs in. I'm using curl, but the backend code for your system is more likely to use whatever interface it has for making HTTPS requests. This request stores a blob of JSON identifying this token in a Worker KV namespace $NAMESPACE_ID with a key of $TOKEN_ID:

curl https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/storage/kv/namespaces/$NAMESPACE_ID/values/$TOKEN_ID \
-X PUT \
-H "X-Auth-Key: $CLOUDFLARE_AUTH_KEY" \
-H "X-Auth-Email: $CLOUDFLARE_AUTH_EMAIL" \
-d '{
  "userId": "bob",
  "expires": "2018-07-11T03:44:12Z"
}'

Your Worker code, which runs on every request, will check if the token the user provides matches one you have stored. A single line of code (TOKEN_STORE.get()) pulls the JSON stored above from the Worker KV

addEventListener('fetch', event => {
 event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
 const token = request.headers.get('Authorization')
 if (!token)
   return new Response("An Authorization header is required", {status: 401})

 const tokenInfo = await TOKEN_STORE.get(token, "json")
 if (!tokenInfo)
   return new Response("Invalid token", {status: 403})
 
 if (Date.parse(tokenInfo.expires) < Date.now())
   return new Response(“Token expired”, {status: 403}) 

 request = new Request(request)
 request.headers.set("User-Id", tokenInfo.userId)
 return fetch(request)
}

A secure authentication system which adds virtually no latency at 1/7th the cost of Amazon API Gateway (with so much more flexibility and power)!

Traditionally you’ve had to decide between showing a super fast static site to your visitors, or being able to include dynamic data customizing your site for each visitor. That customization could be showing different products based on the customer’s profile, A/B testing site variants, or even including the customer’s current shopping cart and account information. Rather than waiting for this information to be served from a central database, it’s now possible to store it close to every visitor, delivering a custom page as quickly as you could a static resource.

For example, let’s say we have translations of our site stored as JSON in the KV store. We can dynamically insert this translation data as Javascript into the HTML of our site. In this example our site has a block which looks like this:

<html>
 <head>
   <script>
     var TRANSLATION_DATA = TRANSLATION DATA HERE
   </script>
 </head>

 ...
</html>

Our worker replaces that text with the content of our translations. You could alternatively use a Javascript templating engine or parse the content of the page to do something like Server-Side Includes.

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const token = request.headers.get('Authorization')


  const translationsPromise = TRANSLATION_DATA.get(country)
  const response = await fetch(request)
  const translations = await translationsPromise

  let newBody = await response.text()
  const ct = response.headers.get('content-type')
  if (ct.startsWith('text/html') && response.status === 200){
    newBody = newBody.replace('TRANSLATION DATA HERE', translations)
  }

  return new Response(newBody, response)
}

Workers is a full programming environment, meaning this is just the beginning. We have customers rendering their entire React App inside Workers, fully bootstrapping the data required to render their site.

The Workers KV store creates a powerful way to configure your running Workers without having to redeploy them. You might want to implement feature flags to enable or disable features at will, or to dynamically update the data your code uses to make decisions. Workers deploy in under 30 seconds, but it’s common to have more data than can easily fit in Worker code. For example, we have customers interested in using Workers KV to block messages from lost or stolen IoT devices before they ever reach their origin:

addEventListener('fetch', event => {
 event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
 const deviceId = request.headers.get('device-id')
 const bannedEntry = await BANNED_DEVICES.get(deviceId)
 if (bannedEntry !== null){
   return new Response("This device has been banned", {status: 403})
 }

 return fetch(request)
}

One thing we’re excited about when we think about the Workers KV store is the potential for building beyond web applications. There are many situations where developers are looking for an easy way to execute code without worrying about provisioning or maintaining infrastructure. One of the most common cases we see is joining systems together.

For example, one of our customers is planning on using Workers KV to connect their Point of Sale system with their Delivery Service’s API:

// The endpoint we have bound this to is polled periodically
addEventListener('fetch', event => {
 event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
 const orderResp = await fetch(`https://api.pos-system.com/orders/active`, {
   headers: {
     'Authorization': 'POS_API_KEY'
   }
 })

 const orders = await orderResp.json()

 for (var i=0; i < orders.length; i++){
   let order = orders[i]

   const syncedData = await SYNCED_ORDERS.get(order.id, "json")

   // If the order data is newer than the last data we sent to the delivery company,
   // update it.
   if (syncedData.modifiedAt < order.modifiedAt) {
     await fetch(`https://api.delivery-system.com/orders/${ order.id }`, {
       method: 'POST',
       body: JSON.stringify(order),
       headers: {
         'Content-Type': 'application/json',
         'Authorization': 'DELIVERY_API_KEY'
       }
     })

     await SYNCED_ORDERS.put(order.id, JSON.stringify(order))
   }
 }

 return new Response("?")
}

Workers KV is launching today in a limited beta. As we get feedback and operational experience we will be relaxing our storage limits and granting access to more users. While we may have more restrictive limits during beta, you can design your applications around the following characteristics:

• Up to 1 billion keys per namespace

• Keys of up to 2 kB

• Values of up to 64 kB

• Eventually consistent, global consistency within 10 seconds

• 100k+ reads per second per key

• Up to one write per second per key

We worked hard to make the pricing of Workers KV easy to understand and affordable for virtually any use case. Your $5 monthly Workers compute minimum includes 1 GB of KV storage and up to 10 million KV reads. If you use less than the 10 million included Worker requests now, you can use KV without paying a single cent more.

Beyond the minimums, Workers KV is billed at $0.50 per GB-month of additional storage and $0.50 per million additional KV reads.

To get beta access sign up here, we can’t wait to see what you build!

Subscribe to the blog for daily updates on all of our announcements.

tl;dr Cloudflare Workers are seven times faster than a default Lambda function for workloads which push the CPU. Workers are six times faster than Lambda@Edge, tested globally.  

The PBKDF2 algorithm is designed to be slow to compute. It's used to hash passwords; its slowness makes it harder for a password cracker to do their job. Its extreme CPU usage also makes it a good benchmark for the CPU performance of a service like Lambda or Cloudflare Workers.

We've written a test based on the Node Crypto (Lambda) and the WebCrypto (Workers) APIs. Our Lambda is deployed to with the default 128MB of memory behind an API Gateway in us-east-1, our Worker is, as always, deployed around the world. I also have our function running in a Lambda@Edge deployment to compare that performance as well. Again, we're using Catchpoint to test from hundreds of locations around the world.

And again, Lambda is slow:

This chart shows what percentage of the requests made in the past 24 hours were faster than a given number of ms:

The 95th percentile speed of Workers is 242ms. Lambda@Edge is 842ms (2.4x slower), Lambda 1346ms (4.6x slower).

Lambdas are billed based on the amount of memory they allocate and the number of CPU ms they consume, rounded up to the nearest hundred ms. Running a function which consumes 800ms of compute will cost me $1.86 per million requests. Workers is $0.50/million flat. Obviously even beyond the cost, my users would have a pretty terrible experience waiting over a second for responses.

As I said, Workers run in almost 160 locations around the world while my Lambda is only running in Northern Virgina. This is something of an unfair comparison, so let's look at just tests in Washington DC:

Unsurprisingly the Lambda and Lambda@Edge performance evens out to show us exactly what speed Amazon throttles our CPU to. At the 50th percentile, Lambda processors appear to be 6-7x slower than the mid-range server cores in our data centers.  

What you might not realize is that the power of the CPU your Lambda gets is a function of how much memory you allocate for it. As you ramp up the memory you get more performance. To test this we allocated a function with 1024MB of memory:

Again this are only tests emanating from Washington DC, and I have filtered points over 500ms.

The performance of both the Lambda and Lambda@Edge tests with 1024MB of memory is now much closer, but remains roughly half that of Workers. Even worse, running my new Lambda through the somewhat byzantine pricing formula reveals that 100ms (the minimum) of my 1024MB Lambda will cost me the exact same $1.86 I was paying before. This makes Lambda 3.7x more expensive than Workers on a per-cycle basis.

Adding more memory than that only adds cost, not speed. Here is a 3008MB instance (the max I can allocate):

This Lambda would be costing me $5.09 per million requests, over 10x more than a Worker, but would still provide less CPU.

In general our supposition is a 128MB Lambda is buying you 1/8th of a CPU core. This scales up as you elect to purchase Lambda's with more memory, until it hits the performance of one of their cores (minus hypervisor overhead) at which point it levels off.

Ultimately Amazon is charging based on the duration of execution, but on top of a much slower execution platform than we expected.

One response we saw to our last post was that Lambda supports running native binaries and that's where its real performance will be exhibited.

It is of course true that our Javascript tests are likely (definitely in the case of Workers) just calling out to an underlying compiled cryptography implementation. But I don't know the details of how Lambda is implemented, perhaps the critics have a point that native binaries could be faster. So I decided to extend my tests.

After beseeching a couple of my colleagues who have more recent C++ experience than I do, I ended up with a Lambda which executes the BoringSSL implementation of PBKDF2 in plain C++14. The results are utterly boring:

A Lambda executing native code is just as slow as one executing Javascript.

As I mentioned, irrespective of the language, all of this cryptography is likely calling the same highly optimized C implementations. In that way it might not be an accurate reflection of the performance of our chosen programming languages (but is an accurate reflection of CPU performance). Many people still believe that code written in languages like Java is faster than Javascript in V8. I decided to disprove that as well, and I was willing to sacrifice to the point where I installed the JDK, and resigned myself to a lifetime of update notifications.

To test application-level performance I dusted off my interview chops and wrote a naïve prime factorization function in both Javascript and Java. I won't weigh in on the wisdom of using new vs old guard languages, but I will say it doesn't buy you much with Lambda:

This is charting two Lambdas, one with 128MB of memory, the other 1024MB. The tests are all from Washington, DC (near both of our Lambdas) to eliminate the advantage of our Worker's global presence. The 128MB instance has 1745ms of 95% percentile wait time.

When we look at tests which originate all around the globe, where billions of Internet users browse every day, we get the best example of the power of a Worker's deployment yet:

When we exclude the competitors from our analysis we are able to use the same techniques to examine the performance of the Workers platform itself and uncover more than a few opportunities for improvement and optimization. That's what we're focused on: making Workers even faster, more powerful, and easier to use.

As you might imagine given these results, the Workers team is growing. If you'd like to work on this or other audacious projects at Cloudflare, reach out.

.twitter-icon { width: 16px; height: 16px; line-height: 1.1; display: inline-block; background-image: url(data:image/svg+xml;utf8;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iaXNvLTg4NTktMSI/Pgo8IS0tIEdlbmVyYXRvcjogQWRvYmUgSWxsdXN0cmF0b3IgMTguMS4xLCBTVkcgRXhwb3J0IFBsdWctSW4gLiBTVkcgVmVyc2lvbjogNi4wMCBCdWlsZCAwKSAgLS0+CjxzdmcgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgdmVyc2lvbj0iMS4xIiBpZD0iQ2FwYV8xIiB4PSIwcHgiIHk9IjBweCIgdmlld0JveD0iMCAwIDYxMiA2MTIiIHN0eWxlPSJlbmFibGUtYmFja2dyb3VuZDpuZXcgMCAwIDYxMiA2MTI7IiB4bWw6c3BhY2U9InByZXNlcnZlIiB3aWR0aD0iMTZweCIgaGVpZ2h0PSIxNnB4Ij4KPGc+Cgk8Zz4KCQk8cGF0aCBkPSJNNjEyLDExNi4yNThjLTIyLjUyNSw5Ljk4MS00Ni42OTQsMTYuNzUtNzIuMDg4LDE5Ljc3MmMyNS45MjktMTUuNTI3LDQ1Ljc3Ny00MC4xNTUsNTUuMTg0LTY5LjQxMSAgICBjLTI0LjMyMiwxNC4zNzktNTEuMTY5LDI0LjgyLTc5Ljc3NSwzMC40OGMtMjIuOTA3LTI0LjQzNy01NS40OS0zOS42NTgtOTEuNjMtMzkuNjU4Yy02OS4zMzQsMC0xMjUuNTUxLDU2LjIxNy0xMjUuNTUxLDEyNS41MTMgICAgYzAsOS44MjgsMS4xMDksMTkuNDI3LDMuMjUxLDI4LjYwNkMxOTcuMDY1LDIwNi4zMiwxMDQuNTU2LDE1Ni4zMzcsNDIuNjQxLDgwLjM4NmMtMTAuODIzLDE4LjUxLTE2Ljk4LDQwLjA3OC0xNi45OCw2My4xMDEgICAgYzAsNDMuNTU5LDIyLjE4MSw4MS45OTMsNTUuODM1LDEwNC40NzljLTIwLjU3NS0wLjY4OC0zOS45MjYtNi4zNDgtNTYuODY3LTE1Ljc1NnYxLjU2OGMwLDYwLjgwNiw0My4yOTEsMTExLjU1NCwxMDAuNjkzLDEyMy4xMDQgICAgYy0xMC41MTcsMi44My0yMS42MDcsNC4zOTgtMzMuMDgsNC4zOThjLTguMTA3LDAtMTUuOTQ3LTAuODAzLTIzLjYzNC0yLjMzM2MxNS45ODUsNDkuOTA3LDYyLjMzNiw4Ni4xOTksMTE3LjI1Myw4Ny4xOTQgICAgYy00Mi45NDcsMzMuNjU0LTk3LjA5OSw1My42NTUtMTU1LjkxNiw1My42NTVjLTEwLjEzNCwwLTIwLjExNi0wLjYxMi0yOS45NDQtMS43MjFjNTUuNTY3LDM1LjY4MSwxMjEuNTM2LDU2LjQ4NSwxOTIuNDM4LDU2LjQ4NSAgICBjMjMwLjk0OCwwLDM1Ny4xODgtMTkxLjI5MSwzNTcuMTg4LTM1Ny4xODhsLTAuNDIxLTE2LjI1M0M1NzMuODcyLDE2My41MjYsNTk1LjIxMSwxNDEuNDIyLDYxMiwxMTYuMjU4eiIgZmlsbD0iIzAwNkRGMCIvPgoJPC9nPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+CjxnPgo8L2c+Cjwvc3ZnPgo=)}

If it is truly the fastest, and it is comparable in price, it should be how every team deploys all of their serverless infrastructure. So I set out to see just how fast Worker execution is and prove it.

tl;dr Workers is much faster than Lambda and Lambda@Edge:

This is a chart showing what percentage of requests to each service were faster than a given number of ms. It is based on thousands of tests from all around the world, evenly sampled over the past 12 hours. At the 95th percentile, Workers is 441% faster than a Lambda function, and 192% faster than Lambda@Edge.  

The functions being tested simply return the current time. All three scripts are available on Github. The testing is being done by a service called Catchpoint which has hundreds of testing locations around the world.

This is every test ran in the last hour, with results over 1500ms filtered out:

You can immediately see that Worker results are tightly clustered around the x-axis, while Lambda and Lambda@Edge show much worse performance.

To be fair, comparing my Lambda, which only runs in us-east-1 (Northern Virginia, USA), to a global service like Workers is a a little unfair. This effect becomes even more clear if I only show tests ran in Australia. Sydney is 9,735 miles (53 light-ms) from our instance in us-east-1. It becomes pretty clear how miserable the experience would be for visitors down south:

As we only run one instance of each test from Australia every five minutes that's not a conclusive amount of data though, so let's look at the percentile distribution for the past 24 hours:

The 50th percentile speed for Workers is 13ms, well faster than a packet could even get half way to Virginia. At the 95th percentile you're looking at 882ms for Lambda, 216ms for Lambda@Edge, and 40ms for Workers. A full 5% of your users will be waiting almost a second for the simplest possible Lambda response, making it impossible to build a responsive application.

As we said, Workers has quite the advantage being deployed everywhere, as compared to Lambda which lives in a single region. (Lambda@Edge has less of an excuse). We believe Worker performance should be great everywhere though, so lets look a little closer to our Lambda instance. First all the tests in North America:

There are, amazingly, visitors who will be waiting over two seconds for a Lambda response:

Most of that delay is DNS however (Route53?). Just showing the time spent waiting for a response (ignoring DNS and connection time) tells a similar story however (filtering points over 300ms):

It's true that Cloudflare has many more points of presence than Lambda@Edge, but not all of this is explained by geographic distribution. To prove it, lets look at the testing location closest to my Northern Virginia-based Lambda function, Washington, DC. Again, looking at the last 24 hours:

With no geographic explanation the 95th percentile of Workers is 126% faster than Lambda and 65% faster than Lambda@Edge. I find this incredible. Please feel free play with the chart yourself.

How is this possible? I have some guesses. Workers is built on V8 isolates, which are significantly faster to spin up (under 5ms) than a full NodeJS process and have a tenth the memory overhead. The effect of having to wait for new processes start is very obvious when you look at the difference in speed for the first request which hits a new Lambda@Edge function:

Workers has also been carefully architected to avoid moving memory and blocking when it could be avoided, complete with our own optimized implementations of the Javascript APIs. Finally, Workers runs on the same thousands of machines which serve Cloudflare traffic around the world, benefiting from over half a decade of experience pushing the limits of our hardware.

This post has been somewhat self-congratulatory, and I apologize for that. We certainly still have a lot to build and a lot we can still do to improve our performance. It was originally going to be about the power of running your functions distributed all around the world, instead of at a single region. What I'm left with though is the belief that Workers is faster, period.

Here is a full chart for the last hour. I've also exported a CSV of all the data for the past 12 hours for you to explore.

Please reproduce the tests I've done here and share in the comments here or on Hacker News. If I've missed anything, we want to hear about it.

I'll also be sharing a price comparison between the various systems soon. Please subscribe to our blog if you'd like to be notified.

Finally, please try Workers!

Additional reading on Cloudflare Workers

• Cryptocurrency API Gateway using Typescript+Workers

• Delivering a Serverless API in 10 minutes using Workers

• T-25 days until Chrome starts flagging HTTP sites as "Not Secure"

• Bootstrapping a Typescript Worker

• Using Webpack to bundle your Workers modules

• Building a serverless Slack bot using Cloudflare Workers

• DroneDeploy and Cloudflare Workers

• Test New Features and Iterate Quickly with Cloudflare Workers

On 10th of April 1814, almost one hundred thousand troops fought the battle of Toulouse in Southern France. The war had ended on April 6th. Messengers delivering news of Napoleon Is abdication and the end of the war wouldn’t reach Toulouse until April 12th.

The issue was not the lack of a rapid communication system in France, it just hadn’t expanded far enough yet. France had an elaborate semaphore system. Arranged all around the French countryside were buildings with mechanical flags which could be rotated to transmit specific characters to the next station in line. When the following station showed the same flag positions as this one, you knew the letter was acknowledged, and you could show the next character. This system allowed roughly one character to be transmitted per minute, with the start of a message moving down the line at almost 900 miles per hour. It wouldn’t expand to Toulouse until 1834 however, twenty years after the Napoleonic battle.

Cappy Telegraph System

It’s should be no secret that money motivates. Stock trading presents one of the most obvious uses of fast long-distance communication. If you can find out about a ship sinking or a higher than expected earnings call before other traders, you can buy or sell the right stocks and make a fortune.

In France, it was strictly forbidden to use the semaphore system for anything other than government business however. Being such a public method of communication, it wasn’t really possible for an enterprising investor to ‘slip in’ a message without discovery. The ‘Blanc brothers’ figured out one method however. They discovered they could bribe the operator to include one extra bit of information, the “Error - cancel last transmitted symbol” control character with a message. If an operative spotted that symbol, they knew it was time to buy.

Semaphore had several advantages over an electric telegraph. For one, there were no lines to cut, making it easier to defend during war. Ultimately though, its slow speed, need for stations every ten miles or so, and complete worthlessness at night and in bad weather made its time on this earth limited.

Ships crossing the Atlantic were never particularly fast. We American’s didn’t learn of the end of our own revolution at the Treaty of Versailles until October 22nd, almost two months after it had been signed. The news came from a ship “thirty-six days out of london”.

Anyone who could move faster could make money. At the end of the American Civil War, Jim Fisk chartered high speed ships to speed to London and short Confederate Bonds before the news could reach the British market. He made a fortune.

It wasn’t long before high speed clipper ships were making the trip with mail and news in twelve or thirteen days regularly. Even then though, there was fierce competition among newspapers to get the information first. New York newspapers like the Herald and the Tribune banded together to form the New York Associated Press (now known just as the Associated Press) to pay for a boat to meet these ships 50 miles off shore. The latest headlines were sent back to shore via pigeon or the growing telegraph system.

Most of the technology used by the morse code telegraph system was built to satisfy the demands of the finance industry.

The first financial indicator was a pointer which sat above the gold exchange in New York. In our era of complex technology, the pointer system has the refreshing quality of being very simple. An operator in the exchange had a button which turned a motor. The longer he held the button down, the further the motor spun, changing the indication. This system had no explicit source of ‘feedback’, beyond the operator watching the indicator and letting go of his button when it looked right.

Soon, other exchanges were clamoring for a similar indicator. Their motors were wired to those of the Gold Exchange. This did not form a particularly reliable system. Numerous boys had to run from site to site, resetting the indicators when they came out of sync from that at the Gold Exchange.

I am crushed for want of means. My stockings all want to see my mother, and my hat is hoary from age.

— Samuel Morse, in his diary

This same technology formed the basis for the original ticker tape machines. A printing telegraph from this era communicated using a system of pulses over a wire. Each pulse would move the print head one ‘step’ on a racheting wheel. Each step would align a different character with the paper to be printed on. A longer pulse over the wire would energize an electromagnet enough to stamp the paper into the print head. Missing a single pulse though would send the printer out of alignment creating a 19th century version of Mojibake.

It was Thomas Edison who invented the ‘automatic rewinder’, which allowed the machines to be synchronized remotely. The first system used a screw drive. If you moved the print head through three full revolutions without printing anything, you would reach the end of the screw and it would stop actually rotating at a known character, aligning the printers. Printing an actual character would reset the screw. A later system of Edisons used the polarity of the wire to reset the system. If you flipped the polarity on the wire, switching negative and positive, the head would continue to turn in response to pulses, but it would stop at a predefined character, allowing you to ‘reset’ any of the printers which may have come out of alignment. This was actually a big enough problem that there is an entire US Patent Classification devoted to ‘Union Devices’ (178/41).

It will therefore be understood from the above explanation that the impression of any given character upon the type-wheel may be produced upon the paper by an operator stations at a distant point, ... simply by transmitting the proper number of electrical impulses of short duration by means of a properly-constructed circuit-breaker, which will cause the type-wheel to revolve without sensibly affecting the impression device. When the desired character is brought opposite the impression-lever the duration of the final current is prolonged, and the electro-magnet becomes fully magnetized, and therefore an impression of the letter or character upon the paper is produced.

— Thomas A. Edison, Patent for the Printing Telegraph 1870

Ticker tape machines used their own vocabulary:

IBM 4S 651/4

Meant 400 shares of IBM had just been sold for $65.25 per share (stocks were priced using fractions, not decimal numbers).

Ticker tape machines delivered a continuous stream of quotes while the market was open. The great accumulation of used ticker tape led to the famous ‘Ticker Tape parades’, where thousands of pounds of the tape would be thrown from windows on Wall Street. Today we still have ticker tape parades, but not the tape itself, the paper is bought specifically to be thrown out the window.

What’s the best way to share the stock ticker tape with a room full of traders? The early solution was a chalkboard where relevant stock trades could be written and updated throughout the day. Men were also employed to read the ticker and remember the numbers, ready to recall the most recent prices when asked.

A better solution came from the Trans-Lux company in 1939 however. They devised a printer which would print on translucent paper. The numbers could then be projected onto a screen from the rear, creating the first large stock ticker everyone could read.

Trans-lux Projection Stock Ticker

This was improved through the creation of the Trans-Lux Jet. The Jet was a continuous tape composed of flippable cards. One side of each card was a bright color while the other was black. As each card passed by a row of electrically-controlled pneumatic jets, some were flipped, writing out a message which would pass along the display just as modern stock tickers do. The system would be controlled using a shift register which would read in the stock quotes and translate them into pneumatic pulses.

The key issue with a stock ticker is you have to be around when a trade of stock you care about is printed. If you miss it, you have to search back through hundreds of feet of tape to get the most recent price. If you couldn’t find the price, the next best option was a call to the trading floor in New York. What traders needed was a way of looking up the most recent quote for any given stock.

In 1960 Jack Scantlin released the Quotron, the first computerized solution. Each brokerage office would become host to a Quotron ‘main unit’, which was a reasonably sized ‘computer’ equipped with a magnetic tape write head and a separate magnetic tape read head. The tape would continually feed while the market was open, the write head keeping track of the current stock trades coming in over the stock ticker lines. When it was time to read a stock value, the tape would be unspooled between the two heads falling into a bucket. This would allow the read head to find the latest value of the stock even as the write head continued to store trades.

Quotron Keypad

Each desk would be equipped with a keypad / printer combination unit which allowed a trader to enter a stock symbol and have the latest quote print onto a slip of paper. A printer was used because electronic displays were too expensive. In the words of engineer Howard Beckwith:

We considered video displays, but the electronics to form characters was too expensive then. I also considered the “Charactron tube” developed by Convair in San Diego that I had used at another company . . . but this also was too expensive, so we looked at the possibility of developing our own printer. As I remember it, I had run across the paper we used in the printer through a project at Electronic Control Systems where I worked prior to joining Scantlin. The paper came in widths of about six inches, and had to be sliced . . . I know Jack Scantlin and I spent hours in the classified and other directories and on the phone finding plastic for the tape tank, motors to drive the tape, pushbuttons, someone to make the desk unit case, and some company that would slice the tape. After we proved the paper could be exposed with a Xenon flash tube, we set out to devise a way to project the image of characters chosen by binary numbers stored in the shift register. The next Monday morning Jack came in with the idea of the print wheel, which worked beautifully.

The main ‘computer’ in each office was primitive by our standards. For one, it didn’t include a microprocessor. It was a hardwired combination of a shift register and some comparison and control logic. The desk units were connected with a 52-wire cable, giving each button on each unit its own wire. This was necessary because they units contained no logic themselves, their printing and button handling logic is all handled in the main computer.

When a broker in the office selected an exchange, entered a stock symbol, and requested a last price on his desk unit, the symbol would be stored in relays in the main unit, and the playback sprocket would begin driving the tape backwards over a read head at about ten feet per second, dumping the tape into the bin between the two heads (market data would continue to be recorded during the read operation). The tape data from the tracks for the selected exchange would be read into a shift register, and when the desired stock symbol was recognized, the register contents would be “frozen,” and the symbol and price would be shifted out and printed on the desk unit.

Only a single broker could use the system at a time:

If no desk units were in use, the main unit supplied power to all desk units in the office, and the exchange buttons on each unit were lit. When any broker pressed a lit button, the main unit disconnected the other desk units, and waited for the request from the selected desk unit. The desk unit buttons were directly connected to the main unit via the cable, and the main unit contained the logic to decode the request. It would then search the tape, as described above, and when it had an answer ready, would start the desk unit paper drive motor, count clock pulses from the desk unit (starting, for each character, when it detected an extra-large, beginning-of-wheel gap between pulses), and transmit a signal to operate the desk unit flash tube at the right time to print each character.

The Quotron system provided a vast improvement over a chalk board, but it was far from perfect. For one, it was limited to the information available over the ticker tape lines, which didn’t include information like the stocks volume, earnings, and dividends. A challenger named Ultronics created a system which used a similar hardwired digital computer, but with a drum memory rather than a magnetic tape.

Drum Memory

The logic was advanced enough to allow the system to calculate volume, high and low for each stock as the data was stored. Rather than store one of these expensive memory units in every brokerage, Ultronics had centralized locations around the US which were connected to brokerages and each other using 1000 bps Dataphone lines.

This system notably used a form of packet addressing, possibly for the first time ever. When each stock quote was returned it included the address of the terminal which had made the request. That terminal was able to identify the responses meant for it based on that address, allowing all the terminals to be connected to the same line.

At one time during system checkout we had a very elusive problem which we couldn’t pin down. In looking over the programs, we realized that the symptoms we were seeing could occur if an unconditional jump instruction failed to jump. We therefore asked CDC whether they had any indication that that instruction occasionally misbehaved. The reply was, “Oh, no. That’s one of the more reliable instructions,” This was our first indication that commands could be ordered by reliability.

— Montgomery Phister, Jr. 1989

Facing competition from the Ultronics quote computers, it was time for Jack Scantlin’s team to create something even more powerful. What they created was the Quotron II. The Quotron II was powered by magnetic core memory, an early form of random-access memory which allowed them to read and store any stock’s value in any order. Unfortunately there wasn’t actually enough memory. They had 24K of memory to store 3000 securities.

One stock sold for over $1000; some securities traded in 32nds of a dollar; the prices to be stored included the previous day’s close, and the day’s open, high, low, and last, together with the total number of shares traded-the volume. Clearly we’d need 15 bits for each price (ten for the $1000, five for the 32nds), or 75 bits for the five prices alone. Then we’d need another 20 for a four-letter stock symbol, and at least another 12 for the volume. That added up to 107 bits (nine words per stock, or 27,000 words for 3000 stocks) in a format that didn’t fit conveniently into 12-bit words.

Their solution was to store most of the stocks in a compressed format. Each stocks previous closing price was stored in 11 bits, and store the other four values as six bit increments from that number. Any stocks priced over $256, stocks which used fractions smaller than eighths, and too large increments, were stored in a separate overflow memory area.

The Quotron II system was connected to several remote sites using eight Dataphone lines which provided a total bandwidth of 16 kbps.

The fundamental system worked by having one 160A computer read stock prices from a punch tape (using about 5000 feet of tape a day) into the common memory. A second 160A responded to quote requests over the Dataphone lines. The remote offices were connected to bankers office using teletype lines which could transmit up to 100 words-per-minute where a device would forward the messages to the requesting terminal.

It’s somewhat comforting to learn that hack solutions are nothing new:

Once the system was in operation, we had our share of troubles. One mysterious system failure had the effect of cutting off service to all customers in the St. Louis area. Investigation revealed that something was occasionally turning off some 160A memory bits which enabled service to that region. The problem was “solved” for a time by installing a patch which periodically reinstated those bits, just in case.

The system was also notable for introducing the +/- tick to represent if a stock had gone up or down since the last trade. It also added some helpful calculated quantities such as the average price change of all NYSE stocks.

The story of Quotron II showcases the value of preparing for things to go wrong even if you’re not exactly sure how they will, and graceful degradation:

Jack Scantlin was worried about this situation, and had installed a feature in the Quotron program which discarded these common-memory programs, thus making more room for exceptions, when the market went haywire. On the day President Kennedy was assassinated, Jack remembers sitting in his office in Los Angeles watching features disappear until brokers could get nothing but last prices.

Those of us who worked on Quotron II didn’t use today’s labels. Multiprogramming, multiprocessor, packet, timesharing-we didn’t think in those terms, and most of us had never even heard them. But we did believe we were breaking new ground; and, as I mentioned earlier, it was that conviction more than any other factor that made the work fascinating, and the time fly.

It’s valuable to remember that as easy as this system might be to create with modern technology, it was a tremendous challenge at the time. “Most of us lived Quotron 12 to 14 hour days, six and a half days a week; but the weeks flew by, and before we turned around twice, five years were gone...”

Anyone who has ever been involved with the demonstration of an on-line process knows what happens next. With everyone crowded around to watch, the previously infallible gear or program begins to fall apart with a spectacular display of recalcitrance. Well so it went. We set the stage, everyone held their breath, and then the first query we keyed in proceeded to pull down the whole software structure.

NASDAQ Terminal

Feeling pressure from the SEC to link all the nation’s securities markets, the National Association of Securities Dealers decided to build an ‘automated quotation service’ for their stocks. Unlike a stock ticker, which provides the price of the last trade of a stock, the purpose of the NASDAQ system was to allow traders to advertise the prices they would accept to other traders. This was extremely valuable, as before the creation of this system, it was left to each trader to strike a deal with their fellow stock brokers, a very different system than the roughly ‘single-price-for-all’ system we have today.

The NASDAQ system was powered by two Univac 1108 computers for redundancy. The central system in Connecticut was connected to regional centers in Atlanta, Chicago, New York and San Francisco where requests were aggregated and disseminated. As of December 1975 there was 20,000 miles of dedicated telephone lines connecting the regional centers to 642 brokerage offices.

Each NASDAQ terminal was composed of a CRT screen and dedicated keyboard. A request for a stock would return the currently available bid and ask price of each ‘market maker’ around the country. The market makers where the centers where stock purchases were aggregated and a price set. The trader could quickly see where the best price was available, and call the market maker to execute his trade. Similarly, the market makers could use the terminal units to update their quotations and transmit the latest values. This type of detailed ‘per-market-maker’ information is actually still a part of the NASDAQ system, but it’s only accessible to paying members.

One thing this system didn’t do is support trading via computer, without calling the money maker on the phone (the AQ in NASDAQ actually stands for ‘Automated Quotations’, no stock purchasing capability was originally intended). This became a problem on Black Monday in 1987 when the stock market lost almost a quarter of its value in a single day. During the collapse, many money makers couldn’t keep up with the selling demand, leaving many small investors facing big loses with no way to sell.

In response the NASDAQ created the Small Order Execution System which allowed small orders of a thousand shares or less to be traded automatically. The theory was these small trades didn’t require the man-to-man blustering and bargaining which was necessary for large-scale trading. Eventually this was phased out, in favor of the nearly all computerized trading based system we have today.

Today over three trillion dollars worth of stocks are traded every month on computerized stock exchanges. The stocks being traded represent over forty billion dollars worth of corporate value. With the right credentials and backing it’s possible to gain or lose billions of dollars in minutes.

These markets make it possible for both the average citizen and billion dollar funds to invest in thousands of companies. In turn, it allows those companies to raise the money they need to (hopefully) grow.

Our next post in this series is on the history of digital communication before the Internet came along. Subscribe to be notified of its release.

/* Hide period after author */ .post-header .meta a { border-right: 5px solid white; margin-right: -5px; position: relative; } /* Post */ body { background-color: white; } pre, code { font-size: inherit; line-height: inherit; } section.primary-content { line-height: 1.6; color: black; } blockquote { padding-bottom: 1.5em; padding-top: 1em; font-style: italic; font-size: 1.2rem; } blockquote.pull-quote-centered { font-size: 1.2em; text-align: center; max-width: 100%; margin-left: auto; margin-right: auto; } blockquote blockquote { margin-left: 1em; padding-left: 1em; border-left: 5px solid rgba(0, 0, 0, 0.2); padding-bottom: 0.5em; padding-top: 0.5em; margin-bottom: 0.5em; margin-top: 0.5em; } p.attribution { color: #666; font-size: 1em; padding-bottom: 1em; } a code.year { text-decoration: underline; } figure.standard { margin: 2em 0; }

QWERTYUIOP

— Text of the first email ever sent, 1971

The ARPANET (a precursor to the Internet) was created “to help maintain U.S. technological superiority and guard against unforeseen technological advances by potential adversaries,” in other words, to avert the next Sputnik. Its purpose was to allow scientists to share the products of their work and to make it more likely that the work of any one team could potentially be somewhat usable by others. One thing which was not considered particularly valuable was allowing these scientists to communicate using this network. People were already perfectly capable of communicating by phone, letter, and in-person meeting. The purpose of a computer was to do massive computation, to augment our memories and empower our minds.

Surely we didn’t need a computer, this behemoth of technology and innovation, just to talk to each other.

The computers which sent (and received) the first email.

The history of computing moves from massive data processing mainframes, to time sharing where many people share one computer, to the diverse collection of personal computing devices we have today. Messaging was first born in the time sharing era, when users wanted the ability to message other users of the same time shared computer.

Unix machines have a command called write which can be used to send messages to other currently logged-in users. For example, if I want to ask Mark out to lunch:

$ write mark
write: mark is logged in more than once; writing to ttys002

Hi, wanna grab lunch?

He will see:

Message from zack@Awesome-Mainframe.local on ttys003 at 10:36 ...
Hi, wanna grab lunch?

This is absolutely hilarious if your coworker happens to be using a graphical tool like vim which will not take kindly to random output on the screen.

When the mail was being developed, nobody thought at the beginning it was going to be the smash hit that it was. People liked it, they thought it was nice, but nobody imagined it was going to be the explosion of excitement and interest that it became. So it was a surprise to everybody, that it was a big hit.

— Frank Heart, director of the ARPANET infrastructure team

An early alternative to Unix called Tenex took this capability one step further. Tenex included the ability to send a message to another user by writing onto the end of a file which only they could read. This is conceptually very simple, you could implement it yourself by creating a file in everyones home directory which only they can read:

mkdir ~/messages
chmod 0442 ~/messages

Anyone who wants to send a message just has to append to the file:

echo "??\n" >> /Users/zack/messages

This is, of course, not a great system because anyone could delete your messages! I trust the Tenex implementation (called SNDMSG) was a bit more secure.

In 1971, the Tenex team had just gotten access to the ARPANET, the network of computers which was a main precursor to the Internet. The team quickly created a program called CPYNET which could be used to send files to remote computers, similar to FTP today.

One of these engineers, Ray Tomlinson, had the idea to combine the message files with CPYNET. He added a command which allowed you to append to a file. He also wired things up such that you could add an @ symbol and a remote machine name to your messages and the machine would automatically connect to that host and append to the right file. In other words, running:

SNDMSG zack@cloudflare

Would append to the /Users/zack/messages file on the host cloudflare. And email was born!

The CPYNET format did not have much of a life outside of Tenex unfortunately. It was necessary to create a standard method of communication which every system could understand. Fortunately, this was also the goal of another similar protocol, FTP. FTP (the File Transfer Protocol) sought to create a single way by which different machines could transfer files over the ARPANET.

FTP originally didn’t include support for email. Around the time it was updated to use TCP (rather than the NCP protocol which ARPANET historically used) the MAIL command was added.

$ ftp
< open bbn

> 220 HELLO, this is the BBN mail service

< MAIL zack

> 354 Type mail, ended by <CRLF>.<CRLF>

< Sup?
< .

> 250 Mail stored

These commands were ultimately borrowed from FTP and formed the basis for the SMTP (Simple Mail Transfer Protocol) protocol in 1982.

The format for defining how a message should be transmitted (and often how it would be stored on disk) was first standardized in 1977:

Date     :  27 Aug 1976 0932-PDT
From     :  Ken Davis <KDavis at Other-Host>
Subject  :  Re: The Syntax in the RFC
To       :  George Jones <Group at Host>,
              Al Neuman at Mad-Host

There’s no way this is ever going anywhere...

Note that at this time the ‘at’ word could be used rather than the ‘@’ symbol. Also note that this use of headers before the message predates HTTP by almost fifteen years. This format remains nearly identical today.

The Fifth Edition of Unix used a very similar format for storing a users email messages on disk. Each user would have a file which contained their messages:

From MAILER-DAEMON Fri Jul  8 12:08:34 1974
From: Author <author@example.com>
To: Recipient <recipient@example.com>
Subject: Save $100 on floppy disks

They’re never gonna go out of style!

From MAILER-DAEMON Fri Jul  8 12:08:34 1974
From: Author <author@example.com>
To: Recipient <recipient@example.com>
Subject: Seriously, buy AAPL

You’ve never heard of it, you’ve never heard of me, but when you see
that stock symbol appear.  Buy it.

- The Future

Each message began with the word ‘From’, meaning if a message happened to contain From at the beginning of a line it needed to be escaped lest the system think that’s the start of a new message:

From MAILER-DAEMON Fri Jul  8 12:08:34 2011
From: Author <author@example.com>
To: Recipient <recipient@example.com>
Subject: Sample message 1

This is the body.
>From (should be escaped).
There are 3 lines.

It was technically possible to interact with your email by simply editing your mailbox file, but it was much more common to use an email client. As you might expect there was a diversity of clients available, but a few are of historical note.

RD was an editor which was created by Lawrence Roberts who was actually the program manager for the ARPANET itself at the time. It was a set of macros on top of the Tenex text editor (TECO), which itself would later become Emacs.

RD was the first client to give us the ability to sort messages, save messages, and delete them. There was one key thing missing though: any integration between receiving a message and sending one. RD was strictly for consuming emails you had received, to reply to a message it was necessary to compose an entirely new message in SNDMSG or another tool.

That innovation came from MSG, which itself was an improvement on a client with the hilarious name BANANARD. MSG added the ability to reply to a message, in the words of Dave Crocker:

My subjective sense was that propagation of MSG resulted in an exponential explosion of email use, over roughly a 6-month period. The simplistic explanation is that people could now close the Shannon-Weaver communication loop with a single, simple command, rather than having to formulate each new message. In other words, email moved from the sending of independent messages into having a conversation.

Email wasn’t just allowing people to talk more easily, it was changing how they talk. In the words of C. R. Linklider and Albert Vezza in 1978:

One of the advantages of the message systems over letter mail was that, in an ARPANET message, one could write tersely and type imperfectly, even to an older person in a superior position and even to a person one did not know very well, and the recipient took no offense... Among the advantages of the network message services over the telephone were the fact that one could proceed immediately to the point without having to engage in small talk first, that the message services produced a preservable record, and that the sender and receiver did not have to be available at the same time.

The most popular client from this era was called MH and was composed of several command line utilities for doing various actions with and to your email.

$ mh

% show

(Message inbox:1)
Return-Path: joed
Received: by mysun.xyz.edu (5.54/ACS)
        id AA08581; Mon, 09 Jan 1995 16:56:39 EST
Message-Id: <9501092156.AA08581@mysun.xyz.edu>
To: angelac
Subject: Here’s the first message you asked for
Date: Mon, 09 Jan 1995 16:56:37 -0600
From: "Joe Doe" <joed>

Hi, Angela!  You asked me to send you a message.  Here it is.
I hope this is okay and that you can figure out how to use
that mail system.

Joe

You could reply to the message easily:

% repl

To: "Joe Doe" <joed>
cc: angelac
Subject: Re: Here’s the first message you asked for
In-reply-to: Your message of "Mon, 09 Jan 1995 16:56:37 -0600."
        <9501092156.AA08581@mysun.xyz.edu>
-------

% edit vi

You could then edit your reply in vim which is actually pretty cool.

Interestingly enough, in June of 1996 the guide “MH & xmh: Email for Users & Programmers” was actually the first book in history to be published on the Internet.

All mail clients suck. This one just sucks less.

— Mutt Slogan

It took several years until terminals became powerful enough, and perhaps email pervasive enough, that a more graphical program was required. In 1986 Elm was introduced, which allowed you to interact with your email more interactively.

Elm Mail Client

This was followed by more graphical TUI clients like Mutt and Pine.

In the words of the University of Washington’s Pine team:

Our goal was to provide a mailer that naive users could use without fear of making mistakes. We wanted to cater to users who were less interested in learning the mechanics of using electronic mail than in doing their jobs; users who perhaps had some computer anxiety. We felt the way to do this was to have a system that didn’t do surprising things and provided immediate feedback on each operation; a mailer that had a limited set of carefully-selected functions.

These clients were becoming gradually easier and easier to use by non-technical people, and it was becoming clear how big of a deal this really was:

We in the ARPA community (and no doubt many others outside it) have come to realize that we have in our hands something very big, and possibly very important. It is now plain to all of us that message service over computer networks has enormous potential for changing the way communication is done in all sectors of our society: military, civilian government, and private.

Its like when I did the referer field. I got nothing but grief for my choice of spelling. I am now attempting to get the spelling corrected in the OED since my spelling is used several billion times a minute more than theirs.

— Phillip Hallam-Baker on his spelling of ’Referer’ 2000

The first webmail client was created by Phillip Hallam-Baker at CERN in 1994. Its creation was early enough in the history of the web that it led to the identification of the need for the Content-Length header in POST requests.

Hotmail was released in 1996. The name was chosen because it included the letters HTML to emphasize it being ‘on the web’ (it was original stylized as ‘HoTMaiL’). When it was launched users were limited to 2MB of storage (at the time a 1.6GB hard drive was $399).

Hotmail was originally implemented using FreeBSD, but in a decision I’m sure every engineer regretted, it was moved to Windows 2000 after the service was bought by Microsoft. In 1999, hackers revealed a security flaw in Hotmail that permitted anybody to log in to any Hotmail account using the password ‘eh’. It took until 2001 for ‘hackers’ to realize you could access other people’s messages by swap usernames in the URL and guessing at a valid message number.

Gmail was famously created in 2004 as a ‘20% project’ of Paul Buchheit. Originally it wasn’t particularly believed in as a product within Google. They had to launch using a few hundred Pentium III computers no one else wanted, and it took three years before they had the resources to accept users without an invitation. It was notable both for being much closer to a desktop application (using AJAX) and for the unprecedented offer of 1GB of mail storage.

US Postal Mail Volume, KPCB

At this point email is a ubiquitous enough communication standard that it’s very possible postal mail as an everyday idea will die before I do. One thing which has not survived well is any attempt to replace email with a more complex messaging tool like Google Wave. With the rise of more targeted communication tools like Slack, Facebook, and Snapchat though, you never know.

There is, of course, a cost to that. The ancestors of the Internet were kind enough to give us a communication standard which is free, transparent, and standardized. It would be a shame to see the tech communication landscape move further and further into the world of locked gardens and proprietary schemas.

We’ll leave you with two quotes:

Mostly because it seemed like a neat idea. There was no directive to ‘go forth and invent e-mail’.

— Ray Tomlinson, answering a question about why he invented e-mail

Permit me to carry the doom-crying one step further. I am curious whether the increasingly easy access to computers by adolescents will have any effect, however small, on their social development. Keep in mind that the social skills necessary for interpersonal relationships are not taught; they are learned by experience. Adolescence is probably the most important time period for learning these skills. There are two directions for a cause-effect relationship. Either people lacking social skills (shy people, etc.) turn to other pasttimes, or people who do not devote enough time to human interactions have difficulty learning social skills. I do not [consider] whether either or both of these alternatives actually occur. I believe I am justified in asking whether computers will compete with human interactions as a way of spending time? Will they compete more effectively than other pasttimes? If so, and if we permit computers to become as ubiquitous as televisions, will computers have some effect (either positive or negative) on personal development of future generations?

— Gary Feldman, 1981

• Use Cloudflare Apps to build tools which can be installed by millions of sites.

  Build an app →

  If you're in San Francisco, London or Austin: work with us.

• Our next post is on the history of the URL!Get notified when new apps and apps-related posts are released:

  Email Address

(function($) {window.fnames = new Array(); window.ftypes = new Array();fnames[0]='EMAIL';ftypes[0]='email';fnames[1]='FNAME';ftypes[1]='text';fnames[2]='LNAME';ftypes[2]='text';}(jQuery));var $mcj = jQuery.noConflict(true); /* Social */ .social { margin-top: 1.3em; } .fb_iframe_widget { padding-right: 1px; } .IN-widget { padding-left: 11px; } /* Hide period after author */ .post-header .meta a { border-right: 5px solid white; margin-right: -5px; position: relative; } /* Post */ body { background-color: white; } pre, code { font-size: inherit; line-height: inherit; } section.primary-content { font-size: 16px; line-height: 1.6; color: black; } blockquote { padding-bottom: 1.5em; padding-top: 1em; font-style: italic; font-size: 1.25rem; } blockquote.pull-quote-centered { font-size: 1.2em; text-align: center; max-width: 100%; margin-left: auto; margin-right: auto; } blockquote blockquote { margin-left: 1em; padding-left: 1em; border-left: 5px solid rgba(0, 0, 0, 0.2); padding-bottom: 0.5em; padding-top: 0.5em; margin-bottom: 0.5em; margin-top: 0.5em; } figure.standard { position: relative; max-width: 100%; margin: 1em auto; text-align: center; z-index: -1; } .figcaption { padding-top: .5em; font-size: .8em; color: #888; font-weight: 300; letter-spacing: .03em; line-height: 1.35; } .figcontent { display: inline-block; } p.attribution { color: #666; font-size: 0.9em; padding-bottom: 1em; } a code.year { text-decoration: underline; } .closing-cards #mc_embed_signup .mc-field-group { margin: 0.75em 0; } .closing-cards #mc_embed_signup input { font-size: 1.5em; height: auto; } .closing-cards #mc_embed_signup input[type="email"] { border: 1px solid #bcbcbc; border-radius: 2px; margin-bottom: 0; } .closing-cards #mc_embed_signup input[type="submit"] { background: #f38020; color: #fff; padding: .8em 1em .8em 1em; white-space: nowrap; line-height: 1.2; text-align: center; border-radius: 2px; border: 0; display: inline-block; text-rendering: optimizeLegibility; -webkit-tap-highlight-color: transparent; -webkit-font-smoothing: subpixel-antialiased; user-select: none; -webkit-appearance: none; appearance: none; letter-spacing: .04em; text-indent: .04em; cursor: pointer; } .closing-cards #mc_embed_signup div.mce_inline_error { background-color: transparent; color: #C33; padding: 0; display: inline-block; font-size: 0.9em; } .closing-cards #mc_embed_signup p:not(:empty) { line-height: 1.5; margin-bottom: 2em; } .closing-cards #mc_embed_signup input[type="email"] { font-size: 20px !important; width: 100% !important; padding: .6em 1em !important; } .closing-cards #mc_embed_signup .mc-field-group { margin: 0 !important; } .closing-cards #mc_embed_signup input[type="submit"] { font-size: 20px !important; margin-top: .5em !important; padding: .6em 1em !important; } .closing-cards #mc_embed_signup div.mce_inline_error { padding: 0; margin: 0; color: #F38020 !important; } aside.section.learn-more { display: none; } .closing-cards { background: #eee; width: 100%; list-style-type: none; margin-left: 0; } .closing-card { width: calc(50% - 10px) !important; font-size: 20px; padding: 1.5em; display: inline-block; box-sizing: border-box; vertical-align: top; } @media (max-width: 788px){ .closing-card { width: 100% !important; } .closing-card + .closing-card { border-top: 10px solid white; } }

In fact, it has been a constant source of delight for me over the past year to get to continually tell hordes (literally) of people who want to – strap yourselves in, here it comes – control what their documents look like in ways that would be trivial in TeX, Microsoft Word, and every other common text processing environment: “Sorry, you’re screwed.”

— Marc Andreessen 1994

When Tim Berners-Lee announced HTML in 1991 there was no method of styling pages. How a given HTML tag was rendered was determined by the browser, often with significant input from the user’s preferences. To many, it seemed like a good idea to create a standard way for pages to ‘suggest’ how they might prefer to be rendered stylistically.

But CSS wouldn’t be introduced for five years, and wouldn’t be fully implemented for ten. This was a period of intense work and innovation which resulted in more than a few competing styling methods that just as easily could have become the standard.

While these languages are obviously not in common use today, we find it fascinating to think about the world that might have been. Even more surprisingly, it happens that many of these other options include features which developers would love to see appear in CSS even today.

In early 1993 the Mosaic browser had not yet reached 1.0. Those browsers that did exist dealt solely with HTML. There was no method of specifying the style of HTML whatsoever, meaning whatever the browser decided an <h1> should look like, that’s what you got.

In June of that year, Robert Raisch made a proposal to the www-talk mailing list to create a “an easily parsable format to deliver stylistic information along with Web documents” which would be called RRP.

@BODY fo(fa=he,si=18)

If you have no idea what this code is doing you are forgiven. This particular rule is setting the font family (fa) to helvetica (he), and the font size (si) to 18 points. It made sense to make the content of this new format as terse as was possible as it was born in the era before gzipping and when connection speeds hovered around 14.4k.

Some interesting things missing from this proposal were any mention of units, all numbers being interpreted based on their context (font sizes were always in points for example). This could be attributed to RRP being designed more as a “set of HINTS or SUGGESTIONS to the renderer” rather than a specification. This was considered necessary because the same stylesheet needed to function for both the common line-mode browsers (like Lynx, and the graphical browsers which were becoming increasingly popular.

Interestingly, RRP did include a method of specifying a columnar layout, a feature which wouldn’t make it to CSS until 2011. For example, three columns, each of width ‘80 units’ would look like this:

@P co(nu=3,wi=80)

It’s a little hard to parse, but not much worse than white-space: nowrap perhaps.

It’s worth noting that RRP did not support any of the ‘cascading’ we associate with stylesheets today. A given document could only have one active stylesheet at a time, which is a logical way to think about styling a document, even if it’s foreign to us today.

Marc Andreessen (the creator of Mosaic, which would become the most popular browser) was aware of the RRP proposal, but it was never implemented by Mosaic. Instead, Mosaic quickly moved (somewhat tragically) down the path of using HTML tags to define style, introducing tags like <FONT> and <CENTER>.

Then why don't you just implement one of the many style sheet proposals that are on the table. This would pretty much solve the problem if done correctly.

So then I get to tell people, "Well, you get to learn this language to write your document, and then you get to learn that language for actually making your document look like you want it to." Oh, they'll love that.

— Marc Andreessen 1994

Contrary to popular perception, Mosaic was not the first graphical browser. It was predated by ViolaWWW, a graphical browser originally written by Pei-Yuan Wei in just four days.

[IMAGE]

Pei-Yuan created a stylesheet language which supports a form of the nested structure we are used to in CSS today:

(BODY fontSize=normal
      BGColor=white
      FGColor=black
  (H1   fontSize=largest
        BGColor=red
        FGColor=white)
)

In this case we are applying color selections to the body and specifically styling H1s which appear within the body. Rather than using repeated selectors to handle the nesting, PWP used a parenthesis system which is evocative of the indentation systems used by languages like Stylus and SASS which are preferred by some developers to CSS today. This makes PWP’s syntax potentially better in at least one way than the CSS language which would eventually become the lingua franca of the web.

PWP is also notable for introducing the method of referring to external stylesheets we still use today:

<LINK REL="STYLE" HREF="URL_to_a_stylesheet">

ViolaWWW was unfortunately written to work chiefly with the X Windowing System which was only popular on Unix systems. When Mosaic was ported to Windows it quickly left Viola in the dust.

HTML is the kind of thing that can only be loved by a computer scientist. Yes, it expresses the underlying structure of a document, but documents are more than just structured text databases; they have visual impact. HTML totally eliminates any visual creativity that a document’s designer might have.

— Roy Smith 1993

The need for a language to express the style of documents long predates the Internet.

As you may know, HTML as we know it was originally based on a pre-Internet language called SGML. In 1987 the US Department of Defense decided to study if SGML could be used to make it easier to store and transmit the huge volume of documentation they deal with. Like any good government project, they wasted no time coming up with a name. The team was originally called the Computer-Aided Logistics Support team, then the Computer-aided Acquisition and Logistics Support team, then finally the Continuous Acquisition and Life-cycle Support initiative. In any case, the initials were CALS.

The CALS team created a language for styling SGML documents called FOSI which is an initialism which undoubtedly stands for some combination of four words. They published a specification for the language which is as comprehensive as it is incomprehensible. It also includes one of the best nonsensical infographics to ever exist on the web.

One inviolate rule of the Internet is: more will always get done if you can prove someone wrong in the process. In 1993, just four days after Pei-Yuan’s proposal, Steven Heaney proposed that rather than “re-inventing the wheel,” it was best to use a variant of FOSI to style the web.

A FOSI document is itself written in SGML, which is actually a somewhat logical move given web developers existing familiarity with the SGML variant HTML. An example document looks like this:

<outspec>
  <docdesc>
    <charlist>
      <font size="12pt" bckcol="white" fontcol="black">
    </charlist>
  </docdesc>
  <e-i-c gi="h1"><font size="24pt" bckcol="red", fontcol="white"></e-i-c>
  <e-i-c gi="h2"><font size="20pt" bckcol="red", fgcol="white"></e-i-c>
  <e-i-c gi="a"><font fgcol="red"></e-i-c>
  <e-i-c gi="cmd kbd screen listing example"><font style="monoser"></e-i-c>
</outspec>

If you’re a bit confused what a docdesc or charlist are, so were the members of www-talk. The only contextual information given was that e-i-c means ‘element in context’. FOSI is notable however for introducing the em unit which has now become the preferred method for people who know more about CSS than you to style things.

The language conflict which was playing out was actually as old as programming itself. It was the battle of functional ‘lisp-style’ syntax vs the syntax of more declarative languages. Pei-Yuan himself described his syntax as “LISP’ish,” but it was only a matter of time until a true LISP variant entered the stage.

For all its complexity, FOSI was actually perceived to be an interim solution to the problem of formatting documents. The long-term plan was to create a language based on the functional programming language Scheme which could enable the most powerful document transformations you could imagine. This language was called DSSSL.

In the words of contributor Jon Bosak:

It’s a mistake to put DSSSL into the same bag as scripting languages. Yes, DSSSL is Turing-complete; yes, it’s a programming language. But a script language (at least the way I use the term) is procedural; DSSSL very definitely is not. DSSSL is entirely functional and entirely side-effect-free.

Nothing ever happens in a DSSSL stylesheet. The stylesheet is one giant function whose value is an abstract, device-independent, nonprocedural description of the formatted document that gets fed as a specification (a declaration, if you will) of display areas to downstream rendering processes.

At its simplest, DSSSL is actually a pretty reasonable styling language:

(element H1
  (make paragraph
    font-size: 14pt
    font-weight: 'bold))

As it was a programming language, you could even define functions:

(define (create-heading heading-font-size)
  (make paragraph
    font-size: heading-font-size
    font-weight: 'bold))

(element h1 (create-heading 24pt))
(element h2 (create-heading 18pt))

And use mathematical constructs in your styling, for example to ‘stripe’ the rows of a table:

(element TR
  (if (= (modulo (child-number) 2)
        0)
    ...   ;even-row
    ...)) ;odd-row

As a final way of kindling your jealousy, DSSSL could treat inherited values as variables, and do math on them:

(element H1
  (make paragraph
    font-size: (+ 4pt (inherited-font-size))))

DSSSL did, unfortunately, have the fatal flaw which would plague all Scheme-like languages: too many parenthesis. Additionally, it was arguably too complete of a spec when it was finally published which made it intimidating to browser developers. The DSSSL spec included over 210 separate styleable properties.

The team did go on to create XSL, a language for document transformation which is no less confusing, but which would be decidedly more popular.

CSS does not include parent selectors (a method of styling a parent based on what children it contains). This fact has been long bemoaned by Stack Overflow posters, but it turns out there is a very good reason for its absence. Particularly in the early days of the Internet, it was considered critically important that the page be renderable before the document has been fully loaded. In other words, we want to be able to render the beginning of the HTML to the page before the HTML which will form the bottom of the page has been fully downloaded.

A parent selector would mean that styles would have to be updated as the HTML document loads. Languages like DSSSL were completely out, as they could perform operations on the document itself, which would not be entirely available when the rendering is to begin.

The first contributor to bring up this issue and propose a workable language was Bert Bos in March of 1995. His proposal also contains an early edition of the ‘smiley’ emoticon :-).

The language itself was somewhat ‘object-oriented’ in syntax:

*LI.prebreak: 0.5
*LI.postbreak: 0.5
*OL.LI.label: 1
*OL*OL.LI.label: A

Using . to signify direct children, and * to specify ancestors.

His language also has the cool property of defining how features like links work in the stylesheet itself:

*A.anchor: !HREF

In this case we specified that the destination of the link element is the value of its HREF attribute. The idea that the behavior of elements like links should be controllable was popular in several proposals. In the era pre-JavaScript, there was not an existing way of controlling such things, so it seemed logical to include it in these new proposals.

One functional proposal, introduced in 1994 by a gentleman with the name ‘C.M. Sperberg-McQueen’, includes the same behavior functionally:

(style a
  (block #f)     ; format as inline phrase
  (color blue)   ; in blue if you’ve got it
  (click (follow (attval 'href)))  ; and on click, follow url

His language also introduced the content keyword as a way of controlling the content of an HTML element from the stylesheet, a concept which was later introduced into CSS 2.1.

Before I talk about the language which actually became CSS, it’s worth mentioning one other language proposal if only because it is in some ways the thing of an early web developer’s dreams.

PSL96 was, in the naming convention of the time, the 1996 edition of the “Presentation Specification Language.” At its core, PSL looks like CSS:

H1 {
  fontSize: 20;
}

It quickly gets more interesting. You could express element position based on not just the sizes specified for them (Width), but the actual (Actual Width) sizes the browser rendered them as:

LI {
  VertPos: Top = LeftSib . Actual Bottom;
}

You’ll also notice you can use the element’s left sibling as a constraint.

You can also add logical expressions to your styles. For example to style only anchor elements which have hrefs:

A {
  if (getAttribute(self, "href") != "") then
    fgColor = "blue";
    underlineNumber = 1;
  endif
}

That styling could be extended to do all manner of things we resort to classes today to accomplish:

LI {
  if (ChildNum(Self) == round(NumChildren(Parent) / 2 + 1)) then
    VertPos: Top = Parent.Top;
    HorizPos: Left = LeftSib.Left + Self.Width;
  else
    VertPos: Top = LeftSib.Actual Bottom;
    HorizPos: Left = LeftSib.Left;
  endif
}

Support for functionality like this could have perhaps truly enabled the dream of separating content from style. Unfortunately this language was plagued by being a bit too extensible, meaning it would have been very possible for its implementation to vary considerably from browser to browser. Additionally, it was published in a series of papers in the academic world, rather than on the www-talk mailing list where most of the functional work was being done. It was never integrated into a mainstream browser.

The language which, at least in name, would directly lead to CSS was called CHSS (Cascading HTML Style Sheets), proposed in 1994 by Håkon W Lie.

Like most good ideas, the original proposal was pretty nutty.

h1.font.size = 24pt 100%
h2.font.size = 20pt 40%

Note the percentages at the end of rules. This percentage referred to how much ‘ownership’ the current stylesheet was taking over this value. If a previous stylesheet had defined the h2 font size as 30pt, with 60% ownership, and this stylesheet styled h2s as 20px 40%, the two values would be combined based on their ownership percentage to get some value around 26pt.

It is pretty clear how this proposal was made in the era of document-based HTML pages, as there is no way compromise-based design would work in our app-oriented world. Nevertheless, it did include the fundamental idea that stylesheets should cascade. In other words, it should be possible for multiple stylesheets to be applied to the same page.

This idea, in its original formulation, was generally considered important because it gave the end user control over what they saw. The original page would have one stylesheet, and the web user would have his or her own stylesheet, and the two would be combined to render the page. Supporting multiple stylesheets was viewed as a method of maintaining the personal-freedom of the web, not as a way of supporting developers (who were still coding individual HTML pages by hand).

The user would even be able to control how much control they gave to the suggestions of the page’s author, as expressed in an ASCII diagram in the proposal:

       User                   Author
Font   o-----x--------------o 64%
Color  o-x------------------o 90%
Margin o-------------x------o 37%
Volume o---------x----------o 50%

Like many of these proposals, it included features which would not make it into CSS for decades, if ever. For example, it was possible to write logical expressions based on the user’s environment:

AGE > 3d ? background.color = pale_yellow : background.color = white
DISPLAY_HEIGHT > 30cm ? http://NYT.com/style : http://LeMonde.fr/style

In a somewhat optimistic sci-fi vision of the future, it was believed your browser would know how relevant a given piece of content was to you, allowing it to show it to you at a larger size:

RELEVANCE > 80 ? h1.font.size *= 1.5

Microsoft is absolutely committed to open standards, especially on the Internet.

— John Ludeman 1994

Håkon Lie went on to simplify his proposal and, working with Bert Bos, published the first version of the CSS spec in December of 1996. Ultimately he would go on to write his doctoral thesis on the creation of CSS, a document which was heroically helpful to me in writing this.

Compared to many of the other proposals, one notable fact of CSS is its simplicity. It can be easily parsed, easily written, and easily read. As with many other examples over the history of the Internet, it was the technology which was easiest for a beginner to pick up which won, rather than those which were most powerful for an expert.

It is itself a reminder of how incidental much of this innovation can be. For example, support for contextual selectors (body ol li) was only added because Netscape already had a method for removing borders from images that were hyperlinks, and it seemed necessary to implement everything the popular browser could do. The functionality itself added a significant delay to the implementation of CSS, as at the time most browsers didn’t keep a ‘stack’ of tags as they parsed HTML. This meant the parsers had to be redesigned to support CSS fully.

Challenges like this (and the widespread use of non-standard HTML tags to define style) meant CSS was not usable until 1997, and was not fully supported by any single browser until March of 2000. As any developer can tell you, browser support wasn’t anywhere close to standards compliant until just a few years ago, more than fifteen years after CSS’ release.

If Netscape 4 ignored CSS rules applied to the <body> element and added random amounts of whitespace to every structural element on your page, and if IE4 got <body> right but bungled padding, what kind of CSS was safe to write? Some developers chose not to write CSS at all. Others wrote one style sheet to compensate for IE4’s flaws and a different style sheet to compensate for the blunders of Netscape 4.

— Jeffrey Zeldman

Internet Explorer 3 famously launched with (somewhat terrible) CSS support. To compete, it was decided that Netscape 4 should also have support for the language. Rather than doubling down on this third (considering HTML and JavaScript) language though, it was decided it should be implemented by converting the CSS into JavaScript, and executing it. Even better, it was decided this ‘JavaScript Style Sheet’ intermediary language should be accessible to web developers.

The syntax is straight JavaScript, with the addition of some styling-specific APIs:

tags.H1.color = "blue";
tags.p.fontSize = "14pt";
with (tags.H3) {
  color = "green";
}

classes.punk.all.color = "#00FF00"
ids.z098y.letterSpacing = "0.3em"

You could even define functions which would be evaluated every time the tag was encountered:

evaluate_style() {
  if (color == "red"){
    fontStyle = "italic";
  } else {
    fontWeight = "bold";
  }
}

tag.UL.apply = evaluate_style();

The idea that we should simplify the dividing line between styles and scripts is certainly reasonable, and is now even experiencing a resurgence of sorts in the React community.

JavaScript was itself a very new language at this time, but via some reverse engineering Internet Explorer had already added support for it into IE3 (as “JScript”). The bigger issue was the community had already rallied around CSS, and Netscape was, at this time, viewed as bullies by much of the standards community. When Netscape did submit JSSS to the standards committee, it fell on deaf ears. Three years later, Netscape 6 dropped support for JSSS and it died a (mostly) quiet death.

Thanks to some public shaming by the W3C, Internet Explorer 5.5 launched with nearly complete CSS1 support in the year 2000. Of course, as we now know, browser CSS implementations were heroically buggy and difficult to work with for at least another decade.

Today the situation has fortunately improved dramatically, allowing developers to finally realize the dream of writing code once and trusting it will function (almost) the same from browser to browser.

Our conclusion from all of this is the realization of just how arbitrary and contextual many of the decisions which govern our current tools were. If CSS was designed the way it is just to satisfy the constraints of 1996, then maybe that gives us permission 20 years later to do things a little differently.

Our goal was to create a way for every website owner to find and install all of the open-source and SaaS tools technical experts use everyday. As developers ourselves, we wanted to make it possible for a developer in her basement to build the next great tool and get it on a million websites (and make a million dollars) the next day. We didn’t want her to succeed because she had the biggest sales or marketing team, or the most name recognition, but because her tool is the best.

When we began talking with Cloudflare, it was about integrating with Eager, not about being acquired. But Cloudflare presented an opportunity to grow so much faster than we had hoped. In fact, we plan on releasing our platform to Cloudflare’s millions of customers as soon as April of next year. That is a scale which we could only dream of last year.

As an app platform, scale is uniquely important to us. With millions of users we can ensure app developers that what they build will get installed and purchased. We can support thousands of apps and generate millions of dollars of revenue for developers. We can hire the best developers in the world and grow our team. We can make the Internet an easier place to build great things.

This blog is perhaps not the best place to bestow Cloudflare with compliments, so I’ll leave it with this. After visiting Cloudflare’s offices in Austin and San Francisco, after meeting its co-founders, after attending the company retreat in Santa Cruz, we decided to join Cloudflare. We have already begun working on reimagining our app platform, turning it into Cloudflare Apps. Beginning in April of next year, any and every developer will be able to use our platform to get their code installed onto millions of websites.

If you want to build amazing tools for all website owners, even for the ones who don’t know how to “copy-paste and embed code”, we’re the platform for you. If you want to build a tool which can make money from small and medium sized businesses, rather than by selling to huge enterprises, we’re for you too.

We’ll be releasing more information about our app development platform in January. If you’d like to get notified when we do, let us know.