In 1864 British computer pioneer Charles Babbage described the first key-value store. It was meant to be part of his Analytical Engine. Sadly, the Analytical Engine, which would have been the first programmable computer, was never built. But Babbage lays out clearly the design for his key-value store in his autobiography. He imagined a read-only store implemented as punched cards. He referred to these as Tables:
I explained that the Tables to be used must, of course, be computed and punched on cards by the machine, in which case they would undoubtedly be correct. I then added that when the machine wanted a tabular number, say the logarithm of a given number, that it would ring a bell and then stop itself. On this, the attendant would look at a certain part of the machine, and find that it wanted the logarithm of a given number, say of 2303. The attendant would then go to the drawer containing the pasteboard cards representing its table of logarithms. From amongst these he would take the required logarithmic card, and place it in the machine.
Punched card illustration from Babbage’s autobiography showing an integer key (2303) and value representing the decimal part of log10(2303) (0.3622939)
Upon this the engine would first ascertain whether the assistant had or had not given him the correct logarithm of the number; if so, it would use it and continue its work. But if the engine found the attendant had given him a wrong logarithm, it would then ring a louder bell, and stop itself. On the attendant again examining the engine, he would observe the words, "Wrong tabular number," and then discover that he really had given the wrong logarithm, and of course he would have to replace it by the right one.
So, a key-value store (in this case mapping integers to their logarithm) implemented as external storage on punched cards with a human assistant as data bus. We’ve come a long way but key-value stores are as useful today as they were 150 years ago.
Today we’re announcing a native key-value store for Cloudflare Workers. We’re calling this Workers KV and this functionality is just the start of a sequence of announcements around storage and databases on the edge.
Values are written into Workers KV via the standard Cloudflare API and they are available within seconds at every one of Cloudflare’s 150+ global PoPs. Stephen and Zack’s technical post goes into more detail about how to use Workers KV. The values written are encrypted while at rest, in transit, and on local disk; they are only decrypted as needed.
Values can also be written from inside a Cloudflare Worker. Cloudflare takes care of synchronizing keys and values across our entire network.
So, what can you do with Workers KV?
Kenton Varda has taken the entire Cap’n Proto web site and implemented it using Workers and Workers KV. The entire site is served from a Worker that accesses static assets stored using Workers KV. That’s pretty neat… his site is entirely ‘serverless’ or rather ‘originless’
Others might not want to go quite so far but here are some use cases:
- Shopping cart storage: an e-commerce site can store a user’s entire shopping cart in a key/value pair using Workers KV. The e-commerce back end software can call the Cloudflare API to save (and retrieve, if necessary) the shopping cart contents. A Worker running on Cloudflare’s network can access the shopping cart from Workers KV and use it to show the visitor its contents and get them to checkout.
Storing the cart using Workers KV is more stable than trying to store it in the browser’s local storage (and can’t be cleared by the user) and completely offloads the storage from the e-commerce site’s engine. - A/B testing: a site can make a per-visitor decision about which profile to use for A/B testing and then store that profile information using Workers KV. This has significant speed advantages (since the A/B decision isn’t delayed until the page load in the browser).
- Authentication verification can be handled in a Worker. For example, a user can log into a web site and details of the login (including the associated authentication token) can be pushed into a Worker KV value and then checked within a Worker. This offloads checking tokens from the backend server meaning that unauthenticated requests can be rejected quickly and authenticated requests passed through.
In fact, the entire authentication flow can be done in a Worker. A Worker can check credentials against an authentication service via an async API call, and then update a Worker KV value that gets replicated globally automatically. - Page construction can be performed in a Worker. A Worker can retrieve a template from a backend server or the cache and then fill in values on the page from Worker KV values. Those values can be updated rapidly by backend applications. This eliminates the need for solutions like Edge-Side Includes.
With the addition of Workers KV, Cloudflare Workers moves closer to being a complete compute platform embedded inside the Internet.
The Third Place for Code
Until the advent of services like Cloudflare Workers there were really two places to run code: on a server (perhaps rented from a cloud provider) and on the end-user’s client (which could be an IoT device, mobile phone or computer).
Servers and clients have very different properties. Server software is quick to update, if you want to change functionality it can be done quickly because you control when server software is modified. On the other hand client software can be slow to update: it might require a firmware flash on an IoT device, a download on a mobile app or reinstallation on a computer. Even web-based software can be slow to update if the user keeps a session open for a long time.
Client software does have a massive latency advantage over server software: anything the user does with the client will be fast because because the latency between the eyeball and CPU is very low. Server’s tend to have large latency to the client, and latency that can vary widely from moment to moment and location to location.
Cloudflare Workers and similar services take the best of server software and client software. Workers are fast to update (10s of seconds for a global change), yet close to the end user (hence low latency). This gives developers a new way to think about building software: what should go on the server, what on the client and what in the Internet?
FaaScinating
The rise of Functions-as-a-Service (FaaS) has caused some confusion because anyone who has spent time with the lambda-calculus or purely functional languages will know that keeping state around turns out to be very, very useful. Truly writing code in a functional style is a big commitment and FaaS services really require state storage to fit in with the majority, imperative style of programming.
Of course, FaaS software can store state in external services and use API calls to get and update it. But that’s likely slow and error prone. Cloudflare Workers KV introduces FaaS-native storage in the form of a key-value store. Workers KV can be written and read externally via the Cloudflare API, or internally directly from a Cloudflare Worker.
We can’t wait to see what people build with Cloudflare Workers and Cloudflare Workers KV. Get hacking! If you want to get started today sign up here.
P.S. — And if you’d like to replicate Babbage’s log table (without the need for a bell and an assistant) here’s how I built a Worker that retrieves the base-10 logarithm of an integer from a Worker KV. If the logarithm was missing it updates the Worker KV store with the computed log.
Step 1: Create a Namespace
Here CF_ACCT contains the Cloudflare account ID, CF_EMAIL the email address associated with the Cloudflare account and CF_KEY the API key. This call creates a namespace called logcard and returns its unique ID (which is needed later)
$ curl "https://api.cloudflare.com/client/v4/accounts/$CF_ACCT/workers/namespaces" \
-X POST \
-H "X-Auth-Email: $CF_EMAIL" \
-H "X-Auth-Key: $CF_KEY" \
-H "Content-Type: application/json" \
--data '{"title": "logcard"}'
{
"result": {
"id": "<REDACTED NAMESPACE ID>",
"title": "logcard"
},
"success": true,
"errors": [],
"messages": []
}Step 2: Upload Code and Bind a Variable to the Namespace
This curl uploads the script babbage.js containing the Worker and names it logcard. It binds the namespace created above to the variable LOGCARD. That variable can be used within the Worker to access the Workers KV namespace created above.
curl "https://api.cloudflare.com/client/v4/accounts/$CF_ACCT/workers/scripts/logcard" \
-X PUT \
-H "X-Auth-Email: $CF_EMAIL" \
-H "X-Auth-Key: $CF_KEY" \
-F 'script=@-;type=application/javascript' -F 'metadata={"body_part": "script", "bindings": [{"name": "LOGCARD", "type": "kv_namespace", "namespace_id": "<REDACTED NAMESPACE ID>"}]};type=application/json' < babbage.js
{
"result": {
"id": "logcard",
"etag": "<REDACTED ETAG>",
"size": 2254,
"modified_on": "2018-09-28T07:33:01.109336Z"
},
"success": true,
"errors": [],
"messages": []
}The code accesses the namespace and retrieves values with const retrieved = await LOGCARD.get(paddedInt). The full code for the Worker can be found here.
I've bound this Worker to the route /logcard on the site https://glowbeamtechnologies.com/. The Worker takes a value between 1 and 9999 in the URI parameter int and outputs a representation of one of Babbage's punched cards containing the decimal part of the base-10 log of int.
It first tries to find the value in the Workers KV, if not found it calculates the value and stores it in the KV for later reuse. You can try it here: https://glowbeamtechnologies.com/logcard?int=2303.
Subscribe to the blog for daily updates on all of our announcements.
