We’ve all experienced the frustrations of slow loading websites, or an app that seems to stall when it needs to call an API for an update. Anything less than instant and your mind wanders to something else...

One way to make things fast is to bring resources as close to the user as possible — this is what Cloudflare has been doing with compute by running within milliseconds of most of the world’s population. But, counterintuitive as it may seem, sometimes bringing compute closer to the user can actually slow applications down. If your application needs to connect to APIs, databases or other resources that aren’t located near the end user then it can be more performant to run the application near the resources instead of the user.

So today we’re excited to announce Smart Placement for Workers and Pages Functions, making every interaction as fast as possible. With Smart Placement, Cloudflare is taking serverless computing to the Supercloud by moving compute resources to optimal locations in order to speed up applications. The best part – it’s completely automatic, without any extra input (like the dreaded “region”) needed.

Smart Placement is available now, in open beta, to all Workers and Pages customers!

Check out our demo on how Smart Placement works!

Cloudflare’s anycast network is built to process requests instantly and close to the user. As a developer, that’s what makes Cloudflare Workers, our serverless compute offering so compelling. Competitors are bounded by “regions” while Workers run everywhere — hence we have one region: Earth. Requests handled entirely by Workers can be processed right then and there, without ever having to hit an origin server.

While this concept of serverless was originally considered to be for lightweight tasks, serverless computing has been seeing a shift in recent years. It’s being used to replace traditional architecture, which relies on origin servers and self-managed infrastructure, instead of simply augmenting it. We’re seeing more and more of these use cases with Workers and Pages users.

With the shift to going serverless and building entire applications on Workers comes a need for data. Storing information about previous actions or events lets you build personalized, interactive applications. Say you need to create user profiles, store which page a user left off at, which SKUs a user has in their cart – all of these are mapped to data points used to maintain state. Backend services like relational databases, key-value stores, blob storage, and APIs all let you build stateful applications.

We have our own growing suite of storage offerings: Workers KV, Durable Objects, D1, R2. As we’re maturing our data products, we think deeply about their interactions with Workers so that you don’t have to! For example, another approach that has better performance in some cases is moving storage, rather than compute close to users. If you’re using Durable Objects to create a real-time game we could move the Durable Objects to minimize latency for all users.

Our goal for the future state is that you set mode = "smart"and we evaluate the optimal placement of all your resources with no additional configuration needed.

Today, the primary use case for Smart Placement is when you’re using non-Cloudflare services like external databases or third party APIs for your applications.

Many backend services, whether they’re self-hosted or managed services, are centralized, meaning that data is stored and managed in a single location. Your users are global and Workers are global, but your backend is centralized.

If your code makes multiple requests to your backend services they could be crossing the globe multiple times, having a big hit on performance. Some services offer data replication and caching which help to improve performance, but also come with trade-offs like data consistency and higher costs that should be weighed against your use case.

The Cloudflare network is ~50ms from 95% of the world’s connected population. Turning this on its head, we're also very close to your backend services.

Let’s understand how moving compute close to your backend services could decrease application latency by walking through an example:

Say you have a user in Sydney, Australia who’s accessing an application running on Workers. This application makes three round trips to a database located in Frankfurt, Germany in order to serve the user’s request.

Intuitively, you can guess that the bottleneck is going to be the time that it takes the Worker to perform multiple round trips to your database. Instead of the Worker being invoked close to the user, what if it was invoked in a data center closest to the database instead?

Let’s put this to the test.

We measured the request duration for a Worker without Smart Placement and compared it to one with Smart Placement enabled. For both tests, we sent 3,500 requests from Sydney to a Worker which does three round trips to an Upstash instance (free-tier) located in eu-central-1 (Frankfurt).

The results are clear! In this example, moving the Worker close to the backend improved application performance by 4-8x.

As a developer, you should focus on what you do best – building applications – without needing to worry about the network decisions that will make your application faster.

Cloudflare has a unique vantage point: our network gathers intelligence around the optimal paths between users, Cloudflare data centers and back-end servers – we have lots of experience in this area with Argo Smart Routing. Smart Placement takes these factors into consideration to automatically place your Worker in the best spot to minimize overall request duration.

So, how does Smart Placement work?

Smart Placement can be enabled on a per-Worker basis under the “Settings” tab or in your wrangler.toml file:

[placement]
mode = "smart"

Once you enable Smart Placement on your Worker or Pages Function, the Smart Placement algorithm analyzes fetch requests (also known as subrequests) that your Worker is making in real time. It then compares these to latency data aggregated by our network. If we detect that on average your Worker makes more than one subrequest to a back-end resource, then your Worker will automatically get invoked from the optimal data center!

There are some back-end services that, for good reason, are not considered by the Smart Placement algorithm:

• Globally distributed services: If the services that your Worker communicates with are geo-distributed in many regions, Smart Placement isn’t a good fit. We automatically rule these out of the Smart Placement optimization.

• Analytics or logging services: Requests to analytics or logging services don’t need to be in the critical path of your application. waitUntil() should be used so that the response back to users isn’t blocked when instrumenting your code. Since waitUntil() doesn’t impact the request duration from a user’s perspective, we automatically rule analytics/logging services out of the Smart Placement optimization.

Refer to our documentation for a list of services not considered by the Smart Placement algorithm.

Once Smart Placement kicks in, you’ll be able to see a new “Request Duration” tab on your Worker. We route 1% of requests without Smart Placement enabled so that you can see its impact on request duration.

And yes, it is really that easy!

Try out Smart Placement by checking out our demo (it’s a lot of fun to play with!). To learn more, visit our developer documentation.

We’re only getting started! We have lots of ideas on how we can improve Smart Placement:

• Support for calculating the optimal location when the application uses multiple back-ends

• Fine-tuned placement (e.g. if your Worker uses multiple back-ends depending on the path. We calculate the optimal placement per path instead of per-Worker)

• Support for TCP based connections

We would like to hear from you! If you have feedback or feature requests, reach out through the Cloudflare Developer Discord.

When OpenAI launched ChatGPT plugins in alpha we knew that it opened the door for new possibilities for both Cloudflare users and developers building on Cloudflare. After the launch, our team quickly went to work seeing what we could build, and today we’re very excited to share with you two new Cloudflare ChatGPT plugins – the Cloudflare Radar plugin and the Cloudflare Docs plugin.

The Cloudflare Radar plugin allows you to talk to ChatGPT about real-time Internet patterns powered by Cloudflare Radar.

The Cloudflare Docs plugin allows developers to use ChatGPT to help them write and build Cloudflare applications with the most up-to-date information from our documentation. It also serves as an open source example of how to build a ChatGPT plugin with Cloudflare Workers.

Let’s do a deeper dive into how each of these plugins work and how we built them.

When ChatGPT introduced plugins, one of their use cases was retrieving real-time data from third-party applications and their APIs and letting users ask relevant questions using natural language.

Cloudflare Radar has lots of data about how people use the Internet, a well-documented public API, an OpenAPI specification, and it’s entirely built on top of Workers, which gives us lots of flexibility for improvements and extensibility. We had all the building blocks to create a ChatGPT plugin quickly. So, that's what we did.

We added an OpenAI manifest endpoint which describes what the plugin does, some branding assets, and an enriched OpenAPI schema to tell ChatGPT how to use our data APIs. The longest part of our work was fine-tuning the schema with good descriptions (written in natural language, obviously) and examples of how to query our endpoints.

Amusingly, the descriptions ended up much improved by the need to explain the API endpoints to ChatGPT. An interesting side effect is that this benefits us humans also.

{
    "/api/v1/http/summary/ip_version": {
        "get": {
            "operationId": "get_SummaryIPVersion",
            "parameters": [
                {
                    "description": "Date range from today minus the number of days or weeks specified in this parameter, if not provided always send 14d in this parameter.",
                    "required": true,
                    "schema": {
                        "type": "string",
                        "example": "14d",
                        "enum": ["14d","1d","2d","7d","28d","12w","24w","52w"]
                    },
                    "name": "dateRange",
                    "in": "query"
                }
            ]
        }
    }

Luckily, itty-router-openapi, an easy and compact OpenAPI 3 schema generator and validator for Cloudflare Workers that we built and open-sourced when we launched Radar 2.0, made it really easy for us to add the missing parts.

import { OpenAPIRouter } from '@cloudflare/itty-router-openapi'

const router = OpenAPIRouter({
  aiPlugin: {
    name_for_human: 'Cloudflare Radar API',
    name_for_model: 'cloudflare_radar',
    description_for_human: "Get data insights from Cloudflare's point of view.",
    description_for_model:
      "Plugin for retrieving the data based on Cloudflare Radar's data. Use it whenever a user asks something that might be related to Internet usage, eg. outages, Internet traffic, or Cloudflare Radar's data in particular.",
    contact_email: 'radar@cloudflare.com',
    legal_info_url: 'https://www.cloudflare.com/website-terms/',
    logo_url: 'https://cdn-icons-png.flaticon.com/512/5969/5969044.png',
  },
})

We incorporated our changes into itty-router-openapi, and now it supports the OpenAI manifest and route, and a few other options that make it possible for anyone to build their own ChatGPT plugin on top of Workers too.

The Cloudflare Radar ChatGPT is available to non-free ChatGPT users or anyone on OpenAI’s plugin's waitlist. To use it, simply open ChatGPT, go to the Plugin store and install Cloudflare Radar.

Once installed, you can talk to it and ask questions about our data using natural language.

When you add plugins to your account, ChatGPT will prioritize using their data based on what the language model understands from the human-readable descriptions found in the manifest and Open API schema. If ChatGPT doesn't think your prompt can benefit from what the plugin provides, then it falls back to its standard capabilities.

Another interesting thing about plugins is that they extend ChatGPT's limited knowledge of the world and events after 2021 and can provide fresh insights based on recent data.

Here are a few examples to get you started:

"What is the percentage distribution of traffic per TLS protocol version?"

"What's the HTTP protocol version distribution in Portugal?"

Now that ChatGPT has context, you can add some variants, like switching the country and the date range.

“How about the US in the last six months?”

You can also combine multiple topics (ChatGPT will make multiple API calls behind the scenes and combine the results in the best possible way).

“How do HTTP protocol versions compare with TLS protocol versions?”

Out of ideas? Ask it “What can I ask the Radar plugin?”, or “Give me a random insight”.

Be creative, too; it understands a lot about our data, and we keep improving it. You can also add date or country filters using natural language in your prompts.

The Cloudflare Docs plugin is a ChatGPT Retrieval Plugin that lets you access the most up-to-date knowledge from our developer documentation using ChatGPT. This means if you’re using ChatGPT to assist you with building on Cloudflare that the answers you’re getting or code that’s being generated will be informed by current best practices and information located within our docs. You can set up and run the Cloudflare Docs ChatGPT Plugin by following the read me in the example repo.

The plugin was built entirely on Workers and uses KV as a vector store. It can also keep its index up-to-date using Cron Triggers, Queues and Durable Objects.

The plugin is a Worker that responds to POST requests from ChatGPT to a /query endpoint. When a query comes in, the Worker converts the query text into an embedding vector via the OpenAI embeddings API and uses this to find, and return, the most relevant document snippets from Cloudflare’s developer documentation.

The way this is achieved is by first converting every document in Cloudflare’s developer documentation on GitHub into embedding vectors (again using OpenAI’s API) and storing them in KV. This storage format allows you to find semantically similar content by doing a similarity search (we use cosine similarity), where two pieces of text that are similar in meaning will result in the two embedding vectors having a high similarity score. Cloudflare’s entire developer documentation compresses to under 5MB when converted to embedding vectors, so fetching these from KV is very quick. We’ve also explored building larger vector stores on Workers, as can be seen in this demo of 1 million vectors stored on Durable Object storage. We’ll be releasing more open source libraries to support these vector store use cases in the near future.

So ChatGPT will query the plugin when it believes the user’s question is related to Cloudflare’s developer tools, and the plugin will return a list of up-to-date information snippets directly from our documentation. ChatGPT can then decide how to use these snippets to best answer the user’s question.

The plugin also includes a “Scheduler” Worker that can periodically refresh the documentation embedding vectors, so that the information is always up-to-date. This is advantageous because ChatGPT’s own knowledge has a cutoff of September 2021 – so it’s not aware of changes in documentation, or new Cloudflare products.

The Scheduler Worker is triggered by a Cron Trigger, on a schedule you can set (eg, hourly), where it will check which content has changed since it last ran via GitHub’s API. It then sends these document paths in messages to a Queue to be processed. Workers will batch process these messages – for each message, the content is fetched from GitHub, and then turned into embedding vectors via OpenAI’s API. A Durable Object is used to coordinate all the Queue processing so that when all the batches have finished processing, the resulting embedding vectors can be combined and stored in KV, ready for querying by the plugin.

This is a great example of how Workers can be used not only for front-facing HTTP APIs, but also for scheduled batch-processing use cases.

We are in a time when technology is constantly changing and evolving, so as you experiment with these new plugins please let us know what you think. What do you like? What could be better? Since ChatGPT plugins are in alpha, changes to the plugins user interface or performance (i.e. latency) may occur. If you build your own plugin, we’d love to see it and if it’s open source you can submit a pull request on our example repo. You can always find us hanging out in our developer discord.