AI has reset expectations and that paradigm is breaking: how we search for information has fundamentally changed.

Users no longer want to search websites the old way. They’re used to interacting with AI systems like Copilot, Claude, and ChatGPT, where they can simply ask a question and get an answer. We’ve moved from search engines to answer engines. 

At the same time, websites now have a new class of visitors, AI agents. Agents face the same pain with keyword search: they have to issue keyword queries, click through links, and scrape pages to piece together answers. But they also need more: a structured way to ask questions and get reliable answers across websites. This means that websites need a way to give the agents they trust controlled access, so that information is retrieved accurately.

Website owners need a way to participate in this shift.

If AI has reset expectations, what comes next? To meet both people and agents where they are, websites need more than incremental upgrades to keyword search. They need a model that makes conversational access to content a first-class part of the web itself.

That’s what we want to deliver: combining an open standard (NLWeb) with the infrastructure (AutoRAG) to make it simple for any website to become AI-ready.

NLWeb is an open project developed by Microsoft that defines a standard protocol for natural-language queries on websites. Each NLWeb instance also operates as a Model Context Protocol (MCP) server. Cloudflare is building to this spec and actively working with Microsoft to extend the standard with the goal to let every site function like an AI app, so users and agents alike can query its contents naturally.

AutoRAG, Cloudflare’s managed retrieval engine, can automatically crawl your website, store the content in R2, and embed it into a managed vector database. AutoRAG keeps the index fresh with continuous re-crawling and re-indexing. Model inference and embedding can be served through Workers AI. Each AutoRAG is paired with an AI Gateway that can provide observability and insights into your AI model usage. This gives you a complete, managed pipeline for conversational search without the burden of managing custom infrastructure.

“Together, NLWeb and AutoRAG let publishers go beyond search boxes, making conversational interfaces for websites simple to create and deploy. This integration will enable every website to easily become AI-ready for both people and trusted agents.” – R.V. Guha, creator of NLWeb, CVP and Technical Fellow at Microsoft. 

We are optimistic this will open up new monetization models for publishers:

"The challenges publishers have faced are well known, as are the risks of AI accelerating the collapse of already challenged business models. However, with NLWeb and AutoRAG, there is an opportunity to reset the nature of relationships with audiences for the better. More direct engagement on Publisher Owned and Operated (O&O) environments, where audiences value the brand and voice of the Publisher, means new potential for monetization. This would be the reset the entire industry needs."  – Joe Marchese, General & Build Partner at Human Ventures.

By combining NLWeb's standard with Cloudflare’s AutoRAG infrastructure, we’re making it possible to  easily bring conversational search to any website.

Simply select your domain in AutoRAG, and it will crawl and index your site for semantic querying. It then deploys a Cloudflare Worker, which acts as the access layer. This Worker implements the NLWeb standard and UI defined by the NLWeb project and exposes your indexed content to both people and AI agents. The Worker includes:

• `/ask` endpoint: The defined standard for how conversational web searches should be served. Powers the conversational UI at the root `/` as well as the embeddable preview at `/snippet.html`. It supports chat history so queries can build on one another within the same session, and includes automatic query decontextualization to improve retrieval quality.

• `/mcp` endpoint: Implements an MCP server that trusted AI agents can connect to for structured access.

With this setup, your site content is immediately available in two ways for you to experiment: through a conversational UI that you can serve to your visitors, and through a structured MCP interface that lets trusted agents query your site reliably on your terms.

Additionally, if you prefer to deploy and host your own version of the NLWeb project, there’s also the option to use AutoRAG as the retrieval engine powering the NLWeb instance.

From your perspective, making your site conversational is just a single click. Behind the scenes, AutoRAG spins up a full retrieval pipeline to make that possible:

1. Crawling and ingestion: AutoRAG explores your site like a search engine, following `sitemap.xml` and `robots.txt` files to understand what pages are available and allowed for crawling. From there, it follows your sitemap to discover pages within your domain (up to 100k pages). Browser Rendering is used to load each page so that it can capture dynamic, JavaScript content. Crawled pages are downloaded into an R2 bucket in your account before being ingested. 

2. Continuous Indexing: Once ingested, the content is parsed and embedded into Vectorize, making it queryable beyond keyword matching through semantic search. AutoRAG automatically re-crawls and re-indexes to keep your knowledge base aligned with your latest content.

3. Access & Observability: A Cloudflare Worker is deployed in your account to serve as the access layer that implements the NLWeb protocol (you can also find the deployable Worker in the Workers templates repository). Workers AI is used to seamlessly power the summarization and decontextualized query capabilities to improve responses. Soon, with the AI Gateway and Secret Store BYO keys, you’ll be able to connect models from any provider and select them directly in the AutoRAG dashboard.

Until now, AutoRAG only supported R2 as a data source. That worked well for structured files, but we needed to make a website itself a first-class data source to be indexed and searchable. Making that possible meant building website crawling into AutoRAG and strengthening the system to handle large, dynamic sources like websites.

Before implementing our web crawler, we needed to improve the reliability of data syncs. Prior users of AutoRAG lacked visibility into when indexing syncs ran and whether they were successful. To fix this, we introduced a Job module to track all syncs, store history, and provide logs. This required two new Durable Objects to be added into AutoRAG’s architecture:

• JobManager runs a complete sync, and its duties include queuing files, embedding content, and keeping the Vectorize database up to date.  To ensure data consistency, only one JobManager can run per RAG at a time, enforced by the RagManager (a Durable Object in our existing architecture), which cancels any running jobs before starting new ones which can be triggered either manually or by a scheduled sync.

• FileManager solved scalability issues we hit when Workers ran out of memory during parallel processing. Originally, a single Durable Object was responsible for handling multiple files, but with a 128MB memory limit it quickly became a bottleneck. The solution was to break the work apart: JobManager now distributes files across many FileManagers, each responsible for a single file. By processing 20 files in parallel through 20 different FileManagers, we expanded effective memory capacity from 128MB to roughly 2.5GB per batch.

With these improvements, we were ready to build the website parser. By reusing our existing R2-based queuing logic, we added crawling with minimal disruption:

1. A JobManager designated for a website crawl begins by reading the sitemaps associated with the RAG configuration.

2. Instead of listing objects from an R2 bucket, it queues each website link into our existing R2-based queue, using the full URL as the R2 object key.

3. From here, the process is nearly identical to our file-based sync. A FileManager picks up the job and checks if the RAG is configured for website parsing.

4. If it is, the FileManager crawls the link and places the page's HTML contents into the user's R2 bucket, again using the URL as the object key.

After these steps, we index the data and serve it at query time. This approach maximized code reuse, and any improvements to our HTML-to-Markdown conversion now benefit both file and website-based RAGs automatically.

Getting your website ready for conversational search through NLWeb and AutoRAG is simple. Here’s how:

1. In the Cloudflare Dashboard, navigate to Compute & AI > AutoRAG.

2. Select Create in AutoRAG, then choose the NLWeb Website quick deploy option.

3. Select the domain from your Cloudflare account that you want indexed.

4. Click Start indexing.

That’s it! You can now try out your NLWeb search experience via the provided link, and test out how it will look on your site by using the embeddable snippet.

We’d love to hear your feedback as you experiment with this new capability and share your thoughts with us at nlweb@cloudflare.com.

With AI Gateway already processing over 3 billion logs and experiencing rapid growth, the number of connections to the platform continues to increase steadily. To help developers manage this scale more effectively, we wanted to offer an alternative to implementing HTTP/2 keep-alive to maintain persistent HTTP(S) connections, thereby avoiding the overhead of repeated handshakes and TLS negotiations with each new HTTP connection to AI Gateway. We understand that implementing HTTP/2 can present challenges, particularly when many libraries and tools may not support it by default and most modern programming languages have well-established WebSocket libraries available.

With this in mind, we used Cloudflare’s Developer Platform and Durable Objects (yes, again!) to build a WebSockets API that establishes a single, persistent connection, enabling continuous communication. 

Through this API, all AI providers supported by AI Gateway can be accessed via WebSocket, allowing you to maintain a single TCP connection between your client or server application and the AI Gateway. The best part? Even if your chosen provider doesn’t support WebSockets, we handle it for you, managing the requests to your preferred AI provider.

By connecting via WebSocket to AI Gateway, we make the requests to the inference service for you using the provider’s supported protocols (HTTPS, WebSocket, etc.), and you can keep the connection open to execute as many inference requests as you would like. 

To make your connection to AI Gateway more secure, we are also introducing authentication for AI Gateway. The new WebSockets API will require authentication. All you need to do is create a Cloudflare API token with the permission “AI Gateway: Run” and send that in the cf-aig-authorization header.

In the flow diagram above:

1️⃣ When Authenticated Gateway is enabled and a valid token is included, requests will pass successfully.

2️⃣ If Authenticated Gateway is enabled, but a request does not contain the required cf-aig-authorization header with a valid token, the request will fail. This ensures only verified requests pass through the gateway. 

3️⃣ When Authenticated Gateway is disabled, the cf-aig-authorization header is bypassed entirely, and any token — whether valid or invalid — is ignored.

We recently used Durable Objects (DOs) to scale our logging solution for AI Gateway, so using WebSockets within the same DOs was a natural fit.

When a new WebSocket connection is received by our Cloudflare Workers, we implement authentication in two ways to support the diverse capabilities of WebSocket clients. The primary method involves validating a Cloudflare API token through the cf-aig-authorization header, ensuring the token is valid for the connecting account and gateway. 

However, due to limitations in browser WebSocket implementations, we also support authentication via the “sec-websocket-protocol” header. Browser WebSocket clients don't allow for custom headers in their standard API, complicating the addition of authentication tokens in requests. While we don’t recommend that you store API keys in a browser, we decided to add this method to add more flexibility to all WebSocket clients.

// Built-in WebSocket client in browsers
const socket = new WebSocket("wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/", [
   "cf-aig-authorization.${AI_GATEWAY_TOKEN}"
]);


// ws npm package
import WebSocket from "ws";
const ws = new WebSocket("wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/",{
   headers: {
       "cf-aig-authorization": "Bearer AI_GATEWAY_TOKEN",
   },
});

After this initial verification step, we upgrade the connection to the Durable Object, meaning that it will now handle all the messages for the connection. Before the new connection is fully accepted, we generate a random UUID, so this connection is identifiable among all the messages received by the Durable Object. During an open connection, any AI Gateway settings passed via headers — such as cf-aig-skip-cache (which bypasses caching when set to true) — are stored and applied to all requests in the session. However, these headers can still be overridden on a per-request basis, just like with the Universal Endpoint today.

Once the connection is established, the Durable Object begins listening for incoming messages. From this point on, users can send messages in the AI Gateway universal format via WebSocket, simplifying the transition of your application from an existing HTTP setup to WebSockets-based communication.

import WebSocket from "ws";
const ws = new WebSocket("wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/",{
   headers: {
       "cf-aig-authorization": "Bearer AI_GATEWAY_TOKEN",
   },
});

ws.send(JSON.stringify({
   type: "universal.create",
   request: {
      "eventId": "my-request",
      "provider": "workers-ai",
      "endpoint": "@cf/meta/llama-3.1-8b-instruct",
      "headers": {
         "Authorization": "Bearer WORKERS_AI_TOKEN",
         "Content-Type": "application/json"
      },
      "query": {
         "prompt": "tell me a joke"
      }
   }
}));

ws.on("message", function incoming(message) {
   console.log(message.toString())
});

When a new message reaches the Durable Object, it’s processed using the same code that powers the HTTP Universal Endpoint, enabling seamless code reuse across Workers and Durable Objects — one of the key benefits of building on Cloudflare.

For non-streaming requests, the response is wrapped in a JSON envelope, allowing us to include additional information beyond the AI inference itself, such as the AI Gateway log ID for that request.

Here’s an example response for the request above:

{
  "type":"universal.created",
  "metadata":{
     "cacheStatus":"MISS",
     "eventId":"my-request",
     "logId":"01JC3R94FRD97JBCBX3S0ZAXKW",
     "step":"0",
     "contentType":"application/json"
  },
  "response":{
     "result":{
        "response":"Why was the math book sad? Because it had too many problems. Would you like to hear another one?"
     },
     "success":true,
     "errors":[],
     "messages":[]
  }
}

For streaming requests, AI Gateway sends an initial message with request metadata telling the developer the stream is starting.

{
  "type":"universal.created",
  "metadata":{
     "cacheStatus":"MISS",
     "eventId":"my-request",
     "logId":"01JC40RB3NGBE5XFRZGBN07572",
     "step":"0",
     "contentType":"text/event-stream"
  }
}

After this initial message, all streaming chunks are relayed in real-time to the WebSocket connection as they arrive from the inference provider. Note that only the eventId field is included in the metadata for these streaming chunks (more info on what this new field is below).

{
  "type":"universal.stream",
  "metadata":{
     "eventId":"my-request",
  }
  "response":{
     "response":"would"
  }
}

This approach serves two purposes: first, all request metadata is already provided in the initial message. Second, it addresses the concurrency challenge of handling multiple streaming requests simultaneously.

With WebSocket connections, client and server can send messages asynchronously at any time. This means the client doesn’t need to wait for a server response before sending another message. But what happens if a client sends multiple streaming inference requests immediately after the WebSocket connection opens?

In this case, the server streams all the inference responses simultaneously to the client. Since everything occurs asynchronously, the client has no built-in way to identify which response corresponds to each request.

To address this, we introduced a new field in the Universal format called eventId, which allows AI Gateway to include a client-defined ID with each message, even in a streaming WebSocket environment.

So, to fully answer the question above: the server streams both responses in parallel chunks, and the client can accurately identify which request each message belongs to based on the eventId.

Once all chunks for a request have been streamed, AI Gateway sends a final message to signal the request’s completion. For added flexibility, this message includes all the metadata again, even though it was also provided at the start of the streaming process.

{
  "type":"universal.done",
  "metadata":{
     "cacheStatus":"MISS",
     "eventId":"my-request",
     "logId":"01JC40RB3NGBE5XFRZGBN07572",
     "step":"0",
     "contentType":"text/event-stream"
  }
}

AI Gateway’s real-time Websocket API is now in beta and open to everyone!

To try it out, copy your gateway universal endpoint URL, and replace the “https://” with “wss://”, like this:

wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/

Then open a WebSocket connection using your Universal Endpoint, and guarantee that it is authenticated with a Cloudflare token with the AI Gateway Run permission.

Here’s an example code using the ws npm package:

import WebSocket from "ws";
const ws = new WebSocket("wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/", {
   headers: {
       "cf-aig-authorization": "Bearer AI_GATEWAY_TOKEN",
   },
});

ws.on("open", function open() {
   console.log("Connected to server.");
   ws.send(JSON.stringify({
      type: "universal.create",
      request: {
         "provider": "workers-ai",
         "endpoint": "@cf/meta/llama-3.1-8b-instruct",
         "headers": {
            "Authorization": "Bearer WORKERS_AI_TOKEN",
            "Content-Type": "application/json"
         },
         "query": {
            "stream": true,
            "prompt": "tell me a joke"
         }
      }
   }));
});


ws.on("message", function incoming(message) {
   console.log(message.toString())
});

Here’s an example code using the built-in browser WebSocket client:

const socket = new WebSocket("wss://gateway.ai.cloudflare.com/v1/my-account-id/my-gateway/", [
   "cf-aig-authorization.${AI_GATEWAY_TOKEN}"
]);

socket.addEventListener("open", (event) => {
  console.log("Connected to server.");
   socket.send(JSON.stringify({
      type: "universal.create",
      request: {
         "provider": "workers-ai",
         "endpoint": "@cf/meta/llama-3.1-8b-instruct",
         "headers": {
            "Authorization": "Bearer WORKERS_AI_TOKEN",
            "Content-Type": "application/json"
         },
         "query": {
            "stream": true,
            "prompt": "tell me a joke"
         }
      }
   }));
});

socket.addEventListener("message", (event) => {
  console.log(event.data);
});

In Q1 2025, we plan to support WebSocket-to-WebSocket connections (using DOs), allowing you to connect to OpenAI's new real-time API directly through our platform. In the meantime, you can deploy this Worker in your account to proxy the requests yourself.

If you have any questions, reach out on our Discord channel. We’re also hiring for AI Gateway, check out Cloudflare Jobs in Lisbon!

AI Gateway is specifically designed to address these pain points. Since its launch in September 2023, AI Gateway has empowered developers and organizations by successfully proxying over 2 billion requests in just one year, as we highlighted during September’s Birthday Week. With AI Gateway, developers can easily store, analyze, and optimize their AI inference requests and responses in real time.

With our initial architecture, AI Gateway faced a significant challenge: the logs, those critical trails of data interactions between applications and AI models, could only be retained for 30 minutes. This limitation was not just a minor inconvenience; it posed a substantial barrier for developers and businesses needing to analyze long-term patterns, ensure compliance, or simply debug over more extended periods.

In this post, we'll explore the technical challenges and strategic decisions behind extending our log storage capabilities from 30 minutes to being able to store billions of logs indefinitely. We'll discuss the challenges of scale, the intricacies of data management, and how we've engineered a system that not only meets the demands of today, but is also scalable for the future of AI development.

AI Gateway is built on Cloudflare Workers, a serverless platform that runs on the Cloudflare network, allowing developers to write small JavaScript functions that can execute at the point of need, near the user, on Cloudflare's vast network of data centers, without worrying about platform scalability.

Our customers use multiple providers and models and are always looking to optimize the way they do inference. And, of course, in order to evaluate their prompts, performance, cost, and to troubleshoot what’s going on, AI Gateway’s customers need to store requests and responses. New requests show up within 15 seconds and customers can check a request’s cost, duration, number of tokens, and provide their feedback (thumbs up or down).

This scales in a way where an account can have multiple gateways and each gateway has its own settings. In our first implementation, a backend worker was responsible for storing Real Time Logs and other background tasks. However, in the rapidly evolving domain of artificial intelligence, where real-time data is as precious as the insights it provides, managing log data efficiently becomes paramount. We recognized that to truly empower our users, we needed to offer a solution where logs weren't just transient records but could be stored permanently. Permanent log storage means developers can now track the performance, security, and operational insights of their AI applications over time, enabling not only immediate troubleshooting but also longitudinal studies of AI behavior, usage trends, and system health.

The diagram above describes our old architecture, which could only store 30 minutes of data.

Tracing the path of a request through the AI Gateway, as depicted in the sequence above:

1. A developer sends a new inference request, which is first received by our Gateway Worker.

2. The Gateway Worker then performs several checks: it looks for cached results, enforces rate limits, and verifies any other configurations set by the user for their gateway. Provided all conditions are met, it forwards the request to the selected inference provider (in this diagram, OpenAI).

3. The inference provider processes the request and sends back the response.

4. Simultaneously, as the response is relayed back to the developer, the request and response details are also dispatched to our Backend Worker. This worker's role is to manage and store the log of this transaction.

Initially, the AI Gateway project stored both request metadata and the actual request bodies in a D1 database. This approach facilitated rapid development in the project's infancy. However, as customer engagement grew, the D1 database began to fill at an accelerating rate, eventually retaining logs for only 30 minutes at a time.

To mitigate this, we first optimized the database schema, which extended the log retention to one hour. However, we soon encountered diminishing returns due to the sheer volume of byte data from the request bodies. Post-launch, it became clear that a more scalable solution was necessary. We decided to migrate the request bodies to R2 storage, significantly alleviating the data load on D1. This adjustment allowed us to incrementally extend log retention to 24 hours.

Consequently, D1 functioned primarily as a log index, enabling users to search and filter logs efficiently. When users needed to view details or download a log, these actions were seamlessly proxied through to R2.

This dual-system approach provided us with the breathing room to contemplate and develop more sophisticated storage solutions for the future.

As our traffic surged, we encountered a growing number of requests from customers wanting to access and compare older logs.

Upon learning that the Durable Objects team was seeking beta testers for their new Durable Objects with SQLite, we eagerly signed up.

Originally, we considered Durable Objects as the ideal solution for expanding our log storage capacity, which required us to shard the logs by a unique string. Initially, this string was the account ID, but during a mid-development load test, we hit a cap at 10 million logs per Durable Object. This limitation meant that each account could only support up to this number of logs.

Given our commitment to the DO migration, we saw an opportunity rather than a constraint. To overcome the 10 million log limit per account, we refined our approach to shard by both account ID and gateway name. This adjustment effectively raised the storage ceiling from 10 million logs per account to 10 million per gateway. With the default setting allowing each account up to 10 gateways, the potential storage for each account skyrocketed to 100 million logs.

This strategic pivot not only enabled us to store a significantly larger number of logs. But also enhanced our flexibility in gateway management. Now, when a gateway is deleted, we can simply remove the corresponding Durable Object.

Additionally, this sharding method isolates high-volume request scenarios. If one customer's heavy usage slows down log insertion, it only impacts their specific Durable Object, thereby preserving performance for other customers.

Taking a glance at the revised architecture diagram, we replaced the Backend Worker with our newly integrated Durable Object. The rest of the request flow remains unchanged, including the concurrent response to the user and the interaction with the Durable Object, which occurs in the fourth step.

Leveraging Cloudflare’s network, our Gateway Worker operates near the user's location, which in turn positions the user's Durable Object close by. This proximity significantly enhances the speed of log insertion and query operations.

As the number of users and requests on AI Gateway grows, managing each unique Durable Object (DO) becomes increasingly complex. New customers join continuously, and we needed an efficient method to track each DO, ensure users stay within their 10 gateway limit, and manage the storage capacity for free users.

To address these challenges, we introduced another layer of control with a new Durable Object we've named the Account Manager. The primary function of the Account Manager is straightforward yet crucial: it keeps user activities in check.

Here's how it works: before any Gateway commits a new log to permanent storage, it consults the Account Manager. This check determines whether the gateway is allowed to insert the log based on the user's current usage and entitlements. The Account Manager uses its own SQLite database to verify the total number of rows a user has and their service level. If all checks pass, it signals the Gateway that the log can be inserted. It was paramount to guarantee that this entire validation process occurred in the background, ensuring that the user experience remains seamless and uninterrupted.

The Account Manager stays updated by periodically receiving data from each Gateway’s Durable Object. Specifically, after every 1000 inference requests, the Gateway sends an update on its total rows to the Account Manager, which then updates its local records. This system ensures that the Account Manager has the most current data when making its decisions.

Additionally, the Account Manager is responsible for monitoring customer entitlements. It tracks whether an account is on a free or paid plan, how many gateways a user is permitted to create, and the log storage capacity allocated to each gateway. 

Through these mechanisms, the Account Manager not only helps in maintaining system integrity but also ensures fair usage across all users of AI Gateway.

As we continue to develop evaluations to fully automatic and, in the future, use Large Language Models (LLMs),  we are now taking the first step towards this goal and launching the open beta phase of comprehensive AI evaluations, centered on Human-in-the-Loop feedback.

This feature empowers users to create bespoke datasets from their application logs, thereby enabling them to score and evaluate the performance, speed, and cost-effectiveness of their models, with a primary focus on LLMs and automated scoring, analyzing the performance of LLMs, providing developers with objective, data-driven insights to refine their models.

To do this, developers require a reliable logging mechanism that persists logs from multiple gateways, storing up to 100 million logs in total (10 million logs per gateway, across 10 gateways). This represents a significant volume of data, as each request made through the AI Gateway generates a log entry, with some log entries potentially exceeding 50 MB in size.

This necessity leads us to work on the expansion of log storage capabilities. Since log storage is limited to 10 million logs per gateway, in future iterations, we aim to scale this capacity by implementing sharded Durable Objects (DO), allowing multiple Durable Objects per gateway to handle and store logs. This scaling strategy will enable us to store significantly larger volumes of logs, providing richer data for evaluations (using LLMs as a judge or from user input), all through AI Gateway.

We are working on improving our existing Universal Endpoint, the next step on an enhanced solution that builds on existing fallback mechanisms to offer greater resilience, flexibility, and intelligence in request management.

Currently, when a provider encounters an error or is unavailable, our system falls back to an alternative provider to ensure continuity. The improved Universal Endpoint takes this a step further by introducing automatic retry capabilities, allowing failed requests to be reattempted before fallback is triggered. This significantly improves reliability by handling transient errors and increasing the likelihood of successful request fulfillment. It will look something like this:

curl --location 'https://aig.example.com/' \
--header 'CF-AIG-TOKEN: Bearer XXXX' \
--header 'Content-Type: application/json' \
--data-raw '[
    {
        "id": "0001",
        "provider": "openai",
        "endpoint": "chat/completions",
        "headers": {
            "Authorization": "Bearer XXXX",
            "Content-Type": "application/json"
        },
        "query": {
            "model": "gpt-3.5-turbo",
            "messages": [
                {
                    "role": "user",
                    "content": "generate a prompt to create cloudflare random images"
                }
            ]
        },
        "option": {
            "retry": 2,
            "delay": 200,
            "onComplete": {
                "provider": "workers-ai",
                "endpoint": "@cf/stabilityai/stable-diffusion-xl-base-1.0",
                "headers": {
                    "Authorization": "Bearer XXXXXX",
                    "Content-Type": "application/json"
                },
                "query": {
                    "messages": [
                        {
                            "role": "user",
                            "content": "<prompt-response id='\''0001'\'' />"
                        }
                    ]
                }
            }
        }
    },
    {
        "provider": "workers-ai",
        "endpoint": "@cf/stabilityai/stable-diffusion-xl-base-1.0",
        "headers": {
            "Authorization": "Bearer XXXXXX",
            "Content-Type": "application/json"
        },
        "query": {
            "messages": [
                {
                    "role": "user",
                    "content": "create a image of a missing cat"
                }
            ]
        }
    }
]'

The request to the improved Universal Endpoint system demonstrates how it handles multiple providers with integrated retry mechanisms and fallback logic. In this example, the first request is sent to a provider like OpenAI, asking it to generate a text-to-image prompt. The “retry” option ensures that transient issues don’t result in immediate failure.

The system’s ability to seamlessly switch between providers while applying retry strategies ensures higher reliability and robustness in managing requests. By leveraging fallback logic, the Improved Universal Endpoint can dynamically adapt to provider failures, ensuring that tasks are completed successfully even in complex, multi-step workflows.

In addition to retry logic, we will have the ability to inspect requests and responses and make dynamic decisions based on the content of the result. This enables developers to create conditional workflows where the system can adapt its behavior depending on the nature of the response, creating a highly flexible and intelligent decision-making process.

If you haven’t yet used AI Gateway, check out our developer documentation on how to get started. If you have any questions, reach out on our Discord channel.

Radar 2.0 was built on the learnings of Radar 1.0 and was launched last month during Cloudflare's Birthday Week as a complete product revamp. We wanted to make it easier for our users to find insights and navigate our data, and overall provide a better and faster user experience.

We're building a Supercloud. Cloudflare's products now include hundreds of features in networking, security, access controls, computing, storage, and more.

This blog will explain how we built the new Radar from an engineering perspective. We wanted to do this to demonstrate that anyone could build a somewhat complex website that involves demanding requirements and multiple architectural layers, do it on top of our stack, and how easy it can be.

Hopefully, this will inspire other developers to switch from traditional software architectures and build their applications using modern, more efficient technologies.

The following diagram is a birds-eye view of the Radar 2.0 architecture. As you can see, it's divided into three main layers:

• The Core layer is where we keep our data lake, data exploration tools, and backend API.

• The Cloudflare network layer is where we host and run Radar and serve the public APIs.

• The Client layer is essentially everything else that runs in your browser. We call it the Radar Web app.

As you can see, there are Cloudflare products everywhere. They provide the foundational resources to host and securely run our code at scale, but also other building blocks necessary to run the application end to end.

By having these features readily available and tightly integrated into our ecosystem and tools, at the distance of a click and a few lines of code, engineering teams don't have to reinvent the wheel constantly and can use their time on what is essential: their app logic.

Let's dig in.

Radar 2.0 is deployed using Cloudflare Pages, our developer-focused website hosting platform. In the early days, you could only host static assets on Pages, which was helpful for many use cases, including integrating with static site generators like Hugo, Jekyll, or Gatsby. Still, it wouldn't solve situations where your application needs some sort of server-side computing or advanced logic using a single deployment.

Luckily Pages recently added support to run custom Workers scripts. With Functions, you can now run server-side code and enable any kind of dynamic functionality you'd typically implement using a separate Worker.

Cloudflare Pages Functions also allow you to use Durable Objects, KV, R2, or D1, just like a regular Worker would. We provide excellent documentation on how to do this and more in our Developer Documentation. Furthermore, the team wrote a blog on how to build a full-stack application that describes all the steps in detail.

Radar 2.0 needs server-side functions for two reasons:

• To render Radar and run the server side of Remix.

• To implement and serve our frontend API.

We use Remix with Cloudflare Pages on Radar 2.0.

Remix follows a server/client model and works under the premise that you can't control the user's network, so web apps must reduce the amount of Javascript, CSS, and JSON they send through the wire. To do this, they move some of the logic to the server.

In this case, the client browser will get pre-rendered DOM components and the result of pre-fetched API calls with just the right amount of JSON, Javascript, and CSS code, rightfully adjusted to the UI needs. Here’s the technical explanation with more detail.

Typically, Remix would need a Node.js server to do all of this, but guess what: It can also run on Cloudflare Workers and Pages.

Here’s the code to get the Remix server running on Workers, using Cloudflare Pages:

import { createPagesFunctionHandler } from "@remix-run/cloudflare-pages";
import * as build from "@remix-run/dev/server-build";

const handleRequest = createPagesFunctionHandler({
  build: {
    ...build,
    publicPath: "/build/",
    assetsBuildDirectory: "public/build",
  },
  mode: process.env.NODE_ENV,
  getLoadContext: (context) => ({
    ...context.env,
    CF: (context.request as any).cf as IncomingRequestCfProperties | undefined,
  }),
});

const handler: ExportedHandler<Env> = {
  fetch: async (req, env, ctx) => {
    const r = new Request(req);
    return handleRequest({
      env,
      params: {},
      request: r,
      waitUntil: ctx.waitUntil,
      next: () => {
        throw new Error("next() called in Worker");
      },
      functionPath: "",
      data: undefined,
    });
  },
};

In Remix, routes handle changes when a user interacts with the app and changes it (clicking on a menu option, for example). A Remix route can have a loader, an action and a default export. The loader handles API calls for fetching data (GET method). The action handles submissions to the server (POST, PUT, PATCH, DELETE methods) and returns the response. The default export handles the UI code in React that’s returned for that route. A route without a default export returns only data.

Because Remix runs both on the server and the client, it can get smart and know what can be pre-fetched and computed server-side and what must go through the network connection, optimizing everything for performance and responsiveness.

Here’s an example of a Radar route, simplified for readability, for the Outage Center page.

import type { MetaFunction } from "@remix-run/cloudflare";
import { useLoaderData } from "@remix-run/react";
import { type LoaderArgs } from "@remix-run/server-runtime";

export async function loader(args: LoaderArgs) {
  const ssr = await initialFetch(SSR_CHARTS, args);
  return { ssr, };
}

export default function Outages() {
  const { ssr } = useLoaderData<typeof loader>();

  return (
    <Page
      filters={["timerange"]}
      title={
        <>
          <Svg use="icon-outages" />
          {t("nav.main.outage-center")}
        </>
      }
    >
      <Grid columns={[1, 1, 1, 1]}>
        <Card.Article colspan={[1, 1, 1, 1]} rowspan={[1, 1, 1, 1]}>
          <Card.Section>
            <Components.InternetOutagesChoropleth ssr={ssr} />
          </Card.Section>
          <Divider />
          <Card.Section>
            <Components.InternetOutagesTable ssr={ssr} />
          </Card.Section>
        </Card.Article>
      </Grid>
    </Page>
  );
}

And here’s what it produces:

Remix and SSR can also help you with your Lighthouse scores and SEO. It can drastically improve metrics like Cumulative Layout Shift, First Contentful Paint and Largest Contentful Paint by reducing the number of fetches and information traveling from the server to the browser and pre-rendering the DOM.

Another project porting their app to Remix is Cloudflare TV. This is how their metrics looked before and after the changes.

Radar’s Desktop Lighthouse score is now nearly 100% on Performance, Accessibility, Best Practices, and SEO.

Another Cloudflare product that we use extensively on Radar 2.0 is Speed. In particular, we want to mention the Early Hints feature. Early Hints is a new web standard that defines a new HTTP 103 header the server can use to inform the browser which assets will likely be needed to render the web page while it's still being requested, resulting in dramatic load times improvements.

You can use Cloudflare Pages with Early Hints.

Radar has two APIs. The backend which has direct access to our data sources, and the frontend, which is available on the Internet.

The backend API was written using Python, Pandas and FastAPI and is protected by Cloudflare Access, JWT tokens and an authenticated origin pull (AOP) configuration. Using Python allows anyone on the team, engineers or data scientists, to collaborate easily and contribute to improving and expanding the API, which is great. Our data science team uses JupyterHub and Jupyter Notebooks as part of their data exploration workflows, which makes prototyping and reusing code, algorithms and models particularly easy and fast.

It then talks to the upstream frontend API via a Strawberry based GraphQL server. Using GraphQL makes it easy to create complex queries, giving internal users and analysts the flexibility they need when building reports from our vast collection of data.

We built Radar's frontend API on top of Cloudflare Workers. This worker has two main functions:

• It fetches data from the backend API using GraphQL, and then transforms it.

• It provides a public REST API that anyone can use, including Radar.

Using a worker in front of our core API allows us to easily add and separate microservices, and also adds notable features like:

• Cloudflare's Cache API allows finer control over what to cache and for how long and supports POST requests and customizable cache control headers, which we use.

• Stale responses using R2. When the backend API cannot serve a request for some reason, and there’s a stale response cached, it’ll be served directly from R2, giving end users a better experience.

• CSV and JSON output formats. The CSV format is convenient and makes it easier for data scientists, analysts, and others to use the API and consume our API data directly from other tools.

One last feature on the frontend API is OpenAPI 3 support. We automatically generate an OpenAPI schema and validate user input with it. This is done through a custom library that we built on top of itty-router, which we also use. Today we’re open sourcing this work.

itty-router-openapi provides an easy and compact OpenAPI 3 schema generator and validator for Cloudflare Workers. Check our GitHub repository for more information and details on how to use it.

Today we’re also launching our developer’s documentation pages for the Radar API where you can find more information about our data license, basic concepts, how to get started and the available API methods. Cloudflare Radar's API is free, allowing academics, data sleuths and other web enthusiasts to investigate Internet usage across the globe, based on data from our global network.

To facilitate using our API, we also put together a Colab Notebook template that you can play with, copy and expand to your use case.

The Radar App is the code that runs in your browser. We've talked about Remix, but what else do we use?

Radar relies on a lot of data visualizations. Things like charts and maps are essential to us. We decided to build our reusable library of visualization components on top of two other frameworks: visx, a "collection of expressive, low-level visualization primitives for React," D3, a powerful JavaScript library for manipulating the DOM based on data, and MapLibre, an open-source map visualization stack.

Here’s one of our visualization components in action. We call it the “PewPew map”.

And here’s the Remix React code for it, whenever we need to use it in a page:

<Card.Section
    title={t("card.attacks.title")}
    description={t("card.attacks.description")}
  >
    <Flex gap={spacing.medium} align="center" justify="flex-end">
      <SegmentedControl
        label="Sort order:"
        name="attacksDirection"
        value={attacksDirection}
        options={[
          { label: t("common.source"), value: "ORIGIN" },
          { label: t("common.target"), value: "TARGET" },
        ]}
      onChange={({ target }: any) => setAttacksDirection(target.value)}
      />
    </Flex>

    <Components.AttacksCombinedChart
      ssr={ssr}
      height={400}
      direction={attacksDirection}
    />
  </Card.Section>

Another change we made to Radar was switching our images and graphical assets to Scalable Vector Graphics. SVGs are great because they're essentially a declarative graphics language. They're XML text files with vectorial information. And so, they can be easily manipulated, transformed, stored, or indexed, and of course, they can be rendered at any size, producing beautiful, crisp results on any device and resolution.

SVGs are also extremely small and efficient in size compared to bitmap formats and support internationalization, making them easier to translate to other languages (localization), thus providing better accessibility.

Here’s an example of a Radar Bubble Chart, inspected, where you can see the SVG code and the strings embedded.

React Cosmos is a "sandbox for developing and testing UI components in isolation." We wanted to use Cosmos with Radar 2.0 because it's the perfect project for it:

1. It has a lot of visual components; some are complex and have many configuration options and features.

2. The components are highly reusable across multiple pages in different contexts with different data.

3. We have a multidisciplinary team; everyone can send a pull request and add or change code in the frontend.

Cosmos acts as a component library where you can see our palette of ready-to-use visualizations and widgets, from simple buttons to complex charts, and you play with their options in real-time and see what happens. Anyone can do it, not only designers or engineers but also other project stakeholders. This effectively improves team communications and makes contributing and iterating quickly.

Here’s a screenshot of our Cosmos in action:

Continuous integration is important for any team doing modern software. Cloudflare Pages provides multiple options to work with CI tools using direct uploads, out of the box. The team has put up documentation and examples on how to do that with GitHub Actions, CircleCI, and Travis, but you can use others.

In our case, we use BitBucket and TeamCity internally to build and deploy our releases. Our workflow automatically builds, tests, and deploys Radar 2.0 within minutes on an approved PR and follow-up merge.

Unit tests are done with Vitest and E2E tests with Playwright. Visual Regression testing is planned and Playwright can also help with that.

Furthermore, we have multiple environments to stage and test our releases before they go live to production. Our CI/CD setup makes it easy to switch from one environment to the other or quickly roll back any undesired deployment.

Again Cloudflare Pages makes it easy to do this using Preview deployments, aliases, or Branch build controls. The same is true for regular Workers using Environments.

Radar 1.0 wasn't particularly fast doing CI/CD, we confess. We had a few episodes when a quick fix could take some good 30 minutes from committing the code to deployment, and we felt frustrated about it.

So we invested a lot in ensuring that the new CI would be fast, efficient, and furious.

One cool thing we ended up doing was fast preview links on any commit pushed to the code repository. Using a combination of intelligent caching during builds and doing asynchronous tests when the commit is outside the normal release branches, we were able to shorten the deployment time to seconds.

This is the notification we get in our chat when anyone pushes code to any branch:

Anyone can follow a thread for a specific branch in the chat and get notified on new changes when they happen.

Blazing-fast builds, preview links and notifications are game-changers. An engineer can go from an idea or a quick fix to showing the result on a link to a product manager or another team member. Anyone can quickly click the link to see the changes on a fully working end-to-end version of Radar.

Cloudflare is committed to web accessibility. Recently we announced how we upgraded Cloudflare’s Dashboard to adhere to industry accessibility standards, but this premise is valid for all our properties. The same is true for localization. In 2020, we internationalized our Dashboard and added support for new languages and locales.

Accessibility and localization go hand in hand and are both important, but they are also different. The Web Content Accessibility Guidelines define many best practices around accessibility, including using color and contrast, tags, SVGs, shortcuts, gestures, and many others. The A11Y project page is an excellent resource for learning more.

Localization, though, also known as L10n, is more of a technical requirement when you start a new project. It's about making sure you choose the right set of libraries and frameworks that will make it easier to add new translations without engineering dependencies or code rewrites.

We wanted Radar to perform well on both fronts. Our design system takes Cloudflare's design and brand guidelines seriously and adds as many A11Y good practices as possible, and the app is fully aware of localization strings across its pages and UI components.

Adding a new language is as easy as translating a single JSON file. Here's a snippet of the en-US.json file with the default American English strings:

{
  "abbr.asn": "Autonomous System Number",
  "actions.chart.download.csv": "Download chart data in CSV",
  "actions.chart.download.png": "Download chart in PNG Format",
  "actions.chart.download.svg": "Download chart in SVG Format",
  "actions.chart.download": "Download chart",
  "actions.chart.maximize": "Maximize chart",
  "actions.chart.minimize": "Minimize chart",
  "actions.chart.share": "Share chart",
  "actions.download.csv": "Download CSV",
  "actions.download.png": "Download PNG",
  "actions.download.svg": "Download SVG",
  "actions.share": "Share",
  "alert.beta.link": "Radar Classic",
  "alert.beta.message": "Radar 2.0 is currently in Beta. You can still use {link} during the transition period.",
  "card.about.cloudflare.p1": "Cloudflare, Inc. ({website} / {twitter}) is on a mission to help build a better Internet. Cloudflare's suite of products protects and accelerates any Internet application online without adding hardware, installing software, or changing a line of code. Internet properties powered by Cloudflare have all web traffic routed through its intelligent global network, which gets smarter with every request. As a result, they see significant improvement in performance and a decrease in spam and other attacks. Cloudflare was named to Entrepreneur Magazine's Top Company Cultures 2018 list and ranked among the World's Most Innovative Companies by Fast Company in 2019.",
  "card.about.cloudflare.p2": "Headquartered in San Francisco, CA, Cloudflare has offices in Austin, TX, Champaign, IL, New York, NY, San Jose, CA, Seattle, WA, Washington, D.C., Toronto, Dubai, Lisbon, London, Munich, Paris, Beijing, Singapore, Sydney, and Tokyo.",
  "card.about.cloudflare.title": "About Cloudflare",
...

You can expect us to release Radar in other languages soon.

Radar Reports are documents that use data exploration and storytelling to analyze a particular theme in-depth. Some reports tend to get updates from time to time. Examples of Radar Reports are our quarterly DDoS Attack Trends, or the IPv6 adoption.

The source of these Reports is Jupyter Notebooks. Our Data Science team works on some use-case or themes with other stakeholders using our internal Jupyter Hub tool. After all the iteration and exploration are done, and the work is signed off, a notebook is produced.

A Jupyter Notebook is a JSON document containing text, source code, rich media such as images or charts, and other metadata. It is the de facto standard for presenting data science projects, and every data scientist uses it.

With Radar 1.0, converting a Jupyter Notebook to a Radar page was a lengthy and manual process implicating many engineering and design resources and causing much frustration to everyone involved. Even updating an already-published notebook would frequently cause trouble for us.

Radar 2.0 changed all of this. We now have a fully automated process that takes a Jupyter Notebook and, as long as it's designed using a list of simple rules and internal guidelines, converts it automatically, hosts the resulting HTML and assets in an R2 bucket, and publishes it on the Reports page.

The conversion to HTML takes into account our design system and UI components, and the result is a beautiful document, usually long-form, perfectly matching Radar's look and feel.

We will eventually open-source this tool so that anyone can use it.

We gave examples of using Cloudflare's products and features to build your next-gen app without worrying too much about things that aren't core to your business or logic. A few are missing, though.

Once the app is up and running, you must protect it from bad traffic and malicious actors. Cloudflare offers you DDoS, WAF, and Bot Management protection out of the box at a click's distance.

For example, here are some of our security rules. This is traffic we don't have to worry about in our app because Cloudflare detects it and acts on it according to our rules.

Another thing we don't need to worry about is redirects from the old site to the new one. Cloudflare has a feature called Bulk Redirects, where you can easily create redirect lists directly on the dashboard.

It's also important to mention that every time we talk about what you can do using our Dashboard, we're, in fact, also saying you can do precisely the same using Cloudflare's APIs. Our Dashboard is built entirely on top of them. And if you're the infrastructure as code kind of person, we have you covered, too; you can use the Cloudflare Terraform provider.

Deploying and managing Workers, R2 buckets, or Pages sites is obviously scriptable too. Wrangler is the command-line tool to do this and more, and it goes the extra mile to allow you to run your full app locally, emulating our stack, on your computer, before deploying.

We hope you enjoyed this Radar team write-up and were inspired to build your next app on top of our Supercloud. We will continue improving and innovating on Radar 2.0 with new features, share our findings and open-sourcing our tools with you.

In the meantime, we opened a Radar room on our Developers Discord Server. Feel free to join it and ask us questions; the team is eager to receive feedback and discuss web technology with you.

You can also follow us on Twitter for more Radar updates.