The road to a more standards-compliant Workers API

Earlier this year, we announced our participation in a new W3C Community Group for the advancement of Web-interoperable API standards. Since then, this new WinterCG has been hard at work identifying the common API standards around which all JavaScript runtimes can build. Today I just want to give a peek at some work the WinterCG has been doing; and show off some of the improvements we have been making in the Workers runtime to increase alignment with Web Platform standards around event handling, task cancellation using AbortController, text encoding and decoding, URL parsing and pattern matching, and streams support.

Right at the start of the WinterCG activity, the group took some time to evaluate and compare the various non-browser JavaScript runtimes such as Node.js, Deno, Bun, and Workers with the purpose of identifying the Web Platform APIs they all had in common. Following a very simple criteria, we looked at the standard APIs that were already implemented and supported by at least two of these runtimes and compiled those into a list that the WinterCG calls the "Minimum Common Web Platform API". This list will serve as the basis for what the community group defines as the minimum set of Web Platform APIs that should be implemented consistently across runtimes that claim to be "Web-interoperable".

The current list is straightforward:

AbortController	ReadableStreamDefaultController
AbortSignal	ReadableStreamDefaultReader
ByteLengthQueuingStrategy	SubtleCrypto
CompressionStream	TextDecoder
CountQueuingStrategy	TextDecoderStream
Crypto	TextEncoder
CryptoKey	TextEncoderStream
DecompressionStream	TransformStream
DOMException	TransformStreamDefaultController
Event	URL
EventTarget	URLPattern
ReadableByteStreamController	URLSearchParams
ReadableStream	WritableStream
ReadableStreamBYOBReader	WritableStreamDefaultController
ReadableStreamBYOBRequest

In addition to these, the WinterCG also expects Web-interoperable runtimes to have implementations of the atob(), btoa(), queueMicrotask(), structuredClone(), setTimeout(), clearTimeout(), setInterval(), clearInterval(), console, and crypto.subtle APIs available on the global scope.

Today, we are happy to say that the Workers runtime has compliant or nearly compliant implementations of every one of these WinterCG Minimum Common Web Platform APIs. Some of these APIs intentionally diverge from the standards either due to backwards compatibility concerns, Workers-specific features, or performance optimizations. Other APIs diverge still because we are still in the process of updating them to align with the specifications.

The Workers runtime has, from the beginning, had the mission to align its developer experience with JavaScript and Web Platform standards as much as possible. Over the past year we have worked hard to continue advancing that mission forward both by improving the standards-compliance of existing APIs such as Event, EventTarget, URL, and streams; and the introduction of new Web Platform APIs such as URLPattern, encoding streams, and compression streams.

The Workers runtime has provided an implementation of the Event and EventTarget Web Platform APIs from the very beginning. These were, however, only limited implementations of what the WHATWG DOM specification defines. Specifically, Workers had only implemented the bare minimum of the Event API that it itself needed to operate.

Today, the Event and EventTarget implementations in Workers provide a more complete implementation.

Let's look at the official definition of Event as defined by the WHATWG DOM standard:

[Exposed=*]
interface Event {
  constructor(DOMString type, optional EventInit eventInitDict = {});
 
  readonly attribute DOMString type;
  readonly attribute EventTarget? target;
  readonly attribute EventTarget? srcElement; // legacy
  readonly attribute EventTarget? currentTarget;
  sequence<EventTarget> composedPath();
 
  const unsigned short NONE = 0;
  const unsigned short CAPTURING_PHASE = 1;
  const unsigned short AT_TARGET = 2;
  const unsigned short BUBBLING_PHASE = 3;
  readonly attribute unsigned short eventPhase;
 
  undefined stopPropagation();
           attribute boolean cancelBubble; // legacy alias of .stopPropagation()
  undefined stopImmediatePropagation();
 
  readonly attribute boolean bubbles;
  readonly attribute boolean cancelable;
           attribute boolean returnValue;  // legacy
  undefined preventDefault();
  readonly attribute boolean defaultPrevented;
  readonly attribute boolean composed;
 
  [LegacyUnforgeable] readonly attribute boolean isTrusted;
  readonly attribute DOMHighResTimeStamp timeStamp;
 
  undefined initEvent(DOMString type, optional boolean bubbles = false, optional boolean cancelable = false); // legacy
};
 
dictionary EventInit {
  boolean bubbles = false;
  boolean cancelable = false;
  boolean composed = false;
};

Web Platform API specifications are always written in terms of a definition language called Web IDL. Every attribute defined in the interface is a property that is exposed on the object. Event objects, then, are supposed to have properties like type, target, srcElement, currentTarget, bubbles, cancelable, returnValue, defaultPrevented, composed, isTrusted, and timeStamp. They are also expected to have methods such as composedPath(), stopPropagation(), and stopImmediatePropagation(). Because most of these were not immediately needed by Workers, most were not provided originally.

Today, all standard, non-legacy properties and methods defined by the specification are available for use:

 const event = new Event('foo', {
    bubbles: false,
    cancelable: true,
    composed: true,
  });
 
  console.log(event.bubbles);
  console.log(event.cancelable);
  console.log(event.composed);
  
  addEventListener('foo', (event) => {
    console.log(event.eventPhase);  // 2 AT_TARGET
    console.log(event.currentTarget);
    console.log(event.composedPath());
  });
 
  dispatchEvent(event);

While we were at it, we also fixed a long standing bug in the implementation of Event that prevented user code from properly subclassing the Event object to create their own custom event types. This change is protected by a compatibility flag that is now enabled by default for all Workers using a compatibility date on or past 2022-01-31.

  class MyEvent extends Event {
    constructor() {
      super('my-event')
    }
 
    get type() { return super.type.toUpperCase() }
  }
 
  const myEvent = new MyEvent();
  // Previously, this would print "my-event" instead of "MY-EVENT" as expected.
  console.log(myEvent.type);

The EventTarget implementation has also been updated to support once handlers (event handlers that are triggered at-most once then automatically unregistered), cancelable handlers (using AbortSignal), and event listener objects, all in line with the standard.

Using a one-time event handler

  addEventListener('foo', (event) => {
    console.log('printed only once');
  }, {
    once: true
  });
 
  dispatchEvent(new Event('foo'));
  dispatchEvent(new Event('foo'));

Once handlers are key for preventing memory leaks in your applications when you know that a particular event is only ever going to be emitted once, or whenever you only care about handling it once. The stored reference to the function or object that is handling the event is removed immediately upon the first invocation, allowing the memory to be garbage collected.

Using a cancelable event handler

  const ac = new AbortController();
 
  addEventListener('foo', (event) => {
    console.log('not printed at all');
  }, {
    signal: ac.signal
  });
 
  ac.abort();
 
  dispatchEvent(new Event('foo'));

Using an event listener object

While passing a function to addEventListener() is the most common case, the standard actually allows an event listener to be an object with a handleEvent() method as well.

  const listener = {
    handleEvent(event) {
      console.log(event.type);
    }
  };
 
  addEventListener('foo', listener);
  addEventListener('bar', listener);
 
  dispatchEvent(new Event('foo'));
  dispatchEvent(new Event('bar'));

As illustrated in the cancelable event example above, we have also introduced an implementation of the AbortController and AbortSignal APIs into Workers. These provide a standard, and interoperable way of signaling cancellation of several kinds of tasks.

The AbortController/AbortSignal pattern is straightforward: An AbortSignal is just a type of EventTarget that will emit a single "abort" event when it is triggered:

  const ac = new AbortController();
 
  ac.signal.addEventListener('abort', (event) => {
    console.log(event.reason);  // 'just because'
  }, { once: true });
 
  ac.abort('just because');

The AbortController is used to actually trigger the abort event, optionally with a reason argument that is passed on to the event. The reason is typically an Error object but can be any JavaScript value.

The AbortSignal can only be triggered once, so the "abort" event should only ever be emitted once.

It is also possible to create AbortSignals that timeout after a specified period of time:

const signal = AbortSignal.timeout(10);

Or an AbortSignal that is pre-emptively triggered immediately on creation (these will never actually emit the "abort" event):

const signal = AbortSignal.abort('for reasons');

Currently, within Workers, AbortSignal and AbortController has been integrated with the EventTarget, fetch(), and streams APIs in alignment with the relevant standard specifications for each.

Using AbortSignal to cancel a fetch()

  const ac = new AbortController();
 
  const res = fetch('https://example.org', {
    signal: ac.signal
  });
 
  ac.abort(new Error('canceled'))
 
  try {
    await res;
  } catch (err) {
    console.log(err);
  }

The Workers runtime has long provided basic implementations of the TextEncoder and TextDecoder APIs. Initially, these were limited to only supporting encoding and decoding of UTF-8 text. The standard definition of TextDecoder, however, defines a much broader range of text encodings that are now fully supported by the Workers implementation. Per the standard, TextEncoder currently only supports UTF-8.

  const win1251decoder = new TextDecoder("windows-1251");
  const bytes = new Uint8Array([
    207, 240, 232, 226, 229, 242, 44, 32, 236, 232, 240, 33,
  ]);
  console.log(win1251decoder.decode(bytes)); // Привет, мир!

In addition to supporting the full range of encodings defined by the standard, Workers also now provides implementations of the TextEncoderStream and TextDecoderStream, which provide TransformStream implementations that apply encoding and decoding to streaming data:

  const { writable, readable } = new TextDecoderStream("windows-1251");
 
  const writer = writable.getWriter();
  writer.write(new Uint8Array([
    207, 240, 232, 226, 229, 242, 44, 32, 236, 232, 240, 33,
  ]));
 
  const reader = readable.getReader();
  const res = await reader.read();
  console.log(res.value); // Привет, мир!

Using the encoding streams requires the use of the transformstream_enable_standard_constructor compatibility flag.

Streaming compression and decompression is also now supported in the runtime using the standard CompressionStream and DecompressionStream APIs.

const ds = new DecompressionStream('gzip');
const decompressedStream = blob.stream().pipeThrough(ds);

const cs = new CompressionStream('gzip');
const compressedStream = blob.stream().pipeThrough(cs);

These are TransformStream implementations that fully conform to the standard definitions. Use of the compression streams does not require a compatibility flag to enable.

Similar to Event, there has been an implementation of the Web Platform standard URL API available within Workers from nearly the beginning. But also like Event, the implementation was not entirely compatible with the standard.

The incompatibilities were subtle, for instance, in the original implementation, the URL string "https://a//b//c//" would be parsed incorrectly as "https://a/b/c" (note that the extra empty path segments are removed) whereas the standard parsing algorithm would produce "https://a//b//c/" as a result. Such inconsistent results were causing interoperability issues with JavaScript written to run across multiple JavaScript runtimes and needed to be fixed.

A new implementation of the URL parsing algorithm has been provided, and as of October 31, 2022 it has been enabled by default for all newly deployed Workers. Older Workers can begin using the new implementation by updating their compatibility dates to 2022-10-31 or by enabling the url_standard compatibility flag.

Along with the updated URL implementation, Workers now provides an implementation of the standard URLPattern API.

URLPattern provides a regular-expression-like syntax for matching a URL string against a pattern. For instance, consider this example taken from the MDN documentation for URLPattern:

  // Matching a pathname
  let pattern1 = new URLPattern('https://example.com/books/:id')
  // same as
  let pattern2 = new URLPattern(
    '/books/:id',
    'https://example.com',
  );
  // or
  let pattern3 = new URLPattern({
    protocol: 'https',
    hostname: 'example.com',
    pathname: '/books/:id',
  });
  // or
  let pattern4 = new URLPattern({
    pathname: '/books/:id',
    baseURL: 'https://example.com',
  });

Last, but absolutely not least, our most significant effort over the past year has been providing new standards compliant implementations of the ReadableStream, WritableStream, and TransformStream APIs.

The Workers runtime has always provided an implementation of these objects but they were never fully conformant to the standard. User code was not capable of creating custom ReadableStream and WritableStream instances, and TransformStreams were limited to simple identity pass-throughs of bytes. The implementations have been updated now to near complete compliance with the standard (near complete because we still have a few edge cases and features we are working on).

The new streams implementation will be enabled by default in all new Workers as of November 30, 2022, or can be enabled earlier using the streams_enable_constructors and transformstream_enable_standard_constructor compatibility flags.

Creating a custom ReadableStream

async function handleRequest(request) {
  const enc = new TextEncoder();

  const rs = new ReadableStream({
    pull(controller) {
      controller.enqueue(enc.encode('hello world'));
      controller.close();
    }
  });

  return new Response(rs);
}

The new implementation supports both "regular" and "bytes" ReadableStream types, supports BYOB readers, and includes performance optimizations for both tee() and pipeThrough().

Creating a custom WritableStream

  const ws = new WritableStream({
    write(chunk) {
      console.log(chunk);  // "hello world"
    }
  });
 
  const writer = ws.getWriter();
  writer.write("hello world");

WritableStreams are fairly simple objects that can accept any JavaScript value written to them.

Creating a custom TransformStream

  const { readable, writable } = new TransformStream({
    transform(chunk, controller) {
      controller.enqueue(chunk.toUpperCase());
    }
  });
 
  const writer = writable.getWriter();
  const reader = readable.getReader();
 
  writer.write("hello world");
 
  const res = await reader.read();
  console.log(res.value);  // "HELLO WORLD"

It has always been possible in Workers to call new TransformStream() (with no arguments) to create a limited version of a TransformStream that only accepts bytes and only acts as a pass-through, passing the bytes written to the writer on to the reader without any modification.

That original implementation is now available within Workers using the IdentityTransformStream class.

  const { readable, writable } = new IdentityTransformStream();
 
  const writer = writable.getWriter();
  const reader = readable.getReader();
 
  const enc = new TextEncoder();
  const dec = new TextDecoder();
 
  writer.write(enc.encode("hello world"));
 
  const res = await reader.read();
  console.log(dec.decode(res.value));  // "hello world"

If your code is using new TransformStream() today as this kind of pass-through, the new implementation will continue to work except for one very important difference: the old, non-standard implementation of new TransformStream() supported BYOB reads on the readable side (i.e. readable.getReader({ mode: 'byob' })). The new implementation (enabled via a compatibility flag and becoming the default on November 30 ) does not support BYOB reads as required by the stream standard.

It is clear that we have made a lot of progress in improving the standards compliance of the Workers runtime over the past year, but there is far more to do. Next we will be turning our attention to the implementation of the fetch() and WebSockets APIs, as well as actively seeking closer alignment with other runtimes through collaboration in the Web-interoperable Runtimes Community Group.

If you are interested in helping drive the implementation of Web Platform APIs forward, and advancing interoperability between JavaScript runtime environments, the Workers Runtime team at Cloudflare is hiring! Reach out, or see our open positions here.