How Does Secondary DNS Work?

If you already understand how Secondary DNS works, please feel free to skip this section. It does not provide any Cloudflare-specific information.

Secondary DNS has many use cases across the Internet; however, traditionally, it was used as a synchronized backup for when the primary DNS server was unable to respond to queries. A more modern approach involves focusing on redundancy across many different nameservers, which in many cases broadcast the same anycasted IP address.

Secondary DNS involves the unidirectional transfer of DNS zones from the primary to the Secondary DNS server(s). One primary can have any number of Secondary DNS servers that it must communicate with in order to keep track of any zone updates. A zone update is considered a change in the contents of a  zone, which ultimately leads to a Start of Authority (SOA) serial number increase. The zone’s SOA serial is one of the key elements of Secondary DNS; it is how primary and secondary servers synchronize zones. Below is an example of what an SOA record might look like during a dig query.

example.com	3600	IN	SOA	ashley.ns.cloudflare.com. dns.cloudflare.com. 
2034097105  // Serial
10000 // Refresh
2400 // Retry
604800 // Expire
3600 // Minimum TTL

Each of the numbers is used in the following way:

1. Serial - Used to keep track of the status of the zone, must be incremented at every change.

2. Refresh - The maximum number of seconds that can elapse before a Secondary DNS server must check for a SOA serial change.

3. Retry - The maximum number of seconds that can elapse before a Secondary DNS server must check for a SOA serial change, after previously failing to contact the primary.

4. Expire - The maximum number of seconds that a Secondary DNS server can serve stale information, in the event the primary cannot be contacted.

5. Minimum TTL - Per RFC 2308, the number of seconds that a DNS negative response should be cached for.

Using the above information, the Secondary DNS server stores an SOA record for each of the zones it is tracking. When the serial increases, it knows that the zone must have changed, and that a zone transfer must be initiated.  

Serial increases can be detected in the following ways:

1. The fastest way for the Secondary DNS server to keep track of a serial change is to have the primary server NOTIFY them any time a zone has changed using the DNS protocol as specified in RFC 1996, Secondary DNS servers will instantly be able to initiate a zone transfer.

2. Another way is for the Secondary DNS server to simply poll the primary every “Refresh” seconds. This isn’t as fast as the NOTIFY approach, but it is a good fallback in case the notifies have failed.

One of the issues with the basic NOTIFY protocol is that anyone on the Internet could potentially notify the Secondary DNS server of a zone update. If an initial SOA query is not performed by the Secondary DNS server before initiating a zone transfer, this is an easy way to perform an amplification attack. There is two common ways to prevent anyone on the Internet from being able to NOTIFY Secondary DNS servers:

1. Using transaction signatures (TSIG) as per RFC 2845. These are to be placed as the last record in the extra records section of the DNS message. Usually the number of extra records (or ARCOUNT) should be no more than two in this case.

2. Using IP based access control lists (ACL). This increases security but also prevents flexibility in server location and IP address allocation.

Generally NOTIFY messages are sent over UDP, however TCP can be used in the event the primary server has reason to believe that TCP is necessary (i.e. firewall issues).

In addition to serial tracking, it is important to ensure that a standard protocol is used between primary and Secondary DNS server(s), to efficiently transfer the zone. DNS zone transfer protocols do not attempt to solve the confidentiality, authentication and integrity triad (CIA); however, the use of TSIG on top of the basic zone transfer protocols can provide integrity and authentication. As a result of DNS being a public protocol, confidentiality during the zone transfer process is generally not a concern.

AXFR is the original zone transfer protocol that was specified in RFC 1034 and RFC 1035 and later further explained in RFC 5936. AXFR is done over a TCP connection because a reliable protocol is needed to ensure packets are not lost during the transfer. Using this protocol, the primary DNS server will transfer all of the zone contents to the Secondary DNS server, in one connection, regardless of the serial number. AXFR is recommended to be used for the first zone transfer, when none of the records are propagated, and IXFR is recommended after that.

IXFR is the more sophisticated zone transfer protocol that was specified in RFC 1995. Unlike the AXFR protocol, during an IXFR, the primary server will only send the secondary server the records that have changed since its current version of the zone (based on the serial number). This means that when a Secondary DNS server wants to initiate an IXFR, it sends its current serial number to the primary DNS server. The primary DNS server will then format its response based on previous versions of changes made to the zone. IXFR messages must obey the following pattern:

1. Current latest SOA

2. Secondary server current SOA

3. DNS record deletions

4. Secondary server current SOA + changes

5. DNS record additions

6. Current latest SOA

Steps 2,3,4,5,6 can be repeated any number of times, as each of those represents one change set of deletions and additions, ultimately leading to a new serial.

IXFR can be done over UDP or TCP, but again TCP is generally recommended to avoid packet loss.

The DNS team loves microservice architecture! When we initially implemented Secondary DNS at Cloudflare, it was done using Mesos Marathon. This allowed us to separate each of our services into several different marathon apps, individually scaling apps as needed. All of these services live in our core data centers. The following services were created:

1. Zone Transferer - responsible for attempting IXFR, followed by AXFR if IXFR fails.

2. Zone Transfer Scheduler - responsible for periodically checking zone SOA serials for changes.

3. Rest API - responsible for registering new zones and primary nameservers.

In addition to the marathon apps, we also had an app external to the cluster:

1. Notify Listener - responsible for listening for notifies from primary servers and telling the Zone Transferer to initiate an AXFR/IXFR.

Each of these microservices communicates with the others through Kafka.

Figure 1: Secondary DNS Microservice Architecture‌‌

Once the zone transferer completes the AXFR/IXFR, it then passes the zone through to our zone builder, and finally gets pushed out to our edge at each of our 200 locations.

Although this current architecture worked great in the beginning, it left us open to many vulnerabilities and scalability issues down the road. As our Secondary DNS product became more popular, it was important that we proactively scaled and reduced the technical debt as much as possible. As with many companies in the industry, Cloudflare has recently migrated all of our core data center services to Kubernetes, moving away from individually managed apps and Marathon clusters.

What this meant for Secondary DNS is that all of our Marathon-based services, as well as our NOTIFY Listener, had to be migrated to Kubernetes. Although this long migration ended up paying off, many difficult challenges arose along the way that required us to come up with unique solutions in order to have a seamless, zero downtime migration.

Although the entire DNS team agreed that kubernetes was the way forward for Secondary DNS, it also introduced several challenges. These challenges arose from a need to properly scale up across many distributed locations while also protecting each of our individual data centers. Since our core does not rely on anycast to automatically distribute requests, as we introduce more customers, it opens us up to denial-of-service attacks.

The two main issues we ran into during the migration were:

1. How do we create a distributed and reliable system that makes use of kubernetes principles while also making sure our customers know which IPs we will be communicating from?

2. When opening up a public-facing UDP socket to the Internet, how do we protect ourselves while also preventing unnecessary spam towards primary nameservers?.

As was previously mentioned, one form of protection in the Secondary DNS protocol is to only allow certain IPs to initiate zone transfers. There is a fine line between primary servers allow listing too many IPs and them having to frequently update their IP ACLs. We considered several solutions:

1. Open source k8s controllers

2. Altering Network Address Translation(NAT) entries

3. Do not use k8s for zone transfers

4. Allowlist all Cloudflare IPs and dynamically update

5. Proxy egress traffic

Ultimately we decided to proxy our egress traffic from k8s, to the DNS primary servers, using static proxy addresses. Shadowsocks-libev was chosen as the SOCKS5 implementation because it is fast, secure and known to scale. In addition, it can handle both UDP/TCP and IPv4/IPv6.

Figure 2: Shadowsocks proxy Setup

The partnership of k8s and Shadowsocks combined with a large enough IP range brings many benefits:

1. Horizontal scaling

2. Efficient load balancing

3. Primary server ACLs only need to be updated once

4. It allows us to make use of kubernetes for both the Zone Transferer and the Local ShadowSocks Proxy.

5. Shadowsocks proxy can be reused by many different Cloudflare services.

The Notify Listener requires listening on static IPs for NOTIFY Messages coming from primary DNS servers. This is mostly a solved problem through the use of k8s services of type loadbalancer, however exposing this service directly to the Internet makes us uneasy because of its susceptibility to attacks. Fortunately DDoS protection is one of Cloudflare's strengths, which lead us to the likely solution of dogfooding one of our own products, Spectrum.

Spectrum provides the following features to our service:

1. Reverse proxy TCP/UDP traffic

2. Filter out Malicious traffic

3. Optimal routing from edge to core data centers

4. Dual Stack technology

Figure 3: Spectrum interaction with Notify Listener

Figure 3 shows two interesting attributes of the system:

1. Spectrum <-> k8s IPv4 only:

2. This is because our custom k8s load balancer currently only supports IPv4; however, Spectrum has no issue terminating the IPv6 connection and establishing a new IPv4 connection.

3. Spectrum <-> k8s routing decisions based of L4 protocol:

4. This is because k8s only supports one of TCP/UDP/SCTP per service of type load balancer. Once again, spectrum has no issues proxying this correctly.

One of the problems with using a L4 proxy in between services is that source IP addresses get changed to the source IP address of the proxy (Spectrum in this case). Not knowing the source IP address means we have no idea who sent the NOTIFY message, opening us up to attack vectors. Fortunately, Spectrum’s proxy protocol feature is capable of adding custom headers to TCP/UDP packets which contain source IP/Port information.

As we are using miekg/dns for our Notify Listener, adding proxy headers to the DNS NOTIFY messages would cause failures in validation at the DNS server level. Alternatively, we were able to implement custom read and write decorators that do the following:

1. Reader: Extract source address information on inbound NOTIFY messages. Place extracted information into new DNS records located in the additional section of the message.

2. Writer: Remove additional records from the DNS message on outbound NOTIFY replies. Generate a new reply using proxy protocol headers.

There is no way to spoof these records, because the server only permits two extra records, one of which is the optional TSIG. Any other records will be overwritten.

Figure 4: Proxying Records Between Notifier and Spectrum‌‌

This custom decorator approach abstracts the proxying away from the Notify Listener through the use of the DNS protocol.  

Although knowing the source IP will block a significant amount of bad traffic, since NOTIFY messages can use both UDP and TCP, it is prone to IP spoofing. To ensure that the primary servers do not get spammed, we have made the following additions to the Zone Transferer:

1. Always ensure that the SOA has actually been updated before initiating a zone transfer.

2. Only allow at most one working transfer and one scheduled transfer per zone.

As shown in figure 1, there are several ways of sending Kafka messages to the Zone Transferer in order to initiate a zone transfer. There is no benefit in having a large backlog of zone transfers for the same zone. Once a zone has been transferred, assuming no more changes, it does not need to be transferred again. This means that we should only have at most one transfer ongoing, and one scheduled transfer at the same time, for any zone.

If we want to limit our number of scheduled messages to one per zone, this involves ignoring Kafka messages that get sent to the Zone Transferer. This is not as simple as ignoring specific messages in any random order. One of the benefits of Kafka is that it holds on to messages until the user actually decides to acknowledge them, by committing that messages offset. Since Kafka is just a queue of messages, it has no concept of order other than first in first out (FIFO). If a user is capable of reading from the Kafka topic concurrently, it is entirely possible that a message in the middle of the queue be committed before a message at the end of the queue.

Most of the time this isn’t an issue, because we know that one of the concurrent readers has read the message from the end of the queue and is processing it. There is one Kubernetes-related catch to this issue, though: pods are ephemeral. The kube master doesn’t care what your concurrent reader is doing, it will kill the pod and it’s up to your application to handle it.

Consider the following problem:

Figure 5: Kafka Partition‌‌

1. Read offset 1. Start transferring zone 1.

2. Read offset 2. Start transferring zone 2.

3. Zone 2 transfer finishes. Commit offset 2, essentially also marking offset 1.

4. Restart pod.

5. Read offset 3 Start transferring zone 3.

If these events happen, zone 1 will never be transferred. It is important that zones stay up to date with the primary servers, otherwise stale data will be served from the Secondary DNS server. The solution to this problem involves the use of a list to track which messages have been read and completely processed. In this case, when a zone transfer has finished, it does not necessarily mean that the kafka message should be immediately committed. The solution is as follows:

1. Keep a list of Kafka messages, sorted based on offset.

2. If finished transfer, remove from list:

3. If the message is the oldest in the list, commit the messages offset.

Figure 6: Kafka Algorithm to Solve Message Loss

This solution is essentially soft committing Kafka messages, until we can confidently say that all other messages have been acknowledged. It’s important to note that this only truly works in a distributed manner if the Kafka messages are keyed by zone id, this will ensure the same zone will always be processed by the same Kafka consumer.

Although Cloudflare has a large global network, as shown above, the zone transferring process does not take place at each of the edge datacenter locations (which would surely overwhelm many primary servers), but rather in our core data centers. In this case, how do we propagate to our edge in seconds? After transferring the zone, there are a couple more steps that need to be taken before the change can be seen at the edge.

1. Zone Builder - This interacts with the Zone Transferer to build the zone according to what Cloudflare edge understands. This then writes to Quicksilver, our super fast, distributed KV store.

2. Authoritative Server - This reads from Quicksilver and serves the built zone.

Figure 7: End to End Secondary DNS‌‌

At the time of writing this post, according to dnsperf.com, Cloudflare leads in global performance for both Authoritative and Resolver DNS. Here, Secondary DNS falls under the authoritative DNS category here. Let’s break down the performance of each of the different parts of the Secondary DNS pipeline, from the primary server updating its records, to them being present at the Cloudflare edge.

1. Primary Server to Notify Listener - Our most accurate measurement is only precise to the second, but we know UDP/TCP communication is likely much faster than that.

2. NOTIFY to Zone Transferer - This is negligible

3. Zone Transferer to Primary Server - 99% of the time we see ~800ms as the average latency for a zone transfer.

Figure 8: Zone XFR latency

4. Zone Transferer to Zone Builder - 99% of the time we see ~10ms to build a zone.

Figure 9: Zone Build time

5. Zone Builder to Quicksilver edge: 95% of the time we see less than 1s propagation.

Figure 10: Quicksilver propagation time

End to End latency: less than 5 seconds on average. Although we have several external probes running around the world to test propagation latencies, they lack precision due to their sleep intervals, location, provider and number of zones that need to run. The actual propagation latency is likely much lower than what is shown in figure 10. Each of the different colored dots is a separate data center location around the world.

Figure 11: End to End Latency

An additional test was performed manually to get a real world estimate, the test had the following attributes:

Primary server: NS1Number of records changed: 1Start test timer event: Change record on NS1Stop test timer event: Observe record change at Cloudflare edge using digRecorded timer value: 6 seconds

Cloudflare serves 15.8 trillion DNS queries per month, operating within 100ms of 99% of the Internet-connected population. The goal of Cloudflare operated Secondary DNS is to allow our customers with custom DNS solutions, be it on-premise or some other DNS provider, to be able to take advantage of Cloudflare's DNS performance and more recently, through Secondary Override, our proxying and security capabilities too. Secondary DNS is currently available on the Enterprise plan, if you’d like to take advantage of it, please let your account team know. For additional documentation on Secondary DNS, please refer to our support article.

Ulaanbaatar (or Ulan Bator; but shortened to UB by many) is the capital of Mongolia and located nearly a mile above sea level just outside the Gobi Desert - a desert that spans a good percentage of Central Asia’s Mongolia. (The rest of the Gobi Desert extends into China). The country is nestled squarely between Russia to the north and China to the south. It’s also home to some of the richest and ancient customs and festivals around. It’s those festivals that successfully draw in the tourists who want to experience something quite unique. Luckily, even with all the tourists, Mongolia has managed to keep its local customs; both in the cities and within its nomadic tribes.

via Wikipedia

History also has drawn explorers and conquerors to and from the region; but more on that later.

Any avid reader of our blogs will know that we frequently explain that the expansion of our network provides customers and end-users with both more capacity and less latency. That goal (covering 95% of the Internet population with 10 milliseconds or less of latency) means that Mongolia was seriously on our radar.

Now we have a data center in Ulaanbaatar, Mongolia, latency into that blue sky country is significantly reduced. Prior to this data center going live we were shipping bits into the country via Hong Kong, a whopping 1,800 miles away (or 50 milliseconds if we talk latency). That's far! We know this new data center is a win-win for both mobile and broadband customers within the country and for Cloudflare customers as a whole.

Ulaanbaatar is city number 154 on Cloudflare’s network. Our expansion into cities like Ulaanbaatar doesn’t just happen instantly; it takes many teams within Cloudflare in order to successfully deploy in a place like this.

However, before deploying, we need to negotiate a place to deploy into. A new city requires a secure data center for us to build into. A bandwidth partner is also required. We need to get access to the local networks and to also acquire cache-fill bandwidth in order to operate our CDN. Once we have those items negotiated, we can focus on the next steps. Any site we build has to match our own stringent security standards (we are PCI/DSS compliant – hence all our data centers need to also be PCI/DSS compliant). That’s a paperwork process, which surprisingly takes longer than most other stages (because we care about security).

Then logistics kicks in. A BOM (Bill of Materials) is created. Serial numbers recorded. Power plugs chosen. Fun fact: Cloudflare data centers are nearly all identical, except the power cables. While we live in a world where fiber optic cables and connectors are universal, independent of location (or speed in some cases), the power connections we receive for our data centers vary widely as we expand around the globe.

The actual shipping is possibly the more interesting part of the logistics process. Getting a few pallets of hardware strapped up and ready to move is only a small part of the process. Paperwork again becomes the all-powerful issue. Each country has its own customs and import process, each shipping company has its own twist on how to process things, and each shipment needs to be coordinated with a receiving party. Our logistics team pulls off this miracle for new sites, upgrades for existing sites, replacement parts for existing sites, all while sometimes calmly listening to mundane on-hold music from around the globe.

Then our hardware arrives! Seriously, this is a biggie and those around the office that follow these new city builds are always celebrating on those days. The logistics team has done their part; now it’s time for the deployment team to kick-in.

The deployment team’s main goal is to get hardware racked-and-stacked in a site where (in most cases) we are contracting out the remote-hands to do this work. Sometimes we send people to a site to build it; however, that’s not a scalable process and hence we use local remote-hands contractors to do this heavy-lifting and cabling. There are diagrams, there are instructions, there are color-coded cables (‘cause the right cable needs to go into the right socket). Depending on the size of the build; it can be just a days work or up-to a weeks worth of work. We vary our data center sizes based on the capacity needs for each city. Once racked-and-stacked there is one more job to get done within the Infrastructure team. They get the initial private network connection enabled and secured. That single connection provides us with the ability to continue to the next step. Actually setting up the network and servers at the new site.

Every new data center site is shipped with zero configuration loaded into network hardware and compute servers. They all ship raw with no Cloudflare knowledge embedded into them. This is by design. The network teams first goal is to configure the routers and switches, which is mainly a bootstrap process in order for the hardware to phone-home and securely request its full configuration setup. We have previously written about our extensive network automation methods. In the case of a new site, it’s not that different. Once the site can communicate back home, it’s clear to the software base that its configuration is out of date. Updates are pushed to the site and network monitoring is automatically enabled.

But let's not paint a perfect rosy picture. There can still be networking issues. Just one is worth pointing-out as it’s a recurring issue and one that plagues the industry globally. Fiber optic cables can sometimes be plugged in with their receive and transmit sides reversed. It’s a 50:50 chance of being right. Sometimes it just amazes us that we can’t get this fixed; but … a quick swap of the two connectors and we are back in business!

No conversation about Mongolia would be valid unless we discuss Genghis Khan. Back in the 13th century, Genghis Khan founded the Mongol Empire. He unified the tribes of what is now Mongolia (and beyond). He established the largest land empire in history and is well described both online and via various TV documentaries (The History Channel doesn’t skimp when it comes to covering him). Genghis Khan was a name of honor that he didn’t receive till 1206. Before that he was just named “Temujin”.

Photo by SarahTz CC by/2.0

Around 30 miles outside Ulaanbaatar is the equestrian statue of Genghis Khan on the banks of the Tuul River in Gorkhi Terelj National Park. Pictured above, this statue is 131 feet tall and built from 250 tons of steel.

We get to announce our new city (and country) data center during a very special time. The Golden Eagle Festival takes place during the first weekend of October (that’s October 6 and 7 this year). It’s a test of a hunter’s speed and agility. In this case the hunters (nomads of Mongolia) are practicing an ancient technique of using Golden Eagles to hunt. It takes place in the far west of the country in the Altai Mountains.

The most famous festival in Mongolia is the Naadam Festival in mid-July. So many things going on within that festival, all of which is a celebration of Mongolian and nomadic culture. The festival celebrates the local sports (wrestling, archery, and horse racing) along with plenty of arts. The opening ceremony can be quite elaborate.

When discussing Mongolia, travelers nearly always want to make sure their itineraries overlap at least one of these festivals!

Last week was Cloudflare’s Birthday Week and we announced many services and products. Rest-assured that everything we announced is instantly available in Ulaanbaatar, Mongolia (data center city 154) just like it’s available everywhere else on Cloudflare’s network.

One final trivia point regarding our Ulaanbaatar data center. With Ulaanbaatar live, we now have datacenters covering the letters A thru Z (from Amsterdam to Zurich), i.e. with U added, every letter is now fully represented.

If you like what you’ve read and want to come join our infrastructure team, our logistics team, our network team, our SRE team, or any of the teams that help with these global expansions, then we would love to see your resume or CV. Look here for job opening.

CC-BY 2.0 image by Steve Jurvetson

Today Cloudflare’s DNS service answers around 1 million queries per second – not including attack traffic – via a global anycast network. Naturally as a growing startup, the technology we used to handle tens or hundreds of thousands of zones a few years ago became outdated over time, and couldn't keep up with the millions we have today. Last year we decided to replace two core elements of our DNS infrastructure: the part of our DNS server that answers authoritative queries and the data pipeline which takes changes made by our customers to DNS records and distributes them to our edge machines across the globe.

The rough architecture of the system can be seen above. We store customer DNS records and other origin server information in a central database, convert the raw data into a format usable by our edge in the middle, and then distribute it to our >100 data centers (we call them PoPs - Points of Presence) using a KV (key/value) store.

The queries are served by a custom DNS server, rrDNS, that we’ve been using and developing for several years. In the early days of Cloudflare, our DNS service was built on top of PowerDNS, but that was phased out and replaced by rrDNS in 2013.

The Cloudflare DNS team owns two elements of the data flow: the data pipeline itself and rrDNS. The first goal was to replace the data pipeline with something entirely new as the current software was starting to show its age; as any >5 year old infrastructure would. The existing data pipeline was originally built for use with PowerDNS, and slowly evolved over time. It contained many warts and obscure features because it was built to translate our DNS records into the PowerDNS format.

In the old system, the data model was fairly simple. We’d store the DNS records roughly in the same structure that they are represented in our UI or API: one entry per resource record (RR). This meant that the data pipeline only had to perform fairly rudimentary encoding tasks when generating the zone data to be distributed to the edge.

Zone metadata and RRs were encoded using a mix of JSON and Protocol Buffers, though we weren’t making particularly good use of the schematized nature of the protocols so the schemas were very bloated and the resulting data ended up being larger than necessary. Not to mention that as the number of total RRs in our database headed north of 100 million, these small differences in encoding made a significant difference in aggregate.

It’s worth remembering here that DNS doesn’t really operate on a per-RR basis when responding to queries. You query for a name and a type (e.g example.com and AAAA) and you’ll be given an RRSet which is a collection of RRs. The old data format had RRSets broken out into multiple RR entries (one key per record) which typically meant multiple roundtrips to our KV store to answer a single query. We wanted to change this and group data by RRSet so that a single request could be made to the KV store to retrieve all the data needed to answer a query. Because Cloudflare optimizes heavily for DNS performance, multiple KV lookups were limiting our ability to make rrDNS go as fast as possible.

In a similar vein, for lookups like A/AAAA/CNAME we decided to group the values into a single “address” key instead of one key per RRset. This further avoids having to perform extra lookups in the most common cases. Squishing keys together also helps reduce memory usage of the cache we use in front of the KV store, since we’re storing more information against a single cache key.

After settling on this new data model, we needed to figure out how to serialize the data and pass it to the edge. As mentioned, we were previously using a mix of JSON and Protocol Buffers, and we decided to replace this with a purely MessagePack-based implementation.

MessagePack is a binary serialization format that is typed, but does not have a strict schema built into the format. In this regard, it can be considered a little like JSON. For both the reader and the writer, extra fields can be present or absent and it’s up to your application code to compensate.

In contrast, Protocol Buffers (or other formats like Cap’n Proto) require a schema for data structures defined in a language agnostic format, and then generate code for the specific implementation. Since DNS already has a large structured schema, we didn’t want to have to duplicate all of this schema in another language and then maintain it. In the old implementation with Protocol Buffers, we’d not properly defined schemas for all DNS types – to avoid this maintenance overhead – which resulted in a very confusing data model for rrDNS.

When looking for new formats we wanted something that would be fast, easy to use and that could integrate easily into the code base and libraries we were already using. rrDNS makes heavy use of the miekg/dns Go library which uses a large collection of structs to represent each RR type, for example:

type SRV struct {
	Hdr      RR_Header
	Priority uint16
	Weight   uint16
	Port     uint16
	Target   string `dns:"domain-name"`
}

When decoding the data written by our pipeline in rrDNS we need to convert the RRs into these structs. As it turns out, the tinylib/msgp library we had been investigating has a rather nice set of code generation tools. This would allow us to auto-generate efficient Go code from the struct definitions without having to maintain another schema definition in another format.

This meant we could work with the miekg RR structs (of which we are already familiar with from rrDNS) in the data pipeline, serialize them straight into binary data, and then deserialize them again at the edge straight into a struct we could use. We didn't need to worry about mapping from one set of structures to another using this technique, which simplified things greatly.

MessagePack also performs incredibly well compared to other formats on the market. Here’s an excerpt from a Go serialization benchmarking test; we can see that on top of the other reasons MessagePack benefits our stack, it outperforms pretty much every other viable cross-platform option.

One unexpected surprise after switching to this new model was that we actually reduced the space required to store the data at the edge by around 9x, which was a significantly higher saving compared to our initial estimates. It just goes to show how much impact a bloated data model can have on a system.

Another very important feature of Cloudflare’s DNS is our ability to propagate zone changes around the globe in a matter of seconds, not minutes or hours. Our existing pipeline was struggling to keep up with the growing number of zones, and with changes to at least 5 zones each second, even at the quietest of times we needed something new.

For a while now we’ve had this monitoring, and we are able to visualize propagation times across the globe. The graph below is taken from our end-to-end monitoring: it makes changes to DNS via our API and watches for the change from various probes around the world. Each dot on the graph represents a particular probe talking to one of our PoPs, and the delay is tracked as the time it took for a change made via our API to be visible externally.

Due to various layers of caches – both inside and outside of our control – we see some banding on 10s intervals under 1 minute, and it fluctuates all the time. For monitoring and alerting of this nature, the granularity we have here is sufficient but it’s something we’d definitely like to improve. In normal operation, new DNS data is actually available to 99% of our global PoPs in under 5s.

In this time frame we can see there were a couple of incidents where delays of a few minutes were visible for a small number of PoPs due to network connectivity, but generally all probes reported stable propagation times.

In contrast, here’s a graph of the old data pipeline for the same period. We can see how the graph represents the growing delay in visible changes for all PoPs at any given time.

With a new data model designed and ready to go, one that better matched our query patterns, we set out implementing a new service to pick up changes to our zones in the central data store, do any needed processing and send the resulting output to our KV store.

The new service (written in our favourite language Go) has been running in production since July 2016, and we’ve now migrated over 99% of Cloudflare customer zones over to it. If we exclude incidents where issues with congestion across the internet affect connectivity to or from a particular location, the new pipeline itself has experienced zero delays thus far.

rrDNS is a modular application, which allows us to write different “filters” that can hand off processing of different types of queries to different code. The Authoritative filter is responsible for taking an incoming DNS query, looking up the zone the query name belongs to, and performing all relevant logic to find the RRSet to send back to the client.

Since we’ve completely revised the underlying DNS data model at our edge, we needed to make significant changes to the “Authoritative Filter” in rrDNS. This too is an old area of the code base that hasn’t significantly changed in a number of years. As with any ageing code base, this brings a number of challenges, so we opted to re-write the filter completely. This allowed us to redesign it from the ground up on our new data model, keeping a keen eye on performance, and to better suit the scale and shape of our DNS traffic today. Starting fresh also made it much easier to build in good development practices, such as high test coverage and better documentation.

We’ve been running the v2 version of the authoritative filter in production alongside the existing code since the later months of 2016, and it has already played a key role in the DNS aspects of our new load balancing product.

The results with the new filter have been great: we’re able to respond to DNS queries on average 3x faster than before, which is excellent news for our customers and improves our ability to mitigate large DNS attacks. We can see here that as the percentage of zones migrated increased, we saw a significant improvement in our average response time.

The most time consuming part of the project was migrating customers from the old system to something entirely new, without impacting customers or anybody noticing what we were doing. Achieving this involved a significant effort from variety of people in our customer facing, support and operations teams. Cloudflare has many offices in different time zones – London, San Francisco, Singapore and Austin – so keeping everyone in sync was key to our success.

Already, as a part of the release process for rrDNS we automatically sample and replay production queries against existing and upcoming code to detect unexpected differences, so naturally we decided to extend this idea for our migration. For any zone to pass the migration test, we compared the possible answers for the entire zone from the old system and the new system. Just one failure would result in the tool skipping the zone.

This allowed us to iteratively test the migration of zones and fix issues as they arose, keeping releases simple and regular. We chose not to do a single – and very scary – switch away from the old system, but run them both in parallel and slowly move zones over keeping them both in sync. Meaning we quickly could migrate zones back in case something unexpected happened.

Once we got going we were safely migrating zones at several hundred thousand per day, and we kept a close eye on how far we were from our initial goal of 99%. The last mile is still in progress, as there is often an element of customer engagement for some complex configurations that need attention.

Replacing a piece of infrastructure this core to Cloudflare took significant effort from a large variety of teams. So what did we gain?

• Average of 3x performance boost in code handling DNS queries

• Faster and more consistent updates to DNS data around the globe

• A much more robust system for SREs to operate and engineers to maintain

• Consolidated feature-set based on today’s requirements, and better documentation of edge case behaviours

• More test coverage, better metrics and higher confidence in our code, making it safer to make changes and develop our DNS products

Now that we’re now able to process our customers DNS more quickly, we’ll soon be rolling out support for a few new RR types and some other exciting new things in the coming months.

Does solving these kinds of technical and operational challenges excite you? Cloudflare is always hiring for talented specialists and generalists within our Engineering, Technical Operations and other teams.

Come learn about Cloudflare’s recent partnership with Google Cloud Platform (GCP) through their CDN Interconnect Program. Cloudflare offers performance and security to over 25,000 Google Cloud Platform customers. The CDN Interconnect program allows Cloudflare’s servers to establish high-speed interconnections with Google Cloud Platform at various locations around the world, accelerating dynamic content while reducing bandwidth and egress billing costs.

We’ll be at booth C7 discussing the benefits of Cloudflare, our partnership with Google Cloud Platform, and handing out Cloudflare SWAG. In addition, our Co-Founder, Michelle Zatlyn, will be presenting “A Cloud Networking Blueprint for Securing Your Workloads” on Thursday, March 9th from 11:20 AM to 12:20 PM at Moscone West, Room 2005.

Google Cloud Platform’s CDN Interconnect program allows select CDN providers to establish direct interconnect links with Google’s edge network at various locations. Customers egressing network traffic from Google Cloud Platform through one of these links will benefit from the direct connectivity to the CDN providers and will be billed according to the lower Google Cloud Interconnect pricing.

Joint customers of Cloudflare and Google Cloud Platform can expect bandwidth savings of up to 75% and receive discounted egress pricing. Egress traffic is traffic flowing from Google Cloud Platform servers to Cloudflare’s servers. The high-speed interconnections between GCP and Cloudflare speed up the delivery of dynamic content for visitors.

As part of this program, 41 Cloudflare data centers are directly connected to Google Cloud Platform’s infrastructure. When one of these Cloudflare data centers requests content from a Google Cloud Platform origin, it’s routed through a high-performance interconnect instead of the public Internet. This dramatically reduces latency for origin requests, and it also enables discounted Google Cloud Platform egress pricing in the US, Europe and Asia regions.

Quizlet and Discord, two prominent joint customers of Cloudflare and Google Cloud Platform, have shared their performance, security, and cost-savings stories.

Discord is a free voice and text chat app designed specifically for gaming. In one year, Discord grew from 25,000 concurrent users to 2.4 million, a 9,000 percent growth. Discord’s 25 million registered users send 100 million messages per day across the platform, requiring a global presence with tremendous amounts of network throughput. As Discord experiences explosive growth, they're thankful Cloudflare helps keep bandwidth & hardware costs down and web performance high.

• Saving $100,000 on annual hardware costs

• Saving $100,000 monthly on Google Cloud Network Egress bill

• Secure traffic even with spikes of websockets events up to 2 million/second

Learn more about Discord’s use of Cloudflare on Google Cloud Platform: https://www.cloudflare.com/case-studies/discord

Quizlet is the world’s largest student and teacher online learning community. Every month, over 20 million active learners from 130 countries practice and master more than 140 million study sets of content on every conceivable subject and topic. Quizlet’s Alexa ranking is 588 globally, and 104 in the United States, ranking it as one of the most highly-trafficked websites.

Quizlet receives performance and security benefits, while saving more than 50 percent on their Google Cloud networking egress bill by using Cloudflare.

• Save 50% on monthly Google Cloud Network Egress Bill

• Reduced daily bandwidth use by 76 percent reduction (or over 10 Tb)

Learn more about Quizlet’s use of Cloudflare on Google Cloud Platform:https://www.cloudflare.com/case-studies/quizlet

Cloudflare’s Co-Founder, Michelle Zatlyn, will be presenting alongside Google and Palo Alto Networks, in a talk titled “A Cloud Networking Blueprint for Securing Your Workloads”.

Thursday, March 9th | 11:20 AM - 12:20 PM | Moscone West, Room 2005

Securing your workloads in the cloud requires shifting away from the traditional “perimeter” security to a “pervasive, hierarchical, scalable” security model. In this session, we discuss cloud networking best practices for securing enterprise and cloud-native workloads on Google Cloud Platform. We describe a network security blueprint that covers securing your virtual networks (VPCs), DDoS protection, using third-party security appliances and services, and visibility and analytics for your deployments. We also highlight Google’s experiences in delivering its own services securely and future trends in cloud network security.

For about half the world (and about half of CloudFlare's data centers) it's already 2013. As our team (most of whom are in San Francisco) get ready to celebrate New Year's Eve, wanted to quickly look back on CloudFlare's 2012. Here are some stats that tell the story of our last year:

• Page views served by CloudFlare in 2012: 679,237,127,874

• Hits served via CloudFlare's network in 2012: 3,691,532,490,107

• Bandwidth served from CloudFlare's network in 2012: 765 Petabytes

• Bandwidth we saved our customers in 2012: 436 Petabytes

• New sites that signed up for CloudFlare in 2012: 573,177

• Threats stopped by CloudFlare in 2012: 281,701,624,076

• New CloudFlare data centers added in 2012: 10

Over 2012, we saw more than 720 million unique IPs connect to CloudFlare's network. Our best estimate is that behind each of those IPs there are 1.8 Internet users. In other words, we saw approximately 1.3 billion Internet users pass through CloudFlare's network in 2012. That's well over half of the Internet's total population of users.

We also saved a ton of time that those Internet users would have otherwise spent waiting for websites to load. If you add up all the time that people would have spent waiting for websites to load had CloudFlare not existed in 2012, you get more than 891 lifetimes worth of time saved. We're really proud of that.

We have a number of improvements, new features, new data centers, and other surprised lined up for 2013. From everyone at CloudFlare, Happy New Year! Here's to an even faster, safer Internet in the year ahead.

Today, CloudFlare adds more than 250 new customers every six hours or so, but getting our first 250 took several months and a lot of faith from our earliest adopters. When we started working on CloudFlare, an employee at a major CDN company warned us that we had no idea all the crazy things people did with their websites. He wasn't kidding. For the first sites that signed up, we usually made them slower and offered little additional protection. But, over time, and with the patience of our first users, we incorporated everything we learned from each new site and built something great.

The core value proposition of CloudFlare has always been that the system gets smarter and faster with each new website that joins. In that sense, CloudFlare is a community. When an attack is launched against any one site, knowledge about that attack is immediately shared across the rest of the network. Similarly, we use data from the performance of sites on CloudFlare to help tune optimizations for each new site that joins.

Today CloudFlare's community is made up of hundreds of thousands of sites, and each new site that joins continues to make the system better. Together we have brought the resources previously reserved only for the Internet giants to the rest of the web, and we've grown into a giant ourselves. We now power more page views per month than Twitter, Amazon.com, Wikipedia, Zynga, AOL, Apple, and Bing — combined. We have big plans for tomorrow and ways we are continuing to work to save the web, but we'll always remember that we couldn't have done it without you.

From the whole CloudFlare team, thank you!

P.S. - Want to have an even bigger impact? We're always hiring and we usually get our best candidates from our existing users.

A few weeks ago the Wall Street Journal named CloudFlare the most innovative Networking & Internet Technology of 2011. We were pretty flattered given that winners from other categories included IBM for Watson (the Jeopardy-playing computer), a company that figured out how to create stem cells from any cell in your body, and another that used algae to produce oil and diesel fuels cheaper than drilling for fossil fuels.

On Tuesday, I traveled down to Redwood City to receive the award at the FASTech conference and pose for this picture which, I'm sure, will cause my mom to write in (probably to customer support) and tell me I need a haircut.

Every day Ian on our team pulls a "stat of the day." He asks the rest of us to guess the answer. Today's was: "How much has CloudFlare grown since January 1, 2011?" It's pretty incredible to realize the answer: our traffic has grown by more than 3,200% since January. And we're not slowing down. In the next few days, we'll cross having saved our users more than 2 petabytes of bandwidth. And, as we just got finished discussing during our Friday all-team B.E.E.R. meeting, if you knew what we had planned in 2012 you'd realize: we're just getting started.

Want to be a part of the ride? We're hiring.

We're ending the week on a high. About an hour ago we crossed 10 billion page views having been powered by CloudFlare over the last 30 days. Those pages were served to about 250 million unique visitors. To put it in perspective, that means more than 13% of worldwide Internet visitors passed through our network at least once in the last month. That's almost 100 million more unique visitors than Twitter, and more than 3 billion more page views than Wikipedia, over the same period. It's also a lot more than we were doing just a month ago ourselves. We're now averaging over 25,000 requests per second. All I can say is: Wow. Great job team!

Kevin, the visual designer who recently joined CloudFlare's team, pulled together this graph the other day. Even having lived through the last 10 months, it's still incredible to see them represented here. We reached 100 million people experiencing the benefits of CloudFlare's network about seven months after our launch. After that, it took us only three more months to cross 200 million. And our growth has continued to accelerate: as of this morning, just a couple weeks after crossing 200 million, more than 230 million people every month now experience a faster, safer Internet because of CloudFlare.

One thing that I find particularly interesting about this graph is a similarity between most of these fast-growing companies. With the notable exception of Zynga (and to a lesser extend Yahoo) they have all built platforms that enhance other people's content. YouTube allowed anyone to publish their own videos. Similarly, Twitter, LinkedIn, and Facebook let anyone share their own personal and professional details with a broad audience. In some ways, Google's platform is backward of the others -- taking other people's content and overlaying an index to make it searchable -- but the core idea of making other people's content better is the same.

Several months ago, I was talking with one of the famous, gray-haired engineers of Silicon Valley and he said, "What CloudFlare really is is the YouTube for websites." I have to confess that didn't entirely click for me until Kevin showed me this graph. There were ways to publish videos online prior to YouTube, but the company made doing so faster, easier, and better. There are ways to publish websites prior toCloudFlare, but what we've built is building a transparent platform to make any website faster, safer, and better.

This is an incredibly rewarding job, but today was particularly fun. We sat one of the conference rooms in the CFHQ and planned the rollout of what we're launching next. The Internet continues to suffer many challenges, but we think we've built a platform that can continue to help solve many of them. Stay tuned...

Today, just 10 months after our public launch, more than 10% of the Internet's visitors experience a faster, safer web because of CloudFlare every month. That's hard for us to imagine, even as we've watched the logs relentlessly grow. Needless to say, we are excited about where we are heading!

Today we announced that we'd raised more capital in order to help grow CloudFlare and it got picked up by a few media outlets. I wanted to give a bit of background on how we got here.

It takes some capital in order to start a company like CloudFlare. While others will undoubtedly try to build a network like ours on top of EC2 or someone else's cloud, we believe you have to build the network from the ground up in order for it to really scale. We had originally raised a little over $2 million to build CloudFlare back in November 2009. We raised this initial Series A from Ray Rothrock at Venrock, and Carl Ledbetter at Pelion Venture Partners. Ray is the leading security investor for the last 20 years, and Carl was CTO at Novell and knows everything there is to know about computer networking. Beyond their industry expertise, they're also both great people who have been a pleasure to work with.

Immediately after our launch at TechCrunch Disrupt in late September 2010, CloudFlare started to grow quickly. Traffic through our network was doubling week over week. It was an exciting time for our team of eight. The quick adoption validated for us that CloudFlare could be a big, meaningful business.

Our public launch also introduced us to many incredible people, including partners, future employees and investors. One of the individuals that we met was Scott Sandell at NEA in November 2010. He was immediately excited about CloudFlare's goal to make the Internet better for every website and web visitor. He was so excited that he became the lead investor for our Series B round where we raised $20M.

Before we took an investment from NEA, we asked two things of Scott. First, we asked that we not announce the new funding until we actually started to spend it. Second, we asked that we continue to stick to our original plan of building CloudFlare at the rate we'd originally intended. Raising money is a means to an end, it is not an end in and of itself. We wanted CloudFlare to focus on our plan and proceed at our own pace.

Ray, Carl, and Scott have been force multipliers to CloudFlare. They have quietly advised as we continued to execute on CloudFlare's original plan and lay the groundwork for the developments we have in store. While VCs can get a bad wrap from entrepreneurs, if you find the right ones then they can be terrific at helping a disruptive new business like ours flourish.

Today we are already more than 20 times the size that we were in November 2010 when we closed the funding round, and our pace of growth is only accelerating. We have just begun to deploy that capital to accomplish two things: 1) further expanding our network to provide better service to our users; and 2) hiring the best employees to build a world class team capable of helping solve the Internet's toughest challenges.

We have great things in store, and we're just getting started. As I've become fond of saying: stay tuned....