Today, Cloudflare can quantitatively confirm that Internet access has been shut down in the Democratic Republic of the Congo, information already reported by many press organisations. This shutdown occurred as the presidential election was taking place on December the 30th, and continues as the results are published.

Sadly, this act is far from unprecedented. We have published many posts about events like this in the past, including a different post about roughly three days of Internet disruption in the Democratic Republic of the Congo less than a year ago. A painfully familiar shape can be seen on our network monitoring platform, showing that the traffic in the country is barely reaching a quarter of its typical level:

Note that the graph is based on UTC and Democratic Republic of the Congo’s capital Kinshasa has the timezone of GMT+1.

The drop in bandwidth started just before midday on 31 December 2018 (around 10:30 UTC, 11:30 local time in Kinshasa). This can be clearly seen if we overlay each 24 hour day over each other:

The red line is 31 December, the gray lines the previous eight days. Looking at today’s overlay bandwidth graph, we can confirm this has continued and is an abnormal behaviour.

Other actors on the Internet have also been reporting similar figures. We hope that we can soon inform our readers the country is normally connected to the Internet again.

While 85 million people live in the country, very few people have internet access (6.21% according to Wikipedia’s List of countries by number of Internet users page). The country is also very large (2,344,858 square kms or 905,355 sq miles) and the 11th largest country in the world - around a quarter the landmass of the USA and nearly twice as big as South Africa. These facts play together and because of limited fiber deployment within the country; there are many places that still use very limited and expensive satellite Internet access. We can see in our bandwidth kgraphs that traffic to these satellite connected locations was not affected by this shutdown:

Note that the bandwidth levels are very low and represent a very small percentage of the overall traffic into Democratic Republic of the Congo.

Comparing that graph to the one from the largest mobile provider in the country; we clearly see the distinct cutoff.

15 months ago we wrote about an outage in Togo, were we noted that this adds Togo to the list of countries like Syria (twice), Iraq, Turkey, Libya, Tunisia, etc that have restricted or revoked Internet access. We have also written about unrest in Gabon (in 2016) and The Gambia (also in 2016). In Gambia’s case, the incumbent president lost the election! In fact we wrote “Rather than clamping down on the opposition by blocking the access to the Internet, it is quite possible that the blackout in Gambia may have infuriated voters and increased the vote against the president.”. Let’s see what happens in Democratic Republic of the Congo.

We'll update this blog once we see changes to these traffic levels. The Congolese government says they will restore internet access after election results are published on January 6th. That’s four days from now.

At Cloudflare, we’ll continue to do our part to try to ensure that vulnerable voices have access to the Internet. Cloudflare’s Project Galileo and Athenian Project help protect at risk websites -- such as those run by human rights organizations, journalists, and government entities reporting election results -- from being knocked offline by cyber attack.

We also support the principles for a Contract for the Web, which urge governments to commit to keeping all of the Internet available, all of the time, and Access Now’s #KeepitOn campaign. We can only hope that these efforts will yield more positive results in 2019.

Photo by Mink Mingle / Unsplash

Cloudflare recently updated the authoritative DNS service to support nine new record types. Since these records are less commonly used than what we previously supported, we thought it would be a good idea to do a brief explanation of each record type and how it is used.

DNSKEY and DS work together to allow you to enable DNSSEC on a child zone (subdomain) that you have delegated to another Nameserver. DS is useful if you are delegating DNS (through an NS record) for a child to a separate system and want to keep using DNSSEC for that child zone; without a DS entry in the parent, the child data will not be validated. We’ve blogged about the details of Cloudflare’s DNSSEC implementation and why it is important in the past, and this new feature allows for more flexible adoption for customers who need to delegate subdomains.

Today, there is no way to restrict which TLS (SSL) certificates are trusted to be served for a host. For example if an attacker were able to maliciously generate an SSL certificate for a host, they could use a on-path attacker attack to appear as the original site. With SSHFP, TLSA, SMIMEA, and CERT, a website owner can configure the exact certificate public key that is allowed to be used on the domain, stored inside the DNS and secured with DNSSEC, reducing the risk of these kinds of attacks working.

It is critically important that if you rely on these types of records that you enable and configure DNSSEC for your domain.

This type of record is an answer to the question “When I’m connecting via SSH to this remote machine, it’s authenticating me, but how do I authenticate it?” If you’re the only person connecting to this machine, your SSH client will compare the fingerprint of the public host key to the one it kept in the known_hosts file during the first connection. However across multiple machines or multiple users from an organization, you need to verify this information against a common source of trust. In essence, you need the equivalent of the authentication that a certificate authority provides by signing an HTTPS certificate, but for SSH. Although it’s possible to set certificate authorities for SSH and to have them sign public host keys, another way is to publish the fingerprint of the keys in the domain via the SSHFP record type.

Again, for these fingerprints to be trustworthy it is important to enable DNSSEC on your zone.

The SSHFP record type is similar to TLSA record. You are specifying the algorithm type, the signature type, and then the signature itself within the record for a given hostname.

If the domain and remote server have SSHFP set and you are running an SSH client (such as OpenSSH 5.1+) that supports it, you can now verify the remote machine upon connection by adding the following parameters to your connection:

❯ ssh -o "VerifyHostKeyDNS=yes" -o "StrictHostKeyChecking=yes" [insertremoteserverhere]

TLSA records were designed to specify which keys are allowed to be used for a given domain when connecting via TLS. They were introduced in the DANE specification and allow domain owners to announce which certificate can and should be used for specific purposes for the domain. While most major browsers do not support TLSA, it may still be valuable for non browser specific applications and services.

For example, I’ve set a TLSA record for the domain hasvickygoneonholiday.com for TCP traffic over port 443. There are a number of ways to generate the record, but the easiest is likely through Shumon Huque’s tool.

For most of the examples in this post we will be using kdig rather than the ubiquitous dig. Generally preinstalled dig versions can be old and may not handle newer record types well. If your queries do not quite match up, you should either upgrade your version of dig or install knot.

;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 2218
;; Flags: qr rd ra ad; QUERY: 1; ANSWER: 2; AUTHORITY: 0; ADDITIONAL: 1

;; QUESTION SECTION:
;; _443._tcp.hasvickygoneonholiday.com. 	IN	TLSA

;; ANSWER SECTION:
_443._tcp.hasvickygoneonholiday.com. 300	IN	TLSA	3 1 1 4E48ED671DFCDF6CBF55E52DBC8B9C9CC21121BD149BC24849D1398DA56FB242
_443._tcp.hasvickygoneonholiday.com. 300	IN	RRSIG	TLSA 13 4 300 20180803233834 20180801213834 35273 hasvickygoneonholiday.com. JvC9mZLfuAyEHZUZdq4n8kyRbF09vwgx4c1fas24Ag925LILr1armjHbr7ZTp8ycS/Go3y3lgyYCuBeW/vT/3w==

;; Received 232 B
;; Time 2018-08-02 15:38:34 PDT
;; From 192.168.1.1@53(UDP) in 28.5 ms

From the above request and response, we can see that a) the response for the zone is secured and signed with DNSSEC (Flag: ad) and that I should be verifying a certificate with the public key (3 1 1) SHA256 hash (3 1 1) of 4E48ED671DFCDF6CBF55E52DBC8B9C9CC21121BD149BC24849D1398DA56FB242. We can use openssl (v1.1.x or higher) to verify the results:

❯ openssl s_client  -connect hasvickygoneonholiday.com:443 -dane_tlsa_domain "hasvickygoneonholiday.com" -dane_tlsa_rrdata "
3 1 1 4e48ed671dfcdf6cbf55e52dbc8b9c9cc21121bd149bc24849d1398da56fb242"
CONNECTED(00000003)
depth=0 C = US, ST = CA, L = San Francisco, O = "CloudFlare, Inc.", CN = hasvickygoneonholiday.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=CA/L=San Francisco/O=CloudFlare, Inc./CN=hasvickygoneonholiday.com
   i:/C=US/ST=CA/L=San Francisco/O=CloudFlare, Inc./CN=CloudFlare Inc ECC CA-2
 1 s:/C=US/ST=CA/L=San Francisco/O=CloudFlare, Inc./CN=CloudFlare Inc ECC CA-2
   i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE7jCCBJSgAwIBAgIQB9z9WxnovNf/lt2Lkrfq+DAKBggqhkjOPQQDAjBvMQsw
...
---
SSL handshake has read 2666 bytes and written 295 bytes
Verification: OK
Verified peername: hasvickygoneonholiday.com
DANE TLSA 3 1 1 ...149bc24849d1398da56fb242 matched EE certificate at depth 0

SMIMEA records function similar to TLSA but are specific to email addresses. The domain for these records should be prefixed by “_smimecert.” and specific formatting is required to attach a SMIMEA record to an email address. The local-part (username) of the email address must be treated in a specific format and SHA-256 hashed as detailed in the RFC. From the RFC example: “ For example, to request an SMIMEA resource record for a user whose email address is "hugh@example.com", an SMIMEA query would be placed for the following QNAME: c93f1e400f26708f98cb19d936620da35eec8f72e57f9eec01c1afd6._smimecert.example.com

CERT records are used for generically storing certificates within DNS and are most commonly used by systems for email encryption. To create a CERT record, you must specify the certificate type, the key tag, the algorithm, and then the certificate, which is either the certificate itself, the CRL, a URL of the certificate, or fingerprint and a URL.

PTR (Pointer) records are pointers to canonical names. They are similar to CNAME in structure, meaning they only contain one FQDN (fully qualified domain name) but the RFC dictates that subsequent lookups are not done for PTR records, the value should just be returned back to the requestor. This is different to a CNAME where a recursive resolver would follow the target of the canonical name. The most common use of a PTR record is in reverse DNS, where you can look up which domains are meant to exist at a given IP address. These are useful for outbound mailservers as well as authoritative DNS servers.

It is only possible to delegate the authority for IP addresses that you own from your Regional Internet Registry (RIR). Creating reverse zones and PTR records for IPs that you can not (or do not) delegate does not serve any practical purpose.

For example, looking up the A record for marek.ns.cloudflare.com gives us the IP of 173.245.59.202.

❯ kdig a marek.ns.cloudflare.com +short
173.245.59.202

Now imagine we want to know if the owner of this IP ‘authorizes’ marek.ns.cloudflare.com to point to it. Reverse Zones are specifically crafted child zones within in-addr.arpa. (for IPv4) and ip6.arpa. (for IPv6) whom are delegated via the Regional Internet Registries to the owners of the IP address space. That is to say if you own a /24 from ARIN, ARIN will delegate the reverse zone space for your /24 to you to control. The IPv4 address is represented inverted as the subdomain in in-addr.arpa. Since Cloudflare owns the IP, we’ve delegated the reverse zone and created a PTR there.

❯ kdig -x 173.245.59.202
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 18658
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; 202.59.245.173.in-addr.arpa.		IN	PTR

;; ANSWER SECTION:
202.59.245.173.in-addr.arpa.	1222	IN	PTR	marek.ns.cloudflare.com.

For completeness, here is the +trace for the 202.59.245.173.in-addr.arpa zone. We can see that the /24 59.245.173.in-addr.arpa has been delegated to Cloudflare from ARIN:

❯ dig 202.59.245.173.in-addr.arpa +trace

; <<>> DiG 9.8.3-P1 <<>> 202.59.245.173.in-addr.arpa +trace
;; global options: +cmd
.			48419	IN	NS	a.root-servers.net.
.			48419	IN	NS	b.root-servers.net.
.			48419	IN	NS	c.root-servers.net.
.			48419	IN	NS	d.root-servers.net.
.			48419	IN	NS	e.root-servers.net.
.			48419	IN	NS	f.root-servers.net.
.			48419	IN	NS	g.root-servers.net.
.			48419	IN	NS	h.root-servers.net.
.			48419	IN	NS	i.root-servers.net.
.			48419	IN	NS	j.root-servers.net.
.			48419	IN	NS	k.root-servers.net.
.			48419	IN	NS	l.root-servers.net.
.			48419	IN	NS	m.root-servers.net.
;; Received 228 bytes from 2001:4860:4860::8888#53(2001:4860:4860::8888) in 25 ms

in-addr.arpa.		172800	IN	NS	e.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	d.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	b.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	f.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	c.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	a.in-addr-servers.arpa.
;; Received 421 bytes from 192.36.148.17#53(192.36.148.17) in 8 ms

173.in-addr.arpa.	86400	IN	NS	u.arin.net.
173.in-addr.arpa.	86400	IN	NS	arin.authdns.ripe.net.
173.in-addr.arpa.	86400	IN	NS	z.arin.net.
173.in-addr.arpa.	86400	IN	NS	r.arin.net.
173.in-addr.arpa.	86400	IN	NS	x.arin.net.
173.in-addr.arpa.	86400	IN	NS	y.arin.net.
;; Received 165 bytes from 199.180.182.53#53(199.180.182.53) in 300 ms

59.245.173.in-addr.arpa. 86400	IN	NS	ns1.cloudflare.com.
59.245.173.in-addr.arpa. 86400	IN	NS	ns2.cloudflare.com.
;; Received 95 bytes from 2001:500:13::63#53(2001:500:13::63) in 188 ms

Naming Authority Pointer Records are used in conjunction with SRV records, generally as a part of the SIP protocol. NAPTR records point to domains to specific services, if available for that domain. Anders Brownworth has an excellent description in detail on his blog. The start of his example, with his permission:

Let’s consider a call to 2125551212@example.com. Given only this address though, we don't know what IP address, port or protocol to send this call to. We don't even know if example.com supports SIP or some other VoIP protocol like H.323 or IAX2. I'm implying that we're interested in placing a call to this URL but if no VoIP service is supported, we could just as easily fall back to emailing this user instead. To find out, we start with a NAPTR record lookup for the domain we were given:

#host -t NAPTR example.com
example.com NAPTR 10 100 "S" "SIP+D2U" "" _sip._udp.example.com.
example.com NAPTR 20 100 "S" "SIP+D2T" "" _sip._tcp.example.com.
example.com NAPTR 30 100 "S" "E2U+email" "!^.*$!mailto:info@example.com!i" _sip._tcp.example.com.

Here we find that example.com gives us three ways to contact example.com, the first of which is "SIP+D2U" which would imply SIP over UDP at _sip._udp.example.com.

Uniform Resource Identifier records are commonly used as a compliment to NAPTR records and per the RFC, can be used to replace SRV records. As such, they contain a Weight and Priority field as well as Target, similar to SRV.

One use case is proposed by this draft RFC is to replace SRV records with URI records for discovering Kerberos key distribution centers (KDC). It minimizes the number of requests over SRV records and allows the domain owner to specify preference for TCP or UDP.

In the below example, it specifies that we should use a KDC on TCP at the default port and UDP on port 89 should the primary connection fail.

❯ kdig URI _kerberos.hasvickygoneonholiday.com
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 8450
;; Flags: qr rd ra; QUERY: 1; ANSWER: 2; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; _kerberos.hasvickygoneonholiday.com. 	IN	URI

;; ANSWER SECTION:
_kerberos.hasvickygoneonholiday.com. 283	IN	URI	1 10 "krb5srv:m:tcp:kdc.hasbickygoneonholiday.com"
_kerberos.hasvickygoneonholiday.com. 283	IN	URI	1 20 "krb5srv:m:udp:kdc.hasbickygoneonholiday.com:89"

Cloudflare now supports CERT, DNSKEY, DS, NAPTR, PTR, SMIMEA, SSHFP, and TLSA in our authoritative DNS products. We would love to hear if you have any interesting example use cases for the new record types and what other record types we should support in the future.

Our DNS engineering teams in London and San Francisco are both hiring if you would like to contribute to the fastest authoritative and recursive DNS services in the world.

Software Engineer

If you read this blog on a regular basis, you probably use the little tool called SSH, especially its ubiquitous and most popular implementation OpenSSH.

Maybe you’re savvy enough to only use it with public/private keys, and therefore protect yourself from dictionary attacks. If you do then you know that in order to configure access to a new host, you need to make a copy of a public key available to that host (usually by writing it to its disk). Managing keys can be painful if you have many hosts, especially when you need to renew one of the keys. What if DNSSEC could help?

CC BY 2.0 image by William Neuheisel

With version 6.2 of OpenSSH came a feature that allows the remote host to retrieve a public key in a customised way, instead of the typical authorized_keys file in the ~/.ssh/ directory. For example, you can gather the keys of a group of users that require access to a number of machines on a single server (for example, an LDAP server), and have all the hosts query that server when they need the public key of the user attempting to log in. This saves a lot of editing of authorized_keys files on each and every host. The downside is that it's necessary to trust the source these hosts retrieve public keys from. An LDAP server on a private network is probably trustworthy (when looked after properly) but for hosts running in the cloud, that’s not really practical.

DNSSEC is helpful here. That's right: now that we can verify responses from a DNS server, we can safely store public keys in DNS records!

So let's say we administer example.com and want to give Alice and Bob access to machines foo, bar and baz in that domain. We'll store their respective public keys in TXT[1] records named alice_pubkey.example.com and bob_pubkey.example.com. To be entirely accurate, it doesn’t really matter which zone these records belong to, but I’ll consider here that we only have one domain. The requirements are:

• the machines need to run OpenSSH server version 6.2 or later

• they also need a DNSSEC validating resolver (we'll use unbound-host)

• Alice and Bob's keys need to be less than 256 characters long (ECDSA or Ed25519 keys will work)

• DNSSEC needs to be correctly set up on the domain example.com (surprise!)

Alice and Bob generate keys like this:

foo:~$ ssh-keygen -t ecdsa

or like this:

foo:~$ ssh-keygen -t ed25519

and then follow the instructions. They will of course provide a non-empty passphrase. Then they send us (or whoever administers the zone file for example.com) the public key file, which may look like this:

ssh-ed25519 AAAAC3N...VY4A= alice@foo

We can strip the comment alice@foo out of that file, and use the rest as the value to create a TXT record with the name alice_pubkey in the domain example.com. Then, retrieving the key is as easy as this:

 foo:~$ unbound-host -t TXT alice_pubkey.example.com
 alice_pubkey.example.com has TXT record “ssh-ed25519 AAAAC3N…”

With -v, unbound-host will show us whether the signature has been verified

foo:~$ unbound-host -v -t TXT alice_pubkey.example.com
alice_pubkey.example.com has TXT record “ssh-ed25519 AAAAC…” (insecure)

With -D, it will actually check the signature:

foo:~$ unbound-host -D -v -t TXT alice_pubkey.example.com
alice_pubkey.example.com has TXT record “ssh-ed25519 AAAAC3N…” (secure)

If no record exists, it will show this:

foo:~$ unbound-host -D -v -t TXT charlie_pubkey.example.com
charlie_pubkey.example.com has no TXT record (secure)

Note that the absence of record is also labelled “secure”, thanks to NSEC.

Let’s prepare to parse that output. The sshd_config man page shows that sshd needs a specific user to run the program that will retrieve public keys. This is following the best practices of privilege separation. Let's call that user pubkeygrab and create an account on foo, bar and baz, giving it just the permissions it needs to work and nothing more:

foo:~$ useradd -m -d /var/empty -s /sbin/nologin pubkeygrab

Then create the script pubkeygrab.sh, and store it on each of the machines. Obviously, we'll make sure only root can edit it:

foo:~$ cat /usr/local/bin/pubkeygrab.sh
#!/bin/sh

USER=$1

/usr/sbin/unbound-host -v -D -t TXT ${USER}_pubkey.example.com \\
     | /usr/bin/grep -v "no TXT record" \\
     | /usr/bin/grep ' (secure)$' \\
     | /usr/bin/sed 's/.* "\(.*\)" (secure)$/\1/'

Now I'm certain that a lot of readers will have something to say about the style or the efficiency of this shell script, I just wrote it that way to highlight what steps need to be taken:

• it retrieves a TXT record, and doesn't output anything if the record doesn't exist

• if unbound-host has not confirmed that the record was correctly DNSSEC signed, it doesn't output anything

• if the above is successful, it filters out the text to return only the public key

• it doesn't try to do anything complex, because complexity is the enemy of security (or at least, that’s a point of view that I share with a few people)

• it works with multiple records

I'm sure you will write your own program to do the above. Just make sure it works only when you want it to. It is critical to ensure that it doesn't return anything at least when:

• a record for the corresponding user doesn't exist

• the records are not signed or not properly signed

• the local copy of the root key (/var/unbound/root.key, here) is corrupted.

Bonus points to you if you find more cases.

Now that you have read the warning above, add the following to /etc/ssh/sshd_config on foo, bar and baz:

AuthorizedKeysCommand /usr/local/bin/pubkeygrab.sh
AuthorizedKeysCommandUser pubkeygrab

and restart sshd. Check that the users alice and bob exist on each machine too. Note that the above change will also apply to all existing users. Now you can go to your CloudFlare account, select the domain example.com, and create the TXT records alice_pubkey and bob_pubkey. Paste their respective public keys in the value field. Soon after, Alice and Bob can log in. Ask Charlie to try too. If the above works for Alice and Bob, but fails for Charlie, congratulations, you have turned CloudFlare into a PKI for SSH.

If you remove the TXT records, Alice and Bob’s access should be revoked, and they should be unable to login, once the TTL of the TXT record is expired. However, note that when the output of pubkeygrab.sh is empty, sshd reverts to the usual AuthorizedKeysFile parameter to find a public key. If Alice and Bob are cheeky and want to keep their access after you removed their TXT records, they just need to copy their public key into that file any time before you ban them. If you don't want that, make sure the AuthorizedKeysFile parameter points to a place Alice and Bob can't write to.

I hope this is showing how interesting DNSSEC can be, and that we have more news on this topic soon.

1. Yes, it would be better to have a dedicated record, instead of overloading TXT records. ↩︎