When I woke up this morning I had no idea I'd be on a video conference with CloudFlare, OpenDNS, Google, GoDaddy, Twitter tech folks all day
— Rajiv Pant (@rajivpant) August 28, 2013
At 1:19pm (PDT) today, a researcher noticed that the New York Times' website wasn't loading. We know the New York Times tech team, so we sent an email to check in. A few minutes later, the CTO of the NYT called us back. Since then, a handful of the CloudFlare team has been holed up in a conference room playing a small part in cleaning up this hack.
Registrars and Registries
To understand the attack, it's important to understand three key entities on the Internet: 1) registries; 2) registrars; and 3) recursive DNS providers. NYTimes.com is under the .com top level domain (TLD). The registry for that TLD is Verisign. When you buy a domain what you are fundamentally buying is the right to set the name servers within a TLD's registry.
You purchase and manage domains through organizations known as registrars. NYTimes.com is managed by a registrar known as MelbourneIT. MelbourneIT has traditionally been known as one of the more secure registrars. In addition to the New York Times, they are also used by large web organizations including Twitter and the Huffington Post.
When you visit NYTimes.com your browser looks up the domain against the Internet's DNS network. The first step in that request is a query to a recursive DNS provider. Most ISPs provide a recursive DNS (it's the two IP addresses often entered when setting up your wifi access point). There are also public services from companies like OpenDNS and Google that provide globally distributed recursive DNS services used by millions of people.
Recursive DNS providers follow the DNS chain, starting at the root, then the TLD registry, then ultimately to whatever is listed as the authoritative name server for the domain. In order to lighten the load upstream, recursive DNS providers cache results for a limited period of time known as a TTL. Compromising any step in the DNS chain would allow an attacker to take over some or all traffic destined for a site. That's exactly what happened today.
Registrar Compromise
The New York Times has confirmed publicly that their registrar was hacked, allegedly by the Syrian Electronic Army. While we've been in contact with MelbourneIT, we don't yet have details on how the attack was accomplished. We do know that the attacker was able to update the name servers for NYTimes.com without authorization, effectively hijacking the site.
The bad records entered by the hackers at MelbourneIT were pushed from the registrar up to the registry, Verisign, which manages the .com TLD. In particular, the NYTimes.com site had its name servers at the registry listed as ns5.boxsecured.com and ns6.boxsecured.com. The correct name servers should have been DNS.EWR1.NYTIMES.COM and DNS.SEA1.NYTIMES.COM. Troublingly, MelbourneIT initially appeared unable to correct the bad entries at the registry.
From screen shots that the Syrian Electronic Army has subsequently posted to its Twitter feed, it appears that the hackers gained access to MelbourneIT's administrative control panel.
While NYT worked on getting the bad records corrected with MelbourneIT, we reached out to two of the largest recursive DNS providers: OpenDNS and Google. Technical teams from CloudFlare, OpenDNS and Google jumped on a conference call and discovered the site to which the NYTimes.com site was redirected was in internet space (the IP addresses) full of phishing and possible malware, although no malware distribution was witnessed. (Earlier, this read: "...discovered what appeared to be malware on the site to which the NYTimes.com site was redirected." The confusion was that the IP range contained malware and phishing according to scans run by OpenDNS. I misinterpreted that to mean that there was malware on the site itself.) OpenDNS and Google's DNS team worked to correct the hacked records for the customers of their recursive DNS services.
The OpenDNS team was also able to look for other domains that had been updated recently to name servers controlled by the Syrian Electronic Army. We discovered several domains that had been updated, including several belonging to Twitter and the Huffington Post. As mentioned above, these organizations also used MelbourneIT, suggesting that the compromise was more than just the NYT's account.
Cleaning Up the Mess
At the registry, Verisign rolled back changes to the name servers and added a so-called registry lock to NYTimes.com. This prevented further changes even if initiated by the registrar. While quick action by OpenDNS and Google limited the impact on their customers, web surfers using other recursive DNS providers continued to be served hacked results. Unfortunately, because recursive DNS servers cache results for a period of time, even after the records were corrected, many name servers were still pointing to the incorrect locations for affected domains.
The registrar of the primary domain the Syrian Electronic Army was using as a name server for the domains they hacked revoked the domain's registration this afternoon. Since the cache TTL on the domain was relatively short, shortly after the domain was revoked traffic largely stopped flowing to the malware infected sites. That did not mean all hacked sites came back online. In some places, DNS recursors continue to have the cached bad records. They will expire over the next 24 hours and traffic to sites will return to normal.
How to Protect Yourself
This was a very spooky attack. MelbourneIT is known for having higher security than most registrars. We are hopeful that they will post the details of the attack as they are discovered so organizations can understand the threat and how to better protect themselves.
An e-mail obtained by Matther Key, an independant journalist, indicates that the hackers used a MelbourneIT domain reseller account as part of the attack. While we are only speculating at this point, it's possible that there was a vulnerability in Melbourne IT's reseller systems that allowed a privilege escalation. This method of attack would allow the attacker to then affect domain names owned by other Melbourne IT customers.
The hack also illustrates the damage that can be done by redirecting a site's DNS. DNS forms the heart of the Internet, not just the web. Email routing, too, depends on DNS to route message to the correct server.
There is one sensible measure that domains at risk should all put in place immediately. It is possible to put what is known as a registry lock in place for your domain. This prevents even the registrar from making changes to the registry automatically. If you run a whois query against your domain, you can see if you have a registry lock in place if it includes three status lines: serverDeleteProhibited, serverTransferProhibited, and serverUpdateProhibited.
Registrars generally do not make it easy to request registry locks because they make processes like automatic renewals more difficult. However, if you have a domain that may be at risk, you should insist that your registrar put a registry lock in place. It's worth noting that while some of Twitter's utility domains were redirected, Twitter.com was not -- and Twitter.com has a registry lock in place.
We spend our time building technical networks, but it's comforting to know that the human network is effective as well.
Well that was a fun day... lots of fantastic people in the tech community is a silver lining
— Jon Oden (@jonjoden) August 28, 2013