Introducing the BPF Tools

In a recent article I described the basic concepts behind the use of Berkeley Packet Filter (aka BSD Packet filter or BPF) bytecode for high performance packet filtering, and the xt_bpf iptables module. In this post I'll explain how we use BPF and xt_bpf as one tool to deal with large scale DDoS attacks.

And, today, CloudFlare is open sourcing the tools we've created to generate and deploy BPF rules.

Our BPF Tools are now available on the CloudFlare Github: https://github.com/cloudflare/bpftools

For installation instructions review the README, but typing make should do most of the work:

$ git clone https://github.com/cloudflare/bpftools.git $ cd bpftools $ make

The BPF Tools repository contains a number of simple Python scripts, some of them focus on analyzing pcap files, others focus more on the generation and use of the BPF bytecode itself:

• pcap2hex, hex2pcap

• parsedns

• bpfgen

• filter

• iptables_bpf, iptables_bpf_chain

We rely on the BPF assembler from the Linux Kernel /tools/net directory. To make your life easier we ship a copy in linux_tools.

Here at CloudFlare we run a very large number of authoritative DNS servers and we constantly deal with malicious actors flooding our servers with, amongst other things, DNS requests. So no surprise that our current BPF Tools focus on DNS traffic, although they are easily adaptable to any other stateless floods.

The BPF Tools should be usable and in working order, but don't expect too much. These small utilities were written to be easily hackable and they will be in a state of constant flux: this is our toolkit after all. Please expect some degree of code instability.

Here's a concrete example of using these tools to identify and filter a DNS attack. This is based on a real world attack but the actual details have been changed for anonymization.

To start you need a pcap savefile containing a traffic dump. For example to capture a pcap of DNS requests we run:

$ sudo tcpdump -pni eth0 -s0 -w example.pcap -vv -c 10000 \ "ip and dst port 53" listening on eth0, link-type EN10MB (Ethernet)

It's important to record the traffic on the EN10MB (Ethernet) device, as the scripts expect to see packets with a 14-byte Ethernet header. If you forget about that and record on the any interface (LINUX_SLL) you can fix the pcap by using the pcap2hex / hex2pcap tools. They are able to amend the layer 2 header and make it look like Ethernet again:

$ cat sll.pcap | ./pcap2hex | ./hex2pcap > ethernet.pcap

Here is a sample output of the pcap2hex tool after we captured requests going to our favorite domain www.example.uk (notice the --ascii flag):

$ cat example.pcap | ./pcap2hex --ascii | head 000ffffff6603c94d5cb47f0080045000056817b4000f91147a3cba204c6 adf53a1aa408003500426dd26366000000010000000000010e697471766d 6e737a656c757a6f6a03777777076578616d706c6502756b000001000100 00291000000080000000 ..S..`<...G...E..V.{@...G....... :....5.Bm.cf...........itqvmnszeluzoj.www.example.uk.......) ........ 000ffffff6603c94d5cb47f008004520004fdf234000f41110107b1e341c adf53a1a84a00035003b4a99e25c00000001000000000001076969766c69 657903777777076578616d706c6502756b00000100010000291000000080 000000 ..S..`<...G...E..O.#@.....{.4...:....5.;J..\.... .......iivliey.www.example.uk.......)........

Taking a look at the traffic, it looks like we captured a flood of requests to <random string>.www.example.uk! We see this kind of flood all the time. I believe the goal of this is to keep our DNS server busy preparing NXDOMAIN responses and not have enough CPU to serve legitimate traffic.

Let's take a closer look at these packets.

With DNS traffic handy we can take a closer look at the details of the DNS requests. For that pick a hex-encoded packet from the output of pcap2hex and pass it to the parsedns script:

$ ./parsedns 000ffffff6603c94d5... ... [.] l4: a408003500426dd2 source port: 41992 destination port: 53 length: 66 [.] l5: 6366000000010000000000010e6974717... id: 0x6366 flags: 0x0000 query op=0 rcode=0 questions: 1 answers: 0 auth: 0 extra: 1 #-46 q[0]: 'itqvmnszeluzoj' 'www' 'example' 'uk' . type=0x0001 class=0x0001 extra[0]: . type=0x0029 class=0x1000 ttl=32768 rrlen=0: bufsize=4096 dnssec_ok_flag

This tool pretty prints a DNS packet and presents all the interesting bits. Sometimes the flooding tools have bugs and set a bit somewhere making it easy to distinguish malicious requests from legitimate DNS queries hitting our servers.

Unfortunately the request above looks pretty normal. We could distinguish the traffic on the EDNS DNS extension but some real recursors also set this flag as well, so this strategy would result in false positives.

Blocking this flood is however, simple - we can safely assume that the www.example.uk domain doesn't have any subdomains, instead of looking at low level bits of DNS packets we can drop all the packets asking for *.www.example.uk.

The tool bpfgen generates the BPF bytecode to do that. This is the most important tool in the repo.

Right now it has three "BPF generators": dns, dns_validate and suffix. We'll focus only on the first one which generates BPF rules matching given DNS domains. To match all the requests matching the pattern *.www.example.uk run:

$ ./bpfgen dns -- *.www.example.uk 18,177 0 0 0,0 0 0 20,12 0 0 0,7 0 0 0,80 0 0 0, ...

That does look pretty cryptic, here's how can you generate an assembly-like BPF syntax:

$ ./bpfgen --assembly dns -- *.www.example.uk ldx 4*([0]&0xf) ; l3_off(0) + 8 of udp + 12 of dns ld #20 add x tax ...

The generated code is way too long to post and explain here, I strongly recommend looking at the bpftools/gen_dns.py file and reviewing the kernel networking/filter.txt documentation.

For more details about the bpfgen tool and its features see the documentation:

$ ./bpfgen --help $ ./bpfgen dns -- --help $ ./bpfgen dns_validate -- --help $ ./bpfgen suffix -- --help

The BPF bytecode generated by bpfgen is somewhat special - it's prepared to be passed to the xt_bpf iptables module and not the usual tcpdump. The bytecode passed to xt_bpf must assume the packet starts from the IP header without any layer 2 header. This is not how it usually works for tcpdump which assumes packets do have a proper layer 2 header. In other words: you can't swap bytecodes between tcpdump and xt_bpf.

To work around that bpfgen has an --offset flag. To create BPF for xt_bpf you can supply the explicit --offset=0 flag:

$ ./bpfgen --offset=0 dns -- *.www.example.uk

To create BPF for tcpdump on Ethernet packets you must supply --offset=14 flag:

$ ./bpfgen --offset=14 dns -- *.www.example.uk

It's always a good idea to test the bytecode before putting it on production servers. For that we have a filter script. It consumes a pcap file, runs it through a tcpdump-like BPF and produces another pcap with only packets that matched given bytecode.

To see what traffic will match our BPF:

$ cat example.pcap \ | ./filter -b "`./bpfgen --offset 14 dns -- *.www.example.uk`" \ | tcpdump -nr - | wc -l 9997

Hooray, our BPF successfully matches 99.97% of the flood we recorded. Now let's see that which packets it will not match:

$ cat example.pcap \ | ./filter -b "`./bpfgen -o 14 --negate dns *.www.example.uk`" \ | tcpdump -nr - | wc -l 3

It's often worthwhile to inspect the matched and unmatched packets and make sure the BPF is indeed correct.

Note: filter uses the usual libpcap infrastructure, that's why it requires the BPF to consume a layer 2 header. We will likely rewrite that code and change filter to use BPF generated for xt_bpf.

With the BPF bytecode tested we can safely deploy it to the servers. The simplest way to do it is to apply an iptables rule manually:

iptables -I INPUT 1 \ --wait -p udp --dport 53 \ -m bpf --bytecode "14,0 0 0 20,177 0 0 0,12... \ -j DROP

(You will need a recent iptables with xt_bpf support.)

This can be very cumbersome. Especially because the --bytecode parameter contains spaces which makes it pretty unfriendly for parsing with bash or ssh.

To speed up the process we have another tool iptables_bpf. It accepts almost the same parameters as bpfgen but, as opposed to printing a raw BPF bytecode, it produces a bash script:

$ ./iptables_bpf dns -- *.example.uk Generated file 'bpf_dns_ip4_any_example_uk.sh'

The generated script is fairly straightforward and at its core it applies an iptables rule like this:

iptables \ --wait -I INPUT 1 \ -i eth0 \ -p udp --dport 53 \ -m set --match-set bpf_dns_ip4_any_example_uk dst \ -m bpf --bytecode "16,177 0 0 0,0 0 0 20,12 ... \ -m comment --comment "dns -- *.example.uk" \ -j DROP

As you can see, it depends on an ipset "match-set" named bpf_dns_ip4_any_example_uk. ipsets are a pretty recent addition to the iptables family and they allow us to control which destination IPs therule will be applied to. We use this for additional safety. When you deploy the generated script by default it will not match any traffic. Only when you add an IP to the ipset will the BPF rulebe executed. To add an IP to the ipset run:

ipset add bpf_dns_ip4_any_example_uk 1.1.1.1/32

Alternatively rerun the script with an IP as a parameter:

$ sudo ./bpf_dns_ip4_any_example_uk.sh 1.1.1.1/32

If things go wrong pass --delete to remove the BPF iptables rule and the ipset:

$ sudo ./bpf_dns_ip4_any_example_uk.sh --delete

Although fairly advanced and I hope practical, this generated script is not really intended as a fit-for-all deployment tool for all BPF scripts. Feel encouraged to tweak it or fork it for your needs.

In extreme cases you might want to chain BPF rules. As an example see the iptables_bpf_chain script, you can run it like this:

$ ./iptables_bpf_chain -w example_uk \ --accept www.example.uk \ --accept ns.example.uk \ --drop any Generated file 'example_uk_ip4.sh'

The generated file will create the iptables chain example_uk and it will add three rules to it: two BPF rules accepting some packets and one rule dropping everything else. The chain will be referenced from the "INPUT" chain in a similar fashion to the previous example. Before using iptables_bpf_chain please do review it carefully.

This article only scratched the surface of our tools. They can do much more, like:

• match IPv6 packets

• do suffix matching

• match domains case insensitively

• perform basic DNS request validation

For details read the documentation with --help.

Fighting packet floods is tough, but with tools in place it can be managed efficiently. The xt_bpf iptables module is very effective and with our BPF generation tools it allows us to drop malicious traffic in iptables before it hits the application.

By sharing these tools we hope to help administrators around the world, we know we are not the only ones fighting packet floods!