Subscribe to receive notifications of new posts:

Quick and dirty annotations for Go stack traces

2015-08-03

1 min read

CloudFlare’s DNS server, RRDNS, is entirely written in Go and typically runs tens of thousands goroutines. Since goroutines are cheap and Go I/O is blocking we run one goroutine per file descriptor we listen on and queue new packets for processing.

CC BY-SA 2.0 image by wiredforlego

When there are thousands of goroutines running, debug output quickly becomes difficult to interpret. For example, last week I was tracking down a problem with a file descriptor and wanted to know what its listening goroutine was doing. With 40k stack traces, good luck figuring out which one is having trouble.

Go stack traces include parameter values, but most Go types are (or are implemented as) pointers, so what you will see passed to the goroutine function is just a meaningless memory address.

We have a couple options to make sense of the addresses: get a heap dump at the same time as the stack trace and cross-reference the pointers, or have a debug endpoint that prints a goroutine/pointer -> IP map. Neither are seamless.

Underscore to the rescue

However, we know that integers are shown in traces, so what we did is first convert IPv4 addresses to their uint32 representation:

// addrToUint32 takes a TCPAddr or UDPAddr and converts its IP to a uint32.
// If the IP is v6, 0xffffffff is returned.
func addrToUint32(addr net.Addr) uint32 {
       var ip net.IP
       switch addr := addr.(type) {
       case *net.TCPAddr:
               ip = addr.IP
       case *net.UDPAddr:
               ip = addr.IP
       case *net.IPAddr:
               ip = addr.IP
       }
       if ip == nil {
               return 0
       }
       ipv4 := ip.To4()
       if ipv4 == nil {
               return math.MaxUint32
       }
       return uint32(ipv4[0])<<24 | uint32(ipv4[1])<<16 | uint32(ipv4[2])<<8 | uint32(ipv4[3])
}

And then pass the IPv4-as-uint32 to the listening goroutine as an _ parameter. Yes, as a parameter with name _; it's known as the blank identifier in Go.

// PacketUDPRead is a goroutine that listens on a specific UDP socket and reads
// in new requests
// The first parameter is the int representation of the listening IP address,
// and it's passed just so it will appear in stack traces
func PacketUDPRead(_ uint32, conn *net.UDPConn, ...) { ... }

go PacketUDPRead(addrToUint32(conn.LocalAddr()), conn, ...)

Now when we get a stack trace, we can just look at the first bytes, convert them back to dotted notation, and know on what IP the goroutine was listening.

goroutine 42 [IO wait]:
	[...]
	/.../request.go:195 +0x5d
rrdns/core.PacketUDPRead(0xc27f000001, 0x2b6328113ad8, 0xc20801ecc0, 0xc208044308, 0xc208e99280, 0xc208ad8180, 0x12a05f200)
	/.../server.go:119 +0x35a
created by rrdns/core.PacketIO
	/.../server.go:230 +0x8be

0xc27f000001 -> remove alignment byte -> 0x7f000001 -> 127.0.0.1

Obviously you can do the same with any piece of information you can represent as an int.

Are you interested in taming the goroutines that run the web? We're hiring in London, San Francisco and Singapore!

Cloudflare's connectivity cloud protects entire corporate networks, helps customers build Internet-scale applications efficiently, accelerates any website or Internet application, wards off DDoS attacks, keeps hackers at bay, and can help you on your journey to Zero Trust.

Visit 1.1.1.1 from any device to get started with our free app that makes your Internet faster and safer.

To learn more about our mission to help build a better Internet, start here. If you're looking for a new career direction, check out our open positions.
RRDNSReliabilityDNSGoProgramming

Follow on X

Filippo Valsorda|@filosottile
Cloudflare|@cloudflare

Related posts

October 09, 2024 1:00 PM

Improving platform resilience at Cloudflare through automation

We realized that we need a way to automatically heal our platform from an operations perspective, and designed and built a workflow orchestration platform to provide these self-healing capabilities across our global network. We explore how this has helped us to reduce the impact on our customers due to operational issues, and the rich variety of similar problems it has empowered us to solve....

September 24, 2024 1:00 PM

Cloudflare partners with Internet Service Providers and network equipment providers to deliver a safer browsing experience to millions of homes

Cloudflare is extending the use of our public DNS resolver through partnering with ISPs and network providers to deliver a safer browsing experience directly to families. Join us in protecting every Internet user from unsafe content with the click of a button, powered by 1.1.1.1 for Families....