Go coverage with external tests

The Go test coverage implementation is quite ingenious: when asked to, the Go compiler will preprocess the source so that when each code portion is executed a bit is set in a coverage bitmap. This is integrated in the go test tool: go test -cover enables it and -coverprofile= allows you to write a profile to then inspect with go tool cover.

This makes it very easy to get unit test coverage, but there's no simple way to get coverage data for tests that you run against the main version of your program, like end-to-end tests.

The proper fix would involve adding -cover preprocessing support to go build, and exposing the coverage profile maybe as a runtime/pprof.Profile, but as of Go 1.6 there’s no such support. Here instead is a hack we've been using for a while in the test suite of RRDNS, our custom Go DNS server.

We create a dummy test that executes main(), we put it behind a build tag, compile a binary with go test -c -cover and then run only that test instead of running the regular binary.

Here's what the rrdns_test.go file looks like:

// +build testrunmain

package main

import "testing"

func TestRunMain(t *testing.T) {
	main()
}

We compile the binary like this

$ go test -coverpkg="rrdns/..." -c -tags testrunmain rrdns

And then when we want to collect coverage information, we execute this instead of ./rrdns (and run our test battery as usual):

$ ./rrdns.test -test.run "^TestRunMain$" -test.coverprofile=system.out

You must return from main() cleanly for the profile to be written to disk; in RRDNS we do that by catching SIGINT. You can still use command line arguments and standard input normally, just note that you will get two lines of extra output from the test framework.

Finally, since you probably also run unit tests, you might want to merge the coverage profiles with gocovmerge (from issue #6909):

$ go get github.com/wadey/gocovmerge
$ gocovmerge unit.out system.out > all.out
$ go tool cover -html all.out

If finding creative ways to test big-scale network services sounds fun, know that we are hiring in London, San Francisco and Singapore.