Subscribe to receive notifications of new posts:

Building the simplest Go static analysis tool

04/27/2016

2 min read

Go native vendoring (a.k.a. GO15VENDOREXPERIMENT) allows you to freeze dependencies by putting them in a vendor folder in your project. The compiler will then look there before searching the GOPATH.

The only annoyance compared to using a per-project GOPATH, which is what we used to do, is that you might forget to vendor a package that you have in your GOPATH. The program will build for you, but it won't for anyone else. Back to the WFM times!

I decided I wanted something, a tool, to check that all my (non-stdlib) dependencies were vendored.

At first I thought of using go list, which Dave Cheney appropriately called a swiss army knife, but while it can show the entire recursive dependency tree (format .Deps), there's no way to know from the templating engine if a dependency is in the standard library.

We could just pass each output back into go list to check for .Standard, but I thought this would be a good occasion to build a very simple static analysis tool. Go's simplicity and libraries make it a very easy task, as you will see.

First, loading the program

We use golang.org/x/tools/go/loader to load the packages passed as arguments on the command line, including the test files based on a flag.

var conf loader.Config
for _, p := range flag.Args() {
    if *tests {
        conf.ImportWithTests(p)
    } else {
        conf.Import(p)
    }
}
prog, err := conf.Load()
if err != nil {
    log.Fatal(err)
}
for p := range prog.AllPackages {
    fmt.Println(p.Path())
}

With these few lines we already replicated go list -f {{ .Deps }}!

The only missing loading feature here is wildcard (./...) support. That code is in the go tool source and it's unexported. There's an issue about exposing it, but for now packages are just copy-pasting it. We'll use a packaged version of that code, github.com/kisielk/gotool:

for _, p := range gotool.ImportPaths(flag.Args()) {

Finally, since we are only interested in the dependency tree today we instruct the parser to only go as far as the imports statements and we ignore the resulting "not used" errors:

conf.ParserMode = parser.ImportsOnly
conf.AllowErrors = true
conf.TypeChecker.Error = func(error) {}

Then, the actual logic

We now have a loader.Program object, which holds references to various loader.PackageInfo objects, which in turn are a combination of package, AST and types information. All you need to perform any kind of complex analysis. Not that we are going to do that today :)

We'll just replicate the go list logic to recognize stdlib packages and remove the packages passed on the command line from the list:

initial := make(map[*loader.PackageInfo]bool)
for _, pi := range prog.InitialPackages() {
    initial[pi] = true
}

var packages []*loader.PackageInfo
for _, pi := range prog.AllPackages {
    if initial[pi] {
        continue
    }
    if len(pi.Files) == 0 {
        continue // virtual stdlib package
    }
    filename := prog.Fset.File(pi.Files[0].Pos()).Name()
    if !strings.HasPrefix(filename, build.Default.GOROOT) ||
        !isStandardImportPath(pi.Pkg.Path()) {
        packages = append(packages, pi)
    }
}

Then we just have to print a warning if any remaining package is not in a /vendor/ folder:

for _, pi := range packages {
    if strings.Index(pi.Pkg.Path(), "/vendor/") == -1 {
        fmt.Println("[!] dependency not vendored:", pi.Pkg.Path())
    }
}

Done! You can find the tool here: https://github.com/FiloSottile/vendorcheck

Further reading

This document maintained by Alan Donovan will tell you more than I'll ever know about the static analysis tooling.

Note that you might be tempted to use go/importer and types.Importer[From] instead of x/go/loader. Don't do that. That doesn't load the source but reads compiled .a files, which can be stale or missing. Static analysis tools that spit out "package not found" for existing packages or, worse, incorrect results because of this are a pet peeve of mine.

If you now feel the urge to write static analysis tools, know that the CloudFlare Go team is hiring in London, San Francisco and Singapore!

We protect entire corporate networks, help customers build Internet-scale applications efficiently, accelerate any website or Internet application, ward off DDoS attacks, keep hackers at bay, and can help you on your journey to Zero Trust.

Visit 1.1.1.1 from any device to get started with our free app that makes your Internet faster and safer.

To learn more about our mission to help build a better Internet, start here. If you're looking for a new career direction, check out our open positions.
ToolsGoProgramming

Follow on X

Filippo Valsorda|@filosottile
Cloudflare|@cloudflare

Related posts

November 29, 2018 9:54 AM

Know your SCM_RIGHTS

As TLS 1.3 was ratified earlier this year, I was recollecting how we got started with it here at Cloudflare. We made the decision to be early adopters of TLS 1.3 a little over two years ago. It was a very important decision, and we took it very seriously....