It's Go Time on Linux

Some interesting changes related to timekeeping in the upcoming Go 1.3 release inspired us to take a closer look at how Go programs keep time with the help of the Linux kernel. Timekeeping is a complex topic and determining the current time isn’t as simple as it might seem at first glance.

Go running on the Linux kernel has been used to build many important systems like RRDNS (our DNS server) at CloudFlare. Accurately, precisely and efficiently determining the time is an important part of many of the these systems.

To see why time is important, consider that humans have had some trouble convincing computers to keep time for them in the recent past. It been a bit more than a decade since we had to dust off our best COBOL programmers to tackle Y2K.

More recently, a bug in the handling of a leap second propagated through the Network Time Protocol (NTP) also took many systems off-line. As we've seen in recent days, NTP is very useful for synchronizing computer clocks and/or DDoSing them. The leap second bug received extensive coverage. Google was ready but many other popular sites were taken offline.

We also have the Year 2038 problem to look forward to. Hopefully there will still be a few engineers around then that remember what this 32-bit thing was all about.

Everything starts with the time package that is part of Go’s standard library. The time package provides types for Time, Duration, Ticker, Timer and various utility functions for manipulating these types.

The most commonly used function in this package is probably the time.Now function, which returns the current time as a Time struct. The Time struct has 3 fields:

type Time struct {
    sec int64
	nsec uintptr
	loc *Location
}

Location contains the timezone information for the time.

Duration is used to express the difference between two Times and to configure timers and tickers.

Timer is useful for implementing a timeout, typically as part of a select statement. Ticker can be used to wake up periodically, usually when you are using select in a for loop.

Go’s time package is used in many other places in the Go standard library. When dealing with socket connections that may go slow or stop sending data completely, one uses the SetDeadline functions that are part of the net.Conn interface.

We love writing tests at CloudFlare, and having unit tests that include some kind of random component can turn up interesting issues. You can use the current time to seed random number generators in tests, using:

rand.New(rand.NewSource(time.Now().UnixNano()))

If you’re generating random numbers for a secure application, you really want to be using the crypto/rand package. Interestingly, even the initialization of crypto/rand.Reader incorporates the current time.

The current time also features when one logs something using the log package.

A very useful service called Sourcegraph turns up more than 6000 examples where time.Now is used. For example, the Camlistore code base calls time.Now in about 130 different places.

With time.Now being as pervasive as it is, have you ever wondered how it works? Time to dive deeper.

The most important changes to the way in which Go programs keep time on Linux was committed on 8 November 2012 in changeset 42c8d3aadc40 Let’s analyse the commit message and the code for some clues:

runtime: use vDSO clock_gettime for time.now & runtime.nanotime
on Linux/amd64. Performance improvement aside, time.Now() now 
gets real nanosecond resolution on supported systems.

To understand this commit message better, we first need to review the system calls available on Linux for obtaining the value of the clock.

In the beginning, there was time and gettimeofday, which existed in SVr4, 4.3BSD and was described in POSIX.1-2001. time returns the number of seconds since the Unix epoch, 1970-01-01 00:00:00 UTC and is defined in C as:

time_t time(time_t *t)

time_t is 4 bytes on 32-bit platforms and 8 bytes on 64-bit platforms, hence the Y2038 problem mentioned above.

gettimeofday returns the number of seconds and microseconds since the epoch and is defined in C as:

int gettimeofday(struct timeval *tv, struct timezone *tz)

gettimeofday populates a struct timeval, which has the following fields:

struct timeval {
	time_t tv_sec; /* seconds */
	suseconds_t tv_usec; /* microseconds */
}

gettimeofday yields timestamps that only have microsecond precision. POSIX.1-2008 marks gettimeofday as obsolete, recommending the use of clock_gettime instead, which is defined in C as:

int clock_gettime(clockid_t clk_id, struct timespec *tp)

clock_gettime populates a struct timespec, which has the following fields:

struct timespec {
	time_t tv_sec; /* seconds */
	long tv_nsec; /* nanoseconds */
}

clock_gettime can yield timestamps that have nanosecond precision. The clock ID parameter determines the type of clock to use. Of interest to us are:

• CLOCK_REALTIME: a system-wide clock that measures real time. This clock is affected by discontinuous jumps in the system time and by incremental adjustments by made using the adjtime function or NTP.

• CLOCK_MONOTONIC: a clock that represents monotonic time since some unspecified starting point. This clock is not affected by discontinous jumps in the system time, but is affected by adjtime and NTP.

• CLOCK_MONOTONIC_RAW: similar to CLOCK_MONOTONIC, but not subject to adjustment by adjtime or NTP.

With this background we can look at the code for time.Now. (Hint: click on the function name in godoc to look at the code yourself.)

The time.Now function is implemented using a function called now that is internal to package time, but is actually provided by the Go runtime. In other words, there is no code for the function in package time itself.

Let’s take a closer look at the Linux implementations for the 386 and amd64 platforms. We see that these functions are implemented in assembler and call a function to retrieve the current time. You might have been expecting to see a system call, i.e. an INT 0x80 instruction on 386 or the SYSCALL instruction on amd64, to the kernel at this point, but Go does something much more interesting on Linux.

The Linux kernel provides Virtual Dynamically linked Shared Objects (vDSO) as a way for user space applications to make low-overhead calls to functions that would normally involve a system call to the kernel.

If you’re writing your application in a language that uses glibc, you are probably already getting your time via vDSO. Go doesn’t use glibc, so it has to implement this functionality in its runtime. The relevant code is in vdso_linux_amd64.c in the runtime package.

Finally, if you’re the kind of person that likes to stare into the bowels of your operating system, here’s the kernel side of the vDSO.

vDSO support for time functions is currently 64-bit only, but a kernel patch is in the works to add them on 32-bit platforms. When this happens, the Go runtime code will have to be updated to take advantage of this.

When frequently calling a function to determine the time, you may be interested to know how long it takes to return. In other words, how now is “now” really? For benchmarking purposes, you can make these system calls directly from Go code. We have prepared a patch to Dave Cheney’s excellent autobench project so that you may benchmark these system calls and other time-related functions yourself.

Benchmarks can also help us measure the time saved by calling gettimeofday and clock_gettime via the vDSO mechanism instead of the traditional system call path.

We’ll also use autobench to compare the performance of different versions of Go for the same set of time functions.

All benchmarks numbers below were obtained on an Intel Core i7-3540M CPU running at its maximum clock speed of 3 GHz. The CPU frequency scaling governor was set to performance mode to ensure reliable benchmark results.

We’ll use the Go 1.2 stable release as a baseline.

BenchmarkSyscallTime and BenchmarkVDSOTime measure the time it takes to make a time system call and vDSO call, respectively:

BenchmarkSyscallTime 38.2 ns/op
BenchmarkVDSOTime    3.85 ns/op

BenchmarkSyscallGettimeofday and BenchmarkVDSOGettimeofday measure the time it takes make to call a gettimeofday system call and vDSO call, respectively:

BenchmarkSyscallGettimeofday 59.3 ns/op
BenchmarkVDSOGettimeofday    23.4 ns/op

BenchmarkTimeNow measures the time it takes to call time.Now, which makes an underlying vDSO call to clock_gettime(CLOCK_REALTIME) and converts the returned value to a time.Time struct:

BenchmarkTimeNow 23.6 ns/op

Using autobench, we can also compare different Go versions against each other. To see how far we've come in the last few years, we also compared Go 1.2 to Go 1.0.3, which was released on Sep 27, 2012. The major difference was in the benchmark for time.Now:

benchmark        old ns/op  new ns/op  delta
BenchmarkTimeNow 406        23         -94.19%

To repeat this test with Go 1.0.3, you'll need the fix in changeset 419dcca62a3d to compile if you are using a recent version of GCC.

The speed at which you can tell time also depending on the clock source being used by your kernel. To see which clock sources you have available, run the following command:

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm

To see which clock source is currently in use, run this command:

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

Time Stamp Counter (TSC) is a 64-bit register that can be read by the RDTSC instruction. It is faster to read than the High Precision Event Timer (HPET). The ACPI Power Management Timer (APCI PMT) is another timer that is found on many motherboards.

We can also use the same benchmarks above to compare the TSC clock source and a HPET clock source. Doing so requires booting Linux with the clocksource=hpet kernel command line parameter. Here are the results:

benchmark                              tsc ns/op   hpet ns/op  delta
BenchmarkSyscallGettimeofday              59          645      +987.69%
BenchmarkVDSOGettimeofday                 23          598      +2455.56%
BenchmarkSyscallClockGettimeRealtime      58          642      +995.56%
BenchmarkSyscallClockGettimeMonotonic     57          641      +1012.85%
BenchmarkTimeNow                          23          598      +2433.90%

As you can see, querying the HPET clock source takes significantly longer.

Not all CPUs are created equal. To see which TSC features your CPU supports, run the following command:

$ cat /proc/cpuinfo | grep tsc

You will see some or all of the following CPU flags related to the TSC:

• tsc

• rdtscp

• constant_tsc

• nonstop_tsc

The tsc flag indicates that your CPU has the 64-bit TSC register, which has been present since the Pentium.

The rdtscp flag indicates that your CPU supports the newer RDTSCP instruction, in addition to the RDTSC instruction. Intel has an interesting whitepaper on How to Benchmark Code Execution Times on Intel IA-32 and IA-64 Instruction Set Architectures with more details about the differences.

The constant_tsc flag indicates that the TSC runs at constant frequency irrespective of the current frequency, voltage or throttling state of the CPU, commonly referred to as its P- and T-state.

The nonstop_tsc flag indicates that TSC does not stop, irrespective of the CPU’s power saving mode, referred to as its C-state.

These features work in conjunction to provide an invariant TSC. There is more discussion over at the Intel Software forums if you’re interested.

There has also been some interesting work on the time front in the upcoming Go 1.3 release. The time.Sleep function, Ticker, and Timer now use clock_gettime(CLOCK_MONOTONIC) on Linux and other platforms. This work has been a good example of the broader community contributing to improve the core of Go, as can be seen in issue 6007, CL 53010043 and discussion on golang-dev.

That’s it for today. If kernels and clocks excite you, CloudFlare is always looking to hire great Go and Linux kernel engineers. See our Careers page.

It would not have been possible to write this article without the help from some other sources. If you want to know more, here’s some recommended reading:

The book Understanding the Linux Kernel by Daniel Bovet and Marco Cesati has an entire chapter on the timekeeping architecture in the kernel.

More about timers on StackOverflow

Read more about vDSOs.

Read more about clock_gettime.

Sometimes bug reports can be a great source of information on details of the kernel and your CPU. Red Hat Bugzilla #474091 was a gold mine of information on the CPU flags for the TSC.

More at Quora on the the benefits of TSC vs HPET.

StackOverflow also had some information on the ACPI PM clock source.

Finally, some discussion of the differences between CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW as used in the Twisted framework for Python.