Tiny C++ Benchmark Framework

Lately I had to improve a little some of my algorithms. I wanted to check, how changes affect to time of algorithms execution. So I started to search benchmark framework for C++. After a while, I have found quite small, but powerful framework - google benchmark [1].

Simple usage

The simplest usage of google benchmark is shown below:

#include <benchmark/benchmark.h>
#include <algorithm>

int complex_computation(int n)
{
  if (n == 0) 
    return 0;

  unsigned int a = 1, b = 1;

  for (unsigned int i=0; i < n-1; i++) 
    {
      std::swap(a, b);
      b += a;
    }

  return b;
}

static void BM_Fibonacci(benchmark::State& state)
{
  int ret;

  while (state.KeepRunning())
    ret |= complex_computation(500);

  CHECK(ret != 0);
}

BENCHMARK(BM_Fibonacci);

int main(int argc, const char* argv[]) 
{
  benchmark::Initialize(&argc, argv);
  benchmark::RunSpecifiedBenchmarks();

  return 0;
}

At the beginning, we have to define function, which will be measured (complex_computation, in my case). Next, we're defining benchmark method - BM_Fibonacci. It has one argument - state (I will talk about it later). In while loop we call our method, until benchmark's working. ret variable is used only because of compiler optimizations (we'd like to avoid removing dummy code). Benchmark method should be later registered (using BENCHMARK macro). In main function, framework should be initialized, and after that, we can run all our benchmark methods.

loganek@hf-gcs-computer:~/Documents/blog-benchmark$ g++ benchmark.cpp -std=c++11 -lbenchmark -lpthread -O2 -o benchmark
loganek@hf-gcs-computer:~/Documents/blog-benchmark$ ./benchmark 
Reading /proc/self/cputime_ns failed. Using getrusage().
Benchmarking on 4 X 2701 MHz CPUs
2014/09/29-00:49:29
CPU scaling is enabled: Benchmark timings may be noisy.
DEBUG: Benchmark      Time(ns)    CPU(ns) Iterations
----------------------------------------------------
DEBUG: BM_Fibonacci        208        225    2222788                                  
loganek@hf-gcs-computer:~/Documents/blog-benchmark$ 

As you can see, there is computation time, cpu and iterations count - everything what we need :) Moreover, we can also get a few informations about machine.

Arguments, range of arguments

Snippet shown above is very simple, but this framework offers users much more. We can define a set of arguments for benchmark. Let's remind previous example. We tried to measure fibonacci implementation. But I hardcoded an argument. Sometimes we'd like to measure one method with different arguments. We don't have to define BM_Fibonacci_5, BM_Fibonacci_42, BM_Fibonacci_87 functions for different arguments. Let's look at the improvement of previous code:

static void BM_Fibonacci(benchmark::State& state)
{
  int ret;
  while (state.KeepRunning())
    ret |= complex_computation(state.range_x());
  CHECK(ret != 0);
}

BENCHMARK(BM_Fibonacci)->Arg(5)->Arg(42)->Arg(87);

and the output is also quite different:

loganek@hf-gcs-computer:~/Documents/blog-benchmark$ ./benchmark 
Reading /proc/self/cputime_ns failed. Using getrusage().
Benchmarking on 4 X 2701 MHz CPUs
2014/09/29-01:20:03
CPU scaling is enabled: Benchmark timings may be noisy.
DEBUG: Benchmark         Time(ns)    CPU(ns) Iterations
-------------------------------------------------------
DEBUG: BM_Fibonacci/5           3         19   25832109                                  
DEBUG: BM_Fibonacci/42         18         35   14398500                                  
DEBUG: BM_Fibonacci/87         44         61    8224413   

We've access for results of every argument, which I passed to a benchmark method.
It is also possible to pass range of arguments to our benchmark method.

BENCHMARK(BM_Fibonacci)->Range(0, 1024);

Defined range runs benchmark for min, max in this range, and every power of 8 from this range. So in my case: 0, 1, 8, 64, 512 and 1024.

Multithreading, other features

Google Benchmark framework gives a few other features for developers:

• multithreading
• benchmark templates
• pair of arguments
• custom reporter class

You may see usage of other features in google benchmark example code [2] or on README page[3].

Summary

That's one of first test framework, which I used for now. Previously I wrote simple methods for measurements, but it wasn't so convenient, and in the long run, it's just waste of time. Moreover, benchmark helps me keep in order my tests, so my code is much cleaner than before.
In the future, I'd like to check also other frameworks; I heard about Celero [4] and hayai [5]. But for now, google benchmark fits my needs.

Links
[1] https://github.com/google/benchmark
[2] https://github.com/google/benchmark/blob/master/test/benchmark_test.cc
[3] https://github.com/google/benchmark/blob/master/README.md
[4] https://github.com/DigitalInBlue/Celero
[5] https://github.com/nickbruun/hayai