# bench-graph

An easy to use package to plot the benchmarking results data generated by
`gauge` or `criterion` (with `--csv` or `--csvraw` options).

## Easy and Flexible Charting

Usually we have a bunch of benchmarks in the benchmark measurement
code and that is not how you may want to present the results in graph. We may
want only a subset of the benchmarks and in a different order, with more
palatable names to present. Maybe we want to plot multiple graphs from the
results file. `bench-graph` hides the low level charting details but allows
configuration of high level parameters, specifically:

* translate the names of the benchmarks before charting
* filter which benchmarks you want to include on the plot
* specify the order of benchmarks in the plot
* specify the scales
* plot multiple charts from the same measurement file
* plot comparison (full or difference) between groups of benchmarks

## Plot Comparisons

We can compare the difference in benchmark results of different versions of
the same package or the difference between similar benchmarks of different
packages. Comparison between many benchmarks can be shown side by side as full
results of both, or as the difference from a baseline.

See the [test
file](https://github.com/composewell/bench-graph/blob/master/test/Main.hs) for
a complicated example. Simpler charts can be plotted with just a few lines of
code. Here is a sample plot from the data in [test/results.csv
](https://github.com/composewell/bench-graph/blob/master/test/results.csv),
showing the time taken by three packages:

[![Full comparison time](https://github.com/composewell/bench-graph/blob/master/sample-charts/csvraw-time-full.svg)](https://github.com/composewell/bench-graph/blob/master/sample-charts/csvraw-time-full.svg)

Here is a plot of the same data but this time showing full time for the first
package and the difference in time (i.e. the excess time taken) for the other
two packages.

[![Delta comparison time](https://github.com/composewell/bench-graph/blob/master/sample-charts/csvraw-time-delta.svg)](https://github.com/composewell/bench-graph/blob/master/sample-charts/csvraw-time-delta.svg)

Just use `stack test` in this package and see all the charts generated in the
`charts` directory.

## Plot Performance Regressions

See the [streamly](https://github.com/composewell/streamly) repo for an example
script using `bench-graph` to automatically plot perforformance changes across
commits. In the repo if you run this command:

```
$ ./bench.sh --help # prints help
$ ./bench.sh --compare --base HEAD^
```

It generates a comparative graph of benchmark changes from the previous commit
to the HEAD commit. The generated graph looks like this, the negative delta
means there is performance improvement:

[![Delta comparison time](https://github.com/composewell/bench-graph/blob/master/sample-charts/composition-scaling.svg)](https://github.com/composewell/bench-graph/blob/master/sample-charts/composition-scaling.svg)

The `bench-graph` code for this can be found
[here](https://github.com/composewell/streamly/blob/master/benchmark/ChartLinear.hs).

The shell script functionality may be included in the `bench-graph` itself in
future.

## Plot Any Fields

Any of the fields available in `gauge`/`criterion` output can be plotted, for
example:

* time
* cycles
* total allocations (allocated)
* bytes copied during GC (bytesCopied)
* CPU time taken by the mutator i.e. other than GC (mutatorCpuSeconds)
* CPU time taken by the garbage collector (gcCpuSeconds)

## Contributions and Feedback

Contributions are welcome! This is just an early draft of the package, with
maybe lots of bugs and lack of required functionality, any feedback on
improvements is welcome. Please raise an issue for anything you want to suggest
or discuss.