bench-graph: Plot and compare benchmarks

[ benchmarking, bsd3, library, performance ] [ Propose Tags ]

Plot benchmarks and compare them. An easy to use package to produce pretty graphs from the csv file generated by criterion or gauge in a few lines of code. It is high level yet pretty flexible in what you can do with it e.g. you can compare groups of benchmarks side by side showing full measurements or a delta, or plot the performance regression of one version of your package against another version.

See the README for comprehensive documentation.


[Skip to Readme]
Versions 0.1.0, 0.1.1, 0.1.2, 0.1.3
Change log Changelog.md
Dependencies base (>=4.8 && <5), Chart (>=1.6 && <2), Chart-diagrams (>=1.6 && <2), csv (==0.1.*), directory (>=1.2 && <1.4), filepath (>=1.3 && <1.5), transformers (>=0.4 && <0.6) [details]
License BSD-3-Clause
Copyright 2017 Composewell Technologies
Author Harendra Kumar
Maintainer harendra.kumar@gmail.com
Category Performance, Benchmarking
Home page https://github.com/composewell/bench-graph
Bug tracker https://github.com/composewell/bench-graph/issues
Source repo head: git clone https://github.com/composewell/bench-graph
Uploaded by harendra at Thu Jul 12 07:58:37 UTC 2018
Distributions NixOS:0.1.3
Downloads 205 total (44 in the last 30 days)
Rating (no votes yet) [estimated by rule of succession]
Your Rating
  • λ
  • λ
  • λ
Status Docs available [build log]
Last success reported on 2018-07-12 [all 1 reports]
Hackage Matrix CI

Modules

[Index]

Downloads

Maintainer's Corner

For package maintainers and hackage trustees


Readme for bench-graph-0.1.1

[back to package description]

bench-graph

Hackage Build Status Windows Build status

An easy to use package to plot the benchmarking results data generated by gauge or criterion (with --csv or --csvraw options).

Easy and Flexible Charting

Usually we have a bunch of benchmarks in the benchmark measurement code and that is not how we may want to present the results in graph. We may want only a subset of the benchmarks and in a different order, with more palatable names to present. Maybe we want to plot multiple graphs from the results file. bench-graph hides the low level charting details but allows configuration of high level parameters, specifically:

  • translate the names of the benchmarks before charting
  • filter which benchmarks you want to include on the plot
  • specify the order of benchmarks in the plot
  • specify the scales
  • plot multiple charts from the same measurement file
  • plot comparison (full or difference) between groups of benchmarks

Plot Comparisons

We can compare the difference in benchmark results of different versions of the same package or the difference between similar benchmarks of different packages. Comparison between many benchmarks can be shown side by side as full results of both, or as the difference from a baseline.

See the test file for a complicated example. Simpler charts can be plotted with just a few lines of code. Here is a sample plot from the data in test/results.csv , showing the time taken by three packages:

Full comparison time

Here is a plot of the same data but this time showing full time for the first package and the difference in time (i.e. the excess time taken) for the other two packages.

Delta comparison time

Just use stack test in this package and see all the charts generated in the charts directory.

Plot Performance Regressions

See the streamly repo for an example script using bench-graph to automatically plot perforformance changes across commits. In the repo if you run this command:

$ ./bench.sh --help # prints help
$ ./bench.sh --compare --base HEAD^

It generates a comparative graph of benchmark changes from the previous commit to the HEAD commit. The generated graph looks like this, the negative delta means there is performance improvement:

Delta comparison time

The bench-graph code for this can be found here.

The shell script functionality may be included in the bench-graph itself in future.

Plot Any Fields

Any of the fields available in gauge/criterion output can be plotted, for example:

  • time
  • cycles
  • total allocations (allocated)
  • bytes copied during GC (bytesCopied)
  • CPU time taken by the mutator i.e. other than GC (mutatorCpuSeconds)
  • CPU time taken by the garbage collector (gcCpuSeconds)

Contributions and Feedback

Contributions are welcome! This is just an early draft of the package, with maybe lots of bugs and lack of required functionality, any feedback on improvements is welcome. Please raise an issue for anything you want to suggest or discuss.