hsbencher: Flexible benchmark runner for Haskell and non-Haskell benchmarks.
Benchmark frameworks are usually very specific to the host language/environment. Hence they are usually about as reusable as compiler passes (that is, not).
hsbencher is an attempt at a reusable benchmark
framework. It knows fairly little about what the benchmarks do, and
is mostly concerned with defining and iterating through
configuration spaces (e.g. varying the number of threads), and
managing the data that results.
Benchmark data is stored in simple text files, and optionally uploaded to Google Fusion Tables.
hsbencher attempts to stradle the divide between language-specific
and language-agnostic by having an extensible set of
hsbencher knows a little about cabal, ghc, and less
about Make, but it can be taught more.
The general philosophy is to have benchmarks follow a simple protocol, for example printing out a line "SELFTIMED: 3.3s" if they wish to report their own timing. The focus is on benchmarks that run long enough to run in their own process. This is typical of parallelism benchmarks and different than the fine grained benchmarks that are well supported by Criterion.
hsbencher is used by creating a script or executable that imports
and provides a list of benchmarks, each of which is decorated with its
parameter space. Below is a minimal example that creates a two-configuration
import HSBencher main = defaultMainWithBechmarks . [ Benchmark "bench1/bench1.cabal" ["1000"] $ . Or [ Set NoMeaning (RuntimeParam "+RTS -qa -RTS") . , Set NoMeaning (RuntimeEnv "HELLO" "yes") ] ]
The output would appear as in this gist: https://gist.github.com/rrnewton/5667800
More examples can be found here: https://github.com/rrnewton/HSBencher/tree/master/example
[Skip to Readme]
|Versions [faq]||1.0, 1.1, 22.214.171.124, 126.96.36.199, 1.2, 1.3.1, 1.3.4, 1.3.6, 1.3.8, 1.3.9, 1.5.1, 1.5.3, 188.8.131.52, 184.108.40.206, 1.12, 1.14, 1.14.1, 1.20, 220.127.116.11, 18.104.22.168, 22.214.171.124, 126.96.36.199|
|Dependencies||async, base (>=4.5 && <=4.7), bytestring, containers, directory, filepath, GenericPretty (>=1.2), handa-gdata (>=0.6.2), http-conduit, hydra-print (>=0.1.0.3), io-streams (>=1.0), mtl, process, random, time, unix [details]|
|Copyright||(c) Ryan Newton 2013|
|Uploaded||by RyanNewton at Wed Jul 10 18:48:54 UTC 2013|
|Downloads||10302 total (546 in the last 30 days)|
|Rating||(no votes yet) [estimated by Bayesian average]|
Docs uploaded by user
Build status unknown [no reports yet]
Add support for Google Fusion Table upload of benchmark data.
Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info
For package maintainers and hackage trustees