hsbencher: Flexible benchmark runner for Haskell and non-Haskell benchmarks.

[ bsd3, development, library ] [ Propose Tags ]

Benchmark frameworks are usually very specific to the host language/environment. Hence they are usually about as reusable as compiler passes (that is, not).

Nevertheless, hsbencher is an attempt at a reusable benchmark framework. It knows fairly little about what the benchmarks do, and is mostly concerned with defining and iterating through configuration spaces (e.g. varying the number of threads), and managing the data that results.

Benchmark data is stored in simple text files, and optionally uploaded to Google Fusion Tables.

hsbencher attempts to stradle the divide between language-specific and language-agnostic by having an extensible set of BuildMethods. As shipped, hsbencher knows a little about cabal, ghc, and less about Make, but it can be taught more.

The general philosophy is to have benchmarks follow a simple protocol, for example printing out a line "SELFTIMED: 3.3s" if they wish to report their own timing. The focus is on benchmarks that run long enough to run in their own process. This is typical of parallelism benchmarks and different than the fine grained benchmarks that are well supported by Criterion.

hsbencher is used by creating a script or executable that imports HSBencher and provides a list of benchmarks, each of which is decorated with its parameter space. Below is a minimal example that creates a two-configuration parameter space:

import HSBencher
main = defaultMainWithBechmarks
.      [ Benchmark "bench1/bench1.cabal" ["1000"] $
.        Or [ Set NoMeaning (RuntimeParam "+RTS -qa -RTS")
.            , Set NoMeaning (RuntimeEnv "HELLO" "yes") ] ]

The output would appear as in this gist: https://gist.github.com/rrnewton/5667800

More examples can be found here: https://github.com/rrnewton/HSBencher/tree/master/example

ChangesLog:


[Skip to Readme]
Versions [faq] 1.0, 1.1, 1.1.0.1, 1.1.0.2, 1.2, 1.3.1, 1.3.4, 1.3.6, 1.3.8, 1.3.9, 1.5.1, 1.5.3, 1.5.3.1, 1.8.0.4, 1.12, 1.14, 1.14.1, 1.20, 1.20.0.1, 1.20.0.2, 1.20.0.3, 1.20.0.5
Dependencies async, base (>=4.5 && <=4.7), bytestring, containers, directory, filepath, GenericPretty (>=1.2), handa-gdata (>=0.6.9.1), http-conduit, hydra-print (>=0.1.0.3), io-streams (>=1.1), mtl, process (>=1.2), random, time, unix [details]
License BSD-3-Clause
Copyright (c) Ryan Newton 2013
Author Ryan Newton
Maintainer rrnewton@gmail.com
Category Development
Uploaded by RyanNewton at Wed Feb 19 14:39:55 UTC 2014
Distributions NixOS:1.20.0.5
Downloads 8042 total (249 in the last 30 days)
Rating (no votes yet) [estimated by rule of succession]
Your Rating
  • λ
  • λ
  • λ
Status Hackage Matrix CI
Docs available [build log]
Successful builds reported [all 1 reports]

Modules

[Index]

Flags

NameDescriptionDefaultType
fusion

Add support for Google Fusion Table upload of benchmark data.

EnabledAutomatic
hydra

Add support for (and dependency on) the hydra-print library.

DisabledAutomatic

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

Maintainer's Corner

For package maintainers and hackage trustees


Readme for hsbencher-1.5.3.1

[back to package description]

See hsbencher.cabal for a general overview.

Protocols for benchmarks to follow

All benchmarks, via all BuildMethods

Benchmarks using any BuildMethod, including BuildMethods added by the end user obey the following conventions:

  • Timing -- complete process runtime is used by default, this can be overridden by having the benchmark print out a line such as SELFTIMED 3.3 on stdout , which would indicate a 3.3 second runtime.

  • Coming soon -- compile time timing and multi-phase timing (e.g. for Accelerate)

  • "shortrun" -- all benchmarks should run and do SOMETHING even if given no runtime arguments. The convention is for quick tests (correctness, not performance) to run in this way. HSBencher supports a --shortrun mode that enables this. Note that it still passes in environment variables and "parameters", it elides only the cmdargs field of the Benchmark record.

Benchmarks using the builtin 'make' BuildMethod

  • make should build the benchmark.
  • make run should run the benchmark
  • compile time arguments are provided with make COMPILE_ARGS='...'
  • runtime arguments are provided with make run RUN_ARGS='...'

Benchmarks using the builtin cabal BuildMethod

  • One .cabal file should be contained in the directory.
  • Either the directory itself, or a file within the directory can be the benchmark "target".
  • ONE executable target should be built when cabal install or cabal-dev install is invoked.
  • compile time arguments are formatted for cabal-install
  • runtime arguments are provided to the resulting executable, raw (i.e. you need to include +RTS -RTS yourself)

Benchmarks using the builtin ghc BuildMethod

  • A single .hs file should be specified as the build target
  • It should build by running ghc --make on the target file; any include directories beyond the one containing the target file must be added explicitly (as CompileParam's)
  • compile time arguments are formatted for ghc command line
  • runtime arguments are provided to the resulting executable, raw (i.e. you need to include +RTS -RTS yourself)