The hsbencher package

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.



Benchmark frameworks are usually very specific to the host language/environment. Hence they are usually about as reusable as compiler passes (that is, not).

Nevertheless, hsbencher is an attempt at a reusable benchmark framework. It knows fairly little about what the benchmarks do, and is mostly concerned with defining and iterating through configuration spaces (e.g. varying the number of threads), and managing the data that results.

Benchmark data is stored in simple text files, and optionally uploaded to Google Fusion Tables.

hsbencher attempts to stradle the divide between language-specific and language-agnostic by having an extensible set of BuildMethods. As shipped, hsbencher knows a little about cabal, ghc, and less about Make, but it can be taught more.

The general philosophy is to have benchmarks follow a simple protocol, for example printing out a line SELFTIMED: 3.3s if they wish to report their own timing. The focus is on benchmarks that run long enough to run in their own process. This is typical of parallelism benchmarks and different than the fine grained benchmarks that are well supported by Criterion.

hsbencher is used by creating a script or executable that imports HSBencher and provides a list of benchmarks, each of which is decorated with its parameter space. Below is a minimal example that creates a two-configuration parameter space:

import HSBencher
main = defaultMainWithBechmarks
.      [ Benchmark "bench1/bench1.cabal" ["1000"] $
.        Or [ Set NoMeaning (RuntimeParam "+RTS -qa -RTS")
.            , Set NoMeaning (RuntimeEnv "HELLO" "yes") ] ]

The output would appear as in this gist:

More examples can be found here:


[Skip to ReadMe]


Versions1.0, 1.1,,, 1.2, 1.3.1, 1.3.4, 1.3.6, 1.3.8, 1.3.9, 1.4.2, 1.5.1, 1.5.3,,, 1.12, 1.14, 1.14.1, 1.20,,,,
Change logNone available
Dependenciesasync, base (>=4.5 && <=4.7), bytestring, containers, directory, filepath, GenericPretty (>=1.2), handa-gdata (>=0.6.2), http-conduit, hydra-print (>=, io-streams (>=1.1), mtl, process, random, time, unix [details]
Copyright(c) Ryan Newton 2013
AuthorRyan Newton
UploadedTue Nov 19 04:42:53 UTC 2013 by RyanNewton



fusionAdd support for Google Fusion Table upload of benchmark data.EnabledAutomatic
hydraAdd support for (and dependency on) the hydra-print library.DisabledAutomatic

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info


Maintainers' corner

For package maintainers and hackage trustees

Readme for hsbencher-1.4.2

See hsbencher.cabal for a general overview.

Protocols for benchmarks to follow

All benchmarks, via all BuildMethods

Benchmarks using any BuildMethod, including BuildMethods added by the end user obey the following conventions:

Benchmarks using the builtin 'make' BuildMethod

Benchmarks using the builtin cabal BuildMethod

Benchmarks using the builtin ghc BuildMethod