See hsbencher.cabal for a general overview.


Protocols for benchmarks to follow
==================================


All benchmarks, via all BuildMethods
------------------------------------

Benchmarks using any BuildMethod, including BuildMethods added by the
end user obey the following conventions:

 * Timing -- complete process runtime is used by default, this can be
   overridden by having the benchmark print out a line such as
   `SELFTIMED 3.3` on stdout , which would indicate a 3.3 second runtime.

 * Coming soon -- compile time timing and multi-phase timing (e.g. for Accelerate) 

 * "shortrun" -- all benchmarks should run and do SOMETHING even if
   given no runtime arguments.  The convention is for quick tests
   (correctness, not performance) to run in this way.  HSBencher
   supports a `--shortrun` mode that enables this.  Note that it still
   passes in environment variables and "parameters", it elides only
   the `cmdargs` field of the `Benchmark` record.

Benchmarks using the builtin 'make' BuildMethod
-----------------------------------------------

 * `make` should build the benchmark.
 * `make run` should run the benchmark
 * compile time arguments are provided with `make COMPILE_ARGS='...'`
 * runtime arguments are provided with `make run RUN_ARGS='...'`

Benchmarks using the builtin cabal BuildMethod
----------------------------------------------

 * One .cabal file should be contained in the directory.
 * Either the directory itself, or a file within the directory can be
   the benchmark "target".
 * ONE executable target should be built when `cabal install` or
   `cabal-dev install` is invoked.
 * compile time arguments are formatted for cabal-install
 * runtime arguments are provided to the resulting executable, raw
   (i.e. you need to include `+RTS -RTS` yourself)

Benchmarks using the builtin ghc BuildMethod
--------------------------------------------

 * A single .hs file should be specified as the build target
 * It should build by running `ghc --make` on the target file; any
   include directories beyond the one containing the target file must
   be added explicitly (as CompileParam's)
 * compile time arguments are formatted for ghc command line 
 * runtime arguments are provided to the resulting executable, raw
   (i.e. you need to include `+RTS -RTS` yourself)