Fibon in a Flash =================================================================== $ git clone git://github.com/dmpots/fibon.git $ cd fibon $ git submodule update --init benchmarks $ cabal configure && cabal build $ ./dist/build/fibon-run/fibon-run Introduction =================================================================== Fibon is a set of tools for running and analysing benchmarks in Haskell. Most importantly, it includes a set of default benchmarks taken from the [Hackage][1] open source repository. Benchmarks ------------------ Fibon makes it easy to use either the fibon benchmarks or your own set of benchmarks. Benchmarks are stored in the `benchmarks/Fibon/Benchmarks` directory. This directory is setup as a [git submodule][2] which means you can easily grab the standard suite or use a suite kept under your own source control. Benchmark Groups ------------------ Benchamarks named and organized into groups based on the filesystem organization. For example, a benchmark in the directory `benchmarks/Fibon/Benchmarks/Hackage/Agum` will have the name `Agum` an be in the benchmark group `Hackage`. Executables ------------------ The fibon package builds three tools 1. `fibon-run` - runs the benchmarks 2. `fibon-analyze` - analyzes the results of a run 2. `fibon-init` - utilty used when adding new benchmarks Size and Tune ------------------ Fibon benchmarks can be run with two different input sizes: `Test` and `Ref`. The `Test` size is useful to make sure that a benchmark can run successsfully, but will not give meaningful timings. Fibon benchmarks can be run under two different tune settings (e.g. compiler optimization settings). The `Base` and `Peak` settings can be configured anyway you want to make the desired comparison. Directory Structure -------------------- Source directories ./benchmarks -- benchmark code ./config -- config files ./lib -- common files used by several executables ./tools -- source code for executables Working directories ./log -- logging output from benchmark runs ./run -- working directory for benchmark runs Getting the Benchmarks =================================================================== The benchmarks are kept in a separate repository as a [git submodule][2]. You can get the fibon benchmarks by updating the submodule from within your fibon working directory $ git submodule update --init benchmarks This will checkout the benchmarks from the fibon-benchmarks repository and place them in your working copy. Running Benchmarks =================================================================== The available benchmarks and configurations are discovered when the fibon package is configured. Benchmarks are searched for in the `benchmarks/Fibon/Benchmarks` directory and configuration files are searched for in the `config` directory. If a configuration file or benchmark is added, you will need to re-run `cabal configure` to make them available to the fibon-run tool. Configuration --------------- Fibon comes with a default configuration. The default configuration will run all benchmarks with the `Base` setting of `-O0` and a `Peak` setting of `-O2` on the `Ref` size. A configuration file can be used to specify more complicated configurations. You can get some example configuration by doing $ git submodule update --init config This will checkout a repository of config files. Note that currently these files contain some user and machine-specific configurations, but should be a useful starting point. You can also selectively run benchmarks, groups, sizes, and tune settings as described below. Running --------------- Benchamarks are run with the `fibon-run` tool. Running `fibon-run` with no arguments will use the default config file. An alternate config file can be specified with the `-c` flag. Also, you can give a list of benchmarks or groups to run on the command line. Use `--help` to see a full list of options. Running the benchmarks will produce some logging to standard out and create four output files in the `log` directory. 1. `*.LOG` - the full log of the run 1. `*.SUMMARY` - the mean runtimes of each benchmark 1. `*.RESULTS` - the full results (passed to `fibon-analyse`) Analyzing Benchmark Results =================================================================== Benchmarks can be analyzed by the `fibon-analyse` tool. $ fibon-analyse log/000.default.RESULTS Adding New Benchmarks =================================================================== TODO Benchmark Notes =================================================================== Ghc610 ChameneosRedux Does not work with -O0. Gets "thread blocked indefinitely" exception Mandelbrot The Test size gives different result, but the Ref size is ok. Think it is just some kind of floating point wibbles. [1]: http://hackage.haskell.org [2]: http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#submodules