Portability | GHC |
---|---|
Stability | experimental |
Maintainer | bos@serpentine.com |
Safe Haskell | None |
Wrappers for compiling and running benchmarks quickly and easily.
See defaultMain
below for an example.
- class Benchmarkable a where
- data Benchmark
- data Pure
- bench :: Benchmarkable b => String -> b -> Benchmark
- bgroup :: String -> [Benchmark] -> Benchmark
- bcompare :: [Benchmark] -> Benchmark
- nf :: NFData b => (a -> b) -> a -> Pure
- whnf :: (a -> b) -> a -> Pure
- nfIO :: NFData a => IO a -> IO ()
- whnfIO :: IO a -> IO ()
- defaultMain :: [Benchmark] -> IO ()
- defaultMainWith :: Config -> Criterion () -> [Benchmark] -> IO ()
- defaultOptions :: [OptDescr (IO Config)]
- parseArgs :: Config -> [OptDescr (IO Config)] -> [String] -> IO (Config, [String])
How to write benchmarks
The Benchmarkable
typeclass represents the class of all code that
can be benchmarked. Every instance must run a benchmark a given
number of times. We are most interested in benchmarking two things:
-
IO
actions. AnyIO
action can be benchmarked directly. - Pure functions. GHC optimises aggressively when compiling with
-O
, so it is easy to write innocent-looking benchmark code that doesn't measure the performance of a pure function at all. We work around this by benchmarking both a function and its final argument together.
Benchmarking IO actions
Benchmarking pure code
Because GHC optimises aggressively when compiling with -O
, it is
potentially easy to write innocent-looking benchmark code that will
only be evaluated once, for which all but the first iteration of
the timing loop will be timing the cost of doing nothing.
To work around this, we provide a special type, Pure
, for
benchmarking pure code. Values of this type are constructed using
one of two functions.
The first is a function which will cause results to be evaluated to head normal form (NF):
nf
::NFData
b => (a -> b) -> a ->Pure
The second will cause results to be evaluated to weak head normal form (the Haskell default):
whnf
:: (a -> b) -> a ->Pure
As both of these types suggest, when you want to benchmark a function, you must supply two values:
- The first element is the function, saturated with all but its last argument.
- The second element is the last argument to the function.
Here is an example that makes the use of these functions clearer. Suppose we want to benchmark the following function:
firstN :: Int -> [Int] firstN k = take k [(0::Int)..]
So in the easy case, we construct a benchmark as follows:
nf
firstN 1000
The compiler will correctly infer that the number 1000 must have
the type Int
, and the type of the expression is Pure
.
Fully evaluating a result
The whnf
harness for evaluating a pure function only evaluates
the result to weak head normal form (WHNF). If you need the result
evaluated all the way to normal form, use the nf
function to
force its complete evaluation.
Using the firstN
example from earlier, to naive eyes it might
appear that the following code ought to benchmark the production
of the first 1000 list elements:
whnf
firstN 1000
Because in this case the result will only be forced until it reaches WHNF, what this would actually benchmark is merely the production of the first list element!
Types
class Benchmarkable a whereSource
A benchmarkable function or action.
Run a function or action the specified number of times.
A benchmark may consist of either a single Benchmarkable
item
with a name, created with bench
, or a (possibly nested) group of
Benchmark
s, created with bgroup
.
A container for a pure function to benchmark, and an argument to supply to it each time it is evaluated.
Constructing benchmarks
:: Benchmarkable b | |
=> String | A name to identify the benchmark. |
-> b | |
-> Benchmark |
Create a single benchmark.
:: String | A name to identify the group of benchmarks. |
-> [Benchmark] | Benchmarks to group under this name. |
-> Benchmark |
Group several benchmarks together under a common name.
bcompare :: [Benchmark] -> BenchmarkSource
Compare benchmarks against a reference benchmark
(The first bench
in the given list).
The results of the comparisons are written to a CSV file specified using the
-r
command line flag. The CSV file uses the following format:
Reference,Name,% faster than the reference
nf :: NFData b => (a -> b) -> a -> PureSource
Apply an argument to a function, and evaluate the result to head normal form (NF).
whnf :: (a -> b) -> a -> PureSource
Apply an argument to a function, and evaluate the result to weak head normal form (WHNF).
nfIO :: NFData a => IO a -> IO ()Source
Perform an action, then evaluate its result to head normal form. This is particularly useful for forcing a lazy IO action to be completely performed.
Perform an action, then evaluate its result to weak head normal form (WHNF). This is useful for forcing an IO action whose result is an expression to be evaluated down to a more useful value.
Running benchmarks
defaultMain :: [Benchmark] -> IO ()Source
An entry point that can be used as a main
function.
import Criterion.Main fib :: Int -> Int fib 0 = 0 fib 1 = 1 fib n = fib (n-1) + fib (n-2) main = defaultMain [ bgroup "fib" [ bench "10" $ whnf fib 10 , bench "35" $ whnf fib 35 , bench "37" $ whnf fib 37 ] ]
:: Config | |
-> Criterion () | Prepare data prior to executing the first benchmark. |
-> [Benchmark] | |
-> IO () |
An entry point that can be used as a main
function, with
configurable defaults.
Example:
import Criterion.Config import qualified Criterion.MultiMap as M import Criterion.Main myConfig = defaultConfig { -- Always GC between runs. cfgPerformGC = ljust True } main = defaultMainWith myConfig (return ()) [ bench "fib 30" $ whnf fib 30 ]
If you save the above example as "Fib.hs"
, you should be able
to compile it as follows:
ghc -O --make Fib
Run "Fib --help"
on the command line to get a list of command
line options.
Other useful code
defaultOptions :: [OptDescr (IO Config)]Source
The standard options accepted on the command line.