Copyright | (c) Ivan Lazar Miljenovic |
---|---|
License | MIT |
Maintainer | Ivan.Miljenovic@gmail.com |
Safe Haskell | None |
Language | Haskell2010 |
Make it easier to compare benchmarks and to test that benchmarks are indeed valid.
At the top level you will probably run the testBench
function, and
create comparisons using compareFunc
or the list-based variants.
For example:
main :: IO () main = testBench $ do -- Compare how long it takes to make a list of the specified length. compareFunc "List length" (\n -> length (replicate n ()) == n) [testWith (@? "Not as long as specified"), benchNormalForm] (mapM_ (\n -> comp ("len == " ++ show n) n) [1..5])
When run, the output will look something like:
Cases: 7 Tried: 7 Errors: 0 Failures: 0 Mean MeanLB MeanUB Stddev StddevLB StddevUB OutlierVariance List length len == 1 323.8 ns 318.6 ns 335.9 ns 23.86 ns 5.834 ns 40.90 ns 83% len == 2 352.8 ns 349.1 ns 358.1 ns 15.05 ns 11.76 ns 19.62 ns 61% len == 3 372.4 ns 358.4 ns 393.8 ns 62.50 ns 39.83 ns 90.85 ns 96% len == 4 396.3 ns 378.4 ns 419.2 ns 67.83 ns 46.71 ns 94.74 ns 96% len == 5 426.0 ns 407.0 ns 459.5 ns 82.23 ns 53.37 ns 110.2 ns 97%
Synopsis
- type TestBench = TestBenchM ()
- testBench :: TestBench -> IO ()
- testBenchWith :: Config -> TestBench -> IO ()
- testBenchConfig :: Config
- collection :: String -> TestBench -> TestBench
- compareFunc :: ProvideParams params a b => String -> (a -> b) -> params -> Comparison a b -> TestBench
- compareFuncList :: (ProvideParams params a b, Show a, Eq b, Show b) => String -> (a -> b) -> params -> [a] -> TestBench
- compareFuncListIO :: (ProvideParams params a (IO b), Show a, Eq b, Show b) => String -> (a -> IO b) -> params -> [a] -> TestBench
- compareFuncListWith :: (ProvideParams params a b, Show a) => (String -> a -> CompParams a b) -> String -> (a -> b) -> params -> [a] -> TestBench
- compareFuncList' :: (ProvideParams params a b, Show a) => String -> (a -> b) -> params -> [a] -> TestBench
- compareFuncAll :: (ProvideParams params a b, Show a, Enum a, Bounded a, Eq b, Show b) => String -> (a -> b) -> params -> TestBench
- compareFuncAllIO :: (ProvideParams params a (IO b), Show a, Enum a, Bounded a, Eq b, Show b) => String -> (a -> IO b) -> params -> TestBench
- compareFuncAllWith :: (ProvideParams params a b, Show a, Enum a, Bounded a, Eq b, Show b) => (String -> a -> CompParams a b) -> String -> (a -> b) -> params -> TestBench
- compareFuncAll' :: (ProvideParams params a b, Show a, Enum a, Bounded a) => String -> (a -> b) -> params -> TestBench
- data CompParams a b
- class ProvideParams cp a b | cp -> a b where
- toParams :: cp -> CompParams a b
- normalForm :: NFData b => CompParams a b
- normalFormIO :: NFData b => CompParams a (IO b)
- benchNormalForm :: NFData b => CompParams a b
- benchIO :: CompParams a (IO b)
- benchNormalFormIO :: NFData b => CompParams a (IO b)
- withBenchMode :: ((a -> b) -> a -> Benchmarkable) -> CompParams a b
- noBenchmarks :: CompParams a b
- baseline :: (Eq b, Show b) => String -> a -> CompParams a b
- baselineIO :: (Eq b, Show b) => String -> a -> CompParams a (IO b)
- baselineWith :: (b -> b -> Assertion) -> String -> a -> CompParams a b
- testWith :: (b -> Assertion) -> CompParams a b
- noTests :: CompParams a b
- weigh :: NFData b => CompParams a b
- weighIO :: NFData b => CompParams a (IO b)
- data GetWeight
- getWeight :: NFData b => (a -> b) -> a -> GetWeight
- getWeightIO :: NFData b => (a -> IO b) -> a -> GetWeight
- type Comparison a b = ComparisonM a b ()
- comp :: String -> a -> Comparison a b
- compBench :: String -> a -> Comparison a b
- compTest :: String -> a -> Comparison a b
- data ComparisonM a b r
- getTestBenches :: TestBench -> IO (Test, EvalForest)
- data Eval = Eval {}
- type EvalTree = LabelTree Eval
- type EvalForest = [EvalTree]
- flattenBenchForest :: EvalForest -> [Benchmark]
- evalForest :: Config -> EvalForest -> IO ()
- nfEq :: (NFData b, Show b, Eq b) => b -> (a -> b) -> String -> a -> TestBench
- whnfEq :: (Show b, Eq b) => b -> (a -> b) -> String -> a -> TestBench
- mkTestBench :: ((a -> b) -> a -> Maybe Benchmarkable) -> ((a -> b) -> a -> Maybe GetWeight) -> (b -> Maybe Assertion) -> (a -> b) -> String -> a -> TestBench
- data TestBenchM r
- type OpTree = LabelTree Operation
- data Operation
- data LabelTree a
- type Depth = Int
Specification and running
type TestBench = TestBenchM () Source #
An environment for combining testing and benchmarking.
testBench :: TestBench -> IO () Source #
Run the specified benchmarks if and only if all tests pass, using a comparison-based format for benchmarking output.
For more control, use getTestBenches
.
testBenchWith :: Config -> TestBench -> IO () Source #
As with testBench
but allow specifying a custom default
Config
parameter rather than testBenchConfig
.
Since: 0.2.0.0
testBenchConfig :: Config Source #
This is the same as defaultConfig
from criterion but with the
verbosity set to Quiet
to avoid unnecessary noise on stdout.
Since: 0.2.0.0
Grouping
Comparisons
compareFunc :: ProvideParams params a b => String -> (a -> b) -> params -> Comparison a b -> TestBench Source #
Compare how various input values (of the same type) behave for a specific function.
By default:
- Results are only evaluated to Weak Head Normal Form. To
fully evaluate results, use
benchNormalForm
. - No tests are performed by default; use either
baseline
ortestWith
to specify one.
List of input values
Rather than manually stating all the arguments - especially if you're either a) dealing with a few different types or b) repeating all the possible targets a few times - it can be helpful to instead use an enum type to indicate all the possible options with a helper type class to generate all the possible benchmarks.
For example, consider a case where you wish to compare data structures
of Word8
values:
import qualified Data.ByteString as SB import qualified Data.ByteString.Lazy as LB import Data.Monoid ((<>)) import Data.Proxy (Proxy(..)) import qualified Data.Sequence as Seq -- | All the types we care about. data SequenceType = List | Sequence | StrictBS | LazyBS deriving (Eq, Ord, Show, Read, Enum, Bounded) -- | The function we actually want to benchmark. listLength :: (Sequential l) => Proxy l -> Int listLength st = len (st `pack` sampleList) -- | How to run a function on our chosen type. chooseType :: SequenceType -> (forall s. (Sequential s) => Proxy s -> k) -> k chooseType List k = k (Proxy :: Proxy [Word8]) chooseType Sequence k = k (Proxy :: Proxy (Seq.Seq Word8)) chooseType StrictBS k = k (Proxy :: Proxy SB.ByteString) chooseType LazyBS k = k (Proxy :: Proxy LB.ByteString) sampleList :: [Word8] sampleList = replicate 1000000 0 -- | A common type class containing all the functions we want to test. class Sequential xs where len :: xs -> Int pack :: Proxy xs -> [Word8] -> xs instance Sequential [Word8] where len = length pack _ = id instance Sequential (Seq.Seq Word8) where len = length pack _ = Seq.fromList instance Sequential SB.ByteString where len = SB.length pack _ = SB.pack instance Sequential LB.ByteString where len = fromIntegral . LB.length pack _ = LB.pack
We can then write as our benchmark:
compareFuncAll
"Packing and length" (flip chooseType listLength)normalForm
This may seem like a lot of up-front work just to avoid having to
write out all the cases manually, but if you write a lot of similar
benchmarks comparing different aspects of these sequential structures
then the chooseType
function ends up being rather trivial to write
(but alas, barring Template Haskell, not possible to easily automate).
Furthermore, you can now be sure that you won't forget a case!
compareFuncList :: (ProvideParams params a b, Show a, Eq b, Show b) => String -> (a -> b) -> params -> [a] -> TestBench Source #
As with compareFunc
but use the provided list of values to base
the benchmarking off of.
This is useful in situations where you create an enumeration type to describe everything you're benchmarking and a function that takes one of these values and evaluates it.
baseline
is used on the first value (if non-empty); the Show
instance is used to provide labels.
Since: 0.2.0.0
compareFuncListIO :: (ProvideParams params a (IO b), Show a, Eq b, Show b) => String -> (a -> IO b) -> params -> [a] -> TestBench Source #
A variant of compareFuncList
that allows for the function to
return an IO
value.
Since: 0.2.0.0
compareFuncListWith :: (ProvideParams params a b, Show a) => (String -> a -> CompParams a b) -> String -> (a -> b) -> params -> [a] -> TestBench Source #
A variant of compareFuncList
where you provide your own
equivalent to baseline
.
Most useful with baselineWith
.
Since: 0.2.0.0
compareFuncList' :: (ProvideParams params a b, Show a) => String -> (a -> b) -> params -> [a] -> TestBench Source #
A variant of compareFuncList
that doesn't use baseline
(allowing you to specify your own test).
Since: 0.2.0.0
compareFuncAll :: (ProvideParams params a b, Show a, Enum a, Bounded a, Eq b, Show b) => String -> (a -> b) -> params -> TestBench Source #
An extension to compareFuncList
that uses the Bounded
and
Enum
instances to generate the list of all values.
Since: 0.2.0.0
compareFuncAllIO :: (ProvideParams params a (IO b), Show a, Enum a, Bounded a, Eq b, Show b) => String -> (a -> IO b) -> params -> TestBench Source #
An extension to compareFuncListIO
that uses the Bounded
and
Enum
instances to generate the list of all values.
Since: 0.2.0.0
compareFuncAllWith :: (ProvideParams params a b, Show a, Enum a, Bounded a, Eq b, Show b) => (String -> a -> CompParams a b) -> String -> (a -> b) -> params -> TestBench Source #
An extension to compareFuncListWith
that uses the Bounded
and
Enum
instances to generate the list of all values.
Since: 0.2.1.0
compareFuncAll' :: (ProvideParams params a b, Show a, Enum a, Bounded a) => String -> (a -> b) -> params -> TestBench Source #
A variant of comapreFuncAll
that doesn't use baseline
(allowing you to specify your own test).
Since: 0.2.0.0
Comparison parameters
data CompParams a b Source #
Monoidally build up the parameters used to control a Comparison
environment.
This will typically be a combination of benchNormalForm
with
either baseline
or testWith
.
Instances
ProvideParams [CompParams a b] a b Source # | |
Defined in TestBench toParams :: [CompParams a b] -> CompParams a b Source # | |
Semigroup (CompParams a b) Source # | |
Defined in TestBench (<>) :: CompParams a b -> CompParams a b -> CompParams a b # sconcat :: NonEmpty (CompParams a b) -> CompParams a b # stimes :: Integral b0 => b0 -> CompParams a b -> CompParams a b # | |
Monoid (CompParams a b) Source # | |
Defined in TestBench mempty :: CompParams a b # mappend :: CompParams a b -> CompParams a b -> CompParams a b # mconcat :: [CompParams a b] -> CompParams a b # | |
ProvideParams (CompParams a b) a b Source # | |
Defined in TestBench toParams :: CompParams a b -> CompParams a b Source # |
class ProvideParams cp a b | cp -> a b where Source #
A convenience class to make it easier to provide CompParams
values.
You can either:
- Provide no parameters with
mempty
- Provide values chained together using
ormappend
<>
- Use the list instance and provide a list of
CompParams
values.
Since: 0.2.0.0
toParams :: cp -> CompParams a b Source #
Instances
ProvideParams [CompParams a b] a b Source # | |
Defined in TestBench toParams :: [CompParams a b] -> CompParams a b Source # | |
ProvideParams (CompParams a b) a b Source # | |
Defined in TestBench toParams :: CompParams a b -> CompParams a b Source # |
normalForm :: NFData b => CompParams a b Source #
A combination of benchNormalForm
and weigh
, taking into
account the common case that you want to consider a value that
can - and should - be evaluated to normal form.
Since: 0.2.0.0
normalFormIO :: NFData b => CompParams a (IO b) Source #
A variant of normalForm
where the results are within IO
.
Since: 0.2.0.0
Control benchmarking
benchNormalForm :: NFData b => CompParams a b Source #
Evaluate all benchmarks to normal form.
benchIO :: CompParams a (IO b) Source #
Evaluate all IO-based benchmarks to weak head normal form.
Since: 0.2.0.0
benchNormalFormIO :: NFData b => CompParams a (IO b) Source #
Evaluate all IO-based benchmarks to normal form.
Since: 0.2.0.0
withBenchMode :: ((a -> b) -> a -> Benchmarkable) -> CompParams a b Source #
Allow specifying how benchmarks should be evaluated. This may
allow usage of methods such as nfIO
, but this has not been
tested as yet.
noBenchmarks :: CompParams a b Source #
Don't run any benchmarks. I'm not sure why you'd want to do this as there's surely easier/better testing environments available, but this way it's possible.
Control testing
baselineIO :: (Eq b, Show b) => String -> a -> CompParams a (IO b) Source #
baselineWith :: (b -> b -> Assertion) -> String -> a -> CompParams a b Source #
A variant of baseline
that lets you specify how to test for
equality.
The first argument to the provided function will be the "baseline" value; the second will be the value being tested.
Since: 0.2.0.0
testWith :: (b -> Assertion) -> CompParams a b Source #
noTests :: CompParams a b Source #
Don't run any tests. This isn't recommended, but could be useful if all you want to do is run comparisons (potentially because no meaningful tests are possible).
Control function weighing
weigh :: NFData b => CompParams a b Source #
Calculate memory usage of the various parameters.
Note that to achieve this, testBench
and associated functions
will run copies of itself to be able to calculate memory usage in
a pristine environment (i.e. without influence of caching from
testing and benchmarking). As such, you may wish to use the
-threaded
GHC option when building your benchmarking
executable.
Since: 0.2.0.0
getWeightIO :: NFData b => (a -> IO b) -> a -> GetWeight Source #
An IO-based variant of getWeight
.
Since: 0.2.0.0
Specify comparisons
type Comparison a b = ComparisonM a b () Source #
comp :: String -> a -> Comparison a b Source #
Benchmark and test (if specified) this value against the specified function.
compBench :: String -> a -> Comparison a b Source #
Only benchmark and possibly weigh (but do not test) this value against the specified function.
compTest :: String -> a -> Comparison a b Source #
Only test (but do not benchmark or weigh) this value against the specified function.
Lower-level types
data ComparisonM a b r Source #
Instances
Monad (ComparisonM a b) Source # | |
Defined in TestBench (>>=) :: ComparisonM a b a0 -> (a0 -> ComparisonM a b b0) -> ComparisonM a b b0 # (>>) :: ComparisonM a b a0 -> ComparisonM a b b0 -> ComparisonM a b b0 # return :: a0 -> ComparisonM a b a0 # fail :: String -> ComparisonM a b a0 # | |
Functor (ComparisonM a b) Source # | |
Defined in TestBench fmap :: (a0 -> b0) -> ComparisonM a b a0 -> ComparisonM a b b0 # (<$) :: a0 -> ComparisonM a b b0 -> ComparisonM a b a0 # | |
Applicative (ComparisonM a b) Source # | |
Defined in TestBench pure :: a0 -> ComparisonM a b a0 # (<*>) :: ComparisonM a b (a0 -> b0) -> ComparisonM a b a0 -> ComparisonM a b b0 # liftA2 :: (a0 -> b0 -> c) -> ComparisonM a b a0 -> ComparisonM a b b0 -> ComparisonM a b c # (*>) :: ComparisonM a b a0 -> ComparisonM a b b0 -> ComparisonM a b b0 # (<*) :: ComparisonM a b a0 -> ComparisonM a b b0 -> ComparisonM a b a0 # | |
MonadIO (ComparisonM a b) Source # | |
Defined in TestBench liftIO :: IO a0 -> ComparisonM a b a0 # |
Manual construction of a TestBench
getTestBenches :: TestBench -> IO (Test, EvalForest) Source #
Obtain the resulting tests and benchmarks from the specified
TestBench
.
type EvalTree = LabelTree Eval Source #
A more explicit tree-like structure for benchmarks than using
Criterion's Benchmark
type.
type EvalForest = [EvalTree] Source #
flattenBenchForest :: EvalForest -> [Benchmark] Source #
Remove the explicit tree-like structure into the implicit one used by Criterion.
Useful for embedding the results into an existing benchmark suite.
evalForest :: Config -> EvalForest -> IO () Source #
Run the specified benchmarks, printing the results (once they're all complete) to stdout in a tabular format for easier comparisons.
Direct benchmarks/tests
nfEq :: (NFData b, Show b, Eq b) => b -> (a -> b) -> String -> a -> TestBench Source #
Create a single benchmark evaluated to normal form, where the results should equal the value specified.
Will also weigh the function.
whnfEq :: (Show b, Eq b) => b -> (a -> b) -> String -> a -> TestBench Source #
Create a single benchmark evaluated to weak head normal form, where the results should equal the value specified.
:: ((a -> b) -> a -> Maybe Benchmarkable) | Define the benchmark to be performed, if any. |
-> ((a -> b) -> a -> Maybe GetWeight) | If a benchmark is performed, should its memory
usage also be calculated? See the
documentation for |
-> (b -> Maybe Assertion) | Should the result be checked? |
-> (a -> b) | |
-> String | |
-> a | |
-> TestBench |
A way of writing custom testing/benchmarking statements. You will probably want to use one of the pre-defined versions instead.
Lower-level types
data TestBenchM r Source #
Instances
Monad TestBenchM Source # | |
Defined in TestBench (>>=) :: TestBenchM a -> (a -> TestBenchM b) -> TestBenchM b # (>>) :: TestBenchM a -> TestBenchM b -> TestBenchM b # return :: a -> TestBenchM a # fail :: String -> TestBenchM a # | |
Functor TestBenchM Source # | |
Defined in TestBench fmap :: (a -> b) -> TestBenchM a -> TestBenchM b # (<$) :: a -> TestBenchM b -> TestBenchM a # | |
Applicative TestBenchM Source # | |
Defined in TestBench pure :: a -> TestBenchM a # (<*>) :: TestBenchM (a -> b) -> TestBenchM a -> TestBenchM b # liftA2 :: (a -> b -> c) -> TestBenchM a -> TestBenchM b -> TestBenchM c # (*>) :: TestBenchM a -> TestBenchM b -> TestBenchM b # (<*) :: TestBenchM a -> TestBenchM b -> TestBenchM a # | |
MonadIO TestBenchM Source # | |
Defined in TestBench liftIO :: IO a -> TestBenchM a # |
A simple labelled rose-tree data structure also containing the depth.
Instances
Functor LabelTree Source # | |
Eq a => Eq (LabelTree a) Source # | |
Ord a => Ord (LabelTree a) Source # | |
Defined in TestBench.LabelTree | |
Read a => Read (LabelTree a) Source # | |
Show a => Show (LabelTree a) Source # | |