hsbencher-1.2: Flexible benchmark runner for Haskell and non-Haskell benchmarks.

Safe HaskellNone




Benchmark building

data FilePredicate Source

A description of a set of files. The description may take one of multiple forms.


WithExtension String

E.g. .hs, WITH the dot.

IsExactly String

E.g. Makefile | SatisfiesPredicate (String -> Bool)

InDirectoryWithExactlyOne FilePredicate

A common pattern. For example, we can build a file foo.c, if it lives in a directory with exactly one Makefile.

PredOr FilePredicate FilePredicate

Logical or.


filePredCheck :: FilePredicate -> FilePath -> IO (Maybe FilePath)Source

This function gives meaning to the FilePred type. It returns a filepath to signal True and Nothing otherwise.

data BuildResult Source

The result of doing a build. Note that compile can will throw an exception if compilation fails.


StandAloneBinary FilePath

This binary can be copied and executed whenever.

RunInPlace (RunFlags -> EnvVars -> CommandDescr)

In this case the build return what you need to do the benchmark run, but the directory contents cannot be touched until after than run is finished.


data BuildMethod Source

A completely encapsulated method of building benchmarks. Cabal and Makefiles are two examples of this. The user may extend it with their own methods.




methodName :: String

Identifies this build method for humans. , buildsFiles :: FilePredicate , canBuild :: FilePath -> IO Bool

canBuild :: FilePredicate

Can this method build a given file/directory?

concurrentBuild :: Bool

More than one build can happen at once. This implies that compile always returns StandAloneBinary.

compile :: PathRegistry -> BuildID -> CompileFlags -> FilePath -> BenchM BuildResult
clean :: PathRegistry -> BuildID -> FilePath -> BenchM ()

Clean any left-over build results.

setThreads :: Maybe (Int -> [ParamSetting])

Synthesize a list of compile/runtime settings that will control the number of threads.


mkBenchmark :: FilePath -> [String] -> BenchSpace a -> Benchmark aSource

Make a Benchmark data structure given the core, required set of fields, and uses defaults to fill in the rest. Takes target, cmdargs, configs.

data Benchmark a Source




target :: FilePath

The target file or direcotry.

cmdargs :: [String]

Command line argument to feed the benchmark executable.

configs :: BenchSpace a

The configration space to iterate over.


Eq a => Eq (Benchmark a) 
Ord a => Ord (Benchmark a) 
Show a => Show (Benchmark a) 
Generic (Benchmark a) 
Out a => Out (Benchmark a) 

Benchmark configuration spaces

data BenchSpace meaning Source

A datatype for describing (generating) benchmark configuration spaces. This is accomplished by nested conjunctions and disjunctions. For example, varying threads from 1-32 would be a 32-way Or. Combining that with profiling on/off (product) would create a 64-config space.

While the ParamSetting provides an *implementation* of the behavior, this datatype can also be decorated with a (more easily machine readable) meaning of the corresponding setting. For example, indicating that the setting controls the number of threads.


And [BenchSpace meaning] 
Or [BenchSpace meaning] 
Set meaning ParamSetting 


Eq meaning => Eq (BenchSpace meaning) 
Ord meaning => Ord (BenchSpace meaning) 
Read meaning => Read (BenchSpace meaning) 
Show meaning => Show (BenchSpace meaning) 
Generic (BenchSpace meaning) 
Out a => Out (BenchSpace a) 

data ParamSetting Source

Different types of parameters that may be set or varied.


RuntimeParam String

String contains runtime options, expanded and tokenized by the shell.

CompileParam String

String contains compile-time options, expanded and tokenized by the shell.

RuntimeEnv String String

The name of the env var and its value, respectively. For now Env Vars ONLY affect runtime.

CmdPath String String

Takes CMD PATH, and establishes a benchmark-private setting to use PATH for CMD. For example `CmdPath ghc ghc-7.6.3`. | Threads Int -- ^ Shorthand: builtin support for changing the number of threads across a number of separate build methods.

enumerateBenchSpace :: BenchSpace a -> [[(a, ParamSetting)]]Source

Exhaustively compute all configurations described by a benchmark configuration space.

compileOptsOnly :: BenchSpace a -> BenchSpace aSource

Strip all runtime options, leaving only compile-time options. This is useful for figuring out how many separate compiles need to happen.

isCompileTime :: ParamSetting -> BoolSource

Is it a setting that affects compile time?

type BuildID = StringSource

A BuildID should uniquely identify a particular (compile-time) configuration, but consist only of characters that would be reasonable to put in a filename. This is used to keep build results from colliding.

makeBuildID :: FilePath -> CompileFlags -> BuildIDSource

Performs a simple reformatting (stripping disallowed characters) to create a build ID corresponding to a set of compile flags. To make it unique we also append the target path.

HSBench Driver Configuration

data Config Source

The global configuration for benchmarking:




benchlist :: [Benchmark DefaultParamMeaning]
benchsetName :: Maybe String

What identifies this set of benchmarks? Used to create fusion table.

benchversion :: (String, Double)

benchlist file name and version number (e.g. X.Y)

threadsettings :: [Int]

A list of #threads to test. 0 signifies non-threaded mode.

maxthreads :: Int
trials :: Int

number of runs of each configuration

shortrun :: Bool
doClean :: Bool
keepgoing :: Bool

keep going after error

pathRegistry :: PathRegistry

Paths to executables.

hostname :: String
startTime :: Integer

Seconds since Epoch.

resultsFile :: String

Where to put timing results.

logFile :: String

Where to put more verbose testing output.

gitInfo :: (String, String, Int)

Branch, revision hash, depth.

buildMethods :: [BuildMethod]

Starts with cabalmakeghc, can be extended by user.

logOut :: OutputStream ByteString
resultsOut :: OutputStream ByteString
stdOut :: OutputStream ByteString
envs :: [[(String, String)]]
argsBeforeFlags :: Bool

A global setting to control whether executables are given their 'flags/params' after their regular arguments. This is here because some executables don't use proper command line parsing.

harvesters :: (LineHarvester, Maybe LineHarvester)

Line harvesters for SELFTIMED and productivity lines.

doFusionUpload :: Bool
fusionConfig :: FusionConfig


type BenchM a = ReaderT Config IO aSource

A monad for benchamrking. This provides access to configuration options, but really, its main purpose is enabling logging.

data FusionConfig Source




fusionTableID :: Maybe TableId

This must be Just whenever doFusionUpload is true.

fusionClientID :: Maybe String
fusionClientSecret :: Maybe String


Subprocesses and system commands

data CommandDescr Source

A self-contained description of a runnable command. Similar to System.Process.CreateProcess but slightly simpler.




command :: CmdSpec

Executable and arguments

envVars :: [(String, String)]

Environment variables to APPEND to current env.

timeout :: Maybe Double

Optional timeout in seconds.

workingDir :: Maybe FilePath

Optional working directory to switch to before running command.

data RunResult Source

Measured results from running a subprocess (benchmark).




realtime :: Double

Benchmark time in seconds, may be different than total process time.

productivity :: Maybe Double


ExitError Int

Contains the returned error code.

data SubProcess Source

A running subprocess.




wait :: IO RunResult
process_out :: InputStream ByteString

A stream of lines.

process_err :: InputStream ByteString

A stream of lines.

newtype LineHarvester Source

Things like SELFTIMED that should be monitored. type Tags = [String]


Benchmark outputs for upload

data BenchmarkResult Source

This contains all the contextual information for a single benchmark run, which makes up a row in a table of benchmark results. Note that multiple trials (actual executions) go into a single BenchmarkResult

emptyBenchmarkResult :: BenchmarkResultSource

A default value, useful for filling in only the fields that are relevant to a particular benchmark.

For convenience; large records call for pretty-printing

doc :: Out a => a -> Doc

doc is the equivalent of show

This is a specialised variant of docPrec, using precedence context zero.