hsbencher-1.1: Flexible benchmark runner for Haskell and non-Haskell benchmarks.

Safe HaskellNone

HSBencher.Types

Contents

Synopsis

Benchmark building

type RunFlags = [String]Source

type CompileFlags = [String]Source

data FilePredicate Source

A description of a set of files. The description may take one of multiple forms.

Constructors

WithExtension String

E.g. .hs, WITH the dot.

IsExactly String

E.g. Makefile | SatisfiesPredicate (String -> Bool)

InDirectoryWithExactlyOne FilePredicate

A common pattern. For example, we can build a file foo.c, if it lives in a directory with exactly one Makefile.

PredOr FilePredicate FilePredicate

Logical or.

Instances

filePredCheck :: FilePredicate -> FilePath -> IO (Maybe FilePath)Source

This function gives meaning to the FilePred type. It returns a filepath to signal True and Nothing otherwise.

data BuildResult Source

The result of doing a build. Note that compile can will throw an exception if compilation fails.

Constructors

StandAloneBinary FilePath

This binary can be copied and executed whenever.

RunInPlace (RunFlags -> EnvVars -> CommandDescr)

In this case the build return what you need to do the benchmark run, but the directory contents cannot be touched until after than run is finished.

Instances

data BuildMethod Source

A completely encapsulated method of building benchmarks. Cabal and Makefiles are two examples of this. The user may extend it with their own methods.

Constructors

BuildMethod 

Fields

methodName :: String

Identifies this build method for humans. , buildsFiles :: FilePredicate , canBuild :: FilePath -> IO Bool

canBuild :: FilePredicate

Can this method build a given file/directory?

concurrentBuild :: Bool

More than one build can happen at once. This implies that compile always returns StandAloneBinary.

compile :: PathRegistry -> BuildID -> CompileFlags -> FilePath -> BenchM BuildResult
 
clean :: PathRegistry -> BuildID -> FilePath -> BenchM ()

Clean any left-over build results.

setThreads :: Maybe (Int -> [ParamSetting])

Synthesize a list of compile/runtime settings that will control the number of threads.

Instances

Benchmark configuration spaces

data Benchmark a Source

Constructors

Benchmark 

Fields

target :: FilePath

The target file or direcotry.

cmdargs :: [String]

Command line argument to feed the benchmark executable.

configs :: BenchSpace a

The configration space to iterate over.

Instances

Eq a => Eq (Benchmark a) 
Ord a => Ord (Benchmark a) 
Show a => Show (Benchmark a) 
Generic (Benchmark a) 
Out a => Out (Benchmark a) 

data BenchSpace meaning Source

A datatype for describing (generating) benchmark configuration spaces. This is accomplished by nested conjunctions and disjunctions. For example, varying threads from 1-32 would be a 32-way Or. Combining that with profiling on/off (product) would create a 64-config space.

While the ParamSetting provides an *implementation* of the behavior, this datatype can also be decorated with a (more easily machine readable) meaning of the corresponding setting. For example, indicating that the setting controls the number of threads.

Constructors

And [BenchSpace meaning] 
Or [BenchSpace meaning] 
Set meaning ParamSetting 

Instances

Eq meaning => Eq (BenchSpace meaning) 
Ord meaning => Ord (BenchSpace meaning) 
Read meaning => Read (BenchSpace meaning) 
Show meaning => Show (BenchSpace meaning) 
Generic (BenchSpace meaning) 
Out a => Out (BenchSpace a) 

data ParamSetting Source

Different types of parameters that may be set or varied.

Constructors

RuntimeParam String

String contains runtime options, expanded and tokenized by the shell.

CompileParam String

String contains compile-time options, expanded and tokenized by the shell.

RuntimeEnv String String

The name of the env var and its value, respectively. For now Env Vars ONLY affect runtime.

CmdPath String String

Takes CMD PATH, and establishes a benchmark-private setting to use PATH for CMD. For example `CmdPath ghc ghc-7.6.3`. | Threads Int -- ^ Shorthand: builtin support for changing the number of threads across a number of separate build methods.

Instances

enumerateBenchSpace :: BenchSpace a -> [[(a, ParamSetting)]]Source

Exhaustively compute all configurations described by a benchmark configuration space.

compileOptsOnly :: BenchSpace a -> BenchSpace aSource

Strip all runtime options, leaving only compile-time options. This is useful for figuring out how many separate compiles need to happen.

isCompileTime :: ParamSetting -> BoolSource

Is it a setting that affects compile time?

toCompileFlags :: [(a, ParamSetting)] -> CompileFlagsSource

toRunFlags :: [(a, ParamSetting)] -> RunFlagsSource

toEnvVars :: [(a, ParamSetting)] -> [(String, String)]Source

toCmdPaths :: [(a, ParamSetting)] -> [(String, String)]Source

type BuildID = StringSource

A BuildID should uniquely identify a particular (compile-time) configuration, but consist only of characters that would be reasonable to put in a filename. This is used to keep build results from colliding.

makeBuildID :: CompileFlags -> BuildIDSource

Performs a simple reformatting (stripping disallowed characters) to create a build ID corresponding to a set of compile flags.

data DefaultParamMeaning Source

Constructors

Threads Int

Set the number of threads.

Variant String

Which schedulerimplementationetc.

NoMeaning 

Instances

HSBench Driver Configuration

data Config Source

The global configuration for benchmarking:

Constructors

Config 

Fields

benchlist :: [Benchmark DefaultParamMeaning]
 
benchsetName :: Maybe String

What identifies this set of benchmarks? Used to create fusion table.

benchversion :: (String, Double)

benchlist file name and version number (e.g. X.Y)

threadsettings :: [Int]

A list of #threads to test. 0 signifies non-threaded mode.

maxthreads :: Int
 
trials :: Int

number of runs of each configuration

shortrun :: Bool
 
doClean :: Bool
 
keepgoing :: Bool

keep going after error

pathRegistry :: PathRegistry

Paths to executables.

hostname :: String
 
startTime :: Integer

Seconds since Epoch.

resultsFile :: String

Where to put timing results.

logFile :: String

Where to put more verbose testing output.

gitInfo :: (String, String, Int)

Branch, revision hash, depth.

buildMethods :: [BuildMethod]

Starts with cabalmakeghc, can be extended by user.

logOut :: OutputStream ByteString
 
resultsOut :: OutputStream ByteString
 
stdOut :: OutputStream ByteString
 
envs :: [[(String, String)]]
 
doFusionUpload :: Bool
 

Instances

type BenchM a = ReaderT Config IO aSource

A monad for benchamrking. This provides access to configuration options, but really, its main purpose is enabling logging.

Subprocesses and system commands

data CommandDescr Source

A self-contained description of a runnable command. Similar to System.Process.CreateProcess but slightly simpler.

Constructors

CommandDescr 

Fields

command :: CmdSpec

Executable and arguments

envVars :: [(String, String)]

Environment variables to APPEND to current env.

timeout :: Maybe Double

Optional timeout in seconds.

workingDir :: Maybe FilePath

Optional working directory to switch to before running command.

Instances

data RunResult Source

Measured results from running a subprocess (benchmark).

Constructors

RunCompleted 

Fields

realtime :: Double

Benchmark time in seconds, may be different than total process time.

productivity :: Maybe Double

Seconds

TimeOut 
ExitError Int

Contains the returned error code.

Instances

data SubProcess Source

A running subprocess.

Constructors

SubProcess 

Fields

wait :: IO RunResult
 
process_out :: InputStream ByteString

A stream of lines.

process_err :: InputStream ByteString

A stream of lines.