HSBencher.Types

All the core types used by the rest of the HSBencher codebase.

Benchmark building

The basic types for describing a single benchmark.

mkBenchmark :: FilePath -> [String] -> BenchSpace a -> Benchmark a

Make a Benchmark data structure given the core, required set of fields, and uses defaults to fill in the rest. Takes target, cmdargs, configs.

data Benchmark a

The all-inclusive datatype for a single Benchmark. Do NOT construct values of this type directly. Rather, you should make your code robust against future addition of fields to this datatype. Use mkBenchmark followed by customizing only the fields you need.

Constructors

Benchmark

Fields

target :: FilePath

Where is the benchmark to run? This must be a single target file or directory. The convention is that this file or directory, when combined with a BuildMethod, provides a self-contained way to build one benchmark. The buildMethods field of the Config had better contain some build method that knows how to handle this file or directory.

cmdargs :: [String]

Command line argument to feed the benchmark executable.

configs :: BenchSpace a

The configration space to iterate over.

progname :: Maybe String

Optional name to use to identify this benchmark, INSTEAD of the basename from target.

benchTimeOut :: Maybe Double

Specific timeout for this benchmark in seconds. Overrides global setting.

Instances

Eq a => Eq (Benchmark a)

Ord a => Ord (Benchmark a)

Show a => Show (Benchmark a)

Generic (Benchmark a)

Out a => Out (Benchmark a)

type RunFlags = [String]

The arguments passed (in a build-method specific way) to the running benchmark.

type CompileFlags = [String]

The arguments passed (in a build-method specific way) into the compilation process.

Build method interface and applicability

A build method is applicable to a subset of target files (FilePredicate) and has a particular interface that HSbencher relies upon.

data BuildMethod

A completely encapsulated method of building benchmarks. Cabal and Makefiles are two examples of this. The user may extend it with their own methods.

Constructors

BuildMethod

Fields

methodName :: String

Identifies this build method for humans. , buildsFiles :: FilePredicate , canBuild :: FilePath -> IO Bool

canBuild :: FilePredicate

Can this method build a given file/directory?

concurrentBuild :: Bool

More than one build can happen at once. This implies that compile always returns StandAloneBinary.

compile :: PathRegistry -> BuildID -> CompileFlags -> FilePath -> BenchM BuildResult

Identify the benchmark to build by its target FilePath. Compile it.

clean :: PathRegistry -> BuildID -> FilePath -> BenchM ()

Clean any left-over build results.

setThreads :: Maybe (Int -> [ParamSetting])

Synthesize a list of compile/runtime settings that will control the number of threads.

Instances

Show BuildMethod

data BuildResult

The result of doing a build. Note that compile can will throw an exception if compilation fails.

Constructors

StandAloneBinary FilePath

This binary can be copied and executed whenever.

RunInPlace (RunFlags -> EnvVars -> CommandDescr)

In this case the build return what you need to do the benchmark run, but the directory contents cannot be touched until after than run is finished.

Instances

Show BuildResult

data FilePredicate

A description of a set of files. The description may take one of multiple forms.

Constructors

WithExtension String

E.g. .hs, WITH the dot.

IsExactly String

E.g. Makefile | SatisfiesPredicate (String -> Bool)

InDirectoryWithExactlyOne FilePredicate

A common pattern. For example, we can build a file foo.c, if it lives in a directory with exactly one Makefile.

PredOr FilePredicate FilePredicate

Logical or.

AnyFile

Instances

Eq FilePredicate

Ord FilePredicate

Show FilePredicate

Generic FilePredicate

Out FilePredicate

filePredCheck :: FilePredicate -> FilePath -> IO (Maybe FilePath)

This function gives meaning to the FilePred type. It returns a filepath to signal True and Nothing otherwise.

Benchmark configuration spaces

Describe how many different ways you want to run your benchmarks.

data BenchSpace meaning

A datatype for describing (generating) benchmark configuration spaces. This is accomplished by nested conjunctions and disjunctions. For example, varying threads from 1-32 would be a 32-way Or. Combining that with profiling on/off (product) would create a 64-config space.

While the ParamSetting provides an *implementation* of the behavior, this datatype can also be decorated with a (more easily machine readable) meaning of the corresponding setting. For example, indicating that the setting controls the number of threads.

Constructors

And [BenchSpace meaning]

Or [BenchSpace meaning]

Set meaning ParamSetting

Instances

Eq meaning => Eq (BenchSpace meaning)

Ord meaning => Ord (BenchSpace meaning)

Read meaning => Read (BenchSpace meaning)

Show meaning => Show (BenchSpace meaning)

Generic (BenchSpace meaning)

Out a => Out (BenchSpace a)

data ParamSetting

Different types of parameters that may be set or varied.

Constructors

RuntimeParam String

String contains runtime options, expanded and tokenized by the shell.

CompileParam String

String contains compile-time options, expanded and tokenized by the shell.

RuntimeEnv String String

The name of the env var and its value, respectively. For now Env Vars ONLY affect runtime.

CmdPath String String

Takes CMD PATH, and establishes a benchmark-private setting to use PATH for CMD. For example `CmdPath ghc ghc-7.6.3`. | Threads Int -- ^ Shorthand: builtin support for changing the number of threads across a number of separate build methods. | TimeOut Double -- ^ Set the timeout for this benchmark.

Instances

Eq ParamSetting

Ord ParamSetting

Read ParamSetting

Show ParamSetting

Generic ParamSetting

Out ParamSetting

enumerateBenchSpace :: BenchSpace a -> [[(a, ParamSetting)]]

Exhaustively compute all configurations described by a benchmark configuration space.

compileOptsOnly :: BenchSpace a -> BenchSpace a

Strip all runtime options, leaving only compile-time options. This is useful for figuring out how many separate compiles need to happen.

isCompileTime :: ParamSetting -> Bool

Is it a setting that affects compile time?

toCompileFlags :: [(a, ParamSetting)] -> CompileFlags

Extract the parameters that affect the compile-time arguments.

toRunFlags :: [(a, ParamSetting)] -> RunFlags

Extract the parameters that affect the runtime arguments.

toEnvVars :: [(a, ParamSetting)] -> [(String, String)]

toCmdPaths :: [(a, ParamSetting)] -> [(String, String)]

type BuildID = String

A BuildID should uniquely identify a particular (compile-time) configuration, but consist only of characters that would be reasonable to put in a filename. This is used to keep build results from colliding.

makeBuildID :: FilePath -> CompileFlags -> BuildID

Performs a simple reformatting (stripping disallowed characters) to create a build ID corresponding to a set of compile flags. To make it unique we also append the target path.

data DefaultParamMeaning

Constructors

Threads Int

Set the number of threads.

Variant String

Which schedulerimplementationetc.

NoMeaning

Instances

Eq DefaultParamMeaning

Ord DefaultParamMeaning

Read DefaultParamMeaning

Show DefaultParamMeaning

Generic DefaultParamMeaning

Out DefaultParamMeaning

HSBencher Driver Configuration

data Config

The global configuration for benchmarking. WARNING! This is an internal data structure. You shouldn't really use it.

Constructors

Config

Fields

benchlist :: [Benchmark DefaultParamMeaning]

benchsetName :: Maybe String

What identifies this set of benchmarks? In some upload backends this is the name of the dataset or table.

benchversion :: (String, Double)

benchlist file name and version number (e.g. X.Y) , threadsettings :: [Int] -- ^ A list of #threads to test. 0 signifies non-threaded mode.

runTimeOut :: Maybe Double

Timeout in seconds for running benchmarks (if not specified by the benchmark specifically)

maxthreads :: Int

In parallel compile/run phases use at most this many threads. Defaults to getNumProcessors.

trials :: Int

number of runs of each configuration

skipTo :: Maybe Int

Where to start in the config space.

runID :: Maybe String

An over-ride for the run ID.

ciBuildID :: Maybe String

The build ID from the continuous integration system.

shortrun :: Bool

An alternate mode to run very small sizes of benchmarks for testing. HSBencher relies on a convention where benchmarks WITHOUT command-line arguments must do a short run.

doClean :: Bool

Invoke the build methods clean operation before compilation.

keepgoing :: Bool

Keep going after error.

pathRegistry :: PathRegistry

Paths to executables

hostname :: String

Manually override the machine hostname. Defaults to the output of the hostname command.

startTime :: Integer

Seconds since Epoch.

resultsFile :: String

Where to put timing results.

logFile :: String

Where to put full, verbose testing output.

gitInfo :: (String, String, Int)

Branch, revision hash, depth.

buildMethods :: [BuildMethod]

Known methods for building benchmark targets. Starts with cabalmakeghc, can be extended by user.

logOut :: OutputStream ByteString

Internal use only

resultsOut :: OutputStream ByteString

Internal use only

stdOut :: OutputStream ByteString

Internal use only A set of environment variable configurations to test

envs :: [[(String, String)]]

argsBeforeFlags :: Bool

A global setting to control whether executables are given their 'flags/params' after their regular arguments. This is here because some executables don't use proper command line parsing.

harvesters :: LineHarvester

A stack of line harvesters that gather RunResult details.

plugIns :: [SomePlugin]

Each plugin, and, if configured, its configuration.

plugInConfs :: Map String SomePluginConf

Maps the plugName to its config.

Instances

Show Config

type BenchM a = ReaderT Config IO a

A monad for benchmarking This provides access to configuration options, but really, its main purpose is enabling logging.

Subprocesses and system commands

data CommandDescr

A self-contained description of a runnable command. Similar to System.Process.CreateProcess but slightly simpler.

Constructors

CommandDescr

Fields

command :: CmdSpec

Executable and arguments

envVars :: [(String, String)]

Environment variables to APPEND to current env.

timeout :: Maybe Double

Optional timeout in seconds.

workingDir :: Maybe FilePath

Optional working directory to switch to before running command.

Instances

Eq CommandDescr

Ord CommandDescr

Read CommandDescr

Show CommandDescr

Generic CommandDescr

data RunResult

Measured results from running a subprocess (benchmark).

Constructors

RunCompleted

Fields

realtime :: Double

Benchmark time in seconds, may be different than total process time.

productivity :: Maybe Double

Seconds

allocRate :: Maybe Word64

Bytes allocated per mutator-second

memFootprint :: Maybe Word64

High water mark of allocated memory, in bytes.

jittime :: Maybe Double

Time to JIT compile the benchmark, counted separately from realtime.

RunTimeOut

ExitError Int

Contains the returned error code.

Instances

Eq RunResult

Show RunResult

emptyRunResult :: RunResult

A default RunResult that is a good starting point for filling in desired fields. (This way, one remains robust to additional fields that are added in the future.)

data SubProcess

A running subprocess.

Constructors

SubProcess

Fields

wait :: IO RunResult

process_out :: InputStream ByteString

A stream of lines.

process_err :: InputStream ByteString

A stream of lines.

newtype LineHarvester

A line harvester takes a single line of input and possible extracts data from it which it can then add to a RunResult.

The boolean result indicates whether the line was used or not.

Constructors

LineHarvester (ByteString -> (RunResult -> RunResult, Bool))

Instances

Show LineHarvester

Monoid LineHarvester

We can stack up line harvesters. ALL of them get to run on each line.

orHarvest :: LineHarvester -> LineHarvester -> LineHarvester

Run the second harvester only if the first fails.

Benchmark outputs for upload

data BenchmarkResult

This contains all the contextual information for a single benchmark run, which makes up a row in a table of benchmark results. Note that multiple trials (actual executions) go into a single BenchmarkResult

Constructors

BenchmarkResult

Fields

_PROGNAME :: String

Which benchmark are we running

_VARIANT :: String

If there are multiple ways to run the benchmark, this shoud record which was used.

_ARGS :: [String]

Command line arguments.

_HOSTNAME :: String

Which machine did we run on?

_RUNID :: String

A unique identifier for the full hsbencher that included this benchmark.

_CI_BUILD_ID :: String

When launched from Jenkins or Travis, it can help to record where we came from.

_THREADS :: Int

If multithreaded, how many CPU threads did this benchmark run with.

_DATETIME :: String

_MINTIME :: Double

Time of the fastest run

_MEDIANTIME :: Double

Time of the median run

_MAXTIME :: Double

Time of the slowest run

_MINTIME_PRODUCTIVITY :: Maybe Double

GC productivity (if recorded) for the mintime run.

_MEDIANTIME_PRODUCTIVITY :: Maybe Double

GC productivity (if recorded) for the mediantime run.

_MAXTIME_PRODUCTIVITY :: Maybe Double

GC productivity (if recorded) for the maxtime run.

_ALLTIMES :: String

Space separated list of numbers, should be one number for each TRIAL

_TRIALS :: Int

How many times to [re]run each benchmark.

_COMPILER :: String

_COMPILE_FLAGS :: String

Flags used during compilation

_RUNTIME_FLAGS :: String

Flags passed at runtime, possibly in addition to ARGS

_ENV_VARS :: String

Environment variables set for this benchmark run

_BENCH_VERSION :: String

If the benchmark *suite* tracks its version number, put it here.

_BENCH_FILE :: String

_UNAME :: String

Information about the host machine that ran the benchmark.

_PROCESSOR :: String

_TOPOLOGY :: String

_GIT_BRANCH :: String

Which branch was the benchmark run from

_GIT_HASH :: String

Which exact revision of the code was run.

_GIT_DEPTH :: Int

How many git commits deep was that rev (rough proxy for age)

_WHO :: String

Was anyone else logged into the machine?

_ETC_ISSUE :: String

Information about the host machine from etcissue

_LSPCI :: String

Information about the host machine from the lspci command

_FULL_LOG :: String

Optionally record the full stdout from the benchmarking process.

_MEDIANTIME_ALLOCRATE :: Maybe Word64

If recorded, the allocation rate of the median run.

_MEDIANTIME_MEMFOOTPRINT :: Maybe Word64

If recorded, the memory footprint (high water mark) of the median run

_ALLJITTIMES :: String

Space separated list of numbers, JIT compile times (if applicable), with a 1-1 correspondence to the exec times in ALLTIMES. Time should not be double counted as JIT and exec time; these should be disjoint.

Instances

Eq BenchmarkResult

Ord BenchmarkResult

Read BenchmarkResult

Show BenchmarkResult

emptyBenchmarkResult :: BenchmarkResult

A default value, useful for filling in only the fields that are relevant to a particular benchmark.

resultToTuple :: BenchmarkResult -> [(String, String)]

Convert the Haskell representation of a benchmark result into a tuple for upload to a typical database backend.

data SomePlugin

Constructors

forall p . Plugin p => SomePlugin p

Instances

Eq SomePlugin

Ord SomePlugin

Show SomePlugin

data SomePluginConf

Keep a full plugin configuration together with the plugin it goes with.

Constructors

forall p . Plugin p => SomePluginConf p (PlugConf p)

Instances

Show SomePluginConf

data SomePluginFlag

Keep a single flag together with the plugin it goes with.

Constructors

forall p . Plugin p => SomePluginFlag p (PlugFlag p)

Instances

Show SomePluginFlag

class (Show p, Eq p, Ord p, Show (PlugFlag p), Ord (PlugFlag p), Typeable (PlugFlag p), Show (PlugConf p), Ord (PlugConf p), Typeable (PlugConf p)) => Plugin p where

An interface for plugins provided in separate packages. These plugins provide new backends for uploading benchmark data.

Associated Types

type PlugFlag p

A configuration flag for the plugin (parsed from the command line)

type PlugConf p

The full configuration record for the plugin.

Methods

plugName :: p -> String

Each plugin must have a unique name.

plugCmdOpts :: p -> (String, [OptDescr (PlugFlag p)])

Options for command line parsing. These should probably be disjoint from the options used by other plugins; so use very specific names.

Finally, note that the String returned here is a header line that is printed before the usage documentation when the benchmark executable is invoked with `-h`.

foldFlags :: p -> [PlugFlag p] -> PlugConf p -> PlugConf p

Process flags and update a configuration accordingly.

defaultPlugConf :: p -> PlugConf p

The default configuration for this plugin.

plugInitialize :: p -> Config -> IO Config

Take any initialization actions, which may include reading or writing files and connecting to network services, as the main purpose of plugin is to provide backends for data upload.

Note that the initialization process can CHANGE the Config (it returns a new one).

plugUploadRow :: p -> Config -> BenchmarkResult -> IO ()

This is the raison d'etre for the class. Upload a single row of benchmark data.

Instances

Plugin DribblePlugin

genericCmdOpts :: Plugin p => p -> [OptDescr SomePluginFlag]

Make the command line flags for a particular plugin generic so that they can be mixed together with other plugins options.

getMyConf :: forall p. Plugin p => p -> Config -> PlugConf p

Retrieve our own Plugin's configuration from the global config. This involves a dynamic type cast.

setMyConf :: forall p. Plugin p => p -> PlugConf p -> Config -> Config

Encapsulate the policy for where/how to inject the Plugin's conf into the global Config.

For convenience -- large records demand pretty-printing

doc :: Out a => a -> Doc

doc is the equivalent of show

This is a specialised variant of docPrec, using precedence context zero.