I9      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~  Safe-Inferred  None*023468:=KMGKeep a full plugin configuration together with the plugin it goes with.9Keep a single flag together with the plugin it goes with.zAn interface for plugins provided in separate packages. These plugins provide new backends for uploading benchmark data.BA configuration flag for the plugin (parsed from the command line) -The full configuration record for the plugin. $Each plugin must have a unique name. Options for command line parsing. These should probably be disjoint from the options used by other plugins; so use very specific names.Finally, note that the String returned here is a header line that is printed before the usage documentation when the benchmark executable is invoked with `-h`. 5Process flags and update a configuration accordingly. Take any initialization actions, which may include reading or writing files and connecting to network services, as the main purpose of plugin is to provide backends for data upload.RNote that the initialization process can CHANGE the Config (it returns a new one).PThis is the raison d'etre for the class. Upload a single row of benchmark data.This contains all the contextual information for a single benchmark run, which makes up a "row" in a table of benchmark results. Note that multiple "trials" (actual executions) go into a single BenchmarkResultWhich benchmark are we runningRIf there are multiple ways to run the benchmark, this shoud record which was used.Command line arguments.Which machine did we run on?HA unique identifier for the full hsbencher that included this benchmark.OWhen launched from Jenkins or Travis, it can help to record where we came from.ZIf multithreaded, how many CPU threads did this benchmark run with, zero otherwise. Time of the fastest runTime of the median runTime of the slowest run2GC productivity (if recorded) for the mintime run.5GC productivity (if recorded) for the mediantime run.2GC productivity (if recorded) for the maxtime run.DSpace separated list of numbers, should be one number for each TRIAL )How many times to [re]run each benchmark."Flags used during compilation#5Flags passed at runtime, possibly in addition to ARGS$0Environment variables set for this benchmark run%@If the benchmark *suite* tracks its version number, put it here.':Information about the host machine that ran the benchmark.*'Which branch was the benchmark run from+)Which exact revision of the code was run.,<How many git commits deep was that rev (rough proxy for age)-(Was anyone else logged into the machine?.(Information about the host machine from etcissue/9Information about the host machine from the lspci command0@Optionally record the full stdout from the benchmarking process.13If recorded, the allocation rate of the median run.2EIf recorded, the memory footprint (high water mark) of the median run3Space separated list of numbers, JIT compile times (if applicable), with a 1-1 correspondence to the exec times in ALLTIMES. Time should not be double counted as JIT and exec time; these should be disjoint.9wA line harvester takes a single line of input and possible extracts data from it which it can then add to a RunResult.>The boolean result indicates whether the line was used or not.;A running subprocess.>A stream of lines.?A stream of lines.@7Measured results from running a subprocess (benchmark).A!Contains the returned error code.DDBenchmark time in seconds, may be different than total process time.ESecondsF"Bytes allocated per mutator-secondG.High water mark of allocated memory, in bytes.HDTime to JIT compile the benchmark, counted separately from realtime.JsA self-contained description of a runnable command. Similar to System.Process.CreateProcess but slightly simpler.LExecutable and argumentsM/Environment variables to APPEND to current env.NOptional timeout in seconds.OBOptional working directory to switch to before running command.PhDoes a crash of the process mean we throw away any data the program already printed? Usually False.Q8Different types of parameters that may be set or varied.RATakes CMD PATH, and establishes a benchmark-private setting to use PATH for CMD. For example `CmdPath "ghc" "ghc-7.6.3"`. | Threads Int -- ^ Shorthand: builtin support for changing the number of threads across a number of separate build methods. | TimeOut Double -- ^ Set the timeout for this benchmark.S]The name of the env var and its value, respectively. For now Env Vars ONLY affect runtime.TString contains compile-time options, expanded and tokenized by the shell. CompileEnv String String -- ^ Establish an environment variable binding during compile time.UzRuntime "args" are like runtime params but are more prominent. They typically are part of the "key" of the benchmark.VEString contains runtime options, expanded and tokenized by the shell.WA BuildID should uniquely identify a particular (compile-time) configuration, but consist only of characters that would be reasonable to put in a filename. This is used to keep build results from colliding.ZWhich schedulerimplementationetc.[Set the number of threads.\A datatype for describing (generating) benchmark configuration spaces. This is accomplished by nested conjunctions and disjunctions. For example, varying threads from 1-32 would be a 32-way Or. Combining that with profiling on/off (product) would create a 64-config space.While the ParamSetting provides an *implementation* of the behavior, this datatype can also be decorated with a (more easily machine readable) meaning of the corresponding setting. For example, indicating that the setting controls the number of threads.`The all-inclusive datatype for a single Benchmark. Do NOT construct values of this type directly. Rather, you should make your code robust against future addition of fields to this datatype. Use 3 followed by customizing only the fields you need.bWhere is the benchmark to run? This must be a single target file or directory. The convention is that this file or directory, when combined with a BuildMethod, provides a self-contained way to build one benchmark. The } field of the gX had better contain some build method that knows how to handle this file or directory.c7Command line argument to feed the benchmark executable.d'The configration space to iterate over.eNOptional name to use to identify this benchmark, INSTEAD of the basename from b.fJSpecific timeout for this benchmark in seconds. Overrides global setting.gwThe global configuration for benchmarking. WARNING! This is an internal data structure. You shouldn't really use it.jjWhat identifies this set of benchmarks? In some upload backends this is the name of the dataset or table.kbenchlist file name and version number (e.g. X.Y) , threadsettings :: [Int] -- ^ A list of #threads to test. 0 signifies non-threaded mode.l\Timeout in seconds for running benchmarks (if not specified by the benchmark specifically)mMIn parallel compile/run phases use at most this many threads. Defaults to getNumProcessors.n$number of runs of each configurationo#Where to start in the config space.p/How many configurations to run before stopping.q1How many times to retry failed benchmark configs.rAn over-ride for the run ID.s4The build ID from the continuous integration system.tAn alternate mode to run very small sizes of benchmarks for testing. HSBencher relies on a convention where benchmarks WITHOUT command-line arguments must do a short run.u=Invoke the build method's clean operation before compilation.vKeep going after error.wPaths to executablesxKManually override the machine hostname. Defaults to the output of the x command.ySeconds since Epoch. zWhere to put timing results.{*Where to put full, verbose testing output.|Branch, revision hash, depth.}@Known methods for building benchmark targets. Starts with cabalmakeghc, can be extended by user.~Internal use onlyInternal use onlyGInternal use only A set of environment variable configurations to testAA global setting to control whether executables are given their 'flags/params'm after their regular arguments. This is here because some executables don't use proper command line parsing.9A stack of line harvesters that gather RunResult details.4Each plugin, and, if configured, its configuration.  Maps the   to its config.{A monad for benchmarking This provides access to configuration options, but really, its main purpose is enabling logging.A completely encapsulated method of building benchmarks. Cabal and Makefiles are two examples of this. The user may extend it with their own methods.pIdentifies this build method for humans. , buildsFiles :: FilePredicate , canBuild :: FilePath -> IO Bool-Can this method build a given file/directory?dMore than one build can happen at once. This implies that compile always returns StandAloneBinary.DIdentify the benchmark to build by its target FilePath. Compile it."Clean any left-over build results.WSynthesize a list of compile/runtime settings that will control the number of threads.(The result of doing a build. Note that 2 can will throw an exception if compilation fails.In this case the build return what you need to do the benchmark run, but the directory contents cannot be touched until after than run is finished.0This binary can be copied and executed whenever.RA description of a set of files. The description may take one of multiple forms. Logical or.hA common pattern. For example, we can build a file foo.c, if it lives in a directory with exactly one Makefile.E.g. Makefile) | SatisfiesPredicate (String -> Bool)E.g. ".hs", WITH the dot.#Maps canonical command names, e.g. ghcn, to absolute system paths. This is a global configuration mechanism that all BuildMethods have access to.SThe arguments passed (in a build-method specific way) into the compilation process.OThe arguments passed (in a build-method specific way) to the running benchmark.#This function gives meaning to the FilePred* type. It returns a filepath to signal True and Nothing otherwise.Make a Benchmark data structure given the core, required set of fields, and uses defaults to fill in the rest. Takes target, cmdargs, configs.UExhaustively compute all configurations described by a benchmark configuration space.*Is it a setting that affects compile time?>Extract the parameters that affect the compile-time arguments.Performs a simple reformatting (stripping disallowed characters) to create a build ID corresponding to a set of compile flags. To make it unique we also append the target path.Strip all runtime options, leaving only compile-time options. This is useful for figuring out how many separate compiles need to happen. A default @ that is a good starting point for filling in desired fields. (This way, one remains robust to additional fields that are added in the future.)1Run the second harvester only if the first fails.cA default value, useful for filling in only the fields that are relevant to a particular benchmark.pConvert the Haskell representation of a benchmark result into a tuple for upload to a typical database backend.{Make the command line flags for a particular plugin generic so that they can be mixed together with other plugins options.eRetrieve our own Plugin's configuration from the global config. This involves a dynamic type cast.If there is no configuration for this plugin currently registered, the default configuration for that plugin is returned.YEncapsulate the policy for where/how to inject the Plugin's conf into the global Config.FWe can stack up line harvesters. ALL of them get to run on each line.  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~       !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~`abcdef\_^]QVUTSRWX[ZYghijklmnopqrstuvwxyz{|}~JKLMNOP@CBADEFGHI;<=>?9: !"#$%&'()*+,-./01234 5876= % !"#$%&'()*+,-./0123458769:;<=>?@ CBADEFGHIJKLMNOPQVUTSRWX[ZY\_^]`abcdefghijklmnopqrstuvwxyz{|}~     NoneThere are three logging destinations we care about. The .dat file, the .log file, and the user's screen (i.e. the user who launched the benchmarks).2Print a message (line) both to stdout and logFile:8Log a line to a particular file and also echo to stdout.(Shorthand for tagged version of logging.8Logging straight to stdout (but with the hsbencher tag).'The tag for printing hsbencher messagse None+=KThe configuration consists only of the location of a single file, which is where the results will be fed. If no file is provided, the default location is selected during plugin initialization..A simple singleton type -- a unique signifier.2A plugin with the basic options (if any) included. NoneM Internal data type.This runs a sub-process and tries to determine how long it took (real time) and how much of that time was spent in the mutator vs. the garbage collector.It is complicated by: aAn additional protocol for the process to report self-measured realtime (a line starting in  SELFTIMED , ditto for JITTIME:)uParsing the output of GHC's "+RTS -s" to retrieve productivity OR using lines of the form "PRODUCTIVITY: XYZ"Note that "+RTS -s" is specific to Haskell/GHC, but the PRODUCTIVITY tag allows non-haskell processes to report garbage collector overhead.^This procedure is currently not threadsafe, because it changes the current working directory.-A simpler and SINGLE-THREADED alternative to [. This is part of the process of trying to debug the HSBencher zombie state (Issue #32).%Check for a SELFTIMED line of output.Check for a line of output of the form "TAG NUM" or "TAG: NUM". Take a function that puts the result into place (the write half of a lens).Retrieve productivity (i.e. percent time NOT garbage collecting) as output from a Haskell program with "+RTS -s". Productivity is a percentage (double between 0.0 and 100.0, inclusive).?Fire a single event after a time interval, then end the stream.This makes the EOS into an explicit-, penultimate message. This way it survives concurrentMergeN. It represents this end of stream by Nothing, but beware the doubly-nested  type.TAlternatioe to the io-streams version which does not allow setting the environment.Stack of harvestersStack of harvesters ! !None""NoneM In seconds.+Global variable holding the main thread id.Create a thread that echos the contents of stdout/stderr InputStreams (lines) to the appropriate places (as designated by the logging facility). Returns an MVar used to synchronize on the completion of the echo thread.Run a command and wait for all output. Log output to the appropriate places. The first argument is a "tag" to append to each output line to make things clearer.iRuns a command through the OS shell and returns stdout split into lines. (Ignore exit code and stderr.)JRuns a command through the OS shell and returns the first line of output.A more persistent version of #.8Create a backup copy of existing results_HOST.dat files. $ $NoneBuild with GNU Make. This is a basic Make protocol; your application may need something more complicated. This assumes targets clean, run, and the default target for building.The variables RUN_ARGS and COMPILE_ARGS are used to pass in the per-benchmark run and compile options, so the Makefile must be written with these conventions in mind. Note that this build method never knows where or if any resulting binaries reside. One effect of that is that this simple build method can never be used for PARALLEL compiles, because it cannot manage where the build-intermediates are stored.aBuild with GHC directly. This assumes that all dependencies are installed and a single call to ghc can build the file.cCompile-time arguments go directly to GHC, and runtime arguments directly to the resulting binary.=Build with cabal. Specifically, this uses "cabal install".'This build method attempts to choose reasonable defaults for benchmarking. It takes control of the output program suffix and directory (setting it to BENCHROOT/bin). It passes compile-time arguments directly to cabal. Likewise, runtime arguments get passed directly to the resulting binary.%Checks whether a  works for a given file matchesMethod :: BuildMethod -> FilePath -> IO Bool matchesMethod BuildMethod{canBuild} path = return $ filePredCheck canBuild pathBOur compilation targets might be either directories or file names.&A simple wrapper for a command that is expected to succeed (and whose output we don't care about). Throws an exception if the command fails. Returns lines of output if successful.%'(&%'(&None2Command line flags to the benchmarking executable.)Command line options.*%Check that the flag setting is valid.>Fill in "static" fields of a BenchmarkResult row based on the g data.]This abstracts over the actions we need to take to properly add an additional plugin to the g.Retrieve the (default) configuration from the environment, it may subsequently be tinkered with. This procedure should be idempotent.)*)* None 3EM+General usage information.,3Build a single benchmark in a single configuration.-0Write the results header out stdout and to disk..dTODO: Eventually this will make sense when all config can be read from the environment, args, files.GIn this version, user provides a list of benchmarks to run, explicitly.'Multiple lines of usage info help docs./<Remove a plugin from the configuration based on its plugName An even more flexible version allows the user to install a hook which modifies the configuration just before bencharking begins. All trawling of the execution environment (command line args, environment variables) happens BEFORE the user sees the configuration.`This function doesn't take a benchmark list separately, because that simply corresponds to the i field of the output g.0+,1234-./56789:;<=>?@0+,1234-./56789:;<=>?@ None  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~A  !"#$%&'()*+,-./0123456789:;<=>?@ABCCDDEFGHIJKLMNOPQRRSTUVWXYZ[\]^_`abcdefgghijklmmnopqrstuvwxyz{|}~       !"#$%&'()*+,-./0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E FGhsbencher-1.12HSBencher.TypesHSBencher.Internal.LoggingHSBencher.Backend.Dribble!HSBencher.Internal.MeasureProcessHSBencher.HarvestersHSBencher.Internal.UtilsHSBencher.Methods.BuiltinHSBencher.Internal.ConfigHSBencher.Internal.AppPaths_hsbencher HSBencherGenericPretty-1.2.1Text.PrettyPrint.GenericPrettydocSomePluginConfSomePluginFlag SomePluginPluginPlugFlagPlugConfplugName plugCmdOpts foldFlagsplugInitialize plugUploadRowBenchmarkResult _PROGNAME_VARIANT_ARGS _HOSTNAME_RUNID _CI_BUILD_ID_THREADS _DATETIME_MINTIME _MEDIANTIME_MAXTIME_MINTIME_PRODUCTIVITY_MEDIANTIME_PRODUCTIVITY_MAXTIME_PRODUCTIVITY _ALLTIMES_TRIALS _COMPILER_COMPILE_FLAGS_RUNTIME_FLAGS _ENV_VARS_BENCH_VERSION _BENCH_FILE_UNAME _PROCESSOR _TOPOLOGY _GIT_BRANCH _GIT_HASH _GIT_DEPTH_WHO _ETC_ISSUE_LSPCI _FULL_LOG_MEDIANTIME_ALLOCRATE_MEDIANTIME_MEMFOOTPRINT _ALLJITTIMES_CUSTOM SomeResult StringResult DoubleResult IntResult LineHarvester SubProcesswait process_out process_err RunResult ExitError RunTimeOut RunCompletedrealtime productivity allocRate memFootprintjittimecustom CommandDescrcommandenvVarstimeout workingDir tolerateError ParamSettingCmdPath RuntimeEnv CompileParam RuntimeArg RuntimeParamBuildIDDefaultParamMeaning NoMeaningVariantThreads BenchSpaceSetOrAnd Benchmarktargetcmdargsconfigsprogname benchTimeOutConfig benchlist benchsetName benchversion runTimeOut maxthreadstrialsskipTorunOnly retryFailedrunID ciBuildIDshortrundoClean keepgoing pathRegistryhostname startTime resultsFilelogFilegitInfo buildMethodslogOut resultsOutstdOutenvsargsBeforeFlags harvestersplugIns plugInConfsBenchM BuildMethod methodNamecanBuildconcurrentBuildcompileclean setThreads BuildResult RunInPlaceStandAloneBinary FilePredicateAnyFilePredOrInDirectoryWithExactlyOne IsExactly WithExtension CompileFlagsRunFlags filePredCheck mkBenchmarkenumerateBenchSpace isCompileTimetoCompileFlags toCmdPaths toEnvVars makeBuildIDcompileOptsOnlyemptyRunResult orHarvestemptyBenchmarkResult resultToTuplegenericCmdOpts getMyConf setMyConfLogDestStdOutLogFile ResultsFileloglogOnlogTchatter hsbencher_tag DribbleConfcsvfile DribblePlugindefaultDribblePluginmeasureProcessmeasureProcessDBGselftimedHarvesterjittimeHarvestertaggedLineHarvesterghcProductivityHarvesterghcAllocRateHarvesterghcMemFootprintHarvestercustomTagHarvesterIntcustomTagHarvesterDoublecustomTagHarvesterStringmy_namedefaultTimeout main_threadidtrim echoStream runLoggedrunLinesrunSL fetchBaseName backupResults makeMethod ghcMethod cabalMethodFlag DisablePlugShowBenchmarks ShowVersionShowHelpGHCPath CabalPath ForceHostName CIBuildIDRunID RetryFailedRunOnlySkipTo NumTrials KeepGoingShortRunNoCleanNoCabalNoRecompBinDir BenchsetNameParBenchall_cli_optionsaugmentResultWithConfig addPlugin getConfigdefaultMainWithBechmarks fullUsageInfodefaultMainModifyConfigcatchIOversionbindirlibdirdatadir libexecdir sysconfdir getBinDir getLibDir getDataDir getLibexecDir getSysconfDirgetDataFileName PathRegistry$fMonoidLineHarvesterTagEnvVarstest1test2test3$fShowSomePluginFlag$fShowSomePluginConf$fOrdSomePlugin$fEqSomePlugin$fShowSomePlugin$fShowSomeResult$fShowLineHarvester$fOutMap$fOutBenchmark$fOutBenchSpace$fOutDefaultParamMeaning$fOutFilePredicate$fOutParamSetting$fShowOutputStream$fShowBuildMethod$fShowBuildResultfileLockuploadBenchResult$fPluginDribblePlugin$fDefaultDribbleConf$fDefaultDribblePlugin ProcessEvt timeOutStreamreifyEOSbase Data.MaybeMayberunInteractiveCommandWithEnv TimerFire ProcessClosedOutLineErrLinedumpRestorMaybetaggedLineHarvesterStrfilepath-1.3.0.2System.FilePath.Posix takeBaseName getCPULoadgetDir runSuccessful inDirectory filesInDircore_cli_options mkPosIntFlaggeneralUsageStr compileOneprintBenchrunHeader defaultMain removePluginhsbencherVersionrunOnerunA_gatherContextrunB_runTrialsrunC_produceOutput doShowHelpcatParallelOutput didCompleteisErrorgetprod getallocrategetmemfootprintgettime getjittimeposInf shortArgsnest