!y-      !"#$%&'()*+,O(c) 2009-2014 Bryan O'Sullivan (c) 2018 Composewell Technologies BSD-styleharendra.kumar@gmail.com experimentalGHC Trustworthy "#16FNT' bench-showThe statistical estimator used to arrive at a single value for a benchmark when samples from multiple experiments are available. bench-showReport the median, outliers and outlier variance using box-plot method. This is the most robust indicator with respect to outliers when successive runs of benchmarks are compared. bench-showlReport the mean and the standard deviation from the mean. This is less robust than median but more precise. bench-showqReport the coefficient of regression, discarding the constant factor, arrived at by linear regression using ordinary least square method. The R-square goodness-of-fit estimate is also reported. It works better when larger number of samples are taken. This cannot be used when the number of samples is less than 2, in that case a mean value is reported instead.- bench-showsAnalysis of the extent to which outliers in a sample affect its standard deviation (and to some extent, its mean).. bench-show"Qualitative description of effect./ bench-show$Brief textual description of effect.0 bench-show@Quantitative description of effect (a fraction between 0 and 1).1 bench-showpA description of the extent to which outliers in the sample data affect the sample mean and standard deviation.2 bench-showLess than 1% effect.3 bench-showBetween 1% and 10%.4 bench-showBetween 10% and 50%.5 bench-show+Above 50% (i.e. measurements are useless).6 bench-showCOutliers from sample data, calculated using the boxplot technique.7 bench-showJMore than 3 times the interquartile range (IQR) below the first quartile.8 bench-show9Between 1.5 and 3 times the IQR below the first quartile.9 bench-show9Between 1.5 and 3 times the IQR above the third quartile.: bench-show3More than 3 times the IQR above the third quartile.; bench-show=Classify outliers in a data set, using the boxplot technique.< bench-showgCompute the extent to which outliers in the sample data affect the sample mean and standard deviation.= bench-show/Count the total number of outliers in a sample.> bench-show%Perform an analysis of a measurement.< bench-showmean bench-showstandard deviation. bench-showNumber of original iterations.&?@ABCDEFGHIJKLMNOPQ-R./012345=STUV!(c) 2018 Composewell TechnologiesBSD3harendra.kumar@gmail.com experimentalGHCNone"#<NH$W bench-showDescribe a relative unit i.e. a unit in terms of another unit. A relative unit has a label and a ratio which when multiplied with the unit gives us the other unit. For example, if the known time unit is seconds, we can describe a millisecond as Unit "ms" (1/1000). bench-show1Configuration governing generation of chart. See *) for the default values of these fields. bench-showProvide more details in the report, especially the standard deviation, outlier variance, R-square estimate and an annotation to indicate the actual method used when using  are reported. bench-showEThe directory where the output graph or report file should be placed. bench-showiReport title, more information like the plotted field name or the presentation style may be added to it.  bench-show/Additional annotations to be added to the title  bench-show7How to determine the layout of the report or the chart.  bench-show"The estimator used for the report.  bench-showThe minimum percentage difference between two runs of a benchmark beyond which the benchmark is flagged to have regressed or improved.  bench-show5Strategy to compare two runs or groups of benchmarks. bench-showFilter and reorder the benchmarking fields. It is invoked with a list of all available benchmarking fields. Only those fields present in the output of this function are plotted and in that order. bench-showThe values in the tuple are (fieldName, RangeMin, RangeMax)+. Specify the min and max range of benchmarking fields. If the field value is outside the range it is clipped to the range limit. For time fields, the range values are in microseconds, and for space fields they are in bytes. The minimum of the range is used to determine the unit for the field. bench-showThe values in the tuple are (fieldName, tick)M. Specify the tick size of the fields to be used for the graphical reports. bench-showFilter, group and translate benchmark names. This function is invoked once for all benchmark names found in the results. It produces a tuple (groupname, benchname), where  groupnameB is the name of the group the benchmark should be placed in, and  benchnameL is the translated benchmark name to be used in the report. If it returns X> for a benchmark, that benchmark is omitted from the results. bench-showWFilter and reorder the benchmark group names. A benchmark group may be assigned using 7; when not assigned, all benchmarks are placed in the default[ group. The input to this function is a list of tuples with benchmark group names and the runIds. The output produced by this function is a filtered and reordered subset of the input. Only those benchmark groups present in the output are rendered and are presented in that order. bench-showFilter and reorder benchmarks.  is provided with a function which is invoked with a sorting column name or index, the function produces the benchmark names and corresponding values for that column which can be used as a sorting criterion. The output of H is a list of benchmarks in the order in which they are to be rendered. bench-showYAdditional annotations that can be optionally added to the title of the report or graph. bench-showTStrategy to compute the difference between two groups of benchmarks being compared. bench-showUse a single estimator to compute the difference between the baseline and the candidate. The estimator that is provided in the  is used. bench-showUse ,  and  estimators for both baseline and candidate, and report the estimator that shows the minimum difference. This is more robust against random variations. bench-show0When sorting and filtering the benchmarks using / we can choose a column as a sort criterion.  is provided with the data for the corresponding column which can be used for sorting the benchmarks. The column could be a group or a field depending on the !. bench-showHSpecify the index of the sort column. Index 0 corresponds to the first value column. In a textual report, the very first column consists of benchmark names, therefore index 0 addresses the second column of the report. bench-showTSpecify the column using the name of the group or the field it represents, and the runIdJ. When just the name is enough to uniquely identify the sort column the Y( constructor can be used, otherwise the Z( constructor is used which can use the runId to disambiguate. In a $4 presentation, just the field name is enough. In a # presentation, when there is a single benchmark run in the input file, just the group name is enough to identify the group, the runIdT defaults to 0. However, when there are multiple runs, a group needs to specify a runId as well. bench-showFieldTick is used only in visual charts to generate the major ticks on the y-axis. You can specify either the size of a tick (!) or the total number of ticks ( ). bench-showVSize of a tick, the unit is microseconds for time fields, and bytes for space fields.  bench-show*Total number of ticks in the range spread.! bench-show8How to present the reports or graphs. Each report presents a number of benchmarks as rows, it may have, (1) a single column presenting the values for a single field, (2) multiple columns presenting values for different fields, or (3) multiple columns presenting values of the same field for different groups." bench-showReports are generated for each group and for each field selected by the configuration. Each report presents benchmarks in a single group with a single column presenting a single field. If there are m fields and n7 groups selected by the configuration then a total of m x n8 reports are generated. Output files are named using -estimator-groupname-fieldname as suffix.# bench-showOne report is generated for each field selected by the configuration. Each report presents a field with all the groups selected by the configuration as columns or clusters. Output files are named using -estimator-fieldname as suffix.$ bench-showOne report is generated for each group selected by the configuration. Each report presents a group with all the fields selected by the configuration as columns or clusters. Output files are named using -estimator-groupname as suffix.% bench-showbHow to show the results for multiple benchmark groups presented in columns or bar chart clusters.& bench-show)Show absolute field values for all groups' bench-showjShow baseline group values as usual and values for the subsequent groups as differences from the baseline( bench-showdShow baseline group values as 100% and values for subsequent groups as a percentage of the baseline) bench-showqShow baseline group values as usual and values for subsequent groups as precentage difference from the baseline* bench-showlDefault configuration. Use this as the base configuration and modify the required fields. The defaults are:  verbose = False title = Nothing titleAnnotations = [TitleField] outputDir = Nothing presentation = Groups Absolute estimator = Median threshold = 3 diffStrategy = MinEstimator selectFields = filter (flip elem ["time", "mean", "maxrss"] . map toLower) fieldRanges = [] fieldTicks = [] classifyBenchmark = Just . ("default",) selectGroups = id selectBenchmarks = f -> either error (map fst) $ f (ColumnIndex 0) K[\]^_`abcdefghijklmWn   !"#$%&'()opq*rstuvwxy!(c) 2018 Composewell TechnologiesBSD3harendra.kumar@gmail.com experimentalGHCNone"#<N+ bench-show*Presents the benchmark results in a CSV input file as graphical bar charts according to the provided configuration. The first parameter is the input file name, the second parameter is the name prefix for the output SVG image file(s). One or more output files may be generated depending on the !_ setting. The last parameter is the configuration to customize the graph, you can start with *I as the base and override any of the fields that you may want to change. For example: )graph "bench-results.csv" "output-graph" * +!(c) 2018 Composewell TechnologiesBSD3harendra.kumar@gmail.com experimentalGHCNone"#<Nr, bench-showPresents the benchmark results in a CSV input file as text reports according to the provided configuration. The first parameter is the input file name. The second parameter, when specified using z, is the name prefix for the output SVG image file(s). One or more output files may be generated with the given prefix depending on the !( setting. When the second parameter is X the reports are printed on the console. The last parameter is the configuration to customize the report, you can start with *I as the base and override any of the fields that you may want to change. For example: #report "bench-results.csv" Nothing * ,$(c) 2017-18 Composewell TechnologiesBSD3harendra.kumar@gmail.com experimentalGHCNone"#-   !"#$%&'()*+,-%&'()!"#$   *,+!(c) 2018 Composewell TechnologiesBSD3harendra.kumar@gmail.comNone"#{       !"#$%&'()*+,-./0123456789:;<=>?@ABCDEEFGHHIJKLLMNOPQRST3UVWXYZ[\Z]^Z]_``abcdefgghijkklmnoYpqrstuvwxyzZ[{|bench-show-0.2.2-inplace BenchShowBenchShow.AnalysisBenchShow.CommonBenchShow.GraphBenchShow.ReportBenchShow.Tutorial EstimatorMedianMean RegressionConfigverbose outputDirtitletitleAnnotations presentation estimator threshold diffStrategy selectFields fieldRanges fieldTicksclassifyBenchmark selectGroupsselectBenchmarksTitleAnnotation TitleFieldTitleEstimator TitleDiff DiffStrategySingleEstimator MinEstimator SortColumn ColumnIndex ColumnName FieldTickTickSize TickCount PresentationSoloGroupsFields GroupStyleAbsoluteDiffPercent PercentDiff defaultConfiggraphreportOutlierVarianceovEffectovDesc ovFraction OutlierEffect UnaffectedSlightModerateSevereOutliers lowSeverelowMildhighMild highSevereclassifyOutliersoutlierVariance countOutliersanalyzeBenchmarkBenchmarkMatrixcolNames rowValuesBenchmarkIterMatrixiterPredColNamesiterRespColNames iterRowValues AnalyzedField analyzedMeananalyzedStdDevanalyzedMediananalyzedOutliersanalyzedOutlierVar analyzedKDEanalyzedRegCoeffanalyzedRegRSq isMaxFieldgetAnalyzedValue foldBenchmark filterSamples RelativeUnitbaseGHC.BaseNothing Data.EitherLeftRight RawReportreportOutputFilereportIdentifier reportRowIds reportColumnsreportAnalyzedreportEstimators ReportColumncolNamecolUnit colValues GroupMatrix groupName groupIndex groupBenches groupMatrix ReportType TextReportGraphicalChart getFieldRange getFieldTickprepareToReportprepareGroupMatricesreportComparingGroupsreportPerGroup diffString makeTitleJust