!җ˹/      !"#$%&'()*+,-.O(c) 2009-2014 Bryan O'Sullivan (c) 2018 Composewell Technologies BSD-styleharendra.kumar@gmail.com experimentalGHC Trustworthy "#27HPV'2 bench-showThe statistical estimator used to arrive at a single value for a benchmark when samples from multiple experiments are available. bench-showReport the median, outliers and outlier variance using box-plot method. This is the most robust indicator with respect to outliers when successive runs of benchmarks are compared. bench-showlReport the mean and the standard deviation from the mean. This is less robust than median but more precise. bench-showqReport the coefficient of regression, discarding the constant factor, arrived at by linear regression using ordinary least square method. The R-square goodness-of-fit estimate is also reported. It works better when larger number of samples are taken. This cannot be used when the number of samples is less than 2, in that case a mean value is reported instead./ bench-showsAnalysis of the extent to which outliers in a sample affect its standard deviation (and to some extent, its mean).0 bench-show"Qualitative description of effect.1 bench-show$Brief textual description of effect.2 bench-show@Quantitative description of effect (a fraction between 0 and 1).3 bench-showpA description of the extent to which outliers in the sample data affect the sample mean and standard deviation.4 bench-showLess than 1% effect.5 bench-showBetween 1% and 10%.6 bench-showBetween 10% and 50%.7 bench-show+Above 50% (i.e. measurements are useless).8 bench-showCOutliers from sample data, calculated using the boxplot technique.9 bench-showJMore than 3 times the interquartile range (IQR) below the first quartile.: bench-show9Between 1.5 and 3 times the IQR below the first quartile.; bench-show9Between 1.5 and 3 times the IQR above the third quartile.< bench-show3More than 3 times the IQR above the third quartile.= bench-show=Classify outliers in a data set, using the boxplot technique.> bench-showgCompute the extent to which outliers in the sample data affect the sample mean and standard deviation.? bench-show/Count the total number of outliers in a sample.@ bench-show%Perform an analysis of a measurement.> bench-showmean bench-showstandard deviation. bench-showNumber of original iterations.&ABCDEFGHIJKLMNOPQRS/T01234567?UVWX!(c) 2018 Composewell TechnologiesBSD3harendra.kumar@gmail.com experimentalGHCNone"#>P2&Y bench-showDescribe a relative unit i.e. a unit in terms of another unit. A relative unit has a label and a ratio which when multiplied with the unit gives us the other unit. For example, if the known time unit is seconds, we can describe a millisecond as Unit "ms" (1/1000). bench-show1Configuration governing generation of chart. See ,) for the default values of these fields. bench-showProvide more details in the report, especially the standard deviation, outlier variance, R-square estimate and an annotation to indicate the actual method used when using  are reported. bench-showEThe directory where the output graph or report file should be placed. bench-showFunction to make a title for the report. The argument to the function is the benchmark field name for which the report is being made.  bench-showDEPRECATED: Please use  instead.iReport title, more information like the plotted field name or the presentation style may be added to it.  bench-showDEPRECATED: Please use  instead./Additional annotations to be added to the title  bench-show7How to determine the layout of the report or the chart.  bench-show"The estimator used for the report.  bench-showThe minimum percentage difference between two runs of a benchmark beyond which the benchmark is flagged to have regressed or improved. bench-show5Strategy to compare two runs or groups of benchmarks. bench-showJOmit the baseline group in normalized relative comparisons i.e. when the ' is * or +. bench-showFilter and reorder the benchmarking fields. It is invoked with a list of all available benchmarking fields. Only those fields present in the output of this function are plotted and in that order. bench-showThe values in the tuple are (fieldName, RangeMin, RangeMax)+. Specify the min and max range of benchmarking fields. If the field value is outside the range it is clipped to the range limit. For time fields, the range values are in microseconds, and for space fields they are in bytes. The minimum of the range is used to determine the unit for the field. bench-showThe values in the tuple are (fieldName, tick)M. Specify the tick size of the fields to be used for the graphical reports. bench-showFilter, group and translate benchmark names. This function is invoked once for all benchmark names found in the results. It produces a tuple (groupname, benchname), where  groupnameB is the name of the group the benchmark should be placed in, and  benchnameL is the translated benchmark name to be used in the report. If it returns Z> for a benchmark, that benchmark is omitted from the results. bench-showWFilter and reorder the benchmark group names. A benchmark group may be assigned using 7; when not assigned, all benchmarks are placed in the default[ group. The input to this function is a list of tuples with benchmark group names and the runIds. The output produced by this function is a filtered and reordered subset of the input. Only those benchmark groups present in the output are rendered and are presented in that order. bench-showFilter and reorder benchmarks. ` takes a function argument, the function is invoked with a sorting column name or index and a '). The output of the function is either a [q value consisting of tuples of the benchmark names and values corresponding to the given column and style or a \ value indicating an error.  can inspect these benchmarks and there values to produce a filtered and sorted list of benchmark names that are to be rendered.CThe style argument is ignored when the report presentation is not %. When style is ZC, the presentation setting specified in the configuration is used.Signature changed in 0.3.0 bench-showYAdditional annotations that can be optionally added to the title of the report or graph. bench-showTStrategy to compute the difference between two groups of benchmarks being compared. bench-showUse a single estimator to compute the difference between the baseline and the candidate. The estimator that is provided in the  is used. bench-showUse ,  and  estimators for both baseline and candidate, and report the estimator that shows the minimum difference. This is more robust against random variations. bench-show0When sorting and filtering the benchmarks using / we can choose a column as a sort criterion.  is provided with the data for the corresponding column which can be used for sorting the benchmarks. The column could be a group or a field depending on the #. bench-showHSpecify the index of the sort column. Index 0 corresponds to the first value column. In a textual report, the very first column consists of benchmark names, therefore index 0 addresses the second column of the report. bench-showTSpecify the column using the name of the group or the field it represents, and the runIdJ. When just the name is enough to uniquely identify the sort column the \( constructor can be used, otherwise the [( constructor is used which can use the runId to disambiguate. In a &4 presentation, just the field name is enough. In a % presentation, when there is a single benchmark run in the input file, just the group name is enough to identify the group, the runIdT defaults to 0. However, when there are multiple runs, a group needs to specify a runId as well.  bench-showFieldTick is used only in visual charts to generate the major ticks on the y-axis. You can specify either the size of a tick (!!) or the total number of ticks (").! bench-showVSize of a tick, the unit is microseconds for time fields, and bytes for space fields." bench-show*Total number of ticks in the range spread.# bench-show8How to present the reports or graphs. Each report presents a number of benchmarks as rows, it may have, (1) a single column presenting the values for a single field, (2) multiple columns presenting values for different fields, or (3) multiple columns presenting values of the same field for different groups.$ bench-showReports are generated for each group and for each field selected by the configuration. Each report presents benchmarks in a single group with a single column presenting a single field. If there are m fields and n7 groups selected by the configuration then a total of m x n8 reports are generated. Output files are named using -estimator-groupname-fieldname as suffix.% bench-showOne report is generated for each field selected by the configuration. Each report presents a field with all the groups selected by the configuration as columns or clusters. Output files are named using -estimator-fieldname as suffix.& bench-showOne report is generated for each group selected by the configuration. Each report presents a group with all the fields selected by the configuration as columns or clusters. Output files are named using -estimator-groupname as suffix.' bench-showHow to show the results for multiple benchmark groups presented in columns or bar chart clusters. In relative comparisons, the first group is considered as the baseline and the subsequent groups are compared against the baseline.Definition changed in 0.3.0 @since 0.2.0( bench-show0Show absolute values of the field for all groups) bench-showlShow baseline group as absolute values and values for the subsequent groups as difference from the baseline* bench-showIf the value of the group being compared is higher than the baseline then display the difference as percentage of baseline otherwise display the difference as a percentage of the group being compared.+ bench-showSIf the value of the group being compared is higher than the baseline then display +(value / baseline value) otherwise display -(baseline value / value)f. This provides a normalized comparison independent of the absolute value of a benchmark. Note that +! can be directly computed using * and vice-versa., bench-showlDefault configuration. Use this as the base configuration and modify the required fields. The defaults are:  verbose = False mkTitle = Nothing titleAnnotations = [TitleField] outputDir = Nothing presentation = Groups Absolute estimator = Median threshold = 3 diffStrategy = SingleEstimator omitBaseline = False selectFields = filter (flip elem ["time", "mean", "maxrss"] . map toLower) fieldRanges = [] fieldTicks = [] classifyBenchmark = Just . ("default",) selectGroups = id selectBenchmarks = f -> either error (map fst) $ f (ColumnIndex 0) Nothing M]^_`abcdefghijklmnoYp   !"#$%&')(*+qrs,tuvwxyz{!(c) 2018 Composewell TechnologiesBSD3harendra.kumar@gmail.com experimentalGHCNone"#>P- bench-show*Presents the benchmark results in a CSV input file as graphical bar charts according to the provided configuration. The first parameter is the input file name, the second parameter is the name prefix for the output SVG image file(s). One or more output files may be generated depending on the #_ setting. The last parameter is the configuration to customize the graph, you can start with ,I as the base and override any of the fields that you may want to change. For example: )graph "bench-results.csv" "output-graph" , -!(c) 2018 Composewell TechnologiesBSD3harendra.kumar@gmail.com experimentalGHCNone"#>Pd. bench-showPresents the benchmark results in a CSV input file as text reports according to the provided configuration. The first parameter is the input file name. The second parameter, when specified using |, is the name prefix for the output SVG image file(s). One or more output files may be generated with the given prefix depending on the #( setting. When the second parameter is Z the reports are printed on the console. The last parameter is the configuration to customize the report, you can start with ,I as the base and override any of the fields that you may want to change. For example: #report "bench-results.csv" Nothing , .$(c) 2017-18 Composewell TechnologiesBSD3harendra.kumar@gmail.com experimentalGHCNone"#/   !"#$%&')(*+,-./')(*+#$%& !"  ,.-!(c) 2018 Composewell TechnologiesBSD3harendra.kumar@gmail.comNone"#˵}       !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGGHIJJKLMNNOPQRSTUV5WXYZ[\]^\_`\_abbcdefghiijklmmnopq[rstuvwxyz{|\]}~'bench-show-0.3.0-DiV8CYH4wYc9T4NDZNIpwo BenchShowBenchShow.AnalysisBenchShow.CommonBenchShow.GraphBenchShow.ReportBenchShow.Tutorial EstimatorMedianMean RegressionConfigverbose outputDirmkTitletitletitleAnnotations presentation estimator threshold diffStrategy omitBaseline selectFields fieldRanges fieldTicksclassifyBenchmark selectGroupsselectBenchmarksTitleAnnotation TitleFieldTitleEstimator TitleDiff DiffStrategySingleEstimator MinEstimator SortColumn ColumnIndex ColumnName FieldTickTickSize TickCount PresentationSoloGroupsFields GroupStyleAbsoluteDiff PercentDiff Multiples defaultConfiggraphreportOutlierVarianceovEffectovDesc ovFraction OutlierEffect UnaffectedSlightModerateSevereOutliers lowSeverelowMildhighMild highSevereclassifyOutliersoutlierVariance countOutliersanalyzeBenchmarkBenchmarkMatrixcolNames rowValuesBenchmarkIterMatrixiterPredColNamesiterRespColNames iterRowValues AnalyzedField analyzedMeananalyzedStdDevanalyzedMediananalyzedOutliersanalyzedOutlierVar analyzedKDEanalyzedRegCoeffanalyzedRegRSq isMaxFieldgetAnalyzedValue foldBenchmark filterSamples RelativeUnitbase GHC.MaybeNothing Data.EitherRightLeft RawReportreportOutputFilereportIdentifier reportRowIds reportColumnsreportAnalyzedreportEstimators ReportColumncolNamecolUnit colValues GroupMatrix groupName groupIndex groupBenches groupMatrix ReportType TextReportGraphicalChart getFieldRange getFieldTickprepareToReportprepareGroupMatricesreportComparingGroupsreportPerGroup diffString makeTitleJust