criterion performance measurements
overview
want to understand this report?
fromList/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.412648800634957e-2 | 2.4483191034136542e-2 | 2.4894721755854433e-2 |
| Standard deviation | 3.974565008725139e-4 | 8.328277308264194e-4 | 1.3162301534044244e-3 |
Outlying measurements have slight (9.738826053603968e-2%) effect on estimated standard deviation.
fromList/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1534213999729655 | 0.1747344616325263 | 0.19318632467569455 |
| Standard deviation | 1.830940818773069e-2 | 2.6927063462579878e-2 | 3.53073842525747e-2 |
Outlying measurements have moderate (0.4725850202791134%) effect on estimated standard deviation.
fromList/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.44046598735917436 | 0.44884157807369723 | 0.4569196548598263 |
| Standard deviation | 0.0 | 1.3741233431720462e-2 | 1.3991639421018238e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
fromList/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.17482854887163338 | 0.1902273143988065 | 0.1996660158334313 |
| Standard deviation | 1.0316342582011275e-2 | 1.6096568751426992e-2 | 2.0485856334169345e-2 |
Outlying measurements have moderate (0.15656378164571164%) effect on estimated standard deviation.
insert/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.081685608369317e-2 | 7.786839635111972e-2 | 8.245090163491764e-2 |
| Standard deviation | 5.217212214837219e-3 | 8.947780516275332e-3 | 1.327769773438149e-2 |
Outlying measurements have moderate (0.38106645984391546%) effect on estimated standard deviation.
insert/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.6276725811442991 | 0.6299008889970471 | 0.6317383521542661 |
| Standard deviation | 0.0 | 2.9038944722208217e-3 | 3.1825795453392907e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
insert/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.41154342948629935 | 0.43711719802339005 | 0.45614781110217767 |
| Standard deviation | 0.0 | 2.9006388485886495e-2 | 3.296198875164503e-2 |
Outlying measurements have moderate (0.1945567702837197%) effect on estimated standard deviation.
insert/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.7192818818632519 | 0.7225154152454741 | 0.724302665866059 |
| Standard deviation | 0.0 | 2.846644785065061e-3 | 3.0956088807120494e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
toList/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.739900467123115e-3 | 4.754439150983831e-3 | 4.7716939619166345e-3 |
| Standard deviation | 3.8809242432882757e-5 | 5.004256956662484e-5 | 6.38833646060042e-5 |
Outlying measurements have slight (2.3242630385487528e-2%) effect on estimated standard deviation.
toList/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.203517981313218e-3 | 6.229740957598831e-3 | 6.267297717580343e-3 |
| Standard deviation | 6.75585645057132e-5 | 9.248787065511906e-5 | 1.443291738870905e-4 |
Outlying measurements have slight (2.629656683710734e-2%) effect on estimated standard deviation.
toList/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.378809973242944e-3 | 4.3916669560981865e-3 | 4.412629062580165e-3 |
| Standard deviation | 3.7395762030671884e-5 | 5.1252968906191644e-5 | 6.846216776756695e-5 |
Outlying measurements have slight (2.271498107084911e-2%) effect on estimated standard deviation.
toList/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3486580240643082e-2 | 2.40099591588223e-2 | 2.4468714534031544e-2 |
| Standard deviation | 9.149215593440743e-4 | 1.0933749701844172e-3 | 1.2850544881348141e-3 |
Outlying measurements have moderate (0.14800815149402494%) effect on estimated standard deviation.
lookup/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.5375915251825834e-2 | 4.5627379312661476e-2 | 4.6042851120592254e-2 |
| Standard deviation | 3.97061966901405e-4 | 5.944508584768441e-4 | 7.600391957398097e-4 |
Outlying measurements have slight (6.632653061224489e-2%) effect on estimated standard deviation.
lookup/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.756404216201929e-2 | 9.830584251050156e-2 | 9.880067970935094e-2 |
| Standard deviation | 5.641929049369217e-4 | 8.984144454276187e-4 | 1.380279968688867e-3 |
Outlying measurements have slight (9.87654320987653e-2%) effect on estimated standard deviation.
lookup/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.614710129666534e-2 | 9.674579019020521e-2 | 9.760639813541648e-2 |
| Standard deviation | 7.860487032418642e-4 | 1.083145982630007e-3 | 1.497649354424678e-3 |
Outlying measurements have slight (9.876543209876543e-2%) effect on estimated standard deviation.
lookup/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.13197397477330355 | 0.1326362211377632 | 0.13396622275669196 |
| Standard deviation | 5.698191064928849e-4 | 1.3759915097588812e-3 | 2.079366445851779e-3 |
Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.
delete/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.240555175744418e-2 | 5.283568718037974e-2 | 5.3199642865652506e-2 |
| Standard deviation | 5.826850456990582e-4 | 7.523974935309557e-4 | 9.42933159555392e-4 |
Outlying measurements have slight (7.100591715976314e-2%) effect on estimated standard deviation.
delete/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.6540663953612682 | 0.6738188573569394 | 0.6912871713142907 |
| Standard deviation | 0.0 | 2.8484685870575382e-2 | 3.0256007296697144e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
delete/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.452512810615416 | 0.459547457216936 | 0.4649249674196938 |
| Standard deviation | 0.0 | 8.261800493807954e-3 | 9.31412088939653e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
delete/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.7485480433581403 | 0.7769275967427282 | 0.801340668195834 |
| Standard deviation | 0.0 | 3.930255830896414e-2 | 4.2284680125588836e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.