criterion performance measurements
overview
want to understand this report?
vertices [0..1000]+circuit [1001..10000]/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.240546981265899e-2 | 2.2507821927745936e-2 | 2.2648833423307438e-2 |
| Standard deviation | 2.14378989880587e-4 | 2.7146407949532223e-4 | 3.406385309386302e-4 |
Outlying measurements have slight (4.75e-2%) effect on estimated standard deviation.
vertices [0..1000]+circuit [1001..10000]/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6337452442743122e-2 | 1.6448874841601898e-2 | 1.6568649794835143e-2 |
| Standard deviation | 2.145094568623755e-4 | 2.767769221959165e-4 | 3.6794868583943813e-4 |
Outlying measurements have slight (3.993055555555542e-2%) effect on estimated standard deviation.
vertices [0..1000]+circuit [1001..10000]/AIM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4439567812357052e-2 | 2.461382464394668e-2 | 2.4860590585141108e-2 |
| Standard deviation | 3.043636924611619e-4 | 4.5022045519306465e-4 | 6.868958502255852e-4 |
Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.
overlays [ edge x (x+1000) | x <- [0..1000] ] + circuit [1001..10000]]/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4433603622492228 | 0.4452256064978428 | 0.44853350586890883 |
| Standard deviation | 2.975117495225277e-4 | 3.201228999868938e-3 | 3.991401673348222e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
overlays [ edge x (x+1000) | x <- [0..1000] ] + circuit [1001..10000]]/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2246105716925134 | 0.22584429764412764 | 0.22812952900049924 |
| Standard deviation | 9.718936435400732e-4 | 2.3082812568272376e-3 | 3.3501836309799858e-3 |
Outlying measurements have moderate (0.13888888888888878%) effect on estimated standard deviation.
overlays [ edge x (x+1000) | x <- [0..1000] ] + circuit [1001..10000]]/AIM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.526019762798334e-2 | 9.597170559392627e-2 | 9.659017276000725e-2 |
| Standard deviation | 7.491469829786145e-4 | 1.0775799891653034e-3 | 1.5306440754204317e-3 |
Outlying measurements have slight (9.876543209876538e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.