criterion performance measurements
overview
want to understand this report?
| |V|=4039 |E|=88234 |SCC|=4039 | |
| |V|=81306 |E|=1768149 |SCC|=12248 |
|V|=4039 |E|=88234 |SCC|=4039/AM-alga
200 205 210 215 220 225 230 235
mean |
2 3 4 5 6 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 202 ms | 217 ms | 258 ms |
| R² goodness-of-fit | 0.915 | 0.980 | 1.000 |
| Mean execution time | 203 ms | 209 ms | 225 ms |
| Standard deviation | 1.01 ms | 12.7 ms | 17.8 ms |
Outlying measurements have moderate (14.7%) effect on estimated standard deviation.
|V|=4039 |E|=88234 |SCC|=4039/AIM-alga
170 175 180 185 190 195 200
mean |
2 3 4 5 6 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 167 ms | 182 ms | 220 ms |
| R² goodness-of-fit | 0.916 | 0.975 | 1.000 |
| Mean execution time | 171 ms | 176 ms | 190 ms |
| Standard deviation | 2.01 ms | 11.7 ms | 16.5 ms |
Outlying measurements have moderate (15.0%) effect on estimated standard deviation.
|V|=4039 |E|=88234 |SCC|=4039/KL-alga
432 433 433 434 434 435 435 436 436
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 430 ms | 436 ms | 443 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 432 ms | 434 ms | 435 ms |
| Standard deviation | 738 μs | 1.50 ms | 1.98 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
|V|=81306 |E|=1768149 |SCC|=12248/AM-alga
7.35 7.4 7.45 7.50
mean |
1 2 2 3 3 4 4 0.5 iters 5 10 15 20 25 30 35 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 7.32 s | 7.44 s | 7.70 s |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 7.36 s | 7.44 s | 7.49 s |
| Standard deviation | 38.0 ms | 74.0 ms | 103 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
|V|=81306 |E|=1768149 |SCC|=12248/AIM-alga
4.35 4.40 4.45 4.5 4.55 4.6 4.65
mean |
1 2 2 3 3 4 4 0.5 iters 5 10 15 20 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.21 s | 4.29 s | 4.43 s |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 4.39 s | 4.47 s | 4.62 s |
| Standard deviation | 8.07 ms | 134 ms | 167 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
|V|=81306 |E|=1768149 |SCC|=12248/KL-alga
58 59 60 61 62 58.5 59.5 60.5 61.5
mean |
1 2 2 3 3 4 4 0.5 iters 50 100 150 200 250 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 49.2 s | 57.5 s | 61.4 s |
| R² goodness-of-fit | 0.995 | 0.997 | 1.000 |
| Mean execution time | 58.3 s | 60.7 s | 61.6 s |
| Standard deviation | 102 ms | 1.64 s | 2.01 s |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.