criterion performance measurements
overview
want to understand this report?
|V|=1628 |E|=26703 |SCC|=690/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.369990457582142e-2 | 9.548189713846603e-2 | 9.681892218833565e-2 |
| Standard deviation | 1.5580329849116722e-3 | 2.50119057605301e-3 | 3.823769033514356e-3 |
Outlying measurements have slight (9.876543209876538e-2%) effect on estimated standard deviation.
|V|=1628 |E|=26703 |SCC|=690/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.386872707581562e-2 | 6.556633412875083e-2 | 6.704676450210162e-2 |
| Standard deviation | 1.906846110533029e-3 | 2.7806389635216394e-3 | 4.072146954059652e-3 |
Outlying measurements have slight (8.43820765566925e-2%) effect on estimated standard deviation.
|V|=87 |E|=404 |SCC|=48/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.878954051483708e-4 | 4.88902989356697e-4 | 4.903647718451131e-4 |
| Standard deviation | 3.0448085416616903e-6 | 4.291207396586043e-6 | 6.728509724510396e-6 |
Outlying measurements have slight (1.1626297577854584e-2%) effect on estimated standard deviation.
|V|=87 |E|=404 |SCC|=48/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.896086604634904e-4 | 3.9609631593352226e-4 | 4.0830226347360343e-4 |
| Standard deviation | 1.491903927482115e-5 | 2.7603477851367328e-5 | 4.566752776668535e-5 |
Outlying measurements have severe (0.6149014919998123%) effect on estimated standard deviation.
|V|=3487 |E|=57949 |SCC|=852/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.294790126880107 | 0.3009453470472848 | 0.3071152258815709 |
| Standard deviation | 6.024071108536542e-3 | 7.861264887908281e-3 | 9.13974279901956e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
|V|=3487 |E|=57949 |SCC|=852/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.21647399232824682 | 0.2252294475704629 | 0.2288044869879943 |
| Standard deviation | 1.1750423462052002e-3 | 7.4530917825890546e-3 | 1.174877839771253e-2 |
Outlying measurements have moderate (0.13888888888888873%) effect on estimated standard deviation.
|V|=349 |E|=3228 |SCC|=78/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.517243508083669e-3 | 4.594978541507301e-3 | 4.704511885442327e-3 |
| Standard deviation | 2.263942612085455e-4 | 3.044716763802864e-4 | 4.5748330053401235e-4 |
Outlying measurements have moderate (0.41300853514729957%) effect on estimated standard deviation.
|V|=349 |E|=3228 |SCC|=78/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.186372842075784e-3 | 4.2667643732439125e-3 | 4.378687198699892e-3 |
| Standard deviation | 2.225492500515911e-4 | 2.8865859348787176e-4 | 4.0741046230576743e-4 |
Outlying measurements have moderate (0.4385148343083135%) effect on estimated standard deviation.
|V|=10000 |E|=100000 |SCC|=2/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.24705319039834045 | 0.2568363680088563 | 0.2701836965500358 |
| Standard deviation | 6.836261582551843e-3 | 1.3975410709344676e-2 | 1.858476831873051e-2 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
|V|=10000 |E|=100000 |SCC|=2/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.25762609027583194 | 0.2679839178744684 | 0.27901963276187114 |
| Standard deviation | 1.0252059409630003e-2 | 1.4084900393659361e-2 | 1.6271827546176242e-2 |
Outlying measurements have moderate (0.15999999999999998%) effect on estimated standard deviation.
|V|=15 |E|=21 |SCC|=15/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.217136185228015e-5 | 3.237558670015057e-5 | 3.25523403332683e-5 |
| Standard deviation | 5.415325616468926e-7 | 6.397232487541014e-7 | 7.749965723525306e-7 |
Outlying measurements have moderate (0.1694562126064402%) effect on estimated standard deviation.
|V|=15 |E|=21 |SCC|=15/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.769282923466225e-5 | 2.786314769366947e-5 | 2.810502522698544e-5 |
| Standard deviation | 5.037352764365779e-7 | 6.583809021865079e-7 | 1.0604399463708775e-6 |
Outlying measurements have moderate (0.22669916796634504%) effect on estimated standard deviation.
|V|=7000 |E|=70000 |SCC|=1/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.14782954353930627 | 0.14924961498650036 | 0.15055015462113713 |
| Standard deviation | 1.5258296568523208e-3 | 2.138202194803505e-3 | 2.776253609267545e-3 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
|V|=7000 |E|=70000 |SCC|=1/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.0830882205805105e-2 | 6.227378360236694e-2 | 6.409531968755218e-2 |
| Standard deviation | 2.189526810253358e-3 | 2.918601589824393e-3 | 4.175025470533906e-3 |
Outlying measurements have moderate (0.15020032372387493%) effect on estimated standard deviation.
|V|=8000 |E|=80000 |SCC|=1/KL-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1649994082891637 | 0.16995032820440287 | 0.17446841885250905 |
| Standard deviation | 4.4269550656315625e-3 | 6.575420841743258e-3 | 1.0710691436800465e-2 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
|V|=8000 |E|=80000 |SCC|=1/AM-alga
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.537693284811738e-2 | 7.73815063181509e-2 | 7.965575168959292e-2 |
| Standard deviation | 2.828854793545781e-3 | 3.609231996674328e-3 | 4.970117098088466e-3 |
Outlying measurements have slight (8.551290713308873e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.