## Iterative Optimization Heuristics Profiler

## Development Team

Hao Wang,

*Leiden Institute of Advanced Computer Science*, h.wang@liacs.leidenuniv.nl.Carola Doerr,

*CNRS and Sorbonne University*, Carola.Doerr@mpi-inf.mpg.de.Furong Ye,

*Leiden Institute of Advanced Computer Science*, f.ye@liacs.leidenuniv.nl.Sander van Rijn,

*Leiden Institute of Advanced Computer Science*, s.j.van.rijn@liacs.leidenuniv.nl.Thomas BĂ¤ck,

*Leiden Institute of Advanced Computer Science*, t.h.w.baeck@liacs.leidenuniv.nl.

## Reference

When using IOHProfiler and parts thereof, please kindly cite this work as

[Hao Wang, Furong Ye, Carola Doerr, Sander van Rijn, Thomas BĂ¤ck: *IOHProfiler: A New Tool for Benchmarking and Profiling Iterative Optimization Heuristics*, arXiv, 2018]

```
@article{IOHProfiler,
author = {Hao Wang, Furong Ye, Carola Doerr, Sander van Rijn, Thomas BĂ¤ck},
title = {IOHProfiler: A New Tool for Benchmarking and Profiling Iterative Optimization Heuristics},
journal = {CoRR},
volume = {tbd},
year = {2018},
url = {tbd}
}
```

## Acknowledgements

Carola Doerr thanks Anne Auger and Dimo Brockhoff from INRIA Saclay, France, for very helpful discussions about the performance criteria used by the COCO (COmparing Continuous Optimisers) platform (https://github.com/numbbo/coco). Furong Ye acknowledges financial support from the China Scholarship Council, CSC No. 201706310143.

## License

This application is governed by the **BSD 3-Clause license**.

BSD 3-Clause License

Copyright © 2018, All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

**Remark**: This is just a mockup, to show that we could put some important/concise documents here.

## R Markdown

This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.

When you click the **Knit** button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:

```
summary(cars)
```

## Including Plots

You can also embed plots, for example:

```
plot(pressure)
```

Note that the `echo = FALSE`

parameter was added to the code chunk to prevent printing of the R code that generated the plot.

Upload Data

Data Processing Prompt

List of Processed Data

Runtime Statistics at Chosen Target Values

This table summarizes for each algorithm and each target value chosen on the left:

- runs: the number of runs that have found at least one solution of the required target quality \(f(x)\),
- mean: the average number of function evaluations needed to find a solution of function value at least \(f(x)\)
- median, \(2\%, 5\%,\ldots,98\%\) : the quantiles of these first-hitting times

When not all runs managed to find the target value, the statistics hold only for those runs that did. That is, the mean value is the mean of the successful runs. Same for the quantiles. An alternative version with simulated restarts is currently in preparation.

Original Runtime Samples

This table shows for each selected algorithm \(A\), each selected target value \(f(x)\), and each run \(r\) the number \(T(A,f(x),r)\) of evaluations performed by the algorithm until it evaluated for the first time a solution of quality at least \(f(x)\).

Expected Runtime (per function)

The ** mean, median
and standard deviation** of the runtime samples
are depicted against the best objective values.
The displayed elements (mean, median, standard deviations)
can be switched on and off by clicking on the legend on the right.
A

**tooltip**and

**toolbar**appears when hovering over the figure.

### Histogram of Fixed-Target Runtimes

This histogram counts how many runs needed between
\(t\) and \(t+1\) function evaluations. The bins
\([t,t+1)\) are chosen automatically. The bin size is determined
by the so-called **Freedmanâ€“Diaconis rule**: \(\text{Bin size}=
2\frac{Q_3 - Q_1}{\sqrt[3]{n}}\), where \(Q_1, Q_3\) are the \(25\%\)
and \(75\%\) percentile of the runtime and \(n\) is the sample size.
The displayed algorithms can be selected by clicking on the legend on the right.
A **tooltip** and **toolbar** appears when hovering over the figure.

### Empirical Probability Mass Function of the Runtime

**Warning! **The
**probability mass function** of the runtime is approximated by the
treating the runtime as a *continuous* random variable and applying the **kernel estimation** (KDE):

The plot shows the distribution of the first hitting
times of the individual runs (dots), and an estimated
distribution of the probability mass function.
The displayed algorithms can be selected by clicking on
the legend on the right. A **tooltip** and **toolbar**
appear when hovering over the figure. This also includes the
option to download the plot as png file. A csv file with the runtime
data can be downlowaded from the
Data Summary tab.

Empirical Cumulative Distribution of the runtime: Aggregation

The evenly spaced target values are:

The fraction of (run,target value)
pairs \((i,v)\) satisfying that the best solution that the algorithm has
found in the \(i\)-th run within the given time budget \(t\) has quality at least
\(v\) is plotted against the available budget \(t\). The displayed elements can be switched
on and off by clicking on the legend on the right. A **tooltip**
and **toolbar** appears when hovering over the figure.

Area Under the ECDF

The **area under the ECDF** is
caculated for the sequence of target values specified on the left. The displayed
values are normalized against the maximal number of function evaluations for
each algorithm.
Intuitively, the larger the area, the better the algorithm.
The displayed algorithms can be selected by clicking on the legend on the right.
A **tooltip** and **toolbar** appears when hovering over the figure.
This also includes the option to download the plot as png file.

Empirical Cumulative Distribution of the Runtime: Single Target

Each EDCF curve shows the proportion of the runs
that have found a solution of at least the required
target value within the budget given by the \(x\)-axis.
The displayed curves can be selected by clicking on the legend on the right. A **tooltip**
and **toolbar** appears when hovering over the figure.
This also includes the option to download the plot as png file.

Target Statistics at Chosen Budget Values

This table summarizes for each algorithm and each **budget** \(B\) chosen on the left:

- runs: the number of runs that have performed at least \(B\) evaluations,
- mean: the average best-so-far function value obtain within a budget of \(B\) evaluations,
- median, \(2\%, 5\%,\ldots,98\%\) : the quantiles of the best function values found within the first \(B\) evaluations.

When not all runs evaluated at least \(B\) search points, the statistics hold for the subset of runs that did. Alternative statistics using simulated restarted algorithms are in preparation.

Original Target Samples

### Histogram of Fixed-Budget Targets

This histogram counts the number of runs whose best-so-far function values within the
first \(B\) evaluations is between \(v_i\) and \(v_{i+1}\). The buckets \([v_i,v_{i+1})\) are chosen automatically
according to the so-called **Freedmanâ€“Diaconis rule**: \(\text{Bin size}= 2\frac{Q_3 - Q_1}{\sqrt[3]{n}}\),
where \(Q_1, Q_3\) are the \(25\%\) and \(75\%\) percentile of the runtime and \(n\) is the sample size.
The displayed algorithms can be selected by clicking on the legend on the right.
A **tooltip** and **toolbar** appears when hovering over the figure.

### Empirical Probability Density Function of Fixed-Budget Function Values

The plot shows, for the budget selected on the left, the distribution
of the best-so-far function values of the individual runs (dots), and an estimated distribution of the probability mass function.
The displayed algorithms can be selected by clicking on the legend on the right. A **tooltip** and **toolbar**
appear when hovering over the figure. A csv file with the runtime data can be downlowaded from the
Data Summary tab.

Expected Target Value (per function)

The ** mean, median
and standard deviation** of the best function values
found with a fixed-budget of evaluations are depicted against the budget.
The displayed elements
can be switched on and off by clicking on the legend on the right.
A

**tooltip**and

**toolbar**appears when hovering over the figure.

Empirical Cumulative Distribution of the Fixed-Budget Values: Aggregation

The evenly spaced budget values are:

The fraction of (run,budget) pairs \((i,B)\) satisfying
that the best solution that the algorithm has found in the \(i\)-th run within the first \(B\) evaluations has quality at
**most** \(v\) is plotted against the target value \(v\). The displayed elements can be switched on and off by clicking on the
legend on the right. A **tooltip** and **toolbar** appears when hovering over the figure.

Area Under the ECDF

The **area under the ECDF** is
caculated for the sequence of budget values specified on the left. The displayed
values are normalized against the maximal target value recorded for
each algorithm. Intuitively, the **smaller** the area, the **better** the algorithm.
The displayed algorithms can be selected by clicking on the legend on the right.
A **tooltip** and **toolbar** appears when hovering over the figure.

Empirical Cumulative Distribution of the Fixed-Budget Values: Single Budgets

Each EDCF curve shows the proportion of the runs that have found within the
given budget B a solution of at least the required target value given by the x-axis. The displayed curves can be selected
by clicking on the legend on the right. A **tooltip** and **toolbar** appears when hovering over the figure.

Expected Parameter Value (per function)

The ** mean or median** of internal parameters of the algorithm
found with a fixed-budget of evaluations are depicted against the budget.
The displayed elements can be switched on and off by clicking on the legend on the right.
A

**tooltip**and

**toolbar**appears when hovering over the figure.

Parameter Statistics at Chosen Target Values

This table summarizes for each algorithm and each target value chosen on the left:

- runs: the number of runs that have found at least one solution of the required target quality \(f(x)\),
- mean: the average number of function evaluations needed to find a solution of function value at least \(f(x)\)
- median, \(2\%, 5\%,\ldots,98\%\) : the quantiles of these first-hitting times

When not all runs managed to find the target value, the statistics hold only for those runs that did. That is, the mean value is the mean of the successful runs. Same for the quantiles. An alternative version with simulated restarts is currently in preparation.