The Design and Analysis of Benchmark Experiments

The Design and Analysis of Benchmark
Experiments
Torsten Hothorn Friedrich Leisch Achim Zeileis
Friedrich-Alexander-Universitat Technische Universitat Wien Wirtschaftsuniversitat Wien
Erlangen-Nurnberg
Kurt Hornik
Wirtschaftsuniversitat Wien
Abstract
The assessment of the performance of learners by means of benchmark experiments is an
established exercise. In practice, benchmark studies are a tool to compare the performance of
several competing algorithms for a certain learning problem. Cross-validation or resampling
techniques are commonly used to derive point estimates of the performances which are com-
pared to identify algorithms with good properties. For several benchmarking problems, test
procedures taking the variability of those point estimates into account have been suggested.
Most of the recently proposed inference procedures are based on special variance estimators
for the cross-validated performance.
We introduce a theoretical framework for inference problems in benchmark experiments
and show that standard statistical test procedures can be used to test for di erences in the
performances. The theory is based on well de ned distributions of performance measures
which can be compared with established tests. To demonstrate the usefulness in practice,
the theoretical results are applied to regression and classi cation benchmark studies based on
arti cial and real world data.
Keywords: model comparison, performance, hypothesis testing, cross-validation, bootstrap ...