Benchmark Health Considered Harmful

BenchmarkHealth Considered Harmful
Craig B. Zilles
Computer Sciences Department, University of Wisconsin - Madison
1210 West Dayton Street, Madison, WI 53706-1685, USA
zilles@cs.wisc.edu
I am not interested in the accuracy of the simulation, butAbstract
rather whether it is a useful benchmark to use for perfor-In the past couple of years, a number of software and
mance studies. The number of waiters only impacts thearchitectural techniques have been proposed for improving
benchmark’s run-time behavior if the list is frequently tra-the performance of linked data structures. These research
versed. The list is traversed on two occasions: 1) once forideas are often evaluated using the Olden benchmark
each iteration, to increment the time each waiter has spentsuite [1]. Frequently, in such experiments, the largest
waiting, and 2) once for each list insertion, because ele-speed-up is attained for the benchmark called health.
ments are inserted at the end of the list (and no tail pointer isThis article demonstrates that this benchmark is a
kept). This means that the monotonically increasing amountmicro-benchmark for enormous linked list traversals, and
of memory used in the simulation is touched (dirtied even)not a good one at that. Given that linked lists of such size
at least once every iteration of the simulation.are not an efficient data structure, it is unlikely that this
Shortly after the simulation begins, the run time is domi-benchmark corresponds to any real program. ...