| 1 |
In June of 1997, Margo Seltzer and Aaron Brown published a paper in |
| 2 |
Sigmetrics called "Operating System Benchmarking in the Wake of Lmbench: |
| 3 |
A Case Study of the Performance of NetBSD on the Intel x86 Architecture". |
| 4 |
|
| 5 |
|
| 6 |
This papers claims to have found flaws in the original lmbench work. |
| 7 |
With the exception of one bug, which we have of course fixed, we find |
| 8 |
the claims inaccurate, misleading, and petty. We don't understand |
| 9 |
what appears to be a pointless attack on something that has obviously |
| 10 |
helped many researchers and industry people alike. lmbench was warmly |
| 11 |
received and is widely used and referenced. We stand firmly behind the |
| 12 |
work and results of the original benchmark. We continue to improve and |
| 13 |
extend the benchmark. Our focus continues to be on providing a useful, |
| 14 |
accurate, portable benchmark suite that is widely used. As always, we |
| 15 |
welcome constructive feedback. |
| 16 |
|
| 17 |
|
| 18 |
To ease the concerns of gentle benchmarkers around the world, we have |
| 19 |
spent at least 4 weeks reverifying the results. We modified lmbench to |
| 20 |
eliminate any effects of |
| 21 |
|
| 22 |
. clock resolution |
| 23 |
. loop overhead |
| 24 |
. timing interface overhead |
| 25 |
|
| 26 |
Our prediction was that that this would not make any difference and our |
| 27 |
prediction was correct. All of the results reported in lmbench 1.x are |
| 28 |
valid except the file reread benchmark which may be 20% optimistic on |
| 29 |
some platforms. |
| 30 |
|
| 31 |
We've spent a great deal of time and energy, for free, at the expense |
| 32 |
of our full time jobs, to address the issues raised by hbench. We feel |
| 33 |
that we were needlessly forced into a lose/lose situation of arguing |
| 34 |
with a fellow researcher. We intend no disrespect towards their work, |
| 35 |
but did not feel that it was appropriate for what we see as incorrect |
| 36 |
and misleading claims to go unanswered. |
| 37 |
|
| 38 |
We wish to move on to the more interesting and fruitful work of extending |
| 39 |
lmbench in substantial ways. |
| 40 |
|
| 41 |
Larry McVoy & Carl Staelin, June 1997 |
| 42 |
|
| 43 |
-------------------------------------------------------------------------- |
| 44 |
|
| 45 |
Detailed responses to their claims: |
| 46 |
|
| 47 |
Claim 1: |
| 48 |
|
| 49 |
"it did not have the statistical rigor and self-consistency |
| 50 |
needed for detailed architectural studies" |
| 51 |
|
| 52 |
Reply: |
| 53 |
|
| 54 |
This is an unsubstantiated claim. There are no numbers which back |
| 55 |
up this claim. |
| 56 |
|
| 57 |
Claim 2: |
| 58 |
|
| 59 |
"with a reasonable compiler, the test designed to read and touch |
| 60 |
data from the file system buffer cache never actually touched |
| 61 |
the data" |
| 62 |
|
| 63 |
Reply: |
| 64 |
|
| 65 |
Yes, this was a bug in lmbench 1.0. It has been fixed. |
| 66 |
On platforms such as a 120 Mhz Pentium, we see change of a 20% |
| 67 |
in the results, i.e., without the bug fix it is about 20% faster. |
| 68 |
|
| 69 |
Claim 3: |
| 70 |
|
| 71 |
This is a multi part claim: |
| 72 |
|
| 73 |
a) gettimeofday() is too coarse. |
| 74 |
|
| 75 |
Reply: |
| 76 |
|
| 77 |
The implication is that there are number of benchmarks in |
| 78 |
lmbench that finish in less time than the clock resolution |
| 79 |
with correspondingly incorrect results. There is exactly one |
| 80 |
benchmark, TCP connection latency, where this is true and that |
| 81 |
is by design, not by mistake. All other tests run long enough |
| 82 |
to overcome 10ms clocks (most modern clocks are microsecond |
| 83 |
resolution). |
| 84 |
|
| 85 |
Seltzer/Brown point out that lmbench 1.x couldn't accurately |
| 86 |
measure the L1/L2 cache bandwidths. lmbench 1.x didn't attempt |
| 87 |
to report L1/L2 cache bandwidths so it would seem a little |
| 88 |
unreasonable to imply inaccuracy in something the benchmark |
| 89 |
didn't measure. It's not hard to get this right by the way, we |
| 90 |
do so handily in lmbench 2.0. |
| 91 |
|
| 92 |
|
| 93 |
b) TCP connection latency is reported as 0 on the DEC Alpha. |
| 94 |
|
| 95 |
Reply: |
| 96 |
|
| 97 |
We could have easily run the TCP latency connection benchmark in |
| 98 |
a loop long enough to overcome the clock resolution. We were, |
| 99 |
and are, well aware of the problem on DEC Alpha boxes. We run |
| 100 |
only a few interations of this benchmark because the benchmark |
| 101 |
causes a large number of sockets to get stuck in TIME_WAIT, |
| 102 |
part of the TCP shutdown protocol. Almost all protocol stacks |
| 103 |
degrade somewhat in performance when there are large numbers of |
| 104 |
old sockets in their queues. We felt that showing the degraded |
| 105 |
performance was not representative of what users would see. |
| 106 |
So we run only for a small number (about 1000) interations and |
| 107 |
report the result. We would not consider changing the benchmark |
| 108 |
the correct answer - DEC needs to fix their clocks if they wish |
| 109 |
to see accurate results for this test. |
| 110 |
|
| 111 |
We would welcome a portable solution to this problem. Reading |
| 112 |
hardware specific cycle counters is not portable. |
| 113 |
|
| 114 |
Claim 4: |
| 115 |
|
| 116 |
"lmbench [..] was inconsistent in its statistical treatment of |
| 117 |
the data" |
| 118 |
... |
| 119 |
"The most-used statistical policy in lmbench is to take the |
| 120 |
minimum of a few repetitions of the measurement" |
| 121 |
|
| 122 |
Reply: |
| 123 |
|
| 124 |
Both of these claims are false, as can be seen by a quick inspection |
| 125 |
of the code. The most commonly used timing method (16/19 tests |
| 126 |
use this) is |
| 127 |
|
| 128 |
start_timing |
| 129 |
do the test N times |
| 130 |
stop_timing |
| 131 |
report results in terms of duration / N |
| 132 |
|
| 133 |
In fact, the /only/ case where a minimum is used is in the |
| 134 |
context switch test. |
| 135 |
|
| 136 |
The claim goes on to try and say that taking the minimum causes |
| 137 |
incorrect results in the case of the context switch test. |
| 138 |
Another unsupportable claim, one that shows a clear lack of |
| 139 |
understanding of the context switch test. The real issue is cache |
| 140 |
conflicts due to page placement in the cache. Page placement is |
| 141 |
something not under our control, it is under the control of the |
| 142 |
operating system. We did not, and do not, subscribe to the theory |
| 143 |
that one should use better ``statistical methods'' to eliminate |
| 144 |
the variance in the context switch benchmark. The variance is |
| 145 |
what actually happened and happens to real applications. |
| 146 |
|
| 147 |
The authors also claim "if the virtually-contiguous pages of |
| 148 |
the buffer are randomly assigned to physical addresses, as they |
| 149 |
are in many systems, ... then there is a good probability that |
| 150 |
pages of the buffer will conflict in the cache". |
| 151 |
|
| 152 |
We agree with the second part but heartily disagree with |
| 153 |
the first. It's true that NetBSD doesn't solve this problem. |
| 154 |
It doesn't follow that others don't. Any vendor supplied |
| 155 |
operating system that didn't do this on a direct mapped L2 |
| 156 |
cache would suffer dramatically compared to it's competition. |
| 157 |
We know for a fact that Solaris, IRIX, and HPUX do this. |
| 158 |
|
| 159 |
A final claim is that they produced a modified version of the |
| 160 |
context switch benchmark that does not have the variance of |
| 161 |
the lmbench version. We could not support this. We ran that |
| 162 |
benchmark on an SGI MP and saw the same variance as the original |
| 163 |
benchmark. |
| 164 |
|
| 165 |
Claim 5: |
| 166 |
|
| 167 |
"The lmbench bandwidth tests use inconsistent methods of accessing |
| 168 |
memory, making it hard to directly compare the results of, say |
| 169 |
memory read bandwidth with memory write bandwidth, or file reread |
| 170 |
bandwidth with memory copy bandwidth" |
| 171 |
... |
| 172 |
"On the Alpha processor, memory read bandwidth via array indexing |
| 173 |
is 26% faster than via pointer indirection; the Pentium Pro is |
| 174 |
67% faster when reading with array indexing, and an unpipelined |
| 175 |
i386 is about 10% slower when writing with pointer indirection" |
| 176 |
|
| 177 |
Reply: |
| 178 |
In reading that, it would appear that they are suggesting that |
| 179 |
their numbers are up to 67% different than the lmbench numbers. |
| 180 |
We can only assume that this was delibrately misleading. |
| 181 |
Our results are identical to theirs. How can this be? |
| 182 |
|
| 183 |
. We used array indexing for reads, so did they. |
| 184 |
They /implied/ that we did it differently, when in fact |
| 185 |
we use exactly the same technique. They get about |
| 186 |
87MB/sec on reads on a P6, so do we. We challenge |
| 187 |
the authors to demonstrate the implied 67% difference |
| 188 |
between their numbers and ours. In fact, we challenge |
| 189 |
them to demonstrate a 1% difference. |
| 190 |
|
| 191 |
. We use pointers for writes exactly because we wanted |
| 192 |
comparable numbers. The read case is a load and |
| 193 |
an integer add per word. If we used array indexing |
| 194 |
for the stores, it would be only a store per word. |
| 195 |
On older systems, the stores can appear to go faster |
| 196 |
because the load/add is slower than a single store. |
| 197 |
|
| 198 |
While the authors did their best to confuse the issue, the |
| 199 |
results speak for themselves. We coded up the write benchmark |
| 200 |
our way and their way. Results for a Intel P6: |
| 201 |
|
| 202 |
pointer array difference |
| 203 |
L1 $ 587 710 18% |
| 204 |
L2 $ 414 398 4% |
| 205 |
memory 53 53 0% |
| 206 |
|
| 207 |
|
| 208 |
Claim 5a: |
| 209 |
The harmonic mean stuff. |
| 210 |
|
| 211 |
Reply: |
| 212 |
They just don't understand modern architectures. The harmonic mean |
| 213 |
theory is fine if and only if the process can't do two things at |
| 214 |
once. Many modern processors can indeed do more than one thing at |
| 215 |
once, the concept is known as super scalar, and can and does include |
| 216 |
load/store units. If the processor supports both outstanding loads |
| 217 |
and outstanding stores, the harmonic mean theory fails. |
| 218 |
|
| 219 |
Claim 6: |
| 220 |
|
| 221 |
"we modified the memory copy bandwidth to use the same size |
| 222 |
data types as the memory read and write benchmark (which use the |
| 223 |
machine's native word size); originally, on 32-bit machines, the |
| 224 |
copy benchmark used 64-bit types whereas the memory read/write |
| 225 |
bandwidth tests used 32- bit types" |
| 226 |
|
| 227 |
Reply: |
| 228 |
|
| 229 |
The change was to use 32 bit types for bcopy. On even relatively |
| 230 |
modern systems, such as a 586, this change has no impact - the |
| 231 |
benchmark is bound by memory sub systems. On older systems, the |
| 232 |
use of multiple load/store instructions, as required for the smaller |
| 233 |
types, resulted in lower results than the meory system could produce. |
| 234 |
|
| 235 |
The processor cycles required actually slow down the results. This |
| 236 |
is still true today for in cache numbers. For example, an R10K |
| 237 |
shows L1 cache bandwidths of 750MB/sec and 377MB/sec with 64 bit |
| 238 |
vs 32 bit loads. It was our intention to show the larger number and |
| 239 |
that requires the larger types. |
| 240 |
|
| 241 |
Perhaps because the authors have not ported their benchmark to |
| 242 |
non-Intel platforms, they have not noticed this. The Intel |
| 243 |
platform does not have native 64 bit types so it does two |
| 244 |
load/stores for what C says is a 64 bit type. Just because it |
| 245 |
makes no difference on Intel does not mean it makes no difference. |