Debian Codegraph @

hbench-REBUTTAL

raw
1 In June of 1997, Margo Seltzer and Aaron Brown published a paper in
2 Sigmetrics called "Operating System Benchmarking in the Wake of Lmbench:
3 A Case Study of the Performance of NetBSD on the Intel x86 Architecture".
4
5
6 This papers claims to have found flaws in the original lmbench work.
7 With the exception of one bug, which we have of course fixed, we find
8 the claims inaccurate, misleading, and petty. We don't understand
9 what appears to be a pointless attack on something that has obviously
10 helped many researchers and industry people alike. lmbench was warmly
11 received and is widely used and referenced. We stand firmly behind the
12 work and results of the original benchmark. We continue to improve and
13 extend the benchmark. Our focus continues to be on providing a useful,
14 accurate, portable benchmark suite that is widely used. As always, we
15 welcome constructive feedback.
16
17
18 To ease the concerns of gentle benchmarkers around the world, we have
19 spent at least 4 weeks reverifying the results. We modified lmbench to
20 eliminate any effects of
21
22 . clock resolution
23 . loop overhead
24 . timing interface overhead
25
26 Our prediction was that that this would not make any difference and our
27 prediction was correct. All of the results reported in lmbench 1.x are
28 valid except the file reread benchmark which may be 20% optimistic on
29 some platforms.
30
31 We've spent a great deal of time and energy, for free, at the expense
32 of our full time jobs, to address the issues raised by hbench. We feel
33 that we were needlessly forced into a lose/lose situation of arguing
34 with a fellow researcher. We intend no disrespect towards their work,
35 but did not feel that it was appropriate for what we see as incorrect
36 and misleading claims to go unanswered.
37
38 We wish to move on to the more interesting and fruitful work of extending
39 lmbench in substantial ways.
40
41 Larry McVoy & Carl Staelin, June 1997
42
43 --------------------------------------------------------------------------
44
45 Detailed responses to their claims:
46
47 Claim 1:
48
49 "it did not have the statistical rigor and self-consistency
50 needed for detailed architectural studies"
51
52 Reply:
53
54 This is an unsubstantiated claim. There are no numbers which back
55 up this claim.
56
57 Claim 2:
58
59 "with a reasonable compiler, the test designed to read and touch
60 data from the file system buffer cache never actually touched
61 the data"
62
63 Reply:
64
65 Yes, this was a bug in lmbench 1.0. It has been fixed.
66 On platforms such as a 120 Mhz Pentium, we see change of a 20%
67 in the results, i.e., without the bug fix it is about 20% faster.
68
69 Claim 3:
70
71 This is a multi part claim:
72
73 a) gettimeofday() is too coarse.
74
75 Reply:
76
77 The implication is that there are number of benchmarks in
78 lmbench that finish in less time than the clock resolution
79 with correspondingly incorrect results. There is exactly one
80 benchmark, TCP connection latency, where this is true and that
81 is by design, not by mistake. All other tests run long enough
82 to overcome 10ms clocks (most modern clocks are microsecond
83 resolution).
84
85 Seltzer/Brown point out that lmbench 1.x couldn't accurately
86 measure the L1/L2 cache bandwidths. lmbench 1.x didn't attempt
87 to report L1/L2 cache bandwidths so it would seem a little
88 unreasonable to imply inaccuracy in something the benchmark
89 didn't measure. It's not hard to get this right by the way, we
90 do so handily in lmbench 2.0.
91
92
93 b) TCP connection latency is reported as 0 on the DEC Alpha.
94
95 Reply:
96
97 We could have easily run the TCP latency connection benchmark in
98 a loop long enough to overcome the clock resolution. We were,
99 and are, well aware of the problem on DEC Alpha boxes. We run
100 only a few interations of this benchmark because the benchmark
101 causes a large number of sockets to get stuck in TIME_WAIT,
102 part of the TCP shutdown protocol. Almost all protocol stacks
103 degrade somewhat in performance when there are large numbers of
104 old sockets in their queues. We felt that showing the degraded
105 performance was not representative of what users would see.
106 So we run only for a small number (about 1000) interations and
107 report the result. We would not consider changing the benchmark
108 the correct answer - DEC needs to fix their clocks if they wish
109 to see accurate results for this test.
110
111 We would welcome a portable solution to this problem. Reading
112 hardware specific cycle counters is not portable.
113
114 Claim 4:
115
116 "lmbench [..] was inconsistent in its statistical treatment of
117 the data"
118 ...
119 "The most-used statistical policy in lmbench is to take the
120 minimum of a few repetitions of the measurement"
121
122 Reply:
123
124 Both of these claims are false, as can be seen by a quick inspection
125 of the code. The most commonly used timing method (16/19 tests
126 use this) is
127
128 start_timing
129 do the test N times
130 stop_timing
131 report results in terms of duration / N
132
133 In fact, the /only/ case where a minimum is used is in the
134 context switch test.
135
136 The claim goes on to try and say that taking the minimum causes
137 incorrect results in the case of the context switch test.
138 Another unsupportable claim, one that shows a clear lack of
139 understanding of the context switch test. The real issue is cache
140 conflicts due to page placement in the cache. Page placement is
141 something not under our control, it is under the control of the
142 operating system. We did not, and do not, subscribe to the theory
143 that one should use better ``statistical methods'' to eliminate
144 the variance in the context switch benchmark. The variance is
145 what actually happened and happens to real applications.
146
147 The authors also claim "if the virtually-contiguous pages of
148 the buffer are randomly assigned to physical addresses, as they
149 are in many systems, ... then there is a good probability that
150 pages of the buffer will conflict in the cache".
151
152 We agree with the second part but heartily disagree with
153 the first. It's true that NetBSD doesn't solve this problem.
154 It doesn't follow that others don't. Any vendor supplied
155 operating system that didn't do this on a direct mapped L2
156 cache would suffer dramatically compared to it's competition.
157 We know for a fact that Solaris, IRIX, and HPUX do this.
158
159 A final claim is that they produced a modified version of the
160 context switch benchmark that does not have the variance of
161 the lmbench version. We could not support this. We ran that
162 benchmark on an SGI MP and saw the same variance as the original
163 benchmark.
164
165 Claim 5:
166
167 "The lmbench bandwidth tests use inconsistent methods of accessing
168 memory, making it hard to directly compare the results of, say
169 memory read bandwidth with memory write bandwidth, or file reread
170 bandwidth with memory copy bandwidth"
171 ...
172 "On the Alpha processor, memory read bandwidth via array indexing
173 is 26% faster than via pointer indirection; the Pentium Pro is
174 67% faster when reading with array indexing, and an unpipelined
175 i386 is about 10% slower when writing with pointer indirection"
176
177 Reply:
178 In reading that, it would appear that they are suggesting that
179 their numbers are up to 67% different than the lmbench numbers.
180 We can only assume that this was delibrately misleading.
181 Our results are identical to theirs. How can this be?
182
183 . We used array indexing for reads, so did they.
184 They /implied/ that we did it differently, when in fact
185 we use exactly the same technique. They get about
186 87MB/sec on reads on a P6, so do we. We challenge
187 the authors to demonstrate the implied 67% difference
188 between their numbers and ours. In fact, we challenge
189 them to demonstrate a 1% difference.
190
191 . We use pointers for writes exactly because we wanted
192 comparable numbers. The read case is a load and
193 an integer add per word. If we used array indexing
194 for the stores, it would be only a store per word.
195 On older systems, the stores can appear to go faster
196 because the load/add is slower than a single store.
197
198 While the authors did their best to confuse the issue, the
199 results speak for themselves. We coded up the write benchmark
200 our way and their way. Results for a Intel P6:
201
202 pointer array difference
203 L1 $ 587 710 18%
204 L2 $ 414 398 4%
205 memory 53 53 0%
206
207
208 Claim 5a:
209 The harmonic mean stuff.
210
211 Reply:
212 They just don't understand modern architectures. The harmonic mean
213 theory is fine if and only if the process can't do two things at
214 once. Many modern processors can indeed do more than one thing at
215 once, the concept is known as super scalar, and can and does include
216 load/store units. If the processor supports both outstanding loads
217 and outstanding stores, the harmonic mean theory fails.
218
219 Claim 6:
220
221 "we modified the memory copy bandwidth to use the same size
222 data types as the memory read and write benchmark (which use the
223 machine's native word size); originally, on 32-bit machines, the
224 copy benchmark used 64-bit types whereas the memory read/write
225 bandwidth tests used 32- bit types"
226
227 Reply:
228
229 The change was to use 32 bit types for bcopy. On even relatively
230 modern systems, such as a 586, this change has no impact - the
231 benchmark is bound by memory sub systems. On older systems, the
232 use of multiple load/store instructions, as required for the smaller
233 types, resulted in lower results than the meory system could produce.
234
235 The processor cycles required actually slow down the results. This
236 is still true today for in cache numbers. For example, an R10K
237 shows L1 cache bandwidths of 750MB/sec and 377MB/sec with 64 bit
238 vs 32 bit loads. It was our intention to show the larger number and
239 that requires the larger types.
240
241 Perhaps because the authors have not ported their benchmark to
242 non-Intel platforms, they have not noticed this. The Intel
243 platform does not have native 64 bit types so it does two
244 load/stores for what C says is a 64 bit type. Just because it
245 makes no difference on Intel does not mean it makes no difference.