Debian Codegraph @

BENCH

raw
1
2 There were no structured benchmarks of icecream, so I started some. I'm benchmarking 5 runs
3 each, throwing away the worst and the best run, and then averaging the rest of the 3.
4 There are two machines in the cluster, both single cpu and both about the same speed (1.7GHz
5 Pentium M). Normally the tests are done via otherwise idle WLAN (54MBit). WLAN has a bad
6 latency and very bad throughput, which gives icecream a hard time, and should be compareable
7 to a loaded cabled network environment. For comparison, I repeated some tests via 100MBit
8 cabled LAN.
9
10 I'm testing linux 2.6.16 (defconfig), which is a set of small C files with sometimes larger
11 C files inbetween. its a tough benchmark because compiling C is rather quick, so remote
12 overhead has to be low.
13
14 No icecream:
15
16 make -j1: 315s
17 make -j2: 322s
18 make -j3: 324s
19 make -j10: 334s
20
21 result: without icecream, starting more compilers than CPUs available is a bad idea.
22
23
24 icecream wrapper, no scheduler.
25
26 make -j1: 323s
27 make -j10: 340s
28
29 result: Overhead of just using icecream without cluster is neglectible (2%) in all cases.
30
31 dirk's no-latency icecream: remote daemon with -m 1:
32
33 make -j1: 418s
34 make -j2: 397s
35 make -j3: 263s
36 make -j10: 230s
37 make -j10/100: 203s
38 make -j100: 231s
39 make -j10/100: 202s
40
41 result: Enabling icecream without parallel compilation is a horrible mistake. icecream
42 must be tuned to detect and compensate this situation better. Maximum performance
43 improvement of icecream is 27% (LAN: 36%) out of the theoretical 50%.
44
45 ======================================================================
46
47 Qt 3.3.6's src/ subdirectory. This is a bunch of medium and large C++ files. It
48 gives icecream good opportunities because compilation time is comparatively low
49 and the preprocessed files are not too large (low STL usage).
50
51
52 No icecream:
53
54 make -j1: 368s
55 make -j3: 363s
56
57 result: apparently there is a small win by starting more compilers than CPUs in parallel.
58 Perhaps the I/O overhead is not neglectible like in the linux kernel case.
59
60 dirk's no-latency icecream, remote daemon with -m 2:
61
62 make -j1: 572s
63 make -j3: 273s
64 make -j10: 269s
65 make -j10/100: 239s
66 make -j100: 282s
67
68 result: again, not compiling in parallel is a deadly sin. trying to overload the cluster
69 with very high parallelism as well. Maximum performance improvement is again 27% and
70 36% for LAN. That these numbers compare equally with the Linux kernel case is astonishing
71 and needs explanation.
72
73 Now, to make sure that the no-latency patch hasn't regressed anything, we're comparing with
74 stock 0.7 (which already has some performance improvements over 0.6):
75
76
77 make -j1: 569s
78 make -j10: 349s
79 make -j10/100: 253s
80 make -j100/100: 278s
81
82 It is remarkable, that 0.7 does not provide much advantage over compiling locally (6% speedup)
83 in a WLAN, while providing the expected 36% speedup for LAN. This proves that no-latency
84 provides significant wins for unstable/bad network connections and does not regress
85 performance for good networking setups. The overall 20% improvement is not bad at all.
86
87 2006-06-16 make-it-cool-branch:
88
89 make -j10/100: 244s
90 make -j1: 376s
91
92 result: correcting process accounting for local jobs makes -j1 fast again (just 2% overhead)
93
94 icecream, always compile remote even though host is not faster:
95
96 make -j10: 538s
97 make -j10/sched: 389s
98
99 As we can see, the scheduler improves performance by 38%.
100
101
102 ======================================================================
103
104 New performance experiments.
105
106 New baseline:
107
108 make-it-cool, both with -m 1, make -j5
109
110 make -j5 : 382s
111 make -j5 : 354s
112 make -j5 : 336s
113 make -j5 : 355s
114
115 remote with -m 2
116
117 make -j5 : 333s
118 make -j5 : 299s
119 make -j5 : 307s
120
121 remote with -m 2, preload scheduler:
122
123 make -j5 : 303s
124 make -j5 : 287s
125 make -j5 : 291s
126
127 remote with -m 1, preload scheduler:
128
129 make -j5 : 287s
130 make -j5 : 288s
131 make -j5 : 289s
132
133 remote with -m 1, preload scheduler, optimized return:
134 make -j5 : 288s
135 make -j5 : 289s
136 make -j5 : 288s
137
138 remote with -m 2, preload scheduler, optimized return:
139 make -j5 : 279s
140 make -j5 : 281s
141 make -j5 : 278s
142
143 As we can see, over-booking the remote slave improves performance by 13%.
144 As the CPU hardly gets faster, it means that we're reducing idle wait time
145 this way.
146
147 One experiment was to pre-load jobs on the remote slave. This means even
148 though all compile slots are filled, it gets one (exactly one) job assigned.
149 The daemon itself will start compilation as soon as it gets a free slot,
150 reducing both client<->CS and CS<->scheduler roundtrip. Overall, it gave an
151 impressive 4% speedup.
152
153 A lot of time is however spent on writing back the compiled object file
154 to the client, and this is dominated by network saturation and latency
155 and not by CPU usage. The obvious solution is to notify the scheduler
156 about a free slot as soon as compilation (but not write-back) has
157 finished. With remote over-booking this resulted in another 3% speedup,
158 while no improvements could be measured in the -m 1 case. Given that
159 it significantly reduced code complexity in the daemon, it should still be
160 an improvement (reduced code size by almost 8%!).
161
162 ======================================================================
163
164 Load detection tunings.
165
166 The biggest problem with accurate load detection is that it is impossible
167 to find out how much cpu time a iceccd child is currently using. All you
168 can get is how much time it used overall, but only when it exited.
169 Which gives you a lot of cpu-usage peaks sprinkled over time, and you have
170 to somehow average that out in a meaningful way.
171
172 Actually, the Linux kernel tracks cpu time, and you can read it in
173 /proc/<pid>/stat for any child. unfortunately it is converted to seconds
174 in there, so resolution isn't much more accurate.
175
176 Watching the old load detector, it became obvious that it once in a
177 while jumps to 1000 because of own jobs eating 100% cpu, but not finishing
178 within the measure timeframe, which causes the host to be blacklisted
179 by the scheduler, even though nothing is wrong with it.
180
181 There are two solutions:
182
183 - trying to get a more accurate usage over time
184
185 - only track usage whenever it is accurate, e.g. a child exited.
186
187 As the second possibility has problems with multi-cpu environments (you'd
188 have to wait for all jobs to finish before doing another idleness sample,
189 which probably reduces throughput considerably), first one was chosen.
190
191 Simplest idea: assume that overall system nice load is at least partially
192 caused by our own childs.
193
194 remote with -m 2:
195 make -j5 : 272s
196 make -j5 : 274s
197 make -j5 : 271s
198
199 Hmm.. 2% win.
200
201 ============================================================================
202
203 New baseline: 0.8.0:
204
205 remote with -m 1:
206 make -j5 : 257s
207
208 without compression:
209
210 make -j5: 442s
211