| 1 |
|
| 2 |
There were no structured benchmarks of icecream, so I started some. I'm benchmarking 5 runs |
| 3 |
each, throwing away the worst and the best run, and then averaging the rest of the 3. |
| 4 |
There are two machines in the cluster, both single cpu and both about the same speed (1.7GHz |
| 5 |
Pentium M). Normally the tests are done via otherwise idle WLAN (54MBit). WLAN has a bad |
| 6 |
latency and very bad throughput, which gives icecream a hard time, and should be compareable |
| 7 |
to a loaded cabled network environment. For comparison, I repeated some tests via 100MBit |
| 8 |
cabled LAN. |
| 9 |
|
| 10 |
I'm testing linux 2.6.16 (defconfig), which is a set of small C files with sometimes larger |
| 11 |
C files inbetween. its a tough benchmark because compiling C is rather quick, so remote |
| 12 |
overhead has to be low. |
| 13 |
|
| 14 |
No icecream: |
| 15 |
|
| 16 |
make -j1: 315s |
| 17 |
make -j2: 322s |
| 18 |
make -j3: 324s |
| 19 |
make -j10: 334s |
| 20 |
|
| 21 |
result: without icecream, starting more compilers than CPUs available is a bad idea. |
| 22 |
|
| 23 |
|
| 24 |
icecream wrapper, no scheduler. |
| 25 |
|
| 26 |
make -j1: 323s |
| 27 |
make -j10: 340s |
| 28 |
|
| 29 |
result: Overhead of just using icecream without cluster is neglectible (2%) in all cases. |
| 30 |
|
| 31 |
dirk's no-latency icecream: remote daemon with -m 1: |
| 32 |
|
| 33 |
make -j1: 418s |
| 34 |
make -j2: 397s |
| 35 |
make -j3: 263s |
| 36 |
make -j10: 230s |
| 37 |
make -j10/100: 203s |
| 38 |
make -j100: 231s |
| 39 |
make -j10/100: 202s |
| 40 |
|
| 41 |
result: Enabling icecream without parallel compilation is a horrible mistake. icecream |
| 42 |
must be tuned to detect and compensate this situation better. Maximum performance |
| 43 |
improvement of icecream is 27% (LAN: 36%) out of the theoretical 50%. |
| 44 |
|
| 45 |
====================================================================== |
| 46 |
|
| 47 |
Qt 3.3.6's src/ subdirectory. This is a bunch of medium and large C++ files. It |
| 48 |
gives icecream good opportunities because compilation time is comparatively low |
| 49 |
and the preprocessed files are not too large (low STL usage). |
| 50 |
|
| 51 |
|
| 52 |
No icecream: |
| 53 |
|
| 54 |
make -j1: 368s |
| 55 |
make -j3: 363s |
| 56 |
|
| 57 |
result: apparently there is a small win by starting more compilers than CPUs in parallel. |
| 58 |
Perhaps the I/O overhead is not neglectible like in the linux kernel case. |
| 59 |
|
| 60 |
dirk's no-latency icecream, remote daemon with -m 2: |
| 61 |
|
| 62 |
make -j1: 572s |
| 63 |
make -j3: 273s |
| 64 |
make -j10: 269s |
| 65 |
make -j10/100: 239s |
| 66 |
make -j100: 282s |
| 67 |
|
| 68 |
result: again, not compiling in parallel is a deadly sin. trying to overload the cluster |
| 69 |
with very high parallelism as well. Maximum performance improvement is again 27% and |
| 70 |
36% for LAN. That these numbers compare equally with the Linux kernel case is astonishing |
| 71 |
and needs explanation. |
| 72 |
|
| 73 |
Now, to make sure that the no-latency patch hasn't regressed anything, we're comparing with |
| 74 |
stock 0.7 (which already has some performance improvements over 0.6): |
| 75 |
|
| 76 |
|
| 77 |
make -j1: 569s |
| 78 |
make -j10: 349s |
| 79 |
make -j10/100: 253s |
| 80 |
make -j100/100: 278s |
| 81 |
|
| 82 |
It is remarkable, that 0.7 does not provide much advantage over compiling locally (6% speedup) |
| 83 |
in a WLAN, while providing the expected 36% speedup for LAN. This proves that no-latency |
| 84 |
provides significant wins for unstable/bad network connections and does not regress |
| 85 |
performance for good networking setups. The overall 20% improvement is not bad at all. |
| 86 |
|
| 87 |
2006-06-16 make-it-cool-branch: |
| 88 |
|
| 89 |
make -j10/100: 244s |
| 90 |
make -j1: 376s |
| 91 |
|
| 92 |
result: correcting process accounting for local jobs makes -j1 fast again (just 2% overhead) |
| 93 |
|
| 94 |
icecream, always compile remote even though host is not faster: |
| 95 |
|
| 96 |
make -j10: 538s |
| 97 |
make -j10/sched: 389s |
| 98 |
|
| 99 |
As we can see, the scheduler improves performance by 38%. |
| 100 |
|
| 101 |
|
| 102 |
====================================================================== |
| 103 |
|
| 104 |
New performance experiments. |
| 105 |
|
| 106 |
New baseline: |
| 107 |
|
| 108 |
make-it-cool, both with -m 1, make -j5 |
| 109 |
|
| 110 |
make -j5 : 382s |
| 111 |
make -j5 : 354s |
| 112 |
make -j5 : 336s |
| 113 |
make -j5 : 355s |
| 114 |
|
| 115 |
remote with -m 2 |
| 116 |
|
| 117 |
make -j5 : 333s |
| 118 |
make -j5 : 299s |
| 119 |
make -j5 : 307s |
| 120 |
|
| 121 |
remote with -m 2, preload scheduler: |
| 122 |
|
| 123 |
make -j5 : 303s |
| 124 |
make -j5 : 287s |
| 125 |
make -j5 : 291s |
| 126 |
|
| 127 |
remote with -m 1, preload scheduler: |
| 128 |
|
| 129 |
make -j5 : 287s |
| 130 |
make -j5 : 288s |
| 131 |
make -j5 : 289s |
| 132 |
|
| 133 |
remote with -m 1, preload scheduler, optimized return: |
| 134 |
make -j5 : 288s |
| 135 |
make -j5 : 289s |
| 136 |
make -j5 : 288s |
| 137 |
|
| 138 |
remote with -m 2, preload scheduler, optimized return: |
| 139 |
make -j5 : 279s |
| 140 |
make -j5 : 281s |
| 141 |
make -j5 : 278s |
| 142 |
|
| 143 |
As we can see, over-booking the remote slave improves performance by 13%. |
| 144 |
As the CPU hardly gets faster, it means that we're reducing idle wait time |
| 145 |
this way. |
| 146 |
|
| 147 |
One experiment was to pre-load jobs on the remote slave. This means even |
| 148 |
though all compile slots are filled, it gets one (exactly one) job assigned. |
| 149 |
The daemon itself will start compilation as soon as it gets a free slot, |
| 150 |
reducing both client<->CS and CS<->scheduler roundtrip. Overall, it gave an |
| 151 |
impressive 4% speedup. |
| 152 |
|
| 153 |
A lot of time is however spent on writing back the compiled object file |
| 154 |
to the client, and this is dominated by network saturation and latency |
| 155 |
and not by CPU usage. The obvious solution is to notify the scheduler |
| 156 |
about a free slot as soon as compilation (but not write-back) has |
| 157 |
finished. With remote over-booking this resulted in another 3% speedup, |
| 158 |
while no improvements could be measured in the -m 1 case. Given that |
| 159 |
it significantly reduced code complexity in the daemon, it should still be |
| 160 |
an improvement (reduced code size by almost 8%!). |
| 161 |
|
| 162 |
====================================================================== |
| 163 |
|
| 164 |
Load detection tunings. |
| 165 |
|
| 166 |
The biggest problem with accurate load detection is that it is impossible |
| 167 |
to find out how much cpu time a iceccd child is currently using. All you |
| 168 |
can get is how much time it used overall, but only when it exited. |
| 169 |
Which gives you a lot of cpu-usage peaks sprinkled over time, and you have |
| 170 |
to somehow average that out in a meaningful way. |
| 171 |
|
| 172 |
Actually, the Linux kernel tracks cpu time, and you can read it in |
| 173 |
/proc/<pid>/stat for any child. unfortunately it is converted to seconds |
| 174 |
in there, so resolution isn't much more accurate. |
| 175 |
|
| 176 |
Watching the old load detector, it became obvious that it once in a |
| 177 |
while jumps to 1000 because of own jobs eating 100% cpu, but not finishing |
| 178 |
within the measure timeframe, which causes the host to be blacklisted |
| 179 |
by the scheduler, even though nothing is wrong with it. |
| 180 |
|
| 181 |
There are two solutions: |
| 182 |
|
| 183 |
- trying to get a more accurate usage over time |
| 184 |
|
| 185 |
- only track usage whenever it is accurate, e.g. a child exited. |
| 186 |
|
| 187 |
As the second possibility has problems with multi-cpu environments (you'd |
| 188 |
have to wait for all jobs to finish before doing another idleness sample, |
| 189 |
which probably reduces throughput considerably), first one was chosen. |
| 190 |
|
| 191 |
Simplest idea: assume that overall system nice load is at least partially |
| 192 |
caused by our own childs. |
| 193 |
|
| 194 |
remote with -m 2: |
| 195 |
make -j5 : 272s |
| 196 |
make -j5 : 274s |
| 197 |
make -j5 : 271s |
| 198 |
|
| 199 |
Hmm.. 2% win. |
| 200 |
|
| 201 |
============================================================================ |
| 202 |
|
| 203 |
New baseline: 0.8.0: |
| 204 |
|
| 205 |
remote with -m 1: |
| 206 |
make -j5 : 257s |
| 207 |
|
| 208 |
without compression: |
| 209 |
|
| 210 |
make -j5: 442s |
| 211 |
|