| 1 |
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" |
| 2 |
"http://www.w3.org/TR/REC-html40/loose.dtd"> |
| 3 |
<html> |
| 4 |
<head><title>benchmark results for distcc</title> |
| 5 |
<link rel="SHORTCUT ICON" href="favicon.ico"> |
| 6 |
<link type="text/css" rel="stylesheet" href="distcc-green.css"> |
| 7 |
<link type="text/html" rel="top" href="index.html"> |
| 8 |
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"></head> |
| 9 |
<body vlink="#003300" text="#000000" link="#003300" bgcolor="#ffffff" alink="#336633"><p><table width="100%"><tr><td valign="top"><div class="navbar"><ul> |
| 10 |
<li><a href="index.html">distcc</a></li> |
| 11 |
<li><a href="http://code.google.com/p/distcc/source/browse/trunk">source</a> |
| 12 |
<li><a href="http://code.google.com/p/distcc/downloads/list">binaries</a></li> |
| 13 |
<li><a href="http://distcc.googlecode.com/svn/trunk/NEWS">news</a></li> |
| 14 |
<li><a href="scenarios.html">scenarios</a></li> |
| 15 |
<li><a href="results.html">testimonials</a></li> |
| 16 |
<li><a href="benchmark.html"><b>benchmarks</b></a></li> |
| 17 |
<li><a href="security.html">security</a></li> |
| 18 |
<li><a href="distcc-lca-2004.html">whitepaper</a></li> |
| 19 |
<li><a href="http://distcc.googlecode.com/svn/trunk/doc/web/man/">Man pages</a></li> |
| 20 |
<li><a href="faq.html">FAQ</a></li> |
| 21 |
|
| 22 |
<p> |
| 23 |
<li><a href="http://code.google.com/p/distcc/issues/list">report a bug</a></li> |
| 24 |
<li><a href="https://lists.samba.org/mailman/listinfo/distcc">mailing list</a> |
| 25 |
<ul><li><a href="mailto:[email protected]">post</a></li> |
| 26 |
<li><a href="http://lists.samba.org/archive/distcc/">archive</a></li> |
| 27 |
<li><a href="news://news.gmane.org/gmane.comp.compilers.distcc">newsgroup</a></li></ul> |
| 28 |
</li> |
| 29 |
<li><a href="http://freshmeat.net/projects/distcc">Freshmeat</a> |
| 30 |
<ul><li><a href="http://freshmeat.net/subscribe/28140">be notified</a></li></ul> |
| 31 |
</li> |
| 32 |
|
| 33 |
<p> |
| 34 |
<li>related:</li> |
| 35 |
<ul> |
| 36 |
<li><a title="How to use distcc and Gentoo" href="http://www.gentoo.org/doc/en/distcc.xml">Gentoo</a></li> |
| 37 |
<li><a title="Script to ease building cross-compilers and toolchains" href="http://kegel.com/crosstool/">crosstool</a></li> |
| 38 |
<li><a title="Cache compiler results" href="https://ccache.dev/">ccache</a></li> |
| 39 |
<li><a title="Centralized control over distcc and ccache" href="http://ccontrol.ozlabs.org/">ccontrol</a></li> |
| 40 |
<li><a title="Smart job scheduler" href="http://dmucs.sourceforge.net/">dmucs</a></li> |
| 41 |
</ul> |
| 42 |
<p><p></td><td valign="top"> |
| 43 |
|
| 44 |
<div class="body"><h1 class="title">benchmark results for distcc</h1> |
| 45 |
|
| 46 |
<p>The following table shows the effect of distcc on several |
| 47 |
real-world compilation tasks; in this case, compiling various |
| 48 |
open-source projects. In each case, we compiled in four modes:</p> |
| 49 |
<ol> |
| 50 |
<li> <b>local_01</b>: all compilation done locally |
| 51 |
<li> <b>dist_h38_j40</b>: using distcc with 38 compilation servers (152 |
| 52 |
CPUs), and <code>make -j40</code>, which gives 40 parallel |
| 53 |
compilations. distcc is used in non-pump mode (local preprocessing). |
| 54 |
<li> <b>pump_h38_j40</b>: like <code>dist_h38_j40</code>, but using |
| 55 |
distcc in pump (remote preprocessing) mode. |
| 56 |
<li> <b>pump_h38_j80</b>: like <code>pump_h38_j40</code>, but using |
| 57 |
<code>make -j80</code> instead of <code>make -j40</code>. |
| 58 |
</ol> |
| 59 |
|
| 60 |
<p>The client machine is an 2.8GHz Intel Pentium 4 machine, with 2G of |
| 61 |
memory, running Linux 2.6.18.5 (modified Ubuntu 6.06). The server |
| 62 |
machines were all 4 CPU 2GHz AMD Dual Core Processor 270 machines, |
| 63 |
with 2G of memory, running Linux 2.6.18.5 (modified Ubuntu 6.06).</p> |
| 64 |
|
| 65 |
<p>Each test was run 5 times. We report the average time over the 5 |
| 66 |
runs, as well as the standard deviation. In order to minimize disk |
| 67 |
caching effects on walltime results, we read all the files in each |
| 68 |
project's tarball before building.</p> |
| 69 |
|
| 70 |
|
| 71 |
<table width=100%> |
| 72 |
<tr valign=bottom align=right><th align=left>Project</th><th align=left>mode</th> |
| 73 |
<th>wall<br>time</th><th>std<br>dev</th><th>cpu<br>time</th><th>cpu<br>%</th> |
| 74 |
<tr><td colspan=6><hr></td></tr> |
| 75 |
|
| 76 |
<tr align=right><td align=left>binutils-2.18</td><td align=left>local_01 </td> |
| 77 |
<td>125.8s</td> <td>4.0s</td> <td>102.9s</td> <td>81.9%</td></tr> |
| 78 |
<tr align=right><td align=left>binutils-2.18</td><td align=left>dist_h38_j40</td> |
| 79 |
<td>35.4s</td> <td>1.0s</td> <td>25.6s</td> <td>72.2%</td></tr> |
| 80 |
<tr align=right><td align=left>binutils-2.18</td><td align=left>pump_h38_j40</td> |
| 81 |
<td>28.7s</td> <td>1.5s</td> <td>16.6s</td> <td>57.7%</td></tr> |
| 82 |
<tr align=right><td align=left>binutils-2.18</td><td align=left>pump_h38_j80</td> |
| 83 |
<td>29.8s</td> <td>1.5s</td> <td>16.6s</td> <td>55.9%</td></tr> |
| 84 |
<tr><td colspan=6><hr></td></tr> |
| 85 |
|
| 86 |
<tr align=right><td align=left>glibc-2.6 </td><td align=left>local_01 </td> |
| 87 |
<td>946.4s</td> <td>5.1s</td> <td>568.7s</td> <td>60.1%</td></tr> |
| 88 |
<tr align=right><td align=left>glibc-2.6 </td><td align=left>dist_h38_j40</td> |
| 89 |
<td>534.0s</td> <td>7.0s</td> <td>390.5s</td> <td>74.7%</td></tr> |
| 90 |
<tr align=right><td align=left>glibc-2.6 </td><td align=left>pump_h38_j40</td> |
| 91 |
<td>357.8s</td> <td>6.5s</td> <td>241.6s</td> <td>67.5%</td></tr> |
| 92 |
<tr align=right><td align=left>glibc-2.6 </td><td align=left>pump_h38_j80</td> |
| 93 |
<td>365.4s</td> <td>4.2s</td> <td>243.0s</td> <td>66.6%</td></tr> |
| 94 |
<tr><td colspan=6><hr></td></tr> |
| 95 |
|
| 96 |
<tr align=right><td align=left>hello-2.1.1 </td><td align=left>local_01 </td> |
| 97 |
<td>0.7s</td> <td>0.1s</td> <td>0.4s</td> <td>62.8%</td></tr> |
| 98 |
<tr align=right><td align=left>hello-2.1.1 </td><td align=left>dist_h38_j40</td> |
| 99 |
<td>0.9s</td> <td>0.2s</td> <td>0.3s</td> <td>40.3%</td></tr> |
| 100 |
<tr align=right><td align=left>hello-2.1.1 </td><td align=left>pump_h38_j40</td> |
| 101 |
<td>1.1s</td> <td>0.1s</td> <td>0.4s</td> <td>31.3%</td></tr> |
| 102 |
<tr align=right><td align=left>hello-2.1.1 </td><td align=left>pump_h38_j80</td> |
| 103 |
<td>1.1s</td> <td>0.0s</td> <td>0.4s</td> <td>31.6%</td></tr> |
| 104 |
<tr><td colspan=6><hr></td></tr> |
| 105 |
|
| 106 |
<tr align=right><td align=left>httpd-2.0.43</td><td align=left>local_01 </td> |
| 107 |
<td>139.8s</td> <td>1.4s</td> <td>117.2s</td> <td>83.9%</td></tr> |
| 108 |
<tr align=right><td align=left>httpd-2.0.43</td><td align=left>dist_h38_j40</td> |
| 109 |
<td>85.1s</td> <td>2.1s</td> <td>51.3s</td> <td>60.4%</td></tr> |
| 110 |
<tr align=right><td align=left>httpd-2.0.43</td><td align=left>pump_h38_j40</td> |
| 111 |
<td>80.1s</td> <td>2.0s</td> <td>33.7s</td> <td>42.1%</td></tr> |
| 112 |
<tr align=right><td align=left>httpd-2.0.43</td><td align=left>pump_h38_j80</td> |
| 113 |
<td>81.7s</td> <td>4.1s</td> <td>33.7s</td> <td>41.4%</td></tr> |
| 114 |
<tr><td colspan=6><hr></td></tr> |
| 115 |
|
| 116 |
<tr align=right><td align=left>linux-2.6.25</td><td align=left>local_01 </td> |
| 117 |
<td>818.3s</td> <td>7.9s</td> <td>543.9s</td> <td>66.5%</td></tr> |
| 118 |
<tr align=right><td align=left>linux-2.6.25</td><td align=left>dist_h38_j40</td> |
| 119 |
<td>203.2s</td> <td>1.7s</td> <td>185.3s</td> <td>91.2%</td></tr> |
| 120 |
<tr align=right><td align=left>linux-2.6.25</td><td align=left>pump_h38_j40</td> |
| 121 |
<td>134.9s</td> <td>5.3s</td> <td>96.5s</td> <td>71.6%</td></tr> |
| 122 |
<tr align=right><td align=left>linux-2.6.25</td><td align=left>pump_h38_j80</td> |
| 123 |
<td>135.6s</td> <td>4.3s</td> <td>97.7s</td> <td>72.1%</td></tr> |
| 124 |
<tr><td colspan=6><hr></td></tr> |
| 125 |
|
| 126 |
<tr align=right><td align=left>samba-3.0.20</td><td align=left>local_01 </td> |
| 127 |
<td>314.0s</td> <td>2.4s</td> <td>258.8s</td> <td>82.4%</td></tr> |
| 128 |
<tr align=right><td align=left>samba-3.0.20</td><td align=left>dist_h38_j40</td> |
| 129 |
<td>101.8s</td> <td>0.6s</td> <td>94.0s</td> <td>92.3%</td></tr> |
| 130 |
<tr align=right><td align=left>samba-3.0.20</td><td align=left>pump_h38_j40</td> |
| 131 |
<td>31.4s</td> <td>2.4s</td> <td>21.1s</td> <td>67.4%</td></tr> |
| 132 |
<tr align=right><td align=left>samba-3.0.20</td><td align=left>pump_h38_j80</td> |
| 133 |
<td>31.8s</td> <td>1.7s</td> <td>21.2s</td> <td>66.9%</td></tr> |
| 134 |
<tr><td colspan=6><hr></td></tr> |
| 135 |
|
| 136 |
</table> |
| 137 |
|
| 138 |
<h2><a name="discussion" href="#discussion">Discussion</a></h2> |
| 139 |
|
| 140 |
<ul> |
| 141 |
|
| 142 |
<li> For all but the smallest projects, <code>distcc</code> results in a |
| 143 |
significant speedup in compilation time over compiling locally. This is |
| 144 |
because distcc is able to compile many source files in parallel. Such |
| 145 |
parallelism may not possible for small projects such as |
| 146 |
<code>hello</code>, which involves four small compilations. |
| 147 |
<br><br></li> |
| 148 |
|
| 149 |
<li> In almost all cases, distcc's new pump mode results in a significant |
| 150 |
speedup in compilation time over non-pump mode. Sometimes, as for |
| 151 |
Samba 3.0, the speedup is more than a factor of three!<br><br></li> |
| 152 |
|
| 153 |
<li> The parallelism of the various projects' Makefiles affects |
| 154 |
the obtained speedup significantly. Makefiles that |
| 155 |
run <code>make</code> sequentially in subdirectories benefit less from |
| 156 |
distcc. They will see little added benefit from distcc's pump mode |
| 157 |
because the sequentiality of their execution allows only some, |
| 158 |
but not many, compilations to be issued near |
| 159 |
simultaneously. <br><br></li> |
| 160 |
|
| 161 |
<li> All these opensource projects are built by running |
| 162 |
<code>configure</code> plus <code>make</code>. In each case, |
| 163 |
we count only the <code>make</code> time, not the |
| 164 |
<code>configure</code> time. However, projects such as |
| 165 |
binutils run extra configuration during the <code>make</code> |
| 166 |
step (for binutils, the initial <code>configure</code> run is |
| 167 |
trivial, and the <code>make</code> command does more intensive |
| 168 |
configuration in various subdirectories before building). This |
| 169 |
will affect times as described above -- especially reducing the |
| 170 |
benefit of pump mode -- since the configuration |
| 171 |
steps are not run in parallel. <br><br></li> |
| 172 |
|
| 173 |
<li> Even when pump mode does not speed up the build much, as for |
| 174 |
<code>httpd</code>, it still reduces the CPU burden on the |
| 175 |
local machine, making it more usable during compiles. Note, |
| 176 |
however, that this is balanced by an increased CPU burden on |
| 177 |
the server machines (about 10% in our tests), and may also |
| 178 |
require more memory on the host machine than non-pump mode |
| 179 |
does. <br><br></li> |
| 180 |
|
| 181 |
<li> With a multi-processor client machine, the speed-ups would have been |
| 182 |
less, both for non-pump distcc over local compilation as for |
| 183 |
distcc-pump over non-pump distcc. Still, with four-processor client |
| 184 |
machines, distcc's pump-mode is up to 2 1/2 times faster than its |
| 185 |
non-pump mode for large projects benchmarked at Google.<br><br></li> |
| 186 |
|
| 187 |
<li> As a side note, while collecting benchmark results, we found sometimes |
| 188 |
that pump mode did not give the expected speedup. On analyzing the |
| 189 |
logs, we discovered the reason: that distcc had encountered an error |
| 190 |
running the test in pump mode, and had fallen back to plain distcc |
| 191 |
mode. This can happen for |
| 192 |
<a href="man/include_server_1.html#TOC_5">several |
| 193 |
reasons</a>. For example, the Linux kernel needed special attention |
| 194 |
because it rewrites header files during the build. |
| 195 |
</ul> |
| 196 |
|
| 197 |
</td> |
| 198 |
</tr></table> |
| 199 |
|
| 200 |
<div class="thanks">Copyright © 2008 Google Inc.<br> |
| 201 |
distcc is a trademark of Martin Pool. |
| 202 |
|
| 203 |
<p>Send comments to <tt>distcc(at)lists.samba.org</tt>. |
| 204 |
|
| 205 |
</body> |
| 206 |
</html> |