| 1 |
<?xml version="1.0" encoding="utf-8" ?> |
| 2 |
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| 3 |
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| 4 |
<head> |
| 5 |
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> |
| 6 |
<meta name="generator" content="Docutils 0.3.1: http://docutils.sourceforge.net/" /> |
| 7 |
<title>distcc, a fast free distributed compiler</title> |
| 8 |
<meta name="date" content="December 2003" /> |
| 9 |
<meta name="author" content="Martin Pool <mbp@sourcefrog.net>" /> |
| 10 |
<link rel="stylesheet" href="default.css" type="text/css" /> |
| 11 |
</head> |
| 12 |
<body> |
| 13 |
<div class="document" id="distcc-a-fast-free-distributed-compiler"> |
| 14 |
<h1 class="title">distcc, a fast free distributed compiler</h1> |
| 15 |
<table class="docinfo" frame="void" rules="none"> |
| 16 |
<col class="docinfo-name" /> |
| 17 |
<col class="docinfo-content" /> |
| 18 |
<tbody valign="top"> |
| 19 |
<tr><th class="docinfo-name">Date:</th> |
| 20 |
<td>December 2003</td></tr> |
| 21 |
<tr><th class="docinfo-name">Author:</th> |
| 22 |
<td>Martin Pool <<a class="reference" href="mailto:mbp@sourcefrog.net">mbp@sourcefrog.net</a>></td></tr> |
| 23 |
<tr class="field"><th class="docinfo-name">Presented at:</th><td class="field-body">linux.conf.au, Adelaide, 2004</td> |
| 24 |
</tr> |
| 25 |
<tr class="field"><th class="docinfo-name">Licence:</th><td class="field-body">Verbatim reproduction of this paper is permitted.</td> |
| 26 |
</tr> |
| 27 |
<tr class="field"><th class="docinfo-name">HTML at:</th><td class="field-body"><A HREF="distcc-lca-2004.html">.../distcc-lca-2004.html</A></td> |
| 28 |
<tr class="field"><th class="docinfo-name">PDF at:</th><td class="field-body"><A HREF="distcc-lca-2004.pdf">.../distcc-lca-2004.pdf</A></td> |
| 29 |
</tr> |
| 30 |
</tbody> |
| 31 |
</table> |
| 32 |
<!-- -*- mode: rst -*- --> |
| 33 |
<div class="section" id="abstract"> |
| 34 |
<h1><a name="abstract">1 Abstract</a></h1> |
| 35 |
<p><a class="reference" href="http://distcc.samba.org/">distcc</a> is a program to distribute compilation of C, C++, Objective |
| 36 |
C and Objective C++ code between several machines on a network. |
| 37 |
distcc is easy to set up, and can build software projects several |
| 38 |
times faster than local compilation. It is distributed under the GNU |
| 39 |
General Public License.</p> |
| 40 |
<p>distcc does not require all machines to share a filesystem, have |
| 41 |
synchronised clocks, or to have the same libraries or header |
| 42 |
files installed. They can even have different processors or |
| 43 |
operating systems, if cross-compilers are installed.</p> |
| 44 |
<p>This paper begins with some background information on compilation and |
| 45 |
prior distributed compilers, then describes the design of distcc, |
| 46 |
presents the outcomes and some applications, and some lessons learned |
| 47 |
that may be useful to other small free software projects.</p> |
| 48 |
</div> |
| 49 |
<div class="section" id="background"> |
| 50 |
<h1><a name="background">2 Background</a></h1> |
| 51 |
<p>It is common for a software developer to have access to more than one |
| 52 |
computer. Perhaps they have a laptop and also a desktop machine, or |
| 53 |
perhaps they work in a group where each member has a workstation. It |
| 54 |
would be nice if the spare computing power could be used to speed up |
| 55 |
software builds.</p> |
| 56 |
<p>distcc offers a straightforward solution. The <tt class="literal"><span class="pre">distccd</span></tt> server runs |
| 57 |
on all the machines, with a low priority and a cap on the number of |
| 58 |
processes. Any client can distribute compilation jobs onto other |
| 59 |
machines. distcc can be installed by any user, without needing root |
| 60 |
access or system-wide changes.</p> |
| 61 |
<p>Faster compilations allows "clobber" builds (<tt class="literal"><span class="pre">make</span> <span class="pre">clean;</span> <span class="pre">make</span> <span class="pre">all</span></tt>) |
| 62 |
to be run more frequently, which can catch bugs that might be hidden |
| 63 |
by incorrect Makefile dependencies. It allows test engineers to get |
| 64 |
builds to test faster. It makes nightly, or hourly check builds more |
| 65 |
feasible, so bugs that are checked in can be caught and fixed with |
| 66 |
less disruption. In general it removes one limitation from the |
| 67 |
developer's work flow and allows them to work the way they want.</p> |
| 68 |
<p>The ability to build quickly is particularly useful when building |
| 69 |
binary software packages with <tt class="literal"><span class="pre">dpkg</span></tt> or <tt class="literal"><span class="pre">rpm</span></tt>. These tools often |
| 70 |
want to build the whole tree from scratch so as to ensure a repeatable |
| 71 |
build. Using distcc and ccache makes repeated package builds faster.</p> |
| 72 |
<p>Fast distributed builds are also useful for operating systems |
| 73 |
distributed primarily in source form, such as <a class="reference" href="http://www.gentoo.org/">Gentoo</a> and <a class="reference" href="http://www.linuxfromscratch.org/">Linux From |
| 74 |
Scratch</a>.</p> |
| 75 |
<div class="section" id="previous-work"> |
| 76 |
<h2><a name="previous-work">2.1 Previous work</a></h2> |
| 77 |
<p>There have been several prior distributed build systems, including |
| 78 |
pvmmake, doozer, and dmake. These systems typically require the build |
| 79 |
directory to be on a networked filesystem shared between all machines |
| 80 |
participating in the build. Furthermore, each machines must have the |
| 81 |
same header files and libraries installed. If one of the header files |
| 82 |
is inconsistent, it can, at best, produce confusing error messages, and |
| 83 |
at worst, it can cause unpredictable crashes when different modules see |
| 84 |
different definitions. All of the machine clocks must be tightly |
| 85 |
synchronized so that the timestamps are accurate.</p> |
| 86 |
<p>This can be reasonable for a dedicated build cluster, but is less |
| 87 |
practical in ad-hoc networks, where users may be able to upgrade |
| 88 |
libraries on their machines at random, or where using NFS may not be |
| 89 |
desirable.</p> |
| 90 |
<p>Another approach is to use kernel-level clustering such as <a class="reference" href="http://www.openmosix.org/">OpenMOSIX</a>. |
| 91 |
This allows any parallel tasks to be distributed, but introduces very |
| 92 |
tight coupling between machines. Systems based on process migration |
| 93 |
may not perform well with the many short-lived processes generated |
| 94 |
during compilation.</p> |
| 95 |
</div> |
| 96 |
<div class="section" id="c-compilation"> |
| 97 |
<h2><a name="c-compilation">2.2 C compilation</a></h2> |
| 98 |
<p>It's worthwhile to briefly discuss how C programs are built. |
| 99 |
Typically, a build tool (such as <em>Make</em>) invokes the C compiler for each |
| 100 |
source file to produce an object file. Within this compilation, there |
| 101 |
are several steps.</p> |
| 102 |
<blockquote> |
| 103 |
<ul class="simple"> |
| 104 |
<li>The source is run through the <tt class="literal"><span class="pre">cpp</span></tt> preprocessor, which folds in |
| 105 |
header files, handles <tt class="literal"><span class="pre">#ifdef</span></tt> and other directives, and produces |
| 106 |
a single preprocessed output file.</li> |
| 107 |
<li>The compiler proper <cite>cc1</cite> reads the preprocessed source and produces |
| 108 |
assembly output.</li> |
| 109 |
<li>The assembler <cite>as</cite> reads the assembly text and produces an object |
| 110 |
file.</li> |
| 111 |
</ul> |
| 112 |
</blockquote> |
| 113 |
<p>One interesting property of the C build process is that the |
| 114 |
preprocessed text plus the compiler command line completely specifies |
| 115 |
the output. By running the preprocessor locally, distcc avoids any |
| 116 |
requirement to have the same headers on all machines.</p> |
| 117 |
<p>Before distcc, the <a class="reference" href="http://www.erikyyy.de/compilercache/">compilercache</a> and <a class="reference" href="https://ccache.dev/">ccache</a> programs used this |
| 118 |
observation to remember the results of building a particular source |
| 119 |
file. This can be useful when trees are cleaned and rebuilt, or when |
| 120 |
similar trees are built in different directories.</p> |
| 121 |
<p>Using GNU make's <tt class="literal"><span class="pre">-j</span></tt> option several jobs are run in |
| 122 |
parallel. Make uses its graph of dependencies between targets to make |
| 123 |
sure that only independent jobs are run in parallel -- or at least it |
| 124 |
does if the Makefile correctly specifies dependencies.</p> |
| 125 |
<p>Although sending a file across a network takes a relatively long time, |
| 126 |
it takes few CPU cycles. On Linux and BSD, distcc uses the |
| 127 |
<tt class="literal"><span class="pre">sendfile</span></tt> system call so that transmission of the preprocessed |
| 128 |
source from the client is done in a single system call.</p> |
| 129 |
</div> |
| 130 |
</div> |
| 131 |
<div class="section" id="design"> |
| 132 |
<h1><a name="design">3 Design</a></h1> |
| 133 |
<p>distcc distributes work from a client machine to any number of |
| 134 |
volunteer machines. (Since this is free software, we prefer the term |
| 135 |
<em>volunteer</em> to the <em>slaves</em> that are used by other systems.)</p> |
| 136 |
<p>distcc consists of a client program and a server. The client analyzes |
| 137 |
the command run, and for jobs that can be distributed it chooses a |
| 138 |
host, runs the preprocessor, sends the request across the network and |
| 139 |
reports the results. The server accepts and handles requests |
| 140 |
containing command lines and source code and responds with object code |
| 141 |
or error messages.</p> |
| 142 |
<div class="section" id="simulating-gcc"> |
| 143 |
<h2><a name="simulating-gcc">3.1 Simulating gcc</a></h2> |
| 144 |
<p>The client is invoked as a wrapper around the compiler by Make. |
| 145 |
Because distcc is invoked in place of gcc, it needs to understand |
| 146 |
every pattern of command line invocations. If the arguments are such |
| 147 |
that the compilation can be run remotely, distcc forms two new sets of |
| 148 |
arguments, one to run the preprocessor locally, and one to run the. |
| 149 |
compiler remotely. If the arguments are not understood by distcc, it |
| 150 |
takes the safe default of running the command locally. Options that |
| 151 |
read or write additional local files such assembly listings or |
| 152 |
profiler tables are run locally</p> |
| 153 |
<p>For commands that need to be run locally, distcc runs the compiler |
| 154 |
locally, avoiding the overhead of the network protocol and of |
| 155 |
separately running the compiler. Early versions of distcc directly |
| 156 |
replaced their process with the local compiler using <tt class="literal"><span class="pre">exec</span></tt>, but in |
| 157 |
current releases the compiler is run as a child process so that its |
| 158 |
progress can be monitored. Some cases are difficult because the |
| 159 |
precise meaning of some options such as <tt class="literal"><span class="pre">-MD</span></tt> has changed between |
| 160 |
versions of gcc.</p> |
| 161 |
<p>The client supports a "masquerade" mode, where a directory such as |
| 162 |
<tt class="literal"><span class="pre">/usr/lib/distcc</span></tt> is filled with symbolic links which have the same |
| 163 |
name as the real compiler, but point to the distcc client. This |
| 164 |
directory is prepended to <tt class="literal"><span class="pre">$PATH</span></tt>, so that when Make executes <tt class="literal"><span class="pre">gcc</span></tt>, |
| 165 |
it really invokes distcc. distcc specially handles this case to work |
| 166 |
out the correct real compiler to invoke, and guards against |
| 167 |
inadvertently running itself recursively. Masquerade mode allows |
| 168 |
distcc to be used with most existing source trees without the need to |
| 169 |
change the Makefile or build process.</p> |
| 170 |
</div> |
| 171 |
<div class="section" id="networking"> |
| 172 |
<h2><a name="networking">3.2 Networking</a></h2> |
| 173 |
<p>The server runs on each of the volunteer machines, either as a |
| 174 |
standalone daemon, through <tt class="literal"><span class="pre">inetd</span></tt> or indirectly through <tt class="literal"><span class="pre">sshd</span></tt>. |
| 175 |
distcc normally runs on TCP port 3632.</p> |
| 176 |
<p>distcc uses a simple custom binary network protocol to encode requests |
| 177 |
and responses. Variable-length fields are preceded by a length word, |
| 178 |
allowing a very simple and efficient implementation.</p> |
| 179 |
<p>The request includes a protocol version, the compiler command-line |
| 180 |
arguments, and the preprocessed source. The response from the server |
| 181 |
includes confirmation of the version, the compiler's exit code, any |
| 182 |
errors or warnings from the compiler, and the object file if any.</p> |
| 183 |
<p>The distcc protocol includes four-letter magic words before each |
| 184 |
element. These words are not strictly necessary because the order of |
| 185 |
elements is always the same, but they make it easier to scan network |
| 186 |
dumps, and provide a protection against networking or program errors.</p> |
| 187 |
<p>distcc relies on TCP or SSH to ensure integrity of the stream and does |
| 188 |
not have a checksum of its own. This is the same level of protection |
| 189 |
as HTTP or NFS, and no failures have been reported to date.</p> |
| 190 |
</div> |
| 191 |
<div class="section" id="scheduler"> |
| 192 |
<h2><a name="scheduler">3.3 Scheduler</a></h2> |
| 193 |
<p>distcc uses a basic load-balancing algorithm to choose a volunteer to |
| 194 |
run each particular job. Scheduling is managed by small state files |
| 195 |
kept in each user's home directory. In addition, each server imposes |
| 196 |
a limit on the number of jobs which will be accepted at any time. |
| 197 |
This prevents any single server being overloaded, and provides basic |
| 198 |
coordination over servers being used by several clients. By default, |
| 199 |
the limit on the number of jobs is set to two greater than the number |
| 200 |
of processors in the server, to allow for up to two jobs being in |
| 201 |
transit across the network at any time.</p> |
| 202 |
<p>The server uses a <em>pre-forking</em> design similar to Apache: the maximum |
| 203 |
number of client processes are forked at startup, and all of them wait |
| 204 |
to accept connections from a client. This puts a limit on |
| 205 |
concurrency, and slightly reduces the latency of accepting a new |
| 206 |
connection.</p> |
| 207 |
<p>Clients keep a temporary note of machines which are unreachable or |
| 208 |
failing. These notes time out after sixty seconds, so machines which |
| 209 |
reboot or are temporarily unreachable will shortly rejoin the cluster. |
| 210 |
Machines which are abruptly terminated or shut down while running jobs |
| 211 |
may cause those particular compilations to fail, which can normally be |
| 212 |
addressed by re-running Make. If <tt class="literal"><span class="pre">distccd</span></tt> is gracefully |
| 213 |
terminated it will complete any running jobs and decline future submissions.</p> |
| 214 |
<p>Client processes are coordinated by Unix lock files held in |
| 215 |
<tt class="literal"><span class="pre">~/.distcc</span></tt>. <tt class="literal"><span class="pre">flocks</span></tt> have the strong advantage that they are |
| 216 |
always released by the kernel when the process terminates, even if it |
| 217 |
aborts because of a signal.</p> |
| 218 |
<p>As with a kernel scheduler, distcc does not have enough information |
| 219 |
about future jobs to make perfect scheduling decisions. In |
| 220 |
particular, make does not communicate the global state of the build |
| 221 |
and how many jobs will be issued in the future. All distcc knows is |
| 222 |
the number of jobs are running at the present moment. If it knew about |
| 223 |
future load, it might schedule jobs in the middle of a build |
| 224 |
differently from the last few jobs. Jobs on the critical path of the |
| 225 |
build and, in particular, the last jobs of a build ought to be scheduled |
| 226 |
for minimum delay rather than maximum throughput, which might mean |
| 227 |
running them locally rather than remotely.</p> |
| 228 |
<p>At the moment the client does not maintain an estimate of the speed of |
| 229 |
volunteer machines. Estimating this accurately is difficult because |
| 230 |
the performance depends on the source and computer in a complex way. |
| 231 |
Different files of similar size can take very different times to |
| 232 |
compile depending on the difficulty of the code.</p> |
| 233 |
<p>The primary shortcoming of the current scheduler is that adding slow |
| 234 |
machines to a build farm can reduce the overall performance. There is |
| 235 |
a significant opportunity cost to waiting for a job that on such a |
| 236 |
slow machine, particularly if it is a dependency for Make to proceed.</p> |
| 237 |
</div> |
| 238 |
<div class="section" id="ssh"> |
| 239 |
<h2><a name="ssh">3.4 SSH</a></h2> |
| 240 |
<p>distcc has the option of using a helper program such as <tt class="literal"><span class="pre">ssh</span></tt> to |
| 241 |
open connections rather than simply opening a TCP socket.</p> |
| 242 |
<p>When OpenSSH is used to open connections, all data is strongly |
| 243 |
encrypted. The volunteer machines do not need any additional |
| 244 |
listening ports or long-running processes. All remote compilations |
| 245 |
run under the account of the user that initiated them.</p> |
| 246 |
<p>Using OpenSSH or another pluggable connection method is a powerful |
| 247 |
Unix design pattern. Just a few hundred lines in distcc to fork the |
| 248 |
connection program and open pipes allows it to use world-class |
| 249 |
encryption and authentication that easily integrates into existing |
| 250 |
networks.</p> |
| 251 |
<p>A key advantage of this pattern is that distcc's security exposure is |
| 252 |
very small. Any connection that reaches distcc is by definition |
| 253 |
authorized either by ssh opening the socket, or because the |
| 254 |
administrator allowed connections from that address. Most of the |
| 255 |
distcc code can therefore be quite trusting of the requests that are |
| 256 |
received.</p> |
| 257 |
<p>If someone wanted to implement a simpler transport that did |
| 258 |
challenge-response password authentication but not encryption, then |
| 259 |
they could plug it in through the <tt class="literal"><span class="pre">DISTCC_SSH</span></tt> variable without |
| 260 |
needing any changes to distcc itself.</p> |
| 261 |
</div> |
| 262 |
<div class="section" id="lzo-compression"> |
| 263 |
<h2><a name="lzo-compression">3.5 LZO compression</a></h2> |
| 264 |
<p>distcc can use the <a class="reference" href="http://www.oberhumer.com/opensource/lzo/">LZO</a> fast compressor to compress requests and |
| 265 |
replies. This option is explicitly set by the client in the host |
| 266 |
list.</p> |
| 267 |
<p>The preprocessed source is typically very gassy because of long runs |
| 268 |
of whitespace and repeated identifiers, and compresses well. Object |
| 269 |
files also compress moderately well.</p> |
| 270 |
<p>Compression improves throughput when the network is the limiting |
| 271 |
factor. This can be the case when the network is very slow, such as a |
| 272 |
10Mbps wired network or a wireless network. Compression can also be |
| 273 |
useful when there are many remote machines, which can saturate even a |
| 274 |
fast Ethernet connection.</p> |
| 275 |
</div> |
| 276 |
</div> |
| 277 |
<div class="section" id="outcomes-and-applications"> |
| 278 |
<h1><a name="outcomes-and-applications">4 Outcomes and applications</a></h1> |
| 279 |
<div class="section" id="a-performance-model"> |
| 280 |
<h2><a name="a-performance-model">4.1 A performance model</a></h2> |
| 281 |
<p>The goal of distcc is to reduce the overall elapsed time to build or |
| 282 |
refresh a complete tree.</p> |
| 283 |
<p>Performance is predicted by the network bandwidth, the time to run the |
| 284 |
preprocessor and the compiler, the relative speeds of the client and |
| 285 |
volunteer computers, and the overhead of the distcc client and |
| 286 |
server. The speed of compilation on a particular machine depends on |
| 287 |
available memory, contention for the CPU, and the speed of the CPU, |
| 288 |
RAM, and bus.</p> |
| 289 |
<p>In many cases other aspects of the build process become the limiting |
| 290 |
factor once compilation is parallelized. For small packages, running |
| 291 |
an <tt class="literal"><span class="pre">autoconf</span></tt> script is much slower than actually compiling the |
| 292 |
package. <tt class="literal"><span class="pre">Make</span></tt> can also be a <a class="reference" href="http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html">limiting factor</a>, because it has |
| 293 |
high overheads and cannot schedule efficiently across source |
| 294 |
directories. One user found that distcc alone doubled build speed, |
| 295 |
but switching to the <a class="reference" href="http://www.scons.org/">SCons construction tool</a> in combination with |
| 296 |
distcc gave a five-fold improvement.</p> |
| 297 |
</div> |
| 298 |
<div class="section" id="estimate-of-feasibility"> |
| 299 |
<h2><a name="estimate-of-feasibility">4.2 Estimate of feasibility</a></h2> |
| 300 |
<p>Before any code was written for distcc, the author did a brief |
| 301 |
feasibility study to see if the idea was practical. This involved |
| 302 |
measuring the elapsed and CPU time for some of the basic operations: |
| 303 |
running the preprocessor and the compiler together and separately, |
| 304 |
copying a file across the network, and starting a simple wrapper |
| 305 |
program. The compiler used 90% or more of the CPU time in most |
| 306 |
trials. This indicated that the bulk of compilation could in theory |
| 307 |
be run remotely, and distcc might win when more than one machine was |
| 308 |
available.</p> |
| 309 |
<p>Feasibility studies such as this are heartily recommended before |
| 310 |
starting a project, especially one where success depends on achieving |
| 311 |
a quantifiable result. Knowing that a concept ought to work can be a |
| 312 |
worthwhile spur to writing a good implementation that achieves the |
| 313 |
predicted result. Conversely, knowing that an idea is impractical can |
| 314 |
avoid sinking time or money into a project that can never win. (It |
| 315 |
might still be worth documenting your results in a web page or email, |
| 316 |
in case the situation changes in the future or somebody else can think |
| 317 |
of an improvement.)</p> |
| 318 |
</div> |
| 319 |
<div class="section" id="is-it-safe"> |
| 320 |
<h2><a name="is-it-safe">4.3 Is it safe?</a></h2> |
| 321 |
<p>In eighteen months of widespread use of distcc to build diverse source |
| 322 |
trees, there have been no reported unexplained failures. Bug reports are due |
| 323 |
to, in order of frequency:</p> |
| 324 |
<blockquote> |
| 325 |
<ol class="arabic simple"> |
| 326 |
<li>Mismatched compiler versions or broken compiler installations.</li> |
| 327 |
<li>Concurrency bugs in a makefile or other build process.</li> |
| 328 |
<li>Known bugs in gcc or other software that are provoked by distcc.</li> |
| 329 |
</ol> |
| 330 |
</blockquote> |
| 331 |
<p>One persistent problem has occurred because of a quirk in the way gcc |
| 332 |
produces debug information. The compiler's working directory is |
| 333 |
inserted into the debug stabs to help in finding the source for the |
| 334 |
program. This is a problem for distcc and ccache because the compiler |
| 335 |
is invoked in a different directory from the preprocessor. This |
| 336 |
should be fixed in an upcoming gcc release by including the |
| 337 |
compilation directory in the preprocessor output.</p> |
| 338 |
</div> |
| 339 |
<div class="section" id="scalability"> |
| 340 |
<h2><a name="scalability">4.4 Scalability</a></h2> |
| 341 |
<p>Reports from users indicate, distcc is nearly linearly scalable for |
| 342 |
small numbers of CPUs. Compiling across three identical machines is |
| 343 |
typically 2.5 to 2.8 times faster than local compilation. Builds |
| 344 |
across sixteen machines have been reported at over ten times faster |
| 345 |
than a local builds. These numbers include the overhead of distcc and |
| 346 |
Make, and the time for non-parallel or non-distributed tasks.</p> |
| 347 |
<p>Amdahl's law states that the possible speedup from parallelizing a |
| 348 |
task is limited by the reciprocal of the fraction of a task which must |
| 349 |
be serialized. distcc is able to parallelize the compilation phase, |
| 350 |
but execution of the makefile, preprocessing, linking, and the distcc |
| 351 |
client must all be local. The balance of these tasks for a particular |
| 352 |
source tree determines how much it will benefit from using distcc.</p> |
| 353 |
<p>Performance tends to plateau between ten and twenty machines. This is |
| 354 |
consistent with measurements of the preprocessor using roughly 5-10% |
| 355 |
of the total CPU time: when twenty preprocessors and distcc clients |
| 356 |
are running, the client is completely saturated and cannot issue any |
| 357 |
more jobs.</p> |
| 358 |
<p>Lack of access to a large test lab has so far prevented a systematic |
| 359 |
exploration of the limits of scalability.</p> |
| 360 |
</div> |
| 361 |
<div class="section" id="compiler-versions"> |
| 362 |
<h2><a name="compiler-versions">4.5 Compiler versions</a></h2> |
| 363 |
<p>Incompatibilities between different releases of gcc proved to be a |
| 364 |
more serious problem than was expected.</p> |
| 365 |
<p>It turns out that it is not even sufficient to check the gcc version |
| 366 |
string. Across different Linux distributions, two gcc binaries |
| 367 |
claiming to be the same version can have incompatible APIs, because of |
| 368 |
distributors merging or backporting different patches. gcc is not |
| 369 |
necessarily compatible across different BSD forks.</p> |
| 370 |
<p>In general C++ causes more trouble than C because the ABI is more |
| 371 |
complex and has been more unstable.</p> |
| 372 |
<p>Incompatibilities typically manifest either as errors reading system |
| 373 |
headers which depend on gcc internals, or as link errors. In the |
| 374 |
absence of a satisfactory way to check compatibility, for the moment |
| 375 |
we just advise people to check compiler versions if an error occurs. |
| 376 |
Some automatic checking may be added in a future version.</p> |
| 377 |
</div> |
| 378 |
<div class="section" id="makefile-bugs"> |
| 379 |
<h2><a name="makefile-bugs">4.6 Makefile bugs</a></h2> |
| 380 |
<p>Because distcc can use CPUs on more than one machine, it is common to |
| 381 |
use much higher <tt class="literal"><span class="pre">-j</span></tt> levels than for local builds, with ten or even |
| 382 |
twenty jobs in flight at once. This will sometimes uncover dormant |
| 383 |
concurrency bugs in Makefiles.</p> |
| 384 |
<p>For example, a source file may depend on an automatically generated |
| 385 |
header, but that dependency may not actually be specified in the |
| 386 |
Makefile. For a non-parallel build, Make might happen to run the |
| 387 |
rules in such a way that the header is always generated first. When |
| 388 |
many concurrent tasks are run, the compile may start earlier and then |
| 389 |
fail because the header is not yet ready. As with threading bugs, |
| 390 |
these failures can be intermittent and hard to track to a root cause. |
| 391 |
Make's <tt class="literal"><span class="pre">--debug</span></tt> will eventually explain the problem after enough |
| 392 |
scrutiny.</p> |
| 393 |
<p>Some large packages such as XFree86 have stubborn Makefile bugs and |
| 394 |
must be built without parallelism. Concurrency bugs have also been |
| 395 |
discovered in tools such as <tt class="literal"><span class="pre">libtool</span></tt> that are called from |
| 396 |
makefiles. Another source tree used a single temporary file name for |
| 397 |
all object files, so that they overwrote each other when run in |
| 398 |
parallel.</p> |
| 399 |
</div> |
| 400 |
<div class="section" id="lifting-compilation"> |
| 401 |
<h2><a name="lifting-compilation">4.7 Lifting compilation</a></h2> |
| 402 |
<p>distcc can be useful even without parallel builds. It is fairly |
| 403 |
common for a developer to be working on a relatively slower machine: |
| 404 |
perhaps on a laptop, or an embedded system, or perhaps a large server |
| 405 |
machine is available. It is quite possible for the benefit in running |
| 406 |
on a faster CPU to be greater than the overhead of running distcc and |
| 407 |
transferring the build across the network.</p> |
| 408 |
<p>Being able to do most of the work of compilation remotely makes it |
| 409 |
feasible to do development entirely on embedded platforms that are |
| 410 |
large enough to run an editor and linker, such as the <a class="reference" href="http://rikkus.info/zaurus.html">Sharp Zaurus</a>.</p> |
| 411 |
<p>distcc support sociable computing by making it possible to do |
| 412 |
development on quite large projects from a laptop, if compilation is |
| 413 |
lifted off to one or more larger and noisier servers in another room.</p> |
| 414 |
</div> |
| 415 |
<div class="section" id="security"> |
| 416 |
<h2><a name="security">4.8 Security</a></h2> |
| 417 |
<p>distcc is intended to be quite secure when used according to the |
| 418 |
documentation.</p> |
| 419 |
<p>Remote execution of compile jobs introduces a trust relationship |
| 420 |
between the client and server machines. The client completely trusts |
| 421 |
the server to compile code correctly. A corrupt server could |
| 422 |
introduce malicious code into the results, or attack the client in |
| 423 |
other ways. The server completely trusts an authorized client. A |
| 424 |
malicious client could execute arbitrary commands on the server.</p> |
| 425 |
<p>The distccd TCP server does not run as root. If started by root, |
| 426 |
before accepting connections it gives away its privileges and changes |
| 427 |
to either the <em>distcc</em> or <em>nobody</em> user, or to a user specified by the |
| 428 |
<tt class="literal"><span class="pre">--user</span></tt> option.</p> |
| 429 |
<p>The distcc client may be run as root. Compiling or doing other |
| 430 |
unnecessary work as root is generally discouraged on Unix, but distcc |
| 431 |
permits it because some build processes require it.</p> |
| 432 |
<p>In TCP mode network transmissions are not encrypted or signed. An |
| 433 |
attacker with passive access to the network traffic can see the source |
| 434 |
or object code. An attacker with active access can modify the source |
| 435 |
or object code, or execute arbitrary commands on the server.</p> |
| 436 |
<p>The TCP server can limit clients by either checking the client |
| 437 |
address, or listening only on particular IP interfaces. However, in |
| 438 |
some circumstances an attacker can spoof connections from a false |
| 439 |
address. On untrusted networks SSH is the only safe choice.</p> |
| 440 |
<p>In SSH mode the server is started by each user under their own account |
| 441 |
on the server. There is no long-lived daemon. Connections are |
| 442 |
authenticated and encrypted. Every user is allowed only the |
| 443 |
privileges that their normal account allows.</p> |
| 444 |
<p>Because the server and client necessarily trust each other, there has |
| 445 |
been no security audit of the code that runs after a connection is |
| 446 |
established. It is possible that a hostile server could gain control |
| 447 |
of a client directly, as well as modifying the object code.</p> |
| 448 |
<p>Some people have proposed that the server should perform |
| 449 |
reasonableness checks on the command or source code submitted from |
| 450 |
clients. This is probably futile because gcc is not secure against |
| 451 |
hostile input, and it might be possible for an attacker to gain |
| 452 |
control of the gcc process through a carefully crafted input file or |
| 453 |
command line. The only safe assumption is that anyone able to submit |
| 454 |
jobs is able to execute arbitrary operations under the account used by |
| 455 |
distccd.</p> |
| 456 |
<p>Alternatively it has been suggested that client should check whether a |
| 457 |
server is trustworthy by re-running compilations on a different |
| 458 |
server. This approach can be useful in some distributed-computing |
| 459 |
problem spaces such as SETI@home, but is not practical for distcc. |
| 460 |
Firstly, the damage from a single compromised object file is very |
| 461 |
high, so every single file would have to be extensively checked. |
| 462 |
Secondly, the performance constraints are such that building every |
| 463 |
single file two or more times would often make distribution worthless. |
| 464 |
Thirdly there would sometimes be false alerts from minor differences |
| 465 |
in gcc patchlevel that are not visible in normal use but that do |
| 466 |
change the output file at a byte level.</p> |
| 467 |
<p>Only one security issue has arisen to date, which was that distcc used |
| 468 |
a predictable temporary directory name. A local attacker by creating |
| 469 |
that directory could cause distcc to fail, but no more damage.</p> |
| 470 |
</div> |
| 471 |
<div class="section" id="distcc-press-gangs"> |
| 472 |
<h2><a name="distcc-press-gangs">4.9 distcc press-gangs</a></h2> |
| 473 |
<p>Systems have been developed which allow non-Unix machines to be |
| 474 |
co-opted to contribute to compilation.</p> |
| 475 |
<p>One such is <a class="reference" href="http://opendoorsoftware.com/cgi/http.pl?p=distccKNOPPIX">distccKNOPPIX</a>, which allows a machine to be rebooted and |
| 476 |
to work as a compiler volunteer without needing any installation onto |
| 477 |
its hard disk. The compiler and distcc server are loaded across http, |
| 478 |
so that they can easily be matched to the versions on the client.</p> |
| 479 |
<p>distcc packages are now available for <a class="reference" href="http://www.cygwin.com/">Cygwin</a>, allowing cross |
| 480 |
compilers to be run on Microsoft systems.</p> |
| 481 |
</div> |
| 482 |
<div class="section" id="good-logging"> |
| 483 |
<h2><a name="good-logging">4.10 Good logging</a></h2> |
| 484 |
<p>Most releases of distcc has discovered bugs in either distcc or other |
| 485 |
software that could not be reproduced on the author's machine. It has |
| 486 |
the honour of helping find a <a class="reference" href="http://www.mail-archive.com/distcc@lists.samba.org/msg00871.html">serious bug in the Linux 2.5 TCP stack</a> by |
| 487 |
being the first program to flood the network fast enough to provoke a |
| 488 |
timer wraparound.</p> |
| 489 |
<p>At the default logging level, distcc emits only warnings and errors, |
| 490 |
such as when a compile host is unreachable or compilation fails.</p> |
| 491 |
<p>When verbose logging is enabled, distcc reports the success or failure |
| 492 |
of most non-trivial system calls and internal functions. Failure can |
| 493 |
indicate either a problem on the machine, or a bug in distcc. In the |
| 494 |
first case, the logs can help the user find out where the problem is. |
| 495 |
If there was a bug in distcc, the trace helps work out where exactly |
| 496 |
the failure occurred, what actions led up to that point, and what the |
| 497 |
consequences were. Many of these bugs are at heart incorrect |
| 498 |
assumptions about, for example, how distcc will be called or how a |
| 499 |
system call behaves on different platforms. The trace often calls out |
| 500 |
which assumptions were incorrect.</p> |
| 501 |
<p>The goal of distcc's error handling is to have any problem cause at |
| 502 |
most one failure. It ought to be possible to look at a verbose trace |
| 503 |
and work out what went wrong and the fix.</p> |
| 504 |
<p>Errors and trace messages from the server are written both into a log |
| 505 |
file, and also into the compiler error stream passed back to the |
| 506 |
client. This makes it more likely that if a problem occurs on the |
| 507 |
server, it will be obvious to the remote user without their having to |
| 508 |
look in the server's log file. This pattern is very useful, though it |
| 509 |
might not be appropriate for software that handles requests from |
| 510 |
untrusted users. Errors during server startup are written to error |
| 511 |
output as well as to the log file, to make it more obvious to the |
| 512 |
administrator that the server did not actually start.</p> |
| 513 |
</div> |
| 514 |
<div class="section" id="generalisations"> |
| 515 |
<h2><a name="generalisations">4.11 Generalisations</a></h2> |
| 516 |
<p>There has been some interest in adapting distcc to compile other |
| 517 |
languages such as Java or C#. These languages have quite different |
| 518 |
mappings from source files to output, and in particular there is no |
| 519 |
separate preprocessing phase. A Java compiler must read class files |
| 520 |
as directed by <tt class="literal"><span class="pre">import</span></tt> statements, and those classes in turn may |
| 521 |
need to be rebuilt from source. A distributed compiler for these |
| 522 |
languages would be possible but might need to be more tightly |
| 523 |
integrated with the compiler than distcc is.</p> |
| 524 |
<p>distcc could be generalised still further to distribute arbitrary |
| 525 |
non-interactive jobs, rather than just compilation. A good step |
| 526 |
towards this would be to move all the intelligence about command lines |
| 527 |
into the client, so that the server just managed temporary files and |
| 528 |
executed specified commands. To date no compelling application for |
| 529 |
this sort of distribution has presented. Scientific and technical |
| 530 |
computing users seem reasonably well served by clustering and |
| 531 |
batch-queueing software.</p> |
| 532 |
</div> |
| 533 |
<div class="section" id="changes-in-gcc"> |
| 534 |
<h2><a name="changes-in-gcc">4.12 Changes in gcc</a></h2> |
| 535 |
<p>gcc version 3.3 includes some performance improvements that make |
| 536 |
distcc less immediately useful.</p> |
| 537 |
<p>One is precompiled headers, whereby the internal state of the compiler |
| 538 |
is dumped out to disk after reading a set of common header files. |
| 539 |
This is useful for programs that include a large and fixed set of |
| 540 |
headers, where parsing the headers takes a large fraction of the |
| 541 |
compile time. At the moment distcc can support this by putting the |
| 542 |
<tt class="literal"><span class="pre">.pch</span></tt> state file on a shared filesystem. A better solution may be |
| 543 |
added to distcc in the future.</p> |
| 544 |
<p>gcc has also recently gained an integrated preprocessor option, to |
| 545 |
avoid the overhead of running the preprocessor as a separate task. |
| 546 |
Since distcc must run the preprocessor separately, it cannot benefit |
| 547 |
from the integrated preprocessor and it's performance compared to gcc |
| 548 |
is slightly worse.</p> |
| 549 |
</div> |
| 550 |
<div class="section" id="worse-is-better"> |
| 551 |
<h2><a name="worse-is-better">4.13 Worse is better</a></h2> |
| 552 |
<p>distcc adheres to the "<a class="reference" href="http://www.jwz.org/doc/worse-is-better.html">Worse is Better</a>" philosophy, summarized by |
| 553 |
Richard Gabriel as four points:</p> |
| 554 |
<blockquote> |
| 555 |
<ul class="simple"> |
| 556 |
<li><em>Simplicity: the design must be simple, both in implementation and |
| 557 |
interface. It is more important for the implementation to be |
| 558 |
simple than the interface. Simplicity is the most important |
| 559 |
consideration in a design.</em></li> |
| 560 |
</ul> |
| 561 |
</blockquote> |
| 562 |
<p>distcc does not require changes to the kernel, the build system, or |
| 563 |
the compiler. It can be described in a couple of pages, and installed |
| 564 |
in a matter of minutes. There is no central coordinating process or |
| 565 |
global knowledge of the cluster's state.</p> |
| 566 |
<p>The most prominent place where the implementation is less simple |
| 567 |
than necesssary is in network IO, which is moderately optimized. |
| 568 |
This is perhaps justifiable as network speed is important to the |
| 569 |
program being useful.</p> |
| 570 |
<p>Masquerading as gcc makes the implementation more complex, because |
| 571 |
distcc must identify that it is masqueraded, and then make |
| 572 |
appropriate adjustments to the path to invoke the real compiler. |
| 573 |
However, masquerading makes the installation and use of distcc much |
| 574 |
easier, and so it is a reasonable tradeoff. If it had turned out |
| 575 |
to be hard to find a reliable and reasonably simple masquerading |
| 576 |
algorithm, the feature might have been dropped.</p> |
| 577 |
<p>distcc is written in C, which makes a correct implementation |
| 578 |
slightly harder and more complex than it would be in a scripting |
| 579 |
language. This is justified because to perform adequately, distcc |
| 580 |
absolutely must have a very small startup cost, which is not the |
| 581 |
case for any popular very-high-level language.</p> |
| 582 |
<blockquote> |
| 583 |
<ul class="simple"> |
| 584 |
<li><em>Correctness: the design must be correct in all observable |
| 585 |
aspects. It is slightly better to be simple than correct.</em></li> |
| 586 |
</ul> |
| 587 |
</blockquote> |
| 588 |
<p>There are no known cases where distcc generates an incorrect and |
| 589 |
unexplained result. The edges cases such as debug information are |
| 590 |
being fixed in a simple manner, by correcting gcc, rather than by |
| 591 |
complicating distcc.</p> |
| 592 |
<p>Almost all free software trees will build, with the restriction |
| 593 |
that trees with concurrency bugs cannot be built in parallel.</p> |
| 594 |
<p>When distcc encounters an unexpected condition, it tries to "fail |
| 595 |
safe" by, for example, running the compilation locally, backing off |
| 596 |
from a particular machine and avoiding accidental recursive |
| 597 |
invocations.</p> |
| 598 |
<blockquote> |
| 599 |
<ul class="simple"> |
| 600 |
<li><em>Consistency: the design must not be overly inconsistent. Consistency |
| 601 |
can be sacrificed for simplicity in some cases, but it is better to |
| 602 |
drop those parts of the design that deal with less common |
| 603 |
circumstances than to introduce either implementational complexity |
| 604 |
or inconsistency.</em></li> |
| 605 |
</ul> |
| 606 |
</blockquote> |
| 607 |
<p>Almost all the code in distcc is common across platforms, and most of |
| 608 |
the code is reached on any particular execution: there are |
| 609 |
relatively few special cases.</p> |
| 610 |
<p>There are some special cases and inconsistencies in the way the |
| 611 |
daemon starts up, in an attempt to make it more friendly to |
| 612 |
administrators and as historical accidents.</p> |
| 613 |
<p>distcc is very consistent with modern free Unix practice in details |
| 614 |
such as responding to <tt class="literal"><span class="pre">--help</span></tt>, using the GNU autoconf system for |
| 615 |
installation, and installing into standard directories.</p> |
| 616 |
<blockquote> |
| 617 |
<ul class="simple"> |
| 618 |
<li><em>Completeness: the design must cover as many important situations as |
| 619 |
is practical. All reasonably expected cases should be |
| 620 |
covered. Completeness can be sacrificed in favor of any other |
| 621 |
quality. In fact, completeness must sacrificed whenever implementation |
| 622 |
simplicity is jeopardized. Consistency can be sacrificed to achieve |
| 623 |
completeness if simplicity is retained; especially worthless is |
| 624 |
consistency of interface.</em></li> |
| 625 |
</ul> |
| 626 |
</blockquote> |
| 627 |
<p>distcc is intentionally not a general-purpose clustering system. |
| 628 |
They are too hard to write, and none that can be installed in a |
| 629 |
matter of minutes. distcc has very incomplete coverage of all the |
| 630 |
tasks people might want to distribute across machines, but it does |
| 631 |
handle a large task that is important to a significant audience.</p> |
| 632 |
<p>Within the space of C compilation, most tasks can be distributed. |
| 633 |
Even if a few jobs during a software build cannot be distributed, |
| 634 |
others run in parallel might be, so there is still an overall |
| 635 |
benefit. distcc's documentation and diagnostic output make it |
| 636 |
reasonably easy to identify problem commands.</p> |
| 637 |
<p>Some gcc options that could in practice be distributed are run |
| 638 |
locally for the sake of implementation simplicity. Commands that |
| 639 |
produce assembly listings are one example: there is no reason why |
| 640 |
the assembly output could not be relocated back to the client, but |
| 641 |
this option is rarely used so the complexity is not justified.</p> |
| 642 |
</div> |
| 643 |
</div> |
| 644 |
<div class="section" id="conclusions"> |
| 645 |
<h1><a name="conclusions">5 Conclusions</a></h1> |
| 646 |
<p>distcc has a simple design and gains large performance improvements |
| 647 |
for software builds. It is reliable and straightforward to install |
| 648 |
and manage.</p> |
| 649 |
<p>A few maxims about demonstration of software projects have been borne |
| 650 |
out by distcc's development. Before development began, a feasibility |
| 651 |
study demonstrated that the concept could work. The design is |
| 652 |
intentionally simple and the scope modestly focused on a subset of |
| 653 |
important problems. Strong logging helps get useful results from |
| 654 |
distributed community testing. Performance features such as |
| 655 |
compression and new network APIs were used selectively in places where |
| 656 |
they are shown to increase distcc's speed.</p> |
| 657 |
<p>As the world's body of source code continues to grow, distcc will help |
| 658 |
get it built, tested and released more quickly and painlessly.</p> |
| 659 |
<!-- LocalWords: distccKNOPPIX compilercache --> |
| 660 |
</div> |
| 661 |
</div> |
| 662 |
</body> |
| 663 |
</html> |