Debian Codegraph @

doc/web/distcc-lca-2004.html

raw
1 <?xml version="1.0" encoding="utf-8" ?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
4 <head>
5 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
6 <meta name="generator" content="Docutils 0.3.1: http://docutils.sourceforge.net/" />
7 <title>distcc, a fast free distributed compiler</title>
8 <meta name="date" content="December 2003" />
9 <meta name="author" content="Martin Pool &lt;mbp&#64;sourcefrog.net&gt;" />
10 <link rel="stylesheet" href="default.css" type="text/css" />
11 </head>
12 <body>
13 <div class="document" id="distcc-a-fast-free-distributed-compiler">
14 <h1 class="title">distcc, a fast free distributed compiler</h1>
15 <table class="docinfo" frame="void" rules="none">
16 <col class="docinfo-name" />
17 <col class="docinfo-content" />
18 <tbody valign="top">
19 <tr><th class="docinfo-name">Date:</th>
20 <td>December 2003</td></tr>
21 <tr><th class="docinfo-name">Author:</th>
22 <td>Martin Pool &lt;<a class="reference" href="mailto:mbp&#64;sourcefrog.net">mbp&#64;sourcefrog.net</a>&gt;</td></tr>
23 <tr class="field"><th class="docinfo-name">Presented at:</th><td class="field-body">linux.conf.au, Adelaide, 2004</td>
24 </tr>
25 <tr class="field"><th class="docinfo-name">Licence:</th><td class="field-body">Verbatim reproduction of this paper is permitted.</td>
26 </tr>
27 <tr class="field"><th class="docinfo-name">HTML at:</th><td class="field-body"><A HREF="distcc-lca-2004.html">.../distcc-lca-2004.html</A></td>
28 <tr class="field"><th class="docinfo-name">PDF at:</th><td class="field-body"><A HREF="distcc-lca-2004.pdf">.../distcc-lca-2004.pdf</A></td>
29 </tr>
30 </tbody>
31 </table>
32 <!-- -*- mode: rst -*- -->
33 <div class="section" id="abstract">
34 <h1><a name="abstract">1   Abstract</a></h1>
35 <p><a class="reference" href="http://distcc.samba.org/">distcc</a> is a program to distribute compilation of C, C++, Objective
36 C and Objective C++ code between several machines on a network.
37 distcc is easy to set up, and can build software projects several
38 times faster than local compilation. It is distributed under the GNU
39 General Public License.</p>
40 <p>distcc does not require all machines to share a filesystem, have
41 synchronised clocks, or to have the same libraries or header
42 files installed. They can even have different processors or
43 operating systems, if cross-compilers are installed.</p>
44 <p>This paper begins with some background information on compilation and
45 prior distributed compilers, then describes the design of distcc,
46 presents the outcomes and some applications, and some lessons learned
47 that may be useful to other small free software projects.</p>
48 </div>
49 <div class="section" id="background">
50 <h1><a name="background">2   Background</a></h1>
51 <p>It is common for a software developer to have access to more than one
52 computer. Perhaps they have a laptop and also a desktop machine, or
53 perhaps they work in a group where each member has a workstation. It
54 would be nice if the spare computing power could be used to speed up
55 software builds.</p>
56 <p>distcc offers a straightforward solution. The <tt class="literal"><span class="pre">distccd</span></tt> server runs
57 on all the machines, with a low priority and a cap on the number of
58 processes. Any client can distribute compilation jobs onto other
59 machines. distcc can be installed by any user, without needing root
60 access or system-wide changes.</p>
61 <p>Faster compilations allows &quot;clobber&quot; builds (<tt class="literal"><span class="pre">make</span> <span class="pre">clean;</span> <span class="pre">make</span> <span class="pre">all</span></tt>)
62 to be run more frequently, which can catch bugs that might be hidden
63 by incorrect Makefile dependencies. It allows test engineers to get
64 builds to test faster. It makes nightly, or hourly check builds more
65 feasible, so bugs that are checked in can be caught and fixed with
66 less disruption. In general it removes one limitation from the
67 developer's work flow and allows them to work the way they want.</p>
68 <p>The ability to build quickly is particularly useful when building
69 binary software packages with <tt class="literal"><span class="pre">dpkg</span></tt> or <tt class="literal"><span class="pre">rpm</span></tt>. These tools often
70 want to build the whole tree from scratch so as to ensure a repeatable
71 build. Using distcc and ccache makes repeated package builds faster.</p>
72 <p>Fast distributed builds are also useful for operating systems
73 distributed primarily in source form, such as <a class="reference" href="http://www.gentoo.org/">Gentoo</a> and <a class="reference" href="http://www.linuxfromscratch.org/">Linux From
74 Scratch</a>.</p>
75 <div class="section" id="previous-work">
76 <h2><a name="previous-work">2.1   Previous work</a></h2>
77 <p>There have been several prior distributed build systems, including
78 pvmmake, doozer, and dmake. These systems typically require the build
79 directory to be on a networked filesystem shared between all machines
80 participating in the build. Furthermore, each machines must have the
81 same header files and libraries installed. If one of the header files
82 is inconsistent, it can, at best, produce confusing error messages, and
83 at worst, it can cause unpredictable crashes when different modules see
84 different definitions. All of the machine clocks must be tightly
85 synchronized so that the timestamps are accurate.</p>
86 <p>This can be reasonable for a dedicated build cluster, but is less
87 practical in ad-hoc networks, where users may be able to upgrade
88 libraries on their machines at random, or where using NFS may not be
89 desirable.</p>
90 <p>Another approach is to use kernel-level clustering such as <a class="reference" href="http://www.openmosix.org/">OpenMOSIX</a>.
91 This allows any parallel tasks to be distributed, but introduces very
92 tight coupling between machines. Systems based on process migration
93 may not perform well with the many short-lived processes generated
94 during compilation.</p>
95 </div>
96 <div class="section" id="c-compilation">
97 <h2><a name="c-compilation">2.2   C compilation</a></h2>
98 <p>It's worthwhile to briefly discuss how C programs are built.
99 Typically, a build tool (such as <em>Make</em>) invokes the C compiler for each
100 source file to produce an object file. Within this compilation, there
101 are several steps.</p>
102 <blockquote>
103 <ul class="simple">
104 <li>The source is run through the <tt class="literal"><span class="pre">cpp</span></tt> preprocessor, which folds in
105 header files, handles <tt class="literal"><span class="pre">#ifdef</span></tt> and other directives, and produces
106 a single preprocessed output file.</li>
107 <li>The compiler proper <cite>cc1</cite> reads the preprocessed source and produces
108 assembly output.</li>
109 <li>The assembler <cite>as</cite> reads the assembly text and produces an object
110 file.</li>
111 </ul>
112 </blockquote>
113 <p>One interesting property of the C build process is that the
114 preprocessed text plus the compiler command line completely specifies
115 the output. By running the preprocessor locally, distcc avoids any
116 requirement to have the same headers on all machines.</p>
117 <p>Before distcc, the <a class="reference" href="http://www.erikyyy.de/compilercache/">compilercache</a> and <a class="reference" href="https://ccache.dev/">ccache</a> programs used this
118 observation to remember the results of building a particular source
119 file. This can be useful when trees are cleaned and rebuilt, or when
120 similar trees are built in different directories.</p>
121 <p>Using GNU make's <tt class="literal"><span class="pre">-j</span></tt> option several jobs are run in
122 parallel. Make uses its graph of dependencies between targets to make
123 sure that only independent jobs are run in parallel -- or at least it
124 does if the Makefile correctly specifies dependencies.</p>
125 <p>Although sending a file across a network takes a relatively long time,
126 it takes few CPU cycles. On Linux and BSD, distcc uses the
127 <tt class="literal"><span class="pre">sendfile</span></tt> system call so that transmission of the preprocessed
128 source from the client is done in a single system call.</p>
129 </div>
130 </div>
131 <div class="section" id="design">
132 <h1><a name="design">3   Design</a></h1>
133 <p>distcc distributes work from a client machine to any number of
134 volunteer machines. (Since this is free software, we prefer the term
135 <em>volunteer</em> to the <em>slaves</em> that are used by other systems.)</p>
136 <p>distcc consists of a client program and a server. The client analyzes
137 the command run, and for jobs that can be distributed it chooses a
138 host, runs the preprocessor, sends the request across the network and
139 reports the results. The server accepts and handles requests
140 containing command lines and source code and responds with object code
141 or error messages.</p>
142 <div class="section" id="simulating-gcc">
143 <h2><a name="simulating-gcc">3.1   Simulating gcc</a></h2>
144 <p>The client is invoked as a wrapper around the compiler by Make.
145 Because distcc is invoked in place of gcc, it needs to understand
146 every pattern of command line invocations. If the arguments are such
147 that the compilation can be run remotely, distcc forms two new sets of
148 arguments, one to run the preprocessor locally, and one to run the.
149 compiler remotely. If the arguments are not understood by distcc, it
150 takes the safe default of running the command locally. Options that
151 read or write additional local files such assembly listings or
152 profiler tables are run locally</p>
153 <p>For commands that need to be run locally, distcc runs the compiler
154 locally, avoiding the overhead of the network protocol and of
155 separately running the compiler. Early versions of distcc directly
156 replaced their process with the local compiler using <tt class="literal"><span class="pre">exec</span></tt>, but in
157 current releases the compiler is run as a child process so that its
158 progress can be monitored. Some cases are difficult because the
159 precise meaning of some options such as <tt class="literal"><span class="pre">-MD</span></tt> has changed between
160 versions of gcc.</p>
161 <p>The client supports a &quot;masquerade&quot; mode, where a directory such as
162 <tt class="literal"><span class="pre">/usr/lib/distcc</span></tt> is filled with symbolic links which have the same
163 name as the real compiler, but point to the distcc client. This
164 directory is prepended to <tt class="literal"><span class="pre">$PATH</span></tt>, so that when Make executes <tt class="literal"><span class="pre">gcc</span></tt>,
165 it really invokes distcc. distcc specially handles this case to work
166 out the correct real compiler to invoke, and guards against
167 inadvertently running itself recursively. Masquerade mode allows
168 distcc to be used with most existing source trees without the need to
169 change the Makefile or build process.</p>
170 </div>
171 <div class="section" id="networking">
172 <h2><a name="networking">3.2   Networking</a></h2>
173 <p>The server runs on each of the volunteer machines, either as a
174 standalone daemon, through <tt class="literal"><span class="pre">inetd</span></tt> or indirectly through <tt class="literal"><span class="pre">sshd</span></tt>.
175 distcc normally runs on TCP port 3632.</p>
176 <p>distcc uses a simple custom binary network protocol to encode requests
177 and responses. Variable-length fields are preceded by a length word,
178 allowing a very simple and efficient implementation.</p>
179 <p>The request includes a protocol version, the compiler command-line
180 arguments, and the preprocessed source. The response from the server
181 includes confirmation of the version, the compiler's exit code, any
182 errors or warnings from the compiler, and the object file if any.</p>
183 <p>The distcc protocol includes four-letter magic words before each
184 element. These words are not strictly necessary because the order of
185 elements is always the same, but they make it easier to scan network
186 dumps, and provide a protection against networking or program errors.</p>
187 <p>distcc relies on TCP or SSH to ensure integrity of the stream and does
188 not have a checksum of its own. This is the same level of protection
189 as HTTP or NFS, and no failures have been reported to date.</p>
190 </div>
191 <div class="section" id="scheduler">
192 <h2><a name="scheduler">3.3   Scheduler</a></h2>
193 <p>distcc uses a basic load-balancing algorithm to choose a volunteer to
194 run each particular job. Scheduling is managed by small state files
195 kept in each user's home directory. In addition, each server imposes
196 a limit on the number of jobs which will be accepted at any time.
197 This prevents any single server being overloaded, and provides basic
198 coordination over servers being used by several clients. By default,
199 the limit on the number of jobs is set to two greater than the number
200 of processors in the server, to allow for up to two jobs being in
201 transit across the network at any time.</p>
202 <p>The server uses a <em>pre-forking</em> design similar to Apache: the maximum
203 number of client processes are forked at startup, and all of them wait
204 to accept connections from a client. This puts a limit on
205 concurrency, and slightly reduces the latency of accepting a new
206 connection.</p>
207 <p>Clients keep a temporary note of machines which are unreachable or
208 failing. These notes time out after sixty seconds, so machines which
209 reboot or are temporarily unreachable will shortly rejoin the cluster.
210 Machines which are abruptly terminated or shut down while running jobs
211 may cause those particular compilations to fail, which can normally be
212 addressed by re-running Make. If <tt class="literal"><span class="pre">distccd</span></tt> is gracefully
213 terminated it will complete any running jobs and decline future submissions.</p>
214 <p>Client processes are coordinated by Unix lock files held in
215 <tt class="literal"><span class="pre">~/.distcc</span></tt>. <tt class="literal"><span class="pre">flocks</span></tt> have the strong advantage that they are
216 always released by the kernel when the process terminates, even if it
217 aborts because of a signal.</p>
218 <p>As with a kernel scheduler, distcc does not have enough information
219 about future jobs to make perfect scheduling decisions. In
220 particular, make does not communicate the global state of the build
221 and how many jobs will be issued in the future. All distcc knows is
222 the number of jobs are running at the present moment. If it knew about
223 future load, it might schedule jobs in the middle of a build
224 differently from the last few jobs. Jobs on the critical path of the
225 build and, in particular, the last jobs of a build ought to be scheduled
226 for minimum delay rather than maximum throughput, which might mean
227 running them locally rather than remotely.</p>
228 <p>At the moment the client does not maintain an estimate of the speed of
229 volunteer machines. Estimating this accurately is difficult because
230 the performance depends on the source and computer in a complex way.
231 Different files of similar size can take very different times to
232 compile depending on the difficulty of the code.</p>
233 <p>The primary shortcoming of the current scheduler is that adding slow
234 machines to a build farm can reduce the overall performance. There is
235 a significant opportunity cost to waiting for a job that on such a
236 slow machine, particularly if it is a dependency for Make to proceed.</p>
237 </div>
238 <div class="section" id="ssh">
239 <h2><a name="ssh">3.4   SSH</a></h2>
240 <p>distcc has the option of using a helper program such as <tt class="literal"><span class="pre">ssh</span></tt> to
241 open connections rather than simply opening a TCP socket.</p>
242 <p>When OpenSSH is used to open connections, all data is strongly
243 encrypted. The volunteer machines do not need any additional
244 listening ports or long-running processes. All remote compilations
245 run under the account of the user that initiated them.</p>
246 <p>Using OpenSSH or another pluggable connection method is a powerful
247 Unix design pattern. Just a few hundred lines in distcc to fork the
248 connection program and open pipes allows it to use world-class
249 encryption and authentication that easily integrates into existing
250 networks.</p>
251 <p>A key advantage of this pattern is that distcc's security exposure is
252 very small. Any connection that reaches distcc is by definition
253 authorized either by ssh opening the socket, or because the
254 administrator allowed connections from that address. Most of the
255 distcc code can therefore be quite trusting of the requests that are
256 received.</p>
257 <p>If someone wanted to implement a simpler transport that did
258 challenge-response password authentication but not encryption, then
259 they could plug it in through the <tt class="literal"><span class="pre">DISTCC_SSH</span></tt> variable without
260 needing any changes to distcc itself.</p>
261 </div>
262 <div class="section" id="lzo-compression">
263 <h2><a name="lzo-compression">3.5   LZO compression</a></h2>
264 <p>distcc can use the <a class="reference" href="http://www.oberhumer.com/opensource/lzo/">LZO</a> fast compressor to compress requests and
265 replies. This option is explicitly set by the client in the host
266 list.</p>
267 <p>The preprocessed source is typically very gassy because of long runs
268 of whitespace and repeated identifiers, and compresses well. Object
269 files also compress moderately well.</p>
270 <p>Compression improves throughput when the network is the limiting
271 factor. This can be the case when the network is very slow, such as a
272 10Mbps wired network or a wireless network. Compression can also be
273 useful when there are many remote machines, which can saturate even a
274 fast Ethernet connection.</p>
275 </div>
276 </div>
277 <div class="section" id="outcomes-and-applications">
278 <h1><a name="outcomes-and-applications">4   Outcomes and applications</a></h1>
279 <div class="section" id="a-performance-model">
280 <h2><a name="a-performance-model">4.1   A performance model</a></h2>
281 <p>The goal of distcc is to reduce the overall elapsed time to build or
282 refresh a complete tree.</p>
283 <p>Performance is predicted by the network bandwidth, the time to run the
284 preprocessor and the compiler, the relative speeds of the client and
285 volunteer computers, and the overhead of the distcc client and
286 server. The speed of compilation on a particular machine depends on
287 available memory, contention for the CPU, and the speed of the CPU,
288 RAM, and bus.</p>
289 <p>In many cases other aspects of the build process become the limiting
290 factor once compilation is parallelized. For small packages, running
291 an <tt class="literal"><span class="pre">autoconf</span></tt> script is much slower than actually compiling the
292 package. <tt class="literal"><span class="pre">Make</span></tt> can also be a <a class="reference" href="http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html">limiting factor</a>, because it has
293 high overheads and cannot schedule efficiently across source
294 directories. One user found that distcc alone doubled build speed,
295 but switching to the <a class="reference" href="http://www.scons.org/">SCons construction tool</a> in combination with
296 distcc gave a five-fold improvement.</p>
297 </div>
298 <div class="section" id="estimate-of-feasibility">
299 <h2><a name="estimate-of-feasibility">4.2   Estimate of feasibility</a></h2>
300 <p>Before any code was written for distcc, the author did a brief
301 feasibility study to see if the idea was practical. This involved
302 measuring the elapsed and CPU time for some of the basic operations:
303 running the preprocessor and the compiler together and separately,
304 copying a file across the network, and starting a simple wrapper
305 program. The compiler used 90% or more of the CPU time in most
306 trials. This indicated that the bulk of compilation could in theory
307 be run remotely, and distcc might win when more than one machine was
308 available.</p>
309 <p>Feasibility studies such as this are heartily recommended before
310 starting a project, especially one where success depends on achieving
311 a quantifiable result. Knowing that a concept ought to work can be a
312 worthwhile spur to writing a good implementation that achieves the
313 predicted result. Conversely, knowing that an idea is impractical can
314 avoid sinking time or money into a project that can never win. (It
315 might still be worth documenting your results in a web page or email,
316 in case the situation changes in the future or somebody else can think
317 of an improvement.)</p>
318 </div>
319 <div class="section" id="is-it-safe">
320 <h2><a name="is-it-safe">4.3   Is it safe?</a></h2>
321 <p>In eighteen months of widespread use of distcc to build diverse source
322 trees, there have been no reported unexplained failures. Bug reports are due
323 to, in order of frequency:</p>
324 <blockquote>
325 <ol class="arabic simple">
326 <li>Mismatched compiler versions or broken compiler installations.</li>
327 <li>Concurrency bugs in a makefile or other build process.</li>
328 <li>Known bugs in gcc or other software that are provoked by distcc.</li>
329 </ol>
330 </blockquote>
331 <p>One persistent problem has occurred because of a quirk in the way gcc
332 produces debug information. The compiler's working directory is
333 inserted into the debug stabs to help in finding the source for the
334 program. This is a problem for distcc and ccache because the compiler
335 is invoked in a different directory from the preprocessor. This
336 should be fixed in an upcoming gcc release by including the
337 compilation directory in the preprocessor output.</p>
338 </div>
339 <div class="section" id="scalability">
340 <h2><a name="scalability">4.4   Scalability</a></h2>
341 <p>Reports from users indicate, distcc is nearly linearly scalable for
342 small numbers of CPUs. Compiling across three identical machines is
343 typically 2.5 to 2.8 times faster than local compilation. Builds
344 across sixteen machines have been reported at over ten times faster
345 than a local builds. These numbers include the overhead of distcc and
346 Make, and the time for non-parallel or non-distributed tasks.</p>
347 <p>Amdahl's law states that the possible speedup from parallelizing a
348 task is limited by the reciprocal of the fraction of a task which must
349 be serialized. distcc is able to parallelize the compilation phase,
350 but execution of the makefile, preprocessing, linking, and the distcc
351 client must all be local. The balance of these tasks for a particular
352 source tree determines how much it will benefit from using distcc.</p>
353 <p>Performance tends to plateau between ten and twenty machines. This is
354 consistent with measurements of the preprocessor using roughly 5-10%
355 of the total CPU time: when twenty preprocessors and distcc clients
356 are running, the client is completely saturated and cannot issue any
357 more jobs.</p>
358 <p>Lack of access to a large test lab has so far prevented a systematic
359 exploration of the limits of scalability.</p>
360 </div>
361 <div class="section" id="compiler-versions">
362 <h2><a name="compiler-versions">4.5   Compiler versions</a></h2>
363 <p>Incompatibilities between different releases of gcc proved to be a
364 more serious problem than was expected.</p>
365 <p>It turns out that it is not even sufficient to check the gcc version
366 string. Across different Linux distributions, two gcc binaries
367 claiming to be the same version can have incompatible APIs, because of
368 distributors merging or backporting different patches. gcc is not
369 necessarily compatible across different BSD forks.</p>
370 <p>In general C++ causes more trouble than C because the ABI is more
371 complex and has been more unstable.</p>
372 <p>Incompatibilities typically manifest either as errors reading system
373 headers which depend on gcc internals, or as link errors. In the
374 absence of a satisfactory way to check compatibility, for the moment
375 we just advise people to check compiler versions if an error occurs.
376 Some automatic checking may be added in a future version.</p>
377 </div>
378 <div class="section" id="makefile-bugs">
379 <h2><a name="makefile-bugs">4.6   Makefile bugs</a></h2>
380 <p>Because distcc can use CPUs on more than one machine, it is common to
381 use much higher <tt class="literal"><span class="pre">-j</span></tt> levels than for local builds, with ten or even
382 twenty jobs in flight at once. This will sometimes uncover dormant
383 concurrency bugs in Makefiles.</p>
384 <p>For example, a source file may depend on an automatically generated
385 header, but that dependency may not actually be specified in the
386 Makefile. For a non-parallel build, Make might happen to run the
387 rules in such a way that the header is always generated first. When
388 many concurrent tasks are run, the compile may start earlier and then
389 fail because the header is not yet ready. As with threading bugs,
390 these failures can be intermittent and hard to track to a root cause.
391 Make's <tt class="literal"><span class="pre">--debug</span></tt> will eventually explain the problem after enough
392 scrutiny.</p>
393 <p>Some large packages such as XFree86 have stubborn Makefile bugs and
394 must be built without parallelism. Concurrency bugs have also been
395 discovered in tools such as <tt class="literal"><span class="pre">libtool</span></tt> that are called from
396 makefiles. Another source tree used a single temporary file name for
397 all object files, so that they overwrote each other when run in
398 parallel.</p>
399 </div>
400 <div class="section" id="lifting-compilation">
401 <h2><a name="lifting-compilation">4.7   Lifting compilation</a></h2>
402 <p>distcc can be useful even without parallel builds. It is fairly
403 common for a developer to be working on a relatively slower machine:
404 perhaps on a laptop, or an embedded system, or perhaps a large server
405 machine is available. It is quite possible for the benefit in running
406 on a faster CPU to be greater than the overhead of running distcc and
407 transferring the build across the network.</p>
408 <p>Being able to do most of the work of compilation remotely makes it
409 feasible to do development entirely on embedded platforms that are
410 large enough to run an editor and linker, such as the <a class="reference" href="http://rikkus.info/zaurus.html">Sharp Zaurus</a>.</p>
411 <p>distcc support sociable computing by making it possible to do
412 development on quite large projects from a laptop, if compilation is
413 lifted off to one or more larger and noisier servers in another room.</p>
414 </div>
415 <div class="section" id="security">
416 <h2><a name="security">4.8   Security</a></h2>
417 <p>distcc is intended to be quite secure when used according to the
418 documentation.</p>
419 <p>Remote execution of compile jobs introduces a trust relationship
420 between the client and server machines. The client completely trusts
421 the server to compile code correctly. A corrupt server could
422 introduce malicious code into the results, or attack the client in
423 other ways. The server completely trusts an authorized client. A
424 malicious client could execute arbitrary commands on the server.</p>
425 <p>The distccd TCP server does not run as root. If started by root,
426 before accepting connections it gives away its privileges and changes
427 to either the <em>distcc</em> or <em>nobody</em> user, or to a user specified by the
428 <tt class="literal"><span class="pre">--user</span></tt> option.</p>
429 <p>The distcc client may be run as root. Compiling or doing other
430 unnecessary work as root is generally discouraged on Unix, but distcc
431 permits it because some build processes require it.</p>
432 <p>In TCP mode network transmissions are not encrypted or signed. An
433 attacker with passive access to the network traffic can see the source
434 or object code. An attacker with active access can modify the source
435 or object code, or execute arbitrary commands on the server.</p>
436 <p>The TCP server can limit clients by either checking the client
437 address, or listening only on particular IP interfaces. However, in
438 some circumstances an attacker can spoof connections from a false
439 address. On untrusted networks SSH is the only safe choice.</p>
440 <p>In SSH mode the server is started by each user under their own account
441 on the server. There is no long-lived daemon. Connections are
442 authenticated and encrypted. Every user is allowed only the
443 privileges that their normal account allows.</p>
444 <p>Because the server and client necessarily trust each other, there has
445 been no security audit of the code that runs after a connection is
446 established. It is possible that a hostile server could gain control
447 of a client directly, as well as modifying the object code.</p>
448 <p>Some people have proposed that the server should perform
449 reasonableness checks on the command or source code submitted from
450 clients. This is probably futile because gcc is not secure against
451 hostile input, and it might be possible for an attacker to gain
452 control of the gcc process through a carefully crafted input file or
453 command line. The only safe assumption is that anyone able to submit
454 jobs is able to execute arbitrary operations under the account used by
455 distccd.</p>
456 <p>Alternatively it has been suggested that client should check whether a
457 server is trustworthy by re-running compilations on a different
458 server. This approach can be useful in some distributed-computing
459 problem spaces such as SETI&#64;home, but is not practical for distcc.
460 Firstly, the damage from a single compromised object file is very
461 high, so every single file would have to be extensively checked.
462 Secondly, the performance constraints are such that building every
463 single file two or more times would often make distribution worthless.
464 Thirdly there would sometimes be false alerts from minor differences
465 in gcc patchlevel that are not visible in normal use but that do
466 change the output file at a byte level.</p>
467 <p>Only one security issue has arisen to date, which was that distcc used
468 a predictable temporary directory name. A local attacker by creating
469 that directory could cause distcc to fail, but no more damage.</p>
470 </div>
471 <div class="section" id="distcc-press-gangs">
472 <h2><a name="distcc-press-gangs">4.9   distcc press-gangs</a></h2>
473 <p>Systems have been developed which allow non-Unix machines to be
474 co-opted to contribute to compilation.</p>
475 <p>One such is <a class="reference" href="http://opendoorsoftware.com/cgi/http.pl?p=distccKNOPPIX">distccKNOPPIX</a>, which allows a machine to be rebooted and
476 to work as a compiler volunteer without needing any installation onto
477 its hard disk. The compiler and distcc server are loaded across http,
478 so that they can easily be matched to the versions on the client.</p>
479 <p>distcc packages are now available for <a class="reference" href="http://www.cygwin.com/">Cygwin</a>, allowing cross
480 compilers to be run on Microsoft systems.</p>
481 </div>
482 <div class="section" id="good-logging">
483 <h2><a name="good-logging">4.10   Good logging</a></h2>
484 <p>Most releases of distcc has discovered bugs in either distcc or other
485 software that could not be reproduced on the author's machine. It has
486 the honour of helping find a <a class="reference" href="http://www.mail-archive.com/distcc&#64;lists.samba.org/msg00871.html">serious bug in the Linux 2.5 TCP stack</a> by
487 being the first program to flood the network fast enough to provoke a
488 timer wraparound.</p>
489 <p>At the default logging level, distcc emits only warnings and errors,
490 such as when a compile host is unreachable or compilation fails.</p>
491 <p>When verbose logging is enabled, distcc reports the success or failure
492 of most non-trivial system calls and internal functions. Failure can
493 indicate either a problem on the machine, or a bug in distcc. In the
494 first case, the logs can help the user find out where the problem is.
495 If there was a bug in distcc, the trace helps work out where exactly
496 the failure occurred, what actions led up to that point, and what the
497 consequences were. Many of these bugs are at heart incorrect
498 assumptions about, for example, how distcc will be called or how a
499 system call behaves on different platforms. The trace often calls out
500 which assumptions were incorrect.</p>
501 <p>The goal of distcc's error handling is to have any problem cause at
502 most one failure. It ought to be possible to look at a verbose trace
503 and work out what went wrong and the fix.</p>
504 <p>Errors and trace messages from the server are written both into a log
505 file, and also into the compiler error stream passed back to the
506 client. This makes it more likely that if a problem occurs on the
507 server, it will be obvious to the remote user without their having to
508 look in the server's log file. This pattern is very useful, though it
509 might not be appropriate for software that handles requests from
510 untrusted users. Errors during server startup are written to error
511 output as well as to the log file, to make it more obvious to the
512 administrator that the server did not actually start.</p>
513 </div>
514 <div class="section" id="generalisations">
515 <h2><a name="generalisations">4.11   Generalisations</a></h2>
516 <p>There has been some interest in adapting distcc to compile other
517 languages such as Java or C#. These languages have quite different
518 mappings from source files to output, and in particular there is no
519 separate preprocessing phase. A Java compiler must read class files
520 as directed by <tt class="literal"><span class="pre">import</span></tt> statements, and those classes in turn may
521 need to be rebuilt from source. A distributed compiler for these
522 languages would be possible but might need to be more tightly
523 integrated with the compiler than distcc is.</p>
524 <p>distcc could be generalised still further to distribute arbitrary
525 non-interactive jobs, rather than just compilation. A good step
526 towards this would be to move all the intelligence about command lines
527 into the client, so that the server just managed temporary files and
528 executed specified commands. To date no compelling application for
529 this sort of distribution has presented. Scientific and technical
530 computing users seem reasonably well served by clustering and
531 batch-queueing software.</p>
532 </div>
533 <div class="section" id="changes-in-gcc">
534 <h2><a name="changes-in-gcc">4.12   Changes in gcc</a></h2>
535 <p>gcc version 3.3 includes some performance improvements that make
536 distcc less immediately useful.</p>
537 <p>One is precompiled headers, whereby the internal state of the compiler
538 is dumped out to disk after reading a set of common header files.
539 This is useful for programs that include a large and fixed set of
540 headers, where parsing the headers takes a large fraction of the
541 compile time. At the moment distcc can support this by putting the
542 <tt class="literal"><span class="pre">.pch</span></tt> state file on a shared filesystem. A better solution may be
543 added to distcc in the future.</p>
544 <p>gcc has also recently gained an integrated preprocessor option, to
545 avoid the overhead of running the preprocessor as a separate task.
546 Since distcc must run the preprocessor separately, it cannot benefit
547 from the integrated preprocessor and it's performance compared to gcc
548 is slightly worse.</p>
549 </div>
550 <div class="section" id="worse-is-better">
551 <h2><a name="worse-is-better">4.13   Worse is better</a></h2>
552 <p>distcc adheres to the &quot;<a class="reference" href="http://www.jwz.org/doc/worse-is-better.html">Worse is Better</a>&quot; philosophy, summarized by
553 Richard Gabriel as four points:</p>
554 <blockquote>
555 <ul class="simple">
556 <li><em>Simplicity: the design must be simple, both in implementation and
557 interface. It is more important for the implementation to be
558 simple than the interface. Simplicity is the most important
559 consideration in a design.</em></li>
560 </ul>
561 </blockquote>
562 <p>distcc does not require changes to the kernel, the build system, or
563 the compiler. It can be described in a couple of pages, and installed
564 in a matter of minutes. There is no central coordinating process or
565 global knowledge of the cluster's state.</p>
566 <p>The most prominent place where the implementation is less simple
567 than necesssary is in network IO, which is moderately optimized.
568 This is perhaps justifiable as network speed is important to the
569 program being useful.</p>
570 <p>Masquerading as gcc makes the implementation more complex, because
571 distcc must identify that it is masqueraded, and then make
572 appropriate adjustments to the path to invoke the real compiler.
573 However, masquerading makes the installation and use of distcc much
574 easier, and so it is a reasonable tradeoff. If it had turned out
575 to be hard to find a reliable and reasonably simple masquerading
576 algorithm, the feature might have been dropped.</p>
577 <p>distcc is written in C, which makes a correct implementation
578 slightly harder and more complex than it would be in a scripting
579 language. This is justified because to perform adequately, distcc
580 absolutely must have a very small startup cost, which is not the
581 case for any popular very-high-level language.</p>
582 <blockquote>
583 <ul class="simple">
584 <li><em>Correctness: the design must be correct in all observable
585 aspects. It is slightly better to be simple than correct.</em></li>
586 </ul>
587 </blockquote>
588 <p>There are no known cases where distcc generates an incorrect and
589 unexplained result. The edges cases such as debug information are
590 being fixed in a simple manner, by correcting gcc, rather than by
591 complicating distcc.</p>
592 <p>Almost all free software trees will build, with the restriction
593 that trees with concurrency bugs cannot be built in parallel.</p>
594 <p>When distcc encounters an unexpected condition, it tries to &quot;fail
595 safe&quot; by, for example, running the compilation locally, backing off
596 from a particular machine and avoiding accidental recursive
597 invocations.</p>
598 <blockquote>
599 <ul class="simple">
600 <li><em>Consistency: the design must not be overly inconsistent. Consistency
601 can be sacrificed for simplicity in some cases, but it is better to
602 drop those parts of the design that deal with less common
603 circumstances than to introduce either implementational complexity
604 or inconsistency.</em></li>
605 </ul>
606 </blockquote>
607 <p>Almost all the code in distcc is common across platforms, and most of
608 the code is reached on any particular execution: there are
609 relatively few special cases.</p>
610 <p>There are some special cases and inconsistencies in the way the
611 daemon starts up, in an attempt to make it more friendly to
612 administrators and as historical accidents.</p>
613 <p>distcc is very consistent with modern free Unix practice in details
614 such as responding to <tt class="literal"><span class="pre">--help</span></tt>, using the GNU autoconf system for
615 installation, and installing into standard directories.</p>
616 <blockquote>
617 <ul class="simple">
618 <li><em>Completeness: the design must cover as many important situations as
619 is practical. All reasonably expected cases should be
620 covered. Completeness can be sacrificed in favor of any other
621 quality. In fact, completeness must sacrificed whenever implementation
622 simplicity is jeopardized. Consistency can be sacrificed to achieve
623 completeness if simplicity is retained; especially worthless is
624 consistency of interface.</em></li>
625 </ul>
626 </blockquote>
627 <p>distcc is intentionally not a general-purpose clustering system.
628 They are too hard to write, and none that can be installed in a
629 matter of minutes. distcc has very incomplete coverage of all the
630 tasks people might want to distribute across machines, but it does
631 handle a large task that is important to a significant audience.</p>
632 <p>Within the space of C compilation, most tasks can be distributed.
633 Even if a few jobs during a software build cannot be distributed,
634 others run in parallel might be, so there is still an overall
635 benefit. distcc's documentation and diagnostic output make it
636 reasonably easy to identify problem commands.</p>
637 <p>Some gcc options that could in practice be distributed are run
638 locally for the sake of implementation simplicity. Commands that
639 produce assembly listings are one example: there is no reason why
640 the assembly output could not be relocated back to the client, but
641 this option is rarely used so the complexity is not justified.</p>
642 </div>
643 </div>
644 <div class="section" id="conclusions">
645 <h1><a name="conclusions">5   Conclusions</a></h1>
646 <p>distcc has a simple design and gains large performance improvements
647 for software builds. It is reliable and straightforward to install
648 and manage.</p>
649 <p>A few maxims about demonstration of software projects have been borne
650 out by distcc's development. Before development began, a feasibility
651 study demonstrated that the concept could work. The design is
652 intentionally simple and the scope modestly focused on a subset of
653 important problems. Strong logging helps get useful results from
654 distributed community testing. Performance features such as
655 compression and new network APIs were used selectively in places where
656 they are shown to increase distcc's speed.</p>
657 <p>As the world's body of source code continues to grow, distcc will help
658 get it built, tested and released more quickly and painlessly.</p>
659 <!-- LocalWords: distccKNOPPIX compilercache -->
660 </div>
661 </div>
662 </body>
663 </html>