Debian Codegraph @

doc/web/faq.html

raw
1 <html>
2 <head><title>distcc frequently asked questions</title>
3 <meta name="author" content="Martin Pool">
4 <meta name="keywords" content="distributed, make, build, gcc,
5 parallel, network, networked, faster, free, fast, paralel, cc, c,
6 C++, Objective C, ObjC, compile, compiler, compilation,
7 simple, GPL, GNU, Linux, BSD, Unix, software, development,
8 developer, TeamBuilder, quick, dmake, pvmake,
9 transparent, easy, workstations, Solaris, IncrediBuild,
10 productivity, speed, time, tool, performance, hardware,
11 agents, nfs, libraries, source, headers, workload,
12 buildfarm, farm, server, cluster, clustered, clustering, MOSIX, OpenMOSIX">
13 <meta name="description" content="distcc is a fast, free distributed C and C++ compiler.">
14 <link rel="SHORTCUT ICON" href="favicon.ico">
15 <link type="text/css" rel="stylesheet" href="distcc-green.css">
16 <link type="text/html" rel="top" href="index.html">
17 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"></head>
18 <body vlink="#003300" text="#000000" link="#003300" bgcolor="#ffffff" alink="#336633"><p><table width="100%"><tr><td valign="top"><div class="navbar"><ul>
19 <li><a href="index.html">distcc</a></li>
20 <li><a href="http://code.google.com/p/distcc/source/browse/trunk">source</a>
21 <li><a href="http://code.google.com/p/distcc/downloads/list">binaries</a></li>
22 <li><a href="http://distcc.googlecode.com/svn/trunk/NEWS">news</a></li>
23 <li><a href="scenarios.html">scenarios</a></li>
24 <li><a href="results.html">testimonials</a></li>
25 <li><a href="benchmark.html">benchmarks</a></li>
26 <li><a href="security.html">security</a></li>
27 <li><a href="distcc-lca-2004.html">whitepaper</a></li>
28 <li><a href="http://distcc.googlecode.com/svn/trunk/doc/web/man/">Man pages</a></li>
29 <li><a href="faq.html"><b>FAQ</b></a></li>
30
31 <p>
32 <li><a href="http://code.google.com/p/distcc/issues/list">report a bug</a></li>
33 <li><a href="https://lists.samba.org/mailman/listinfo/distcc">mailing list</a>
34 <ul><li><a href="mailto:[email protected]">post</a></li>
35 <li><a href="http://lists.samba.org/archive/distcc/">archive</a></li>
36 <li><a href="news://news.gmane.org/gmane.comp.compilers.distcc">newsgroup</a></li></ul>
37 </li>
38 <li><a href="http://freshmeat.net/projects/distcc">Freshmeat</a>
39 <ul><li><a href="http://freshmeat.net/subscribe/28140">be notified</a></li></ul>
40 </li>
41
42 <p>
43 <li>related:</li>
44 <ul>
45 <li><a title="How to use distcc and Gentoo" href="http://www.gentoo.org/doc/en/distcc.xml">Gentoo</a></li>
46 <li><a title="Script to ease building cross-compilers and toolchains" href="http://kegel.com/crosstool/">crosstool</a></li>
47 <li><a title="Cache compiler results" href="https://ccache.dev/">ccache</a></li>
48 <li><a title="Centralized control over distcc and ccache" href="http://ccontrol.ozlabs.org/">ccontrol</a></li>
49 <li><a title="Smart job scheduler" href="http://dmucs.sourceforge.net/">dmucs</a></li>
50 </ul>
51 <p><p></td>
52 <td valign="top">
53 <div class="body">
54 <h1 class="title">distcc frequently asked questions</h1>
55
56
57 <p><b>Question not answered here? Check the <a
58 href="http://lists.samba.org/archive/distcc/">mailing list
59 archives</A> or send email to <a
60 href="mailto:[email protected]">distcc (at)
61 lists.samba.org</A></b>
62
63
64 <h3>What compilers are supported?</h3>
65
66 <p>gcc is fully supported, building C, C++, Objective C and Objective
67 C++. Other gcc languages such as Java are not supported. All recent
68 versions of gcc are thought to work, though later versions tend to work
69 better.
70
71 <p>Intel's <em>icc</em> compiler is somewhat compatible with gcc and
72 works with distcc, but some problems have been reported.
73
74 <p>Sun's proprietary C compiler is reported to work with distcc in C
75 mode. It appears that Sun CC cannot compile C++ templates
76 correctly when using a separate preprocessor, so it is generally
77 not practical to compile C++ using Sun CC and distcc.
78
79 <p>
80 The main feature required by distcc is that the compiler must be able
81 to run the preprocessor separately, and then compile the preprocessor
82 output from a file. This was a basic part of the original design of
83 C, but some compilers seem to have lost the ability to do this.
84 However, this feature is not required if you use distcc's "pump" mode.
85
86 Secondarily, distcc is currently hardcoded to suit gcc's behaviour and
87 command-line syntax, so only compilers that act like gcc will work.
88 This could in principle be changed.
89
90
91 <h3>How to build gcc with distcc?</h3>
92
93 <p>
94 gcc uses an unusual three-stage process to verify that the compiler
95 can recompile itself with the same results. Because of bugs in the
96 gcc integrated preprocessor, compiling locally can produce a
97 program that is functionally identical, but not byte-for-byte
98 identical, with a program compiled remotely.
99
100 <p>
101 This can apparently be fixed by specifying <tt>127.0.0.1</tt> in the
102 host list, rather than <tt>localhost</tt>. This causes distcc to run
103 compilation "remotely" onto the same machine, which should be the
104 same as compiling remotely.
105
106 <p>
107 You will also need to make sure that the build directory is mounted
108 at the same location on all machines so that the new compiler can
109 be located.
110
111 <p>
112 (More information or instructions here would be welcome.)
113
114
115
116 <h3>distcc gets slower when I add slow machines to the cluster</h3>
117
118
119
120
121 <p>
122 Make sure you put the preferred (fastest/closest/least loaded)
123 machines at the start of the <tt>DISTCC_HOSTS</tt> list. This is
124 particularly important when running <tt>./configure</tt> scripts
125 because all compilation will be done on the first machine listed.
126 Normally this should be <tt>localhost</tt>, but if another machine is
127 much faster then perhaps not.
128
129 <p>
130 To some extent this is still an open bug that I hope to address in
131 a future version. It would be nice if distcc could automatically
132 detect the best distribution, but it doesn't do that yet.
133
134
135 <h3>Restarting distccd on reboot</h3>
136
137
138 <p>
139 You may have <tt>distccd</tt> installed on a machine where you don't
140 have root, and want it to restart when the machine reboots.
141
142 <p>
143 One way to do this, <a href="http://lists.samba.org/pipermail/distcc/2002q4/000417.html">suggested by Shane McDaniel</a>,
144 is to put it in
145 your per-user <tt>crontab</tt>. <tt>distccd</tt> will be started at
146 regular intervals, but will exit if something is already listening
147 on the port. Remember to set your <tt>PATH</tt> in the crontab so
148 that <tt>distccd</tt> and all necessary compilers can be found.
149
150
151
152 <h3>Choosing a userid</h3>
153
154 <blockquote>
155 It would be awesome if you could run the daemon as a specific
156 user in daemon mode.
157 </blockquote>
158
159 <p>
160 It is awesome! :-) From distcc 1.1, root can use the <tt>--user</tt>
161 option to cause distcc to start as a particular user.
162 <p>
163 <tt># distccd --user nobody </tt>
164
165 <p>
166 If distccd is started by root and no user is specified, it will
167 change to the user <tt>distcc</tt> if possible, or otherwise to
168 <tt>nobody</tt>. I recommend you create this user when installing the
169 package.
170
171 <p>
172 If distccd is started by a non-root user then it will continue
173 running as that user.
174
175
176
177
178 <h3>Why not use a distributed-make-style system?</h3>
179
180
181
182
183
184 <p>
185 A few people have extended <tt>make</tt> to allow it to distribute jobs
186 across several machines. Projects I know of include Graydon Hoare's
187 <a href=http://www.venge.net/graydon/cgi-bin/viewcvs.cgi/src/doozer/">Doozer</a>,
188 Sun's <tt>dmake</tt>, <tt>pvmmake</tt>, and <tt>ppmake</tt>. GNU Make apparently has
189 internal hooks to add distribution mechanisms, which I think is how
190 <tt>ppmake</tt> and <tt>pvmmake</tt> work.
191
192 <p>
193 Unlike distcc, most of programs have no special knowledge of C: they
194 just schedule jobs remotely or locally. The advantage of this is
195 that you can distribute all different jobs, such as linking, or
196 building documentation, or compiling programs in other languages.
197
198 <p>
199 The disadvantage of this approach is because it relies on running
200 tasks on any node, all relevant aspects of the nodes must be the
201 same. This typically means that all machines must have a shared
202 filesystem mounted at the same location, that they must all have
203 exactly the same compiler, headers and libraries installed, their
204 clocks must be in sync, and typically that they must all have the
205 same OS and CPU architecture.
206
207 <p>
208 In some situations, such as a lab of centrally managed machines,
209 this is quite practical. However, many people have a less
210 homogeneous environment: perhaps some machines run a different OS
211 release, or developers are allowed to upgrade libraries on their own
212 machines, or perhaps you just don't want to run NFS.
213
214 <p>
215 In this situation distcc is much easier to set up. You don't even
216 need root on the volunteer machines, let alone a mandate to move
217 <tt>/home</tt> onto NFS.
218
219 <p>
220 (You might get away with having slightly different headers or
221 libraries, but the potential for confusion is so great that I think
222 you'd be crazy to try.)
223
224 <p>
225 By Amdahl's law, a distributed make system could in principle be
226 faster than distcc, because it can distribute many different jobs.
227 In practice, however, for many projects compiling C or C++ takes
228 over 80% of the time. Many of the other jobs, such as linking,
229 cannot be parallelized anyhow.
230
231 <p>
232 I don't know how their performance compares but I would be interested
233 to hear.
234
235
236 <h3>Has anybody yet thought of integrating distcc with ccache?</h3>
237
238 <p>
239 If you don't use distcc's "pump" mode, then
240 they work pretty well as separate programs that call each other.
241 You can either set <tt>CC='ccache distcc gcc'</tt>, or arrange for
242 both ccache and distcc to be "masqueraded" on the path. (See the
243 manual for information on how to install this.)
244
245 <p>
246 Normally it is better for ccache to be run before distcc.
247
248 <p>
249 This is very nearly as efficient as making them both part of a
250 single program. The preprocessor will only run once and the
251 preprocessed source will be passed from ccache straight to distcc.
252 Having them separate allows them to be tested, debugged and
253 released separately.
254
255 <p>
256 Unless there's a really strong argument to do otherwise, having two
257 smaller programs is better practice, or at least more to my taste,
258 than making one monolithic one.
259
260 <p>
261 However, all of that said, now that we have "pump" mode, the
262 trade-offs have changed. Using ccache prevents the use of "pump"
263 mode. It would make sense to integrate caching into distcc
264 so that you can get distributed preprocessing, distributed
265 compilation, and caching all at the same time. This would make
266 a great project for someone...
267
268 <p>
269 Also, it would be nice to allow a cache of
270 compiled files to be shared across several users or machines.
271
272 <p>
273 Joerg Beyer started a project called <a href="http://gecc.sourceforge.net/">gecc</a>
274 to explore this architecture, but development appears to have stalled in 2002.
275
276 <h3>Temporary files in strange location?</h3>
277 <p>
278 <blockquote>
279 distcc always tries to create subdirs in /root/tmp for its temp files. How do I get around
280 this?
281 </blockquote>
282
283 <p>
284 distcc respects the <tt>$TMPDIR</tt> environment variable when creating its
285 scratch directory. I suspect you have that set in root's .profile.
286 If you unset it in the shell script that launches xinetd, or set it to
287 something not in root's path, then it should be fine.
288
289
290
291 <h3>Server dies after a few connections when run from inetd</h3>
292
293
294 <p>
295 <blockquote>
296 i noticed that when you start a make process, the remote distccd receives
297 10-15 connections from my host (which is good) and those processes die after
298 a while. this is a long compilation (kdelibs) and after the initial
299 processes die, i cannot see any new one's coming in. is this normal ?
300 it seems as if after a while (2-3 minuets) distcc stops working and only my
301 local gcc is still compiling.
302 </blockquote>
303
304
305 <p>
306
307 If you're running distccd from inetd, then it may be that inetd
308 thinks that the service is "looping" because of all the rapid
309 connections. You need to increase the maximum connection rate.
310 See the inetd manual.
311
312 <p>
313 On traditional BSD inetd, you can do this by changing the word <tt>t</tt> to something like <tt>nowait.1000</tt>
314
315 <p>
316 Alternatively, run distccd with <tt>--daemon</tt>, rather than from
317 inetd.
318
319
320
321
322 <h3>Running distccd on a firewall?</h3>
323
324
325 <p>
326 <blockquote>
327 another machine i want to add to this "cluster" is my firewall. its not a
328 very powerful one, but it can help speed things up a little more ;)
329 are there any knows security issues with distcc ? how stupid will it be to run
330 it on a server that acts as a firewall ?
331 </blockquote>
332
333
334
335 <p>
336 It depends on your security profile, but it's not completely
337 unreasonable. Hopefully your firewall already has iptables and
338 tcpwrappers protection against connections from the outside world.
339 Just make sure that nobody else can connect to the distccd port.
340
341
342
343 <h3>What -j level to use?</h3>
344
345 <p>
346 <blockquote>
347 when starting a compilation, the howto says to use -j8 .
348 is this optimal ? anything else i should be using for better
349 performance ?
350 </blockquote>
351
352 <p>
353 For plain (that is, non-pump) mode, you should use about twice the total
354 number of CPUs available, but it depends on your network, program being
355 compiled, available memory, etc. Experiment with different values.
356
357 <p>
358 Client machines tend to be saturated at about 10-20 jobs, so using
359 values above -j10 is rarely useful. (Perhaps higher levels would work
360 well on clients with two or more CPUs and very fast network
361 connections, memory and disk.)
362
363 <p>
364 The advice above is all for plain mode; for "pump" mode, you
365 may see benefit from higher -j values. Please experiment and
366 <a href="mailto:[email protected]">let us know</a> what works best.
367
368
369 <h3>Should I include localhost in the host list?</h3>
370
371 <p>
372 Naturally, compiling on the machine that drives
373 compilation has low overhead.
374
375 <p>
376 But, if a large number of machines are part of the
377 distcc "farm", I suspect most of the driving machine's
378 time would be better spent doing preprocessing &amp;
379 feeding only, and as such increase the chances of
380 always having something ready to handle to machines
381 that finish their distcc compilation jobs.
382
383 <p>
384 I'm just hoping to collect some opinions on this
385 matter. For a large project and for a cluster of about 8
386 machines, is it actually better to dedicated the
387 driving machine to do preprocessing only? (and so not
388 include it in the hosts file).
389
390 <p>
391 It will depend on your source tree, your network, your
392 compiler, and your makefile (or alternative) just what fraction of the
393 work must be done locally, which is the most important thing here. At
394 about the 3-4 machine level it may be worth putting localhost last; at
395 8-10 machines it may be better to leave it out altogether.
396
397
398 <h3>Different gcc versions?</h3>
399
400 <p>
401 <blockquote>
402 if the host machine is using gcc 3.2, can the other machines use an older
403 version of gcc ? 2.9.x for example ?
404 </blockquote>
405
406
407 <p>
408 distcc doesn't care. However, in some circumstances, particularly
409 for C++, gcc object files compiled with one version of gcc are not
410 compatible with those compiled by another. This is true even if
411 they are built on the same machine.
412 <p>
413 It is usually best to make sure that every compiler name maps to a
414 reasonably similar version on every machine. You can either make
415 sure that <tt>gcc</tt> is the same everywhere, or use a
416 version-qualified compiler name, such as <tt>gcc-3.2</tt> or
417 <tt>i386-redhat-linux-gcc-3.2.2</tt>.
418 <p>
419 In particular, object files can be incompatible because:
420 <ol>
421 <li> Calling conventions or ABIs have changed, particularly for
422 C++.
423
424 <li> Different versions optimize differently.
425
426 <li> The header files specifically detect the version of gcc being
427 used, to change optimization or to work around bugs. The
428 Linux kernel does this.
429 </ol>
430 <p>
431 In all these cases you would need to "make clean" if you upgraded
432 your gcc.
433 <p>
434 It is hard to generalize, but using gcc versions which have the
435 first two components the same (e.g. 3.2.1 and 3.2.2) is usually OK.
436 To be safe, use the exact same release on all machines.
437 <p>
438 It is also a good idea to have the assembler versions be the same
439 too, although the problems there are slightly less complex.
440
441
442
443
444 <h3>Shouldn't distcc check gcc versions?</h3>
445
446
447 <p>
448 It might be a good idea for distcc to check the version of gcc on
449 all the volunteer machines. However this turns out to be hard to do
450 completely reliably, because some programs which both call themselves
451 "gcc 3.2.1" behave differently, presumably because of vendor
452 patches or because vendors have shipped pre-release code.
453 <p>
454 So since automatic detection would not be a reliable solution, for
455 the moment we depend on the user to make sure they have compatible
456 versions installed.
457
458
459
460
461 <h3>Using different platforms?</h3>
462 <p>
463 <blockquote>
464 I have a mixed network environment. MacOS X, Linux, and Windows (with
465 cygwin). Seeing distcc really piqued my interest. Using gcc on all of my
466 environments, can I set up distcc to span across this variety of operating
467 systems?
468 </blockquote>
469 <p>
470 It should be reasonably straightforward. Of course you will need to
471 either install or build appropriate cross compilers for each machine.
472 <p>
473 For example, on each volunteer machine, build an x86-linux cross
474 compiler and (this is important) install it as "<tt>i386-redhat-linux-gcc-3.2.2</tt>" or
475 something similar, using the appropriate gcc configuration options to
476 set the name. You also need to make a link to that name on the Linux
477 machine.
478 <p>
479 Then from Linux, run "<tt>distcc i386-redhat-linux-gcc-3.2.2</tt>".
480 <p>
481 Repeat as appropriate for every combination you want to use.
482 <p>
483 On Windows, you need to use the Cygwin (or perhaps Mingw?)
484 software, which provides a Unix-like environment for running gcc.
485 <p>
486 The easiest way to build a cross compiler is to use
487 Dan Kegel's
488 <a href="http://kegel.com/crosstool/">crosstool</a>.
489 <p>
490 Brian Mosher reports that
491 <a href="http://www.xraylith.wisc.edu/~khan/software/gnu-win32/cygwin-to-linux-cross-howto.txt">Mumit Khan's description of building a Cygwin-hosted Linux toolchain</a> works well.
492 <a href="http://lists.samba.org/pipermail/distcc/2002q4/000469.html">Mike Santy suggests a slightly different approach</a>.
493
494
495
496 <h3>Compiling between different i386 Unixes?</h3>
497 <p>
498 <blockquote>
499 Do I need to install cross-compilers to distribute builds between
500 different operating systems on the same CPU architecture? For
501 example, <a href="http://groups.google.com/groups?hl=en&amp;lr=&amp;ie=UTF-8&amp;oe=UTF-8&amp;threadm=opro4i72qrekdofe%40news.lysator.liu.se&amp;rnum=3&amp;prev=/groups%3Fq%3Ddistcc%2Bopenbsd%26ie%3DUTF-8%26oe%3DUTF-8%26hl%3Den%26btnG%3DGoogle%2BSearch">OpenBSD i386 and NetBSD i386</a>.
502 </blockquote>
503
504 <p>
505 In general, yes you do. Even on the same system, there can be
506 incompatibilites between the output of compilers such as
507
508 <ul>
509 <li> Different object file formats, or variations of the format.
510 (OpenBSD uses a.out, but most Unixes use ELF.)
511 <li> Different patches applied to the compiler, even if it claims
512 to be the same version.
513 </ul>
514
515 <h3>What's hard about synchronizing clocks?</h3>
516
517 <p>
518 <blockquote>
519 Why do you say it's a feature of distcc that the machine clocks
520 don't need to be synchronized? It's easy to do: you just install an
521 NTP client.
522 </blockquote>
523 <p>
524 It's not terribly hard, I agree. But it's not quite as trivial as
525 you might think:
526 <p>
527 <ol>
528 <li> Installing an NTP client requires root access on <em>all</em> the
529 machines, not just your own workstation. (There's no way around
530 it, because it needs to change the machine's clock.)
531 <li> You need to be able to reach a reliable timeserver, which is
532 sometimes a problem with firewalls.
533 <li> If one of the machines does get out of sync, then builds may
534 be incorrect or you may get errors. You may not notice until
535 after problems have occurred. You can't fix it without
536 intervention from the administrator, who may be on holidays.
537 </ol>
538 <p>
539 What's not there can't break.
540
541
542 <h3>What does "listening on 0.0.0.0:3632" mean?</h3>
543 <p>
544 This message means it's listening for connections on a wildcard IP
545 address, so clients coming through any network interface should be
546 able to connect, subject to <tt>--allow</tt> rules.
547 <p>
548 If you have more than one interface and want to only allow
549 connections from one of them then use the <tt>--listen</tt> or <tt>w</tt> options, or both.
550
551
552
553
554
555 <h3>How can I use SSH connection multiplexing?</h3>
556
557 <p>
558 <blockquote>
559 Can I use ssh connection sharing to reduce the overhead of opening SSH connections?
560 </blockquote>
561 <p>
562 Yes. Using SSH connection sharing can reduce the overhead of establishing SSH connections by a factor of 10.
563 <p>
564 Make sure you have a recent SSH (only on client side). I think you need OpenSSH 4 or later.
565 <pre>
566 $ ssh -V
567 OpenSSH_4.6p1 Debian-5ubuntu0.1, OpenSSL 0.9.8e 23 Feb 2007
568 </pre>
569
570 <p>
571 Create a file <code>~/.ssh/config</code>, and add this:
572 <pre>
573 Host *
574 ControlMaster auto
575 ControlPath ~/.ssh_tmp/master-%r@%h:%p
576 </pre>
577
578 <p>
579 Then create the master SSH connection:
580 <pre>
581 ssh -fMN <var>hostname</var>
582 </pre>
583
584 <p>
585 Subsequent connections to that host will now fly!
586
587 <p>
588 For best results, create master SSH connections for each host in your distcc host list.
589
590 <p>
591 For more information on ssh connection sharing,
592 see <a href="http://www.linux.com/feature/54498">here</a>.
593
594 <h3>Will compiling with distcc magically make my program distributed?</h3>
595 <p>
596 <blockquote>
597 All objects
598 files were created , but when I issue command for execution as
599 <tt>./executable</tt>
600 then program takes so much time as without distcc. Kindly tell me that
601 how this execution time can be reduced.
602 All things work Properly , But all processors of systems are not utilized
603 during EXECUTION of programs.
604 </blockquote>
605 <p>
606 (This has to be the wierdest question in the list, but it really was sent.)
607 <p>
608 distcc makes building the program take less time. The program that
609 is produced is the same as for a local build.
610 <p>
611 If you want your program to be distributed at run time across several
612 machines, you need to design in parallelism and distribution,
613 possibly using a framework such as MPI or OpenMOSIX.
614 <p>
615 What you seem to be asking for is to mechanically transform an
616 arbitrary C program into a distributed parallel program. distcc
617 doesn't do anything like that. It is perhaps not quite impossible
618 (people have written parallelizing compilers) but it's very very
619 hard.
620
621 <h3>make-kpkg with distcc?</h3>
622 <p>
623 <blockquote>
624 How to compile the debian kernel
625 with make-kpkg using distcc?
626 </blockquote>
627
628 <p>
629 Use the <tt>CONCURRENCY_LEVEL</tt> environment variable eg
630 <tt>make-kpkg kernel_image CONCURRENCY_LEVEL=4</tt>
631 after installing distcc into a masquerade directory.
632
633
634 <h3>How to use an SSH TCP port other than 22?</h3>
635 <p>
636 <blockquote>
637 I want to use two machines over ssh. One machine has a different port than
638 22. But I can't set a port in the host specifications for ssh.
639 </blockquote>
640 <p>
641 Use something like this in your <tt>~/.ssh/config</tt>:
642 <p>
643 <tt>
644 Host bertie
645 <br>Port 2202
646 </tt>
647
648 <h3>How do I build a cross compiler to Mac OS X?</h3>
649
650
651 <p>
652 Dara Hazeghi says:
653 <p>
654 <blockquote>Basically, you have to download the same source
655 version compiler as the one on your OS X box, build it
656 on you Linux PC, etc. The assembler now works on Linux
657 as well, so that other issue you had should be moot.
658 <p>
659 That is, you should compile the Apple source on your non-Apple
660 machine. Apple have some patches that are not in the upstream gcc
661 release so a compiler built from the gnu.org source will not be
662 compatible.
663 </blockquote>
664
665
666 <h3>Randomly patched kernel crashes/hangs</h3>
667 <p>
668 <blockquote>
669 I installed applied some
670 <a href="http://bugs.gentoo.org/show_bug.cgi?id=36230">random</a> <a href="http://lists.samba.org/archive/distcc/2004q1/002035.html">unstable</a> kernel patches, and now my machine hangs/crashes/corrupts data when I run distcc!
671 </blockquote>
672
673 <p>
674 If your kernel crashes, it is
675 <a href="http://lists.samba.org/archive/distcc/2004q1/002040.html">by definition</a>
676 a kernel bug.
677 <p>
678 If you have applied any patches that are not in the <tt>kernel.org</tt>
679 stable release, your first step should be to back them out and see
680 if the problem is still reproducible.
681
682
683
684 <h3>distccd sometimes unable to find gcc</h3>
685 <p>
686
687 distccd inherits its <tt>PATH</tt> from whichever process starts it,
688 and uses this to find compilers. You may need to make sure the <tt>H</tt> is set appropriately by the script that starts distccd. You
689 can also set <tt>DISTCCD_PATH</tt>, which overrides <tt>PATH</tt> and
690 bypasses checks for distcc masquerade directories. The daemon's path
691 is logged when <tt>--verbose</tt> is given.
692
693
694 <h3>Error about "jobserver unavailable"</h3>
695
696
697 <p>
698 This indicates a problem with your Makefile or shell. See the <a href="http://mail.gnu.org/archive/html/help-make/2001-09/msg00005.html">explanation of this error</a> from GNU make.
699 <p>
700 This can also happen when make is unable to create the fifo it uses
701 to communicate among parallel processes, because TMPDIR is not
702 accessible. Check that the variable is set properly (or unset), and
703 that the directory has the right permissions.
704
705
706 <h3>Files written to NFS filesystems are corrupt</h3>
707 <p>
708 <blockquote>
709 <a href="http://lists.samba.org/pipermail/distcc/2003q3/001556.html">List post</a>: Writing object files from distcc to an NFS directory
710 can cause corrupt output: object files will be full of zeros.
711 </blockquote>
712 <p>
713 This is a bug in the Linux kernel NFS client interaction between
714 <tt>mmap</tt> and <tt>rename</tt>. It can be avoided by using the
715 <a href="http://article.gmane.org/gmane.linux.nfs/3578"><tt>no_subtree_check</tt></a> export option on the NFS server.
716 distcc 2.18 no longer uses mmap to receive files and may not
717 suffer this bug so strongly, but setting <tt>no_subtree_check</tt> is
718 still recommended.
719
720
721 <h3>distccmon doesn't work on NFS</h3>
722 <p>
723 <blockquote>
724 When I set DISTCC_DIR to a directory on an NFS server, or have HOME on
725 an NFS server, the monitors give errors or just don't work.
726 </blockquote>
727 <p>
728 The monitor checks that the processes recorded in the state files
729 are actually running (using kill -0). If the processes aren't
730 running on the same machine, this doesn't work.
731 <p>
732 In any case, having DISTCC_DIR on NFS is likely to cause problems
733 with locking. Please set DISTCC_DIR to a local directory instead.
734
735
736 <h3>gcc's DEPENDENCIES_OUTPUT option is broken</h3>
737 <p>
738 Programs that use gcc's <a href="http://lists.samba.org/pipermail/distcc/2003q3/001547.html">DEPENDENCIES_OUTPUT</a> option don't work with ccache.
739 <p>
740 This should be fixed in <a href="https://ccache.dev/">ccache 2.3</a>. There is no problem with distcc.
741
742
743 <h3>distcc fails to build on OS X</h3>
744 <p>
745 <blockquote>
746 I'm trying to compile Distcc on OS X (gcc 3.1) and it fails with the
747 following:
748 </blockquote>
749 <p>
750 <tt>
751 gcc -DHAVE_CONFIG_H -D_GNU_SOURCE -I./popt -I./src
752 "-DSYSCONFDIR=\"/usr/local/etc\"" -g -O2 -W -Wall -W -Wimplicit
753 -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings
754 -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes
755 -Wnested-externs -o src/climasq.o -c src/climasq.c
756 <br>src/climasq.c:69: only 1 arg to macro 'rs_log_error' (2 expected)
757 <br>src/climasq.c:114: only 1 arg to macro 'rs_trace' (2 expected)
758 <br>cpp-precomp: warning: errors during smart preprocessing, retrying in
759 basic mode
760 <br>make: *** [src/climasq.o] Error 1
761 </tt>
762
763 <p>
764 There is a bug in some versions of Apple's gcc. Building with
765 <tt>CFLAGS="-no-cpp-precomp"</tt> should fix it. It should be automatically
766 corrected in distcc 2.8 and later.
767
768
769
770 <h3>Problems with <tt>gcc -MD</h3></tt>
771 <p>
772 <blockquote>
773 I'm using 0.12, and having trouble with dependency file creation. The
774 makefiles that are set up to use gcc's -MD option aren't working. I'm
775 getting some of the .d files in their proper directory, and not others.
776 </blockquote>
777 <p>
778 This problem can only occur if you're using gcc 3.0 or later, have
779 the source and object files in different directories or under
780 different names, and you are using <tt>-MD</tt> or <tt>-MMD</tt> but
781 not <tt>-MF</tt>.
782 <p>
783 The workaround is to change the Makefile to explicitly specify with
784 <tt>-MF</tt> the file which should receive the dependency information.
785 Many Makefiles already do this. Note that the <tt>-MF</tt> option is
786 only available in gcc 3.0 and later.
787 <p>
788 Because the behaviour of <tt>-MD</tt> has changed from gcc2 to gcc3
789 there is no perfect solution available at the moment.
790
791
792
793 <h3>gdb can't find source files</h3>
794 <blockquote>
795 When I try to debug with gdb an executable compiled with distcc, gdb doesn't
796 find the source of the object to be debugged, unless that source is in the
797 directory from which I start gdb.
798 </blockquote>
799 <p>
800
801 Unfortunately this is caused by a bug in gcc, which I hope will be
802 fixed in a future release. gcc embeds the directory where the
803 compiler (cc1) was run, when it really ought to record the directory
804 the source came from.
805 <p>
806 You can work around it for now by using the "<tt>directory</tt>" command
807 in gdb to tell it where to find the source, or by passing an absolute
808 file name when compiling.
809 <p>
810 Tim Janik has an <a href="http://www.gtk.org/~timj/patches/">unofficial patch</a> for distcc which works around this but I think I
811 won't merge it because it's better to fix it in gcc.
812 <p>
813 This is Debian <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=148957&amp;repeatmerged=yes">#148957</a>.
814 <p>
815 There was a <a href="http://gcc.gnu.org/ml/gcc-patches/2002-08/msg01411.html">discussion about this bug</a> on the gcc-patches mailing list.
816 This can affect other programs which rely on debug stabs, such as
817 <tt>addr2line</tt>, and it results in object files not being
818 byte-for-byte identical when they include the source directory. The
819 same bug affects <a href="https://ccache.dev/">ccache</a>.
820
821 <h3>TCP_CORK in linux-2.2</h3>
822 <p>
823 Linux 2.2 has a bug to do with <tt>TCP_CORK</tt> sockets getting stuck in
824 the <tt>FIN_WAIT1</tt> state. distcc 0.10 tries to work around it but it
825 is not completely possible; if it causes trouble set
826 <tt>DISTCC_CORK=0</tt>.
827
828
829 <h3>Hung sockets in Linux 2.5</h3>
830 <p>
831 distcc seems to produce a problem in Linux 2.5 where one machine
832 thinks the socket is CLOSED, and the other thinks it is
833 ESTABLISHED. As a result the transfer hangs.
834 <p>
835 The CONNECTED/ESTABLISHED state should *never* be reachable by a correct TCP
836 implementation. distcc is triggering a bug in the 2.5 TCP stack,
837 which was <a href="http://groups.google.com/groups?hl=en&amp;lr=&amp;ie=UTF-8&amp;oe=UTF-8&amp;threadm=20030604050011%242b4f%40gated-at.bofh.it&amp;rnum=1&amp;prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26q%3Ddistcc%2B%2Bdavem%26btnG%3DGoogle%2BSearch">found and fixed</a> in 2.5 in June 2003.
838
839
840 <h3>libtool and trouble building KDE</h3>
841 <p>
842 Trying to build KDE with <tt>make CC=distcc CXX=distcc -j5</tt> fails
843 with a libtool error.
844 <p>
845 The version of libtool included with some KDE releases is buggy.
846 <p>
847 The best workaround is to install distcc in "masquerade mode".
848 Wayne Davison writes:
849 <blockquote>
850 <p>
851 To get distcc going, just follow the instructions in the docs to setup a
852 "masquerade" dir, add it to the start of your PATH, and then never
853 fiddle with CC and/or CXX again (i.e. undefine them). Note that you
854 need to be running a 2.x version of distcc for masquerade support to be
855 integrated by default (e.g. 2.0.1). (Side note: if you're using a
856 binary RPM of distcc, make sure that the maintainer has made a
857 masquerade dir a part of the installed config. If not, ask them to do
858 so, as it is the easiest way to make distcc compatible with the widest
859 range of packages.)
860 <p>
861 To be explicit, do something very similar (or identical) to this:
862 <p>
863 <pre>
864 # mkdir -p /usr/lib/distcc/bin
865 # cd /usr/lib/distcc/bin
866 # ln -s ../../../bin/distcc cc
867 # ln -s ../../../bin/distcc c++
868 # ln -s ../../../bin/distcc gcc
869 # ln -s ../../../bin/distcc g++
870 </pre>
871 <p>
872 You can add links to any other compiler names you use on your system as
873 well.
874 <p>
875 The other setup requirements remain unchanged: add DISTCC_HOSTS in your
876 environment and run make with the -j5 (or whatever) option, perhaps by
877 setting MAKEFLAGS in your environment.
878 </blockquote>
879
880
881 <h3>Linux 2.2 makefile strangeness</h3>
882 <p>
883 Linux 2.2.21 cannot be built with distcc because of bugs in the kernel's
884 Makefile. (<tt>$CC</tt> is set to the compiler name plus the computed
885 options.) It may be possible to use distcc in masquerade mode.
886
887
888 <h3>Compiler command line parsing problems?</h3>
889
890
891 <p>
892 distcc parses the compiler command line to work out what
893 operation is being invoked, and what are the input and output
894 files. The semantics of <tt>gcc</tt> command lines is fairly complex,
895 and has changed slightly over time. There may be some valid
896 command lines that distcc understands differently to gcc, though
897 none are known at the moment. These ought to cause the command to
898 be run locally, but it is possible that it would cause a failure.
899 Either case should be reported as a bug.
900
901 <h3>Can't handle local file access</h3>
902
903 <p>
904 distcc can't handle compilers that need to read other files from
905 the local filesystem. This might be a problem with such things as
906 profile-directed optimizers. distcc tries to detect such commands
907 and run them locally, but there may be cases which are not handled
908 properly.
909
910 <h3>Huge files</h3>
911 <p>
912 distcc's protocol and file IO would probably have trouble with
913 source or object files over 2GB in size. I've never heard of a .c
914 or .o file that large, and I rather suspect gcc would not handle
915 them well either.
916
917
918 <h3>KDE builds slowly with <tt>--enable-final</tt></h3>
919 <p>
920 Bernardo Innocenti says
921 <p>
922 <blockquote> Using the --enable-final configure option of KDE makes distcc almost
923 useless.
924 </blockquote>
925 <p>
926
927 Frerich Raabe explains:
928 <blockquote>
929 --enable-final makes the build system concatenate all sourcefiles in a
930 directory (say, Konqueror's sourcefiles) into one big file.
931 <p>
932 Technically, this is achieved by creating a dummy file which simply
933 includes every C++ sourcefile. The advantage of this is that the
934 compile a- takes less time since there is only little scattered file
935 opening involved and b- produces usually more optimized code, since
936 the compiler can see more code at once and thus there are more
937 chances to optimize. Of course this eats a lot more memory, but that
938 is not an issue nowadays.
939 <p>
940 Now, it's clear why this makes distcc useless: there is just one huge file
941 per project, and outsourcing that file via distcc to other nodes will just
942 delay the build since the sourcecode (and it's a lot) has to be transferred
943 over the network, and there is no way to pararellize this.
944 <p>
945 To avoid this, configure with <tt>--disable-final</tt>.
946 </blockquote>
947 <h3>C++ code that uses <tt>#pragma implementation</tt> doesn't seem to
948 work properly.</h3>
949 <p>
950 That pragma can't work with distcc because it introduces
951 dependencies between the source and local filenames. It is
952 strongly deprecated in gcc and there are no plans to support it in
953 distcc.
954 <h3>Data corruption in compiles</h3>
955 <p>
956 Typical symptoms include linker errors, weird syntax errors or
957 compiler crashes, caused by corrupt object or source files.
958 <p>
959 This has occurred several times in the past, typically because of
960 kernel bugs such as
961 Gentoo <a href="http://bugs.gentoo.org/show_bug.cgi?id=36230">#36320</a>.
962 See if you can reproduce the problem using a standard kernel.org
963 kernel.
964
965
966 <h3>Linux kernel panic using gigabit ethernet</h3>
967 <p>
968 There is a
969 <a href="http://lkml.org/lkml/2004/5/3/192">bug in the Linux 2.4.26 kernel</a>
970 [<a href="http://groups.google.com/groups?hl=en&amp;lr=&amp;ie=UTF-8&amp;threadm=1Gu6a-7hD-11%40gated-at.bofh.it&amp;rnum=1&amp;prev=/groups%3Fq%3Ddistcc%2520panic%26meta%3Dsite%253Dgroups">Google groups thread</a>] that is triggered by using distcc. It is timing-related and seems
971 more likely to occur with a gigabit network adapter.
972 <p>
973 <a href="http://groups.google.com/groups?hl=en&amp;lr=&amp;ie=UTF-8&amp;selm=1Qr7f-5aV-31%40gated-at.bofh.it&amp;rnum=23">This patch may fix the problem.</a> If it doesn't, please report your
974 problem to the kernel mailing list.
975 <p>
976 You may be able to avoid it by setting these environment variables
977 on both the client and the server:
978 <p>
979 <tt>DISTCC_MMAP=0 DISTCC_SENDFILE=0 </tt>
980 <p>
981 This bug is present in the Fedora Core 1 kernel and is
982 <a href="https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=114192">Red Hat bug #114192</a>.
983 <h3>Sun Workshop CC does not work with distcc</h3>
984 <p>
985 <blockquote>
986 I am trying to use distcc on a network of Solaris 2.7 workstations and I
987 also ran into this problem. I modified the src just like you did but I
988 still have problems compiling certain C++ files - specifically those that
989 include STL headers.
990 <p>
991 Has anyone figured out how to prevent the re-inclusion of the headers?
992 </blockquote>
993
994 <p>
995 P. Christeas says:
996 <p>
997 <blockquote> After some time, the use of distcc with SunWS seems an unresolved issue. Of
998 course, I exclude any attempt to modify the STL headers (which in fact are
999 Sun's version of STL headers).
1000 The correct statement is that _Sun's CC does not behave correctly when the
1001 preprocessor is involved as a separate process_.
1002 <p>
1003 This compiler should still be marked as 'unsupported'. The only way (I know
1004 of) to make some use of a compile farm is to exclusively mark the few files
1005 that *do* compile, and send those through distcc.
1006 Hint: it seems that templates and CORBA implementation source do generally
1007 break the precompiler.
1008 </blockquote>
1009 <p>
1010 So <a href="compilers.html">Sun CC will not work for C++</a> with distcc, only for
1011 C.
1012
1013
1014 <h3>configure has trouble when hosts are down</h3>
1015 <p>
1016 <blockquote>
1017 I'm having a strange problem. When no distcc-server is available and
1018 distcc decides to run a task locally, it fails.
1019 <p>
1020 <tt>
1021 configure:5648: checking for i386-redhat-linux-gcc-3.3.2 option to produce PIC
1022 <br>configure:5825: result: -fPIC
1023 <br>configure:5833: checking if i386-redhat-linux-gcc-3.3.2 PIC flag -fPIC works
1024 <br>configure:5854: i386-redhat-linux-gcc-3.3.2 -c -O2 -march=i386
1025 <br>-mcpu=i686 -fPIC -DPIC conftest.c &gt;&amp;5
1026 <br>distcc[32203] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
1027 <br>configure:5858: $? = 0
1028 <br>configure:5866: result: no
1029 </tt>
1030 <p>
1031 For some reason configure thinks -fPIC is not supported although it works
1032 if a distcc-server is started. This causes some software to not work when
1033 build with distcc (but without distcc-servers). If build without distcc,
1034 it works (as -fPIC is supported).
1035 <p>
1036 What could be the cause ? I'm running the latest distcc.
1037 </blockquote>
1038
1039 <p>
1040 autoconf interprets any message from the compiler as a failure for
1041 some tests, even if compilation succeeds. This is fixed in distcc
1042 2.18 by always building autoconf tests locally.
1043
1044
1045
1046 <h3>distcc only listens for IPv6 connections on BSD</h3>
1047 <p>
1048 <blockquote>
1049 I use distcc from NetBSD's
1050 pkgsrc, which sets --enable-rfc2553 by default. When
1051 distccd is run as "distccd --daemon --user nobody",
1052 distccd seems to *only* be listening on an IPv6
1053 socket.
1054 </blockquote>
1055 <p>
1056 By default some BSD systems do not allow applications to accept both
1057 IPv4 and IPv6 connections through a single server socket. See <a href="http://cert.uni-stuttgart.de/archive/bugtraq/2002/08/msg00319.html">this message</a>.
1058 <p>
1059 There are several options
1060
1061 <ol>
1062 <li> Don't build distcc with the <tt>--enable-rfc2553</tt> option unless you need to support IPv6.
1063 <li> Tell distcc to explicitly listen on either the IPv4 or IPv6 address, unless you need to accept connections over both protocols.
1064 <li> Set the <tt>net.inet6.ip6.v6only</tt> sysctl to 0. (This affects
1065 all programs on the system, see the BSD manual for details.)
1066 <li> Run two copies of distcc listening on both ports.
1067 </ol>
1068
1069
1070 <h3>distcc fails with "No locks available"</h3>
1071 <p>
1072 This can happen if your home directory is on an NFS filesystem and
1073 NFS locking is not working. Locking in NFS is supported by a separate
1074 protocol and daemon which can be down even if other file operations
1075 are working OK.
1076 <p>
1077 To fix this, either fix NFS locking on your system, or set
1078 <tt>DISTCC_DIR</tt> to point to a local disk. If your system doesn't have
1079 a local hard disk you can just use <tt>tmpfs</tt> or something similar,
1080 because none of the files in that directory need to persist across
1081 reboots.
1082
1083
1084 <h3>How to stop distcc using a particular machine?</h3>
1085 <p>
1086 <blockquote>
1087 I want to stop my machine accepting distcc jobs because I need it for
1088 something else.
1089 </blockquote>
1090 <p>
1091 The easiest way is to just shut down the <tt>distccd</tt> server. When
1092 it comes back up, clients will start using it again.
1093
1094
1095 <h3>distcc only builds on one machine at a time</h3>
1096 <p>
1097 <blockquote>
1098 If I set
1099 <tt>DISTCC_HOSTS="remote_host localhost"</tt>
1100 then the project seems to be compiled only on the remote host (I see traces
1101 in ethereal and compilation is slow).
1102 </blockquote>
1103 <p>
1104 You are probably not using the <tt>-j</tt> option to make or scons, so
1105 it's only compiling one file at a time. Only the first host in the
1106 list will be used in this case.
1107
1108
1109
1110 <h3>Errors with temporary files on Cygwin</h3>
1111
1112
1113 <p>
1114 Some Windows compilers can't handle the default <tt>TMPDIR</tt> setting
1115 when distcc is run under Cygwin. To fix this, put something like
1116 <tt>export TMPDIR=c:/temp</tt> in <tt>/etc/profile</tt>.
1117
1118
1119 <h3>Server won't start on Windows 98 or ME</h3>
1120
1121
1122 <p>
1123 For some reason the server does not work properly on Windows 98 unless it is
1124 invoked with <tt>--no-detach</tt>.
1125
1126
1127 <h3>Make fails with "file not found"</h3>
1128
1129
1130 <p>
1131 If Make works for non-parallel builds, but fails when you use <tt>j</tt> and distcc, then there is probably a concurrency bug in your
1132 Makefile. It is similar to a threading bug in C,
1133 C++ or Java when a program is run on a multiprocessor system.
1134 <p>
1135 Sometimes it will work with <tt>-j</tt> without distcc, but fail when
1136 you use distcc. This is probably because distcc lets the
1137 compilation run faster and therefore more jobs run in parallel.
1138 <p>
1139 You need to just fix your makefile, or not build that particular
1140 directory in parallel. It is very unlikely this is a bug in GNU make
1141 or distcc.
1142
1143
1144
1145 <h3>Internet-wide distributed compilation</h3>
1146 <p>
1147
1148 <blockquote>
1149 I'm a newbie with distcc and linux. I have been able to bootstrap (with
1150 lots of effort) Gentoo. I'm having only one computer on my lan that I
1151 control, so can't install distcc on the other host. I'm looking for a
1152 WAN of public distcc (I would grant access to mine also). I would like
1153 have a vast distributed network of distcc that would be public. It may
1154 use some ssh. I don't even know if this possible.
1155 <p>
1156 Won't it be faster then a laptop and a desktop if we have N hosts? I
1157 have heard of sub-ether that create a virtual NIC -- as far as I could
1158 understand. Any body know if it work with distcc?
1159
1160 </blockquote>
1161 <p>
1162 This could be done, but some preparatory work is needed:
1163 <p>
1164 This really has to be done across SSH or a VPN; sending commands
1165 across the public net unencrypted would be insanely insecure.
1166 <p>
1167 Connections across the public internet are slower than on a LAN in
1168 both throughput and bandwidth. It's probably only feasible if the
1169 machines involved are on the same continent (preferably the same
1170 state) and have at least 512kbit connections.
1171 <p>
1172 The more serious problem is compiling on possibly untrusted
1173 machines. You need to assume that anyone who sends jobs can take
1174 over control of the user running distcc: if it's run within a UML
1175 instance or some similar sandbox, and each user has their own
1176 instance, that might be OK. Clients also need to implicitly trust
1177 all the servers not to give them corrupted or malicious data back.
1178 <p>
1179 There is also the organizational issue of just locating machines to
1180 use and the right public key to get into them.
1181 <p>
1182 Probably the best way to prototype this is to find a few friends who
1183 live in the same country. Set up ssh access into each others
1184 machines, and run distcc over that to try compilation. Please report
1185 your results to the distcc mailing list. If this works well, then we
1186 can look at scaling it up to run on larger groups or between
1187 potentially untrusted machines.
1188
1189
1190
1191
1192
1193
1194 <h3>What is the relationship between Apple Xcode and distcc?</h3>
1195 <p>
1196 Apple's <a href="http://developer.apple.com/tools/xcode/">Xcode</a>
1197 development kit includes a fork of an old version of distcc, plus
1198 Apple patches to locate machines using Rendezvous and a small GUI to
1199 configure it.
1200 <p>
1201 The source is available from
1202 <a href="http://www.opensource.apple.com/">opensource.apple.com</a>.
1203
1204 <h3>How can I avoid compiling on workstations when they're in use</h3>
1205
1206
1207 <p>
1208 This <a href="http://lists.samba.org/archive/distcc/2006q1/003271.html"><tt>watch-ssaver</tt> script</a>
1209 from <a href="http://sourceforge.net/projects/dmucs">DMUCS</a>
1210 enables distcc only when the screensaver is running.