| 1 |
<html> |
| 2 |
<head><title>distcc frequently asked questions</title> |
| 3 |
<meta name="author" content="Martin Pool"> |
| 4 |
<meta name="keywords" content="distributed, make, build, gcc, |
| 5 |
parallel, network, networked, faster, free, fast, paralel, cc, c, |
| 6 |
C++, Objective C, ObjC, compile, compiler, compilation, |
| 7 |
simple, GPL, GNU, Linux, BSD, Unix, software, development, |
| 8 |
developer, TeamBuilder, quick, dmake, pvmake, |
| 9 |
transparent, easy, workstations, Solaris, IncrediBuild, |
| 10 |
productivity, speed, time, tool, performance, hardware, |
| 11 |
agents, nfs, libraries, source, headers, workload, |
| 12 |
buildfarm, farm, server, cluster, clustered, clustering, MOSIX, OpenMOSIX"> |
| 13 |
<meta name="description" content="distcc is a fast, free distributed C and C++ compiler."> |
| 14 |
<link rel="SHORTCUT ICON" href="favicon.ico"> |
| 15 |
<link type="text/css" rel="stylesheet" href="distcc-green.css"> |
| 16 |
<link type="text/html" rel="top" href="index.html"> |
| 17 |
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"></head> |
| 18 |
<body vlink="#003300" text="#000000" link="#003300" bgcolor="#ffffff" alink="#336633"><p><table width="100%"><tr><td valign="top"><div class="navbar"><ul> |
| 19 |
<li><a href="index.html">distcc</a></li> |
| 20 |
<li><a href="http://code.google.com/p/distcc/source/browse/trunk">source</a> |
| 21 |
<li><a href="http://code.google.com/p/distcc/downloads/list">binaries</a></li> |
| 22 |
<li><a href="http://distcc.googlecode.com/svn/trunk/NEWS">news</a></li> |
| 23 |
<li><a href="scenarios.html">scenarios</a></li> |
| 24 |
<li><a href="results.html">testimonials</a></li> |
| 25 |
<li><a href="benchmark.html">benchmarks</a></li> |
| 26 |
<li><a href="security.html">security</a></li> |
| 27 |
<li><a href="distcc-lca-2004.html">whitepaper</a></li> |
| 28 |
<li><a href="http://distcc.googlecode.com/svn/trunk/doc/web/man/">Man pages</a></li> |
| 29 |
<li><a href="faq.html"><b>FAQ</b></a></li> |
| 30 |
|
| 31 |
<p> |
| 32 |
<li><a href="http://code.google.com/p/distcc/issues/list">report a bug</a></li> |
| 33 |
<li><a href="https://lists.samba.org/mailman/listinfo/distcc">mailing list</a> |
| 34 |
<ul><li><a href="mailto:[email protected]">post</a></li> |
| 35 |
<li><a href="http://lists.samba.org/archive/distcc/">archive</a></li> |
| 36 |
<li><a href="news://news.gmane.org/gmane.comp.compilers.distcc">newsgroup</a></li></ul> |
| 37 |
</li> |
| 38 |
<li><a href="http://freshmeat.net/projects/distcc">Freshmeat</a> |
| 39 |
<ul><li><a href="http://freshmeat.net/subscribe/28140">be notified</a></li></ul> |
| 40 |
</li> |
| 41 |
|
| 42 |
<p> |
| 43 |
<li>related:</li> |
| 44 |
<ul> |
| 45 |
<li><a title="How to use distcc and Gentoo" href="http://www.gentoo.org/doc/en/distcc.xml">Gentoo</a></li> |
| 46 |
<li><a title="Script to ease building cross-compilers and toolchains" href="http://kegel.com/crosstool/">crosstool</a></li> |
| 47 |
<li><a title="Cache compiler results" href="https://ccache.dev/">ccache</a></li> |
| 48 |
<li><a title="Centralized control over distcc and ccache" href="http://ccontrol.ozlabs.org/">ccontrol</a></li> |
| 49 |
<li><a title="Smart job scheduler" href="http://dmucs.sourceforge.net/">dmucs</a></li> |
| 50 |
</ul> |
| 51 |
<p><p></td> |
| 52 |
<td valign="top"> |
| 53 |
<div class="body"> |
| 54 |
<h1 class="title">distcc frequently asked questions</h1> |
| 55 |
|
| 56 |
|
| 57 |
<p><b>Question not answered here? Check the <a |
| 58 |
href="http://lists.samba.org/archive/distcc/">mailing list |
| 59 |
archives</A> or send email to <a |
| 60 |
href="mailto:[email protected]">distcc (at) |
| 61 |
lists.samba.org</A></b> |
| 62 |
|
| 63 |
|
| 64 |
<h3>What compilers are supported?</h3> |
| 65 |
|
| 66 |
<p>gcc is fully supported, building C, C++, Objective C and Objective |
| 67 |
C++. Other gcc languages such as Java are not supported. All recent |
| 68 |
versions of gcc are thought to work, though later versions tend to work |
| 69 |
better. |
| 70 |
|
| 71 |
<p>Intel's <em>icc</em> compiler is somewhat compatible with gcc and |
| 72 |
works with distcc, but some problems have been reported. |
| 73 |
|
| 74 |
<p>Sun's proprietary C compiler is reported to work with distcc in C |
| 75 |
mode. It appears that Sun CC cannot compile C++ templates |
| 76 |
correctly when using a separate preprocessor, so it is generally |
| 77 |
not practical to compile C++ using Sun CC and distcc. |
| 78 |
|
| 79 |
<p> |
| 80 |
The main feature required by distcc is that the compiler must be able |
| 81 |
to run the preprocessor separately, and then compile the preprocessor |
| 82 |
output from a file. This was a basic part of the original design of |
| 83 |
C, but some compilers seem to have lost the ability to do this. |
| 84 |
However, this feature is not required if you use distcc's "pump" mode. |
| 85 |
|
| 86 |
Secondarily, distcc is currently hardcoded to suit gcc's behaviour and |
| 87 |
command-line syntax, so only compilers that act like gcc will work. |
| 88 |
This could in principle be changed. |
| 89 |
|
| 90 |
|
| 91 |
<h3>How to build gcc with distcc?</h3> |
| 92 |
|
| 93 |
<p> |
| 94 |
gcc uses an unusual three-stage process to verify that the compiler |
| 95 |
can recompile itself with the same results. Because of bugs in the |
| 96 |
gcc integrated preprocessor, compiling locally can produce a |
| 97 |
program that is functionally identical, but not byte-for-byte |
| 98 |
identical, with a program compiled remotely. |
| 99 |
|
| 100 |
<p> |
| 101 |
This can apparently be fixed by specifying <tt>127.0.0.1</tt> in the |
| 102 |
host list, rather than <tt>localhost</tt>. This causes distcc to run |
| 103 |
compilation "remotely" onto the same machine, which should be the |
| 104 |
same as compiling remotely. |
| 105 |
|
| 106 |
<p> |
| 107 |
You will also need to make sure that the build directory is mounted |
| 108 |
at the same location on all machines so that the new compiler can |
| 109 |
be located. |
| 110 |
|
| 111 |
<p> |
| 112 |
(More information or instructions here would be welcome.) |
| 113 |
|
| 114 |
|
| 115 |
|
| 116 |
<h3>distcc gets slower when I add slow machines to the cluster</h3> |
| 117 |
|
| 118 |
|
| 119 |
|
| 120 |
|
| 121 |
<p> |
| 122 |
Make sure you put the preferred (fastest/closest/least loaded) |
| 123 |
machines at the start of the <tt>DISTCC_HOSTS</tt> list. This is |
| 124 |
particularly important when running <tt>./configure</tt> scripts |
| 125 |
because all compilation will be done on the first machine listed. |
| 126 |
Normally this should be <tt>localhost</tt>, but if another machine is |
| 127 |
much faster then perhaps not. |
| 128 |
|
| 129 |
<p> |
| 130 |
To some extent this is still an open bug that I hope to address in |
| 131 |
a future version. It would be nice if distcc could automatically |
| 132 |
detect the best distribution, but it doesn't do that yet. |
| 133 |
|
| 134 |
|
| 135 |
<h3>Restarting distccd on reboot</h3> |
| 136 |
|
| 137 |
|
| 138 |
<p> |
| 139 |
You may have <tt>distccd</tt> installed on a machine where you don't |
| 140 |
have root, and want it to restart when the machine reboots. |
| 141 |
|
| 142 |
<p> |
| 143 |
One way to do this, <a href="http://lists.samba.org/pipermail/distcc/2002q4/000417.html">suggested by Shane McDaniel</a>, |
| 144 |
is to put it in |
| 145 |
your per-user <tt>crontab</tt>. <tt>distccd</tt> will be started at |
| 146 |
regular intervals, but will exit if something is already listening |
| 147 |
on the port. Remember to set your <tt>PATH</tt> in the crontab so |
| 148 |
that <tt>distccd</tt> and all necessary compilers can be found. |
| 149 |
|
| 150 |
|
| 151 |
|
| 152 |
<h3>Choosing a userid</h3> |
| 153 |
|
| 154 |
<blockquote> |
| 155 |
It would be awesome if you could run the daemon as a specific |
| 156 |
user in daemon mode. |
| 157 |
</blockquote> |
| 158 |
|
| 159 |
<p> |
| 160 |
It is awesome! :-) From distcc 1.1, root can use the <tt>--user</tt> |
| 161 |
option to cause distcc to start as a particular user. |
| 162 |
<p> |
| 163 |
<tt># distccd --user nobody </tt> |
| 164 |
|
| 165 |
<p> |
| 166 |
If distccd is started by root and no user is specified, it will |
| 167 |
change to the user <tt>distcc</tt> if possible, or otherwise to |
| 168 |
<tt>nobody</tt>. I recommend you create this user when installing the |
| 169 |
package. |
| 170 |
|
| 171 |
<p> |
| 172 |
If distccd is started by a non-root user then it will continue |
| 173 |
running as that user. |
| 174 |
|
| 175 |
|
| 176 |
|
| 177 |
|
| 178 |
<h3>Why not use a distributed-make-style system?</h3> |
| 179 |
|
| 180 |
|
| 181 |
|
| 182 |
|
| 183 |
|
| 184 |
<p> |
| 185 |
A few people have extended <tt>make</tt> to allow it to distribute jobs |
| 186 |
across several machines. Projects I know of include Graydon Hoare's |
| 187 |
<a href=http://www.venge.net/graydon/cgi-bin/viewcvs.cgi/src/doozer/">Doozer</a>, |
| 188 |
Sun's <tt>dmake</tt>, <tt>pvmmake</tt>, and <tt>ppmake</tt>. GNU Make apparently has |
| 189 |
internal hooks to add distribution mechanisms, which I think is how |
| 190 |
<tt>ppmake</tt> and <tt>pvmmake</tt> work. |
| 191 |
|
| 192 |
<p> |
| 193 |
Unlike distcc, most of programs have no special knowledge of C: they |
| 194 |
just schedule jobs remotely or locally. The advantage of this is |
| 195 |
that you can distribute all different jobs, such as linking, or |
| 196 |
building documentation, or compiling programs in other languages. |
| 197 |
|
| 198 |
<p> |
| 199 |
The disadvantage of this approach is because it relies on running |
| 200 |
tasks on any node, all relevant aspects of the nodes must be the |
| 201 |
same. This typically means that all machines must have a shared |
| 202 |
filesystem mounted at the same location, that they must all have |
| 203 |
exactly the same compiler, headers and libraries installed, their |
| 204 |
clocks must be in sync, and typically that they must all have the |
| 205 |
same OS and CPU architecture. |
| 206 |
|
| 207 |
<p> |
| 208 |
In some situations, such as a lab of centrally managed machines, |
| 209 |
this is quite practical. However, many people have a less |
| 210 |
homogeneous environment: perhaps some machines run a different OS |
| 211 |
release, or developers are allowed to upgrade libraries on their own |
| 212 |
machines, or perhaps you just don't want to run NFS. |
| 213 |
|
| 214 |
<p> |
| 215 |
In this situation distcc is much easier to set up. You don't even |
| 216 |
need root on the volunteer machines, let alone a mandate to move |
| 217 |
<tt>/home</tt> onto NFS. |
| 218 |
|
| 219 |
<p> |
| 220 |
(You might get away with having slightly different headers or |
| 221 |
libraries, but the potential for confusion is so great that I think |
| 222 |
you'd be crazy to try.) |
| 223 |
|
| 224 |
<p> |
| 225 |
By Amdahl's law, a distributed make system could in principle be |
| 226 |
faster than distcc, because it can distribute many different jobs. |
| 227 |
In practice, however, for many projects compiling C or C++ takes |
| 228 |
over 80% of the time. Many of the other jobs, such as linking, |
| 229 |
cannot be parallelized anyhow. |
| 230 |
|
| 231 |
<p> |
| 232 |
I don't know how their performance compares but I would be interested |
| 233 |
to hear. |
| 234 |
|
| 235 |
|
| 236 |
<h3>Has anybody yet thought of integrating distcc with ccache?</h3> |
| 237 |
|
| 238 |
<p> |
| 239 |
If you don't use distcc's "pump" mode, then |
| 240 |
they work pretty well as separate programs that call each other. |
| 241 |
You can either set <tt>CC='ccache distcc gcc'</tt>, or arrange for |
| 242 |
both ccache and distcc to be "masqueraded" on the path. (See the |
| 243 |
manual for information on how to install this.) |
| 244 |
|
| 245 |
<p> |
| 246 |
Normally it is better for ccache to be run before distcc. |
| 247 |
|
| 248 |
<p> |
| 249 |
This is very nearly as efficient as making them both part of a |
| 250 |
single program. The preprocessor will only run once and the |
| 251 |
preprocessed source will be passed from ccache straight to distcc. |
| 252 |
Having them separate allows them to be tested, debugged and |
| 253 |
released separately. |
| 254 |
|
| 255 |
<p> |
| 256 |
Unless there's a really strong argument to do otherwise, having two |
| 257 |
smaller programs is better practice, or at least more to my taste, |
| 258 |
than making one monolithic one. |
| 259 |
|
| 260 |
<p> |
| 261 |
However, all of that said, now that we have "pump" mode, the |
| 262 |
trade-offs have changed. Using ccache prevents the use of "pump" |
| 263 |
mode. It would make sense to integrate caching into distcc |
| 264 |
so that you can get distributed preprocessing, distributed |
| 265 |
compilation, and caching all at the same time. This would make |
| 266 |
a great project for someone... |
| 267 |
|
| 268 |
<p> |
| 269 |
Also, it would be nice to allow a cache of |
| 270 |
compiled files to be shared across several users or machines. |
| 271 |
|
| 272 |
<p> |
| 273 |
Joerg Beyer started a project called <a href="http://gecc.sourceforge.net/">gecc</a> |
| 274 |
to explore this architecture, but development appears to have stalled in 2002. |
| 275 |
|
| 276 |
<h3>Temporary files in strange location?</h3> |
| 277 |
<p> |
| 278 |
<blockquote> |
| 279 |
distcc always tries to create subdirs in /root/tmp for its temp files. How do I get around |
| 280 |
this? |
| 281 |
</blockquote> |
| 282 |
|
| 283 |
<p> |
| 284 |
distcc respects the <tt>$TMPDIR</tt> environment variable when creating its |
| 285 |
scratch directory. I suspect you have that set in root's .profile. |
| 286 |
If you unset it in the shell script that launches xinetd, or set it to |
| 287 |
something not in root's path, then it should be fine. |
| 288 |
|
| 289 |
|
| 290 |
|
| 291 |
<h3>Server dies after a few connections when run from inetd</h3> |
| 292 |
|
| 293 |
|
| 294 |
<p> |
| 295 |
<blockquote> |
| 296 |
i noticed that when you start a make process, the remote distccd receives |
| 297 |
10-15 connections from my host (which is good) and those processes die after |
| 298 |
a while. this is a long compilation (kdelibs) and after the initial |
| 299 |
processes die, i cannot see any new one's coming in. is this normal ? |
| 300 |
it seems as if after a while (2-3 minuets) distcc stops working and only my |
| 301 |
local gcc is still compiling. |
| 302 |
</blockquote> |
| 303 |
|
| 304 |
|
| 305 |
<p> |
| 306 |
|
| 307 |
If you're running distccd from inetd, then it may be that inetd |
| 308 |
thinks that the service is "looping" because of all the rapid |
| 309 |
connections. You need to increase the maximum connection rate. |
| 310 |
See the inetd manual. |
| 311 |
|
| 312 |
<p> |
| 313 |
On traditional BSD inetd, you can do this by changing the word <tt>t</tt> to something like <tt>nowait.1000</tt> |
| 314 |
|
| 315 |
<p> |
| 316 |
Alternatively, run distccd with <tt>--daemon</tt>, rather than from |
| 317 |
inetd. |
| 318 |
|
| 319 |
|
| 320 |
|
| 321 |
|
| 322 |
<h3>Running distccd on a firewall?</h3> |
| 323 |
|
| 324 |
|
| 325 |
<p> |
| 326 |
<blockquote> |
| 327 |
another machine i want to add to this "cluster" is my firewall. its not a |
| 328 |
very powerful one, but it can help speed things up a little more ;) |
| 329 |
are there any knows security issues with distcc ? how stupid will it be to run |
| 330 |
it on a server that acts as a firewall ? |
| 331 |
</blockquote> |
| 332 |
|
| 333 |
|
| 334 |
|
| 335 |
<p> |
| 336 |
It depends on your security profile, but it's not completely |
| 337 |
unreasonable. Hopefully your firewall already has iptables and |
| 338 |
tcpwrappers protection against connections from the outside world. |
| 339 |
Just make sure that nobody else can connect to the distccd port. |
| 340 |
|
| 341 |
|
| 342 |
|
| 343 |
<h3>What -j level to use?</h3> |
| 344 |
|
| 345 |
<p> |
| 346 |
<blockquote> |
| 347 |
when starting a compilation, the howto says to use -j8 . |
| 348 |
is this optimal ? anything else i should be using for better |
| 349 |
performance ? |
| 350 |
</blockquote> |
| 351 |
|
| 352 |
<p> |
| 353 |
For plain (that is, non-pump) mode, you should use about twice the total |
| 354 |
number of CPUs available, but it depends on your network, program being |
| 355 |
compiled, available memory, etc. Experiment with different values. |
| 356 |
|
| 357 |
<p> |
| 358 |
Client machines tend to be saturated at about 10-20 jobs, so using |
| 359 |
values above -j10 is rarely useful. (Perhaps higher levels would work |
| 360 |
well on clients with two or more CPUs and very fast network |
| 361 |
connections, memory and disk.) |
| 362 |
|
| 363 |
<p> |
| 364 |
The advice above is all for plain mode; for "pump" mode, you |
| 365 |
may see benefit from higher -j values. Please experiment and |
| 366 |
<a href="mailto:[email protected]">let us know</a> what works best. |
| 367 |
|
| 368 |
|
| 369 |
<h3>Should I include localhost in the host list?</h3> |
| 370 |
|
| 371 |
<p> |
| 372 |
Naturally, compiling on the machine that drives |
| 373 |
compilation has low overhead. |
| 374 |
|
| 375 |
<p> |
| 376 |
But, if a large number of machines are part of the |
| 377 |
distcc "farm", I suspect most of the driving machine's |
| 378 |
time would be better spent doing preprocessing & |
| 379 |
feeding only, and as such increase the chances of |
| 380 |
always having something ready to handle to machines |
| 381 |
that finish their distcc compilation jobs. |
| 382 |
|
| 383 |
<p> |
| 384 |
I'm just hoping to collect some opinions on this |
| 385 |
matter. For a large project and for a cluster of about 8 |
| 386 |
machines, is it actually better to dedicated the |
| 387 |
driving machine to do preprocessing only? (and so not |
| 388 |
include it in the hosts file). |
| 389 |
|
| 390 |
<p> |
| 391 |
It will depend on your source tree, your network, your |
| 392 |
compiler, and your makefile (or alternative) just what fraction of the |
| 393 |
work must be done locally, which is the most important thing here. At |
| 394 |
about the 3-4 machine level it may be worth putting localhost last; at |
| 395 |
8-10 machines it may be better to leave it out altogether. |
| 396 |
|
| 397 |
|
| 398 |
<h3>Different gcc versions?</h3> |
| 399 |
|
| 400 |
<p> |
| 401 |
<blockquote> |
| 402 |
if the host machine is using gcc 3.2, can the other machines use an older |
| 403 |
version of gcc ? 2.9.x for example ? |
| 404 |
</blockquote> |
| 405 |
|
| 406 |
|
| 407 |
<p> |
| 408 |
distcc doesn't care. However, in some circumstances, particularly |
| 409 |
for C++, gcc object files compiled with one version of gcc are not |
| 410 |
compatible with those compiled by another. This is true even if |
| 411 |
they are built on the same machine. |
| 412 |
<p> |
| 413 |
It is usually best to make sure that every compiler name maps to a |
| 414 |
reasonably similar version on every machine. You can either make |
| 415 |
sure that <tt>gcc</tt> is the same everywhere, or use a |
| 416 |
version-qualified compiler name, such as <tt>gcc-3.2</tt> or |
| 417 |
<tt>i386-redhat-linux-gcc-3.2.2</tt>. |
| 418 |
<p> |
| 419 |
In particular, object files can be incompatible because: |
| 420 |
<ol> |
| 421 |
<li> Calling conventions or ABIs have changed, particularly for |
| 422 |
C++. |
| 423 |
|
| 424 |
<li> Different versions optimize differently. |
| 425 |
|
| 426 |
<li> The header files specifically detect the version of gcc being |
| 427 |
used, to change optimization or to work around bugs. The |
| 428 |
Linux kernel does this. |
| 429 |
</ol> |
| 430 |
<p> |
| 431 |
In all these cases you would need to "make clean" if you upgraded |
| 432 |
your gcc. |
| 433 |
<p> |
| 434 |
It is hard to generalize, but using gcc versions which have the |
| 435 |
first two components the same (e.g. 3.2.1 and 3.2.2) is usually OK. |
| 436 |
To be safe, use the exact same release on all machines. |
| 437 |
<p> |
| 438 |
It is also a good idea to have the assembler versions be the same |
| 439 |
too, although the problems there are slightly less complex. |
| 440 |
|
| 441 |
|
| 442 |
|
| 443 |
|
| 444 |
<h3>Shouldn't distcc check gcc versions?</h3> |
| 445 |
|
| 446 |
|
| 447 |
<p> |
| 448 |
It might be a good idea for distcc to check the version of gcc on |
| 449 |
all the volunteer machines. However this turns out to be hard to do |
| 450 |
completely reliably, because some programs which both call themselves |
| 451 |
"gcc 3.2.1" behave differently, presumably because of vendor |
| 452 |
patches or because vendors have shipped pre-release code. |
| 453 |
<p> |
| 454 |
So since automatic detection would not be a reliable solution, for |
| 455 |
the moment we depend on the user to make sure they have compatible |
| 456 |
versions installed. |
| 457 |
|
| 458 |
|
| 459 |
|
| 460 |
|
| 461 |
<h3>Using different platforms?</h3> |
| 462 |
<p> |
| 463 |
<blockquote> |
| 464 |
I have a mixed network environment. MacOS X, Linux, and Windows (with |
| 465 |
cygwin). Seeing distcc really piqued my interest. Using gcc on all of my |
| 466 |
environments, can I set up distcc to span across this variety of operating |
| 467 |
systems? |
| 468 |
</blockquote> |
| 469 |
<p> |
| 470 |
It should be reasonably straightforward. Of course you will need to |
| 471 |
either install or build appropriate cross compilers for each machine. |
| 472 |
<p> |
| 473 |
For example, on each volunteer machine, build an x86-linux cross |
| 474 |
compiler and (this is important) install it as "<tt>i386-redhat-linux-gcc-3.2.2</tt>" or |
| 475 |
something similar, using the appropriate gcc configuration options to |
| 476 |
set the name. You also need to make a link to that name on the Linux |
| 477 |
machine. |
| 478 |
<p> |
| 479 |
Then from Linux, run "<tt>distcc i386-redhat-linux-gcc-3.2.2</tt>". |
| 480 |
<p> |
| 481 |
Repeat as appropriate for every combination you want to use. |
| 482 |
<p> |
| 483 |
On Windows, you need to use the Cygwin (or perhaps Mingw?) |
| 484 |
software, which provides a Unix-like environment for running gcc. |
| 485 |
<p> |
| 486 |
The easiest way to build a cross compiler is to use |
| 487 |
Dan Kegel's |
| 488 |
<a href="http://kegel.com/crosstool/">crosstool</a>. |
| 489 |
<p> |
| 490 |
Brian Mosher reports that |
| 491 |
<a href="http://www.xraylith.wisc.edu/~khan/software/gnu-win32/cygwin-to-linux-cross-howto.txt">Mumit Khan's description of building a Cygwin-hosted Linux toolchain</a> works well. |
| 492 |
<a href="http://lists.samba.org/pipermail/distcc/2002q4/000469.html">Mike Santy suggests a slightly different approach</a>. |
| 493 |
|
| 494 |
|
| 495 |
|
| 496 |
<h3>Compiling between different i386 Unixes?</h3> |
| 497 |
<p> |
| 498 |
<blockquote> |
| 499 |
Do I need to install cross-compilers to distribute builds between |
| 500 |
different operating systems on the same CPU architecture? For |
| 501 |
example, <a href="http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=opro4i72qrekdofe%40news.lysator.liu.se&rnum=3&prev=/groups%3Fq%3Ddistcc%2Bopenbsd%26ie%3DUTF-8%26oe%3DUTF-8%26hl%3Den%26btnG%3DGoogle%2BSearch">OpenBSD i386 and NetBSD i386</a>. |
| 502 |
</blockquote> |
| 503 |
|
| 504 |
<p> |
| 505 |
In general, yes you do. Even on the same system, there can be |
| 506 |
incompatibilites between the output of compilers such as |
| 507 |
|
| 508 |
<ul> |
| 509 |
<li> Different object file formats, or variations of the format. |
| 510 |
(OpenBSD uses a.out, but most Unixes use ELF.) |
| 511 |
<li> Different patches applied to the compiler, even if it claims |
| 512 |
to be the same version. |
| 513 |
</ul> |
| 514 |
|
| 515 |
<h3>What's hard about synchronizing clocks?</h3> |
| 516 |
|
| 517 |
<p> |
| 518 |
<blockquote> |
| 519 |
Why do you say it's a feature of distcc that the machine clocks |
| 520 |
don't need to be synchronized? It's easy to do: you just install an |
| 521 |
NTP client. |
| 522 |
</blockquote> |
| 523 |
<p> |
| 524 |
It's not terribly hard, I agree. But it's not quite as trivial as |
| 525 |
you might think: |
| 526 |
<p> |
| 527 |
<ol> |
| 528 |
<li> Installing an NTP client requires root access on <em>all</em> the |
| 529 |
machines, not just your own workstation. (There's no way around |
| 530 |
it, because it needs to change the machine's clock.) |
| 531 |
<li> You need to be able to reach a reliable timeserver, which is |
| 532 |
sometimes a problem with firewalls. |
| 533 |
<li> If one of the machines does get out of sync, then builds may |
| 534 |
be incorrect or you may get errors. You may not notice until |
| 535 |
after problems have occurred. You can't fix it without |
| 536 |
intervention from the administrator, who may be on holidays. |
| 537 |
</ol> |
| 538 |
<p> |
| 539 |
What's not there can't break. |
| 540 |
|
| 541 |
|
| 542 |
<h3>What does "listening on 0.0.0.0:3632" mean?</h3> |
| 543 |
<p> |
| 544 |
This message means it's listening for connections on a wildcard IP |
| 545 |
address, so clients coming through any network interface should be |
| 546 |
able to connect, subject to <tt>--allow</tt> rules. |
| 547 |
<p> |
| 548 |
If you have more than one interface and want to only allow |
| 549 |
connections from one of them then use the <tt>--listen</tt> or <tt>w</tt> options, or both. |
| 550 |
|
| 551 |
|
| 552 |
|
| 553 |
|
| 554 |
|
| 555 |
<h3>How can I use SSH connection multiplexing?</h3> |
| 556 |
|
| 557 |
<p> |
| 558 |
<blockquote> |
| 559 |
Can I use ssh connection sharing to reduce the overhead of opening SSH connections? |
| 560 |
</blockquote> |
| 561 |
<p> |
| 562 |
Yes. Using SSH connection sharing can reduce the overhead of establishing SSH connections by a factor of 10. |
| 563 |
<p> |
| 564 |
Make sure you have a recent SSH (only on client side). I think you need OpenSSH 4 or later. |
| 565 |
<pre> |
| 566 |
$ ssh -V |
| 567 |
OpenSSH_4.6p1 Debian-5ubuntu0.1, OpenSSL 0.9.8e 23 Feb 2007 |
| 568 |
</pre> |
| 569 |
|
| 570 |
<p> |
| 571 |
Create a file <code>~/.ssh/config</code>, and add this: |
| 572 |
<pre> |
| 573 |
Host * |
| 574 |
ControlMaster auto |
| 575 |
ControlPath ~/.ssh_tmp/master-%r@%h:%p |
| 576 |
</pre> |
| 577 |
|
| 578 |
<p> |
| 579 |
Then create the master SSH connection: |
| 580 |
<pre> |
| 581 |
ssh -fMN <var>hostname</var> |
| 582 |
</pre> |
| 583 |
|
| 584 |
<p> |
| 585 |
Subsequent connections to that host will now fly! |
| 586 |
|
| 587 |
<p> |
| 588 |
For best results, create master SSH connections for each host in your distcc host list. |
| 589 |
|
| 590 |
<p> |
| 591 |
For more information on ssh connection sharing, |
| 592 |
see <a href="http://www.linux.com/feature/54498">here</a>. |
| 593 |
|
| 594 |
<h3>Will compiling with distcc magically make my program distributed?</h3> |
| 595 |
<p> |
| 596 |
<blockquote> |
| 597 |
All objects |
| 598 |
files were created , but when I issue command for execution as |
| 599 |
<tt>./executable</tt> |
| 600 |
then program takes so much time as without distcc. Kindly tell me that |
| 601 |
how this execution time can be reduced. |
| 602 |
All things work Properly , But all processors of systems are not utilized |
| 603 |
during EXECUTION of programs. |
| 604 |
</blockquote> |
| 605 |
<p> |
| 606 |
(This has to be the wierdest question in the list, but it really was sent.) |
| 607 |
<p> |
| 608 |
distcc makes building the program take less time. The program that |
| 609 |
is produced is the same as for a local build. |
| 610 |
<p> |
| 611 |
If you want your program to be distributed at run time across several |
| 612 |
machines, you need to design in parallelism and distribution, |
| 613 |
possibly using a framework such as MPI or OpenMOSIX. |
| 614 |
<p> |
| 615 |
What you seem to be asking for is to mechanically transform an |
| 616 |
arbitrary C program into a distributed parallel program. distcc |
| 617 |
doesn't do anything like that. It is perhaps not quite impossible |
| 618 |
(people have written parallelizing compilers) but it's very very |
| 619 |
hard. |
| 620 |
|
| 621 |
<h3>make-kpkg with distcc?</h3> |
| 622 |
<p> |
| 623 |
<blockquote> |
| 624 |
How to compile the debian kernel |
| 625 |
with make-kpkg using distcc? |
| 626 |
</blockquote> |
| 627 |
|
| 628 |
<p> |
| 629 |
Use the <tt>CONCURRENCY_LEVEL</tt> environment variable eg |
| 630 |
<tt>make-kpkg kernel_image CONCURRENCY_LEVEL=4</tt> |
| 631 |
after installing distcc into a masquerade directory. |
| 632 |
|
| 633 |
|
| 634 |
<h3>How to use an SSH TCP port other than 22?</h3> |
| 635 |
<p> |
| 636 |
<blockquote> |
| 637 |
I want to use two machines over ssh. One machine has a different port than |
| 638 |
22. But I can't set a port in the host specifications for ssh. |
| 639 |
</blockquote> |
| 640 |
<p> |
| 641 |
Use something like this in your <tt>~/.ssh/config</tt>: |
| 642 |
<p> |
| 643 |
<tt> |
| 644 |
Host bertie |
| 645 |
<br>Port 2202 |
| 646 |
</tt> |
| 647 |
|
| 648 |
<h3>How do I build a cross compiler to Mac OS X?</h3> |
| 649 |
|
| 650 |
|
| 651 |
<p> |
| 652 |
Dara Hazeghi says: |
| 653 |
<p> |
| 654 |
<blockquote>Basically, you have to download the same source |
| 655 |
version compiler as the one on your OS X box, build it |
| 656 |
on you Linux PC, etc. The assembler now works on Linux |
| 657 |
as well, so that other issue you had should be moot. |
| 658 |
<p> |
| 659 |
That is, you should compile the Apple source on your non-Apple |
| 660 |
machine. Apple have some patches that are not in the upstream gcc |
| 661 |
release so a compiler built from the gnu.org source will not be |
| 662 |
compatible. |
| 663 |
</blockquote> |
| 664 |
|
| 665 |
|
| 666 |
<h3>Randomly patched kernel crashes/hangs</h3> |
| 667 |
<p> |
| 668 |
<blockquote> |
| 669 |
I installed applied some |
| 670 |
<a href="http://bugs.gentoo.org/show_bug.cgi?id=36230">random</a> <a href="http://lists.samba.org/archive/distcc/2004q1/002035.html">unstable</a> kernel patches, and now my machine hangs/crashes/corrupts data when I run distcc! |
| 671 |
</blockquote> |
| 672 |
|
| 673 |
<p> |
| 674 |
If your kernel crashes, it is |
| 675 |
<a href="http://lists.samba.org/archive/distcc/2004q1/002040.html">by definition</a> |
| 676 |
a kernel bug. |
| 677 |
<p> |
| 678 |
If you have applied any patches that are not in the <tt>kernel.org</tt> |
| 679 |
stable release, your first step should be to back them out and see |
| 680 |
if the problem is still reproducible. |
| 681 |
|
| 682 |
|
| 683 |
|
| 684 |
<h3>distccd sometimes unable to find gcc</h3> |
| 685 |
<p> |
| 686 |
|
| 687 |
distccd inherits its <tt>PATH</tt> from whichever process starts it, |
| 688 |
and uses this to find compilers. You may need to make sure the <tt>H</tt> is set appropriately by the script that starts distccd. You |
| 689 |
can also set <tt>DISTCCD_PATH</tt>, which overrides <tt>PATH</tt> and |
| 690 |
bypasses checks for distcc masquerade directories. The daemon's path |
| 691 |
is logged when <tt>--verbose</tt> is given. |
| 692 |
|
| 693 |
|
| 694 |
<h3>Error about "jobserver unavailable"</h3> |
| 695 |
|
| 696 |
|
| 697 |
<p> |
| 698 |
This indicates a problem with your Makefile or shell. See the <a href="http://mail.gnu.org/archive/html/help-make/2001-09/msg00005.html">explanation of this error</a> from GNU make. |
| 699 |
<p> |
| 700 |
This can also happen when make is unable to create the fifo it uses |
| 701 |
to communicate among parallel processes, because TMPDIR is not |
| 702 |
accessible. Check that the variable is set properly (or unset), and |
| 703 |
that the directory has the right permissions. |
| 704 |
|
| 705 |
|
| 706 |
<h3>Files written to NFS filesystems are corrupt</h3> |
| 707 |
<p> |
| 708 |
<blockquote> |
| 709 |
<a href="http://lists.samba.org/pipermail/distcc/2003q3/001556.html">List post</a>: Writing object files from distcc to an NFS directory |
| 710 |
can cause corrupt output: object files will be full of zeros. |
| 711 |
</blockquote> |
| 712 |
<p> |
| 713 |
This is a bug in the Linux kernel NFS client interaction between |
| 714 |
<tt>mmap</tt> and <tt>rename</tt>. It can be avoided by using the |
| 715 |
<a href="http://article.gmane.org/gmane.linux.nfs/3578"><tt>no_subtree_check</tt></a> export option on the NFS server. |
| 716 |
distcc 2.18 no longer uses mmap to receive files and may not |
| 717 |
suffer this bug so strongly, but setting <tt>no_subtree_check</tt> is |
| 718 |
still recommended. |
| 719 |
|
| 720 |
|
| 721 |
<h3>distccmon doesn't work on NFS</h3> |
| 722 |
<p> |
| 723 |
<blockquote> |
| 724 |
When I set DISTCC_DIR to a directory on an NFS server, or have HOME on |
| 725 |
an NFS server, the monitors give errors or just don't work. |
| 726 |
</blockquote> |
| 727 |
<p> |
| 728 |
The monitor checks that the processes recorded in the state files |
| 729 |
are actually running (using kill -0). If the processes aren't |
| 730 |
running on the same machine, this doesn't work. |
| 731 |
<p> |
| 732 |
In any case, having DISTCC_DIR on NFS is likely to cause problems |
| 733 |
with locking. Please set DISTCC_DIR to a local directory instead. |
| 734 |
|
| 735 |
|
| 736 |
<h3>gcc's DEPENDENCIES_OUTPUT option is broken</h3> |
| 737 |
<p> |
| 738 |
Programs that use gcc's <a href="http://lists.samba.org/pipermail/distcc/2003q3/001547.html">DEPENDENCIES_OUTPUT</a> option don't work with ccache. |
| 739 |
<p> |
| 740 |
This should be fixed in <a href="https://ccache.dev/">ccache 2.3</a>. There is no problem with distcc. |
| 741 |
|
| 742 |
|
| 743 |
<h3>distcc fails to build on OS X</h3> |
| 744 |
<p> |
| 745 |
<blockquote> |
| 746 |
I'm trying to compile Distcc on OS X (gcc 3.1) and it fails with the |
| 747 |
following: |
| 748 |
</blockquote> |
| 749 |
<p> |
| 750 |
<tt> |
| 751 |
gcc -DHAVE_CONFIG_H -D_GNU_SOURCE -I./popt -I./src |
| 752 |
"-DSYSCONFDIR=\"/usr/local/etc\"" -g -O2 -W -Wall -W -Wimplicit |
| 753 |
-Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings |
| 754 |
-Waggregate-return -Wstrict-prototypes -Wmissing-prototypes |
| 755 |
-Wnested-externs -o src/climasq.o -c src/climasq.c |
| 756 |
<br>src/climasq.c:69: only 1 arg to macro 'rs_log_error' (2 expected) |
| 757 |
<br>src/climasq.c:114: only 1 arg to macro 'rs_trace' (2 expected) |
| 758 |
<br>cpp-precomp: warning: errors during smart preprocessing, retrying in |
| 759 |
basic mode |
| 760 |
<br>make: *** [src/climasq.o] Error 1 |
| 761 |
</tt> |
| 762 |
|
| 763 |
<p> |
| 764 |
There is a bug in some versions of Apple's gcc. Building with |
| 765 |
<tt>CFLAGS="-no-cpp-precomp"</tt> should fix it. It should be automatically |
| 766 |
corrected in distcc 2.8 and later. |
| 767 |
|
| 768 |
|
| 769 |
|
| 770 |
<h3>Problems with <tt>gcc -MD</h3></tt> |
| 771 |
<p> |
| 772 |
<blockquote> |
| 773 |
I'm using 0.12, and having trouble with dependency file creation. The |
| 774 |
makefiles that are set up to use gcc's -MD option aren't working. I'm |
| 775 |
getting some of the .d files in their proper directory, and not others. |
| 776 |
</blockquote> |
| 777 |
<p> |
| 778 |
This problem can only occur if you're using gcc 3.0 or later, have |
| 779 |
the source and object files in different directories or under |
| 780 |
different names, and you are using <tt>-MD</tt> or <tt>-MMD</tt> but |
| 781 |
not <tt>-MF</tt>. |
| 782 |
<p> |
| 783 |
The workaround is to change the Makefile to explicitly specify with |
| 784 |
<tt>-MF</tt> the file which should receive the dependency information. |
| 785 |
Many Makefiles already do this. Note that the <tt>-MF</tt> option is |
| 786 |
only available in gcc 3.0 and later. |
| 787 |
<p> |
| 788 |
Because the behaviour of <tt>-MD</tt> has changed from gcc2 to gcc3 |
| 789 |
there is no perfect solution available at the moment. |
| 790 |
|
| 791 |
|
| 792 |
|
| 793 |
<h3>gdb can't find source files</h3> |
| 794 |
<blockquote> |
| 795 |
When I try to debug with gdb an executable compiled with distcc, gdb doesn't |
| 796 |
find the source of the object to be debugged, unless that source is in the |
| 797 |
directory from which I start gdb. |
| 798 |
</blockquote> |
| 799 |
<p> |
| 800 |
|
| 801 |
Unfortunately this is caused by a bug in gcc, which I hope will be |
| 802 |
fixed in a future release. gcc embeds the directory where the |
| 803 |
compiler (cc1) was run, when it really ought to record the directory |
| 804 |
the source came from. |
| 805 |
<p> |
| 806 |
You can work around it for now by using the "<tt>directory</tt>" command |
| 807 |
in gdb to tell it where to find the source, or by passing an absolute |
| 808 |
file name when compiling. |
| 809 |
<p> |
| 810 |
Tim Janik has an <a href="http://www.gtk.org/~timj/patches/">unofficial patch</a> for distcc which works around this but I think I |
| 811 |
won't merge it because it's better to fix it in gcc. |
| 812 |
<p> |
| 813 |
This is Debian <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=148957&repeatmerged=yes">#148957</a>. |
| 814 |
<p> |
| 815 |
There was a <a href="http://gcc.gnu.org/ml/gcc-patches/2002-08/msg01411.html">discussion about this bug</a> on the gcc-patches mailing list. |
| 816 |
This can affect other programs which rely on debug stabs, such as |
| 817 |
<tt>addr2line</tt>, and it results in object files not being |
| 818 |
byte-for-byte identical when they include the source directory. The |
| 819 |
same bug affects <a href="https://ccache.dev/">ccache</a>. |
| 820 |
|
| 821 |
<h3>TCP_CORK in linux-2.2</h3> |
| 822 |
<p> |
| 823 |
Linux 2.2 has a bug to do with <tt>TCP_CORK</tt> sockets getting stuck in |
| 824 |
the <tt>FIN_WAIT1</tt> state. distcc 0.10 tries to work around it but it |
| 825 |
is not completely possible; if it causes trouble set |
| 826 |
<tt>DISTCC_CORK=0</tt>. |
| 827 |
|
| 828 |
|
| 829 |
<h3>Hung sockets in Linux 2.5</h3> |
| 830 |
<p> |
| 831 |
distcc seems to produce a problem in Linux 2.5 where one machine |
| 832 |
thinks the socket is CLOSED, and the other thinks it is |
| 833 |
ESTABLISHED. As a result the transfer hangs. |
| 834 |
<p> |
| 835 |
The CONNECTED/ESTABLISHED state should *never* be reachable by a correct TCP |
| 836 |
implementation. distcc is triggering a bug in the 2.5 TCP stack, |
| 837 |
which was <a href="http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=20030604050011%242b4f%40gated-at.bofh.it&rnum=1&prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26q%3Ddistcc%2B%2Bdavem%26btnG%3DGoogle%2BSearch">found and fixed</a> in 2.5 in June 2003. |
| 838 |
|
| 839 |
|
| 840 |
<h3>libtool and trouble building KDE</h3> |
| 841 |
<p> |
| 842 |
Trying to build KDE with <tt>make CC=distcc CXX=distcc -j5</tt> fails |
| 843 |
with a libtool error. |
| 844 |
<p> |
| 845 |
The version of libtool included with some KDE releases is buggy. |
| 846 |
<p> |
| 847 |
The best workaround is to install distcc in "masquerade mode". |
| 848 |
Wayne Davison writes: |
| 849 |
<blockquote> |
| 850 |
<p> |
| 851 |
To get distcc going, just follow the instructions in the docs to setup a |
| 852 |
"masquerade" dir, add it to the start of your PATH, and then never |
| 853 |
fiddle with CC and/or CXX again (i.e. undefine them). Note that you |
| 854 |
need to be running a 2.x version of distcc for masquerade support to be |
| 855 |
integrated by default (e.g. 2.0.1). (Side note: if you're using a |
| 856 |
binary RPM of distcc, make sure that the maintainer has made a |
| 857 |
masquerade dir a part of the installed config. If not, ask them to do |
| 858 |
so, as it is the easiest way to make distcc compatible with the widest |
| 859 |
range of packages.) |
| 860 |
<p> |
| 861 |
To be explicit, do something very similar (or identical) to this: |
| 862 |
<p> |
| 863 |
<pre> |
| 864 |
# mkdir -p /usr/lib/distcc/bin |
| 865 |
# cd /usr/lib/distcc/bin |
| 866 |
# ln -s ../../../bin/distcc cc |
| 867 |
# ln -s ../../../bin/distcc c++ |
| 868 |
# ln -s ../../../bin/distcc gcc |
| 869 |
# ln -s ../../../bin/distcc g++ |
| 870 |
</pre> |
| 871 |
<p> |
| 872 |
You can add links to any other compiler names you use on your system as |
| 873 |
well. |
| 874 |
<p> |
| 875 |
The other setup requirements remain unchanged: add DISTCC_HOSTS in your |
| 876 |
environment and run make with the -j5 (or whatever) option, perhaps by |
| 877 |
setting MAKEFLAGS in your environment. |
| 878 |
</blockquote> |
| 879 |
|
| 880 |
|
| 881 |
<h3>Linux 2.2 makefile strangeness</h3> |
| 882 |
<p> |
| 883 |
Linux 2.2.21 cannot be built with distcc because of bugs in the kernel's |
| 884 |
Makefile. (<tt>$CC</tt> is set to the compiler name plus the computed |
| 885 |
options.) It may be possible to use distcc in masquerade mode. |
| 886 |
|
| 887 |
|
| 888 |
<h3>Compiler command line parsing problems?</h3> |
| 889 |
|
| 890 |
|
| 891 |
<p> |
| 892 |
distcc parses the compiler command line to work out what |
| 893 |
operation is being invoked, and what are the input and output |
| 894 |
files. The semantics of <tt>gcc</tt> command lines is fairly complex, |
| 895 |
and has changed slightly over time. There may be some valid |
| 896 |
command lines that distcc understands differently to gcc, though |
| 897 |
none are known at the moment. These ought to cause the command to |
| 898 |
be run locally, but it is possible that it would cause a failure. |
| 899 |
Either case should be reported as a bug. |
| 900 |
|
| 901 |
<h3>Can't handle local file access</h3> |
| 902 |
|
| 903 |
<p> |
| 904 |
distcc can't handle compilers that need to read other files from |
| 905 |
the local filesystem. This might be a problem with such things as |
| 906 |
profile-directed optimizers. distcc tries to detect such commands |
| 907 |
and run them locally, but there may be cases which are not handled |
| 908 |
properly. |
| 909 |
|
| 910 |
<h3>Huge files</h3> |
| 911 |
<p> |
| 912 |
distcc's protocol and file IO would probably have trouble with |
| 913 |
source or object files over 2GB in size. I've never heard of a .c |
| 914 |
or .o file that large, and I rather suspect gcc would not handle |
| 915 |
them well either. |
| 916 |
|
| 917 |
|
| 918 |
<h3>KDE builds slowly with <tt>--enable-final</tt></h3> |
| 919 |
<p> |
| 920 |
Bernardo Innocenti says |
| 921 |
<p> |
| 922 |
<blockquote> Using the --enable-final configure option of KDE makes distcc almost |
| 923 |
useless. |
| 924 |
</blockquote> |
| 925 |
<p> |
| 926 |
|
| 927 |
Frerich Raabe explains: |
| 928 |
<blockquote> |
| 929 |
--enable-final makes the build system concatenate all sourcefiles in a |
| 930 |
directory (say, Konqueror's sourcefiles) into one big file. |
| 931 |
<p> |
| 932 |
Technically, this is achieved by creating a dummy file which simply |
| 933 |
includes every C++ sourcefile. The advantage of this is that the |
| 934 |
compile a- takes less time since there is only little scattered file |
| 935 |
opening involved and b- produces usually more optimized code, since |
| 936 |
the compiler can see more code at once and thus there are more |
| 937 |
chances to optimize. Of course this eats a lot more memory, but that |
| 938 |
is not an issue nowadays. |
| 939 |
<p> |
| 940 |
Now, it's clear why this makes distcc useless: there is just one huge file |
| 941 |
per project, and outsourcing that file via distcc to other nodes will just |
| 942 |
delay the build since the sourcecode (and it's a lot) has to be transferred |
| 943 |
over the network, and there is no way to pararellize this. |
| 944 |
<p> |
| 945 |
To avoid this, configure with <tt>--disable-final</tt>. |
| 946 |
</blockquote> |
| 947 |
<h3>C++ code that uses <tt>#pragma implementation</tt> doesn't seem to |
| 948 |
work properly.</h3> |
| 949 |
<p> |
| 950 |
That pragma can't work with distcc because it introduces |
| 951 |
dependencies between the source and local filenames. It is |
| 952 |
strongly deprecated in gcc and there are no plans to support it in |
| 953 |
distcc. |
| 954 |
<h3>Data corruption in compiles</h3> |
| 955 |
<p> |
| 956 |
Typical symptoms include linker errors, weird syntax errors or |
| 957 |
compiler crashes, caused by corrupt object or source files. |
| 958 |
<p> |
| 959 |
This has occurred several times in the past, typically because of |
| 960 |
kernel bugs such as |
| 961 |
Gentoo <a href="http://bugs.gentoo.org/show_bug.cgi?id=36230">#36320</a>. |
| 962 |
See if you can reproduce the problem using a standard kernel.org |
| 963 |
kernel. |
| 964 |
|
| 965 |
|
| 966 |
<h3>Linux kernel panic using gigabit ethernet</h3> |
| 967 |
<p> |
| 968 |
There is a |
| 969 |
<a href="http://lkml.org/lkml/2004/5/3/192">bug in the Linux 2.4.26 kernel</a> |
| 970 |
[<a href="http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&threadm=1Gu6a-7hD-11%40gated-at.bofh.it&rnum=1&prev=/groups%3Fq%3Ddistcc%2520panic%26meta%3Dsite%253Dgroups">Google groups thread</a>] that is triggered by using distcc. It is timing-related and seems |
| 971 |
more likely to occur with a gigabit network adapter. |
| 972 |
<p> |
| 973 |
<a href="http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&selm=1Qr7f-5aV-31%40gated-at.bofh.it&rnum=23">This patch may fix the problem.</a> If it doesn't, please report your |
| 974 |
problem to the kernel mailing list. |
| 975 |
<p> |
| 976 |
You may be able to avoid it by setting these environment variables |
| 977 |
on both the client and the server: |
| 978 |
<p> |
| 979 |
<tt>DISTCC_MMAP=0 DISTCC_SENDFILE=0 </tt> |
| 980 |
<p> |
| 981 |
This bug is present in the Fedora Core 1 kernel and is |
| 982 |
<a href="https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=114192">Red Hat bug #114192</a>. |
| 983 |
<h3>Sun Workshop CC does not work with distcc</h3> |
| 984 |
<p> |
| 985 |
<blockquote> |
| 986 |
I am trying to use distcc on a network of Solaris 2.7 workstations and I |
| 987 |
also ran into this problem. I modified the src just like you did but I |
| 988 |
still have problems compiling certain C++ files - specifically those that |
| 989 |
include STL headers. |
| 990 |
<p> |
| 991 |
Has anyone figured out how to prevent the re-inclusion of the headers? |
| 992 |
</blockquote> |
| 993 |
|
| 994 |
<p> |
| 995 |
P. Christeas says: |
| 996 |
<p> |
| 997 |
<blockquote> After some time, the use of distcc with SunWS seems an unresolved issue. Of |
| 998 |
course, I exclude any attempt to modify the STL headers (which in fact are |
| 999 |
Sun's version of STL headers). |
| 1000 |
The correct statement is that _Sun's CC does not behave correctly when the |
| 1001 |
preprocessor is involved as a separate process_. |
| 1002 |
<p> |
| 1003 |
This compiler should still be marked as 'unsupported'. The only way (I know |
| 1004 |
of) to make some use of a compile farm is to exclusively mark the few files |
| 1005 |
that *do* compile, and send those through distcc. |
| 1006 |
Hint: it seems that templates and CORBA implementation source do generally |
| 1007 |
break the precompiler. |
| 1008 |
</blockquote> |
| 1009 |
<p> |
| 1010 |
So <a href="compilers.html">Sun CC will not work for C++</a> with distcc, only for |
| 1011 |
C. |
| 1012 |
|
| 1013 |
|
| 1014 |
<h3>configure has trouble when hosts are down</h3> |
| 1015 |
<p> |
| 1016 |
<blockquote> |
| 1017 |
I'm having a strange problem. When no distcc-server is available and |
| 1018 |
distcc decides to run a task locally, it fails. |
| 1019 |
<p> |
| 1020 |
<tt> |
| 1021 |
configure:5648: checking for i386-redhat-linux-gcc-3.3.2 option to produce PIC |
| 1022 |
<br>configure:5825: result: -fPIC |
| 1023 |
<br>configure:5833: checking if i386-redhat-linux-gcc-3.3.2 PIC flag -fPIC works |
| 1024 |
<br>configure:5854: i386-redhat-linux-gcc-3.3.2 -c -O2 -march=i386 |
| 1025 |
<br>-mcpu=i686 -fPIC -DPIC conftest.c >&5 |
| 1026 |
<br>distcc[32203] (dcc_build_somewhere) Warning: failed to distribute, running locally instead |
| 1027 |
<br>configure:5858: $? = 0 |
| 1028 |
<br>configure:5866: result: no |
| 1029 |
</tt> |
| 1030 |
<p> |
| 1031 |
For some reason configure thinks -fPIC is not supported although it works |
| 1032 |
if a distcc-server is started. This causes some software to not work when |
| 1033 |
build with distcc (but without distcc-servers). If build without distcc, |
| 1034 |
it works (as -fPIC is supported). |
| 1035 |
<p> |
| 1036 |
What could be the cause ? I'm running the latest distcc. |
| 1037 |
</blockquote> |
| 1038 |
|
| 1039 |
<p> |
| 1040 |
autoconf interprets any message from the compiler as a failure for |
| 1041 |
some tests, even if compilation succeeds. This is fixed in distcc |
| 1042 |
2.18 by always building autoconf tests locally. |
| 1043 |
|
| 1044 |
|
| 1045 |
|
| 1046 |
<h3>distcc only listens for IPv6 connections on BSD</h3> |
| 1047 |
<p> |
| 1048 |
<blockquote> |
| 1049 |
I use distcc from NetBSD's |
| 1050 |
pkgsrc, which sets --enable-rfc2553 by default. When |
| 1051 |
distccd is run as "distccd --daemon --user nobody", |
| 1052 |
distccd seems to *only* be listening on an IPv6 |
| 1053 |
socket. |
| 1054 |
</blockquote> |
| 1055 |
<p> |
| 1056 |
By default some BSD systems do not allow applications to accept both |
| 1057 |
IPv4 and IPv6 connections through a single server socket. See <a href="http://cert.uni-stuttgart.de/archive/bugtraq/2002/08/msg00319.html">this message</a>. |
| 1058 |
<p> |
| 1059 |
There are several options |
| 1060 |
|
| 1061 |
<ol> |
| 1062 |
<li> Don't build distcc with the <tt>--enable-rfc2553</tt> option unless you need to support IPv6. |
| 1063 |
<li> Tell distcc to explicitly listen on either the IPv4 or IPv6 address, unless you need to accept connections over both protocols. |
| 1064 |
<li> Set the <tt>net.inet6.ip6.v6only</tt> sysctl to 0. (This affects |
| 1065 |
all programs on the system, see the BSD manual for details.) |
| 1066 |
<li> Run two copies of distcc listening on both ports. |
| 1067 |
</ol> |
| 1068 |
|
| 1069 |
|
| 1070 |
<h3>distcc fails with "No locks available"</h3> |
| 1071 |
<p> |
| 1072 |
This can happen if your home directory is on an NFS filesystem and |
| 1073 |
NFS locking is not working. Locking in NFS is supported by a separate |
| 1074 |
protocol and daemon which can be down even if other file operations |
| 1075 |
are working OK. |
| 1076 |
<p> |
| 1077 |
To fix this, either fix NFS locking on your system, or set |
| 1078 |
<tt>DISTCC_DIR</tt> to point to a local disk. If your system doesn't have |
| 1079 |
a local hard disk you can just use <tt>tmpfs</tt> or something similar, |
| 1080 |
because none of the files in that directory need to persist across |
| 1081 |
reboots. |
| 1082 |
|
| 1083 |
|
| 1084 |
<h3>How to stop distcc using a particular machine?</h3> |
| 1085 |
<p> |
| 1086 |
<blockquote> |
| 1087 |
I want to stop my machine accepting distcc jobs because I need it for |
| 1088 |
something else. |
| 1089 |
</blockquote> |
| 1090 |
<p> |
| 1091 |
The easiest way is to just shut down the <tt>distccd</tt> server. When |
| 1092 |
it comes back up, clients will start using it again. |
| 1093 |
|
| 1094 |
|
| 1095 |
<h3>distcc only builds on one machine at a time</h3> |
| 1096 |
<p> |
| 1097 |
<blockquote> |
| 1098 |
If I set |
| 1099 |
<tt>DISTCC_HOSTS="remote_host localhost"</tt> |
| 1100 |
then the project seems to be compiled only on the remote host (I see traces |
| 1101 |
in ethereal and compilation is slow). |
| 1102 |
</blockquote> |
| 1103 |
<p> |
| 1104 |
You are probably not using the <tt>-j</tt> option to make or scons, so |
| 1105 |
it's only compiling one file at a time. Only the first host in the |
| 1106 |
list will be used in this case. |
| 1107 |
|
| 1108 |
|
| 1109 |
|
| 1110 |
<h3>Errors with temporary files on Cygwin</h3> |
| 1111 |
|
| 1112 |
|
| 1113 |
<p> |
| 1114 |
Some Windows compilers can't handle the default <tt>TMPDIR</tt> setting |
| 1115 |
when distcc is run under Cygwin. To fix this, put something like |
| 1116 |
<tt>export TMPDIR=c:/temp</tt> in <tt>/etc/profile</tt>. |
| 1117 |
|
| 1118 |
|
| 1119 |
<h3>Server won't start on Windows 98 or ME</h3> |
| 1120 |
|
| 1121 |
|
| 1122 |
<p> |
| 1123 |
For some reason the server does not work properly on Windows 98 unless it is |
| 1124 |
invoked with <tt>--no-detach</tt>. |
| 1125 |
|
| 1126 |
|
| 1127 |
<h3>Make fails with "file not found"</h3> |
| 1128 |
|
| 1129 |
|
| 1130 |
<p> |
| 1131 |
If Make works for non-parallel builds, but fails when you use <tt>j</tt> and distcc, then there is probably a concurrency bug in your |
| 1132 |
Makefile. It is similar to a threading bug in C, |
| 1133 |
C++ or Java when a program is run on a multiprocessor system. |
| 1134 |
<p> |
| 1135 |
Sometimes it will work with <tt>-j</tt> without distcc, but fail when |
| 1136 |
you use distcc. This is probably because distcc lets the |
| 1137 |
compilation run faster and therefore more jobs run in parallel. |
| 1138 |
<p> |
| 1139 |
You need to just fix your makefile, or not build that particular |
| 1140 |
directory in parallel. It is very unlikely this is a bug in GNU make |
| 1141 |
or distcc. |
| 1142 |
|
| 1143 |
|
| 1144 |
|
| 1145 |
<h3>Internet-wide distributed compilation</h3> |
| 1146 |
<p> |
| 1147 |
|
| 1148 |
<blockquote> |
| 1149 |
I'm a newbie with distcc and linux. I have been able to bootstrap (with |
| 1150 |
lots of effort) Gentoo. I'm having only one computer on my lan that I |
| 1151 |
control, so can't install distcc on the other host. I'm looking for a |
| 1152 |
WAN of public distcc (I would grant access to mine also). I would like |
| 1153 |
have a vast distributed network of distcc that would be public. It may |
| 1154 |
use some ssh. I don't even know if this possible. |
| 1155 |
<p> |
| 1156 |
Won't it be faster then a laptop and a desktop if we have N hosts? I |
| 1157 |
have heard of sub-ether that create a virtual NIC -- as far as I could |
| 1158 |
understand. Any body know if it work with distcc? |
| 1159 |
|
| 1160 |
</blockquote> |
| 1161 |
<p> |
| 1162 |
This could be done, but some preparatory work is needed: |
| 1163 |
<p> |
| 1164 |
This really has to be done across SSH or a VPN; sending commands |
| 1165 |
across the public net unencrypted would be insanely insecure. |
| 1166 |
<p> |
| 1167 |
Connections across the public internet are slower than on a LAN in |
| 1168 |
both throughput and bandwidth. It's probably only feasible if the |
| 1169 |
machines involved are on the same continent (preferably the same |
| 1170 |
state) and have at least 512kbit connections. |
| 1171 |
<p> |
| 1172 |
The more serious problem is compiling on possibly untrusted |
| 1173 |
machines. You need to assume that anyone who sends jobs can take |
| 1174 |
over control of the user running distcc: if it's run within a UML |
| 1175 |
instance or some similar sandbox, and each user has their own |
| 1176 |
instance, that might be OK. Clients also need to implicitly trust |
| 1177 |
all the servers not to give them corrupted or malicious data back. |
| 1178 |
<p> |
| 1179 |
There is also the organizational issue of just locating machines to |
| 1180 |
use and the right public key to get into them. |
| 1181 |
<p> |
| 1182 |
Probably the best way to prototype this is to find a few friends who |
| 1183 |
live in the same country. Set up ssh access into each others |
| 1184 |
machines, and run distcc over that to try compilation. Please report |
| 1185 |
your results to the distcc mailing list. If this works well, then we |
| 1186 |
can look at scaling it up to run on larger groups or between |
| 1187 |
potentially untrusted machines. |
| 1188 |
|
| 1189 |
|
| 1190 |
|
| 1191 |
|
| 1192 |
|
| 1193 |
|
| 1194 |
<h3>What is the relationship between Apple Xcode and distcc?</h3> |
| 1195 |
<p> |
| 1196 |
Apple's <a href="http://developer.apple.com/tools/xcode/">Xcode</a> |
| 1197 |
development kit includes a fork of an old version of distcc, plus |
| 1198 |
Apple patches to locate machines using Rendezvous and a small GUI to |
| 1199 |
configure it. |
| 1200 |
<p> |
| 1201 |
The source is available from |
| 1202 |
<a href="http://www.opensource.apple.com/">opensource.apple.com</a>. |
| 1203 |
|
| 1204 |
<h3>How can I avoid compiling on workstations when they're in use</h3> |
| 1205 |
|
| 1206 |
|
| 1207 |
<p> |
| 1208 |
This <a href="http://lists.samba.org/archive/distcc/2006q1/003271.html"><tt>watch-ssaver</tt> script</a> |
| 1209 |
from <a href="http://sourceforge.net/projects/dmucs">DMUCS</a> |
| 1210 |
enables distcc only when the screensaver is running. |