| 1 |
<HTML> |
| 2 |
<!-- this file was generated by troffcvt and tc2html --> |
| 3 |
<HEAD> |
| 4 |
<TITLE> |
| 5 |
include_server manual page |
| 6 |
</TITLE> |
| 7 |
</HEAD> |
| 8 |
<BODY> |
| 9 |
<H1> |
| 10 |
include_server manual page<BR> |
| 11 |
</H1> |
| 12 |
<P> |
| 13 |
<H2> |
| 14 |
Table of Contents |
| 15 |
</H2> |
| 16 |
<!-- INSERT TOC HERE, MAYBE --> |
| 17 |
<!-- TOC BEGIN --> |
| 18 |
<UL> |
| 19 |
<LI> |
| 20 |
<A HREF=#TOC_1> NAME</A> |
| 21 |
<LI> |
| 22 |
<A HREF=#TOC_2> SYNOPSIS</A> |
| 23 |
<LI> |
| 24 |
<A HREF=#TOC_3> DESCRIPTION</A> |
| 25 |
<LI> |
| 26 |
<A HREF=#TOC_4> OPTION SUMMARY</A> |
| 27 |
<LI> |
| 28 |
<A HREF=#TOC_5> INCLUDE SERVER SYMPTOMS AND ISSUES</A> |
| 29 |
<LI> |
| 30 |
<A HREF=#TOC_6> DISTCC DISCREPANCY SYMPTOMS</A> |
| 31 |
<LI> |
| 32 |
<A HREF=#TOC_7> EXIT CODES</A> |
| 33 |
<LI> |
| 34 |
<A HREF=#TOC_8> ENVIRONMENT VARIABLES</A> |
| 35 |
<LI> |
| 36 |
<A HREF=#TOC_9> BUGS</A> |
| 37 |
<LI> |
| 38 |
<A HREF=#TOC_10> AUTHOR</A> |
| 39 |
<LI> |
| 40 |
<A HREF=#TOC_11> LICENCE</A> |
| 41 |
<LI> |
| 42 |
<A HREF=#TOC_12> SEE ALSO</A> |
| 43 |
</UL> |
| 44 |
<!-- TOC END --> |
| 45 |
<BR> |
| 46 |
<H2> |
| 47 |
<A NAME=TOC_1> |
| 48 |
NAME</A> |
| 49 |
</H2> |
| 50 |
include_server.py - conservative approximation of include dependencies |
| 51 |
for C/C++<BR> |
| 52 |
<H2> |
| 53 |
<A NAME=TOC_2> |
| 54 |
SYNOPSIS</A> |
| 55 |
</H2> |
| 56 |
<B>include_server</B> --port <I> INCLUDE_SERVER_PORT [OPTIONS]</I><BR> |
| 57 |
<H2> |
| 58 |
<A NAME=TOC_3> |
| 59 |
DESCRIPTION</A> |
| 60 |
</H2> |
| 61 |
include_server.py starts an include server process. This process |
| 62 |
answers queries from <B>distcc</B>(1) clients about what files |
| 63 |
to include in C/C++ compilations. The include_server.py command |
| 64 |
itself terminates as soon as the include server has been spawned.<BR> |
| 65 |
<P> |
| 66 |
The INCLUDE_SERVER_PORT argument is the name of a socket used |
| 67 |
for all communication between distcc clients and the include server. |
| 68 |
The <B>pump</B>(1) command is responsible for creating the socket |
| 69 |
location, for passing it to this script, and for passing it to |
| 70 |
all distcc clients via the environment variable named INCLUDE_SERVER_PORT.<BR> |
| 71 |
<P> |
| 72 |
The protocol used by the include server uses distcc's RPC implementation. |
| 73 |
Each distcc request consists of (1) the current directory and |
| 74 |
(2) the list of arguments of the compilation command.<BR> |
| 75 |
<P> |
| 76 |
If the include server is able to process the request, then it |
| 77 |
answers the distcc client by sending a list of filepaths. The |
| 78 |
filepaths are those of the compressed source and header files |
| 79 |
found to be necessary for compilation through include analysis. |
| 80 |
The list also comprises symbolic links and even dummy files needed |
| 81 |
for the compilation server to construct an accurate replica of |
| 82 |
the parts of the filesystem needed for compilation. In this way, |
| 83 |
a needed header file like /path/foo.h is compressed, renamed, |
| 84 |
and stored in a temporary location, such as /dev/shm/tmpiAvfGv.include_server-9368-1/path/foo.h.lzo. |
| 85 |
The distcc client will pass these files on to a compilation server, |
| 86 |
where they will be uncompressed and mounted temporarily.<BR> |
| 87 |
<P> |
| 88 |
If the include server is not able to process the request, then |
| 89 |
it returns the empty list to the distcc client.<BR> |
| 90 |
<P> |
| 91 |
There are two kinds of failures that relate to the include server. |
| 92 |
The include server may fail to compute the includes or fail in |
| 93 |
other ways, see section <B>INCLUDE SERVER SYMPTOMS</B>. Also, |
| 94 |
the compilation on the remove server may fail due to inadequacy |
| 95 |
of the calculated include closure, but then succeed when locally |
| 96 |
retried, see section <B>DISTCC DISCREPANCY SYMPTOMS</B>.<BR> |
| 97 |
<H2> |
| 98 |
<A NAME=TOC_4> |
| 99 |
OPTION SUMMARY</A> |
| 100 |
</H2> |
| 101 |
The following options are understood by include_server.py.<BR> |
| 102 |
<DL> |
| 103 |
<DT> |
| 104 |
<B>-dPAT, --debug_pattern=PAT</B> |
| 105 |
</DT> |
| 106 |
<DD> |
| 107 |
Bit vector for turning on warnings and debugging<BR> |
| 108 |
1 = warnings<BR> |
| 109 |
2 = trace some functions<BR> |
| 110 |
other powers of two: see include_server/basics.py.<BR> |
| 111 |
</DD> |
| 112 |
<DT> |
| 113 |
<B>-e, --email</B> |
| 114 |
</DT> |
| 115 |
<DD> |
| 116 |
Send email to 'distcc-pump-errors' or if defined, the value of |
| 117 |
environment variable DISTCC_EMAILLOG_WHOM_TO_BLAME, when include |
| 118 |
server gets in trouble. The default is to not send email.<BR> |
| 119 |
</DD> |
| 120 |
<DT> |
| 121 |
<B>--email_bound NUMBER</B> |
| 122 |
</DT> |
| 123 |
<DD> |
| 124 |
Maximal number of emails to send (in addition to a final email). |
| 125 |
Default: 3.<BR> |
| 126 |
</DD> |
| 127 |
<DT> |
| 128 |
<B>--no-email</B> |
| 129 |
</DT> |
| 130 |
<DD> |
| 131 |
Do not send email. This is the default.<BR> |
| 132 |
</DD> |
| 133 |
<DT> |
| 134 |
<B>--path_observation_re=RE</B> |
| 135 |
</DT> |
| 136 |
<DD> |
| 137 |
Issue warning message whenever a filename is resolved to a realpath |
| 138 |
that is matched by RE, which is a regular expression in Python |
| 139 |
syntax. This is useful for finding out where files included actually |
| 140 |
come from. Use RE="" to find them all. Note: warnings |
| 141 |
must be enabled with at least -d1.<BR> |
| 142 |
</DD> |
| 143 |
<DT> |
| 144 |
<B>--pid_file FILEPATH</B> |
| 145 |
</DT> |
| 146 |
<DD> |
| 147 |
The pid of the include server is written to file FILEPATH. This |
| 148 |
allows a script such a <B>pump</B> to tear down the include server.<BR> |
| 149 |
</DD> |
| 150 |
<DT> |
| 151 |
<B>-s, --statistics</B> |
| 152 |
</DT> |
| 153 |
<DD> |
| 154 |
Print information to stdout about include analysis.<BR> |
| 155 |
</DD> |
| 156 |
<DT> |
| 157 |
<B>--stat_reset_triggers=LIST</B> |
| 158 |
</DT> |
| 159 |
<DD> |
| 160 |
Flush stat caches when the timestamp of any filepath in LIST changes |
| 161 |
or the filepath comes in or out of existence. LIST is a colon |
| 162 |
separated string of filepaths, possibly containing simple globs |
| 163 |
(as allowed by Python's glob module). Print a warning whenever |
| 164 |
such a change happens (if warnings are enabled). This option allows |
| 165 |
limited exceptions to distcc_pump's normal assumption that source |
| 166 |
files are not modified during the build.<BR> |
| 167 |
</DD> |
| 168 |
<DT> |
| 169 |
<B>-t, --time</B> |
| 170 |
</DT> |
| 171 |
<DD> |
| 172 |
Print elapsed, user, and system time to stderr.<BR> |
| 173 |
</DD> |
| 174 |
<DT> |
| 175 |
<B>--unsafe_absolute_includes</B> |
| 176 |
</DT> |
| 177 |
<DD> |
| 178 |
Do preprocessing on the compilation server even if includes of |
| 179 |
absolute filepaths are encountered. Normally the include-server |
| 180 |
will fall back on local preprocessing if it detects any absolute |
| 181 |
includes. Thus, this flag is useful for preventing such fallbacks |
| 182 |
when the absolute includes are a false alarm, either because the |
| 183 |
absolute include is discarded during preprocessing or because |
| 184 |
the absolutely included file exists on the compilation servers.<BR> |
| 185 |
</DD> |
| 186 |
<DT> |
| 187 |
<BR> |
| 188 |
</DT> |
| 189 |
<DD> |
| 190 |
More precisely, with --unsafe_absolute_includes absolute includes |
| 191 |
are ignored for the purposes of gathering the include closure. |
| 192 |
Using this option may lead to incorrect results because (1) the |
| 193 |
header may actually be included on the compilation server and |
| 194 |
it may not be the same as on the client, (2) the include directives |
| 195 |
of the header are not further analyzed.<BR> |
| 196 |
</DD> |
| 197 |
<DT> |
| 198 |
<BR> |
| 199 |
</DT> |
| 200 |
<DD> |
| 201 |
The option is useful for compiling code that has such hardcoded |
| 202 |
absolute locations of header files inside conditional directives |
| 203 |
(e.g. "#ifdef") that render the includes irrelevant. |
| 204 |
More precisely, these includes must be eliminated during preprocessing |
| 205 |
for the actual configuration. Then the question of existence of |
| 206 |
the header file is moot and the remote compilation is sound. This |
| 207 |
is often the case if such includes are meant for unusual configurations |
| 208 |
different from the actual configuration.<BR> |
| 209 |
</DD> |
| 210 |
<DT> |
| 211 |
<B>-v, --verify</B> |
| 212 |
</DT> |
| 213 |
<DD> |
| 214 |
Verify that files in CPP closure are contained in closure calculated |
| 215 |
by include processor.<BR> |
| 216 |
</DD> |
| 217 |
<DT> |
| 218 |
<B>-w, --write_include_closure</B> |
| 219 |
</DT> |
| 220 |
<DD> |
| 221 |
Write a .d_approx file which lists all the included files calculated |
| 222 |
by the include server; with -x, additionally write the included |
| 223 |
files as calculated by CPP to a .d_exact file.<BR> |
| 224 |
</DD> |
| 225 |
<DT> |
| 226 |
<B>-x, --exact_analysis</B> |
| 227 |
</DT> |
| 228 |
<DD> |
| 229 |
Use CPP instead, do not omit system headers files.<BR> |
| 230 |
</DD> |
| 231 |
</DL> |
| 232 |
<H2> |
| 233 |
<A NAME=TOC_5> |
| 234 |
INCLUDE SERVER SYMPTOMS AND ISSUES</A> |
| 235 |
</H2> |
| 236 |
<BR> |
| 237 |
The most likely messages and warnings to come from the include |
| 238 |
processor are listed below.<BR> |
| 239 |
<P> |
| 240 |
<DL> |
| 241 |
<DT> |
| 242 |
<B>Preprocessing locally. Include server not covering: Couldn't |
| 243 |
determine default system include directories</B> |
| 244 |
</DT> |
| 245 |
<DD> |
| 246 |
To determine the default system header directories, the include |
| 247 |
server runs the compiler once for each language needed during |
| 248 |
its session. This message indicates that the compiler specified |
| 249 |
to distcc is not present on the client.<BR> |
| 250 |
</DD> |
| 251 |
</DL> |
| 252 |
<P> |
| 253 |
<DL> |
| 254 |
<DT> |
| 255 |
<B>Preprocessing locally. Include server not covering: Bailing |
| 256 |
out because include server spent more than ...s user time handling |
| 257 |
request</B> |
| 258 |
</DT> |
| 259 |
<DD> |
| 260 |
In uncommon situations, the include server fails to analyze very |
| 261 |
complicated macro expressions. The distcc client will use plain |
| 262 |
distcc mode.<BR> |
| 263 |
</DD> |
| 264 |
</DL> |
| 265 |
<P> |
| 266 |
<DL> |
| 267 |
<DT> |
| 268 |
<B>Warning: Filepath must be relative but isn't</B> |
| 269 |
</DT> |
| 270 |
<DD> |
| 271 |
The include server does not accept absolute filepaths, such as |
| 272 |
/usr/include/stdio.h, in include directives, because there is |
| 273 |
no guarantee that this header on the compilation server machine |
| 274 |
will be the same as that on the client. The include server gives |
| 275 |
up analyzing the include closure. The distcc client cannot use |
| 276 |
pump-mode.<BR> |
| 277 |
</DD> |
| 278 |
<DT> |
| 279 |
<BR> |
| 280 |
</DT> |
| 281 |
<DD> |
| 282 |
To overcome this problem in a not always reliable way, set the |
| 283 |
environment variable INCLUDE_SERVER_ARGS='--unsafe_absolute_includes' |
| 284 |
when invoking the pump script to pass the --unsafe_absolute_includes |
| 285 |
option to the include server.<BR> |
| 286 |
</DD> |
| 287 |
</DL> |
| 288 |
<P> |
| 289 |
<DL> |
| 290 |
<DT> |
| 291 |
<B>Warning: Absolute filepath ... was IGNORED</B> |
| 292 |
</DT> |
| 293 |
<DD> |
| 294 |
The --unsafe_absolute_includes is in use. This situation happens |
| 295 |
under the same circumstances as when "Filepath must be relative |
| 296 |
but isn't" is issued, but in this case the include will provide |
| 297 |
an answer to the distcc client.<BR> |
| 298 |
</DD> |
| 299 |
</DL> |
| 300 |
<P> |
| 301 |
<DL> |
| 302 |
<DT> |
| 303 |
<B>Warning: Path '/PATH/FILE' changed/came into existence/no longer |
| 304 |
exists</B> |
| 305 |
</DT> |
| 306 |
<DD> |
| 307 |
These warnings are issued when using stat reset triggers. Because |
| 308 |
/PATH/FILE changed, the include server clears its caches; the |
| 309 |
new version of the file (or the lack of it) renders the include |
| 310 |
analysis invalid. This message can usually be ignored; it does |
| 311 |
signify a somewhat precarious use of files by the build system. |
| 312 |
It is recommended to fix the build system so that files are not |
| 313 |
rewritten.<BR> |
| 314 |
</DD> |
| 315 |
</DL> |
| 316 |
<P> |
| 317 |
<DL> |
| 318 |
<DT> |
| 319 |
<B>Warning: For translation unit ..., lookup of file ... resolved |
| 320 |
to ... whose realpath is ...</B> |
| 321 |
</DT> |
| 322 |
<DD> |
| 323 |
This warning occurs with --path_observation_re when a new realpath |
| 324 |
matching a source or header file is observed.<BR> |
| 325 |
</DD> |
| 326 |
</DL> |
| 327 |
<H2> |
| 328 |
<A NAME=TOC_6> |
| 329 |
DISTCC DISCREPANCY SYMPTOMS</A> |
| 330 |
</H2> |
| 331 |
<BR> |
| 332 |
The interactions between the build system, distcc, and the include |
| 333 |
server is somewhat complex. When a distcc commands receives a |
| 334 |
failing compilation from the remote server it retries the compilation |
| 335 |
locally. This section discusses the causes of discrepancies between |
| 336 |
remote and local compilation. These are flagged by the demotion |
| 337 |
message:<BR> |
| 338 |
<P> |
| 339 |
<B>__________Warning: ... pump-mode compilation(s) failed on server,</B> |
| 340 |
<B>but succeeded locally.</B><BR> |
| 341 |
<B>__________Distcc-pump was demoted to plain mode.</B> <B> See |
| 342 |
the Distcc Discrepancy Symptoms section in the include_server(1) |
| 343 |
man</B> <B> page.</B><BR> |
| 344 |
<P> |
| 345 |
The pump script issues this message at the end of the build. This |
| 346 |
means that for at least one distcc invocation a local compilation |
| 347 |
succeeded after the remote compilation failed. Each distcc invocation |
| 348 |
for which such a discrepancy occurred in turn also issues a message |
| 349 |
such as:<BR> |
| 350 |
<P> |
| 351 |
<B>Warning: remote compilation of '...' failed,</B> <B> retried |
| 352 |
locally and got a different result.</B><BR> |
| 353 |
<P> |
| 354 |
The demotion makes subsequent distcc invocations use plain distcc |
| 355 |
mode. Thus preprocessing will take place on the local machine |
| 356 |
for the remainder of the build. This technique prevents very |
| 357 |
slow builds where all compilations end up on the local machine |
| 358 |
after failing remotely.<BR> |
| 359 |
<P> |
| 360 |
Of course, if the local compilations fails after the remote failures, |
| 361 |
then the distcc invocation exits with the non-zero status of the |
| 362 |
local compilation. The error messages printed are also those of |
| 363 |
the local compilation.<BR> |
| 364 |
<P> |
| 365 |
The fallback behavior for distcc-pump mode to local compilation |
| 366 |
can be disabled by setting the environment variable DISTCC_FALLBACK |
| 367 |
to 0, which makes the distcc command fail as soon as the remote |
| 368 |
compilation has failed. This setting is very useful for debugging |
| 369 |
why the remote compilation went wrong, because now the output |
| 370 |
from the server will be printed.<BR> |
| 371 |
<P> |
| 372 |
Next we discuss the possible causes of discrepancies.<BR> |
| 373 |
<P> |
| 374 |
<DL> |
| 375 |
<DT> |
| 376 |
<B>The user changed a source or header file during the build.</B> |
| 377 |
</DT> |
| 378 |
<DD> |
| 379 |
This yields inconsistent results of course.<BR> |
| 380 |
</DD> |
| 381 |
</DL> |
| 382 |
<P> |
| 383 |
<DL> |
| 384 |
<DT> |
| 385 |
<B>A source or header file changed during the build.</B> |
| 386 |
</DT> |
| 387 |
<DD> |
| 388 |
The build system rewrites a file. For Linux kernel 2.6, this |
| 389 |
happens for 'include/linux/compile.h' and 'include/asm/asm-offsets.h'. |
| 390 |
This condition is fixed by letting the include server know that |
| 391 |
it must reset its caches when a stat of any of the files changes. |
| 392 |
Practically, this is done by gathering the files in a colon-separated |
| 393 |
list and then setting the INCLUDE_SERVER_ARGS environment variable |
| 394 |
when invoking the pump script, so that it passes the <B> --stat_reset_triggers</B> |
| 395 |
option; for example,<BR> |
| 396 |
INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"<BR> |
| 397 |
</DD> |
| 398 |
</DL> |
| 399 |
<P> |
| 400 |
<DL> |
| 401 |
<DT> |
| 402 |
<B>A header file is potentially included, but does not exist, |
| 403 |
and is then later included.</B> |
| 404 |
</DT> |
| 405 |
<DD> |
| 406 |
This occurs when some header foo.h includes another header file |
| 407 |
trick.h, but the trick.h file has not yet been generated and the |
| 408 |
inclusion is actually ignored because of preprocessing directives. |
| 409 |
The include server will probe for the existence of trick.h, because |
| 410 |
it overapproximates all possible ways directives actually evaluate. |
| 411 |
The file trick.h is determined not to exist. If it is later generated, |
| 412 |
and then really included, then the include server will falsely |
| 413 |
believe that the file still does not exist. The solution to this |
| 414 |
problem is to make the build system generate trick.h before the |
| 415 |
first time any header file is included that makes a syntactic |
| 416 |
reference to trick.h<BR> |
| 417 |
</DD> |
| 418 |
</DL> |
| 419 |
<P> |
| 420 |
<DL> |
| 421 |
<DT> |
| 422 |
<B>The include server was started with --unsafe_absolute_includes</B>.<B> |
| 423 |
</DT> |
| 424 |
<DD> |
| 425 |
This is a problem if there are header files locally that do not |
| 426 |
exist remotely and that are actually used. Such includes are |
| 427 |
often protected by conditional directives that evaluate so that |
| 428 |
are actually used on only specific and often uncommon platforms. |
| 429 |
If you are not compiling for such a platform, then it may be correct |
| 430 |
to use --unsafe_absolute_include</B>.<BR> |
| 431 |
</DD> |
| 432 |
</DL> |
| 433 |
<P> |
| 434 |
<DL> |
| 435 |
<DT> |
| 436 |
<B>The include server has calculated the wrong includes.</B> |
| 437 |
</DT> |
| 438 |
<DD> |
| 439 |
We do not know of such a situation.<BR> |
| 440 |
</DD> |
| 441 |
</DL> |
| 442 |
<H2> |
| 443 |
<A NAME=TOC_7> |
| 444 |
EXIT CODES</A> |
| 445 |
</H2> |
| 446 |
The exit code of include_server.py is usually 0. That the include |
| 447 |
server has been started properly is communicated through the existence |
| 448 |
of the pid_file.<BR> |
| 449 |
<H2> |
| 450 |
<A NAME=TOC_8> |
| 451 |
ENVIRONMENT VARIABLES</A> |
| 452 |
</H2> |
| 453 |
<BR> |
| 454 |
<B>DISTCC_EMAILLOG_WHOM_TO_BLAME</B> The email address to use |
| 455 |
for include server automated emails. The default is 'distcc-pump-errors' |
| 456 |
(which is an email address that probably will not exist in your |
| 457 |
domain).<BR> |
| 458 |
<P> |
| 459 |
Additionally, the invocation of the compiler may use additional |
| 460 |
environment variables.<BR> |
| 461 |
<H2> |
| 462 |
<A NAME=TOC_9> |
| 463 |
BUGS</A> |
| 464 |
</H2> |
| 465 |
If you think you have found a distcc bug, please see the file |
| 466 |
<I>reporting-bugs.txt</I> in the documentation directory for information |
| 467 |
on how to report it.<BR> |
| 468 |
<P> |
| 469 |
In distcc-pump mode, the include server is unable to handle certain |
| 470 |
very complicated computed includes as found in parts of the Boost |
| 471 |
library. The include server will time out and distcc will revert |
| 472 |
to plain mode.<BR> |
| 473 |
<P> |
| 474 |
Other known bugs may be documented on <I> http://code.google.com/p/distcc/</I><BR> |
| 475 |
<H2> |
| 476 |
<A NAME=TOC_10> |
| 477 |
AUTHOR</A> |
| 478 |
</H2> |
| 479 |
The include server was written by Nils Klarlund, with assistance |
| 480 |
from Fergus Henderson, Manos Renieris, and Craig Silverstein. |
| 481 |
Please report bugs to <[email protected]>.<BR> |
| 482 |
<H2> |
| 483 |
<A NAME=TOC_11> |
| 484 |
LICENCE</A> |
| 485 |
</H2> |
| 486 |
You are free to use distcc. distcc (including this manual) may |
| 487 |
be copied, modified or distributed only under the terms of the |
| 488 |
GNU General Public Licence version 2 or later. distcc comes with |
| 489 |
absolutely no warrany. A copy of the GPL is included in the file |
| 490 |
COPYING.<BR> |
| 491 |
<H2> |
| 492 |
<A NAME=TOC_12> |
| 493 |
SEE ALSO</A> |
| 494 |
</H2> |
| 495 |
<B>distcc</B>(1), <B>distccd</B>(1), <B>include_server</B>(1), |
| 496 |
and <B>gcc</B>(1). http://code.google.com/p/distcc/ https://ccache.dev/<BR> |
| 497 |
</BODY> |
| 498 |
</HTML> |