Debian Codegraph @

subprojects/meson-docs/docs/markdown/Simple-comparison.md

raw
1 # A simple comparison
2
3 In this experiment we generated one thousand C files with contents
4 that looked like this.
5
6 ```c
7 #include<stdio.h>
8 #include"header.h"
9
10 int func23() { return 0; }
11 ```
12
13 The function number was different in each file. In addition there was
14 a main C file that just called each function in turn. We then
15 generated build system files for *Meson*, *CMake*, *SCons*, *Premake*
16 and *Autotools* that compiled these files into a single executable.
17
18 With this we measured three different things. The first is
19 configuration time, that is, the time the build system takes to
20 generate necessary build files. This is usually called the *configure
21 step*. The time was measured in seconds.
22
23 The second thing to measure was the build time. This should be limited
24 by the compiler and in the optimal case should be the same for every
25 build system. Four parallel processes were used in this test.
26
27 The third thing we measured was the empty build time. This measures
28 how much time the build system takes to check the states of all source
29 files because if any of them could potentially cause a rebuild.
30
31 Since CMake has two different backends, Make and Ninja, we ran the
32 tests on both of them. All tests were run on a 2011 era MacBook Pro
33 running Ubuntu 13/04. The tests were run multiple times and we always
34 took the fastest time.
35
36 Here are the results for configuration time.
37
38 ![Configuration times](images/conftime.png)
39
40 The reason SCons got zero seconds on this test is because you cannot
41 separate configure and build steps. They run as one unit. Autotools is
42 the clear loser of this test as it is over an order of magnitude
43 slower than the second slowest one. This configuration time includes
44 both autogen and configure. All other systems take less than one
45 second to do this setup, which is fast enough for a human being to
46 interpret as instantaneous.
47
48 ![Build times](https://raw.githubusercontent.com/wiki/jpakkane/meson/buildtime.png)
49
50 Build times are a bit more even. SCons is the slowest, being almost
51 ten seconds slower than the second slowest. Some of it is work from
52 the configure step but it still has the worst performance. Premake is
53 the fastest Make-based build system narrowly beating out Autotools.
54 Both Ninja-based build systems are faster than all non-Ninja ones with
55 Meson being slightly faster. In practice the difference is minimal.
56 The advantages of Ninja can be seen by comparing CMake's times when
57 using Make or Ninja. It is possible to shave off 3.5 seconds (over
58 20%) of the total build time just by changing the backend. The
59 project's CMake configuration files don't need any changes.
60
61 ![No-op time](https://raw.githubusercontent.com/wiki/jpakkane/meson/emptytime.png)
62
63 Empty build times reflect the performance of regular build times.
64 SCons is again the slowest taking over three seconds compared to
65 Meson, which takes only 0.03 seconds, a difference of two orders of
66 magnitude. Even Autotools, the fastest Make-based system, is almost
67 one order of magnitude slower. Ninja holds the top spots just like in
68 the previous test.
69
70 Conclusions
71 -----
72
73 Build system performance matters. Even with this extremely simple
74 example we can find differences between various popular build systems.
75 As the project size increases, these differences grow even larger.
76 (The author has witnessed no-op build times of 30 seconds for Make
77 versus less than one second for Ninja when compiling the Clang
78 compiler.) Keeping incremental build times low is one of the major
79 keys of programmer productivity as it allows developers to iterate
80 faster and stay in the creative zone.
81
82 Original scripts
83 -----
84
85 Those who want to run these experiments themselves can download the scripts here:
86
87 * [Generator script](https://raw.githubusercontent.com/wiki/jpakkane/meson/gen_src.py)
88 * [Measurement script](https://raw.githubusercontent.com/wiki/jpakkane/meson/measure.py)