| 1 |
# A simple comparison |
| 2 |
|
| 3 |
In this experiment we generated one thousand C files with contents |
| 4 |
that looked like this. |
| 5 |
|
| 6 |
```c |
| 7 |
#include<stdio.h> |
| 8 |
#include"header.h" |
| 9 |
|
| 10 |
int func23() { return 0; } |
| 11 |
``` |
| 12 |
|
| 13 |
The function number was different in each file. In addition there was |
| 14 |
a main C file that just called each function in turn. We then |
| 15 |
generated build system files for *Meson*, *CMake*, *SCons*, *Premake* |
| 16 |
and *Autotools* that compiled these files into a single executable. |
| 17 |
|
| 18 |
With this we measured three different things. The first is |
| 19 |
configuration time, that is, the time the build system takes to |
| 20 |
generate necessary build files. This is usually called the *configure |
| 21 |
step*. The time was measured in seconds. |
| 22 |
|
| 23 |
The second thing to measure was the build time. This should be limited |
| 24 |
by the compiler and in the optimal case should be the same for every |
| 25 |
build system. Four parallel processes were used in this test. |
| 26 |
|
| 27 |
The third thing we measured was the empty build time. This measures |
| 28 |
how much time the build system takes to check the states of all source |
| 29 |
files because if any of them could potentially cause a rebuild. |
| 30 |
|
| 31 |
Since CMake has two different backends, Make and Ninja, we ran the |
| 32 |
tests on both of them. All tests were run on a 2011 era MacBook Pro |
| 33 |
running Ubuntu 13/04. The tests were run multiple times and we always |
| 34 |
took the fastest time. |
| 35 |
|
| 36 |
Here are the results for configuration time. |
| 37 |
|
| 38 |
 |
| 39 |
|
| 40 |
The reason SCons got zero seconds on this test is because you cannot |
| 41 |
separate configure and build steps. They run as one unit. Autotools is |
| 42 |
the clear loser of this test as it is over an order of magnitude |
| 43 |
slower than the second slowest one. This configuration time includes |
| 44 |
both autogen and configure. All other systems take less than one |
| 45 |
second to do this setup, which is fast enough for a human being to |
| 46 |
interpret as instantaneous. |
| 47 |
|
| 48 |
 |
| 49 |
|
| 50 |
Build times are a bit more even. SCons is the slowest, being almost |
| 51 |
ten seconds slower than the second slowest. Some of it is work from |
| 52 |
the configure step but it still has the worst performance. Premake is |
| 53 |
the fastest Make-based build system narrowly beating out Autotools. |
| 54 |
Both Ninja-based build systems are faster than all non-Ninja ones with |
| 55 |
Meson being slightly faster. In practice the difference is minimal. |
| 56 |
The advantages of Ninja can be seen by comparing CMake's times when |
| 57 |
using Make or Ninja. It is possible to shave off 3.5 seconds (over |
| 58 |
20%) of the total build time just by changing the backend. The |
| 59 |
project's CMake configuration files don't need any changes. |
| 60 |
|
| 61 |
 |
| 62 |
|
| 63 |
Empty build times reflect the performance of regular build times. |
| 64 |
SCons is again the slowest taking over three seconds compared to |
| 65 |
Meson, which takes only 0.03 seconds, a difference of two orders of |
| 66 |
magnitude. Even Autotools, the fastest Make-based system, is almost |
| 67 |
one order of magnitude slower. Ninja holds the top spots just like in |
| 68 |
the previous test. |
| 69 |
|
| 70 |
Conclusions |
| 71 |
----- |
| 72 |
|
| 73 |
Build system performance matters. Even with this extremely simple |
| 74 |
example we can find differences between various popular build systems. |
| 75 |
As the project size increases, these differences grow even larger. |
| 76 |
(The author has witnessed no-op build times of 30 seconds for Make |
| 77 |
versus less than one second for Ninja when compiling the Clang |
| 78 |
compiler.) Keeping incremental build times low is one of the major |
| 79 |
keys of programmer productivity as it allows developers to iterate |
| 80 |
faster and stay in the creative zone. |
| 81 |
|
| 82 |
Original scripts |
| 83 |
----- |
| 84 |
|
| 85 |
Those who want to run these experiments themselves can download the scripts here: |
| 86 |
|
| 87 |
* [Generator script](https://raw.githubusercontent.com/wiki/jpakkane/meson/gen_src.py) |
| 88 |
* [Measurement script](https://raw.githubusercontent.com/wiki/jpakkane/meson/measure.py) |