| 1 |
--- |
| 2 |
title: Design rationale |
| 3 |
... |
| 4 |
|
| 5 |
This is the original design rationale for Meson. The syntax it |
| 6 |
describes does not match the released version |
| 7 |
== |
| 8 |
|
| 9 |
A software developer's most important tool is the editor. If you talk |
| 10 |
to coders about the editors they use, you are usually met with massive |
| 11 |
enthusiasm and praise. You will hear how Emacs is the greatest thing |
| 12 |
ever or how vi is so elegant or how Eclipse's integration features |
| 13 |
make you so much more productive. You can sense the enthusiasm and |
| 14 |
affection that the people feel towards these programs. |
| 15 |
|
| 16 |
The second most important tool, even more important than the compiler, |
| 17 |
is the build system. |
| 18 |
|
| 19 |
Those are pretty much universally despised. |
| 20 |
|
| 21 |
The most positive statement on build systems you can usually get (and |
| 22 |
it might require some coaxing) is something along the lines of *well, |
| 23 |
it's a terrible system, but all other options are even worse*. It is |
| 24 |
easy to see why this is the case. For starters, commonly used free |
| 25 |
build systems have obtuse syntaxes. They use for the most part global |
| 26 |
variables that are set in random locations so you can never really be |
| 27 |
sure what a given line of code does. They do strange and unpredictable |
| 28 |
things at every turn. |
| 29 |
|
| 30 |
Let's illustrate this with a simple example. Suppose we want to run a |
| 31 |
program built with GNU Autotools under GDB. The instinctive thing to |
| 32 |
do is to just run `gdb programname`. The problem is that this may or |
| 33 |
may not work. In some cases the executable file is a binary whereas at |
| 34 |
other times it is a wrapper shell script that invokes the real binary |
| 35 |
which resides in a hidden subdirectory. GDB invocation fails if the |
| 36 |
binary is a script but succeeds if it is not. The user has to remember |
| 37 |
the type of each one of his executables (which is an implementation |
| 38 |
detail of the build system) just to be able to debug them. Several |
| 39 |
other such pain points can be found in [this blog |
| 40 |
post](http://voices.canonical.com/jussi.pakkanen/2011/09/13/autotools/). |
| 41 |
|
| 42 |
Given these idiosyncrasies it is no wonder that most people don't want |
| 43 |
to have anything to do with build systems. They'll just copy-paste |
| 44 |
code that works (somewhat) in one place to another and hope for the |
| 45 |
best. They actively go out of their way not to understand the system |
| 46 |
because the mere thought of it is repulsive. Doing this also provides |
| 47 |
a kind of inverse job security. If you don't know tool X, there's less |
| 48 |
chance of finding yourself responsible for its use in your |
| 49 |
organisation. Instead you get to work on more enjoyable things. |
| 50 |
|
| 51 |
This leads to a vicious circle. Since people avoid the tools and don't |
| 52 |
want to deal with them, very few work on improving them. The result is |
| 53 |
apathy and stagnation. |
| 54 |
|
| 55 |
Can we do better? |
| 56 |
-- |
| 57 |
|
| 58 |
At its core, building C and C++ code is not a terribly difficult |
| 59 |
task. In fact, writing a text editor is a lot more complicated and |
| 60 |
takes more effort. Yet we have lots of very high quality editors but |
| 61 |
only few build systems with questionable quality and usability. |
| 62 |
|
| 63 |
So, in the grand tradition of own-itch-scratching, I decided to run a |
| 64 |
scientific experiment. The purpose of this experiment was to explore |
| 65 |
what would it take to build a "good" build system. What kind of syntax |
| 66 |
would suit this problem? What sort of problems would this application |
| 67 |
need to solve? What sort of solutions would be the most appropriate? |
| 68 |
|
| 69 |
To get things started, here is a list of requirements any modern |
| 70 |
cross-platform build system needs to provide. |
| 71 |
|
| 72 |
### 1. Must be simple to use |
| 73 |
|
| 74 |
One of the great virtues of Python is the fact that it is very |
| 75 |
readable. It is easy to see what a given block of code does. It is |
| 76 |
concise, clear and easy to understand. The proposed build system must |
| 77 |
be syntactically and semantically clean. Side effects, global state |
| 78 |
and interrelations must be kept at a minimum or, if possible, |
| 79 |
eliminated entirely. |
| 80 |
|
| 81 |
### 2. Must do the right thing by default |
| 82 |
|
| 83 |
Most builds are done by developers working on the code. Therefore the |
| 84 |
defaults must be tailored towards that use case. As an example the |
| 85 |
system shall build objects without optimization and with debug |
| 86 |
information. It shall make binaries that can be run directly from the |
| 87 |
build directory without linker tricks, shell scripts or magic |
| 88 |
environment variables. |
| 89 |
|
| 90 |
### 3. Must enforce established best practices |
| 91 |
|
| 92 |
There really is no reason to compile source code without the |
| 93 |
equivalent of `-Wall`. So enable it by default. A different kind of |
| 94 |
best practice is the total separation of source and build |
| 95 |
directories. All build artifacts must be stored in the build |
| 96 |
directory. Writing stray files in the source directory is not |
| 97 |
permitted under any circumstances. |
| 98 |
|
| 99 |
### 4. Must have native support for platforms that are in common use |
| 100 |
|
| 101 |
A lot of free software projects can be used on non-free platforms such |
| 102 |
as Windows or OSX. The system must provide native support for the |
| 103 |
tools of choice on those platforms. In practice this means native |
| 104 |
support for Visual Studio and XCode. Having said IDEs invoke external |
| 105 |
builder binaries does not count as native support. |
| 106 |
|
| 107 |
### 5. Must not add complexity due to obsolete platforms |
| 108 |
|
| 109 |
Work on this build system started during the Christmas holidays of 2012. |
| 110 |
This provides a natural hard cutoff line of 2012/12/24. Any |
| 111 |
platform, tool or library that was not in active use at that time is |
| 112 |
explicitly not supported. These include Unixes such as IRIX, SunOS, |
| 113 |
OSF-1, Ubuntu versions older than 12/10, GCC versions older than 4.7 |
| 114 |
and so on. If these old versions happen to work, great. If they don't, |
| 115 |
not a single line of code will be added to the system to work around |
| 116 |
their bugs. |
| 117 |
|
| 118 |
### 6. Must be fast |
| 119 |
|
| 120 |
Running the configuration step on a moderate sized project must not |
| 121 |
take more than five seconds. Running the compile command on a fully up |
| 122 |
to date tree of 1000 source files must not take more than 0.1 seconds. |
| 123 |
|
| 124 |
### 7. Must provide easy to use support for modern sw development features |
| 125 |
|
| 126 |
An example is precompiled headers. Currently no free software build |
| 127 |
system provides native support for them. Other examples could include |
| 128 |
easy integration of Valgrind and unit tests, test coverage reporting |
| 129 |
and so on. |
| 130 |
|
| 131 |
### 8. Must allow override of default values |
| 132 |
|
| 133 |
Sometimes you just have to compile files with only given compiler |
| 134 |
flags and no others, or install files in weird places. The system must |
| 135 |
allow the user to do this if he really wants to. |
| 136 |
|
| 137 |
Overview of the solution |
| 138 |
-- |
| 139 |
|
| 140 |
Going over these requirements it becomes quite apparent that the only |
| 141 |
viable approach is roughly the same as taken by CMake: having a domain |
| 142 |
specific language to declare the build system. Out of this declaration |
| 143 |
a configuration is generated for the backend build system. This can be |
| 144 |
a Makefile, Visual Studio or XCode project or anything else. |
| 145 |
|
| 146 |
The difference between the proposed DSL and existing ones is that the |
| 147 |
new one is declarative. It also tries to work on a higher level of |
| 148 |
abstraction than existing systems. As an example, using external |
| 149 |
libraries in current build systems means manually extracting and |
| 150 |
passing around compiler flags and linker flags. In the proposed system |
| 151 |
the user just declares that a given build target uses a given external |
| 152 |
dependency. The build system then takes care of passing all flags and |
| 153 |
settings to their proper locations. This means that the user can focus |
| 154 |
on his own code rather than marshalling command line arguments from |
| 155 |
one place to another. |
| 156 |
|
| 157 |
A DSL is more work than the approach taken by SCons, which is to |
| 158 |
provide the system as a Python library. However it allows us to make |
| 159 |
the syntax more expressive and prevent certain types of bugs by |
| 160 |
e.g. making certain objects truly immutable. The end result is again |
| 161 |
the same: less work for the user. |
| 162 |
|
| 163 |
The backend for Unix requires a bit more thought. The default choice |
| 164 |
would be Make. However it is extremely slow. It is not uncommon on |
| 165 |
large code bases for Make to take several minutes just to determine |
| 166 |
that nothing needs to be done. Instead of Make we use |
| 167 |
[Ninja](https://ninja-build.org/), which is extremely fast. The |
| 168 |
backend code is abstracted away from the core, so other backends can |
| 169 |
be added with relatively little effort. |
| 170 |
|
| 171 |
Sample code |
| 172 |
-- |
| 173 |
|
| 174 |
Enough design talk, let's get to the code. Before looking at the |
| 175 |
examples we would like to emphasize that this is not in any way the |
| 176 |
final code. It is proof of concept code that works in the system as it |
| 177 |
currently exists (February 2013), but may change at any time. |
| 178 |
|
| 179 |
Let's start simple. Here is the code to compile a single executable |
| 180 |
binary. |
| 181 |
|
| 182 |
```meson |
| 183 |
project('compile one', 'c') |
| 184 |
executable('program', 'prog.c') |
| 185 |
``` |
| 186 |
|
| 187 |
This is about as simple as one can get. First you declare the project |
| 188 |
name and the languages it uses. Then you specify the binary to build |
| 189 |
and its sources. The build system will do all the rest. It will add |
| 190 |
proper suffixes (e.g. '.exe' on Windows), set the default compiler |
| 191 |
flags and so on. |
| 192 |
|
| 193 |
Usually programs have more than one source file. Listing them all in |
| 194 |
the function call can become unwieldy. That is why the system supports |
| 195 |
keyword arguments. They look like this. |
| 196 |
|
| 197 |
```meson |
| 198 |
project('compile several', 'c') |
| 199 |
sourcelist = ['main.c', 'file1.c', 'file2.c', 'file3.c'] |
| 200 |
executable('program', sources : sourcelist) |
| 201 |
``` |
| 202 |
|
| 203 |
External dependencies are simple to use. |
| 204 |
|
| 205 |
```meson |
| 206 |
project('external lib', 'c') |
| 207 |
libdep = find_dep('extlibrary', required : true) |
| 208 |
sourcelist = ['main.c', 'file1.c', 'file2.c', 'file3.c'] |
| 209 |
executable('program', sources : sourcelist, dep : libdep) |
| 210 |
``` |
| 211 |
|
| 212 |
In other build systems you have to manually add the compile and link |
| 213 |
flags from external dependencies to targets. In this system you just |
| 214 |
declare that extlibrary is mandatory and that the generated program |
| 215 |
uses that. The build system does all the plumbing for you. |
| 216 |
|
| 217 |
Here's a slightly more complicated definition. It should still be |
| 218 |
understandable. |
| 219 |
|
| 220 |
```meson |
| 221 |
project('build library', 'c') |
| 222 |
foolib = shared_library('foobar', sources : 'foobar.c',\ |
| 223 |
install : true) |
| 224 |
exe = executable('testfoobar', 'tester.c', link : foolib) |
| 225 |
add_test('test library', exe) |
| 226 |
``` |
| 227 |
|
| 228 |
First we build a shared library named foobar. It is marked |
| 229 |
installable, so running `meson install` installs it to the library |
| 230 |
directory (the system knows which one so the user does not have to |
| 231 |
care). Then we build a test executable which is linked against the |
| 232 |
library. It will not be installed, but instead it is added to the list |
| 233 |
of unit tests, which can be run with the command `meson test`. |
| 234 |
|
| 235 |
Above we mentioned precompiled headers as a feature not supported by |
| 236 |
other build systems. Here's how you would use them. |
| 237 |
|
| 238 |
```meson |
| 239 |
project('pch demo', 'cpp') |
| 240 |
executable('myapp', 'myapp.cpp', pch : 'pch/myapp.hh') |
| 241 |
``` |
| 242 |
|
| 243 |
The main reason other build systems cannot provide pch support this |
| 244 |
easily is because they don't enforce certain best practices. Due to |
| 245 |
the way include paths work, it is impossible to provide pch support |
| 246 |
that always works with both in-source and out-of-source |
| 247 |
builds. Mandating separate build and source directories makes this and |
| 248 |
many other problems a lot easier. |
| 249 |
|
| 250 |
Get the code |
| 251 |
-- |
| 252 |
|
| 253 |
The code for this experiment can be found at [the Meson |
| 254 |
repository](https://github.com/mesonbuild/meson). It should be noted |
| 255 |
that (at the time of writing) it is not a build system. It is only |
| 256 |
a proposal for one. It does not work reliably yet. You probably |
| 257 |
should not use it as the build system of your project. |
| 258 |
|
| 259 |
All that said I hope that this experiment will eventually turn into a |
| 260 |
full blown build system. For that I need your help. Comments and |
| 261 |
especially patches are more than welcome. |