| 1 |
# Building `sys/unix` |
| 2 |
|
| 3 |
The sys/unix package provides access to the raw system call interface of the |
| 4 |
underlying operating system. See: https://godoc.org/golang.org/x/sys/unix |
| 5 |
|
| 6 |
Porting Go to a new architecture/OS combination or adding syscalls, types, or |
| 7 |
constants to an existing architecture/OS pair requires some manual effort; |
| 8 |
however, there are tools that automate much of the process. |
| 9 |
|
| 10 |
## Build Systems |
| 11 |
|
| 12 |
There are currently two ways we generate the necessary files. We are currently |
| 13 |
migrating the build system to use containers so the builds are reproducible. |
| 14 |
This is being done on an OS-by-OS basis. Please update this documentation as |
| 15 |
components of the build system change. |
| 16 |
|
| 17 |
### Old Build System (currently for `GOOS != "linux"`) |
| 18 |
|
| 19 |
The old build system generates the Go files based on the C header files |
| 20 |
present on your system. This means that files |
| 21 |
for a given GOOS/GOARCH pair must be generated on a system with that OS and |
| 22 |
architecture. This also means that the generated code can differ from system |
| 23 |
to system, based on differences in the header files. |
| 24 |
|
| 25 |
To avoid this, if you are using the old build system, only generate the Go |
| 26 |
files on an installation with unmodified header files. It is also important to |
| 27 |
keep track of which version of the OS the files were generated from (ex. |
| 28 |
Darwin 14 vs Darwin 15). This makes it easier to track the progress of changes |
| 29 |
and have each OS upgrade correspond to a single change. |
| 30 |
|
| 31 |
To build the files for your current OS and architecture, make sure GOOS and |
| 32 |
GOARCH are set correctly and run `mkall.sh`. This will generate the files for |
| 33 |
your specific system. Running `mkall.sh -n` shows the commands that will be run. |
| 34 |
|
| 35 |
Requirements: bash, go |
| 36 |
|
| 37 |
### New Build System (currently for `GOOS == "linux"`) |
| 38 |
|
| 39 |
The new build system uses a Docker container to generate the go files directly |
| 40 |
from source checkouts of the kernel and various system libraries. This means |
| 41 |
that on any platform that supports Docker, all the files using the new build |
| 42 |
system can be generated at once, and generated files will not change based on |
| 43 |
what the person running the scripts has installed on their computer. |
| 44 |
|
| 45 |
The OS specific files for the new build system are located in the `${GOOS}` |
| 46 |
directory, and the build is coordinated by the `${GOOS}/mkall.go` program. When |
| 47 |
the kernel or system library updates, modify the Dockerfile at |
| 48 |
`${GOOS}/Dockerfile` to checkout the new release of the source. |
| 49 |
|
| 50 |
To build all the files under the new build system, you must be on an amd64/Linux |
| 51 |
system and have your GOOS and GOARCH set accordingly. Running `mkall.sh` will |
| 52 |
then generate all of the files for all of the GOOS/GOARCH pairs in the new build |
| 53 |
system. Running `mkall.sh -n` shows the commands that will be run. |
| 54 |
|
| 55 |
Requirements: bash, go, docker |
| 56 |
|
| 57 |
## Component files |
| 58 |
|
| 59 |
This section describes the various files used in the code generation process. |
| 60 |
It also contains instructions on how to modify these files to add a new |
| 61 |
architecture/OS or to add additional syscalls, types, or constants. Note that |
| 62 |
if you are using the new build system, the scripts/programs cannot be called normally. |
| 63 |
They must be called from within the docker container. |
| 64 |
|
| 65 |
### asm files |
| 66 |
|
| 67 |
The hand-written assembly file at `asm_${GOOS}_${GOARCH}.s` implements system |
| 68 |
call dispatch. There are three entry points: |
| 69 |
``` |
| 70 |
func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) |
| 71 |
func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr) |
| 72 |
func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) |
| 73 |
``` |
| 74 |
The first and second are the standard ones; they differ only in how many |
| 75 |
arguments can be passed to the kernel. The third is for low-level use by the |
| 76 |
ForkExec wrapper. Unlike the first two, it does not call into the scheduler to |
| 77 |
let it know that a system call is running. |
| 78 |
|
| 79 |
When porting Go to a new architecture/OS, this file must be implemented for |
| 80 |
each GOOS/GOARCH pair. |
| 81 |
|
| 82 |
### mksysnum |
| 83 |
|
| 84 |
Mksysnum is a Go program located at `${GOOS}/mksysnum.go` (or `mksysnum_${GOOS}.go` |
| 85 |
for the old system). This program takes in a list of header files containing the |
| 86 |
syscall number declarations and parses them to produce the corresponding list of |
| 87 |
Go numeric constants. See `zsysnum_${GOOS}_${GOARCH}.go` for the generated |
| 88 |
constants. |
| 89 |
|
| 90 |
Adding new syscall numbers is mostly done by running the build on a sufficiently |
| 91 |
new installation of the target OS (or updating the source checkouts for the |
| 92 |
new build system). However, depending on the OS, you may need to update the |
| 93 |
parsing in mksysnum. |
| 94 |
|
| 95 |
### mksyscall.go |
| 96 |
|
| 97 |
The `syscall.go`, `syscall_${GOOS}.go`, `syscall_${GOOS}_${GOARCH}.go` are |
| 98 |
hand-written Go files which implement system calls (for unix, the specific OS, |
| 99 |
or the specific OS/Architecture pair respectively) that need special handling |
| 100 |
and list `//sys` comments giving prototypes for ones that can be generated. |
| 101 |
|
| 102 |
The mksyscall.go program takes the `//sys` and `//sysnb` comments and converts |
| 103 |
them into syscalls. This requires the name of the prototype in the comment to |
| 104 |
match a syscall number in the `zsysnum_${GOOS}_${GOARCH}.go` file. The function |
| 105 |
prototype can be exported (capitalized) or not. |
| 106 |
|
| 107 |
Adding a new syscall often just requires adding a new `//sys` function prototype |
| 108 |
with the desired arguments and a capitalized name so it is exported. However, if |
| 109 |
you want the interface to the syscall to be different, often one will make an |
| 110 |
unexported `//sys` prototype, and then write a custom wrapper in |
| 111 |
`syscall_${GOOS}.go`. |
| 112 |
|
| 113 |
### types files |
| 114 |
|
| 115 |
For each OS, there is a hand-written Go file at `${GOOS}/types.go` (or |
| 116 |
`types_${GOOS}.go` on the old system). This file includes standard C headers and |
| 117 |
creates Go type aliases to the corresponding C types. The file is then fed |
| 118 |
through godef to get the Go compatible definitions. Finally, the generated code |
| 119 |
is fed though mkpost.go to format the code correctly and remove any hidden or |
| 120 |
private identifiers. This cleaned-up code is written to |
| 121 |
`ztypes_${GOOS}_${GOARCH}.go`. |
| 122 |
|
| 123 |
The hardest part about preparing this file is figuring out which headers to |
| 124 |
include and which symbols need to be `#define`d to get the actual data |
| 125 |
structures that pass through to the kernel system calls. Some C libraries |
| 126 |
preset alternate versions for binary compatibility and translate them on the |
| 127 |
way in and out of system calls, but there is almost always a `#define` that can |
| 128 |
get the real ones. |
| 129 |
See `types_darwin.go` and `linux/types.go` for examples. |
| 130 |
|
| 131 |
To add a new type, add in the necessary include statement at the top of the |
| 132 |
file (if it is not already there) and add in a type alias line. Note that if |
| 133 |
your type is significantly different on different architectures, you may need |
| 134 |
some `#if/#elif` macros in your include statements. |
| 135 |
|
| 136 |
### mkerrors.sh |
| 137 |
|
| 138 |
This script is used to generate the system's various constants. This doesn't |
| 139 |
just include the error numbers and error strings, but also the signal numbers |
| 140 |
and a wide variety of miscellaneous constants. The constants come from the list |
| 141 |
of include files in the `includes_${uname}` variable. A regex then picks out |
| 142 |
the desired `#define` statements, and generates the corresponding Go constants. |
| 143 |
The error numbers and strings are generated from `#include <errno.h>`, and the |
| 144 |
signal numbers and strings are generated from `#include <signal.h>`. All of |
| 145 |
these constants are written to `zerrors_${GOOS}_${GOARCH}.go` via a C program, |
| 146 |
`_errors.c`, which prints out all the constants. |
| 147 |
|
| 148 |
To add a constant, add the header that includes it to the appropriate variable. |
| 149 |
Then, edit the regex (if necessary) to match the desired constant. Avoid making |
| 150 |
the regex too broad to avoid matching unintended constants. |
| 151 |
|
| 152 |
### internal/mkmerge |
| 153 |
|
| 154 |
This program is used to extract duplicate const, func, and type declarations |
| 155 |
from the generated architecture-specific files listed below, and merge these |
| 156 |
into a common file for each OS. |
| 157 |
|
| 158 |
The merge is performed in the following steps: |
| 159 |
1. Construct the set of common code that is identical in all architecture-specific files. |
| 160 |
2. Write this common code to the merged file. |
| 161 |
3. Remove the common code from all architecture-specific files. |
| 162 |
|
| 163 |
|
| 164 |
## Generated files |
| 165 |
|
| 166 |
### `zerrors_${GOOS}_${GOARCH}.go` |
| 167 |
|
| 168 |
A file containing all of the system's generated error numbers, error strings, |
| 169 |
signal numbers, and constants. Generated by `mkerrors.sh` (see above). |
| 170 |
|
| 171 |
### `zsyscall_${GOOS}_${GOARCH}.go` |
| 172 |
|
| 173 |
A file containing all the generated syscalls for a specific GOOS and GOARCH. |
| 174 |
Generated by `mksyscall.go` (see above). |
| 175 |
|
| 176 |
### `zsysnum_${GOOS}_${GOARCH}.go` |
| 177 |
|
| 178 |
A list of numeric constants for all the syscall number of the specific GOOS |
| 179 |
and GOARCH. Generated by mksysnum (see above). |
| 180 |
|
| 181 |
### `ztypes_${GOOS}_${GOARCH}.go` |
| 182 |
|
| 183 |
A file containing Go types for passing into (or returning from) syscalls. |
| 184 |
Generated by godefs and the types file (see above). |