| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
######### |
| 4 |
UML HowTo |
| 5 |
######### |
| 6 |
|
| 7 |
.. contents:: :local: |
| 8 |
|
| 9 |
************ |
| 10 |
Introduction |
| 11 |
************ |
| 12 |
|
| 13 |
Welcome to User Mode Linux |
| 14 |
|
| 15 |
User Mode Linux is the first Open Source virtualization platform (first |
| 16 |
release date 1991) and second virtualization platform for an x86 PC. |
| 17 |
|
| 18 |
How is UML Different from a VM using Virtualization package X? |
| 19 |
============================================================== |
| 20 |
|
| 21 |
We have come to assume that virtualization also means some level of |
| 22 |
hardware emulation. In fact, it does not. As long as a virtualization |
| 23 |
package provides the OS with devices which the OS can recognize and |
| 24 |
has a driver for, the devices do not need to emulate real hardware. |
| 25 |
Most OSes today have built-in support for a number of "fake" |
| 26 |
devices used only under virtualization. |
| 27 |
User Mode Linux takes this concept to the ultimate extreme - there |
| 28 |
is not a single real device in sight. It is 100% artificial or if |
| 29 |
we use the correct term 100% paravirtual. All UML devices are abstract |
| 30 |
concepts which map onto something provided by the host - files, sockets, |
| 31 |
pipes, etc. |
| 32 |
|
| 33 |
The other major difference between UML and various virtualization |
| 34 |
packages is that there is a distinct difference between the way the UML |
| 35 |
kernel and the UML programs operate. |
| 36 |
The UML kernel is just a process running on Linux - same as any other |
| 37 |
program. It can be run by an unprivileged user and it does not require |
| 38 |
anything in terms of special CPU features. |
| 39 |
The UML userspace, however, is a bit different. The Linux kernel on the |
| 40 |
host machine assists UML in intercepting everything the program running |
| 41 |
on a UML instance is trying to do and making the UML kernel handle all |
| 42 |
of its requests. |
| 43 |
This is different from other virtualization packages which do not make any |
| 44 |
difference between the guest kernel and guest programs. This difference |
| 45 |
results in a number of advantages and disadvantages of UML over let's say |
| 46 |
QEMU which we will cover later in this document. |
| 47 |
|
| 48 |
|
| 49 |
Why Would I Want User Mode Linux? |
| 50 |
================================= |
| 51 |
|
| 52 |
|
| 53 |
* If User Mode Linux kernel crashes, your host kernel is still fine. It |
| 54 |
is not accelerated in any way (vhost, kvm, etc) and it is not trying to |
| 55 |
access any devices directly. It is, in fact, a process like any other. |
| 56 |
|
| 57 |
* You can run a usermode kernel as a non-root user (you may need to |
| 58 |
arrange appropriate permissions for some devices). |
| 59 |
|
| 60 |
* You can run a very small VM with a minimal footprint for a specific |
| 61 |
task (for example 32M or less). |
| 62 |
|
| 63 |
* You can get extremely high performance for anything which is a "kernel |
| 64 |
specific task" such as forwarding, firewalling, etc while still being |
| 65 |
isolated from the host kernel. |
| 66 |
|
| 67 |
* You can play with kernel concepts without breaking things. |
| 68 |
|
| 69 |
* You are not bound by "emulating" hardware, so you can try weird and |
| 70 |
wonderful concepts which are very difficult to support when emulating |
| 71 |
real hardware such as time travel and making your system clock |
| 72 |
dependent on what UML does (very useful for things like tests). |
| 73 |
|
| 74 |
* It's fun. |
| 75 |
|
| 76 |
Why not to run UML |
| 77 |
================== |
| 78 |
|
| 79 |
* The syscall interception technique used by UML makes it inherently |
| 80 |
slower for any userspace applications. While it can do kernel tasks |
| 81 |
on par with most other virtualization packages, its userspace is |
| 82 |
**slow**. The root cause is that UML has a very high cost of creating |
| 83 |
new processes and threads (something most Unix/Linux applications |
| 84 |
take for granted). |
| 85 |
|
| 86 |
* UML is strictly uniprocessor at present. If you want to run an |
| 87 |
application which needs many CPUs to function, it is clearly the |
| 88 |
wrong choice. |
| 89 |
|
| 90 |
*********************** |
| 91 |
Building a UML instance |
| 92 |
*********************** |
| 93 |
|
| 94 |
There is no UML installer in any distribution. While you can use off |
| 95 |
the shelf install media to install into a blank VM using a virtualization |
| 96 |
package, there is no UML equivalent. You have to use appropriate tools on |
| 97 |
your host to build a viable filesystem image. |
| 98 |
|
| 99 |
This is extremely easy on Debian - you can do it using debootstrap. It is |
| 100 |
also easy on OpenWRT - the build process can build UML images. All other |
| 101 |
distros - YMMV. |
| 102 |
|
| 103 |
Creating an image |
| 104 |
================= |
| 105 |
|
| 106 |
Create a sparse raw disk image:: |
| 107 |
|
| 108 |
# dd if=/dev/zero of=disk_image_name bs=1 count=1 seek=16G |
| 109 |
|
| 110 |
This will create a 16G disk image. The OS will initially allocate only one |
| 111 |
block and will allocate more as they are written by UML. As of kernel |
| 112 |
version 4.19 UML fully supports TRIM (as usually used by flash drives). |
| 113 |
Using TRIM inside the UML image by specifying discard as a mount option |
| 114 |
or by running ``tune2fs -o discard /dev/ubdXX`` will request UML to |
| 115 |
return any unused blocks to the OS. |
| 116 |
|
| 117 |
Create a filesystem on the disk image and mount it:: |
| 118 |
|
| 119 |
# mkfs.ext4 ./disk_image_name && mount ./disk_image_name /mnt |
| 120 |
|
| 121 |
This example uses ext4, any other filesystem such as ext3, btrfs, xfs, |
| 122 |
jfs, etc will work too. |
| 123 |
|
| 124 |
Create a minimal OS installation on the mounted filesystem:: |
| 125 |
|
| 126 |
# debootstrap buster /mnt http://deb.debian.org/debian |
| 127 |
|
| 128 |
debootstrap does not set up the root password, fstab, hostname or |
| 129 |
anything related to networking. It is up to the user to do that. |
| 130 |
|
| 131 |
Set the root password - the easiest way to do that is to chroot into the |
| 132 |
mounted image:: |
| 133 |
|
| 134 |
# chroot /mnt |
| 135 |
# passwd |
| 136 |
# exit |
| 137 |
|
| 138 |
Edit key system files |
| 139 |
===================== |
| 140 |
|
| 141 |
UML block devices are called ubds. The fstab created by debootstrap |
| 142 |
will be empty and it needs an entry for the root file system:: |
| 143 |
|
| 144 |
/dev/ubd0 ext4 discard,errors=remount-ro 0 1 |
| 145 |
|
| 146 |
The image hostname will be set to the same as the host on which you |
| 147 |
are creating its image. It is a good idea to change that to avoid |
| 148 |
"Oh, bummer, I rebooted the wrong machine". |
| 149 |
|
| 150 |
UML supports vector I/O high performance network devices which have |
| 151 |
support for some standard virtual network encapsulations like |
| 152 |
Ethernet over GRE and Ethernet over L2TPv3. These are called vecX. |
| 153 |
|
| 154 |
When vector network devices are in use, ``/etc/network/interfaces`` |
| 155 |
will need entries like:: |
| 156 |
|
| 157 |
# vector UML network devices |
| 158 |
auto vec0 |
| 159 |
iface vec0 inet dhcp |
| 160 |
|
| 161 |
We now have a UML image which is nearly ready to run, all we need is a |
| 162 |
UML kernel and modules for it. |
| 163 |
|
| 164 |
Most distributions have a UML package. Even if you intend to use your own |
| 165 |
kernel, testing the image with a stock one is always a good start. These |
| 166 |
packages come with a set of modules which should be copied to the target |
| 167 |
filesystem. The location is distribution dependent. For Debian these |
| 168 |
reside under /usr/lib/uml/modules. Copy recursively the content of this |
| 169 |
directory to the mounted UML filesystem:: |
| 170 |
|
| 171 |
# cp -rax /usr/lib/uml/modules /mnt/lib/modules |
| 172 |
|
| 173 |
If you have compiled your own kernel, you need to use the usual "install |
| 174 |
modules to a location" procedure by running:: |
| 175 |
|
| 176 |
# make INSTALL_MOD_PATH=/mnt/lib/modules modules_install |
| 177 |
|
| 178 |
This will install modules into /mnt/lib/modules/$(KERNELRELEASE). |
| 179 |
To specify the full module installation path, use:: |
| 180 |
|
| 181 |
# make MODLIB=/mnt/lib/modules modules_install |
| 182 |
|
| 183 |
At this point the image is ready to be brought up. |
| 184 |
|
| 185 |
************************* |
| 186 |
Setting Up UML Networking |
| 187 |
************************* |
| 188 |
|
| 189 |
UML networking is designed to emulate an Ethernet connection. This |
| 190 |
connection may be either point-to-point (similar to a connection |
| 191 |
between machines using a back-to-back cable) or a connection to a |
| 192 |
switch. UML supports a wide variety of means to build these |
| 193 |
connections to all of: local machine, remote machine(s), local and |
| 194 |
remote UML and other VM instances. |
| 195 |
|
| 196 |
|
| 197 |
+-----------+--------+------------------------------------+------------+ |
| 198 |
| Transport | Type | Capabilities | Throughput | |
| 199 |
+===========+========+====================================+============+ |
| 200 |
| tap | vector | checksum, tso | > 8Gbit | |
| 201 |
+-----------+--------+------------------------------------+------------+ |
| 202 |
| hybrid | vector | checksum, tso, multipacket rx | > 6GBit | |
| 203 |
+-----------+--------+------------------------------------+------------+ |
| 204 |
| raw | vector | checksum, tso, multipacket rx, tx" | > 6GBit | |
| 205 |
+-----------+--------+------------------------------------+------------+ |
| 206 |
| EoGRE | vector | multipacket rx, tx | > 3Gbit | |
| 207 |
+-----------+--------+------------------------------------+------------+ |
| 208 |
| Eol2tpv3 | vector | multipacket rx, tx | > 3Gbit | |
| 209 |
+-----------+--------+------------------------------------+------------+ |
| 210 |
| bess | vector | multipacket rx, tx | > 3Gbit | |
| 211 |
+-----------+--------+------------------------------------+------------+ |
| 212 |
| fd | vector | dependent on fd type | varies | |
| 213 |
+-----------+--------+------------------------------------+------------+ |
| 214 |
| vde | vector | dep. on VDE VPN: Virt.Net Locator | varies | |
| 215 |
+-----------+--------+------------------------------------+------------+ |
| 216 |
|
| 217 |
* All transports which have tso and checksum offloads can deliver speeds |
| 218 |
approaching 10G on TCP streams. |
| 219 |
|
| 220 |
* All transports which have multi-packet rx and/or tx can deliver pps |
| 221 |
rates of up to 1Mps or more. |
| 222 |
|
| 223 |
* GRE and L2TPv3 allow connections to all of: local machine, remote |
| 224 |
machines, remote network devices and remote UML instances. |
| 225 |
|
| 226 |
|
| 227 |
Network configuration privileges |
| 228 |
================================ |
| 229 |
|
| 230 |
The majority of the supported networking modes need ``root`` privileges. |
| 231 |
For example, for vector transports, ``root`` privilege is required to fire |
| 232 |
an ioctl to setup the tun interface and/or use raw sockets where needed. |
| 233 |
|
| 234 |
This can be achieved by granting the user a particular capability instead |
| 235 |
of running UML as root. In case of vector transport, a user can add the |
| 236 |
capability ``CAP_NET_ADMIN`` or ``CAP_NET_RAW`` to the uml binary. |
| 237 |
Thenceforth, UML can be run with normal user privilges, along with |
| 238 |
full networking. |
| 239 |
|
| 240 |
For example:: |
| 241 |
|
| 242 |
# sudo setcap cap_net_raw,cap_net_admin+ep linux |
| 243 |
|
| 244 |
Configuring vector transports |
| 245 |
=============================== |
| 246 |
|
| 247 |
All vector transports support a similar syntax: |
| 248 |
|
| 249 |
If X is the interface number as in vec0, vec1, vec2, etc, the general |
| 250 |
syntax for options is:: |
| 251 |
|
| 252 |
vecX:transport="Transport Name",option=value,option=value,...,option=value |
| 253 |
|
| 254 |
Common options |
| 255 |
-------------- |
| 256 |
|
| 257 |
These options are common for all transports: |
| 258 |
|
| 259 |
* ``depth=int`` - sets the queue depth for vector IO. This is the |
| 260 |
amount of packets UML will attempt to read or write in a single |
| 261 |
system call. The default number is 64 and is generally sufficient |
| 262 |
for most applications that need throughput in the 2-4 Gbit range. |
| 263 |
Higher speeds may require larger values. |
| 264 |
|
| 265 |
* ``mac=XX:XX:XX:XX:XX`` - sets the interface MAC address value. |
| 266 |
|
| 267 |
* ``gro=[0,1]`` - sets GRO off or on. Enables receive/transmit offloads. |
| 268 |
The effect of this option depends on the host side support in the transport |
| 269 |
which is being configured. In most cases it will enable TCP segmentation and |
| 270 |
RX/TX checksumming offloads. The setting must be identical on the host side |
| 271 |
and the UML side. The UML kernel will produce warnings if it is not. |
| 272 |
For example, GRO is enabled by default on local machine interfaces |
| 273 |
(e.g. veth pairs, bridge, etc), so it should be enabled in UML in the |
| 274 |
corresponding UML transports (raw, tap, hybrid) in order for networking to |
| 275 |
operate correctly. |
| 276 |
|
| 277 |
* ``mtu=int`` - sets the interface MTU |
| 278 |
|
| 279 |
* ``headroom=int`` - adjusts the default headroom (32 bytes) reserved |
| 280 |
if a packet will need to be re-encapsulated into for instance VXLAN. |
| 281 |
|
| 282 |
* ``vec=0`` - disable multipacket IO and fall back to packet at a |
| 283 |
time mode |
| 284 |
|
| 285 |
Shared Options |
| 286 |
-------------- |
| 287 |
|
| 288 |
* ``ifname=str`` Transports which bind to a local network interface |
| 289 |
have a shared option - the name of the interface to bind to. |
| 290 |
|
| 291 |
* ``src, dst, src_port, dst_port`` - all transports which use sockets |
| 292 |
which have the notion of source and destination and/or source port |
| 293 |
and destination port use these to specify them. |
| 294 |
|
| 295 |
* ``v6=[0,1]`` to specify if a v6 connection is desired for all |
| 296 |
transports which operate over IP. Additionally, for transports that |
| 297 |
have some differences in the way they operate over v4 and v6 (for example |
| 298 |
EoL2TPv3), sets the correct mode of operation. In the absence of this |
| 299 |
option, the socket type is determined based on what do the src and dst |
| 300 |
arguments resolve/parse to. |
| 301 |
|
| 302 |
tap transport |
| 303 |
------------- |
| 304 |
|
| 305 |
Example:: |
| 306 |
|
| 307 |
vecX:transport=tap,ifname=tap0,depth=128,gro=1 |
| 308 |
|
| 309 |
This will connect vec0 to tap0 on the host. Tap0 must already exist (for example |
| 310 |
created using tunctl) and UP. |
| 311 |
|
| 312 |
tap0 can be configured as a point-to-point interface and given an IP |
| 313 |
address so that UML can talk to the host. Alternatively, it is possible |
| 314 |
to connect UML to a tap interface which is connected to a bridge. |
| 315 |
|
| 316 |
While tap relies on the vector infrastructure, it is not a true vector |
| 317 |
transport at this point, because Linux does not support multi-packet |
| 318 |
IO on tap file descriptors for normal userspace apps like UML. This |
| 319 |
is a privilege which is offered only to something which can hook up |
| 320 |
to it at kernel level via specialized interfaces like vhost-net. A |
| 321 |
vhost-net like helper for UML is planned at some point in the future. |
| 322 |
|
| 323 |
Privileges required: tap transport requires either: |
| 324 |
|
| 325 |
* tap interface to exist and be created persistent and owned by the |
| 326 |
UML user using tunctl. Example ``tunctl -u uml-user -t tap0`` |
| 327 |
|
| 328 |
* binary to have ``CAP_NET_ADMIN`` privilege |
| 329 |
|
| 330 |
hybrid transport |
| 331 |
---------------- |
| 332 |
|
| 333 |
Example:: |
| 334 |
|
| 335 |
vecX:transport=hybrid,ifname=tap0,depth=128,gro=1 |
| 336 |
|
| 337 |
This is an experimental/demo transport which couples tap for transmit |
| 338 |
and a raw socket for receive. The raw socket allows multi-packet |
| 339 |
receive resulting in significantly higher packet rates than normal tap. |
| 340 |
|
| 341 |
Privileges required: hybrid requires ``CAP_NET_RAW`` capability by |
| 342 |
the UML user as well as the requirements for the tap transport. |
| 343 |
|
| 344 |
raw socket transport |
| 345 |
-------------------- |
| 346 |
|
| 347 |
Example:: |
| 348 |
|
| 349 |
vecX:transport=raw,ifname=p-veth0,depth=128,gro=1 |
| 350 |
|
| 351 |
|
| 352 |
This transport uses vector IO on raw sockets. While you can bind to any |
| 353 |
interface including a physical one, the most common use it to bind to |
| 354 |
the "peer" side of a veth pair with the other side configured on the |
| 355 |
host. |
| 356 |
|
| 357 |
Example host configuration for Debian: |
| 358 |
|
| 359 |
**/etc/network/interfaces**:: |
| 360 |
|
| 361 |
auto veth0 |
| 362 |
iface veth0 inet static |
| 363 |
address 192.168.4.1 |
| 364 |
netmask 255.255.255.252 |
| 365 |
broadcast 192.168.4.3 |
| 366 |
pre-up ip link add veth0 type veth peer name p-veth0 && \ |
| 367 |
ifconfig p-veth0 up |
| 368 |
|
| 369 |
UML can now bind to p-veth0 like this:: |
| 370 |
|
| 371 |
vec0:transport=raw,ifname=p-veth0,depth=128,gro=1 |
| 372 |
|
| 373 |
|
| 374 |
If the UML guest is configured with 192.168.4.2 and netmask 255.255.255.0 |
| 375 |
it can talk to the host on 192.168.4.1 |
| 376 |
|
| 377 |
The raw transport also provides some support for offloading some of the |
| 378 |
filtering to the host. The two options to control it are: |
| 379 |
|
| 380 |
* ``bpffile=str`` filename of raw bpf code to be loaded as a socket filter |
| 381 |
|
| 382 |
* ``bpfflash=int`` 0/1 allow loading of bpf from inside User Mode Linux. |
| 383 |
This option allows the use of the ethtool load firmware command to |
| 384 |
load bpf code. |
| 385 |
|
| 386 |
In either case the bpf code is loaded into the host kernel. While this is |
| 387 |
presently limited to legacy bpf syntax (not ebpf), it is still a security |
| 388 |
risk. It is not recommended to allow this unless the User Mode Linux |
| 389 |
instance is considered trusted. |
| 390 |
|
| 391 |
Privileges required: raw socket transport requires `CAP_NET_RAW` |
| 392 |
capability. |
| 393 |
|
| 394 |
GRE socket transport |
| 395 |
-------------------- |
| 396 |
|
| 397 |
Example:: |
| 398 |
|
| 399 |
vecX:transport=gre,src=$src_host,dst=$dst_host |
| 400 |
|
| 401 |
|
| 402 |
This will configure an Ethernet over ``GRE`` (aka ``GRETAP`` or |
| 403 |
``GREIRB``) tunnel which will connect the UML instance to a ``GRE`` |
| 404 |
endpoint at host dst_host. ``GRE`` supports the following additional |
| 405 |
options: |
| 406 |
|
| 407 |
* ``rx_key=int`` - GRE 32-bit integer key for rx packets, if set, |
| 408 |
``txkey`` must be set too |
| 409 |
|
| 410 |
* ``tx_key=int`` - GRE 32-bit integer key for tx packets, if set |
| 411 |
``rx_key`` must be set too |
| 412 |
|
| 413 |
* ``sequence=[0,1]`` - enable GRE sequence |
| 414 |
|
| 415 |
* ``pin_sequence=[0,1]`` - pretend that the sequence is always reset |
| 416 |
on each packet (needed to interoperate with some really broken |
| 417 |
implementations) |
| 418 |
|
| 419 |
* ``v6=[0,1]`` - force IPv4 or IPv6 sockets respectively |
| 420 |
|
| 421 |
* GRE checksum is not presently supported |
| 422 |
|
| 423 |
GRE has a number of caveats: |
| 424 |
|
| 425 |
* You can use only one GRE connection per IP address. There is no way to |
| 426 |
multiplex connections as each GRE tunnel is terminated directly on |
| 427 |
the UML instance. |
| 428 |
|
| 429 |
* The key is not really a security feature. While it was intended as such |
| 430 |
its "security" is laughable. It is, however, a useful feature to |
| 431 |
ensure that the tunnel is not misconfigured. |
| 432 |
|
| 433 |
An example configuration for a Linux host with a local address of |
| 434 |
192.168.128.1 to connect to a UML instance at 192.168.129.1 |
| 435 |
|
| 436 |
**/etc/network/interfaces**:: |
| 437 |
|
| 438 |
auto gt0 |
| 439 |
iface gt0 inet static |
| 440 |
address 10.0.0.1 |
| 441 |
netmask 255.255.255.0 |
| 442 |
broadcast 10.0.0.255 |
| 443 |
mtu 1500 |
| 444 |
pre-up ip link add gt0 type gretap local 192.168.128.1 \ |
| 445 |
remote 192.168.129.1 || true |
| 446 |
down ip link del gt0 || true |
| 447 |
|
| 448 |
Additionally, GRE has been tested versus a variety of network equipment. |
| 449 |
|
| 450 |
Privileges required: GRE requires ``CAP_NET_RAW`` |
| 451 |
|
| 452 |
l2tpv3 socket transport |
| 453 |
----------------------- |
| 454 |
|
| 455 |
_Warning_. L2TPv3 has a "bug". It is the "bug" known as "has more |
| 456 |
options than GNU ls". While it has some advantages, there are usually |
| 457 |
easier (and less verbose) ways to connect a UML instance to something. |
| 458 |
For example, most devices which support L2TPv3 also support GRE. |
| 459 |
|
| 460 |
Example:: |
| 461 |
|
| 462 |
vec0:transport=l2tpv3,udp=1,src=$src_host,dst=$dst_host,srcport=$src_port,dstport=$dst_port,depth=128,rx_session=0xffffffff,tx_session=0xffff |
| 463 |
|
| 464 |
This will configure an Ethernet over L2TPv3 fixed tunnel which will |
| 465 |
connect the UML instance to a L2TPv3 endpoint at host $dst_host using |
| 466 |
the L2TPv3 UDP flavour and UDP destination port $dst_port. |
| 467 |
|
| 468 |
L2TPv3 always requires the following additional options: |
| 469 |
|
| 470 |
* ``rx_session=int`` - l2tpv3 32-bit integer session for rx packets |
| 471 |
|
| 472 |
* ``tx_session=int`` - l2tpv3 32-bit integer session for tx packets |
| 473 |
|
| 474 |
As the tunnel is fixed these are not negotiated and they are |
| 475 |
preconfigured on both ends. |
| 476 |
|
| 477 |
Additionally, L2TPv3 supports the following optional parameters. |
| 478 |
|
| 479 |
* ``rx_cookie=int`` - l2tpv3 32-bit integer cookie for rx packets - same |
| 480 |
functionality as GRE key, more to prevent misconfiguration than provide |
| 481 |
actual security |
| 482 |
|
| 483 |
* ``tx_cookie=int`` - l2tpv3 32-bit integer cookie for tx packets |
| 484 |
|
| 485 |
* ``cookie64=[0,1]`` - use 64-bit cookies instead of 32-bit. |
| 486 |
|
| 487 |
* ``counter=[0,1]`` - enable l2tpv3 counter |
| 488 |
|
| 489 |
* ``pin_counter=[0,1]`` - pretend that the counter is always reset on |
| 490 |
each packet (needed to interoperate with some really broken |
| 491 |
implementations) |
| 492 |
|
| 493 |
* ``v6=[0,1]`` - force v6 sockets |
| 494 |
|
| 495 |
* ``udp=[0,1]`` - use raw sockets (0) or UDP (1) version of the protocol |
| 496 |
|
| 497 |
L2TPv3 has a number of caveats: |
| 498 |
|
| 499 |
* you can use only one connection per IP address in raw mode. There is |
| 500 |
no way to multiplex connections as each L2TPv3 tunnel is terminated |
| 501 |
directly on the UML instance. UDP mode can use different ports for |
| 502 |
this purpose. |
| 503 |
|
| 504 |
Here is an example of how to configure a Linux host to connect to UML |
| 505 |
via L2TPv3: |
| 506 |
|
| 507 |
**/etc/network/interfaces**:: |
| 508 |
|
| 509 |
auto l2tp1 |
| 510 |
iface l2tp1 inet static |
| 511 |
address 192.168.126.1 |
| 512 |
netmask 255.255.255.0 |
| 513 |
broadcast 192.168.126.255 |
| 514 |
mtu 1500 |
| 515 |
pre-up ip l2tp add tunnel remote 127.0.0.1 \ |
| 516 |
local 127.0.0.1 encap udp tunnel_id 2 \ |
| 517 |
peer_tunnel_id 2 udp_sport 1706 udp_dport 1707 && \ |
| 518 |
ip l2tp add session name l2tp1 tunnel_id 2 \ |
| 519 |
session_id 0xffffffff peer_session_id 0xffffffff |
| 520 |
down ip l2tp del session tunnel_id 2 session_id 0xffffffff && \ |
| 521 |
ip l2tp del tunnel tunnel_id 2 |
| 522 |
|
| 523 |
|
| 524 |
Privileges required: L2TPv3 requires ``CAP_NET_RAW`` for raw IP mode and |
| 525 |
no special privileges for the UDP mode. |
| 526 |
|
| 527 |
BESS socket transport |
| 528 |
--------------------- |
| 529 |
|
| 530 |
BESS is a high performance modular network switch. |
| 531 |
|
| 532 |
https://github.com/NetSys/bess |
| 533 |
|
| 534 |
It has support for a simple sequential packet socket mode which in the |
| 535 |
more recent versions is using vector IO for high performance. |
| 536 |
|
| 537 |
Example:: |
| 538 |
|
| 539 |
vecX:transport=bess,src=$unix_src,dst=$unix_dst |
| 540 |
|
| 541 |
This will configure a BESS transport using the unix_src Unix domain |
| 542 |
socket address as source and unix_dst socket address as destination. |
| 543 |
|
| 544 |
For BESS configuration and how to allocate a BESS Unix domain socket port |
| 545 |
please see the BESS documentation. |
| 546 |
|
| 547 |
https://github.com/NetSys/bess/wiki/Built-In-Modules-and-Ports |
| 548 |
|
| 549 |
BESS transport does not require any special privileges. |
| 550 |
|
| 551 |
VDE vector transport |
| 552 |
-------------------- |
| 553 |
|
| 554 |
Virtual Distributed Ethernet (VDE) is a project whose main goal is to provide a |
| 555 |
highly flexible support for virtual networking. |
| 556 |
|
| 557 |
http://wiki.virtualsquare.org/#/tutorials/vdebasics |
| 558 |
|
| 559 |
Common usages of VDE include fast prototyping and teaching. |
| 560 |
|
| 561 |
Examples: |
| 562 |
|
| 563 |
``vecX:transport=vde,vnl=tap://tap0`` |
| 564 |
|
| 565 |
use tap0 |
| 566 |
|
| 567 |
``vecX:transport=vde,vnl=slirp://`` |
| 568 |
|
| 569 |
use slirp |
| 570 |
|
| 571 |
``vec0:transport=vde,vnl=vde:///tmp/switch`` |
| 572 |
|
| 573 |
connect to a vde switch |
| 574 |
|
| 575 |
``vecX:transport=\"vde,vnl=cmd://ssh remote.host //tmp/sshlirp\"`` |
| 576 |
|
| 577 |
connect to a remote slirp (instant VPN: convert ssh to VPN, it uses sshlirp) |
| 578 |
https://github.com/virtualsquare/sshlirp |
| 579 |
|
| 580 |
``vec0:transport=vde,vnl=vxvde://234.0.0.1`` |
| 581 |
|
| 582 |
connect to a local area cloud (all the UML nodes using the same |
| 583 |
multicast address running on hosts in the same multicast domain (LAN) |
| 584 |
will be automagically connected together to a virtual LAN. |
| 585 |
|
| 586 |
*********** |
| 587 |
Running UML |
| 588 |
*********** |
| 589 |
|
| 590 |
This section assumes that either the user-mode-linux package from the |
| 591 |
distribution or a custom built kernel has been installed on the host. |
| 592 |
|
| 593 |
These add an executable called linux to the system. This is the UML |
| 594 |
kernel. It can be run just like any other executable. |
| 595 |
It will take most normal linux kernel arguments as command line |
| 596 |
arguments. Additionally, it will need some UML-specific arguments |
| 597 |
in order to do something useful. |
| 598 |
|
| 599 |
Arguments |
| 600 |
========= |
| 601 |
|
| 602 |
Mandatory Arguments: |
| 603 |
-------------------- |
| 604 |
|
| 605 |
* ``mem=int[K,M,G]`` - amount of memory. By default in bytes. It will |
| 606 |
also accept K, M or G qualifiers. |
| 607 |
|
| 608 |
* ``ubdX[s,d,c,t]=`` virtual disk specification. This is not really |
| 609 |
mandatory, but it is likely to be needed in nearly all cases so we can |
| 610 |
specify a root file system. |
| 611 |
The simplest possible image specification is the name of the image |
| 612 |
file for the filesystem (created using one of the methods described |
| 613 |
in `Creating an image`_). |
| 614 |
|
| 615 |
* UBD devices support copy on write (COW). The changes are kept in |
| 616 |
a separate file which can be discarded allowing a rollback to the |
| 617 |
original pristine image. If COW is desired, the UBD image is |
| 618 |
specified as: ``cow_file,master_image``. |
| 619 |
Example:``ubd0=Filesystem.cow,Filesystem.img`` |
| 620 |
|
| 621 |
* UBD devices can be set to use synchronous IO. Any writes are |
| 622 |
immediately flushed to disk. This is done by adding ``s`` after |
| 623 |
the ``ubdX`` specification. |
| 624 |
|
| 625 |
* UBD performs some heuristics on devices specified as a single |
| 626 |
filename to make sure that a COW file has not been specified as |
| 627 |
the image. To turn them off, use the ``d`` flag after ``ubdX``. |
| 628 |
|
| 629 |
* UBD supports TRIM - asking the Host OS to reclaim any unused |
| 630 |
blocks in the image. To turn it off, specify the ``t`` flag after |
| 631 |
``ubdX``. |
| 632 |
|
| 633 |
* ``root=`` root device - most likely ``/dev/ubd0`` (this is a Linux |
| 634 |
filesystem image) |
| 635 |
|
| 636 |
Important Optional Arguments |
| 637 |
---------------------------- |
| 638 |
|
| 639 |
If UML is run as "linux" with no extra arguments, it will try to start an |
| 640 |
xterm for every console configured inside the image (up to 6 in most |
| 641 |
Linux distributions). Each console is started inside an |
| 642 |
xterm. This makes it nice and easy to use UML on a host with a GUI. It is, |
| 643 |
however, the wrong approach if UML is to be used as a testing harness or run |
| 644 |
in a text-only environment. |
| 645 |
|
| 646 |
In order to change this behaviour we need to specify an alternative console |
| 647 |
and wire it to one of the supported "line" channels. For this we need to map a |
| 648 |
console to use something different from the default xterm. |
| 649 |
|
| 650 |
Example which will divert console number 1 to stdin/stdout:: |
| 651 |
|
| 652 |
con1=fd:0,fd:1 |
| 653 |
|
| 654 |
UML supports a wide variety of serial line channels which are specified using |
| 655 |
the following syntax |
| 656 |
|
| 657 |
conX=channel_type:options[,channel_type:options] |
| 658 |
|
| 659 |
|
| 660 |
If the channel specification contains two parts separated by comma, the first |
| 661 |
one is input, the second one output. |
| 662 |
|
| 663 |
* The null channel - Discard all input or output. Example ``con=null`` will set |
| 664 |
all consoles to null by default. |
| 665 |
|
| 666 |
* The fd channel - use file descriptor numbers for input/output. Example: |
| 667 |
``con1=fd:0,fd:1.`` |
| 668 |
|
| 669 |
* The port channel - start a telnet server on TCP port number. Example: |
| 670 |
``con1=port:4321``. The host must have /usr/sbin/in.telnetd (usually part of |
| 671 |
a telnetd package) and the port-helper from the UML utilities (see the |
| 672 |
information for the xterm channel below). UML will not boot until a client |
| 673 |
connects. |
| 674 |
|
| 675 |
* The pty and pts channels - use system pty/pts. |
| 676 |
|
| 677 |
* The tty channel - bind to an existing system tty. Example: ``con1=/dev/tty8`` |
| 678 |
will make UML use the host 8th console (usually unused). |
| 679 |
|
| 680 |
* The xterm channel - this is the default - bring up an xterm on this channel |
| 681 |
and direct IO to it. Note that in order for xterm to work, the host must |
| 682 |
have the UML distribution package installed. This usually contains the |
| 683 |
port-helper and other utilities needed for UML to communicate with the xterm. |
| 684 |
Alternatively, these need to be complied and installed from source. All |
| 685 |
options applicable to consoles also apply to UML serial lines which are |
| 686 |
presented as ttyS inside UML. |
| 687 |
|
| 688 |
Starting UML |
| 689 |
============ |
| 690 |
|
| 691 |
We can now run UML. |
| 692 |
:: |
| 693 |
|
| 694 |
# linux mem=2048M umid=TEST \ |
| 695 |
ubd0=Filesystem.img \ |
| 696 |
vec0:transport=tap,ifname=tap0,depth=128,gro=1 \ |
| 697 |
root=/dev/ubda con=null con0=null,fd:2 con1=fd:0,fd:1 |
| 698 |
|
| 699 |
This will run an instance with ``2048M RAM`` and try to use the image file |
| 700 |
called ``Filesystem.img`` as root. It will connect to the host using tap0. |
| 701 |
All consoles except ``con1`` will be disabled and console 1 will |
| 702 |
use standard input/output making it appear in the same terminal it was started. |
| 703 |
|
| 704 |
Logging in |
| 705 |
============ |
| 706 |
|
| 707 |
If you have not set up a password when generating the image, you will have to |
| 708 |
shut down the UML instance, mount the image, chroot into it and set it - as |
| 709 |
described in the Generating an Image section. If the password is already set, |
| 710 |
you can just log in. |
| 711 |
|
| 712 |
The UML Management Console |
| 713 |
============================ |
| 714 |
|
| 715 |
In addition to managing the image from "the inside" using normal sysadmin tools, |
| 716 |
it is possible to perform a number of low-level operations using the UML |
| 717 |
management console. The UML management console is a low-level interface to the |
| 718 |
kernel on a running UML instance, somewhat like the i386 SysRq interface. Since |
| 719 |
there is a full-blown operating system under UML, there is much greater |
| 720 |
flexibility possible than with the SysRq mechanism. |
| 721 |
|
| 722 |
There are a number of things you can do with the mconsole interface: |
| 723 |
|
| 724 |
* get the kernel version |
| 725 |
* add and remove devices |
| 726 |
* halt or reboot the machine |
| 727 |
* Send SysRq commands |
| 728 |
* Pause and resume the UML |
| 729 |
* Inspect processes running inside UML |
| 730 |
* Inspect UML internal /proc state |
| 731 |
|
| 732 |
You need the mconsole client (uml\_mconsole) which is a part of the UML |
| 733 |
tools package available in most Linux distritions. |
| 734 |
|
| 735 |
You also need ``CONFIG_MCONSOLE`` (under 'General Setup') enabled in the UML |
| 736 |
kernel. When you boot UML, you'll see a line like:: |
| 737 |
|
| 738 |
mconsole initialized on /home/jdike/.uml/umlNJ32yL/mconsole |
| 739 |
|
| 740 |
If you specify a unique machine id on the UML command line, i.e. |
| 741 |
``umid=debian``, you'll see this:: |
| 742 |
|
| 743 |
mconsole initialized on /home/jdike/.uml/debian/mconsole |
| 744 |
|
| 745 |
|
| 746 |
That file is the socket that uml_mconsole will use to communicate with |
| 747 |
UML. Run it with either the umid or the full path as its argument:: |
| 748 |
|
| 749 |
# uml_mconsole debian |
| 750 |
|
| 751 |
or |
| 752 |
|
| 753 |
# uml_mconsole /home/jdike/.uml/debian/mconsole |
| 754 |
|
| 755 |
|
| 756 |
You'll get a prompt, at which you can run one of these commands: |
| 757 |
|
| 758 |
* version |
| 759 |
* help |
| 760 |
* halt |
| 761 |
* reboot |
| 762 |
* config |
| 763 |
* remove |
| 764 |
* sysrq |
| 765 |
* help |
| 766 |
* cad |
| 767 |
* stop |
| 768 |
* go |
| 769 |
* proc |
| 770 |
* stack |
| 771 |
|
| 772 |
version |
| 773 |
------- |
| 774 |
|
| 775 |
This command takes no arguments. It prints the UML version:: |
| 776 |
|
| 777 |
(mconsole) version |
| 778 |
OK Linux OpenWrt 4.14.106 #0 Tue Mar 19 08:19:41 2019 x86_64 |
| 779 |
|
| 780 |
|
| 781 |
There are a couple actual uses for this. It's a simple no-op which |
| 782 |
can be used to check that a UML is running. It's also a way of |
| 783 |
sending a device interrupt to the UML. UML mconsole is treated internally as |
| 784 |
a UML device. |
| 785 |
|
| 786 |
help |
| 787 |
---- |
| 788 |
|
| 789 |
This command takes no arguments. It prints a short help screen with the |
| 790 |
supported mconsole commands. |
| 791 |
|
| 792 |
|
| 793 |
halt and reboot |
| 794 |
--------------- |
| 795 |
|
| 796 |
These commands take no arguments. They shut the machine down immediately, with |
| 797 |
no syncing of disks and no clean shutdown of userspace. So, they are |
| 798 |
pretty close to crashing the machine:: |
| 799 |
|
| 800 |
(mconsole) halt |
| 801 |
OK |
| 802 |
|
| 803 |
config |
| 804 |
------ |
| 805 |
|
| 806 |
"config" adds a new device to the virtual machine. This is supported |
| 807 |
by most UML device drivers. It takes one argument, which is the |
| 808 |
device to add, with the same syntax as the kernel command line:: |
| 809 |
|
| 810 |
(mconsole) config ubd3=/home/jdike/incoming/roots/root_fs_debian22 |
| 811 |
|
| 812 |
remove |
| 813 |
------ |
| 814 |
|
| 815 |
"remove" deletes a device from the system. Its argument is just the |
| 816 |
name of the device to be removed. The device must be idle in whatever |
| 817 |
sense the driver considers necessary. In the case of the ubd driver, |
| 818 |
the removed block device must not be mounted, swapped on, or otherwise |
| 819 |
open, and in the case of the network driver, the device must be down:: |
| 820 |
|
| 821 |
(mconsole) remove ubd3 |
| 822 |
|
| 823 |
sysrq |
| 824 |
----- |
| 825 |
|
| 826 |
This command takes one argument, which is a single letter. It calls the |
| 827 |
generic kernel's SysRq driver, which does whatever is called for by |
| 828 |
that argument. See the SysRq documentation in |
| 829 |
Documentation/admin-guide/sysrq.rst in your favorite kernel tree to |
| 830 |
see what letters are valid and what they do. |
| 831 |
|
| 832 |
cad |
| 833 |
--- |
| 834 |
|
| 835 |
This invokes the ``Ctl-Alt-Del`` action in the running image. What exactly |
| 836 |
this ends up doing is up to init, systemd, etc. Normally, it reboots the |
| 837 |
machine. |
| 838 |
|
| 839 |
stop |
| 840 |
---- |
| 841 |
|
| 842 |
This puts the UML in a loop reading mconsole requests until a 'go' |
| 843 |
mconsole command is received. This is very useful as a |
| 844 |
debugging/snapshotting tool. |
| 845 |
|
| 846 |
go |
| 847 |
-- |
| 848 |
|
| 849 |
This resumes a UML after being paused by a 'stop' command. Note that |
| 850 |
when the UML has resumed, TCP connections may have timed out and if |
| 851 |
the UML is paused for a long period of time, crond might go a little |
| 852 |
crazy, running all the jobs it didn't do earlier. |
| 853 |
|
| 854 |
proc |
| 855 |
---- |
| 856 |
|
| 857 |
This takes one argument - the name of a file in /proc which is printed |
| 858 |
to the mconsole standard output |
| 859 |
|
| 860 |
stack |
| 861 |
----- |
| 862 |
|
| 863 |
This takes one argument - the pid number of a process. Its stack is |
| 864 |
printed to a standard output. |
| 865 |
|
| 866 |
******************* |
| 867 |
Advanced UML Topics |
| 868 |
******************* |
| 869 |
|
| 870 |
Sharing Filesystems between Virtual Machines |
| 871 |
============================================ |
| 872 |
|
| 873 |
Don't attempt to share filesystems simply by booting two UMLs from the |
| 874 |
same file. That's the same thing as booting two physical machines |
| 875 |
from a shared disk. It will result in filesystem corruption. |
| 876 |
|
| 877 |
Using layered block devices |
| 878 |
--------------------------- |
| 879 |
|
| 880 |
The way to share a filesystem between two virtual machines is to use |
| 881 |
the copy-on-write (COW) layering capability of the ubd block driver. |
| 882 |
Any changed blocks are stored in the private COW file, while reads come |
| 883 |
from either device - the private one if the requested block is valid in |
| 884 |
it, the shared one if not. Using this scheme, the majority of data |
| 885 |
which is unchanged is shared between an arbitrary number of virtual |
| 886 |
machines, each of which has a much smaller file containing the changes |
| 887 |
that it has made. With a large number of UMLs booting from a large root |
| 888 |
filesystem, this leads to a huge disk space saving. |
| 889 |
|
| 890 |
Sharing file system data will also help performance, since the host will |
| 891 |
be able to cache the shared data using a much smaller amount of memory, |
| 892 |
so UML disk requests will be served from the host's memory rather than |
| 893 |
its disks. There is a major caveat in doing this on multisocket NUMA |
| 894 |
machines. On such hardware, running many UML instances with a shared |
| 895 |
master image and COW changes may cause issues like NMIs from excess of |
| 896 |
inter-socket traffic. |
| 897 |
|
| 898 |
If you are running UML on high-end hardware like this, make sure to |
| 899 |
bind UML to a set of logical CPUs residing on the same socket using the |
| 900 |
``taskset`` command or have a look at the "tuning" section. |
| 901 |
|
| 902 |
To add a copy-on-write layer to an existing block device file, simply |
| 903 |
add the name of the COW file to the appropriate ubd switch:: |
| 904 |
|
| 905 |
ubd0=root_fs_cow,root_fs_debian_22 |
| 906 |
|
| 907 |
where ``root_fs_cow`` is the private COW file and ``root_fs_debian_22`` is |
| 908 |
the existing shared filesystem. The COW file need not exist. If it |
| 909 |
doesn't, the driver will create and initialize it. |
| 910 |
|
| 911 |
Disk Usage |
| 912 |
---------- |
| 913 |
|
| 914 |
UML has TRIM support which will release any unused space in its disk |
| 915 |
image files to the underlying OS. It is important to use either ls -ls |
| 916 |
or du to verify the actual file size. |
| 917 |
|
| 918 |
COW validity. |
| 919 |
------------- |
| 920 |
|
| 921 |
Any changes to the master image will invalidate all COW files. If this |
| 922 |
happens, UML will *NOT* automatically delete any of the COW files and |
| 923 |
will refuse to boot. In this case the only solution is to either |
| 924 |
restore the old image (including its last modified timestamp) or remove |
| 925 |
all COW files which will result in their recreation. Any changes in |
| 926 |
the COW files will be lost. |
| 927 |
|
| 928 |
Cows can moo - uml_moo : Merging a COW file with its backing file |
| 929 |
----------------------------------------------------------------- |
| 930 |
|
| 931 |
Depending on how you use UML and COW devices, it may be advisable to |
| 932 |
merge the changes in the COW file into the backing file every once in |
| 933 |
a while. |
| 934 |
|
| 935 |
The utility that does this is uml_moo. Its usage is:: |
| 936 |
|
| 937 |
uml_moo COW_file new_backing_file |
| 938 |
|
| 939 |
|
| 940 |
There's no need to specify the backing file since that information is |
| 941 |
already in the COW file header. If you're paranoid, boot the new |
| 942 |
merged file, and if you're happy with it, move it over the old backing |
| 943 |
file. |
| 944 |
|
| 945 |
``uml_moo`` creates a new backing file by default as a safety measure. |
| 946 |
It also has a destructive merge option which will merge the COW file |
| 947 |
directly into its current backing file. This is really only usable |
| 948 |
when the backing file only has one COW file associated with it. If |
| 949 |
there are multiple COWs associated with a backing file, a -d merge of |
| 950 |
one of them will invalidate all of the others. However, it is |
| 951 |
convenient if you're short of disk space, and it should also be |
| 952 |
noticeably faster than a non-destructive merge. |
| 953 |
|
| 954 |
``uml_moo`` is installed with the UML distribution packages and is |
| 955 |
available as a part of UML utilities. |
| 956 |
|
| 957 |
Host file access |
| 958 |
================== |
| 959 |
|
| 960 |
If you want to access files on the host machine from inside UML, you |
| 961 |
can treat it as a separate machine and either nfs mount directories |
| 962 |
from the host or copy files into the virtual machine with scp. |
| 963 |
However, since UML is running on the host, it can access those |
| 964 |
files just like any other process and make them available inside the |
| 965 |
virtual machine without the need to use the network. |
| 966 |
This is possible with the hostfs virtual filesystem. With it, you |
| 967 |
can mount a host directory into the UML filesystem and access the |
| 968 |
files contained in it just as you would on the host. |
| 969 |
|
| 970 |
*SECURITY WARNING* |
| 971 |
|
| 972 |
Hostfs without any parameters to the UML Image will allow the image |
| 973 |
to mount any part of the host filesystem and write to it. Always |
| 974 |
confine hostfs to a specific "harmless" directory (for example ``/var/tmp``) |
| 975 |
if running UML. This is especially important if UML is being run as root. |
| 976 |
|
| 977 |
Using hostfs |
| 978 |
------------ |
| 979 |
|
| 980 |
To begin with, make sure that hostfs is available inside the virtual |
| 981 |
machine with:: |
| 982 |
|
| 983 |
# cat /proc/filesystems |
| 984 |
|
| 985 |
``hostfs`` should be listed. If it's not, either rebuild the kernel |
| 986 |
with hostfs configured into it or make sure that hostfs is built as a |
| 987 |
module and available inside the virtual machine, and insmod it. |
| 988 |
|
| 989 |
|
| 990 |
Now all you need to do is run mount:: |
| 991 |
|
| 992 |
# mount none /mnt/host -t hostfs |
| 993 |
|
| 994 |
will mount the host's ``/`` on the virtual machine's ``/mnt/host``. |
| 995 |
If you don't want to mount the host root directory, then you can |
| 996 |
specify a subdirectory to mount with the -o switch to mount:: |
| 997 |
|
| 998 |
# mount none /mnt/home -t hostfs -o /home |
| 999 |
|
| 1000 |
will mount the host's /home on the virtual machine's /mnt/home. |
| 1001 |
|
| 1002 |
hostfs as the root filesystem |
| 1003 |
----------------------------- |
| 1004 |
|
| 1005 |
It's possible to boot from a directory hierarchy on the host using |
| 1006 |
hostfs rather than using the standard filesystem in a file. |
| 1007 |
To start, you need that hierarchy. The easiest way is to loop mount |
| 1008 |
an existing root_fs file:: |
| 1009 |
|
| 1010 |
# mount root_fs uml_root_dir -o loop |
| 1011 |
|
| 1012 |
|
| 1013 |
You need to change the filesystem type of ``/`` in ``etc/fstab`` to be |
| 1014 |
'hostfs', so that line looks like this:: |
| 1015 |
|
| 1016 |
/dev/ubd/0 / hostfs defaults 1 1 |
| 1017 |
|
| 1018 |
Then you need to chown to yourself all the files in that directory |
| 1019 |
that are owned by root. This worked for me:: |
| 1020 |
|
| 1021 |
# find . -uid 0 -exec chown jdike {} \; |
| 1022 |
|
| 1023 |
Next, make sure that your UML kernel has hostfs compiled in, not as a |
| 1024 |
module. Then run UML with the boot device pointing at that directory:: |
| 1025 |
|
| 1026 |
ubd0=/path/to/uml/root/directory |
| 1027 |
|
| 1028 |
UML should then boot as it does normally. |
| 1029 |
|
| 1030 |
Hostfs Caveats |
| 1031 |
-------------- |
| 1032 |
|
| 1033 |
Hostfs does not support keeping track of host filesystem changes on the |
| 1034 |
host (outside UML). As a result, if a file is changed without UML's |
| 1035 |
knowledge, UML will not know about it and its own in-memory cache of |
| 1036 |
the file may be corrupt. While it is possible to fix this, it is not |
| 1037 |
something which is being worked on at present. |
| 1038 |
|
| 1039 |
Tuning UML |
| 1040 |
============ |
| 1041 |
|
| 1042 |
UML at present is strictly uniprocessor. It will, however spin up a |
| 1043 |
number of threads to handle various functions. |
| 1044 |
|
| 1045 |
The UBD driver, SIGIO and the MMU emulation do that. If the system is |
| 1046 |
idle, these threads will be migrated to other processors on a SMP host. |
| 1047 |
This, unfortunately, will usually result in LOWER performance because of |
| 1048 |
all of the cache/memory synchronization traffic between cores. As a |
| 1049 |
result, UML will usually benefit from being pinned on a single CPU, |
| 1050 |
especially on a large system. This can result in performance differences |
| 1051 |
of 5 times or higher on some benchmarks. |
| 1052 |
|
| 1053 |
Similarly, on large multi-node NUMA systems UML will benefit if all of |
| 1054 |
its memory is allocated from the same NUMA node it will run on. The |
| 1055 |
OS will *NOT* do that by default. In order to do that, the sysadmin |
| 1056 |
needs to create a suitable tmpfs ramdisk bound to a particular node |
| 1057 |
and use that as the source for UML RAM allocation by specifying it |
| 1058 |
in the TMP or TEMP environment variables. UML will look at the values |
| 1059 |
of ``TMPDIR``, ``TMP`` or ``TEMP`` for that. If that fails, it will |
| 1060 |
look for shmfs mounted under ``/dev/shm``. If everything else fails use |
| 1061 |
``/tmp/`` regardless of the filesystem type used for it:: |
| 1062 |
|
| 1063 |
mount -t tmpfs -ompol=bind:X none /mnt/tmpfs-nodeX |
| 1064 |
TEMP=/mnt/tmpfs-nodeX taskset -cX linux options options options.. |
| 1065 |
|
| 1066 |
******************************************* |
| 1067 |
Contributing to UML and Developing with UML |
| 1068 |
******************************************* |
| 1069 |
|
| 1070 |
UML is an excellent platform to develop new Linux kernel concepts - |
| 1071 |
filesystems, devices, virtualization, etc. It provides unrivalled |
| 1072 |
opportunities to create and test them without being constrained to |
| 1073 |
emulating specific hardware. |
| 1074 |
|
| 1075 |
Example - want to try how Linux will work with 4096 "proper" network |
| 1076 |
devices? |
| 1077 |
|
| 1078 |
Not an issue with UML. At the same time, this is something which |
| 1079 |
is difficult with other virtualization packages - they are |
| 1080 |
constrained by the number of devices allowed on the hardware bus |
| 1081 |
they are trying to emulate (for example 16 on a PCI bus in qemu). |
| 1082 |
|
| 1083 |
If you have something to contribute such as a patch, a bugfix, a |
| 1084 |
new feature, please send it to ``[email protected]``. |
| 1085 |
|
| 1086 |
Please follow all standard Linux patch guidelines such as cc-ing |
| 1087 |
relevant maintainers and run ``./scripts/checkpatch.pl`` on your patch. |
| 1088 |
For more details see ``Documentation/process/submitting-patches.rst`` |
| 1089 |
|
| 1090 |
Note - the list does not accept HTML or attachments, all emails must |
| 1091 |
be formatted as plain text. |
| 1092 |
|
| 1093 |
Developing always goes hand in hand with debugging. First of all, |
| 1094 |
you can always run UML under gdb and there will be a whole section |
| 1095 |
later on on how to do that. That, however, is not the only way to |
| 1096 |
debug a Linux kernel. Quite often adding tracing statements and/or |
| 1097 |
using UML specific approaches such as ptracing the UML kernel process |
| 1098 |
are significantly more informative. |
| 1099 |
|
| 1100 |
Tracing UML |
| 1101 |
============= |
| 1102 |
|
| 1103 |
When running, UML consists of a main kernel thread and a number of |
| 1104 |
helper threads. The ones of interest for tracing are NOT the ones |
| 1105 |
that are already ptraced by UML as a part of its MMU emulation. |
| 1106 |
|
| 1107 |
These are usually the first three threads visible in a ps display. |
| 1108 |
The one with the lowest PID number and using most CPU is usually the |
| 1109 |
kernel thread. The other threads are the disk |
| 1110 |
(ubd) device helper thread and the SIGIO helper thread. |
| 1111 |
Running ptrace on this thread usually results in the following picture:: |
| 1112 |
|
| 1113 |
host$ strace -p 16566 |
| 1114 |
--- SIGIO {si_signo=SIGIO, si_code=POLL_IN, si_band=65} --- |
| 1115 |
epoll_wait(4, [{EPOLLIN, {u32=3721159424, u64=3721159424}}], 64, 0) = 1 |
| 1116 |
epoll_wait(4, [], 64, 0) = 0 |
| 1117 |
rt_sigreturn({mask=[PIPE]}) = 16967 |
| 1118 |
ptrace(PTRACE_GETREGS, 16967, NULL, 0xd5f34f38) = 0 |
| 1119 |
ptrace(PTRACE_GETREGSET, 16967, NT_X86_XSTATE, [{iov_base=0xd5f35010, iov_len=832}]) = 0 |
| 1120 |
ptrace(PTRACE_GETSIGINFO, 16967, NULL, {si_signo=SIGTRAP, si_code=0x85, si_pid=16967, si_uid=0}) = 0 |
| 1121 |
ptrace(PTRACE_SETREGS, 16967, NULL, 0xd5f34f38) = 0 |
| 1122 |
ptrace(PTRACE_SETREGSET, 16967, NT_X86_XSTATE, [{iov_base=0xd5f35010, iov_len=2696}]) = 0 |
| 1123 |
ptrace(PTRACE_SYSEMU, 16967, NULL, 0) = 0 |
| 1124 |
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=16967, si_uid=0, si_status=SIGTRAP, si_utime=65, si_stime=89} --- |
| 1125 |
wait4(16967, [{WIFSTOPPED(s) && WSTOPSIG(s) == SIGTRAP | 0x80}], WSTOPPED|__WALL, NULL) = 16967 |
| 1126 |
ptrace(PTRACE_GETREGS, 16967, NULL, 0xd5f34f38) = 0 |
| 1127 |
ptrace(PTRACE_GETREGSET, 16967, NT_X86_XSTATE, [{iov_base=0xd5f35010, iov_len=832}]) = 0 |
| 1128 |
ptrace(PTRACE_GETSIGINFO, 16967, NULL, {si_signo=SIGTRAP, si_code=0x85, si_pid=16967, si_uid=0}) = 0 |
| 1129 |
timer_settime(0, 0, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=2830912}}, NULL) = 0 |
| 1130 |
getpid() = 16566 |
| 1131 |
clock_nanosleep(CLOCK_MONOTONIC, 0, {tv_sec=1, tv_nsec=0}, NULL) = ? ERESTART_RESTARTBLOCK (Interrupted by signal) |
| 1132 |
--- SIGALRM {si_signo=SIGALRM, si_code=SI_TIMER, si_timerid=0, si_overrun=0, si_value={int=1631716592, ptr=0x614204f0}} --- |
| 1133 |
rt_sigreturn({mask=[PIPE]}) = -1 EINTR (Interrupted system call) |
| 1134 |
|
| 1135 |
This is a typical picture from a mostly idle UML instance. |
| 1136 |
|
| 1137 |
* UML interrupt controller uses epoll - this is UML waiting for IO |
| 1138 |
interrupts: |
| 1139 |
|
| 1140 |
epoll_wait(4, [{EPOLLIN, {u32=3721159424, u64=3721159424}}], 64, 0) = 1 |
| 1141 |
|
| 1142 |
* The sequence of ptrace calls is part of MMU emulation and running the |
| 1143 |
UML userspace. |
| 1144 |
* ``timer_settime`` is part of the UML high res timer subsystem mapping |
| 1145 |
timer requests from inside UML onto the host high resolution timers. |
| 1146 |
* ``clock_nanosleep`` is UML going into idle (similar to the way a PC |
| 1147 |
will execute an ACPI idle). |
| 1148 |
|
| 1149 |
As you can see UML will generate quite a bit of output even in idle. The output |
| 1150 |
can be very informative when observing IO. It shows the actual IO calls, their |
| 1151 |
arguments and returns values. |
| 1152 |
|
| 1153 |
Kernel debugging |
| 1154 |
================ |
| 1155 |
|
| 1156 |
You can run UML under gdb now, though it will not necessarily agree to |
| 1157 |
be started under it. If you are trying to track a runtime bug, it is |
| 1158 |
much better to attach gdb to a running UML instance and let UML run. |
| 1159 |
|
| 1160 |
Assuming the same PID number as in the previous example, this would be:: |
| 1161 |
|
| 1162 |
# gdb -p 16566 |
| 1163 |
|
| 1164 |
This will STOP the UML instance, so you must enter `cont` at the GDB |
| 1165 |
command line to request it to continue. It may be a good idea to make |
| 1166 |
this into a gdb script and pass it to gdb as an argument. |
| 1167 |
|
| 1168 |
Developing Device Drivers |
| 1169 |
========================= |
| 1170 |
|
| 1171 |
Nearly all UML drivers are monolithic. While it is possible to build a |
| 1172 |
UML driver as a kernel module, that limits the possible functionality |
| 1173 |
to in-kernel only and non-UML specific. The reason for this is that |
| 1174 |
in order to really leverage UML, one needs to write a piece of |
| 1175 |
userspace code which maps driver concepts onto actual userspace host |
| 1176 |
calls. |
| 1177 |
|
| 1178 |
This forms the so-called "user" portion of the driver. While it can |
| 1179 |
reuse a lot of kernel concepts, it is generally just another piece of |
| 1180 |
userspace code. This portion needs some matching "kernel" code which |
| 1181 |
resides inside the UML image and which implements the Linux kernel part. |
| 1182 |
|
| 1183 |
*Note: There are very few limitations in the way "kernel" and "user" interact*. |
| 1184 |
|
| 1185 |
UML does not have a strictly defined kernel-to-host API. It does not |
| 1186 |
try to emulate a specific architecture or bus. UML's "kernel" and |
| 1187 |
"user" can share memory, code and interact as needed to implement |
| 1188 |
whatever design the software developer has in mind. The only |
| 1189 |
limitations are purely technical. Due to a lot of functions and |
| 1190 |
variables having the same names, the developer should be careful |
| 1191 |
which includes and libraries they are trying to refer to. |
| 1192 |
|
| 1193 |
As a result a lot of userspace code consists of simple wrappers. |
| 1194 |
E.g. ``os_close_file()`` is just a wrapper around ``close()`` |
| 1195 |
which ensures that the userspace function close does not clash |
| 1196 |
with similarly named function(s) in the kernel part. |
| 1197 |
|
| 1198 |
Using UML as a Test Platform |
| 1199 |
============================ |
| 1200 |
|
| 1201 |
UML is an excellent test platform for device driver development. As |
| 1202 |
with most things UML, "some user assembly may be required". It is |
| 1203 |
up to the user to build their emulation environment. UML at present |
| 1204 |
provides only the kernel infrastructure. |
| 1205 |
|
| 1206 |
Part of this infrastructure is the ability to load and parse fdt |
| 1207 |
device tree blobs as used in Arm or Open Firmware platforms. These |
| 1208 |
are supplied as an optional extra argument to the kernel command |
| 1209 |
line:: |
| 1210 |
|
| 1211 |
dtb=filename |
| 1212 |
|
| 1213 |
The device tree is loaded and parsed at boottime and is accessible by |
| 1214 |
drivers which query it. At this moment in time this facility is |
| 1215 |
intended solely for development purposes. UML's own devices do not |
| 1216 |
query the device tree. |
| 1217 |
|
| 1218 |
Security Considerations |
| 1219 |
----------------------- |
| 1220 |
|
| 1221 |
Drivers or any new functionality should default to not |
| 1222 |
accepting arbitrary filename, bpf code or other parameters |
| 1223 |
which can affect the host from inside the UML instance. |
| 1224 |
For example, specifying the socket used for IPC communication |
| 1225 |
between a driver and the host at the UML command line is OK |
| 1226 |
security-wise. Allowing it as a loadable module parameter |
| 1227 |
isn't. |
| 1228 |
|
| 1229 |
If such functionality is desirable for a particular application |
| 1230 |
(e.g. loading BPF "firmware" for raw socket network transports), |
| 1231 |
it should be off by default and should be explicitly turned on |
| 1232 |
as a command line parameter at startup. |
| 1233 |
|
| 1234 |
Even with this in mind, the level of isolation between UML |
| 1235 |
and the host is relatively weak. If the UML userspace is |
| 1236 |
allowed to load arbitrary kernel drivers, an attacker can |
| 1237 |
use this to break out of UML. Thus, if UML is used in |
| 1238 |
a production application, it is recommended that all modules |
| 1239 |
are loaded at boot and kernel module loading is disabled |
| 1240 |
afterwards. |