| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
==================== |
| 4 |
Considering hardware |
| 5 |
==================== |
| 6 |
|
| 7 |
:Author: Sebastian Andrzej Siewior <[email protected]> |
| 8 |
|
| 9 |
The way a workload is handled can be influenced by the hardware it runs on. |
| 10 |
Key components include the CPU, memory, and the buses that connect them. |
| 11 |
These resources are shared among all applications on the system. |
| 12 |
As a result, heavy utilization of one resource by a single application |
| 13 |
can affect the deterministic handling of workloads in other applications. |
| 14 |
|
| 15 |
Below is a brief overview. |
| 16 |
|
| 17 |
System memory and cache |
| 18 |
----------------------- |
| 19 |
|
| 20 |
Main memory and the associated caches are the most common shared resources among |
| 21 |
tasks in a system. One task can dominate the available caches, forcing another |
| 22 |
task to wait until a cache line is written back to main memory before it can |
| 23 |
proceed. The impact of this contention varies based on write patterns and the |
| 24 |
size of the caches available. Larger caches may reduce stalls because more lines |
| 25 |
can be buffered before being written back. Conversely, certain write patterns |
| 26 |
may trigger the cache controller to flush many lines at once, causing |
| 27 |
applications to stall until the operation completes. |
| 28 |
|
| 29 |
This issue can be partly mitigated if applications do not share the same CPU |
| 30 |
cache. The kernel is aware of the cache topology and exports this information to |
| 31 |
user space. Tools such as **lstopo** from the Portable Hardware Locality (hwloc) |
| 32 |
project (https://www.open-mpi.org/projects/hwloc/) can visualize the hierarchy. |
| 33 |
|
| 34 |
Avoiding shared L2 or L3 caches is not always possible. Even when cache sharing |
| 35 |
is minimized, bottlenecks can still occur when accessing system memory. Memory |
| 36 |
is used not only by the CPU but also by peripheral devices via DMA, such as |
| 37 |
graphics cards or network adapters. |
| 38 |
|
| 39 |
In some cases, cache and memory bottlenecks can be controlled if the hardware |
| 40 |
provides the necessary support. On x86 systems, Intel offers Cache Allocation |
| 41 |
Technology (CAT), which enables cache partitioning among applications and |
| 42 |
provides control over the interconnect. AMD provides similar functionality under |
| 43 |
Platform Quality of Service (PQoS). On Arm64, the equivalent is Memory |
| 44 |
System Resource Partitioning and Monitoring (MPAM). |
| 45 |
|
| 46 |
These features can be configured through the Linux Resource Control interface. |
| 47 |
For details, see Documentation/filesystems/resctrl.rst. |
| 48 |
|
| 49 |
The perf tool can be used to monitor cache behavior. It can analyze |
| 50 |
cache misses of an application and compare how they change under |
| 51 |
different workloads on a neighboring CPU. Even more powerful, the perf |
| 52 |
c2c tool can help identify cache-to-cache issues, where multiple CPU |
| 53 |
cores repeatedly access and modify data on the same cache line. |
| 54 |
|
| 55 |
Hardware buses |
| 56 |
-------------- |
| 57 |
|
| 58 |
Real-time systems often need to access hardware directly to perform their work. |
| 59 |
Any latency in this process is undesirable, as it can affect the outcome of the |
| 60 |
task. For example, on an I/O bus, a changed output may not become immediately |
| 61 |
visible but instead appear with variable delay depending on the latency of the |
| 62 |
bus used for communication. |
| 63 |
|
| 64 |
A bus such as PCI is relatively simple because register accesses are routed |
| 65 |
directly to the connected device. In the worst case, a read operation stalls the |
| 66 |
CPU until the device responds. |
| 67 |
|
| 68 |
A bus such as USB is more complex, involving multiple layers. A register read |
| 69 |
or write is wrapped in a USB Request Block (URB), which is then sent by the |
| 70 |
USB host controller to the device. Timing and latency are influenced by the |
| 71 |
underlying USB bus. Requests cannot be sent immediately; they must align with |
| 72 |
the next frame boundary according to the endpoint type and the host controller's |
| 73 |
scheduling rules. This can introduce delays and additional latency. For example, |
| 74 |
a network device connected via USB may still deliver sufficient throughput, but |
| 75 |
the added latency when sending or receiving packets may fail to meet the |
| 76 |
requirements of certain real-time use cases. |
| 77 |
|
| 78 |
Additional restrictions on bus latency can arise from power management. For |
| 79 |
instance, PCIe with Active State Power Management (ASPM) enabled can suspend |
| 80 |
the link between the device and the host. While this behavior is beneficial for |
| 81 |
power savings, it delays device access and adds latency to responses. This issue |
| 82 |
is not limited to PCIe; internal buses within a System-on-Chip (SoC) can also be |
| 83 |
affected by power management mechanisms. |
| 84 |
|
| 85 |
Virtualization |
| 86 |
-------------- |
| 87 |
|
| 88 |
In a virtualized environment such as KVM, each guest CPU is represented as a |
| 89 |
thread on the host. If such a thread runs with real-time priority, the system |
| 90 |
should be tested to confirm it can sustain this behavior over extended periods. |
| 91 |
Because of its priority, the thread will not be preempted by lower-priority |
| 92 |
threads (such as SCHED_OTHER), which may then receive no CPU time. This can |
| 93 |
cause problems if a lower-priority thread is pinned to a CPU already occupied by |
| 94 |
a real-time task and unable to make progress. Even if a CPU has been isolated, |
| 95 |
the system may still (accidentally) start a per‑CPU thread on that CPU. |
| 96 |
Ensuring that a guest CPU goes idle is difficult, as it requires avoiding both |
| 97 |
task scheduling and interrupt handling. Furthermore, if the guest CPU does go |
| 98 |
idle but the guest system is booted with the option **idle=poll**, the guest |
| 99 |
CPU will never enter an idle state and will instead spin until an event |
| 100 |
arrives. |
| 101 |
|
| 102 |
Device handling introduces additional considerations. Emulated PCI devices or |
| 103 |
VirtIO devices require a counterpart on the host to complete requests. This |
| 104 |
adds latency because the host must intercept and either process the request |
| 105 |
directly or schedule a thread for its completion. These delays can be avoided if |
| 106 |
the required PCI device is passed directly through to the guest. Some devices, |
| 107 |
such as networking or storage controllers, support the PCIe SR-IOV feature. |
| 108 |
SR-IOV allows a single PCIe device to be divided into multiple virtual functions, |
| 109 |
which can then be assigned to different guests. |
| 110 |
|
| 111 |
Networking |
| 112 |
---------- |
| 113 |
|
| 114 |
For low-latency networking, the full networking stack may be undesirable, as it |
| 115 |
can introduce additional sources of delay. In this context, XDP can be used |
| 116 |
as a shortcut to bypass much of the stack while still relying on the kernel's |
| 117 |
network driver. |
| 118 |
|
| 119 |
The requirements are that the network driver must support XDP- preferably using |
| 120 |
an "skb pool" and that the application must use an XDP socket. Additional |
| 121 |
configuration may involve BPF filters, tuning networking queues, or configuring |
| 122 |
qdiscs for time-based transmission. These techniques are often |
| 123 |
applied in Time-Sensitive Networking (TSN) environments. |
| 124 |
|
| 125 |
Documenting all required steps exceeds the scope of this text. For detailed |
| 126 |
guidance, see the TSN documentation at https://tsn.readthedocs.io. |
| 127 |
|
| 128 |
Another useful resource is the Linux Real-Time Communication Testbench |
| 129 |
https://github.com/Linutronix/RTC-Testbench. |
| 130 |
The goal of this project is to validate real-time network communication. It can |
| 131 |
be thought of as a "cyclictest" for networking and also serves as a starting |
| 132 |
point for application development. |