| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
=========================== |
| 4 |
How realtime kernels differ |
| 5 |
=========================== |
| 6 |
|
| 7 |
:Author: Sebastian Andrzej Siewior <[email protected]> |
| 8 |
|
| 9 |
Preface |
| 10 |
======= |
| 11 |
|
| 12 |
With forced-threaded interrupts and sleeping spin locks, code paths that |
| 13 |
previously caused long scheduling latencies have been made preemptible and |
| 14 |
moved into process context. This allows the scheduler to manage them more |
| 15 |
effectively and respond to higher-priority tasks with reduced latency. |
| 16 |
|
| 17 |
The following chapters provide an overview of key differences between a |
| 18 |
PREEMPT_RT kernel and a standard, non-PREEMPT_RT kernel. |
| 19 |
|
| 20 |
Locking |
| 21 |
======= |
| 22 |
|
| 23 |
Spinning locks such as spinlock_t are used to provide synchronization for data |
| 24 |
structures accessed from both interrupt context and process context. For this |
| 25 |
reason, locking functions are also available with the _irq() or _irqsave() |
| 26 |
suffixes, which disable interrupts before acquiring the lock. This ensures that |
| 27 |
the lock can be safely acquired in process context when interrupts are enabled. |
| 28 |
|
| 29 |
However, on a PREEMPT_RT system, interrupts are forced-threaded and no longer |
| 30 |
run in hard IRQ context. As a result, there is no need to disable interrupts as |
| 31 |
part of the locking procedure when using spinlock_t. |
| 32 |
|
| 33 |
For low-level core components such as interrupt handling, the scheduler, or the |
| 34 |
timer subsystem the kernel uses raw_spinlock_t. This lock type preserves |
| 35 |
traditional semantics: it disables preemption and, when used with _irq() or |
| 36 |
_irqsave(), also disables interrupts. This ensures proper synchronization in |
| 37 |
critical sections that must remain non-preemptible or with interrupts disabled. |
| 38 |
|
| 39 |
Execution context |
| 40 |
================= |
| 41 |
|
| 42 |
Interrupt handling in a PREEMPT_RT system is invoked in process context through |
| 43 |
the use of threaded interrupts. Other parts of the kernel also shift their |
| 44 |
execution into threaded context by different mechanisms. The goal is to keep |
| 45 |
execution paths preemptible, allowing the scheduler to interrupt them when a |
| 46 |
higher-priority task needs to run. |
| 47 |
|
| 48 |
Below is an overview of the kernel subsystems involved in this transition to |
| 49 |
threaded, preemptible execution. |
| 50 |
|
| 51 |
Interrupt handling |
| 52 |
------------------ |
| 53 |
|
| 54 |
All interrupts are forced-threaded in a PREEMPT_RT system. The exceptions are |
| 55 |
interrupts that are requested with the IRQF_NO_THREAD, IRQF_PERCPU, or |
| 56 |
IRQF_ONESHOT flags. |
| 57 |
|
| 58 |
The IRQF_ONESHOT flag is used together with threaded interrupts, meaning those |
| 59 |
registered using request_threaded_irq() and providing only a threaded handler. |
| 60 |
Its purpose is to keep the interrupt line masked until the threaded handler has |
| 61 |
completed. |
| 62 |
|
| 63 |
If a primary handler is also provided in this case, it is essential that the |
| 64 |
handler does not acquire any sleeping locks, as it will not be threaded. The |
| 65 |
handler should be minimal and must avoid introducing delays, such as |
| 66 |
busy-waiting on hardware registers. |
| 67 |
|
| 68 |
|
| 69 |
Soft interrupts, bottom half handling |
| 70 |
------------------------------------- |
| 71 |
|
| 72 |
Soft interrupts are raised by the interrupt handler and are executed after the |
| 73 |
handler returns. Since they run in thread context, they can be preempted by |
| 74 |
other threads. Do not assume that softirq context runs with preemption |
| 75 |
disabled. This means you must not rely on mechanisms like local_bh_disable() in |
| 76 |
process context to protect per-CPU variables. Because softirq handlers are |
| 77 |
preemptible under PREEMPT_RT, this approach does not provide reliable |
| 78 |
synchronization. |
| 79 |
|
| 80 |
If this kind of protection is required for performance reasons, consider using |
| 81 |
local_lock_nested_bh(). On non-PREEMPT_RT kernels, this allows lockdep to |
| 82 |
verify that bottom halves are disabled. On PREEMPT_RT systems, it adds the |
| 83 |
necessary locking to ensure proper protection. |
| 84 |
|
| 85 |
Using local_lock_nested_bh() also makes the locking scope explicit and easier |
| 86 |
for readers and maintainers to understand. |
| 87 |
|
| 88 |
|
| 89 |
per-CPU variables |
| 90 |
----------------- |
| 91 |
|
| 92 |
Protecting access to per-CPU variables solely by using preempt_disable() should |
| 93 |
be avoided, especially if the critical section has unbounded runtime or may |
| 94 |
call APIs that can sleep. |
| 95 |
|
| 96 |
If using a spinlock_t is considered too costly for performance reasons, |
| 97 |
consider using local_lock_t. On non-PREEMPT_RT configurations, this introduces |
| 98 |
no runtime overhead when lockdep is disabled. With lockdep enabled, it verifies |
| 99 |
that the lock is only acquired in process context and never from softirq or |
| 100 |
hard IRQ context. |
| 101 |
|
| 102 |
On a PREEMPT_RT kernel, local_lock_t is implemented using a per-CPU spinlock_t, |
| 103 |
which provides safe local protection for per-CPU data while keeping the system |
| 104 |
preemptible. |
| 105 |
|
| 106 |
Because spinlock_t on PREEMPT_RT does not disable preemption, it cannot be used |
| 107 |
to protect per-CPU data by relying on implicit preemption disabling. If this |
| 108 |
inherited preemption disabling is essential and if local_lock_t cannot be used |
| 109 |
due to performance constraints, brevity of the code, or abstraction boundaries |
| 110 |
within an API then preempt_disable_nested() may be a suitable alternative. On |
| 111 |
non-PREEMPT_RT kernels, it verifies with lockdep that preemption is already |
| 112 |
disabled. On PREEMPT_RT, it explicitly disables preemption. |
| 113 |
|
| 114 |
Timers |
| 115 |
------ |
| 116 |
|
| 117 |
By default, an hrtimer is executed in hard interrupt context. The exception is |
| 118 |
timers initialized with the HRTIMER_MODE_SOFT flag, which are executed in |
| 119 |
softirq context. |
| 120 |
|
| 121 |
On a PREEMPT_RT kernel, this behavior is reversed: hrtimers are executed in |
| 122 |
softirq context by default, typically within the ktimersd thread. This thread |
| 123 |
runs at the lowest real-time priority, ensuring it executes before any |
| 124 |
SCHED_OTHER tasks but does not interfere with higher-priority real-time |
| 125 |
threads. To explicitly request execution in hard interrupt context on |
| 126 |
PREEMPT_RT, the timer must be marked with the HRTIMER_MODE_HARD flag. |
| 127 |
|
| 128 |
Memory allocation |
| 129 |
----------------- |
| 130 |
|
| 131 |
The memory allocation APIs, such as kmalloc() and alloc_pages(), require a |
| 132 |
gfp_t flag to indicate the allocation context. On non-PREEMPT_RT kernels, it is |
| 133 |
necessary to use GFP_ATOMIC when allocating memory from interrupt context or |
| 134 |
from sections where preemption is disabled. This is because the allocator must |
| 135 |
not sleep in these contexts waiting for memory to become available. |
| 136 |
|
| 137 |
However, this approach does not work on PREEMPT_RT kernels. The memory |
| 138 |
allocator in PREEMPT_RT uses sleeping locks internally, which cannot be |
| 139 |
acquired when preemption is disabled. Fortunately, this is generally not a |
| 140 |
problem, because PREEMPT_RT moves most contexts that would traditionally run |
| 141 |
with preemption or interrupts disabled into threaded context, where sleeping is |
| 142 |
allowed. |
| 143 |
|
| 144 |
What remains problematic is code that explicitly disables preemption or |
| 145 |
interrupts. In such cases, memory allocation must be performed outside the |
| 146 |
critical section. |
| 147 |
|
| 148 |
This restriction also applies to memory deallocation routines such as kfree() |
| 149 |
and free_pages(), which may also involve internal locking and must not be |
| 150 |
called from non-preemptible contexts. |
| 151 |
|
| 152 |
IRQ work |
| 153 |
-------- |
| 154 |
|
| 155 |
The irq_work API provides a mechanism to schedule a callback in interrupt |
| 156 |
context. It is designed for use in contexts where traditional scheduling is not |
| 157 |
possible, such as from within NMI handlers or from inside the scheduler, where |
| 158 |
using a workqueue would be unsafe. |
| 159 |
|
| 160 |
On non-PREEMPT_RT systems, all irq_work items are executed immediately in |
| 161 |
interrupt context. Items marked with IRQ_WORK_LAZY are deferred until the next |
| 162 |
timer tick but are still executed in interrupt context. |
| 163 |
|
| 164 |
On PREEMPT_RT systems, the execution model changes. Because irq_work callbacks |
| 165 |
may acquire sleeping locks or have unbounded execution time, they are handled |
| 166 |
in thread context by a per-CPU irq_work kernel thread. This thread runs at the |
| 167 |
lowest real-time priority, ensuring it executes before any SCHED_OTHER tasks |
| 168 |
but does not interfere with higher-priority real-time threads. |
| 169 |
|
| 170 |
The exception are work items marked with IRQ_WORK_HARD_IRQ, which are still |
| 171 |
executed in hard interrupt context. Lazy items (IRQ_WORK_LAZY) continue to be |
| 172 |
deferred until the next timer tick and are also executed by the irq_work/ |
| 173 |
thread. |
| 174 |
|
| 175 |
RCU callbacks |
| 176 |
------------- |
| 177 |
|
| 178 |
RCU callbacks are invoked by default in softirq context. Their execution is |
| 179 |
important because, depending on the use case, they either free memory or ensure |
| 180 |
progress in state transitions. Running these callbacks as part of the softirq |
| 181 |
chain can lead to undesired situations, such as contention for CPU resources |
| 182 |
with other SCHED_OTHER tasks when executed within ksoftirqd. |
| 183 |
|
| 184 |
To avoid running callbacks in softirq context, the RCU subsystem provides a |
| 185 |
mechanism to execute them in process context instead. This behavior can be |
| 186 |
enabled by setting the boot command-line parameter rcutree.use_softirq=0. This |
| 187 |
setting is enforced in kernels configured with PREEMPT_RT. |
| 188 |
|
| 189 |
Spin until ready |
| 190 |
================ |
| 191 |
|
| 192 |
The "spin until ready" pattern involves repeatedly checking (spinning on) the |
| 193 |
state of a data structure until it becomes available. This pattern assumes that |
| 194 |
preemption, soft interrupts, or interrupts are disabled. If the data structure |
| 195 |
is marked busy, it is presumed to be in use by another CPU, and spinning should |
| 196 |
eventually succeed as that CPU makes progress. |
| 197 |
|
| 198 |
Some examples are hrtimer_cancel() or timer_delete_sync(). These functions |
| 199 |
cancel timers that execute with interrupts or soft interrupts disabled. If a |
| 200 |
thread attempts to cancel a timer and finds it active, spinning until the |
| 201 |
callback completes is safe because the callback can only run on another CPU and |
| 202 |
will eventually finish. |
| 203 |
|
| 204 |
On PREEMPT_RT kernels, however, timer callbacks run in thread context. This |
| 205 |
introduces a challenge: a higher-priority thread attempting to cancel the timer |
| 206 |
may preempt the timer callback thread. Since the scheduler cannot migrate the |
| 207 |
callback thread to another CPU due to affinity constraints, spinning can result |
| 208 |
in livelock even on multiprocessor systems. |
| 209 |
|
| 210 |
To avoid this, both the canceling and callback sides must use a handshake |
| 211 |
mechanism that supports priority inheritance. This allows the canceling thread |
| 212 |
to suspend until the callback completes, ensuring forward progress without |
| 213 |
risking livelock. |
| 214 |
|
| 215 |
In order to solve the problem at the API level, the sequence locks were extended |
| 216 |
to allow a proper handover between the the spinning reader and the maybe |
| 217 |
blocked writer. |
| 218 |
|
| 219 |
Sequence locks |
| 220 |
-------------- |
| 221 |
|
| 222 |
Sequence counters and sequential locks are documented in |
| 223 |
Documentation/locking/seqlock.rst. |
| 224 |
|
| 225 |
The interface has been extended to ensure proper preemption states for the |
| 226 |
writer and spinning reader contexts. This is achieved by embedding the writer |
| 227 |
serialization lock directly into the sequence counter type, resulting in |
| 228 |
composite types such as seqcount_spinlock_t or seqcount_mutex_t. |
| 229 |
|
| 230 |
These composite types allow readers to detect an ongoing write and actively |
| 231 |
boost the writer’s priority to help it complete its update instead of spinning |
| 232 |
and waiting for its completion. |
| 233 |
|
| 234 |
If the plain seqcount_t is used, extra care must be taken to synchronize the |
| 235 |
reader with the writer during updates. The writer must ensure its update is |
| 236 |
serialized and non-preemptible relative to the reader. This cannot be achieved |
| 237 |
using a regular spinlock_t because spinlock_t on PREEMPT_RT does not disable |
| 238 |
preemption. In such cases, using seqcount_spinlock_t is the preferred solution. |
| 239 |
|
| 240 |
However, if there is no spinning involved i.e., if the reader only needs to |
| 241 |
detect whether a write has started and not serialize against it then using |
| 242 |
seqcount_t is reasonable. |