| 1 |
Real-time application monitors |
| 2 |
============================== |
| 3 |
|
| 4 |
- Name: rtapp |
| 5 |
- Type: container for multiple monitors |
| 6 |
- Author: Nam Cao <[email protected]> |
| 7 |
|
| 8 |
Description |
| 9 |
----------- |
| 10 |
|
| 11 |
Real-time applications may have design flaws such that they experience |
| 12 |
unexpected latency and fail to meet their time requirements. Often, these flaws |
| 13 |
follow a few patterns: |
| 14 |
|
| 15 |
- Page faults: A real-time thread may access memory that does not have a |
| 16 |
mapped physical backing or must first be copied (such as for copy-on-write). |
| 17 |
Thus a page fault is raised and the kernel must first perform the expensive |
| 18 |
action. This causes significant delays to the real-time thread |
| 19 |
- Priority inversion: A real-time thread blocks waiting for a lower-priority |
| 20 |
thread. This causes the real-time thread to effectively take on the |
| 21 |
scheduling priority of the lower-priority thread. For example, the real-time |
| 22 |
thread needs to access a shared resource that is protected by a |
| 23 |
non-pi-mutex, but the mutex is currently owned by a non-real-time thread. |
| 24 |
|
| 25 |
The `rtapp` monitor detects these patterns. It aids developers to identify |
| 26 |
reasons for unexpected latency with real-time applications. It is a container of |
| 27 |
multiple sub-monitors described in the following sections. |
| 28 |
|
| 29 |
Monitor pagefault |
| 30 |
+++++++++++++++++ |
| 31 |
|
| 32 |
The `pagefault` monitor reports real-time tasks raising page faults. Its |
| 33 |
specification is:: |
| 34 |
|
| 35 |
RULE = always (RT imply not PAGEFAULT) |
| 36 |
|
| 37 |
To fix warnings reported by this monitor, `mlockall()` or `mlock()` can be used |
| 38 |
to ensure physical backing for memory. |
| 39 |
|
| 40 |
This monitor may have false negatives because the pages used by the real-time |
| 41 |
threads may just happen to be directly available during testing. To minimize |
| 42 |
this, the system can be put under memory pressure (e.g. invoking the OOM killer |
| 43 |
using a program that does `ptr = malloc(SIZE_OF_RAM); memset(ptr, 0, |
| 44 |
SIZE_OF_RAM);`) so that the kernel executes aggressive strategies to recycle as |
| 45 |
much physical memory as possible. |
| 46 |
|
| 47 |
Monitor sleep |
| 48 |
+++++++++++++ |
| 49 |
|
| 50 |
The `sleep` monitor reports real-time threads sleeping in a manner that may |
| 51 |
cause undesirable latency. Real-time applications should only put a real-time |
| 52 |
thread to sleep for one of the following reasons: |
| 53 |
|
| 54 |
- Cyclic work: real-time thread sleeps waiting for the next cycle. For this |
| 55 |
case, only the `clock_nanosleep` syscall should be used with `TIMER_ABSTIME` |
| 56 |
(to avoid time drift) and `CLOCK_MONOTONIC` (to avoid the clock being |
| 57 |
changed). No other method is safe for real-time. For example, threads |
| 58 |
waiting for timerfd can be woken by softirq which provides no real-time |
| 59 |
guarantee. |
| 60 |
- Real-time thread waiting for something to happen (e.g. another thread |
| 61 |
releasing shared resources, or a completion signal from another thread). In |
| 62 |
this case, only futexes (FUTEX_LOCK_PI, FUTEX_LOCK_PI2 or one of |
| 63 |
FUTEX_WAIT_*) should be used. Applications usually do not use futexes |
| 64 |
directly, but use PI mutexes and PI condition variables which are built on |
| 65 |
top of futexes. Be aware that the C library might not implement conditional |
| 66 |
variables as safe for real-time. As an alternative, the librtpi library |
| 67 |
exists to provide a conditional variable implementation that is correct for |
| 68 |
real-time applications in Linux. |
| 69 |
|
| 70 |
Beside the reason for sleeping, the eventual waker should also be |
| 71 |
real-time-safe. Namely, one of: |
| 72 |
|
| 73 |
- An equal-or-higher-priority thread |
| 74 |
- Hard interrupt handler |
| 75 |
- Non-maskable interrupt handler |
| 76 |
|
| 77 |
This monitor's warning usually means one of the following: |
| 78 |
|
| 79 |
- Real-time thread is blocked by a non-real-time thread (e.g. due to |
| 80 |
contention on a mutex without priority inheritance). This is priority |
| 81 |
inversion. |
| 82 |
- Time-critical work waits for something which is not safe for real-time (e.g. |
| 83 |
timerfd). |
| 84 |
- The work executed by the real-time thread does not need to run at real-time |
| 85 |
priority at all. This is not a problem for the real-time thread itself, but |
| 86 |
it is potentially taking the CPU away from other important real-time work. |
| 87 |
|
| 88 |
Application developers may purposely choose to have their real-time application |
| 89 |
sleep in a way that is not safe for real-time. It is debatable whether that is a |
| 90 |
problem. Application developers must analyze the warnings to make a proper |
| 91 |
assessment. |
| 92 |
|
| 93 |
The monitor's specification is:: |
| 94 |
|
| 95 |
RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST)) |
| 96 |
|
| 97 |
RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD) |
| 98 |
and ((not WAKE) until RT_FRIENDLY_WAKE) |
| 99 |
|
| 100 |
RT_VALID_SLEEP_REASON = FUTEX_WAIT |
| 101 |
or RT_FRIENDLY_NANOSLEEP |
| 102 |
|
| 103 |
RT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP |
| 104 |
and NANOSLEEP_TIMER_ABSTIME |
| 105 |
and NANOSLEEP_CLOCK_MONOTONIC |
| 106 |
|
| 107 |
RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO |
| 108 |
or WOKEN_BY_HARDIRQ |
| 109 |
or WOKEN_BY_NMI |
| 110 |
or KTHREAD_SHOULD_STOP |
| 111 |
|
| 112 |
ALLOWLIST = BLOCK_ON_RT_MUTEX |
| 113 |
or FUTEX_LOCK_PI |
| 114 |
or TASK_IS_RCU |
| 115 |
or TASK_IS_MIGRATION |
| 116 |
|
| 117 |
Beside the scenarios described above, this specification also handle some |
| 118 |
special cases: |
| 119 |
|
| 120 |
- `KERNEL_THREAD`: kernel tasks do not have any pattern that can be recognized |
| 121 |
as valid real-time sleeping reasons. Therefore sleeping reason is not |
| 122 |
checked for kernel tasks. |
| 123 |
- `KTHREAD_SHOULD_STOP`: a non-real-time thread may stop a real-time kernel |
| 124 |
thread by waking it and waiting for it to exit (`kthread_stop()`). This |
| 125 |
wakeup is safe for real-time. |
| 126 |
- `ALLOWLIST`: to handle known false positives with the kernel. |
| 127 |
- `BLOCK_ON_RT_MUTEX` is included in the allowlist due to its implementation. |
| 128 |
In the release path of rt_mutex, a boosted task is de-boosted before waking |
| 129 |
the rt_mutex's waiter. Consequently, the monitor may see a real-time-unsafe |
| 130 |
wakeup (e.g. non-real-time task waking real-time task). This is actually |
| 131 |
real-time-safe because preemption is disabled for the duration. |
| 132 |
- `FUTEX_LOCK_PI` is included in the allowlist for the same reason as |
| 133 |
`BLOCK_ON_RT_MUTEX`. |