Debian Codegraph @

Documentation/tools/rtla/rtla-timerlat-top.rst

raw
1 .. |tool| replace:: timerlat top
2
3 ====================
4 rtla-timerlat-top
5 ====================
6 -------------------------------------------
7 Measures the operating system timer latency
8 -------------------------------------------
9
10 :Manual section: 1
11
12 SYNOPSIS
13 ========
14 **rtla timerlat top** [*OPTIONS*] ...
15
16 DESCRIPTION
17 ===========
18
19 .. include:: common_timerlat_description.txt
20
21 The **rtla timerlat top** displays a summary of the periodic output
22 from the *timerlat* tracer. It also provides information for each
23 operating system noise via the **osnoise:** tracepoints that can be
24 seen with the option **-T**.
25
26 OPTIONS
27 =======
28
29 .. include:: common_timerlat_options.txt
30
31 .. include:: common_top_options.txt
32
33 .. include:: common_options.txt
34
35 .. include:: common_timerlat_aa.txt
36
37 **--aa-only** *us*
38
39 Set stop tracing conditions and run without collecting and displaying statistics.
40 Print the auto-analysis if the system hits the stop tracing condition. This option
41 is useful to reduce rtla timerlat CPU, enabling the debug without the overhead of
42 collecting the statistics.
43
44 EXAMPLE
45 =======
46
47 In the example below, the timerlat tracer is dispatched in cpus *1-23* in the
48 automatic trace mode, instructing the tracer to stop if a *40 us* latency or
49 higher is found::
50
51 # timerlat -a 40 -c 1-23 -q
52 Timer Latency
53 0 00:00:12 | IRQ Timer Latency (us) | Thread Timer Latency (us)
54 CPU COUNT | cur min avg max | cur min avg max
55 1 #12322 | 0 0 1 15 | 10 3 9 31
56 2 #12322 | 3 0 1 12 | 10 3 9 23
57 3 #12322 | 1 0 1 21 | 8 2 8 34
58 4 #12322 | 1 0 1 17 | 10 2 11 33
59 5 #12322 | 0 0 1 12 | 8 3 8 25
60 6 #12322 | 1 0 1 14 | 16 3 11 35
61 7 #12322 | 0 0 1 14 | 9 2 8 29
62 8 #12322 | 1 0 1 22 | 9 3 9 34
63 9 #12322 | 0 0 1 14 | 8 2 8 24
64 10 #12322 | 1 0 0 12 | 9 3 8 24
65 11 #12322 | 0 0 0 15 | 6 2 7 29
66 12 #12321 | 1 0 0 13 | 5 3 8 23
67 13 #12319 | 0 0 1 14 | 9 3 9 26
68 14 #12321 | 1 0 0 13 | 6 2 8 24
69 15 #12321 | 1 0 1 15 | 12 3 11 27
70 16 #12318 | 0 0 1 13 | 7 3 10 24
71 17 #12319 | 0 0 1 13 | 11 3 9 25
72 18 #12318 | 0 0 0 12 | 8 2 8 20
73 19 #12319 | 0 0 1 18 | 10 2 9 28
74 20 #12317 | 0 0 0 20 | 9 3 8 34
75 21 #12318 | 0 0 0 13 | 8 3 8 28
76 22 #12319 | 0 0 1 11 | 8 3 10 22
77 23 #12320 | 28 0 1 28 | 41 3 11 41
78 rtla timerlat hit stop tracing
79 ## CPU 23 hit stop tracing, analyzing it ##
80 IRQ handler delay: 27.49 us (65.52 %)
81 IRQ latency: 28.13 us
82 Timerlat IRQ duration: 9.59 us (22.85 %)
83 Blocking thread: 3.79 us (9.03 %)
84 objtool:49256 3.79 us
85 Blocking thread stacktrace
86 -> timerlat_irq
87 -> __hrtimer_run_queues
88 -> hrtimer_interrupt
89 -> __sysvec_apic_timer_interrupt
90 -> sysvec_apic_timer_interrupt
91 -> asm_sysvec_apic_timer_interrupt
92 -> _raw_spin_unlock_irqrestore
93 -> cgroup_rstat_flush_locked
94 -> cgroup_rstat_flush_irqsafe
95 -> mem_cgroup_flush_stats
96 -> mem_cgroup_wb_stats
97 -> balance_dirty_pages
98 -> balance_dirty_pages_ratelimited_flags
99 -> btrfs_buffered_write
100 -> btrfs_do_write_iter
101 -> vfs_write
102 -> __x64_sys_pwrite64
103 -> do_syscall_64
104 -> entry_SYSCALL_64_after_hwframe
105 ------------------------------------------------------------------------
106 Thread latency: 41.96 us (100%)
107
108 The system has exit from idle latency!
109 Max timerlat IRQ latency from idle: 17.48 us in cpu 4
110 Saving trace to timerlat_trace.txt
111
112 In this case, the major factor was the delay suffered by the *IRQ handler*
113 that handles **timerlat** wakeup: *65.52%*. This can be caused by the
114 current thread masking interrupts, which can be seen in the blocking
115 thread stacktrace: the current thread (*objtool:49256*) disabled interrupts
116 via *raw spin lock* operations inside mem cgroup, while doing write
117 syscall in a btrfs file system.
118
119 The raw trace is saved in the **timerlat_trace.txt** file for further analysis.
120
121 Note that **rtla timerlat** was dispatched without changing *timerlat* tracer
122 threads' priority. That is generally not needed because these threads have
123 priority *FIFO:95* by default, which is a common priority used by real-time
124 kernel developers to analyze scheduling delays.
125
126 SEE ALSO
127 --------
128 **rtla-timerlat**\(1), **rtla-timerlat-hist**\(1)
129
130 *timerlat* tracer documentation: <https://www.kernel.org/doc/html/latest/trace/timerlat-tracer.html>
131
132 AUTHOR
133 ------
134 Written by Daniel Bristot de Oliveira <[email protected]>
135
136 .. include:: common_appendix.txt