Debian Codegraph @

Documentation/trace/coresight/panic.rst

raw
1 ===================================================
2 Using Coresight for Kernel panic and Watchdog reset
3 ===================================================
4
5 Introduction
6 ------------
7 This documentation is about using Linux coresight trace support to
8 debug kernel panic and watchdog reset scenarios.
9
10 Coresight trace during Kernel panic
11 -----------------------------------
12 From the coresight driver point of view, addressing the kernel panic
13 situation has four main requirements.
14
15 a. Support for allocation of trace buffer pages from reserved memory area.
16 Platform can advertise this using a new device tree property added to
17 relevant coresight nodes.
18
19 b. Support for stopping coresight blocks at the time of panic
20
21 c. Saving required metadata in the specified format
22
23 d. Support for reading trace data captured at the time of panic
24
25 Allocation of trace buffer pages from reserved RAM
26 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
27 A new optional device tree property "memory-region" is added to the
28 Coresight TMC device nodes, that would give the base address and size of trace
29 buffer.
30
31 Static allocation of trace buffers would ensure that both IOMMU enabled
32 and disabled cases are handled. Also, platforms that support persistent
33 RAM will allow users to read trace data in the subsequent boot without
34 booting the crashdump kernel.
35
36 Note:
37 For ETR sink devices, this reserved region will be used for both trace
38 capture and trace data retrieval.
39 For ETF sink devices, internal SRAM would be used for trace capture,
40 and they would be synced to reserved region for retrieval.
41
42
43 Disabling coresight blocks at the time of panic
44 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
45 In order to avoid the situation of losing relevant trace data after a
46 kernel panic, it would be desirable to stop the coresight blocks at the
47 time of panic.
48
49 This can be achieved by configuring the comparator, CTI and sink
50 devices as below::
51
52 Trigger on panic
53 Comparator --->External out --->CTI -->External In---->ETR/ETF stop
54
55 Saving metadata at the time of kernel panic
56 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
57 Coresight metadata involves all additional data that are required for a
58 successful trace decode in addition to the trace data. This involves
59 ETR/ETF/ETB register snapshot etc.
60
61 A new optional device property "memory-region" is added to
62 the ETR/ETF/ETB device nodes for this.
63
64 Reading trace data captured at the time of panic
65 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
66 Trace data captured at the time of panic, can be read from rebooted kernel
67 or from crashdump kernel using a special device file /dev/crash_tmc_xxx.
68 This device file is created only when there is a valid crashdata available.
69
70 General flow of trace capture and decode in case of kernel panic
71 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
72 1. Enable source and sink on all the cores using the sysfs interface.
73 ETR sinks should have trace buffers allocated from reserved memory,
74 by selecting "resrv" buffer mode from sysfs.
75
76 2. Run relevant tests.
77
78 3. On a kernel panic, all coresight blocks are disabled, necessary
79 metadata is synced by kernel panic handler.
80
81 System would eventually reboot or boot a crashdump kernel.
82
83 4. For platforms that supports crashdump kernel, raw trace data can be
84 dumped using the coresight sysfs interface from the crashdump kernel
85 itself. Persistent RAM is not a requirement in this case.
86
87 5. For platforms that supports persistent RAM, trace data can be dumped
88 using the coresight sysfs interface in the subsequent Linux boot.
89 Crashdump kernel is not a requirement in this case. Persistent RAM
90 ensures that trace data is intact across reboot.
91
92 Coresight trace during Watchdog reset
93 -------------------------------------
94 The main difference between addressing the watchdog reset and kernel panic
95 case are below,
96
97 a. Saving coresight metadata need to be taken care by the
98 SCP(system control processor) firmware in the specified format,
99 instead of kernel.
100
101 b. Reserved memory region given by firmware for trace buffer and metadata
102 has to be in persistent RAM.
103 Note: This is a requirement for watchdog reset case but optional
104 in kernel panic case.
105
106 Watchdog reset can be supported only on platforms that meet the above
107 two requirements.
108
109 Sample commands for testing a Kernel panic case with ETR sink
110 -------------------------------------------------------------
111
112 1. Boot Linux kernel with "crash_kexec_post_notifiers" added to the kernel
113 bootargs. This is mandatory if the user would like to read the tracedata
114 from the crashdump kernel.
115
116 2. Enable the preloaded ETM configuration::
117
118 #echo 1 > /sys/kernel/config/cs-syscfg/configurations/panicstop/enable
119
120 3. Configure CTI using sysfs interface::
121
122 #./cti_setup.sh
123
124 #cat cti_setup.sh
125
126
127 cd /sys/bus/coresight/devices/
128
129 ap_cti_config () {
130 #ETM trig out[0] trigger to Channel 0
131 echo 0 4 > channels/trigin_attach
132 }
133
134 etf_cti_config () {
135 #ETF Flush in trigger from Channel 0
136 echo 0 1 > channels/trigout_attach
137 echo 1 > channels/trig_filter_enable
138 }
139
140 etr_cti_config () {
141 #ETR Flush in from Channel 0
142 echo 0 1 > channels/trigout_attach
143 echo 1 > channels/trig_filter_enable
144 }
145
146 ctidevs=`find . -name "cti*"`
147
148 for i in $ctidevs
149 do
150 cd $i
151
152 connection=`find . -name "ete*"`
153 if [ ! -z "$connection" ]
154 then
155 echo "AP CTI config for $i"
156 ap_cti_config
157 fi
158
159 connection=`find . -name "tmc_etf*"`
160 if [ ! -z "$connection" ]
161 then
162 echo "ETF CTI config for $i"
163 etf_cti_config
164 fi
165
166 connection=`find . -name "tmc_etr*"`
167 if [ ! -z "$connection" ]
168 then
169 echo "ETR CTI config for $i"
170 etr_cti_config
171 fi
172
173 cd ..
174 done
175
176 Note: CTI connections are SOC specific and hence the above script is
177 added just for reference.
178
179 4. Choose reserved buffer mode for ETR buffer::
180
181 #echo "resrv" > /sys/bus/coresight/devices/tmc_etr0/buf_mode_preferred
182
183 5. Enable stop on flush trigger configuration::
184
185 #echo 1 > /sys/bus/coresight/devices/tmc_etr0/stop_on_flush
186
187 6. Start Coresight tracing on cores 1 and 2 using sysfs interface
188
189 7. Run some application on core 1::
190
191 #taskset -c 1 dd if=/dev/urandom of=/dev/null &
192
193 8. Invoke kernel panic on core 2::
194
195 #echo 1 > /proc/sys/kernel/panic
196 #taskset -c 2 echo c > /proc/sysrq-trigger
197
198 9. From rebooted kernel or crashdump kernel, read crashdata::
199
200 #dd if=/dev/crash_tmc_etr0 of=/trace/cstrace.bin
201
202 10. Run opencsd decoder tools/scripts to generate the instruction trace.
203
204 Sample instruction trace dump
205 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
206
207 Core1 dump::
208
209 A etm4_enable_hw: ffff800008ae1dd4
210 CONTEXT EL2 etm4_enable_hw: ffff800008ae1dd4
211 I etm4_enable_hw: ffff800008ae1dd4:
212 d503201f nop
213 I etm4_enable_hw: ffff800008ae1dd8:
214 d503201f nop
215 I etm4_enable_hw: ffff800008ae1ddc:
216 d503201f nop
217 I etm4_enable_hw: ffff800008ae1de0:
218 d503201f nop
219 I etm4_enable_hw: ffff800008ae1de4:
220 d503201f nop
221 I etm4_enable_hw: ffff800008ae1de8:
222 d503233f paciasp
223 I etm4_enable_hw: ffff800008ae1dec:
224 a9be7bfd stp x29, x30, [sp, #-32]!
225 I etm4_enable_hw: ffff800008ae1df0:
226 910003fd mov x29, sp
227 I etm4_enable_hw: ffff800008ae1df4:
228 a90153f3 stp x19, x20, [sp, #16]
229 I etm4_enable_hw: ffff800008ae1df8:
230 2a0003f4 mov w20, w0
231 I etm4_enable_hw: ffff800008ae1dfc:
232 900085b3 adrp x19, ffff800009b95000 <reserved_mem+0xc48>
233 I etm4_enable_hw: ffff800008ae1e00:
234 910f4273 add x19, x19, #0x3d0
235 I etm4_enable_hw: ffff800008ae1e04:
236 f8747a60 ldr x0, [x19, x20, lsl #3]
237 E etm4_enable_hw: ffff800008ae1e08:
238 b4000140 cbz x0, ffff800008ae1e30 <etm4_starting_cpu+0x50>
239 I 149.039572921 etm4_enable_hw: ffff800008ae1e30:
240 a94153f3 ldp x19, x20, [sp, #16]
241 I 149.039572921 etm4_enable_hw: ffff800008ae1e34:
242 52800000 mov w0, #0x0 // #0
243 I 149.039572921 etm4_enable_hw: ffff800008ae1e38:
244 a8c27bfd ldp x29, x30, [sp], #32
245
246 ..snip
247
248 149.052324811 chacha_block_generic: ffff800008642d80:
249 9100a3e0 add x0,
250 I 149.052324811 chacha_block_generic: ffff800008642d84:
251 b86178a2 ldr w2, [x5, x1, lsl #2]
252 I 149.052324811 chacha_block_generic: ffff800008642d88:
253 8b010803 add x3, x0, x1, lsl #2
254 I 149.052324811 chacha_block_generic: ffff800008642d8c:
255 b85fc063 ldur w3, [x3, #-4]
256 I 149.052324811 chacha_block_generic: ffff800008642d90:
257 0b030042 add w2, w2, w3
258 I 149.052324811 chacha_block_generic: ffff800008642d94:
259 b8217882 str w2, [x4, x1, lsl #2]
260 I 149.052324811 chacha_block_generic: ffff800008642d98:
261 91000421 add x1, x1, #0x1
262 I 149.052324811 chacha_block_generic: ffff800008642d9c:
263 f100443f cmp x1, #0x11
264
265
266 Core 2 dump::
267
268 A etm4_enable_hw: ffff800008ae1dd4
269 CONTEXT EL2 etm4_enable_hw: ffff800008ae1dd4
270 I etm4_enable_hw: ffff800008ae1dd4:
271 d503201f nop
272 I etm4_enable_hw: ffff800008ae1dd8:
273 d503201f nop
274 I etm4_enable_hw: ffff800008ae1ddc:
275 d503201f nop
276 I etm4_enable_hw: ffff800008ae1de0:
277 d503201f nop
278 I etm4_enable_hw: ffff800008ae1de4:
279 d503201f nop
280 I etm4_enable_hw: ffff800008ae1de8:
281 d503233f paciasp
282 I etm4_enable_hw: ffff800008ae1dec:
283 a9be7bfd stp x29, x30, [sp, #-32]!
284 I etm4_enable_hw: ffff800008ae1df0:
285 910003fd mov x29, sp
286 I etm4_enable_hw: ffff800008ae1df4:
287 a90153f3 stp x19, x20, [sp, #16]
288 I etm4_enable_hw: ffff800008ae1df8:
289 2a0003f4 mov w20, w0
290 I etm4_enable_hw: ffff800008ae1dfc:
291 900085b3 adrp x19, ffff800009b95000 <reserved_mem+0xc48>
292 I etm4_enable_hw: ffff800008ae1e00:
293 910f4273 add x19, x19, #0x3d0
294 I etm4_enable_hw: ffff800008ae1e04:
295 f8747a60 ldr x0, [x19, x20, lsl #3]
296 E etm4_enable_hw: ffff800008ae1e08:
297 b4000140 cbz x0, ffff800008ae1e30 <etm4_starting_cpu+0x50>
298 I 149.046243445 etm4_enable_hw: ffff800008ae1e30:
299 a94153f3 ldp x19, x20, [sp, #16]
300 I 149.046243445 etm4_enable_hw: ffff800008ae1e34:
301 52800000 mov w0, #0x0 // #0
302 I 149.046243445 etm4_enable_hw: ffff800008ae1e38:
303 a8c27bfd ldp x29, x30, [sp], #32
304 I 149.046243445 etm4_enable_hw: ffff800008ae1e3c:
305 d50323bf autiasp
306 E 149.046243445 etm4_enable_hw: ffff800008ae1e40:
307 d65f03c0 ret
308 A ete_sysreg_write: ffff800008adfa18
309
310 ..snip
311
312 I 149.05422547 panic: ffff800008096300:
313 a90363f7 stp x23, x24, [sp, #48]
314 I 149.05422547 panic: ffff800008096304:
315 6b00003f cmp w1, w0
316 I 149.05422547 panic: ffff800008096308:
317 3a411804 ccmn w0, #0x1, #0x4, ne // ne = any
318 N 149.05422547 panic: ffff80000809630c:
319 540001e0 b.eq ffff800008096348 <panic+0xe0> // b.none
320 I 149.05422547 panic: ffff800008096310:
321 f90023f9 str x25, [sp, #64]
322 E 149.05422547 panic: ffff800008096314:
323 97fe44ef bl ffff8000080276d0 <panic_smp_self_stop>
324 A panic: ffff80000809634c
325 I 149.05422547 panic: ffff80000809634c:
326 910102d5 add x21, x22, #0x40
327 I 149.05422547 panic: ffff800008096350:
328 52800020 mov w0, #0x1 // #1
329 E 149.05422547 panic: ffff800008096354:
330 94166b8b bl ffff800008631180 <bust_spinlocks>
331 N 149.054225518 bust_spinlocks: ffff800008631180:
332 340000c0 cbz w0, ffff800008631198 <bust_spinlocks+0x18>
333 I 149.054225518 bust_spinlocks: ffff800008631184:
334 f000a321 adrp x1, ffff800009a98000 <pbufs.0+0xbb8>
335 I 149.054225518 bust_spinlocks: ffff800008631188:
336 b9405c20 ldr w0, [x1, #92]
337 I 149.054225518 bust_spinlocks: ffff80000863118c:
338 11000400 add w0, w0, #0x1
339 I 149.054225518 bust_spinlocks: ffff800008631190:
340 b9005c20 str w0, [x1, #92]
341 E 149.054225518 bust_spinlocks: ffff800008631194:
342 d65f03c0 ret
343 A panic: ffff800008096358
344
345 Perf based testing
346 ------------------
347
348 Starting perf session
349 ~~~~~~~~~~~~~~~~~~~~~
350 ETF::
351
352 perf record -e cs_etm/panicstop,@tmc_etf1/ -C 1
353 perf record -e cs_etm/panicstop,@tmc_etf2/ -C 2
354
355 ETR::
356
357 perf record -e cs_etm/panicstop,@tmc_etr0/ -C 1,2
358
359 Reading trace data after panic
360 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
361 Same sysfs based method explained above can be used to retrieve and
362 decode the trace data after the reboot on kernel panic.