| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
================ |
| 4 |
CoreSight - Perf |
| 5 |
================ |
| 6 |
|
| 7 |
:Author: Carsten Haitzler <[email protected]> |
| 8 |
:Date: June 29th, 2022 |
| 9 |
|
| 10 |
Perf is able to locally access CoreSight trace data and store it to the |
| 11 |
output perf data files. This data can then be later decoded to give the |
| 12 |
instructions that were traced for debugging or profiling purposes. You |
| 13 |
can log such data with a perf record command like:: |
| 14 |
|
| 15 |
perf record -e cs_etm//u testbinary |
| 16 |
|
| 17 |
This would run some test binary (testbinary) until it exits and record |
| 18 |
a perf.data trace file. That file would have AUX sections if CoreSight |
| 19 |
is working correctly. You can dump the content of this file as |
| 20 |
readable text with a command like:: |
| 21 |
|
| 22 |
perf report --stdio --dump -i perf.data |
| 23 |
|
| 24 |
You should find some sections of this file have AUX data blocks like:: |
| 25 |
|
| 26 |
0x1e78 [0x30]: PERF_RECORD_AUXTRACE size: 0x11dd0 offset: 0 ref: 0x1b614fc1061b0ad1 idx: 0 tid: 531230 cpu: -1 |
| 27 |
|
| 28 |
. ... CoreSight ETM Trace data: size 73168 bytes |
| 29 |
Idx:0; ID:10; I_ASYNC : Alignment Synchronisation. |
| 30 |
Idx:12; ID:10; I_TRACE_INFO : Trace Info.; INFO=0x0 { CC.0 } |
| 31 |
Idx:17; ID:10; I_ADDR_L_64IS0 : Address, Long, 64 bit, IS0.; Addr=0x0000000000000000; |
| 32 |
Idx:26; ID:10; I_TRACE_ON : Trace On. |
| 33 |
Idx:27; ID:10; I_ADDR_CTXT_L_64IS0 : Address & Context, Long, 64 bit, IS0.; Addr=0x0000FFFFB6069140; Ctxt: AArch64,EL0, NS; |
| 34 |
Idx:38; ID:10; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE |
| 35 |
Idx:39; ID:10; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE |
| 36 |
Idx:40; ID:10; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEEEEEEEEEEEEEE |
| 37 |
Idx:41; ID:10; I_ATOM_F6 : Atom format 6.; EEEEEEEEEEEN |
| 38 |
... |
| 39 |
|
| 40 |
If you see these above, then your system is tracing CoreSight data |
| 41 |
correctly. |
| 42 |
|
| 43 |
To compile perf with CoreSight support in the tools/perf directory do:: |
| 44 |
|
| 45 |
make CORESIGHT=1 |
| 46 |
|
| 47 |
This requires OpenCSD to build. You may install distribution packages |
| 48 |
for the support such as libopencsd and libopencsd-dev or download it |
| 49 |
and build yourself. Upstream OpenCSD is located at: |
| 50 |
|
| 51 |
https://github.com/Linaro/OpenCSD |
| 52 |
|
| 53 |
For complete information on building perf with CoreSight support and |
| 54 |
more extensive usage look at: |
| 55 |
|
| 56 |
https://github.com/Linaro/OpenCSD/blob/master/HOWTO.md |
| 57 |
|
| 58 |
|
| 59 |
Kernel CoreSight Support |
| 60 |
------------------------ |
| 61 |
|
| 62 |
You will also want CoreSight support enabled in your kernel config. |
| 63 |
Ensure it is enabled with:: |
| 64 |
|
| 65 |
CONFIG_CORESIGHT=y |
| 66 |
|
| 67 |
There are various other CoreSight options you probably also want |
| 68 |
enabled like:: |
| 69 |
|
| 70 |
CONFIG_CORESIGHT_LINKS_AND_SINKS=y |
| 71 |
CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y |
| 72 |
CONFIG_CORESIGHT_CATU=y |
| 73 |
CONFIG_CORESIGHT_SINK_TPIU=y |
| 74 |
CONFIG_CORESIGHT_SINK_ETBV10=y |
| 75 |
CONFIG_CORESIGHT_SOURCE_ETM4X=y |
| 76 |
CONFIG_CORESIGHT_CTI=y |
| 77 |
CONFIG_CORESIGHT_CTI_INTEGRATION_REGS=y |
| 78 |
|
| 79 |
Please refer to the kernel configuration help for more information. |
| 80 |
|
| 81 |
Fine-grained tracing with AUX pause and resume |
| 82 |
---------------------------------------------- |
| 83 |
|
| 84 |
Arm CoreSight may generate a large amount of hardware trace data, which |
| 85 |
will lead to overhead in recording and distract users when reviewing |
| 86 |
profiling result. To mitigate the issue of excessive trace data, Perf |
| 87 |
provides AUX pause and resume functionality for fine-grained tracing. |
| 88 |
|
| 89 |
The AUX pause and resume can be triggered by associated events. These |
| 90 |
events can be ftrace tracepoints (including static and dynamic |
| 91 |
tracepoints) or PMU events (e.g. CPU PMU cycle event). To create a perf |
| 92 |
session with AUX pause / resume, three configuration terms are |
| 93 |
introduced: |
| 94 |
|
| 95 |
- "aux-action=start-paused": it is specified for the cs_etm PMU event to |
| 96 |
launch in a paused state. |
| 97 |
- "aux-action=pause": an associated event is specified with this term |
| 98 |
to pause AUX trace. |
| 99 |
- "aux-action=resume": an associated event is specified with this term |
| 100 |
to resume AUX trace. |
| 101 |
|
| 102 |
Example for triggering AUX pause and resume with ftrace tracepoints:: |
| 103 |
|
| 104 |
perf record -e cs_etm/aux-action=start-paused/k,syscalls:sys_enter_openat/aux-action=resume/,syscalls:sys_exit_openat/aux-action=pause/ ls |
| 105 |
|
| 106 |
Example for triggering AUX pause and resume with PMU event:: |
| 107 |
|
| 108 |
perf record -a -e cs_etm/aux-action=start-paused/k \ |
| 109 |
-e cycles/aux-action=pause,period=10000000/ \ |
| 110 |
-e cycles/aux-action=resume,period=1050000/ -- sleep 1 |
| 111 |
|
| 112 |
Perf test - Verify kernel and userspace perf CoreSight work |
| 113 |
----------------------------------------------------------- |
| 114 |
|
| 115 |
When you run perf test, it will do a lot of self tests. Some of those |
| 116 |
tests will cover CoreSight (only if enabled and on ARM64). You |
| 117 |
generally would run perf test from the tools/perf directory in the |
| 118 |
kernel tree. Some tests will check some internal perf support like: |
| 119 |
|
| 120 |
Check Arm CoreSight trace data recording and synthesized samples |
| 121 |
Check Arm SPE trace data recording and synthesized samples |
| 122 |
|
| 123 |
Some others will actually use perf record and some test binaries that |
| 124 |
are in tests/shell/coresight and will collect traces to ensure a |
| 125 |
minimum level of functionality is met. The scripts that launch these |
| 126 |
tests are in the same directory. These will all look like: |
| 127 |
|
| 128 |
CoreSight / ASM Pure Loop |
| 129 |
CoreSight / Memcpy 16k 10 Threads |
| 130 |
CoreSight / Thread Loop 10 Threads - Check TID |
| 131 |
etc. |
| 132 |
|
| 133 |
These perf record tests will not run if the tool binaries do not exist |
| 134 |
in tests/shell/coresight/\*/ and will be skipped. If you do not have |
| 135 |
CoreSight support in hardware then either do not build perf with |
| 136 |
CoreSight support or remove these binaries in order to not have these |
| 137 |
tests fail and have them skip instead. |
| 138 |
|
| 139 |
These tests will log historical results in the current working |
| 140 |
directory (e.g. tools/perf) and will be named stats-\*.csv like: |
| 141 |
|
| 142 |
stats-asm_pure_loop-out.csv |
| 143 |
stats-memcpy_thread-16k_10.csv |
| 144 |
... |
| 145 |
|
| 146 |
These statistic files log some aspects of the AUX data sections in |
| 147 |
the perf data output counting some numbers of certain encodings (a |
| 148 |
good way to know that it's working in a very simple way). One problem |
| 149 |
with CoreSight is that given a large enough amount of data needing to |
| 150 |
be logged, some of it can be lost due to the processor not waking up |
| 151 |
in time to read out all the data from buffers etc.. You will notice |
| 152 |
that the amount of data collected can vary a lot per run of perf test. |
| 153 |
If you wish to see how this changes over time, simply run perf test |
| 154 |
multiple times and all these csv files will have more and more data |
| 155 |
appended to it that you can later examine, graph and otherwise use to |
| 156 |
figure out if things have become worse or better. |
| 157 |
|
| 158 |
This means sometimes these tests fail as they don't capture all the |
| 159 |
data needed. This is about tracking quality and amount of data |
| 160 |
produced over time and to see when changes to the Linux kernel improve |
| 161 |
quality of traces. |
| 162 |
|
| 163 |
Be aware that some of these tests take quite a while to run, specifically |
| 164 |
in processing the perf data file and dumping contents to then examine what |
| 165 |
is inside. |
| 166 |
|
| 167 |
You can change where these csv logs are stored by setting the |
| 168 |
PERF_TEST_CORESIGHT_STATDIR environment variable before running perf |
| 169 |
test like:: |
| 170 |
|
| 171 |
export PERF_TEST_CORESIGHT_STATDIR=/var/tmp |
| 172 |
perf test |
| 173 |
|
| 174 |
They will also store resulting perf output data in the current |
| 175 |
directory for later inspection like:: |
| 176 |
|
| 177 |
perf-asm_pure_loop-out.data |
| 178 |
perf-memcpy_thread-16k_10.data |
| 179 |
... |
| 180 |
|
| 181 |
You can alter where the perf data files are stored by setting the |
| 182 |
PERF_TEST_CORESIGHT_DATADIR environment variable such as:: |
| 183 |
|
| 184 |
PERF_TEST_CORESIGHT_DATADIR=/var/tmp |
| 185 |
perf test |
| 186 |
|
| 187 |
You may wish to set these above environment variables if you wish to |
| 188 |
keep the output of tests outside of the current working directory for |
| 189 |
longer term storage and examination. |