| 1 |
<?xml version='1.0'?> |
| 2 |
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis- |
| 3 |
open.org/docbook/xml/4.5/docbookx.dtd" [ |
| 4 |
]> |
| 5 |
|
| 6 |
<chapter id="userspace-probing"> |
| 7 |
<title>User-space Probing</title> |
| 8 |
|
| 9 |
<para> |
| 10 |
SystemTap initially focused on kernel-space probing. Because |
| 11 |
there are many instances where user-space probing can help diagnose |
| 12 |
a problem, SystemTap 0.6 added support to allow probing user-space processes. |
| 13 |
SystemTap can probe the entry into and return from a |
| 14 |
function in user-space processes, probe predefined markers in |
| 15 |
user-space code, and monitor user-process events. |
| 16 |
</para> |
| 17 |
|
| 18 |
<para> |
| 19 |
SystemTap requires the uprobes module to perform user-space probing. If your |
| 20 |
Linux kernel is version 3.5 or higher, it already includes uprobes. |
| 21 |
To verify that the current kernel supports uprobes natively, run the following command: |
| 22 |
</para> |
| 23 |
|
| 24 |
<screen><command>grep CONFIG_UPROBES /boot/config-`uname -r`</command></screen> |
| 25 |
|
| 26 |
<para> |
| 27 |
If uprobes is integrated, the output of this command is as follows: |
| 28 |
</para> |
| 29 |
|
| 30 |
<screen>CONFIG_UPROBES=y</screen> |
| 31 |
|
| 32 |
<!-- FIXME How does uprobes work for cross compilation and |
| 33 |
using a compile server? --> |
| 34 |
|
| 35 |
<section id="uevents"> |
| 36 |
<title>User-Space Events</title> |
| 37 |
|
| 38 |
<indexterm> |
| 39 |
<primary>Events</primary> |
| 40 |
<secondary>user-space</secondary> |
| 41 |
</indexterm> |
| 42 |
|
| 43 |
<para> |
| 44 |
All user-space event probes begin with <emphasis>process</emphasis>. |
| 45 |
You can limit the process events to a specific running process by specifying the process ID. |
| 46 |
You can also limit the process events to monitor a particular executable |
| 47 |
by specifying the path to the executable (PATH). |
| 48 |
SystemTap makes use of the <envar>PATH</envar> environment variable, which allows |
| 49 |
you to use both the name used on the command-line to start the executable and |
| 50 |
the absolute path to the executable. |
| 51 |
</para> |
| 52 |
<para> |
| 53 |
Several of the user-space probe events limit their scope to a |
| 54 |
particular executable name (PATH), because |
| 55 |
SystemTap must use debug information to statically analyze where to |
| 56 |
place the probes. But for many user-space probe events, the process ID |
| 57 |
and executable name are optional. |
| 58 |
Any <command>process</command> event in the list below that include process ID or |
| 59 |
the path to the executable must include those arguments. |
| 60 |
The process ID and path to the executable are optional for the |
| 61 |
<command>process</command> events that do not list them: |
| 62 |
</para> |
| 63 |
|
| 64 |
<variablelist> |
| 65 |
|
| 66 |
<varlistentry> |
| 67 |
<term>process("<replaceable>PATH</replaceable>").function("<replaceable>function</replaceable>")</term> |
| 68 |
<listitem> |
| 69 |
<para> |
| 70 |
The entry to the user-space function <replaceable>function</replaceable> for the executable <replaceable>PATH</replaceable>. |
| 71 |
This event is the user-space analogue of the |
| 72 |
<command>kernel.function("<replaceable>function</replaceable>")</command> event. |
| 73 |
It allows wildcards for the function <replaceable>function</replaceable> |
| 74 |
and <literal>.return</literal> suffix. |
| 75 |
</para> |
| 76 |
</listitem> |
| 77 |
</varlistentry> |
| 78 |
|
| 79 |
<varlistentry> |
| 80 |
<term>process("<replaceable>PATH</replaceable>").statement("<replaceable>statement</replaceable>")</term> |
| 81 |
<listitem> |
| 82 |
<para> |
| 83 |
The earliest instruction in the code for <replaceable>statement</replaceable>. This is the user-space analogue of |
| 84 |
<command>kernel.statement("<replaceable>statement</replaceable>")</command>. |
| 85 |
</para> |
| 86 |
</listitem> |
| 87 |
</varlistentry> |
| 88 |
|
| 89 |
<varlistentry> |
| 90 |
<term>process("<replaceable>PATH</replaceable>").mark("<replaceable>marker</replaceable>")</term> |
| 91 |
<listitem> |
| 92 |
<para>The static probe point <replaceable>marker</replaceable> defined in |
| 93 |
<replaceable>PATH</replaceable>. |
| 94 |
You can use wildcards for <replaceable>marker</replaceable> to specify |
| 95 |
multiple marks with a single probe. |
| 96 |
The static probe points may also have numbered arguments ($1, $2, and so on) |
| 97 |
available to the probe.</para> |
| 98 |
<para>A variety of user-space packages such as Java include these static |
| 99 |
probe points. |
| 100 |
Most packages that provide static probe points also |
| 101 |
provide aliases for the raw user-space mark events. |
| 102 |
Below is one such alias for the x86_64 Java hotspot JVM: |
| 103 |
</para> |
| 104 |
<programlisting>probe hotspot.gc_begin = |
| 105 |
process("/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so").mark("gc__begin")</programlisting> |
| 106 |
</listitem> |
| 107 |
</varlistentry> |
| 108 |
|
| 109 |
<varlistentry> |
| 110 |
<term>process.begin</term> |
| 111 |
<listitem> |
| 112 |
<para>A user-space process is created. |
| 113 |
You can limit this to a particular process ID or a full path to the |
| 114 |
executable.</para> |
| 115 |
</listitem> |
| 116 |
</varlistentry> |
| 117 |
|
| 118 |
<varlistentry> |
| 119 |
<term>process.thread.begin</term> |
| 120 |
<listitem> |
| 121 |
<para>A user-space thread is created. |
| 122 |
You can limit this to a particular process ID or a full path to the |
| 123 |
executable.</para> |
| 124 |
</listitem> |
| 125 |
</varlistentry> |
| 126 |
|
| 127 |
<varlistentry> |
| 128 |
<term>process.end</term> |
| 129 |
<listitem> |
| 130 |
<para>A user-space process dies. |
| 131 |
You can limit this to a particular process ID or a full path to the |
| 132 |
executable.</para> |
| 133 |
</listitem> |
| 134 |
</varlistentry> |
| 135 |
|
| 136 |
<varlistentry> |
| 137 |
<term>process.thread.end</term> |
| 138 |
<listitem> |
| 139 |
<para>A user-space thread is destroyed. |
| 140 |
You can limit this to a particular process ID or a full path to the |
| 141 |
executable.</para> |
| 142 |
</listitem> |
| 143 |
</varlistentry> |
| 144 |
|
| 145 |
<varlistentry> |
| 146 |
<term>process.syscall</term> |
| 147 |
<listitem> |
| 148 |
<para>A user-space process makes a system call. |
| 149 |
The system call number is available in the <varname>$syscall</varname> context variable, and |
| 150 |
the fist six arguments are available in <varname>$arg1</varname> through <varname>$arg6</varname>. |
| 151 |
The <literal>.return</literal> suffix places the probe at the return from the system call. |
| 152 |
For <command>syscall.return</command>, the return value is available through the |
| 153 |
<varname>$return</varname> context variable.</para> |
| 154 |
<para>You can limit this to a particular process ID or a full path to the |
| 155 |
executable.</para> |
| 156 |
</listitem> |
| 157 |
</varlistentry> |
| 158 |
|
| 159 |
</variablelist> |
| 160 |
|
| 161 |
|
| 162 |
</section> |
| 163 |
|
| 164 |
<section id="utargetvariable"> |
| 165 |
<title>Accessing User-Space Target Variables</title> |
| 166 |
|
| 167 |
<indexterm> |
| 168 |
<primary>target variables</primary> |
| 169 |
<secondary>user-space</secondary> |
| 170 |
</indexterm> |
| 171 |
|
| 172 |
<para> |
| 173 |
You can access user-space target variables in the same manner as described in |
| 174 |
<xref linkend="targetvariables"/>. |
| 175 |
In Linux, however, there are separate address spaces for the user and |
| 176 |
kernel code. |
| 177 |
When using the <literal>-></literal> operator, SystemTap accesses the appropriate address space. |
| 178 |
</para> |
| 179 |
<para> |
| 180 |
For pointers to base types such as integers and strings, |
| 181 |
there are a number of functions listed below to access user-space data. |
| 182 |
The first argument for each functions is the pointer to the data item. |
| 183 |
</para> |
| 184 |
|
| 185 |
<variablelist> |
| 186 |
|
| 187 |
<varlistentry> |
| 188 |
<term>user_char(<replaceable>address</replaceable>)</term> |
| 189 |
<listitem> |
| 190 |
<para> |
| 191 |
Obtains the character at <replaceable>address</replaceable> for the current |
| 192 |
user process. |
| 193 |
</para> |
| 194 |
</listitem> |
| 195 |
</varlistentry> |
| 196 |
|
| 197 |
<varlistentry> |
| 198 |
<term>user_short(<replaceable>address</replaceable>)</term> |
| 199 |
<listitem> |
| 200 |
<para> |
| 201 |
Obtains the short integer at <replaceable>address</replaceable> for the current |
| 202 |
user process. |
| 203 |
</para> |
| 204 |
</listitem> |
| 205 |
</varlistentry> |
| 206 |
|
| 207 |
<varlistentry> |
| 208 |
<term>user_int(<replaceable>address</replaceable>)</term> |
| 209 |
<listitem> |
| 210 |
<para> |
| 211 |
Obtains the integer at <replaceable>address</replaceable> for the current |
| 212 |
user process. |
| 213 |
</para> |
| 214 |
</listitem> |
| 215 |
</varlistentry> |
| 216 |
|
| 217 |
<varlistentry> |
| 218 |
<term>user_long(<replaceable>address</replaceable>)</term> |
| 219 |
<listitem> |
| 220 |
<para> |
| 221 |
Obtains the long integer at <replaceable>address</replaceable> for the current |
| 222 |
user process. |
| 223 |
</para> |
| 224 |
</listitem> |
| 225 |
</varlistentry> |
| 226 |
|
| 227 |
<varlistentry> |
| 228 |
<term>user_string(<replaceable>address</replaceable>)</term> |
| 229 |
<listitem> |
| 230 |
<para> |
| 231 |
Obtains the string at <replaceable>address</replaceable> for the current |
| 232 |
user process. |
| 233 |
</para> |
| 234 |
</listitem> |
| 235 |
</varlistentry> |
| 236 |
|
| 237 |
<varlistentry> |
| 238 |
<term>user_string_n(<replaceable>address</replaceable>, <replaceable>n</replaceable>)</term> |
| 239 |
<listitem> |
| 240 |
<para> |
| 241 |
Obtains the string at <replaceable>address</replaceable> for the current |
| 242 |
user process and limits the string to <replaceable>n</replaceable> bytes. |
| 243 |
</para> |
| 244 |
</listitem> |
| 245 |
</varlistentry> |
| 246 |
|
| 247 |
</variablelist> |
| 248 |
|
| 249 |
</section> |
| 250 |
|
| 251 |
<section id="ustack"> |
| 252 |
<title>User-Space Stack Backtraces</title> |
| 253 |
|
| 254 |
<indexterm> |
| 255 |
<primary>Stack backtrace</primary> |
| 256 |
<secondary>user-space</secondary> |
| 257 |
</indexterm> |
| 258 |
|
| 259 |
<!-- |
| 260 |
FIXME why is this useful? |
| 261 |
FIXME describe how it works |
| 262 |
FIXME cautions about debuginfo |
| 263 |
FIXME when was -d -ldd options introduced? |
| 264 |
FIXME description of how to use the stack backtracing |
| 265 |
--> |
| 266 |
|
| 267 |
<para> |
| 268 |
The probe point (<function>pp</function>) function indicates which |
| 269 |
particular event triggered the SystemTap event handler. |
| 270 |
A probe on the entry into a function would list the function name. |
| 271 |
However, in many cases the same probe point event may be triggered by |
| 272 |
many different modules in the program; this is particularly true for |
| 273 |
functions in shared libraries. A SystemTap backtrace of the user-space |
| 274 |
stack can provide additional context on how the probe point event is |
| 275 |
triggered. |
| 276 |
</para> |
| 277 |
|
| 278 |
<para> |
| 279 |
The user-space stack backtrace generation is complicated |
| 280 |
by the compiler producing code optimized to eliminate stack frame |
| 281 |
pointers. |
| 282 |
However, the compiler also includes information in the debug information |
| 283 |
section to allow debugging tools to produce stack backtraces. |
| 284 |
SystemTap user-space stack backtrace mechanism makes use of that |
| 285 |
debug information to walk the stack to generate stack traces for 32-bit |
| 286 |
and 64-bit x86 processors; other processor architectures do not yet |
| 287 |
support the use of debug information to unwind the user-space stack. |
| 288 |
To ensure that the needed debug information is used to produce the |
| 289 |
user-space stack backtraces, use the |
| 290 |
<option>-d <replaceable>executable</replaceable></option> option for |
| 291 |
executables and <option>--ldd</option> for shared libraries. |
| 292 |
</para> |
| 293 |
|
| 294 |
<para> |
| 295 |
For example, you can use the user-space backtrack functions to see how |
| 296 |
the <function>xmalloc</function> function is |
| 297 |
being called by the <command>ls</command> command. With the |
| 298 |
debuginfo for the <command>ls</command> command installed, the following |
| 299 |
SystemTap command provides a backtrace each time the |
| 300 |
<function>xmalloc</function> function is called: |
| 301 |
</para> |
| 302 |
|
| 303 |
<screen><command>stap -d /bin/ls --ldd \</command> |
| 304 |
<command>-e 'probe process("ls").function("xmalloc") {print_usyms(ubacktrace())}' \</command> |
| 305 |
<command>-c "ls /"</command></screen> |
| 306 |
|
| 307 |
<para> |
| 308 |
When executed, this command produces output similar to the following: |
| 309 |
</para> |
| 310 |
|
| 311 |
<screen> |
| 312 |
bin dev lib media net proc sbin sys var |
| 313 |
boot etc lib64 misc op_session profilerc selinux tmp |
| 314 |
cgroup home lost+found mnt opt root srv usr |
| 315 |
0x4116c0 : xmalloc+0x0/0x20 [/bin/ls] |
| 316 |
0x4116fc : xmemdup+0x1c/0x40 [/bin/ls] |
| 317 |
0x40e68b : clone_quoting_options+0x3b/0x50 [/bin/ls] |
| 318 |
0x4087e4 : main+0x3b4/0x1900 [/bin/ls] |
| 319 |
0x3fa441ec5d : __libc_start_main+0xfd/0x1d0 [/lib64/libc-2.12.so] |
| 320 |
0x402799 : _start+0x29/0x2c [/bin/ls] |
| 321 |
0x4116c0 : xmalloc+0x0/0x20 [/bin/ls] |
| 322 |
0x4116fc : xmemdup+0x1c/0x40 [/bin/ls] |
| 323 |
0x40e68b : clone_quoting_options+0x3b/0x50 [/bin/ls] |
| 324 |
0x40884a : main+0x41a/0x1900 [/bin/ls] |
| 325 |
0x3fa441ec5d : __libc_start_main+0xfd/0x1d0 [/lib64/libc-2.12.so] |
| 326 |
... |
| 327 |
</screen> |
| 328 |
|
| 329 |
<para> |
| 330 |
For more details on the functions available for user-space stack |
| 331 |
backtraces, refer to <filename>ucontext-symbols.stp</filename> and |
| 332 |
<filename>ucontext-unwind.stp</filename> tapsets. You can also find |
| 333 |
the description of the functions in the aforementioned tapsets in the |
| 334 |
<citetitle>SystemTap Tapset Reference Manual</citetitle>. |
| 335 |
</para> |
| 336 |
|
| 337 |
|
| 338 |
</section> |
| 339 |
|
| 340 |
</chapter> |