Debian Codegraph @

testsuite/systemtap.examples/keyword-index.txt

raw
1 SYSTEMTAP EXAMPLES INDEX BY KEYWORD
2 (see also index.txt)
3
4
5 For systemtap version 5.4.
6
7 = _BEST =
8
9 general/helloworld.stp - SystemTap "Hello World" Program
10 keywords: _best simple bpf
11
12 A basic "Hello World" program implemented in SystemTap script. It
13 prints out "hello world" message and then immediately exits.
14
15 # stap helloworld.stp
16
17
18 general/para-callgraph.stp - Callgraph Tracing with Arguments
19 keywords: _best trace callgraph
20
21 Print a timed per-thread microsecond-timed callgraph, complete with
22 function parameters and return values. The first parameter names the
23 function probe points to trace. The optional second parameter names
24 the probe points for trigger functions, which acts to enable tracing
25 for only those functions that occur while the current thread is
26 nested within the trigger.
27
28 # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \
29 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true"
30
31
32 general/varwatch.stp - Watch a Variable Changing Value in a Thread
33 keywords: _best monitoring
34
35 This script places a set of probes (specified by $1), each of which
36 monitors the state of some context $variable expression (specified by
37 $2). Whenever the value changes, with respect to the active thread,
38 the event is traced.
39
40 # stap -w varwatch.stp 'kernel.statement("do_sys_open@fs/open.c:*")' \
41 '$$vars' -T 1
42
43
44 general/whythefail.stp - Why did the function fail?
45 keywords: speculation monitoring function trace _best
46
47 The whythefail.stp script prints a statement-execution trace for a
48 given function, but only for those runs of the function that ended up
49 with a (configurable) post-return condition.
50
51 # stap whythefail.stp kernel do_sys_open '$return < 0' -c 'cat \
52 /root/no-such-file || true'
53
54
55 network/netfilter_summary.stp - System-Wide Count of Network Packets by IPs
56 keywords: _best network traffic
57
58 The script watches all IPv4 network traffic on the system. On exit
59 the script prints a list showing the number of packets sent along
60 source IP address / destination IP address pair encountered, and the
61 total number of bytes sent among the pair. The list is ordered from
62 greatest to least number of packets seen among the source/destination
63 pairs.
64
65 # stap netfilter_summary.stp -T 1
66
67
68 network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs
69 keywords: _best network traffic json
70
71 The script watches all IPv4 network traffic on the system. The data
72 is output in JSON format and includes the number of packets sent
73 along source IP address / destination IP address pair encountered,
74 and the total number of bytes sent among the pair.
75
76 # stap netfilter_summary_json.stp -T 1
77
78
79 process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task
80 keywords: _best process scheduler time tracepoint interrupt
81
82 The cycle_thief.stp script instruments the scheduler and IRQ handler
83 to determine which processes and interrupts are competing with the
84 specified task for the cpu cycles. This script uses the '-c' or '-x'
85 options to focus on a specific task. The script output the number of
86 times the task migrates between processors, histograms showing the
87 length of time on and off processor, lists of processes running while
88 the task is off the processor, and the interrupts that occurred while
89 the task was running.
90
91 # stap cycle_thief.stp -T 1
92
93
94 process/proctop.stp - Periodically Print Process Information With History
95 keywords: process scheduler _best
96
97 Every 5 seconds, print out a list of 25 processes that took the most
98 system time with information about the processes. Includes
99 information on processes that may have exited while the script was
100 running. The script contains configuration options listed in the
101 script source.
102
103 # stap proctop.stp -T 1
104
105
106 process/strace.stp - Trace system calls
107 keywords: _best process syscall
108
109 The script loosely emulates strace, when applied to individual
110 processes or hierarchies (via -c/-x), or the entire system (without
111 -c/-x). A few output configuration parameters may be set with -G.
112
113 # stap strace.stp -c "cat /proc/sys/vm/* || true"
114
115
116 process/thread-business.stp - monitor syscall history
117 keywords: _best process syscall
118
119 Prints a periodic tabular report about the counts of syscall activity
120 of all threads on the system, along with a textual
121 recent-syscall-history for each
122
123 # stap thread-business.stp -T 10
124
125
126 profiling/fntimes.stp - Show Functions Taking Longer Than Usual
127 keywords: _best profiling
128
129 The fntimes.stp script monitors the execution time history of a given
130 function family (assumed non-recursive). Each time (beyond a warmup
131 interval) is then compared to the historical maximum. If it exceeds
132 a certain threshold (250%), a message is printed.
133
134 # stap fntimes.stp 'kernel.function("sys_*")' -T 7
135
136
137 profiling/latencytap.stp - Show Reasons and Durations for Processes Sleeping
138 keywords: _best profiling
139
140 The latencytap.stp script collects data on the intervals processes
141 are deactivated (sleeping). The script categorizes the reasons for
142 the sleeps by analyzing the backtraces and displays a sorted list of
143 the top 20 causes from largest total sum time sleeping to smallest.
144 The output is updated every 30 seconds. The script needs to be
145 compiled with the '--all-modules' option to produce reasons for
146 sleeps caused by modules. Optionally, this script can be used with
147 the '-c' or '-x' options to focus on a specific PID.
148
149 # stap latencytap.stp --all-modules -T 1
150
151
152 profiling/linetimes.stp - Show Time Spent on Each Line of a Function
153 keywords: profiling _best
154
155 The linetimes.stp script takes two arguments: where to find the
156 function and the function name. linetimes.stp will instrument each
157 line in the function. It will print out the number of times that the
158 function is called, a table with the average and maximum time each
159 line takes, and control flow information when the script exits.
160
161 # stap linetimes.stp kernel do_nanosleep -T 1
162
163
164 profiling/pf4.stp - Profile Kernel/User Backtraces
165 keywords: _best profiling backtrace
166
167 The pf4.stp script sets up time-based sampling. Every five seconds it
168 prints out a sorted list with the top twenty kernel and/or user stack
169 backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d
170 MODULE, -d /PATH/TO/EXEC to add more symbolic info.
171
172 # stap pf4.stp -T 6 --all-modules --ldd
173
174
175 profiling/thread-times.stp - Profile Kernel Functions
176 keywords: _best profiling
177
178 The thread-times.stp script sets up time-based sampling. Every five
179 seconds it prints out a sorted list with the top twenty threads
180 occupying the CPUs, broken down as a percentage of user and kernel
181 time.
182
183 # stap thread-times.stp -T 1
184
185
186 stapgames/2048.stp - 2048
187 keywords: _best stapgames
188
189 The modern classic 2048 sliding-tiles game, using local keyboard and
190 ansi animation.
191
192 # stap -p4 2048.stp
193
194
195 virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons
196 keywords: _best virtualization kvm
197
198 The kvm_service_time.stp script tracks the statistics about the
199 amount of time that the processor left the guest virtual machine for
200 each exit reason (for example fixing up a page table or handling an
201 IO operation). When the script exits it prints out the number of
202 times each exit reason was encountered, the total duration of time it
203 left the guest VM, the minimum time, the average time, and the
204 maximum time in microseconds for that exit reason. On Linux 2.6.38
205 and newer kernel the script can automatically determine whether it is
206 running on Intel or AMD processors. For older kernels with a
207 kernel.trace("kvm_exit") tracepoint that does not have the $isa
208 parameter you can explicitly state the kvm type with a "-G kvm=intel"
209 or "-G kvm=amd" on the command line.
210
211 # stap kvm_service_time.stp -T 1
212
213
214 = APPLICATION =
215
216 apps/libguestfs_log.stp - Trace libguestfs startup
217 keywords: application trace bpf
218
219 The libguestfs_log.stp script prints a log of when various
220 libgueststartup steps are encountered. The first column is
221 microseconds since the script started. The second column is the time
222 elapsed in microseconds between the previous and current events and
223 the third column is the event name.
224
225 # stap libguestfs_log.stp -T 1
226
227
228 apps/stap_time.stp - Provide elapsed times for Passes of SystemTap script compilation
229 keywords: application trace
230
231 SystemTap has multiple passes to convert the text of a SystemTap
232 script into instrumentation that actually collect data on the system.
233 The stap_time.stp script uses the markers in SystemTap to note the
234 amount of time that each of the passes requires. When SystemTap
235 completes pass 4 (compiling the instrumentation into a kernel module)
236 the script print out the script name followed by the amount of time
237 in milliseconds required for Pass 0 (command line option parsing),
238 Pass 1 (script parsing), Pass 2 (elaboration), Pass 3 (code
239 generation), and Pass 4 (module compilation). The resulting data can
240 be analyzed to determine if there are issues with the amount of time
241 that systemtap takes to generate instrumentation.
242
243 # stap stap_time.stp -T 1
244
245
246 = BACKTRACE =
247
248 general/watchdog.stp - Watchdog Timer for Arbitrary Events
249 keywords: watchdog backtrace
250
251 The watchdog.stp script provides a watchdog timer mechanism for
252 arbitrary events. The script takes three arguments: the events to
253 start watchdog timer, the event to stop the watchdog timer, and the
254 time in millseconds for the watchdog. If the watchdog timer is
255 exceed, the script will trigger a stack backtrace of the user-process
256 that timed out using pstack. This script can be used to diagnose what
257 the userspace application is doing when a slower than expected
258 operation occurs.
259
260 # stap watchdog.stp 'syscall.nanosleep' 'syscall.nanosleep.return' 1000 \
261 -T 1
262
263
264 interrupt/scf.stp - Tally Backtraces for Inter-Processor Interrupt
265 keywords: interrupt backtrace
266
267 The Linux kernel function smp_call_function causes expensive
268 inter-processor interrupts (IPIs). The scf.stp script tallies the
269 processes and backtraces causing the interprocessor interrupts to
270 identify the cause of the expensive IPI. On exit the script prints
271 the tallies in descending frequency.
272
273 # stap scf.stp -T 1
274
275
276 io/io_submit.stp - Tally Reschedule Reason During AIO io_submit Call
277 keywords: io backtrace
278
279 When a reschedule occurs during an AIO io_submit call, accumulate the
280 traceback in a histogram. When the script exits prints out a sorted
281 list from most common to least common backtrace.
282
283 # stap io_submit.stp -T 1
284
285
286 memory/last_100_frees.stp - Log recent free(3) calls.
287 keywords: memory process backtrace
288
289 This script reports on the last few free(3) libc calls done by
290 processes (possibly restricted by stap -x/-c), along with a userspace
291 backtrace at those moments.
292
293 # stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd
294
295
296 process/auditbt.stp - Generate backtraces for kernel audit events
297 keywords: monitoring security backtrace
298
299 Attaches to the kernel audit-log paths (also used by libaudit), and
300 log every record being sent, along with a user-space backtrace of the
301 process that caused it.
302
303 # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true"
304
305
306 process/pstrace_exec.stp - Print trace of process ancestors for matching exec commands
307 keywords: process backtrace
308
309 The pstrace_exec.stp script watches each exec operation. If the exec
310 contains a substring that matches the script's command-line argument,
311 it prints out that process and all of its ancestors.
312
313 # stap pstrace_exec.stp -T 1 bash
314
315
316 process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations
317 keywords: io scheduler backtrace
318
319 The script monitors the time that threads spend in waiting for IO
320 operations (in "D" state) in the wait_for_completion function. If a
321 thread spends over 10ms, its name and backtrace is printed, and later
322 so is the total delay.
323
324 # stap sleepingBeauties.stp -T 1
325
326
327 profiling/pf4.stp - Profile Kernel/User Backtraces
328 keywords: _best profiling backtrace
329
330 The pf4.stp script sets up time-based sampling. Every five seconds it
331 prints out a sorted list with the top twenty kernel and/or user stack
332 backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d
333 MODULE, -d /PATH/TO/EXEC to add more symbolic info.
334
335 # stap pf4.stp -T 6 --all-modules --ldd
336
337
338 = BPF =
339
340 apps/libguestfs_log.stp - Trace libguestfs startup
341 keywords: application trace bpf
342
343 The libguestfs_log.stp script prints a log of when various
344 libgueststartup steps are encountered. The first column is
345 microseconds since the script started. The second column is the time
346 elapsed in microseconds between the previous and current events and
347 the third column is the event name.
348
349 # stap libguestfs_log.stp -T 1
350
351
352 general/ansi_colors.stp - Color Table for ansi_set_color()
353 keywords: format bpf
354
355 The script prints a table showing the available color combinations
356 for the ansi_set_color() function in the ansi.stp tapset.
357
358 # stap ansi_colors.stp
359
360
361 general/ansi_colors2.stp - Show Attribues in Table for ansi_set_color()
362 keywords: format bpf
363
364 The script prints a table showing the available attributes (bold,
365 underline, and inverse) with color combinations for the
366 ans_set_color() function in the ansi.stp tapset.
367
368 # stap ansi_colors2.stp
369
370
371 general/helloworld.stp - SystemTap "Hello World" Program
372 keywords: _best simple bpf
373
374 A basic "Hello World" program implemented in SystemTap script. It
375 prints out "hello world" message and then immediately exits.
376
377 # stap helloworld.stp
378
379
380 general/sizeof.stp - Print the Size of a C Type
381 keywords: statistics memory bpf
382
383 This script prints the size of a type, based on dwarf debuginfo for
384 any kernel or userspace module, or trial-compilation of a given
385 header file name.
386
387 # stap sizeof.stp FILE '</usr/include/stdio.h>'
388
389
390 general/stopwatches.stp - See the amount of wall clock time a process spends in various states
391 keywords: time bpf
392
393 The stopwatch.stp script illustrates how to use multiple stopwatches
394 record how much wallclock time a process spends in kernel- and
395 user-space. On exit the script prints out the time in seconds,
396 milliseconds, microseconds, and nanoseconds. Note that this output of
397 this script is not directly comparable to the time command because
398 time records the time that the process is actually active in kernel-
399 and user-space.
400
401 # stap stopwatches.stp -T 1
402
403
404 memory/cachestat.stp - Count Page Cache Hits and Misses
405 keywords: memory bpf
406
407 Monitors hits and misses to the page cache and reports a count every
408 5 seconds. Based on a bpftrace tool by David Valin.
409
410 # stap cachestat.stp -T 1
411
412
413 memory/mmfilepage.stp - Track Virtual Memory System Actions on File Backed Pages
414 keywords: memory bpf
415
416 The mmfilepage.stp script uses the virtual memory tracepoints
417 available in some kernels to track the number of faults, copy on
418 writes mapping, and unmapping operations for file backed pages. When
419 the script is terminated the counts are printed for each process that
420 allocated pages while the script was running. The mmfilepage.stp
421 script is useful in debugging leaks in the mapped file regions of a
422 process.
423
424 # stap mmfilepage.stp -T 1
425
426
427 memory/mmwriteback.stp - Track Virtual Memory System Writing to Disk
428 keywords: memory bpf
429
430 The mmwriteback.stp script uses the virtual memory tracepoints
431 available in some kernels to report all of the file writebacks that
432 occur form kupdate, pdflush and kjournald while the script is
433 running. It's useful in determining where writes are coming from on
434 a supposedly idle system that is experiencing unexpected IO.
435
436 # stap mmwriteback.stp -T 1
437
438
439 network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space
440 keywords: network tcp process bpf
441
442 The sk_stream-wait_memory.stp prints a time stamp, executable, and
443 pid each time a process blocks due to the send buffer being full. A
444 similar entry is printed each time a process continues because there
445 is room in the buffer.
446
447 # stap sk_stream_wait_memory.stp -T 1
448
449
450 profiling/syscallsrw.stp - Provide a tally of read and write syscalls run on the system
451 keywords: prometheus syscall bpf
452
453 The syscallsrw.stp script tallies the read and write syscalls. This
454 is a demo script for stapbpf prometheus-exporter scripts.
455
456
457 = CALLGRAPH =
458
459 general/callgraph.stp - Callgraph Tracing
460 keywords: simple trace callgraph
461
462 Print a timed per-thread microsecond-timed nested callgraph. The
463 first parameter names the function probe points to trace.
464
465 # stap callgraph.stp 'kernel.function("*@fs/proc*.c")' -c "cat \
466 /proc/sys/vm/* || true"
467
468
469 general/para-callgraph-verbose.stp - Callgraph Tracing with Verbose Arguments
470 keywords: trace callgraph
471
472 Print a timed per-thread microsecond-timed callgraph, complete with
473 pretty-printed function parameters and return values. The first
474 parameter names the function probe points to trace. The optional
475 second parameter names the probe points for trigger functions, which
476 acts to enable tracing for only those functions that occur while the
477 current thread is nested within the trigger.
478
479 # stap para-callgraph-verbose.stp 'kernel.function("*@fs/proc*.c")' \
480 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true"
481
482
483 general/para-callgraph.stp - Callgraph Tracing with Arguments
484 keywords: _best trace callgraph
485
486 Print a timed per-thread microsecond-timed callgraph, complete with
487 function parameters and return values. The first parameter names the
488 function probe points to trace. The optional second parameter names
489 the probe points for trigger functions, which acts to enable tracing
490 for only those functions that occur while the current thread is
491 nested within the trigger.
492
493 # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \
494 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true"
495
496
497 general/tcl-trace.stp - Callgraph Tracing of Tcl code
498 keywords: callgraph
499
500 Print a timed per-thread microsecond-timed callgraph of executing Tcl
501 code.
502
503 # stap tcl-trace.stp -T 5
504
505
506 = CONTAINER =
507
508 profiling/container_check.stp - Monitor capabilities and syscalls used by a process and it children
509 keywords: profiling container syscall
510
511 The container_check.stp script monitors the use of linux capablities
512 and optionally forbidden syscalls by a process and its children. On
513 exit the script prints out lists showing the capabilies used by each
514 executable, which syscall used specific capabilites for each
515 executable, a list of forbidden syscalls used, and details on any
516 syscalls that failed during monitoring. This script is designed to
517 help diagnose issues caused by restricted capabilies and syscalls
518 when running an application in a container. If the script warns
519 about skipped probes, the number of active kretprobes may need to be
520 increased with "-DKRETACTIVE=100" option on the command line
521
522 # stap container_check.stp -c "ping -c 1 sourceware.org || true"
523
524
525 = DIAGRAM =
526
527 process/pstree.stp - Generates a process diagram in DOT form.
528 keywords: process diagram
529
530 The pstree.stp script generates a process diagram in DOT form. For
531 instance, it may be useful on a 'make' command to see all the
532 processes that are started.
533
534 # stap pstree.stp -T 1
535
536
537 = DISK =
538
539 io/deviceseeks.stp - Histograms of Seek Behavior for Each Device
540 keywords: disk
541
542 The deviceseeks.stp script generates a periodic histogram showing the
543 frequency of different sized seeks (in sectors) on each device, or
544 optional given device-name regexp
545
546 # stap deviceseeks.stp -T 1 sda
547
548
549 io/disktop.stp - Summarize Disk Read/Write Traffic
550 keywords: disk
551
552 Get the status of reading/writing disk every 5 seconds, output top
553 ten entries during that period.
554
555 # stap disktop.stp -T 1
556
557
558 io/enospc.stp - Report an disk out-of-space condition.
559 keywords: disk filesystem
560
561 This script monitors a filesystem implementations for early internal
562 indications of ENOSPC, and reports these to the system logger and the
563 systemtap console.
564
565 # stap enospc.stp -T 1
566
567
568 io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices
569 keywords: io monitoring disk simple
570
571 The mbrwatch.stp script reports any attempted reads/writes of the
572 first few sectors of a raw block device.
573
574 # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \
575 /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \
576 head -1`"
577
578
579 network/nfsd_unlink.stp - Find Which Client Is Removing NFS Files on Server
580 keywords: nfs disk
581
582 The nfsd_unlink.stp script lists the ip address and file name each
583 time time a file is being removed or unlinked by the nfsd. This
584 script is run on the nfs server.
585
586 # stap nfsd_unlink.stp -T 1
587
588
589 = DYNINST =
590
591 apps/php-trace.stp - Tracing of PHP code execution
592 keywords: trace dyninst
593
594 Trace of executing PHP code using the enabled markers.
595
596 # stap php-trace.stp -c 'php -f hello.php'
597
598
599 = FILE =
600
601 io/iotime.stp - Trace Time Spent in Read and Write for Files
602 keywords: profiling syscall io file
603
604 The script watches each open, close, read, and write syscalls on the
605 system. For each file the scripts observes opened it accumulates the
606 amount of wall clock time spent in read and write operations and the
607 number of bytes read and written. When a file is closed the script
608 prints out a pair of lines for the file. Both lines begin with a
609 timestamp in microseconds, the PID number, and the executable name in
610 parentheses. The first line with the "access" keyword lists the file
611 name, the attempted number of bytes for the read and write
612 operations. The second line with the "iotime" keyword list the file
613 name and the number of microseconds accumulated in the read and write
614 syscalls.
615
616 # stap iotime.stp -T 1
617
618
619 process/pfiles.stp - Print Process File Descriptors
620 keywords: process file
621
622 Run pfiles.stp to produce a human-readable summary of all open file
623 descriptors of a given process. Specify the process-id as -x PID for
624 fastest performance.
625
626 # stap -g pfiles.stp -x $$
627
628
629 = FILESYSTEM =
630
631 general/badname.stp - Bad Filename Filter
632 keywords: filesystem guru
633
634 The badname.stp script shows how one could prevent the creation of
635 files with undesirable names using guru mode.
636
637 # stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \
638 denied"
639
640
641 io/enospc.stp - Report an disk out-of-space condition.
642 keywords: disk filesystem
643
644 This script monitors a filesystem implementations for early internal
645 indications of ENOSPC, and reports these to the system logger and the
646 systemtap console.
647
648 # stap enospc.stp -T 1
649
650
651 lwtools/fslatency-nd.stp - Measure the distribution of file system synchronous read and write latency (non-debuginfo)
652 keywords: io filesystem
653
654 This dynamically traces two common file system functions:
655 do_sync_read() and do_sync_write(), and reports a histogram
656 distribution of latency. Many, but not all, file systems and
657 workloads use these functions. Tracing their time provides one view
658 of suffered file system latency.
659
660 # stap fslatency-nd.stp 1 1
661
662
663 lwtools/fsslower-nd.stp - Trace slow file system synchronous reads and writes (non-debuginfo)
664 keywords: io filesystem
665
666 This dynamically traces two common file system functions:
667 do_sync_read() and do_sync_write(), and shows details of each call
668 that is slower than a threshold. Many, but not all, file systems and
669 workloads use these functions. Tracing their time provides one view
670 of suffered file system latency.
671
672 # stap fsslower-nd.stp -T 1
673
674
675 = FLOATINGPOINT =
676
677 general/floatingpoint.stp - SystemTap "Floating Point" Program
678 keywords: simple floatingpoint
679
680 A basic "Floating Point" program implemented in SystemTap script. It
681 extracts floating point from benchmark-sdt section in main.cxx and
682 takes one pseudo floating point, printing out results of various
683 simple floating point operations.
684
685 # stap floatingpoint.stp -c "stap --benchmark-sdt"
686
687
688 = FORMAT =
689
690 general/ansi_colors.stp - Color Table for ansi_set_color()
691 keywords: format bpf
692
693 The script prints a table showing the available color combinations
694 for the ansi_set_color() function in the ansi.stp tapset.
695
696 # stap ansi_colors.stp
697
698
699 general/ansi_colors2.stp - Show Attribues in Table for ansi_set_color()
700 keywords: format bpf
701
702 The script prints a table showing the available attributes (bold,
703 underline, and inverse) with color combinations for the
704 ans_set_color() function in the ansi.stp tapset.
705
706 # stap ansi_colors2.stp
707
708
709 = FUNCTION =
710
711 general/func_time_stats.stp - Function Time Statistics
712 keywords: function statistics
713
714 The func_time_stats.stp script tracks the wall clock time for each
715 invocation of the function probe listed as the first command line
716 argument. When the script exits it prints out the minimum, average,
717 and maximum times in microseconds followed by a count of times that
718 the function was called and a histogram showing the distributions of
719 times.
720
721 # stap func_time_stats.stp 'syscall.read' -T 1
722
723
724 general/whythefail.stp - Why did the function fail?
725 keywords: speculation monitoring function trace _best
726
727 The whythefail.stp script prints a statement-execution trace for a
728 given function, but only for those runs of the function that ended up
729 with a (configurable) post-return condition.
730
731 # stap whythefail.stp kernel do_sys_open '$return < 0' -c 'cat \
732 /root/no-such-file || true'
733
734
735 profiling/functioncallcount.stp - Count Times Functions Are Called
736 keywords: profiling function
737
738 The functioncallcount.stp script takes one argument, a list of
739 functions to probe. The script will run and count the number of times
740 that each of the functions on the list is called. On exit the script
741 will print a sorted list from most frequently to least frequently
742 called function.
743
744 # stap -w functioncallcount.stp "*@fs/*.c" -T 1
745
746
747 profiling/sched_switch.stp - Display the Task Switches Happening in the Scheduler
748 keywords: profiling function
749
750 The sched_switch.stp script takes two arguments, first argument can
751 be "pid" or "name" to indicate what is being passed as second
752 argument. The script will trace the process based on pid/name and
753 print the scheduler switches happening with the process. If no
754 arguments are passed, it displays all the scheduler switches. This
755 can be used to understand which tasks schedule out the current
756 process being traced, and when it gets scheduled in again.
757
758 # stap sched_switch.stp -T 1
759
760
761 = FUTEX =
762
763 process/futexes.stp - System-Wide Futex Contention
764 keywords: syscall locking futex
765
766 The script watches the futex syscall on the system. On exit the
767 futex's address, the number of contentions, and the average time for
768 each contention on the futex are printed from lowest pid number to
769 highest.
770
771 # stap futexes.stp -T 1
772
773
774 process/futexes2.stp - System-Wide Shared Futex Contention
775 keywords: syscall locking futex
776
777 The script watches just shared futex syscalls on the system. On exit
778 the futex's key, the number of contentions, and the average time for
779 each contention on the futex are printed from lowest pid number to
780 highest.
781
782 # stap futexes2.stp -T 1
783
784
785 = GURU =
786
787 general/badname.stp - Bad Filename Filter
788 keywords: filesystem guru
789
790 The badname.stp script shows how one could prevent the creation of
791 files with undesirable names using guru mode.
792
793 # stap -g badname.stp -c "touch /tmp/myXXXbadnameXXXfile.$$ 2>&1 | grep \
794 denied"
795
796
797 general/keyhack.stp - Hack the keyboard
798 keywords: simple guru
799
800 This script makes it appear that one's keyboard is broken, by
801 changing keycodes at the kernel device driver level. Annoy your
802 friends!
803
804 # stap -g keyhack.stp -c 'sleep 5'
805
806
807 io/eatmydata.stp - disable fsync
808 keywords: io guru simple
809
810 Suppresses fsync() syscalls from processes identified by stap -c/-x
811 by turning them into presumed-faster fsync() on some dummy or other
812 file descriptor
813
814 # stap -g eatmydata.stp -c 'strace ls || true'
815
816
817 io/ttyspy.stp - Monitor TTY Typing
818 keywords: io tty monitoring guru
819
820 The ttyspy.stp script uses tty_audit hooks to monitor recent typing
821 activity on the system, printing a scrolling record of recent
822 keystrokes, on a per-tty basis.
823
824 # stap --skip-badvars -g ttyspy.stp -T 1
825
826
827 network/netfilter_drop.stp - System-Wide Network Packet Dropping Tool
828 keywords: network packets guru
829
830 The script drops the specified number of packets of the specified
831 protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified,
832 all incoming packets are dropped. The number of packets to drop can
833 be specified with a positive integer. A value of 0 indicates that
834 packets should be dropped until the user manually exits.
835
836 # stap -g netfilter_drop.stp TCP 1 -T 2
837
838
839 network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10
840 keywords: network tcp socket guru
841
842 Run the tcp_init_cwnd.stp script in the background to override a
843 kernel's default tcp cwnd value to 10, which has been found to
844 improve latency for web server type workloads. The script prints a
845 count of cwnd value changes when it is stopped.
846
847 # stap -g tcp_init_cwnd.stp -T 1
848
849
850 process/noptrace.stp - Disable ptrace from Hierarchies of Processes
851 keywords: process security guru
852
853 Blocks ptrace(2) attempts from processes identified by stap -c/-x, as
854 also specifiable from /proc/systemtap/stap_XXX/ control files.
855 Processes may be added or removed from the blocked list.
856
857 # stap -g noptrace.stp -c 'strace ls || true'
858
859
860 process/threadstacks.stp - Override default new-pthread stack sizes
861 keywords: thread guru
862
863 Overrides default NPTL pthread_create stack size for all new threads
864 created by target processes. Reports one line per process when the
865 related glibc variable __default_stacksize is updated. Moot for
866 glibc versions that support $LIBC_PTHREAD_DEFAULT_STACKSIZE_NP.
867
868 # stap -g threadstacks.stp -Gsize=65536 -T 1 -d `which stap`
869
870
871 security-band-aids/cve-2008-0600.stp - cve-2008-0600 security band-aid
872 keywords: security guru
873
874 historical emergency security band-aid, for reference/education only
875
876
877 security-band-aids/cve-2011-4127.stp - cve-2011-4127 security band-aid
878 keywords: security guru
879
880 historical emergency security band-aid, for reference/education only
881
882
883 security-band-aids/cve-2012-0056.stp - cve-2012-0056 security band-aid
884 keywords: security guru
885
886 historical emergency security band-aid, for reference/education only
887
888
889 security-band-aids/cve-2013-2094.stp - cve-2013-2094 security band-aid
890 keywords: security guru
891
892 historical emergency security band-aid, for reference/education only
893
894
895 security-band-aids/cve-2014-7169.stp - cve-2014-7169 security band-aid
896 keywords: security guru
897
898 historical emergency security band-aid, for reference/education only
899
900
901 security-band-aids/cve-2015-0235.stp - cve-2015-0235 security band-aid
902 keywords: security guru
903
904 historical emergency security band-aid, for reference/education only
905
906
907 security-band-aids/cve-2015-3456.stp - cve-2015-3456 security band-aid
908 keywords: security guru
909
910 historical emergency security band-aid, for reference/education only
911
912
913 security-band-aids/cve-2015-7547.stp - cve-2015-7547 security band-aid
914 keywords: security guru
915
916 historical emergency security band-aid, for reference/education only
917
918
919 security-band-aids/cve-2016-0728.stp - cve-2016-0728 security band-aid
920 keywords: security guru
921
922 historical emergency security band-aid, for reference/education only
923
924
925 security-band-aids/cve-2016-5195.stp - cve-2016-5195 security band-aid
926 keywords: security guru
927
928 historical emergency security band-aid, for reference/education only
929
930
931 security-band-aids/cve-2017-6074.stp - cve-2017-6074 security band-aid
932 keywords: security guru
933
934 historical emergency security band-aid, for reference/education only
935
936
937 security-band-aids/cve-2018-1000001.stp - cve-2018-1000001 security band-aid
938 keywords: security guru
939
940 historical emergency security band-aid, for reference/education only
941
942
943 security-band-aids/cve-2018-14634.stp - cve-2018-14634 security band-aid
944 keywords: security guru
945
946 historical emergency security band-aid, for reference/education only
947
948
949 security-band-aids/cve-2018-6485.stp - cve-2018-6485 security band-aid
950 keywords: security guru
951
952 historical emergency security band-aid, for reference/education only
953
954
955 security-band-aids/cve-2021-4034.stp - cve-2021-4034 security band-aid
956 keywords: security guru
957
958 historical emergency security band-aid, for reference/education only
959
960
961 security-band-aids/cve-2021-4155.stp - cve-2021-4155 security band-aid
962 keywords: security guru
963
964 historical emergency security band-aid, for reference/education only
965
966
967 security-band-aids/cve-2023-4911.stp - cve-2023-4911 security band-aid
968 keywords: security guru
969
970 historical emergency security band-aid, for reference/education only
971
972
973 = INTERACTIVE =
974
975 general/eventcount.stp - Interactive Count Specified Events
976 keywords: interactive statistics thread process
977
978 The script periodically prints a count of specified events and their
979 related tid's over the course of execution. Numerous configuration
980 options exist to control filtering / reporting, some of which can be
981 modified at runtime. See the script source for more information.
982
983 # stap eventcount.stp 'syscall_any' -c 'sleep 3'
984
985
986 general/sizeof_interactive.stp - Interactive Print the Size of a C Type
987 keywords: statistics memory interactive
988
989 This script prints the size of a type, based on dwarf debuginfo for
990 any kernel or userspace module, or trial-compilation of a given
991 header file name. Types and corresponding locations are provided to
992 the script at runtime via keyboard input. The format of the input is
993 identical to that of sizeof.stp (see script source for more
994 information). Types and locations can be repeatedly given until the
995 process is terminated.
996
997 # echo "FILE '</usr/include/stdio.h>'" | stap sizeof_interactive.stp -T 1
998
999
1000 = INTERRUPT =
1001
1002 interrupt/interrupts-by-dev.stp - Record Interrupts on a Per-Device Basis
1003 keywords: interrupt
1004
1005 The interrupts-by-dev.stp script profiles interrupts received by each
1006 device per 100 ms.
1007
1008 # stap interrupts-by-dev.stp -T 1
1009
1010
1011 interrupt/scf.stp - Tally Backtraces for Inter-Processor Interrupt
1012 keywords: interrupt backtrace
1013
1014 The Linux kernel function smp_call_function causes expensive
1015 inter-processor interrupts (IPIs). The scf.stp script tallies the
1016 processes and backtraces causing the interprocessor interrupts to
1017 identify the cause of the expensive IPI. On exit the script prints
1018 the tallies in descending frequency.
1019
1020 # stap scf.stp -T 1
1021
1022
1023 process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task
1024 keywords: _best process scheduler time tracepoint interrupt
1025
1026 The cycle_thief.stp script instruments the scheduler and IRQ handler
1027 to determine which processes and interrupts are competing with the
1028 specified task for the cpu cycles. This script uses the '-c' or '-x'
1029 options to focus on a specific task. The script output the number of
1030 times the task migrates between processors, histograms showing the
1031 length of time on and off processor, lists of processes running while
1032 the task is off the processor, and the interrupts that occurred while
1033 the task was running.
1034
1035 # stap cycle_thief.stp -T 1
1036
1037
1038 = IO =
1039
1040 general/alias_suffixes.stp - Count I/O Syscalls using Alias Suffixes
1041 keywords: io statistics
1042
1043 alias_suffixes.stp is a demonstration of how alias suffixes in the
1044 systemtap language might be used. The script tracks the wall clock
1045 time for each invocation of the system calls open, close, read, and
1046 write. When the script exists it prints out the minimum, average, and
1047 maximum times in microseconds for each system call, followed by a
1048 count of times that each syscall was invoked and a histogram showing
1049 the distributions of times.
1050
1051 # stap alias_suffixes.stp -T 1
1052
1053
1054 io/capture_ssl_master_secrets.stp - Capture SSL/TLS master secrets from gnutls and openssl users
1055 keywords: io monitoring
1056
1057 Capture SSL/TLS master secrets from gnutls and openssl users for easy
1058 traffic decryption
1059
1060 # stap capture_ssl_master_secrets.stp -c 'sleep 3'
1061
1062
1063 io/eatmydata.stp - disable fsync
1064 keywords: io guru simple
1065
1066 Suppresses fsync() syscalls from processes identified by stap -c/-x
1067 by turning them into presumed-faster fsync() on some dummy or other
1068 file descriptor
1069
1070 # stap -g eatmydata.stp -c 'strace ls || true'
1071
1072
1073 io/inodewatch.stp - Monitoring Reads and Writes to a File
1074 keywords: io
1075
1076 The inodewatch.stp outputs the executable name and process id each
1077 time a read or write occurs to the specified inode on the specified
1078 major/minor device.
1079
1080 # stap inodewatch.stp 0x08 0x01 100 -T 1
1081
1082
1083 io/inodewatch2.stp - Monitoring Attribute Changes to a File
1084 keywords: io
1085
1086 The inodewatch2.stp script outputs the executable name, process id,
1087 and attributes each time the attributes are changed on the specified
1088 inode on the specified major/minor device.
1089
1090 # stap inodewatch2.stp 0x08 0x01 100 -T 1
1091
1092
1093 io/io_submit.stp - Tally Reschedule Reason During AIO io_submit Call
1094 keywords: io backtrace
1095
1096 When a reschedule occurs during an AIO io_submit call, accumulate the
1097 traceback in a histogram. When the script exits prints out a sorted
1098 list from most common to least common backtrace.
1099
1100 # stap io_submit.stp -T 1
1101
1102
1103 io/ioblktime.stp - Average Time Block IO Requests Spend in Queue
1104 keywords: io
1105
1106 The ioblktime.stp script tracks the amount of time that each block IO
1107 requests spend waiting for completion. The script computes the
1108 average waiting time for block IO per device and prints list every 10
1109 seconds. In some cases there can be too many outstanding block IO
1110 operations and the script may exceed the default number of
1111 MAXMAPENTRIES allowed. In this case the allowed number can be
1112 increased with "-DMAXMAPENTRIES=10000" option on the stap command
1113 line.
1114
1115 # stap ioblktime.stp -T 1
1116
1117
1118 io/iodevstats.stp - List Executables Reading and Writing the Most Data by Device
1119 keywords: io profiling
1120
1121 The iodevstats.stp script measures the amount of data successfully
1122 read and written by all the executables for each io device on the
1123 system. The output is sorted from greatest sum of bytes read and
1124 written to a device by an executable to the least. The output
1125 contains device major/minor number, the count of operations (reads
1126 and writes), the totals and averages for the number of bytes read and
1127 written.
1128
1129 # stap iodevstats.stp -T 1
1130
1131
1132 io/iostat-scsi.stp - IO Statistics for SCSI Devices
1133 keywords: io profiling scsi
1134
1135 The iostat-scsi.stp script provides a breakdown of the number of blks
1136 read and written on the machine's various SCSI devices. The script
1137 takes one argument which is the number of seconds between reports.
1138
1139 # stap -g iostat-scsi.stp 1 -T 1
1140
1141
1142 io/iostats.stp - List Executables Reading and Writing the Most Data
1143 keywords: io profiling
1144
1145 The iostat.stp script measures the amount of data successfully read
1146 and written by all the executables on the system. The output is
1147 sorted from most greatest sum of bytes read and written by an
1148 executable to the least. The output contains the count of operations
1149 (opens, reads, and writes), the totals and averages for the number of
1150 bytes read and written.
1151
1152 # stap iostats.stp -T 1
1153
1154
1155 io/iotime.stp - Trace Time Spent in Read and Write for Files
1156 keywords: profiling syscall io file
1157
1158 The script watches each open, close, read, and write syscalls on the
1159 system. For each file the scripts observes opened it accumulates the
1160 amount of wall clock time spent in read and write operations and the
1161 number of bytes read and written. When a file is closed the script
1162 prints out a pair of lines for the file. Both lines begin with a
1163 timestamp in microseconds, the PID number, and the executable name in
1164 parentheses. The first line with the "access" keyword lists the file
1165 name, the attempted number of bytes for the read and write
1166 operations. The second line with the "iotime" keyword list the file
1167 name and the number of microseconds accumulated in the read and write
1168 syscalls.
1169
1170 # stap iotime.stp -T 1
1171
1172
1173 io/iotop.stp - Periodically Print IO Activity by Process Name
1174 keywords: io
1175
1176 Every five seconds print out the top ten executables generating I/O
1177 traffic during that interval sorted in descending order.
1178
1179 # stap iotop.stp -T 1
1180
1181
1182 io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices
1183 keywords: io monitoring disk simple
1184
1185 The mbrwatch.stp script reports any attempted reads/writes of the
1186 first few sectors of a raw block device.
1187
1188 # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \
1189 /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \
1190 head -1`"
1191
1192
1193 io/nfs_func_users.stp - Tally the Number of NFS Functions Used by Each Process
1194 keywords: io profiling
1195
1196 The nfs_func_users.stp script counts the uses of NFS functions in the
1197 kernel on a per process bases. The output is sorted from the process
1198 with the greatest number of NFS functions called to the least. The
1199 output contains the executable name, the process number, and the
1200 total number of NFS functions called by the process.
1201
1202 # stap nfs_func_users.stp -T 1
1203
1204
1205 io/slowvfs.stp - Trace slow vfs opens.
1206 keywords: io simple
1207
1208 This script prints a line for every kernel vfs_open operation that
1209 takes longer than a configurable number of microseconds. Highly
1210 contended or remote filesystems are likelier to hit this.
1211
1212 # stap slowvfs.stp -G sloth=10 -c 'find /proc >/dev/null || true'
1213
1214
1215 io/switchfile.stp - Switch log files
1216 keywords: io
1217
1218 Every second print a log message and switch log files every 5
1219 seconds.
1220
1221 # stap -o /tmp/switchfile.stp.out switchfile.stp -T 8
1222
1223
1224 io/traceaio.stp - Trace calls to io_submit
1225 keywords: io
1226
1227 Trace calls to io_submit, logging iocbs and iovecs.
1228
1229 # stap traceaio.stp -x 1 -T 1
1230
1231
1232 io/traceio.stp - Track Cumulative IO Activity by Process Name
1233 keywords: io
1234
1235 Every second print out the top ten executables sorted in descending
1236 order based on cumulative I/O traffic observed.
1237
1238 # stap traceio.stp -T 1
1239
1240
1241 io/traceio2.stp - Watch IO Activity on a Particular Device
1242 keywords: io
1243
1244 Print out the executable name and process number as reads and writes
1245 to the specified device occur.
1246
1247 # stap traceio2.stp 0x0801 -T 1
1248
1249
1250 io/traceio3.stp - Trace VFS writes to files
1251 keywords: io
1252
1253 Print a list of files being written to at the kernel VFS layer.
1254 Optionally, accept a regular expression to match file names against.
1255
1256 # stap traceio3.stp /dev -T 5
1257
1258
1259 io/ttyspy.stp - Monitor TTY Typing
1260 keywords: io tty monitoring guru
1261
1262 The ttyspy.stp script uses tty_audit hooks to monitor recent typing
1263 activity on the system, printing a scrolling record of recent
1264 keystrokes, on a per-tty basis.
1265
1266 # stap --skip-badvars -g ttyspy.stp -T 1
1267
1268
1269 lwtools/biolatency-nd.stp - Measure block I/O latency distribution (non-debuginfo)
1270 keywords: io
1271
1272 This measures block I/O latency (storage I/O, ie, disk I/O), and
1273 shows the distribution as a histogram. This can be useful to identify
1274 the characteristics of I/O latency, beyond the averages shown by
1275 iostat(1). For example, to study I/O latency outliers, or multi-modal
1276 distributions.
1277
1278 # stap biolatency-nd.stp 1 1
1279
1280
1281 lwtools/bitesize-nd.stp - Measure block I/O size distribution (non-debuginfo)
1282 keywords: io
1283
1284 This uses the kernel tracepoint block_rq_insert to read the size of
1285 I/O. The output includes the name of the process or thread that was
1286 on-CPU when the I/O request was inserted on the issue queue.
1287
1288 # stap bitesize-nd.stp -T 1
1289
1290
1291 lwtools/execsnoop-nd.stp - Trace process exec() with command line argument details (non-debuginfo)
1292 keywords: io
1293
1294 This can identify if CPU is consumed by short-lived processes, by
1295 tracing new process execution. It works by tracing exec() from the
1296 fork()->exec() sequence, which means it will not catch new processes
1297 that only fork(). It will also show every exec(), including those if
1298 a process re-execs.
1299
1300 # stap execsnoop-nd.stp -T 1
1301
1302
1303 lwtools/fslatency-nd.stp - Measure the distribution of file system synchronous read and write latency (non-debuginfo)
1304 keywords: io filesystem
1305
1306 This dynamically traces two common file system functions:
1307 do_sync_read() and do_sync_write(), and reports a histogram
1308 distribution of latency. Many, but not all, file systems and
1309 workloads use these functions. Tracing their time provides one view
1310 of suffered file system latency.
1311
1312 # stap fslatency-nd.stp 1 1
1313
1314
1315 lwtools/fsslower-nd.stp - Trace slow file system synchronous reads and writes (non-debuginfo)
1316 keywords: io filesystem
1317
1318 This dynamically traces two common file system functions:
1319 do_sync_read() and do_sync_write(), and shows details of each call
1320 that is slower than a threshold. Many, but not all, file systems and
1321 workloads use these functions. Tracing their time provides one view
1322 of suffered file system latency.
1323
1324 # stap fsslower-nd.stp -T 1
1325
1326
1327 lwtools/killsnoop-nd.stp - Trace kill() signals showing process and signal details (non-debuginfo)
1328 keywords: io
1329
1330 This traces signals system-wide, including those sent by the kill(1)
1331 command, and shows various details.
1332
1333 # stap killsnoop-nd.stp -T 1
1334
1335
1336 lwtools/opensnoop-nd.stp - Trace open() syscalls showing filenames (non-debuginfo)
1337 keywords: io
1338
1339 This traces the open() syscall system-wide, to show which files are
1340 being opened, and by who.
1341
1342 # stap opensnoop-nd.stp -T 1
1343
1344
1345 lwtools/rwtime-nd.stp - Summarize read() and write() syscall latency (non-debuginfo)
1346 keywords: io
1347
1348 This traces read() and write() syscalls, producing a histogram
1349 summary of their durations (aka latencies).
1350
1351 # stap rwtime-nd.stp -T 1
1352
1353
1354 lwtools/syscallbypid-nd.stp - Count syscalls with process details (non-debuginfo)
1355 keywords: io
1356
1357 This traces syscalls system-wide, and produces a summary report
1358 showing their counts by process ID, process name, and syscall types.
1359
1360 # stap syscallbypid-nd.stp -T 1
1361
1362
1363 process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations
1364 keywords: io scheduler backtrace
1365
1366 The script monitors the time that threads spend in waiting for IO
1367 operations (in "D" state) in the wait_for_completion function. If a
1368 thread spends over 10ms, its name and backtrace is printed, and later
1369 so is the total delay.
1370
1371 # stap sleepingBeauties.stp -T 1
1372
1373
1374 virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port
1375 keywords: virtualization qemu kvm io
1376
1377 The qemu_io.stp script tallies the number of times each of the IO
1378 port on the guest virtual machines is touched by a input or output
1379 operation. When the script exits, it prints a count of the number of
1380 times each IO port read and written.
1381
1382 # stap qemu_io.stp -T 1
1383
1384
1385 = JSON =
1386
1387 network/net_xmit_json.stp - Tracks time between packet queue and transmit.
1388 keywords: network statistics json
1389
1390 This script tracks time between packet queue and transmit. The
1391 information is provided to userspace via procfs in JSON format.
1392
1393 # stap net_xmit_json.stp -T 1
1394
1395
1396 network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs
1397 keywords: _best network traffic json
1398
1399 The script watches all IPv4 network traffic on the system. The data
1400 is output in JSON format and includes the number of packets sent
1401 along source IP address / destination IP address pair encountered,
1402 and the total number of bytes sent among the pair.
1403
1404 # stap netfilter_summary_json.stp -T 1
1405
1406
1407 = KVM =
1408
1409 virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons
1410 keywords: _best virtualization kvm
1411
1412 The kvm_service_time.stp script tracks the statistics about the
1413 amount of time that the processor left the guest virtual machine for
1414 each exit reason (for example fixing up a page table or handling an
1415 IO operation). When the script exits it prints out the number of
1416 times each exit reason was encountered, the total duration of time it
1417 left the guest VM, the minimum time, the average time, and the
1418 maximum time in microseconds for that exit reason. On Linux 2.6.38
1419 and newer kernel the script can automatically determine whether it is
1420 running on Intel or AMD processors. For older kernels with a
1421 kernel.trace("kvm_exit") tracepoint that does not have the $isa
1422 parameter you can explicitly state the kvm type with a "-G kvm=intel"
1423 or "-G kvm=amd" on the command line.
1424
1425 # stap kvm_service_time.stp -T 1
1426
1427
1428 virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events
1429 keywords: virtualization qemu kvm
1430
1431 The qemu_count.stp script tallies the number of times each of the
1432 user-space qemu probepoints is encountered. When the script exits, it
1433 prints a list of the number of times each user-space qemu probepoint
1434 is encountered.
1435
1436 # stap qemu_count.stp -T 1
1437
1438
1439 virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port
1440 keywords: virtualization qemu kvm io
1441
1442 The qemu_io.stp script tallies the number of times each of the IO
1443 port on the guest virtual machines is touched by a input or output
1444 operation. When the script exits, it prints a count of the number of
1445 times each IO port read and written.
1446
1447 # stap qemu_io.stp -T 1
1448
1449
1450 = LIMITS =
1451
1452 general/cpu_throttle.stp - Monitor Intel processors for throttling due to power or thermal limits
1453 keywords: prometheus limits
1454
1455 Intel processors have hardware mechanisms that will reduce the
1456 effective processors speed to avoid exceeding set thermal or power
1457 constraints. The cpu_throttle.stp script monitors when CPU throttling
1458 occurs and accumulates the amount of time in milliseconds that each
1459 processor is throttled. The script makes the information available
1460 via procfs in Prometheus readable format.
1461
1462 # stap cpu_throttle.stp -T 1
1463
1464
1465 memory/overcommit.stp - Log Failed Process Memory Allocation Due to Overcommit Limits
1466 keywords: memory limits
1467
1468 The overcommit.stp script prints a line each time the kernel refuses
1469 a memory allocation request from a process because of
1470 /proc/sys/vm/overcommit* limits.
1471
1472 # stap overcommit.stp -T 1
1473
1474
1475 process/rlimit_nofile.stp - Trace processes running out of file descriptors
1476 keywords: limits
1477
1478 This script watches processes being scheduled and which try to
1479 allocate a file descriptor without luck.
1480
1481 # stap rlimit_nofile.stp -T 1
1482
1483
1484 = LOCKING =
1485
1486 locks/bkl.stp - Tracing Contention on Big Kernel Lock
1487 keywords: locking
1488
1489 The bkl.stp script can help determine whether the Big Kernel Lock
1490 (BKL) is causing serialization on a multiprocessor system due to
1491 excessive contention of the BKL. The bkl.stp script takes two
1492 arguments. The first one is optional, and used to enable backtraces,
1493 and print them once a process has been holding the BKL for a user
1494 specified number of nseconds is reached. The second option is
1495 compulsory and is the number of processes waiting for the Big Kernel
1496 Lock (BKL). When the number of processes waiting for the BKL is
1497 reached or exceeded, the script will print a time stamp, the number
1498 of processes waiting for the BKL, the holder of the BKL, and the
1499 amount of time the BKL was held. If backtraces are enabled, a
1500 backtrace will be printed as well.
1501
1502 # stap bkl.stp -T 1 1
1503
1504
1505 locks/bkl_stats.stp - Per Process Statistics on Big Kernel Lock Use
1506 keywords: locking
1507
1508 The bkl_stats.stp script can indicate which processes have excessive
1509 waits for the Big Kernel Lock (BKL) and which processes are taking
1510 the BKL for long periods of time. The bkl_stats.stp script prints
1511 lists of all the processes that require the BKL. Every five seconds
1512 two tables are printed out. The first table lists the processes that
1513 waited for the BKL followed by the number of times that the process
1514 waited, the minimum time of the wait, the average and the maximum
1515 time waited. The second table lists has similar information for the
1516 time spent in holding the lock for each of the processes.
1517
1518 # stap bkl_stats.stp -T 1
1519
1520
1521 process/futexes.stp - System-Wide Futex Contention
1522 keywords: syscall locking futex
1523
1524 The script watches the futex syscall on the system. On exit the
1525 futex's address, the number of contentions, and the average time for
1526 each contention on the futex are printed from lowest pid number to
1527 highest.
1528
1529 # stap futexes.stp -T 1
1530
1531
1532 process/futexes2.stp - System-Wide Shared Futex Contention
1533 keywords: syscall locking futex
1534
1535 The script watches just shared futex syscalls on the system. On exit
1536 the futex's key, the number of contentions, and the average time for
1537 each contention on the futex are printed from lowest pid number to
1538 highest.
1539
1540 # stap futexes2.stp -T 1
1541
1542
1543 process/mutex-contention.stp - pthread mutex contention analysis
1544 keywords: locking
1545
1546 Tracks pthread-mutex initialization/use and underlying futex
1547 operations, to identify (with backtraces/symbol-names) the mutexes
1548 suffering most contention. Invoke with "-d SHLIB --ldd", perhaps
1549 with -DMAXMAPENTRIES=NNNN for some large NNNN, if the arrays overflow
1550 due to heavy activity.
1551
1552 # stap mutex-contention.stp -T 1
1553
1554
1555 process/semop-watch.stp - Watch semop(2)/semtimedop(2) operations
1556 keywords: process locking
1557
1558 Prints a timed trace of semop(2)/semtimedop(2) syscalls
1559
1560 # stap semop-watch.stp -c 'sleep 2'
1561
1562
1563 = MEMORY =
1564
1565 general/sizeof.stp - Print the Size of a C Type
1566 keywords: statistics memory bpf
1567
1568 This script prints the size of a type, based on dwarf debuginfo for
1569 any kernel or userspace module, or trial-compilation of a given
1570 header file name.
1571
1572 # stap sizeof.stp FILE '</usr/include/stdio.h>'
1573
1574
1575 general/sizeof_interactive.stp - Interactive Print the Size of a C Type
1576 keywords: statistics memory interactive
1577
1578 This script prints the size of a type, based on dwarf debuginfo for
1579 any kernel or userspace module, or trial-compilation of a given
1580 header file name. Types and corresponding locations are provided to
1581 the script at runtime via keyboard input. The format of the input is
1582 identical to that of sizeof.stp (see script source for more
1583 information). Types and locations can be repeatedly given until the
1584 process is terminated.
1585
1586 # echo "FILE '</usr/include/stdio.h>'" | stap sizeof_interactive.stp -T 1
1587
1588
1589 memory/cachestat.stp - Count Page Cache Hits and Misses
1590 keywords: memory bpf
1591
1592 Monitors hits and misses to the page cache and reports a count every
1593 5 seconds. Based on a bpftrace tool by David Valin.
1594
1595 # stap cachestat.stp -T 1
1596
1597
1598 memory/glibc-malloc.stp - Overview glibc malloc internal operations
1599 keywords: memory process
1600
1601 This script reports on internal statistics of the glibc malloc
1602 implementation, as used by a process restricted by stap -x/-c
1603
1604 # stap glibc-malloc.stp -c 'stap --dump-functions'
1605
1606
1607 memory/hugepage_clear_delays.stp - Summarize the time each process spends clearing huge pages
1608 keywords: memory
1609
1610 When a huge page is initially used by a process it must be cleared of
1611 its original contents to avoid leaking information between processes.
1612 On x86_64 machines the huge pages are 2MB in size, 512 times larger
1613 than a normal 4KB page. Thus, clearing a huge page can delay program
1614 execution by a hundred or more microseconds and maybe noticeable to
1615 latency sensitive programs. This script will tally the number of
1616 times that each process triggers a huge page clear, the total time in
1617 microseconds spent clearing the pages, and the average time in
1618 microseconds of the huge page clear.
1619
1620 # stap hugepage_clear_delays.stp -T 1
1621
1622
1623 memory/hugepage_collapse.stp - Log the kernel collapsing normal pages into a huge page
1624 keywords: memory
1625
1626 The kernel may attempt to group many normal sized pages into a single
1627 huge page to improve TLB performance. This operation may take a
1628 significant amount of time and affect program performance. You can
1629 identify if huge page collapse operations are occurring with the
1630 systemtap hugepage_collapse.stp script.
1631
1632 # stap hugepage_collapse.stp -T 1
1633
1634
1635 memory/hugepage_cow_delays.stp - Summarize the time each process spends doing copy on write for huge pages
1636 keywords: memory
1637
1638 The system may attempt to reduce memory use by having two or more
1639 difference processes share the same writeable page in memory.
1640 However, when one of the processes write to shared page that is
1641 marked copy on write (COW) the page must be duplicated and then
1642 modified. On x86_64 machines the huge pages are 2MB in size, 512
1643 times larger than a normal 4KB page. Thus, copy on write operation
1644 on a huge page can delay program execution by a hundred or more
1645 microseconds and maybe noticeable to latency sensitive programs.
1646 This script will tally the number of times that each process triggers
1647 a huge page copy on write, the total time in microseconds spent
1648 copying the pages, and the average time in microseconds of the huge
1649 page copy.
1650
1651 # stap hugepage_cow_delays.stp -T 1
1652
1653
1654 memory/hugepage_split.stp - Log the kernel splitting huge pages into normal sized pages
1655 keywords: memory
1656
1657 Because some portions of the kernel code only work with normal-sized
1658 pages the kernel may convert a huge page into a set of normal-sized
1659 pages using a split operation. This operation may be relatively
1660 expensive and noticeable to latency sensitive programs. You can
1661 identify if split operations are occurring with the systemtap
1662 hugepage_split.stp script.
1663
1664 # stap hugepage_split.stp -T 1
1665
1666
1667 memory/hw_watch_addr.stp - Watch a Kernel Address Using Breakpoint Hardware
1668 keywords: memory watchpoint
1669
1670 The script will watch accesses to a single kernel address and prints
1671 a traceback each time the address is accessed. This script needs to
1672 be run as root to allow access to the breakpoint hardware.
1673
1674 # stap --all-modules hw_watch_addr.stp 0x`grep "vm_dirty_ratio" \
1675 /proc/kallsyms | awk '{print $1}'` -T 5
1676
1677
1678 memory/hw_watch_sym.stp - Watch a Kernel Symbol Using Breakpoint Hardware
1679 keywords: memory watchpoint
1680
1681 The script will watch accesses to the starting address of a single
1682 kernel symbol and prints a traceback each time the symbol is
1683 accessed. This script needs to be run as root to allow access to the
1684 breakpoint hardware.
1685
1686 # stap --all-modules hw_watch_sym.stp vm_dirty_ratio -T 5
1687
1688
1689 memory/kmalloc-top - Show Paths to Kernel Malloc Invocations
1690 keywords: memory
1691
1692 The kmalloc-top perl program runs a small systemtap script to collect
1693 stack traces for each call to the kmalloc function and counts the
1694 time that each stack trace is observed. When kmalloc-top exits it
1695 prints out sorted list. The output can be filtered to print only the
1696 first N stack traces (-t), stack traces with a minimum counts (-m),
1697 or exclude certain stack traces (-e).
1698
1699 # ./kmalloc-top -c "sleep 1"
1700
1701
1702 memory/last_100_frees.stp - Log recent free(3) calls.
1703 keywords: memory process backtrace
1704
1705 This script reports on the last few free(3) libc calls done by
1706 processes (possibly restricted by stap -x/-c), along with a userspace
1707 backtrace at those moments.
1708
1709 # stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd
1710
1711
1712 memory/mmanonpage.stp - Track Virtual Memory System Actions on Anonymous Pages
1713 keywords: memory
1714
1715 The mmanonpage.stp script uses the virtual memory tracepoints
1716 available in some kernels to track the number of faults, user space
1717 frees, page ins, copy on writes and unmaps for anonymous pages. When
1718 the script is terminated the counts are printed for each process that
1719 allocated pages while the script was running. This script displays
1720 the anonymous page statistics for each process that ran while the
1721 script is active. It's useful in debugging leaks in the anonymous
1722 regions of a process.
1723
1724 # stap mmanonpage.stp -T 1
1725
1726
1727 memory/mmfilepage.stp - Track Virtual Memory System Actions on File Backed Pages
1728 keywords: memory bpf
1729
1730 The mmfilepage.stp script uses the virtual memory tracepoints
1731 available in some kernels to track the number of faults, copy on
1732 writes mapping, and unmapping operations for file backed pages. When
1733 the script is terminated the counts are printed for each process that
1734 allocated pages while the script was running. The mmfilepage.stp
1735 script is useful in debugging leaks in the mapped file regions of a
1736 process.
1737
1738 # stap mmfilepage.stp -T 1
1739
1740
1741 memory/mmreclaim.stp - Track Virtual Memory System Page Reclamation
1742 keywords: memory
1743
1744 The mmreclaim.stp script uses the virtual memory tracepoints
1745 available in some kernels to track page reclaim activity that
1746 occurred while the script was running. It's useful in debugging
1747 performance problems that occur due to page reclamation.
1748
1749 # stap mmreclaim.stp -T 1
1750
1751
1752 memory/mmwriteback.stp - Track Virtual Memory System Writing to Disk
1753 keywords: memory bpf
1754
1755 The mmwriteback.stp script uses the virtual memory tracepoints
1756 available in some kernels to report all of the file writebacks that
1757 occur form kupdate, pdflush and kjournald while the script is
1758 running. It's useful in determining where writes are coming from on
1759 a supposedly idle system that is experiencing unexpected IO.
1760
1761 # stap mmwriteback.stp -T 1
1762
1763
1764 memory/numa_faults.stp - Summarize Process Misses across NUMA Nodes
1765 keywords: memory numa
1766
1767 The numa_faults.stp script tracks the read and write pages faults for
1768 each process. When the script exits it prints out the total read and
1769 write pages faults for each process. The script also provide a break
1770 down of page faults per node for each process. This script is useful
1771 for determining whether the program has good locality (page faults
1772 limited to a single node) on a NUMA computer.
1773
1774 # stap numa_faults.stp -T 1
1775
1776
1777 memory/overcommit.stp - Log Failed Process Memory Allocation Due to Overcommit Limits
1778 keywords: memory limits
1779
1780 The overcommit.stp script prints a line each time the kernel refuses
1781 a memory allocation request from a process because of
1782 /proc/sys/vm/overcommit* limits.
1783
1784 # stap overcommit.stp -T 1
1785
1786
1787 memory/pfaults.stp - Generate Log of Major and Minor Page Faults
1788 keywords: memory
1789
1790 The pfaults.stp script generates a simple log for each major and
1791 minor page fault that occurs on the system. Each line contains a
1792 timestamp (in microseconds) when the page fault servicing was
1793 completed, the pid of the process, the address of the page fault, the
1794 type of access (read or write), the type of fault (major or minor),
1795 and the elapsed time for page fault. This log can be examined to
1796 determine where the page faults are occurring.
1797
1798 # stap pfaults.stp -T 1
1799
1800
1801 memory/swapmonitor.stp - Summarize swap operations
1802 keywords: memory swap
1803
1804 The swapmonitor.stp script is an example to monitor "swap in/out" in
1805 RHEL. This stap monitor the call of "swap_readpage" and
1806 "swap_writepage", and then printf the timeframe, func, execname, pid
1807 and the count with 10 seconds interval.
1808
1809 # stap swapmonitor.stp -T 5
1810
1811
1812 memory/vm.tracepoints.stp - Collect Slab Allocation Statistics
1813 keywords: memory statistics
1814
1815 The script will probe all memory slab/slub allocations and collects
1816 information about the size of the object (bytes requested) and
1817 user-space process in execution. When run over a period of time, it
1818 helps to correlate kernel-space memory consumption owing to
1819 user-space processes.
1820
1821 # stap vm.tracepoints.stp -T 10
1822
1823
1824 = MONITORING =
1825
1826 general/graphs.stp - Graphing Disk and CPU Utilization with gnuplot
1827 keywords: monitoring
1828
1829 The script tracks the disk and CPU utilization. It prints a stream
1830 of data which, when piped directly into gnuplot, draws historical
1831 curves for each.
1832
1833 # stap graphs.stp -T 1
1834
1835
1836 general/measureinterval.stp - Measure intervals between events
1837 keywords: monitoring
1838
1839 The measureinteval.stp script allows quick creation of
1840 instrumentation to measure quantity between different probe points,
1841 aggregate the measurements, and display the results. The first
1842 argument to the script is the quantity to measure such as time in
1843 microseconds via "gettimeofday_us()". There could be multiple
1844 intervals being measured concurrently and the second argument is the
1845 means to match the start probe points listed in the fourth argument
1846 with the end probes listed in the fifth argument. The third argument
1847 is how to display the aggregated data. The data could be displayed
1848 as a sum (@sum), an average (@avg), or a histogram (@hist_log). The
1849 fourth and fifth arguments are lists of probes marking the start and
1850 end of an interval. An optional sixth argument allows grouping of
1851 the intervals. One could use "tid()" for the matching argument to
1852 measure the intervals on a per-thread basis, but then use
1853 "execname()" for the optional argument to group the measurements by
1854 executable name. If there is no sixth argument, all the intervals
1855 are aggregated into a single entry.
1856
1857 # stap measureinterval.stp "gettimeofday_us()" "tid()" @sum vm.pagefault \
1858 vm.pagefault.return "execname()" -c 'sleep 3'
1859
1860
1861 general/varwatch.stp - Watch a Variable Changing Value in a Thread
1862 keywords: _best monitoring
1863
1864 This script places a set of probes (specified by $1), each of which
1865 monitors the state of some context $variable expression (specified by
1866 $2). Whenever the value changes, with respect to the active thread,
1867 the event is traced.
1868
1869 # stap -w varwatch.stp 'kernel.statement("do_sys_open@fs/open.c:*")' \
1870 '$$vars' -T 1
1871
1872
1873 general/whythefail.stp - Why did the function fail?
1874 keywords: speculation monitoring function trace _best
1875
1876 The whythefail.stp script prints a statement-execution trace for a
1877 given function, but only for those runs of the function that ended up
1878 with a (configurable) post-return condition.
1879
1880 # stap whythefail.stp kernel do_sys_open '$return < 0' -c 'cat \
1881 /root/no-such-file || true'
1882
1883
1884 io/capture_ssl_master_secrets.stp - Capture SSL/TLS master secrets from gnutls and openssl users
1885 keywords: io monitoring
1886
1887 Capture SSL/TLS master secrets from gnutls and openssl users for easy
1888 traffic decryption
1889
1890 # stap capture_ssl_master_secrets.stp -c 'sleep 3'
1891
1892
1893 io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices
1894 keywords: io monitoring disk simple
1895
1896 The mbrwatch.stp script reports any attempted reads/writes of the
1897 first few sectors of a raw block device.
1898
1899 # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \
1900 /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \
1901 head -1`"
1902
1903
1904 io/ttyspy.stp - Monitor TTY Typing
1905 keywords: io tty monitoring guru
1906
1907 The ttyspy.stp script uses tty_audit hooks to monitor recent typing
1908 activity on the system, printing a scrolling record of recent
1909 keystrokes, on a per-tty basis.
1910
1911 # stap --skip-badvars -g ttyspy.stp -T 1
1912
1913
1914 process/auditbt.stp - Generate backtraces for kernel audit events
1915 keywords: monitoring security backtrace
1916
1917 Attaches to the kernel audit-log paths (also used by libaudit), and
1918 log every record being sent, along with a user-space backtrace of the
1919 process that caused it.
1920
1921 # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true"
1922
1923
1924 process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading
1925 keywords: process monitoring syscall tracepoint
1926
1927 The procmod_watcher.stp script monitors calls to fork(), exec(),
1928 exit(), init_module(), and delete_module(). Event-specific details
1929 are also printed out (e.g. for exec(), the file being exec'ed). This
1930 script does not require debuginfo.
1931
1932 # stap procmod_watcher.stp -T 1
1933
1934
1935 = NANOSLEEP =
1936
1937 process/sleeptime.stp - Trace Time Spent in Nanosleep Syscalls
1938 keywords: syscall nanosleep
1939
1940 The script watches each nanosleep syscall on the system. At the end
1941 of each nanosleep syscall the script prints out a line with a
1942 timestamp in microseconds, the pid, the executable name in
1943 parentheses, the "nanosleep:" key, and the duration of the sleep in
1944 microseconds.
1945
1946 # stap sleeptime.stp -T 1
1947
1948
1949 = NETWORK =
1950
1951 lwtools/accept2close-nd.stp - Show socket lifespan, from accept() to close() (non-debuginfo)
1952 keywords: network socket
1953
1954 This traces socket duration from the accept() syscall to close(), and
1955 provides details on the lifespan of these passive connections,
1956 showing the distribution as a histogram.
1957
1958 # stap accept2close-nd.stp -T 1
1959
1960
1961 network/autofs4.stp - Watch autofs4 Operations
1962 keywords: network nfs
1963
1964 Trace key autofs4 operations such as mounting or unmounting remote
1965 filesystems.
1966
1967 # stap autofs4.stp -T 1
1968
1969
1970 network/connect_stat.stp - Show Process Ancestry for IP Connections
1971 keywords: network socket process
1972
1973 The connect_stat.stp script prints a task's entire ancestry (parent
1974 process name/uid/gid) whenever it attempts an outgoing socket
1975 connection to a given IP address.
1976
1977 # stap connect_stat.stp 127.0.0.1 -T 1
1978
1979
1980 network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel
1981 keywords: network tracepoint socket
1982
1983 Every five seconds the dropwatch.stp script lists the number of
1984 socket buffers freed at locations in the kernel.
1985
1986 # stap -d kernel dropwatch.stp -T 1
1987
1988
1989 network/net_xmit_json.stp - Tracks time between packet queue and transmit.
1990 keywords: network statistics json
1991
1992 This script tracks time between packet queue and transmit. The
1993 information is provided to userspace via procfs in JSON format.
1994
1995 # stap net_xmit_json.stp -T 1
1996
1997
1998 network/netdev.stp - Trace Activity on Network Devices
1999 keywords: network traffic
2000
2001 The netdev.stp script traces configuration and transmit/receive
2002 activity on network devices.
2003
2004 # stap netdev.stp -T 1
2005
2006
2007 network/netfilter_drop.stp - System-Wide Network Packet Dropping Tool
2008 keywords: network packets guru
2009
2010 The script drops the specified number of packets of the specified
2011 protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified,
2012 all incoming packets are dropped. The number of packets to drop can
2013 be specified with a positive integer. A value of 0 indicates that
2014 packets should be dropped until the user manually exits.
2015
2016 # stap -g netfilter_drop.stp TCP 1 -T 2
2017
2018
2019 network/netfilter_summary.stp - System-Wide Count of Network Packets by IPs
2020 keywords: _best network traffic
2021
2022 The script watches all IPv4 network traffic on the system. On exit
2023 the script prints a list showing the number of packets sent along
2024 source IP address / destination IP address pair encountered, and the
2025 total number of bytes sent among the pair. The list is ordered from
2026 greatest to least number of packets seen among the source/destination
2027 pairs.
2028
2029 # stap netfilter_summary.stp -T 1
2030
2031
2032 network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs
2033 keywords: _best network traffic json
2034
2035 The script watches all IPv4 network traffic on the system. The data
2036 is output in JSON format and includes the number of packets sent
2037 along source IP address / destination IP address pair encountered,
2038 and the total number of bytes sent among the pair.
2039
2040 # stap netfilter_summary_json.stp -T 1
2041
2042
2043 network/nettop.stp - Periodic Listing of Processes Using Network Interfaces
2044 keywords: network traffic
2045
2046 Every five seconds the nettop.stp script prints out a list of
2047 processed (PID and command) with the number of packets sent/received
2048 and the amount of data sent/received by the process during that
2049 interval.
2050
2051 # stap nettop.stp -T 1
2052
2053
2054 network/packet_contents.stp - Network packet contents
2055 keywords: network traffic
2056
2057 The packet_contents.stp script displays the length of each network
2058 packet and its contents in both hexadecimal and ASCII. Systemtap
2059 strings are MAXSTRINGLEN in length by default which may not be enough
2060 for larger packets. In order to print larger packets, this limit can
2061 be increased by passing in the "-DMAXSTRINGLEN=65536" command line
2062 option.
2063
2064 # stap packet_contents.stp -T 1
2065
2066
2067 network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space
2068 keywords: network tcp process bpf
2069
2070 The sk_stream-wait_memory.stp prints a time stamp, executable, and
2071 pid each time a process blocks due to the send buffer being full. A
2072 similar entry is printed each time a process continues because there
2073 is room in the buffer.
2074
2075 # stap sk_stream_wait_memory.stp -T 1
2076
2077
2078 network/socket-trace.stp - Trace Functions Called in Network Socket Code
2079 keywords: network socket
2080
2081 The script instruments each of the functions in the Linux kernel's
2082 net/socket.c file. The script prints out trace data. The first
2083 element of a line is time delta in microseconds from the previous
2084 entry. This is followed by the command name and the PID. The "->" and
2085 "<-" indicates function entry and function exit, respectively. The
2086 last element of the line is the function name.
2087
2088 # stap socket-trace.stp -T 1
2089
2090
2091 network/socktop - Periodically Summarize Socket Activity on the System
2092 keywords: network socket
2093
2094 The socktop script periodically prints out a list of the processes
2095 with the highest socket activity. Command line options for the
2096 script allow filtering to focus on particular types of sockets. The
2097 "-h" option lists socktop script's filtering options.
2098
2099 # ./socktop -c 1
2100
2101
2102 network/stp_dump.stp - Dump of STP packets
2103 keywords: network traffic
2104
2105 The stp_dump.stp prints out the packet contents. Each block contains
2106 the STP protocol ID, version ID, flags, root and bridge MAC
2107 addresses, and various times.
2108
2109 # stap stp_dump.stp -T 5
2110
2111
2112 network/tcp_connections.stp - Track Creation of Incoming TCP Connections
2113 keywords: network tcp socket
2114
2115 The tcp_connections.stp script prints information for each new
2116 incoming TCP connection accepted by the computer. The information
2117 includes the UID, the command accepting the connection, the PID of
2118 the command, the port the connection is on, and the IP address of the
2119 originator of the request.
2120
2121 # stap tcp_connections.stp -T 1
2122
2123
2124 network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10
2125 keywords: network tcp socket guru
2126
2127 Run the tcp_init_cwnd.stp script in the background to override a
2128 kernel's default tcp cwnd value to 10, which has been found to
2129 improve latency for web server type workloads. The script prints a
2130 count of cwnd value changes when it is stopped.
2131
2132 # stap -g tcp_init_cwnd.stp -T 1
2133
2134
2135 network/tcp_trace.stp - TCP Connection Tracing Utility
2136 keywords: network trace
2137
2138 This scripts traces a given TCP connection based on the filter
2139 parameters given by the user. The indexing is done by the 4 tuples
2140 local address, remote address, local port, remote port.
2141
2142 # stap tcp_trace.stp 127.0.0.1:*-127.0.0.1:* timeout=1
2143
2144
2145 network/tcpdumplike.stp - Dump of Received UDP/TCP Packets
2146 keywords: network traffic
2147
2148 The tcpdumplike.stp prints out a line for each TCP & UDP packet
2149 received. Each line includes the source and destination IP addresses,
2150 the source and destination ports, and flags.
2151
2152 # stap tcpdumplike.stp -T 1
2153
2154
2155 network/tcpipstat.stp - Display Network Statistics for Individual TCP Sockets
2156 keywords: network statistics
2157
2158 The tcpipstat script collects and displays network statistics related
2159 to individual TCP sockets or groups of sockets. The statistics that
2160 are collected are simular to that of the command netstat -s, only
2161 sorted and grouped by individual sockets.
2162
2163 # stap tcpipstat.stp timeout=1
2164
2165
2166 network/who_sent_it.stp - Trace threads sending network traffic to given host and/or port
2167 keywords: network trace traffic simple
2168
2169 This script traces outgoing network packets using the netfilter
2170 probes (not requiring debuginfo), printing the source thread name/id
2171 and destination host:port. It may be filtered with the_dport and
2172 the_daddr globals, e.g., to watch only for DNS traffic (port 53),
2173 and/or only to the localhost (127.0.0.1).
2174
2175 # stap who_sent_it.stp -G the_dport=53 -c "ping -c 1 sourceware.org || \
2176 true"
2177
2178
2179 = NFS =
2180
2181 network/autofs4.stp - Watch autofs4 Operations
2182 keywords: network nfs
2183
2184 Trace key autofs4 operations such as mounting or unmounting remote
2185 filesystems.
2186
2187 # stap autofs4.stp -T 1
2188
2189
2190 network/nfsd-recent.stp - Keep track of NFS server statistics
2191 keywords: nfs statistics
2192
2193 This script tracks all nfsd server operations by client_ip address,
2194 and periodically lists those clients that have made recent requests.
2195 It's a way of finding out which nfs clients might be considered still
2196 connected.
2197
2198 # stap nfsd-recent.stp -T 1
2199
2200
2201 network/nfsd-trace.stp - Trace NFSD requests with file names.
2202 keywords: nfs trace
2203
2204 This script traces all nfsd server operations by client_ip address,
2205 operation, and complete file name (if possible).
2206
2207 # stap nfsd-trace.stp -T 1
2208
2209
2210 network/nfsd_unlink.stp - Find Which Client Is Removing NFS Files on Server
2211 keywords: nfs disk
2212
2213 The nfsd_unlink.stp script lists the ip address and file name each
2214 time time a file is being removed or unlinked by the nfsd. This
2215 script is run on the nfs server.
2216
2217 # stap nfsd_unlink.stp -T 1
2218
2219
2220 network/nfsdtop.stp - Keep track of NFS server statistics
2221 keywords: nfs statistics
2222
2223 The nfsdtop.stp script gathers and displays NFS lookups,
2224
2225 # stap nfsdtop.stp -T 1
2226
2227
2228 = NUMA =
2229
2230 memory/numa_faults.stp - Summarize Process Misses across NUMA Nodes
2231 keywords: memory numa
2232
2233 The numa_faults.stp script tracks the read and write pages faults for
2234 each process. When the script exits it prints out the total read and
2235 write pages faults for each process. The script also provide a break
2236 down of page faults per node for each process. This script is useful
2237 for determining whether the program has good locality (page faults
2238 limited to a single node) on a NUMA computer.
2239
2240 # stap numa_faults.stp -T 1
2241
2242
2243 = PACKETS =
2244
2245 network/netfilter_drop.stp - System-Wide Network Packet Dropping Tool
2246 keywords: network packets guru
2247
2248 The script drops the specified number of packets of the specified
2249 protocol. Valid protocols are TCP, UDP, or ALL. If ALL is specified,
2250 all incoming packets are dropped. The number of packets to drop can
2251 be specified with a positive integer. A value of 0 indicates that
2252 packets should be dropped until the user manually exits.
2253
2254 # stap -g netfilter_drop.stp TCP 1 -T 2
2255
2256
2257 = PROCESS =
2258
2259 general/eventcount.stp - Interactive Count Specified Events
2260 keywords: interactive statistics thread process
2261
2262 The script periodically prints a count of specified events and their
2263 related tid's over the course of execution. Numerous configuration
2264 options exist to control filtering / reporting, some of which can be
2265 modified at runtime. See the script source for more information.
2266
2267 # stap eventcount.stp 'syscall_any' -c 'sleep 3'
2268
2269
2270 memory/glibc-malloc.stp - Overview glibc malloc internal operations
2271 keywords: memory process
2272
2273 This script reports on internal statistics of the glibc malloc
2274 implementation, as used by a process restricted by stap -x/-c
2275
2276 # stap glibc-malloc.stp -c 'stap --dump-functions'
2277
2278
2279 memory/last_100_frees.stp - Log recent free(3) calls.
2280 keywords: memory process backtrace
2281
2282 This script reports on the last few free(3) libc calls done by
2283 processes (possibly restricted by stap -x/-c), along with a userspace
2284 backtrace at those moments.
2285
2286 # stap last_100_frees.stp -c "stap -V" -d `which stap` --ldd
2287
2288
2289 network/connect_stat.stp - Show Process Ancestry for IP Connections
2290 keywords: network socket process
2291
2292 The connect_stat.stp script prints a task's entire ancestry (parent
2293 process name/uid/gid) whenever it attempts an outgoing socket
2294 connection to a given IP address.
2295
2296 # stap connect_stat.stp 127.0.0.1 -T 1
2297
2298
2299 network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space
2300 keywords: network tcp process bpf
2301
2302 The sk_stream-wait_memory.stp prints a time stamp, executable, and
2303 pid each time a process blocks due to the send buffer being full. A
2304 similar entry is printed each time a process continues because there
2305 is room in the buffer.
2306
2307 # stap sk_stream_wait_memory.stp -T 1
2308
2309
2310 process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task
2311 keywords: _best process scheduler time tracepoint interrupt
2312
2313 The cycle_thief.stp script instruments the scheduler and IRQ handler
2314 to determine which processes and interrupts are competing with the
2315 specified task for the cpu cycles. This script uses the '-c' or '-x'
2316 options to focus on a specific task. The script output the number of
2317 times the task migrates between processors, histograms showing the
2318 length of time on and off processor, lists of processes running while
2319 the task is off the processor, and the interrupts that occurred while
2320 the task was running.
2321
2322 # stap cycle_thief.stp -T 1
2323
2324
2325 process/errsnoop.stp - Tabulate System Call Errors
2326 keywords: process syscall
2327
2328 Prints a periodic tabular report about failing system calls, by
2329 process and by syscall failure. The first optional argument
2330 specifies the reporting interval (in seconds, default 5); the second
2331 optional argument gives a screen height (number of lines in the
2332 report, default 20).
2333
2334 # stap errsnoop.stp 1 10 -T 1
2335
2336
2337 process/forktracker.stp - Trace Creation of Processes
2338 keywords: process scheduler
2339
2340 The forktracker.stp script prints out a time-stamped entry showing
2341 each fork and exec operation on the machine. This can be useful to
2342 determine what process is creating a flurry of short-lived processes.
2343
2344 # stap forktracker.stp -T 1
2345
2346
2347 process/ltrace.stp - uprobes-based ltrace
2348 keywords: process
2349
2350 The ltrace.stp script lists calls that the designated process makes
2351 through PLTs (procedure linkage tables), generally into shared
2352 libraries.
2353
2354 # stap ltrace.stp -c ls || echo PR14738
2355
2356
2357 process/noptrace.stp - Disable ptrace from Hierarchies of Processes
2358 keywords: process security guru
2359
2360 Blocks ptrace(2) attempts from processes identified by stap -c/-x, as
2361 also specifiable from /proc/systemtap/stap_XXX/ control files.
2362 Processes may be added or removed from the blocked list.
2363
2364 # stap -g noptrace.stp -c 'strace ls || true'
2365
2366
2367 process/pfiles.stp - Print Process File Descriptors
2368 keywords: process file
2369
2370 Run pfiles.stp to produce a human-readable summary of all open file
2371 descriptors of a given process. Specify the process-id as -x PID for
2372 fastest performance.
2373
2374 # stap -g pfiles.stp -x $$
2375
2376
2377 process/plimit.stp - Print Resource Limits of Process
2378 keywords: process
2379
2380 The script prints a variety of resource limits for a given pid, like
2381 /proc/$$/limits on recent kernels.
2382
2383 # stap -g plimit.stp $$
2384
2385
2386 process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading
2387 keywords: process monitoring syscall tracepoint
2388
2389 The procmod_watcher.stp script monitors calls to fork(), exec(),
2390 exit(), init_module(), and delete_module(). Event-specific details
2391 are also printed out (e.g. for exec(), the file being exec'ed). This
2392 script does not require debuginfo.
2393
2394 # stap procmod_watcher.stp -T 1
2395
2396
2397 process/proctop.stp - Periodically Print Process Information With History
2398 keywords: process scheduler _best
2399
2400 Every 5 seconds, print out a list of 25 processes that took the most
2401 system time with information about the processes. Includes
2402 information on processes that may have exited while the script was
2403 running. The script contains configuration options listed in the
2404 script source.
2405
2406 # stap proctop.stp -T 1
2407
2408
2409 process/psig.stp - Print Process File Descriptors
2410 keywords: process signals
2411
2412 Run psig.stp to produce a human-readable summary of the signal
2413 handling configuration of a given process. Specify the process-id as
2414 -x PID for fastest performance.
2415
2416 # stap -DMAXACTION=10000 -g psig.stp -x $$
2417
2418
2419 process/pstrace_exec.stp - Print trace of process ancestors for matching exec commands
2420 keywords: process backtrace
2421
2422 The pstrace_exec.stp script watches each exec operation. If the exec
2423 contains a substring that matches the script's command-line argument,
2424 it prints out that process and all of its ancestors.
2425
2426 # stap pstrace_exec.stp -T 1 bash
2427
2428
2429 process/pstree.stp - Generates a process diagram in DOT form.
2430 keywords: process diagram
2431
2432 The pstree.stp script generates a process diagram in DOT form. For
2433 instance, it may be useful on a 'make' command to see all the
2434 processes that are started.
2435
2436 # stap pstree.stp -T 1
2437
2438
2439 process/sched-latency.stp - track wakeup-to-dispatch latency stats
2440 keywords: process scheduler time tracepoint
2441
2442 This script periodically reports a histogram of the latency between a
2443 task (thread) being woken up and it actually being dispatched to a
2444 CPU: the amount of time it's spent in the runnable queue.
2445
2446 # stap sched-latency.stp -T 7
2447
2448
2449 process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints
2450 keywords: process scheduler time tracepoint
2451
2452 The schedtimes.stp script instruments the scheduler to track the
2453 amount of time that each process spends in running, sleeping,
2454 queuing, and waiting for io. On exit the script prints out the
2455 accumulated time for each state of processes observed. Optionally,
2456 this script can be used with the '-c' or '-x' options to focus on a
2457 specific PID and its children.
2458
2459 # stap schedtimes.stp -T 1
2460
2461
2462 process/semop-watch.stp - Watch semop(2)/semtimedop(2) operations
2463 keywords: process locking
2464
2465 Prints a timed trace of semop(2)/semtimedop(2) syscalls
2466
2467 # stap semop-watch.stp -c 'sleep 2'
2468
2469
2470 process/spawn_seeker.stp - Track Creation of Processes by process and execname
2471 keywords: process scheduler
2472
2473 The spawn_seeker.stp script every minute (and on exit) prints out the
2474 local time and sorted lists of which processes and executables
2475 spawned tasks during the previous minute. This can be useful to
2476 determine what process is creating a flurry of short-lived processes.
2477 When a process exits its count of tasks created is added to its
2478 parent's count to better account for the indirect task creation by
2479 children processes. For more detailed examination of task creation
2480 consider using forktracker.stp.
2481
2482 # stap spawn_seeker.stp -T 1
2483
2484
2485 process/strace.stp - Trace system calls
2486 keywords: _best process syscall
2487
2488 The script loosely emulates strace, when applied to individual
2489 processes or hierarchies (via -c/-x), or the entire system (without
2490 -c/-x). A few output configuration parameters may be set with -G.
2491
2492 # stap strace.stp -c "cat /proc/sys/vm/* || true"
2493
2494
2495 process/thread-business.stp - monitor syscall history
2496 keywords: _best process syscall
2497
2498 Prints a periodic tabular report about the counts of syscall activity
2499 of all threads on the system, along with a textual
2500 recent-syscall-history for each
2501
2502 # stap thread-business.stp -T 10
2503
2504
2505 process/wait4time.stp - Trace Time Spent in wait4 Syscalls
2506 keywords: syscall process
2507
2508 The script watches each wait4 syscall on the system. At the end of
2509 each wait4 syscall the script prints out a line with a timestamp in
2510 microseconds, the pid, the executable name in parentheses, the
2511 "wait4:" key, the duration of the wait and the PID that the wait4 was
2512 waiting for. If the waited for PID is not specified , it is "-1".
2513
2514 # stap wait4time.stp -T 1
2515
2516
2517 profiling/syscallerrorsbypid.stp - Provide a per-process syscall error tally on the system
2518 keywords: prometheus process syscall tracepoint
2519
2520 The syscallerrorsbypid.stp script tallies syscall errors for each
2521 running process. This information can be useful to whether there are
2522 excessive errors for various processes on the system. The script
2523 makes the information available via procfs in Prometheus readable
2524 format. When a process exits its data will be eliminated from the
2525 prometheus output. To avoid exceeding the storage limitations of
2526 SystemTap older entries maybe overwritten by newer entries. This can
2527 lead to some active process syscall error counts disappearing and/or
2528 later reappearing with a lower value. Also note that the script does
2529 not properly name syscalls for 32-bit applications running on 64-bit
2530 machines.
2531
2532 # stap syscallerrorsbypid.stp -T 1
2533
2534
2535 profiling/syscalllatency.stp - Provide a per-process accumulation of syscall latency on the system
2536 keywords: prometheus process syscall tracepoint
2537
2538 The syscalllatency.stp script accumulates syscall latency for each
2539 running process. This information can be useful to whether excessive
2540 time is being spent in particular syscalls on the system. The script
2541 makes the information available via procfs in Prometheus readable
2542 format. When a process exits its data will be eliminated from the
2543 prometheus output. To avoid exceeding the storage limitations of
2544 SystemTap older entries maybe overwritten by newer entries. This can
2545 lead to some active process syscall error counts disappearing and/or
2546 later reappearing with a lower value. Also note that the script does
2547 not properly name syscalls for 32-bit applications running on 64-bit
2548 machines.
2549
2550 # stap syscalllatency.stp -T 1
2551
2552
2553 profiling/syscallsbypid.stp - Provide a per-process syscall tally on the system
2554 keywords: prometheus process syscall tracepoint
2555
2556 The syscallsbypid.stp script tallies each syscall for each running
2557 process. This information can be useful to determine the activity of
2558 various processes on the system. The script makes the information
2559 available via procfs in Prometheus readable format. When a process
2560 exits its data will be eliminated from the prometheus output. To
2561 avoid exceeding the storage limitations of SystemTap older entries
2562 maybe overwritten by newer entries. This can lead to some active
2563 process syscall counts disappearing and/or later reappearing with a
2564 lower value. Also note that the script does not properly name
2565 syscalls for 32-bit applications running on 64-bit machines.
2566
2567 # stap syscallsbypid.stp -T 1
2568
2569
2570 = PROFILING =
2571
2572 general/tcl-funtop.stp - Profile Tcl calls
2573 keywords: profiling
2574
2575 The tcl-funtop.stp script displays a top-like view of most frequently
2576 used Tcl functions for the each five second interval.
2577
2578 # stap tcl-funtop.stp -T 5
2579
2580
2581 io/iodevstats.stp - List Executables Reading and Writing the Most Data by Device
2582 keywords: io profiling
2583
2584 The iodevstats.stp script measures the amount of data successfully
2585 read and written by all the executables for each io device on the
2586 system. The output is sorted from greatest sum of bytes read and
2587 written to a device by an executable to the least. The output
2588 contains device major/minor number, the count of operations (reads
2589 and writes), the totals and averages for the number of bytes read and
2590 written.
2591
2592 # stap iodevstats.stp -T 1
2593
2594
2595 io/iostat-scsi.stp - IO Statistics for SCSI Devices
2596 keywords: io profiling scsi
2597
2598 The iostat-scsi.stp script provides a breakdown of the number of blks
2599 read and written on the machine's various SCSI devices. The script
2600 takes one argument which is the number of seconds between reports.
2601
2602 # stap -g iostat-scsi.stp 1 -T 1
2603
2604
2605 io/iostats.stp - List Executables Reading and Writing the Most Data
2606 keywords: io profiling
2607
2608 The iostat.stp script measures the amount of data successfully read
2609 and written by all the executables on the system. The output is
2610 sorted from most greatest sum of bytes read and written by an
2611 executable to the least. The output contains the count of operations
2612 (opens, reads, and writes), the totals and averages for the number of
2613 bytes read and written.
2614
2615 # stap iostats.stp -T 1
2616
2617
2618 io/iotime.stp - Trace Time Spent in Read and Write for Files
2619 keywords: profiling syscall io file
2620
2621 The script watches each open, close, read, and write syscalls on the
2622 system. For each file the scripts observes opened it accumulates the
2623 amount of wall clock time spent in read and write operations and the
2624 number of bytes read and written. When a file is closed the script
2625 prints out a pair of lines for the file. Both lines begin with a
2626 timestamp in microseconds, the PID number, and the executable name in
2627 parentheses. The first line with the "access" keyword lists the file
2628 name, the attempted number of bytes for the read and write
2629 operations. The second line with the "iotime" keyword list the file
2630 name and the number of microseconds accumulated in the read and write
2631 syscalls.
2632
2633 # stap iotime.stp -T 1
2634
2635
2636 io/nfs_func_users.stp - Tally the Number of NFS Functions Used by Each Process
2637 keywords: io profiling
2638
2639 The nfs_func_users.stp script counts the uses of NFS functions in the
2640 kernel on a per process bases. The output is sorted from the process
2641 with the greatest number of NFS functions called to the least. The
2642 output contains the executable name, the process number, and the
2643 total number of NFS functions called by the process.
2644
2645 # stap nfs_func_users.stp -T 1
2646
2647
2648 profiling/container_check.stp - Monitor capabilities and syscalls used by a process and it children
2649 keywords: profiling container syscall
2650
2651 The container_check.stp script monitors the use of linux capablities
2652 and optionally forbidden syscalls by a process and its children. On
2653 exit the script prints out lists showing the capabilies used by each
2654 executable, which syscall used specific capabilites for each
2655 executable, a list of forbidden syscalls used, and details on any
2656 syscalls that failed during monitoring. This script is designed to
2657 help diagnose issues caused by restricted capabilies and syscalls
2658 when running an application in a container. If the script warns
2659 about skipped probes, the number of active kretprobes may need to be
2660 increased with "-DKRETACTIVE=100" option on the command line
2661
2662 # stap container_check.stp -c "ping -c 1 sourceware.org || true"
2663
2664
2665 profiling/errno.stp - Show Which Processes and System Calls Return Errors Most Frequently
2666 keywords: profiling
2667
2668 On exit the errno.stp script provides a sorted list showing which
2669 combination of PID, system call, and error occur most frequently.
2670
2671 # stap errno.stp -T 1
2672
2673
2674 profiling/fileline-profile.stp - Profile Kernel/User Functions
2675 keywords: profiling
2676
2677 The fileline-profile.stp script ends by printing out a sorted list of
2678 the top twenty kernel and/or user processes providing file:line
2679 information, if available, from the samples addresses gathered over
2680 the time period the script is run. Use any of --ldd, --all-modules,
2681 -d MODULE, -d /PATH/TO/EXEC to add more symbolic info. To include the
2682 symbol name in the output, specify guru mode (-g) and add
2683 symbolname="yes" to the stap command.
2684
2685 # stap fileline-profile.stp -T 6 --all-modules --ldd
2686
2687
2688 profiling/fntimes.stp - Show Functions Taking Longer Than Usual
2689 keywords: _best profiling
2690
2691 The fntimes.stp script monitors the execution time history of a given
2692 function family (assumed non-recursive). Each time (beyond a warmup
2693 interval) is then compared to the historical maximum. If it exceeds
2694 a certain threshold (250%), a message is printed.
2695
2696 # stap fntimes.stp 'kernel.function("sys_*")' -T 7
2697
2698
2699 profiling/functioncallcount.stp - Count Times Functions Are Called
2700 keywords: profiling function
2701
2702 The functioncallcount.stp script takes one argument, a list of
2703 functions to probe. The script will run and count the number of times
2704 that each of the functions on the list is called. On exit the script
2705 will print a sorted list from most frequently to least frequently
2706 called function.
2707
2708 # stap -w functioncallcount.stp "*@fs/*.c" -T 1
2709
2710
2711 profiling/ioctl_handler.stp - Monitor which executables use ioctl syscalls and what kernel code is handling the ioctl
2712 keywords: profiling
2713
2714 The ioctl systemcall is used to manipulate devices setting or special
2715 files. The way that ioctl syscalls are handled depend greatly on the
2716 device the special file is associated with. Using strace to monitor
2717 the open and ioctl syscalls may not give a good indication of what
2718 kernel code is actually handling the ioctl operations. The
2719 ioctl_handler.stp script is designed to provide more details. On exit
2720 the ioctl_handler.stp script provides a count of the ioctl syscalls
2721 for each executable run on the system. If there was some special
2722 device driver code used to handle to the ioctl, the output will have
2723 a tally of the times the function name and module was called for that
2724 executable. The "--all-modules" option should be included on the
2725 command line so the script can provide function name information.
2726
2727 # stap ioctl_handler.stp --all-modules -T 1
2728
2729
2730 profiling/latencytap.stp - Show Reasons and Durations for Processes Sleeping
2731 keywords: _best profiling
2732
2733 The latencytap.stp script collects data on the intervals processes
2734 are deactivated (sleeping). The script categorizes the reasons for
2735 the sleeps by analyzing the backtraces and displays a sorted list of
2736 the top 20 causes from largest total sum time sleeping to smallest.
2737 The output is updated every 30 seconds. The script needs to be
2738 compiled with the '--all-modules' option to produce reasons for
2739 sleeps caused by modules. Optionally, this script can be used with
2740 the '-c' or '-x' options to focus on a specific PID.
2741
2742 # stap latencytap.stp --all-modules -T 1
2743
2744
2745 profiling/linetimes.stp - Show Time Spent on Each Line of a Function
2746 keywords: profiling _best
2747
2748 The linetimes.stp script takes two arguments: where to find the
2749 function and the function name. linetimes.stp will instrument each
2750 line in the function. It will print out the number of times that the
2751 function is called, a table with the average and maximum time each
2752 line takes, and control flow information when the script exits.
2753
2754 # stap linetimes.stp kernel do_nanosleep -T 1
2755
2756
2757 profiling/perf.stp - Show performance ratios using perf.counter to access performance counters
2758 keywords: profiling
2759
2760 On exit the perf.stp script provides a sorted list showing cycles per
2761 insn, branches per insn, and cache refs per insn
2762
2763 # stap -w perf.stp -c "find /usr/bin -name \"l*\" -printf \"%h/%f %s %Cx \
2764 %Ck%CM %Y\n\""
2765
2766
2767 profiling/periodic.stp - Show the Period of the Various Timers on the System
2768 keywords: profiling
2769
2770 The periodic.stp script uses the kernel.trace("timer_expire_entry")
2771 tracepoint to collect data on period and frequency of the various
2772 timers on the system. The script displays a sorted list of the
2773 timers observed on the system from most frequent to least frequent.
2774 The script needs to be compiled with the '--all-modules' option to
2775 produce list the function names. Optionally, this script can be used
2776 with a numerical argument to indicate the interval in seconds between
2777 printing output.
2778
2779 # stap periodic.stp --all-modules -T 1
2780
2781
2782 profiling/pf2.stp - Profile Kernel Functions
2783 keywords: profiling
2784
2785 The pf2.stp script sets up time-based sampling. Every five seconds it
2786 prints out a sorted list with the top ten kernel functions with
2787 samples.
2788
2789 # stap pf2.stp -T 1
2790
2791
2792 profiling/pf3.stp - Profile Kernel/User Functions
2793 keywords: profiling
2794
2795 The pf3.stp script sets up time-based sampling. Every five seconds it
2796 prints out a sorted list with the top twenty kernel and/or user
2797 functions with samples. Use any of --ldd, --all-modules, -d MODULE,
2798 -d /PATH/TO/EXEC to add more symbolic info.
2799
2800 # stap pf3.stp -T 6 --all-modules --ldd
2801
2802
2803 profiling/pf4.stp - Profile Kernel/User Backtraces
2804 keywords: _best profiling backtrace
2805
2806 The pf4.stp script sets up time-based sampling. Every five seconds it
2807 prints out a sorted list with the top twenty kernel and/or user stack
2808 backtraces (on a per-cpu basis). Use any of --ldd, --all-modules, -d
2809 MODULE, -d /PATH/TO/EXEC to add more symbolic info.
2810
2811 # stap pf4.stp -T 6 --all-modules --ldd
2812
2813
2814 profiling/sched_switch.stp - Display the Task Switches Happening in the Scheduler
2815 keywords: profiling function
2816
2817 The sched_switch.stp script takes two arguments, first argument can
2818 be "pid" or "name" to indicate what is being passed as second
2819 argument. The script will trace the process based on pid/name and
2820 print the scheduler switches happening with the process. If no
2821 arguments are passed, it displays all the scheduler switches. This
2822 can be used to understand which tasks schedule out the current
2823 process being traced, and when it gets scheduled in again.
2824
2825 # stap sched_switch.stp -T 1
2826
2827
2828 profiling/thread-times.stp - Profile Kernel Functions
2829 keywords: _best profiling
2830
2831 The thread-times.stp script sets up time-based sampling. Every five
2832 seconds it prints out a sorted list with the top twenty threads
2833 occupying the CPUs, broken down as a percentage of user and kernel
2834 time.
2835
2836 # stap thread-times.stp -T 1
2837
2838
2839 profiling/timeout.stp - Show Processes Doing Polling Operations
2840 keywords: profiling
2841
2842 The timeout.stp script is based on a blog entry
2843 (http://udrepper.livejournal.com/19041.html) mentioning a need for a
2844 tool to help developers find applications that are polling. The
2845 timeout.stp script monitors systemcall used for polling and records
2846 the systemcalls that timed out rather than returned because some
2847 action occurred. The script updates the screen once a second with the
2848 top twenty processes.
2849
2850 # stap timeout.stp -T 1
2851
2852
2853 profiling/topsys.stp - Show Processes Doing Polling Operations
2854 keywords: profiling
2855
2856 The topsys.stp script lists out the top twenty systemcalls for the
2857 previous 5 seconds. The output is sorted from most frequent to least
2858 frequent.
2859
2860 # stap topsys.stp -T 1
2861
2862
2863 profiling/ucalls.stp - Profile method invocations in Java, Perl, Php, Python, Ruby, and Tcl scripts
2864 keywords: profiling syscall
2865
2866 The ucalls.stp script is modeled after the BCC ucalls script
2867 (https://github.com/iovisor/bcc/blob/master/tools/lib/ucalls.py) by
2868 Sasha Goldshtein. The ucalls.stp script monitors the process
2869 indicated by the -x or -c option. When the scripts exits it prints
2870 out information about the number of times that each method is invoked
2871 for code written in Java, Perl, Php, Python, Ruby, and Tcl. Include
2872 the word "syscalls" on the command line to count syscalls invoked by
2873 the process. If you want latency information, include "latency" on
2874 the command line. Note that the latency option does not work for
2875 recursive functions
2876
2877 # stap ucalls.stp syscalls latency -c "python -c 'print(\"hello world\")' \
2878 || true"
2879
2880
2881 = PROMETHEUS =
2882
2883 general/also_ran.stp - Keep a tally of executables run on the system
2884 keywords: prometheus
2885
2886 The also_ran.stp script tallies each time a executable is started or
2887 a shared library is loaded for execution. This information can be
2888 useful to determine what software is actually being used on the
2889 system. The script makes the information available via procfs in
2890 Prometheus readable format.
2891
2892 # stap also_ran.stp -T 1
2893
2894
2895 general/cpu_throttle.stp - Monitor Intel processors for throttling due to power or thermal limits
2896 keywords: prometheus limits
2897
2898 Intel processors have hardware mechanisms that will reduce the
2899 effective processors speed to avoid exceeding set thermal or power
2900 constraints. The cpu_throttle.stp script monitors when CPU throttling
2901 occurs and accumulates the amount of time in milliseconds that each
2902 processor is throttled. The script makes the information available
2903 via procfs in Prometheus readable format.
2904
2905 # stap cpu_throttle.stp -T 1
2906
2907
2908 profiling/syscallerrorsbypid.stp - Provide a per-process syscall error tally on the system
2909 keywords: prometheus process syscall tracepoint
2910
2911 The syscallerrorsbypid.stp script tallies syscall errors for each
2912 running process. This information can be useful to whether there are
2913 excessive errors for various processes on the system. The script
2914 makes the information available via procfs in Prometheus readable
2915 format. When a process exits its data will be eliminated from the
2916 prometheus output. To avoid exceeding the storage limitations of
2917 SystemTap older entries maybe overwritten by newer entries. This can
2918 lead to some active process syscall error counts disappearing and/or
2919 later reappearing with a lower value. Also note that the script does
2920 not properly name syscalls for 32-bit applications running on 64-bit
2921 machines.
2922
2923 # stap syscallerrorsbypid.stp -T 1
2924
2925
2926 profiling/syscalllatency.stp - Provide a per-process accumulation of syscall latency on the system
2927 keywords: prometheus process syscall tracepoint
2928
2929 The syscalllatency.stp script accumulates syscall latency for each
2930 running process. This information can be useful to whether excessive
2931 time is being spent in particular syscalls on the system. The script
2932 makes the information available via procfs in Prometheus readable
2933 format. When a process exits its data will be eliminated from the
2934 prometheus output. To avoid exceeding the storage limitations of
2935 SystemTap older entries maybe overwritten by newer entries. This can
2936 lead to some active process syscall error counts disappearing and/or
2937 later reappearing with a lower value. Also note that the script does
2938 not properly name syscalls for 32-bit applications running on 64-bit
2939 machines.
2940
2941 # stap syscalllatency.stp -T 1
2942
2943
2944 profiling/syscallsbypid.stp - Provide a per-process syscall tally on the system
2945 keywords: prometheus process syscall tracepoint
2946
2947 The syscallsbypid.stp script tallies each syscall for each running
2948 process. This information can be useful to determine the activity of
2949 various processes on the system. The script makes the information
2950 available via procfs in Prometheus readable format. When a process
2951 exits its data will be eliminated from the prometheus output. To
2952 avoid exceeding the storage limitations of SystemTap older entries
2953 maybe overwritten by newer entries. This can lead to some active
2954 process syscall counts disappearing and/or later reappearing with a
2955 lower value. Also note that the script does not properly name
2956 syscalls for 32-bit applications running on 64-bit machines.
2957
2958 # stap syscallsbypid.stp -T 1
2959
2960
2961 profiling/syscallsrw.stp - Provide a tally of read and write syscalls run on the system
2962 keywords: prometheus syscall bpf
2963
2964 The syscallsrw.stp script tallies the read and write syscalls. This
2965 is a demo script for stapbpf prometheus-exporter scripts.
2966
2967
2968 = QEMU =
2969
2970 virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events
2971 keywords: virtualization qemu kvm
2972
2973 The qemu_count.stp script tallies the number of times each of the
2974 user-space qemu probepoints is encountered. When the script exits, it
2975 prints a list of the number of times each user-space qemu probepoint
2976 is encountered.
2977
2978 # stap qemu_count.stp -T 1
2979
2980
2981 virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port
2982 keywords: virtualization qemu kvm io
2983
2984 The qemu_io.stp script tallies the number of times each of the IO
2985 port on the guest virtual machines is touched by a input or output
2986 operation. When the script exits, it prints a count of the number of
2987 times each IO port read and written.
2988
2989 # stap qemu_io.stp -T 1
2990
2991
2992 = REGEX =
2993
2994 general/regex.stp - Report opened files whose names match a given regex
2995 keywords: regex
2996
2997 Uses the regex functionality to detect opened files whose names match
2998 a pattern given on the command line. If no command line parameter is
2999 given, demonstrate by filtering for files that end with an extension
3000 showing them to be an archive.
3001
3002 # stap regex.stp -T 1
3003
3004
3005 = RETRANSMISSION =
3006
3007 network/tcp_retransmission.stp - print tcp retransmission packet
3008 keywords: tcp retransmission
3009
3010 The tcp_retransmission.stp prints out a line for each tcp
3011 retransmission packet
3012
3013 # stap tcp_retransmission.stp -T 1
3014
3015
3016 = SCHEDULER =
3017
3018 process/chng_cpu.stp - Monitor Changes in Processor Executing a Task
3019 keywords: scheduler
3020
3021 The chng_cpu.stp script takes an argument which is the executable
3022 name of the task it should monitor. Each time a task with that
3023 executable name is found running on a different processor, the script
3024 prints out the thread id (tid), the executable name, the processor
3025 now running the task, the thread state, and a backtrace showing the
3026 kernel functions that triggered the running of the task on the
3027 processor.
3028
3029 # stap chng_cpu.stp -T 1 bash
3030
3031
3032 process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task
3033 keywords: _best process scheduler time tracepoint interrupt
3034
3035 The cycle_thief.stp script instruments the scheduler and IRQ handler
3036 to determine which processes and interrupts are competing with the
3037 specified task for the cpu cycles. This script uses the '-c' or '-x'
3038 options to focus on a specific task. The script output the number of
3039 times the task migrates between processors, histograms showing the
3040 length of time on and off processor, lists of processes running while
3041 the task is off the processor, and the interrupts that occurred while
3042 the task was running.
3043
3044 # stap cycle_thief.stp -T 1
3045
3046
3047 process/forktracker.stp - Trace Creation of Processes
3048 keywords: process scheduler
3049
3050 The forktracker.stp script prints out a time-stamped entry showing
3051 each fork and exec operation on the machine. This can be useful to
3052 determine what process is creating a flurry of short-lived processes.
3053
3054 # stap forktracker.stp -T 1
3055
3056
3057 process/migrate.stp - Track the Migration of Specific Executables
3058 keywords: scheduler
3059
3060 The migrate.stp script takes an argument which is the executable name
3061 of the task it should monitor. Each time a task with that executable
3062 name migrates between processors an entry is printed with the process
3063 id (pid), the executable name, the processor off loading the task,
3064 and the process taking the task. Note that the task may or may not be
3065 executing at the time of the migration.
3066
3067 # stap migrate.stp -T 1 bash
3068
3069
3070 process/proctop.stp - Periodically Print Process Information With History
3071 keywords: process scheduler _best
3072
3073 Every 5 seconds, print out a list of 25 processes that took the most
3074 system time with information about the processes. Includes
3075 information on processes that may have exited while the script was
3076 running. The script contains configuration options listed in the
3077 script source.
3078
3079 # stap proctop.stp -T 1
3080
3081
3082 process/sched-latency.stp - track wakeup-to-dispatch latency stats
3083 keywords: process scheduler time tracepoint
3084
3085 This script periodically reports a histogram of the latency between a
3086 task (thread) being woken up and it actually being dispatched to a
3087 CPU: the amount of time it's spent in the runnable queue.
3088
3089 # stap sched-latency.stp -T 7
3090
3091
3092 process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints
3093 keywords: process scheduler time tracepoint
3094
3095 The schedtimes.stp script instruments the scheduler to track the
3096 amount of time that each process spends in running, sleeping,
3097 queuing, and waiting for io. On exit the script prints out the
3098 accumulated time for each state of processes observed. Optionally,
3099 this script can be used with the '-c' or '-x' options to focus on a
3100 specific PID and its children.
3101
3102 # stap schedtimes.stp -T 1
3103
3104
3105 process/sleepingBeauties.stp - Generate Backtraces of Threads Waiting for IO Operations
3106 keywords: io scheduler backtrace
3107
3108 The script monitors the time that threads spend in waiting for IO
3109 operations (in "D" state) in the wait_for_completion function. If a
3110 thread spends over 10ms, its name and backtrace is printed, and later
3111 so is the total delay.
3112
3113 # stap sleepingBeauties.stp -T 1
3114
3115
3116 process/spawn_seeker.stp - Track Creation of Processes by process and execname
3117 keywords: process scheduler
3118
3119 The spawn_seeker.stp script every minute (and on exit) prints out the
3120 local time and sorted lists of which processes and executables
3121 spawned tasks during the previous minute. This can be useful to
3122 determine what process is creating a flurry of short-lived processes.
3123 When a process exits its count of tasks created is added to its
3124 parent's count to better account for the indirect task creation by
3125 children processes. For more detailed examination of task creation
3126 consider using forktracker.stp.
3127
3128 # stap spawn_seeker.stp -T 1
3129
3130
3131 = SCSI =
3132
3133 io/iostat-scsi.stp - IO Statistics for SCSI Devices
3134 keywords: io profiling scsi
3135
3136 The iostat-scsi.stp script provides a breakdown of the number of blks
3137 read and written on the machine's various SCSI devices. The script
3138 takes one argument which is the number of seconds between reports.
3139
3140 # stap -g iostat-scsi.stp 1 -T 1
3141
3142
3143 = SECURITY =
3144
3145 process/auditbt.stp - Generate backtraces for kernel audit events
3146 keywords: monitoring security backtrace
3147
3148 Attaches to the kernel audit-log paths (also used by libaudit), and
3149 log every record being sent, along with a user-space backtrace of the
3150 process that caused it.
3151
3152 # stap auditbt.stp -d /usr/bin/sudo --ldd -c "sudo true"
3153
3154
3155 process/noptrace.stp - Disable ptrace from Hierarchies of Processes
3156 keywords: process security guru
3157
3158 Blocks ptrace(2) attempts from processes identified by stap -c/-x, as
3159 also specifiable from /proc/systemtap/stap_XXX/ control files.
3160 Processes may be added or removed from the blocked list.
3161
3162 # stap -g noptrace.stp -c 'strace ls || true'
3163
3164
3165 security-band-aids/cve-2008-0600.stp - cve-2008-0600 security band-aid
3166 keywords: security guru
3167
3168 historical emergency security band-aid, for reference/education only
3169
3170
3171 security-band-aids/cve-2011-4127.stp - cve-2011-4127 security band-aid
3172 keywords: security guru
3173
3174 historical emergency security band-aid, for reference/education only
3175
3176
3177 security-band-aids/cve-2012-0056.stp - cve-2012-0056 security band-aid
3178 keywords: security guru
3179
3180 historical emergency security band-aid, for reference/education only
3181
3182
3183 security-band-aids/cve-2013-2094.stp - cve-2013-2094 security band-aid
3184 keywords: security guru
3185
3186 historical emergency security band-aid, for reference/education only
3187
3188
3189 security-band-aids/cve-2014-7169.stp - cve-2014-7169 security band-aid
3190 keywords: security guru
3191
3192 historical emergency security band-aid, for reference/education only
3193
3194
3195 security-band-aids/cve-2015-0235.stp - cve-2015-0235 security band-aid
3196 keywords: security guru
3197
3198 historical emergency security band-aid, for reference/education only
3199
3200
3201 security-band-aids/cve-2015-3456.stp - cve-2015-3456 security band-aid
3202 keywords: security guru
3203
3204 historical emergency security band-aid, for reference/education only
3205
3206
3207 security-band-aids/cve-2015-7547.stp - cve-2015-7547 security band-aid
3208 keywords: security guru
3209
3210 historical emergency security band-aid, for reference/education only
3211
3212
3213 security-band-aids/cve-2016-0728.stp - cve-2016-0728 security band-aid
3214 keywords: security guru
3215
3216 historical emergency security band-aid, for reference/education only
3217
3218
3219 security-band-aids/cve-2016-5195.stp - cve-2016-5195 security band-aid
3220 keywords: security guru
3221
3222 historical emergency security band-aid, for reference/education only
3223
3224
3225 security-band-aids/cve-2017-6074.stp - cve-2017-6074 security band-aid
3226 keywords: security guru
3227
3228 historical emergency security band-aid, for reference/education only
3229
3230
3231 security-band-aids/cve-2018-1000001.stp - cve-2018-1000001 security band-aid
3232 keywords: security guru
3233
3234 historical emergency security band-aid, for reference/education only
3235
3236
3237 security-band-aids/cve-2018-14634.stp - cve-2018-14634 security band-aid
3238 keywords: security guru
3239
3240 historical emergency security band-aid, for reference/education only
3241
3242
3243 security-band-aids/cve-2018-6485.stp - cve-2018-6485 security band-aid
3244 keywords: security guru
3245
3246 historical emergency security band-aid, for reference/education only
3247
3248
3249 security-band-aids/cve-2021-4034.stp - cve-2021-4034 security band-aid
3250 keywords: security guru
3251
3252 historical emergency security band-aid, for reference/education only
3253
3254
3255 security-band-aids/cve-2021-4155.stp - cve-2021-4155 security band-aid
3256 keywords: security guru
3257
3258 historical emergency security band-aid, for reference/education only
3259
3260
3261 security-band-aids/cve-2023-4911.stp - cve-2023-4911 security band-aid
3262 keywords: security guru
3263
3264 historical emergency security band-aid, for reference/education only
3265
3266
3267 = SIGNALS =
3268
3269 process/psig.stp - Print Process File Descriptors
3270 keywords: process signals
3271
3272 Run psig.stp to produce a human-readable summary of the signal
3273 handling configuration of a given process. Specify the process-id as
3274 -x PID for fastest performance.
3275
3276 # stap -DMAXACTION=10000 -g psig.stp -x $$
3277
3278
3279 process/sig_by_pid.stp - Signal Counts by Process ID
3280 keywords: signals
3281
3282 Print signal counts by process ID in descending order.
3283
3284 # stap sig_by_pid.stp -T 1
3285
3286
3287 process/sig_by_proc.stp - Signal Counts by Process Name
3288 keywords: signals
3289
3290 Print signal counts by process name in descending order.
3291
3292 # stap sig_by_proc.stp -T 1
3293
3294
3295 process/sigkill.stp - Track SIGKILL Signals
3296 keywords: signals
3297
3298 The script traces any SIGKILL signals. When that SIGKILL signal is
3299 sent to a process, the script prints out the signal name, the
3300 destination executable and process ID, the executable name and user
3301 ID that sents the signal.
3302
3303 # stap sigkill.stp -T 1
3304
3305
3306 process/sigmon.stp - Track a Particular Signal to a Specific Process
3307 keywords: signals
3308
3309 The script watches for a particular signal sent to a specific
3310 process. When that signal is sent to the specified process, the
3311 script prints out the PID and executable of the process sending the
3312 signal, the PID and executable name of the process receiving the
3313 signal, and the signal number and name.
3314
3315 # stap sigmon.stp -T 1 SIGKILL
3316
3317
3318 = SIMPLE =
3319
3320 general/callgraph.stp - Callgraph Tracing
3321 keywords: simple trace callgraph
3322
3323 Print a timed per-thread microsecond-timed nested callgraph. The
3324 first parameter names the function probe points to trace.
3325
3326 # stap callgraph.stp 'kernel.function("*@fs/proc*.c")' -c "cat \
3327 /proc/sys/vm/* || true"
3328
3329
3330 general/floatingpoint.stp - SystemTap "Floating Point" Program
3331 keywords: simple floatingpoint
3332
3333 A basic "Floating Point" program implemented in SystemTap script. It
3334 extracts floating point from benchmark-sdt section in main.cxx and
3335 takes one pseudo floating point, printing out results of various
3336 simple floating point operations.
3337
3338 # stap floatingpoint.stp -c "stap --benchmark-sdt"
3339
3340
3341 general/helloworld.stp - SystemTap "Hello World" Program
3342 keywords: _best simple bpf
3343
3344 A basic "Hello World" program implemented in SystemTap script. It
3345 prints out "hello world" message and then immediately exits.
3346
3347 # stap helloworld.stp
3348
3349
3350 general/key.stp - make keyboard noises
3351 keywords: simple
3352
3353 For fans of Leroy Anderson and Typewriters only, this script arranges
3354 to play a click or a zing for various keystrokes.
3355
3356 # stap key.stp -c 'sleep 5'
3357
3358
3359 general/keyhack.stp - Hack the keyboard
3360 keywords: simple guru
3361
3362 This script makes it appear that one's keyboard is broken, by
3363 changing keycodes at the kernel device driver level. Annoy your
3364 friends!
3365
3366 # stap -g keyhack.stp -c 'sleep 5'
3367
3368
3369 general/pyexample.stp - SystemTap python 2 support tapset
3370 keywords: simple
3371
3372 A python support tapset that displays backtraces and variable values
3373
3374 # PYTHONPATH=$PYEXECDIR stap -g --suppress-time-limits -c \
3375 '/usr/bin/python3 -m HelperSDT pyexample.py 35' pyexample.stp python2
3376
3377
3378 general/pyexample.stp - SystemTap python 3 support tapset
3379 keywords: simple
3380
3381 A python support tapset that displays backtraces and variable values
3382
3383 # PYTHONPATH=$PY3EXECDIR stap -g --suppress-time-limits -c \
3384 '/usr/bin/python3 -m HelperSDT pyexample.py 35' pyexample.stp python3
3385
3386
3387 io/eatmydata.stp - disable fsync
3388 keywords: io guru simple
3389
3390 Suppresses fsync() syscalls from processes identified by stap -c/-x
3391 by turning them into presumed-faster fsync() on some dummy or other
3392 file descriptor
3393
3394 # stap -g eatmydata.stp -c 'strace ls || true'
3395
3396
3397 io/mbrwatch.stp - Monitor Read/Write of the Boot Sector Area of Block Devices
3398 keywords: io monitoring disk simple
3399
3400 The mbrwatch.stp script reports any attempted reads/writes of the
3401 first few sectors of a raw block device.
3402
3403 # stap mbrwatch.stp -c "dd of=/dev/null count=1 if=/dev/`grep -v major \
3404 /proc/partitions | grep . | grep -v 'sr[0-9]' | awk '{print $4}' | \
3405 head -1`"
3406
3407
3408 io/slowvfs.stp - Trace slow vfs opens.
3409 keywords: io simple
3410
3411 This script prints a line for every kernel vfs_open operation that
3412 takes longer than a configurable number of microseconds. Highly
3413 contended or remote filesystems are likelier to hit this.
3414
3415 # stap slowvfs.stp -G sloth=10 -c 'find /proc >/dev/null || true'
3416
3417
3418 network/who_sent_it.stp - Trace threads sending network traffic to given host and/or port
3419 keywords: network trace traffic simple
3420
3421 This script traces outgoing network packets using the netfilter
3422 probes (not requiring debuginfo), printing the source thread name/id
3423 and destination host:port. It may be filtered with the_dport and
3424 the_daddr globals, e.g., to watch only for DNS traffic (port 53),
3425 and/or only to the localhost (127.0.0.1).
3426
3427 # stap who_sent_it.stp -G the_dport=53 -c "ping -c 1 sourceware.org || \
3428 true"
3429
3430
3431 = SOCKET =
3432
3433 general/socket-events.stp - Trace Socket Activity of a Process
3434 keywords: socket syscall
3435
3436 Prints the life cycle of all sockets associated with a process. This
3437 includes bytes and timing. The timing information that is tracked
3438 includes event completion relative to the start of said event and the
3439 end of the previous event. Currently tracks read, write, recv, send,
3440 connect and close.
3441
3442 # stap socket-events.stp
3443
3444
3445 lwtools/accept2close-nd.stp - Show socket lifespan, from accept() to close() (non-debuginfo)
3446 keywords: network socket
3447
3448 This traces socket duration from the accept() syscall to close(), and
3449 provides details on the lifespan of these passive connections,
3450 showing the distribution as a histogram.
3451
3452 # stap accept2close-nd.stp -T 1
3453
3454
3455 network/connect_stat.stp - Show Process Ancestry for IP Connections
3456 keywords: network socket process
3457
3458 The connect_stat.stp script prints a task's entire ancestry (parent
3459 process name/uid/gid) whenever it attempts an outgoing socket
3460 connection to a given IP address.
3461
3462 # stap connect_stat.stp 127.0.0.1 -T 1
3463
3464
3465 network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel
3466 keywords: network tracepoint socket
3467
3468 Every five seconds the dropwatch.stp script lists the number of
3469 socket buffers freed at locations in the kernel.
3470
3471 # stap -d kernel dropwatch.stp -T 1
3472
3473
3474 network/socket-trace.stp - Trace Functions Called in Network Socket Code
3475 keywords: network socket
3476
3477 The script instruments each of the functions in the Linux kernel's
3478 net/socket.c file. The script prints out trace data. The first
3479 element of a line is time delta in microseconds from the previous
3480 entry. This is followed by the command name and the PID. The "->" and
3481 "<-" indicates function entry and function exit, respectively. The
3482 last element of the line is the function name.
3483
3484 # stap socket-trace.stp -T 1
3485
3486
3487 network/socktop - Periodically Summarize Socket Activity on the System
3488 keywords: network socket
3489
3490 The socktop script periodically prints out a list of the processes
3491 with the highest socket activity. Command line options for the
3492 script allow filtering to focus on particular types of sockets. The
3493 "-h" option lists socktop script's filtering options.
3494
3495 # ./socktop -c 1
3496
3497
3498 network/tcp_connections.stp - Track Creation of Incoming TCP Connections
3499 keywords: network tcp socket
3500
3501 The tcp_connections.stp script prints information for each new
3502 incoming TCP connection accepted by the computer. The information
3503 includes the UID, the command accepting the connection, the PID of
3504 the command, the port the connection is on, and the IP address of the
3505 originator of the request.
3506
3507 # stap tcp_connections.stp -T 1
3508
3509
3510 network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10
3511 keywords: network tcp socket guru
3512
3513 Run the tcp_init_cwnd.stp script in the background to override a
3514 kernel's default tcp cwnd value to 10, which has been found to
3515 improve latency for web server type workloads. The script prints a
3516 count of cwnd value changes when it is stopped.
3517
3518 # stap -g tcp_init_cwnd.stp -T 1
3519
3520
3521 = SPECULATION =
3522
3523 general/whythefail.stp - Why did the function fail?
3524 keywords: speculation monitoring function trace _best
3525
3526 The whythefail.stp script prints a statement-execution trace for a
3527 given function, but only for those runs of the function that ended up
3528 with a (configurable) post-return condition.
3529
3530 # stap whythefail.stp kernel do_sys_open '$return < 0' -c 'cat \
3531 /root/no-such-file || true'
3532
3533
3534 = STAPGAMES =
3535
3536 stapgames/2048.stp - 2048
3537 keywords: _best stapgames
3538
3539 The modern classic 2048 sliding-tiles game, using local keyboard and
3540 ansi animation.
3541
3542 # stap -p4 2048.stp
3543
3544
3545 stapgames/block.stp - block breaker game
3546 keywords: stapgames
3547
3548 A block game where you progressively break the ceiling blocks until
3549 clearing the level
3550
3551 # stap -p4 -Itapset/ block.stp
3552
3553
3554 stapgames/eater.stp - eater game
3555 keywords: stapgames
3556
3557 walk through a maze, eat stuff
3558
3559 # stap -p4 -Itapset/ eater.stp
3560
3561
3562 stapgames/lifegame.stp - life game
3563 keywords: stapgames
3564
3565 watch as your creation morphes into different forms
3566
3567 # stap -p4 -Itapset/ lifegame.stp
3568
3569
3570 stapgames/pingpong.stp - pingpong game
3571 keywords: stapgames
3572
3573 A simulated ball bounces around the terminal reflecting at the edges
3574
3575 # stap -p4 -Itapset/ pingpong.stp
3576
3577
3578 = STATISTICS =
3579
3580 general/alias_suffixes.stp - Count I/O Syscalls using Alias Suffixes
3581 keywords: io statistics
3582
3583 alias_suffixes.stp is a demonstration of how alias suffixes in the
3584 systemtap language might be used. The script tracks the wall clock
3585 time for each invocation of the system calls open, close, read, and
3586 write. When the script exists it prints out the minimum, average, and
3587 maximum times in microseconds for each system call, followed by a
3588 count of times that each syscall was invoked and a histogram showing
3589 the distributions of times.
3590
3591 # stap alias_suffixes.stp -T 1
3592
3593
3594 general/eventcount.stp - Interactive Count Specified Events
3595 keywords: interactive statistics thread process
3596
3597 The script periodically prints a count of specified events and their
3598 related tid's over the course of execution. Numerous configuration
3599 options exist to control filtering / reporting, some of which can be
3600 modified at runtime. See the script source for more information.
3601
3602 # stap eventcount.stp 'syscall_any' -c 'sleep 3'
3603
3604
3605 general/func_time_stats.stp - Function Time Statistics
3606 keywords: function statistics
3607
3608 The func_time_stats.stp script tracks the wall clock time for each
3609 invocation of the function probe listed as the first command line
3610 argument. When the script exits it prints out the minimum, average,
3611 and maximum times in microseconds followed by a count of times that
3612 the function was called and a histogram showing the distributions of
3613 times.
3614
3615 # stap func_time_stats.stp 'syscall.read' -T 1
3616
3617
3618 general/sizeof.stp - Print the Size of a C Type
3619 keywords: statistics memory bpf
3620
3621 This script prints the size of a type, based on dwarf debuginfo for
3622 any kernel or userspace module, or trial-compilation of a given
3623 header file name.
3624
3625 # stap sizeof.stp FILE '</usr/include/stdio.h>'
3626
3627
3628 general/sizeof_interactive.stp - Interactive Print the Size of a C Type
3629 keywords: statistics memory interactive
3630
3631 This script prints the size of a type, based on dwarf debuginfo for
3632 any kernel or userspace module, or trial-compilation of a given
3633 header file name. Types and corresponding locations are provided to
3634 the script at runtime via keyboard input. The format of the input is
3635 identical to that of sizeof.stp (see script source for more
3636 information). Types and locations can be repeatedly given until the
3637 process is terminated.
3638
3639 # echo "FILE '</usr/include/stdio.h>'" | stap sizeof_interactive.stp -T 1
3640
3641
3642 memory/vm.tracepoints.stp - Collect Slab Allocation Statistics
3643 keywords: memory statistics
3644
3645 The script will probe all memory slab/slub allocations and collects
3646 information about the size of the object (bytes requested) and
3647 user-space process in execution. When run over a period of time, it
3648 helps to correlate kernel-space memory consumption owing to
3649 user-space processes.
3650
3651 # stap vm.tracepoints.stp -T 10
3652
3653
3654 network/net_xmit_json.stp - Tracks time between packet queue and transmit.
3655 keywords: network statistics json
3656
3657 This script tracks time between packet queue and transmit. The
3658 information is provided to userspace via procfs in JSON format.
3659
3660 # stap net_xmit_json.stp -T 1
3661
3662
3663 network/nfsd-recent.stp - Keep track of NFS server statistics
3664 keywords: nfs statistics
3665
3666 This script tracks all nfsd server operations by client_ip address,
3667 and periodically lists those clients that have made recent requests.
3668 It's a way of finding out which nfs clients might be considered still
3669 connected.
3670
3671 # stap nfsd-recent.stp -T 1
3672
3673
3674 network/nfsdtop.stp - Keep track of NFS server statistics
3675 keywords: nfs statistics
3676
3677 The nfsdtop.stp script gathers and displays NFS lookups,
3678
3679 # stap nfsdtop.stp -T 1
3680
3681
3682 network/tcpipstat.stp - Display Network Statistics for Individual TCP Sockets
3683 keywords: network statistics
3684
3685 The tcpipstat script collects and displays network statistics related
3686 to individual TCP sockets or groups of sockets. The statistics that
3687 are collected are simular to that of the command netstat -s, only
3688 sorted and grouped by individual sockets.
3689
3690 # stap tcpipstat.stp timeout=1
3691
3692
3693 = SWAP =
3694
3695 memory/swapmonitor.stp - Summarize swap operations
3696 keywords: memory swap
3697
3698 The swapmonitor.stp script is an example to monitor "swap in/out" in
3699 RHEL. This stap monitor the call of "swap_readpage" and
3700 "swap_writepage", and then printf the timeframe, func, execname, pid
3701 and the count with 10 seconds interval.
3702
3703 # stap swapmonitor.stp -T 5
3704
3705
3706 = SYSCALL =
3707
3708 general/socket-events.stp - Trace Socket Activity of a Process
3709 keywords: socket syscall
3710
3711 Prints the life cycle of all sockets associated with a process. This
3712 includes bytes and timing. The timing information that is tracked
3713 includes event completion relative to the start of said event and the
3714 end of the previous event. Currently tracks read, write, recv, send,
3715 connect and close.
3716
3717 # stap socket-events.stp
3718
3719
3720 io/iotime.stp - Trace Time Spent in Read and Write for Files
3721 keywords: profiling syscall io file
3722
3723 The script watches each open, close, read, and write syscalls on the
3724 system. For each file the scripts observes opened it accumulates the
3725 amount of wall clock time spent in read and write operations and the
3726 number of bytes read and written. When a file is closed the script
3727 prints out a pair of lines for the file. Both lines begin with a
3728 timestamp in microseconds, the PID number, and the executable name in
3729 parentheses. The first line with the "access" keyword lists the file
3730 name, the attempted number of bytes for the read and write
3731 operations. The second line with the "iotime" keyword list the file
3732 name and the number of microseconds accumulated in the read and write
3733 syscalls.
3734
3735 # stap iotime.stp -T 1
3736
3737
3738 process/errsnoop.stp - Tabulate System Call Errors
3739 keywords: process syscall
3740
3741 Prints a periodic tabular report about failing system calls, by
3742 process and by syscall failure. The first optional argument
3743 specifies the reporting interval (in seconds, default 5); the second
3744 optional argument gives a screen height (number of lines in the
3745 report, default 20).
3746
3747 # stap errsnoop.stp 1 10 -T 1
3748
3749
3750 process/futexes.stp - System-Wide Futex Contention
3751 keywords: syscall locking futex
3752
3753 The script watches the futex syscall on the system. On exit the
3754 futex's address, the number of contentions, and the average time for
3755 each contention on the futex are printed from lowest pid number to
3756 highest.
3757
3758 # stap futexes.stp -T 1
3759
3760
3761 process/futexes2.stp - System-Wide Shared Futex Contention
3762 keywords: syscall locking futex
3763
3764 The script watches just shared futex syscalls on the system. On exit
3765 the futex's key, the number of contentions, and the average time for
3766 each contention on the futex are printed from lowest pid number to
3767 highest.
3768
3769 # stap futexes2.stp -T 1
3770
3771
3772 process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading
3773 keywords: process monitoring syscall tracepoint
3774
3775 The procmod_watcher.stp script monitors calls to fork(), exec(),
3776 exit(), init_module(), and delete_module(). Event-specific details
3777 are also printed out (e.g. for exec(), the file being exec'ed). This
3778 script does not require debuginfo.
3779
3780 # stap procmod_watcher.stp -T 1
3781
3782
3783 process/sleeptime.stp - Trace Time Spent in Nanosleep Syscalls
3784 keywords: syscall nanosleep
3785
3786 The script watches each nanosleep syscall on the system. At the end
3787 of each nanosleep syscall the script prints out a line with a
3788 timestamp in microseconds, the pid, the executable name in
3789 parentheses, the "nanosleep:" key, and the duration of the sleep in
3790 microseconds.
3791
3792 # stap sleeptime.stp -T 1
3793
3794
3795 process/strace.stp - Trace system calls
3796 keywords: _best process syscall
3797
3798 The script loosely emulates strace, when applied to individual
3799 processes or hierarchies (via -c/-x), or the entire system (without
3800 -c/-x). A few output configuration parameters may be set with -G.
3801
3802 # stap strace.stp -c "cat /proc/sys/vm/* || true"
3803
3804
3805 process/syscalls_by_pid.stp - System-Wide Count of Syscalls by PID
3806 keywords: syscall
3807
3808 The script watches all syscall on the system. On exit the script
3809 prints a list showing the number of systemcalls executed by each PID
3810 ordered from greatest to least number of syscalls.
3811
3812 # stap syscalls_by_pid.stp -T 1
3813
3814
3815 process/syscalls_by_proc.stp - System-Wide Count of Syscalls by Executable
3816 keywords: syscall
3817
3818 The script watches all syscall on the system. On exit the script
3819 prints a list showing the number of systemcalls executed by each
3820 executable ordered from greatest to least number of syscalls.
3821
3822 # stap syscalls_by_proc.stp -T 1
3823
3824
3825 process/syscalltimes - System-Wide Syscall Statistics with Filtering
3826 keywords: syscall
3827
3828 Combination shell/systemtap script to measure system call counts and
3829 times. Can be filtered by process IDs, process names and users.
3830
3831 # ./syscalltimes -c 'sleep 1'
3832
3833
3834 process/thread-business.stp - monitor syscall history
3835 keywords: _best process syscall
3836
3837 Prints a periodic tabular report about the counts of syscall activity
3838 of all threads on the system, along with a textual
3839 recent-syscall-history for each
3840
3841 # stap thread-business.stp -T 10
3842
3843
3844 process/wait4time.stp - Trace Time Spent in wait4 Syscalls
3845 keywords: syscall process
3846
3847 The script watches each wait4 syscall on the system. At the end of
3848 each wait4 syscall the script prints out a line with a timestamp in
3849 microseconds, the pid, the executable name in parentheses, the
3850 "wait4:" key, the duration of the wait and the PID that the wait4 was
3851 waiting for. If the waited for PID is not specified , it is "-1".
3852
3853 # stap wait4time.stp -T 1
3854
3855
3856 profiling/container_check.stp - Monitor capabilities and syscalls used by a process and it children
3857 keywords: profiling container syscall
3858
3859 The container_check.stp script monitors the use of linux capablities
3860 and optionally forbidden syscalls by a process and its children. On
3861 exit the script prints out lists showing the capabilies used by each
3862 executable, which syscall used specific capabilites for each
3863 executable, a list of forbidden syscalls used, and details on any
3864 syscalls that failed during monitoring. This script is designed to
3865 help diagnose issues caused by restricted capabilies and syscalls
3866 when running an application in a container. If the script warns
3867 about skipped probes, the number of active kretprobes may need to be
3868 increased with "-DKRETACTIVE=100" option on the command line
3869
3870 # stap container_check.stp -c "ping -c 1 sourceware.org || true"
3871
3872
3873 profiling/syscallerrorsbypid.stp - Provide a per-process syscall error tally on the system
3874 keywords: prometheus process syscall tracepoint
3875
3876 The syscallerrorsbypid.stp script tallies syscall errors for each
3877 running process. This information can be useful to whether there are
3878 excessive errors for various processes on the system. The script
3879 makes the information available via procfs in Prometheus readable
3880 format. When a process exits its data will be eliminated from the
3881 prometheus output. To avoid exceeding the storage limitations of
3882 SystemTap older entries maybe overwritten by newer entries. This can
3883 lead to some active process syscall error counts disappearing and/or
3884 later reappearing with a lower value. Also note that the script does
3885 not properly name syscalls for 32-bit applications running on 64-bit
3886 machines.
3887
3888 # stap syscallerrorsbypid.stp -T 1
3889
3890
3891 profiling/syscalllatency.stp - Provide a per-process accumulation of syscall latency on the system
3892 keywords: prometheus process syscall tracepoint
3893
3894 The syscalllatency.stp script accumulates syscall latency for each
3895 running process. This information can be useful to whether excessive
3896 time is being spent in particular syscalls on the system. The script
3897 makes the information available via procfs in Prometheus readable
3898 format. When a process exits its data will be eliminated from the
3899 prometheus output. To avoid exceeding the storage limitations of
3900 SystemTap older entries maybe overwritten by newer entries. This can
3901 lead to some active process syscall error counts disappearing and/or
3902 later reappearing with a lower value. Also note that the script does
3903 not properly name syscalls for 32-bit applications running on 64-bit
3904 machines.
3905
3906 # stap syscalllatency.stp -T 1
3907
3908
3909 profiling/syscallsbypid.stp - Provide a per-process syscall tally on the system
3910 keywords: prometheus process syscall tracepoint
3911
3912 The syscallsbypid.stp script tallies each syscall for each running
3913 process. This information can be useful to determine the activity of
3914 various processes on the system. The script makes the information
3915 available via procfs in Prometheus readable format. When a process
3916 exits its data will be eliminated from the prometheus output. To
3917 avoid exceeding the storage limitations of SystemTap older entries
3918 maybe overwritten by newer entries. This can lead to some active
3919 process syscall counts disappearing and/or later reappearing with a
3920 lower value. Also note that the script does not properly name
3921 syscalls for 32-bit applications running on 64-bit machines.
3922
3923 # stap syscallsbypid.stp -T 1
3924
3925
3926 profiling/syscallsrw.stp - Provide a tally of read and write syscalls run on the system
3927 keywords: prometheus syscall bpf
3928
3929 The syscallsrw.stp script tallies the read and write syscalls. This
3930 is a demo script for stapbpf prometheus-exporter scripts.
3931
3932
3933 profiling/ucalls.stp - Profile method invocations in Java, Perl, Php, Python, Ruby, and Tcl scripts
3934 keywords: profiling syscall
3935
3936 The ucalls.stp script is modeled after the BCC ucalls script
3937 (https://github.com/iovisor/bcc/blob/master/tools/lib/ucalls.py) by
3938 Sasha Goldshtein. The ucalls.stp script monitors the process
3939 indicated by the -x or -c option. When the scripts exits it prints
3940 out information about the number of times that each method is invoked
3941 for code written in Java, Perl, Php, Python, Ruby, and Tcl. Include
3942 the word "syscalls" on the command line to count syscalls invoked by
3943 the process. If you want latency information, include "latency" on
3944 the command line. Note that the latency option does not work for
3945 recursive functions
3946
3947 # stap ucalls.stp syscalls latency -c "python -c 'print(\"hello world\")' \
3948 || true"
3949
3950
3951 = TCP =
3952
3953 network/sk_stream_wait_memory.stp - Track Start and Stop of Processes Due to Network Buffer Space
3954 keywords: network tcp process bpf
3955
3956 The sk_stream-wait_memory.stp prints a time stamp, executable, and
3957 pid each time a process blocks due to the send buffer being full. A
3958 similar entry is printed each time a process continues because there
3959 is room in the buffer.
3960
3961 # stap sk_stream_wait_memory.stp -T 1
3962
3963
3964 network/tcp_connections.stp - Track Creation of Incoming TCP Connections
3965 keywords: network tcp socket
3966
3967 The tcp_connections.stp script prints information for each new
3968 incoming TCP connection accepted by the computer. The information
3969 includes the UID, the command accepting the connection, the PID of
3970 the command, the port the connection is on, and the IP address of the
3971 originator of the request.
3972
3973 # stap tcp_connections.stp -T 1
3974
3975
3976 network/tcp_init_cwnd.stp - Increase Initial TCP Congestion Window to 10
3977 keywords: network tcp socket guru
3978
3979 Run the tcp_init_cwnd.stp script in the background to override a
3980 kernel's default tcp cwnd value to 10, which has been found to
3981 improve latency for web server type workloads. The script prints a
3982 count of cwnd value changes when it is stopped.
3983
3984 # stap -g tcp_init_cwnd.stp -T 1
3985
3986
3987 network/tcp_retransmission.stp - print tcp retransmission packet
3988 keywords: tcp retransmission
3989
3990 The tcp_retransmission.stp prints out a line for each tcp
3991 retransmission packet
3992
3993 # stap tcp_retransmission.stp -T 1
3994
3995
3996 = THREAD =
3997
3998 general/eventcount.stp - Interactive Count Specified Events
3999 keywords: interactive statistics thread process
4000
4001 The script periodically prints a count of specified events and their
4002 related tid's over the course of execution. Numerous configuration
4003 options exist to control filtering / reporting, some of which can be
4004 modified at runtime. See the script source for more information.
4005
4006 # stap eventcount.stp 'syscall_any' -c 'sleep 3'
4007
4008
4009 process/threadstacks.stp - Override default new-pthread stack sizes
4010 keywords: thread guru
4011
4012 Overrides default NPTL pthread_create stack size for all new threads
4013 created by target processes. Reports one line per process when the
4014 related glibc variable __default_stacksize is updated. Moot for
4015 glibc versions that support $LIBC_PTHREAD_DEFAULT_STACKSIZE_NP.
4016
4017 # stap -g threadstacks.stp -Gsize=65536 -T 1 -d `which stap`
4018
4019
4020 = TIME =
4021
4022 general/stopwatches.stp - See the amount of wall clock time a process spends in various states
4023 keywords: time bpf
4024
4025 The stopwatch.stp script illustrates how to use multiple stopwatches
4026 record how much wallclock time a process spends in kernel- and
4027 user-space. On exit the script prints out the time in seconds,
4028 milliseconds, microseconds, and nanoseconds. Note that this output of
4029 this script is not directly comparable to the time command because
4030 time records the time that the process is actually active in kernel-
4031 and user-space.
4032
4033 # stap stopwatches.stp -T 1
4034
4035
4036 process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task
4037 keywords: _best process scheduler time tracepoint interrupt
4038
4039 The cycle_thief.stp script instruments the scheduler and IRQ handler
4040 to determine which processes and interrupts are competing with the
4041 specified task for the cpu cycles. This script uses the '-c' or '-x'
4042 options to focus on a specific task. The script output the number of
4043 times the task migrates between processors, histograms showing the
4044 length of time on and off processor, lists of processes running while
4045 the task is off the processor, and the interrupts that occurred while
4046 the task was running.
4047
4048 # stap cycle_thief.stp -T 1
4049
4050
4051 process/sched-latency.stp - track wakeup-to-dispatch latency stats
4052 keywords: process scheduler time tracepoint
4053
4054 This script periodically reports a histogram of the latency between a
4055 task (thread) being woken up and it actually being dispatched to a
4056 CPU: the amount of time it's spent in the runnable queue.
4057
4058 # stap sched-latency.stp -T 7
4059
4060
4061 process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints
4062 keywords: process scheduler time tracepoint
4063
4064 The schedtimes.stp script instruments the scheduler to track the
4065 amount of time that each process spends in running, sleeping,
4066 queuing, and waiting for io. On exit the script prints out the
4067 accumulated time for each state of processes observed. Optionally,
4068 this script can be used with the '-c' or '-x' options to focus on a
4069 specific PID and its children.
4070
4071 # stap schedtimes.stp -T 1
4072
4073
4074 = TRACE =
4075
4076 apps/gmalloc_watch.stp - Tracing glib2 memory allocations
4077 keywords: trace
4078
4079 The gmalloc_watch.stp script from Colin Walters' blog
4080 (https://blog.verbum.org/2011/03/19/analyzing-memory-use-with-systemt
4081 ap/) traces the allocation of glib2 memory using the markers in
4082 glib2.
4083
4084 # stap gmalloc_watch.stp -T 1
4085
4086
4087 apps/libguestfs_log.stp - Trace libguestfs startup
4088 keywords: application trace bpf
4089
4090 The libguestfs_log.stp script prints a log of when various
4091 libgueststartup steps are encountered. The first column is
4092 microseconds since the script started. The second column is the time
4093 elapsed in microseconds between the previous and current events and
4094 the third column is the event name.
4095
4096 # stap libguestfs_log.stp -T 1
4097
4098
4099 apps/php-trace.stp - Tracing of PHP code execution
4100 keywords: trace dyninst
4101
4102 Trace of executing PHP code using the enabled markers.
4103
4104 # stap php-trace.stp -c 'php -f hello.php'
4105
4106
4107 apps/stap_time.stp - Provide elapsed times for Passes of SystemTap script compilation
4108 keywords: application trace
4109
4110 SystemTap has multiple passes to convert the text of a SystemTap
4111 script into instrumentation that actually collect data on the system.
4112 The stap_time.stp script uses the markers in SystemTap to note the
4113 amount of time that each of the passes requires. When SystemTap
4114 completes pass 4 (compiling the instrumentation into a kernel module)
4115 the script print out the script name followed by the amount of time
4116 in milliseconds required for Pass 0 (command line option parsing),
4117 Pass 1 (script parsing), Pass 2 (elaboration), Pass 3 (code
4118 generation), and Pass 4 (module compilation). The resulting data can
4119 be analyzed to determine if there are issues with the amount of time
4120 that systemtap takes to generate instrumentation.
4121
4122 # stap stap_time.stp -T 1
4123
4124
4125 general/callgraph.stp - Callgraph Tracing
4126 keywords: simple trace callgraph
4127
4128 Print a timed per-thread microsecond-timed nested callgraph. The
4129 first parameter names the function probe points to trace.
4130
4131 # stap callgraph.stp 'kernel.function("*@fs/proc*.c")' -c "cat \
4132 /proc/sys/vm/* || true"
4133
4134
4135 general/para-callgraph-verbose.stp - Callgraph Tracing with Verbose Arguments
4136 keywords: trace callgraph
4137
4138 Print a timed per-thread microsecond-timed callgraph, complete with
4139 pretty-printed function parameters and return values. The first
4140 parameter names the function probe points to trace. The optional
4141 second parameter names the probe points for trigger functions, which
4142 acts to enable tracing for only those functions that occur while the
4143 current thread is nested within the trigger.
4144
4145 # stap para-callgraph-verbose.stp 'kernel.function("*@fs/proc*.c")' \
4146 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true"
4147
4148
4149 general/para-callgraph.stp - Callgraph Tracing with Arguments
4150 keywords: _best trace callgraph
4151
4152 Print a timed per-thread microsecond-timed callgraph, complete with
4153 function parameters and return values. The first parameter names the
4154 function probe points to trace. The optional second parameter names
4155 the probe points for trigger functions, which acts to enable tracing
4156 for only those functions that occur while the current thread is
4157 nested within the trigger.
4158
4159 # stap para-callgraph.stp 'kernel.function("*@fs/proc*.c")' \
4160 'kernel.function("vfs_read")' -c "cat /proc/sys/vm/* || true"
4161
4162
4163 general/whythefail.stp - Why did the function fail?
4164 keywords: speculation monitoring function trace _best
4165
4166 The whythefail.stp script prints a statement-execution trace for a
4167 given function, but only for those runs of the function that ended up
4168 with a (configurable) post-return condition.
4169
4170 # stap whythefail.stp kernel do_sys_open '$return < 0' -c 'cat \
4171 /root/no-such-file || true'
4172
4173
4174 network/nfsd-trace.stp - Trace NFSD requests with file names.
4175 keywords: nfs trace
4176
4177 This script traces all nfsd server operations by client_ip address,
4178 operation, and complete file name (if possible).
4179
4180 # stap nfsd-trace.stp -T 1
4181
4182
4183 network/tcp_trace.stp - TCP Connection Tracing Utility
4184 keywords: network trace
4185
4186 This scripts traces a given TCP connection based on the filter
4187 parameters given by the user. The indexing is done by the 4 tuples
4188 local address, remote address, local port, remote port.
4189
4190 # stap tcp_trace.stp 127.0.0.1:*-127.0.0.1:* timeout=1
4191
4192
4193 network/who_sent_it.stp - Trace threads sending network traffic to given host and/or port
4194 keywords: network trace traffic simple
4195
4196 This script traces outgoing network packets using the netfilter
4197 probes (not requiring debuginfo), printing the source thread name/id
4198 and destination host:port. It may be filtered with the_dport and
4199 the_daddr globals, e.g., to watch only for DNS traffic (port 53),
4200 and/or only to the localhost (127.0.0.1).
4201
4202 # stap who_sent_it.stp -G the_dport=53 -c "ping -c 1 sourceware.org || \
4203 true"
4204
4205
4206 = TRACEPOINT =
4207
4208 network/dropwatch.stp - Watch Where Socket Buffers Are Freed in the Kernel
4209 keywords: network tracepoint socket
4210
4211 Every five seconds the dropwatch.stp script lists the number of
4212 socket buffers freed at locations in the kernel.
4213
4214 # stap -d kernel dropwatch.stp -T 1
4215
4216
4217 process/cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task
4218 keywords: _best process scheduler time tracepoint interrupt
4219
4220 The cycle_thief.stp script instruments the scheduler and IRQ handler
4221 to determine which processes and interrupts are competing with the
4222 specified task for the cpu cycles. This script uses the '-c' or '-x'
4223 options to focus on a specific task. The script output the number of
4224 times the task migrates between processors, histograms showing the
4225 length of time on and off processor, lists of processes running while
4226 the task is off the processor, and the interrupts that occurred while
4227 the task was running.
4228
4229 # stap cycle_thief.stp -T 1
4230
4231
4232 process/procmod_watcher.stp - Monitor process creation/termination and module [un]loading
4233 keywords: process monitoring syscall tracepoint
4234
4235 The procmod_watcher.stp script monitors calls to fork(), exec(),
4236 exit(), init_module(), and delete_module(). Event-specific details
4237 are also printed out (e.g. for exec(), the file being exec'ed). This
4238 script does not require debuginfo.
4239
4240 # stap procmod_watcher.stp -T 1
4241
4242
4243 process/sched-latency.stp - track wakeup-to-dispatch latency stats
4244 keywords: process scheduler time tracepoint
4245
4246 This script periodically reports a histogram of the latency between a
4247 task (thread) being woken up and it actually being dispatched to a
4248 CPU: the amount of time it's spent in the runnable queue.
4249
4250 # stap sched-latency.stp -T 7
4251
4252
4253 process/schedtimes.stp - Track Time Processes Spend in Various States Using Tracepoints
4254 keywords: process scheduler time tracepoint
4255
4256 The schedtimes.stp script instruments the scheduler to track the
4257 amount of time that each process spends in running, sleeping,
4258 queuing, and waiting for io. On exit the script prints out the
4259 accumulated time for each state of processes observed. Optionally,
4260 this script can be used with the '-c' or '-x' options to focus on a
4261 specific PID and its children.
4262
4263 # stap schedtimes.stp -T 1
4264
4265
4266 profiling/syscallerrorsbypid.stp - Provide a per-process syscall error tally on the system
4267 keywords: prometheus process syscall tracepoint
4268
4269 The syscallerrorsbypid.stp script tallies syscall errors for each
4270 running process. This information can be useful to whether there are
4271 excessive errors for various processes on the system. The script
4272 makes the information available via procfs in Prometheus readable
4273 format. When a process exits its data will be eliminated from the
4274 prometheus output. To avoid exceeding the storage limitations of
4275 SystemTap older entries maybe overwritten by newer entries. This can
4276 lead to some active process syscall error counts disappearing and/or
4277 later reappearing with a lower value. Also note that the script does
4278 not properly name syscalls for 32-bit applications running on 64-bit
4279 machines.
4280
4281 # stap syscallerrorsbypid.stp -T 1
4282
4283
4284 profiling/syscalllatency.stp - Provide a per-process accumulation of syscall latency on the system
4285 keywords: prometheus process syscall tracepoint
4286
4287 The syscalllatency.stp script accumulates syscall latency for each
4288 running process. This information can be useful to whether excessive
4289 time is being spent in particular syscalls on the system. The script
4290 makes the information available via procfs in Prometheus readable
4291 format. When a process exits its data will be eliminated from the
4292 prometheus output. To avoid exceeding the storage limitations of
4293 SystemTap older entries maybe overwritten by newer entries. This can
4294 lead to some active process syscall error counts disappearing and/or
4295 later reappearing with a lower value. Also note that the script does
4296 not properly name syscalls for 32-bit applications running on 64-bit
4297 machines.
4298
4299 # stap syscalllatency.stp -T 1
4300
4301
4302 profiling/syscallsbypid.stp - Provide a per-process syscall tally on the system
4303 keywords: prometheus process syscall tracepoint
4304
4305 The syscallsbypid.stp script tallies each syscall for each running
4306 process. This information can be useful to determine the activity of
4307 various processes on the system. The script makes the information
4308 available via procfs in Prometheus readable format. When a process
4309 exits its data will be eliminated from the prometheus output. To
4310 avoid exceeding the storage limitations of SystemTap older entries
4311 maybe overwritten by newer entries. This can lead to some active
4312 process syscall counts disappearing and/or later reappearing with a
4313 lower value. Also note that the script does not properly name
4314 syscalls for 32-bit applications running on 64-bit machines.
4315
4316 # stap syscallsbypid.stp -T 1
4317
4318
4319 = TRAFFIC =
4320
4321 network/netdev.stp - Trace Activity on Network Devices
4322 keywords: network traffic
4323
4324 The netdev.stp script traces configuration and transmit/receive
4325 activity on network devices.
4326
4327 # stap netdev.stp -T 1
4328
4329
4330 network/netfilter_summary.stp - System-Wide Count of Network Packets by IPs
4331 keywords: _best network traffic
4332
4333 The script watches all IPv4 network traffic on the system. On exit
4334 the script prints a list showing the number of packets sent along
4335 source IP address / destination IP address pair encountered, and the
4336 total number of bytes sent among the pair. The list is ordered from
4337 greatest to least number of packets seen among the source/destination
4338 pairs.
4339
4340 # stap netfilter_summary.stp -T 1
4341
4342
4343 network/netfilter_summary_json.stp - System-Wide Count of Network Packets by IPs
4344 keywords: _best network traffic json
4345
4346 The script watches all IPv4 network traffic on the system. The data
4347 is output in JSON format and includes the number of packets sent
4348 along source IP address / destination IP address pair encountered,
4349 and the total number of bytes sent among the pair.
4350
4351 # stap netfilter_summary_json.stp -T 1
4352
4353
4354 network/nettop.stp - Periodic Listing of Processes Using Network Interfaces
4355 keywords: network traffic
4356
4357 Every five seconds the nettop.stp script prints out a list of
4358 processed (PID and command) with the number of packets sent/received
4359 and the amount of data sent/received by the process during that
4360 interval.
4361
4362 # stap nettop.stp -T 1
4363
4364
4365 network/packet_contents.stp - Network packet contents
4366 keywords: network traffic
4367
4368 The packet_contents.stp script displays the length of each network
4369 packet and its contents in both hexadecimal and ASCII. Systemtap
4370 strings are MAXSTRINGLEN in length by default which may not be enough
4371 for larger packets. In order to print larger packets, this limit can
4372 be increased by passing in the "-DMAXSTRINGLEN=65536" command line
4373 option.
4374
4375 # stap packet_contents.stp -T 1
4376
4377
4378 network/stp_dump.stp - Dump of STP packets
4379 keywords: network traffic
4380
4381 The stp_dump.stp prints out the packet contents. Each block contains
4382 the STP protocol ID, version ID, flags, root and bridge MAC
4383 addresses, and various times.
4384
4385 # stap stp_dump.stp -T 5
4386
4387
4388 network/tcpdumplike.stp - Dump of Received UDP/TCP Packets
4389 keywords: network traffic
4390
4391 The tcpdumplike.stp prints out a line for each TCP & UDP packet
4392 received. Each line includes the source and destination IP addresses,
4393 the source and destination ports, and flags.
4394
4395 # stap tcpdumplike.stp -T 1
4396
4397
4398 network/who_sent_it.stp - Trace threads sending network traffic to given host and/or port
4399 keywords: network trace traffic simple
4400
4401 This script traces outgoing network packets using the netfilter
4402 probes (not requiring debuginfo), printing the source thread name/id
4403 and destination host:port. It may be filtered with the_dport and
4404 the_daddr globals, e.g., to watch only for DNS traffic (port 53),
4405 and/or only to the localhost (127.0.0.1).
4406
4407 # stap who_sent_it.stp -G the_dport=53 -c "ping -c 1 sourceware.org || \
4408 true"
4409
4410
4411 = TTY =
4412
4413 io/ttyspy.stp - Monitor TTY Typing
4414 keywords: io tty monitoring guru
4415
4416 The ttyspy.stp script uses tty_audit hooks to monitor recent typing
4417 activity on the system, printing a scrolling record of recent
4418 keystrokes, on a per-tty basis.
4419
4420 # stap --skip-badvars -g ttyspy.stp -T 1
4421
4422
4423 = VIRTUALIZATION =
4424
4425 virtualization/kvm_service_time.stp - Time Statistics on KVM Exit Reasons
4426 keywords: _best virtualization kvm
4427
4428 The kvm_service_time.stp script tracks the statistics about the
4429 amount of time that the processor left the guest virtual machine for
4430 each exit reason (for example fixing up a page table or handling an
4431 IO operation). When the script exits it prints out the number of
4432 times each exit reason was encountered, the total duration of time it
4433 left the guest VM, the minimum time, the average time, and the
4434 maximum time in microseconds for that exit reason. On Linux 2.6.38
4435 and newer kernel the script can automatically determine whether it is
4436 running on Intel or AMD processors. For older kernels with a
4437 kernel.trace("kvm_exit") tracepoint that does not have the $isa
4438 parameter you can explicitly state the kvm type with a "-G kvm=intel"
4439 or "-G kvm=amd" on the command line.
4440
4441 # stap kvm_service_time.stp -T 1
4442
4443
4444 virtualization/qemu_count.stp - Tally the Number of User-Space QEMU Events
4445 keywords: virtualization qemu kvm
4446
4447 The qemu_count.stp script tallies the number of times each of the
4448 user-space qemu probepoints is encountered. When the script exits, it
4449 prints a list of the number of times each user-space qemu probepoint
4450 is encountered.
4451
4452 # stap qemu_count.stp -T 1
4453
4454
4455 virtualization/qemu_io.stp - Tally the Number of User-Space QEMU IO on Each IO Port
4456 keywords: virtualization qemu kvm io
4457
4458 The qemu_io.stp script tallies the number of times each of the IO
4459 port on the guest virtual machines is touched by a input or output
4460 operation. When the script exits, it prints a count of the number of
4461 times each IO port read and written.
4462
4463 # stap qemu_io.stp -T 1
4464
4465
4466 = WATCHDOG =
4467
4468 general/watchdog.stp - Watchdog Timer for Arbitrary Events
4469 keywords: watchdog backtrace
4470
4471 The watchdog.stp script provides a watchdog timer mechanism for
4472 arbitrary events. The script takes three arguments: the events to
4473 start watchdog timer, the event to stop the watchdog timer, and the
4474 time in millseconds for the watchdog. If the watchdog timer is
4475 exceed, the script will trigger a stack backtrace of the user-process
4476 that timed out using pstack. This script can be used to diagnose what
4477 the userspace application is doing when a slower than expected
4478 operation occurs.
4479
4480 # stap watchdog.stp 'syscall.nanosleep' 'syscall.nanosleep.return' 1000 \
4481 -T 1
4482
4483
4484 = WATCHPOINT =
4485
4486 memory/hw_watch_addr.stp - Watch a Kernel Address Using Breakpoint Hardware
4487 keywords: memory watchpoint
4488
4489 The script will watch accesses to a single kernel address and prints
4490 a traceback each time the address is accessed. This script needs to
4491 be run as root to allow access to the breakpoint hardware.
4492
4493 # stap --all-modules hw_watch_addr.stp 0x`grep "vm_dirty_ratio" \
4494 /proc/kallsyms | awk '{print $1}'` -T 5
4495
4496
4497 memory/hw_watch_sym.stp - Watch a Kernel Symbol Using Breakpoint Hardware
4498 keywords: memory watchpoint
4499
4500 The script will watch accesses to the starting address of a single
4501 kernel symbol and prints a traceback each time the symbol is
4502 accessed. This script needs to be run as root to allow access to the
4503 breakpoint hardware.
4504
4505 # stap --all-modules hw_watch_sym.stp vm_dirty_ratio -T 5
4506
4507