| 1 |
This directory contains sets of scripts for testing plain kprobes or |
| 2 |
systemtap probes on a set of kernel functions. It tries to bisect the |
| 3 |
list if a probe on a set of kernel functions crashes the system. |
| 4 |
|
| 5 |
Here's a workflow: |
| 6 |
|
| 7 |
1) First put a list of functions to probe in a file, by default called |
| 8 |
'probes.all'. If you'd like a list of all kernel functions, you can |
| 9 |
run 'gen_code_all.sh'. |
| 10 |
|
| 11 |
2a) To test plain kprobes, run: |
| 12 |
|
| 13 |
# ./kprobes_test.py |
| 14 |
|
| 15 |
2b) To test systemtap probes, run: |
| 16 |
|
| 17 |
# ./stap_probes_test.py |
| 18 |
|
| 19 |
3) Wait for the script to finish. Feel free to watch kprobes_test.log |
| 20 |
for progress. If the system crashes, restart it. Either |
| 21 |
'kprobe_test.py' or 'stap_probes_test.py' should get automatically run |
| 22 |
when the system gets restarted. |
| 23 |
|
| 24 |
If you'd like help monitoring the target system, you can use |
| 25 |
'monitor_system.py', which will try to automatically reboot a remote |
| 26 |
system if the system can't be reached. |
| 27 |
|
| 28 |
4) When the script (either ./kprobes_test.py or ./stap_probes_test.py) |
| 29 |
finishes, you'll see the following output files. |
| 30 |
|
| 31 |
- probes.unregistered is a file that contains the list of kernel |
| 32 |
functions where a probe couldn't be registered. |
| 33 |
- probes.untriggered is a file that contains the list of kernel |
| 34 |
functions where a probe could be registered, but the kernel |
| 35 |
function wasn't actually called. So, we don't really know whether |
| 36 |
these functions passed or failed. |
| 37 |
- probes.passed is a file that contains the list of kernel functions |
| 38 |
where a probe was registered and the kerenel function was called |
| 39 |
with no crash. |
| 40 |
- probes.failed is a file that contains the list of kernel functions |
| 41 |
where a probe was registered and the kernel function was called |
| 42 |
that caused a crash. |
| 43 |
|
| 44 |
|
| 45 |
Here are the original instructions for 'gen_passlist', which tested |
| 46 |
systemtap probe safety. |
| 47 |
|
| 48 |
==== |
| 49 |
|
| 50 |
To start a kprobes test run: |
| 51 |
runtest passlist.exp |
| 52 |
|
| 53 |
---- |
| 54 |
Here is an implementation of generating passlist for safe-mode probes |
| 55 |
based on the discussion in former thread |
| 56 |
(http://sourceware.org/ml/systemtap/2006-q3/msg00574.html). |
| 57 |
|
| 58 |
Its main idea is: |
| 59 |
1) Fetch a group of probe points from probes.pending, probe them and |
| 60 |
run some workloads(e.g. runltp) parallely meanwhile. |
| 61 |
|
| 62 |
2) If the probe test ends without crash, those actually triggered |
| 63 |
probe points are moved into probes.passed and those untriggered |
| 64 |
are into probes.untriggered; |
| 65 |
If the probe test crashes the system, it will be resumed |
| 66 |
automatically after system reboot. Those probe points which have |
| 67 |
been triggered are also moved into probes.passed, but those |
| 68 |
untriggered ones are moved into probes.failed. |
| 69 |
|
| 70 |
3) Repeat the above until probes.pending becomes empty, then: |
| 71 |
Normally, probes.pending is reinitialized from probes.failed and |
| 72 |
probes.untriggered, then start the next iteration; |
| 73 |
But if max running level (now 3) is reached, or probes.pending, |
| 74 |
probes.failed and probes.untriggered are all empty, stop the |
| 75 |
whole test. |
| 76 |
|
| 77 |
To be able to resume after a crash, this test will append its startup |
| 78 |
codes to the end of /etc/rc.d/rc.local at the beginning and remove |
| 79 |
them at the end of the test. |
| 80 |
|
| 81 |
The gen_tapset_all.sh is used to generate a "probes.all" file based |
| 82 |
on current tapset of systemtap. |
| 83 |
|
| 84 |
It is suggested to use a script in a remote server to restart the |
| 85 |
test machine automatically in case it is crashed by the probe test. |
| 86 |
|
| 87 |
Instructions: |
| 88 |
0) Please jump to 6) directly if you use all default settings |
| 89 |
1) Define your list of probe points in a file named "probes.all"; |
| 90 |
otherwise, it will be automatically extracted from |
| 91 |
stap -p2 -e 'probe kernel.function("*"){} |
| 92 |
2) Define your own workloads by changing the "benchs" list variable; |
| 93 |
otherwise, several LTP testcases will be used |
| 94 |
3) Define how may times you want the test to be iterated by changing |
| 95 |
MAX_RUNNING_LEVEL variable; otherwise, 3 is the default |
| 96 |
4) Define the group size for different iteration level by changing |
| 97 |
proper_current_size() function; otherwise, they will be set in |
| 98 |
a decreasing order based on the length of probes.all |
| 99 |
5) Remove /stp_genpasslist_running if you want to start a test from |
| 100 |
the beginning after a system crash |
| 101 |
6) Start a test: |
| 102 |
runtest passlist.exp |