| 1 |
Examples of biolatency-nd.stp, the Linux SystemTap version. |
| 2 |
|
| 3 |
|
| 4 |
Measuring block I/O (storage I/O, ie, disk I/O) latency until Ctrl-C: |
| 5 |
|
| 6 |
# ./biolatency-nd.stp |
| 7 |
Tracing block I/O... Hit Ctrl-C to end. |
| 8 |
^C |
| 9 |
bio latency (ns): |
| 10 |
value |-------------------------------------------------- count |
| 11 |
131072 | 0 |
| 12 |
262144 | 0 |
| 13 |
524288 |@@ 5 |
| 14 |
1048576 |@@ 4 |
| 15 |
2097152 |@@@@@@@@@@@@@@@@@@@@@@@@ 49 |
| 16 |
4194304 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 55 |
| 17 |
8388608 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 82 |
| 18 |
16777216 |@@@@@@@@@@@@@@@@@@ 37 |
| 19 |
33554432 | 0 |
| 20 |
67108864 | 0 |
| 21 |
|
| 22 |
This shows a distribution between aout 1 and 16 milliseconds, which is expected |
| 23 |
for current rotational disk I/O. |
| 24 |
|
| 25 |
|
| 26 |
A more interesting distribution: |
| 27 |
|
| 28 |
# ./biolatency-nd.stp |
| 29 |
Tracing block I/O... Hit Ctrl-C to end. |
| 30 |
^C |
| 31 |
bio latency (ns): |
| 32 |
value |-------------------------------------------------- count |
| 33 |
32768 | 0 |
| 34 |
65536 | 0 |
| 35 |
131072 | 10 |
| 36 |
262144 |@@@@@@@@@ 450 |
| 37 |
524288 |@@@ 159 |
| 38 |
1048576 | 26 |
| 39 |
2097152 | 35 |
| 40 |
4194304 | 18 |
| 41 |
8388608 |@@ 112 |
| 42 |
16777216 |@@@@ 195 |
| 43 |
33554432 |@@@@@@@@@ 452 |
| 44 |
67108864 |@@@@@@@@@@@@@@@@@@@@@@ 1065 |
| 45 |
134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 2398 |
| 46 |
268435456 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 2008 |
| 47 |
536870912 | 0 |
| 48 |
1073741824 | 0 |
| 49 |
|
| 50 |
This shows a bimodal distribution with a low-latency mode around 0.26 |
| 51 |
milliseconds, which may be on-disk cache hits, and then a high-latency mode |
| 52 |
between around 16 and 268 milliseconds. Such high latency is likely |
| 53 |
caused by queueing, which can be investigated further using custom |
| 54 |
SystemTap invocations as well as the "avgqu-sz" column from iostat(1). |
| 55 |
|
| 56 |
|
| 57 |
biolatency-nd.stp accepts an optional interval and count as arguments. Measuring |
| 58 |
block I/O latency, and printing a report every second, five times: |
| 59 |
|
| 60 |
# ./biolatency-nd.stp 1 5 |
| 61 |
Tracing block I/O... Output every 1 secs. |
| 62 |
|
| 63 |
bio latency (ns): |
| 64 |
value |-------------------------------------------------- count |
| 65 |
32768 | 0 |
| 66 |
65536 | 0 |
| 67 |
131072 |@@ 28 |
| 68 |
262144 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 486 |
| 69 |
524288 |@@@@@@@@@@@@@@@@@@@ 198 |
| 70 |
1048576 |@@@@@@@@@ 98 |
| 71 |
2097152 |@@@@@@ 61 |
| 72 |
4194304 |@@@ 35 |
| 73 |
8388608 |@ 17 |
| 74 |
16777216 | 3 |
| 75 |
33554432 | 0 |
| 76 |
67108864 | 0 |
| 77 |
|
| 78 |
|
| 79 |
bio latency (ns): |
| 80 |
value |-------------------------------------------------- count |
| 81 |
32768 | 0 |
| 82 |
65536 | 0 |
| 83 |
131072 |@@ 22 |
| 84 |
262144 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 505 |
| 85 |
524288 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 370 |
| 86 |
1048576 |@@@@@@@@ 91 |
| 87 |
2097152 |@@ 31 |
| 88 |
4194304 |@ 20 |
| 89 |
8388608 |@ 20 |
| 90 |
16777216 | 4 |
| 91 |
33554432 | 0 |
| 92 |
67108864 | 0 |
| 93 |
|
| 94 |
|
| 95 |
bio latency (ns): |
| 96 |
value |-------------------------------------------------- count |
| 97 |
32768 | 0 |
| 98 |
65536 | 0 |
| 99 |
131072 |@@@@ 70 |
| 100 |
262144 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 826 |
| 101 |
524288 |@@@@@@@@@@@@@@@@@@@ 333 |
| 102 |
1048576 |@@@ 52 |
| 103 |
2097152 | 11 |
| 104 |
4194304 | 7 |
| 105 |
8388608 |@ 17 |
| 106 |
16777216 | 5 |
| 107 |
33554432 | 0 |
| 108 |
67108864 | 0 |
| 109 |
|
| 110 |
|
| 111 |
bio latency (ns): |
| 112 |
value |-------------------------------------------------- count |
| 113 |
32768 | 0 |
| 114 |
65536 | 0 |
| 115 |
131072 |@@ 35 |
| 116 |
262144 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 559 |
| 117 |
524288 |@@@@@@@@@@@@@@@@@@@@@@ 270 |
| 118 |
1048576 |@@@@@@ 82 |
| 119 |
2097152 |@@ 34 |
| 120 |
4194304 |@ 17 |
| 121 |
8388608 |@ 19 |
| 122 |
16777216 | 9 |
| 123 |
33554432 | 2 |
| 124 |
67108864 | 0 |
| 125 |
134217728 | 0 |
| 126 |
|
| 127 |
|
| 128 |
bio latency (ns): |
| 129 |
value |-------------------------------------------------- count |
| 130 |
32768 | 0 |
| 131 |
65536 | 0 |
| 132 |
131072 |@@@ 57 |
| 133 |
262144 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 877 |
| 134 |
524288 |@@@@@@@@@@@@@@@ 270 |
| 135 |
1048576 |@@@@ 85 |
| 136 |
2097152 |@@ 46 |
| 137 |
4194304 | 7 |
| 138 |
8388608 | 15 |
| 139 |
16777216 | 10 |
| 140 |
33554432 | 0 |
| 141 |
67108864 | 0 |
| 142 |
|
| 143 |
These disk I/O were much faster, with a mode around 0.25 ms. |