| 1 |
================================================================ |
| 2 |
Documentation for Kdump - The kexec-based Crash Dumping Solution |
| 3 |
================================================================ |
| 4 |
|
| 5 |
This document includes overview, setup, installation, and analysis |
| 6 |
information. |
| 7 |
|
| 8 |
Overview |
| 9 |
======== |
| 10 |
|
| 11 |
Kdump uses kexec to quickly boot to a dump-capture kernel whenever a |
| 12 |
dump of the system kernel's memory needs to be taken (for example, when |
| 13 |
the system panics). The system kernel's memory image is preserved across |
| 14 |
the reboot and is accessible to the dump-capture kernel. |
| 15 |
|
| 16 |
You can use common commands, such as cp, scp or makedumpfile to copy |
| 17 |
the memory image to a dump file on the local disk, or across the network |
| 18 |
to a remote system. |
| 19 |
|
| 20 |
Kdump and kexec are currently supported on the x86, x86_64, ppc64, |
| 21 |
s390x, arm and arm64 architectures. |
| 22 |
|
| 23 |
When the system kernel boots, it reserves a small section of memory for |
| 24 |
the dump-capture kernel. This ensures that ongoing Direct Memory Access |
| 25 |
(DMA) from the system kernel does not corrupt the dump-capture kernel. |
| 26 |
The kexec -p command loads the dump-capture kernel into this reserved |
| 27 |
memory. |
| 28 |
|
| 29 |
On x86 machines, the first 640 KB of physical memory is needed for boot, |
| 30 |
regardless of where the kernel loads. For simpler handling, the whole |
| 31 |
low 1M is reserved to avoid any later kernel or device driver writing |
| 32 |
data into this area. Like this, the low 1M can be reused as system RAM |
| 33 |
by kdump kernel without extra handling. |
| 34 |
|
| 35 |
On PPC64 machines first 32KB of physical memory is needed for booting |
| 36 |
regardless of where the kernel is loaded and to support 64K page size |
| 37 |
kexec backs up the first 64KB memory. |
| 38 |
|
| 39 |
For s390x, when kdump is triggered, the crashkernel region is exchanged |
| 40 |
with the region [0, crashkernel region size] and then the kdump kernel |
| 41 |
runs in [0, crashkernel region size]. Therefore no relocatable kernel is |
| 42 |
needed for s390x. |
| 43 |
|
| 44 |
All of the necessary information about the system kernel's core image is |
| 45 |
encoded in the ELF format, and stored in a reserved area of memory |
| 46 |
before a crash. The physical address of the start of the ELF header is |
| 47 |
passed to the dump-capture kernel through the elfcorehdr= boot |
| 48 |
parameter. Optionally the size of the ELF header can also be passed |
| 49 |
when using the elfcorehdr=[size[KMG]@]offset[KMG] syntax. |
| 50 |
|
| 51 |
With the dump-capture kernel, you can access the memory image through |
| 52 |
/proc/vmcore. This exports the dump as an ELF-format file that you can |
| 53 |
write out using file copy commands such as cp or scp. You can also use |
| 54 |
makedumpfile utility to analyze and write out filtered contents with |
| 55 |
options, e.g with '-d 31' it will only write out kernel data. Further, |
| 56 |
you can use analysis tools such as the GNU Debugger (GDB) and the Crash |
| 57 |
tool to debug the dump file. This method ensures that the dump pages are |
| 58 |
correctly ordered. |
| 59 |
|
| 60 |
Setup and Installation |
| 61 |
====================== |
| 62 |
|
| 63 |
Install kexec-tools |
| 64 |
------------------- |
| 65 |
|
| 66 |
1) Login as the root user. |
| 67 |
|
| 68 |
2) Download the kexec-tools user-space package from the following URL: |
| 69 |
|
| 70 |
http://kernel.org/pub/linux/utils/kernel/kexec/kexec-tools.tar.gz |
| 71 |
|
| 72 |
This is a symlink to the latest version. |
| 73 |
|
| 74 |
The latest kexec-tools git tree is available at: |
| 75 |
|
| 76 |
- git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git |
| 77 |
- http://www.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git |
| 78 |
|
| 79 |
There is also a gitweb interface available at |
| 80 |
http://www.kernel.org/git/?p=utils/kernel/kexec/kexec-tools.git |
| 81 |
|
| 82 |
More information about kexec-tools can be found at |
| 83 |
http://horms.net/projects/kexec/ |
| 84 |
|
| 85 |
3) Unpack the tarball with the tar command, as follows:: |
| 86 |
|
| 87 |
tar xvpzf kexec-tools.tar.gz |
| 88 |
|
| 89 |
4) Change to the kexec-tools directory, as follows:: |
| 90 |
|
| 91 |
cd kexec-tools-VERSION |
| 92 |
|
| 93 |
5) Configure the package, as follows:: |
| 94 |
|
| 95 |
./configure |
| 96 |
|
| 97 |
6) Compile the package, as follows:: |
| 98 |
|
| 99 |
make |
| 100 |
|
| 101 |
7) Install the package, as follows:: |
| 102 |
|
| 103 |
make install |
| 104 |
|
| 105 |
|
| 106 |
Build the system and dump-capture kernels |
| 107 |
----------------------------------------- |
| 108 |
There are two possible methods of using Kdump. |
| 109 |
|
| 110 |
1) Build a separate custom dump-capture kernel for capturing the |
| 111 |
kernel core dump. |
| 112 |
|
| 113 |
2) Or use the system kernel binary itself as dump-capture kernel and there is |
| 114 |
no need to build a separate dump-capture kernel. This is possible |
| 115 |
only with the architectures which support a relocatable kernel. As |
| 116 |
of today, i386, x86_64, ppc64, arm and arm64 architectures support |
| 117 |
relocatable kernel. |
| 118 |
|
| 119 |
Building a relocatable kernel is advantageous from the point of view that |
| 120 |
one does not have to build a second kernel for capturing the dump. But |
| 121 |
at the same time one might want to build a custom dump capture kernel |
| 122 |
suitable to his needs. |
| 123 |
|
| 124 |
Following are the configuration setting required for system and |
| 125 |
dump-capture kernels for enabling kdump support. |
| 126 |
|
| 127 |
System kernel config options |
| 128 |
---------------------------- |
| 129 |
|
| 130 |
1) Enable "kexec system call" or "kexec file based system call" in |
| 131 |
"Processor type and features.":: |
| 132 |
|
| 133 |
CONFIG_KEXEC=y or CONFIG_KEXEC_FILE=y |
| 134 |
|
| 135 |
And both of them will select KEXEC_CORE:: |
| 136 |
|
| 137 |
CONFIG_KEXEC_CORE=y |
| 138 |
|
| 139 |
2) Enable "sysfs file system support" in "Filesystem" -> "Pseudo |
| 140 |
filesystems." This is usually enabled by default:: |
| 141 |
|
| 142 |
CONFIG_SYSFS=y |
| 143 |
|
| 144 |
Note that "sysfs file system support" might not appear in the "Pseudo |
| 145 |
filesystems" menu if "Configure standard kernel features (expert users)" |
| 146 |
is not enabled in "General Setup." In this case, check the .config file |
| 147 |
itself to ensure that sysfs is turned on, as follows:: |
| 148 |
|
| 149 |
grep 'CONFIG_SYSFS' .config |
| 150 |
|
| 151 |
3) Enable "Compile the kernel with debug info" in "Kernel hacking.":: |
| 152 |
|
| 153 |
CONFIG_DEBUG_INFO=Y |
| 154 |
|
| 155 |
This causes the kernel to be built with debug symbols. The dump |
| 156 |
analysis tools require a vmlinux with debug symbols in order to read |
| 157 |
and analyze a dump file. |
| 158 |
|
| 159 |
Dump-capture kernel config options (Arch Independent) |
| 160 |
----------------------------------------------------- |
| 161 |
|
| 162 |
1) Enable "kernel crash dumps" support under "Processor type and |
| 163 |
features":: |
| 164 |
|
| 165 |
CONFIG_CRASH_DUMP=y |
| 166 |
|
| 167 |
And this will select VMCORE_INFO and CRASH_RESERVE:: |
| 168 |
CONFIG_VMCORE_INFO=y |
| 169 |
CONFIG_CRASH_RESERVE=y |
| 170 |
|
| 171 |
2) Enable "/proc/vmcore support" under "Filesystems" -> "Pseudo filesystems":: |
| 172 |
|
| 173 |
CONFIG_PROC_VMCORE=y |
| 174 |
|
| 175 |
(CONFIG_PROC_VMCORE is set by default when CONFIG_CRASH_DUMP is selected.) |
| 176 |
|
| 177 |
Dump-capture kernel config options (Arch Dependent, i386 and x86_64) |
| 178 |
-------------------------------------------------------------------- |
| 179 |
|
| 180 |
1) On i386, enable high memory support under "Processor type and |
| 181 |
features":: |
| 182 |
|
| 183 |
CONFIG_HIGHMEM4G |
| 184 |
|
| 185 |
2) With CONFIG_SMP=y, usually nr_cpus=1 need specified on the kernel |
| 186 |
command line when loading the dump-capture kernel because one |
| 187 |
CPU is enough for kdump kernel to dump vmcore on most of systems. |
| 188 |
|
| 189 |
However, you can also specify nr_cpus=X to enable multiple processors |
| 190 |
in kdump kernel. |
| 191 |
|
| 192 |
With CONFIG_SMP=n, the above things are not related. |
| 193 |
|
| 194 |
3) A relocatable kernel is suggested to be built by default. If not yet, |
| 195 |
enable "Build a relocatable kernel" support under "Processor type and |
| 196 |
features":: |
| 197 |
|
| 198 |
CONFIG_RELOCATABLE=y |
| 199 |
|
| 200 |
4) Use a suitable value for "Physical address where the kernel is |
| 201 |
loaded" (under "Processor type and features"). This only appears when |
| 202 |
"kernel crash dumps" is enabled. A suitable value depends upon |
| 203 |
whether kernel is relocatable or not. |
| 204 |
|
| 205 |
If you are using a relocatable kernel use CONFIG_PHYSICAL_START=0x100000 |
| 206 |
This will compile the kernel for physical address 1MB, but given the fact |
| 207 |
kernel is relocatable, it can be run from any physical address hence |
| 208 |
kexec boot loader will load it in memory region reserved for dump-capture |
| 209 |
kernel. |
| 210 |
|
| 211 |
Otherwise it should be the start of memory region reserved for |
| 212 |
second kernel using boot parameter "crashkernel=Y@X". Here X is |
| 213 |
start of memory region reserved for dump-capture kernel. |
| 214 |
Generally X is 16MB (0x1000000). So you can set |
| 215 |
CONFIG_PHYSICAL_START=0x1000000 |
| 216 |
|
| 217 |
5) Make and install the kernel and its modules. DO NOT add this kernel |
| 218 |
to the boot loader configuration files. |
| 219 |
|
| 220 |
Dump-capture kernel config options (Arch Dependent, ppc64) |
| 221 |
---------------------------------------------------------- |
| 222 |
|
| 223 |
1) Enable "Build a kdump crash kernel" support under "Kernel" options:: |
| 224 |
|
| 225 |
CONFIG_CRASH_DUMP=y |
| 226 |
|
| 227 |
2) Enable "Build a relocatable kernel" support:: |
| 228 |
|
| 229 |
CONFIG_RELOCATABLE=y |
| 230 |
|
| 231 |
Make and install the kernel and its modules. |
| 232 |
|
| 233 |
Dump-capture kernel config options (Arch Dependent, arm) |
| 234 |
---------------------------------------------------------- |
| 235 |
|
| 236 |
- To use a relocatable kernel, |
| 237 |
Enable "AUTO_ZRELADDR" support under "Boot" options:: |
| 238 |
|
| 239 |
AUTO_ZRELADDR=y |
| 240 |
|
| 241 |
Dump-capture kernel config options (Arch Dependent, arm64) |
| 242 |
---------------------------------------------------------- |
| 243 |
|
| 244 |
- Please note that kvm of the dump-capture kernel will not be enabled |
| 245 |
on non-VHE systems even if it is configured. This is because the CPU |
| 246 |
will not be reset to EL2 on panic. |
| 247 |
|
| 248 |
crashkernel syntax |
| 249 |
=========================== |
| 250 |
1) crashkernel=size@offset |
| 251 |
|
| 252 |
Here 'size' specifies how much memory to reserve for the dump-capture kernel |
| 253 |
and 'offset' specifies the beginning of this reserved memory. For example, |
| 254 |
"crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory |
| 255 |
starting at physical address 0x01000000 (16MB) for the dump-capture kernel. |
| 256 |
|
| 257 |
The crashkernel region can be automatically placed by the system |
| 258 |
kernel at run time. This is done by specifying the base address as 0, |
| 259 |
or omitting it all together:: |
| 260 |
|
| 261 |
crashkernel=256M@0 |
| 262 |
|
| 263 |
or:: |
| 264 |
|
| 265 |
crashkernel=256M |
| 266 |
|
| 267 |
If the start address is specified, note that the start address of the |
| 268 |
kernel will be aligned to a value (which is Arch dependent), so if the |
| 269 |
start address is not then any space below the alignment point will be |
| 270 |
wasted. |
| 271 |
|
| 272 |
2) range1:size1[,range2:size2,...][@offset] |
| 273 |
|
| 274 |
While the "crashkernel=size[@offset]" syntax is sufficient for most |
| 275 |
configurations, sometimes it's handy to have the reserved memory dependent |
| 276 |
on the value of System RAM -- that's mostly for distributors that pre-setup |
| 277 |
the kernel command line to avoid a unbootable system after some memory has |
| 278 |
been removed from the machine. |
| 279 |
|
| 280 |
The syntax is:: |
| 281 |
|
| 282 |
crashkernel=<range1>:<size1>[,<range2>:<size2>,...][@offset] |
| 283 |
range=start-[end] |
| 284 |
|
| 285 |
For example:: |
| 286 |
|
| 287 |
crashkernel=512M-2G:64M,2G-:128M |
| 288 |
|
| 289 |
This would mean: |
| 290 |
|
| 291 |
1) if the RAM is smaller than 512M, then don't reserve anything |
| 292 |
(this is the "rescue" case) |
| 293 |
2) if the RAM size is between 512M and 2G (exclusive), then reserve 64M |
| 294 |
3) if the RAM size is larger than 2G, then reserve 128M |
| 295 |
|
| 296 |
3) crashkernel=size,high and crashkernel=size,low |
| 297 |
|
| 298 |
If memory above 4G is preferred, crashkernel=size,high can be used to |
| 299 |
fulfill that. With it, physical memory is allowed to be allocated from top, |
| 300 |
so could be above 4G if system has more than 4G RAM installed. Otherwise, |
| 301 |
memory region will be allocated below 4G if available. |
| 302 |
|
| 303 |
When crashkernel=X,high is passed, kernel could allocate physical memory |
| 304 |
region above 4G, low memory under 4G is needed in this case. There are |
| 305 |
three ways to get low memory: |
| 306 |
|
| 307 |
1) Kernel will allocate at least 256M memory below 4G automatically |
| 308 |
if crashkernel=Y,low is not specified. |
| 309 |
2) Let user specify low memory size instead. |
| 310 |
3) Specified value 0 will disable low memory allocation:: |
| 311 |
|
| 312 |
crashkernel=0,low |
| 313 |
|
| 314 |
4) crashkernel=size,cma |
| 315 |
|
| 316 |
Reserve additional crash kernel memory from CMA. This reservation is |
| 317 |
usable by the first system's userspace memory and kernel movable |
| 318 |
allocations (memory balloon, zswap). Pages allocated from this memory |
| 319 |
range will not be included in the vmcore so this should not be used if |
| 320 |
dumping of userspace memory is intended and it has to be expected that |
| 321 |
some movable kernel pages may be missing from the dump. |
| 322 |
|
| 323 |
A standard crashkernel reservation, as described above, is still needed |
| 324 |
to hold the crash kernel and initrd. |
| 325 |
|
| 326 |
This option increases the risk of a kdump failure: DMA transfers |
| 327 |
configured by the first kernel may end up corrupting the second |
| 328 |
kernel's memory. |
| 329 |
|
| 330 |
This reservation method is intended for systems that can't afford to |
| 331 |
sacrifice enough memory for standard crashkernel reservation and where |
| 332 |
less reliable and possibly incomplete kdump is preferable to no kdump at |
| 333 |
all. |
| 334 |
|
| 335 |
Boot into System Kernel |
| 336 |
----------------------- |
| 337 |
1) Update the boot loader (such as grub, yaboot, or lilo) configuration |
| 338 |
files as necessary. |
| 339 |
|
| 340 |
2) Boot the system kernel with the boot parameter "crashkernel=Y@X". |
| 341 |
|
| 342 |
On x86 and x86_64, use "crashkernel=Y[@X]". Most of the time, the |
| 343 |
start address 'X' is not necessary, kernel will search a suitable |
| 344 |
area. Unless an explicit start address is expected. |
| 345 |
|
| 346 |
On ppc64, use "crashkernel=128M@32M". |
| 347 |
|
| 348 |
On s390x, typically use "crashkernel=xxM". The value of xx is dependent |
| 349 |
on the memory consumption of the kdump system. In general this is not |
| 350 |
dependent on the memory size of the production system. |
| 351 |
|
| 352 |
On arm, the use of "crashkernel=Y@X" is no longer necessary; the |
| 353 |
kernel will automatically locate the crash kernel image within the |
| 354 |
first 512MB of RAM if X is not given. |
| 355 |
|
| 356 |
On arm64, use "crashkernel=Y[@X]". Note that the start address of |
| 357 |
the kernel, X if explicitly specified, must be aligned to 2MiB (0x200000). |
| 358 |
|
| 359 |
Load the Dump-capture Kernel |
| 360 |
============================ |
| 361 |
|
| 362 |
After booting to the system kernel, dump-capture kernel needs to be |
| 363 |
loaded. |
| 364 |
|
| 365 |
Based on the architecture and type of image (relocatable or not), one |
| 366 |
can choose to load the uncompressed vmlinux or compressed bzImage/vmlinuz |
| 367 |
of dump-capture kernel. Following is the summary. |
| 368 |
|
| 369 |
For i386 and x86_64: |
| 370 |
|
| 371 |
- Use bzImage/vmlinuz if kernel is relocatable. |
| 372 |
- Use vmlinux if kernel is not relocatable. |
| 373 |
|
| 374 |
For ppc64: |
| 375 |
|
| 376 |
- Use vmlinux |
| 377 |
|
| 378 |
For s390x: |
| 379 |
|
| 380 |
- Use image or bzImage |
| 381 |
|
| 382 |
For arm: |
| 383 |
|
| 384 |
- Use zImage |
| 385 |
|
| 386 |
For arm64: |
| 387 |
|
| 388 |
- Use vmlinux or Image |
| 389 |
|
| 390 |
If you are using an uncompressed vmlinux image then use following command |
| 391 |
to load dump-capture kernel:: |
| 392 |
|
| 393 |
kexec -p <dump-capture-kernel-vmlinux-image> \ |
| 394 |
--initrd=<initrd-for-dump-capture-kernel> --args-linux \ |
| 395 |
--append="root=<root-dev> <arch-specific-options>" |
| 396 |
|
| 397 |
If you are using a compressed bzImage/vmlinuz, then use following command |
| 398 |
to load dump-capture kernel:: |
| 399 |
|
| 400 |
kexec -p <dump-capture-kernel-bzImage> \ |
| 401 |
--initrd=<initrd-for-dump-capture-kernel> \ |
| 402 |
--append="root=<root-dev> <arch-specific-options>" |
| 403 |
|
| 404 |
If you are using a compressed zImage, then use following command |
| 405 |
to load dump-capture kernel:: |
| 406 |
|
| 407 |
kexec --type zImage -p <dump-capture-kernel-bzImage> \ |
| 408 |
--initrd=<initrd-for-dump-capture-kernel> \ |
| 409 |
--dtb=<dtb-for-dump-capture-kernel> \ |
| 410 |
--append="root=<root-dev> <arch-specific-options>" |
| 411 |
|
| 412 |
If you are using an uncompressed Image, then use following command |
| 413 |
to load dump-capture kernel:: |
| 414 |
|
| 415 |
kexec -p <dump-capture-kernel-Image> \ |
| 416 |
--initrd=<initrd-for-dump-capture-kernel> \ |
| 417 |
--append="root=<root-dev> <arch-specific-options>" |
| 418 |
|
| 419 |
Following are the arch specific command line options to be used while |
| 420 |
loading dump-capture kernel. |
| 421 |
|
| 422 |
For i386 and x86_64: |
| 423 |
|
| 424 |
"1 irqpoll nr_cpus=1 reset_devices" |
| 425 |
|
| 426 |
For ppc64: |
| 427 |
|
| 428 |
"1 maxcpus=1 noirqdistrib reset_devices" |
| 429 |
|
| 430 |
For s390x: |
| 431 |
|
| 432 |
"1 nr_cpus=1 cgroup_disable=memory" |
| 433 |
|
| 434 |
For arm: |
| 435 |
|
| 436 |
"1 maxcpus=1 reset_devices" |
| 437 |
|
| 438 |
For arm64: |
| 439 |
|
| 440 |
"1 nr_cpus=1 reset_devices" |
| 441 |
|
| 442 |
Notes on loading the dump-capture kernel: |
| 443 |
|
| 444 |
* By default, the ELF headers are stored in ELF64 format to support |
| 445 |
systems with more than 4GB memory. On i386, kexec automatically checks if |
| 446 |
the physical RAM size exceeds the 4 GB limit and if not, uses ELF32. |
| 447 |
So, on non-PAE systems, ELF32 is always used. |
| 448 |
|
| 449 |
The --elf32-core-headers option can be used to force the generation of ELF32 |
| 450 |
headers. This is necessary because GDB currently cannot open vmcore files |
| 451 |
with ELF64 headers on 32-bit systems. |
| 452 |
|
| 453 |
* The "irqpoll" boot parameter reduces driver initialization failures |
| 454 |
due to shared interrupts in the dump-capture kernel. |
| 455 |
|
| 456 |
* You must specify <root-dev> in the format corresponding to the root |
| 457 |
device name in the output of mount command. |
| 458 |
|
| 459 |
* Boot parameter "1" boots the dump-capture kernel into single-user |
| 460 |
mode without networking. If you want networking, use "3". |
| 461 |
|
| 462 |
* We generally don't have to bring up a SMP kernel just to capture the |
| 463 |
dump. Hence generally it is useful either to build a UP dump-capture |
| 464 |
kernel or specify maxcpus=1 option while loading dump-capture kernel. |
| 465 |
Note, though maxcpus always works, you had better replace it with |
| 466 |
nr_cpus to save memory if supported by the current ARCH, such as x86. |
| 467 |
|
| 468 |
* You should enable multi-cpu support in dump-capture kernel if you intend |
| 469 |
to use multi-thread programs with it, such as parallel dump feature of |
| 470 |
makedumpfile. Otherwise, the multi-thread program may have a great |
| 471 |
performance degradation. To enable multi-cpu support, you should bring up an |
| 472 |
SMP dump-capture kernel and specify maxcpus/nr_cpus options while loading it. |
| 473 |
|
| 474 |
* For s390x there are two kdump modes: If an ELF header is specified with |
| 475 |
the elfcorehdr= kernel parameter, it is used by the kdump kernel as it |
| 476 |
is done on all other architectures. If no elfcorehdr= kernel parameter is |
| 477 |
specified, the s390x kdump kernel dynamically creates the header. The |
| 478 |
second mode has the advantage that for CPU and memory hotplug, kdump has |
| 479 |
not to be reloaded with kexec_load(). |
| 480 |
|
| 481 |
* For s390x systems with many attached devices the "cio_ignore" kernel |
| 482 |
parameter should be used for the kdump kernel in order to prevent allocation |
| 483 |
of kernel memory for devices that are not relevant for kdump. The same |
| 484 |
applies to systems that use SCSI/FCP devices. In that case the |
| 485 |
"allow_lun_scan" zfcp module parameter should be set to zero before |
| 486 |
setting FCP devices online. |
| 487 |
|
| 488 |
Kernel Panic |
| 489 |
============ |
| 490 |
|
| 491 |
After successfully loading the dump-capture kernel as previously |
| 492 |
described, the system will reboot into the dump-capture kernel if a |
| 493 |
system crash is triggered. Trigger points are located in panic(), |
| 494 |
die(), die_nmi() and in the sysrq handler (ALT-SysRq-c). |
| 495 |
|
| 496 |
The following conditions will execute a crash trigger point: |
| 497 |
|
| 498 |
If a hard lockup is detected and "NMI watchdog" is configured, the system |
| 499 |
will boot into the dump-capture kernel ( die_nmi() ). |
| 500 |
|
| 501 |
If die() is called, and it happens to be a thread with pid 0 or 1, or die() |
| 502 |
is called inside interrupt context or die() is called and panic_on_oops is set, |
| 503 |
the system will boot into the dump-capture kernel. |
| 504 |
|
| 505 |
On powerpc systems when a soft-reset is generated, die() is called by all cpus |
| 506 |
and the system will boot into the dump-capture kernel. |
| 507 |
|
| 508 |
For testing purposes, you can trigger a crash by using "ALT-SysRq-c", |
| 509 |
"echo c > /proc/sysrq-trigger" or write a module to force the panic. |
| 510 |
|
| 511 |
Write Out the Dump File |
| 512 |
======================= |
| 513 |
|
| 514 |
After the dump-capture kernel is booted, write out the dump file with |
| 515 |
the following command:: |
| 516 |
|
| 517 |
cp /proc/vmcore <dump-file> |
| 518 |
|
| 519 |
or use scp to write out the dump file between hosts on a network, e.g:: |
| 520 |
|
| 521 |
scp /proc/vmcore remote_username@remote_ip:<dump-file> |
| 522 |
|
| 523 |
You can also use makedumpfile utility to write out the dump file |
| 524 |
with specified options to filter out unwanted contents, e.g:: |
| 525 |
|
| 526 |
makedumpfile -l --message-level 1 -d 31 /proc/vmcore <dump-file> |
| 527 |
|
| 528 |
Analysis |
| 529 |
======== |
| 530 |
|
| 531 |
Before analyzing the dump image, you should reboot into a stable kernel. |
| 532 |
|
| 533 |
You can do limited analysis using GDB on the dump file copied out of |
| 534 |
/proc/vmcore. Use the debug vmlinux built with -g and run the following |
| 535 |
command:: |
| 536 |
|
| 537 |
gdb vmlinux <dump-file> |
| 538 |
|
| 539 |
Stack trace for the task on processor 0, register display, and memory |
| 540 |
display work fine. |
| 541 |
|
| 542 |
Note: GDB cannot analyze core files generated in ELF64 format for x86. |
| 543 |
On systems with a maximum of 4GB of memory, you can generate |
| 544 |
ELF32-format headers using the --elf32-core-headers kernel option on the |
| 545 |
dump kernel. |
| 546 |
|
| 547 |
You can also use the Crash utility to analyze dump files in Kdump |
| 548 |
format. Crash is available at the following URL: |
| 549 |
|
| 550 |
https://github.com/crash-utility/crash |
| 551 |
|
| 552 |
Crash document can be found at: |
| 553 |
https://crash-utility.github.io/ |
| 554 |
|
| 555 |
Trigger Kdump on WARN() |
| 556 |
======================= |
| 557 |
|
| 558 |
The kernel parameter, panic_on_warn, calls panic() in all WARN() paths. This |
| 559 |
will cause a kdump to occur at the panic() call. In cases where a user wants |
| 560 |
to specify this during runtime, /proc/sys/kernel/panic_on_warn can be set to 1 |
| 561 |
to achieve the same behaviour. |
| 562 |
|
| 563 |
Trigger Kdump on add_taint() |
| 564 |
============================ |
| 565 |
|
| 566 |
The kernel parameter panic_on_taint facilitates a conditional call to panic() |
| 567 |
from within add_taint() whenever the value set in this bitmask matches with the |
| 568 |
bit flag being set by add_taint(). |
| 569 |
This will cause a kdump to occur at the add_taint()->panic() call. |
| 570 |
|
| 571 |
Write the dump file to encrypted disk volume |
| 572 |
============================================ |
| 573 |
|
| 574 |
CONFIG_CRASH_DM_CRYPT can be enabled to support saving the dump file to an |
| 575 |
encrypted disk volume (only x86_64 supported for now). User space can interact |
| 576 |
with /sys/kernel/config/crash_dm_crypt_keys for setup, |
| 577 |
|
| 578 |
1. Tell the first kernel what logon keys are needed to unlock the disk volumes, |
| 579 |
# Add key #1 |
| 580 |
mkdir /sys/kernel/config/crash_dm_crypt_keys/7d26b7b4-e342-4d2d-b660-7426b0996720 |
| 581 |
# Add key #1's description |
| 582 |
echo cryptsetup:7d26b7b4-e342-4d2d-b660-7426b0996720 > /sys/kernel/config/crash_dm_crypt_keys/description |
| 583 |
|
| 584 |
# how many keys do we have now? |
| 585 |
cat /sys/kernel/config/crash_dm_crypt_keys/count |
| 586 |
1 |
| 587 |
|
| 588 |
# Add key #2 in the same way |
| 589 |
|
| 590 |
# how many keys do we have now? |
| 591 |
cat /sys/kernel/config/crash_dm_crypt_keys/count |
| 592 |
2 |
| 593 |
|
| 594 |
# To support CPU/memory hot-plugging, reuse keys already saved to reserved |
| 595 |
# memory |
| 596 |
echo true > /sys/kernel/config/crash_dm_crypt_key/reuse |
| 597 |
|
| 598 |
2. Load the dump-capture kernel |
| 599 |
|
| 600 |
3. After the dump-capture kerne get booted, restore the keys to user keyring |
| 601 |
echo yes > /sys/kernel/crash_dm_crypt_keys/restore |
| 602 |
|
| 603 |
Contact |
| 604 |
======= |
| 605 |
|
| 606 |
- [email protected] |
| 607 |
|
| 608 |
GDB macros |
| 609 |
========== |
| 610 |
|
| 611 |
.. include:: gdbmacros.txt |
| 612 |
:literal: |