How do I configure kexec/kdump on Red Hat Enterprise Linux 5?

Release Found: Red Hat Enterprise Linux 5

Note: for virtualized guests, xendump must be used. Please see the article How do I configure Xendump on Red Hat Enterprise Linux 5?.

Overview

Kexec is a fastboot mechanism that allows booting a Linux kernel from the context of an already running kernel without going through the BIOS. The BIOS can be very time consuming, especially on big servers with numerous peripherals. This can save a lot of time for developers who end up booting a machine numerous times.

Kdump is a new kernel crash dumping mechanism and is very reliable. The crash dump is captured from the context of a freshly booted kernel and not from the context of the crashed kernel. Kdump uses kexec to boot into a second kernel whenever the system crashes. This second kernel, often called a capture kernel, boots with very little memory and captures the dump image.

The first kernel reserves a section of memory that the second kernel uses to boot. Be aware that the kdump reserves a significant amount of memory at the boot time, which changes the actual minimum memory requirements of Red Hat Enterprise Linux 5. To compute the actual minimum memory requirements for a system, refer to http://www.RedHat.com/rhel/details/limits/ for the listed minimum memory requirements and add the amount of memory used by kdump to determine the actual minimum memory requirements.

Kexec enables booting the capture kernel without going through BIOS hence the contents of the first kernel's memory are preserved, which is essentially the kernel crash dump.

Installing Kdump

Verify the kexec-tools package is installed:

# rpm -q kexec-tools

If it is not installed proceed to install it via yum:

# yum install kexec-tools

Specifying Kdump Location

The location of the Kdump vmcore must be specified in /etc/kdump.conf. Not specifying the vmcore location will result in undefined behavior. You can either dump directly to a device, to a file, or to some location on the network via NFS or SSH.

Dumping Directly to a Device

You can configure Kdump to dump directly to a device by using the raw directive in kdump.conf. The syntax to be used is

raw devicename

For example:

raw /dev/sda1

Please be aware that this will overwrite any data that was previously on the device.

Dumping to a file on Disk

Kdump can be configured to mount a partition and dump to a file on disk. This is done by specifying the filesystem type, followed by

the partition in kdump.conf. The partition may be specified as a device name, a filesystem label, or UUID in the same manner as /etc/fstab.

The default directory in which the core will be dumped is /var/crash/%DATE/ where %DATE is the current date at the time of the cash dump. For example:

ext3 /dev/sda1

will mount /dev/sda1 as an ext3 device and dump the core to /var/crash/, whereas

ext3 LABEL=/boot

will mount the device that is ext3 with the label /boot. On most Red Hat Enterprise Linux installations, this will be the /boot directory. The easiest way to find how to specify the device is to look at what you're currently using in /etc/fstab. The default directory in which the core will be dumped is /var/crash/%DATE/ where %DATE is the current date at the time of the crash dump. This can be changed by using the path directive in kdump.conf. For example:

ext3 UUID=f15759be-89d4-46c4-9e1d-1b67e5b5da82

path /usr/local/cores

will dump the vmcore to /usr/local/cores/ instead of the default /var/crash/ location.

Dumping to a Network Device using NFS

To configure kdump to dump to an NFS mount, edit /etc/kdump.conf and add a line with the following format:

net :/nfs/mount

For example:

net nfs.example.com:/export/vmcores

This will dump the vmcore to /export/vmcores/var/crash/[hostname]-[date] on the server nfs.example.com. The client system must have access to write to this mount point.

Dumping to a Network Device using SSH

SSH has the advantage of encrypting network communications while dumping. For this reason this is the best solution when you're required to dump a vmcore across a publicly accessible network such as the Internet or a corporate WAN.

net user@

For example:

net kdump@crash.example.com

In this case, kdump will use scp to connect to the crash.example.com server using the kdump user. It will copy the vmcore to the /var/crash/[host]-[date]/ directory. The kdump user will need the necessary write permissions on the remote server.

To make this change take effect, run service kdump propagate, which should result in output similar to the following:

Generating new ssh keys... done,

kdump@crash.example.com's password:

/root/.ssh/kdump_id_rsa.pub has been added to

~kdump/.ssh/authorized_keys2 on crash.example.com

Specifying Page Selection and Compression

On large memory systems, it is advisable to both discard pages that are not needed and to compress remaining pages. This is done in kdump.conf with the core_collector command. At this point in them the only fully supported core collector is makedumpfile. The options can be viewed with makedumpfile --help. The -d option specifies which types of pages should be left out. The option is a bit mask, having each page type specified like so:

zero pages    = 1

cache pages    = 2

cache private = 4

user  pages    = 8

free  pages    = 16

In general, these pages don't contain relevent information. To set all these flags and leave out these pages, use a value of -d 31. The -c tells makedumpfile to compress the remaining data pages.

#core_collector makedumpfile -d 1           # throw out zero pages (containing no data)

#core_collector makedumpfile -d 31          # throw out all trival pages

#core_collector makedumpfile -c             # compress all pages, but leave them all

core_collector makedumpfile -d 31 -c        # throw out trival pages and compress (recommended)

Keep in mind that using the -d and -c options will marginally increase the ammount of time required to gather a cores.

Adding Boot Parameters

Modify some boot parameters to reserve a chunk of memory for the capture kernel. For i386 and x86_64 architectures, edit /etc/grub.conf, and append crashkernel=128M@16M to the end of the kernel line.

Note: It may be possible to use less than 128M, but testing with only 64M has proven unreliable. This is an example of /etc/grub.conf with the kdump options added:

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE:  You do not have a /boot partition.  This means that

#           all kernel and initrd paths are relative to /, eg.

#           root (hd0,0)

#           kernel /boot/vmlinuz-version ro root=/dev/hda1

#           initrd /boot/initrd-version.img

#boot=/dev/hda

default=0

timeout=5

splashimage=(hd0,0)/boot/grub/splash.xpm.gz

hiddenmenu

title Red Hat Enterprise Linux Client (2.6.17-1.2519.4.21.el5)

root (hd0,0)

kernel /boot/vmlinuz-2.6.17-1.2519.4.21.el5 ro root=LABEL=/ rhgb quiet crashkernel=128M@16M

initrd /boot/initrd-2.6.17-1.2519.4.21.el5.img

Testing

After making the above changes, reboot the system. The 128M of memory (starting 16M into the memory) is left untouched by the normal system, reserved for the capture kernel. Take note that the output of free -m shows 128M less memory than without this parameter, which is expected.

Now that the reserved memory region is set up, turn on the kdump init script and start the service:

#  chkconfig kdump on

#  service kdump start

This should load the kernel image image via kexec, leaving the system ready to capture a vmcore upon crashing. To test this, force-crash the system using sysrq:

# echo "c" > /proc/sysrq-trigger

This causes the kernel to panic, followed by the system restarting into the kdump kernel. When the boot process gets to the point where it starts the kdump service, the vmcore should be copied out to disk to the location you specified in the /etc/kdump.conf file.

NOTE: Console frame-buffers and X are not properly supported. On a system typically run with something like "vga=791" in the kernel config line or with X running, console video will be garbled when a kernel is booted via kexec. Note that the kdump kernel should still be able to create a dump, and when the system reboots, video should be restored to normal.