Date: 01-09-2022

Enabling Core Dumps on Red Hat Linux Systems

Core dumps are an essential part of analyzing slapd crashes. Unfortunately, Red Hat systems do not save the Symas OpenLDAP core dumps by default.

Some Red Hat installations have the abrt (abort) service installed. This service manages core dump saving and processing. The gdb debugging program is also required for analyzing core dumps.

Note: RHEL 7 systems use the yum command to manage package installations. RHEL 8 and 9 systems use the dnf command to manage packages. The dnf command is backwards-compatible with yum commands, so either command may be used on RHEL 8 and 9 systems. In this document we'll use the yum command.

Install the gdb debugger

The gdb debugger is required for reading core dump files.

Check for installation:

yum list installed | grep gdb

Install if necessary:

yum install gdb

Checking for the abrt service

To check if the abrt service is installed, run:

yum list installed | grep abrt

Configuring systems with abrtd

Install and configure the abrt-ccpp package

The abrt-ccpp package is necessary for processing Symas Openldap core dumps.

Check for installation:

yum list installed | grep abrt-addon-ccpp

Install the package if necessary:

yum install abrt-addon-ccpp

When the abrt-ccpp package is installed, the ccpp hook need to be installed.

Install the ccpp hook:

abrt-install-ccpp-hook install

Verify that the installation succeeded (the result from this command should be "0"):

abrt-install-ccpp-hook is-installed; echo $?;

Update the file /etc/abrt/plugins/CCpp.conf with the following settings:

MaxCoreFileSize = 0
CreateCoreBacktrace = yes
SaveFullCore = yes

Configure abrtd

Update the file /etc/abrt/abrt-action-save-package-data.conf with the following settings to enable third-party package processing:

OpenGPGCheck = no
ProcessUnpackaged = yes

Depending on the size of the LDAP database the core dumps from slapd may be too large for the default core dump location; this may truncate the core dump, rendering it useless and cause disk space related issues. Also, by default the size of the core dump file is limited to a certain size, again truncating the file and rendering it useless.

Update the file /etc/abrt/abrt.conf with the following settings:

DumpLocation = <path to target directory>
MaxCrashReportSize = 0

Configure systemd slapd overrides for core dump size

Core dump size may be restricted by the slapd service settings in systemd. This may cause the core dump files to be truncated and rendered useless.

Update (create if necessary) the file /etc/systemd/system/solservice.service.d/override.conf with the following settings:

[service]
LimitCORE=infinity

Configure selinux

The configured selinux policy may prevent abrtd from writing core dumps. Check the current mode of selinux:

sestatus

If the 'Current mode' is 'permissive' no action needs to be taken.

If the 'Current mode' is 'enforcing', update the file /etc/selinux/config with the following settingL:

SELINUX=permissive

After updating the selinux mode, the server must be rebooted before continuing.

Service Configuration

Ensure that the abrt and abrt-ccpp services are enabled:

systemctl status abrtd.service
systemctl status abrt-ccpp.service

If either are disabled, execute:

systemctl enable abrtd.service
systemctl enable abrt-ccpp.service

Apply all changes to slapd, abrt and abrt-ccpp are applied to systemd:

systemctl daemon-reload

Restart the necessary services:

systemctl restart abrtd
systemctl restart abrt-ccpp
systemctl restart slapd

Check the status of the services:

systemctl status abrtd
systemctl status abrt-ccpp
systemctl status slapd

Configuring systems without abrtd

Install the lz4 compression utility

Core dumps are automatically compressed using the lz4 compression utility. Core dump files must be decompressed prior to analyzing the core dump in the gdb debugger. The lz4 compression utility is not installed by default on some RHEL and RHEL derivative linux distributions.

Check if lz4 is installed:

yum list installed | grep 'lz4\.'

Install the package if necessary:

yum install lz4

Configure systemd slapd overrides for core dump size

Core dump size may be restricted by the slapd service settings in systemd. This may cause the core dump files to be truncated and rendered useless.

Update (create if necessary) the file /etc/systemd/system/solservice.service.d/override.conf with the following settings:

[service]
LimitCORE=infinity

Configure sysctl to set core file location and permissions

Core dump file sizes may exceed the amount of available disk space in the default save location, /var/crash. The core file location should be set to a disk/partition that has enough space to hold the core dump file.

Slapd may use the setuid function to allow slapd to be run by a user/group other than root. By default, services that use setuid are forbidden from saving core dump files.

For both of these items, update the file /etc/sysctl.conf with the following settings:

kernel.core_pattern=<path-to-core-dump-location>/core.%e-%t-%p-%i-%s
kernel.core_uses_pid=1
fs.suid_dumpable=2

Implement the configuration changes

For the new settings to take effect, execute the following:

sysctl -p
systemctl daemon-reexec
systemctl daemon-reload
systemctl restart slapd

Testing core dumps

If possible, these new settings should be tested in a non-production environment.

Make sure slapd is running:

systemctl status slapd

Kill slapd with a SIGSEV signal:

kill -11 `pidof slapd`

Check the core dump directory, as set in /etc/abrt/abrt.conf:

ls -lt <core dump location>

If the core dump is successfully written, you should see directores/files like:

# Systems with abrtd:
<path-to-target-directory>/ccpp-2022-09-01-11:47:20-1829

# Systems without abrtd
<path-to-target-directory>/core.slapd.XXXXX