Core Dumps on Linux: Debugging Crashes With coredumpctl

Core Dumps on Linux: Debugging Crashes With coredumpctl

A program crashes with a segmentation fault, and by the time you look, it is gone. A core dump is the evidence: a snapshot of the process’s memory and registers at the moment it died. With it, a debugger can show exactly which line crashed and the state that led there, even for crashes you cannot reproduce.

How the kernel produces a core dump

When a process receives a signal whose default action is to dump core, most commonly SIGSEGV (invalid memory access), SIGABRT (an abort(), often from a failed assertion), SIGBUS, SIGFPE, or SIGILL, the kernel writes out its memory. Our guide to process signals covers these.

Where it goes is set by a kernel parameter:

cat /proc/sys/kernel/core_pattern
Value starts withMeaning
`/usr/lib/systemd/systemd-coredump`
`/usr/share/apport/apport`
core or a pathWritten as a file, traditionally in the process’s working directory

A crash also shows its exit status as 128 plus the signal number (139 for SIGSEGV); see exit codes explained.

coredumpctl

On systems using systemd-coredump, crashes are stored compressed in /var/lib/systemd/coredump/ and indexed in the journal. coredumpctl is the interface:

coredumpctl list
TIME                          PID  UID  GID SIG     COREFILE EXE
Tue 2026-09-29 14:02:11 CDT  4812 1000 1000 SIGSEGV present  /usr/bin/myapp
Tue 2026-09-29 16:40:03 CDT  9120 1000 1000 SIGABRT present  /usr/bin/python3.14

COREFILE says whether the dump is still on disk (present), was removed for space (missing), or was never stored.

coredumpctl info 4812          # details and a backtrace, if symbols are available
coredumpctl info myapp         # the most recent crash of a program
coredumpctl gdb 4812           # open the dump in gdb
coredumpctl dump 4812 -o core.myapp   # extract the core file

Debugging the dump with gdb

coredumpctl gdb myapp

Inside gdb, the essentials:

(gdb) bt                 # backtrace: the call stack at the crash
(gdb) bt full            # with local variables
(gdb) frame 2            # move to a frame in the stack
(gdb) info locals        # local variables there
(gdb) print ptr          # inspect a variable
(gdb) thread apply all bt   # backtraces for every thread

The backtrace is usually what a bug report needs. Our gdb debugging basics guide covers gdb itself.

Debug symbols with debuginfod

A backtrace full of ?? means debug symbols are missing: distributions strip them from packages to save space and ship them separately. debuginfod fetches the right symbols on demand, matched by the binary’s build ID:

# Fedora and Arch typically set this already; check with: echo $DEBUGINFOD_URLS
export DEBUGINFOD_URLS="https://debuginfod.debian.net"      # Debian
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"      # Ubuntu
export DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org/"   # Fedora
export DEBUGINFOD_URLS="https://debuginfod.archlinux.org"   # Arch

gdb then downloads symbols automatically the first time it needs them, and the backtrace shows function names, files, and line numbers.

Ubuntu and Apport

Ubuntu routes crashes to Apport, which writes reports to /var/crash/ and offers to send them to Ubuntu’s error tracker. To inspect a report locally:

ls /var/crash/
apport-unpack /var/crash/_usr_bin_myapp.1000.crash /tmp/myapp-crash
gdb /usr/bin/myapp /tmp/myapp-crash/CoreDump

Installing systemd-coredump switches Ubuntu to the systemd handler if you prefer coredumpctl.

Making sure dumps are created

Core size limit. Shells may set the core size limit to 0:

ulimit -c              # 0 means disabled for this shell
ulimit -c unlimited    # enable for programs started from this shell

For services, systemd sets the limit with LimitCORE= in the unit. See ulimit and resource limits.

setuid programs do not dump core by default, for security (controlled by fs.suid_dumpable).

Controlling disk use and retention

Dumps can be large. /etc/systemd/coredump.conf controls systemd-coredump:

[Coredump]
Storage=external       # store on disk (default); 'none' to keep only the journal entry
Compress=yes
ProcessSizeMax=2G      # do not dump processes larger than this
ExternalSizeMax=2G
MaxUse=4G              # total space for dumps

Old dumps are cleaned up by systemd-tmpfiles after a few days by default.

Privacy and security

A core dump contains everything the process had in memory: passwords, keys, tokens, and personal data can all be in there. On systems handling sensitive data:

  • Keep /var/lib/systemd/coredump/ readable only by root (the default)
  • Limit retention
  • Consider Storage=none unless you are actively debugging
  • Be careful before attaching a core dump to a public bug report

When a program does not crash but misbehaves, strace and ltrace show what it is doing, and perf shows where it spends time.

Frequently Asked Questions

What is a core dump?

A file containing a snapshot of a process’s memory, registers, and state at the moment it crashed. Loading it into a debugger shows exactly where the program was and what it was doing, which makes it possible to diagnose crashes that are hard to reproduce.

Where are core dumps stored on Linux?

On distributions using systemd-coredump, which includes Fedora, Arch, and Debian when the package is installed, they are stored compressed in /var/lib/systemd/coredump and indexed in the journal. Ubuntu sends crashes to Apport instead, which writes reports to /var/crash.

How do I list recent crashes?

Run coredumpctl list. It shows each crash with its time, PID, signal, whether the core file is still present, and the executable. coredumpctl info followed by a PID or program name shows details including a backtrace when symbols are available.

Why does gdb show question marks instead of function names?

Debug symbols are missing. Distributions ship them separately. Enabling debuginfod, by setting DEBUGINFOD_URLS to your distribution’s server, lets gdb download the matching symbols automatically. Fedora, Debian, Ubuntu, and Arch all run debuginfod servers.

Why is no core dump created when my program crashes?

Common reasons are a core size limit of zero from ulimit -c, the program being setuid, the crash being handled by Apport on Ubuntu, or systemd-coredump storage being disabled. Check cat /proc/sys/kernel/core_pattern to see where the kernel sends dumps.

Are core dumps a security risk?

They can be. A core dump contains the process’s memory, which may include passwords, keys, and personal data. Restrict access to the dump directory, limit retention, and disable dumps on systems that handle sensitive data unless you need them, by setting Storage=none in coredump.conf.