gdb inspects a running or crashed program: where it is, what the call stack looks like, and what the variables contain.

You need debug symbols. Without them you get addresses and question marks.

Symbols first

gcc -g -Og -o myprog myprog.c

-g includes debug information. -Og optimises without interfering with debugging, which is what you want during development. Full -O2 inlines and reorders code so reported line numbers stop matching your source and variables show as <optimized out>.

For a distribution binary, install the matching debug package:

# Debian and Ubuntu
sudo apt install nginx-dbgsym

# Fedora and RHEL
sudo dnf debuginfo-install nginx

Check what a binary has:

file ./myprog
# "with debug_info, not stripped" is what you want

The commands that cover most sessions

gdb ./myprog

(gdb) break main              # or: b main
(gdb) break myprog.c:42
(gdb) run                     # or: r, with arguments after it
(gdb) next                    # or: n, step over calls
(gdb) step                    # or: s, step into calls
(gdb) continue                # or: c
(gdb) print variable          # or: p
(gdb) backtrace               # or: bt
(gdb) quit

next versus step is the distinction people mix up. next runs a function call as one operation; step goes inside it.

finish runs to the end of the current function and prints the return value, which is useful when you have stepped into something by accident.

Reading a backtrace

This is the single most valuable thing gdb does.

(gdb) bt
#0  0x00007ffff7a4e2a1 in __strlen_avx2 () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000555555555234 in process_name (name=0x0) at myprog.c:42
#2  0x00005555555552f8 in handle_request (req=0x5555555592a0) at myprog.c:67
#3  0x0000555555555401 in main (argc=2, argv=0x7fffffffe3c8) at myprog.c:89

Read it bottom to top: main called handle_request, which called process_name, which called strlen and crashed.

Frame 1 is the important one: name=0x0. A null pointer was passed to process_name, and strlen dereferenced it. The bug is in whatever should have set name, not in strlen.

Move between frames and inspect:

(gdb) frame 2
(gdb) print *req
(gdb) info locals
(gdb) info args

bt full prints local variables for every frame at once, which is frequently faster than walking them.

Core dumps

A core dump is the memory image of a crashed process. It lets you debug a crash after the fact, which is the only practical approach for something that fails rarely or only in production.

Enabling them

ulimit -c unlimited              # this shell only
ulimit -c                        # check

Permanently:

# /etc/security/limits.conf
* soft core unlimited

For a systemd service:

[Service]
LimitCORE=infinity

Our ulimit guide covers the wider set of limits.

Finding them

On systemd systems, cores go to the journal rather than the working directory, which is why people conclude core dumps are disabled when they are not:

coredumpctl list
coredumpctl info 12345
coredumpctl gdb 12345         # opens it in gdb directly
coredumpctl dump 12345 > core.12345

coredumpctl gdb is the fast path: it finds the core, finds the binary, loads both.

Where cores go is controlled by:

cat /proc/sys/kernel/core_pattern
# |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

The leading pipe means the core is handed to that program rather than written to a file.

Debugging a core from another machine

The binary must match exactly. A core dump loaded against a different build produces a backtrace that is confidently wrong, with plausible-looking function names at incorrect offsets, which wastes a great deal of time.

gdb ./myprog core.12345
(gdb) bt full

Collect three things from the affected host: the core, the exact binary, and its debug symbols.

Attaching to a running process

gdb -p 4823

The process pauses while you inspect and resumes on continue or detach.

Most distributions restrict this:

cat /proc/sys/kernel/yama/ptrace_scope
# 1 = only children may be traced

Either use root, or lower it temporarily:

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Set it back afterwards. The restriction exists because ptrace on arbitrary processes lets one compromised process read another’s memory, including credentials.

This is the technique for a hung process: attach, bt, and see what it is waiting on.

Watchpoints and conditionals

(gdb) watch counter                  # break when counter changes
(gdb) break myprog.c:42 if x > 100   # conditional breakpoint
(gdb) break process_name if name == 0
(gdb) info breakpoints
(gdb) delete 2

Conditional breakpoints are what make gdb usable on a loop that runs a million times and fails once. Hardware watchpoints are fast; software ones are extremely slow, and gdb tells you which you got.

Configuration worth having

# ~/.gdbinit
set print pretty on
set pagination off
set history save on
set confirm off

set print pretty on formats structs readably, which makes a genuine difference.

GEF, pwndbg, and gdb-dashboard are front ends that show registers, stack, and disassembly automatically on every stop. Worth installing if you do this often.

For a terminal interface built in:

(gdb) layout src
(gdb) layout split

Where gdb is the wrong tool

Memory errors. Valgrind or AddressSanitizer find use-after-free and buffer overruns far better, because gdb only shows you the crash and not the earlier corruption that caused it.

gcc -fsanitize=address -g -o myprog myprog.c
valgrind --leak-check=full ./myprog

Syscall-level questions. Our strace and ltrace guide covers those, and eBPF and bpftrace covers doing it without stopping the process.

Performance. perf and profilers, not a debugger.

gdb is for “what is the state of this program at this moment”. For that, nothing else comes close.

Frequently Asked Questions

Why does my gdb backtrace show question marks instead of function names?

The binary has no debug symbols, either because it was compiled without -g or because they were stripped. Install the matching debug symbol package from your distribution, usually named with a -dbg or -debuginfo suffix, or rebuild with -g and without stripping.

Where do core dumps go on a modern Linux system?

On systems using systemd they go to the journal and are read with coredumpctl rather than appearing as files in the working directory. Run coredumpctl list to see recent crashes and coredumpctl gdb to open the most recent one directly in the debugger.

Can I debug a program that is already running?

Yes. Use gdb -p followed by the process ID to attach to a running process, which pauses it while you inspect. On most distributions ptrace_scope restricts this to child processes, so you either need root or must temporarily lower the yama ptrace_scope setting.

Should I compile with optimization when debugging?

Prefer -Og, which enables optimizations that do not interfere with debugging. Full -O2 reorders and inlines code so the debugger reports line numbers that do not correspond to what you wrote, and variables are frequently reported as optimized out.

What is the difference between step and next in gdb?

step enters function calls, stopping at the first line inside the called function. next executes the call as a single operation and stops on the following line in the current function. Use next to move through a function and step when you want to descend into a call.

How do I debug a crash I cannot reproduce locally?

Enable core dumps on the affected system, retrieve the core file along with the exact binary and its debug symbols, and load all three into gdb. The binary must match precisely, because a core dump from a different build produces a backtrace that is wrong in ways that are not obvious.