Too many open files is one of the most common errors on a busy Linux server, and the fix is usually described as “run ulimit -n 65535.” That works in your shell and does nothing for the service that was failing, which is where the confusion starts.

Soft and hard limits

Every limit has two values. The soft limit is what is actually enforced. The hard limit is the ceiling the soft limit may be raised to.

Any user may lower either, or raise the soft limit up to the hard limit. Only root may raise the hard limit.

ulimit -a            # all soft limits
ulimit -Ha           # all hard limits
ulimit -n            # soft limit on open files
ulimit -Hn           # hard limit on open files

The limits worth knowing

-n, open files. The one that bites. Every socket, every open file, every pipe counts. A web server with thousands of connections needs thousands of descriptors.

-u, processes. Caps processes per user, which is basic fork bomb protection.

-c, core file size. Often 0, which is why a crash produces no core dump to debug.

-v and -m, virtual and resident memory. Rarely used now, since cgroups do this better.

-s, stack size. Occasionally relevant for deeply recursive programs.

Setting them for a shell

ulimit -n 8192          # this shell and its children, until it exits

To persist for login sessions, use /etc/security/limits.conf or a drop-in:

# /etc/security/limits.d/99-custom.conf
alice    soft    nofile    8192
alice    hard    nofile    65535
*        soft    nofile    4096
*        hard    nofile    65535
@devs    soft    nofile    8192

Columns are domain, type, item, value. A @ prefix means a group.

This works through pam_limits, so it applies to anything going through PAM: SSH logins, console logins, su. Confirm PAM is configured to use it:

grep pam_limits /etc/pam.d/*

Why services ignore all of that

A systemd service is not a login session. It does not go through PAM, does not read your shell profile, and does not consult limits.conf.

Set the limit in the unit:

sudo systemctl edit nginx

[Service]
LimitNOFILE=65535
LimitNPROC=4096
LimitCORE=infinity

sudo systemctl daemon-reload
sudo systemctl restart nginx

For a default across all units, use /etc/systemd/system.conf:

DefaultLimitNOFILE=65535:524288

The colon form is soft:hard.

This single point accounts for most of the wasted time on this topic. Someone raises limits.conf, restarts the service, sees no change, raises it further, and never touches the file that matters.

Verifying, properly

Do not check ulimit in your shell and assume it reflects the service. Read the process directly:

cat /proc/$(pgrep -f nginx | head -1)/limits

That is authoritative. It shows soft and hard values for every limit as the kernel sees them for that process.

Count what it is actually using:

ls /proc/1234/fd | wc -l

Comparing that number with the limit tells you whether you have a capacity problem or a leak.

The system-wide ceiling

Separately from per-process limits there is a global one:

sysctl fs.file-max          # system-wide maximum
cat /proc/sys/fs/file-nr    # allocated, free, maximum

Raise it if needed via sysctl:

fs.file-max = 2097152

A per-process limit above the system maximum accomplishes nothing, which is the other half of why raising one number sometimes has no effect.

Leak or capacity

Before raising a limit, work out which problem you have.

# Watch a process's descriptor count over time
watch -n 5 'ls /proc/1234/fd | wc -l'

Steady growth that never falls is a leak. The application opens something and never closes it, and a higher limit only delays the failure. lsof will show you what it is holding:

sudo lsof -p 1234 | awk '{print $5}' | sort | uniq -c | sort -rn

A large count of sock entries usually means unclosed connections. A large count of REG entries means files.

A count that rises with load and falls afterwards is genuine capacity, and raising the limit is the correct response.

Sensible values

For a busy web server or proxy, 65535 open files is a reasonable and common setting. Descriptors are cheap; the historical defaults of 1024 come from an era of very different hardware.

For a database, follow the vendor’s recommendation, which is usually specific and usually higher than you expect.

For a desktop, the defaults are fine and you will never notice.

Frequently Asked Questions

What is the difference between a soft and hard limit?

The soft limit is what is enforced. The hard limit is the ceiling the soft limit can be raised to. Any user can lower either or raise the soft limit up to the hard limit, but only root can raise the hard limit.

Why does my ulimit setting not apply to my service?

Because systemd services do not run a login shell and never read your shell profile or limits.conf in the way you expect. Set LimitNOFILE in the unit file instead, which is the only place that reliably applies to a service.

What causes too many open files?

Either a genuine need for more descriptors, common on busy servers, or a file descriptor leak where a program opens files and never closes them. Check the count against the limit with ls /proc/PID/fd before raising anything.

Does raising the file descriptor limit have a downside?

Very little. Descriptors are cheap and modern defaults are conservative for historical reasons. The real risk is masking a leak, which will simply take longer to hit the new ceiling and fail at a less convenient time.

Where does limits.conf actually apply?

Through pam_limits, which means it applies to login sessions: SSH logins, console logins, and anything going through PAM. It does not apply to systemd services, which is the single most common misunderstanding about it.

How do I see the limits of a running process?

Read /proc/PID/limits, which shows every limit with its current soft and hard values. This is authoritative, unlike checking ulimit in your shell, which tells you only about your shell.