'Too Many Open Files' Won't Go Away? You Are Editing the Wrong Limit

The box has 16GB of RAM free, the CPUs are half idle, and your app is falling over. The logs fill with Too many open files, requests start timing out, and the database throws connection errors it never threw before. So you do the obvious thing: you edit /etc/security/limits.conf, bump the limit to 65536, save it, restart the service, and watch it happen again. Nothing changed.

You are editing the wrong limit. That file is real, the number in it is real, and for a service started by systemd it does almost nothing. This is the single most common dead end with this error, and it is why the internet is full of "raise your ulimit" answers that quietly do not work for the people who need them most.

Here is the actual layering, how to set the limit where your process will really see it, and how to prove the number reached the process instead of hoping it did.

sudo systemctl edit your-service
# add:
# [Service]
# LimitNOFILE=65536
sudo systemctl daemon-reload
sudo systemctl restart your-service

The exact error: Too many open files

It shows up in several shapes depending on who hit the wall first, but the phrase is the same:

socket: Too many open files
accept4: Too many open files
OSError: [Errno 24] Too many open files
java.io.IOException: Too many open files
[emerg] 1234#1234: too many open files

Every one of these is the kernel returning EMFILE: the calling process has hit its per-process cap on open file descriptors. On Linux a "file" here means far more than files on disk. Every open socket, every pipe, every database connection, every log handle counts. A busy web app with a connection pool and a few thousand concurrent sockets can blow through a 1024 default without touching a single file on disk.

Why raising limits.conf does nothing for a systemd service

There are several independent places a file-descriptor limit can be set, and they do not talk to each other. The number that matters is the one on the process that actually crashed, and only one layer sets that number for any given process.

/etc/security/limits.conf is enforced by PAM, the pluggable authentication stack. It applies when a user logs in: an SSH session, a console login, a su or sudo -i. That is why the classic test is so misleading. You SSH in, run ulimit -n, see your shiny 65536, and conclude the change worked. But your nginx or your app was not started by a login. It was started by systemd at boot, which never went through PAM, so it never read limits.conf at all. The limit your service runs under was decided entirely by systemd.

There is a second trap even after you find systemd. Modern systemd (since v240) hands services a generous hard limit, but the soft limit, the one most programs actually respect unless they deliberately raise it themselves, is commonly still 1024. A process sees the soft limit as its ceiling. So "systemd already sets a high limit" is only half true: the hard cap is high, the effective soft cap is often low, and LimitNOFILE= is what raises both. Never assume; the /proc/<pid>/limits file shows you both columns, and that is the only number that counts.

The fix, step by step

Step 1: Find the process and read its real limit

Before changing anything, see what the crashing process is actually running under. Do not read a fresh shell's ulimit; read the process itself:

pidof nginx        # or: pgrep -f your-app  -> gives a PID, e.g. 1234
cat /proc/1234/limits | grep 'open files'
# Max open files            1024      524288    files

The two numbers are soft and hard. If the first column is 1024, that is your ceiling regardless of how much RAM the box has, and regardless of what limits.conf says.

Step 2: Raise the limit on the systemd unit

systemctl edit creates a drop-in override so you are not hand-editing a vendor unit file that a package update will overwrite:

sudo systemctl edit nginx

In the editor that opens, add exactly this:

[Service]
LimitNOFILE=65536

Save and close. This writes /etc/systemd/system/nginx.service.d/override.conf. Setting a single value applies it to both soft and hard limits, which is what you want for a service. If you need them different, use LimitNOFILE=soft:hard, for example LimitNOFILE=65536:524288.

Step 3: Reload systemd and restart the service

sudo systemctl daemon-reload
sudo systemctl restart nginx

A restart is mandatory, not optional. The limit is fixed at process start; daemon-reload alone updates systemd's view but does not re-limit an already-running process. This is the same restart-is-required discipline as any other unit change, which is why a unit file that actually keeps your app running is worth getting right once.

Step 4: If a supervisor sits in between, fix it there too

systemd sets the limit on the process it launches directly. If that process is a supervisor and your real workers are its children, the supervisor's own limit is what the workers inherit.

• pm2: pm2 caps its children unless raised. Set the limit for the pm2 daemon itself (its systemd unit, pm2-<user>.service, gets the same LimitNOFILE= treatment), then pm2 restart all --update-env so workers re-fork under the new ceiling.
• supervisord: add minfds=65536 under [supervisord] in supervisord.conf, then restart supervisord. Its programs inherit that.
• Docker: the daemon-managed default may be low. Set it per container with --ulimit nofile=65536:65536, or in compose under ulimits: nofile:, and confirm inside the container with the same /proc/1/limits check.

Verification

Read the limit off the running worker again after the restart. This is the whole game: not what a config file says, but what the process reports:

cat /proc/$(pgrep -f your-app | head -1)/limits | grep 'open files'
# Max open files            65536     65536     files

Then watch the actual descriptor count under load so you know you have headroom, not just a bigger number:

# how many fds this process currently holds
ls /proc/$(pgrep -f your-app | head -1)/fd | wc -l
# or, more detail:
lsof -p $(pgrep -f your-app | head -1) | wc -l

If that count climbs toward your new limit and keeps climbing, keep reading; you have a leak, not a limit.

What people get wrong

Editing limits.conf and stopping there. Already covered, but it earns repeating because it is where most of an evening goes. limits.conf governs login sessions. Your service is not a login session. If you take one thing from this article: verify with /proc/<pid>/limits, never with ulimit -n in an unrelated shell.

Setting fs.file-max and thinking that is the fix. fs.file-max in sysctl is the system-wide ceiling across all processes combined. Raising it does not raise your per-process limit; the per-process cap is almost always the one you are hitting first. Set it only if the system-wide number is genuinely small, which on a modern box it is not.

Treating a raised limit as the cure when it is really a leak. If descriptors only ever grow and never drop, some code path is opening sockets or files and never closing them. Raising the limit just moves the crash from Tuesday to Friday. A connection that is never returned to the pool, a client that is created per request instead of reused, an unclosed file in a loop: these are the usual suspects. This mirrors the memory story where a bad limit and a real leak look identical until you measure, which is exactly the trap in OOMKilled exit code 137. Measure the trend before you decide.

When it is still broken

• The number did not change on the process. Re-read /proc/<pid>/limits. If it still says 1024, you edited a different unit than the one running, or you forgot the restart, or a supervisor is overriding it. Confirm the exact unit with systemctl status your-service and check its Drop-In: line lists your override.
• It is a per-user limit, and the service runs as that user via a login shell. Some setups launch through su - appuser, which does go through PAM. In that narrow case limits.conf does apply, and you set both. Check how the process is actually started before assuming.
• It is a leak. Sample ls /proc/<pid>/fd | wc -l every minute under load. A steady climb that never recovers is a leak. Then lsof -p <pid> and look at what dominates: hundreds of CLOSE_WAIT sockets point at connections you open and forget to close; repeated identical file paths point at a handle opened in a hot loop.
• The database is the one complaining. If MySQL or MariaDB logs it, the fix is the same LimitNOFILE on the database unit, plus the engine's own open_files_limit, which the server caps to what the OS grants it at startup. Raise the OS limit first, then the engine setting, then restart.

The discipline that ends this class of incident is boring and reliable: find the actual PID, read its actual limit, change the limit at the layer closest to that PID, restart, and read the limit again. Config files lie about what a running process sees. The /proc filesystem does not.