Your App Keeps Dying: A systemd Unit File That Actually Keeps It Running

Karani GeoffreyJuly 21, 20268 min read

Every team I have helped in Nairobi has the same file somewhere on their server. It is four lines long, it lives in /etc/systemd/system/, and it was copied from a Stack Overflow answer in 2019. It works right up until the night it does not, and then nobody can explain why the app is "running" but the port is closed.

My thesis is simple and slightly unpopular: Restart=always is not reliability. It is a way of not finding out why your app crashed. A good unit file does four jobs - it starts the process correctly, it restarts it with a backoff that gives dependencies time to recover, it stops the process from eating the whole machine, and it makes the failure legible in the journal. Let us build one.

The unit file everyone copies, and why it bites

[Unit]
Description=My app

[Service]
ExecStart=node server.js
Restart=always

[Install]
WantedBy=multi-user.target

Three real bugs are hiding here. ExecStart uses a relative path and a bare command name - systemd does not run a login shell, so your PATH is not your PATH, and nvm does not exist. There is no User=, so this runs as root. And Restart=always with the default one-hundred-millisecond RestartSec means a crash-looping app will hammer your database a hundred times a second until systemd's start rate limiter gives up and leaves it dead.

The unit file I actually deploy

[Unit]
Description=Billing API (Node)
Documentation=https://runbooks.example.co.ke/billing
After=network-online.target postgresql.service
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=simple
User=billing
Group=billing
WorkingDirectory=/srv/billing
EnvironmentFile=/etc/billing/env
ExecStart=/usr/bin/node /srv/billing/server.js
ExecReload=/bin/kill -HUP $MAINPID

Restart=on-failure
RestartSec=5s
RestartSteps=5
RestartMaxDelaySec=120
TimeoutStopSec=30

# Resource ceilings
MemoryHigh=640M
MemoryMax=768M
TasksMax=256
CPUQuota=150%
OOMPolicy=stop

# Sandbox
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/srv/billing/uploads /var/lib/billing
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictNamespaces=true
LockPersonality=true

[Install]
WantedBy=multi-user.target

Now the reasoning, line by line, because copying this without understanding it puts you back where you started.

StartLimit lives in [Unit], not [Service]

This trips up a lot of people. StartLimitIntervalSec and StartLimitBurst moved to the [Unit] section years ago. Older systemd parses them in [Service] for compatibility but logs a warning and the semantics get confusing. Put them in [Unit]. What they mean: "if this unit is started more than 5 times within 300 seconds, stop trying." When you hit it you get the message everyone has seen at 2am:

Jul 21 02:14:09 web01 systemd[1]: billing.service: Start request repeated too quickly.
Jul 21 02:14:09 web01 systemd[1]: billing.service: Failed with result 'exit-code'.
Jul 21 02:14:09 web01 systemd[1]: Failed to start billing.service - Billing API (Node).

The unit is now in a failed state and will not be retried, ever, until you clear it:

sudo systemctl reset-failed billing.service
sudo systemctl start billing.service

That rate limiter is a feature. It is systemd telling you the problem is not transient. If you find yourself raising StartLimitBurst to 50, you are silencing a smoke alarm.

Restart=on-failure plus exponential backoff

Restart=on-failure restarts on a non-zero exit code, an uncaught signal, a watchdog timeout, or a timeout on start/stop. It does not restart when your app exits cleanly with status 0, which is exactly what you want - a clean exit is usually a deliberate shutdown or a migration script finishing.

RestartSec=5s sets the base delay. On systemd 254 and newer (Ubuntu 24.04 LTS ships 255, check with systemctl --version) you also get RestartSteps= and RestartMaxDelaySec=, which turn that flat 5 seconds into a geometric backoff from 5s up to 120s over 5 steps. This matters on a small VPS: if Postgres is recovering from an unclean shutdown, a backoff gives it room instead of drowning it in connection attempts.

Rule of thumb: use Restart=on-failure for services, Restart=always only for things that legitimately exit 0 and must come straight back (rare), and Restart=no for one-shot migration units.

Type= is about when systemd thinks you are "up"

Type=simple - systemd considers the service started the moment it forks the process. Fast, and lying. Anything ordered After= your service will start before your app has bound its port.
Type=exec - like simple, but systemd waits until the binary has actually been executed successfully. Catches "binary not found" as a start failure. A strictly better default than simple.
Type=notify - your app calls sd_notify(READY=1) when it is genuinely ready to serve. This is the correct answer, and libraries exist for Python, Go and Node.
Type=forking - for old daemons that background themselves. Requires PIDFile=. Avoid if you can.

If you cannot use notify, at least use exec. If your dependent service needs your app to be listening, use a socket unit or a readiness check in ExecStartPost= rather than pretending simple means ready.

Resource caps: the setting that saves the whole box

The most common way a Kenyan VPS falls over is not a security incident. It is one Node process leaking until the kernel OOM killer picks a victim - and the kernel often picks MariaDB, because MariaDB is the biggest thing in memory. MemoryMax=768M puts the leaking service in a cgroup with a hard ceiling so the OOM kill happens inside that cgroup and takes down only the guilty party.

MemoryHigh= is the softer sibling: crossing it throttles the process and pushes it to reclaim, rather than killing it. Set MemoryHigh about 15 percent below MemoryMax so you get backpressure before execution. Watch it live:

systemd-cgtop -m
cat /sys/fs/cgroup/system.slice/billing.service/memory.current
cat /sys/fs/cgroup/system.slice/billing.service/memory.events

memory.events gives you counters for high, max, oom and oom_kill. If oom_kill is non-zero, your cap is too tight or your app leaks. That is a real number you can graph, which is more than most teams have.

Sandboxing without breaking your app

ProtectSystem=strict mounts the entire filesystem read-only for this service except /dev, /proc, /sys, and whatever you list in ReadWritePaths=. This is the single highest-value hardening line, and it is also the one that will break your deploy at 11pm if the app writes a cache file somewhere you forgot. Roll it out on staging first and watch for EROFS errors.

Score your unit before and after:

systemd-analyze security billing.service
systemd-analyze verify /etc/systemd/system/billing.service

One warning: do not add MemoryDenyWriteExecute=true to anything with a JIT. Node's V8, the JVM, PyPy and .NET all need writable-executable pages. You will get an instant, confusing crash. The same caution applies to aggressive SystemCallFilter= lists - start with SystemCallFilter=@system-service and nothing more.

Exit codes that tell you exactly what is wrong

When systemctl status shows a numeric status like 203/EXEC, that is systemd's own vocabulary, not your app's. Learn these four and you will solve most unit-file bugs in ten seconds:

Status	Meaning	Fix
200/CHDIR	WorkingDirectory does not exist	Create it, or fix the path
203/EXEC	ExecStart binary missing or not executable	Use an absolute path, `chmod +x`
217/USER	The User= does not exist	`useradd --system --no-create-home billing`
226/NAMESPACE	A sandbox path is invalid	A directory in ReadWritePaths= is missing

Here is what that looks like in the wild:

$ systemctl status billing
× billing.service - Billing API (Node)
     Loaded: loaded (/etc/systemd/system/billing.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2026-07-21 02:14:09 EAT; 3min ago
    Process: 21455 ExecStart=/usr/bin/node /srv/billing/server.js (code=exited, status=203/EXEC)
   Main PID: 21455 (code=exited, status=203/EXEC)
        CPU: 4ms

That is not a Node problem. /usr/bin/node does not exist because the app was installed with nvm under a user's home directory. Either install Node system-wide or point ExecStart at the real interpreter path.

Never edit vendor units - use drop-ins

If the package ships the unit (nginx, mariadb, redis), do not edit the file in /lib/systemd/system/. It will be overwritten on upgrade. Use a drop-in:

sudo systemctl edit mariadb.service

That opens an empty override at /etc/systemd/system/mariadb.service.d/override.conf. Add only what you are changing:

[Service]
MemoryMax=1200M
LimitNOFILE=32768

Then confirm what systemd has actually merged - this is the command that ends arguments:

systemctl cat mariadb.service
systemctl show mariadb.service -p MemoryMax -p Restart -p RestartSec

The deploy loop

sudoedit /etc/systemd/system/billing.service
sudo systemd-analyze verify /etc/systemd/system/billing.service
sudo systemctl daemon-reload
sudo systemctl enable --now billing.service
journalctl -u billing.service -f -n 50

Forgetting daemon-reload is the most common self-inflicted wound in this whole article. systemd will happily keep running the old unit and you will swear the file is not being read.

Environment files, not hardcoded secrets

sudo install -o root -g billing -m 0640 /dev/null /etc/billing/env
sudo tee /etc/billing/env > /dev/null <<'EOF'
NODE_ENV=production
PORT=8081
DATABASE_URL=postgres://billing:s3cr3t@127.0.0.1:5432/billing
EOF

No quotes needed and no shell expansion happens - systemd is not bash. ${VAR} inside the file will not be expanded, and a trailing comment on a value line becomes part of the value. Mode 0640 with group ownership means the service can read it and other users on the box cannot.

What "healthy" should mean to you

A service that systemd reports as active (running) can still be completely useless - deadlocked, out of database connections, serving 502s. If your app supports it, wire up the watchdog:

Type=notify
WatchdogSec=30s
Restart=on-failure

Your app must then call sd_notify(WATCHDOG=1) at least every 15 seconds from a code path that proves it is alive - inside the request loop, not from a background timer that keeps ticking while the pool is exhausted. If the ping stops, systemd kills and restarts the service. That is the difference between "the process exists" and "the service works", and it is the whole point.

Start with the unit file above. Add the sandbox lines one at a time. Look at memory.events once a week. You will spend far fewer nights guessing.

#systemd#linux#devops#vps#reliability