Ubuntu 16.04 Killed My Upstart Job: Converting an init Script to systemd

A client in Nairobi asked me to move their invoicing app from Ubuntu 14.04 to 16.04 over a weekend. The upgrade itself was uneventful. The reboot was not. The machine came back, Nginx came back, the database came back, and the application that pays the bills did not. I ran the command I had run a thousand times before and got this:

$ sudo initctl status invoices
initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

That message is the whole story. Ubuntu 16.04 is the first long term support release where systemd is PID 1, not Upstart. Upstart is not crashed, not misconfigured, not missing a package. It is simply not running, and nothing is reading the job file in /etc/init/ that has started this app since 2014.

This is the first thing I am publishing here, and it is a good one to start with, because it is the shape of most of the work I get called for: nothing is broken, something changed, and the documentation on the internet is one release out of date. My position is that the conversion is not a chore to be survived. An Upstart job describes how to launch a process. A systemd unit describes how a process fits into the machine, and you get ordering, restart policy, resource limits and logs for the same twenty lines of text.

sudo systemctl daemon-reload
sudo systemctl enable --now app.service
systemctl status app.service
sudo journalctl -u app.service -n 50 --no-pager

Why /etc/init stopped working but /etc/init.d still does

This trips people up, because half a migrated server keeps working and the other half does not, which feels random. It is not.

Ubuntu switched its default init system to systemd in 15.04, and 16.04 is the first LTS to carry that change. systemd was built knowing it would inherit decades of SysV init scripts, so it ships a generator that reads /etc/init.d/ at boot and synthesises a unit for each script it finds. Your ancient /etc/init.d/legacy-thing therefore still starts, and service legacy-thing restart still works, because service is now a compatibility wrapper that forwards to systemctl.

There is no such generator for Upstart. The .conf files in /etc/init/ are in a format only Upstart parses, and Upstart is no longer running to parse them. So the SysV script from 2009 survives the upgrade and the modern Upstart job from 2014 does not. That is the joke.

The old job, and the unit that replaces it

Here is a representative Upstart job, close enough to the one I found on that server:

description "Invoices API"

start on runlevel [2345]
stop on runlevel [!2345]

setuid invoices
setgid invoices
chdir /srv/invoices

env NODE_ENV=production
env PORT=8080

respawn
respawn limit 10 5

exec /usr/bin/node server.js

Every stanza has a systemd equivalent. This is the translation table I keep coming back to:

Applied to the job above, the unit is:

[Unit]
Description=Invoices API
After=network.target postgresql.service

[Service]
Type=simple
User=invoices
Group=invoices
WorkingDirectory=/srv/invoices
Environment=NODE_ENV=production
Environment=PORT=8080
ExecStart=/usr/bin/node /srv/invoices/server.js
Restart=on-failure
RestartSec=5
StartLimitBurst=10
StartLimitInterval=60

[Install]
WantedBy=multi-user.target

One caveat on the last two lines: 16.04 ships systemd 229, where the start rate limit belongs in [Service]. Newer systemd releases moved it to [Unit], so if you copy this unit onto a much newer machine one day, check systemctl --version before you trust the placement.

Two things in there are not a straight translation, and they are the two that matter.

ExecStart uses absolute paths for both the interpreter and the script. Upstart ran your job through a shell that had inherited a reasonable PATH. systemd does not run a login shell at all, so ~/.profile, ~/.bashrc and anything nvm put there do not exist as far as your service is concerned. If you cannot bring yourself to hardcode the path, at least set it explicitly with Environment=PATH=....

And respawn became Restart=on-failure, not Restart=always. on-failure restarts on a non-zero exit or a signal, and leaves a clean exit alone, which is what you want for anything that can legitimately be told to shut down.

Step 1: Find what is actually missing

Before writing anything, list the Upstart jobs the machine still carries. They are all now inert:

ls -1 /etc/init/*.conf

Ignore the ones shipped by distribution packages, which have systemd units of their own already. What you care about are the files you or a predecessor wrote by hand for applications.

Step 2: Write the unit

sudo nano /etc/systemd/system/invoices.service

Unit files you write live in /etc/systemd/system/. Files under /lib/systemd/system/ belong to packages and will be overwritten on upgrade, so never edit those directly.

Step 3: Load it and start it

sudo systemctl daemon-reload
sudo systemctl enable --now invoices.service

daemon-reload is the step people forget: systemd caches unit files and will keep using the old copy until you tell it to re-read the directory. enable creates the symlink that makes the service start at boot, and --now also starts it immediately. Expected output from enable:

Created symlink from /etc/systemd/system/multi-user.target.wants/invoices.service to /etc/systemd/system/invoices.service.

Step 4: Retire the old job

sudo mv /etc/init/invoices.conf /root/invoices.conf.upstart-backup

Keep the backup for a week, then delete it. Leaving a dead .conf in place guarantees that in eighteen months somebody edits it, restarts nothing, and cannot work out why their change had no effect.

Verification

Three commands, in this order:

systemctl is-enabled invoices.service   # -> enabled
systemctl is-active invoices.service    # -> active
sudo journalctl -u invoices.service -n 20 --no-pager

Then do the thing nobody wants to do on a Friday: reboot the machine and check again. A service that starts when you type systemctl start and not at boot is a service that will be down after the next power cut, and where I work the power cut is not hypothetical.

What people get wrong

Reinstalling upstart-sysv to make Upstart PID 1 again. It is possible, and it is a trap. You are now running a configuration that receives less testing than any other Ubuntu install on the planet, and every future package that ships a systemd unit gets slightly stranger. You are also postponing an hour of work to a moment when you will have less time.

Putting the start command in /etc/rc.local with an ampersand. It starts the process at boot, so it looks like it works. Nothing supervises it, nothing restarts it when it dies, nothing captures its output, and stopping it means finding the PID by hand. This is not a service, it is a process that happens to exist.

Running the app in screen or tmux. Same problem with an extra layer, and it survives exactly until the next reboot, at which point you find out it was never going to come back on its own.

Reaching for Restart=always immediately. Blanket restarts turn a crash into a busy loop that hides the crash. Start with on-failure, and if you find yourself raising the restart limits, go and read the journal instead. The restart is not the fix.

When it is still broken

systemctl status gives you an exit code, and the codes are specific:

• status=203/EXEC - systemd could not execute the binary in ExecStart. Wrong path, not executable, or a script without a shebang line.
• status=200/CHDIR - WorkingDirectory does not exist, or the service user cannot enter it.
• status=217/USER - the user named in User= does not exist on this machine. Common after restoring a unit file from another server.
• status=1/FAILURE - the process ran and exited non-zero. This one is genuinely your application, so go to the journal.

Two more tools worth knowing. systemctl cat invoices.service prints the unit exactly as systemd has it, which settles arguments about whether your edit was loaded. And systemd-analyze verify /etc/systemd/system/invoices.service parses the file and complains about typos in directive names, which is how you discover that WorkingDirectory with a lowercase d was silently ignored all afternoon.