Read-Only File System on a Live Server: Diagnose Before You Remount

Your application starts failing writes. Logs stop appending. A deploy dies with "Read-only file system". You SSH in, try to touch a file, and the shell tells you the same thing. The server is up, it answers pings, but it has quietly turned itself into a read-only appliance while you were asleep.

Somewhere on the internet is the answer everyone reaches for first: mount -o remount,rw /. It often appears to work. The filesystem goes writable again, the app recovers, everyone moves on. And that is exactly the trap, because the kernel did not do this at random. It flipped to read-only on purpose, as a last-ditch move to stop a failing disk from corrupting your data further. Remounting rw without asking why is telling the smoke alarm to be quiet while the kitchen is on fire.

Before you remount anything, you spend ninety seconds finding out what triggered it. That decision, diagnose first, is the entire difference between a five-minute recovery and a restored-from-backup afternoon.

The exact message you are seeing

In the kernel log it looks like one of these:

EXT4-fs error (device dm-1): ext4_journal_check_start:83: Detected aborted journal
EXT4-fs (dm-1): Remounting filesystem read-only

And every write, from your app or your shell, comes back as:

touch: cannot touch '/var/tmp/test': Read-only file system

That second line is the symptom you searched for. The first two lines are the actual event, and they are the ones that tell you whether this is a five-minute fix or a failing disk. Most write-ups quote only the symptom, which is why most write-ups give you the dangerous advice.

Why the kernel does this on purpose

ext4 (and ext3 before it) carries an error policy in the superblock, mirrored by the errors= option in /etc/fstab. The default on most distributions is errors=remount-ro: if the filesystem hits an error it cannot safely continue through, remount it read-only immediately. The logic is brutal and correct: a filesystem that keeps writing after it has detected inconsistency turns a recoverable problem into an unrecoverable one. Read-only freezes the damage.

So the remount is an effect. The cause is whatever the kernel logged just before it: an aborted journal, a block I/O error, a device that stopped responding, or metadata that failed a consistency check. On a cloud VPS the underlying block storage is virtualised, so the failure can originate on the hypervisor side and still surface to your guest kernel as an I/O error. Same symptom, and you still must not just remount and walk away.

There is a modern wrinkle worth knowing. On newer kernels (5.10 and later) the behaviour around auto-remounting read-write on reboot changed, so admins who remember old boxes silently coming back writable can be surprised when a reboot leaves the filesystem read-only or forces a check instead. Do not rely on "a reboot will sort it out". Find the cause.

The fix, step by step

Step 1: Read the kernel log before touching anything

The single most important command. Look at what the kernel said, timestamped:

dmesg -T | tail -40
journalctl -k -b | grep -i -E "ext4|i/o error|remount|ata|nvme"

You are triaging between two worlds:

What the log shows	What it means	Next move
I/O error, ata bus error, nvme ... timeout, sector read failures	Hardware or block storage is failing	Stop writing, image the disk, replace or migrate
aborted journal after a hard power loss, no I/O errors	Likely a transient inconsistency	fsck offline, then remount
Repeated remount-ro right after each rw attempt	Underlying fault is still active	Do not keep remounting; go to rescue mode

If there are no I/O errors at all and the trigger was a single unclean shutdown, you are probably in the recoverable world. If you see sector-level read failures or the controller resetting, you are not, and every write you attempt makes recovery harder.

Step 2: Check the disk's own health

Ask the drive whether it is dying. SMART is authoritative in a way that guessing is not:

sudo smartctl -a /dev/sda   # or /dev/nvme0, etc.

Watch Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable, and the overall health assessment. Non-zero and climbing pending sectors, or a failed self-assessment, means the hardware is the cause and no amount of remounting fixes it. On a cloud VPS you may not get real SMART data because the disk is virtual; there, correlate with your provider's status page and the guest dmesg I/O errors instead.

Step 3: If the cause is transient, fsck offline

You cannot safely repair a mounted filesystem, and you must never try. Repairing metadata while the kernel is also using it corrupts it further. For a non-root filesystem, unmount and check:

sudo umount /dev/sdb1
sudo fsck -y /dev/sdb1

For the root filesystem you cannot unmount it while running from it. Boot into the rescue/single-user environment (your provider's rescue console, or a recovery ISO) where root is not mounted, then fsck it there. Scheduling a check at next boot with touch /forcefsck or fsck.mode=force on the kernel command line is the safe way to have init run fsck before root is mounted rw.

Step 4: Only now, remount read-write

If, and only if, the log showed a transient cause, SMART is clean, and fsck completed without unfixable errors, bring the filesystem back:

sudo mount -o remount,rw /

If it flips straight back to read-only, stop. That is the kernel telling you the fault is still live. Do not loop on remounting; go back to rescue mode.

Verifying it actually recovered

Prove writes work and that the mount is genuinely rw, not just quiet for a moment:

touch /var/tmp/rw-test && echo "writable" && rm /var/tmp/rw-test
findmnt -no OPTIONS /    # should list rw, not ro

Then confirm no new errors are being logged under real load:

dmesg -T --level=err,warn | tail -20

A clean write test plus a silent kernel log over a few minutes of normal traffic is your evidence. A write test that passes while dmesg is still spitting I/O errors is not recovery; it is borrowed time.

What people get wrong

Remounting rw as the first move. This is the headline mistake. It clears the symptom and lets a failing disk keep writing, converting recoverable data loss into total loss. The remount is step four, not step one, and it is conditional on everything above it.

Running fsck on the mounted root filesystem. People try fsck / on a live system to "fix it in place". fsck on a mounted, in-use filesystem can and does corrupt it. Root must be checked from an environment where it is not mounted rw, meaning rescue or single-user mode.

Assuming a reboot fixes it. On older boxes a reboot often came back writable and people learned to treat that as the fix. On current kernels the reboot behaviour changed, and rebooting a failing disk just interrupts your diagnosis. If the cause is hardware, the reboot buys nothing and may lose the log you needed.

Editing fstab to errors=continue so it stops happening. This is disabling the safety net because the alarm is annoying. errors=continue lets the filesystem keep operating through detected inconsistencies, which is precisely how you turn a bad sector into a shredded directory tree. Know what your fstab says, but do not weaken it to hide a real fault.

When it is still broken

• It flips back to ro immediately. The fault is active. Stop writing entirely, and if the data matters, image the disk with ddrescue to healthy storage before attempting any further repair.
• SMART is clean but errors persist. Suspect the layer between the disk and ext4: a failing SATA cable, a flaky controller, or on a VPS a problem on the host. On virtual storage, open a ticket with the provider and quote the guest dmesg I/O errors.
• The read-only was actually "no space left". A full disk or exhausted inodes can look like write failures too. Rule it out with df -h and df -i; a related trap is covered in Postgres PANIC: no space left on device and in MySQL 'The table is full', both of which surface as write errors that are not disk failure.
• It is an overlay or container layer. Inside containers a read-only rootfs can be a deliberate mount option, not a fault. Check whether the mount was set ro on purpose before you go hunting for a dying disk.

The rule is short enough to keep in your head at 2am: read the log, check the disk, repair offline, remount last. The server going read-only is the kernel doing you a favour. Do not undo the favour before you understand it.