What bench migrate actually does, and the rollback you'll wish you had

The message you do not want, on a client's system, at 16:40 on a Friday:

Executing your_app.patches.v1_4.backfill_supplier_ledger in dev.client.local (_a1b2c3d4e5f6a7b8)
Traceback (most recent call last):
  ...
pymysql.err.OperationalError: (1054, "Unknown column 'supplier_group' in 'field list'")

Twelve patches were due. Six ran. The seventh died. The instinct is to fix the patch and run bench migrate again, and quite often that works. But it is worth understanding why it sometimes does not, because the reason is structural and it is the single most important thing to know before you automate a deployment: a failed migration is not a migration that did not happen. It is a migration that half happened, and there is no command that undoes it.

This is the one irreversible step in the pipeline that this series builds towards. If you are heading for a deployment to a client server you cannot log in to, then this is the step you cannot fix by hand afterwards, which means the protection has to be in the script.

What it does, in order

Under the surface, bench --site dev.client.local migrate constructs a site migration and runs it in a fixed sequence.

One ordering detail catches people repeatedly: fixtures are synchronised after the patches. If you have a post-model-sync patch that expects a custom field defined in a fixture, that field does not exist yet on the first migrate after you add it. The patch fails, or worse, quietly writes nothing.

patches.txt, and why a patch never runs twice

Your app declares its patches in your_app/patches.txt, an INI-shaped file with two sections:

[pre_model_sync]
your_app.patches.v1_0.rename_customer_ref_field

[post_model_sync]
your_app.patches.v1_4.backfill_supplier_ledger

They run top to bottom within each section. What decides whether a given patch runs at all is a single lookup against a DocType called Patch Log.

You can read the record directly:

bench --site dev.client.local mariadb

SELECT patch, skipped, creation
FROM `tabPatch Log`
ORDER BY creation DESC
LIMIT 10;

That table is the source of truth about what has run on this site. It is also the reason for a rule I now treat as absolute: never edit a patch that has shipped. On your machine it will re-run because your Patch Log has no row for it. On the client's site it will not, because theirs does. You will then be debugging a difference that exists only in a table nobody thought to look at.

Why re-running does not undo anything

Two mechanics combine here, and either alone would be survivable.

The first is the per-patch commit above. When a patch throws, Frappe rolls back and re-raises, but the rollback only covers the transaction that was open - the current patch. Everything before it is committed.

The second is that MariaDB does not roll back DDL. If your patch ran an ALTER TABLE and then failed three lines later on a data update, the rollback reverts the data update and leaves the column exactly where the ALTER put it. The schema has moved and the data has not. That is the state that "no amount of re-running fixes", because on the second attempt the patch tries to add a column that now exists and dies differently.

So the honest summary is: re-running bench migrate resumes, it does not retry. That is genuinely useful when the failure was environmental - a missing Python package, a worker holding a lock - and it is no help at all when the failure left the schema somewhere the patch does not expect.

--skip-failing does less than the name suggests

The temptation, mid-incident, is bench --site dev.client.local migrate --skip-failing. Know what it does before you reach for it. It catches the exception, writes a Patch Log row with skipped = 1 and the full traceback, and continues to the next patch.

Because the executed-patches lookup only counts rows where skipped = 0, that patch is not permanently skipped. It is retried on the next migrate. So the flag is really "keep going and keep the traceback", which is a reasonable thing to want and a bad thing to leave in a deployment script. Use it interactively when you are diagnosing. Never put it in automation, because a deploy that reports success while quietly deferring a data backfill is worse than one that fails loudly.

The discipline before the command

Three things, in this order, every time. None of them is optional and all of them belong in the script rather than in your memory.

cd /home/frappe/frappe-bench

# 1. Refuse to start if background jobs are still in flight
bench --site dev.client.local ready-for-migration

# 2. Close the site to users
bench --site dev.client.local set-maintenance-mode on

# 3. Back up the database AND the files, together
bench --site dev.client.local backup --with-files

The first command is underused. It checks for pending background jobs three times, a second apart, and exits non-zero if any are queued. A migration that starts while a worker is midway through writing invoices is a genuinely bad day, and this is a one-line guard against it.

The second matters because bench migrate does not do it for you. Frappe sets an internal flag during migration, but nothing stops a user saving a Sales Order while the schema is being altered underneath them.

The third produces four files, named by timestamp and site:

ls -1t sites/dev.client.local/private/backups | head -4

20260727_083000-dev_client_local-private-files.tar
20260727_083000-dev_client_local-files.tar
20260727_083000-dev_client_local-database.sql.gz
20260727_083000-dev_client_local-site_config_backup.json

--with-files is the part people skip, and it is the part that makes the backup usable. Frappe keeps uploaded documents on disk, not in the database. A database-only restore gives you rows pointing at attachments that are no longer where the rows say they are, and you will not notice until a user opens an invoice next week.

The actual recovery

The restore itself, in full. Run it as frappe, from the bench directory:

cd /home/frappe/frappe-bench
B=sites/dev.client.local/private/backups/20260727_083000-dev_client_local

bench --site dev.client.local restore "${B}-database.sql.gz" \
  --with-public-files "${B}-files.tar" \
  --with-private-files "${B}-private-files.tar"

Then put the code back where the data expects it, and reopen the site:

cd apps/your_app
git checkout --force v1.3.0
cd /home/frappe/frappe-bench
bench build --app your_app
bench --site dev.client.local clear-cache
bench --site dev.client.local set-maintenance-mode off
sudo supervisorctl restart frappe-bench-web: frappe-bench-workers:

Four practical notes on that restore, all learned by being surprised.

It drops and recreates the database, so it needs database root credentials. If root_password is not in common_site_config.json, it prompts, and a prompt in an unattended script is a hang rather than an error. Pass --db-root-password explicitly, or accept that restore is a human step. I have deliberately kept it a human step.

If the backup was taken by a newer version of the app than the code currently checked out, restore detects the downgrade and asks for confirmation, aborting unless you pass --force. That is a good safety net and another reason this does not belong inside automation.

Paths are resolved relative to the site's backup directory and the bench's parent as well as the current directory, so passing just the filename usually works. Passing the full relative path always works, which is what a script should do.

And if backups are encrypted, restore reads the key from the site config automatically, or takes --encryption-key. If the site config is the thing you lost, that key is what stands between you and an unreadable backup. Store it somewhere that is not the server.

What this costs

Taking a backup with files before every migration is not free. On a site with several years of scanned delivery notes, the tar is large and the backup adds minutes to every deploy, on a disk that has to have room for it. Teams notice this and start skipping it for "small" changes, which is exactly when it bites, because small changes are the ones nobody reads carefully.

It is also worth being honest that a nightly bench backup is a recovery point, not a recovery strategy. If losing a day of transactions is unacceptable to this client, the answer is binlogs, and why bench backup is not a backup strategy covers point-in-time recovery for ERPNext properly.

The compromise I have settled on: always take it, keep the last three, and delete older ones on a schedule the client controls rather than from the deploy script. A deploy script that deletes backups is one bad glob away from being the incident.

The other cost is honesty with the client. A deployment that includes a migration is not zero downtime, and saying so up front is much better than discovering it together. Maintenance mode is on for the duration, and on a real ERPNext site with a few hundred thousand rows that is usually seconds to a couple of minutes, not hours.

All of this assumes you know which directory the migration is even running in. If apps/, sites/ and env/ are not yet a clear picture, the anatomy of a Frappe bench is the prerequisite. And the deploy script that wraps every one of these commands, including the trap that lifts maintenance mode when the migration fails, is in the full no-access deployment walkthrough.