Python 2 Dies in January: The Migration Plan I Wish I Had Started Sooner

Python 2.7 stops being maintained on 1 January 2020. That is four and a half months from today, it has been announced for years, and I still open codebases every month that run on it. Two of them are payroll systems.

Let me be precise about the risk, because the panic version of this story is not useful. Nothing breaks on 1 January. Your application will start on 2 January exactly as it did on 31 December. What ends is upstream maintenance: no more security fixes for the interpreter, and libraries dropping Python 2 from their next release. The failure mode is not a crash, it is a slow closing of doors. Six months in, a dependency has a vulnerability and its fix is Python 3 only. A year in, your operating system has stopped shipping the interpreter and you are compiling it yourself.

The migration itself is not hard. It is long, it is boring, and it goes badly when done in the wrong order. Here is the order I use.

Phase 1: find out what is actually blocking you

Do not read your code first. Read your dependencies, because one abandoned library can hold the whole project hostage and it changes the plan entirely:

pip install caniusepython3
caniusepython3 -r requirements.txt

It walks the dependency tree and prints the packages with no Python 3 release, plus what depends on them. A blocked leaf package is an afternoon of finding a replacement. A blocked web framework is a different project altogether, and you need to know that before you promise a date.

If you are on Django, the version matters more than anything in your own code. Django 1.11 is the last release supporting Python 2, and its extended support ends in April 2020. Upgrading Django and porting to Python 3 at the same time is two migrations at once; do the Django upgrade to 1.11 first, on Python 2, and ship that.

Phase 2: tests before changes

You are about to make thousands of small edits to code you did not write, in a language that will silently change the meaning of some of them. That is exactly the situation tests exist for. You do not need full coverage. You need the paths where being wrong costs money: invoicing, tax, stock movements, anything that writes to a ledger.

While you are here, turn on the warnings the interpreter already offers:

python2.7 -3 -W all yourscript.py
pylint --py3k yourpackage/

The -3 flag makes Python 2 warn about constructs whose behaviour changes in Python 3, and pylint --py3k is a checker for the same class of problem. Between them you get a work list from the tools instead of from a stack trace in production.

Phase 3: straddle both versions

The temptation is to open a branch, convert everything with 2to3, and merge it in three months. I have watched that fail twice. The branch cannot be deployed, so normal work continues on master, and the two diverge until merging costs more than the migration.

Instead, make the code run on both interpreters and keep shipping every week. futurize, from the python-future project, does this: it applies the safe transformations and adds compatibility imports rather than producing Python 3 only output.

pip install future
futurize --stage1 -w -n yourpackage/    # safe, no behaviour change
# run tests, commit, deploy
futurize --stage2 -w -n yourpackage/    # adds six and future imports

Stage one is genuinely safe: print statements become functions, exception syntax is modernised, obvious dead idioms are fixed. Ship it. Stage two changes semantics in places, so review every hunk and take it module by module.

Put this at the top of each module you touch:

from __future__ import absolute_import, division, print_function, unicode_literals

The division import is the important one and I will come back to it.

Then run your suite under both interpreters on every commit:

[tox]
envlist = py27, py36

[testenv]
deps = -rrequirements.txt
       pytest
commands = pytest {posargs}

The four bugs that actually bite

Python 2	Python 3	How it fails
5 / 2 gives 2	5 / 2 gives 2.5	Silently, in totals and pro-rata
str and unicode mix freely	str and bytes never mix	Loudly, with TypeError
dict.iteritems()	dict.items()	Loudly, with AttributeError
sorted([1, None]) works	raises TypeError	Loudly, but only on odd data

Only the first one is silent, and that is why it is the dangerous one. In Python 2, total / count on two integers truncates. In Python 3 it does not. A report that has always rounded down now produces decimals, or an invoice line that was correct becomes correct in a different way, and nobody notices for a quarter. This is the same class of problem as a database quietly answering an ambiguous query: the code runs, the number is different, and no error is raised. I wrote about the database version of that in the piece on ONLY_FULL_GROUP_BY. Grep for / in any module that touches money, and decide each one deliberately between / and //.

The loud one you will meet most is text and bytes:

TypeError: a bytes-like object is required, not 'str'

The rule that makes this stop being confusing: decode at the edges, keep text in the middle. Anything arriving from a socket, a file opened in binary mode, or a subprocess is bytes. Decode it once, at the point of arrival, with an explicit encoding. Everything inside your program is str. Encode again only when writing out. Programs that decode in the middle of business logic are the ones that produce this at three in the morning:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128)

Phase 4: the flip

When tox is green under both, the switch is a deployment change, not a code change. Change the interpreter in the virtualenv, redeploy, and change nothing else that day. Keep the Python 2 environment on the server until you are satisfied, because the rollback is then a symlink rather than a rebuild.

Watch for the things tests do not cover: cron jobs, one off scripts in /usr/local/bin, the deploy script itself, and any shebang line that says #!/usr/bin/python. That last one is a trap in waiting, because on newer distributions the plain python command is on its way out entirely.

Verification

python --version
pip freeze | wc -l
tox                      # green on py27 and py36
grep -rn '#!/usr/bin/python$' . --include='*.py'

Then re-run last month's reports on both versions and compare the output files byte for byte. If the numbers differ, you have found a division bug, and finding it in a diff is much cheaper than finding it in an audit.

What people get wrong

The long-lived conversion branch. Described above. Straddle and ship instead.

Running 2to3 and calling it done. It converts syntax, not meaning. It cannot know whether a division was meant to truncate, and it will not tell you that your data now sorts differently.

Migrating without tests because "the code is simple". The code is not simple, it is familiar, and those are different things.

Assuming the date is the deadline. The real deadline is set by your dependencies, and it is usually earlier. A library that dropped Python 2 last month already blocks you from taking its security fixes.

If you truly cannot finish in time

Sometimes the honest answer is that a system will still be on Python 2 in January. Then do these three things, in this order. Pin every dependency to a known good version and vendor them, so a fresh install still works when the index moves on. Put the application behind something that is maintained, so unpatched code is not directly exposed to the internet. And write down, on one page for whoever pays for it, what specifically becomes risky and when, because "it is end of life" means nothing to a business owner and "no security fixes for the software that holds your customer records" means a great deal.