[tor-project] TPA-RFC-27: Python 2 end of life

Thu May 5 19:19:47 UTC 2022

Summary: Python 2 is officially unsupported by TPA. Major breakage to
unfixed code is expected after the Debian bullseye upgrade completes
(May-July 2022), and definite breakage will occur when Python 2
support is completely dropped in Debian bookworm (some time in 2023).

# Background

[Python 2.7.18][] was released on April 20th 2020. It was the last
Python 2 release that will ever happen, and Python 2 is now
unsupported, end of life, [dead][].

[Python 2.7.18]: https://www.python.org/downloads/release/python-2718/
[dead]: https://www.enricozini.org/blog/2020/python/python-2-is-dead/

## Status of Python 2 in Debian

It was originally thought that the Debian 11 "bullseye" [release][]
(on August 14th 2021) would not support Python 2 at all, but it was
actually released with *some* Python 2 support.

[release]: https://www.debian.org/News/2021/20210814

However, an analysis from anarcat about the [Python 2 modules shipped
in bullseye][] shows that a large number of Python 2 modules were
actually removed from Debian 11. Out of the 2699 "python 2" packages
in Debian buster (packages starting with `python2?-`, excluding `-doc`
and `-dbg`), 2616 were removed. Therefore, only 90 such packages
remain in Debian bullseye, a 97% reduction.

[Python 2 modules shipped in bullseye]: https://people.debian.org/~anarcat/python2-in-debian/

As a consequence, odds are that your Python 2 code will just stop
working after the bullseye upgrade, if it uses one of the [modules
missing from bullseye][]. Which, really, means if it uses anything
outside the standard library that is not vendored with your code
(e.g. in a "virtualenv"), because the odds of that module being one of
the few 90 modules are pretty low.

[modules missing from bullseye]: https://people.debian.org/~anarcat/python2-in-debian/python2-removed-bullseye.txt

The next Debian stable release (12, code name "bookworm") doesn't yet
have a clear plan to remove Python 2, but it's likely to shrink the
list of Python 2 modules even further. It is currently down to 79
packages.

Bookworm also does not currently ship the magic [python-is-python2][]
package, which ensures the existence of `/usr/bin/python`. This means
any script with such a header will start breaking in Debian bookworm:

    #!/usr/bin/python

[python-is-python2]: https://packages.debian.org/bullseye/python-is-python2

# Status of Python 2 in TPA

We currently are in the middle of a the [Debian 11 bullseye
upgrade][], so we have both Debian 10 and Debian 11 machines, which
means we have actually Python 2.7.16 and 3.7.3 (buster) and Python
2.7.18 and 3.9.2 (bullseye) currently deployed.

In any case, we have two "reasonable" versions of Python 2 (2.7+) and
Python 3 (3.5+) available everywhere, it should be fairly easy to
target Python 3 for ports, without having to concern ourselves with
Python 2 support any longer.

[Debian 11 bullseye upgrade]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/5

We do not currently knowingly deploy any Python 2 module in the above
list, although it's possible some packages are manually installed on
some host.

The TPA code base still has a lot of Python 2 code, particularly on
the LDAP server, but there's a lot of Python 2 code floating around
the infrastructure. We haven't performed an audit of the code and are
fixing issues as they come up as part of the Python 2 upgrade
procedure.

Other services have not been examined. So far, most services actually
run under Python 3, or have been found to be already ported and just
needing a redeployment (see
[tpo/network-health/metrics/exit-scanner#40004][] for an example).

[tpo/network-health/metrics/exit-scanner#40004]: https://gitlab.torproject.org/tpo/network-health/metrics/exit-scanner/-/issues/40004

# Proposal

After the Debian 11 "bullseye" upgrade, TPA will *not* support Python
2 modules that were removed from Debian. Any program using such a
module will need to be ported to Python 3, as the packages shipping
those modules will be removed as part of the upgrade procedure. The
`/usr/bin/python` binary will remain, for now, as the 2.7.18
executable.

After the Debian 12 "bookworm" upgrade, Python 2 will be completely
removed from servers. Any program using Python 2 will likely stop
working completely as the `/usr/bin/python` command will be removed.

The `/usr/bin/python` command *may* eventually be replaced by the
Python 3 interpreter, but that will not be before the bookworm upgrade
procedure begins, and only if the lack of a `python` binary is too
problematic for users.

## Timeline

Debian 11 bullseye upgrades should complete by July 1st 2022, but most
upgrades should complete by the second week of May 2022, that is next
week, starting on May 9th 2022 and continuing during the week.

A grace period *may* be given to certain projects that cannot
immediately port their code to Python 3, by keeping Python 2 modules
from Debian buster installed, even after the bullseye upgrade. Those
modules will *definitely* be removed by July 1st 2022, however.

Debian 12 bookworm upgrades are currently scheduled to begin some time
in 2023 and should be completed before July 2024. An actual schedule
will be proposed in a future announcement. When this change will be
deployed, Python 2 will be gone from TPA servers.

## Affected users

This affects any service admin who deploys Python code on TPA
infrastructure.

# Alternatives considered

We have considered just ignoring this problem, and in fact that was
the approach with the [original Debian 11 bullseye upgrade
proposal][]. Although it didn't state it explicitly, it didn't have
any plan for the Python 2 upgrade.

And indeed, the issue about the [Python end of life][] was postponed
to the [Debian 12 bookworm upgrade milestone][], because it was
believed Python 2 would just keep working in Debian 11. Unfortunately,
the [second batch of upgrades][] showed the situation was much more
severe than we expected, and required a more radical approach.

[second batch of upgrades]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40692
[Python end of life]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33949
[original Debian 11 bullseye upgrade proposal]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-20-bullseye-upgrades

Another alternative to porting your code to Python 3 is actually to
use the PyPy interpreter, which still supports Python 2 (and is
actually still struggling with its Python 3 port). However, we
strongly discourage this approach, and `pypy` is not currently
installed on any TPA server.

GitLab CI users may be able to ignore this issue by using containers
that do ship Python 2. Note that we *may*, in the future, implement
controls on the container images deployed from GitLab CI to avoid
using old, unsupported software in this way, exactly for this kind of
situation. But for now there are no such controls. We strongly
discourage the use of outdated software, including containers, inside
your tool chain, in general.

# Costs

Staff. 

There is no estimate on the volume of Python 2 code left to upgrade. A
study of this should probably be performed at some point, but so far
we have assumed this wasn't a problem, so we are dealing with this on
a case-by-case basis.

# Approval

TPA.

# Deadline

This proposal will welcome comments until Tuesday May 10th, at which
point it will be considered adopted and the Debian bullseye upgrades
will resume.

We acknowledge this is an extremely short deadline (~5 days), but we
have actually planned those Debian bullseye upgrade for a while, and
somehow expected there wouldn't be much Python 2 code lying around. We
hope that the exception for Python 2 modules (until July 1st) will be
sufficient mitigation for us to continue with the bullseye upgrades in
a timely manner.

# Status

This proposal is currently in the `proposed` state.

# References

 * [Discussions about this proposal][], comments welcome here!
 * [Debian 11 bullseye upgrade milestone][]
 * [Debian 12 bookworm upgrade milestone][]

[Discussions about this proposal]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33949
[Debian 11 bullseye upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/5
[Debian 12 bookworm upgrade milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/6

-- 
Antoine Beaupré
torproject.org system administration
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20220505/6a641f75/attachment-0001.sig>