[tor-project] bullseye upgrades, batch 2 status update

[Antoine Beaupré]

TL;DR: upgrade is a little slower, snag on PostgreSQL and you may need
to upgrade to Python3 earlier than expected.

On 2022-04-27 11:25:16, Antoine Beaupré wrote:
> Reminder: the bullseye upgrade run is continuing in May.

A little update on the progress of the bullseye upgrades... As you can
see here, progress has been a little slower than the first batch:

https://gitlab.torproject.org/tpo/tpa/team/-/issues/40692

During the first week of the first batch, we had most of the servers
done ("only three left!"). It was pretty impressive. But in this first
week, we were a little slower: we only did a third.

That was partly due to people's availability: I was away on Monday, and
kez was also less available. Plus, we had kernel reboots to handle,
which took a day off lavamind's.

But it was also due to the complexity of this second batch: most of the
servers are managed by service admins and have more moving parts and
legacy.

In particular, it seems we might be having unexpected trouble with the
PostgreSQL 13 upgrade, which is a little disappointing. Materculae is
showing signs of increase memory usage, including an OOM last
night. That issue is tracked here and we welcome any input from
PostgreSQL nerds:

https://gitlab.torproject.org/tpo/tpa/team/-/issues/40750

We also hit a few problems with the Python 2 deprecation. Normally, we
haven't announced removing support for Python 2 just yet, and Debian
bullseye did ship with Python 2.7, even though it's been dead since
April 2020. But bullseye *does* ship with a bunch of Python 2 *modules*
removed. So far, we have found that we needed to deal with those
removals:

 * python-dateutil
 * python-dnspython
 * python-psycopg2
 * python-stem

In general, buster shipped with 3470 Python 2 packages, and bullseye
brought that list down to *only* 766! So there are a *lot* more packages
like this that may cause problem on our servers. The details of which
packages those are is available here:

https://people.debian.org/~anarcat/python2-in-debian/

We don't actually *know* of any such packages left: all the ones that we
had specified in Puppet are noted above and have been replaced with
their Python 3 equivalent, and the service admins have fixed their
service.

But we don't actually manage everything through Puppet, so it is
perfectly possible that you are relying on a dependency that will be
removed in the pending Python upgrade.

So if you rely on any Python script which relies on Debian packages, now
is a good time to make sure it works with the following header:

    #!/usr/bin/python3

... and you can do this right now, even before we upgrade your service
to Python 3.

I plan on making a formal RFC to clarify this situation as well, today,
so that this gets to a broader audience.

We plan to resume the bullseye upgrades next week, as we don't do major
changes like this before the weekend.

Thank you for your attention!

-- 
Antoine Beaupré
torproject.org system administration
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20220505/93347161/attachment.sig>