[tor-project] minutes from the sysadmin meeting

[Antoine Beaupré]

Ahoy!

Well it's just been way too long since we've done this, but we've done
it! Here's our short sysadmin minutes from today's meeting.

# Roll call: who's there and emergencies

onionoo-backend running out of disk space ([tpo/tpa/team#41343][])

[tpo/tpa/team#41343]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41343

# Dashboard cleanup

Normal per-user check-in:

 * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&ssignee_username=anarcat
 * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&assignee_username=kez
 * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&assignee_username=lavamind

General dashboards:

 * https://gitlab.torproject.org/tpo/tpa/team/-/boards/117
 * https://gitlab.torproject.org/groups/tpo/web/-/boards
 * https://gitlab.torproject.org/groups/tpo/tpa/-/boards

Nextcloud roadmap / spreadsheet.

Overall, it seems we are as you would expect when returning from a
rather chaotic vacation. Backlog is large, but things seem to be under
control.

We added SVN back on the roadmap after one too many tickets asking for
setup.

# Metrics of the month

 * hosts in Puppet: 89, LDAP: 89, Prometheus exporters: 166
 * number of Apache servers monitored: 37, hits per second: 626
 * number of self-hosted nameservers: 6, mail servers: 10
 * pending upgrades: 1, reboots: 0
 * average load: 0.69, memory available: 3.58 TiB/4.98 TiB, running processes: 424
 * disk free/total: 53.19 TiB/126.72 TiB
 * bytes sent: 403.47 MB/s, received: 269.04 MB/s
 * planned bullseye upgrades completion date: 2024-08-02
 * [GitLab tickets][]: 196 tickets including...
   * open: 0
   * icebox: 163
   * needs information: 5
   * backlog: 13
   * next: 9
   * doing: 4
   * needs review: 2
   * (closed: 3301)

 [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards

Upgrade prediction graph lives at:

https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm/

Now also available as the main Grafana dashboard. Head to
<https://grafana.torproject.org/>, change the time period to 30 days,
and wait a while for results to render.

# Number of the month: 42

34 machines were upgraded from bullseye to bookworm in the two first
days of last week! We calculated this was an average of 20 minutes per
host to upgrade.

The trick, of course, is that things often break *after* the upgrade,
and that "fixing" time is not counted here. That said, last estimate
for this was one hour per machine, and we're doing a whole fleet
upgrade every 2-3 years, which means about ten hours of work saved per
year.

But the number of the month is, of course, 42, as we now have an equal
number of bookworm and bullseye machine, after the upgrade. And that
number is, naturally, [42][].

See also https://xkcd.com/1205/ which, interestingly, we fall out of
scope of.

[42]: https://en.wikipedia.org/wiki/42

-- 
Antoine Beaupré
torproject.org system administration