[tor-project] minutes from the sysadmin meeting

[Antoine Beaupré]

Hi everyone,

We had our monthly sysadmin meeting (a bit late for various reasons)
for February, and here are the minutes.

# Roll call: who's there and emergencies

No emergencies: we have an upcoming maintenance about chi-san-01 which
will require a server shutdown at the end of the meeting.

Present: anarcat, gaba, kez, lavamind

# Storage brainstorm

The idea is to just throw ideas for this ticket:

https://gitlab.torproject.org/tpo/tpa/team/-/issues/40478

anarcat went to explain the broad strokes around the current storage
problems (lack of space, performance issues) and solutions we're
looking for (specific to some service, but also possibly applicable
everywhere without creating new tools to learn)

We specifically focused on the storage problems on gitlab-02,
naturally, since that's where the problem is most manifest.

lavamind suggested that there were basically two things we could do:

 1. go through each project one at a time to see how changing certain
    options would affect retention (e.g. "keep latest artifacts")

 2. delete all artifacts older than 30 or 60 days, regardless of
    policy about retention (e.g. keep latest), could or could not
    include job logs

other things we need to do:

 * encourage people to: "please delete stale branches if you do have
   that box checked"
 * talk with jim and mike about the 45GB of old artifacts
 * draft new RFC about artifact retention about deleting old artifacts
   and old jobs (option two above)

We also considered unchecking the "keep latest artifacts" box at the admin level, but this would disable the feature in all projects with no option to opt-in, so it's not really an option.

We considered the following technologies for the broader problem:

 * S3 object storage for gitlab
 * ceph block storage for ganeti
 * filesystem snapshots for gitlab / metrics servers backups

prototype with image/cache storage on runners? safer because easy to rollback

We'll look at setting up a VM with minio for testing. We could first
test the service with the CI runners image/cache storage backends,
which can easily be rebuilt/migrated if we want to drop that test.

This would disregard the block storage problem, but we could pretend
this would be solved at the service level eventually (e.g. redesign the
metrics storage, split up the gitlab server). Anyways, migrating away
from DRBD to Ceph is a major undertaking that would require a lot of
work. It would also be part of the largest "[trusted high performance
cluster][]" work that we recently de-prioritized.

 [trusted high performance cluster]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/2

# Other discussions

We should process the pending TPA-RFCs, particularly TPA-RFC-16, about
the i18 lektor plugin rewrite.

# Next meeting

Our regular schedule would bring us to March 7th, 18:00UTC.

# Metrics of the month

 * hosts in Puppet: 88, LDAP: 88, Prometheus exporters: 143
 * number of Apache servers monitored: 25, hits per second: 253
 * number of self-hosted nameservers: 6, mail servers: 8
 * pending upgrades: 0, reboots: 0
 * average load: 2.10, memory available: 3.98 TiB/5.07 TiB, running processes: 722
 * disk free/total: 35.81 TiB/83.21 TiB
 * bytes sent: 296.17 MB/s, received: 182.11 MB/s
 * planned bullseye upgrades completion date: 2024-12-01
 * [GitLab tickets][]: 166 tickets including...
   * open: 1
   * icebox: 149
   * needs information: 2
   * backlog: 7
   * next: 5
   * doing: 2
   * (closed: 2613)
    
 [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards

Upgrade prediction graph lives at:

https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye/

# Number of the month

-3 months. Since the last report, our bullseye upgrade completion date
moved backwards by three months, from 2024-09-07 to 2024-12-01. That's
because we haven't started yet, but it's interesting that it's seems
to be moving back faster than time itself... We'll look at deploying a
perpetual movement time machine on top of this contraption in the next
meeting.

-- 
Antoine Beaupré
torproject.org system administration
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20220215/5f5f8039/attachment.sig>