[tor-project] minutes from the sysadmin meeting

[Antoine Beaupré]

Hello! Here's our monthly meeting minutes. Our meeting schedule is
rather affected by those pesky northern-hemisphere-summer-holidays, we
should go back to your regular programming in September.

# Roll call: who's there and emergencies

anarcat, gaba, kez, lavamind, no emergencies.

# Dashboard review

We didn't have time to do a full quarterly review (of Q2), and people
are heading out to vacations anyways so there isn't much we can do
about late things. But we reviewed the dashboards to make sure nothing
drops to the floor with the vacations. We started with the per user
dashboards:

 * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&assignee_username=anarcat
 * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&assignee_username=kez
 * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&assignee_username=lavamind

... as we usually do during our weekly checkins ("what are you working
on this week, do you need help"). Then moved on to the more general
dashboards:

 * https://gitlab.torproject.org/tpo/tpa/team/-/boards/117
 * https://gitlab.torproject.org/groups/tpo/web/-/boards
 * https://gitlab.torproject.org/groups/tpo/tpa/-/boards

Normally we should be making new OKRs for Q3/Q4 at this time, but it
doesn't look like we have the cycles to build that system right now,
and it doesn't look like anyone else is doing so either in other
teams. We are aware of the problem and will work on figuring out how
to do roadmapping later.

Anarcat nevertheless did a quick review of the roadmap and found that
the bullseye upgrade might be a priority. He opened [issue
tpo/tpa/team#40837][] to make sure the 13 machines remaining to
upgrade are properly covered by Debian LTS while we finish the
upgrades.

[issue tpo/tpa/team#40837]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40837

The other big pending change is the email services improvements, but
that has been deferred to TPA-RFC-31, the outsourcing of email
services, which is still being drafted.

# TPA-RFC-33: monitoring requirements adoption

Anarcat had already read aloud the requirements in the last meeting,
so he spared us from this exercise. Instead we reviewed the [changes
proposed by lavamind][] which mostly seem good. Kez still has to look
at the proposal, and their input would be crucial as someone less
familiar with our legacy stuff: new pair of eyes will be useful!

[changes proposed by lavamind]: https://gitlab.torproject.org/tpo/tpa/wiki-replica/-/merge_requests/34

Otherwise the requirements seem to be mostly agreed on, and anarcat
will move ahead with a proposal for the monitoring system that will
try to address those.

# Vacations and next meeting

As anarcat and lavamind both have vacations during the month of
August, there's no overlap when we can do a 3-way meeting, apart at
the very end of the month, a week before what will be the september
meeting. So we cancel the meeting for august, next meeting is in
september.

Regarding holidays, it should be noted that only one person of the
team is out at a time, unless someone is out sick. And that can
happen, but we can usually withstand a temporary staff outage. So
we'll have two people around all August, just at reduced capacity.

For triage of the week rotation, rotation will be changed to keep
anarcat on rotation an extra week this week, so that things even out
during the vacations (two weeks each):

 * week 31 (this week): anarcat
 * week 32 (next week): kez, anarcat on vacation
 * week 33: lavamind, anarcat on vacation
 * week 34: anarcat, lavamind on vacation
 * week 35: kez, lavamind on vacation
 * week 36 (september): lavamind, everyone back

# Metrics of the month

 * hosts in Puppet: 96, LDAP: 96, Prometheus exporters: 164
 * number of Apache servers monitored: 30, hits per second: 298
 * number of self-hosted nameservers: 6, mail servers: 9
 * pending upgrades: 0, reboots: 0
 * average load: 2.16, memory available: 4.72 TiB/5.86 TiB, running processes: 883
 * disk free/total: 29.47 TiB/91.36 TiB
 * bytes sent: 420.66 MB/s, received: 298.98 MB/s
 * planned bullseye upgrades completion date: 2022-09-27
 * [GitLab tickets][]: 184 tickets including...
   * open: 0
   * icebox: 151
   * backlog: 20
   * next: 9
   * doing: 2
   * needs review: 1
   * needs information: 1
   * (closed: 2807)

 [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards

Upgrade prediction graph lives at https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye/

# Date of the month

September 27! We moved back the estimated Debian bullseye completion
date by almost three weeks, from 2022-10-14, to 2022-09-27. This is
bound to slow down however, with the vacations coming up, and all the
remaining server needing an upgrade being the "hard" ones. Still, we
can dream, can we?

-- 
Antoine Beaupré
torproject.org system administration
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20220726/599b9ae3/attachment.sig>