[tor-bugs] #29682 [Internal Services/Tor Sysadmin Team]: remove traces munin-node everywhere

[Tor Bug Tracker & Wiki]

#29682: remove traces munin-node everywhere
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  defect                               |         Status:
                                                 |  assigned
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:  #29681                               |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by anarcat):

 fun facts found while ripping that stuff out:

  1. there is a DSA-specific munin package, with the following plugins:

      * `spamassassin`: ham/spam/total counts, looks for `spamd:
        ((processing|checking) message|identified spam|clean message)`
        in `mail.log`, could be replaced with
        [https://github.com/google/mtail mtail]
      * `postgres-wal-traffic_`: output of `psql -p "$port" --no-align
 --command 'SELECT * FROM pg_current_xlog_insert_location()'  --tuples-only
 --quiet | tr -d /`, probably covered by the
 [https://github.com/wrouesnel/postgres_exporter postgres exporter]
      * `ksm_scans`: output of `/sys/kernel/mm/ksm/full_scans`, see
        [https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.html KSM
 docs]
      * `ksm`: same, but with `pages_shared`, `_unshared`, `_volatile`,
 `_sharing`, possibly covered by the node exporter, but hardly seems
 critical in any case
      * `vsftpd`: logtail of `/var/log/ftp/vsftpd.log` looking for
 upload/download/login/delete/connexions, would require a custom mtail
 plugin as well
      * `bind`: logtail of `/var/log/daemon.log`, looking for queries
        etc, easy replacement with the
        [https://github.com/digitalocean/bind_exporter/ Prometheus
 exporter]
      * `apache_servers`: apache server-status, equivalent of the
        apache exporter, already deployed

  2. there's a packet counting script in `ferm` which seem to count per-IP
 packet stats from iptables:
     {{{
     $munin_ips = split(regsubst($v4ips, '([^,]+)', 'ip_\1', 'G'), ',')
     munin::check { $munin_ips:
         script => "ip_";
     }
     if $v6ips {
         $munin6_ips = split(regsubst($v6ips, '([^,]+)', 'ip_\1', 'G'),
 ',')
         munin::check { $munin6_ips: script => 'ip_', }
     }
 }}}
     i have just removed those, without a replacement.

  3. hiding in the haproxy puppet module is another munin plugin. there is
 also a [https://github.com/prometheus/haproxy_exporter prometheus exporter
 for haproxy] which we can eventually deploy to replace this. in the
 meantime, it was deleted

  4. the VM image installer (`modules/roles/files/virt/tor-install-VM`) has
 noises about setting up VM-specific plugins: `echo '  for i in
 `/usr/local/sbin/vm_du_ suggest`; do ln -vsf /usr/local/sbin/vm_du_
 /etc/munin/plugins/vm_du_$i; done'`. that file does not seem to be
 deployed through Puppet, and consists of a script checking the disk space
 of all VMs. It looks something like this on `kvm4` right now:

     {{{
 #!/bin/bash
 # -*- sh -*-

 MUNIN_LIBDIR=${MUNIN_LIBDIR:-/usr/share/munin}
 . $MUNIN_LIBDIR/plugins/plugin.sh

 BASE=/srv/vmstore

 VM=${0##*vm_du_}
 #VM=${VM//_/.}

 case $1 in
     autoconf)
         if [[ -d "$BASE" ]]; then
             echo yes
             exit 0
         else
             echo "no ($BASE not found)"
             exit 0
         fi
         ;;
     suggest)
         if [[ -d "$BASE" ]]; then
             find "$BASE" -mindepth 1 -maxdepth 1 -type d -a ! -name
 lost+found -printf '%f\n' # | tr . _
         fi
         exit 0
         ;;
     config)
         echo "graph_title disk usage VM $VM"
         echo 'graph_args --base 1024 --lower-limit 0'
         echo 'graph_vlabel bytes'
         echo 'graph_category disk'
         echo 'graph_total Total'

         find "$BASE/$VM" -mindepth 1 -maxdepth 1 -type f |
         while read fn; do
             label="${fn##*/}"
             label=${label//./_}
             name=${label//-/_}
             echo "$name.label $label"
             echo "$name.cdef $name,1024,*"
         done
         exit 0
         ;;
 esac

 find "$BASE/$VM" -mindepth 1 -maxdepth 1 -type f -printf '%f %k\n' |
 while read fn du; do
   fn=${fn//[.-]/_}
   echo "$fn.value $du"
 done
 }}}

     that is covered by #29816.

  5. the munin-common package doesn't remove its own user/group by default
 so I did that by hand. there's a possibility that some files are leftover
 in /var or /etc, but I am ready to assume the consequence of a possible
 UID reuse there to remove an extra account from all servers

  6. normally, the package removal process should have removed all of
 /etc/munin/plugins, but there are some leftovers sometimes, e.g. on `oo-
 hetzner-03`:

     {{{
 diskstats     fw_forwarded_local  if_err_eth0  ip_38.229.72.27
 ntp_kernel_err       ntp_kernel_pll_off  postfix_mailvolume  threads
 fw_conntrack  fw_packets          if_eth0      netstat
 ntp_kernel_pll_freq  postfix_mailqueue   proc_pri            users
 }}}

     Those are all symlinks to builtin plugins, so I think they can be
 safely removed and have done so.

  7. nagios was watching that munin was running everywhere in its static
 configuration, I have removed that check as well

 All those changes will take some time to propagate everywhere, which will
 make Nagios noisy for a little while. Tomorrow, it will be possible to
 remove remaining Munin code from Puppet entirely, assuming all nodes will
 have run Puppet correctly.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29682#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online