[tor-bugs] #32461 [Internal Services/Tor Sysadmin Team]: do not write logs on caching servers

[Tor Bug Tracker & Wiki]

#32461: do not write logs on caching servers
-----------------------------------------------------+-----------------
     Reporter:  anarcat                              |      Owner:  tpa
         Type:  defect                               |     Status:  new
     Priority:  Medium                               |  Milestone:
    Component:  Internal Services/Tor Sysadmin Team  |    Version:
     Severity:  Normal                               |   Keywords:
Actual Points:                                       |  Parent ID:
       Points:                                       |   Reviewer:
      Sponsor:                                       |
-----------------------------------------------------+-----------------
 In #32239, a caching system was deployed with nginx. To get hit rate
 ratios, log files are written to disk, with IP address and user agents
 anonymized. That's okay-ish: it's not as well anonymized as our apache log
 files because it's not possible to have a per-day granularity in
 timestamps.

 From there, mtail wakes up once in a while and parses those logfiles and
 counts things, which are exposed as metrics picked up by prometheus. That
 in turn gives us pretty Prometheus graphs and makes us feel better about
 ourselves.

 But ideally, we wouldn't have log files at all and pipe things directly
 into mtail. But we don't want to hang the webserver while waiting for
 mtail (which can be a little flaky), so the typical way to deal with this
 is to pipe logs first in syslog.

 I couldn't immediately figure out how to do this during deployment so I'm
 opening this ticket to make sure we eventually operate that conversion.

 One problem I had is the syslog-ng config sends all logs to the central
 logging server. If we start pushing web hits into syslog, this could
 become unwieldy, to say the least, in terms of performance mostly, but
 also privacy.

 It's also not clear to me how to send logs from syslog into mtail without
 hitting the disk in the first place.

 So the checklist is:

  1. how to send logs from nginx to syslog (`access_log
 syslog:server=unix:/dev/log,facility=local3,tag=nginx_access extended;`
 seems to be the magic config in nginx)
  2. how to avoid sending those logs to the central server
  3. how to send those logs (and only those) into mtail

 All of this should be automatically configured in Puppet as well.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32461>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online