[tor-bugs] #30731 [Circumvention/Snowflake]: Publish post-sanitization broker logs

[Tor Bug Tracker & Wiki]

#30731: Publish post-sanitization broker logs
-------------------------------------+-----------------------------------
 Reporter:  dcf                      |          Owner:  (none)
     Type:  task                     |         Status:  needs_information
 Priority:  Medium                   |      Milestone:
Component:  Circumvention/Snowflake  |        Version:
 Severity:  Normal                   |     Resolution:
 Keywords:                           |  Actual Points:
Parent ID:                           |         Points:
 Reviewer:  phw                      |        Sponsor:
-------------------------------------+-----------------------------------

Comment (by dcf):

 Replying to [comment:3 cohosh]:
 > These graphs are really nice. Thanks for doing this! I'm wondering if we
 want something similar to the information in these graphs for the ongoing
 metrics collected from the broker (#21315).

 Yes, I think it would be great to have similar graphs using up-to-date
 logs. The scripts in attachment:broker-logs.zip:ticket:30693​ should still
 work with the current sanitized logs. We should potentially think about
 making the logs better reflect what we want to measure, because currently
 they require a fair bit of inference. Here's the main classification
 function from the `process` script that tries to regularize the log
 messages:
 {{{#!go
 var regexpRecvSnowflake = regexp.MustCompile(`^Received
 snowflake:\s*([\w/+]*)`)
 var regexpProxyNoClient = regexp.MustCompile(`^Proxy ([\w/+]*) did not
 receive a Client offer.`)

         if strings.HasPrefix(msg, "ACME hostnames: ") {
                 err = w.Write([]string{timestamp, "start", "", "", ""})
         } else if strings.HasPrefix(msg, "http: TLS handshake error ") {
                 err = w.Write([]string{timestamp, "error", "", "", "tls"})
         } else if strings.HasPrefix(msg, "http2: server: error ") {
                 err = w.Write([]string{timestamp, "error", "", "",
 "http2"})
         } else if strings.HasPrefix(msg, "http2: received GOAWAY ") {
                 err = w.Write([]string{timestamp, "error", "", "",
 "http2"})
         } else if msg == "Starting HTTP-01 listener" {
         } else if msg == "Invalid data." {
                 err = w.Write([]string{timestamp, "error", "", "",
 "invalid-data"})
         } else if msg == "Mismatched IDs!" {
                 err = w.Write([]string{timestamp, "proxy-gets-none", "",
 "", "mismatched-ids"})
         } else if msg == "Passing client offer to snowflake proxy." {
                 err = w.Write([]string{timestamp, "client-offers", "", "",
 ""})
         } else if msg == "Client: No snowflake proxies available." {
                 err = w.Write([]string{timestamp, "client-gets-none", "",
 "", "no-proxies"})
         } else if msg == "Client: Timed out." {
                 err = w.Write([]string{timestamp, "client-gets-none", "",
 "", "timeout"})
         } else if matches := regexpRecvSnowflake.FindStringSubmatch(msg);
 matches != nil {
                 err = w.Write([]string{timestamp, "proxy-polls",
 idFor(matches[1]), "", ""})
         } else if matches := regexpProxyNoClient.FindStringSubmatch(msg);
 matches != nil {
                 err = w.Write([]string{timestamp, "proxy-gets-none",
 idFor(matches[1]), "", "no-clients"})
         } else if msg == "Passing client offer to snowflake." {
                 err = w.Write([]string{timestamp, "proxy-gets-offer", "",
 "", ""})
         } else if strings.HasPrefix(msg, "Received answer: ") {
                 err = w.Write([]string{timestamp, "proxy-answers", "", "",
 ""})
         } else if msg == "Client: Retrieving answer" {
                 err = w.Write([]string{timestamp, "client-gets-answer",
 "", "", ""})
         } else {
                 return fmt.Errorf("cannot parse line")
         }
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/30731#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online