[tor-bugs] #32576 [Circumvention/Snowflake]: Fix race condition in snowflake broker (was: Snowflake broker not serving metrics correctly)

[Tor Bug Tracker & Wiki]

#32576: Fix race condition in snowflake broker
-------------------------------------+---------------------------
 Reporter:  cohosh                   |          Owner:  cohosh
     Type:  defect                   |         Status:  assigned
 Priority:  Very High                |      Milestone:
Component:  Circumvention/Snowflake  |        Version:
 Severity:  Normal                   |     Resolution:
 Keywords:  metrics                  |  Actual Points:
Parent ID:                           |         Points:
 Reviewer:                           |        Sponsor:  Sponsor28
-------------------------------------+---------------------------

Old description:

> Looks like the snowflake broker isn't serving metrics since 2019-11-14
> (which happens to coincide with the host migration.
>
> At first I thought this was a DNS issue and filed #32570, but even when I
> manually fetch https://snowflake-broker.torproject.net/metrics I'm
> getting outdated metrics.
>
> I've attached the result of the above fetch as well as the metrics file
> located on the snowflake broker host that the broker *should* be reading
> from.

New description:

 There is a race condition with the snowflake heap that has been causing
 the broker to crash several times a day. This race condition has existed
 in the broker for several years, but some recent updates as well as the
 host migration managed to shake it loose.

 ----

 This race condition is causing the snowflake broker to crash repeatedly
 and often since the migration. We noticed because CollecTor stopped
 collecting metrics since the restart on 14 November 2019.

--

Comment (by cohosh):

 Just updating the summary of this ticket to reflect the actual problem.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32576#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online