[metrics-bugs] #21932 [Metrics/metrics-lib]: Stop relying on the platform's default charset

[Tor Bug Tracker & Wiki]

#21932: Stop relying on the platform's default charset
---------------------------------+-----------------------------------
 Reporter:  karsten              |          Owner:  metrics-team
     Type:  defect               |         Status:  needs_information
 Priority:  Medium               |      Milestone:
Component:  Metrics/metrics-lib  |        Version:
 Severity:  Normal               |     Resolution:
 Keywords:                       |  Actual Points:
Parent ID:                       |         Points:
 Reviewer:                       |        Sponsor:
---------------------------------+-----------------------------------

Comment (by karsten):

 I also looked a bit more at the code.  I did not find any code in metrics-
 lib that would convert strings to US-ASCII before writing them to disk.
 Regarding the three charsets, those are not produced by metrics-lib but by
 Tor relays and kept unchanged by CollecTor which only reads and writes
 byte arrays.  The issue discussed here only affects reading and parsing
 descriptors, not writing descriptors.

 I also looked at archived descriptors to see which descriptor types are
 affected.  I only found relay server descriptors and version 1 Tor
 directories from 2007 and before that internally contain relay server
 descriptors as well.  The latter are parsed in metrics-lib using the same
 code that also parses server descriptors.  (I didn't find bridge server
 descriptors to be affected, but the reason is that CollecTor's bridgedescs
 module converts them to ASCII, which is something we should reconsider,
 too.)  In summary, if we fix server descriptor parsing in metrics-lib,
 we're good.

 So, you have a point that this ticket doesn't only address becoming
 independent of platform charset settings, but that is also suggests
 switching from ASCII (default charset on Debian stable) to UTF8.  I think
 we should do this anyway.  Though it's potentially surprising and also
 backwards-incompatible, so I believe we'll have to call the next release
 2.0.0 if we want to put this in.  Works for me, but what do you think?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/21932#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online