[tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances
Tor Bug Tracker & Wiki
blackhole at torproject.org
Wed Oct 26 07:33:55 UTC 2016
#18910: distributing descriptors accross CollecTor instances
-------------------------------+---------------------------------
Reporter: iwakeh | Owner: iwakeh
Type: enhancement | Status: needs_review
Priority: High | Milestone: CollecTor 1.1.0
Component: Metrics/CollecTor | Version:
Severity: Normal | Resolution:
Keywords: ctip | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
-------------------------------+---------------------------------
Comment (by karsten):
Here's a fresh bug report: synchronizing relay descriptors takes a few
''hours'' on average on my laptop, and I think I know why. Whenever we
append to an existing file in `recent/`, we copy the entire file content
to a temporary file, append the new content there, and then rename. And
we do that for every single server descriptor or extra-info descriptor we
receive. That is bad for two reasons. First reason is that it obviously
doesn't scale, with some sync runs taking 4 hours or longer. Second
reason is that files in `recent/` shouldn't change after they're included
in `index.json` (with the inglorious exception of Torperf measurements),
which is why we should append all descriptors to a temporary file and
rename when we're sure we're done. See also
`ArchiveWriter.cleanUpRsyncDirectory()` where we're renaming temporary
files to destination files at the very end of the update run.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/18910#comment:88>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
More information about the tor-bugs
mailing list