[tor-bugs] #13720 [Ooni]: Investigate possible performance improvements to the ooni-pipeline

[Tor Bug Tracker & Wiki]

#13720: Investigate possible performance improvements to the ooni-pipeline
-----------------------------+---------------------
     Reporter:  hellais      |      Owner:  hellais
         Type:  enhancement  |     Status:  new
     Priority:  normal       |  Milestone:
    Component:  Ooni         |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+---------------------

Comment (by dcf):

 Here is another data point using many more reports. All the http_requests
 reports until 2015-07-17, 2,758 of them, 33 GB compressed.
 {{{
 yaml.load_all(f, Loader=yaml.CSafeLoader)
 real    133m23.210s
 user    147m24.860s
 sys     2m8.624s
 }}}
 This is faster than processing 1.5 GB of files using yaml.safe_load_all
 (and the processing program is even doing more work now). The processing
 program grabs the bodies of all responses, runs a bunch of regexes against
 them, and optionally saves the bodies to files.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/13720#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online