[tor-dev] Trouble with onionperf visualize and S61 performance experiments

[Karsten Loesing]

On 2020-11-23 16:49, George Kadianakis wrote:
> Karsten Loesing <karsten at torproject.org> writes:
> 
>> On 2020-11-03 17:16, Karsten Loesing wrote:
>>> On 2020-11-03 15:01, George Kadianakis wrote:
>>>> Hello Karsten,
>>>
>>> Hi George!
>>
>> Hi again!
>>
>>>> hope you are doing well!
>>>>
>>>> I've been working on the S61 performance experiments [0] and I would appreciate
>>>> some help with onionperf.
>>>>
>>>> I have done various onionperf measurements using something the following command:
>>>>   $ onionperf measure -i --tgen ~/tgen/build/src/tgen --tor ~/onionperf/tor/src/app/tor --drop-guards 10
>>>>
>>>> I put each of the measurements on a different directory and now I want
>>>> to analyze them and derive the CDF-TTFB graphs etc. I attempted doing
>>>> that using the following calls:
>>>>
>>>>  $ onionperf analyze --tgen ./tgen-client/onionperf.tgen.log --torctl ./tor-client/onionperf.torctl.log
>>>>  $ onionperf visualize --data onionperf.analysis.json.xz "test"
>>>>
>>>> Unfortunately, the 'visualize' call can fail for the attached 'onionperf-mbps.json.xz':
>>>>
>>>>   $ onionperf visualize --data onionperf.analysis.json.xz "Test Measurements"
>>>>   2020-11-03 15:51:31 1604411491.540736 [onionperf] [INFO] loading analysis results from /user/tmp/onionperf/analysis/onionperf.analysis.json.xz
>>>>   2020-11-03 15:51:31 1604411491.577864 [onionperf] [INFO] done!
>>>>   2020-11-03 15:51:31 1604411491.586845 [onionperf] [INFO] NumExpr defaulting to 8 threads.
>>>>   /user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/onionperf/visualization.py:251: UserWarning: Attempting to set identical left == right == -1e-06 results in singular transformations; automatically expanding.
>>>>   /user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/onionperf/visualization.py:251: UserWarning: Attempting to set identical left == right == -1e-06 results in singular transformations; automatically expanding.
>>>>   Traceback (most recent call last):
>>>>     File "/user/.local/bin/onionperf", line 4, in <module>
>>>>       __import__('pkg_resources').run_script('OnionPerf==0.8', 'onionperf')
>>>>     File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 650, in run_script
>>>>       self.require(requires)[0].run_script(script_name, ns)
>>>>     File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1453, in run_script
>>>>       exec(script_code, namespace, namespace)
>>>>     File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/EGG-INFO/scripts/onionperf", line 622, in <module>
>>>>     File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/EGG-INFO/scripts/onionperf", line 382, in main
>>>>     File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/EGG-INFO/scripts/onionperf", line 522, in visualize
>>>>     File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/onionperf/visualization.py", line 48, in plot_all
>>>>     File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/onionperf/visualization.py", line 205, in __plot_throughput_ecdf
>>>>     File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/onionperf/visualization.py", line 235, in __draw_ecdf
>>>>     File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 5000, in dropna
>>>>       raise KeyError(list(np.compress(check, subset)))
>>>>   KeyError: ['mbps']
>>>
>>> Indeed, that's a bug in the visualize mode.
>>>
>>> However, before it fails it writes a .csv file that tells us why: none
>>> of the measurements are successful! I'm seeing lots of TOR/CANT_ATTACH
>>> errors in that file. There's something wrong in your measurement setup.
>>> If you fix that, you'll be able to visualize the results.
>>
>> Did you figure out what went wrong? Do you need help figuring that out?
>>
>>> (We should still fix the bug and produce a nicer error message.)
>>
>> I'm going to file an issue and start working on a possible fix tomorrow.
>>
>> All the best,
>> Karsten
>>
> 
> Hey hey Karsten,

Hello!

> yes there was an issue with the port forwarding (or the incoming IP
> addr) and tgen could not do its thing, and I didn't realize because
> there were not any errors exposed to this effect.

Understood. I'm not sure how to make it easier to spot configuration
errors like this in the future. In fact, we had a very similar problem
with one of the long-running instances, op-hk5, and it took us two weeks
to figure that out. I just opened a issue to discuss possible fixes:

https://gitlab.torproject.org/tpo/metrics/onionperf/-/issues/40011

I also opened an issue to avoid the tracebacks you ran into in the future:

https://gitlab.torproject.org/tpo/metrics/onionperf/-/issues/40012

> In any case, I fixed this and then onionperf worked just fine. For
> example see here https://gitlab.torproject.org/tpo/core/tor/-/issues/40157#note_2714605
> 
> So no worries about this, it's all good on this front.
> 
> Also onionperf has been perfoming just fine in general for the purposes
> of #40157 so far.

Awesome! Please let me know if there's anything else causing trouble. Or
just open an issue.

> Cheers!
> (and welcome back (?))

Yes, I'm back!

Cheers!
Karsten