[tor-dev] Raising exceptions in add_event_listener() threads (was Re: HSv3 descriptor work in stem)

[teor]

Hi George,

> On 28 Nov 2019, at 02:04, George Kadianakis <desnacked at riseup.net> wrote:
> 
> Hello Damian (and list),
> 
> here is another question about an issue I have encountered while
> developing onionbalance v3.
> 
> In particular, I'm fetching HS descriptors using HSFETCH and then adding
> an add_event_listener() event to a function that does the descriptor
> parsing and handling as follows:
> 
>        controller.add_event_listener(handle_new_desc_content_event, EventType.HS_DESC_CONTENT)
> 
> The problem is that the handle_new_desc_content_event() callback has
> grown to a non-trivial size and complexity, since it needs to parse the
> descriptor, put it in the right places, and tick off the right
> checkboxes.
> 
> Since its size has increased, so has the number of bugs and errors that
> are appearing during development. The problem is that because the
> callback is running on a separate thread (?) any errors and exceptions
> that get raised in that thread never surface to the my console and hence
> I don't see them. This means that I need to do very tedious printf
> debugging to find the exact place and type of error everytime something
> happens.
> 
> What's the proper way to do debugging and development in callbacks like
> that? Is there a way to make the exceptions float back to the main
> thread or something? Or any other tips?

In general, there are two ways to avoid exceptions disappearing in python
threads, a code change, and a design change.

Code Change

The code change catches all exceptions in a thread, you should be able to
do it right now. (Or Damian could implement it in stem, every time a thread
is launched.)

1. Wrap all code called in a thread in a try/catch block
2. Catch every exception
3. Log it
4. Stop the thread/process/...

The code looks a bit like this:

try:
    run_thread()
except e:
    print("Error in thread")
    log_error()
    # Pick one of these actions
    # Terminate the process immediately
    os._exit(1)
    # Re-raise the error - does this terminate the thread?
    raise
    # Stop the thread
    return 1

You'll want a log_error() function like this one:
https://github.com/privcount/privcount/blob/master/privcount/log.py#L19

Another alternative is to join() threads when they terminate, and re-raise
the exception in the main thread. But the code is a lot more complex:
https://stackoverflow.com/a/6874161

Design Change

The design change moves all the work to the main thread, and just stores data/
sets a flag from other threads. That's not always possible, but it's usually a
good idea to limit work in threads, because it avoids races, delays, and
deadlocks.

In this case, you could store the data in the fetch thread, and then do the
processing the next time the main thread tries to access that data, or anything
related to it.

T