[tor-project] [TSoP] stem.client - status report #4

[Dave Rolek]

Hi everyone!

This is my fourth status report for the stem.client TSoP project [1].
You may see my previous reports on the tor-project ml. [2][3][4]

This report is much of a week late - I'm quite sorry about that.


# Updates

I've still been feeling ill, and it's been a bit unpredictable for when
symptoms flare up, so that's been tough to deal with and work around.
(I'm still moving forward, though, just not as effectively as I'd like.)

Some of my progress comes in longer-term plans than I can effectively
showcase here, but I'll try.

## GitHub mirroring

I think #26197 [5] sparked a lot of discussion on Tor's use of GitHub as
a mirror and means to accept PRs. Preliminary discussions when trying to
move the ticket forward by bringing it up in a network-team meeting
revealed a bit of uncertainty into how mirroring was done, and (iiuc)
eventually led to Isa getting involved to survey all the teams on
current and future usage.

It looks like things are working out well for stem now :), and overall
Tor's GitHub usage seems better understood and better standardized!

(For instance, the most recent PR was marked as auto-merged/closed by
'torproject-pusher' today [6])

I'll continue to be involved as needed to facilitate stem and nyx needs
w.r.t. GitHub repos. (Damian mentioned he's got a lot on his plate
already and asked if I could take the lead on this.)

nyx still doesn't have a repo, so I'll be sure to follow up on this.

## stem.client - re-roadmapping

I continued to push some lingering things in code review [7] and
architecture [8] discussions forward.

That led to a reconsideration of goals and roadmap between Damian, teor,
and me.

Based on the primary goal of prototype allowing exit traffic, I've
mapped out the main tasks of the remainder of the project as follows:

1. refactor: Relay cell encryption/decryption/processing
2. centralize ORPort reads/sends (demux/mux) at connection level
3. implement required RelayCell subclasses (e.g. parsing of decrypted
body)
4. <potentially: other RelayCell subclasses>
5. use these RELAY cells in Circuit / Stream construction/management
6. re-envision Circuit.send() and other methods (e.g. Circuit.extend())
7. refactor into Stream class, subclass(es)
8. introduce "dumb" path selection for exit traffic

This roadmap will be re-evaluated as progress on #1-3 is made.

## stem.client - further review, fixes

A few tickets/PRs have been done to follow up on Damian's changes [9],
which followed up on the previous commits I did from code review. [10]

These two tickets/PRs include bug fixes and some other improvements:
* NetinfoCell doesn't preserve unused content during unpacking [11]
* Cell unused content is ignored while packing [12][6] 

## stem.client - RELAY cells

I've started outlining the class hierarchy for the required RELAY cells,
tackling roadmap items #1-3. It's still in very early stages, so I
don't have changes ready to push.

## stem, general

Nothing to show this time around. I've taken minor notes for 'dev'
tickets, but I won't be pursuing much of these yet, in the interest of
focusing on the roadmap first.

## tor protocol, general

I'm still glancing through the tor-design paper [13][14][15] and updates
[16][17][18] while reading the specification. I think I've read through
enough of the paper now that I won't need to glance often, but I'm still
keeping it in mind.

As I've been working through the spec, I've been tracking changes to
make that improve it. Some of these are just shorthand notes (so I don't
have them available to share in an understandable way), but others have
made it to (probably) solid commits.

I've pushed a WIP branch to GitHub [19], which I will be rebasing /
force-pushing as necessary. *Please* feel free to check it out - just
don't rely on it to be git-stable.


# What's next

## stem.client

I'll be working on the roadmap mentioned above.

The first thing I'll be working out into commits is moving some of the
bytes-level handling of RELAY cells out of the Circuit class and into
cell.py, since this handling currently seems to violate the abstractions
we want.

## stem, general

I still have a few 'dev' tickets I'd like to work on. For a while I've
wanted to work on the tox configuration, but I may break that up into
more important and nice-to-have changes to it.

## tor-spec

As aforementioned, I'll continue to push forward on improvements to
tor-spec.txt.

Again, you may see what I've got mostly push-ready in my WIP branch on
GitHub.

I'll bring this up in the next network-team meeting and figure out the
best logistics for merging these in.


# Other Tor things

Not too much on the side - still an advocate for Tor in-person when
people chat with me, but it hasn't happened as much in the past few
weeks.

Still trying to attend the Community team meetings when I can.


# Closing

As with before, I've been active and reachable over IRC, where my nick
is dmr. Please don't hesitate to reach out to me via IRC or email!

I've been *much* less proactive in reading scrollback recently, in order
to combat some of the reduced capacity due to illness. Please mention my
nick if you'd like to get my attention, otherwise it's *very* unlikely
I'll see it outside of a meeting!

Thanks,
Dave


[1]
https://lists.torproject.org/pipermail/tor-dev/2018-April/013090.html
[2]
https://lists.torproject.org/pipermail/tor-project/2018-May/001811.html
[3]
https://lists.torproject.org/pipermail/tor-project/2018-June/001830.html
[4]
https://lists.torproject.org/pipermail/tor-project/2018-June/001858.html
[5] https://trac.torproject.org/projects/tor/ticket/26197
[6] https://github.com/torproject/stem/pull/5
[7] https://trac.torproject.org/projects/tor/ticket/26227
[8] https://trac.torproject.org/projects/tor/ticket/26226
[9]
https://gitweb.torproject.org/stem.git/log/?id=3818cf41cae98ae7558f5002ef3a5152ede5b2fb
[10] https://github.com/torproject/stem/pull/1
[11] https://trac.torproject.org/projects/tor/ticket/26684
[12] https://trac.torproject.org/projects/tor/ticket/26766
[13] https://www.torproject.org/docs/documentation.html.en#DesignDoc
[14] https://svn.torproject.org/svn/projects/design-paper/tor-design.pdf
[15]
https://svn.torproject.org/svn/projects/design-paper/tor-design.html
[16]
https://blog.torproject.org/blog/top-changes-tor-2004-design-paper-part-1
[17]
https://blog.torproject.org/blog/top-changes-tor-2004-design-paper-part-2
[18]
https://blog.torproject.org/blog/top-changes-tor-2004-design-paper-part-3
[19] https://github.com/dmr-x/torspec/commits/WIP-torspec-improvements