[tor-dev] Draft patchset for PT2.0 support (ticket #21816)

Sun Aug 20 21:48:43 UTC 2017

Good day, tor-dev;

As some of you know, I've been working on a patchset for Tor to allow
it to participate in Pluggable Transports 2.0 configuration, primarily
the new JSON Parameter Block SOCKS method (at least that's what I've
been calling it in the absence of a more official name), almost but
not quite as described in section 3.3.4 of PT2 draft 2 [1]---i.e.,
basically #21816 [2].

[1] https://www.pluggabletransports.info/assets/PTSpecV2Draft2.pdf
[2] https://trac.torproject.org/projects/tor/ticket/21816

I'm attaching a draft patchset which adds this functionality, with the
intent of getting feedback and making remaining cleanups or other
modifications necessary to get it merged into Tor. I have successfully
completed circuits through an obfs4 bridge using both obfs4proxy (PT1)
and a version of shapeshifter-dispatcher (PT2) using a patched Tor. I've
tried to follow the local style, but the preferred implementation
strategies aren't always clear, and of course I'd appreciate any reports
of other problems.

A forked Git repository is also available on Bitbucket [5][6], which
will be updated as I make remaining changes.

[5] https://bitbucket.org/DasyatidPrime/tor-rtt2017-21816.git (Git)
[6] https://bitbucket.org/DasyatidPrime/tor-rtt2017-21816/src (Web)

More implementation details are below if you're interested in this;
thanks for your attention. I'll try to be around on IRC more during
the week, so feel free to ping me there as well.

-RTT

... Details:

Not visible above, related to the target functionality:

  - I'm assuming the SOCKS method includes a response with an
    analogous structure to RFC 1929; PT2 draft 2 doesn't specify
    one. I've cleared this with blanu, and that's intended for the
    next draft of the PT2 specification.

  - Similarly, the length prefix is in fact big-endian; the example in
    PT2 draft 2 is wrong, though the text is correct.

  - We're looking into getting an IANA assignment for the method
    number. Technically this probably meets the requirements for the
    private use block, but I feel like interop might be easier later
    on, and it could simplify code paths in places to have a registered
    number (mainly if it's possible to decouple the method negotiation
    from the configuration-version plumbing). I believe this is still
    in limbo.

  - For the PT2 side, I've been testing against my branch of
    shapeshifter-dispatcher [3] compiled with my branch of
    shapeshifter-ipc [4] since there were some breaking changes to
    Shapeshifter upstream which were otherwise preventing it from
    interoperating with Tor. I'm planning to help merge fixes into
    Shapeshifter upstream as I can; it's my current understanding that
    there aren't any other PT2 managed transport implementations to
    test against.

[3]
https://github.com/OperatorFoundation/shapeshifter-dispatcher/tree/rtt2017
[4] https://github.com/OperatorFoundation/shapeshifter-ipc/tree/rtt2017

Issues on my radar currently (comments appreciated):

  - We probably want unit tests for the (limited) JSON encoding
    functions, and for the factored-out RFC 1929 encoding functions.
    Anything else that looks feasibly testable here?

  - It's not clear to me whether negotiating a PT2 configuration
    version still allows PT1-style RFC 1929 parameter encoding so that
    managed transports can support the new configuration version and
    the new SOCKS method separately. I've assumed it might be
    possible, so far, but that's not being tested against anything.

  - I currently restrict parameters to ASCII to avoid either writing
    a JSON encoder that can spit out invalid JSON or writing a JSON
    encoder that has to validate incoming UTF-8. The impression I've
    gotten is that this is probably okay, but if there are counter-
    examples, I can put in UTF-8 passthrough.

  - The commit sequence isn't the cleanest. How high a priority is it
    to reorder/combine patch hunks to make a cleaner one?

  - We still need a 'changes' file. (What would be an appropriate
    heading for this? Is this a minor feature, for instance?)

A few other questions:

  - Is there an effective way of doing automated testing of the SOCKS
    state machine currently in Tor? I didn't see anything obvious in
    the test directory. This seems like the most fragile part,
    especially since both the original and modified versions are not
    very explicit in their state machine nature and are split between
    multiple files.

  - Can there ever be more than one managed_proxy_t to a transport
    name? More generally, is there a relational diagram of the main
    Tor data structures somewhere? A lot of the way the plumbing for
    state and configuration information is set up feels kind of
    fragile.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-PT2.0-Add-JSON-encoding-functions-for-new-arg-blocks.patch
Type: text/x-patch
Size: 5967 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0012.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-PT2.0-Add-WANT_AUTH_METHOD_PT2-flows-to-SOCKS-handli.patch
Type: text/x-patch
Size: 17530 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0013.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-PT2.0-Update-connection_proxy_state_to_string-for-ne.patch
Type: text/x-patch
Size: 731 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0014.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-PT2.0-Add-note-of-limit-of-proxy_state-bitfield-near.patch
Type: text/x-patch
Size: 839 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0015.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0005-PT2.0-Change-bridge-parameter-length-check-from-erro.patch
Type: text/x-patch
Size: 1232 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0016.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0006-PT2.0-Adjust-outgoing-SOCKS-auth-negotiation-for-PT.patch
Type: text/x-patch
Size: 2729 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0017.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0007-PT2.0-Remove-failing-test-for-huge-bridge-lines.patch
Type: text/x-patch
Size: 1588 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0018.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0008-PT2.0-Add-version-2-to-configuration-negotiation.patch
Type: text/x-patch
Size: 2544 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0019.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0009-PT2.0-Fail-if-we-have-PT-parameters-and-can-t-send-t.patch
Type: text/x-patch
Size: 1326 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0020.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0010-PT2.0-Add-test-for-conf-proto-2-before-negotiating-m.patch
Type: text/x-patch
Size: 7069 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0021.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0011-PT2.0-Use-symbolic-constants-for-SOCKS-methods.patch
Type: text/x-patch
Size: 5193 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0022.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0012-PT2.0-Fix-whitespace.patch
Type: text/x-patch
Size: 1898 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0023.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20170820/f315b8f7/attachment-0001.sig>