[tor-dev] When RFC 7686 and transparent proxies collide

[kaizushi at cock.li]

On 2023-11-03 16:29, rhatto wrote:
> On Thu, Sep 14, 2023 at 08:22:22PM +1000, Matt Jolly wrote:
>> I recently inadvertently opened a much larger can of worms than I'd
>> intended when fixing a bug reported downstream where cURL would,
>> when configured with certain DNS backends, fail to resolve .onion
>> addresses.
>> 
>> [...]
> 
> Hi Matt,
> 
> Thanks for the detailed problem description and references, and I'm 
> sorry to
> hear about the trouble you're getting when trying to fix it.
> 
>> I've asked in the Tor Specifications issue (inspired by Silvio's
>> suggestions), and in the cURL issue, but I seem to be getting nowhere
>> and the impacted users are clamouring for a quick band-aid solution,
>> which I feel will work out worse for everyone in the long run:
>> > How can client applications (safely):
>> >
>> > 1.discover that they're in a Tor-enabled environment
>> > 2.resolve onion services only via Tor in that circumstance
>> > 3.not leak .onion resolution attempts at all
>> >
>> > Right now, not making these requests in the first place is the
>> > safest (and correct) thing to do, however inconvenient it may be.
>> > Rather than immediately trying to come up with a band-aid approach
>> > to this problem, a sane mechanism needs to be implemented to:
>> >
>> > 1.prevent each application from coming up with their own solution
>> > 2.prevent inconsistency in .onion resolution (i.e. no "oh it only
>> > leaks if DO_ONION_RESOLUTION is set")
>> > 3.provide a standardised mechanism for applications that want to be Tor
>> > aware to discover that they're in a Tor-enabled environment.
>> I'm not particularly attached to that last point, but it's worth 
>> discussing.
>> 
>> On a related note:
>> -is the use of a transparent proxy recommended?
> 
> I consider it a valid use case. The only concern I'm aware is about
> proxy leaks. I don't see any reason not to recommend transparent Tor
> proxies if using a configuration that avoids such leakage.
> 
>> -is there a sane alternative that involves as minimal configuration
>> as possible for these users?
> 
> Can't think of anything right now besides what was already covered in
> length in the threads you mentioned. Instead, I'll focus in the overall
> problem.
> 
>> I'm not sure what the best way forward is here, but I'm hoping that
>> actual Tor developers might have a useful opinion on the matter, or
>> at least be able to point me in the right direction.
> 
> The long analysis below does not necessarily represents an "official 
> Tor
> position". It may be frustrating that there is no "official" position
> right now, maybe because this still need further discussion, and also
> because it may be a hard topic.
> 
> # Interpreting RFC 7686
> 
> Let's start with an hermeneutical exercise, by reading carefully what
> RFC 7686 says:
> 
>> 2.  Application Software: Applications (including proxies) that
>>     implement the Tor protocol MUST recognize .onion names as special
>>     by either accessing them directly or using a proxy (e.g., SOCKS
>>     [RFC1928]) to do so.  Applications that do not implement the Tor
>>     protocol SHOULD generate an error upon the use of .onion and
>>     SHOULD NOT perform a DNS lookup.
> 
> Applications MUST support .onion either directly or through a proxy. 
> The RFC
> gives SOCKS as an example, but it could as well have mentioned 
> transparent
> proxies. So applications could easily allow environment variables, 
> command line
> flags, configuration parameters or "environment detection" to indicate 
> that
> they support .onion domains.
> 
> I don't see why an upstream shouldn't allow users and operating systems
> toggle the behavior regarding RFC 7686. Seems like the best default
> approach so far is to generate an error, but nothing is blocking
> applications to expose ways users can indicate that their systems acts
> as proxies for .onion.
> 
> I suggest we take a corollary from this RFC portion: applications 
> implementing
> the Tor protocol are those which either access .onion addresses 
> directly or use
> a proxy that supports Tor. If the application does not support any way 
> to
> properly resolve .onion addresses, then it must make sure to refuse any
> attempts to resolve .onion addresses. Nothing is blocking an 
> application to
> operate in two modes: rejecting .onion resolution attempts by default, 
> except
> if it's told to do so.
> 
> I agree that "implement the Tor protocol" is rather vague. The "Tor 
> protocol"
> might mean a lot of stuff, but I'm assuming here the _intention_ behind 
> this
> statement is this: in order to build a Tor-enabled application, one has 
> to
> interface with Tor somehow (being a Tor client or using a proxy).
> 
> As for curl, it seems to me that upstream wants uniform behavior with 
> all DNS
> backend libraries. And c-ares is rejecting .onion resolution, so for 
> the sake
> of uniformity curl is just rejecting it for all backends.
> 
> But the RFC explicitly mentions "Applications" and not "Libraries". 
> Shall we
> assume then that the RFC is referring to curl, but not to c-ares?
> 
> Would c-ares maintainers be free do follow whatever they prefer?
> 
> Or this is not a concern for c-ares, since it's a library and not an
> application? In this case they don't need to comply with the RFC, and
> could just rollback this change, so curl maintainers could be more open
> to accept toggling the RFC behavior.
> 
> We could say that users interact with applications, and these rely on
> libraries. By the RFC interpretation above, it's in the application 
> level that
> .onion resolution should be allowed or not. It just don't make any 
> statement
> about what libraries SHOULD or SHOULD NOT do, or even how a system 
> resolver
> should behave.
> 
> I don't really think curl maintainers should accept a toggle only if 
> c-ares is
> rolled back, but maybe this could help change their minds.
> 
> Alternatively, c-ares could provide some optional initialization
> parameter indicating it should actually try to resolve an .onion
> address, if that's feasible.
> 
> Also note the use of SHOULD in the RFC. As of BCP 14 / RFC 8174:
> 
>> 3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
>>    may exist valid reasons in particular circumstances to ignore a
>>    particular item, but the full implications must be understood and
>>    carefully weighed before choosing a different course.
>> 
>> 4. SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
>>    there may exist valid reasons in particular circumstances when the
>>    particular behavior is acceptable or even useful, but the full
>>    implications should be understood and the case carefully weighed
>>    before implementing any behavior described with this label.
> 
> From the discussion so far, there are enough "valid reasons in 
> particular
> circumstances" when things can be done differently. I haven't followed 
> RFC
> 7686's process, but maybe the authors and reviewers had carefully 
> chosen the
> terminology to acknowledge that there might be special cases -- such as
> transparent proxies -- where the RFC requirements can be relaxed, given 
> that
> implementors know what they're doing.
> 
> Hope this gives enough arguments to argue for implementing a tunable to 
> account
> for the transparent proxy use case in curl and in other software.
> 
> # Standardization
> 
> What seems more difficult is to standardize a way that any application 
> from any
> operating system could follow some setting, or auto-detect it's behind 
> an
> .onion-enabled proxy.
> 
> I don't see an easy way to solve this that would make everyone happy in 
> the
> short term.
> 
> But it would be possible to split proposals by timeframe (short, mid 
> and long
> term) and scope (specifications, libraries, applications, operating 
> systems).
> 
> Short timescales and "band-aid" fixes usually happen within libraries 
> and
> applications.
> 
> Operating system changes are somehow mid-term.
> 
> Tor specs ranges from quick fixes to mid or long term.
> 
> IETF standards are long term. Could take years to have a RFC amendment,
> and years more to implementors follow the it. RFC 7686 could have
> improved wording and be more detailed or nuanced in the possibilities.
> Right now we don't have any concrete plans or people engaged in 
> amending
> the RFC, but here and there people are discussing this. Example:
> https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/85
> 
> As for the quick fixes, I don't see nothing beyond advocating patches
> implementing flags, environment variables, parameters etc for a RFC
> 7686-compliant tunable. Unfortunately this have to be done in a 
> case-by-case
> basis, but having RFC 7686 may actually be an advantage when requesting 
> such
> changes. I imagine that not having the RFC would make way harder to 
> propose
> anything related.
> 
> # Approaches for transproxy auto-discovery
> 
> Now let's talk a bit about long-term, idealized fixes.
> 
> What follows is just brainstorming.
> 
> It could be considered some method for transproxy auto-discovery, be it
> some widespread standard or only followed by a subset of related
> applications and operating systems.
> 
> An .onion auto-detection feature could be implemented on applications
> by trying to detect some special-use domain like "proxy.onion"
> in two ways:
> 
> 1. By supporting a local hostname similar to "localhost" in many 
> systems. If a
>    name "proxy.onion" resolves locally (say to 127.0.0.1), then an
>    application (or library) could assume it's behind a proxy, be it 
> SOCKS5h or
>    in a system with Tor transparent proxy.
> 
> 2. By having an .onion-aware DNS resolver return a valid entry for
>    "proxy.onion":
>   * That won't leak the address, but may leak to the DNS that someone
>     is using a given library, which may be harmless if it's widespread.
> 
>   * The query could be for a TXT record, which could give more space 
> for
>     custom replies, but could be incompatible with the option 1 above.
> 
> Upon initialization, an application could do this test to determine
> whether it should try to resolve .onion addresses.
> 
> The Tor client would have to change as well, in order to support the 
> special
> "proxy.onion" address instead of triggering a malformed .onion address 
> error.
> 
>> Thanks for your time,
>> Cheers,
>> Matt
> 
> Thanks for all your effort and dedication put into this issue :)

They blatantly don't care and deny the issue, and don't care that it 
breaks a feature Tor has had for over a decade. They respond with 
fallacies and irrational garbage. They don't acknowledge that an 
environment variable would fix the issue and tacitly admit not to care.

Fortunately I run Gentoo and this stupid bit of code is very easy to fix 
on that distribution. I made this patch and used /etc/portage/patches to 
fix it, which is a page on the Gentoo Wiki: https://bpa.st/Q4VQc - I 
think this illustrates how simple this issue is as well as fixing it for 
all users.

However, the transproxy feature allows for people to idiot proof their 
systems and share with people which is supposed to be what open source 
is about. I hope as many people call out this kind of ignorant behavior 
for what it clearly is. You can see the depths of their stupidity and 
denial here...

https://github.com/curl/curl/discussions/11125