[tor-dev] Running a Separate Tor Network

[Tom Ritter]

On 22 October 2014 05:48, Roger Dingledine <arma at mit.edu> wrote:
>> What I had to do was make one of my Directory Authorities an exit -
>> this let the other nodes start building circuits through the
>> authorities and upload descriptors.
>
> This part seems surprising to me -- directory authorities always publish
> their dirport whether they've found it reachable or not, and relays
> publish their descriptors directly to the dirport of each directory
> authority (not through the Tor network).
>
> So maybe there's a bug that you aren't describing, or maybe you are
> misunderstanding what you saw?
>
> See also https://trac.torproject.org/projects/tor/ticket/11973
>
>> Another problem I ran into was that nodes couldn't conduct
>> reachability tests when I had exits that were only using the Reduced
>> Exit Policy - because it doesn't list the ORPort/DirPort!  (I was
>> using nonstandard ports actually, but indeed the reduced exit policy
>> does not include 9001 or 9030.)  Looking at the current consensus,
>> there are 40 exits that exit to all ports, and 400-something exits
>> that use the ReducedExitPolicy.  It seems like 9001 and 9030 should
>> probably be added to that for reachability tests?
>
> The reachability tests for the ORPort involve extending the circuit to
> the ORPort -- which doesn't use an exit stream. So your relays should
> have been able to find themselves reachable, and published a descriptor,
> even with no exit relays in the network.

I think I traced down the source of the behavior I saw.  In brief, I
don't think reachability tests happen when there are no Exit nodes
because of a quirk in the bootstrapping process, where we never think
we have a minimum of directory information:

Nov 09 22:10:26.000 [notice] I learned some more directory
information, but not enough to build a circuit: We need more
descriptors: we have 5/5, and can only build 0% of likely paths. (We
have 100% of guards bw, 100% of midpoint bw, and 0% of exit bw.)

In long form: https://trac.torproject.org/projects/tor/ticket/13718




>> Continuing in this thread, another problem I hit was that (I believe)
>> nodes expect the 'Stable' flag when conducting certain reachability
>> tests.  I'm not 100% certain - it may not prevent the relay from
>> uploading a descriptor, but it seems like if no acceptable exit node
>> is Stable - some reachability tests will be stuck.  I see these sorts
>> of errors when there is no stable Exit node (the node generating the
>> errors is in fact a Stable Exit though, so it clearly uploaded its
>> descriptor and keeps running):
>
> In consider_testing_reachability() we call
>
>     circuit_launch_by_extend_info(CIRCUIT_PURPOSE_TESTING, ei,
>                             CIRCLAUNCH_NEED_CAPACITY|CIRCLAUNCH_IS_INTERNAL);
>
> So the ORPort reachability test doesn't require the Stable flag.

You're right, reachability doesn't depend on Stable, sorry.



>> I then added auth5 to a second DirAuth (auth2) as a trusted DirAuth.
>> This results in a consensus for auth1, auth2, and auth5 - but auth3
>> and auth4 did not sign it or produce a consensus.  Because the
>> consensus was only signed by 2 of the 4 Auths (e.g., not a majority) -
>> it was rejected by the relays (which did not list auth5).
>
> Right -- when you change the set of directory authorities, you need to
> get a sufficient clump of them to change all at once. This coordination
> has been a real hassle as we grow the number of directory authorities,
> and it's one of the main reasons we don't have more currently.

I'm going to try thinking more about this problem.



> This was fixed in git commit c03cfc05, and I think the fix went into
> Tor 0.2.4.13-alpha. What ancient version is your man page from?

/looks sheepish
I was using http://linux.die.net/man/1/tor because it's very quick to
pull up :-p


>>  And how there _is no_
>> V3AuthInitialVotingInterval?  And that you can't modify these
>> parameters without turning on TestingTorParameters (despite the fact
>> that they will be used without TestingTorNetwork?)  And also,
>> unrelated to the naming, these parameters are a fallback case for when
>> we don't have a consensus, but if they're not kept in sync with
>> V3AuthVotingInterval and their kin - the DirAuth can wind up
>> completely out of sync and be unable to recover (except by luck).
>
> Yeah, don't mess with them unless you know what you're doing.
>
> As for the confusing names, you're totally right:
> https://trac.torproject.org/projects/tor/ticket/11967

Ahha.


>>  - The Directory Authority information is a bit out of date.
>> Specifically, I was most confused by V1 vs V2 vs V3 Directories.  I am
>> not sure if the actual network's DirAuths set V1AuthoritativeDirectory
>> or V2AuthoritativeDirectory - but I eventually convinced myself that
>> only V3AuthoritativeDirectory was needed.
>
> Correct. Can you submit a ticket to fix this, wherever you found it?
> Assuming it wasn't from your ancient man page that is? :)

It was.



>>  - The networkstatus-bridges file is not included in the tor man page
>
> Yep. Please file a ticket.

https://trac.torproject.org/projects/tor/ticket/13713

>>  - I feel like the log message "Consensus includes unrecognized
>> authority" (currently info) is worthy of being upgraded to notice.
>
> I don't think this is wise -- it is fine to have a consensus that has
> been signed by a newer authority than you know about, so long as it has
> enough signatures from ones you do know about.
>
> If we made this a notice, then every time we added a new authority,
> all the users running stable would see scary-sounding log messages and
> report them to us over and over.

That's fair.


>>  - I wanted the https://consensus-health.torproject.org/ page for my
>> network, but didn't want to run the java code, so I ported it to
>> python.  This project is growing, and right now I've been editing
>> consensus_health_checker.py as well.
>> https://github.com/tomrittervg/doctor/commits/python-website  I have a
>> few more TODOs for it (like download statistics), but it's coming
>> along.
>
> Neat! Karsten has been wanting to get rid of the consensus-health page
> for a while now. Maybe you want to run the replacement?

Yes, I think that is going to happen:
https://trac.torproject.org/projects/tor/ticket/13637


>> Finally, something I wanted to ask after was the idea of a node (an
>> OR, not a client) belonging to two or more Tor networks.  From the POV
>> of the node operator, I would see it as a node would add some config
>> lines (maybe 'AdditionalDirServer' to add to, rather than redefining,
>> the default DirServers), and it would upload its descriptors to those
>> as well, fetch a consensus from all AdditionalDirServers, and allow
>> connections from and to nodes in either.  I'm still reading through
>> the code to see which areas would be particularly confusing in the
>> context of multiple consensuses, but I thought I'd throw it out there.
>
> This idea should work in theory. In fact, back when Ironkey was
> running their own Tor network, I joked periodically about just
> dumping the cached-descriptors file from their network into moria1's
> cached-descriptors file. I think that by itself would have been sufficient
> to add all of those relays into our Tor network.

Curious.  I will try running this idea down a bit more.

> We're slowly accumulating situations where we want all the relays to
> know about all the relays (e.g. RefuseUnknownExits), but I don't think
> the world ends when it isn't quite true.

Sure, but that only matters if you're trying to bridge Tor networks
without cooperation - a Tor node that wants to sit on two networks
wouldn't have a problem knowing about all the nodes in each.  And a
Tor client using network A or B would route only through that network.
I didn't imagine them as interleaving, I imagined them as separate,
with some relay operators opting to move traffic for both.

-tom