[anti-censorship-team] Trying to understand multi-bridge dynamics post after Fastly/Cloudflare domain front change

[David Fifield]

On Thu, Sep 21, 2023 at 09:26:58PM -0600, David Fifield wrote:
> I made a graph of the bandwidth on the two bridges since this started
> happening.
> 
> The two vertical lines mark:
> 2023-09-20 14:00:00	earliest known case of domain resolving to Cloudflare
> 2023-09-21 18:00:00	change to foursquare.com in rdsys
> 			https://gitlab.torproject.org/tpo/anti-censorship/rdsys-admin/-/merge_requests/17
> 
> 1. snowflake-02 bandwidth has dwindled to almost nothing. Seriously
>    almost nothing: it's around 3 MB/s currently.
> 2. There's a huge almost instantaneous step in snowflake-01 at around
>    2023-09-21 13:00:00. At first, I thought this might have been a
>    consequence of the rdsys change, but it's about 5 hours earlier than
>    that. What could it be? Some unrelated unblocking event that just
>    happened to happen while this domain stuff is happening?

The non-use of snowflake-02 continues -- see the attached graph. I'm
racking my brain trying to understand that is. snowflake-01 usage has
decreased a lot too -- the graph appears to be at about the same level,
but you can see it's not brickwalled at the upper end of the range as it
was before. Even ignoring the step anomaly at 2023-09-21 13:00:00, it
didn't go to zero like snowflake-02 did.

It may be that whatever decides whether you get a Fastly or a Cloudflare
edge server correlates highly with whether your client is capable of
using snowflake-02. My working assumption, so far, has been that Tor
Browser has multi-bridge support since 12.0 (2022-12-07), while Orbot
only has multi-bridge support in the unreleased Orbot 17
(https://github.com/guardianproject/orbot/releases/tag/17.0.0-BETA-2-tor.0.4.7.11
is the first beta release to have it). If Tor Browser users are mostly
on desktop, and mobile users are mostly on mobile/cellular, and DNS
resolution for cdn.sstatic.net also correlates with desktop vs. mobile,
then that could explain it. It would mean that ~100% of Tor Browser
users are getting a Cloudflare IP address, and <100% of Orbot users are.

But it's not the case that Orbot 17 is totally unreleased. The Play
Store currently has 16.6.3-RC-1-tor.0.4.7.10 released 2022-11-01:
https://web.archive.org/web/20230925022736/https://play.google.com/store/apps/details?id=org.torproject.android
But Orbot 17 betas are available (most recent is 2023-08-09
https://github.com/guardianproject/orbot/releases/tag/17.1.0-BETA-3-tor.0.4.7.14),
and version 17 is in F-Droid:
http://meetbot.debian.net/tor-meeting/2023/tor-meeting.2023-09-21-15.57.log.html#l-132
	<dcf1> Orbot 17 has both bridges, but it's not released yet, except in beta, afaik. I walways thought that was the cause of the low use of snowflake-02, that we were just waiting for Orbot to make a full release of v17. But maybe it is more complicated.
	<meskio> mmm, I have here orbot 17, so I guess I'm using the beta...
	<meskio> is in fdroid
So even if the correlation hypothesis were correct, I wouldn't expect
snowflake-02 to drop as far as it has.

Maybe the bridge selection at the client is not as random as we intend?
Even though there are two bridge lines, maybe tor systematically prefers
the one that's listed first? The idea here is that maybe snowflake-02
only gets used when snowflake-01 is past its capacity and starts to fail
connection attempts. With the suddenly reduced overall level of users,
there's enough headroom that snowflake-02 essentially never gets used.

A possible explanation for the sudden step in snowflake-01 usage at
2023-09-21 13:00:00 is that there's a population of Snowflake clients
out there other than the ones we are responsible for. Whoever is
distributing the clients for that population may have noticed the
cdn.sstatic.net change and deployed their own mitigation, separate from
anything we have done. The step only took about 15 minutes (see the
second attached graph), which is a pretty fast deployment. If that other
imagined deployment only knows about snowflake-01, that could explain
why the step appears in snowflake-01's graph and not snowflake-02's. It
still doesn't explain why, before the step, snowflake-01 took a big hit
to users but did not go to zero, while snowflake-02 kept declining.

Maybe we have an undetected bug in multi-proxy support that favors
snowflake-01? The broker is supposed to reject proxies that do not have
multi-bridge support since 2022-10-03:
https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40193
But maybe it's not working the way it's supposed to? Maybe it's easier
to get a proxy for snowflake-01 than for snowflake-02?

Maybe there's something wrong with the snowflake-02 bridge? I've been
using snowflake-02 all day today (using AMP cache rendezvous). In the
morning, it did seem to be a little screwy -- I couldn't get a YouTube
video to play without frequent stops. One time, I happened to notice
these messages in the log; they may be unrelated, but perhaps there is
some weird interaction with Conflux:
	2023-09-24 19:03:16.550 [NOTICE] Failed to find node for hop #1 of our path. Discarding this circuit.
	2023-09-24 19:03:16.552 [NOTICE] Our circuit 0 (id: 38) died due to an invalid selected path, purpose Unlinked conflux circuit. This may be a torrc configuration issue, or a bug.
	2023-09-24 19:22:50.237 [NOTICE] Failed to find node for hop #1 of our path. Discarding this circuit.
I checked the bridge to ensure that the expected version of the server
software was deployed (commit 0a6aeda9), and it was.

While I was using Tor Browser, I let it upgrade to 13.0a5. 13.0a5 has a
fix to the default bridge lines, but I uses a manual bridge line so I
would only be on snowflake-02. After the upgrade, it started working
better and I could watch YouTube as normal. Maybe it was just a
concidence that the upgrade to 13.0a5 seemed to improve the performance?
In any case, from watching bandwidth on the bridge, it looks like I've
had the bridge mostly to myself all day.

Just for good measure, I upgraded tor on snowflake-02 from
0.4.7.13-1~focal+1 to 0.4.8.6-1~focal+1 at 2023-09-24 20:48:25.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: snowflake-bandwidth-20230925.png
Type: image/png
Size: 32073 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20230924/e1890b90/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: snowflake-bandwidth-20230921-zoom.png
Type: image/png
Size: 19274 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20230924/e1890b90/attachment-0003.png>