[tor-bugs] #29280 [Core Tor/Tor]: Use Chutney in Tor's CI

[Tor Bug Tracker & Wiki]

#29280: Use Chutney in Tor's CI
-------------------------------------------------+-------------------------
 Reporter:  cohosh                               |          Owner:  (none)
     Type:  task                                 |         Status:
                                                 |  needs_revision
 Priority:  High                                 |      Milestone:  Tor:
                                                 |  0.4.1.x-final
Component:  Core Tor/Tor                         |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:  CI, PTs 029-backport 034-backport    |  Actual Points:  1
  035-backport 040-backport, network-team-       |
  roadmap-2019-Q1Q2                              |
Parent ID:  #29267                               |         Points:  1
 Reviewer:  teor                                 |        Sponsor:
                                                 |  Sponsor19
-------------------------------------------------+-------------------------
Changes (by teor):

 * status:  needs_review => needs_revision
 * points:   => 1


Comment:

 From a commit message:
 > Previously we had "make check" launched whenever DISTCHECK was
 > false.  Now we'd like to turn it off in a few other circumstances,
 > like running chutney.  Maybe stem too?

 Yes, let's turn off "make check" for stem.

 The default network in chutney is bridges+hs-v2, but I discovered in
 #29729 that it is really unstable in chutney's CI. (Sometimes all the jobs
 work, and sometimes multiple jobs fail. I think it might be due to network
 or CPU load issues on machines.)

 So let's change to a subset of the CI tests that work in chutney?

 So for 0.2.9, let's use:
 * bridges-min
 * hs-v2-min
 And for 0.3.4 and later, let's add:
 * hs-v3-min
 Each test should only take 30-60 seconds, so I think it's worth running
 these 3 tests. (That's about 2 extra minutes per branch, or an 18 extra
 minutes for a full merge-forward from 0.2.9.)

 Here's the full list of all the tests in chutney from #29729:
 https://github.com/torproject/chutney/pull/25/files#diff-
 354f30a63fb0907d4ad57269548329e3R29

 We're missing:
 * basic-min
 * single-onion-v23
 * and all the IPv6 tests
   * We can only run IPv6 tests in macOS Travis, because Linux Travis
 doesn't support IPv6.
 I opened #30212 to add them later.

 Then let's let CI run for a while to evaluate the chutney tests.

 If we think chutney is too unstable in Tor's Chutney job, or Chutney's CI,
 we could:
 * run chutney in offline mode (#30183), or
 * add a sleep after failure in test-network --forgiving (#30210).

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29280#comment:16>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online