[tor-qa] Automated tests run on new releases and emails

[Nicolas Vigier]

Hello,

Starting from now, when a new build for a new release is available, the
test suite will be run automatically, and the results sent to this
mailing list.

They are also published on this page:
https://people.torproject.org/~boklm/tbbtests/

The detection of new builds works like this:

- the most recent signed tag on the master branch on
  builders/tor-browser-bundle.git is used to find the current version
  that needs testing

- the build directory of the tbb builders (currently this is 'mikeperry',
  'gk', 'linus', 'erinn', 'boklm'. Let me know if someone needs to be
  added to that list) on people.tpo is scanned to find if the version is
  available.

- the 5 first letters of the sha256sum of the sha256sums.txt file is
  used as a build identifier. The same build is not tested twice. When a
  new build is uploaded by someone, the sha256sums.txt and build
  identifier change, so the tests are run again.

The tests are run on all the platforms the test suite supports
(currently a few Linux distributions, soon Windows and later Mac OS X).

Because of these changes that I'm pushing now, the tests on current 3.6
version will be run again (although that version is already released)
and results sent here.

There are sometimes some false positives in the tests:

- in the 'check' test, this error:
  NoSuchElementException: Message: u'Unable to locate element:
  {"method":"css selector","selector":"h1.on"}'
  It might be because check sometimes think we are not using tor when we
  are.

- sometimes the https-everywhere or https-everywhere-disabled tests fail
  because of a timeout while loading the http page we expect https-e
  will redirect to https (currently http://www.mediawiki.org/ is used in
  the tests). 

I'm going to try to remove those false positive errors, but for now they
should be ignored.

The test results on new releases are sent to this mailing list, but the
tests results on the nightly builds are sent only to Georg, Mike and me
(starting today). If anyone else wants to receive them, let me know.

Nicolas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-qa/attachments/20140502/93796f78/attachment.sig>