[tor-dev] [GSoC] HTTPS Everywhere's Secure Ruleset Updater - Final Report

[Red]

Hello everyone!

Today is the official "pencils down" date for Google Summer of Code, so
I am going to outline the results of my work this Summer as a way of
concluding my project.

The first thing I had done at the start of my project was learn about
Mozilla's XPCOM components and how they are used to interface with Gecko
and other Firefox internals from extension code.  I also learned about
special files included with any extension such as install.rdf which are
used to describe an extension's version information and
signature-verifying public key, among other things. This lead me to
understanding how extensions are updated and discover the kinds of
limitations I would find in trying to update HTTPS Everywhere's rulesets
library without changing the rest of the extension.

The next thing I did was draft the specification for the format of the
JSON manifest used to describe ruleset updates and explain how the
process of updating the rulesets library would work.  This work has
since been built on and improved, and the existing specification
document can be found in my fork of the extension, and is linked below[0].
This document describes everything one needs to know about the ruleset
updater. It details:

* How an object signing certificate can be generated using certutil from
the NSS toolset.
* How to create an update.json manifest file using the
ruleset_update_manifest.py utility script I created.[1]
* How to sign the update.json file using pk1sign and an object signing
certificate.
* How update information is fetched and how download failures are handled.
* How the contents of update.json and the rulesets library database file
are verified to be authentic.
* What the significance of each field in update.json is.
* Approximately how the ruleset updating mechanism functions in pseudocode.

After the details of how updating would work were outlined in
updateJSONSpec.md, I began implementing the ruleset updater module code
for the extension.  I also started working on a way to write tests for
HTTPS Everywhere that I would initially use to test my code. I
discovered that Firefox extensions could import XPCOM components
exported by other extensions added to a user's browser, and so began
work on a separate extension[2] that could be installed and used solely
for testing HTTPS Everywhere.  This work was soon improved upon by my
mentor, Yan, who had the idea of creating a skeleton of an extension
inside of HTTPS Everywhere's own codebase[3].  This extension used the
more modern Addon SDK, which includes a unit testing suite, enabling
HTTPS Everywhere developers to start writing proper unit tests that
could be included in the HTTPS-Everywhere repository and run from the
command line.

While the unit testing environment I now had available was very
effective, I had run into some surprising difficulty with Mozilla's
nsIDataSignatureVerified XPCOM component, used to verify the signatures
we were generating.  By this point, Yan and I had agreed to try to
compute the signature over the SHA256 hash of update.json's contents, so
that this data could be more easily transcribed from a regular work
computer to the airgapped machine EFF uses to house an offline signing
key.  It was at this point that we found that the key and signatures
OpenSSL were generating were, somehow, incompatible with
nsIDataSignatureVerifier, and Yan managed to get the verification unit
test to pass by using certutil and pk1sign.  The same problem of
verifying signatures was encountered again when I got around to testing
the actual ruleset updater code again, and had to generate new data,
despite using certutil and pk1sign.  Quite unfortunately, this problem
drained almost four whole weeks of my time, as I had been looking into
solutions involving porting a Firefox extension signing tool called
Uhura (written in Perl), which meant also trying to understand what the
tool was doing with ASN.1 specifications, among other things.  Finally,
I found that the nsIDataSignatureVerifier tool successfully verified the
signature I had created for testing when it was not the hash of
update.json's contents that were signed and supplied to the verifier,
but the contents of the file themselves.  The component is poorly
documented and, after reaching out to developers in Mozilla's ecosystem,
I found that it isn't very well understood in general.  As such, even
now I do not really understand why the verifier did not behave as
expected in this scenario.  Regardless, while it is somewhat less
convenient here to sign update.json, the signature verification process
works.

Finally, I was able to move on to solve the problems of downloading and
verifying the authenticity of the rulesets database file, and of
reinitializing the extension's rulesets with those of the new database
file. These problems were technically nuanced, but have been solved. 
The ruleset updater has been integrated into my fork of the extension's
repository on github.  My tests of its performance have passed, and I am
pleased to say that it appears to be working.  I have since submitted a
pull request[4] to have my work reviewed and merged, and have even made
changes requested by Yan, who has reviewed my work already.

I'd like to conclude by giving my thanks to everyone in both of these
mailing lists.  I have received some invaluable support and advice that
has had a profoundly positive impact on my experience.  I learned a lot
from everyone that lent a hand, and I feel the project has benefited
greatly from each of the contributions I received to my efforts.  My
mentor, Yan, has also been a pleasure to work with.  Our weekly meetings
added very significant structure to my workflow, helped to keep me
organized, and were also a great opportunity for me to pique her brain
for ideas and possible solutions to some of the tougher problems I've
encountered.  I sincerely regard this as having been a positive
experience, and I hope that everyone who has helped me until now is
satisfied with my efforts and the actual results of my work.

Thanks, everyone!

[0] -
https://github.com/redwire/https-everywhere/blob/master/doc/updateJSONSpec.md
[1] -
https://github.com/redwire/https-everywhere/blob/master/utils/ruleset_update_manifest.py
[2] - https://github.com/redwire/https-everywhere-testing
[3] -
https://github.com/EFForg/https-everywhere/tree/master/https-everywhere-tests
[4] - https://github.com/EFForg/https-everywhere/pull/437

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 276 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20140818/278ec289/attachment.sig>