[tor-dev] Tor consensus documents now in a public append-only verifiable log

[Linus Nordberg]

Hi,

One of the directory authoritites (maatuska) is submitting new consensus
documents to a public append-only verifiable log, similar to what
Certificate Transparency uses.

If you think this is exciting and want to help out with this experiment,
consider setting up a monitor and start looking for anomalies, or at
least an auditor verifying proper log behaviour. Or help making the
submission code in Tor more useful (f.ex. submitting to the log's .onion
address, submitting votes too, making the POST chunked).

Below are two examples of how to investigate what's in the log so far.


Fetching one document, using curl
---------------------------------

Here's an example of how to fetch the 80th entry in the log:

    $ curl -sA '' -x socks4a://localhost:9050/ 'http://mvkhztpvqcxpdbn3.onion/open/gaol/v1/get-entries?start=79&end=79'

The result is a large JSON document (~740kB):

--8<---------------cut here---------------start------------->8---
{
   "entries" : [
      {
         "leaf_input" : "AAAAAAFTA+pimwAACHcGH4sIAAAAAAACA+y96XLi2rYu+H8/hX7WDTKN+sa
[...]
rUcfehxefa/xjpqq45E6dZEPzPosd1cg8mSGi03G/r+LikK1rl7GQ/I7yOzvfvGm3jeWrD/g6++f2yN9oeI7dH+31/w3O7wS+HI8YAAAA",
         "extra_data" : "AAAA"
      }
   ]
}
--8<---------------cut here---------------end--------------->8---

"leaf_input" contains a gziped, base64 encoded Tor consensus
document. Note that this format _will_ change in the future!


Fetching all documents, using fetchallblobs.py from catlfish
------------------------------------------------------------

Given a local copy of branch gaol6 of catlfish in my private repo [1]
and a copy of the public key for the log [2], here's how to download all
the entries from the log and store them in ZIP files in directory
'gaol-store':

    $ torsocks catlfish/tools/fetchallblobs.py --store gaol-store --publickey logs gaol.ct.nordu.net.pem http://mvkhztpvqcxpdbn3.onion/

The base64 encoded entry can be extracted using this awk script:

    /-----END BLOB-----/{blob=0}
    //{if (blob) print}
    /-----BEGIN BLOB-----/{blob=1}

Feeding that to base64 -d and gzip -d should produce a Tor consensus
document. For example:

    $ cat 00000080 | awk '/-----END BLOB-----/{blob=0}//{if (blob) print}/-----BEGIN BLOB-----/{blob=1}'| base64 -d | gzip -d

[1] $ git clone -b gaol6 https://git.nordu.net/user/linus/catlfish.git
[2] https://www.ct.nordu.net/gaol.ct.nordu.net.pem https://www.ct.nordu.net/gaol.ct.nordu.net.pem.asc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20160224/0f3bf1c5/attachment-0001.sig>