[tor-dev] Stem Proc Integration Tests

[Damian Johnson]

> Erik and I have finished the proc.py integration tests

Great! I'm looking forward to checking them out.

> Tomorrow, we will be starting the next Stem task regarding the Tor Export project. Do you have any recommendations on where we should start exploring in the Stem documentation?

Hmm, this concerns translating Descriptor subclasses [1] into a csv
(and I suppose loading them back up). First step would be to google
around to see if there's a builtin function what'll do this or help
with it. If not then this might simply include dumping out their
__dict__...

>>> class Foo:
...   def __init__(self):
...     self.foo = 5
...     self.bar = "hello"
...   def set_bar(self, value):
...     self.bar = value
...
>>> f = Foo()
>>> f.__dict__
{'foo': 5, 'bar': 'hello'}
>>> ",".join([str(v) for v in f.__dict__.values()])
'5,hello'

You probably want to sort the keys and use that order, but besides
that this should do the trick. Just add a unit test and you're done.
:)

If we want to convert csvs *back* into Descriptors then that's a bit more work.

> Also, what do you anticipate being necessary when it comes to parsing consensus entries?

A NetworkStatusEntry class (and its tests) would be similar to the
ServerDescriptor [2] and ExtraInfoDescriptor [3], but a fair bit
easier (it only has six entries). To implement this you'd...

1. Read over the ServerDescriptor and ExtraInfoDescriptor to figure
out how we're tackling the parsing and verification in python.
2. Read over Karsten's NetworkStatusEntryImpl class [4] to see how
he's handling this in metrics-lib.
3. Read the spec for these entries [5]. Do we need both a
NetworkStatusEntryV2 and NetworkStatusEntryV3 class? Exactly how
should we model this? Karsten: thoughts?
4. Go to 'https://metrics.torproject.org/data.html' to get some test
data, so you see what the things that you'll be parsing looks like.
5. Write a NetworkStatusEntry class with the goal of "figure out each
and every way that data might be malformed". A large part of this
class is to verify that the data we're being given is perfectly valid
according to the spec, so read the spec and implement exactly what it
says. If there's some ambiguity in the spec or you see data that
doesn't conform to the spec then let us know. That's a tor bug.
6. Write unit and integ tests similar to what the other descriptors have...
  - unit tests are the majority of the work, and exercise all the use
cases that you can think of against mock objects
  - integ tests are pretty short, and just run the parser against some
test data from the metrics archive and the cached consensus

Let me know if you have any questions! -Damian

PS. Karsten: Do we want to call this "NetworkStatusEntry" or something
else? It seems like "ConsensusEntry" would be more intuitive, but
maybe this would just spawn confusion.

[1] https://gitweb.torproject.org/stem.git/blob/HEAD:/stem/descriptor/__init__.py#l87
[2] https://gitweb.torproject.org/stem.git/blob/HEAD:/stem/descriptor/server_descriptor.py#l128
[3] https://gitweb.torproject.org/stem.git/blob/HEAD:/stem/descriptor/extrainfo_descriptor.py#l174
[4] https://gitweb.torproject.org/metrics-lib.git/blob/HEAD:/src/org/torproject/descriptor/impl/NetworkStatusEntryImpl.java
[5] https://gitweb.torproject.org/torspec.git/blob/HEAD:/dir-spec.txt#l1334