[or-cvs] r18692: {torflow} Add perf measurements readme and clean up the buildtimes scr (in torflow/trunk: . CircuitAnalysis/BuildTimes NetworkScanners)
mikeperry at seul.org
mikeperry at seul.org
Wed Feb 25 18:58:14 UTC 2009
Author: mikeperry
Date: 2009-02-25 13:58:13 -0500 (Wed, 25 Feb 2009)
New Revision: 18692
Added:
torflow/trunk/README.PerfMeasurements
Modified:
torflow/trunk/CircuitAnalysis/BuildTimes/buildtimes.py
torflow/trunk/CircuitAnalysis/BuildTimes/dist_check.py
torflow/trunk/CircuitAnalysis/BuildTimes/shufflebt.py
torflow/trunk/NetworkScanners/speedracer.py
torflow/trunk/README
Log:
Add perf measurements readme and clean up the buildtimes
scripts a little. Also touch up main README to reference new
documentation.
Modified: torflow/trunk/CircuitAnalysis/BuildTimes/buildtimes.py
===================================================================
--- torflow/trunk/CircuitAnalysis/BuildTimes/buildtimes.py 2009-02-25 10:31:17 UTC (rev 18691)
+++ torflow/trunk/CircuitAnalysis/BuildTimes/buildtimes.py 2009-02-25 18:58:13 UTC (rev 18692)
@@ -205,7 +205,7 @@
return guard_slices,ncircuits,max_circuits,begin,end,pslice,dirname
def usage():
- print 'usage: statscontroller.py [-b <#begin percentile>] [-e <end percentile] [-s <percentile slice size>] [-g] -n <# circuits> -d <output dir name>'
+ print 'usage: buildtimes.py [-b <#begin percentile>] [-e <end percentile] [-s <percentile slice size>] [-g] -n <# circuits> -d <output dir name> [-c <max concurrent circuits>]'
sys.exit(1)
def guardslice(guard_slices,p,s,end,ncircuits,max_circuits,dirname):
Modified: torflow/trunk/CircuitAnalysis/BuildTimes/dist_check.py
===================================================================
--- torflow/trunk/CircuitAnalysis/BuildTimes/dist_check.py 2009-02-25 10:31:17 UTC (rev 18691)
+++ torflow/trunk/CircuitAnalysis/BuildTimes/dist_check.py 2009-02-25 18:58:13 UTC (rev 18692)
@@ -180,10 +180,10 @@
elif disk_only:
continue
- if router.rank_history and not ranks or ranks and not router.rank_history:
+ if ranks and not router.rank_history:
print "WARN: Rank storage mismatch for "+router.idhex
continue
- if router.bw_history and not bws or bws and not router.bw_history:
+ if bws and not router.bw_history:
print "WARN: Bw storage mismatch for "+router.idhex
continue
Modified: torflow/trunk/CircuitAnalysis/BuildTimes/shufflebt.py
===================================================================
--- torflow/trunk/CircuitAnalysis/BuildTimes/shufflebt.py 2009-02-25 10:31:17 UTC (rev 18691)
+++ torflow/trunk/CircuitAnalysis/BuildTimes/shufflebt.py 2009-02-25 18:58:13 UTC (rev 18692)
@@ -249,8 +249,8 @@
def shuffle(sort,truncate,filename,newfile):
if not sort and truncate is None: return
- sortlocation = '/usr/local/bin/sort' #peculiarity of fallon's system
- #sortlocation = 'sort'
+ #sortlocation = '/usr/local/bin/sort' #peculiarity of fallon's system
+ sortlocation = 'sort'
if sort and truncate:
cmd = sortlocation + ' -R ' + filename + ' | head -n ' + truncate + ' > ' + newfile
elif sort and not truncate:
Modified: torflow/trunk/NetworkScanners/speedracer.py
===================================================================
--- torflow/trunk/NetworkScanners/speedracer.py 2009-02-25 10:31:17 UTC (rev 18691)
+++ torflow/trunk/NetworkScanners/speedracer.py 2009-02-25 18:58:13 UTC (rev 18692)
@@ -24,14 +24,17 @@
sys.path.append("./libs")
from SocksiPy import socks
+user_agent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
+
# Some constants for measurements
-user_agent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
url = "http://svn.torproject.org/svn/tor/trunk/doc/design-paper/tor-design.pdf"
-count = 25
-save_every = 5
start_pct = 0
stop_pct = 78
+# Slice size:
pct_step = 3
+# Number of fetches per slice:
+count = 25
+save_every = 5
class MetatrollerException(Exception):
"Metatroller does not accept this command."
Modified: torflow/trunk/README
===================================================================
--- torflow/trunk/README 2009-02-25 10:31:17 UTC (rev 18691)
+++ torflow/trunk/README 2009-02-25 18:58:13 UTC (rev 18692)
@@ -53,9 +53,10 @@
SoaT provides tests for ensuring the honesty of exit nodes. It can check for
invalid exit policies and perform tests to see whether nodes are changing
-the requested content for several protocols. See pydoc for details:
+the requested content for several protocols. See the soat README for
+details:
-# cd NetworkScanners ; pydoc soat
+# cd NetworkScanners ; cat README.ExitScanning
Important note: soat requires the pyopenssl python module to run.
You can check if pyopenssl package is available for your distribution
@@ -70,38 +71,43 @@
of directory-reported node bandwidth and compiles statistics for each of
these tiers itself and via metatroller.
+A README for gathering performance metrics with speedracer can be
+found in README.PerfMeasurements.
-4. Node Monitors
+4. Circuit Analysis
-4a. NodeMonitors/nodemon.py
+4a. Build Time Anlysis
-Nodemon uses TorCtl to monitor ORCONN events on a Tor node to measure
-how much bandwidth is going to each peer node. It also measures how many
-circuits are killed by OR connections being shut down prematurely.
-
-4b. NodeMonitors/moniTor.py
-
-This is a "top-like" interface for Tor information It's goal at the
-start is to just tell you basic information In the future, you may be
-able to control Tor with it.
-
-
-5. Circuit Analysis
-
-5a. Build Time Anlysis
-
The set of scripts under CircuitAnalysis/BuildTimes gathers and plots
circuit construction times and fits a pareto curve to the results.
+A README for gathering performance metrics with buildtimes can be
+found in README.PerfMeasurements.
-5b. OP Addon
+4b. OP Addon
+
In CircuitAnalysis/OPAddon is a utility designed to run along side a Tor
client to build a map of latencies between nodes on the network, and to
then use this map for future routing decisions. See
CircuitAnalysis/OPAddon/README for more info.
+5. Node Monitors
+6a. NodeMonitors/nodemon.py
+
+Nodemon uses TorCtl to monitor ORCONN events on a Tor node to measure
+how much bandwidth is going to each peer node. It also measures how many
+circuits are killed by OR connections being shut down prematurely.
+
+6b. NodeMonitors/moniTor.py
+
+This is a "top-like" interface for Tor information It's goal at the
+start is to just tell you basic information In the future, you may be
+able to control Tor with it.
+
+
+
Each of these components has a lot of room for improvement. Please see
the TODO file for more information.
Added: torflow/trunk/README.PerfMeasurements
===================================================================
--- torflow/trunk/README.PerfMeasurements (rev 0)
+++ torflow/trunk/README.PerfMeasurements 2009-02-25 18:58:13 UTC (rev 18692)
@@ -0,0 +1,256 @@
+ How to Run the TorFlow Performance Tools
+
+
+
+I. Introduction
+
+There are two main client-side performance measurement tools in TorFlow:
+speedracer and buildtimes. Speedracer is meant for gathering average
+stream capacity of nodes and buildtimes is meant for gathering
+statistics on circuit construction speeds and success rates.
+
+
+
+II. SpeedRacer
+
+
+Speedracer functions by dividing the Tor network into groups of nodes of
+similar advertised capacity and then fetching the same URL over and over
+again via 2-hop circuits consisting of nodes in that group.
+
+A. Configuring SpeedRacer
+
+At the time of this writing, it has the following configuration
+parameters at the top of its sourcefile in NetworkScanners/speedracer.py:
+
+url = "http://svn.torproject.org/svn/tor/trunk/doc/design-paper/tor-design.pdf"
+start_pct = 0
+stop_pct = 78
+pct_step = 3
+count = 25
+save_every = 5
+
+The URL you specify should be big enough to amortize out TCP slow-start.
+Shoot for somewhere between 200k-1M. The tor-design.pdf may actually be
+a little on the small side to properly measure capacities of faster
+nodes.
+
+start_pct and stop_pct are the start and stop points for the run, in
+terms of the rankings of nodes by their bandwidth. Lower percentiles
+are faster.
+
+pct_step is the size of the slices in percentile units.
+
+count is the number of URL fetches to do for each slice.
+
+save_every is used for saving incremental results to check for
+convergence. Results will be saved after each multiple of 'save_every'
+fetches. The incremental results are cumulative.
+
+B. Running SpeedRacer
+
+Like soat, speedracer should be given its own Tor that is not performing
+any other stream activity. It will also require Tor 0.2.1.13 (r18556) or
+later.
+
+First, start up tor:
+
+# ~/src/tor-trunk/src/or/tor -f ~/src/torflow-trunk/torrc >& tor.log &
+
+Then, start up the Metatroller:
+
+# ~/src/torflow-trunk/metatroller.py >& mt.log &
+
+Finally, start up speedracer:
+
+# cd ~/src/torflow-trunk/NetworkScanners
+# ./speedracer.py >& speed.log &
+
+
+C. Reading the Tea Leaves
+
+SpeedRacer outputs a lot of statistics in aggregate form in
+./NetworkScanners/data/speedraces/stats-<pct_start>-<pct_end>-<n>-<time>
+
+pct_start and pct_end denote the range of the slice. N denotes the
+number of fetches so far, and time is the timestamp of that run. The
+results are cumulative, so the n=10 file will contain the results from
+n=5 in addition to 5 more fetches.
+
+The statistics stored with each node are indicated in the key at the top
+of each stat file.
+
+For the purposes of speedracer, the interesting statistics are the EB
+stat and the BR stat. The EB stat is the average stream capacity we
+observe for a node, and the BR stat is the ratio of a node's advertised
+bandwidth to its average stream capacity.
+
+For ease of review, the nodes are sorted and printed in lists according
+to a few different metrics. For speedracer, the most useful list is the
+first one, but the others are useful for buildtimes, where these same
+stat files are also available. The data being displayed is the same, it
+is just reordered in each list. These lists are:
+
+1. Bandwidth Ratios
+
+This list is sorted by the ratio of advertised bandwidth to average
+stream capacity (the BR stat). Nodes at the top of this list advertise a
+disproportionately large amount of bandwidth in comparison to what they
+actually were seen to carry over streams used to fetch the URL (the EB
+stat).
+
+2. Failed Counts
+
+This list is less interesting for speedracer. In it, the nodes are
+sorted by the sum of stream and circuit failures (SF and CF,
+respectively). Stream failures are primarily attributed to exit nodes,
+where as circuit failures are attributed to the extender and the
+extendee at the time of failure.
+
+3. Suspected Counts
+
+This list is sorted by 'suspected' failure counts (SS and CS). Suspected
+failure counts are attributed to each node that was a member of the
+path at the time of failure.
+
+Some failures (such as timeouts) are only attributed as 'suspected' to
+all nodes in the path, and as such do not show up in the 'failed'
+counts for nodes.
+
+4. Fail Rates
+
+This list is sorted by the rate of failures per hour of node uptime.
+
+5. Suspect Rates
+
+This list is sorted by the rate of suspected failures per hour of
+node uptime.
+
+6. Failed Reasons
+
+This list groups nodes by their failure reason, and sorts the reasons by
+most prevalent, and sorts the nodes within these lists.
+
+7. Suspect Reasons
+
+This is the same as the failed reasons, except it is sorted by
+'suspected' counts.
+
+
+
+III. Buildtimes
+
+Buildtimes lives in
+torflow-trunk/CircuitAnalysis/BuildTimes/buildtimes.py. It functions by
+creating circuits over and over again through percentile slices of the
+network, similar to speedracer.
+
+A. Running Buildtimes
+
+Buildtimes can actually be run concurrently with one of either
+speedracer or soat using the same Tor process. It can also be run on a
+Tor process that is being used for normal client activity.
+
+Running it is a lot simpler too. It does not require the metatroller
+(but again, it is fine to run the metatroller concurrently). The
+full_run.sh script will run 3 different buildtimes invocations and
+output the results to the 'slices' subdirectory.
+
+Currently, these runs are:
+
+./buildtimes.py -n 10000 -s 3 -e 93 -c 15 -d ./slices
+./buildtimes.py -n 10000 -s 3 -g -e 50 -c 30 -d ./slices
+./buildtimes.py -n 100000 -s 93 -c 100 -d ./slices
+
+This will first run 10k circuits on each 3% slice from 0-93%, with at
+most 15 concurrent circuits at a time. The results from this run are
+split into their percentile ranges.
+
+The second run will only apply the percentile restrictions to the first
+hop, and ensure that this hop has the guard flag. The rest of the
+network will be selected for the 2nd and 3rd hop using Tor's
+bandwidth-weighted selection distribution. The results from this run
+will have a g appended to their percentile ranges.
+
+The final run will create 100k circuits over the entire Tor network,
+using Guard flagged nodes for the first hop, and a bandwidth-weighted
+selection mechanism for all three hops. The results from this run will
+be the only ones with 100000 in their filenames.
+
+In all three runs, the third node is chosen if it allows one of either
+80 or 443. This is done to approximate the effect of Tor's circuit
+prediction mechanism on the typical Tor user. Since Web traffic makes
+up the bulk of Tor traffic by connection, it is likely that the typical
+user's Tor client will prefer to pre-build circuits serving 80 or 443.
+
+
+B. Reading the Tea Leaves
+
+Buildtimes outputs a lot of data. Each of the three runs output a debug
+log via the output redirection in full_run.sh.
+
+Additionally, each percentile slice from each run has its own set of 9
+data files:
+
+1. .agg
+
+This is the aggregate stats file that has the same format as described
+above for speedracer. This time, circuit failure counts and reasons are
+the most interesting items here.
+
+2. .nodes
+
+This file provides a well-formed record of which nodes were used in which
+positions for each circuit ID.
+
+3. .buildtimes
+
+These are the total circuit creation times, indexed by circuit ID.
+
+4. .extendtimes
+
+These are the individual node extend times, indexed by circuit ID.
+
+5. .failed
+
+This file provides a list of failed circuit IDs and their nodes, but currently
+with no reason codes.
+
+6. .ranks
+
+This file records the history of advertised bandwidth and the ranks of
+nodes over the course of the run.
+
+7. .uptime
+
+This file outputs the uptime of each node over the course of the run.
+
+8. .check
+
+This file contains verification information for the selection mechanism. It
+provides min/avg/max percentile ranks, selection counts, uptime, and counts
+for flag presence to verify restrictions.
+
+9. .log
+
+This is the full control port log file.
+
+
+C. Graphing Results
+
+The shufflebt.py script provides histogram graphing for the results and
+doing basic checks on convergence of this histogram for limited sample
+sizes. It takes a .buildtimes file as input, and an optional number of
+circuits to truncate/shuffle at:
+
+usage: shufflebt.py [-n <number of circuits>] [-s] [-g] [-k <k value>] [-d
+outdirname] [-r <res in ms>] <list of filenames>
+
+So for example, to randomly select (shuffle) 1000 circuits and graph the
+result:
+
+# ./shufflebt.py -d ./slices -n 1000 -s -g ./slices/0-93.100000.buildtimes
+eog ./slices/0-93.100000.buildtimes.shuffled.res100.png
+
+
+
More information about the tor-commits
mailing list