[ooni-dev] M-Lab docs

[Stephen Soltesz]

Also requested was a description of typical package deployment and
experiment management on M-Lab.

Typical package management follows these basic steps:

1) build rpm package from source repos in http://github.com/m-lab-tools/
2) copy and sign rpm package to M-Lab yum repository

Now per-machine, the slice (i.e. experiment VM) is instantiated:

3) upon slice vm creation on an M-lab server, the first-phase of
initialization includes bootstrapping the yum configuration of the vm's
filesystem (i.e. /etc/yum.conf and /etc/yum.slice.d/slice.repo list).
These import the public signing key, and point to CentOS mirrors and the
M-Lab slice package repository.
4) the second phase of slice initialization then tries to install the
slice package:
   i.e. yum install mlab_ooni
5) on success, the service starts.  on failure, stop.
6) M-Lab uses external monitoring to identify failed services and inform
a directory service like mlab-ns of available servers.

Since the rpm packages are effectively public, they do not include any
private information.  Some experiments have conditional deployment. i.e.
if site==nuq01, then run an additional service.  As well, the initialize
scripts for a package could generate per-machine key pairs.

The scenario I remember us talking about in Berlin involved M-Lab
harvesting the public part of the generated slice keys, or the onion
urls from the ooni slice, and publishing them in some way, through
mlab-ns, or other service like the ooni bouncer.

Does that sound workable for oonib?

Best,
Stephen

On 08/30/2013 04:50 PM, Meredith Whittaker wrote:
> Hi all, 
> 
> As an action from today's call, here's a collection of most pertinent
> M-lab docs. If you're looking for something that doesn't seem to be
> here, or if you have questions, don't hesitate to ask Stephen and me. 
> 
> Cheers,
> Meredith 
> 
> *Documents: *
> 
>   * Platform architecture doc
>     <http://measurementlab.net/sites/default/files/HowtoContributetoM-LabServerInfrastructure.pdf>.
>     The intended audience is hosting partners (who donate
>     space/power/connectivity/hardware). 
>   * Policies and procedures for approval
>     <http://measurementlab.net/sites/default/files/SubmissionguidelinesforM-Labexperiments.pdf> of
>     new experiments. Note that I wrote this, and that it may not be
>     specific enough, but I can answer any questions. Note that OONI has
>     been approved as meeting these criteria per the Steering Committee
>     process. 
>   * Server map <http://measurementlab.net/mlab_sites>. Get a look at the
>     global M-Lab footprint. 
>   * List of existing M-Lab
>     <http://measurementlab.net/measurement-lab-tools> measurement tools.
>     Code repos and documentation for individual tools should be linked
>     from here. If they're not, let me know and I'll add this. 
>   * Access existing M-Lab data <http://measurementlab.net/data>,
>     tarballs and BigQuery. 
>   * Blog post describing the process to visualize data using BigQuery
>     <http://dmadev.com/2012/11/19/visualizing-m-lab-data-with-bigquery/>, a
>     good introduction both to the tool and the data structure. If you'd
>     like whitelisted access to M-Lab data on BigQuery, let me know and
>     I'll make it happen. 
>   * Founding document
>     <http://measurementlab.net/sites/default/files/mlab_intro_and_server_requirements.pdf>,
>     which will give you background on M-Lab's founding motivations. Many
>     of the facts quoted here have changed (e.g. number of servers), but
>     as history this is instructive. 
>   * Existing visualizations <http://measurementlab.net/visualization>,
>     which are fun and may be instructive. 
> 
> 
> -- 
> 
> Meredith Whittaker
> Program Manager, Google Research
> Google NYC
> 
> 
>