[ooni-talk] Update re: OONI PostgreSQL MetaDB

[Maria Xynou]

Hi friends,

In light of the (unplanned) shutdown of some servers, we have had to work
towards migrating our infrastructure (hosting the OONI data processing
pipeline) and updating the OONI PostgreSQL metaDB to adapt it to the
requirements of our new data processing pipeline.

As a result, we will have to stop updating the OONI PostgreSQL metaDB (
https://github.com/ooni/sysadmin/blob/master/docs/metadb-sharing.md) WAL
snapshots currently available on S3.

If you are an OONI PostgreSQL metaDB user, we apologize for the short
notice, and we encourage you to read the email below for further
information.

# Migration impacting the OONI PostgreSQL metaDB

We are now replacing the batch OONI data processing pipeline with our new
fast-path pipeline, which analyzes and publishes OONI measurements from
around the world in near real-time (the measurements are published as soon
as the report is closed).

To this end, we have built a new PostgreSQL metaDB, which is powered by the
latest version of PostgreSQL (11) available in debian stable, and which
uses a different set of tables (based on our improved, fast-path pipeline).

While we are excited to migrate over to a new and improved pipeline
(particularly since this is expected to enhance the performance of OONI
Explorer!), this unfortunately also means that the current OONI PostgreSQL
MetaDB, which is powered by an older version ofPostgreSQL and which depends
on the batch OONI data processing pipeline, will be affected.

Specifically: If you are an OONI PostgreSQL MetaDB user, you will still
have access to all previous OONI data, but you will unfortunately not
receive any updates once the migration has completed (by 31st October 2020
latest).

Going forward, we recommend removing the line in your PostgreSQL
configuration that does sync (as we will not be updating the WAL snapshots).

We might eventually restart exporting database tables, but by using the
more modern logical replication feature of PostgreSQL 11.

If you are an OONI PostgreSQL MetaDB user and unsure how to proceed,
we encourage
you to reach out and share details about your use case (so that we can also
factor it into our future decisions).

We will be happy to help you. You can reach us via email (
team at openobservatory.org) or via the OONI Slack channel (
https://slack.ooni.org/).

In the meanwhile, please consider accessing OONI data via the ooni-data
Amazon S3 bucket: https://ooni-data.s3.amazonaws.com/

Please refer to this blog post to see how to fetch the raw data on S3:
https://ooni.org/post/mining-ooni-data/

We sincerely apologise for any inconvenience this may cause.

Thanks for using the OONI PostgreSQL metaDB, and thanks for your patience
throughout this process.

Please contact us with any questions you may have.

Thanks,

~ OONI team.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/ooni-talk/attachments/20201020/e27c7e94/attachment.htm>