[tor-dev] Proposal 315: Updating the list of fields required in directory documents

[Nick Mathewson]

Filename: 315-update-dir-required-fields.txt
Title: Updating the list of fields required in directory documents
Author: Nick Mathewson
Created: 23 April 2020
Status: Open

1. Introduction

   When we add a new field to a directory document, we must at first
   describe it as "optional", since older Tor implementations will
   not generate it.  When those implementations are obsolete and
   unsupported, however, we can safely describe those fields as
   "required", since they are always included in practice.

   Making fields required is not just a matter of bookkeeping: it
   helps prevent bugs in two ways.  First, it simplifies our code.
   Second, it makes our code's requirements match our assumptions
   about the network.

   Here I'll describe a general policy for making fields required
   when LTS versions become unsupported, and include a list of
   fields that should become required today.

   This document does not require to us to make all optional fields
   required -- only those which we intend that all Tor instances
   should always generate and expect.

   When we speak of making a field "required", we are talking about
   describing it as "required" in dir-spec.txt, so that any document
   missing that field is no longer considered well-formed.

2. When fields should become required

   We have three relevant kinds of directory documents: those
   generated by relays, those generated by authorities, and those
   generated by onion services.

   Relays generate extrainfo documents and routerdesc documents.
   For these, we can safely make a field required when it is always
   generated by all relay versions that the authorities allow to
   join the network.  To avoid partitioning, authorities should
   start requiring the field before any relays or clients do.

   (If a relay field indicates the presence of a now-required
   feature, then instead of making the field mandatory, we may
   change the semantics so that the field is assumed to be
   present. Later we can remove the option.)

   Authorities generate authority certificates, votes, consensus
   documents, and microdescriptors.  For these, we can safely make a
   field required once all authorities are generating it, and we are
   confident that we do not plan to downgrade those authorities.

   Onion services generate service descriptors.  Because of the risk
   of partitioning attacks, we should not make features in service
   descriptors required without a phased process, described in the
   following section.

2.1. Phased addition of onion service descriptor changes

   Phase one: we add client and service support for the new field,
   but have this support disabled by default. By default, services
   should not generate the new field, and clients should not parse
   it when it is present.  This behavior is controlled by a pair of
   network parameters.  (If the feature is at all complex, the
   network parameters should describe a _minimum version_ that
   should enable the feature, so that we can later enable it only in
   the versions where the feature is not buggy.)

   During this phase, we can manually override the defaults on
   particular clients and services to test the new field.

   Phase two: authorities use the network parameters to enable the
   client support and the service support.  They should only do this
   once enough clients and services have upgraded to a version that
   supports the feature.

   Phase three: once all versions that support the feature are
   obsolete and unsupported, the feature may be marked as required
   in the specifications, and the network parameters ignored.

   Phase four: once all versions that used the network parameters
   are obsolete and unsupported, authorities may stop including
   those parameters in their votes.

3. Directory fields that should become required.

   These fields in router descriptors should become required:
      * identity-ed25519
      * master-key-ed25519
      * onion-key-crosscert
      * ntor-onion-key
      * ntor-onion-key-crosscert
      * router-sig-ed25519
      * proto

   These fields in router descriptors should become "assumed present":
      * hidden-service-dir

   These fields in extra-info documents should become required:
      * identity-ed25519
      * router-sig-ed25519

   The following fields in microdescriptors should become
   required:
      * ntor-onion-key

   The following fields in votes and consensus documents should
   become required:
      * pr