[tor-dev] Nearly everything in the Tor source code is moving in 0.3.5

[Nick Mathewson]

On Thu, Jul 5, 2018 at 7:47 PM, Nick Mathewson <nickm at torproject.org> wrote:
> Hello, friends!

Oh no!  I hit send too early.  More info follows below.

> For quite a while now, the program "tor" has been built from source
> code in just two directories: src/common and src/or.
>
> This has become more-or-less untenable, for a few reasons -- most
> notably of which is that it has led our code to become more
> spaghetti-ish than I can endorse with a clean conscience.
>
> So to fix that, we've gone and done a huge code movement in our git
> master branch, which will land in a release once Tor 0.3.5.1-alpha is
> out.
>
> Here's what we did:
>
>   * src/common has been turned into a set of static libraries.  These
> all live in the "src/lib/*" directories.  The dependencies between
> these libraries should have no cycles.  The libraries are:
>
>     arch -- Headers to handle architectural differences
>     cc -- headers to handle differences among compilers
>     compress -- wraps zlib, zstd, lzma
>     container -- high-level container types
>     crypt_ops -- Cryptographic operations. Planning to split this into
> a higher and lower level library
>     ctime -- Operations that need to run in constant-time. (Properly,
> data-invariant time)
>     defs -- miscelaneous definitions needed throughout Tor.
>     encoding -- transforming one data type into another, and various
> data types into strings.
>     err -- lowest-level error handling, in cases where we can't use
> the logs because something that the logging system needs has broken.
>     evloop -- Generic event-loop handling logic
>     fdio -- Low-level IO wrapper functions for file descriptors.
>     fs -- Operations on the filesystem
>     intmath -- low-level integer math and misc bit-twiddling hacks
>     lock -- low-level locking code
>     log -- Tor's logging module.  This library sits roughly halfway up
> the library dependency diagram, since everything it depends on has to
> be carefully crafted to *not* log.
>     malloc -- Low-level wrappers for the platform memory allocation functions.
>     math -- Higher-level mathematical functions, and floating-point math
>     memarea -- An arena allocator
>     meminfo -- Functions for querying the current process's memory
> status and resources
>     net -- Networking compatibility and convenience code
>     osinfo -- Querying information about the operating system
>     process -- Launching and querying the status of other processes
>     sandbox -- Backend for the linux seccomp2 sandbox
>     smartlist_core -- The lowest-level of the smartlist_t data type.
> Separated from the rest of the containers library because the logging
> subsystem depends on it.
>     string -- Compatibility and convenience functions for manipulating
> C strings.
>     term -- Terminal-related functions (currently limited to a getpass
> function).
>     testsupport -- Macros for mocking, unit tests, etc.
>     thread -- Higher-level thread compatibility code
>     time -- Higher-level time management code, including format
> conversions and monotonic time
>     tls -- Our wrapper around our TLS library
>     trace -- Formerly src/trace -- a generic event tracing API
>     wallclock -- Low-level time code, used by the log module.
>
>   * To ensure that the dependency graph in src/common remains under
> control, there is a tool that you can run called "make
> check-includes".  It verifies that each module in Tor only includes
> the headers that it is permitted to include, using a per-directory
> ".may_include" file.
>
>   * The src/or/or.h header has been split into numerous smaller
> headers.  Notably, many important structures are now declared in a
> header called foo_st.h, where "foo" is the name of the structure.
>
>   * The src/or directory, which had most of Tor's code, had been split
> up into several directories.  This is still a work in progress:  This
> code has not itself been refactored, and its dependency graph is still
> a tangled web.  I hope we'll be working on that over the coming
> releases, but it will take a while to do.
>
>     The new top-level source directories are:
>
>      src/core -- Code necessary to actually perform or use onion routing.
>      src/feature -- Code used only by some onion routing
> configurations, or only for a special purpose.
>      src/app -- Top-level code to run, invoke, and configure the
> lower-level code
>
>    The new second-level source directories are:
>      src/core/crypto -- High-level cryptographic protocols used in Tor
>      src/core/mainloop -- Tor's event loop, connection-handling, and
> traffic-routing code.
>      src/core/or -- Parts related to handling onion routing itself
>      src/core/proto -- Support for encoding

       src/core/proto -- support for encoding and decoding different
wire protocols

   src/feature/api -- Support for making Tor embeddable
   src/feature/client -- Functionality which only Tor clients need
   src/feature/control -- Controller implementation
   src/feature/dirauth -- Directory authority
   src/feature/dircache -- Directory cache
   src/feature/dirclient -- Directory client
   src/feature/dircommon -- Shared code between the other directory modules
   src/feature/hibernate -- Hibernating when Tor is out of bandwidth
or shutting down
   src/feature/hs -- v3 onion service implementation
   src/feature/hs_common -- shared code between both onion service
implementations
   src/feature/nodelist -- storing and accessing the list of relays on
the network.
   src/feature/relay -- code that only relay servers and exit servers need.
   src/feature/rend -- v2 onion service implementation
   src/feature/stats -- statistics and history

   src/app/config -- configuration and state for Tor
   src/app/main -- Top-level functions to invoke the rest or Tor.

>
>   * The "tor" executable is now built in src/app/tor rather than src/or/tor.
>
>   * There are more static libraries than before that you need to build
> into your application if you want to embed Tor.  Rather than
> maintaining this list yourself, I recommend that you run "make
> show-libs" to have Tor emit a list of what you need to link.