[tor-project] Sponsor 38 update (Shadow simulator)

[Jim Newsome]

Sponsor 38:
https://trac.torproject.org/projects/tor/wiki/org/sponsors/Sponsor38

We've been primarily focused on prototyping the new process-based
simulation architecture https://github.com/shadow/shadow/milestone/16:

-   General libc + syscall interposition plumbing is fairly stable now,
    including marshaling pointer arguments across the Shadow/Plugin
    process boundary, in both the ptrace and the LD_PRELOAD
    interposition mechanisms.

-   Settled on a logging design and implemented it. In particular we
    needed to address the problem of logging from our "shim" code that's
    LD_PRELOADed into the plugin's process. Since it's no longer in
    Shadow's process, we can't directly use Shadow's logger, which
    references internal Shadow data structures and in-memory queues. To
    further complicate things, some code runs both inside Shadow and
    inside the LD_PRELOADed library, and that code should use Shadow's
    logger when and only when running inside Shadow. We developed a
    simple plugabble logger that writes to stdout by default, uses
    Shadow's logging when running inside Shadow, and a "shim logger"
    when running inside the LD_PRELOADed shim. This also allows us to
    use the same logging API inside tests and support libraries.

-   We've gotten more syscalls and parts of libc implemented, including
    syscalls from unistd.h and most of the socket.h syscalls. These are
    the basic interfaces used by almost every network program to handle
    communication and file descriptor i/o. We are nearly ready to start
    running some multi-process networking benchmarks.

We've also made some general improvements on the public Shadow branch
https://github.com/shadow/shadow/pulls?page=2&q=is%3Apr+is%3Aclosed+updated%3A%3E%3D2020-03-19:

-   Added both debug and release builds to our continuous integration
    test matrix, helping us catch more bugs before they get merged.

-   Got rid of the monolithic shadow.h header, substantially speeding up
    incremental builds, and improving compatibility with code analysis
    tools.

-   Documentation cleanup and additions, including adding contributor
    guidelines.

-   Reduced flakiness in multi-process networking tests by moving away
    from hard-coded server ports.

-   Added additional integration tests to cover functionality being
    implemented in the new process-based model.

-   Migrated from Python 2 to 3.

We also spent some time strategizing about how to use Github
"milestones" and "issues" to help us plan and prioritize development
work and to keep the community updated on our progress. We are still in
the early stages and hope to make more effective use of these planning
tools over time.

Finally, we're thrilled to welcome Steve Engler to the project. Steve is
starting today: Monday, June 1.