[tor-project] Sponsor 38 (Shadow simulator) update

[Jim Newsome]

We've shared the latest project update at 
https://github.com/shadow/shadow/discussions/1485; copied below:

This is part of a series of periodic updates of development in Shadow. 
This work is sponsored by the NSF 
<https://github.com/shadow/shadow/blob/main/docs/nsf_sponsorship.md>. 
Previous update: 2021-04 
<https://github.com/shadow/shadow/discussions/1274>.


    Shadow 2.0.0-pre release

Shadow 2.0, which features the new process-based architecture 
<https://github.com/shadow/shadow/projects/1> is now in pre-release! 
We've merged <https://github.com/shadow/shadow/pull/1441> the |dev| 
branch into |main|, and tagged v2.0.0-pre.1 
<https://github.com/shadow/shadow/releases/tag/v2.0.0-pre.1>. For users, 
this means:

  * Users cloning the repository at head will get Shadow 2.0 (pre).
  * We've /mostly/ stabilized some major changes to command-line
    options, config files, and topology files; enough so that users
    preparing to set up new simulations, or perhaps even heavily modify
    existing simulations or scripts, are better off targeting 2.0 than
    the Shadow 1.x series.
  * Because this is a pre-release, we're still reserving the right to
    make breaking changes without further bumping the version number,
    but we expect such changes to be minimal.

Users migrating existing simulations and scripts should have a look at 
our migration guide 
<https://github.com/shadow/shadow/blob/main/docs/migrating_from_1x.md>.


    Progress since the last update

We're still experimenting a bit with how to use GitHub's various project 
management tools. We've started using "projects" to represent Shadow 
releases. For example, you can see the issues for the v2.0.0 release 
here <https://github.com/shadow/shadow/projects/3>.

Notable changes since the April update: (By milestone; #'s are GitHub PR 
or issue #'s):

  * Minimum viable prototype of process-based simulation architecture
    <https://github.com/shadow/shadow/milestone/16>: Closed, now that
    Shadow 2.x runs Tor simulations, and we've cross-validated
    simulation results with Shadow 1.x.
      o Added new internal APIs for accessing managed process memory,
        and used them to reduce the number of data copies. (#1264,
        #1280, #1290)
  * Fully support process-based simulation architecture
    <https://github.com/shadow/shadow/milestone/22>
      o This milestone is a bit of a catch-all, and includes a number of
        stability and performance improvements
        <https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-04-06..2021-07-28+sort%3Acreated-asc+milestone%3A%22Fully+support+process-based+simulation+architecture%22>
        that we wanted to finish before the full 2.0 release.
      o We're in the process of adding seccomp-based syscall
        interposition in
        combination with preload-based interposition. More on that in
        "What's
        next", below. So far we've already implemented and enabled it
        (#1452, #1471).
  * Convert existing modules to Rust
    <https://github.com/shadow/shadow/milestone/28>
      o We've ported several of Shadow's core internal objects into Rust
        (#1325, #1336, #1367, #1376, #1434).
      o We've reduced our reliance on globals to fit better in the Rust
        ownership model (#1386, #1393, #1396)
  * Updates to and new formats for Shadow configuration
    <https://github.com/shadow/shadow/milestone/27>
      o We've changed the configuration file format from XML to YAML
        (#1278).
      o We've changed the topology format from graphml (XML) to gml
        (YAML-like)
        (#1369).
      o Lots of updates to configuration options and defaults, making
        things clearer, giving more sensible defaults, and fixing
        parsing bugs. (#1263, #1266, #1270, #1304, #1305).
      o Additionally, we've /unified/ much of the configuration and
        command-line parsing to reduce code duplication and
        inconsistencies, and migrated it to Rust. (#1258, #1286)
      o Most numbers in our configuration and command-line options can
        now include explicit units. (#1300)
      o Made updates to tornettools to stay consistent with Shadow's new
        config option and network graph formats, and other 2.x changes
        (tornettools#7 <https://github.com/shadow/tornettools/pull/7>,
        tornettools#8 <https://github.com/shadow/tornettools/pull/8>,
        tornettools#17 <https://github.com/shadow/tornettools/pull/17>)
  * Code health and maintenance
    <https://github.com/shadow/shadow/milestone/24>:
      o Fixed some sources of nondeterminism in simulation results (#1460).
      o Improve and automate Tor simulation test (#1411, #1458).
      o Updated documentation for Shadow 2.0. (#1412, #1417, #1406, #1403)
      o Changed default install prefix from |~/.shadow| to |~/.local|.
        (#1404)
      o More
        <https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-04-06..2021-07-28+milestone%3A%22Code+health+and+maintenance%22+sort%3Acreated-asc+>


    What's next

We'll be continuing to gain experience running Shadow experiments with 
the 2.0.0 pre-release. We will be working to fix bugs, tie up some loose 
ends and smooth out some rough edges, and generally stabilize the code 
as we continue to prepare for a 2.0.0 stable release.

We're currently tracking outstanding work on the 2.0.0 release page 
<https://github.com/shadow/shadow/projects/3>.


      Seccomp

In the course of our work on the 2.x architecture we've largely been 
exploring two system call interposition mechanisms that allow us to 
co-opt running processes into a simulation - |LD_PRELOAD| (similar to 
what Shadow 1.x uses), and |ptrace|. While preload-based interposition 
has better performance, and plays better with debuggers, we've largely 
been focusing on the ptrace-based approach because it can reliably catch 
all syscalls without having to re-implement large parts of libc and/or 
rely on undocumented glibc implementation details. This has allowed us 
to work out all of the Shadow internals that needed to change to support 
interposition at the Linux system call interface.

Recently, though, we added seccomp-based interposition to the 
preload-based interposition mode. This reliably catches any syscalls 
that would otherwise be missed by our |LD_PRELOAD| wrappers. We also 
have a working prototype of this mode of operation with /thread 
support/, which was previously the biggest missing functionality in 
preload-mode. Assuming thread support gets merged as expected and no 
other problems crop up, this is likely to become the new default mode of 
operation; it's faster than ptrace, and it allows us to trace the 
processes running in Shadow using debugging tools such as |gdb|.

Happy simulating!

The Shadow team 
<https://github.com/shadow/shadow/blob/master/docs/nsf-sponsorship.md#people>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20210629/a85368a4/attachment.htm>