[tor-project] shadow update 2023-09-27

[Jim Newsome]

Mirrored from <https://github.com/shadow/shadow/discussions/3187>:

This is part of a series of periodic updates of development in Shadow. 
This work is sponsored by the 
[NSF](https://shadow.github.io/docs/guide/nsf_sponsorship.html). 
Previous update: 
[2023-06](https://github.com/shadow/shadow/discussions/3061).
 

We've merged [74 non-dependabot pull 
requests](https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2023-06-30..2023-09-27+-author%3Aapp%2Fdependabot+) 
and closed [8 
issues](https://github.com/shadow/shadow/issues?q=closed%3A2023-06-30..2023-09-27+is%3Aissue+) 
since our previous update.

# Latest improvements

## Spawning processes

Shadow now supports running programs that spawn additional processes. 
Technically, this means it now supports `fork`, `vfork`, `execve`, and 
other related syscalls. Some uses for this:

* The primary way for `tor` to work with [pluggable 
transports](https://blog.torproject.org/tor-heart-bridges-and-pluggable-transports/) 
is for the `tor` process to dynamically spawn the pluggable transport 
process. This is currently the *only* way that pluggable transports are 
supported in `arti`. This now works in `shadow`.

* Orchestration of multiple processes on a single host in `shadow` can 
now be done using e.g. `sh` or `python` scripts, instead of having to 
specify every process directly in `shadow`'s configuration file.

* `shadow` can now run other software that spawns worker or helper 
processes.

Another exciting application for this in `shadow` *development* is that 
we can now more easily run third party test suites, which typically 
spawn multiple test processes. For example, running `tor`'s own 
self-tests helps validate that `shadow` is correctly emulating the 
platform functionality that `tor` uses; currently [more than 99% of the 
tests pass](https://github.com/shadow/shadow/issues/3168)!

## New TCP stack

Shadow's experimental new TCP stack is merged, and can be enabled with 
the experimental command-line flag `--use-new-tcp`. This new 
implementation is in Rust instead of C, and is developed to be very 
testable (e.g. in its own crate decoupled from the rest of Shadow, and 
with pluggable system dependencies). While some functionality is still 
being finished, we expect it will be much easier to maintain and to 
validate its correctness than the previous C implementation. Improved 
support for pcap file output also makes it easier to review Shadow's 
simulated network traffic.

## Shim stability

We've occasionally run into problems due to `shadow`'s preloaded shim 
calling into `libc`. This is unsafe because some initialization can run 
before `libc` itself is fully initialized, and much of the code runs in 
a seccomp signal handler, running afoul of 
[async-signal-safety](https://man7.org/linux/man-pages/man7/signal-safety.7.html).

Luckily, Rust has a rich ecosystem of code that doesn't depend on libc 
(`no_std` code). We've made substantial progress in migrating the shim's 
C code to `no_std` Rust code.

## Socket API improvements

Our UDP socket implementation has been migrated from C to Rust, and 
along the way we've made many improvements to the socket API for UDP, 
TCP, and Unix sockets. We've added support for the following:

* `sendmsg`, `recvmsg`, and `shutdown` syscall support for UDP sockets
* `MSG_TRUNC` support for UDP and Unix sockets
* `MSG_PEEK` support for UDP sockets
* `SO_DOMAIN`, `SO_PROTOCOL`, and `SO_ACCEPTCONN` socket options for TCP 
and UDP sockets
* `SIOCGSTAMP` ioctl support for TCP and UDP sockets

We've also made various fixes and improvements to existing socket 
functionality so that Shadow more accurately follows the behaviour of Linux.

## C to Rust migration

We continue to progress our migration from C to Rust. In addition to 
Rust progress in the shim and in the new TCP stack, other notable 
migrations since the last update include the shared memory allocator, 
epoll and timerfd descriptors and syscalls, UDP sockets, and several 
time-related syscalls. Our current status is 74% Rust code and 21.4% C 
code (much of this being C tests) according to Github statistics.

# Release status

We expect to release Shadow 3.1.0 some time in the next few weeks.

# Project status - NSF grant is wrapping up

The last three years of development on Shadow have been sponsored by an 
[NSF](https://shadow.github.io/docs/guide/nsf_sponsorship.html) grant. 
That grant ends at the end of this month (September 2023). This isn't 
the end of development on Shadow — Rob Jansen (@robgjansen) at the U.S. 
Naval Research Laboratory continues to head the project. Over the course 
of this project, The Tor Project has incorporated Shadow into its own 
research and testing workflows, and is likely to continue contributing 
as well. We look forward to continue improving and maintaining Shadow, 
but at a slower pace of development since we no longer expect 
contributions from programmers singularly dedicated to Shadow 
development. Jim Newsome (@sporksmith) is continuing to be a Tor Project 
employee but shifting focus to other Tor projects; Steven Engler 
(@stevenengler) is open to opportunities.

Happy simulating!
The Shadow team