[tor-project] TPA-RFC-58: Podman CI runner deployment, help needed

[Antoine Beaupré]

Summary: I deployed a new GitLab CI runner backed by Podman instead of
Docker, we hope it will improve the stability and our capacity at
building images, but I need help testing it.

# Background

We've been having [stability issues][] with the Docker runners for a
while now.  We also started looking again at container image builds,
which are currently failing without Kaniko.

[stability issues]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41295
[ships instructions on how to build containers inside Podman]: https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-build-container-images-from-a-dockerfile

# Proposal

## Testers needed

I need help testing the new runner. Right now it's marked as not
running "untagged jobs", so it's unlikely to pick your CI jobs and run
them. It would be great if people could test the new runner. 

See the [GitLab tag documentation][] for how to add tags to your
configuration. It's basically done by adding a `tags` field to the
`.gitlab-ci.yml` file.

Note that in TPA's [ci-test gitlab-ci.yaml file][], we use a
`TPA_TAG_VALUE` variable to be able to pass arbitrary tags down into
the jobs without having to constantly change the .yaml file, which
might be a useful addition to your workflow.

The tag to use is `podman`.

You can send any job you want to the `podman` runner, but we'd like to
test a broad variety of things before we put it in production, but
especially image buildings. Upstream even has a [set of instructions
to build packages inside podman][].

[ci-test gitlab-ci.yaml file]: https://gitlab.torproject.org/tpo/tpa/ci-test/-/blob/main/.gitlab-ci.yml
[GitLab tag documentation]: https://docs.gitlab.com/ee/ci/yaml/#tags
[set of instructions to build packages inside podman]: https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-build-container-images-from-a-dockerfile

## Long term plan

If this goes well, we'd like to converge towards using `podman` for
all workloads. It's better packaged in Debian, and better designed,
than Docker. It also allows us to run containers as non-root.

That, however, is not part of this proposal. We're already running
Podman for another service (MinIO) but we're not proposing to
*convert* all existing services to `podman`. If things work well
enough for a long enough period (say 30 days), we might turn off the
older Docker running instead.

# Alternatives considered

To fix the stability issues in Docker, it might be possible to upgrade
to the latest upstream package and abandon the packages from
Debian.org. We're hoping that will not be necessary thanks to Podman.

To build images, we could create a "privileged" runner. For now, we're
hoping Podman will make building container images easier. If we do
create a privileged runner, it needs to take into account the long
term [tiered runner approach](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41044).

# Deadline

The service is already available, and will be running untagged jobs in
two weeks unless an objection is raised.

# Status

This proposal is currently in the `proposed` state.

# References

Feedback can be provided in the [discussion issue][].

[discussion issue]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41296

-- 
Antoine Beaupré
torproject.org system adminicd stration
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20230816/6f22a515/attachment.sig>