Runner tests

Test workflows for the hosted Forgejo Actions runners of Digilab. Each workflow covers one use case of the pipelines that the active GitLab projects use today. Issue: #1354. Decision: ADR 0009.

Each job runs two times, one time on each executor. A matrix sets the label:

Label	Executor	Issue
default	Kubernetes executor. Each job is a pod in tn-digilab-ops-forgejo-jobs.	#1348
vm	KubeVirt executor. Each job gets a new VM with its own Docker daemon.	#1351

CONTEXT.md explains the terms, for example the VM image and the job image.

A red job on one executor is not always a bug. Results tells which failures are expected, and why.

Use cases

Workflow	Use case	Starts on
00-security-context.yml	The job runs as a user that is not root, with no capabilities, no service account token and no route to the Kubernetes API	push
01-go-rust.yml	Go build, vet, test, lint. Rust build and test	push
02-node.yml	Node.js build and test, with actions/checkout	push
03-python.yml	Python test with pytest	push
04-java.yml	Java build and test with Maven	push
05a-image-kaniko.yml	Image build and push with the archived upstream kaniko	push
05b-image-kaniko-fork.yml	Image build and push with our kaniko fork	push
05c-image-ko.yml	Go image build and push with ko, one architecture	push
05d-image-buildkit.yml	Image build and push with rootless BuildKit, from a Dockerfile	push
06-services.yml	A test with a PostgreSQL service container	push
07-helm-kustomize-tofu.yml	Helm lint and template, Kustomize build, OpenTofu plan	push
08-release-write-back.yml	Push a commit back, create a release, push a Helm chart	push to main
09-security.yml	Trivy scan for vulnerabilities, secrets and misconfigurations	push
10a-multiarch-ko.yml	amd64 and arm64 in one OCI image index, with ko	push
10b-multiarch-buildah.yml	amd64 and arm64 in one OCI image index, with Buildah and QEMU	push
11-registry.yml	Push a scratch image with the bot token, then pull it without credentials	push
11b-docker-build-push.yml	docker build with a RUN step, docker push, then pull and run the image, with the Docker daemon of the VM	push
12-registry-throughput.yml	Measure the upload and download speed of a job to the internet and the Forgejo registry	push to throughput/ or the workflow, and by hand

You can start each workflow by hand too. Names that contain the run number, for example image tags, branches and chart versions, end with the label. Therefore the two jobs do not overwrite each other.

Results

Run of 2026-10-29 on main, 46 jobs.

Workflow	Job	default	vm
00	no-root	✅	❌ expected: the job is root in the VM. The test checks the hardening of default only
01	go, go-lint, rust	✅	✅
02	node	✅	✅
03	python	✅	✅
04	maven	✅	✅
05a	kaniko	❌ expected: kaniko needs root	✅
05b	kaniko-fork	❌ no image yet	❌ no image yet
05c	ko	✅	✅
05d	buildkit	❌ expected: no user namespaces	✅
06	postgres	✅	✅
07	helm-kustomize, tofu	✅	✅
08	write-back, release, helm-push	✅	✅
09	trivy	✅	✅
10a	ko-multiarch	✅	✅
10b	buildah-multiarch	❌ expected: no user namespaces	✅
11	push, pull	✅	✅
11b	docker	❌ expected: no Docker daemon	✅
12	throughput	✅	✅

05b uses registry.gitlab.com/digilab.overheid.nl/ecosystem/forgejo/kaniko/executor:debug. That image does not exist yet: the fork needs BuildKit for its own build, so it needs the label vm first.

Build tools

Each row tells where a tool works, and why.

Tool	default	vm	Why
Go, Rust, Node.js, Python, Maven	✅	✅	Normal processes. They need no root. On default, use an image that has the tools, because the job cannot install packages
Helm, Kustomize, OpenTofu	✅	✅	Normal processes
crane, skopeo, oras	✅	✅	They talk to the registry API only. They need no daemon and no root
ko	✅	✅	ko builds Go images without a daemon and without root. It cross-compiles, so it makes multi-arch images without QEMU
Trivy	✅	✅	A normal process. It downloads its database of about 130 MB at each run
Service containers (services:)	✅	✅	On default, each service runs as uid 1000. PostgreSQL then needs PGDATA: /tmp/pgdata
actions/checkout and other JavaScript actions	✅	✅	On default, the job image must contain node. On vm, the runner runs the action in the job container
kaniko	❌	✅	kaniko unpacks the base image over / and changes owners. That needs root. default runs no job as root (ADR 0018)
BuildKit, rootless	❌	✅	Rootless BuildKit needs user namespaces. default does not permit them (max_user_namespaces is 0, Pod Security restricted)
Buildah	❌	✅	Buildah needs user namespaces or a privileged container. default ignores options: --privileged
docker build, docker buildx, Docker-in-Docker	❌	✅	They need a Docker daemon. default has none. On vm, the job uses the Docker daemon of its own VM
QEMU emulation (binfmt), for example a multi-arch RUN step	❌	✅	Registering binfmt needs a privileged container. The VM has qemu-user-static
apt-get install, apk add	❌	✅	They need root. On vm, the job is root in its container

Use default when a job can run on it. default starts faster and has stricter isolation. Use vm only for the tools that need root, a Docker daemon or user namespaces.

Rules for the label vm

The KubeVirt executor gives each job a new VM. The VM is deleted after the job. Therefore:

• The job is root in its container and can use the Docker daemon of the VM (docker_host: automount, privileged: true).
• The VM can reach the internet on port 443 only. DNS goes through the Cilium DNS proxy. Plain HTTP, SSH and NTP are blocked.
• The VM cannot reach other pods, the nodes or the Kubernetes API. It can reach Forgejo.
• A job waits for a free VM. See Timings.

Timings

Runs of 2026-10-29 and 2026-10-30: all jobs started at the same time on each executor.

Setup	default wait, max	vm wait, max
4 jobs on default; 3 VM slots, cloud-init, poll 30 s	104 s	609 s
9 VM slots, quota room for the old VMs	100 s	349 s
+ own VM image (ADR 0024)	100 s	243 s
+ poll every 5 s, 4 warm VMs	103 s (median 50 s)	120 s (median 50 s)
+ 12 jobs on default, 24 jobs with test 11b	26 s (median 4 s)	147 s (median 63 s)

The run time of one job is 2–83 s on both executors.

A job on vm waits longer in a burst, because each job gets a new VM. A VM from our own VM image (ADR 0024) is ready in 16–19 s; with cloud-init it took about 55–60 s.

The pool has 9 slots (maxConcurrent), 3 on each node, and keeps 4 warm VMs ready (minIdle). The quota has room for each VM and its successor, so a new VM does not wait for the old one to shut down. The operator asks Forgejo for queued jobs every 5 s, so a single job on a warm VM starts within about 5 s. In a burst, the jobs after the first 4 wait for a new VM or a free slot. The Kubernetes executor runs 12 jobs at the same time.

The steps are in #1406. Ideas for later are in #1409.

Image pulls

Run of 2026-10-29, own VM image, 4 warm VMs, poll every 5 s:

	default	vm
Pull of one job image	about 0.6 s	1.5–34 s, median about 15 s
All pulls of the 23 jobs	about 14 s	338 s
Run time of the 19 jobs that pass on both, with the pull	378 s	474 s
The same, without the pull	378 s	174 s

The nodes of default keep the images, so containerd only checks the manifest. Each VM of vm starts with an empty image store, so each job pulls its image again. Without the pull, a job on vm runs faster than on default, because the Kubernetes executor starts each step through the Kubernetes API. A pull-through cache in the cluster removes most of the pull time (#1390).

To measure again, query the table action_run_job in the Forgejo database: started - created is the wait, and stopped - started is the run time.

Rules for the label default

The Kubernetes executor runs no job as root (ADR 0018). Each container of a job pod, also each service container, runs as uid 1000 with no capabilities. The namespace enforces Pod Security restricted. Therefore:

• A job cannot install packages with apk add or apt-get install. Use an image that has the tools, or put a binary in /tmp/bin and add that directory to $GITHUB_PATH.
• HOME is /tmp. Most images have no user with uid 1000.
• A PostgreSQL service needs PGDATA: /tmp/pgdata. The default PGDATA belongs to uid 999.
• A Dockerfile build needs the label vm. See Build tools.

Secrets

The automatic token of a job cannot push packages. The workflows that push an image or a chart need two secrets of the organisation:

Secret	Value
REGISTRY_USER	The bot user of the organisation
REGISTRY_TOKEN	A token of that bot user with the scope write:package

Registry throughput

12-registry-throughput.yml is a measurement, not a functional test. It answers one question: how fast can a job on the runner send data to a registry and get it back?

The job measures two targets from the same job pod:

Target	What it measures
internet	An upload to speed.cloudflare.com and a download of a 100 MB file from nbg1-speed.hetzner.com. This is the bandwidth of the cluster to the internet, without a registry.
forgejo	A push to and a pull from the registry of this Forgejo instance, on its public hostname.

For the registry target, the job does these steps:

1. It makes one image layer of random data, 200 MB by default (SIZE_MB).
2. It pushes the layer as a new image with crane.
3. It pulls the same layer back.
4. It deletes the image.

Random data does not compress, and each run makes new data. Therefore the registry cannot skip the upload. The job compresses the layer before it starts the clock, so the result contains only the network transfer and the registry.

The last step shows a table in MB/s. Read it like this:

• If the upload is low to both targets, the limit is the bandwidth of the cluster.
• If the upload is low to the registry only, the limit is the registry or the route to it.

Each run pushes and deletes 200 MB in the registry. Start the workflow by hand when you need a new measurement.

Push by hand

You can also push an image from your own computer. Use the scratch image in registry/.

1. In Forgejo, go to Settings > Applications and create a token with only the scope write:package.

2. Log in to the registry with your username and the token:

   docker login code.digitaledienst.overheid.nl -u <username>
   

3. Build and push the image:

   docker build --platform linux/amd64 -t code.digitaledienst.overheid.nl/<organisation>/hello:latest registry
   docker push code.digitaledienst.overheid.nl/<organisation>/hello:latest
   

   You need write access to the packages of that organisation.

4. Pull the image without an account. This works for a public organisation:

   docker logout code.digitaledienst.overheid.nl
   docker pull code.digitaledienst.overheid.nl/<organisation>/hello:latest
   

Before you push your own images:

• The organisation decides who can see an image, not the repository. An image under a public organisation is public, also when it comes from a private repository.
• Everything in a public image is public. Do not put secrets in an image.

The full rules are in docs/container-registry.md.

Notes

• Most jobs check out the repository with git clone and the automatic token. Only 02-node.yml uses actions/checkout, because a JavaScript action needs node in the job image.
• The Kubernetes plugin resolves a relative working-directory from /, not from the workspace. The workflows therefore use ${{ forgejo.workspace }}/<directory>.
• kaniko replaces the file system of the job container. The kaniko jobs end with mkdir -p /workspace, so the runner can finish the job.
• Chainguard publishes the images of its kaniko fork only for customers. Therefore 05b uses our own build of the fork (#1347). That build does not publish an image yet.
• kaniko builds one architecture for each job. A multi-arch image in one job needs ko (10a) or Buildah with QEMU in a VM (10b).
• This repository is mirrored to GitLab. Make changes on Forgejo.