Runbook: Self-Hosted Actions Runner

Purpose: Run CI on fleet hardware instead of GitHub-hosted runners, so the job bills zero Actions minutes — and get back to hosted safely when the machine is unavailable When to use: Included Actions minutes are the binding constraint (#1648, #1690), or you want CI executing on real fleet hardware Prerequisites: The repo is private (non-negotiable — see Security constraints); admin on the repo; gh authenticated (gh auth status); the target machine reachable over Tailscale SSH and listed in machines.json; curl, tar, git, and jq on the target; sudo on the target for the service step Estimated time: 10 minutes, plus one CI run to confirm

Everything here goes through one script, scripts/fleet/runner.sh. Each mutation has an inverse, and none of it is hand-edited state:

Command	What it does
`runner.sh status`	Who is registered, is it online, what do the workflows currently target
`runner.sh install --machine <id>`	Register the machine and start its runner service
`runner.sh target <label>`	Point the workflows at a runner label
`runner.sh target hosted`	Point them back at GitHub-hosted `ubuntu-latest`
`runner.sh remove --machine <id>`	Reset the CI target, stop the service, deregister

Every mode acts on this repo unless you pass --repo <owner/name>, which points it at any other repo you have admin on (#1814) — see Another repo.

The CI switch itself is a single repo variable, CI_RUNNER_LABEL, read by every workflow — ci.yml, docs-deploy.yml, and auto-release.yml, plus the two scaffolded templates they ship from (#1782):

yaml

runs-on: ${{ vars.CI_RUNNER_LABEL || 'ubuntu-latest' }}

An unset variable evaluates to the empty string, so unset is the hosted default — there is no separate on/off flag that can disagree with it.

One workflow honouring the switch is not enough, and the failure mode is quiet: a hosted job on a repo with no available Actions minutes fails in about three seconds with no steps and no runner assigned, which reads as an ordinary red X. docs-deploy and auto-release sat like that for weeks — the docs site published nothing and no release was cut — while ci.yml, already on the variable, stayed green and made the repo look healthy. tests/test-fleet-runner.sh now fails if any workflow pins a job to ubuntu-latest.

What a runner must carry. Node comes from actions/setup-node (no system Node needed), but the workflows assume the host provides jq, gh, git, rsync, and tar. ci.yml and docs-deploy.yml assert jq up front so a missing binary fails with a sentence rather than a parse error deep inside a build.

Steps

1. Check the current state

bash

scripts/fleet/runner.sh status

Verify: the repo reports private: true, and the CI targets line names either ubuntu-latest (variable unset) or a label with an online runner behind it. Exit code 0 means consistent; exit 3 means CI is pointed at a label nothing is serving — jump to When the runner is offline.

If the repo is not private, stop. Do not register a runner.

2. Preview the registration

bash

scripts/fleet/runner.sh install --machine farm-01 --print

This mints no token, touches nothing, and needs no network. It prints the two phases that install will run, so you can read exactly what lands on the machine before it lands:

Phase A (unprivileged): download the runner tarball into ~/actions-runner-bob on the target, then config.sh --unattended --replace to register it. This is the phase that carries the registration token, and it is streamed over ssh stdin so the token never enters an argument vector on either machine.
Phase B (privileged): svc.sh install + svc.sh start, run as the unprivileged login user. Carries no secret, which is why it can safely be a separate invocation with a sudo tty.

Pin a specific runner release with RUNNER_VERSION if you need to; by default the target resolves the latest release itself.

3. Register the machine

bash

scripts/fleet/runner.sh install --machine farm-01

Phase B will prompt for the sudo password on the target. Re-running the command is safe: --replace re-registers the same runner name rather than accumulating duplicates.

Verify:

bash

scripts/fleet/runner.sh status

The runner appears as farm-01 online [self-hosted-bob]. CI still targets ubuntu-latest at this point — registration and switching are deliberately separate, so a half-finished setup never redirects CI.

4. Point CI at the runner

bash

scripts/fleet/runner.sh target self-hosted-bob

The command refuses if no online runner carries that label (--force overrides, for the case where you know the machine is mid-boot).

Verify: push a commit or re-run the CI workflow, then open the run. The job header names the self-hosted runner rather than a GitHub-hosted image, and the run consumes no billable minutes. gh run list --limit 1 should show it completing, not sitting in queued.

5. Go back to hosted

Any time — a machine reboot, a maintenance window, or a red run you suspect is environmental:

bash

scripts/fleet/runner.sh target hosted

The next run uses ubuntu-latest again. Nothing needs deregistering to do this.

6. Remove the runner entirely

bash

scripts/fleet/runner.sh remove --machine farm-01

Order matters and the script enforces it: the CI target is reset first, then the service is stopped, then the runner is deregistered. Removing a runner while CI still points at its label is the one sequence that strands the repo.

Another repo

Add --repo <owner/name> to any command above and it acts on that repo instead of this one. Nothing else about the procedure changes — same phases, same guards, same inverse:

bash

scripts/fleet/runner.sh status  --repo paulirv/peabod-www
scripts/fleet/runner.sh install --repo paulirv/peabod-www --machine farm-01
scripts/fleet/runner.sh target  self-hosted-bob --repo paulirv/peabod-www

Patch the target repo's workflows first. The switch is a repo variable that does nothing unless the workflows read it. A repo whose jobs still say runs-on: ubuntu-latest will keep going to hosted after you set the variable, with no error anywhere — the same silent failure #1782 describes. Convert every workflow in the repo, not just ci.yml:

yaml

runs-on: ${{ vars.CI_RUNNER_LABEL || 'ubuntu-latest' }}

Each repo gets its own runner home. A runner directory holds exactly one .runner config, so two repos cannot share one — registering a second repo into an occupied directory replaces the first runner rather than adding to it. The path is derived as ~/actions-runner-<repo-name>, overridable with --dir. This repo is the one exception: its runner predates the convention and stays at ~/actions-runner-bob, which runner.sh resolves automatically.

The service units are namespaced by repo already (actions.runner.<owner>-<repo>.<machine>.service), so several runners coexist on one machine:

bash

systemctl list-units 'actions.runner.*'

What one machine serving several repos costs you. Each runner is a separate process holding a job slot, and they compete for the same cores, disk, and network as whatever else the machine does — on farm-01 that includes warp-drive and cdfork sessions. Concurrency, workspace isolation, and disk reaping across repos are tracked in #1732; until that lands, treat "how many repos can this box gate at once" as an open question rather than a settled one.

Deploy workflows are a separate decision from CI. A self-hosted runner persists between jobs, so a deploy job's credentials land on long-lived disk where the next job can read them. The standing recommendation (#1725) is to keep CD on hosted runners or run it from a workstation, and move only the gate to the farm.

Service control on the target

The runner runs as a service (systemd on Linux, launchd on macOS) out of ~/actions-runner-bob:

bash

cd ~/actions-runner-bob
sudo ./svc.sh status
sudo ./svc.sh stop
sudo ./svc.sh start

Updates are automatic. GitHub self-updates the runner application, so there is no routine upgrade step. If an update fails or the install is corrupted, the repair is a re-register: runner.sh install --machine farm-01 re-downloads and replaces in place.

When the runner is offline

A job assigned to an offline runner queues — it does not fail. There is no red check to notice, no error to read, and no automatic fallback: the run simply sits in Queued. GitHub's documented limit is that "a job can be in the queue for 24 hours before it is automatically cancelled," so a weekend outage silently eats a day of delivery before anything visibly goes wrong.

Detect it:

bash

scripts/fleet/runner.sh status    # exit 3 when CI targets a label with no online runner
gh run list --limit 5             # a run stuck in `queued` corroborates

Because status exits non-zero in exactly this state, it works as a gate in scripts and as a fast first check when CI "isn't running."

Recover it — restore delivery first, diagnose second:

scripts/fleet/runner.sh target hosted — the next run goes to ubuntu-latest.
Re-run the queued workflow (gh run rerun <id>); it picks up the hosted runner.
Now diagnose the machine: is it up, is it on the tailnet, is the service running (sudo ./svc.sh status)?
When it is healthy again, scripts/fleet/runner.sh target self-hosted-bob.

If Actions is down or capped entirely — a different failure from this one — the fallback is Local Delivery Fallback.

Security constraints

A self-hosted runner is a machine on your network executing code from a repo. Four constraints keep that acceptable; the first is absolute.

The repo must stay private. On a public repo, anyone who opens a fork pull request can execute arbitrary code on the runner — GitHub warns against this explicitly, and it is a full compromise of the host, not a sandbox escape. runner.sh install refuses to register against a public repo, and runner.sh status prints a security warning if it ever finds runners attached to one. If this repo is ever made public, remove the runner before flipping visibility.
The runner user is unprivileged. config.sh runs as the login user, and the service is installed for that same user — never root. Jobs inherit that user's access, so do not run the runner as an account with broader reach than CI needs.
The registration token never enters an argument vector. It is streamed to the target over ssh stdin, so it does not appear in ps output, a shell history, or a process listing on either machine. Registration tokens are short-lived, but a leaked one is enough to attach a hostile runner to the repo. The privileged phase is separated precisely so no secret ever needs a sudo tty.
No secrets beyond what CI needs. The workflow requires nothing but the toolchain (node, python, jq, make, git); it reads no repo secrets. Keep it that way — a self-hosted runner persists between jobs, so anything a job writes to disk or environment is visible to the next one. Deploy credentials belong on hosted runners or out-of-band, not here.

The runner is also not a clean-room: unlike a hosted image, the filesystem persists across jobs. Treat a green run on it as evidence the code works on that machine, and keep the hosted path as the reproducible reference.

Runbook: Self-Hosted Actions Runner ​

Steps ​

1. Check the current state ​

2. Preview the registration ​

3. Register the machine ​

4. Point CI at the runner ​

5. Go back to hosted ​

6. Remove the runner entirely ​

Another repo ​

Service control on the target ​

When the runner is offline ​

Security constraints ​

See also ​

Runbook: Self-Hosted Actions Runner

Steps

1. Check the current state

2. Preview the registration

3. Register the machine

4. Point CI at the runner

5. Go back to hosted

6. Remove the runner entirely

Another repo

Service control on the target

When the runner is offline

Security constraints

See also