Verification & Testing System

TL;DR. make check validates the system (deps, schemas, symlinks, hooks, provisions); make test runs the suites; make ci runs both — what GitHub Actions runs. One command to verify everything.

Bob's infrastructure-as-code verification layer. One command to validate the entire system.

make check    # verify everything
make test     # run all tests
make ci       # both (what GitHub Actions runs)

Quick Start

bash

cd ~/projects/bigbrain    # or wherever BOB_SOURCE is
make help                 # see all targets
make check-deps           # am I set up correctly?
make check                # is everything healthy?
make test                 # do all tests pass?

Targets

Verification (`make check`)

make check runs all five checks in order. Stops at first failure.

Target	What It Checks	Speed
`check-deps`	node >= 18, jq, git, make installed	instant
`check-schemas`	All JSON configs valid against schemas	instant
`check-manifest-meta`	Every `provisions/<project>.json` carries its own `_meta.project` — never the `_default` template's (#1869)	instant
`check-symlinks`	Symlinks in the current repo resolve (`--all` / `make check-fleet` for every project)	~1s
`check-hooks`	All hooks in settings.json exist + executable	instant
`check-provisions`	The current repo's symlinks match its manifest (`--all` for the fleet)	~1s
`check-scenarios-drift`	Every command/flag/path/count named by the workday scenarios page still ships (#843)	instant
`check-abs-paths`	No machine-absolute path (`/Users/<name>/`, `/home/<name>/`) in any synced or ledgered file, and no absolute-target symlink in a synced tree (#1924). Allow-list with a `why` per entry; `--apply` migrates	instant
`check-host`	This machine's `BOB_HOME` matches source on every deploy-managed surface, every symlink resolves here, toolchain preconditions hold, settings hooks converged (#1921). Skipped under `$CI` — it grades the host, not the checkout. Alias `make doctor` / `bob doctor`	~0.2s
`check-fleet`	`check-symlinks` + `check-provisions` across all registered projects (#294)	~2s

Testing (`make test`)

Target	What It Tests	Count
`test-schemas`	Schema validator unit tests (type, enum, pattern, etc.)	34
`test-warp-drive`	Warp-drive state machine transitions	99
`test-checks`	Integration tests for all check scripts	13
`test-scenarios-drift`	The scenarios drift-check catches a broken reference and honours the allowlist (#843)	5
`test-render-scenarios`	The scenarios two-render split strips/keeps the friction margin and is idempotent (#844)	5

CI (`make ci`)

Runs check then test. This is the target GitHub Actions calls on every push and PR to master.

Check Details

check-deps

Verifies the four required tools are installed:

  PASS  node 25.5.0 (>= 18)
  PASS  jq 1.7.1
  PASS  git 2.48.1
  PASS  make (GNU Make 3.81)

If something fails, the output includes a fix command:

  FAIL  node 16.20.0 (need >= 18)
        Fix: Install Node.js 18+ via nvm or brew

check-schemas

Validates every JSON configuration file against its JSON Schema definition:

File	Schema
`projects.json`	`schemas/projects.schema.json`
`settings.json`	`schemas/settings.schema.json`
`provisions/*.json` (17 files)	`schemas/manifest.schema.json`

If something fails, the output shows the exact path and error:

  FAIL  provisions/bodmail.json
        $._meta.project: expected type string, got number
        $.skills[3]: duplicate item "webapp-testing"

check-manifest-meta

Guards manifest identity (#1869). cdprov --init --auto once copied provisions/_default.json verbatim when no stack was detected, leaving _meta.project: "_default" and _meta.path: null — a name that collides across projects in audit.js / cdprov status and a path nothing can resolve. The generator now stamps both; this gate keeps the corpus honest:

FAIL — _meta.project missing, "_default", or not equal to the manifest's basename.
WARN (non-fatal) — _meta.path is null; set it ("$HOME/projects/<name>" convention) or re-run cdprov --init --auto from the project.

Exempt: _default*.json (templates), _bob-home*.json (the BOB_HOME surface, #411) and ports.json (the port-slot registry, #197).

To fix: Edit the JSON file to match the schema. The error path ($._meta.project) tells you exactly where the problem is.

check-symlinks

Scans every project listed in projects.json, finds all symlinks in each project's .claude/ directory, and verifies each one resolves to a real file.

  FAIL  /Users/you/Sites/myproject/.claude/commands/sprint.md -> /Users/you/.claude/registry/commands/sprint.md (dangling)

To fix dangling symlinks:

bash

# Option 1: Re-provision the project (recommended)
cdprov --refresh /path/to/project

# Option 2: Remove the stale symlink manually
rm /path/to/.claude/commands/sprint.md

Projects that don't exist on disk or lack a .claude/ directory are silently skipped.

check-hooks

Reads settings.json, extracts every hook command path (from SessionStart, PreToolUse, PostToolUse, Stop), and verifies:

The script file exists
The script is executable (chmod +x)

Also checks the statusLine.command if configured.

  PASS  /Users/you/.claude/hooks/temporal-context.sh
  FAIL  /Users/you/.claude/hooks/missing-hook.sh
        Fix: File not found — remove from settings.json or create the script

To fix:

bash

# If the hook should exist:
chmod +x ~/.claude/hooks/the-hook.sh

# If the hook was removed:
# Edit settings.json and remove the hook entry

check-provisions

For each project in projects.json, loads its manifest (provisions/<project>.json or provisions/_default.json) and verifies that every declared skill, command, and agent is:

Present in the registry (registry/skills/, registry/commands/, registry/agents/)
Symlinked into the project's .claude/ directory
Not dangling

  PASS  bodmail (bodmail.json)
  FAIL  myproject: commands/sprint declared in _default.json but not symlinked
  FAIL  myproject: agents/code-expert symlink is dangling

Failure types and fixes:

Failure	Meaning	Fix
`declared but missing from registry`	Manifest references something that doesn't exist	Remove from manifest or add to registry
`declared but not symlinked`	Item exists in registry but wasn't linked to project	Run `cdprov --refresh /path/to/project`
`symlink is dangling`	Symlink exists but target was moved/deleted	`cdprov --refresh` if the target still exists; `cdprov --prune` if it's gone/cross-machine (see below)

check-abs-paths

State that deploy.sh syncs to every machine, or that a ledger records, rots the moment it carries one machine's home directory. The two recorded shapes are a symlink inside a synced tree whose target is absolute — rsync -l copies the link verbatim, so a runbook link written on the Mac pointed at /Users/paulirving/... on the farm (#1914) — and a manifest whose _meta.path exists on one machine only (#1919). The #1475 LINK_ROOT rule (see the handbook) removes the cause for registry links at write time; cdi's universal links and cdprov's universal retargets follow it too (every link BoB writes targets $BOB_HOME, never the source checkout), and check-abs-paths catches everything else at commit time.

check-abs-paths runs scripts/checks/abs-path-lint.js:

Scope — skills/, registry/, runbooks/, templates/, provisions/, scripts/, hooks/, bin/ and every *.json at the repo root. docs/ is deliberately excluded: prose quotes real paths as examples.
Files fail on any literal /Users/<name>/ or /home/<name>/. Symlinks fail on any absolute target — a synced link cannot be derived from $HOME on the receiving machine, so it must be relative.
Allow-list — scripts/checks/abs-path-allow.json. Each entry is { "path", "pattern"?, "why" }; path is a file or directory prefix, pattern an optional substring the offending line must contain. An entry without a why is a config error (exit 2) — this is a list of documentation examples, not a suppression list, and there is no implicit comment exemption (a comment is where a copy-pasted real path hides best).
Migration — --apply (dry-run by default; the report always shows what it would do) rewrites a JSON string value that is an absolute home path to the $HOME/... form the hand-authored manifests use and provision.sh manifest_project_path writes (#1869), and rewrites a symlink whose absolute target lies inside the repo to the equivalent relative target. Each rewrite is reported; a hit in a script, or a link pointing outside the repo, is listed as unmigratable for a human. Idempotent.
Contract — --json emits abs-path-lint/1: {hits[], allowed, rewrites[], unmigratable[], ok}. --root DIR lints another tree (fixtures). Exit 0 clean · 1 hit(s) remain · 2 usage/config error.

abs-path-lint — /Users/.../bigbrain
  scope: skills registry runbooks templates provisions scripts hooks bin *.json   allow-list: scripts/checks/abs-path-allow.json (1 allowed hit(s))
  would rewrite 1 ledgered path(s) (dry run — pass --apply):
    [would-rewrite] provisions/seebod.json _meta.path: /home/farm/Sites/seebod -> $HOME/Sites/seebod
  FAIL  1 machine-absolute path(s) in synced/ledgered files:
    provisions/seebod.json:4  /home/farm/  "path": "/home/farm/Sites/seebod",

Where a path is legitimately machine-specific, derive it at use time — $BOB_HOME for deployed tooling, $HOME for project roots (the port ledger's contractHome, projects.json), $BOB_SOURCE for the checkout — or record a project id and resolve it through projects.json on the current machine.

check-host (`bob doctor`)

make check grades the checkout; nothing graded the deployed runtime a machine actually runs (#1914). A laptop can sit months behind source on scripts/, carry registry symlinks that point at another user's home, or run the real gh ahead of the forge shim — and every other check stays green. check-host runs scripts/checks/host-drift.js against this machine's BOB_HOME (#1921). Same entry point from bin/: bob doctor [--json] [--project <dir>] [--section a,b].

Four sections, each graded independently and each naming its own remediation:

Section	What it grades	Fix it names
`surfaces`	Every deploy-managed directory and file — the list is derived from `deploy.sh`'s own `sync_dir` / `sync_file` lines, so the check and the deploy cannot disagree about what is managed. Reports `missing` / `stale` / `extra` (extras count: `deploy.sh` syncs with `--delete`). Exclusions are an explicit list (`.DS_Store`, `__pycache__`, `node_modules`, `.git`), never a heuristic	`scripts/deploy.sh`
`symlinks`	Every symlink under `BOB_HOME` (runtime dirs like `projects/`, `cache/` skipped) and under the current project's `.claude/` resolves on this machine. A dangling target, or a target under a foreign home (`/Users/someone-else/…`, `/home/…` on a Mac), fails and names the link	`scripts/deploy.sh` (BOB_HOME) · `cdprov refresh` (project links)
`toolchain`	`node`, `jq`, `gh` on PATH; `BOB_HOME/bin/gh` resolves ahead of the real `gh` (the forge shim must win, #1748); the one guarded `bob-shell.zsh` line is in `~/.zshrc`	install the tool · `scripts/bob-install.sh`
`settings_hooks`	`settings.json` hook convergence — `hook-drift.js` (#1442/#1918) included as one section, not re-implemented	`scripts/deploy.sh` (reconciles `.hooks`)

Exit code is the answer: 0 converged, 1 drift, 2 error/usage. --json emits a versioned bob-host-drift/1 document (verdict, drift_count, sections[]), which the fleet snapshot carries per machine as summary.host_drift and audit.js view rolls up as a "BOB_HOME drift" table — so a machine behind source is visible from any other machine, with the fix named.

BOB_HOME drift — /Users/you/.claude vs /Users/you/projects/bigbrain
  FAIL  surfaces: 2/23 surface(s) behind source (4 file delta)
        scripts/                 missing 1  stale 2  extra 0
        CLAUDE.md                missing 0  stale 1  extra 0
          missing  scripts/fleet/settings-hooks-reconcile.js
          stale    scripts/deploy.sh — content differs from source
        Fix: scripts/deploy.sh
  PASS  symlinks: 134 link(s) resolve on this machine
  PASS  toolchain: node, jq, gh present; forge shim first on PATH; shell guard installed
  PASS  settings_hooks: 25 declared hook(s) wired

Drift: 4 — run scripts/deploy.sh, then re-run make check-host.

Not graded in CI ($CI set → SKIP), for the same reason as check-hooks scope 3 (#1708): the verdict would depend on when the runner host last ran deploy.sh, not on the commit under test. The signal is not lost — the fleet audit owns per-machine drift.

check-scenarios-drift

Keeps the narrative workday scenarios page honest: it re-grounds every BoB surface the page names against the actually-shipped inventory, so the page can't quietly rot into fiction against BoB's daily churn (#843). The shipped command/skill/agent inventory is reused from the same harvester the docs-site reference generator uses (scripts/docs-site/lib/gen-reference.js) — the check and the reference can never disagree about what ships. It is deterministic and side-effect-free (pure filesystem reads, no network, no deploy), consistent with the "audits stay cheap" rule.

What it verifies — three backticked syntactic classes plus prose counts:

Class	Example	Grounded against
Slash command	`/warp-drive`	a shipped command or skill name
Repo file/dir path	`scripts/fleet/audit.js`	the file/dir exists on disk
CLI flag	`--from-issues`	the literal appears in a shipped command/script source
Surface count	"18 universal commands"	the live `gen-reference` inventory count

Bare words (cap, todo, cdprov) are not checked — in prose they are indistinguishable from domain nouns, and repo aliases like cdprov resolve through shell config outside the repo. In-repo CLI entrypoints are still covered, via their scripts/… / bin/… paths.

  STALE  docs/explanation/workday-scenarios.md:84  /nonexistent-command   (no such command or skill)

FAIL — 1 stale reference(s) in docs/explanation/workday-scenarios.md.

How to fix a failure:

Renamed/removed surface — update the reference in the page to the surface's current name (or drop it).
Deliberate forward-reference — when the page intentionally names a surface a sibling requirement will add (e.g. /update-journey, built by #846), declare it in an HTML comment in the page so the allowance is explicit and visible:
```

```
Remove the allowance once the surface ships.

Run it directly with make check-scenarios-drift (it is part of make check).

Repairing drift (prune & repair-fleet)

check-fleet reports drift; these repair it (#303).

`cdprov --prune` — per-project

Removes dangling symlinks under a project's .claude/ that point into BoB (*/.claude/*, $BOB_SOURCE, $BOB_HOME). It is dry-run by default; pass --yes (or --apply) to delete. Each candidate is classified:

removed-tooling — target under this machine's BoB home/source but gone (e.g. the retired PM-file / sprint / requirements / risk items).
cross-machine — an absolute target under a different home's .claude (e.g. a repo provisioned on another box: /home/farm/.claude/...).
other — any other dangling BoB symlink.

Safety guarantees (covered by make test-prune): prune never removes a resolving symlink, a real (non-symlink) file, or a foreign symlink (one whose target is not a BoB path). It is idempotent.

bash

cdprov --prune            # dry run — preview what would be removed, by class
cdprov --prune --yes      # apply

`make repair-fleet` — every project

Walks projects.json and, per live project, runs cdprov --prune then (on apply) cdprov --refresh to re-link currently-declared items. Dry-run unless ARGS=--yes, and best-effort — a failure in one project warns and the run continues.

bash

make repair-fleet            # dry run across the whole fleet (prune preview)
make repair-fleet ARGS=--yes # apply: prune + refresh every registered project
make check-fleet             # confirm remaining drift afterwards

The prune→refresh order matters: prune clears stale/cross-machine links first, then refresh recreates the items the manifest still declares.

JSON Schemas

Located in schemas/. Each defines the exact structure a config file must follow.

manifest.schema.json

For provisions/*.json files:

json

{
  "_meta": {
    "project": "string (required)",
    "path": "string | null",
    "stack": ["string"]
  },
  "skills": ["string (unique)"],
  "commands": ["string (unique)"],
  "agents": ["string (unique)"]
}

projects.schema.json

For projects.json:

json

{
  "_meta": {
    "description": "string (required)",
    "updated": "YYYY-MM-DD"
  },
  "projects": [{
    "name": "string (required)",
    "path": "string (required)",
    "group": "string (required)",
    "active": "boolean (required)"
  }]
}

settings.schema.json

For settings.json:

permissions.allow / permissions.deny — string arrays
hooks — keyed by event name (SessionStart, PreToolUse, PostToolUse, Stop, Notification, SubagentStop)
Each hook entry has type: "command", command: string, optional timeout: integer
statusLine — type + command
enabledPlugins — map of string: boolean

File Layout

~/.claude/
├── Makefile                          # Entry point — run `make help`
├── schemas/
│   ├── manifest.schema.json          # Provisions manifest schema
│   ├── projects.schema.json          # Project registry schema
│   └── settings.schema.json          # Settings schema
├── scripts/checks/
│   ├── check-deps.sh                 # Dependency verification
│   ├── check-hooks.sh                # Hook script verification
│   ├── check-manifest-meta.sh        # Manifest identity (_meta.project / path, #1869)
│   ├── check-provisions.sh           # Manifest-vs-reality verification
│   ├── check-schemas.js              # JSON Schema validation runner
│   └── schema-validator.js           # Shared validator module (zero deps)
├── tests/
│   ├── test-checks.sh                # Integration tests (13 tests)
│   └── test-schemas.js               # Schema validator unit tests (34 tests)
└── .github/workflows/
    └── ci.yml                        # GitHub Actions — runs on push/PR

CI

.github/workflows/ci.yml runs on pushes and PRs to master:

Checkout repo
Setup Node.js 22 (the repo is ESM-first and relies on Node's ES-module auto-detection)
Verify the runner toolchain (jq) and install a pinned PyYAML for the versioning tests
make check-deps check-schemas check-hooks check-intake (skip symlinks/provisions — no projects on a CI runner)
make test
make docs-check

CI skips check-symlinks and check-provisions because they validate local provisioning state — the cdi-created symlinks in a working copy — which doesn't exist in a bare CI checkout (.claude/ symlinks are gitignored). Since #294 these checks default to the current repo (so make check in any working copy is no longer failed by drift in other registered projects); the cross-project sweep moved behind --all / make check-fleet. They remain local/dev gates by design.

Two lanes: source and docs (#1693)

CI is not the only gate, and it deliberately does not run on every push. The repo has two workflow lanes, split by path:

Lane	Workflow	Triggers on	Runs
Source	`ci.yml`	everything except `docs/`, `runbooks/`, `README.md`, `CLAUDE.md`	the checks, the full test suite, `docs-check`
Docs	`docs-deploy.yml`	exactly those doc paths (plus the docs-site tooling)	site build, flat-link verify, `docs-check`, deploy

A docs-only change therefore runs the docs lane alone rather than dragging a 240-target suite behind it, and make docs-check is what makes that safe: the doc naming, filename-convention, and anchor gates are prerequisites of that target, not separate CI steps, so they fire from either lane. Both lanes also carry a per-ref concurrency group with cancel-in-progress, so a burst of pushes bills one run rather than one per commit.

Two consequences worth knowing:

Keep the two path lists in step. A path that is in ci.yml's paths-ignore but not in docs-deploy.yml's trigger paths is gated by nothing. make test-ci-lane-split asserts the containment and fails if it ever drifts.
A few test targets read the real docs tree, and those do not run on a docs-only push — they run on the next push that touches source. No release can be cut in between, because auto-release.yml is itself filtered to the source surface. The site-breaking risks (build failure, orphan, drift) are all covered by the docs lane, which does run.

The release bot's own commit is excluded from both lanes: its subject carries [skip ci] (see Versioning), since a version bump plus a regenerated changelog gives either lane nothing to decide.

Common Workflows

After cloning Bob on a new machine

bash

cd ~/projects/bigbrain    # BOB_SOURCE
make check-deps           # verify tools are installed
make check-schemas        # verify configs are valid
make check-host           # this machine's BOB_HOME matches source (or: bob doctor)

After changing settings.json or a hook

bash

make check-hooks check-schemas

After changing a provision manifest

bash

make check-schemas check-provisions

After adding a new project

bash

# Add to projects.json, then:
make check-schemas      # validates projects.json structure
make check-symlinks     # checks new project's symlinks
make check-provisions   # checks new project matches manifest

Before pushing changes to Bob

bash

make ci     # runs everything

Investigating a project that feels broken

bash

make check-symlinks     # find dangling symlinks
make check-provisions   # find manifest mismatches
cdprov --refresh /path/to/project   # fix it

Rendering the scenarios page (two-render split, #844)

The narrative workday scenarios page carries an introspection layer — the "friction margin" that is honest about where BoB's seams are clumsy. That honesty makes a poor outsider-facing brochure, so the friction content is wrapped in HTML-comment markers and stripped or kept at render time, from a single source (no forked copies to keep in sync):

<!-- friction:TYPE -->   ...introspection-only content...   <!-- /friction -->

TYPE is one of pain-point / automation-opportunity / candidate-feature / section. Produce either render with scripts/render-scenarios.js:

bash

# Outsider-facing: friction blocks removed, clean narrative.
node scripts/render-scenarios.js --mode showcase       [--out showcase.md]

# Maintainer-facing: friction content kept (markers stripped).
node scripts/render-scenarios.js --mode introspection  [--out introspection.md]

The transform is pure and deterministic — same input always yields the same output, and re-running it is idempotent. HTML comments (the friction fences, the authoring note, the drift-allow marker) are internal metadata and are stripped from both renders; only the friction content differs between them. The test-render-scenarios target proves showcase-strips / introspection-keeps / idempotent.

Design Principles

Zero dependencies — only node, jq, git, make (all pre-installed or one brew install)
Every check exits 0 (pass) or 1 (fail) — no ambiguity
Actionable output — every failure includes a fix instruction
Idempotent — safe to run any target as many times as you want
Fast — full make check completes in under 5 seconds
Composable — run individual targets or combine them: make check-deps check-hooks

Verification & Testing System ​

Quick Start ​

Targets ​

Verification (make check) ​

Testing (make test) ​

CI (make ci) ​

Check Details ​

check-deps ​

check-schemas ​

check-manifest-meta ​

check-symlinks ​

check-hooks ​

check-provisions ​

check-abs-paths ​

check-host (bob doctor) ​

check-scenarios-drift ​

Repairing drift (prune & repair-fleet) ​

cdprov --prune — per-project ​

make repair-fleet — every project ​

JSON Schemas ​

manifest.schema.json ​

projects.schema.json ​

settings.schema.json ​

File Layout ​

CI ​

Two lanes: source and docs (#1693) ​

Common Workflows ​

After cloning Bob on a new machine ​

After changing settings.json or a hook ​

After changing a provision manifest ​

After adding a new project ​

Before pushing changes to Bob ​

Investigating a project that feels broken ​

Rendering the scenarios page (two-render split, #844) ​

Design Principles ​

Verification & Testing System

Quick Start

Targets

Verification (`make check`)

Testing (`make test`)

CI (`make ci`)

Check Details

check-deps

check-schemas

check-manifest-meta

check-symlinks

check-hooks

check-provisions

check-abs-paths

check-host (`bob doctor`)

check-scenarios-drift

Repairing drift (prune & repair-fleet)

`cdprov --prune` — per-project

`make repair-fleet` — every project

JSON Schemas

manifest.schema.json

projects.schema.json

settings.schema.json

File Layout

CI

Two lanes: source and docs (#1693)

Common Workflows

After cloning Bob on a new machine

After changing settings.json or a hook

After changing a provision manifest

After adding a new project

Before pushing changes to Bob

Investigating a project that feels broken

Rendering the scenarios page (two-render split, #844)

Design Principles