add-runner-eval

---
name: add-runner-eval
description: Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.
---

# Add a Runner Eval

Use this skill for the **Runner Evals** family: a real Runner/provider session
against a seeded mock control plane. Product browser/server/database/Daytona
coverage belongs in [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md).

Locate the Paperclip checkout using `PAPERCLIP_ROOT` when supplied, or
`git rev-parse --show-toplevel` from a checkout. From outside Git, inspect the
workspace roots (for example `~/paperclipai/paperclip`) and verify that the
selected root contains `packages/paperclip-runner` and `tests/runner-e2e`.
Locate `paperclip-evals` using `PAPERCLIP_EVALS_ROOT` or a discovered sibling;
a worktree's parent directory need not contain that repository. Read
`doc/evals.md` and `packages/paperclip-runner/docs/runner-protocol-live-evals.md`,
then inspect the nearest existing case, roster, schema, and report test before
editing. Definitions and authored cases belong in the sibling
`paperclip-evals/evals/paperclip-runner`; Runner integration, aggregation,
viewer, and publication behavior belongs in `packages/paperclip-runner`.
Keep the control-plane boundary explicit in names and documentation.

The sibling eval README is the concrete map: cases live under `cases/`,
company fixtures under `fixtures/`, runtime/model settings under `configs/`,
selections under `rosters/`, and maintained campaign membership under
`campaigns/live-direct-full.json`. Update inventory/coverage mappings when the
program requires them; a new file alone does not join the maintained campaign.
From the Evals repository root, adapt these provider-free checks to the case
and roster you changed. Run the reliability-plan validator only when that
separate plan changes:

```sh
python3 evals/paperclip-runner/tools/eval_program.py validate \
  --case evals/paperclip-runner/cases/get-task-context.json \
  --config evals/paperclip-runner/configs/live-codex-pinned.json
python3 evals/paperclip-runner/tools/run_live_roster.py validate \
  --roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case
python3 evals/paperclip-runner/tools/run_live_campaign.py validate \
  --campaign evals/paperclip-runner/campaigns/live-direct-full.json
python3 evals/paperclip-runner/tools/reliability_campaign.py validate \
  --plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json
```

Use nearby positive and negative grader cases/fixtures to calibrate the new
assertion, including malformed or missing evidence where the grader must fail
closed. Preserve the existing machine disposition and grade; product,
model/provider, grading, and infrastructure labels are analytical annotations,
not instructions to rewrite classifiers.

Author one bounded case with a deterministic semantic assertion and an
inspectable result. Declare its expected operation, state effect, provider
lane/profile, timeout and retry policy, and any required evidence. Do not grade
hidden reasoning, infer success from a provider terminal message, or invent
conversation/tool evidence. Public output follows the reviewed projection:
sanitized fixture conversation and allowlisted tool outcomes may be published;
raw trusted artifacts, credentials, secrets, private references, and hidden
reasoning may not.

Validate without provider calls first using the commands above and the relevant
report/render validation documented in the Runner docs. When a live run is
authorized, pin the Paperclip commit and exact 40-character
`paperclip-evals` commit, select the smallest useful roster, and retain the
complete provenance and cost record.

Update authoritative detailed docs when the contract or command changes, then
link from `doc/evals.md` rather than duplicating the Runner runbook. Keep public
reports immutable and use the reviewed projection; sanitized fixture
conversation and allowlisted tool outcomes may be public, while credentials,
secrets, private references, raw trusted payloads, and hidden reasoning must not
be exposed.