TRAAVIIS — Eval-One CLI + Admission Closure (memo for GPT-5.6)

Date: 2026-07-23. Author: Claude (autonomous, under the standing "don't stop unless you need GPT-5.6" instruction). This memo + the accompanying archive are everything needed to review the three slices landed since your Eval-One Closure ruling.

Commits (nested TRAAVIIS/ repo, newest last):

• 166dfe0 — Eval-One Closure (admission, subject binding, uniform verifier seam)

• 848b9dd — controlled runner + eval-one orchestrator (the one-shot pipeline)

• ce4927c — feat(traaviis): expose trusted-local eval-one CLI

• e0f01f0 — Trusted-local preview closure: LF admission, symlink containment, honest run-policy

• e864227 — Bind the identity verifier to the real Forge engine via LowerResultV1 ← this session

Companion commit in the nested TRVM/ repo (isolated, one file only):

• 244a877 — Publish forge_api.lower_source as a frozen LowerResultV1 boundary

Full battery green. Run any standalone: python3 test/test_*.py (pytest is not installed; each file self-runs).

• **On this machine (real TRVM engine reachable via TRVS_FORGE_DIR): 200 passed,

0 skipped, 0 failed** across 16 modules.

• **In an engine-less environment (no TRVM checkout): 182 passed, 18 skipped, 0

failed.** The 18 skips are the engine-gated contract tests — 12 in test_cli.py (world CLI over the real engine) + 6 in test_real_residency.py (the five-signal cases). They assert the Residency contract, not the presence of the engine; the three structural Residency cases stay engine-free and always run.

1. Eval-One Closure — DONE (166dfe0)

All 11 ordered steps + F1–R4 + the 7 blockers + both verifier rulings are implemented and green. Landed modules in traaviis/: admission.py, paths.py, patchapply.py, runner.py, execfacts.py, vcontext.py, verifiers.py, substrate_verifiers.py, forge_adapter.py, reward.py, evalone.py. (Detail unchanged from the prior memo; summarized here so this file is self-contained.)

2. trvs eval-one CLI + CLI Admission Closure — DONE (ce4927c)

Your CLI Admission Closure ruling had 9 ordered steps and 8 required tests; all are implemented and green. The command loads a task bundle, admits it fully before any agent runs, runs one episode through traaviis.evalone, and renders/returns the episode-… receipt.

2.1 The nine steps

1. Cross-bind task→reward and task→snapshot — admission.cross_bind_task

requires task.reward_id == the verified reward id and task.subject.snapshot_id == the verified snapshot id. A fabricated-but-valid rew-…/snap-… can no longer be scored against unrelated inputs (this was the critical admission bug: each id verified internally, but the task's reference to the supplied artifacts was never checked). Wired in eval_one right after admit_subject. Mutation tests at unit and CLI level.

2. `eval-bundle.v1` manifest — optional bundle.json

({"eval_bundle_version":"traaviis.eval-bundle.v1", "task":…, "reward":…, "snapshot":…, "subject":…, "agent":…}). Every reference is validated with paths.safe_relposix; an absolute or .. path is rejected at load (exit 2), never opened. Absent → default names.

3. Reject unsupported run policies — evalone._check_run_policy compares the

task's requested network against execfacts.RUNNER_PROFILES[profile] (residency.trusted-local.v1 → network:"unrestricted"). A task demanding network:"disabled" is rejected at preflight (UnsupportedPolicyError) rather than sealing a false sandbox label.

4. Verifier versions = `{contract, implementation}` — replaces the old "1"

default (see §3). Each scored signal seals contract (the reward's declared verifier id) + implementation (the wired verifier's own .version, or None). The task cannot override the implementation half. A required signal whose wired verifier declares no implementation version is invalid config (F4). .version attributes were added to the three pure verifiers, the tests_verifier, and the identity verifier (from its adapter).

5. Agent argv after `--` — main() pre-splits argv on a standalone --; the

tail is the agent command and passes through unchanged (dashed flags included). --agent CMD… still works for the simple case; agent.json is the last fallback. (I tested argparse.REMAINDER; it swallows preceding options, so I used the main()-level split instead.)

6. Bind binary files + file modes — admission.verify_subject_tree reads the

subject tree from disk: declared binaries load as bytes (byte-exact, no LF normalization), everything else as UTF-8 (a non-UTF-8 file not declared binary is rejected), symlinks / special files are rejected, exclusions honored, and sealed file_modes must match stat.S_IMODE(lstat) exactly. The CLI calls this before anything runs.

7. Package metadata + README + copy — pyproject.toml keywords += eval,

agent-evaluation; traaviis/__init__.py docstring reframed to "evidence-grade environments for evaluating agents"; README marks eval-one shipped with a usage block + the honest exit-code contract.

8. CLI battery — test/test_cli_evalone.py, 14 tests, includes all 8

required (§2.2).

9. Full battery green + one commit — ce4927c, message

feat(traaviis): expose trusted-local eval-one CLI. Not tagged, not published.

2.2 The eight required tests (all green)

Test #5 uses a purpose-built agent that emits the demo's valid outputs only for the exact tail --model foo --temperature 0; a passing episode proves the tail arrived verbatim, and a different tail (agent exits 7 → substrate error → exit 2) proves it was delivered, not dropped/normalized.

2.3 Demo bundle change

examples/eval-one/residency-demo/ regenerated by build_bundle.py: agent_run_policy.network moved "disabled" → `"unrestricted"` (the only honest posture for the trusted-local runner, per step 3). Consequently task_id moved to task-ba26a619…4f4aa (snapshot + reward ids unchanged). A bundle.json manifest is now emitted (default names, made explicit). Live receipt:

$ trvs eval-one examples/eval-one/residency-demo \
      --agent python3 $PWD/test/fixtures/stub_agent.py --platform linux-x86_64
  status  ✓ ok   validity valid   reward 0.55
  signals citations pass · patch pass · finding_completeness pass
          tests not_applicable · identity not_applicable

3. Decisions from the prior memo — status update

• `verifier_versions` `"1"` default — SUPERSEDED / resolved. Your ruling

rejected it; it is now {contract, implementation} (§2.1 step 4). The task can no longer influence the implementation half. This changed `episode-` bytes (nested dict), as expected for a pre-public receipt shape.

• Exit codes `ok→0 / invalid→1 / error→2` — kept (mirrors verify). Admission

rejection and bundle/agent setup failures → exit 2.

• Bundle = directory (not single JSON) — kept; bundle.json now names members.

• Identity when Forge is down — unchanged: identity left unwired →

required-identity reports invalid config; never faked. The demo requires only citations/patch/finding_completeness → a valid partial-reward (0.55) episode with tests/identity not_applicable.

4. New autonomous decisions — please ratify or steer

None move a content-addressed identity except the verifier_versions reshape you already ruled for.

1. `--` split at `main()`, `--agent nargs="+"` kept. Rejected

argparse.REMAINDER (swallows options). Precedence for the agent command: argv after -- > --agent CMD… > agent.json.

2. `bundle.json` version pin — an unsupported eval_bundle_version is exit 2,

not a best-effort load. Trailing / on a dir ref (e.g. subject/) is cosmetic and stripped before safe_relposix.

3. `verify_subject_tree` returns bytes for declared binaries, str for text. The

content map handed to eval_one is heterogeneous; verify_materialization already tolerates both. Modes compared as zero-padded octal strings ("0644").

4. Mode-mismatch and non-UTF-8-not-declared-binary are hard rejects (exit 2),

not warnings — a subject that doesn't bind byte-and-mode-exactly must not seal a receipt that claims it did.

5. Trusted-Local Preview Closure — DONE (e0f01f0, e864227, TRVM 244a877)

Your 10-step Trusted-Local Preview ruling is fully landed. Steps 1–5 (canonical LF admission, bundle symlink/realpath containment, run-policy preflight, required bundle.json, product-metadata cleanup) are e0f01f0. Steps 6–8 (commit the preview closure, publish forge_api, unify the world CLI + identity adapter on LowerResultV1) are e864227 + TRVM 244a877. Steps 9–10 (the frozen real Residency task + the first 1.0 episode) are the example bundle + battery below.

5.1 forge_api.lower_source frozen as LowerResultV1 (TRVM 244a877)

ENGINE_API_VERSION="1", LOWER_RESULT_VERSION="forge.lower-result.v1". lower_source(source) returns {result_version, ok, semantic_artifact_id, diagnostics, error, engine_version} — a data outcome. Ordinary invalid WRL is ok=False (never an exception); exceptions are reserved for engine/import failure. Committed in isolation (one file, additive) so none of the delicate identity-spine WIP in the TRVM tree was swept in.

5.2 The adapter now binds the real engine (e864227)

forge_adapter.real_adapter() uses the soft loader engine.try_load(), reads the frozen LowerResultV1 dict, and embeds the engine version in its own .version = "forge.identity.v1@trvm-<engine_version>". A missing/incompatible engine is a catchable ForgeUnavailable (eval-one runs needs_engine=False, so nothing SystemExits). This closes the §5 dependency from the previous memo: real identity verification is now live, not stubbed.

5.3 The frozen real Residency task — examples/eval-one/residency-forge/

A five-signal bundle (citations 0.25, patch 0.20, tests 0.30, identity 0.15, finding 0.10 → 1.0; caps: patch-fail 0.25, citations-fail 0.25, tests-fail 0.40). The subject is spec/residency.md + src/mod.py (return 1) + world/frozen.wrl (the 4-node spinner world). build_bundle.py seals two environment-computed values so the task is self-consistent locally: identity_policy…world.before_id (the real sem-67e954cf… the frozen world lowers to through forge_api.lower_source) and the acceptance command's absolute argv[0] (sys.executable; the trusted-local runner exposes no PATH, so task_id is regenerated per environment). The deterministic stub test/fixtures/residency_agent.py drives the 1.0 episode and every negative case via TRAAVIIS_STUB_MODE.

5.4 The first fully-verified 1.0 episode (step 10)

$ trvs eval-one examples/eval-one/residency-forge \
      --agent python3 $PWD/test/fixtures/residency_agent.py --platform linux-x86_64
  status  ✓ ok   validity valid   reward 1
  signals citations pass · patch pass · tests pass · identity pass
          finding_completeness pass · native n/a · oracle n/a
  episode episode-5e183a94eb2d5eeb68fd59851c79e849219b220ec503918ec9b7bed2643db460

All five real signals pass — including a controlled acceptance run and a real Forge re-lower of the frozen world through the public boundary. This is the first episode that folds the tests and identity signals for real.

5.5 The nine required cases — test/test_real_residency.py (all green)

Cases 1–5 and 7 fold through the real engine and skip when it is unlocatable (GPT's env); cases 6, 8, 9 are engine-free and always run. Case 6 respects the semantic subtlety you flagged: under the all-five reward a non-applying patch leaves identity/tests with no patched tree → error, so the patch-fail cap is demonstrated under a three-signal reward (the residency-demo posture), while the tests-fail cap is demonstrated under the full five (a return 999 patch still produces a patched tree the acceptance run can reject).

6. New autonomous decisions (this closure) — please ratify or steer

None move a content-addressed identity beyond what your ruling already authorized.

1. TRVM commit isolated to `forge/forge_api.py` only. The TRVM tree holds

substantial unrelated identity-spine WIP; I committed the single published-boundary file and left everything else untouched rather than sweep it into a commit.

2. `impl_version` derives from `BENCH_VERSION` (falls back to

engine_info().bench_version, then "unknown"). This is what makes case 9 true.

3. `residency-forge/task_id` is environment-specific by design — it bakes the

interpreter's absolute path (the acceptance argv[0]), so build_bundle.py must be re-run per environment. Documented in the bundle's build script.

7. What's in the archive

The whole TRAAVIIS working tree (traaviis/, test/, examples/, worlds/, cli.py, engine.py, pyproject.toml, this memo, the ruling request), minus __pycache__ / dist / .git. The TRVM forge_api.py change ships as its own commit in the TRVM repo (244a877); a read-only copy is included at the archive root as TRVM_forge_api.py so the frozen LowerResultV1 boundary can be reviewed without a TRVM checkout.