One outsider. One real repository task. The atlas and whatever AI assistant they already use. No new code, no Runefort, no Academy.
This is the cheapest step in ACADEMY.md and the only one that can invalidate the Practice layer. It currently sits behind work that exists to build the thing it might invalidate, which is why it was pulled into its own lane.
Not "do you like Academy". Not satisfaction. One question:
Can a competent outsider complete a real task in this codebase using only the atlas and an assistant they already have?
And if not — what specifically was missing at the moment they stopped?
Both answers are useful and neither is a failure:
| Outcome | What it means | What Practice becomes |
|---|---|---|
| They finish, without much friction | The scarce thing was never a workspace | Practice is a governed execution and evidence surface around the tools they already use — observe, constrain, record, verify. Not an IDE |
| They finish, but the trace shows repeated thrash in one place | That specific deficit is the product | Practice targets it directly, and the tiled floor may or may not be the right shape for it |
| They do not finish | Something structural is missing | B2 names it. Do not guess |
Do not run this hoping for a particular answer. The version where the outsider succeeds easily is the one that most changes the architecture, and it is a good outcome.
The task selection is the part most likely to invalidate the result, so it gets criteria rather than judgement in the moment. A usable task satisfies all five:
Verifiable completion. A test passes, a value changes, a command exits zero. Not "they
understood it" — something a third party can check without interpretation.
Genuinely requires understanding the repo. The answer depends on how this system is put
together, not on general programming knowledge.
Not solvable by grep alone. If a single well-chosen search finds it, it measures search,
not comprehension.
Not blocked by documentation that is merely wrong. A task that fails because a doc lies is
a documentation bug, not a finding about Practice. Fix the doc and pick another task.
Not dependent on private context. Nothing that requires knowing a decision made in a chat,
a preference never written down, or the author's memory.
Two more constraints on scope:
Sized 30–90 minutes for someone who knows the stack. Shorter measures nothing; longer
measures endurance and their calendar.
Real. Take it from the actual backlog. A task invented to be pedagogically neat will
produce a pedagogically neat result that predicts nothing.
Offered as starting points; each still needs checking against the five criteria before use. They differ sharply in information value, so for a first B1 prefer the disagreement-axis task or a pre-screened raw-fallback defect: both have unambiguous completion criteria while still forcing the participant to build a model of the system.
| Candidate | Verifiable by | Watch out for |
|---|---|---|
Rule three documents in the review backlog by adding visibility: front matter, and make the build report the backlog shrink | build output: 24 unruled decreases | Too signposted for a first run. The build already names the backlog, so it mostly tests whether someone can follow an exposed mechanism |
Add a sixth axis to disagree.py and make the self-test cover it | python3 disagree.py --self-test passes with the new case | Best first candidate. Requires understanding why the axes are kept separate, how alignment works, and how the self-test encodes the invariant |
Find why a chosen doc renders as raw fallback instead of parsed BendScript, and fix it | the build's fell back to raw view count drops | Good if pre-screened. Depends entirely on which doc; confirm first that it is a real defect and not a documentation bug, or criterion 4 is violated |
Make @runefort/core build a dist/ that satisfies the README quickstart | the quickstart HTML loads without a 404 | Largest of the four; may exceed 90 minutes |
The trace matters more than the outcome. Record all of it — the participant will not be able to articulate what was missing, but the trace will show it.
Where they searched. The atlas, GitHub, the filesystem, or straight to the assistant.
What they asked the assistant, verbatim. Their first question is the strongest single
signal of what the docs failed to make obvious.
Every place they hesitated for more than ~30 seconds, and what they did next.
What evidence changed their next action — the moment something they read altered their
plan. These moments are what Read exists to produce; if none occur, Read is not yet doing its job.
**Every point where they asked you something.** Each one is a deficit the system should have
covered, and each one contaminates the run — note it and answer minimally.
Wall-clock to completion, or the point and reason they stopped.
Screen recording plus a timestamped note file is enough. No instrumentation to build.
Give them the atlas URL and the task statement. Nothing else — no walkthrough, no orientation.
Before they touch anything, ask one question and write down the answer verbatim:
"Based on the task and the atlas, where do you think you'll need to look first?"
Then say nothing. Do not correct it, do not hint, do not react.
This is the baseline the whole run is measured against. Without it you can see that they found the answer but not what changed their mind — and the difference matters more than the outcome does. If their first guess was wrong and reading the atlas corrected it, Read is working. If their first guess was wrong and the assistant corrected it while the atlas went unread, then the documentation did none of the explanatory work and Academy Read is solving a problem somebody else already solved.
Say explicitly: using any AI assistant is expected, not cheating. The comparison is against
how they would really work.
Do not help unless they are fully stuck. When you do, record it (§2).
Stop at 90 minutes regardless. An unfinished run is a result, not a failed session.
n=1 is the correct size for this. It is a falsification test, not a study. If one competent outsider sails through, the "people need a workspace" premise is already in trouble and no larger sample is needed to justify re-examining it.
Two questions, in this order, immediately after:
"At the point you got stuck, what did you need to know?" — a named deficit, not a feeling.
"Where did you expect to find it?" — this is the one that locates the fix. If they expected
it somewhere the atlas does not go, that is a Read problem; if they expected it in a place that would have to observe the running system, that is a Practice problem, and it is the strongest available evidence for the governed-evidence-surface reading.
Do not ask whether they liked it. Do not ask what features they want.
Append the outcome to ACADEMY.md §7 (what would falsify this) — including, and especially, if the result argues against the current design. The whole point of running B1 before building Practice is that the answer is allowed to change what gets built.
State the result in the same evidence classes used everywhere else in this project: measured (what the trace shows), inferred (what it suggests), unmeasured (what one run cannot tell you).