ComputeDriven benchmark review and Fable assignment

Review date: 2026-09-17 Reviewed repository: /home/travis/ProjectAmp2/computedriven Reviewed commit: 8603bd5c1681d4ba611b556033fbc5a125467f54 Coding owner: Claude Fable. Codex reviews contracts, commits, controls, and claims; Codex does not edit benchmark source.

Executive verdict

The benchmark program is unusually disciplined and is now good enough to produce a defensible first competitive result, but it has not produced that result yet.

The strongest achievement is not a speed number. It is that the harness has repeatedly caught its own false claims: boot latency mistaken for scheduler latency, post-deadline work, incomparable counter domains, refusal counted as starvation, a start after the measurement window called “in flight,” and quadratic analysis. Those repairs make the future result much more credible.

The current evidence supports these claims:

1. The five adapters implement the same bounded semantic state machine and reproduce the oracle on controlled traces.

2. The “progress over utilization” thesis is measurable: aggregate throughput can look excellent while some Loci make no progress.

3. CLOCK-AGE can reduce movement and preserve progress bounds relative to RR-FAIR inside ComputeDriven.

4. The stage-0.2 window, replay, conservation, counter-domain, and start-boundary controls now hold independently.

The current evidence does not support these claims:

• ComputeDriven is faster than C.

• ComputeDriven is faster or better than Tokio.

• ComputeDriven is a better actor runtime than BEAM/Elixir.

• The production cd-kernellet has the performance shown by computedriven-policy.

• Any shared-host smoke number is a competitive result.

The realistic near-term bet is not “beat optimized C at one transition.” It is: complete the same useful workload with fewer migrations, copies, queue operations, and starvation, while retaining bounded semantic progress. A raw transition can approach C; a system can beat a C-based scheduler by proving that less physical work is necessary.

What the current smoke suggests, without treating it as evidence

The retained stage-0.2 smoke is diagnostic only, but it identifies the likely strengths and weaknesses worth testing officially.

The smoke therefore does not reveal one winner. It reveals a plausible Pareto shape: ComputeDriven may be strong at bounded progress, intermittent readiness, and avoiding unnecessary movement; weak at peak closed-loop throughput, skewed open-loop tails, and policy overhead.

Independent acceptance performed in this review

On clean commit 8603bd5:

• tools/fcb_battery.py --selftest: OK.

• git diff --check: clean.

• Quick C1–C9 plus start-boundary controls: CONTROLS HELD across Linux, Go, BEAM, computedriven-policy, and Seastar.

• The rerun was correctness-only on a shared host. Its performance counts are non-evidentiary.

Decision: accept stage 0.2 as the correctness basis for the frozen five-row Tier A battery. Continue to hold the official run until the standing dedicated-host procedure and fixed sentinel reference are satisfied.

Kernellet-cost contract decision: revision 2 is accepted for implementation, with the amendments below. It correctly separates production-kernellet transition cost from scheduler outcomes and from full actor/runtime claims.

Weakness and bottleneck map

Benchmark/evidence bottlenecks

1. No official dedicated-host five-system Tier A result exists.

2. The production kernellet is not in the scheduler-outcome comparison; the current row is a Rust policy adapter.

3. Tier B matched-residency adapters are missing for BEAM and Seastar.

4. Energy is named but unavailable under current RAPL permissions.

5. Linux kernel scheduler work is outside the user-only instruction domain, so one instruction frontier cannot honestly include every row.

6. Single-carrier tests say little about multicore scaling, stealing, NUMA, cross-core wakes, or coordination cost.

7. The harness lacks Tokio, an optimized C event-loop baseline, and a minimal custom Rust executor baseline.

8. The scheduler benchmark does not test I/O readiness, timers, cancellation, blocking interference, or shutdown—the work Tokio actually exists to do.

9. The BEAM row tests scheduling shape, not the actor platform: supervision, links/monitors, selective receive, crash containment, mailbox growth, distribution, and upgrades are absent.

10. No outside machine has reproduced the competitive harness.

Likely implementation bottlenecks

1. Home/policy work can dominate the very small semantic transition.

2. CLOCK-AGE selection and bookkeeping may spend more instructions than the movement it avoids at small populations.

3. Source-to-adapter transport and queueing dominate sparse/open-loop traces and can make source lateness reject rows.

4. Skewed demand produces weaker tails and some mailbox refusal in the policy adapter.

5. One-thread execution prevents evaluating the surface planner and carrier placement thesis.

6. computedriven-policy duplicates kernellet step shape instead of executing the production image.

7. There is no complete actor lifecycle above the floor with measurable supervision/recovery semantics.

MVP bottlenecks

The research runtime does not reach MVP merely by winning a microbenchmark. The MVP needs one useful workload that exercises the distinctive promise end to end:

• create and address many Loci;

• submit useful work with bounded admission;

• observe per-Locus progress and starvation;

• interrupt or crash part of the workload;

• recover, refuse, or replay correctly;

• expose the result through the product without hand-operated repair;

• retain state across restart;

• show a measurable user benefit such as lower tail latency, less memory/movement, or less intervention.

Until this vertical witness exists, benchmark work is research validation rather than MVP completion.

The claim ladder

Claims must be made only at the level their campaign measures.

1. Transition claim: production kernellet vs optimized C/Rust/Go/Erlang class-S loop. Metrics: user-domain instructions, cycles, cache traffic, footprint. No scheduler claim.

2. Scheduler claim: equal traces and semantics on one carrier. Metrics: completed semantic progress, p50/p99/p99.9 ready-to-progress latency, worst-Locus delay, refusals, movement, memory, physical cost. No actor-platform claim.

3. Async-runtime claim: Tokio/current-thread and multithread under wake, timer, I/O, cancellation, blocking, and overload workloads. No supervision/distribution claim.

4. Actor claim: BEAM/OTP actor behavior including mailbox and failure lifecycle. “Better” means a Pareto result across performance, bounded progress, isolation, recovery, and footprint—not just messages/second.

5. MVP claim: one real workflow is faster, more reliable, or requires less operator intervention than its baseline.

Fable implementation program

Fable owns all source changes. Each phase is a separate clean commit and stops for Codex review. Do not combine campaigns into one score or one “winner” sentence.

Phase 1 — Freeze and execute the existing Tier A battery

No feature coding. Preserve commit 8603bd5 as the five-system baseline.

1. Run the full controls and repository guards once from the exact clean commit.

2. On the dedicated host procedure, calibrate the fixed sentinel reference.

3. Run the prespecified five-repeat Tier A battery with bracketing sentinels.

4. Retain every raw B and 2B run and both audits.

5. Publish medians, bootstrap intervals, RSD, rejected blocks, Pareto frontiers, and raw provenance.

6. Stop on any sentinel, isolation, source-lateness, replay, conservation, counter-edge, or deadline failure. Do not search for a favorable window.

This phase answers only: how the five existing one-carrier scheduler shapes compare under the frozen traces.

Phase 2 — Production kernellet versus optimized C

Implement docs/fcb-kernellet-cost-contract.md revision 2. This is the direct response to “are we faster than C?”

Required order:

1. Build the fixed class-S vector file and independent expected outputs.

2. Build the production-kernellet vector driver through the real ABI.

3. Build C, Rust/cd-core, Go, and Erlang comparators.

4. Prove the skip-copy-out and skip-generation-check mutants are rejected.

5. Build the K/2K/4K runner and linearity gate.

6. Add footprint measurement and exact toolchain/build provenance.

7. Run only after Codex reviews the code and a dedicated-host execution is authorized.

Contract amendments:

• Pin and record the exact compiler and runtime binaries by path and hash, not version string alone.

• Add an optimized C sensitivity row at -O3 -march=native beside the portable -O2 row. Neither silently replaces the other.

• Add a “digest disabled” diagnostic only to estimate measurement tax; it is never an admissible competitor because partial work would be invisible.

• Report effect sizes and raw K/2K/4K points even when the marginal is admitted.

• Do not call a lower static image size a lower live-system footprint.

Success is not required to mean fewer instructions than C. A valuable result could be: within a bounded factor of C per transition while the scheduler campaign completes useful demand with much less movement and bounded starvation.

Phase 3 — Tokio and executor campaign

Create a new campaign; do not silently add Tokio to the already frozen five-row receipt.

Subjects:

• Tokio current-thread runtime.

• Tokio multi-thread runtime with one worker (separates scheduler shape from OS threading).

• Tokio multi-thread runtime at 2, 4, and available physical-core counts.

• A minimal Rust executor control using the standard Wake/Waker interface.

• computedriven-policy on one carrier, then the real multi-carrier path only when that path exists.

All rows must execute the same semantic transition and common digest. Add workloads:

1. local spawn → one poll → complete;

2. external wake and local wake;

3. self-wake hot task beside cold tasks;

4. fan-out/fan-in;

5. timer storm with latency measured from intended expiry;

6. socketpair or loopback readiness with coordinated-omission protection;

7. cancellation before start, while queued, and while in progress;

8. one blocking/non-yielding task beside bounded tasks;

9. bounded queue overload and recovery;

10. scale sweep across tasks, runnable fraction, payload size, and carriers.

Primary metrics remain semantic progress and its distribution. Tokio runtime metrics—poll counts, queue depths, forced yields, busy duration, parks, and steals—are diagnostics used to explain results, not substitute outcomes.

Required controls:

• a future that never returns Pending exposes cooperative blocking;

• a lost-wake mutant is detected by conservation and zero progress;

• a spurious wake changes poll count but not semantic progress;

• cancellation cannot commit after its cancellation boundary;

• timer latency is measured from intended expiry, not observation;

• task count and maximum poll duration are bounded for any fairness claim.

Phase 4 — Actor model campaign against BEAM/OTP

Do not claim “better than Elixir” from scheduler throughput. Build the minimum comparable actor lifecycle first.

Common actor semantics:

• spawn/terminate;

• stable address or identity;

• ordered per-sender messages;

• bounded mailbox with an explicit refusal/backpressure outcome;

• request/reply correlation;

• monitor/link equivalent;

• parent/supervisor restart policy;

• state recovery from a committed checkpoint or replay record;

• a visible dead-letter/refusal path;

• deterministic digest of committed actor state.

Workloads:

1. ping-pong and unidirectional message rate;

2. N-actor ring, while explicitly noting that only one actor may be runnable;

3. fan-out/fan-in;

4. hot sender plus cold actors;

5. selective-receive scan or an honestly different ComputeDriven primitive;

6. bounded-mailbox overload;

7. actor crash and supervised restart;

8. crash storm with unrelated actors continuing;

9. large-message copy/share boundary;

10. node/process restart and state recovery, initially single-machine.

Metrics:

• useful messages or completed requests per second;

• ready-to-progress p50/p99/p99.9 and maximum;

• zero-progress actors and minimum progress per actor;

• mailbox depth, refusals, and recovery time;

• spawn/terminate cost and live bytes per idle/active actor;

• bytes copied per committed message;

• unrelated-actor latency during crash/restart;

• lost, duplicated, and out-of-order committed messages (must be zero under the declared contract);

• operator intervention required.

BEAM receives its native strengths: one scheduler and normal multicore configurations, reductions, per-process heaps, links/monitors, and OTP supervision where that is the compared feature. ComputeDriven may win with stronger admission, lineage, or replay, but it cannot omit recovery behavior and still claim a better actor model.

Phase 5 — Matched residency and multicore

Finish Tier B for BEAM and Seastar before making a cross-system residency claim. Add Tokio only after the Tier A Tokio controls hold.

Then evaluate:

• hot-state capacities 1, 8, 32, and all-resident;

• uniform, skewed, waiting/re-ready, and phase-changing working sets;

• 1/2/4/N carriers;

• cross-core wakes, migrations, steals, bytes moved, and cache/NUMA effects;

• scaling efficiency and the point at which policy coordination costs exceed saved movement.

The key graph is a Pareto frontier over semantic progress, worst-Locus delay, and physical cost at each carrier count—not utilization.

Phase 6 — MVP progress benchmark

Choose one real workflow from Super/WEK/WRL/TRVM and freeze an external baseline. The benchmark must include setup, execution, interruption, recovery, and a user-visible result.

Measure:

• wall time to a correct accepted result;

• p99 time from intent to visible progress;

• failures refused before corruption;

• successful recovery rate and recovery time;

• bytes moved/persisted and peak memory;

• manual interventions and operator minutes;

• restart/replay determinism;

• second-machine reproduction.

This phase decides MVP readiness. A runtime microbenchmark cannot substitute for it.

Priority order

1. Execute the frozen official Tier A five-row battery.

2. Implement and review the kernellet-vs-C campaign.

3. Build the MVP vertical witness.

4. Add the Tokio campaign.

5. Build actor lifecycle semantics, then compare with BEAM/OTP.

6. Finish matched residency and multicore scaling.

7. Add energy only when permissions and attribution are defensible.

The MVP witness is intentionally ahead of a broad actor shootout. Beating BEAM is a multi-year platform claim; demonstrating one useful workload with bounded progress and recoverability is a near-term product claim.

Fable stop conditions

Stop and return evidence to Codex when any of these occurs:

• a comparator cannot reproduce the common ledger/digest;

• an apparently fair comparison requires hiding a native runtime service;

• a counter domain differs and cannot be made comparable;

• source lateness, coordinated omission, or measurement overhead dominates a row;

• the benchmark adds semantics for one row that another row is not allowed to use;

• a mutation survives the control intended to catch it;

• a campaign starts producing product code instead of its declared benchmark;

• a result tempts a broader claim than the subject actually measured.

What a good outcome looks like

Within the next bounded cycle, the project should have:

1. one official five-system scheduler-outcome receipt;

2. one real-kernellet transition-cost receipt against optimized C;

3. one MVP workflow receipt;

4. a reviewed Tokio contract and adapter controls, even if the full run comes later.

That would answer the immediate questions honestly:

• Faster than C? Answered separately for transition cost and for whole-workload physical cost.

• Better than Tokio? Answered for specified wake/I/O/timer/cancellation/fairness workloads, not by one throughput number.

• Better actor model than Elixir? Not until lifecycle and failure semantics exist; early results can still show a superior bounded-progress or backpressure property.

• Closer to MVP? Yes only if the real vertical workflow becomes repeatable, recoverable, and externally reproducible.