<!-- HEADER:AUTO — everything between the HEADER:AUTO markers is REGENERATED by scripts/make-e-review-bundle.sh from a live test run and from this file's own section headings. Do not hand-edit it; edit the generator. FINDING N70. --> <!-- HEADER:AUTO:BEGIN --> Reviewed: E_COMPUTEDRIVEN_OPUS_REVIEW_BRIEF.md Rounds: 1 (review) · 2 (hardening) · 3 (implementation hardening) · 4 (the platform boundary, and idempotency across time) · 5 (acceptance atomicity, and deployed-configuration truth) · 6 (canonical batch identity, and fanout truth) · 7 (canonical identity, and serialized acceptance) · 8 (acceptance evidence, and the outbox lifecycle) · 9 (harness truth, and event identity) · 10 (the vendor-repair race, and the fakes' own gaps) · 11 (the acceptance status, and configuration truth) · 12 (the deployment description catches up with the code) · 13 (falsifying the deployment procedure itself) · 14 (staging bootstrap closure) · 15 (bootstrap evidence closure) Latest: round 15 — bootstrap evidence closure Status: Phase 0 complete. Nothing is deployed. No Cloudflare resource has ever been created. Battery: 500 falsifiers, 0 failing — cd prototype && npm run test:all, which runs the suite at today's date AND five years on. Hermetic against the network (N33) and against the clock (N66) — precisely: INVARIANT UNDER A FIVE-YEAR EPOCH SHIFT, which is narrower than "nothing reads ambient time" and is the honest size of the claim (N72). This bundle will not build unless both runs are green. Header generated: 2026-08-24 by scripts/make-e-review-bundle.sh. <!-- HEADER:AUTO:END -->
THIS HEADER HAS NOW GONE STALE TWICE, IN THE DOCUMENT WHOSE SUBJECT IS CLAIM DISCIPLINE. After round 4 it said "150 falsifiers" while 187 were passing, and the round-4 reviewer caught it. The paragraph that replaced it said the number "is now derived by the bundle script from a live run, which is the only form of it that cannot go stale" — and then hand-typed 288 twice more, so round 8 shipped a bundle whose Round-8 banner sat above a header claiming rounds 2–7. The round-8 reviewer caught that one (N70). Stating the fix is not applying it. The block above is now generated: the round list is read from this file's own
# N.headings and the battery count from the run the generator refuses to proceed without. There is no number in this header that a human typed. Round 2 is §5. Round 3 is §6. Round 4 is §7. Round 5 is §8. Round 6 is §9. Round 7 is §10. Round 8 is §11. Round 9 is §12. Round 3 answers findings Travis raised by reading the implementation rather than the findings document — which is why they are subtler and why one of them (N13) retracts a justification round 2 wrote down as fact. Earlier rounds' text is left as written except where it was WRONG; C6 (round 1) and N13's premise (round 2) are corrected in place and say so.
The brief is good. Its instincts are right in the places that usually go wrong: it isolates the analytics failure domain from the product one, it refuses to call a two-hour-old snapshot "live", it puts the ack after the insert, and it says "measure it" wherever it quotes a price. The findings below are mostly about the gap between what it specifies and what would run.
Everything marked FIXED is fixed in `prototype/` and has a runnable falsifier. cd prototype && npm test → see the generated header above for the live count, and npm run test:all runs the same battery at two different dates (N66).
E is not in the revision register. It appears in no entry of REVISION_REGISTER.md and in
no line of STACK_SITEMAP.md, STACK_PLANNING.md or CLOUD_V1.md (grepped 2026-08-23). §38 calls itself a "Final architecture ruling", but in this tree a ruling is an R-number. E needs one, particularly because of (2).
ClickHouse Cloud contradicts R4. R4 ruled *"Cloudflare for everything; Fly keeps only
Keycloak"* (2026-08-15, Travis). ClickHouse Cloud on AWS us-east-1 is a third provider. This may well be the right call — analytics genuinely is a different failure domain — but §3 asked for exactly this kind of conflict to be reported before it is "fixed", and the brief never mentions R4.
No site in the portfolio has a privacy policy, and none has any analytics. Turning on the
first /_e/* route makes that every site's problem on the same day. See C4.
Measured 2026-08-23.
| Question (§36) | Answer | ||||||||
|---|---|---|---|---|---|---|---|---|---|
Q2 · Does e.computedriven.com exist? | No. dig returns NXDOMAIN — no DNS record of any kind. The directory e.computedriven.com/ contained the brief and nothing else. | ||||||||
| Q1 · What code overlaps this system? | None. No analytics vendor appears anywhere: grepping the tree for `posthog\ | amplitude\ | gtag\ | googletagmanager\ | plausible\ | cloudflareinsights\ | matomo\ | mixpanel\ | segment` across all HTML/JS/TS returns zero hits. E is not replacing anything. |
| Q7 · What consent mechanism should E reuse? | There is none to reuse. No consent banner, no consent API, no Sec-GPC or doNotTrack handling, and no privacy page anywhere in the portfolio (find -iname 'privacy*.html' → empty). | ||||||||
| Q3 · First instrumentation target? | `computedriven.com` — it is live (200, 164,656 bytes, byte-identical to the local index.html), it is a single static page, and it is the zone E's own subdomain would sit under. cloud.computedriven.com is NXDOMAIN and undeployed; the rest are separate repos. | ||||||||
Q4 · Can all owned domains expose /_e/*? | Structurally yes, operationally no — not yet. All portfolio sites are static HTML on Cloudflare. But each *.com/ directory is its own git repo (agents/SITES.md §0.1), so this is ~21 repos to touch, not one. | ||||||||
Q5 · Where should e-schema / e-sdk live? | Not in a workspace — there isn't one. The root package.json has no workspaces key; it is an orchestration script file that shells into subdirectories. Nine separate package-lock.json files exist. The convention here is sibling directories with their own package.json, which is what prototype/ follows. | ||||||||
| Q6 · Which existing IDs are safe analytics identifiers? | principals.id and organizations.id from the Cloud control plane — R10 makes them provider-independent roots, with external IdP bindings isolated in principal_identities. That is exactly the opaque-and-stable property §18 needs. Note they do not exist in a deployed database yet. | ||||||||
| Q8 · dev/staging/prod separation? | Nothing exists. The only wrangler configs in the tree are worker/ (declared, never deployed), workbench/, toolboxhvac.com/, and an archived one. There is no CI/deploy pipeline for Workers at all — sites deploy by git push to Pages. | ||||||||
| Q13/Q14 · active minutes, does the homepage wake ClickHouse? | Unanswerable without a deployment. Both are Phase 5/6 measurements. Flagged so they are not quietly treated as answered. |
Convention note for whoever builds this: worker/wrangler.toml establishes the house style for undeployed infrastructure — declare the bindings so the architecture is reviewable, and make every placeholder id deliberately implausible so it cannot be copied into a real deploy. The prototype's two wrangler files follow it.
Severity: BLOCKER = the system cannot run as specified · HIGH = silent incorrectness · MED = a gap that will cost a round · LOW = worth knowing.
event_id is a ULID in §8 and a UUID in §12 — FIXED§8's envelope shows "event_id": "01J..." (a ULID: Crockford base32, 26 chars). §12 declares event_id UUID. toUUID('01JABC…') throws — the first insert fails, and nothing before that point reveals it. batch_id has the same conflict between §13 and §12.
Resolved as UUIDv7, which keeps every property the ULID was chosen for (client-generated, time-sortable, no coordination) while fitting the UUID column at 16 bytes instead of 26. The alternative — keep ULIDs, declare FixedString(26) — is equally coherent and costs 10 bytes/row. This is a decision to confirm, not one I should settle. See packages/e-schema/ids.mjs.
This is the largest defect, and it is structural rather than a typo.
§14 mandates pull → insert → confirmed → ack, which is correct — the alternative loses data. That ordering deliberately creates a window in which a worker inserts, dies, and its message is redelivered. §28 test 16 then requires: "Worker crash after DB insert but before ACK is safe/idempotent."
Nothing in the brief deduplicates anything. §12 specifies ENGINE = MergeTree, which has no deduplication. insert_deduplication_token appears nowhere in the document.
Two further facts, neither of which the brief states:
On ClickHouse Cloud, ENGINE = MergeTree is silently substituted with SharedMergeTree,
which is replicated and therefore can deduplicate. That substitution is the only reason test 16 could ever pass — and it still requires the token to be set, which nothing does.
The same DDL anywhere else — a local container, CI, self-hosted — gets a plain unreplicated
MergeTree with no deduplication at all. So the test would pass in production and fail locally, for a reason invisible in the schema.
Resolved in packages/e-clickhouse/insert.mjs, and it takes three parts, all required:
a deterministic dedup token derived from batch_id (which §13 already guarantees is stable);
deterministic row order before blocking — ClickHouse dedups per block, and Cloudflare
Queues does not guarantee order, so a redelivery that reorders rows would split them into different blocks and dedup would silently miss. (The Cloud lane already paid for assuming queue order once — R41.)
a bounded block count, because the dedup window is finite and overrunning it makes dedup
stop working with no error anywhere. (Round 1 wrote "default 1000 blocks" here as if it were a fact. It was an assumption, and round 2's N4 replaced it with declare-then-verify — see §5.2.)
Falsifiers in test/flush.test.mjs, including §28.16-DEFECT, which runs the crash scenario against the brief's design as written and asserts that it does double-count — six rows for three events, invisible to a DISTINCT count.
PARTITION BY on a client-controlled timestamp is a queue-poisoning DoS — FIXED§8 makes occurred_at a browser field. §12 partitions the table by it. A client sending events spread across 230 years produces ~2,800 partitions from one batch, which exceeds ClickHouse's max_partitions_per_insert_block (default 100) and fails the whole insert — including every valid event in it. Under §14's ack-last rule the poison batch is then never acked, and it redelivers on every flush forever, blocking the pipeline behind it.
Unauthenticated browser input, one request, permanent stall.
Resolved three ways: partition by received_at (our clock), clamp occurred_at at ingest to [received − 7d, received + 1h], and store occurred_at_clamped so a clamp is itself data — a rise in it is either a broken client or an attempt.
CLAUDE.md's standing direction: "R7 has since removed D1 from the architecture: Pigsty owns application state, and D1 holds none." §15 lists D1 among the snapshot-store candidates. Resolved: the prototype uses KV, which is the right shape regardless — one key, written once per flush, read on every page load.
/live route contradicts the brief's own honesty rule§21 creates /live and, four paragraphs later, says "Do not market V1 features as live"; §28 test 39 is the falsifier for a problem introduced on the same page. Suggest /pipeline or /architecture until the Durable Object plane in §23 actually exists.
§17 puts PostHog/Amplitude fanout after the ClickHouse insert — i.e. inside the 2-hour flush. Every vendor event therefore arrives ≥2h late, which breaks PostHog session stitching. Meanwhile §9 says PostHog also receives its own direct high-resolution browser path. So PostHog sees the same user action twice, by two paths, ~2 hours apart. §17's dedup instruction ("use stable E event_id downstream") only works if the direct browser path also carries E's event_id — which the brief never requires and the SDK has no reason to do.
Not resolved in the prototype — this is an architecture decision. The options are: fan out from the ingest worker instead (immediate, but couples acceptance to vendor availability); fan out from a separate queue consumer (a second push consumer that doesn't touch ClickHouse); or accept the delay and drop PostHog's direct path. Recommend the middle one.
consent object does not say what consent.analytics: false means — FIXED§8 puts consent in the envelope and §12 stores it in ClickHouse. The brief never states what happens when it is false. Both readings are bad: reject at ingest (then why is the field stored?), or store it — which persists analytics data for a user who declined analytics, i.e. the precise thing consent exists to prevent.
Resolved: consent gates. analytics !== true is refused at ingest with consent_withheld, nothing is stored, and the SDK defaults consent to denied so an integration that forgets to set it collects nothing rather than everything.
§25 requires "a deletion path by opaque identity" and "retention policy configuration". §12's DDL has neither lightweight deletes nor a TTL, so erasure is an ALTER TABLE … DELETE mutation that rewrites parts. Resolved: allow_experimental_lightweight_delete = 1 and a 25-month TTL on received_at in ddl.sql.
Not in the brief; found while building the prototype, in the prototype's own flush worker. Kept because it is the most dangerous shape in the design.
A Cron Trigger gets 15 min CPU at an interval ≥ 1 hour and 30 seconds below it. The flush worker reserves time before its deadline so an in-flight insert can finish. With a fixed 60 s reserve against a 30 s budget, the deadline lands in the past: the pull loop exits on its first check, reports success, and flushes nothing — on every run, forever. The queue then grows until retention expires and the platform silently discards everything.
The failure is reachable by editing one cron expression for fresher data, which §6 actively invites. And it hides inside its own normal case: ok: true, queue_drained, 0 events is exactly what a healthy idle pipeline looks like.
Resolved: the reserve scales with the budget, and a budget below a floor refuses loudly with budget_too_small rather than doing nothing quietly.
| # | Brief's claim | Verdict |
|---|---|---|
| B1 | §7 · "Queue operations are metered in 64 KB chunks" | Correct. "Each 64 KB of data written, read, or deleted" = one billable operation. The <64 KB target is well-founded. |
| B2 | §5 · "Pull batch size supports up to 100 messages" | Correct — matches the documented max consumer batch size. |
| B3 | §5 · "Paid retention configurable up to 14 days" | **Correct, but it is configured, not default.** A queue left at its default sits inside §6's CRITICAL band (8h) rather than far outside it. Phase 3's "paid retention configured" is load-bearing; keep it. |
| B4 | §13 · "hard max bytes" | Never given a number. The wall is 128 KB per message. The prototype rejects at 96 KB at ingest, leaving room for the fields the worker adds, so our own additions can never be what breaches the platform limit. |
| B5 | §5 · why a push consumer can't buffer 2h | The argument is right but not the strongest one. §5 reasons from delivery delay. The decisive fact is that max_batch_timeout caps at 60 seconds — a push consumer cannot be configured to buffer for two hours at all. Worth stating, because it turns a judgement call into a constraint. |
| B6 | Cron CPU budget | Not mentioned anywhere. 15 min at ≥1h interval, 30 s below. See A9. |
| B7 | §14 · "until time budget reached" | Budget never named. 15 min wall clock; visibility timeout max 12h; consumer wall clock 15 min. All compatible with the 2h design — worth writing down since §14 depends on a number it doesn't state. |
| B8 | §12 · ENGINE = MergeTree | ClickHouse Cloud does substitute SharedMergeTree, and it is replicated. Load-bearing for A2, and unstated. |
| B9 | §10/§16 · ClickHouse Cloud Basic prices | Could not verify — clickhouse.com/pricing is client-rendered and returns no figures to a fetch. Not disputed; not confirmed either. §16's own instruction ("Do not promise these numbers. Measure them.") is the right discipline — keep it and re-check first-party before any published claim. |
C1 · HIGH · R4 conflict. See §0.2 above.
C2 · MED · E is in no planning doc. See §0.1 above.
C3 · MED · No DNS. e.computedriven.com is NXDOMAIN.
C4 · HIGH · No privacy policy exists anywhere in the portfolio, and no consent mechanism.
§25 says "privacy documentation based on implementation" — correct, and it means the first /_e/* route cannot ship until at least one privacy page exists. Because §20's pattern rolls the same endpoint across ~21 sites, this becomes every site's obligation at once. This is the practical gate on Phase 2, and the brief does not identify it as one.
C5 · HIGH · §20's anti-spoofing rule sits beside a schema that contradicts it — FIXED.
§20 says derive site_id from the route; §8's envelope still carries it as a client field. A rule in prose next to a schema that disagrees is a rule that gets lost. Resolved structurally: the client's value is not validated, it is overwritten, and the SDK does not send it at all.
C6 · MED · No privacy-signal handling.
CORRECTED 2026-08-23 (round 2, N9). This line originally read "No `Sec-GPC` / `doNotTrack` handling — FIXED". Only Sec-GPC was implemented. Claiming two and shipping one is the exact failure this register exists to catch, and it was caught on review rather than here. They are also not interchangeable: GPC is a deliberate, legally meaningful opt-out, while DNT is deprecated and was historically sent on-by-default — which is why the industry stopped honouring it and why GPC exists at all. Now: both are honoured, as a stated policy (HONOUR_SEC_GPC / HONOUR_DNT in limits.mjs), because E's payload is entirely optional analytics and there is no product function lost by over-honouring. The response names which signal opted the request out.
C7 · MED · The §19 journey token is replayable. No algorithm named, no key rotation, no
jti. A signed token with only iat/exp can be replayed inside its window to link an unrelated visitor into someone else's journey — which falsifies §28 test 32. Not resolved; needs a single-use marker or a binding to the destination session. Flagged for GPT.
C8 · MED · No ingest response contract — FIXED. No status code, CORS policy, or body was
specified. The prototype returns 202 with {accepted, rejected}, a restrictive origin allowlist on the fallback endpoint only, and no-store.
C9 · LOW · §30's cost dashboard needs inputs §15 forbids it from fetching. The cost panel's
numbers must be computed at flush time and written into the snapshot; nothing says who does it.
C10 · LOW · §3's numbered list is `1. 2. 3. 3. 3. 3. …` — fifteen items, thirteen numbered 3.
Renders fine; the source is wrong, and it makes the questions impossible to cite by number.
prototype/ — zero dependencies, node --test, 55 passing falsifiers.
packages/e-schema/ limits.mjs (every platform number, CITED vs CHOSEN, with dates)
ids.mjs · vocabulary.mjs · envelope.mjs (validate/redact/clamp)
packages/e-sdk/ sdk.mjs — microbatcher, consent gating, identity lifecycle
packages/e-clickhouse/ ddl.sql (4 corrections, annotated) · insert.mjs (the A2 closure)
workers/cd-e-ingest/ edge acceptance · site_id derived from route
workers/cd-e-flush/ the 2h drain · insert→confirm→ack · A9 guard
test/ fakes.mjs + 3 suites, mapped to §28's numbered falsifiersThe design rule throughout: every platform touch point is injected, so the acceptance path and the drain algorithm are testable without a Cloudflare account. test/fakes.mjs models a queue that redelivers, reorders and expires leases, and a ClickHouse that deduplicates and can be constructed not to — which is how A2 is demonstrated rather than asserted.
Run it:
cd e.computedriven.com/prototype && npm testNo `e.computedriven.com` website. §34 Phase 1 asks for a marketing shell; it needs the nav
(amp-nav.js, owned exclusively by lane N per agents/SITES.md §0.2, and being edited by another session as I write this), a DNS decision, and an R-number. Doing it now would collide.
No deploy. No wrangler deploy, no account resources, no ClickHouse service. Both wrangler
files follow worker/'s convention: real binding shapes, deliberately implausible ids.
A6 (vendor fanout) and C7 (token replay) are analysed, not fixed — both are architecture
decisions rather than defects with an obvious repair.
Travis reviewed the round-1 prototype, independently reran the battery, and ruled: keep Workers + Queues + ClickHouse; one more adversarial round before deploying anything. What follows is that round.
| # | Decision |
|---|---|
| 1 | ClickHouse Cloud is a narrow, explicit exception to R4 — not a repeal. E is an isolated analytical data plane with no product or control-plane authority. Needs its own revision-register entry; see §5.5. |
| 2 | UUIDv7 approved. 16-byte native ClickHouse UUID, client-generatable, time-ordered. A1 is closed. |
| 3 | Privacy is a deployment gate, rolled out one site at a time. computedriven.com first; observe; then expand. Do not touch ~21 repos at once. |
| 4 | A2/A3 approved in principle — deterministic dedup token, deterministic row ordering, received_at partitioning, occurred_at clamped and the clamp recorded. |
| 5 | A6 resolved with a second, immediate vendor queue. See §5.3. |
| 6 | C7 (cross-domain journey tokens) deferred. When built: single-use jti, bound to destination/session, backed by a Durable Object rather than KV because one-time redemption wants strong consistency. |
| # | Finding | Status |
|---|---|---|
| N1 | The ingest acknowledgment boundary. ctx.waitUntil(queue.send()) is non-blocking and its send errors are implicitly ignored, so round 1's 202 meant only "we parsed it". The browser then discarded its batch on a promise that was never made. | FIXED — await both sends before answering; a failure returns 503 naming the batch_id to retry with. |
| N2 | Batch identity was owned by the wrong side. The SDK emptied its buffer before transport succeeded, and the Worker minted a fresh batch_id per HTTP request — while ClickHouse dedup keys on batch_id. A browser retry therefore produced a batch identical in every field except the one dedup depends on, with stable event_ids that nothing was comparing. | FIXED — the SDK generates batch_id once, holds the batch until confirmed, and retries the same ids; the Worker validates and preserves, and refuses a malformed one rather than replacing it. Pending buffer is bounded and drops are counted. |
| N3 | The browser was authoritative for authenticated identity. Round 1's envelope accepted user_id/account_id straight from an unauthenticated client. | FIXED — browser is authoritative for anonymous_id/session_id/journey_id only. user_id/account_id come from a signed identity assertion minted by the auth boundary and verified at ingest. A client that declares them is rejected, not stripped — stripping leaves a misconfigured integration shipping data that looks exactly like a site with no logged-in users. Mirrors R21 exactly. |
| N4 | The dedup window was an unverified assumption. Round 1 hard-coded 1000 blocks "the default"; ClickHouse's current guidance says 100. Correctness must not rest on either. | FIXED — declare (conservatively, at 100) → read system.merge_tree_settings → guard with min(declared, actual) → publish it in every flush snapshot. An unreadable setting is recorded, not silently assumed. |
| N5 | `allow_experimental_lightweight_delete = 1` is stale, and the 25-month TTL was being canonised as a technical constant. | FIXED — flag removed. TTL kept but marked UNRULED: it is a privacy/product decision, and it must be the same number the privacy page publishes. |
| N6 | Cron edge case. The deadline was checked once per pull, then up to 100 messages ran sequentially — a pull starting just inside the deadline could overrun the invocation window. | FIXED — deadline checked per message, with bounded concurrency (default 8). |
| N7 | Benchmark the concurrency. | DONE — bench/concurrency.mjs sweeps 1/4/8/16/32 and reports active-minutes per million events, the unit the cost goal is in. Refuses to be quoted without --live. |
| N8 | Vendor dedup keys. | DONE, and PostHog is not what was assumed — see §5.4. |
| N9 | Claim discipline: "Sec-GPC / doNotTrack — FIXED" when only GPC shipped. | FIXED — C6 corrected in place; both signals now honoured as a declared policy, and the response names which one fired. |
| N10 | Attribution values were unbounded. A query-string key allowlist says nothing about values: [email protected] and a megabyte of garbage both passed round 1 untouched. source_domain is LowCardinality(String) — a type whose whole argument is a small distinct set. | FIXED — hostname normalisation, length caps on UTM/referrer/path, and value-level scrubbing of emails, JWTs and key-shaped runs. |
┌─► e-events-clickhouse
│ PULL consumer · ~2h batches → ClickHouse sleeps
Browser ─► cd-e-ingest ────┤
└─► e-events-vendors
PUSH consumer · ~10s → PostHog · AmplitudeOne active consumer per queue, which is also why this cannot be expressed as two consumers on one queue. cd-e-ingest writes to both and awaits both before 202. A partial failure returns 503 and the browser retries — which is only safe because of N2: the same batch_id dedups the ClickHouse leg and the same event_id dedups the vendor leg. Round 1 could not have offered this.
Checked against PostHog's own documentation rather than assumed:
Dedup is on the four-tuple (uuid, event name, timestamp, distinct_id) — all four must
match. A retry that regenerates the timestamp does not deduplicate, which is why the SDK freezes occurred_at at track() time and nothing downstream rewrites it.
It is eventual — it happens during background ClickHouse merges, so duplicates are visible
for a while regardless.
PostHog states it is not guaranteed, "due to how merges work in ClickHouse".
Amplitude is the clean case: insert_id is a documented dedup key, and E's immutable event_id maps straight onto it.
So: send all four members stably, and treat PostHog dedup as hygiene, not a guarantee. No E correctness claim rests on it. This is what brief §1's "PostHog and Amplitude are downstream analytical lenses, not sources of truth" costs in practice — and it is worth knowing before someone reconciles two dashboards and concludes the pipeline is broken.
REVISION_REGISTER.md was at R43 when round 1 began and R74 four hours later — two sessions are appending to it live, and my memory records ruling numbers colliding with a parallel session before. So the entry is written here rather than pasted into a table another session is actively extending. Claim the next free number at paste time; do not trust the one below.
| R75 | **E is an isolated analytical data plane, and a narrow exception to R4.** ClickHouse Cloud is admitted as a third provider for E ONLY. E holds no product or control-plane authority: it may not be read by authorization, entitlement, billing or World logic, and the product plane must never require a read from it to function. R4 stands unrepealed everywhere else — "Cloudflare for everything; Fly keeps only Keycloak" — and this entry is the exception that proves it is still being enforced rather than quietly eroded. The reason for the exception is the reason ClickHouse was chosen: an independent scale and failure domain, and learning the analytics stack while mistakes are cheap | 2026-08-23 | Travis |
Both were introduced by the fix for N10 — the change that exists to handle hostile input.
N11 · HIGH · ReDoS in the redactor. The first draft's email pattern used an unbounded local part ([A-Z0-9._%+-]+@). A long run containing no @ makes the engine match the whole run, fail, and backtrack one character at a time — O(n²). One megabyte of A is ~10¹² steps, inside the function whose entire job is processing hostile browser input, on a Worker with a 30-second CPU limit. It hung the test runner, which is how it was found: a test written for something else. Fixed two ways, since either alone is thin — quantifiers are bounded, and redactValue truncates before it scans.
N12 · MED · The redactor ate the data it exists to protect. The first draft redacted any 32+ character run of [A-Za-z0-9_-]. That class includes hyphens, and summer-sale-2026-north-america-launch is 37 characters — so the redactor would have silently destroyed the attribution values E exists to collect, in the field most likely to carry them. Fixed with an explicit entropy test (mixed case and a digit) instead of length alone. The tradeoff is now stated in the code rather than implied: a credential that happens to be all-lowercase with no digits still gets through, and that is the conservative side of a judgement, not an oversight.
Both are the same shape as A9 in round 1 — the defect was in the thing built to prevent the previous defect. That is now three rounds running, and it is worth naming as a pattern rather than noticing it again next time.
C7 — cross-domain journey token replay. Deferred by decision; not needed to instrument one site.
§10/§16 ClickHouse prices — still unverified; the pricing page is client-rendered and returns
no figures to a fetch. Do not publish a cost claim from them.
§36 Q12/Q13 — does Basic actually scale to zero under this workload, and what active-minutes
per day does a 2h cadence produce. Unanswerable without spending money; they are the first things the deployment should measure.
The TTL (N5) and the `INSERT_CONCURRENCY` default (N7) are both placeholders awaiting a
real decision and a real measurement respectively.
Travis read the round-2 prototype rather than its findings, reran the battery independently (107/107), and raised eight further findings. The architecture was reaffirmed: keep Workers + Queues + ClickHouse. One of these findings retracts a justification round 2 wrote down as fact.
| # | Finding | Status |
|---|---|---|
| N13 | The insert strategy was fighting ClickHouse, on a premise that was false. Round 2 set async_insert: 0 and justified it: "it would ack to us before the data is durable". Verified false — wait_for_async_insert defaults to 1, and ClickHouse "only acknowledges the insert after the data is successfully flushed to disk". Round 2 disabled a correct mechanism on a misunderstanding and then recorded the misunderstanding as a reason, which is worse than the setting. The shape mattered too: browser batches cap at ~25 events and the flush inserts one message at a time, so ROWS_PER_BLOCK = 10_000 never split a single block — every insert was ≤25 rows, and thousands of tiny parts per flush create merge pressure that keeps a supposedly sleeping service awake. | FIXED — async_insert: 1, wait_for_async_insert: 1, set explicitly. Plus a version floor: dedup applies to async inserts only from a recent enough server, so supportsAsyncInsertDedup() is checked each flush and published. Without it, the fix for N13 would have silently reintroduced A2. |
| N14 | **There was no session model, only a session id.** The SDK minted anonymous_id and session_id in its constructor with no persistence, so every document was a new visitor and a new session — on a portfolio of static multi-page sites, where every funnel spans documents. | FIXED — first-party persistence (after consent), session_id is a UUIDv7 minted at session start, 30-minute inactivity gap, 24-hour ceiling, throttled heartbeat so a synchronous localStorage write does not land on every click. |
| N15 | Identity was read at SEND time, not frozen at SEAL time — with a reproduced witness. User A's batch fails and stays pending; A logs out, B logs in; the retry goes up under B's assertion. A subtler variant: identify() created no batch boundary, so pre-login and post-login events shared one batch and ingest stamped authenticated identity onto events that were anonymous when they happened. | FIXED — the assertion is captured into the immutable sealed batch and never re-read; identify() seals first. And the deeper repair: E no longer rewrites history at all — see §6.2. |
| N16 | `sendBeacon` quietly reopened N1's hole. Round 2 established that 202 means both queue writes succeeded, then used sendBeacon on pagehide and treated its true as confirmation, deleting the batch. sendBeacon returning true means only that the user agent queued the data. | FIXED — fetch(..., {keepalive: true}), which also survives unload and exposes the response, so only a real 2xx confirms. Sealed batches are persisted and replayed by the next document with the same ids; if the first send actually landed, the replay is harmless because of A2 and N8. |
| N17 | `views UInt64` in an `AggregatingMergeTree` does not aggregate. The engine combines rows by merging aggregate states; a plain integer carries no instruction for how. Two inserts touching the same (site_id, hour, path) — the normal case — left whichever value won, not the sum. Pageviews would have been under-reported forever, plausibly, with no error. | FIXED — views SimpleAggregateFunction(sum, UInt64); sessions keeps the full AggregateFunction(uniq) because distinct-counting cannot be summed. signups_daily was already correct for a reason now written down, so nobody "fixes" it into the same shape: SummingMergeTree sums plain columns by definition. |
| N18 | The queue lease was shorter than the drain it protects. A 10-minute visibility timeout against a 15-minute wall clock, with the worker draining to ~minute 14: a message pulled at minute 2 became eligible for redelivery while the invocation holding it was still running. Dedup protects correctness, so this is waste rather than corruption — but it is waste in exactly the window the design exists to keep short, and it makes duplicate metrics noisy enough to hide a real problem. | FIXED — VISIBILITY_TIMEOUT_MS = 25 min, with assertLeaseOutlastsDrain() refusing before pulling anything if the invariant is violated. |
| N19 | Scrubbing before URL-decoding is a trivial bypass. ?utm_campaign=travis%40example.com survived round 2 untouched while the literal form was redacted — the parser scrubbed the encoded text, in which there is no @ to find. | FIXED — decode (bounded, repeated until stable, so %2540 does not buy another layer) then redact. Also `journey_id` is now shape-bounded: C7 is open, nothing verifies issuance, so V1 accepts only our own jny_ format and drops anything else rather than admitting arbitrary strings into the column. |
| N20 | Vendor session/identity was wrong in both vendors. Amplitude got hashToInt(session_id) — a valid integer that meant nothing, since Amplitude defines session_id as the session start time in ms. PostHog got e_session_id, a property it has never heard of, so E's events took no part in session aggregation or replay correlation. And $set_once does not establish PostHog's identity relationship. | FIXED — one canonical UUIDv7 session id, two correct derivations: PostHog $session_id = the UUID; Amplitude session_id = the ms timestamp read out of it. identity_linked now emits a PostHog `$identify` with `$anon_distinct_id`, ordered before the event it derives from. |
`session_id` is now validated as a UUIDv7 at ingest, not merely produced as one by the SDK. If the vendor mapping requires the shape, accepting anything else means knowingly shipping broken data to two vendors.
An anonymous pricing_viewed stays anonymous forever. That is what actually happened, and backfilling user_id onto it corrupts the epistemic state of a historical event to make a join easier.
anon_abc ── identity_linked ──► principal_xyzThe link is its own fact, with its own timestamp and its own linked_anonymous_id. Analytics can say "this earlier anonymous event belongs to the journey that later became principal_xyz" without claiming we knew who they were at the time.
This is also the shape the eventual Super projection needs, since a projection has to be able to state what was known when — and it is the same distinction the Cloud lane draws in R27 between a presented and an authenticated claim.
The defect keeps being in the thing built to prevent the previous defect.
A9 — the budget guard whose fixed reserve exceeded a 30-second budget.
N11/N12 — a ReDoS in the redactor, and a redactor that ate the attribution data it protects.
N13 — turning on async_insert would have silently un-done A2 on an older server. That
is why it ships with a version floor rather than a comment.
N16 — sendBeacon reopening the durable-acceptance hole N1 had just closed, one layer
down, in the same round.
Every round has now found at least one of these, and three of the four were found by the battery rather than by reading. That is the argument for the battery, and it is the reason to keep writing falsifiers for the fix and not only for the original bug.
C7 cross-domain journey tokens (deferred by ruling; journey_id is shape-bounded until then)
§10/§16 ClickHouse prices — unverified; the pricing page is client-rendered
Retention TTL — a privacy decision that must equal the number the privacy page publishes
`INSERT_CONCURRENCY` — needs npm run bench:live
Round 3 is the end of what fakes can decide. These need the first staging deployment:
async-insert + dedup behaviour on SharedMergeTree, at the real service version
the materialized views, including the N17 fix under multi-insert merges
cold-start latency and actual scale-to-zero behaviour
insert concurrency (bench:live)
part counts and merge pressure per 2-hour flush
compressed bytes per event
active minutes per day -> the cost model's only real inputThe prototype is instrumented for every one of them: the flush snapshot already publishes dedup_window, dedup_window_source, clickhouse_version, async_insert_dedup_supported, insert_concurrency, visibility_timeout_ms, block counts and durations — so the first staging run produces measurements rather than requiring new code to produce them.
Travis relayed a review of the round-3 prototype which independently reran the battery (150/150) and raised eight findings, retracting round 3's claim that "fakes are finished." That retraction was correct. Two of the eight are demonstrable on this machine for nothing, and both would otherwise have been discovered by paying for a staging deployment.
Verifying the eight produced four more (N29–N32) and one correction to the review itself (N26). Every one of the four is the same shape round 3 named and then walked into again.
| # | Finding | Status |
|---|---|---|
| N21 | The first real Cloudflare pull would have thrown, on every run, forever. The adapter read typeof m.body === 'string' ? JSON.parse(m.body) : m.body, which reads as defensive and handles neither case. Cloudflare documents the pull body as base64-encoded for the json and bytes content types, and json is the default for any Worker after the 2024-03-18 compatibility date — which E is. So m.body is a base64 string for every message, JSON.parse throws SyntaxError on the first character, the flush records pull_failed, and the queue fills to retention while the dashboard says the pipeline is merely unhealthy. Round 3 also discarded `attempts`, `id`, `timestamp_ms` and `metadata` — and attempts is the only input N22's repair needs. | FIXED — packages/e-schema/queue-wire.mjs decodes per metadata['CF-Content-Type'], refuses v8 by name, and preserves the whole envelope. |
| N22 | "A crash after a confirmed insert is safe because it dedupes" was true only for a retry that arrives SOON. Round 3's guard asked will this batch alone emit more blocks than the window holds? — always "no" at ≤25 events per message. The question that decides correctness is will this batch's token still be remembered when Cloudflare redelivers? The dedup log is a rolling global history; a token is evicted when either bound passes. Cloudflare is explicitly at-least-once. | FIXED — two parts, neither sufficient alone: N31 bounds the redelivery horizon in time, and an existence probe gated on attempts > 1 asks the table rather than trusting Keeper. The Keeper window is now an optimisation, not a proof obligation. |
| N23 | The prerequisites were checked and then ignored. Round 3 read the window, read the version, appended an error string when the version was too old — and drained anyway. That is observability after corruption. | FIXED — the drain refuses before the first pull if the version is not confirmed at or above the floor, if the window could not be read, or if N31's horizon invariant fails. A pull is the irreversible step; the Queue is what makes failing closed affordable. |
| N24 | A 15-minute credential frozen into a 24-hour retry. N15 froze the identity assertion into the sealed batch; N16 persisted sealed batches for 24 h. Composed: an authenticated batch that waits 20 minutes replays with an assertion that expired 5 minutes ago, 401s, and repeats until it is dropped. The batch cannot be re-signed — that is what N15 forbids. | FIXED — the TTL now covers the pending horizon, and the credential is re-described as what it is: an analytics attestation with no product authority, carrying scp: e-analytics. See §7.3 on why the prose about key custody was also wrong. |
| N25 | Withdrawal erased the copy and kept the original. Round 3 cleared the queue, pending batches and storage — everything written down — and left anonymous_id, session_id, identity_assertion and journey_id live in memory. Deny → re-grant in one document recovered the same visitor. The SPA case (a banner toggled without navigation) is the normal one. | FIXED — withdrawal is an identity boundary, like logout(). New ids are minted immediately and stay memory-only until analytics is permitted again. |
| N26 | The review argued the floor should be 26.1. Keeping 26.2 — and the reasoning matters more than the number. See §7.2. | CORRECTED, floor unchanged |
| N27 | One vendor's outage duplicated into the other. Ack-if-both-succeeded on a shared queue meant an Amplitude outage retried the message and re-sent to PostHog every time — and round 2's own N8 establishes that Amplitude's insert_id makes its retries clean while PostHog's dedup is a four-tuple, eventual, and documented as not guaranteed. The duplicates landed on the vendor least able to absorb them, caused by the outage of the vendor best able to. | FIXED — e-events-posthog and e-events-amplitude, one consumer each, batch.queue selects the vendor. Retries are isolated per destination. |
| N28 | A flag shipped `true` inside a comment arguing for `false`. Round 3 wrote "nothing verifies issuance, this is not a security control, C7 is deferred" and shipped E_JOURNEY_IDS_ENABLED = true. Separately, properties was bounded by size but not by schema, so any event could carry any 64 key names into a JSON column that materialises discovered paths into physical subcolumns. | FIXED — journey ids off until C7 issues one; a per-event property registry beside vocabulary.mjs. Unknown keys are dropped and counted, not rejected: refusing the event to punish one key loses the funnel to protect the column. |
| N29 | E never turned deduplication on. Round 3 set async_insert: 1 and passed a dedup token, and stopped — because dedup "is the default". It is the default for synchronous inserts. ClickHouse's 26.2 changelog: "Deduplication is now enabled by default for all inserts. Previously it was off for async inserts and materialized views, but on for sync inserts." Below 26.2 the token was accepted and ignored. This is N4's own lesson — never hard-code a provider default — violated inside the fix for N13: round 3 read the window from the server and assumed the switch. | FIXED — deduplicate_insert plus the legacy pair plus deduplicate_blocks_in_dependent_materialized_views, stated explicitly. |
| N30 | N4's protocol was reading a setting that does not govern E's inserts. ClickHouse keeps separate dedup bookkeeping for async inserts. Round 3 read replicated_deduplication_window — the sync one — while every insert is async. The number in the dashboard was real, was read from the server, and described a mechanism this pipeline does not use. A setting name is not a type, so nothing connected N13's change to N4's reader. | FIXED — both families are read, and the active one is selected by async_insert rather than by whichever name got typed first. The snapshot publishes which family it read. |
| N31 | Three rounds counted blocks; nobody read the other bound. replicated_deduplication_window_seconds defaults to 3600 — one hour, against a two-hour flush cadence. The async variant is 604 800 (7 days). Had async_insert ever been reverted — which N13's own error message offers as the remedy for an old server — every token would have expired an hour before the flush that could have used it. The suggested fallback was a silent-duplication switch. | FIXED — queue retention < dedup seconds is asserted before the first pull. Retention is declared at 4 days, not the 14-day maximum: 14 would be double the dedup memory, and "use the maximum" is the obvious safe-looking choice that breaks it. Note the direction reverses — for blocks the conservative declaration is smaller, for seconds it is a larger horizon and a smaller assumed memory. |
| N32 | The `batch_id` column was never populated. Found by writing the falsifier for N22's fix. events_v1 declares batch_id UUID and three rounds of prose are narrated around it — but every one of those statements is about the token, which is passed as an insert setting. normalizeEvent() emits no batch_id; the flush inserts batch.events verbatim; the column took the all-zeros default on every row, silently, because nothing reads it. What it breaks is every question the table can be asked about a batch — N22's probe first of all, which would have returned 0 for a batch demonstrably present and inserted it again. A fix for silent duplication that silently duplicates. | FIXED — stamped in insertBatchIdempotent, which already takes batchId for the token, so a caller cannot insert a batch without saying which batch it is. Plus a bloom-filter index, since the probe is a point lookup on a column unrelated to the sort key. |
The round-3 fixture is why N32 survived. aRow() minted a random batch_id per row, so every row looked correctly stamped and no test could see that nothing in the pipeline stamps it. The fixture now omits the field, exactly as normalizeEvent() does.
The review argued the floor should be 26.1, citing ClickHouse's 26.1 release note: deduplication "now works end-to-end for asynchronous inserts and their dependent materialized views."
That note is accurate and the conclusion does not follow, because capability and default are different events:
26.1 the capability lands — async+MV dedup is made to WORK
26.2 the default flips — "enabled by default for all inserts. Previously it
was off for async inserts and materialized views"
26.3 async inserts become enabled by defaultLowering the floor to 26.1 would have admitted servers where the mechanism exists and is switched off — which is A2, re-opened by the fix for N26. Since round 3 set no dedup switch at all (N29), 26.1 would have deduplicated nothing.
With N29's settings now explicit, 26.1 would in fact behave. The floor stays at 26.2 anyway, because a floor that depends on remembering to send three legacy setting names is a floor resting on a comment — and E targets ClickHouse Cloud, which is past both.
This is worth recording plainly: the pattern reached the reviewer. A2 was nearly re-opened by the correction to a finding about A2. That is now the sixth instance.
Round 2 wrote: "the signing key belongs to the auth boundary, and only the ingest Worker holds the verification key." With HMAC-SHA256 that sentence describes something that does not exist. The verification key is the signing key, so an ingest Worker able to verify an assertion is equally able to mint one for any principal on any site. There is no issuer/verifier split — only a shared secret in two places.
That is an acceptable V1 position (ingest and auth are the same trust domain) and it must be stated as what it is. Claiming a separation the cryptography does not provide is N9's defect — writing "FIXED" for something half-shipped — in a place where the consequence is an authority bug.
The real boundary is now scope: scp: e-analytics, checked at verify. A product API must refuse a token carrying it; E refuses one without it. Asymmetric signing is the upgrade that would make the custody claim true, and the verify path already takes the key as a parameter.
The defect keeps being in the thing built to prevent the previous defect.
A9 — the budget guard whose fixed reserve exceeded a 30-second budget
N11/N12 — a ReDoS in the redactor, and a redactor that ate the attribution data it protects
N13 — async_insert would have silently un-done A2 on an older server
N16 — sendBeacon reopening the hole N1 had just closed, in the same round
N29/N30 — N13's change moved which setting governs; N4's read-don't-assume protocol kept reading the old one, and nobody turned the mechanism on at all
N32 — found by writing the falsifier for N22's fix, and it would have made that fix duplicate
N26 — the proposed correction to a finding about A2 would have re-opened A2
Four of the seven were found by the battery rather than by reading. Write the falsifier for the fix. Round 4 did, and it produced N32.
No website, no DNS, no ClickHouse account, no deploy. The review's framing was right: close the provider wire contract and cross-time idempotency locally first, so staging money is spent measuring rather than on bugs already provable here.
Unchanged from §6.4, plus:
C7 cross-domain journey tokens — now the reason journey_id is off, not merely bounded
§10/§16 ClickHouse prices — still unverified; the pricing page is client-rendered
Retention TTL — a privacy decision that must equal the number the privacy page publishes
`INSERT_CONCURRENCY` — needs npm run bench:live
The `batch_id` bloom filter's granularity — 1 is a guess biased toward the point lookup; it is a real index with a real cost and staging should measure it
N24's TTL is a judgement, not a measurement. A day-long analytics attestation is defensible only because it authorises nothing. If E ever gains an authority, this number is the first thing to revisit.
§36 Q12/Q13 — scale-to-zero and active-minutes still need a deployment
Unchanged from §6.5, with two additions the round-4 findings created:
that `deduplicate_insert` is accepted (not silently ignored) at the service version
the ACTUAL replicated_deduplication_window_*_for_async_inserts on ClickHouse Cloud
-- the 10000/604800 defaults are documented; Cloud may override either
the batch_id bloom filter's real selectivity under the probe
async-insert + dedup behaviour on SharedMergeTree, at the real service version
the materialized views, including the N17 fix under multi-insert merges
cold-start latency and actual scale-to-zero behaviour
insert concurrency (bench:live)
part counts and merge pressure per 2-hour flush
compressed bytes per event
active minutes per day -> the cost model's only real inputThe flush snapshot publishes every field above that a run can produce, so the first staging run measures rather than requiring new code.
The round-4 review reran the battery and got 186/187, against a saved log claiming 187/187, and caught the §1 header still saying 150. Both were real: one non-hermetic test, and a stale count in the document about claim discipline. It then raised nine findings, all nine of which reproduce.
| # | Finding | Status |
|---|---|---|
| N33 | The battery stopped being hermetic. test/round4.test.mjs called the real vendor Worker, which called deliver() with its default fetchImpl = fetch — so an ACK/retry assertion made a live request to https://us.i.posthog.com/batch/ with the fake key 'ph'. It passed here 3/3 and failed on the reviewer's machine 3/3, from identical code. The seam already existed one layer down and the Worker did not thread it through: a seam below the layer under test is not a seam. | FIXED — env.FETCH through the Worker, and test/no-net.mjs replaces the global fetch with a throwing stub for the whole battery via --import. Not a convention; enforced. |
| N34 | BLOCKER. The probe could not run under the credentials this design specifies. The DDL grants e_ingest INSERT + SELECT(received_at), and ClickHouse SELECT privileges are column-scoped. SELECT count() ... WHERE batch_id = ? is a permission error under exactly those grants — which round 4 caught, turned into null, and inserted through. The mechanism was present in every test and absent in production, the worst of the two arrangements. | FIXED — GRANT SELECT(batch_id, received_at), still two non-PII columns, and assertProbeCapability() runs before the first pull so the grant cannot silently regress. |
| N35 | BLOCKER. `attempts > 1` is not "this batch is a duplicate." It answers did Cloudflare redeliver this message, a strict subset. The ingest Worker's own partial fanout produced the counterexample: canonical send OK, Amplitude send fails, 503, browser retries the same batch_id — a second message in the already-successful canonical queue, both with `attempts = 1`. Reproduced: A rows = 2, probe calls = 0. | FIXED — presence is asked for every pulled microbatch, as one grouped query per pull against the bloom filter round 4 added. Idempotency no longer depends on where the duplicate came from. |
| N36 | The cumulative window guard raced. Round 4 checked blocks_cumulative and incremented it after the await, so at INSERT_CONCURRENCY = 8 all eight workers passed one budget. Reproduced against the shipped code: window 2, blocks_cumulative 9, errors: [] — it reported the right reason for stopping while having written 4.5× the budget it enforces. Its falsifier passed because it was written with concurrency: 1, the one configuration under which the guard holds. | FIXED — reserved synchronously, no suspension point between check and reserve; the reservation is returned on failure. |
| N37 | N4's protocol was applied to one side of the invariant only. queue retention < dedup seconds read the right-hand side from the server and supplied the left from a constant — and nothing in the repository sets that value on the queue either. 4 days is Cloudflare's current default, so it was right by luck. The maximum is 14 days, double the 7-day dedup memory, one dashboard field away. | FIXED — readQueueRetention() reads the deployed value; an unreadable retention refuses the drain rather than falling back; the snapshot records which key answered. |
| N38 | N24's fix did not compose either. The TTL runs from mint, the pending horizon from seal. A tab open 23 hours seals a batch with ~1 hour of usable retry against a 24-hour window. Round 4 turned "always broken" into "usually fine" — harder to notice, no more correct, and the same defect one layer along. | FIXED — the invariant is now exp − sealed_at ≥ PENDING_MAX_AGE_MS, checked at seal. An assertion that cannot survive the horizon is not frozen; the batch seals anonymously and the drop is counted. The event is never lost — only the identity label, and E does not rewrite history anyway (N15). |
| N39 | A missing vendor secret was treated as successful delivery and permanently ACKed. {ok: true, skipped: true} was defensible when one queue fanned out to two vendors. With N27's dedicated queues it became a data-loss path: one forgotten `wrangler secret put` silently discards that vendor's entire stream while every invocation looks healthy. | FIXED — a queue that delivered a message requires its vendor to be configured; otherwise retry() and a logged error. A deliberate disable means not producing to the queue at all. |
| N40 | The two vendor queues were not configured alike. TOML array-of-tables: keys belong to the entry above them, so PostHog inherited platform defaults and had no DLQ — Cloudflare retries three times and then permanently deletes. N27 exists to give the vendors independent failure domains and gave them different data-loss behaviour. Staging had no DLQ on either. A visual diff reads as symmetric; the semantics are not. | FIXED — both consumers configured explicitly, separate -dlq queues per vendor (a shared one destroys N27's "queue name identifies one vendor" invariant), staging included. |
| N41 | The size limit was bypassable and the fallback unbounded. There is an early Content-Length check — the review's "applied after the entire request has already been materialized" is slightly stronger than the code — but ?? '0' means an absent header passes, and Content-Length is absent under chunked encoding. Past that gate, await request.text() materialises whatever arrives, against a 100 MB platform ceiling and 128 MB of Worker memory. | FIXED — readBounded() counts bytes off the stream, stops one byte past the limit, and cancels. |
202 Accepted means — the architectural changeThe review's last point was the most important and it is right.
Round 4 awaited all three queue writes and returned 503 if any failed, treating the fanout as atomic. It is not, and pretending it is put the cost on the wrong leg:
ClickHouse queue OK
PostHog queue OK
Amplitude queue FAILS
↓
503 -> the browser resubmits CANONICAL history
↓
a second message in the already-successful canonical queue [this IS N35]
and a second delivery to PostHog, whose dedup is eventual
and documented as NOT GUARANTEED [N8]So a transient Amplitude enqueue problem manufactured a canonical duplicate and a visible vendor duplicate, in the vendor least able to absorb one.
202 now means exactly one thing: the canonical event is durably accepted.
This does not weaken N1, it makes N1 precise. N1's finding was that 202 must not mean "we parsed it"; it still must mean a durable boundary was crossed. What N1 never answered is which boundary, and the architecture already answers it — ClickHouse is the source of truth, the vendors are lenses, and a lens can be rebuilt from the source while the source cannot be rebuilt from a lens. A canonical failure is still 503 with retry_with_same_batch_id. A vendor enqueue failure is reported in the 202 body as vendor_enqueue_failed and reconciled downstream — never by asking the browser to resend history it has already been told was accepted.
Rounds 1–4: the defect is in the thing built to prevent the previous defect. That still holds — N35 is inside N22's fix, N36 inside N22's other fix, N38 inside N24's fix, N39 inside N27's fix.
But round 5's other half is new, and it is worth separating:
rounds 1-4 a guarantee failed when composed with adjacent CODE
round 5 a guarantee failed when composed with adjacent CONFIGURATION
a column grant N34
a queue retention N37
a vendor secret N39
a dead letter queue N40
a Worker's fetch seam N33No amount of reading the source catches those, because the source is not where the fact lives. That is the argument for scripts/check-queue-config.mjs and for assertProbeCapability(): the things that decide correctness must be read from the system that enforces them, on both sides of every invariant that spans two providers. N4 said this about ClickHouse in round 2. It took until round 5 to apply it symmetrically.
And one thing no falsifier could have caught: the header of this document claimed 150 while the battery ran 187. The count is now derived from a live run by the bundle script.
No website, no DNS, no ClickHouse account, no deploy. Strictly N33–N41 plus the 202 change.
C7 cross-domain journey tokens — the reason journey_id is off
§10/§16 ClickHouse prices — unverified; the pricing page is client-rendered
Retention TTL — a privacy decision that must equal the number the privacy page publishes
`INSERT_CONCURRENCY` — needs npm run bench:live
The exact Cloudflare queue-API JSON key for retention. queue-config.mjs tries three
documented spellings and publishes which answered, because there is no account to check against. The first staging run settles it in one line.
The `batch_id` bloom filter's granularity and real selectivity under the always-on probe. N35
made this a per-pull query on every flush rather than a rare one, so its cost is now on the hot path and is unmeasured.
`assertionOutlivesPendingHorizon()` reads an unverified payload. Correct here — the SDK has no
key and N3 forbids it being authoritative — but it means a malformed assertion seals anonymously rather than being rejected. That is the safe direction; it is still a judgement.
The N39 rule assumes a vendor queue implies a configured vendor. Disabling a vendor now means
removing the producer binding and the queue. Nothing enforces that pairing yet.
Unchanged from §7.7, plus what round 5 added:
the real Cloudflare queue-API key and shape for message retention [N37]
that GRANT SELECT(batch_id, received_at) is sufficient for the probe [N34]
the bloom filter's selectivity under a per-pull IN(...) of up to 100 ids [N35]
the cost of that probe in the flush's active-minutes budget [N35]
that `deduplicate_insert` is accepted, not silently ignored [N29]
the ACTUAL replicated_deduplication_window_*_for_async_inserts on Cloud [N30]
async-insert + dedup on SharedMergeTree at the real service version
the materialized views, including N17 under multi-insert merges
cold start / scale-to-zero, insert concurrency, part pressure, compression
active minutes per day -> the cost model's only real inputscripts/check-queue-config.mjs answers the first tranche read-only, without spending anything, the moment credentials exist.
The round-5 review ran the shipped battery independently, got 216/216, agreed the 202 ruling was correct, and then found seven more composition defects by reading the fixes. All seven verified against the shipped code. Verifying them produced three more of our own — one of which (N50) answers the experiment the review asked for, and answers it in the opposite direction to the question.
Battery: 259 falsifiers, 0 failing. 43 new, one per finding plus its witness.
| # | Severity | What it is |
|---|---|---|
| N42 | HIGH | vendor_enqueue_failed was produced, transmitted, and discarded by res.ok; nothing durable ever recorded the gap, and "reconciled from ClickHouse" named no mechanism |
| N43 | HIGH | the fallback route derived the site from a constant, so every site's events were filed under e |
| N44 | MED | a same-pull duplicate reserved a block budget and continued without returning it |
| N45 | HIGH | a row COUNT was standing in for a batch's content identity — same batch_id, different events, second batch ACKed into oblivion |
| N46 | HIGH | identity_linked could seal with its assertion dropped: a canonical fact asserting a link to nobody |
| N47 | MED | attempts was persisted and lastAttemptAt was not, so a reload restored the retry count and erased the backoff |
| N48 | LOW/MED | every rejection message said UUIDv7; every check was isUuid. Stale 202 semantics in both wrangler files |
| N49 | HIGH | ours — and the fallback route destroyed all authenticated traffic, not just misfiled it |
| N50 | HIGH | ours — insert deduplication is scoped per partition, and E's partition key is stamped per attempt |
| N51 | MED | ours — a count mismatch was read as a torn write when it is equally a vocabulary deploy, and that reading halts every later flush |
N43 is that the cross-origin route answered site_id = "e" for every caller. Composed with N3 it is worse than misfiling.
verifyAssertion(token, secret, expectedAudience) binds an assertion to one site, and expectedAudience is the site derived from the route. On the fallback route that was the constant e. So an assertion minted for a person on computedriven.com carries aud: "computedriven", meets a route that has decided the site is e, and fails:
401 bad_identity_assertion / wrong_audienceon every authenticated batch. 401 is not 2xx, so the SDK never confirms — the batch is retried from this document and every later one until the 24-hour horizon drops it. Anonymous traffic was misfiled; logged-in traffic was destroyed, after consuming the entire retry budget.
Two correct mechanisms, composed. N3 binds identity to a site; N43 derives the site from config. Together they reject exactly the traffic identity exists for.
The repair: site identity on that route comes from a server-issued site key in the path (/v1/sites/<site_key>/events), resolved against a server-side map. Origin is a CORS control and touches nothing else. The key is an identifier, not a credential — it ships in browser JavaScript. What it buys over Origin is that we issue it, resolve it, and can revoke it; calling it "authenticated" would be the N9 defect.
The review's proposal was (site_id, batch_id, batch_fingerprint). That is right. The part it did not specify is where the fingerprint is computed, and computing it in the wrong place converts this fix into N51.
Ingest sees the accepted subset, not the batch, and what validation accepts is a function of the deployed vocabulary — §9 invites retiring event names, and N28 built the property registry that makes such changes routine. A fingerprint recomputed at ingest would therefore differ between an original and a retry that straddles a deploy, and the retry would read as an identity conflict.
The browser's sealed set is immutable (N15) and is already what batch_id names, so the fingerprint is minted once, at seal, by the SDK, carried unchanged, and copied — never recomputed — by ingest. FNV-1a 128 rather than SHA-256 because #seal() is synchronous and making it async would put a suspension point between "read the queue" and "empty the queue", which is N36 one layer down.
It is an accident-and-confusion check, not a signature. The question a digest cannot answer — "can I make my batch look like somebody else's?" — is answered by scoping: presence is keyed on site_id too, and site_id comes from the route, so a caller can only ever collide with itself.
The review asked whether the always-on presence probe (N35) has made N31 (retention < dedup seconds) and N36 (the cumulative block budget) redundant as correctness gates rather than diagnostics. Answering it turned up a third bound nobody had read, in ClickHouse's own documentation of the feature:
deduplication "applies per partition: if the two queries write rows to different partitions, both survive."
E partitions by toYYYYMM(received_at) — A3, and correctly, because a browser's clock must not carve the storage layout. But `received_at` is stamped by the ingest Worker per ATTEMPT, so it is a property of the attempt, not of the batch. The pending horizon is twenty-four hours:
23:50, 31 Aug attempt 1 accepted received_at -> partition 202608
00:10, 1 Sep attempt 2 accepted received_at -> partition 202609
same batch_id, same token, DIFFERENT PARTITIONS
-> "both survive"Twelve times a year there is a 24-hour window in which the token is not evictable but inapplicable — and no guard in the flush measures it, because every guard measures when the token expires, not whether it applies.
So the experiment's answer is not "the window gates can relax". It is:
PRIMARY, and at times the ONLY mechanism
the presence probe — queries the TABLE, which has no partition predicate,
so it sees rows wherever they landed
RACE PROTECTION ONLY
the dedup token — free, immediate, and structurally unable to hold
across a partition boundary
WINDOW GATES (N31/N36)
bound the durability of a mechanism that is sometimes not running.
They cannot be the proof of anything.Two falsifiers establish which mechanism holds the property: with the token switched off entirely (dedupWindowSeconds: 0) the probe alone prevents the duplicate; with the probe removed the token alone does not.
THE CONCLUSION IS RECORDED, NOT ENACTED. Demoting N31 from a refusal to a diagnostic trades a hard stop for a metric, and in this tree that is an R-number and Travis's call — not a side effect of a test passing. The gates still refuse, and a falsifier pins that behaviour so it cannot drift while the question is open.
Open for ruling:
Do N31/N36 become diagnostics, with the probe (N34/N35) as the sole hard gate?
Should received_at be frozen at first acceptance and carried on retry, so a batch has one
partition for its whole life? That would restore the token across month boundaries — and it re-opens A3, because the value would then come from a client-held field.
Round 5 named the move from composed with adjacent CODE to composed with adjacent CONFIGURATION. Round 6 is a third thing, and it is the one that generalises furthest:
AN IDENTITY WAS STANDING IN FOR A DIFFERENT IDENTITY, AND THE SUBSTITUTION WAS INVISIBLE BECAUSE BOTH SIDES WERE REAL VALUES.
a row count standing in for a batch's content identity (N45)
a route constant standing in for the site (N43), and therefore for the audience (N49)
an event named identity_linked standing in for a link (N46)
a table-wide token standing in for a per-partition one (N50)
a torn write standing in for a vocabulary deploy (N51)
Each substitution type-checks, reads correctly, and produces a plausible number. None of them fails loudly, and four of the five are silent data defects rather than errors.
And the defect kept being in the thing built to prevent the previous defect, for the seventh round: N44 is N36's reservation missing from one exit of N36's own loop; N45 is the presence proof N35 introduced; N46 is N38's repair; N49 is inside N43's fix; N51 is inside N35's quarantine.
Two defects were found in round 6's own work while writing its falsifiers, which is the ALWAYS WRITE THE FALSIFIER FOR THE FIX rule paying out again:
The new FakeClickHouse probe regex used ([^)]*\)[^)]*) for the tuple list and could not cross
more than two ) characters. It matched a pull of one batch and a pull of two, and silently failed on three or more — a non-matching probe query falls through and returns nothing, which reads downstream as no batch is present. The entire N45/N35 closure would have tested green while being unreachable in every realistic pull.
A ${partition} ${token} template picked up a literal NUL byte in place of the space, so the
fake stored tokens under one separator and looked them up under another. It also made grep silent on those exact lines, which is how it survived two rounds of reading.
Unchanged from §8.4, and now explicitly: no E website, no Super chat, no cross-domain journeys, no portfolio rollout. C7 stays deferred and E_JOURNEY_IDS_ENABLED stays false.
Everything in §8.5, minus what round 6 closed, plus:
The two rulings in §9.4.
`vendor_repair_v1` has a writer and no reader. The repair record is durable and the
reconciliation job that consumes WHERE repaired_at IS NULL does not exist. That is deliberate — the record had to exist before anything could be built on it — but until the job ships, E can say which events a vendor is missing and cannot yet re-deliver them.
The site-key map has no issuance path. E_SITE_KEYS is a secret an operator writes by hand.
Fine for one site; it needs a real registry before "general fallback for external integrations" is a true description.
FNV-1a 128 is not collision-resistant against a motivated caller. Scoping makes that
uninteresting today (a caller can only collide with itself, on its own site). It stops being uninteresting the moment a site key is shared across tenants.
`batch_fingerprint` adds 32 bytes per row and a groupUniqArray to the hot-path probe.
Neither is measured.
Unchanged from §8.6, plus:
the real cost of groupUniqArray(2) + tuple IN over the bloom filter [N45]
whether SharedMergeTree's dedup is partition-scoped in the same way [N50]
— the docs say per partition; ClickHouse Cloud substitutes the engine,
and round 4 already learned that substitution changes the answer
the vendor_repair_v1 write's share of the flush's active minutes [N42]This is the boundary. Round 6 was the last round the fakes can decide — and round 3 said that too, and was wrong, so the correct form is the one §7.7 uses: these specific questions now need a real instance, and the local list is empty for the first time since round 1.
The round-6 review unpacked the bundle, read the implementation rather than the findings, ran the battery independently (259/259), confirmed N49/N50/N51, and disputed one sentence: that the local list was empty. It was not. Four more — and two of them are inside round 6's own repairs.
Battery: 288 falsifiers, 0 failing. 29 new.
| # | Severity | What it is |
|---|---|---|
| N52 | HIGH | the "content fingerprint" hashed only the event IDs, so N45's own defect survived one level down — and round 6's test for it varied the ids, the single field the function read |
| N53 | MED/HIGH | N48 diagnosed event_id and batch_id in writing and then fixed only batch_id |
| N54 | BLOCKER | presence is check-then-act and nothing serialized it across drainers — while N50 had just removed the mechanism silently covering that |
| N55 | HIGH | schema_version: 1 froze nothing; a deploy changed what a shipped version means, and round 6's N51 accommodated the consequence instead of removing the cause |
Round 5's was composed with adjacent configuration. Round 6's was an identity stood in for a different identity. Round 7's is narrower and worse:
THE REPAIR NAMED THE RIGHT PROPERTY AND IMPLEMENTED A WEAKER ONE.
N45 said "a row count is not a content identity" and hashed only the ids
N48 said "event_id and batch_id — the two ids the whole closure keys on" and fixed one
N51 said "the fingerprint is immune to what ingest decides" and used that immunity to **ACK a
divergence between the source and its lenses**
Each comment is an accurate description of a property the code does not have. Reading the code for what it claims confirms it; only running the claim falsifies it. That is why the reviewer's move — read the implementation, not the findings — found four things six rounds of reading findings did not.
Round 7 committed the same defect twice more, both caught by its own battery: schemaFor() replaced a typed Set membership test with a hasOwnProperty lookup and started accepting the string "1"; and a comment about -0 said the opposite of the (correct) line beneath it.
The digest now covers the full sealed events plus the frozen identity assertion, through a deterministic canonicalEncode (recursive key sort, type tags, length prefixes — so distinct structures cannot share an encoding, which plain concatenation allows).
The first draft encoded the events array in its given order, on the reasoning that array order is content. That broke closure 2b — "a reordered redelivery still deduplicates end to end" — and the break was the finding, not a fixture to update. The token derives from the fingerprint and identifies a block; blocks are built by orderDeterministically(), which sorts rows by event_id so a redelivery in a different order produces byte-identical blocks. So:
fingerprint order-SENSITIVE + blocks order-INSENSITIVE
-> two byte-identical block sets carrying DIFFERENT tokens
-> dedup silently missesThe rule the two layers must share:
The fingerprint must be invariant under exactly the transformations the insert path is invariant under — no more, and no less.
The insert path sorts the event list by event_id and touches nothing else. So the fingerprint sorts the event list by event_id and touches nothing else; arrays nested in properties keep their order, because nothing downstream reorders them.
-0 normalises to 0 for a related reason: #restorePending() re-derives a fingerprint from storage that has been through JSON.stringify, where -0 is already "0". Distinguishing them would make a restored batch stop being itself because the page reloaded.
Still FNV-1a 128, still explicitly not a signature. Scoping by site_id means a caller can only collide with itself on its own site — an argument that fails the moment one site key is shared across tenants, which is when SHA-256 becomes required rather than preferable.
EFlushCoordinator, a Durable Object holding one lease: {epoch, holder, acquired_at, expires_at}. Acquired after the dedup prerequisites and before the first pull (N23: a pull is the irreversible step), released on every exit, only by the current epoch. KV was rejected — eventual consistency means both readers can see "no lease" in exactly the contended case.
Cloudflare's warning against a global high-throughput singleton is respected and worth stating precisely: this object sees two requests per flush at a two-hour cadence. No event, no batch and no ClickHouse I/O passes through it, and nothing is held inside blockConcurrencyWhile() across a network call.
What the fence does and does not guarantee. A true fence would have the store reject a stale epoch; ClickHouse has no such facility. So the fence is a check the holder makes on itself (now + margin < expires_at, before each pull and each insert), which bounds the overlap to one in-flight insert rather than to zero. Without a lease the window is the whole drain — up to fifteen minutes of two drainers probing and inserting freely. Within the residual window the dedup token still covers the same-partition case, which is every case except the month boundary. If that residual is ever judged too large, the next step is a claim row in ClickHouse keyed on the batch identity — not a longer margin.
The fence is checked before the deadline. Both stop the drain and both are safe, so the order only decides what the snapshot says — and under the shipped numbers the deadline (14 min) always arrives first, making a second-place fence unreachable in exactly the case it exists for. Reporting time_budget there would hide the only fact an operator needs. The battery caught this: fence_stops: 0.
schema_version was a label, not a version. The rules it named lived in module-level globals, so a deploy changed what {"schema_version": 1} means. Round 7's own N53 did exactly that, making v1 reject UUIDv4 event ids it had previously accepted — a correct fix that silently redefined a shipped version.
The consequence round 6 accommodated rather than removed:
sealed batch [A, B]
attempt 1, pre-deploy v1 rejects B -> ClickHouse [A], vendors [A] (202 lost)
deploy B joins v1
attempt 2, SAME batch v1 accepts B -> vendors [A, B]
-> flush: 1 row, fingerprint MATCHES
-> N51 ACKs it, B never reaches ClickHouse
CLICKHOUSE A POSTHOG / AMPLITUDE A, BA semantic event in a lens and not in the source — the inverse of the sentence the architecture rests on. schemas.mjs freezes each version's vocabulary, property registry, acceptance-gating limits and id formats, with a digest the battery asserts so an edit fails npm test naming the two legal moves: mint v2, or use a new event name. The limits are in the schema because freezing the vocabulary and leaving PROPERTIES_MAX_BYTES tunable leaves the same divergence reachable by a different knob; OCCURRED_AT_MAX_SKEW_* are deliberately not, because they clamp rather than reject (A3) and so change a value, never a count.
With drift impossible, N51's accommodation is withdrawn. A count mismatch is unexplainable again, and unexplainable is a quarantine (N23).
1. `received_at` stays server-stamped per attempt — ruled NO on freezing it. Carrying it back on a retry would make the storage partition depend on a browser-held field, which is A3 walked back in. The epistemic split is kept: occurred_at is the client's claim, received_at is our observation of this accepted attempt. N50's consequence is absorbed by making the probe primary and serializing acceptance, not by making a server fact client-supplied.
2. N31/N36 → diagnostics: agreed in principle, NOT YET ENACTED. The reviewer ruled "yes, but only after N54 is closed with strong single-drainer coordination." N54 is closed against fakes. The coordinator has never run on Cloudflare, no Durable Object has ever been created, and "acceptance is serialized" currently rests on a class in this repository behaving the way its own test says.
Round 3 said "this is the end of what fakes can decide" and rounds 4, 5, 6 and 7 each falsified it for free. So the demotion waits for the first staging flush to show a real lease acquired, refused and released. A falsifier pins the current behaviour until then.
The N31/N36 demotion, gated on a real lease (above).
`vendor_repair_v1` has a writer and no reader — unchanged from §9.7.
Site-key issuance is manual; E_SITE_KEYS is a secret written by hand.
FNV-1a 128 is not collision-resistant. Adequate while site_id scoping means a caller can only
collide with itself; SHA-256 is required before one site key is shared across tenants.
The fence's residual window is one insert. Bounded, stated, unmeasured.
The DO adds a call to the flush's critical path and its latency is unmeasured.
`batch_fingerprint` adds 32 bytes per row plus a groupUniqArray on the hot-path probe.
Unchanged from §9.8, plus:
that EFlushCoordinator's lease behaves under REAL overlapping cron invocations [N54]
the DO's acquire/release latency inside the flush's active-minutes budget [N54]
whether SharedMergeTree's dedup is partition-scoped as the docs describe [N50]The threshold, as the reviewer set it and as round 7 accepts it: N52–N55 closed, the two rulings made, then staging. No further general architecture round unless staging's own evidence uncovers a correctness hole.
The round-7 review read the implementation rather than the findings, reran the battery independently (288/288), agreed with both N50 rulings — and then invoked round 7's own exception to "then staging": nine more, each reproduced locally before it was reported. Round 8 reproduced all nine again from scratch before touching anything, and found a tenth while checking whether one of the proposed repairs was safe.
Battery: 338 falsifiers, 0 failing. 50 new.
| # | Severity | What it is |
|---|---|---|
| N56 | BLOCKER | the content fingerprint was still a client claim — validated for shape and copied, so N45 was bypassable without finding a hash collision at all |
| N57 | HIGH | the encoding was injective over JS values, and the fingerprint is about wire values, which are a different set — in two unrelated ways |
| N58 | BLOCKER | a missing E_FLUSH_COORDINATOR binding meant "coordination unnecessary" rather than "coordination unprovable", so a wrangler typo silently reopened N54 |
| N59 | HIGH | logout() deleted the durable outbox while a send was still in flight |
| N60 | HIGH | a late consent grant left a persisted outbox unread until the 24-hour rule dropped it |
| N61 | HIGH | undefined from a transport meant "durably accepted" |
| N62 | MED/HIGH | the vendor repair log inherited N50's partition hole, and its comment said it did not |
| N63 | HIGH (cost) | an empty queue woke ClickHouse three times, and the test denying it was counting inserts |
| N64 | MED/HIGH | the lease's expiry was decided using the clock of the caller asking to take it |
| N65 | HIGH | found locally — the canonical queue had no dead letter queue and Cloudflare's default retry limit of 3, while the disposable lenses had 5 and a DLQ each |
Round 7's was the repair named the right property and implemented a weaker one. Round 8's is one layer out from that:
THE EVIDENCE WAS ASSERTED BY THE PARTY IT WAS EVIDENCE ABOUT.
the fingerprint was minted by the sender and never checked (N56)
the delivery acknowledgement was whatever the transport returned, and returning nothing counted (N61)
the lease's expiry was decided with the clock of the caller asking to take it (N64)
Three different layers, one shape: a proof supplied by the thing it was supposed to constrain. Each was individually defensible when written — the SDK does mint the fingerprint, the transport does know whether it succeeded, the Worker does know the time — and the defence is the error. Knowing a fact is not the same as being the right party to attest it.
N57 and N63 are the round's other half and rhyme differently: both are claims that were true of one representation and false of the one that matters. "Injective" was true of JS strings and false of the bytes. "An empty queue does not keep ClickHouse awake" was true of inserts and false of queries.
The review filed these together and they behave differently, so they are separated here.
(a) Lone surrogates are an injectivity failure, not a browser/server mismatch. Every unpaired UTF-16 surrogate encodes to the same UTF-8 bytes — measured, "\uD800" and "\uD801" both give ef bf bd — so canonicalEncode distinguished them and fnv1a128Hex could not.
But JSON.stringify escapes a lone surrogate to the six ASCII characters \ud800 (well-formed JSON.stringify, ES2019), so it survives a real UTF-8 wire round trip intact — also measured. Both ends therefore computed the same colliding value. Nothing disagreed; two distinct wire batches simply shared one identity. That is what makes it load-bearing for N56: the verification can be bypassed without touching the hash, by mutating one lone surrogate to another.
Fixed by normalising, not refusing. A string truncated across a surrogate pair — s.slice(0, 100) through an emoji — is the ordinary way a real analytics property acquires one, and refusing would drop genuine events for a defect of ours. Normalising before the fingerprint makes the encoding injective over the values that can actually cross the wire, which is the set the fingerprint is about.
(b) `[undefined]` and sparse holes ARE a mismatch. JSON.stringify([undefined]) is "[null]", and so is a hole. Round 7's encoder gave array-undefined its own tag, so the browser fingerprinted a value the server can only ever receive as something else.
Latent until now, because #restorePending() prefers the stored fingerprint. N56 makes it fatal: ingest recomputes from the parsed body, gets a different digest, and refuses the batch on every attempt until the horizon drops it. So this is not "also worth fixing" alongside N56 — shipping N56 without it converts a silent mis-fingerprint into silent event loss. They ship together.
The rule both halves serve:
The fingerprint describes the JSON value that crosses the network, not the JavaScript object that produced it.
toWireValue() is where a JS value becomes a wire value, at track(), before anything is queued. It mirrors JSON.stringify exactly where that is lossy but harmless, and refuses where JSON.stringify would invent a value we could not explain — a NaN that becomes null is a measurement that becomes an absence, which is A3 walked back in.
The battery found two real bugs in this repair. Array.prototype.map skips holes, so the first draft of the sparse-hole guard could not see the one case it was written for, and the first draft of toWireValue returned a still-sparse array that merely looked right under JSON.stringify. Both are indexed loops now. The round's own pattern, one level in: a guard that named the right property and used a primitive that could not implement it.
The review's diagnosis is exactly right, including that the test denying it was counting insertCalls while three ClickHouse queries ran underneath. Measured before: queue depth 0 | pulls 1 | inserts 0 | QUERIES 3. After: queries 0.
Its correction to N23 is also right, and worth restating because round 4 got it slightly too strong: a pull only leases a message; the ACK is what destroys it. An unacked message returns after the visibility timeout, which N18 already guarantees outlasts the drain.
But the repair as proposed would have introduced data loss, and this is the one place round 8 departs from the review. "A pull is reversible" is true once and false repeatedly. Cloudflare counts delivery attempts, and at the retry limit — default 3 — a message with no dead letter queue is, in Cloudflare's own words, deleted permanently. So preflighting after a non-empty pull spends one retry per cron cycle during a ClickHouse outage, and at a two-hour cadence that destroys the canonical backlog six hours into an outage which the old ordering survived for the full four-day retention. That is a straight reversal of §28.14.
Checking whether that hazard was real is what produced N65. The repair therefore ships as two halves that are only safe together:
empty pull -> free. no message, so no attempt to count. <- all of the cost benefit
non-empty pull -> costs one attempt if the preflight then refuses <- needs N65N63 and N65 ship together or neither ships.
cd-e-vendors/wrangler.toml gives PostHog and Amplitude max_retries = 5 and a dedicated dead letter queue each. The canonical queue has a pull consumer, so it has no [[queues.consumers]] block in any wrangler.toml, and nothing else set either value:
e-events-posthog max_retries 5 DLQ e-events-posthog-dlq
e-events-amplitude max_retries 5 DLQ e-events-amplitude-dlq
e-events-clickhouse max_retries 3 DLQ NONE <-- the canonical oneThe queue carrying the canonical history — the one whose integrity eight rounds exist to protect, the one the vendor tables are explicitly lenses onto — was the only one that silently destroyed its contents, at the lowest retry budget of the three. The architecture's priority order, implemented backwards. Nobody decided that; nobody decided anything, and the default decided.
scripts/check-queue-config.mjs missed it because its stated invariant list says "each vendor queue has its own dead letter queue" — N40's scope, faithfully implemented, never widened when the canonical queue's own durability became load-bearing.
And the fix has an honest limit that lands somewhere interesting. Pinning max_retries at Cloudflare's ceiling of 100 does not hold at every cadence:
120 min -> needs 48 attempts ok
60 min -> needs 96 ok
58 min -> needs 100 ok, exactly at the ceiling
30 min -> needs 192 THE RETRY BUDGET BECOMES BINDINGThat boundary is the same one-hour interval A9 found, arrived at independently two rounds and one review apart: below an hour the flush gets 30 seconds of CPU instead of 15 minutes (A9) and the retry budget starts expiring the backlog before retention does (N65). Neither is visible in the cron expression, and brief §6 actively invites tightening it. assertRetryBudgetCoversRetention() refuses rather than degrades.
There was one forget(), it deleted all three storage keys, and both logout() and consent withdrawal called it. But they are asking different questions:
CONSENT WITHDRAWAL "stop holding my data" -> erase EVERYTHING, sealed work included
LOGOUT "different person from now" -> rotate the IDENTITY; the sealed work is
already the other person's and is owed to themN16 spent a round making sealed batches outlive the document, and logout() deleted the durable copy of one still in flight — reproduced: in-memory pending 1, persistent pending absent. A navigation in that window loses the batch outright, which is the exact failure N16 exists to prevent, opened by the call most likely to be followed by a navigation.
Splitting it is safe because of N15: every pending batch carries the assertion frozen when it was sealed, so a batch retried after logout stays attributed to the person who generated it and cannot become the next person's. The falsifier asserts that directly.
The comment defending undefined → confirmed cited round 5's contract. E has never been deployed. There is no shipped transport to be compatible with, so what was being preserved was compatibility with the prototype's own fakes — the cheapest possible moment to break it, defended as though it were the most expensive.
The failure mode is now a duplicate delivery rather than a lost batch, and duplicates are what eight rounds of idempotency work exist to absorb: same batch_id, same event_ids, same fingerprint, so the presence probe drops it. Loss is the one failure the architecture cannot repair.
No new architecture. Every finding was closed in place; nothing was re-designed.
`vendor_repair_v1` still has no reader. N62 gives it a stable identity and a probe before the
reader exists, which is cheaper than fixing the reader, but the reader is still not written.
FNV-1a 128 is unchanged. Still not collision-resistant, still adequate only while site_id
scoping means a caller can collide only with itself. SHA-256 remains the upgrade.
Nothing was deployed. No Cloudflare resource was created and no money was spent.
The N31/N36 demotion, unchanged: gated on a real lease acquired, refused and released on
Cloudflare.
`vendor_repair_v1` has a writer and no reader — unchanged from §9.7.
Site-key issuance is manual.
FNV-1a 128 is not collision-resistant — SHA-256 before one site key is shared across tenants.
The fence's residual window is one insert. Bounded, stated, unmeasured.
The DO adds a call to the flush's critical path, latency unmeasured.
N63's non-empty branch spends a delivery attempt on a refusal. Bounded by N65's budget,
recoverable via the DLQ, and unmeasured against a real queue.
`toWellFormed` is ES2024. The regex fallback is asserted equivalent to the built-in across the
battery's cases, but has never run on an actual old browser.
Unchanged from §10.8, plus:
that a Cloudflare pull WITHOUT an ack really does increment `attempts` [N63/N65]
whether an EMPTY pull leaves the attempt counters untouched, as reasoned [N63]
that max_retries and dead_letter_queue can be set on an http_pull consumer [N65]
the real skew between the Worker's clock and the DO's, now measured [N64]The threshold. Round 7's was "N52–N55 closed, the two rulings made, then staging", and the review declined to cross it on the strength of nine locally reproduced holes. Round 8 closes all ten and adds no architecture. The remaining questions in §11.10 are all provider questions — every one of them is a fact about Cloudflare or ClickHouse that no fake can settle, which is a different category from the last eight rounds, where the fakes kept finding real defects for free.
Round 8's own view: this is the staging round. But note what N65 did to that confidence — it was found not by reviewing the design but by checking whether a proposed repair was safe, in a file nobody had reason to reopen. The honest statement is that the fakes have stopped finding architecture defects and have not stopped finding configuration ones, and staging is where configuration becomes observable.
The round-8 review unpacked the bundle, read the implementation, and reran the battery — and got 337/338 where the bundle recorded 338/338. Same code, same machine, one day later.
That single discrepancy is the round. Four more findings followed from reading the round-8 repairs, and the reviewer's constraint was explicit: close these, change no architecture, then create the staging resources. Round 9 does exactly that and adds nothing.
Battery: 360 falsifiers, 0 failing — at today's date AND five years on. 22 new.
| # | Severity | What it is |
|---|---|---|
| N66 | MED/HIGH | the battery, asserted hermetic since N33, verified assertions against the wall clock — so a green run decayed on a timer |
| N67 | HIGH | the deployed-config checker — the thing round 8 said staging needs — crashed with a ReferenceError on its healthy path |
| N68 | HIGH | event_id had a stronger contract in the vendor adapter than in E itself; nothing enforced uniqueness, and Amplitude dedups on it |
| N69 | MED/HIGH | batch_version named a version and selected nothing; ingest overwrote whatever arrived with a hard-coded 1 |
| N70 | LOW | the generated bundle carried a Round-8 banner over a header still saying "rounds 2–7" and "288 falsifiers" |
Round 8's was the evidence was asserted by the party it was evidence about. Round 9's is narrower, and it is about us rather than about the system:
THE THINGS THAT CHECK THE SYSTEM WERE NOT THEMSELVES CHECKED.
the battery had a guard against the network and none against the clock (N66)
the deployment gate had never been executed (N67)
the bundle generator derived its test count from a live run and its **narrative from a hand-typed
header** (N70)
Every one of these is this project's own discipline — executable, not prose; derived, not typed — applied rigorously to the product and not at all to the tooling that judges it. Nine rounds were spent making the system prove its claims while the harness making those claims was exempt.
N68 is the exception, and the only genuine semantic question in the round.
verifyAssertion defaulted nowSeconds to Math.floor(Date.now() / 1000). Every test caller passed an explicit time; the one caller that did not was the ingest Worker, which is the only caller that matters. The fixture mints at NOW = 1_787_500_000_000 with a ~24h15m lifetime, so:
2026-08-23 338 / 338 the bundle was generated here
2026-08-24 337 / 338 same code, same machine, different datetest/no-net.mjs closes the network because an ambient dependency is one a test must not silently acquire. The clock is the same kind of dependency and had no such guard.
The fix is structural, not a corrected default. nowSeconds is now required — a missing one is a TypeError, deliberately not an AssertionError, so a caller's bug cannot read as a rejected credential. The Worker takes one observation, requestNow(env), and uses it for both assertion expiry and received_at; the falsifier asserts the clock is read exactly once per request. That buys a second thing worth having: an assertion can no longer expire between being verified and the batch it authorised being stamped.
And the guard is now permanent. test/time-travel.mjs shifts Date.now(), new Date() and the zero-argument Date constructor five years forward and the whole battery runs again — a shift rather than a freeze, so every interval is preserved and only an absolute-date dependency can break. npm run test:all runs both, and the bundle generator refuses to build unless both are green.
The guard was verified against the defect rather than assumed to catch it. With round 8's code restored exactly, on a simulated 2026-08-23:
DEFECTIVE code, real date 338 / 338 <- what round 8 shipped
DEFECTIVE code, +5y time travel 337 / 338 <- what would have caught it, the same dayThe canonical-queue block read found.settings?.message_retention_period, and found is declared with const inside the vendor loop forty lines below. Reproduced against a mocked healthy account:
ReferenceError: found is not defined
scripts/check-queue-config.mjs:203Two things make this worse than an ordinary typo. First, it crashes only when the account is healthy — the block runs only if the canonical queue exists, so a missing queue would have returned earlier and survived. Second, round 9 wrote it in round 8 and could not have noticed, because the script short-circuits without credentials and there were none.
A deployment gate with no test is prose that looks like code — the argument rounds 5 through 8 kept making about the system, aimed at the wrong file.
checkQueueConfig(queues) is now a pure exported function — API payload in, verdict out — with the CLI a thin wrapper that fetches, prints and exits. Nine falsifiers drive it across a healthy account and every N65 misconfiguration. The value it should have used was already in scope: seconds, the number extractRetentionSeconds() had already normalised across the provider's spellings. And when that is null the script has already concluded retention is unprovable, so substituting the declared constant would be N37 exactly — comparing a fact against a wish and reporting the wish as checked. It refuses instead.
The reviewer also confirmed externally that Cloudflare's API exposes dead_letter_queue and settings.max_retries on an http_pull consumer, which removes one of §11.10's open provider questions: N65's provisioning is not asking for a feature that does not exist.
THE RULING: within a site, `event_id` identifies one immutable logical event. Canonical event identity is `(site_id, event_id)`.
It needed to be made because the vendor adapter had already made it by implication and E had not. normalizeEvent runs per event with no cross-event state, so a batch could carry one id twice with different payloads — reproduced, ok: true, accepted: 2, both surviving.
The consequence is lens-vs-source again, from the opposite direction to N55. Amplitude's adapter maps insert_id: event.event_id, and Amplitude ignores a later event with the same insert_id for seven days:
CLICKHOUSE (source) AMPLITUDE (lens)
X page_viewed X page_viewed
X pricing_viewed (dropped as a duplicate)The whole batch is refused, not the second event dropped. Two differing payloads claiming one identity is an integrity conflict, not an invalid optional property — and N56's fingerprint covers both, so a partial accept would produce a message whose contents no longer match the identity it was admitted under. A malformed id remains a per-event rejection, because two events with no identity are not the same identity twice.
What this does not close, stated plainly: the same `event_id` in two different batches. Canonical idempotency is batch-scoped (N35/N45), so both are accepted today. The reviewer is right that UUIDv7 improbability is not an argument here, because this is a publicly writable endpoint. The options are a per-event presence query, a compact identity ledger, or something else — and the hot path is the reason not to choose blind: a two-hour drain can hold thousands of events where the batch probe handles at most ~100 identities.
So it is a staging measurement, and there is a falsifier pinning the current behaviour so the gap cannot be mistaken for a closed one and so whichever mechanism is chosen has a test to turn red first.
batch_version: 1 was on every request, never read, and then overwritten by a hard-coded 1 in the output message. Measured: missing → 1, 0 → 1, 2 → 1, 999 → 1, "1" → 1.
That is precisely N55's content — a version label that does not select semantics makes a compatibility claim nothing enforces — at the envelope layer that N55 did not reach. It is latent only while one version exists: the moment a v2 envelope ships, a v2 sender talking to a v1 deployment is accepted, silently reinterpreted, and relabelled v1, and N56's fingerprint would not catch it because the fingerprint covers the events and the assertion, not the envelope.
Numeric 1 or refused. No coercion — the string "1" fails, because round 7 already had schemaFor() start accepting "1" through a lookup that did not check the type. The constant now lives in one place that both the SDK and ingest import, so they cannot drift.
Exactly what the reviewer scoped, and nothing else:
No architecture changes. Workers, Queues and ClickHouse are untouched.
No rate limiter. The brief calls for one and the prototype still has none. Agreed as a
public-production gate, not a staging gate — staging exists to learn provider behaviour before E is exposed broadly.
No cross-batch event-identity mechanism. Ruled, pinned, deferred to staging measurement (§12.5).
Nothing deployed. No Cloudflare resource created; no money spent.
The N31/N36 demotion, gated on a real Durable Object lease.
Cross-batch `event_id` reuse (§12.5) — ruled, unenforced, pinned by a falsifier.
No rate limiting or bot controls — public-production gate.
`vendor_repair_v1` has a writer and no reader.
FNV-1a 128 is not collision-resistant — SHA-256 before one site key is shared across tenants.
Site-key issuance is manual.
N63's non-empty branch spends a delivery attempt on a refusal — bounded by N65, unmeasured.
`toWellFormed` is ES2024; the fallback is asserted equivalent but has never run on an old browser.
active ClickHouse minutes per day under the two-hour cadence THE COST QUESTION
whether EFlushCoordinator's lease behaves under real overlapping crons [N54]
whether SharedMergeTree dedup is partition-scoped as documented [N50]
that a pull WITHOUT an ack increments `attempts`, and an empty pull does not [N63/N65]
the real skew between the Worker's clock and the Durable Object's [N64]
what a per-event identity check costs on a full two-hour drain [N68]
cold start, merge pressure, compression, bloom-filter/probe costThe threshold, and round 9 accepts the reviewer's line without amendment: N66–N70, then staging.
The remaining questions are now all provider questions or explicitly-deferred measurements. Round 8 said that too and was wrong by five — but the five were of a different kind: three were tooling defects in the harness rather than the system, one was a contract the vendor layer had already decided, and one was a document header. The system's design has survived nine rounds without a component being replaced. What is left is the thing no local phase can produce: whether two-hour buffering actually lets ClickHouse sleep enough to land in the $10–$30/month neighbourhood.
A patch, not a round, scoped exactly as the round-9 reviewer scoped it: close N71, clean two stale claims, stop. The reviewer independently reran both batteries (360/360 and the +5y run), confirmed the derived header now agrees with reality, and checked Cloudflare's current API docs against N65's provisioning — then found one real defect in N62's repair.
Reproducing it turned up a second, in the fakes.
Battery: 374 falsifiers, 0 failing — at today's date and five years on. 14 new.
| # | Severity | What it is |
|---|---|---|
| N71 | MED/HIGH | N62's presence probe runs once per pull, and the repair path had no inFlight set — two copies of one repair in one pull both read "absent" and both inserted, into different partitions |
| N72 | LOW | "hermetic against the clock" overstated what the time-shift battery proves |
| N73 | LOW | the prototype README was still at 288 falsifiers and a round-7 layout |
| N74 | HIGH | found locally — the fakes could neither partition a repair row nor answer the repair presence query, so both halves of N62 were untestable and one of them had never worked |
Round 8 gave repairs a stable identity and a presence probe, and closed the case it was written for: the same repair re-detected in a later pull. It did not close two copies in one pull, because the probe runs once before the workers start and all eight then measure themselves against that one snapshot:
probe -> R absent
worker A worker B
absent absent
INSERT R INSERT R
202608 202609 <- detected_at straddles a monthand the dedup token cannot span partitions, which is the whole of N50 and the whole reason N62 exists. Reproduced against the shipped flush: 2 rows, 2 partitions, `vendor_repairs_recorded: 2`, no errors.
The canonical path has had inFlight since N35 for exactly this shape. The repair path was given the probe and not the set, so N62 solved one half of a two-part problem and read as though it had solved both. It is N54's TOCTOU one scope smaller: N54 serialises across drainers, this serialises within one.
The twin is left unacked rather than made to wait — the same choice the canonical path makes (N44). Waiting would put one worker's progress on another's latency inside a budgeted window, and the queue already has a mechanism for "come back later". The next flush's probe then answers from the table definitively: measured, flush 1 writes one row and acks one message, flush 2 reports vendor_repairs_deduped: 1 and acks the twin. One logical repair, one row.
Found while building the N71 witness, and it is the larger finding.
First half — the partition key. partitionOf() read rows[0].received_at and nothing else. events_v1 partitions on received_at; vendor_repair_v1 partitions on detected_at, and a repair row has no received_at at all. Measured against the shipped fake:
partitionOf([{received_at: '2026-08-31T23:59Z'}]) -> '202608'
partitionOf([{detected_at: '2026-08-31T23:59Z'}]) -> 'unpartitioned'
partitionOf([{detected_at: '2026-09-01T00:01Z'}]) -> 'unpartitioned'So every repair row landed in one phantom bucket, the dedup token was effectively global for that table, and the exact hazard N62 exists to close was unrepresentable. Round 8 added a presence probe for a mechanism its own fakes modelled as unnecessary, and the tests passed either way.
It also masked N71. The first witness reported vendor_repairs_recorded: 2 with one row in the table — the flush really did insert twice and the fake collapsed them. A fake that hides a defect while printing the counter that proves it is worse than one that cannot see it.
Second half, and worse — the probe had no handler. classifyRepairPresence() issues a query the fake did not recognise, so it fell through to return []. An empty result is not "the query failed"; it is "none of these repairs are recorded" — the answer insert.mjs calls the most dangerous possible one in the comment directly above the function. So the probe reported ABSENT for every repair, forever, and N62's cross-pull closure had never detected anything. Witnessed end to end: a repair written by one flush was written again by the next.
Round 8's N62 tests passed presence in as a literal Set. They proved insertVendorRepair behaves correctly given a presence set and never proved the set is ever populated — two halves of one mechanism, each tested against a stub of the other.
This is round 6's own sentence, left true of a second table for two rounds:
A fake that cannot represent the mechanism's scope cannot falsify a claim about its scope.
partitionOf is table-aware now, from a map the battery asserts against ddl.sql; and an unrecognised query against events_v1 or vendor_repair_v1 throws rather than answering [], so a missing fake is loud.
Round 9 called the battery "hermetic against the clock" by analogy with N33's network guard. The analogy overstates it, and in this project that is itself a finding: N9 is the rule that a claim must not exceed its evidence, and it was broken by the sentence describing the fix for the last time it was broken.
no-net.mjs removes the network. The time-shift harness does not remove the clock, and ambient reads remain on purpose — uuidv7's default, the SDK's default now, requestNow()'s fallback, and the coordinator's own clock, which N64 requires to be real. So the code was right and the sentence was wrong. What is proved:
The battery is invariant under a five-year wall-clock epoch shift.
What is not: that nothing reads ambient time (false), invariance under a clock running at a different rate, or one that moves backward mid-run. Those are worth having and are not what this is.
The README said npm test # 288 falsifiers and listed test files up to round7.test.mjs, through two rounds. N70 fixed exactly this defect in `FINDINGS.md` and left the copy of it one directory down — N53's shape ("diagnosed two, fixed one"), applied to documents.
The number is not corrected. It is removed. FINDINGS.md's header is generated from a live run, so exactly one place in the repository states a count and nothing can disagree with it.
Cross-batch `event_id` reuse (§12.5) — ruled, unenforced, pinned by a falsifier, mechanism to
be chosen on measured cost.
No rate limiting or bot controls — public-production gate, agreed.
`vendor_repair_v1` has a writer and no reader.
The N31/N36 demotion, gated on a real Durable Object lease.
FNV-1a 128 is not collision-resistant — SHA-256 before one site key is shared across tenants.
Site-key issuance is manual.
N63's non-empty branch spends a delivery attempt on a refusal — bounded by N65, unmeasured.
First staging traffic is controlled and synthetic, not public. The reviewer's condition, and it follows directly from §13.6: cross-batch event identity is unenforced and there is no rate limiting, so an arbitrary caller could deliberately reuse ids against the very mechanism we are trying to measure.
staging SDK / synthetic sender -> staging ingest Worker -> real Queues
-> real Durable Object -> real ClickHouse Cloud BasicACTIVE CLICKHOUSE MINUTES PER DAY under the two-hour cadence <- the cost question
EFlushCoordinator's lease under real overlapping crons [N54]
whether SharedMergeTree dedup is partition-scoped as documented [N50]
that an unacked pull increments `attempts`, and an empty pull does not [N63/N65]
Worker-to-Durable-Object clock skew, now measurable [N64]
what a per-event identity check costs on a full two-hour drain [N68]
cold start, merge pressure, compression, bloom-filter/probe costRound 10 makes no architectural change and adds no mechanism. Ten rounds and 374 falsifiers have not replaced Workers, Queues, Durable Objects or ClickHouse.
One thing is worth stating plainly rather than as confidence. Round 8 said the remaining questions were all provider questions and was wrong by five; round 9 said it and was wrong by four. Both times the residue was in the harness rather than the system — the battery's clock, the deployment gate, the bundle header, and now the fakes themselves. That is a real pattern and it is nearly exhausted, because the fakes have now been audited against the DDL and made loud where they were silent. But the honest form of the claim is not "there is nothing left"; it is that what is left is no longer answerable here, which is a different sentence and the one that justifies staging.
The final pre-staging patch, scoped exactly as the round-10 reviewer scoped it: N75, N76, two stale comments, stop. The reviewer declined another adversarial architecture round and was right to — nothing here is a design question. Both findings were reproduced against the shipped code before being patched, and reproducing N76 turned up a third.
Battery: 396 falsifiers, 0 failing — at today's date and five years on. 22 new.
| # | Severity | What it is |
|---|---|---|
| N75 | HIGH | the protocol says 202 and browserTransport implemented res.ok, so 200/201/204/299 all released the batch — and the same Worker answers 200 on /health |
| N76 | HIGH (lens) | a missing vendor binding was indistinguishable from a vendor deliberately turned off; and a stale E_EVENTS_VENDORS binding created a third delivery path to both vendors at once |
| N77 | HIGH | found locally — the consumer's unknown-queue fallback bypassed N39's missing-credential guard entirely and ACKed in silence |
| cleanup | LOW | the ingest header and the batch_fingerprint DDL comment each described a superseded protocol |
Measured against the shipped transport:
HTTP 200 -> confirmed true HTTP 299 -> confirmed true
HTTP 201 -> confirmed true HTTP 300 -> confirmed false
HTTP 202 -> confirmed true HTTP 503 -> confirmed false
HTTP 204 -> confirmed trueFive rounds went into making 202 mean one exact thing (N1, N5, N16, N48, N61). The comment directly above the defect even stated the intent correctly — "202 releases the batch" — sitting on top of res.ok, which releases it on a hundred statuses.
The witness was already deployed in the same file. /health answers 200 to a POST. Point the SDK endpoint at it — a plausible copy-paste, and exactly the sort of thing a staging config does by accident — and every batch is confirmed and deleted while nothing has crossed the canonical Queue boundary. That is the class of false acknowledgement N1, N16, N39 and N61 each closed one instance of, left open in the fifth place.
Why it survived ten rounds, and this is the part worth keeping. The battery had two transport tests. They were named:
'N2 browserTransport confirms only on a 2xx'
'N16 the transport no longer uses sendBeacon, and confirms only on a real 2xx'and between them they exercised 202 and 503 — the two statuses on which res.ok and status === 202 give the same answer. A test named for a wider contract than it checks cannot fail when the code implements the wider contract. Both names now say 202, and the sweep that can actually tell the two rules apart is in round11.test.mjs.
202 is returned from two places, and only one of them is an acceptance:
202 = E HAS TERMINALLY HANDLED THIS BATCH.
either the canonical event is durably accepted
or it was suppressed by an authoritative privacy signal (Sec-GPC / DNT, N9)
In BOTH cases the browser must not retry.
EVERY other status leaves the batch pending.The privacy branch returns 202 with accepted: 0 having deliberately stored nothing. Collapsing it into "durably accepted" would have the transport claiming a row exists for an event that was intentionally never written. Answering 204 there instead would be tidier prose and a worse protocol: the SDK would keep the batch and re-send it every session for 24 hours, against a user who asked not to be measured.
The value is now ACCEPTED_STATUS in e-schema/limits.mjs, imported by the side that produces it and the side that requires it, because N75 is those two sides keeping separate copies of one boundary. A falsifier asserts neither file contains a re-typed 202 in executable code.
canonical queue present, both vendor bindings absent:
status 202
canonical messages 1
vendor_enqueue_failed [] <- nothing failed
vendor_repair_recorded false <- so nothing to repairEvery counter reports health. PostHog receives nothing, forever. A Wrangler typo that drops the producer is observationally identical to a deployment that turned PostHog off on purpose.
This is N39's own ruling left standing on the other side of the boundary. N39 found that a missing secret must not look like a successful delivery, and fixed it in the consumer. Its rule — "a deliberate disable is expressed by not producing to the queue at all" — is sound only if something independent says the omission was deliberate. Nothing did. The producer went on inferring intent from a thing being missing.
So the repair is not a new mechanism; it is the missing input to the one N39 already assumed:
enabled + binding present -> enqueue
enabled + binding ABSENT -> vendor_enqueue_failed + a durable vendor_repair row
not enabled -> do not enqueue, and report `vendors_disabled`
named but unknown -> `vendor_config_errors`. NOT a repair row.An unknown name gets no repair record on purpose: a repair row means these events still need delivering, and nothing can ever deliver to a vendor with no adapter — the row would be permanently un-repairable, which is worse than the typo.
The round-5 ruling is preserved without amendment. A vendor configuration failure still never turns a canonical 202 into a 503. We record the lens gap; we do not ask the browser to resend canonical history over it.
The absent case is the whole design, so it is stated rather than defaulted. E_ENABLED_VENDORS unset means every known vendor is expected; "" means none, on purpose. Both are guesses and they fail in opposite directions: "none" restores this finding exactly (a deploy that forgot the line delivers to nobody and looks healthy), while "all" makes a forgotten line produce a loud false alarm an operator fixes in one edit. Over-reporting is recoverable; silence is not — which is N61's sentence, applied to configuration.
E_EVENTS_VENDORS is deleted, not deprecated. Measured, a stale binding alongside the two dedicated ones produced a third delivery path: canonical 1, posthog 1, amplitude 1, legacy 1 — the same events delivered to each vendor twice, through a queue whose consumer then had to guess which vendor it meant. Nothing is deployed; this is the cheapest moment there will ever be to close the contract, and round 8 already made this exact argument (N61: "E HAS NEVER BEEN DEPLOYED, so the compatibility being preserved was with the prototype's own fakes").
Found while reproducing N76, and it is the sharpest of the three. Three states, one message each, measured against the shipped consumer:
queue credentials fetches result
e-events-posthog absent 0 retry + loud console.error <- N39 works
e-events-vendors absent 0 ACK, IN SILENCE <- N39 BYPASSED
e-events-vendors present 2 ACK, both vendors called <- N27 resurrectedThe middle row is N39's own defect, reached through the fallback N39 shipped beside. deliver() reports a missing credential as {ok: true, skipped: true}. The guard that catches that reads const leg = vendor ? legs[vendor] : null — so with no vendor identified there is no leg to inspect, relevant.every(l => l.ok) is true because a skipped leg reports ok, and the message is acked. One forgotten secret plus one queue rename discards that vendor's stream permanently while every invocation looks healthy. That is, word for word, the sentence N39 exists to make false.
The bottom row rebuilds N27's shared failure domain from a queue rename: an Amplitude outage retries the message and re-sends to PostHog, whose dedup is eventual and documented as not guaranteed.
And the battery was pinning it. §28.33-36 asserted:
assert.equal(acked.length, 1, 'no vendors configured -> nothing to fail -> ack');against a batch with no queue name, written in round 2 — the round before N39 — which is why N39 never noticed that its own guard had a hole beside it. The test has been rewritten to assert what §28.33-36 is actually about: a failure retries, a success acks.
No fallback now. An unrecognised queue is refused, logged, and retried rather than acked, so it reaches the DLQ if the misconfiguration persists. Failing closed is safe because batch.queue is not optional — Cloudflare's MessageBatch declares readonly queue: string (Queues JavaScript API docs, checked 2026-08-24), so the only way to reach the refusal in production is a queue this consumer genuinely does not serve.
A DEFAULT THAT ANSWERS A QUESTION IT WAS NEVER ASKED.
res.ok answers "did something 2xx" asked: "did E accept this"
.filter(b => b) answers "is there a binding" asked: "should this vendor be reachable"
?? null -> both answers "which vendors exist" asked: "which vendor is THIS message for"Each is correct-looking. Each is wider than the question. And each fails in the direction that loses data rather than the direction that duplicates it — which is the distinction rounds 1 through 10 spent themselves on (N1, N16, N39, N61), found three more times in the places nothing had yet pointed at.
Every round-11 witness was run against a deliberately reverted copy of the code it names. One did not fail there on the first attempt.
The /health witness read unconfirmedEvents immediately after await c.flush(). But flush() is deliberately synchronous from the caller's view — §28.8b, so navigation never waits on it — and returns a summary rather than the delivery promise. So the await resolves before the transport settles, and the assertion measured the pre-settlement state: 1 under the fix and 1 under the defect. It passed against both.
That is N75's own sin — an assertion that cannot distinguish the two behaviours it is named for — committed inside the test written to close N75. It waits on onDelivery now, which is the moment the batch's fate is actually decided, and it fails against the reverted copy.
A falsifier that has not been run against the defect it names is a claim, not evidence.
Cross-batch `event_id` reuse (§12.5) — ruled, unenforced, pinned by a falsifier.
No rate limiting or bot controls — public-production gate.
`vendor_repair_v1` has a writer and no reader.
The N31/N36 demotion, gated on a real Durable Object lease.
FNV-1a 128 is not collision-resistant — SHA-256 before one site key is shared across tenants.
Site-key issuance is manual.
N63's non-empty branch spends a delivery attempt on a refusal — bounded by N65, unmeasured.
Round 11 adds nothing to this list and removes nothing from it. It is a protocol/configuration patch, not an architecture round.
ACTIVE CLICKHOUSE MINUTES PER DAY under the two-hour cadence <- the cost question
EFlushCoordinator's lease under real overlapping crons [N54]
whether SharedMergeTree dedup is partition-scoped as documented [N50]
that an unacked pull increments `attempts`, and an empty pull does not [N63/N65]
Worker-to-Durable-Object clock skew, now measurable [N64]
what a per-event identity check costs on a full two-hour drain [N68]
cold start, merge pressure, compression, bloom-filter/probe costEleven rounds and 396 falsifiers have not replaced Workers, Queues, Durable Objects or ClickHouse. The round-10 reviewer's conclusion stands and round 11 does not soften it: continuing to simulate Cloudflare now has lower information value than turning the real staging plane on, with controlled synthetic traffic (§13.7).
The round-11 reviewer read the files as an actual --env staging deployment rather than as application code, declined another architecture round, and found three seams. One was a staging blocker. All three were reproduced against the shipped files before being patched.
Battery: 415 falsifiers, 0 failing — at today's date and five years on. 19 new.
| # | Severity | What it is |
|---|---|---|
| N78 | BLOCKER | cd-e-flush's [env.staging] declared a name, a coordinator and a cron and nothing else — no vars, no KV — so a staging deploy would pull from accounts/undefined/queues/undefined |
| N79 | HIGH | the deployment gate hardcoded production queue names, so the one check standing between us and staging could not certify staging |
| N80 | MED/HIGH | N77's "unknown queues fail closed" was a prefix rule, which trusted every hyphen suffix — including the dead letter queues |
| cleanup | LOW | package.json said "round 10" during round 11; cd-e-ingest/wrangler.toml kept the superseded one-line 202 claim |
The pattern is not round 11's. Round 11's was a default that answers a question it was never asked. This one is:
THE APPLICATION SEMANTICS ARE MORE MATURE THAN THE DEPLOYMENT DESCRIPTION.
Eleven rounds hardened what the code does. Nothing had yet read the files that say which code runs where the way the deploy command reads them. Every finding here is in a .toml, a name, or a script, and none of them questions Workers, Queues, Durable Objects or ClickHouse.
Round 11 learned this rule one file over and wrote the reasoning into cd-e-ingest/wrangler.toml: Wrangler named environments do not inherit non-inheritable keys. Then it applied the rule to the file it was looking at and not to the others. Parsed directly:
top-level vars present staging vars NONE
top-level E_SNAPSHOTS present staging E_SNAPSHOTS NONE
top-level DO present staging DO presentwrangler deploy --env staging would have produced a flush Worker whose pull URL formed as https://api.cloudflare.com/client/v4/accounts/undefined/queues/undefined/messages and whose ClickHouse endpoint formed as undefined/?query=.... The missing E_SNAPSHOTS is the quiet half: snapshot writing is deliberately optional in the flush, so the dashboard's data would simply never appear with nothing reporting a problem.
Audited all three configs rather than the one named, and cd-e-vendors had it too: [env.staging] declared no vars, so POSTHOG_HOST was absent. That one does not produce an undefined URL — adapters.mjs reads it as env.POSTHOG_HOST ?? 'https://us.i.posthog.com' — which is worse in a smaller way: a deployment that deliberately configured a different PostHog host (EU region, a proxy) would have staging silently post to the US endpoint with every leg reporting ok. A default standing in for a configured value is still a default.
Staging gets its own KV namespace, not production's. Sharing one would have a staging flush overwrite the live dashboard's single snapshot key with synthetic data, silently — the same "one shared thing, two failure domains" defect as N27, applied to the read side, and brief §26's rule.
`secrets.required` is now declared in all three configs. Listing required secrets in a comment is how E has described them for seven rounds and a comment cannot refuse a deploy. Wrangler validates the declared names at wrangler deploy and fails with the missing ones:
missing secret -> BAD: deploy succeeds, the first cron finds out
GOOD: deployment itself refuseswhich is N39 ("a missing secret must not look like a successful delivery") moved from first-message to deploy-time, the cheapest place there is. E_ASSERTION_SECRET earns this especially: absent, it does not fail closed on its own — every authenticated batch 401s and retries to the 24-hour horizon while anonymous traffic flows normally, which is N49's exact failure shape and invisible in aggregate volume.
Checked, rather than assumed: migrations and triggers are inheritable; vars, durable_objects, kv_namespaces, queues and secrets are not (Wrangler configuration docs, 2026-08-24). A check demanding that migrations be restated would be noise, and noise in a deployment gate is how a real finding gets scrolled past.
The battery can now read a wrangler.toml. The reviewer found N78 by parsing TOML; nothing in the battery had ever read one as configuration — round 11's registry test greps it for substrings, which answers does this string appear and not is this key declared under that table. There is a table-header scanner in test/wrangler-toml.mjs now, and the general check ("every worker restates every non-inheritable key in every environment it declares") runs against all three files. The scanner refuses the TOML constructs it cannot read rather than approximating them, because N74 is the finding that a fake which cannot represent the mechanism reports the claim as proven anyway.
Reproduced against a healthy mocked staging account — HTTP pull consumer, 4-day retention, max_retries 100, canonical DLQ, vendor DLQs, 25-minute visibility, everything correct:
FAIL e-events-clickhouse does not exist
FAIL e-events-amplitude does not exist
FAIL e-events-posthog does not existThree failures on an account with nothing wrong with it. Not a provider uncertainty and not something staging would have taught us — locally decidable, and it is the first thing the staging sequence runs.
It is N67 again. That finding was "a deployment gate with no test is prose that looks like code", and round 9 gave this file falsifiers. Every one of them drove it at production names, so they proved the logic and could not notice the logic was pointed at the wrong account.
The environment is a parameter of the pure function now, defaulting to production so every round-9 falsifier keeps its meaning, and all six names derive from the grammar. The printed provisioning commands follow `--env` too — that is the quiet half. A staging preparation session that printed production wrangler queues create lines is one copy-paste from creating the production queues early, and a queue that exists is a queue this gate stops asking about, so that mistake is quieter than the missing check was.
An unknown --env is refused, not defaulted: --env staigng defaulting to production would report every staging queue missing and point an operator one step away from the fault.
e-events-posthog-staging -> posthog <- intended
e-events-posthog-dlq -> posthog <- NOT intended
e-events-posthog-dlq-staging -> posthog <- NOT intended
e-events-posthog-typo -> posthog <- NOT intended
e-events-posthog-old -> posthog <- NOT intended
e-events-amplitude-dlq -> amplitude <- NOT intendedRound 11 stated the contract as "anything other than the queues this Worker serves → loud refusal + retry" and implemented name === stem || name.startsWith(stem + '-'), which is not that sentence — it is anything beginning with a vendor stem and a hyphen, an infinite set.
The dead letter queue is the case that matters. Attach cd-e-vendors to e-events-posthog-dlq by mistake — a plausible dashboard slip, and exactly what N40 created those queues for — and N77 was supposed to say "I do not serve this queue". It said "PostHog, carry on", re-delivering batches that had already exhausted their retry budget, forever, with no path out because a DLQ has no DLQ.
And the round-11 falsifier had the same hole it was testing for. It asserted vendorForQueue('e-events-posthogus') === null and called that "a near-miss name does not match", with a comment about the separator being required. Both the code and the test agreed on the one case that cannot distinguish a prefix rule from a closed set. That is N75's own finding — "a test named for a wider contract than it checks cannot fail when the code implements the wider contract" — committed in the round that named it, one file away, by the same hand.
Membership is generated, not parsed. packages/e-schema/queue-names.mjs owns a closed grammar (<base>[-dlq][-<environment>], environments a closed list); matchQueueName() constructs every valid name for a base and compares equality. Nothing inspects a suffix, so no suffix can be admitted by accident. A DLQ matches the grammar and is still refused, and the refusal now says which kind of not-ours it is — an operator who attached the consumer to a dead letter queue must not be sent hunting for a typo that is not there.
Round 12 adds nothing to the list and removes nothing from it. Cross-batch event_id reuse, rate limiting, vendor_repair_v1's missing reader, the N31/N36 demotion, FNV-1a's collision resistance, manual site-key issuance, and N63's refusal cost all stand exactly as §14.7 left them.
The reviewer's order, accepted without amendment:
create staging queues -> create staging DLQs -> configure the HTTP pull consumer
-> npm run check:queues:staging MUST BE GREEN
-> create the staging KV -> deploy EFlushCoordinator
-> deploy ingest / flush / vendors with --env staging
-> set STAGING secrets (deploy now REFUSES without them, N78)
-> controlled synthetic traffic ONLY -> force failures -> measure ClickHouseThe first experiment answers the original economic question and nothing else:
How many minutes per day does ClickHouse actually stay active when twelve two-hour windows are available — including cold start and post-insert background merge time?
Cross-batch event_id enforcement and rate limiting stay out of staging deliberately (§13.6). Those are reasons for controlled synthetic traffic, not reasons to keep Cloudflare imaginary.
Twelve rounds and 415 falsifiers have not replaced Workers, Queues, Durable Objects or ClickHouse. What is left is not answerable here.
The round-12 reviewer read the new staging instructions against the current Wrangler CLI and API and found four tooling defects. Two would have blocked the first staging setup. Every claim below was verified against the wrangler actually installed on this machine (4.115.0) rather than against documentation — including the failure text.
Battery: 435 falsifiers, 0 failing — at today's date and five years on. 20 new.
| # | Severity | What it is |
|---|---|---|
| N81 | BLOCKER | secrets.required + wrangler secret put is a bootstrap loop on a Worker that has never existed — deploy refuses, and secret put has nothing to put a secret on |
| N82 | BLOCKER | the generated HTTP-pull command used --max-retries, which is not an option on that command; the one line that sets N65's retry budget would not have executed |
| N83 | HIGH | the queue gate asserted "has a DLQ" rather than *"has the right DLQ"* — and printed the wrong name inside its own ok line |
| N84 | MED | the queue inventory read page one of a paginated endpoint |
The pattern, and it is worth stating plainly: the system has largely stopped failing review; the deployment procedure is what is being falsified now. That is the natural continuation of round 12 — round 12 made the deployment description match the code, and this round makes the deployment procedure match the tools that execute it. It is a good boundary to reach before the first deployment rather than after it.
$ wrangler queues consumer http add fake-queue-xyz --max-retries 100
✘ [ERROR] Unknown arguments: max-retries, maxRetriesThe flag is --message-retries. It fails at parse time rather than creating a consumer with the wrong setting, which is the better of the two failures and still a blocker: N65 is the finding that Cloudflare's default of 3 retries destroys the canonical backlog six hours into an outage the queue is retained four days for, and this command is the only place that number is ever set — N40 established that HTTP pull cannot be declared in a wrangler.toml at all.
The trap is real and worth naming, because the next person will be tempted to "fix" the checker instead: the API field is max_retries and the CLI flag is --message-retries. Both are correct for their own surface. The gate reads the API field and wrote the CLI flag, and the two being differently spelled is exactly how one became the other.
That string had been in the repository since round 5, inside the file whose entire purpose is "turn deployment truth into executable instructions". Nothing could catch it because a printed command is prose until something checks it — N67 aimed at the gate's output rather than at its logic. Commands are generated by functions now, every flag is checked against a surface transcribed from wrangler --help at a named version, and the builder refuses to return a command carrying an unverified flag rather than returning one and logging about it.
A first draft of the round-13 falsifier caught something I would have shipped: the gate's own explanation of N82 printed --max-retries three lines below a block of copy-pasteable commands. Being right about history does not make a bad flag safe to print next to good ones. The history is in the source and here; the terminal gets commands.
wrangler deploy --env staging -> refuses, required secret missing
wrangler secret put X --env staging -> the Worker does not exist yet (API 10007)Round 12 added secrets.required on purpose and it does exactly what it says. For a Worker that has never existed, that turns round 12's own instructions into a loop. Cloudflare carries this as workers-sdk issue #14258, closed against a fix to the error guidance rather than to the behaviour. The path that closes the loop is wrangler deploy --secrets-file <file>, which uploads the initial secrets with the first version.
The instructions were wrong only in the state we are about to enter. After the first version exists, secret put is the correct rotation command and always was.
The repair is not a corrected paragraph. N81 is an ordering defect and round 12 shipped the sequence as prose in §15.6; prose cannot be run, so an ordering defect in it is found by a careful reader or by an operator at 2am. scripts/staging-plan.mjs makes each step declare what must already exist and what it brings into existence, and a falsifier walks the plan from an empty account asserting nothing is ever required before something provides it. It would have caught N81 mechanically — and the falsifier demonstrates that by running round 12's order and watching the two steps require each other.
The ephemeral secret files are gitignored before any exists. The ordering is the point: the moment such a file is worth creating is a moment under deployment pressure, which is the worst moment to be deciding whether it is ignored.
Reproduced against the shipped round-12 checker, with PostHog's staging queue wired to Amplitude's DLQ:
ok: true problems: []
ok e-events-posthog-staging has a dead letter queue (found e-events-amplitude-dlq-staging)The gate printed the wrong name inside its own `ok` line. That is worse than not gathering the evidence: a reader scanning for FAIL sees a green run whose text contradicts it. Both vendors' failure domains terminate in one DLQ there — N27's shared failure domain rebuilt one layer down, in the place N40 built specifically to prevent it.
namesFor() had derived the expected pairing since round 12 and checkQueueConfig() dropped the field when destructuring. The check is positional now, and the two DLQs are additionally required to differ. A DLQ string is also weak evidence on its own: Cloudflare creates a named dead letter queue automatically if it does not exist when a consumer is configured, so seeing a name proves a name was typed, not that the intended topology exists.
List Queues is paginated: per_page defaults to 20, maximum 100, and result_info.total_pages carries the rest. An E queue on page two reads as does not exist — nothing misconfigured, the verifier incomplete. That gets likelier as the account accumulates infrastructure that has nothing to do with E.
per_page is set to the documented maximum and every page is fetched — not one or the other. The maximum alone is a bet that the account stays under 100 queues; the loop alone is correct but pays five times the requests. The loop is bounded independently of total_pages, because a loop bounded only by a number the other side supplies is a loop the other side controls.
The reviewer is right that the operator tooling is not conceptually zero-dependency: it rests on secrets.required, --secrets-file, the named-environment inheritance rules, and a specific command surface.
Adding wrangler as a devDependency would put a large tree behind a battery that currently needs none — npm test runs on a clean checkout, and that is worth keeping. The middle path: record the version, record the surface, check our commands against the surface in the battery, and check the ambient wrangler against the version at provisioning time. A mismatch is a printed note rather than a refusal, because a gate that refused to run on a version bump would be turned off, and a gate that is off catches nothing.
Round 13 adds nothing and removes nothing. Cross-batch event_id reuse, rate limiting, vendor_repair_v1's missing reader, the N31/N36 demotion, FNV-1a's collision resistance, manual site-key issuance, and N63's refusal cost all stand exactly as §14.7 left them.
npm run plan:staging prints it, order-checked, from zero resources:
queues + DLQs -> canonical HTTP pull consumer (--message-retries 100)
-> check:queues:staging MUST BE GREEN -> staging KV -> real resource ids
-> FIRST deploy each Worker WITH --secrets-file
-> controlled synthetic traffic -> force duplicate / outage / lease cases
-> measure ClickHouseThe first thing wanted from that deployment is not a dashboard. It is this table, and the local battery cannot manufacture a single row of it:
| Measurement | Real result |
|---|---|
| cold resume latency | ? |
| active insert time per 2h flush | ? |
| post-insert background activity | ? |
| active minutes/day | ? |
| parts produced per flush | ? |
| compressed bytes/event | ? |
| 1/4/8/16/32 concurrency winner | ? |
| batch-presence probe cost | ? |
| per-event identity probe projected cost | ? |
| actual monthly ClickHouse run rate | ? |
Thirteen rounds and 435 falsifiers have still not produced a reason to replace Workers, Queues, Durable Objects or ClickHouse. Stop reviewing the imaginary staging environment and make it real.
The round-13 reviewer accepted the architecture for the fourteenth time, reran the battery independently (435/435, and 435/435 under the +5-year shift), and then read scripts/staging-plan.mjs as an executable object rather than as a document. Round 13 built that file specifically so an ordering defect would be caught mechanically. It caught N81. It could not catch what was inside it.
Battery: 474 falsifiers, 0 failing — at today's date and five years on. 39 new.
| # | Severity | What it is |
|---|---|---|
| N85 | BLOCKER | the queue gate and the vendor Worker form a bootstrap cycle — and the ordering proof could not see it, because its state tokens were coarser than the state they modelled |
| N86 | BLOCKER | the "MUST BE GREEN" gate exits 0 when it did not run, so the checkpoint was satisfiable by an unset variable |
| N87 | BLOCKER | the plan jumps to a CLICKHOUSE_URL nobody created — and ddl.sql creates a ROLE where CLICKHOUSE_USER needs a USER |
| N88 | HIGH | the gate certifies a vendor queue's DLQ and says nothing about who consumes it, or on what terms |
| N89 | MED | assertWrangler() was named in a comment and never written |
| GUARD | MED | a manual plan step certified itself |
Two more were found while implementing these, and both are recorded below because neither came from review: the "healthy account" fixture three rounds had been proving the green path with (§17.7), and a user name the plan and its generator spelled differently (§17.8).
The through-line is round 13's, one level up: the code has matured faster than the infrastructure bootstrap. Every finding this round is in resource creation, ordering, credentials or verification. None is in E's event semantics. Fourteen rounds and 474 falsifiers have still not produced a reason to replace Workers, Queues, Durable Objects or ClickHouse.
Round 13's order was:
create vendor queues + DLQs -> ... -> FULL queue gate MUST BE GREEN -> deploy cd-e-vendorsA vendor queue's dead letter queue, batch size and retry budget are properties of its worker consumer configuration, and that configuration comes into existence when the consumer Worker is deployed — not when the Queue and DLQ resources are created. So the gate required what only the step after it could provide. Reproduced against the shipped round-13 checker, in exactly the pre-deploy state round 13's own plan creates:
FAIL e-events-amplitude-staging has exactly one dead letter queue (found NONE)
FAIL e-events-amplitude-staging uses ITS OWN ... (found NONE)
FAIL e-events-posthog-staging has exactly one dead letter queue (found NONE)
FAIL e-events-posthog-staging uses ITS OWN ... (found NONE)Four failures on an account with nothing wrong with it. This is N81's shape, inside the file built to make N81 impossible.
validatePlan() missed it because queues.vendors claimed to provide queue:vendor-dlqs, and that token means the DLQ resources exist, not the consumers pointing at them are configured. The model could prove an ordering about its own vocabulary and nothing more. The tokens are per-consumer now: the vendor deploy declares consumer:vendor-posthog and consumer:vendor-amplitude, and the full gate requires them.
The gate has a scope. predeploy certifies everything true before any Worker exists; full adds the vendor consumer topology. cd-e-vendors deploys first, between them — safe for a reason worth stating rather than assuming: nothing is producing messages yet. cd-e-ingest is the only producer and it is deployed two steps later, so a vendor consumer that is live and idle is exactly what is wanted. It exists so the gate can certify it.
And a scoped gate must name the subset it did not cover. predeploy returns a populated deferred list and prints it under a heading that says these are unchecked. Without that, a green pre-deploy run reads exactly like a green full one — which is N86's defect rebuilt inside the gate N86 is about.
$ unset CF_ACCOUNT_ID CF_QUEUE_TOKEN
$ npm run check:queues:staging
E queue configuration check — NOT RUN.
$ echo $?
0So the plan's MUST BE GREEN could not distinguish every invariant holds from I checked nothing. It was especially reachable because round 13's plan had no step that created or exported `CF_QUEUE_TOKEN` at all — Cloudflare requires Queues Read and Write for an HTTP pull consumer, because ACK mutates queue state, so the credential the gate needs was one the plan never provisioned. cloudflare.auth is now step 1 and both gates require it.
A command named check may never return success for "not checked". The informational output was genuinely useful, so it kept its text and lost its verb: --describe prints it and exits 0, check without credentials exits 2. Two questions, two commands, two exit codes — and 2 rather than 1 because a caller must be able to tell the queues are wrong from I could not look.
The first run of the fixed gate exited 1. EXIT_NOT_RUN was declared beside its finding, two hundred lines below the await main() call, so the command whose entire finding is "this must not exit 0 when it did not run" died on
ReferenceError: Cannot access 'EXIT_NOT_RUN' before initializationand exited 1 — still not 0, still wrong, and wrong in the way that reads as the queues are broken. It was caught because the round-14 falsifier spawns the script and reads the real exit code. A falsifier asserting EXIT_NOT_RUN === 2 would have passed against a script that crashed before reaching it. That is N67 once more: the gate's own execution is the thing to test.
Round 13 claimed the plan ran from zero resources. Its config.ids step said to fill CLICKHOUSE_URL with "the ids the previous steps returned", and there was no previous ClickHouse step. The whole plan was queues, DLQs, a pull consumer, a gate, KV, config, secrets, Workers, traffic.
Underneath that, a second one. ddl.sql ends with CREATE ROLE IF NOT EXISTS e_ingest and creates no database user. cd-e-flush authenticates over HTTP Basic as CLICKHOUSE_USER, and a ClickHouse role is not an authenticating principal — it is a bundle of grants that must be granted to a user. cd-e-flush/wrangler.toml made the conflation explicit:
CLICKHOUSE_USER the `e_ingest` role — INSERT only (brief §11)which names a role in the slot that holds a user. So CLICKHOUSE_USER had no value it could be filled with — not a wrong value, no value: the object it named was never created by anything here, and the DDL that should have created it carefully created something adjacent.
Four steps now: clickhouse.service → clickhouse.schema → clickhouse.ingest-user → clickhouse.verified. The user is not created in ddl.sql, deliberately: that file is committed and applied by hand, and a CREATE USER ... IDENTIFIED BY '...' line in it is either a placeholder somebody deploys verbatim or a secret in git. npm run clickhouse:principal:staging generates CREATE USER / GRANT role TO user / ALTER USER ... DEFAULT ROLE with a fresh password, prints it once, and stores nothing. The DEFAULT ROLE line is the easiest to omit and the hardest to diagnose: without it the login succeeds and every insert fails on a privilege the user demonstrably holds.
clickhouse.verified runs the flush's own preflight queries as the real principal, not a list invented for the step — N34 is the finding that a capability verified with a query the flush does not issue is a capability nobody checked. Two of the six read system.*, and whether a non-default ClickHouse Cloud user may do that is not settled here: ClickHouse's grant behaviour on system tables is inconsistent by engine (ClickHouse/ClickHouse#35239) and there is no instance to settle it against. Which is exactly why they are run rather than asserted. Both are marked DEGRADES rather than BLOCKS, because readVersion() and readDedupConfig() catch their errors and return null — a missing capability there does not fail loudly, it removes a check, and an operator who is not told will read silence as success.
N83 taught this gate that has a DLQ < has the right DLQ. The next line was never written. A DLQ says where a failure lands and nothing about who failed. Reproduced against the shipped round-13 checker, with both vendor queues consumed by a Worker that is not ours, on settings that are not ours, with the correct DLQs:
ok: true problems: []A green gate over a topology in which nothing E wrote is running. The settings are not tidiness: max_retries is the vendor leg's entire durability budget and batch_size is what makes a fanout batch legal against the vendor's own limits — N37's lesson, on the vendor leg instead of on retention.
Two spelling traps came with it, and both are N82's shape:
wrangler.toml | Cloudflare API |
|---|---|
max_batch_size = 25 | settings.batch_size = 25 |
max_batch_timeout = 10 | settings.max_wait_time_ms = 10000 — seconds → milliseconds |
max_retries = 5 | settings.max_retries = 5 |
A checker comparing the TOML number to the API number directly reads a correct queue as wrong by a factor of a thousand, and the natural fix for that — loosening the comparison — is how a gate stops being one. The map is written down and a falsifier drives the real TOML through it.
And the consumer's Worker name is itself ambiguous, so it is read as a set. Cloudflare's List Queues consumer object carries script; the Consumers subresource documents script_name; some SDK surfaces nest it under settings. Checked against the API reference 2026-08-24 and not resolved — the documentation genuinely shows more than one and there is no account to settle it against. Every spelling is read, and a consumer from which none resolves is a FAILURE: I could not determine who consumes this queue is not evidence that the right Worker consumes it. When a real account settles the spelling, narrow the list and record the date; do not leave it wide because it works.
Round 13's comment listed the middle path as four things and shipped three. There was no assertWrangler() anywhere in the bundle; the only runtime use of a version was wranglerVersionNote(process.env.WRANGLER_VERSION) — a variable wrangler does not set — inside the no-credentials branch N86 has since established should not have been claiming anything. A function named in a doc comment is prose, exactly as a printed command is. It exists now, with an injected runner so the battery never spawns a process, and it stays a note rather than a refusal: a gate that refused to run on a version bump would be turned off, and a gate that is off catches nothing.
config.ids had no command and provides: ['config:ids'], and validatePlan() treated that as a fact. It is an intention. Some placeholders fail loudly — a bad KV id makes wrangler deploy refuse because the binding cannot resolve — but [vars] are plain strings Wrangler does not validate:
CF_ACCOUNT_ID = "PLACEHOLDER-NOT-A-REAL-ACCOUNT-ID"
E_QUEUE_ID = "PLACEHOLDER-NOT-A-REAL-STAGING-QUEUE-ID"
CLICKHOUSE_URL = "PLACEHOLDER-NOT-A-REAL-STAGING-CLICKHOUSE-HOST"Those three deploy successfully and fail two hours later inside a cron trigger — the worst place for it, because a cron failure is observed by whoever is reading logs rather than by whoever ran the deploy. npm run check:config:staging refuses while any survives, and validatePlan() now rejects any manual step whose output a later step depends on unless a later check verifies it.
The placeholder sweep is scoped to the environment being deployed, and that is a decision rather than an oversight: wrangler deploy --env staging reads [env.staging.*], and production's [vars] are not read by it (N78). A sweep failing on those would make the first staging deployment impossible until production was also filled in — and a gate nobody can pass is a gate that gets removed. They are reported as notes instead.
Not from the review. Found while implementing N85 and N88, and the more uncomfortable of the two self-inflicted findings.
Rounds 9, 12 and 13 each built their own healthyAccount() and used it to assert the green path — assert.deepEqual(r.problems, []), "the correctly wired account is still green". None of the three contained a single dead letter queue as a resource, and none named the Worker consuming a vendor queue. Every DLQ assertion in the gate read a dead_letter_queue string off a consumer, and Cloudflare creates a queue with that name if one is absent — so the evidence was self-fulfilling, and the fixture agreed with the gate about what an account consists of while being the thing used to test the gate.
A fixture that models only what the code already checks cannot falsify the code's coverage — only its logic. That is N74 (a fake that cannot represent the mechanism reports the claim as proven anyway) pointed at the deployment gate's fixture rather than at the flush's fakes. It is also why N88 was reachable by a reviewer and not by the battery: the battery had no way to express the account N88 describes. One builder now, in test/queue-accounts.mjs, with preDeployAccount() alongside it — the state N85 is about, which nothing could previously express either.
Also not from the review — found by reading the first printed plan. The plan titled its step create the e_ingest_worker USER from the bare constant while the generator emitted e_ingest_worker_staging. An operator following the plan would have created a user, been told a different name, and put whichever they trusted into CLICKHOUSE_USER. Derived from one function now. N73's rule about counts, applied to an identifier.
Round 14 adds nothing and removes nothing. Cross-batch event_id reuse, rate limiting, vendor_repair_v1's missing reader, the N31/N36 demotion, FNV-1a's collision resistance, manual site-key issuance, and N63's refusal cost all stand exactly as §14.7 left them.
The events_v1 25-month TTL is still unruled and must be confirmed before the first production event and before any privacy page quotes a figure (§DDL, round 2 N5).
npm run plan:staging prints it, order-checked, from zero resources:
Cloudflare auth (account + Queues READ/WRITE token)
-> queues + DLQs -> canonical HTTP pull consumer (--message-retries 100)
-> CHECKPOINT 1: check:queues --scope predeploy MUST HAVE RUN AND BE GREEN
-> ClickHouse Basic -> ddl.sql -> the ingest USER -> prove the USER can do the flush's work
-> staging KV -> real config values
-> CHECKPOINT: check:config no placeholder survives
-> secret files -> FIRST deploy cd-e-vendors WITH --secrets-file
-> vendor consumers now exist
-> CHECKPOINT 2: check:queues --scope full MUST HAVE RUN AND BE GREEN
-> deploy cd-e-ingest -> deploy cd-e-flush
-> controlled synthetic traffic -> force duplicate / outage / lease cases
-> measure ClickHouseThe table round 13 asked for is unchanged, and the local battery still cannot manufacture a single row of it:
| Measurement | Real result |
|---|---|
| cold resume latency | ? |
| active insert time per 2h flush | ? |
| post-insert background activity | ? |
| active minutes/day | ? |
| parts produced per flush | ? |
| compressed bytes/event | ? |
| 1/4/8/16/32 concurrency winner | ? |
| batch-presence probe cost | ? |
| per-event identity probe projected cost | ? |
| actual monthly ClickHouse run rate | ? |
The reviewer's judgement stands and this round agrees with it: stop reviewing the imaginary staging environment and make it real. What round 14 changes is that the sequence which does that is now one an operator can execute from zero without hitting a cycle, a missing component, or a gate that reports success for having done nothing.
The round-14 reviewer accepted the architecture for the fifteenth time, reran both batteries (474/474, and 474/474 at +5y), and then asked a question round 14 had not: does the evidence each gate produces actually join to the thing it is evidence about?
Battery: 500 falsifiers, 0 failing — at today's date and five years on. 26 new.
| # | Severity | What it is |
|---|---|---|
| N90 | BLOCKER | the ClickHouse principal verifier printed queries that are not the ones the flush runs — and labelled the two most important ones the opposite of what the flush does with them |
| N91 | HIGH | the config gate proves an id's shape; the queue gate holds that id's true value; nothing joined them |
| N92 | BLOCKER | the plan's one "Cloudflare credential" is two different authorities, and the names it exports do not authenticate wrangler at all |
| N93 | HIGH | the gate never proved the runtime token can pull — only that it can list |
| N94 | MED | plan:staging reports success when wrangler cannot run |
Every one of these is the same shape, and it is a different shape from round 14's:
verified query <-/-> actual runtime query (N90)
verified queue <-/-> configured queue id (N91)
operator credential <-/-> operator operation (N92)
runtime token <-/-> runtime capability (N93)
plan success <-/-> executable toolchain (N94)Two true facts, no join between them. Round 14's findings were about what a gate checks; round 15's are about whether the thing checked is the thing used.
N87 was "the plan verifies a component it never created". The fix added a ClickHouse principal verifier and hand-transcribed the flush's queries into it. The hand got it wrong twice.
the flush reads (ASYNC family — ASYNC_INSERT_SETTINGS is in force):
replicated_deduplication_window_for_async_inserts
replicated_deduplication_window_seconds_for_async_inserts
the verifier told the operator to run:
non_replicated_deduplication_window <- INVENTED. In no family at all.
replicated_deduplication_window_seconds <- the SYNC family'sSo an operator could run every printed query, watch them all succeed, deploy, and have the flush refuse every drain. A verifier claiming to exercise the production query was exercising a different query — which is N34's finding exactly, rebuilt by the fix for a different one.
And the classification was backwards on the two that matter. Both were marked DEGRADES, and the generated instructions told the operator in so many words that "nothing downstream will ever complain". proveDedupPrerequisites() refuses the drain on either — supportsAsyncInsertDedup(null) !== true is refusal (a), dedup.blocks === null is refusal (b). The verifier stated the opposite of the truth about the one thing it exists to check. The reviewer is right about why the round-14 falsifier missed it: it checked that each entry named a source and regex-checked that the two readers contained catch. It never compared the SQL, and it never followed those caught nulls into the refusal branch.
The repair is not two corrected strings. dedupConfigSql(), versionSql() and probeCapabilitySql() live in insert.mjs; the runtime readers call them and so does the provisioning printer. A falsifier asserts the strings are identical — not equivalent, since equivalence is what a transcription claims — and captures what the readers actually issue.
The classification is no longer a label anyone types. test/round15.test.mjs drives the real flush with each capability denied and reads stopped_because. All three stop the drain. FakeClickHouse gained deniedQueries to make that expressible: it could model a column-scoped grant on events_v1 and could not model "this user may not read system.merge_tree_settings" — so no test could ask what the flush does in the state N90 is about.
Round 14's own falsifier had pinned the wrong answer and was green, because the code and the test agreed with each other and neither had derived the classification from behaviour. That assertion is replaced, with what it used to say recorded beside it.
The two zero-row INSERT canaries are typed expressions now (SELECT toUUID(...) WHERE 0) rather than SELECT * FROM system.numbers LIMIT 0 — the old form also tested access to system.numbers and a UInt64→UUID coercion, so a failure would not have meant what the operation is named for.
Round 14 was candid in the source: it checks shape and "does NOT prove any id is the right id". That became load-bearing the moment the queue gate started fetching the account. Filled with syntactically valid nonsense, the gate returned {ok: true, problems: []}.
This is worse than a typo. cd-e-flush builds its pull and ACK URLs straight from CF_ACCOUNT_ID and E_QUEUE_ID. A valid id belonging to a different queue in the same account does not fail to start — it makes the flush drain someone else's queue and ack someone else's messages. List Queues returns queue_name and queue_id in the same object, so the join is free.
check:config --bind performs it: account id, canonical queue name → queue id, KV namespace id, and the ClickHouse URL the principal was actually verified against (E_VERIFIED_CLICKHOUSE_URL) — because otherwise an operator can verify service A, configure service B, and every other gate stays green.
Round 14's step 1 said to export CF_ACCOUNT_ID and CF_QUEUE_TOKEN, then run wrangler queues create. Those do not authenticate wrangler. They are E's own variables. Wrangler reads CLOUDFLARE_API_TOKEN / CLOUDFLARE_ACCOUNT_ID or the wrangler login OAuth credential; the CF_-prefixed names are on Cloudflare's deprecated list. On a clean machine the plan stops at step 2, and the name collision makes it read as a permissions problem rather than an authentication one.
The two authorities are also not the same size and must not become so:
OPERATOR — creates queues and KV namespaces, deploys Workers. Stays on the operator's machine. Never a Worker secret.
RUNTIME — Queues Read + Write on the canonical queue. Becomes cd-e-flush's CF_QUEUE_TOKEN and lives at the edge.
The tempting shortcut is widening the runtime token until it can also provision. That puts an estate-deploying credential at the edge — the opposite of what ddl.sql spends forty lines establishing for ClickHouse. Step 1 is now wrangler auth token, verified against the wrangler on this machine rather than against documentation.
Round 15's own falsifier caught two steps I had missed: consumer.pull and all three deploys run wrangler commands and were still requiring no operator authority.
This file already carried the admission, in N86's refusal text: "a read-only token passes this check and then cannot run the pull consumer it just certified." And that is what it did — the gate's entire provider interaction was a GET.
Checkpoint 1 is an unusually safe place to prove otherwise, and the reviewer is right that it is the only safe place: it runs before cd-e-ingest exists, so the canonical queue has no producer. A pull with batch_size: 1 exercises the write-side authorization and consumes nothing. It also proves the queue id, which is N91's join from the other direction.
A message returned at that point is a finding, not a pass — nothing should be producing yet.
N89 ruled that a version mismatch is a note, not a refusal, and that ruling was right: a gate that refuses on a version bump gets turned off. Round 14 then treated a missing executable the same way, printing "nothing below can run until it is installed" and exiting 0. A human reads the warning; an automation reads the number. plan:staging exits 2 now, and --describe prints the sequence anyway — the same split N86 made between check and describe.
Cross-batch event_id reuse, rate limiting, vendor_repair_v1's missing reader, the N31/N36 demotion, FNV-1a's collision resistance, manual site-key issuance, and N63's refusal cost all stand. The events_v1 25-month TTL is still unruled and must be confirmed before the first production event and before any privacy page quotes a figure.
One new open question, and it is a staging question rather than a local one. ClickHouse 26.2 began migrating toward a unified insert-deduplication hash (insert_deduplication_version), while current documentation still describes replicated retry dedup in terms of the main replicated windows. The local model now agrees with itself — the runtime and the verifier read the same settings — but which dedup regime actually governs the ClickHouse Cloud version we are given is not decidable from here. The right move is the one this round's lesson implies:
share the verifier with the runtime locally; discover the provider regime empirically.
Staging should report the exact service version and settings and exercise retry behaviour against them, rather than growing a more elaborate local model of what ought to govern.
wrangler OPERATOR authentication (wrangler auth token)
-> create the RUNTIME Queues token — Read AND Write, and nothing else
-> queues + DLQs -> canonical HTTP pull consumer
-> CHECKPOINT 1: topology + THE TOKEN REALLY PULLS (safe: no producer exists)
-> ClickHouse Basic -> ddl.sql -> the ingest USER
-> prove the USER can run THE FLUSH'S OWN prerequisite queries
-> staging KV -> real config values
-> CHECKPOINT: check:config --bind every id joined to the account
-> secret files -> FIRST deploy cd-e-vendors
-> CHECKPOINT 2: the FULL queue topology, now that the consumers exist
-> deploy cd-e-ingest -> deploy cd-e-flush
-> controlled synthetic traffic -> measure realityThe measurement table is unchanged and still entirely question marks. The first number wanted back is the one the reviewer keeps naming: actual ClickHouse active minutes/day across the twelve two-hour flush windows, including cold resume and post-insert merge activity. That is what tells us whether the sleep/buffer cost thesis works at all.
Fifteen rounds and 500 falsifiers have still not produced a reason to replace Workers, Queues, Durable Objects or ClickHouse. The failure profile has changed completely: nothing this round was about event semantics, and nothing was about what a gate checks. It was all about whether two things that are both true are about the same object.