Version: 1.0 Status: Implementation-ready Last updated: 2026-01-21 Audience: Engineering (primary)
This document defines an end-to-end testing plan and concrete acceptance criteria for implementing webhost.systems from scratch with confidence. It covers unit, integration, and E2E tests across the control plane (auth, CRUD, deploy, billing, limits), data plane (Cloudflare + AgentCore), and integrity-sensitive ingress points (telemetry, billing webhooks).
Normative language: MUST, MUST NOT, SHOULD, MAY.
Confidence over coverage: prioritize tests that prevent high-severity regressions (tenant isolation, billing integrity, secrets leakage, limit enforcement, deploy/invoke correctness).
Defense-in-depth: validate the same invariant at multiple layers where it matters:
auth + ownership checks in every endpoint,
runtime adapter checks (entitlement and state),
telemetry signature verification,
billing webhook signature verification.
A feature is considered complete when:
acceptance criteria in this document are met,
critical paths have automated tests at the appropriate level (unit/integration/E2E),
failure modes are tested (bad input, provider failure, retry/idempotency, unauthorized access),
logs and errors are sanitized (no secrets).
Unit tests (fast, many): pure functions, validators, mappers, cost models, limit checks, signature verification.
Integration tests (medium): control plane functions with real DB (or local emulator), provider adapters against mocked HTTP and/or sandbox accounts, webhook verification.
E2E tests (slow, few): full user flows through UI/API with a real auth session and at least one real runtime environment.
MUST maintain at least:
dev: local development (may use provider mocks)
staging: end-to-end test environment with sandbox provider credentials
prod: production
SHOULD:
Separate credentials per environment (Cloudflare token, AWS credentials, billing webhook secret).
Separate billing provider “store/project” per environment to avoid accidental production entitlements.
MUST provision sandbox accounts for:
Cloudflare (Workers + Durable Objects)
AWS (AgentCore; plus any required secret mechanism)
Billing provider (webhook signature testing)
Maintain a minimal “fixture agent” artifact that:
supports invoke/v1 protocol
returns deterministic output
can optionally create a session and echo the session id
emits a telemetry event (if Pattern A is used)
can be deployed to both providers with minimal changes
includes an AgentCore-targeted variant (or feature flag) suitable for validating TypeScript deployment and tool-enabled invocation scenarios:
Code Interpreter tool invocation path (expected to perform a deterministic computation)
Browser tool invocation path (expected to fetch a deterministic target like example.com and return a stable string such as the page title)
clear, test-detectable markers in output indicating tool usage occurred (e.g., “tool_used:code_interpreter”)
The fixture agent MUST NOT include any real secrets; use ephemeral test secrets.
MUST unit test:
Agent create/update validation:
name constraints
runtimeProvider enum validation
envVarKeys rules
Manifest parsing and validation:
required fields present
protocol supported (invoke/v1)
runtime matches selected runtimeProvider
entrypoint present
Payload limits:
max request size
max messages
max message length
Acceptance:
Invalid inputs consistently return INVALID_REQUEST with field-level issues (or equivalent structured detail).
MUST unit test provider error mapping tables:
Cloudflare deploy errors -> DEPLOYMENT_FAILED
Cloudflare invoke errors -> RUNTIME_ERROR
AgentCore deploy errors -> DEPLOYMENT_FAILED
AgentCore invoke errors -> RUNTIME_ERROR
Test cases MUST cover:
auth/credential failures (retryable=false)
transient provider failures (retryable=true)
invalid config/manifest (retryable=false)
Acceptance:
No raw provider errors are passed through to clients.
Error envelopes always include code, safe message, and retryable.
MUST unit test:
HMAC signature generation matches verification
invalid signatures are rejected
replay protections if implemented (timestamp window, eventId dedupe)
canonicalization rules:
signature is computed over raw request bytes (not re-serialized JSON)
Acceptance:
A single-byte change in payload invalidates signature.
Signature verification is deterministic and constant-time where practical (or uses library that is).
MUST unit test:
signature verification function with known good vectors (valid payload, valid signature)
invalid signature rejection
idempotency handling (same event delivered twice)
Acceptance:
A webhook without valid signature cannot change entitlements.
Duplicate events do not create conflicting subscription state.
MUST unit test:
tier gating for runtimeProvider (AgentCore disabled on lower tiers)
limit checks for requests/tokens/computeMs
boundary conditions:
exactly at limit allowed/blocked as specified
above limit blocked with LIMIT_EXCEEDED
concurrency model assumptions:
if you maintain counters, test atomic increments behavior (at least in a deterministic simulation)
Acceptance:
Limit enforcement decisions are consistent and deterministic given the same inputs.
MUST unit test:
cost estimators for each runtime (Cloudflare, AgentCore):
deterministic given same usage
monotonic: higher usage never yields lower cost
stable rounding rules (avoid drift across aggregates)
Acceptance:
Same event inputs always produce identical costUsdEstimated within strict equality (or within a defined rounding epsilon).
Integration tests verify interactions between:
control plane functions + DB
provider adapters + external APIs (mocked and/or sandbox)
telemetry ingestion + DB + aggregation
billing webhooks + entitlements + gating
MUST test:
Create user on first login (identity mapping)
Agent CRUD:
create/list/get/update/disable/delete (soft delete recommended)
tenant isolation (cannot see others’ agents)
Deployment record creation:
immutable fields remain unchanged
status transitions are correct
versioning monotonic per agent
Active deployment pointer:
set on successful deploy
rollback/activate updates activeDeploymentId
Acceptance:
DB invariants are maintained for all operations, including failure paths.
MUST test:
Accept valid signed telemetry event:
event persisted
ownership validated (userId/agentId/deploymentId consistent)
Reject invalid signature:
event not persisted
audit record created (if audit log implemented)
Reject ownership mismatch (spoof attempt):
event not persisted
Acceptance:
Telemetry endpoint cannot be used to write events for other tenants.
MUST test:
Given a set of raw telemetry events for a period:
aggregation creates/updates billingUsage correctly
per-runtime breakdown correct
repeated aggregation is idempotent (same inputs -> same outputs)
Acceptance:
Aggregation can be rerun without double-counting.
MUST test:
Valid webhook updates subscription tier and entitlements
Invalid webhook rejected
Replay of the same webhook does not double-apply changes
Downgrade behavior (if implemented):
user over limit is blocked on next invoke (or grace period if configured)
Acceptance:
Entitlements change only via verified webhooks.
MUST test adapters with provider API mocks to validate:
correct request shapes sent to providers
correct mapping of provider responses into normalized outputs
correct error normalization
idempotency on deploy (retry same deploymentId)
gating behavior (adapter refuses when not entitled)
Acceptance:
Adapters implement the Runtime Provider Interface contract correctly even when provider behaviors vary.
SHOULD test (at least in staging nightly):
Cloudflare:
deploy fixture agent
invoke stateless
invoke sessionful (sessionId roundtrip)
telemetry emitted and accepted
AgentCore:
deploy fixture agent to AgentCore using the TypeScript AWS SDK (control-plane adapter path)
invoke (with and without session) and verify:
the adapter correctly maps opaque sessionId ↔ provider session identifier
session expiration/unknown session returns a normalized RUNTIME_ERROR with a safe message and retryable=false
telemetry produced (adapter-side or in-workload, depending on design)
tool-enabled invocation tests (only if the deployment enables these capabilities; otherwise skip with a clear reason):
Code Interpreter: prompt asks for a deterministic computation (e.g., fibonacci(10) or sum 1..100) and asserts the expected numeric result appears
Browser tool: prompt asks to visit https://example.com and extract a deterministic string (e.g., page title “Example Domain”)
usage asserts include toolCalls > 0 (or provider-specific equivalent normalized into usage)
Acceptance:
A real deploy+invoke works end-to-end for each provider in staging.
For AgentCore, TypeScript deployment + tool-enabled invocation scenarios pass when the tier and deployment configuration enable them.
E2E tests validate the “it works for a user” flows. These can be implemented via browser automation + API checks or API-only E2E flows with authenticated sessions.
Steps:
Authenticate (create a new user if needed).
Create agent with runtimeProvider=cloudflare and envVarKeys including OPENAI_API_KEY (value not yet set).
Verify agent appears in list and detail view.
Assertions:
Agent created with status created.
Tenant isolation: agent is only visible to the authenticated user.
Steps:
Set secrets for the agent (write-only).
Deploy fixture agent bundle.
Wait/poll until deployment status is active.
Invoke the agent with a simple prompt.
Verify response is correct and deterministic.
Assertions:
Deployment record exists and transitions deploying -> active.
Agent activeDeploymentId set to deployment.
Invocation returns output.text and traceId.
Telemetry event is recorded and attributable to user/agent/deployment.
No secret values appear in logs or responses.
Steps:
After one or more invocations, query usage endpoint for current period.
Query agent metrics series for a time range including invocation.
Assertions:
Usage totals show requests >= number of invocations performed.
Metrics series includes the invocation bucket with tokens/computeMs populated (estimated allowed).
Cost is labeled as estimated (UI/contract semantics).
Steps:
Deploy fixture agent v1.
Deploy fixture agent v2 (different deterministic output).
Invoke and observe v2 output.
Activate v1 deployment (rollback).
Invoke and observe v1 output.
Assertions:
Agent activeDeploymentId updates correctly.
Invocation routing follows active deployment.
Deployment history preserved and immutable.
Setup:
Set an intentionally low requests limit for a test tier, or create a special test user entitlement override.
Steps:
Invoke agent until the limit is reached.
Attempt one additional invocation.
Assertions:
Extra invocation is blocked with LIMIT_EXCEEDED.
Error envelope includes limitType, periodKey, and safe message.
No provider invocation occurs after limit exceeded (verify via mock or provider logs where feasible).
Setup:
User is on a tier with agentcoreEnabled=false.
Steps:
Create agent with runtimeProvider=agentcore (or attempt to change to agentcore).
Attempt to deploy to AgentCore.
Attempt to invoke.
Assertions:
Deploy blocked with LIMIT_EXCEEDED (or a dedicated entitlement error) before provider calls.
Invoke blocked.
UI reflects gating appropriately (nice-to-have, but backend enforcement is authoritative).
Setup:
User on tier with agentcoreEnabled=true.
Steps:
Create agent with runtimeProvider=agentcore.
Deploy fixture agent to AgentCore.
Invoke and verify response.
Verify telemetry recorded with runtimeProvider=agentcore.
Assertions:
Works end-to-end and usage shows up in period totals with correct runtime breakdown.
For each endpoint that accepts agentId/deploymentId:
Attempt to access another user’s resource id.
Assertions:
Returns NOT_FOUND (preferred) or UNAUTHORIZED consistently.
No resource data is leaked (even partial metadata).
Endpoints to include:
agent get/update/disable/delete
deployments list/get/activate/logs
metrics series endpoints
usage endpoints scoped by agent/deployment
Set secrets with distinctive sentinel values.
Trigger common operations: deploy, invoke, errors, metrics.
Search logs, error responses, telemetry events, audit logs, and UI-visible fields.
Assertions:
Sentinel values never appear in:
error messages returned to client,
stored logs/metrics payloads,
audit log metadata.
Send webhook payload with invalid signature.
Attempt to elevate tier.
Assertions:
Tier unchanged.
Response is UNAUTHENTICATED (or equivalent).
An audit log entry is created (optional but recommended).
Send telemetry with invalid signature.
Send telemetry with valid signature but mismatched attribution (agentId from other user).
Send telemetry replay (same signature/payload) if anti-replay implemented.
Assertions:
Invalid signature rejected.
Ownership mismatch rejected.
Replay either deduped or rejected (depending on implementation).
Simulate provider error:
Cloudflare API failure
AgentCore API failure
Assertions:
Deployment status becomes failed.
Agent status becomes error (or remains unchanged if you prefer per-agent status behavior).
Error message is sanitized and actionable.
Retrying deploy with same idempotency key is safe (does not create duplicate deployments or inconsistent versions).
Simulate invoke failure:
provider timeout
runtime exception
tool error (AgentCore tools if applicable)
Assertions:
Invocation returns RUNTIME_ERROR with retryable correctly set.
Telemetry indicates errors=1 and errorClass appropriate.
System does not crash and does not leak provider internals.
Simulate telemetry ingestion temporarily down.
For Pattern A (runtime emits), telemetry call fails.
Assertions:
Invocation still returns success to client.
The system records an internal error or audit event for telemetry failure (if designed).
No secret leakage occurs.
(Optional) Retry behavior is bounded and does not cause runaway loops.
Concurrent invocations at/near limit boundary.
Concurrent deploy attempts on the same agent.
Assertions:
Limit checks behave conservatively; small overruns may be tolerated but must not be systematic.
Concurrent deploy is prevented or fails with CONFLICT.
Deployment versions remain monotonic and unique.
Measure p50/p95 for:
auth + routing overhead
provider invoke time (separately where possible)
Acceptance target (initial):
control-plane routing overhead small relative to provider time; track as a metric rather than hard-failing builds.
Burst telemetry ingestion tests (e.g., N events/sec).
Verify DB write capacity and aggregation staleness under load.
Acceptance:
Telemetry ingestion remains stable or degrades gracefully (429/retryable) without corrupting data.
A release to production MUST NOT proceed unless:
Tenant isolation tests pass for all relevant endpoints.
Webhook verification tests pass and no unsigned webhook can change tier.
Telemetry signature verification tests pass and spoofed telemetry is rejected.
Secrets leakage tests pass (no sentinel secret appears anywhere it shouldn’t).
Deploy+invoke E2E passes on Cloudflare in staging.
Limit enforcement E2E passes (requests limit at minimum).
Rollback E2E passes.
Error normalization is consistent (no raw provider errors in client responses).
If AgentCore is enabled in production:
AgentCore staging E2E deploy+invoke MUST pass.
The platform is considered v1-complete when all of the following are true:
A user can authenticate, create an agent, set secrets (write-only), deploy to Cloudflare, invoke it, and see usage/metrics in the dashboard.
Deployments are recorded as immutable versions.
An agent has a single active deployment pointer.
Rollback updates the pointer and changes invocation routing immediately.
Requests limits are enforced at invocation time and block further invocations when exceeded.
AgentCore deploy/invoke is gated by tier entitlements and cannot be bypassed by direct API calls.
Every invocation produces a telemetry event (or is accounted for deterministically).
Telemetry ingestion verifies signatures and ownership.
Billing webhooks verify signatures and are idempotent.
No plaintext secrets are stored in the primary DB.
No secret values appear in logs, telemetry, error responses, or audit metadata.
Provider failures do not corrupt state.
Errors are normalized and sanitized.
Retries are safe for deploy and telemetry ingestion.
Validation: agents, manifest, invoke payload
Error mapping: provider -> normalized
Signature verify: telemetry and billing
Limit checks: thresholds and boundary cases
Cost estimation: determinism + monotonicity
CRUD + tenant isolation
Deploy record + active pointer
Telemetry ingestion + ownership cross-check
Aggregation idempotency
Webhook signature + idempotency
Adapter with mocks: deploy/invoke shapes + error mapping
Create agent -> set secrets -> deploy -> invoke -> view usage
Limit enforcement blocks
Rollback changes routing
AgentCore deploy+invoke (entitled user)
Streaming invoke (if implemented)