Version: 1.0 Status: Implementation-ready Last updated: 2026-01-21 Audience: Engineering (primary)
This document specifies how webhost.systems captures observability data, performs usage metering, estimates cost, defines subscription tiers, and enforces limits at deploy and invoke time.
Normative language: MUST, MUST NOT, SHOULD, MAY.
The platform MUST:
Attribute all billable usage to {userId, agentId, deploymentId, runtimeProvider, timestamp}.
Capture enough telemetry to:
enforce plan limits,
display usage in the dashboard,
compute estimated cost (per runtime and total).
Prevent telemetry spoofing and billing manipulation.
Enforce limits before incurring provider cost whenever possible (pre-checks at invocation gateway).
Provide operational observability:
deployment outcomes,
invocation success/error rates,
latency and availability tracking,
basic tracing with traceId.
Perfect cost reconciliation against provider invoices (v1 uses estimated costs).
Full distributed tracing across all provider internals (v1 uses correlation IDs + structured events).
Public invocations with API keys (may be post-v1; requirements here assume authenticated invocations).
requests: count of invocations.
tokens: LLM tokens used (provider-reported preferred, otherwise estimated).
computeMs: compute time for the invocation (wall time or billed time, depending on provider; must be consistent per provider).
costUsdEstimated: estimated USD cost (deterministic given the event inputs).
billing period: the usage accounting window used for limits and billing UX.
v1 recommendation: calendar month (YYYY-MM) for simplicity.
retention window: time horizon raw telemetry and logs are kept (tier-based).
Raw telemetry events (append-only) are the source of truth for usage.
Aggregates (billingUsage) are derived views and may be recomputed.
cloudflare: Cloudflare Workers + Durable Objects (default tier).
TypeScript-first: Workers and Durable Objects are commonly developed and deployed in TypeScript.
agentcore: AWS Bedrock AgentCore (premium/enterprise tier).
TypeScript-first: AgentCore supports TypeScript end-to-end for management and invocation (e.g., via AWS SDK clients) and has a TypeScript tools ecosystem (e.g., Code Interpreter and Browser tooling integrations).
Note: tool availability and “built-in services” are provider features; webhost.systems should expose them as runtime capabilities and gate them by tier where appropriate.
At minimum, the system MUST collect:
Invocations:
requests
errors
errorClass (normalized)
runtimeProvider
Usage:
llmTokens (reported or estimated)
computeMs
provider-specific counters (optional but recommended)
Cost:
costUsdEstimated (per event, then aggregated)
v1 MUST provide:
deployment logs: status transitions + sanitized error messages.
invocation errors: last N error summaries per agent/deployment (may be derived from telemetry events).
v1 SHOULD provide:
structured logs with:
timestamp
level
traceId
agentId/deploymentId
message
an external log store pointer (logsRef) rather than storing large logs in the primary DB.
Every invocation MUST have a traceId generated by the invocation gateway.
The traceId SHOULD be propagated to the runtime provider (headers/metadata where possible).
The traceId MUST be included in telemetry events whenever possible.
Telemetry events MUST be emitted per invocation. The canonical event shape (logical) is:
attribution:
userId
agentId
deploymentId (SHOULD be present; MAY be null only for transitional legacy paths)
runtimeProvider
time:
timestamp (invocation end time)
ingestion time (server adds)
counters:
requests (usually 1)
llmTokens
computeMs
errors (0/1 typical)
errorClass (optional)
provider details (optional, runtime-specific)
costUsdEstimated
traceId (optional but strongly recommended)
Security requirements for telemetry authenticity and anti-spoofing are defined in:
40_SECURITY_SECRETS_COMPLIANCE.md (telemetry signing)
10_API_CONTRACTS.md (telemetry ingestion endpoint contract)
Each runtime provider MUST choose one emission pattern (or both) and be consistent:
Pattern A — In-workload emission (preferred for Cloudflare DO):
After completing invocation, the Worker/DO signs and sends the telemetry event to the control plane ingestion endpoint.
Advantages: captures compute timing and provider-specific counters from within the runtime.
Constraint: requires outbound call to control plane.
Pattern B — Adapter-side emission (fallback / interim):
After receiving provider response, the control plane runtime adapter emits telemetry using the information returned by the provider.
Advantages: simpler runtime code.
Constraint: may miss provider-specific counters; must not undercount usage.
On telemetry ingestion, the control plane MUST:
Verify the request is authentic (signature / secret per deployment).
Validate required fields and enums.
Cross-check ownership:
deploymentId belongs to agentId,
agentId belongs to userId.
Persist the event as append-only.
Update incremental counters (optional optimization) OR rely on scheduled aggregation.
Telemetry ingestion SHOULD support deduplication to reduce double-counting from retries:
Preferred: include eventId generated by runtime and store it for a short dedupe window.
Alternative: dedupe using (deploymentId, traceId) if traceId uniqueness is guaranteed per invocation.
If implementing dedupe, do not reject legitimate distinct events.
If telemetry ingestion is temporarily unavailable:
Runtime SHOULD retry with exponential backoff (bounded).
Runtime SHOULD not block user responses indefinitely while retrying telemetry.
If telemetry fails entirely, invocation still succeeds, but usage may be undercounted; this is acceptable only if:
the system logs the telemetry failure internally,
the platform tracks “telemetry drop rate” as an internal health metric,
you add a mitigation plan post-v1 (buffering/queue).
The system MUST produce aggregates:
per user per billing period:
totals: requests, tokens, computeMs, costUsdEstimated
breakdown by runtimeProvider
optionally per agent per time bucket (hour/day) for dashboard charts
v1 supports either (choose one; both acceptable):
Strategy 1 — Scheduled recompute (simple):
Periodically scan raw telemetry for a user and update billingUsage.
Pros: simplest correctness model.
Cons: expensive at scale without partitioning.
Strategy 2 — Incremental counters + periodic reconciliation (recommended):
On telemetry ingestion, increment per-period counters (and optionally per-day counters).
A scheduled job periodically reconciles by scanning raw events for the last N hours/days to correct drift.
Pros: fast reads for limit checks; scalable.
Cons: more implementation complexity.
Dashboard usage views MAY be eventually consistent (minutes).
Limit enforcement MUST be conservative (fail closed when uncertain) OR use a “soft buffer” (see §8.6) to reduce false blocks.
Cost is estimated until reconciled with provider billing exports.
The estimator MUST be deterministic given event inputs.
Each telemetry event MUST include costUsdEstimated (even if 0), to avoid complex recomputation on reads.
If provider returns tokens, use them. If not, estimate tokens using a deterministic tokenizer approximation:
v1 recommendation: a heuristic based on character count (documented as estimate), or a stable tokenizer library if available in your runtime.
Keep the estimator consistent across invocations to maintain predictable limits.
Cloudflare cost may include:
Workers CPU / duration (depending on plan)
Durable Objects compute and storage operations
Workers AI calls (if used)
Egress (if relevant; usually excluded from v1 estimate)
v1 approach:
Define a CloudflareCostModel with configurable constants, e.g.:
USD_PER_MILLION_REQUESTS
USD_PER_GB_SECOND (if using duration-based billing)
USD_PER_DO_OP (optional)
USD_PER_WORKERS_AI_CALL (optional)
Compute costUsdEstimated = f(requests, computeMs, provider.cloudflare.*)
Important:
If you cannot confidently model Cloudflare components, v1 MAY set Cloudflare cost estimate to:
token-based (if you use a model with token cost), plus
a small fixed overhead per request,
and label clearly as estimate.
AgentCore cost may include (depending on how you configure and bill the runtime):
runtime session duration (or billed runtime execution time)
compute resources (vCPU/memory shape, if applicable)
tool usage (e.g., code execution and browser automation), which may be:
billed implicitly as runtime execution time, and/or
billed explicitly per tool invocation (provider-dependent; model only if applicable)
token/model usage (if the platform provides models; BYOK token cost is informational)
v1 approach:
Define an AgentCoreCostModel with configurable constants (do not hardcode pricing into logic):
USD_PER_SESSION_MS or USD_PER_HOUR_BY_INSTANCE_SHAPE
USD_PER_TOOL_INVOCATION (optional; only if the provider bills tools separately)
optional token pricing constants only for platform-provided model billing (not BYOK)
Compute costUsdEstimated = f(tokens, sessionDurationMs, toolInvocations, instanceShape) using whichever inputs are available from telemetry.
Implementation note (TypeScript): AgentCore is TypeScript-capable end-to-end; the adapter can capture and normalize tool/session usage surfaced by the runtime/tooling SDKs into the telemetry schema.
There are two modes:
Mode A — BYOK (Bring your own key):
The platform does not pay the model provider; token cost is informational only.
Limits may still be enforced by tokens to protect platform compute or to provide guardrails.
Cost estimate SHOULD reflect platform costs only (runtime compute, tools), not customer’s model invoice.
Mode B — Platform-provided model billing (post-v1 or optional):
Token cost becomes part of platform cost estimate and billing.
This requires accurate token counting and potentially provider reconciliation.
v1 recommendation:
Start with BYOK + platform runtime costs (as estimates).
Keep the schema ready for Mode B.
free
starter
pro
enterprise
Each tier defines:
maxRequestsPerPeriod
maxTokensPerPeriod (reported or estimated)
maxComputeMsPerPeriod
agentcoreEnabled (boolean)
These flags control which AgentCore features the platform is allowed to enable for a user’s deployments:
memoryEnabled (boolean)
codeInterpreterEnabled (boolean)
browserEnabled (boolean)
Notes:
These are entitlements, not per-request knobs. The control plane may still choose to disable a capability for a specific deployment even when entitled, but MUST NOT enable a capability when not entitled.
These flags SHOULD be reflected in the data model (agent/providerConfig and deployment/providerRef) to support auditing, debugging, and consistent enforcement.
If you enable AgentCore tools for any tier, the tier SHOULD also define explicit monthly quotas to prevent surprise costs:
maxToolCallsPerPeriod (total tool invocations across code interpreter + browser + any future tools)
maxCodeExecutionSecondsPerPeriod (time spent in code execution sandbox)
maxBrowserSessionsPerPeriod (count of distinct browser tool sessions)
If these quotas are not implemented in v1:
the platform MUST still emit telemetry with tool usage fields when available, and
the platform SHOULD gate tool enablement to only the highest tier to reduce risk.
telemetryRetentionDays
logsRetentionDays
maxAgents
maxDeploymentsPerAgent
maxConcurrentInvocations
any other operational caps required for abuse prevention and cost control
invocation limits per billing period:
maxRequests
maxTokens
maxComputeMs
retention:
raw telemetry retention days
logs retention days
runtime access:
whether AgentCore runtime is enabled
optional: maxAgents, maxDeployments, maxEnvVars, etc.
The system MUST implement entitlements as configuration, not hardcoded literals.
Example entitlement table (illustrative; not pricing commitment):
free:
maxRequests: X
maxTokens: Y
maxComputeMs: Z
agentcoreEnabled: false
memoryEnabled: false
codeInterpreterEnabled: false
browserEnabled: false
telemetryRetentionDays: 7
logsRetentionDays: 7
starter:
higher limits
agentcoreEnabled: false (or limited, if you explicitly want a small allocation)
memoryEnabled: false
codeInterpreterEnabled: false
browserEnabled: false
telemetryRetentionDays: 14
logsRetentionDays: 14
pro:
higher limits
agentcoreEnabled: optional (platform choice; if enabled, tools still SHOULD remain off by default in v1)
memoryEnabled: optional (only meaningful if agentcoreEnabled=true)
codeInterpreterEnabled: false (recommended v1)
browserEnabled: false (recommended v1)
telemetryRetentionDays: 30
logsRetentionDays: 30
enterprise:
custom limits
agentcoreEnabled: true
memoryEnabled: true
codeInterpreterEnabled: true
browserEnabled: true
maxToolCallsPerPeriod: <set value>
maxCodeExecutionSecondsPerPeriod: <set value>
maxBrowserSessionsPerPeriod: <set value>
telemetryRetentionDays: 90
logsRetentionDays: 90
Deployments to agentcore MUST be rejected when agentcoreEnabled=false.
Invocations routed to agentcore MUST be rejected when agentcoreEnabled=false (defense in depth).
UI MUST hide or disable AgentCore options when not entitled, but backend enforcement is authoritative.
Capability gating (AgentCore):
If memoryEnabled=false, the control plane MUST NOT enable AgentCore memory features for any deployment, and MUST reject deployments that request/require memory.
If codeInterpreterEnabled=false, the control plane MUST NOT enable Code Interpreter for any deployment, and MUST reject deployments that request/require it.
If browserEnabled=false, the control plane MUST NOT enable Browser tooling for any deployment, and MUST reject deployments that request/require it.
Tool quota enforcement (if tool entitlements exist):
If a tier defines maxToolCallsPerPeriod / maxCodeExecutionSecondsPerPeriod / maxBrowserSessionsPerPeriod, the invocation path MUST enforce these limits (preferably pre-invocation; otherwise block subsequent invocations once exceeded).
Telemetry SHOULD include tool usage counters so quotas can be enforced and shown in the dashboard.
When a user upgrades/downgrades:
Entitlements MUST update via verified billing webhook (or server-side admin action).
Limit checks MUST use the current tier entitlements effective at time of invocation.
If user is downgraded below current usage:
The platform MUST enforce limits immediately (block further invocations) OR
Allow a grace period (explicitly configured). v1 recommendation: block immediately to keep implementation simple.
To avoid “false blocks” due to telemetry latency:
Implement a small buffer margin per limit (e.g., allow +1% or +N requests) and show it as “pending usage”.
OR enforce limits using a fast incremental counter updated at request time (preferred).
If you implement buffering:
Apply the same policy consistently across tiers.
Ensure it cannot be exploited for unlimited usage.
Limits MUST be enforced at:
Deploy time:
runtime gating (AgentCore enabled),
optional limits on number of agents/deployments.
Invoke time (primary cost control):
check budgets before calling provider runtime,
reject with LIMIT_EXCEEDED when exceeded.
At minimum, invocation gateway MUST enforce:
requests budget
tokens budget (estimated or reported)
computeMs budget (estimated)
Important nuance:
You cannot know actual tokens/computeMs before invocation.
Therefore v1 MUST implement predictive reservation or post-charge approach:
Option A — Conservative pre-reservation (recommended):
Compute a worst-case or predicted cost for the request:
reserve requests += 1
reserve tokens += predictedTokens
reserve computeMs += predictedComputeMs
If budget would exceed, reject before provider call.
After invocation, reconcile with actual usage (if actual < reserved, release difference; if actual > reserved, charge difference).
Option B — Request-only pre-check + post-charge (acceptable for MVP):
Pre-check only requests limit before provider call.
After invocation, post telemetry updates tokens/compute.
If tokens/compute exceed limits, block subsequent requests.
Risk: a user can exceed token/compute budget within a single request.
v1 recommendation:
Implement Option B initially (simpler), then upgrade to Option A as you scale/care about cost precision.
If you allow AgentCore (potentially expensive), prefer Option A at least for AgentCore tiers.
At invocation start:
Resolve user and agent; ensure agent is active and has an active deployment.
Load entitlements for user tier.
Load current period usage totals for user (from billingUsage or incremental counters).
If any enforced limit is already exceeded:
reject immediately with LIMIT_EXCEEDED.
Proceed to runtime invocation.
After invocation:
Ensure telemetry event is recorded.
If you maintain incremental counters:
update counters atomically (idempotency-aware).
If user exceeded limits post-charge:
mark user/account state such that subsequent invocations are blocked.
Because invocations may be concurrent, enforcement MUST address race conditions.
Minimum acceptable v1 behavior:
Enforce with best-effort and accept small overruns under concurrency (documented).
Ensure you never undercount systematically.
Recommended behavior:
Maintain a per-user per-period counter with atomic increments (or transactional update) for requests.
If implementing token/compute pre-reservations, those counters must also be atomic.
If the usage store is unavailable (cannot read entitlements/usage):
The system SHOULD fail closed for expensive runtimes (AgentCore) and MAY fail open for low-cost runtimes (Cloudflare free tier) depending on risk tolerance.
v1 recommendation: fail closed for all invocations to avoid surprise costs and inconsistent state.
Invocation gateway MUST return a normalized error envelope:
code: LIMIT_EXCEEDED
message safe for display
details include:
limitType (requests/tokens/computeMs)
periodKey
current and limit values (if known)
suggested action (upgrade tier)
Dashboard MUST show:
Current tier and billing period usage:
totals: requests, tokens, computeMs, estimated cost
per-runtime breakdown
percent of limits used
Per-agent metrics:
requests and errors over time
last deployed time
active deployment version
Deployment history:
statuses, timestamps, error messages
Recent errors list:
timestamp, traceId, error class, runtime provider
For charting:
Provide server-side bucketing into minute | hour | day.
For each bucket, compute:
requests sum
tokens sum
computeMs sum
errors sum
costUsdEstimated sum
UI MUST label costs as:
“Estimated cost” unless reconciled.
If you later add reconciliation:
show both estimated and reconciled values and explain difference.
The system MUST implement retention for:
raw telemetry events
logs (deployment and invocation logs)
Recommended minimum:
free: 7 days
starter: 14 days
pro: 30 days
enterprise: 90+ days
Retention MUST be implemented as a server-side scheduled job that:
deletes raw telemetry events older than retention cutoff per user tier (or per event bucket policy),
deletes logs older than retention cutoff,
preserves aggregated billing usage for at least 13 months (recommended) irrespective of tier.
When an agent is deleted (soft delete recommended):
new invocations MUST be blocked immediately.
existing telemetry and aggregates MAY remain until retention sweeps remove them (or be removed immediately if you implement stronger deletion guarantees).
If a user requests full deletion (post-v1), you need a stronger data deletion workflow.
Estimated costs are good enough for limits and UX initially, but billing requires reconciliation if you charge based on actual provider spend.
Periodically ingest provider billing exports:
Cloudflare billing export (if available)
AWS cost and usage reports
Map provider line items back to {deploymentId, userId} using:
provider tags/labels on resources,
naming conventions,
per-deployment resource references.
Store reconciled cost values in:
a new costReconciliation table, OR
additional fields in billingUsage (e.g., costUsdReconciled).
To enable reconciliation later, v1 MUST:
tag all provider resources with:
userId
agentId
deploymentId
store provider resource identifiers (providerRef) on deployments.
The control plane SHOULD track (internal-only):
telemetry ingestion success rate and latency
telemetry drop rate (emit attempt failed)
invocation gateway latency (p50/p95/p99) by runtime provider
deployment success rate and deploy duration
error rates by errorClass and runtime provider
limit-exceeded rate (helps tune entitlements)
[ ] Telemetry ingestion validates signature and ownership.
[ ] Telemetry events persisted append-only with required attribution fields.
[ ] Billing usage aggregates exist per user per period with per-runtime breakdown.
[ ] Invocation gateway enforces tier gating and limits (at least requests; preferably more).
[ ] Runtime provider resources are tagged with userId/agentId/deploymentId.
[ ] UI shows usage totals and limits with clear “estimated” labeling.
[ ] Retention job deletes raw telemetry and logs per tier.
[ ] Idempotent telemetry ingestion (eventId or traceId dedupe).
[ ] Incremental counters for fast limit checks under concurrency.
[ ] Streaming invocation includes traceId and emits final usage summary.
[ ] Alerting hooks for abnormal error rate or telemetry drop rate.
[ ] MUST NOT store plaintext secrets in telemetry events or logs.
[ ] MUST NOT accept telemetry without integrity verification.
[ ] MUST NOT allow clients to set or modify subscription tier directly.
This module is v1-complete when:
A successful invocation produces exactly one persisted telemetry event (or deduped as one logical event) with correct attribution.
The dashboard shows current-period totals and per-runtime breakdown within an acceptable delay (minutes).
Invocations are blocked when the user exceeds configured limits, returning LIMIT_EXCEEDED.
AgentCore deploy/invoke is blocked when the user is not entitled.
Telemetry ingestion rejects spoofed events (invalid signature) and mismatched ownership.
Retention sweeps remove raw telemetry and logs beyond tier limits while keeping aggregates for longer-term billing reporting.