Production coverage: what of it is actually here¶
A list of production-engineering concepts — rate limiting, circuit breakers, Kubernetes, backpressure, exactly-once, p99, on-call — reads as a checklist. This document is the honest answer to it: for every concept, what exists in this repository, where, and what the evidence for that claim is.
It exists because the alternative is worse. A README that says "production-grade" tells you nothing; a matrix of green ticks tells you less. What follows is a map with a citation in every cell, so that a reader can check any line and find either the code and its tests, or a sentence saying plainly that the thing is not here.
Scope of the claims in this file. Every cell was written against the tree at commit c04fe0e,
re-checked after the deployment, load-measurement and refusal-reader work that landed on top of it,
and verified by reading the named path (the table was produced alongside a repository-wide search for
each term, and any cell whose path did not exist would have been a false claim). Measured cells
name the artefact that holds the measurement — for the performance cells that is
docs/measurements/2026-10-15-2core/, which keeps the raw
reports so a number here can be checked against the run that produced it. Cells describe this
repository as it is, not a deployment, not a product, and not a promise. Where an artefact cannot be
executed in the build environment, the cell says so instead of implying a run.
Every term the brief named is answered, and that is checked mechanically. The vocabulary these
maps were written against is 130 terms — 51 testing kinds and 79 production-engineering concepts —
and each one is answered in one of the three maps, found by a search rather than by eye: a term
answered nowhere is a gap in the documents, not a judgement call. This map answers them in 77 rows,
because a row can cover more than one term (TCP and UDP share one, as do index and query
optimization). What the search proves is the weakest claim in this file — that a reader who looks
for the term lands on a cell — and the cell's own words decide the rest, because "answered" includes
out of scope and not applicable, spelled out.
The other two maps. This one is the mechanism axis. Testing is the taxonomy axis — 44 kinds of testing, and what is deliberately not covered. Resilience practice coverage is the practice axis — 22 SRE practices (stress, fuzzing, chaos monkey, game days, red teaming, FMEA, rejuvenation, post-mortems) and which of them this repository actually practises. Standards is the source axis — the RFC, paper or canonical practice behind a mechanism, and the places this project deliberately departs from it.
How to read a cell¶
| status | meaning | what you should conclude |
|---|---|---|
| built | The mechanism exists in src/docket/ and is exercised by the suite (unit tests, contract tests, the game-day phases, or CI). The cell names the file and, where useful, the test. |
You can run it today and expect the named behaviour. |
| measured | Docket does not implement the mechanism, but records a decision, a number or an outcome about it into the ledger or the receipt. | You get evidence, not enforcement. |
| modelled | A validated artefact exists in deploy/ — a container recipe, a manifest, a chart — and tools/deploy_check.py checks it statically. The chart has additionally been rendered by Helm itself (v3.16.3, 2026-10-15) with the two renders compared; nothing in this environment can build a container or apply a manifest to a cluster, so no cell in this row claims a deployment. |
You have a starting point with known static guarantees, and the honest list of what is untested. |
| reasoned | The concept matters to Docket's users and it is addressed in writing — an ADR, a threat-model entry, a "not covered, and why" bullet — with the boundary stated. | Read the referenced document and decide whether the boundary fits you. |
| out of scope | Deliberately not attempted. Belongs to another layer (a broker, a cluster, a CDN) or would contradict what Docket is. | Don't look for it here; the one-line reason tells you where it does live. |
A cell is never "partly built". Where a mechanism works in one direction and not another — SSE is received but not streamed, latency is measured but not under load — the cell says exactly that.
1. Infrastructure and deployment¶
| concept | status | where it lives | what that means here |
|---|---|---|---|
| Rate limiting | built | src/docket/resilience.py (RateLimiter: token bucket + sliding 60 s window), policy keys max_calls_per_second / max_calls_per_minute, Retry-After on the 429 (src/docket/proxy.py:590), tests tests/test_resilience.py, tests/test_invariants.py::test_the_boundary_records_the_retry_hint_it_sent, game-day phase rate-limit |
The boundary throttles the agent, not the world: a looping agent is refused at the boundary and the refusal is evidence. Retry-After is a real header, and the reference agent's backoff mode honours it. |
| Caching | built (two different things, both visible) | src/docket/cache.py (ResponseCache, LRU + TTL + a hit ceiling), policy [cache], CLI --cache / --cache-ttl / --cache-max-entries / --cache-max-hits, event model.cache_hit, receipt member cache, three verifier checks, docs/caching.md, tests/test_cache.py (74 tests), a measured load point in docs/measurements/2026-10-15-2core/cache-report.json; and the provider-side kind: src/docket/pricing.py quotes the per-model cached_input rate and explain.py's cache-miss rule prices the repeats that were not cached |
Two caches, deliberately not confused: the boundary answers an identical request without contacting the provider (every hit an event, the saving recomputed by docket verify from the origin it replays), while provider prompt caching is a discount on tokens and stays where it belongs — usage.cached_input_tokens in the rows. Off unless the policy says otherwise, because a recorder that changes what your agent receives uninvited is not a recorder you can trust. The named hazard is a cached loop that spends nothing (no dollar cap can end it), which is why max_hits exists as a brake on the cache's own answers. |
| Load balancing | out of scope | — | The boundary forwards to one base URL per provider (DEFAULT_UPSTREAMS, 11 providers plus Ollama). A pool in front of providers would change which account answered a call without changing the receipt, which is the sort of invisible decision this tool exists to make visible. |
| Reverse proxies | built | src/docket/proxy.py (RecordingProxy, absolute-form routing, do_CONNECT tunnelling in --http-proxy mode, util.local_http_url loopback guard), docs/integrations.md#agents-that-ignore-base-urls-http-proxy, tests/test_security.py |
It is a reverse proxy with one opinionated job: record and govern model traffic. A request in absolute form is routed by host; CONNECT is allowed or denied by the egress policy and recorded either way. |
| API gateways | reasoned | docs/comparison.md#why-not-just-build-this-into-a-gateway |
The honest position: a corporate gateway already enforces budgets for key-based traffic; a large share of 2026 agent usage runs on subscriptions with no key to route. Docket sits where the agent runs, works without a key, and produces something a third party can verify — three things a gateway does not do. |
| CI/CD | built | .github/workflows/ci.yml (lint + bandit, tests on Python 3.11/3.12/3.13, Node conformance, a package job that installs the built wheel into a clean venv and runs it), .github/workflows/release.yml, docs.yml, .github/dependabot.yml, Makefile |
The suite, the independent verifier, the committed vectors and the built artefact are all checked on every push. The package job is the one that catches "works from the source tree, not from the wheel". |
| Docker | modelled | deploy/Dockerfile, deploy/compose.yaml, tools/deploy_check.py, tests/test_deploy_check.py, docs/deployment.md |
A two-stage, non-root, read-only-root-friendly image with no credentials baked in and STOPSIGNAL SIGTERM (so a replaced container closes its receipt), plus a three-service compose smoke test whose verifier waits for the run and mounts evidence read-only. Not built or run here: no container runtime exists in this environment; the checker validates the recipe statically, and its own tests break every invariant on purpose. |
| Kubernetes | modelled | deploy/kubernetes/ (Deployment, Service, PVC, ConfigMap, NetworkPolicy), docs/integrations.md#kubernetes--containers |
The documented pattern is one pod: agent and boundary side by side, sharing the workspace and loopback. The manifests encode the hardening this suite can check statically (non-root, no privilege escalation, read-only root, resource limits, probes). Never applied to a cluster here. |
| Service discovery | out of scope | — | Docket's own clients are loopback-only by construction (util.local_http_url refuses a non-loopback address, and 0.0.0.0 is not accepted as a destination), so there is nothing to discover: a recorder that resolves names can be aimed at a network destination by an environment variable. |
2. Backend and distributed systems¶
| concept | status | where it lives | what that means here |
|---|---|---|---|
| Circuit breakers | built | src/docket/resilience.py (CircuitBreaker: threshold, doubling cooldown, half-open), policy keys breaker_threshold / breaker_cooldown_seconds / breaker_max_cooldown_seconds, docket_breaker_state / docket_breaker_open metrics, game-day phase circuit-breaker |
Stops hammering an upstream that is failing, refuses by name, and records the opening and the state transitions. The audit for that phase reads the ledger, not the intention. |
| Timeouts | built | upstream_timeout on the boundary (src/docket/proxy.py, default 300 s), Caps.max_wall_seconds, the reference agent's per-call timeout, docs/policy-reference.md |
Three different timeouts for three different things: the upstream read, the run's wall clock, and the agent's own call. The first is recorded as model.error, the second ends the run with exit 42 and breached, the third is the agent's business. |
| Retries | measured | every attempt is its own model.request event — tests/test_invariants.py::test_a_replayed_request_is_recorded_as_two_calls_and_never_deduplicated; Retry-After on refusals (src/docket/proxy.py); finding rule retry-storm |
Docket does not retry for the agent and does not collapse retries in the record. Two identical calls are two events with two sequence numbers, because a deduplicating evidence tool reports a run that did not happen. retry-storm is the finding that names the pattern. |
| Exponential backoff | built | DOCKET_AGENT_MODE=backoff in src/docket/scriptagent.py, DOCKET_AGENT_MAX_SLEEP, DOCKET_AGENT_SEED, tests in tests/test_invariants.py |
In the reference agent, not in the boundary: full jitter under a cap, doubling per refusal, Retry-After honoured, and the ceiling plus the hint written into the ledger as a note event. Capped because an agent that sleeps for an hour is a hung agent. |
| Idempotency | reasoned | ADR docs/adr/0005-concurrency-and-one-writer-at-a-time.md; for restores, src/docket/snapshot.py + tests/test_snapshot_restore.py (applying twice is a no-op) |
Two answers, deliberately different. Evidence is never idempotent: a duplicate call is recorded twice, because the duplicate is the finding. Restores are idempotent: the same plan applied twice leaves the same bytes and says so. |
| Message queues | out of scope | — | A broker is a different boundary from the one Docket records. Docket has no queue, no consumer group, no delivery semantics to document, and adding one would not make a single receipt more verifiable. |
| Pub/sub | out of scope | — | Same reason. The nearest real thing is the boundary's own event endpoints (/docket/<kind>: /docket/note, /docket/decision, /docket/model-call) — ingestion, not subscription. |
| Event-driven architecture | measured | src/docket/sdk.py and the HTTP recording endpoints, docs/integrations.md#any-language--the-http-recording-endpoints, the event vocabulary in src/docket/events.py |
An agent (or any language beside it) can post step-level events — decisions, tool calls, its own model calls made outside the boundary — and they land in the same hash-chained ledger with their provenance marked. The record is event-shaped; Docket is not a bus. |
| Distributed transactions | out of scope | — | There is nothing to transact across: one run, one process tree, one ledger file. |
| Saga / compensating actions | out of scope | — | The nearest thing Docket does have is a compensating action of exactly one kind: --rollback-on-breach restores the watched tree from a content-addressed snapshot, and records the per-path outcome. That is recovery, not a saga coordinator. |
| Dead-letter queues | out of scope | — | The equivalent problem — what happens to an event that cannot be written — is handled synchronously and honestly instead: fail_fast_evidence (stop the run) or a declared gap in the receipt (ops.evidence), never a silent drop. tests/test_ledger_faults.py covers ENOSPC, EIO, a closed handle and an OS error on open. |
| Cron jobs | modelled | .github/dependabot.yml (weekly), the game-day command designed to be run on a timer, docket verify --json as the CI gate |
No scheduler is shipped, because process scheduling belongs to your platform. The pieces that make a scheduled job useful are there: a machine-readable verdict, a non-zero exit on failure, and a post-mortem document. |
| WebSockets | out of scope | — | The boundary speaks HTTP/1.1 request/response. A websocket upgrade is not routed, framed or recorded; an agent that needs one is outside the boundary, and the receipt says what the boundary covered rather than implying completeness. |
| Long polling | measured | as an ordinary HTTP request: a long-poll GET is recorded with its duration like any other call | There is no special handling, and it does not need it: a slow response is a slow response, and its latency is in the row. |
| SSE (server-sent events) | measured, with a stated limit | src/docket/proxy.py (upstream read loop bounded by MAX_RESPONSE_BUFFER = 4 MiB; the whole stream is read before the response is written), src/docket/fakeupstream.py, tests/test_proxy.py |
Usage inside a stream is tallied when the provider reports it (docs/integrations.md). What is not true: incremental delivery. The boundary buffers the stream, so time-to-first-token is not measured and no streaming latency number is claimed — the latency of an SSE call is the buffered total, and a stream larger than 4 MiB is refused rather than silently truncated. |
3. Databases and concurrency¶
| concept | status | where it lives | what that means here |
|---|---|---|---|
| Indexing | built (the one index there is) | RFC 6962-style Merkle tree over the ledger: src/docket/canon.py, inclusion proofs in src/docket/receipt.py, checked by all three verifiers (docs/verification.md, spec/docket-receipt-1.md §7) |
The ledger has no secondary index and does not want one: Ledger.read is a streaming line scan. The Merkle tree is the structure that matters — it proves that one usage row is in this ledger in O(log n) hashes, without shipping the ledger. |
| Query optimisation | reasoned | src/docket/verify.py, src/docket/report.py, src/docket/explain.py are single-pass readers |
There is no query planner and no query language: report, explain and verify each make one pass over the events. The honest consequence: reading a 10-million-event ledger holds it in memory — and that consequence is now a number rather than a worry, because the soak measured the boundary's own in-memory event list at ≈1.1 KiB per recorded event (24,014 events, +26.5 MiB; 12,014 events, +13.4 MiB), holding to within 1% across both runs. Bounded by max_events / max_ledger_bytes (default 256 MiB), and named in docs/testing.md as unmeasured at ten million events rather than presented as fine. |
| Replication | out of scope | — | Nothing in Docket is a server, so there is no replica to lag. The receipt is the replication primitive: digest-addressed, self-contained (it embeds the price table and policy), copyable, and verifiable by anyone offline. |
| Leader election | out of scope | — | One writer per ledger is the primitive (see Distributed locks below); a consensus protocol on top of it is not needed, because two processes never need to agree on a fact that a hash chain already fixes. |
| CAP theorem | reasoned | ADR 0005; src/docket/ledger.py (fsync before return) |
The choice is made and stated: writes are synchronous and local — the event is on disk before the request proceeds — and there is no network consensus anywhere. A run whose evidence directory is on a slow network filesystem will therefore be slow, which is the correct trade for evidence. |
| Eventual consistency | out of scope | — | A receipt is a point-in-time document about one run; a reader either verifies it or does not. There is no merge, no conflict resolution, and no "eventually" in which two receipts have to agree. |
| Optimistic locking | built (in one place) | the chain itself: each event names prev (the previous event's hash), so a writer at a stale position cannot extend the chain without detection (Ledger.chain_error(), tests/test_ledger.py) |
This is version-check-on-write in its cryptographic form. It is not a general concurrency-control mechanism and is not claimed as one. |
| Pessimistic locking | built | advisory exclusive lock per ledger file: fcntl.flock on POSIX, msvcrt/O_EXCL fallback elsewhere, LedgerLockedError naming the holder (src/docket/ledger.py), ADR 0005, tests/test_ledger_concurrency.py |
One writer at a time on a ledger file, with the refusal naming the process that holds it. tests/test_invariants.py drives both halves: two runs in one directory get two ledgers, and a second open of one file is refused and then allowed after release. |
| Distributed locks | out of scope | ADR 0005 | The lock is advisory, local, and per file. A distributed lock would require consensus and a service; a hash chain plus a single writer gives the property that was actually wanted (no two heads) without either. |
| Race conditions | built (and this is where the testing is) | tests/test_ledger_concurrency.py (8 writer threads), tests/test_proxy_concurrency.py (parallel calls through the real boundary: chain, totals, limiter, breaker, cap), tests/test_invariants.py (200 events from 8 threads through one handle; two supervisors, one directory), ADR 0005 |
The claims are executed, not argued: one event per sequence number, single chain, complete lines, totals that match the sum of the rows, no duplicate sequence number. |
| Deadlocks | reasoned | ADR 0005 (lock ordering); src/docket/supervisor.py |
There is exactly one lock whose acquisition can block (the append lock), it is held for one append and never across an upstream call, a child process wait, or a health probe. The metrics and readiness paths take no ledger lock. A deadlock would require a second lock; there is no second lock. |
| Memory leaks | reasoned, and this is a named gap | src/docket/ledger.py (close() releases the handle and the lock; tests/test_invariants.py asserts _handle is None after close), LedgerFullError, max_ledger_bytes / max_events |
Nothing accumulates after a run ends: handles and locks are released and the receipt holds derived values, not live objects. But a live run holds its events in memory by design, so a long run's memory grows with its event count until the byte ceiling refuses more. There is no soak test that would catch a slow leak over hours — docs/testing.md#not-covered-and-why says so, and no leak-free claim is made. |
| Garbage collection | out of scope | — | CPython's refcounting is the collector; nothing here manages it. The relevant promise is the weaker, checkable one above: a closed ledger holds no file descriptors. |
| Thread safety | built | ThreadingHTTPServer with daemon_threads=True, threading.Lock in the limiter, breaker, health state and stats, the ledger's append lock, tests/test_proxy_concurrency.py, tests/test_ledger_concurrency.py, tests/test_invariants.py |
Every shared structure that a request thread touches is either lock-protected or immutable after construction, and the concurrency tests use the real HTTP server rather than a mocked handler. |
| Backpressure | built, in one direction only | the ledger's ceilings: max_events / max_ledger_bytes → the boundary answers 503 with code: max_events and the supervisor stops the run (src/docket/supervisor.py, tests/test_ledger_contracts.py); MIN_FREE_BYTES readiness floor; MAX_RESPONSE_BUFFER refusing an oversized upstream body |
The agent is pushed back on when the evidence cannot keep up — which is the direction that matters for this tool. The other direction is not implemented: the boundary buffers an upstream response rather than streaming it, so it cannot apply backpressure to a provider that is sending faster than the reader consumes. |
4. Performance and scalability¶
| concept | status | where it lives | what that means here |
|---|---|---|---|
| Autoscaling | out of scope | — | There is no service to scale: a run is one supervisor process and one boundary in it. |
| Horizontal scaling | built (of verification) | the receipt is a file; Python and Node verifiers are both offline and stateless (src/docket/verify.py, verify-js/verify.mjs) |
The scaling axis Docket has is the one that matters for adoption: any number of independent parties can verify the same receipt, concurrently, without a service, a network call or a shared secret. |
| Vertical scaling | measured (and it is not CPU-bound) | tools/loadtest.py + src/docket/loaddriver.py, reports in docs/measurements/2026-10-15-2core/ |
On 2 vCPUs, 8 threads got 175.93 calls/s unpaced with a client p50 of 44 ms, while a paced 20 calls/s soak saw p50 4 ms: the cost under concurrency is the append lock and the fsync per event, not the processor (ADR 0005). More cores would not make the recorder faster; a faster disk would. Nothing here is tuned or claims a target — that is one measured point on one host. |
| CDN | out of scope | — | Nothing is served. The verifier is a single dependency-free file that works from a local disk, which is the property that replaces a CDN here. |
| Edge caching | out of scope | — | Same. verify-js/verify.mjs can be dropped into a browser or an edge function (it is pure ES modules over the WebCrypto-free parts of the format); that is a consequence of being dependency-free, not a caching strategy. |
| Cache invalidation | reasoned (and answered for evidence) | the receipt embeds the price table, the policy digest and the format tag: src/docket/receipt.py, spec/docket-receipt-1.md §4–5 |
The classic problem — a cached artefact that depends on changing data — is answered by making the receipt self-contained: prices, policy and format travel with the evidence, so a receipt from January still verifies in December even if prices.json has moved on. The verifier recomputes costs with the embedded table and says which one it used. |
| Feature flags | out of scope (by design) | — | No flags. The analogue is policy: every knob is explicit, defaults are stated, and the effective policy is recorded in the receipt with a digest. A silent kill switch would be a way to change behaviour without leaving evidence, which is the opposite of the tool's purpose. |
| Database migrations | out of scope | — | There is no database. The ledger format is a line of canonical JSON; a reader that meets a field it does not know preserves it (src/docket/events.py) rather than dropping it. |
| Schema versioning | built | docket/receipt/v1 (src/docket/__init__.py, src/docket/receipt.py), the version-tagged format process in CONTRIBUTING.md#changing-the-format, docs/adr/0001…0005, the independent verifier bound to the same tag |
A receipt names the format it is in, and a verifier fails on a format it does not implement rather than guessing (src/docket/verify.py: unrecognised tag → format: fail). Changing the format is a documented, reviewed act with committed conformance vectors (spec/vectors/) that both implementations must regenerate identically. |
| Disaster recovery | built (the drill, not a promise) | docket doctor (diagnosis + --quarantine-torn-tail repair), --snapshot / --rollback-on-breach / docket restore, the game-day phase that SIGKILLs the supervisor mid-run, tests/test_invariants.py::test_a_killed_supervisor_leaves_evidence_and_no_lie |
Reported as executed behaviour, with the numbers that are measurable: every recorded event is fsynced before the request proceeds, so the recorded prefix survives a kill (recovery point = the last recorded event); a torn tail is classified, and repaired only by an explicit command that moves bytes to a sidecar rather than deleting them. No RTO claim: nothing here measures how long a recovery takes on a real system. |
| Backups | built | content-addressed snapshot store (src/docket/snapshot.py), manifests, docket restore with confinement checks, tests/test_snapshot_confinement.py |
What is backed up is the watched tree, not the evidence: snapshots exist so that a breached run can be undone. The evidence itself is never overwritten by any path in this tool — historical observations are append-only, and a restore writes its own separate record. |
| Failover | out of scope (deliberate) | 11 provider base URLs in DEFAULT_UPSTREAMS, one per provider, no secondary |
There is no automatic provider failover. Switching providers silently would change what a receipt means (different account, different price, different tokeniser); such a switch belongs in the operator's router, above Docket. |
| Multi-region | out of scope | — | A receipt is regional only in the sense that its data is local. |
| Chaos engineering | built | docket chaos (fault grammar: upstream failures, latency, malformed responses, rate limits), docket game-day (9 phases), FaultInjector with a recorded seed, tests/test_chaos.py (34 tests) |
Injections are explicit, seeded and recorded in the ledger, so a run with deliberate faults can never be mistaken for a normal run's evidence. The nine phases each assert their expectations against the ledger, the receipt and the disk — not against the intention. |
| Cost optimisation | built | src/docket/pricing.py + prices.json (per-model input/output/cached rates), findings cache-miss, fat-context, premium-for-small-calls, concentration, runaway-loop, retry-storm (src/docket/explain.py), caps in policy.py, --budget-mode quota |
This is the one production concept the tool exists for: the receipt says what a run cost and on which events that rests; docket explain names the spending patterns with attributed impact; the caps stop the spend. |
| Cold starts | measured | python3 -m docket at commit c04fe0e: 114 ms median (3 runs) for --version, against 14 ms for a bare interpreter on the same box; zero runtime dependencies (pyproject.toml, no imports outside the standard library) |
The whole cold-start cost is interpreter startup plus one import sweep of a stdlib-only package (import docket.cli measures 111 ms on the same box, i.e. indistinguishable from the CLI startup it dominates). No lazy-loading tricks, no plugin discovery, nothing that would make the first call slower than the second. |
| Serverless limits | reasoned | docs/integrations.md#kubernetes--containers, the loopback guard in src/docket/util.py |
The boundary needs loopback and the same filesystem view as the agent, to snapshot it and to write evidence. A single-container task satisfies both; a FaaS runtime with no shared filesystem and no local listener does not, and nothing here claims otherwise. |
| Latency | measured, now including under load | src/docket/latency.py, latency_ms on usage rows, the derived latency member, check latency-summary in all three verifiers, docs/verification.md; load numbers in docs/measurements/2026-10-15-2core/ |
Per-call round trips, recorded in whole milliseconds, summarised under a stated rule (nearest-rank, ceil(p/100·n)). Recomputed by the verifier from the rows the receipt itself cites, so a stated p99 that the rows contradict fails verification. tools/loadtest.py now records the same measurement under an 8-thread load and writes its own summary with the receipt's rule, which is why the two can be compared at all. |
| Throughput | measured | tools/loadtest.py, python -m docket.loaddriver, make load; reports in docs/measurements/2026-10-15-2core/ |
175.93 calls/s at 8 threads unpaced, 20.00 calls/s paced for ten minutes, 0 drops and 0 errors in either, on 2 vCPUs with a scripted provider. This is a measurement of the recorder on one host, not a target and not a curve: the bottleneck under concurrency is the append lock plus fsync, and a real provider would dominate the wall clock. The verification side has a second throughput number: the JS verifier checks a receipt in milliseconds with no dependencies. |
| p99 / tail latency | measured, at two load points, with a stated limit | p50_ms / p95_ms / p99_ms / max_ms in the receipt; docket report prints the tail line; tests/test_latency.py; the load reports in docs/measurements/2026-10-15-2core/ |
Two observed tails: p50 4 ms / p99 7 ms at a paced 20 calls/s over 12,002 calls, and p50 44 ms / p99 53 ms with 8 threads unpaced over 2,000 calls — the second is the price of the append lock, and both are this host. Neither is a load-curve tail (one concurrency point, one rate), and max_ms is always reported beside p99 because with few samples the two are the same value and a reader should be able to see that. |
5. Networking and operations¶
| concept | status | where it lives | what that means here |
|---|---|---|---|
| Network partitions | measured | breaker + timeouts + model.error recording (src/docket/proxy.py, src/docket/resilience.py), game-day phase upstream-failure, tests/test_proxy.py |
A partition between the boundary and the provider is a run of failed calls: each is recorded as an error, the breaker opens after the threshold, and the refusal is evidence. A partition between the agent and the boundary is recorded on the agent side as a connection failure — the boundary cannot record what never arrived, and the receipt's ledger section is scoped to what it saw. |
| Clock skew | built | wall clock for timestamps (RFC 3339 UTC) with order carried by seq and prev, monotonic clock for caps and durations (src/docket/supervisor.py), tests/test_invariants.py (a battery of clock tests) |
Two clocks with two jobs, stated: a reader compares ts to their own clock, and ordering never depends on it — the chain is the order. Caps are measured on the monotonic clock, so a VM whose clock steps forwards or backwards cannot extend a run; a clock that steps backwards is clamped so no duration can be negative, and it cannot manufacture a breach. Retry-After as an HTTP-date is deliberately ignored (a local clock behind the provider's would make the wait negative; a negative sleep is a busy loop). |
| DNS | measured (host, not address) | egress decisions by hostname (src/docket/policy.py Egress.decide, recorded as net.allowed / net.denied), provider base URLs in the receipt's policy, tests/test_url_guards.py |
The host is what the policy decides on and what the ledger records. The resolved IP is not recorded — that would need interception at the resolver, and claiming otherwise would be a lie about scope. Docket's own clients never resolve anything (loopback-only guard). |
| TCP vs UDP | reasoned | HTTP over TCP only (http.client upstream, ThreadingHTTPServer locally; no UDP socket anywhere in src/) |
There is no UDP path, no DNS server, no QUIC. The one TCP-level property that is actually tested is the invariant that a stopped boundary releases its listening socket and can be re-bound immediately (tests/test_invariants.py) — the practical face of SO_REUSEADDR and TIME_WAIT. |
| HTTP/2 | out of scope, and a real limitation | HTTP/1.1 in both directions (BaseHTTPRequestHandler + http.client) |
A provider or agent that requires HTTP/2 (or a gRPC-only provider) cannot be recorded by this boundary. The --http-proxy path tunnels TLS with CONNECT, so an agent that speaks h2 to its provider still gets its traffic governed (egress decisions are made and recorded) but not its usage tallied. That distinction is the honest one and is worth reading twice. |
| HTTP/3 | out of scope | — | Same boundary, stricter: QUIC is UDP-based and nothing here speaks it. |
| gRPC | out of scope | — | Needs HTTP/2 framing plus a non-JSON body, and usage extraction here is JSON-based. An agent calling a gRPC inference service is outside the boundary's reach; the receipt says what it covered rather than estimating what it did not. |
| Webhooks | reasoned | docket verify --json, the CI patterns in docs/integrations.md#ci, and the receipt as a file |
Docket does not call out to a URL — no callback configuration, no outbound notification, nothing to point at an attacker-controlled endpoint. The integration is the inverse and simpler: a CI job runs the agent, verifies the receipt and fails, and the receipt is an artefact like any other. |
| API versioning | built | the receipt's format tag with a refuse-on-unknown verifier (docket/receipt/v1), provider paths under /openai/v1, /anthropic/… (src/docket/proxy.py), docs/policy-reference.md |
Both senses are handled: the evidence format is versioned and a verifier refuses a tag it does not implement; the provider API versions are part of routing, and the boundary records which provider and model a call went to. |
| Semantic versioning | built | CHANGELOG.md (Keep a Changelog + SemVer 2.0.0, with the receipt format versioned separately by its tag), pyproject.toml (0.1.3), GOVERNANCE.md for who decides |
The version policy is written down, including the part most projects get wrong: a format change is not a minor version bump here, it is a documented act under CONTRIBUTING.md#changing-the-format with regenerated vectors. |
| IaC | modelled | deploy/ (Dockerfile, compose, Kubernetes manifests, Helm chart) + tools/deploy_check.py, make deploy-check, a CI step |
Declarative deployment artefacts with a checker that enforces the invariants this project can actually verify statically — and, for the chart, a comparison against the real tool (helm lint / helm template, v3.16.3) rather than a second opinion written by the same hand: non-root, no privilege escalation, read-only root filesystem, resource requests and limits, probes, no :latest, no hostPath, and a rendered chart whose ${...} substitutions resolve to declared values. What is not claimed: that a cluster accepted them — nothing here has one. The Helm half of this was executed; the container half (docker build) and the cluster half were not, and neither is claimed. |
| Terraform | out of scope | — | No module is shipped, on purpose. A Terraform configuration cannot be validated without downloading a provider, nothing in this environment can apply it, and shipping unvalidated infrastructure code under a banner of verified evidence would be exactly the failure this project exists to argue against. The manifests in deploy/ are the interface; a provider module can wrap them without pretending to have been tested here. |
| Helm charts | modelled (and rendered by Helm itself) | deploy/helm/docket/ (Chart.yaml, values.yaml, templates/), rendered and checked by tools/deploy_check.py, tests/test_deploy_check.py; helm lint and helm template run against Helm v3.16.3 (2026-10-15) — deploy/README.md records the commands, the result (5 documents, identical to the static renderer's, field for field) and the defect that measuring found |
The chart is deliberately value-substitution only — no include beyond names — so a deterministic renderer can be written and the result can be validated as real Kubernetes documents. Real Helm has now rendered it, and the renderers agree, which is what makes the static checks evidence rather than a parallel universe: they disagreed at first, because Helm hands a template a float64 for every number in values.yaml and Go prints a large one as 6.7108864e+07 — a policy value that TOML reads as a float. The chart writes its integers through | int64 now, the checker renders numbers the way Helm does, and it refuses an integer key that arrives as a float. helm install has still never run: there is no cluster. |
| Build caching | reasoned | .github/workflows/ci.yml (pip cache), pyproject.toml (hatchling; the sdist ships src spec tests docs examples verify-js tools — PUBLISH.md), tools/mutate.py (its sandbox copy excludes .git, dist, build, __pycache__) |
Nothing here needs a build cache worth documenting: there is no compilation step, the package is pure Python, and the artefacts are small. The immutability property that matters is checked instead — the built sdist is byte-comparable and its file list is compared to git ls-files. |
| Dependency hell | built (as a stance, and it is tested) | pyproject.toml: zero runtime dependencies; two optional extras — sign (cryptography) and deploy (pyyaml) — make the Ed25519 and deployment-checker tests run instead of skip (1,106 passed / 47 skipped with neither installed, 1,153 passed / 0 skipped with both, four fresh venvs measured 2026-10-16 on the tree these documents describe, and the exact four-cell matrix is in docs/testing.md); the CI package job installs from dist/ with --no-index; verify-js/ is the second implementation with no dependency at all |
Three answers to the same problem: the tool has nothing to upgrade; each optional dependency degrades to a named skip with a reason rather than an import error — which is enforced by a test, not asserted, because the deployment checker used to fail its own 28 tests in an environment without PyYAML, and a tool that could not run exited 1 ("found a problem") where its own contract said 2 ("could not run"); and the format has an independent implementation a JavaScript shop can run without installing Python. |
| Production incidents | built (as an artefact and a corpus) | docs/incident-corpus.md (real, cited incidents and what they cost), docket game-day post-mortem with per-phase expectations, docs/evidence.md |
A run's receipt is designed to be the artefact an incident review asks for — what the agent did, what it cost, on which events that rests, and whether the evidence is intact. The post-mortem the game day writes is a real document with met/unmet expectations, produced from the ledger rather than from memory. |
| On-call | measured, plus an ops surface | /docket/healthz, /docket/readyz (readiness reads the ledger's writability and the free-space floor), /docket/metrics (Prometheus text: docket_model_calls_total, docket_cost_usd, docket_events_total, docket_breaker_state, docket_ledger_degraded, …) — src/docket/proxy.py, docs/cli.md#the-boundarys-operations-endpoints, docs/deployment.md |
An on-call engineer gets three things that are usually missing from an agent run: a live cost meter, a readiness signal that turns red before evidence is lost, and a record that can be verified after the fact by someone who does not trust the machine it ran on. The endpoints are loopback-only by design, so they are read from inside the host or pod (probes, a sidecar, kubectl exec) — a peer that can reach the boundary could post events into a live run. There is no pager integration and no alert routing: that belongs to whatever is already waking you up. |
| Post-mortems | built | docket game-day --out <dir> writes a post-mortem (the unhyphenated postmortem spelling is the same document); docs/testing.md describes the nine phases and what each asserts; docs/incident-corpus.md for the external record |
The post-mortem is generated from the run's own evidence with the expectations declared in advance per phase, so it cannot be written to a conclusion. Blameless by construction: it reports what the system did, not who did it. |
The shape of the gaps¶
Three kinds of absence show up in the tables above, and they are not the same thing:
- A mechanism that is not here because it belongs to another layer — a broker, a cluster autoscaler, a CDN, a distributed lock service, Terraform state. Nothing in a receipt gets better if Docket grows one of these, and pretending otherwise would be the sort of scope inflation that makes an evidence tool untrustworthy.
- A mechanism deliberately refused, with the reason written down — deduplicating retries (it would fabricate a run), provider failover in the boundary (it would change what the evidence means), service discovery (it would make a recorder aimable at a network destination), feature flags (it would be a way to change behaviour without evidence). These are the cells most worth reading, because they are where the design has an opinion.
- A mechanism that is here but is not measured at the scale its users will hit — endurance
beyond ten minutes, throughput as a curve rather than the one measured point, latency past
eight threads, memory over hours, a real provider's API changes, and every claim in the
modelled row until someone runs it on a cluster. Three of those used to be unmeasured
outright; the load harness (
make load,make soak, reports indocs/measurements/2026-10-15-2core/) moved them to measured once, on one host, anddocs/testing.md("Not covered, and why") is the long form of what is still open — the honest half of this document.
If you are evaluating Docket for a production agent, the useful reading is: the evidence path is
built and tested to the standard described in docs/testing.md; the enforcement path
is real but deliberately narrow (caps, policy, limits, breaker, limiter, egress); and the
infrastructure around it is your platform's job, with recipes here that have been checked as far as
this environment allows and labelled at exactly that level.
Keeping this honest¶
Every cell names a path. To falsify one, open the path:
- a built cell should have code and a test file you can run (
make test); - a measured cell should produce a member in a receipt you can generate with
docket demo; - a modelled cell should pass
make deploy-check, and the checker's own tests should fail when the artefact is broken (tests/test_deploy_check.pybreaks each invariant on purpose); - a reasoned cell should have a document or an ADR that argues the boundary, not a shrug;
- an out of scope cell should have a one-line reason you either accept or don't — and if you
don't, that is a design conversation worth opening as an issue, which is what the roadmap is for
(
ROADMAP.md).
If a cell here is wrong, it is a defect of the same kind this project exists to prevent, and issues that correct this file are the most welcome kind.