Resilience practice coverage: which of it is actually practised¶
A list of resilience-engineering practices — stress testing, FMEA, red teaming, chaos monkey, software rejuvenation, post-mortems — reads as a checklist, and a checklist is easy to satisfy on paper. This document is the honest answer to it: for each practice, what this repository actually does, where it happens, and how strong the evidence is.
It is the companion to two other maps, and the three of them answer different questions:
| map | the question it answers | size |
|---|---|---|
docs/production-coverage.md |
the mechanisms — rate limiting, backpressure, p99, Helm charts — is each one here? | 77 concepts, one row each |
docs/testing.md |
the kinds of testing — boundary-value, mutation, soak, accessibility — which are done, and what is deliberately not covered | 44 kinds |
| this file | the practices — the SRE/discipline side: injecting failure on purpose, watching, recovering, hardening | 22 practices |
docs/standards.md |
the sources — which norm or paper each mechanism follows, and where Docket departs from it | 20 mechanisms |
How to read a row. Statuses are the same five the other map's legend defines, and they mean the
same thing here: built (exists and is exercised by the suite or CI), measured (a number is
recorded rather than a mechanism provided), modelled (a validated artefact exists and is checked
statically, but nothing in this environment can deploy it), reasoned (addressed in writing with
the boundary stated), out of scope (deliberately not attempted, with the layer it belongs to
named). A cell is never "partly built": where something holds in one direction and not another, the
cell says which. Every claim below was checked against the tree at commit 7e97619 plus the work
that landed on top of it; where a number is quoted, the artefact that holds it is named.
1. Failing on purpose¶
| practice | status | where it lives | what that means here |
|---|---|---|---|
| Stress testing | built | tools/loadtest.py (make load: 2,000 calls over 8 threads, unpaced; make soak: 10 minutes paced at 20 calls/s), tests/test_loadtest.py (37) |
A real supervised run is driven by real threads through a real boundary, and the client reports throughput and p50/p95/p99. What it is not: a stress curve. Nothing here ramps load until something breaks, and every number is a fact about this 2-core host — docs/measurements/2026-10-15-2core/ keeps the raw reports so the numbers can be re-checked rather than believed |
| Fuzzing | built | tools/fuzz.py --target {canon,policy,receipt,ledger,proof,faultspec}, tests/test_fuzz_regressions.py (98) |
Seeded, offline, deterministic mutation of the documents Docket parses — canonical JSON, policy files, receipts, ledger lines, inclusion proofs, fault specs. Every input that has ever crashed it is kept as a regression test, which is the part that matters: a fuzzer you cannot re-run is an anecdote. Not coverage-guided (no libFuzzer or AFL harness; it is a Python mutator over structured documents) |
| Chaos engineering | built | src/docket/chaos.py, docket chaos, tests/test_chaos.py (34) |
A fault grammar (drop:0.25, truncate=0.8, delay=0ms, slow=2s, status=503) applied to a live run, seeded so an experiment is reproducible, and recorded in the ledger as a chaos run plus in the receipt as ops.chaos_run — so a reader can tell an experiment from production traffic |
| Chaos monkey | built | game day supervisor-killed phase; src/docket/scriptagent.py modes (loop, retry, backoff) |
The monkey is both ends of the wire: the supervisor is SIGKILLed mid-run while the agent is a real child process, and the reference agent can be told to behave like the failure (repeat itself, retry a refusal, hammer a rate limit). Both are seeded, so a failure that is found once can be reproduced on purpose |
| Game days | built | docket game-day — nine phases: baseline, upstream-failure, circuit-breaker, upstream-latency, rate-limit, ledger-pressure, rollback, cache-brake, supervisor-killed; runs in CI |
Each phase declares what it expects before it runs, then runs for real and is audited against the ledger, the receipt and the filesystem; a phase that misses an expectation is reported as a miss, never as a pass, and a check whose condition never arose is reported as skipped rather than counted. The output is a written post-mortem per phase. The phases are independent experiments: each one's policy is a copy of the base with exactly one deliberate change, which a test asserts by building every phase against a base another phase has already been used with — the property that was false when cache-brake ran for the first time under the ledger-pressure phase's ceiling, and the reason it is now asserted rather than assumed |
| Red teaming | built | docs/threat-model.md (eight adversaries), the "red teaming" row in docs/testing.md, tests/test_security.py (40), tests/test_url_guards.py (43) |
Adversarial work with a record of what it won: a CONNECT tunnel that skipped egress checks, a secret that reached the ledger through a header, a receipt that verified after a field was edited, a cap bypassed by a request shape — each is now a failing test that stays failing. The URL guards are checked by walking the package's syntax tree, so a call site added later cannot skip one |
2. Watching¶
| practice | status | where it lives | what that means here |
|---|---|---|---|
| Dynamic analysis | built | the whole suite: 1205 tests that bind sockets, spawn children, fill a tmpfs until ENOSPC, SIGKILL a supervisor, and step a clock |
The behaviour claims are made against a running system. The static half is ruff and bandit in CI, and the two are kept apart on purpose: a linter cannot tell you that a cap refused a call |
| Vulnerability scanning | built | bandit -q -c pyproject.toml -r src tools in CI with per-class written suppressions; the URL-guard walk above; pip-audit over the development toolchain; SECURITY.md |
Two different questions: is the code dangerous (bandit, with every suppression justified by class rather than blanket) and is a dependency known-bad (pip-audit, which audits the dev/CI toolchain — the installable package has none). Not covered: container image scanning, because no container is built in this environment, and the container recipe is checked statically instead |
| Synthetic monitoring | built | docket selftest (19 probes), run in CI on every push, and docket doctor for the health of a machine and the ledgers already on it |
This is the practice with the most literal implementation in the tree: probes that use the product the way a deployment does — forward a call, refuse a denied model, cross a cap, get throttled, open and close a breaker, inject a fault, hit the cache and miss it, watch a ledger write fail against a real kernel limit, verify a receipt and catch a tampered one — and print what they observed, not just a verdict. It is what a cron job or a post-deploy job would run |
| Proactive monitoring | built | /docket/metrics (Prometheus text), /docket/healthz, /docket/readyz, cap.warning at warn_at, readiness against min_free_bytes |
The boundary exposes the numbers a dashboard needs — requests, model calls, denials, cost, events, rate-limited count, breaker state, ledger degraded — and warns before a ceiling rather than only at it. Honest boundary: alert rules are not in this repository. The Helm chart carries probes and a Service; the alerting that consumes these metrics belongs to the operator's monitoring stack |
| Observability | built | the ledger (one JSON object per event, hash-chained), the receipt, docket explain, docket report, docket diff, the metrics endpoint, the diagnostic bundle |
Every decision the boundary makes is an event, and the receipt is a bounded summary a reader can verify offline. Content is off by default (--record-bodies opts in): the observability story is about spend, refusals, scopes and evidence, and it deliberately does not turn the audit trail into a transcript of your prompts |
| FMEA | built | docs/resilience.md — sixteen failure modes rated severity/occurrence/detection, ordered by risk, each naming the artefact that detects it |
Added because the practice was named and not practised: there was prose about failure and no analysis of it. The ratings are judgements and the page says so; the detector column is the evidence, and a test reads the table and asserts every path it names exists |
3. Recovering, limiting, hardening¶
| practice | status | where it lives | what that means here |
|---|---|---|---|
| Circuit breaking | built | src/docket/resilience.py (threshold 5 consecutive-ish failures, 30 s cooldown, exponential to 300 s), tests/test_resilience.py (60), game day circuit-breaker |
Opening the breaker is recorded, the state is exposed as a metric, and the two subtleties that break naive implementations are pinned by tests: a breaker that is open must not block a cache hit, and its thresholds must be strict (< versus <= here was a mutation survivor until a test was written for it) |
| Rate limiting | built | RateLimiter in src/docket/resilience.py, [resilience] max_calls_per_second / max_calls_per_minute, game day rate-limit |
Refusals are visible to the agent in its own language — 429 with Retry-After — and each throttle is an event in the ledger rather than a silent wait. Pacing is max(next_call + pace, now), and a bad configuration is a construction-time ValueError, never a mid-run surprise |
| Automated rollback | built | --snapshot, --rollback-on-breach, docket restore (dry run by default), tests/test_snapshot_restore.py (26), game day rollback |
The snapshot is taken of the paths the run touches, and the rollback fires automatically when a cap is breached — the case where nobody is watching. A restore writes its own record beside the evidence instead of editing the run's records, and confinement is checked before any write |
| Disaster recovery | built (single host) | append-only, fsynced ledger; torn-tail classification in docket doctor; --quarantine-torn-tail; docket restore; offline verification |
Recovery here means the record and the tree survive the run dying, which is the part a recorder can own: a killed run leaves a readable prefix rather than a corrupt file, the tail is classified with a byte offset rather than guessed at, and a receipt verifies with no network and no Docket server. What is out of scope, and named as such in docs/production-coverage.md: off-site backup, replication, failover and multi-region — those belong to the storage layer the operator runs |
| Software rejuvenation | reasoned | bounded state by construction: [resilience] max_events / max_ledger_bytes, cache TTL + max_entries + max_hits; the soak's memory measurement in docs/measurements/2026-10-15-2core/ |
There is no long-lived daemon here to restart: the boundary lives for one run and exits, so the failure mode rejuvenation addresses — accumulated state with no bound — is answered by ceilings instead. The measured part: memory tracks events recorded, not time elapsed, at 1.04–1.14 KiB per event across a 6× range of run lengths. The honest limit is on the same page: ten million events would be roughly 11 GB before a ceiling stopped it, which is why the ceilings exist. No restart-based rejuvenation is offered, and a deployment that runs the boundary as a long-lived service is doing something this project does not document |
| Degradation testing | built | tests/test_ledger_faults.py (29: ENOSPC on a real filesystem, EIO, a closed handle, a read-only directory, modes re-tightened on open), fail_fast_evidence, receipt ledger_complete / unwritten_events / first_failing_seq / write_failures, game day ledger-pressure and cache-brake |
The interesting behaviour under pressure is refusing to continue pretending: evidence that cannot be written stops the run by policy, the receipt states the gap instead of leaving a reader to infer it, and the cache's max_hits brake turns caching off rather than letting it grow. Each of these is asserted against a real failure, not a mocked one |
| Blast-radius mitigation | built / modelled | per-run caps in money and calls; one ledger per run with an exclusive lock (ADR 0005); workspace confinement and _exclude_own_evidence; [egress]; the child gets its own process group; Helm NetworkPolicy, non-root, read-only root filesystem |
The radius is the run: an agent that goes wrong spends until its own cap, writes in the directory it was pointed at, reaches the hosts it was allowed, and cannot touch another run's evidence. The cluster half is modelled, not deployed — tools/deploy_check.py checks it statically and says what is unchecked |
| IaC testing | built | tools/deploy_check.py (16 checks over deploy/ and the workflows) and tests/test_deploy_check.py (44) in CI; tools/helm_fidelity.py + tests/test_helm_fidelity.py (17) comparing the checker's chart render against a real helm template — 5/5 documents identical on 2026-10-15 |
Infrastructure as code is only as good as the test that breaks it on purpose: each check has a fixture that violates exactly one invariant, and the most useful defect this found so far is written up as a post-mortem in deploy/README.md — numbers from a values file arrive in Helm as float64, so a chart that renders 1000000 where the policy expected 1000000 bytes was silently wrong |
| Post-mortems | built | the game-day report (expectation, observation, verdict per phase) — postmortems, in the unhyphenated spelling a search box does not care about; "Found by testing" in docs/testing.md; docs/incident-corpus.md |
Three kinds, kept apart: post-mortems of this repository's defects, a running record of what testing actually caught (including the ones a person found by trying something), and a graded corpus of external incidents — the Denial-of-Wallet cases that motivate the project — with confidence grades A–C on every entry, so a reader can weigh a vendor blog differently from a named security team's report |
| Infrastructure hardening | built / modelled | deploy/Dockerfile (non-root user, a comment instead of a fake HEALTHCHECK, STOPSIGNAL SIGTERM), deploy/helm/docket/templates/ (runAsNonRoot, seccompProfile, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, dropped capabilities, NetworkPolicy), ledger/directory modes 0600/0700 enforced on open, URL guards, bandit |
Hardening is visible where it can be checked: the container recipe never runs as root and does not claim a liveness check it cannot perform, the manifests are read-only by default, and the evidence files are owner-only — asserted by tests that create them permissively and watch Docket tighten them. Host and cluster hardening beyond the manifests is the operator's, and is not claimed |
What this page does not claim¶
- No deployment. Nothing here has been built as a container, applied to a cluster, or run
behind a real load balancer. The
modelledcells are static: a file exists, a checker reads it, and the checker's own tests break each invariant on purpose. That is all. - No provider. Every performance and soak number comes from the offline fake upstream in this repository, on a 2-core host. They describe Docket's own overhead, not a real model's latency.
- No incident here was ever "handled". The nine game-day phases and the chaos experiments are deliberate, seeded failures in a test environment; the post-mortems they produce are about the tool, not about an outage someone lived through.
- Ratings are judgements. Where a row quotes a rating (the FMEA table), it is an opinion with a reason attached, and the page it lives on says so. The checkable parts — paths, tests, measured numbers — are the parts a test asserts.