Metadata-Version: 2.5
Name: mayhem-cli
Version: 0.9.1
Summary: General-purpose chaos engineering and resilience-testing framework: a chaos experiment orchestration engine.
Author: Mayhem Engineering
License: Proprietary
Requires-Python: >=3.12
Requires-Dist: click>=8.1
Requires-Dist: kubernetes>=31
Requires-Dist: pydantic>=2.9
Requires-Dist: pyyaml>=6.0
Requires-Dist: structlog>=24.1
Requires-Dist: typer>=0.12
Description-Content-Type: text/markdown

# Mayhem

**Chaos engineering for Docker and Podman, with separately scoped Kubernetes
planning and execution seams.** The documented quickstart uses a compose
blueprint. Mayhem compiles a declarative `kind: drill` document into a frozen,
safety-gated plan, injects supported faults through a capability-aware toolkit,
and derives the run verdict from recorded observations.

Everything lands in SQLite — steps, probes, criteria evaluations, decisions —
so nothing is ever "trust me, it worked."

Kubernetes source support is layered and must not be collapsed into one
"supported" claim: manifest planning, planner support, executor support, and
live resolution are separate states. The legacy `KubernetesAdapter` remains
unavailable, and catalog-only faults are not executable merely because they are
defined. See the [Kubernetes status](#kubernetes-status) section and the
[documentation authority index](docs/README.md).

```
compose blueprint ─▶ topology graph ─▶ compile drill spec ─▶ frozen plan
                                                              │
                        verdict ◀── machine criteria ◀── inject → observe → recover
                                                              │
                                                SQLite evidence + decision trace
```

- [Quickstart](#quickstart)
- [A Run in One Screen](#a-run-in-one-screen)
- [Authoring a Drill](#authoring-a-drill)
- [Configuration](#configuration)
- [CLI Reference](#cli-reference)
- [Campaigns](#campaigns)
- [What's new in v0.9.0](#whats-new-in-v090)
- [Safety Model](#safety-model)
- [Exit Codes](#exit-codes)
- [Architecture](#architecture)
- [Status](#status)
- [Development](#development)

---

## Quickstart

**Prerequisites**

- Python 3.12+
- Docker with Compose v2, or Podman selected with `--podman`

The Kubernetes client ships as a default dependency of the distribution, so
there is no separate install extra. The Kubernetes paths still need a reachable
cluster, a valid kubeconfig context, and the capabilities a fault family
declares. The checked-in documentation does not certify any live cluster.

**Install**

```bash
pip install mayhem-cli        # one bundle; console command is `mayhem`
```

**Run the bundled example**

A complete, self-contained six-service stack lives in
[`examples/testCase/`](examples/testCase/).

```bash
cd examples/testCase
docker compose up -d                          # 0. bring the stack up
mayhem discover topology --compose docker-compose.yml   # 1. blueprint → live graph
mayhem prepare validate mayhem.yaml --compose docker-compose.yml  # 2. compile + safety gates (injects nothing)
mayhem run mayhem.yaml --compose docker-compose.yml --execute  # 3. inject → observe → recover → verdict
```

Omit `--compose` and Mayhem auto-detects `docker-compose.yml` (or
`compose.yml`) in the current directory.

`validate` and `plan` compile from the compose blueprint and do not inject a
fault. Topology construction may still inspect an available container runtime
for current-state data. `run` applies its impact gate and records the actual
execution outcome.

The bundled spec exercises the compose-supported fault families used by the
example stack. The [`examples/k8s`](examples/k8s) directory is a separate
manifest-backed planning example. Its presence does not prove live-cluster
execution, and the current catalog is larger than the original nine-fault
example. See [Targets (cross-runtime)](docs/drill-spec.md#targets-cross-runtime)
for the authored target syntax.

---

## A Run in One Screen

A clean run needs no interpretation — the verdict is one line away.

```
$ mayhem run mayhem.yaml --compose docker-compose.yml --execute

# Run r-process-drill-8f2a1c
**status**: completed
**verdict**: pass
**success criteria**: ALL PASS
- [PASS] status:api-up.status PASS (expected 200, got 200)
- [PASS] latency:api-up.latency_ms PASS (31.2 <= 500.0)
**observations**: 2/2 sources collected
**decisions**: 5 governing decision revisions (snapshot in the run row)
**wall**: 32.4s

run r-process-drill-8f2a1c — inspect with `mayhem inspect history r-process-drill-8f2a1c`
```

Reading the transcript, top to bottom:

1. **Status** — the run's machine state: `completed` (or `failed`/`aborted`
   with a non-zero exit and dirty-lease warnings).
2. **Verdict** — `pass`/`fail` derived from the success criteria you declared
   in the spec; `undecided` when a run aborts or the spec has no success block.
3. **Success criteria** — every criterion evaluated against real observations,
   one line each, so a failure tells you exactly what drifted.
4. **Observations** — how many configured evidence sources actually delivered
   data (probes, logs, metrics …).
5. **Decisions** — the governing decision revisions that shaped this run; the
   decision trace is queryable afterward via `mayhem inspect history`.
6. **Copy-paste handle** — the run id for the follow-up commands below.

From there: `mayhem inspect runs` lists recent runs, `mayhem inspect runs --run <run-id>`
shows full recorded metadata, and `mayhem inspect history <run-id>` replays the
complete event journal (every step, probe sample, and lease for that run).
Add `--debug` to `mayhem run` to stream each step live as it happens
(`[ok] injected proc.pause 10s into testcase-api`,
`[ok] recovered ... (compensation ok)`).

---

## Authoring a Drill

A drill is one `kind: drill` YAML file — the complete DSL reference lives in
[`docs/drill-spec.md`](docs/drill-spec.md). The shape:

```yaml
apiVersion: "mayhem/v1"
kind: drill
name: checkout-recovery
hypothesis: "checkout stays available while cart writes are throttled"

config:
  risk_ceiling: high
  max_faults: 1
  timeout: 30m
  recovery: true

containers:
  cart-api:
    faults:
      - fault: net.latency
        duration: 10s
        params:
          delay_ms: 300
          jitter_ms: 25

execution:
  - sequential: [cart-api]
  - check:
      - http: http://cart-api:8080/_health
        expect:
          status: 200
```

Validate with `mayhem prepare validate mayhem.yaml`; unknown parameters, out-of-range
durations, untargetable node kinds, and capability gaps are all compile-time
errors — before anything is injected.

Faults are drawn from the catalog (net.latency, proc.kill, net.packet_loss,
TLS failure, container pause, HTTP error injection, dependency and database
faults, k8s.pod_kill, k8s.node_drain, …). The full per-fault reference —
every id, its capabilities, risk level, and compensation contract — is in
[`docs/drill-spec.md`](docs/drill-spec.md#fault-catalog).

The `containers:` block above is the docker-family authoring shape. To fault a
Kubernetes workload or node — or mix runtimes in one spec — use the
cross-runtime `targets:` block instead (exactly one of `containers:` /
`targets:` defines a spec); see
[Targets (cross-runtime)](docs/drill-spec.md#targets-cross-runtime).

---

## Configuration

Runtime policies live in `mayhem.yaml` — the *configuration* file, distinct
from a `kind: drill` spec — auto-detected in the cwd or given with `--config`.
The effective view is one command away: `mayhem prepare config show` prints the
resolved configuration and provenance; `mayhem prepare config validate` refuses
unknown keys, a missing or wrong `apiVersion`, and out-of-range sections before
anything runs.

```yaml
apiVersion: mayhem/v1        # required; anything else is rejected
policy:
  allow_faults: null         # null = whole catalog; set to restrict
  deny_faults: []            # fault ids never injectable
  risk_ceiling: null         # tightened by the drill ceiling at plan time
  allow_critical: false      # config-side half of the critical opt-in
  critical_fault_acks: []    # per-fault acks; critical faults need allow_critical + ack + --allow-critical
blast_radius:
  max_services_pct: 50.0
  max_hosts: 2
  max_concurrent_faults: 3
  max_duration_per_fault_s: 300.0
  forbidden_fault_pairs: []  # pairs such as [net.packet_loss, net.bandwidth]
storage:
  path: mayhem.db
  artifacts_dir: .mayhem/artifacts
toolkit:
  binaries: {}               # pin a named tool's binary
runtime: docker              # docker | podman | kubernetes
target:
  containers: []             # explicit discovery targets without compose
kubernetes:
  context: null              # kubeconfig context
  namespace: null            # null = no namespace filter
recovery_grace: 300.0
log_level: INFO              # DEBUG | INFO | WARNING | ERROR
maniac:
  level: 2
  run_level: 10
  seed: null
```

Layering, in increasing precedence: **built-in defaults → selected YAML →
`mayhem.{profile}.yaml` → allowlisted environment variables → programmatic
CLI overrides**. Profile overlays are separate files selected with `--profile
NAME`; there is no `profiles:` key inside the base file. The environment layer
only honours `MAYHEM_STORAGE_PATH`, `MAYHEM_ARTIFACTS_DIR`, and
`MAYHEM_LOG_LEVEL`. The current CLI uses `--config` and `--profile` to select
layers; it does not expose a generic flag that maps arbitrary fields into
configuration.

Drill-level `config.risk_ceiling` composes with the policy ceiling and can
only tighten it.

The full configuration reference is in
[`docs/config.md`](docs/config.md).

---

## CLI surface

The active CLI is workflow-oriented: `discover`, `prepare`, `experiment`, `run`, `inspect`, `recover`, and `extend`. Guided `init` and `doctor` are active, and all legacy root commands and aliases have been removed. Exit codes, machine-readable fields, and database migrations remain stable.

The command inventory, global options, and stable exit codes are generated from
[`src/mayhem/cli/command_registry.py`](src/mayhem/cli/command_registry.py) and
[`src/mayhem/cli/exit_codes.py`](src/mayhem/cli/exit_codes.py).

Root options precede the command. Unique prefixes work at the root and in the workflow groups.

| Command group | Purpose |
|---------------|---------|
| `mayhem discover` | Discover topology, engines, faults, and capabilities. |
| `mayhem prepare` | Validate configuration, prepare dependencies, and compile plans. |
| `mayhem experiment` | Show, validate, and explore authored experiments. |
| `mayhem run`, `mayhem maniac` | Execute authored or randomized drills. |
| `mayhem inspect` | Inspect runs, history, coverage, next actions, leases, and diagnostics. |
| `mayhem recover`, `mayhem janitor` | Recover runs and clean leases. |
| `mayhem extend` | Inspect and extend faults, capabilities, dependencies, and providers. |
| `mayhem game-day` | Plan and run controlled game-day sessions with named approvals. |
| `mayhem bundle` | Verify a portable evidence bundle offline. |
| `mayhem completion` | Print a bash/zsh/fish completion script. |
| `mayhem campaign`, `mayhem commands`, `mayhem init`, `mayhem doctor`, `mayhem verify` | Manage campaigns, inspect the command map, onboard, diagnose, and verify evidence. |

Use each command's current `--help` output for accepted arguments. `mayhem commands show`
prints the live command map, and `tests/unit/test_cli_exhaustive_matrix.py` fails
when a registered command is missing from the active surface.

---

## Campaigns

A campaign groups authored drill-spec paths and runs them sequentially.

```bash
mayhem campaign create black-friday \
  --description "BFCM chaos" --hypothesis "checkout survives every single-fault failure"
mayhem campaign add-experiment black-friday mayhem.yaml
mayhem campaign add-experiment black-friday checkout-recovery.yaml
mayhem campaign start black-friday
mayhem campaign run black-friday --compose docker-compose.yml --execute
```

The current CLI creates campaigns in `draft`, `approve` moves a draft to
`approved`, `start` begins a draft or approved campaign, `run` executes the
stored spec paths, `pause` and `resume` stop and continue a running campaign,
and `archive` or `abort` sets the corresponding terminal status. The current
command surface does not expose campaign scheduling, priority ordering, or
policy/window editing. See the
`mayhem commands show`.

---

## What's new in v0.9.0

v0.9.0 is a truth-and-evidence release. It adds no new fault-injection power; it
makes what Mayhem already does **verifiable**. Every new surface below is
plan-only until explicitly executed, and every claim is recorded in evidence.

| Area | What it gives you | Where |
|------|-------------------|-------|
| Explicit intent | Every mutating command needs an approval; `--dry-run` previews and returns before any engine exists. | `mayhem run --execute`, `src/mayhem/cli/command_registry.py` |
| Runtime context | Engine, target, and namespace resolved once and propagated; a target-type mismatch is refused **before** any lease or subprocess. | `mayhem doctor`, evidence `k8s_context` |
| Capability truth | What each fault can actually do per engine, with a reason and a source of truth for every blocked row. | `mayhem discover capabilities [--explain] [--blocked]` |
| Replay capsules | A versioned, digest-checked capsule per run; export and validate offline. | `mayhem inspect replay export\|validate RUN_ID` |
| Redaction | Redaction enforced at the write boundary, so a directly-built envelope cannot leak; counts-only metrics in evidence. | evidence `redaction_metrics` |
| SLO criteria | Provider-neutral thresholds with explicit units and windows; a missing observation **fails** rather than passes. | `slo:` in the [drill spec](docs/drill-spec.md#slo-criteria-v090) |
| Scenarios | Typed variables, time windows, and conditional steps compiled to a deterministic plan. | `mayhem experiment compose` |
| Coverage graph | Service × fault × engine coverage with blocked runs excluded, plus baseline diffs. | `mayhem inspect graph`, `mayhem inspect coverage-diff` |
| Campaign resume | Durable checkpoints; a verified experiment is never repeated without `--retry-verified`. | `mayhem campaign resume-plan` |
| Residual impact | Before/after comparison proving the system came back; an unavailable source reads `unavailable`, never `clean`. | `mayhem inspect residual` |
| Game days | Sessions with a freeze window, named approvers, and dual control for critical faults. | `mayhem game-day` |
| Provider sandbox | Default grant is read-only; signed fault packs; unsigned is development-only. | `src/mayhem/providers/`(docs/provider-sdk.md) |
| Observability | Read-only Prometheus/Loki connectors (bounded timeout, response-size cap, redacted errors) plus local OpenTelemetry spans. | `src/mayhem/observability/` |
| Evidence bundles | Hash-chained, offline-verifiable bundles of a run's evidence. | `mayhem bundle verify PATH` |

Two honesty rules are worth stating plainly, because they change what output
means:

- **No backend means no effect.** `start_load`, `stop_load`, and `notify` have
  no executor yet. They report `acknowledged_no_backend` and the run is *not*
  clean — they never look like success.
- **No live verification is claimed without a live run.** The capability
  dashboard reports `live=false` for every row in this repository; `live=true`
  can only come from a recorded live run.

### Shell completion

`mayhem completion SHELL` prints a completion script to stdout. Nothing is
installed and nothing is executed — you choose where it goes. Completion is
dynamic, so the candidates always match the commands your installed version
actually has:

```bash
mayhem completion bash >> ~/.bashrc
mayhem completion zsh  > "${fpath[1]}/_mayhem"
mayhem completion fish > ~/.config/fish/completions/mayhem.fish
```

`bash` needs **bash >= 4.4**; macOS ships 3.2, where dynamic completion is not
available — use `zsh` or `fish` there, or install a newer bash. The generated
bash script says so in its header.

```bash
mayhem --version                     # installed version, or 0.0.0+source in a checkout
mayhem discover capabilities --explain k8s.pod_kill
mayhem inspect replay export RUN_ID --out bundle/
mayhem bundle verify bundle/
```

### v0.9.0 status

The v0.9.0 planning package was scaffolding and has been removed; the behavior
it described is what is implemented and documented above. What remains verified:

| Gate | Result |
|------|--------|
| `pytest tests/unit tests/integration tests/e2e` | PASS |
| Coverage of the v0.9.0 modules | 100% on 14 of 15; `domain/scenarios.py` 99% (3 unreachable defensive lines) |
| `uv build --sdist --wheel` + `scripts/verify_release_artifacts.py` | PASS (9 checks) |
| Wheel install, import, console scripts, `mayhem --version`, `python -m mayhem` | PASS |
| GitHub Actions on `v0.9.0` | PASS — unit, integration, e2e, schema, ruff-fatal, package build, wheel smoke, artifact verification |
| Repository-wide `ruff` / `mypy` / `lint-imports` | Pre-existing failures, advisory only (not release-gating) |
| Live Kubernetes / remote-agent execution | Not claimed, and not enabled by default |

---

## Safety Model

- **Risk ceilings.** Every catalog fault carries a risk level; injection is
  refused when either the policy or the drill ceiling is exceeded.
  `critical`-risk faults (e.g. `k8s.node_drain`) need a **triple opt-in**:
  `policy.allow_critical: true`, a per-fault ack in `policy.critical_fault_acks`,
  and the `--allow-critical` CLI flag.
- **Concurrency budget.** `max_faults` caps simultaneously-injected faults;
  a wider `parallel:` step queues into rounds.
- **Duration caps.** Per-fault `duration` beyond the catalog maximum is a
  compile error.
- **Capability gating.** Faults declare the capabilities they need
  (docker engine, kubernetes_engine, net_admin, process control, …); the plan
  is proven against the live graph by the impact gate before run — never
  assumed.
- **Compensation contracts.** Reversible faults run their declared inverse;
  irreversible ones are followed by workload reconciliation. A failed round
  aborts-and-recovers its own faults first, then propagates.
- **Lease hygiene.** Fault rounds hold leases with a TTL; `mayhem janitor`
  sweeps orphaned leases and expires pending runs, `mayhem recover` repairs a
  run's orphans on demand.

---

## Exit Codes

The stable identifiers and numeric values are defined in
[`src/mayhem/cli/exit_codes.py`](src/mayhem/cli/exit_codes.py) and documented in
`src/mayhem/cli/exit_codes.py`. The deterministic
documentation test rejects identifiers that are not declared in source.

---

## Architecture

The pipeline is staged so everything expensive is done up front and execution
is as small as possible:

1. **Discover** — compose discovery builds a Docker/Podman graph. Kubernetes
   has separate live-discovery and offline-manifest providers; the manifest
   provider creates logical placeholders, not live pod selections.
2. **Prepare** — `mayhem prepare config` layering (defaults → selected YAML → separate
   profile overlay → allowlisted environment values → programmatic overrides),
   plus topology, drift detection, and target revalidation.
3. **Compile & plan** — the drill spec becomes a frozen `ExecutionPlan` with
   step sequences, per-fault compensations, success criteria, and observability
   sources; fault, target, capability, and duration inputs are validated
   against the current models.
4. **Execute** — supported runtimes execute inject → hold → compensate rounds
   and record evidence. Kubernetes planner/executor presence does not by itself
   establish a reachable cluster or an available capability.
5. **Recover & report** — `mayhem janitor` sweeps orphaned leases; `mayhem
   recover status|plan|execute`, `mayhem inspect runs`, and `mayhem inspect
   history` replay recorded evidence.

**Documentation**

| Document | Contents |
|----------|----------|
| [`docs/README.md`](docs/README.md) | Documentation authority, classifications, source-of-truth map, and Kubernetes status vocabulary. |
| [`docs/drill-spec.md`](docs/drill-spec.md) | Drill DSL reference. |
| [`docs/config.md`](docs/config.md) | Current layered configuration contract. |
| `src/mayhem/cli/command_registry.py` | Current commands, options, and stable exit codes. |
| `src/mayhem/schemas/output_v1.json`(docs/reference/output-schema.md) | Versioned machine-output envelope and the v0.9.0 compatibility boundary. |
| [`docs/fault-catalog/`](docs/fault-catalog/README.md)(docs/reference/fault-catalog.md) | Checked Kubernetes fault catalog status snapshot and capability gates. |
| [`docs/compensation.md`](docs/compensation.md) | Compensation lifecycle and verification contracts. |

---

## Status

| Area | Status |
|------|--------|
| Compose-oriented Docker/Podman workflow | Documented user path |
| Layered configuration, drill planning, SQLite evidence, CLI exit codes | Current checked-in references |
| Kubernetes manifest topology | Supported as an offline planning input; blueprint pods are placeholders |
| Kubernetes planner | Supports normalized `targets:` scopes and frozen-plan metadata |
| Kubernetes executor/resolver seams | Present in source and unit-tested with fakes; availability is runtime/capability dependent |
| Legacy `KubernetesAdapter` | Compatibility seam only; reports unavailable |
| Live Kubernetes cluster acceptance | Not claimed by repository documentation |
| Catalog-only Kubernetes faults | `k8s.image_pull_slow`; excluded from the available-fault register and refused before mutation |
| v0.9.0 core truth work (intent, admission, capability truth, replay, redaction) | Implemented; unit + integration tested |
| v0.9.0 expansion work (coverage graph, SLOs, scenarios, resume, residual, game day, sandbox, connectors, bundles) | Implemented; plan-only until executed |
| Web UI / REST API | Planned |

### Kubernetes status

The [examples/k8s README](examples/k8s/README.md) and
[documentation authority index](docs/README.md#kubernetes-status-vocabulary)
explain the separate Kubernetes states. Historical discovery reports and the
discovery reports are retained in git history for traceability and must not be
treated as live-cluster evidence.

---

## Development

```bash
# Clone, then sync the dev dependency group (requires https://docs.astral.sh/uv/):
uv sync --group dev

# Run all tests
uv run pytest

# Check the release truth baseline (docs, Justfile, packaging vs. source)
uv run pytest tests/unit/test_release_contract.py

# Build the sdist and wheel
uv build

# Lint
uv run ruff check src/

# Type check (strict is configured per module)
uv run mypy
```
