Metadata-Version: 2.4
Name: bootproof
Version: 0.2.0
Summary: Pre-reboot boot verification: boot a snapshot in an isolated VM, run the service-up checklist, and print GO or NO-GO before you reboot the real box
Author: Calas Automations Inc.
License: MIT
Keywords: reboot,boot,verification,qemu,kvm,systemd,ci
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# bootproof — pre-reboot boot verification

Every sysadmin fears rebooting a long-running server. Services that
"always worked" fail after a kernel update, a config edit, or a cert
rotation — and you only find out when prod is down.

bootproof snapshots the machine, boots the snapshot in an isolated
VM, runs a service-up checklist (ports listening, systemd units
active, health endpoints green), and gives a go/no-go **before**
you reboot the real box. One command: `bootproof check`.

The verdict is the whole UX: a single GO or NO-GO, with every
failing check named exactly — which port, which unit, which
endpoint, and what was observed.

**Status:** bootproof has not yet been run against a real KVM guest.
The `--boot qemu` path is unit-tested for command shape and failure
modes only; see "What was live-tested here vs. simulated" below.

## Install (local)

```sh
pip install .
# or, for development:
pip install -e .
# or without installing at all:
PYTHONPATH=. python -m bootproof check ...
```

Requires Python 3.9+. No third-party dependencies, no keys, no
accounts, no telemetry, no phone-home. Linux-only in v1.

## Usage

```sh
# Write example files (checklist, snapshots, two fixtures)
bootproof init --output bootproof-examples

# List available snapshots
bootproof snapshots --snapshots snapshots.json

# Draft a checklist from this machine (review it before use)
bootproof capture            # writes bootproof-checklist.yaml

# Static host boot checks, no VM (read-only)
bootproof preflight

# Validate a checklist without booting anything
bootproof validate --checklist bootproof-checklist.yaml

# The verdict: boot a snapshot, run the checklist
bootproof check --checklist bootproof-checklist.yaml \
  --snapshots snapshots.json --snapshot snap-2026-10-27 \
  --fixture fixture-go.json

# Real QEMU boot, with guest unit checks over SSH
bootproof check --checklist bootproof-checklist.yaml \
  --snapshot-dir /var/lib/libvirt/images --boot qemu \
  --ssh-user root --ssh-key ~/.ssh/id_ed25519

# Machine-readable output on every command
bootproof check ... --json
```

### Drafting a checklist: `bootproof capture`

`bootproof capture` writes a DRAFT `bootproof-checklist.yaml` from
this machine. It never runs a check.

- Ports: listening TCP sockets from `/proc/net/tcp` and
  `/proc/net/tcp6`. Ports bound to `0.0.0.0`, `::` or loopback are
  written with host `127.0.0.1`; a port bound only to one specific
  address is written with that address.
- Units: `systemctl list-units --type=service --state=running`,
  kept only if `systemctl is-enabled` says the unit starts at boot.
  Running-but-disabled units are listed as comments, not checks.
  Without `systemctl`, only ports are captured and the file says so.
- The first line reads `DRAFT captured from <host> at <UTC>; review
  before use.` An existing file is not overwritten without
  `--force` (exit 2).

### Static host checks: `bootproof preflight`

`bootproof preflight` reads the host's files and boots nothing. Its
report says `boot: preflight (static host checks — not a boot)`.
Same verdict rules and exit codes as `check`.

- **fstab:** every `UUID=`, `LABEL=` and `PARTUUID=` entry must exist
  under `/dev/disk/by-*`, unless its options include `nofail` or
  `noauto`. An unresolved entry is FAIL: a real boot would stop in
  emergency mode.
- **initramfs:** every `/boot/vmlinuz-<v>` needs an `initrd.img-<v>`,
  `initramfs-<v>.img` or `initrd-<v>`. A missing one is FAIL.
- **/boot space:** under 50 MB free is WARN. A warning does not
  block GO.
- **dkms:** if `dkms` is installed, `dkms status` must show every
  module `installed` for the newest `/boot` kernel (by version
  sort). Anything else (for example `built`) is FAIL, naming the
  module. Without `dkms` the check is reported as skipped, not passed.

`--root DIR` inspects another filesystem root instead of `/`.

### Checklist file

JSON, or a small YAML subset (no PyYAML needed — stdlib only).
Three check types:

```yaml
checks:
  - type: port
    port: 443
  - type: port
    port: 5432
  - type: unit
    name: nginx.service
  - type: http
    url: http://127.0.0.1:8080/health
    expect_status: 200
    body_contains: ok
```

Defaults: port host `127.0.0.1`, http `expect_status` 200,
per-check timeout 5s. An empty checklist is an error, not a GO —
a verdict over zero checks would verify nothing (enforced in the
verdict engine itself, not just the CLI loader, so library use
cannot bypass it).

### QEMU mode: guest probing, not host probing (v0.2.0)

In `--boot qemu` mode, checklist checks are probed against the
**guest**, never the host:

- **Port forwarding map:** `QemuRunner` allocates a host port for
  every loopback port/HTTP check in the checklist and forwards it
  into the guest (`hostfwd`). Observed text names the *guest*
  port, never the host port.
- **Unit checks over SSH:** `--ssh-user` + `--ssh-key` run
  `systemctl is-active` inside the guest over the forwarded SSH
  port (BatchMode, no password prompts, missing `ssh` binary is a
  probe error). Without credentials, a unit check in qemu mode is
  an ERROR — host units are never substituted for guest units.
- **Non-loopback and HTTPS checks are errors in qemu mode:** a
  guest's LAN IP is not reachable from the host without bridging,
  and TLS over a forwarded port cannot present the guest's
  certificate. Use loopback `http://` URLs.
- **Readiness:** with SSH credentials, readiness is an actual SSH
  command success (not just a TCP connect); without them, it is a
  TCP connect to the forwarded SSH port.
- **Disk format** is detected from the snapshot suffix (`.img` /
  `.raw` → raw, `.vmdk` → vmdk, else qcow2); **`-machine
  accel=kvm`** is added only when `/dev/kvm` exists (otherwise
  QEMU falls back to TCG — slow, but functional).
- **YAML `#` comments** are stripped only when the `#` starts the
  line, follows whitespace, and is not inside quotes — URL
  fragments (`page#frag`) and `body_contains` values containing
  `#` are data, not comments.
- **Per-check latency** is measured and reported (slowest 3 checks
  in the text report, per-check `seconds` in JSON), and the report
  names the **disk path** that was booted.

### Snapshot sources

- `--snapshots FILE`: JSON file, `{"snapshots": [{"id", "name",
  "source", "path"?}]}`. The fixture source used in this sandbox.
- `--snapshot-dir DIR`: scans for `*.qcow2` / `*.img` / `*.raw`
  disk images; each entry carries its path, which is what a QEMU
  boot needs.

`--snapshot` may be omitted when the source holds exactly one
snapshot. A missing id, or several snapshots with no id given, is
an error — never a silent default.

### Exit codes (CI-usable)

| Code | Meaning |
|------|---------|
| 0 | GO — every check passed |
| 1 | NO-GO — one or more checks observed down |
| 2 | Error — missing snapshot, bad checklist, boot timeout, or any check that could not be verified |

A GO requires every check to pass. A check that cannot be
determined (probe error, target not booted) is an error result,
never a pass — bootproof never prints GO on unverified checks.

## Example GO output

```
$ bootproof check --checklist examples/bootproof-checklist.yaml \
    --snapshots examples/snapshots.json \
    --fixture examples/fixture-go.json
bootproof verdict: GO
snapshot: snap-2026-10-27
boot: simulated
checks: 4 passed, 0 failed, 0 errored, 4 total
GO — every service-up check passed. Safe to reboot.
  PASS: port 443 listening on 127.0.0.1 — port 443 on 127.0.0.1 listening
  PASS: port 5432 listening on 127.0.0.1 — port 5432 on 127.0.0.1 listening
  PASS: unit nginx.service active — unit nginx.service active
  PASS: endpoint http://127.0.0.1:8080/health returns 200 containing 'ok' — http://127.0.0.1:8080/health HTTP 200
```

## Example NO-GO output (failing check named)

```
$ bootproof check --checklist examples/bootproof-checklist.yaml \
    --snapshots examples/snapshots.json \
    --fixture examples/fixture-nogo.json
bootproof verdict: NO-GO
snapshot: snap-2026-10-27
boot: simulated
checks: 3 passed, 1 failed, 0 errored, 4 total
NO-GO — the following checks failed:
  FAIL: port 5432 listening on 127.0.0.1 — port 5432 on 127.0.0.1 closed
  PASS: port 443 listening on 127.0.0.1 — port 443 on 127.0.0.1 listening
  PASS: unit nginx.service active — unit nginx.service active
  PASS: endpoint http://127.0.0.1:8080/health returns 200 containing 'ok' — http://127.0.0.1:8080/health HTTP 200
```

Exit code 1. `--json` gives the same verdict as JSON with a
`boot_mode` field, a per-check `status`, and the `observed` value
for each check.

## Error honesty

```
$ bootproof check ... --snapshot nope ...
error: snapshot not found: nope (available: snap-2026-10-27)
# exit 2

$ bootproof check ... --boot qemu   # on a host without QEMU
error: QEMU binary not found: qemu-system-x86_64 — install qemu-system and enable KVM (/dev/kvm) to run a real boot
# exit 2 — never a silent fallback to simulation
```

Other specific errors: ambiguous snapshot inventory, missing
checklist file, unknown check type, bad port/status values, boot
timeout (guest not reachable within `--timeout`, default 120s).

## What was live-tested here vs. simulated — read this first

This build was developed in a sandbox with **no `/dev/kvm`, no
QEMU, no `virsh`, no restic/borg**. That constraint shaped what
could be proven live, and bootproof labels it everywhere instead
of papering over it:

**Live-tested in this sandbox (real code, real execution):**
- The full verdict engine: GO / NO-GO / ERROR, mixed results,
  error-beats-fail, exit codes 0/1/2.
- Checklist parsing and validation: JSON + YAML subset, all three
  check types, pass and fail cases per type, empty/unknown/malformed
  checklists rejected.
- Real local probes: TCP port checks against a live socket server
  and HTTP checks against a live local HTTP server (pass, wrong
  status, unreachable — all observed, not mocked).
- Snapshot inventory: fixture JSON source and directory scan of
  real `*.qcow2`/`*.img` files.
- Error honesty: missing snapshot, ambiguous snapshot, missing
  source, bad checklist, boot timeout (via fixture boot delay),
  missing QEMU binary.
- End-to-end CLI runs through `python -m bootproof` and through an
  installed `bootproof` entry point in a clean venv.
- `capture` against fixture `/proc/net/tcp*` text and a fake
  `systemctl`; never run on a real host here.
- `preflight` against fixture directory trees (fstab, `/dev/disk`,
  `/boot`) and a fake `dkms`; never run on a real host here.

**Fixture / simulated (labelled `boot: simulated` in every report):**
- The booted guest itself. `--boot simulated` (the default) answers
  probes from a fixture JSON file describing guest state. An
  unlisted port/unit/endpoint fails loudly — fixture silence is
  absence, never success.
- The QEMU boot path could not execute here: `QemuRunner` and
  `build_qemu_command` are real code (command construction with
  `snapshot=on` so the guest never writes the source image, user
  networking with SSH forwarding, readiness loop, timeout kill,
  teardown), unit-tested for command shape and failure modes, but
  no guest was actually booted in this sandbox. **v0.2.0 fixed the
  two real-path correctness holes the panel found in this code**
  (host probing instead of guest probing for ports and units —
  see "QEMU mode" above); the fix is unit-tested against real
  sockets and fake `ssh` binaries, but a real guest boot still
  awaits a host with KVM + a guest image.

**What a real host needs for the full path:**
- Linux host with KVM (`/dev/kvm`) and `qemu-system-x86_64`
  installed.
- A snapshot with a disk path (directory snapshot source, or a
  snapshots file entry carrying `path`).
- `bootproof check --checklist ... --snapshot-dir <dir> --boot qemu`
  (loopback checklist ports/URLs are forwarded into the guest
  automatically; add `--ssh-user` + `--ssh-key` for guest unit
  checks and SSH-based readiness).
- On such a host the report reads `boot: qemu`, not `boot:
  simulated`. Until then, treat every verdict here as a
  checklist-engine verdict over simulated guest state — which is
  exactly what its `boot:` line says.

`--local` is a third mode: it probes the current machine directly
(no snapshot, no boot) as a checklist dry run, labelled
`boot: local`. It is not a boot verification.

In `--local` mode a unit check also runs `systemctl is-enabled`.
`enabled`, `static`, `indirect`, `generated`, `alias` and
`enabled-runtime` pass. `disabled`, `masked` and `linked` fail with
"active now but <state> — will not start at boot unless pulled in by
another unit". Any other answer is an error, never a pass. Set
`require_enabled: false` on a unit check to skip this. QEMU mode
does not check it: the guest boot itself is the proof.

## What v1 does NOT do (deliberately)

- Fleet dashboards, scheduled checks, team-tier features — the
  C$19/mo tier is a later product, not v1.
- Windows support — Windows boot semantics are a second product.
- VMware estates, restic/borg/Proxmox snapshot sources — the
  `SnapshotSource` interface is defined for them, but they are not
  built in v1.
- Any paid gating.

## Tests

```sh
python -m pytest -q
```
