Metadata-Version: 2.5
Name: fleet-sensor-baseline
Version: 0.2.1
Summary: Which units differ from their cohort, and what changed across time.
Project-URL: Homepage, https://github.com/james-sheen/fleet-sensor-baseline
Project-URL: Source, https://github.com/james-sheen/fleet-sensor-baseline
Project-URL: Issues, https://github.com/james-sheen/fleet-sensor-baseline/issues
Author: James Sheen
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: baseline,bmc,drift,fleet,hardware,openbmc,redfish,sensors
Requires-Python: >=3.10
Provides-Extra: collect
Requires-Dist: bmc-sensor-audit<0.3,>=0.2.0; extra == 'collect'
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Description-Content-Type: text/markdown

# fleet-sensor-baseline

**Which units differ from their cohort, and what changed on this unit across time.**

`bmc-sensor-audit` judges one machine against one declaration. This layer holds the
list of machines and the history of captures, and answers the two questions the
referee cannot.

**Released — 0.2.1**, tagged `v0.2.1`, Apache-2.0, on PyPI as
`fleet-sensor-baseline`.

**0.2.1 closes the seam with the referee, and fixes a validator that could
not check the one file it most needed to.** `baseline --for-referee` writes the
referee's declaration-source format, which both projects described and neither
produced; it comes out as a candidate nobody has signed, and it does not declare
the sensors the cohort disagreed about. Separately, `validate` refused every
`targets/2` file while naming `targets/2` among the formats it read — so the
format that exists to carry `pin_sha256` was the one the checker would not check.

**0.2.0 is a compatibility break, and it changes an answer.** The presence
threshold was a single line, so a sensor either belonged to the baseline or was
foreign to it — and at rack scale the majority won. On 24 trays where 2 lost a
sensor, the sensor left the baseline and the **22 healthy trays** were reported
as carrying something unexpected while the 2 that broke came back clean. There
is a third state now: at or above `--present-threshold` it is expected, at or
below `--absent-threshold` it is foreign, and between them the **cohort
disagrees with itself**, which is a fact about the cohort and is charged to no
unit. The baseline format goes to `fleet-baseline/2`; a `/1` file is refused
rather than upgraded, because what is needed to judge was dropped when it was
derived. `walk_ref` is derived instead of required, and an absent referee is
told from a mute one.

---

## The problem

A tray reported forty-five sensors before a firmware update and forty-two after it.
Every remaining sensor is healthy. Every threshold is in range. A single-machine
audit finds nothing, because the machine agrees with itself — only the previous
capture disagrees, and nothing was holding it.

Scale that sideways: five trays in a rack of two thousand quietly stopped exposing
a fan sensor after a board revision. Each one, audited alone, looks fine.

Those are the two axes here. **Vertical** is one unit across time. **Horizontal**
is one unit against its cohort.

## Install

```
pip install fleet-sensor-baseline
```

The core has **no dependencies**. Ingest, baseline, outliers, drift, verdict and
validate are JSON and arithmetic, so the vertical and horizontal axes run on a jump
host with nothing provisioned. The collector needs the referee on PATH:

```
pip install "fleet-sensor-baseline[collect]"
```

## Use

```
# file the captures somebody gathered
fleet-sensor-baseline ingest --store fleet-store --payloads walks/ records/*.json

# what changed on one unit, across time and firmware
fleet-sensor-baseline drift --store fleet-store --unit h-0042

# derive a cohort baseline, then find the units that differ from it
fleet-sensor-baseline baseline --store fleet-store --model GB200-NVL-tray \
    --firmware-range ">=1.4,<1.5" --out baseline.json
fleet-sensor-baseline outliers --store fleet-store --baseline baseline.json

# the fleet run: every expected unit must have reported
fleet-sensor-baseline verdict --store fleet-store --expect-units racks.txt

# check any artifact against the format it declares
fleet-sensor-baseline validate baseline.json

# hand the cohort's baseline back to the referee, as a candidate
fleet-sensor-baseline baseline --store fleet-store --model GB200-NVL-tray \
    --out baseline.json --for-referee declaration.json

# walk a rack of BMCs yourself, serially, and file what comes back
fleet-sensor-baseline collect --targets rack-17.json --store fleet-store \
    --collector-id rack-17
```

### Handing a baseline back to the referee

`--for-referee PATH` writes the cohort's baseline a second time, in
`bmc-sensor-audit/fleet-baseline/1` — the referee's own declaration-source
format, for `coverage --declaration`. It is the third source in that tool's
precedence, under entity-manager and under a reviewed `pdr/1`.

**It comes out as a candidate.** `"reviewed": null`, so the referee refuses to
consume it until a person adds their name and a date. The conversion is not the
review: a baseline derived from a fleet of unprovisioned boards is an empty
declaration that reads healthy against every other unprovisioned board, and at
fleet scale consensus makes that answer look corroborated. A marker this tool
could write is a marker nobody put their name to, so there is no flag to write
one.

**The sensors the cohort disagreed about are not declared.** That band exists
because a proportion is a coarse instrument at rack scale; the referee's format
has two states, not three, and declaring a divergent sensor would expect it of
every unit while 8 percent of the fleet does not have it — charging each of
those units a finding for a fact about the cohort. That is the 0.1.x inversion,
reached through a different door. They are written into the file under
`divergent_not_declared`, which the referee ignores by its own `/1` rule, and
named on stderr at export: the reviewer is the reader who needs them.

The cohort must be scoped to one model — `--model`, or `--for-referee-platform`
to name it directly — because the referee requires a platform on every
declaration and a cohort spanning models has no single answer.

`tests/test_seam.py::TestTheExportedDeclarationIsWhatTheRefereeReads` runs the
real `bmc-sensor-audit` against a file this emits: refused as a candidate,
consumed once reviewed, and the unit that lost the divergent sensor coming back
clean — with the counterfactual that declares it anyway and watches the same
unit get charged.

### Asking before walking

```
fleet-sensor-baseline collect --targets rack-17.json --store fleet-store \
    --collector-id rack-17 --etag-cache
```

The referee keeps one collection-ETag cache per BMC **surface** in the store and
asks each machine whether its sensor set changed before walking it — a handful of
requests instead of one per sensor.

**It proves membership, and a record filed from a skip says so.** The record
reuses the previous capture's payload and carries
`{"basis": "collection-etag", "proves": "membership"}`. That is exactly what
`drift`, `outliers` and `verdict` need, since all three read the name set; it is
not enough for a threshold audit, and the record is explicit rather than looking
like a fresh walk. A BMC that does not implement ETags is walked every time.

Needs `bmc-sensor-audit>=0.1.2`.

`collect` is the only subcommand that needs the referee on PATH. It walks one
BMC at a time with exponential backoff, because AST2600-class BMCs measure a
Redfish walk in seconds and a central plane that fans out to ten thousand of
them is a denial of service with a scheduler. **A walk that fails is a record,
emitted** — a unit that could not be walked reports *as* could-not-walk and
keeps its place in the denominator.

Credentials are named, never stored: a target carries `password_env`, the name
of an environment variable, and a `password` key in a targets file is refused
rather than ignored.

**A certificate pin is the opposite, and belongs in the file.** A SHA-256
fingerprint of a public certificate is public — anyone who can connect can
compute it — so it is not a credential but an *expectation*, and one that
changes should show up in a review diff. Declare it per target in a
`targets/2` file:

```json
{"unit_key": "h-0042", "base_url": "https://192.0.2.1",
 "pin_sha256": "AB:CD:...:EF"}
```

**A pin needs `targets/2` and version 2 exists for exactly that.** A reader
predating the key would ignore it and connect unpinned — a declaration met with
silence — and it refuses an unknown format outright, so the bump turns a silent
downgrade into a refusal.

A CA bundle is a path on one operator's disk and usually fleet-wide, so it is
`collect --cafile PATH` rather than a field in a list that goes into version
control.

## Exit codes

| code | means |
|---|---|
| `0` | clean |
| `1` | findings, and they are named |
| `2` | the check could not be completed, and it says which part |

Precedence is `max`, and **`2` beats `1` deliberately**. A run that found three
outliers and could not reach a fourth unit has not found three outliers; it has
found three and does not know about the fourth. Anything outside `{0,1,2}` from a
subprocess is read as `2` with the raw code kept beside it — *"exited 127"* is the
useful half of that sentence.

**"N units could not be walked" is printed, never omitted.** A fleet report that
renders incompleteness as silence renders a dead collector as a clean rack.

## What this is not

1. **Never a dashboard.** The verdict is an exit code. Visualisation may exist and
   is never the source of judgment. Silence cannot impersonate a pass.
2. **Never metric values.** This audits presence and configuration, not readings.
3. **Never majority-truth over declarations.** A fleet-derived baseline is an
   *additional, labeled, downgraded* declaration source — reachable through
   `baseline --for-referee`, which writes the referee's own format as a
   candidate nobody has signed yet. Wherever a manufacturer declaration exists,
   it wins.
4. **Never identity inside the referee's artifacts.** The walk payload carries no
   `unit_key`. Identity binds here.

### Point 3 is the one that matters

A baseline derived from a fleet **cannot see an absence the whole cohort shares.**
Two thousand trays that all lost the same sensor in the same firmware agree with
each other perfectly, and consensus reports them clean.

So every `fleet-baseline/2` carries this sentence as **part of the format**, and
every consumer prints it verbatim:

> This baseline was derived from the fleet, not declared by a manufacturer. It
> cannot see an absence the whole cohort shares.

`tests/test_scenarios.py::TestS2CommonModeBlindness` demonstrates the blindness on
purpose, and pairs it with `bmc-sensor-audit coverage` finding the same absence
against a manufacturer declaration. The pair of assertions *is* the precedence
rule, made executable.

## Formats

Six, each versioned and each with a shipped validator, because **the person who
receives the file is the one who needs to check it**:

| format | is |
|---|---|
| `fleet-sensor-baseline/fleet-record/1` | one capture of one unit |
| `fleet-sensor-baseline/fleet-baseline/1` | superseded; refused, because it dropped the sensors the cohort disagreed about |
| `fleet-sensor-baseline/fleet-baseline/2` | a derived declaration, labeled as such, carrying `divergent` |
| `fleet-sensor-baseline/summary/1` | a verdict, in the family's one vocabulary |
| `fleet-sensor-baseline/targets/1` | the collector's rack list |
| `fleet-sensor-baseline/targets/2` | the same, and may declare `pin_sha256` |

`fleet-sensor-baseline validate PATH` checks any of them, dispatching on the format
key the file declares rather than on a shape guessed from the fields present.

See [docs/formats.md](docs/formats.md).

## When the cohort disagrees with itself

A sensor is **expected** at or above `--present-threshold` (0.99), **foreign** at
or below `--absent-threshold` (0.01), and between the two the cohort simply
disagrees. That middle band is reported once, against the cohort, naming the
minority -- and charged to no unit.

The band is not a refinement, it is a correction. Without it a sensor on 22 of 24
trays was below 0.99, so it left the baseline entirely, and every unit that
**had** it was reported as carrying something unexpected:

    tray-01 .. tray-24 (22 of them)   findings -- unexpected: Fan_CPU_2
    tray-06, tray-07                  clean

The 22 healthy trays were the outliers and the two that had lost the sensor
passed. A proportion is a coarse instrument at rack scale -- 0.99 of 24 is
23.76, so one deviant unit crosses it, and `--floor` admits cohorts from 20. The
README's own opening case inverted between 20 and about 100 units.

It now reads:

    divergent: Fan_CPU_2 present on 22 of 24 unit(s). The cohort disagrees with
    itself, so this is reported here and charged to no unit; the 2 that do not
    report it: tray-06, tray-07

`tests/test_scenarios.py::TestS6TheCohortThatDisagreesWithItself` pins it.

## Storage

An append-only JSONL index and a content-addressed store. Corrections are new
lines, never edits; the reader takes the latest per surface and capture time.
**The store does not deduplicate captures, and an earlier version of this line
said it did.** A `walk/1` carries per-fetch latencies and a capture time, so two
walks of one unchanged machine never share a digest. What avoids re-storing an
unchanged walk is not walking it — see `collect --etag-cache` below.

**No time-series database in 0.x**, and that is a measurement rather than a
preference: configuration drift is per-boot and per-firmware-event, not
per-second. Revisit when cross-month queries exist and are slow.

## A unit is a tuple

On NVIDIA-class platforms one physical unit answers on more than one BMC — a host
BMC and an HMC behind bmcweb aggregation. Two records differing only in `satellite`
are two surfaces of one machine, not two machines. Pairing across time is surface
to surface; presence across the cohort is the union across a unit's surfaces.

## The boundary

This package **never imports `bmc_sensor_audit`**. It reads exit codes, stdout and
the files the tool writes. One module is exempt — `collect/backends/mock.py` — and
the distinction is the design: it stands in for the thing being walked, not the
thing doing the walking.

It imports the referee's mock BMC *and* its `RedfishClient` and `walk_chassis`, so
the payloads it produces are real `walk/1` files written by the real walker rather
than approximations of them. The exemption is file-level and that is the actual
contract `tests/test_boundary.py` enforces; an earlier version of this sentence
named only the mock, which was narrower than the code and would have made a
correct file look like a violation.

`tests/test_boundary.py` asserts both halves by reading the source: that nothing
else imports it, and that the mock backend still does. Without the second, the
first would pass by finding nothing.

## Tests

| | count |
|---|---|
| tests collected | 341 |
| of those, requiring `bmc-sensor-audit` | 37 |

**The predicate**: `pytest --collect-only` over the test files git tracks, and
the same again with `-m seam` for the second row. Collection rather than a pass
tally, because a skip count is true only on the machine that measured it —
`tests/test_readme_counts.py` derives both and fails if either drifts.

Run it dependency-free and the 37 skip. Install the referee and they run.

**No pass/skip tally is quoted here on purpose.** The first version of this
section did, and both numbers were wrong within a day — not because tests
changed, but because the repository gained a tag and one check stopped skipping.
A tally is a fact about the machine that measured it; the collected counts above
are facts about the suite, and something derives them.

Every skip **says in prose why it could not run, and exits clean**. *Could not
check* is a different answer from *found nothing*, and a suite that reported the
two identically would let a missing dependency read as a green seam.

The dependency-free lane's walk fixtures are hand-built, which is a liability this
suite names rather than hides: a fixture written from a reading of a format agrees
with that format by construction. `tests/test_seam.py` is the answer — it generates
a walk with the referee's own reader and asserts the fixture still matches.

## Upstream

Pinned at `bmc-sensor-audit>=0.2.0,<0.3`, and the floor is **derived, not
chosen**. It has moved three times, each time to consume something reported from
here:

- `>=0.1.2` for `--password-env`, so a credential never crosses argv, and for
  declaring an added aggregation prefix in one entry.
- `>=0.1.5` for `--pin-sha256` and `--cafile` — and for the two defects taking
  them found. A pin on an `http://` target was silently ignored (urllib picks a
  handler by scheme, so the pinned handler was never consulted), and once that
  was refused, the refusal escaped as a traceback exiting `1` — *findings* — so
  a collector read a misconfigured flag as a machine with problems.
- `>=0.2.0` for `--version`, which is what makes the referee probe enforcing. It
  reads the version from the referee that answers on `PATH`, not the one pip
  resolved into this environment, because those are different objects and the
  guard that checked the second one passed in exactly the case it existed for.
  Below this floor the probe can only say it could not tell.
- `>=0.1.3` for the `OUTCOME` line. The collector used to tell a skipped walk
  from a real one by matching a printed **sentence**, because that was the only
  signal there was — it worked and rested on nothing. 0.1.3 publishes a declared
  line as contract, and `tests/test_seam.py` now checks this layer's accepted
  values against the referee's own declared set rather than restating them.

**Four gaps reported from here were fixed there**, and 0.1.2 is where they
landed: conditional requests, certificate pinning, a password off argv, and a
prefix that was *added* being declarable at all. Raising the floor is what turns
them from a note into a capability. See
[docs/upstream-asks.md](docs/upstream-asks.md).

**None of the four was a bug.** Each was a surface that was never there — which
a test suite cannot find, because it asks whether what exists is correct. Only a
second program with a real job discovers that what it needed was missing.

## Licence

Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
