Metadata-Version: 2.4
Name: mutmut-ratchet
Version: 0.4.0
Summary: Shared mutmut CI tooling: PR scoping, time-balanced sharding, and a per-file mutation-score ratchet.
Author: Andrew Berry
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/rtl-433-hass/mutmut-ratchet
Project-URL: Repository, https://github.com/rtl-433-hass/mutmut-ratchet
Project-URL: Issues, https://github.com/rtl-433-hass/mutmut-ratchet/issues
Keywords: mutmut,mutation-testing,ci,ratchet,sharding,pytest
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.14
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: mutmut>=3.6
Dynamic: license-file

# mutmut-ratchet

Shared CI tooling that makes a slow [mutmut](https://github.com/boxed/mutmut)
run usable as a **blocking** pull-request gate: scope the run to what a PR
touched, fan it across a time-balanced matrix, and fail only on a real per-file
mutation-score regression.

It is the de-duplicated form of the `scripts/mutation_*.py` helpers that were
copy-pasted between [`rtl-433-hass/rtl_433`](https://github.com/rtl-433-hass/rtl_433)
and [`rtl-433-hass/pyrtl_433`](https://github.com/rtl-433-hass/pyrtl_433). Only
the per-repo constants differed; those are now configuration.

## Why

A full-package mutmut run is slow (tens of minutes), and its score is not
perfectly reproducible. Naively gating on it either blocks every PR for an hour
or fails spuriously. The five subcommands fix that:

| Subcommand | What it does |
| --- | --- |
| `targets` | Turns `git diff --name-only` into the modules worth mutating, or escalates to a full run |
| `shards`  | Deterministic LPT partition of the package into N **time**-balanced shards |
| `stats`   | Reduces mutmut's on-disk `mutants/*.meta` into a comparable JSON payload |
| `ratchet` | Compares that payload to a committed baseline with a mutant-denominated tolerance band |
| `timings` | Records the measured per-file mutmut runtime that `shards` balances on |

Two design choices carry most of the value:

* **Shards balance by measured time, not mutant count.** Per-mutant test time
  varies several-fold across modules, so a count-balanced split still leaves a
  slow pole that the gate waits on. Modules are sorted heaviest first (ties by
  path) and dropped into the lightest bin (ties by lowest index) — plain LPT,
  within 4/3 of optimal makespan and fully reproducible, so every matrix job
  computes the identical assignment with no coordination.
* **The ratchet's tolerance band is denominated in mutants.** A flat percentage
  is wrong in both directions: 2% is ~13 mutants on a 630-mutant module but
  *zero* mutants on a 29-mutant one. The band is
  `max(tolerance_fraction × total, tolerance_mutants)` mutants, converted back to
  score space. That absorbs both run-to-run noise and the scoped-vs-full gap (a
  scoped run is a slight lower bound: some mutants are only killed by tests in
  other files) while still failing on a genuine regression, which typically
  costs far more mutants than the band. The baseline only ever ratchets upward.

## Install

```bash
uv add --dev mutmut-ratchet
```

`mutmut` itself is a dependency (the tooling reads mutmut's module walk and its
per-file meta through `mutmut.__main__`), so installing this pulls it in. Pin
the exact mutmut version you test against in your own dev group if you care
which one CI uses.

## Configure

All settings live in your `pyproject.toml` under `[tool.mutmut_ratchet]`. Only
`package_path` is required; unknown keys and wrong types are hard errors, so a
typo can never silently weaken the gate.

```toml
[tool.mutmut_ratchet]
package_path = "my_package"
escalate_paths = ["pyproject.toml", "uv.lock", "tests/conftest.py"]

[tool.mutmut_ratchet.explicit_test_sources]
"tests/test_urls.py" = ["_urls.py"]
"tests/test_mut_client_floor.py" = ["client.py"]
```

| Key | Type | Default | Meaning |
| --- | --- | --- | --- |
| `package_path` | string | *(required)* | Repo-relative path of the package under mutation, e.g. `pyrtl_433` or `custom_components/rtl_433`. |
| `package_dotted` | string | `package_path` with `/` → `.` | Dotted name mutmut uses in mutant filters. Set it only when the two differ (e.g. a `src/` layout). |
| `baseline` | string | `"scripts/mutation_baseline.json"` | Committed per-file score baseline. Relative paths anchor to the `pyproject.toml`'s directory. |
| `timings` | string | `"scripts/mutation_timings.json"` | Committed per-file mutmut runtime profile, used as the shard bin-pack weight. |
| `escalate_paths` | list of strings | `["pyproject.toml", "tests/conftest.py"]` | Changed paths that force a **full** run because they can move results package-wide. Exact repo-relative matches. |
| `explicit_test_sources` | table of string → list of strings | `{}` | Test file → the source modules (relative to `package_path`) it exercises, for tests the `test[_mut]_<name>.py` → `<name>.py` convention cannot resolve. Genuinely broad tests are deliberately *left out* so they still escalate. |
| `tolerance_fraction` | float | `0.02` | Fractional half of the band; scales the cushion on large files. |
| `tolerance_mutants` | int | `3` | Absolute half of the band, in mutants; protects small files where a percentage rounds to nothing. |
| `precision` | int | `6` | Decimals scores are rounded to before comparison. |
| `floor` | float | `0.70` | Advisory floor recorded into a freshly written baseline payload. |
| `fallback_seconds_per_mutant` | float | `1.0` | Weight used when neither a timing nor any timing profile exists at all (a fresh checkout). |

A committed baseline carries the `tolerance_fraction` / `tolerance_mutants` it
was written with, and those win over the config defaults — so retuning the band
is a baseline edit, not a workflow edit. An explicit `--tolerance-*` flag beats
both.

Precedence throughout is **CLI flag → `pyproject.toml` → built-in default**.

## Commands

Every subcommand also accepts `--config PYPROJECT` (default: the nearest
`pyproject.toml` at or above the cwd), `--package-path`, and `--package-dotted`.

```
mutmut-ratchet targets [--base REF] [PATH ...]
```
Maps changed paths to targets. With no arguments, reads whitespace-separated
paths from stdin. Prints four lines: `all` or `scoped`; the mutmut filter
patterns; the source paths; and the functions the scope was narrowed *to*
(empty when nothing was). Exit 0.

`--base REF` narrows the scope to the **changed functions** rather than whole
modules, by diffing against `REF` and mapping each hunk to the function that
contains it. A one-line edit to a 750-line module then mutates one function
instead of all of it. Three rules keep that sound:

* A changed line belonging to no mutable function — module-level code, an
  import, a class body, a nested class's method — escalates that file to its
  whole module, because such a change can affect anything in the file.
* A source pulled in because one of its *tests* changed is never narrowed: a
  weakened test can free a mutant anywhere in the module it exercises.
* No line information at all (no `--base`, or git cannot answer) falls back to
  whole modules, never to an empty filter.

Line 4 feeds `shards --restrict-functions`. A module that stays whole
contributes nothing to it — the shard falls back to that module's own pattern
when no name mentions it — so this never has to enumerate a file's functions.

The diff is only taken once the scope is known to be narrowable, and only over
the sources in it, so an escalated or docs-only PR never pays for it.

Function-scoped runs need `--mode functions` to gate; see below.

Patterns are derived the same way `shards` derives them, so a package
`__init__.py` in scope emits the `x_*` / `xǁ*` trampoline patterns its
mutants are actually named after — mutmut drops the `__init__` segment from
mutant names, so a `<pkg>.sub.__init__.*` filter would match none of them and
silently run zero mutants.

```
mutmut-ratchet shards --shard N --of M [--restrict PATH ...] [--restrict-functions NAME ...] [--baseline P] [--timings P]
```
Prints two lines: this shard's filter patterns and its source paths (both blank
for an empty shard). `--restrict` intersects the shard with a scoped set without
changing the global assignment, so a scoped PR fans across the same shards.
`--restrict-functions` (line 4 of `targets --base`) narrows the patterns
further, to those functions; a module in the shard that none of them name keeps
its whole-module pattern. Exit 0, or 2 for out-of-range `--shard`/`--of`.

```
mutmut-ratchet stats [--paths PATH ...]
```
Prints the stats JSON on stdout, with a `files` block and a `functions` block
(`{path: {function: tally}}`). `--paths` is **required after a filtered `mutmut
run`**: mutants outside the filter stay "not checked", which would otherwise
read as 0%. Exit 0.

Every tally also reports `not_checked` — the subset of `survived` that mutmut
recorded but never executed. That is what a *function*-scoped run needs: no
list of what the run covered has to be threaded in from outside, because a
function whose mutants are entirely not-checked was plainly not measured, and
`--mode functions` skips it. It is also how a partial file tally is kept out of
the baseline.

```
mutmut-ratchet ratchet --mode floor|strict|functions --stats FILE [--baseline P] [--update] [--tolerance-fraction F] [--tolerance-mutants N] [--tolerance-survivors N]
```
`floor` is the CI gate (improvements never fail); `strict` also fails on upward
drift, as a local check that the committed baseline is still representative.
`--update` writes current scores back, keeping the higher of baseline/current
per file, and refuses while regressions stand. Exit 0 pass, 1 regression/drift,
2 missing stats or baseline.

`functions` is the gate for a **function-scoped** run. A per-file score is a
partial measurement when only some of a file's functions were mutated — the
rest come back "not checked", counted as survived — so it cannot be compared
against a whole-file baseline. Each mutated function's score can. Two rules:

* A function already in the baseline fails only when it gains survivors beyond
  `--tolerance-survivors` (default 0). Counting survivors rather than comparing
  scores is what makes this *no new survivors*: a function that grows by ten
  well-tested lines keeps its count, and touching a function that already had
  survivors does not block the PR.
* A function not in the baseline must reach `floor`, since there is nothing to
  compare against and new untested code is the case the gate exists for.

The default tolerance is 0 — *no new survivors* means none. The per-file band
exists because a file's score is compared across runs and across the
scoped/full divide, where a mutant or two genuinely moves; this gate counts
survivors in the handful of functions a PR changed, where that does not apply.

The one mechanism that looks like noise is worth being precise about: mutmut
counts a timeout as a kill, but its budget is `(estimated_test_time +
timeout_constant) × timeout_multiplier` — 15× plus a second, by default. A
mutant must run an order of magnitude longer than the tests it runs under to
trip that, which is an unbounded loop rather than a busy runner, and an
unbounded loop trips it every time. What *could* still flip is a genuinely
nondeterministic test, and a band would only hide that in the functions a PR is
actively changing. Raise it per repository if a suite really does flap, and
treat needing to as the finding it is.

Rolling this onto an existing baseline is a bootstrap: with `--update`, a
baseline that has no `functions` block records one without gating, and every
run after that gates against it.

```
mutmut-ratchet timings [--out PATH]
```
Writes the runtime profile after a **full** `mutmut run`. Exit 0, or 2 when no
timing data exists yet.

## Worked GitHub Actions example

Scoped on PRs, full on push/schedule, sharded six ways either way, with one
aggregated status check.

```yaml
name: Mutation

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: "0 3 * * *"

permissions: {}

jobs:
  mutation:
    name: "Mutation floor — shard ${{ matrix.shard }}"
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        shard: [0, 1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v5
        with:
          # Need history back to the base branch to diff a PR's changed files.
          fetch-depth: 0
      - uses: astral-sh/setup-uv@v6
        with:
          enable-cache: true
      - run: uv sync --locked --dev

      - name: Determine scope for this trigger
        id: scope
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            git fetch --no-tags origin "${{ github.base_ref }}"
            changed=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD")
            uv run mutmut-ratchet targets $changed > targets.txt
            cat targets.txt
            {
              echo "mode=$(sed -n '1p' targets.txt)"
              echo "paths=$(sed -n '3p' targets.txt)"
            } >> "$GITHUB_OUTPUT"
          else
            echo "mode=all" >> "$GITHUB_OUTPUT"
            echo "paths=" >> "$GITHUB_OUTPUT"
          fi

      - name: Determine this shard's modules
        id: shard
        run: |
          if [ "${{ steps.scope.outputs.mode }}" = "all" ]; then
            uv run mutmut-ratchet shards --shard ${{ matrix.shard }} --of 6 > shard.txt
          elif [ -n "${{ steps.scope.outputs.paths }}" ]; then
            uv run mutmut-ratchet shards --shard ${{ matrix.shard }} --of 6 --restrict ${{ steps.scope.outputs.paths }} > shard.txt
          else
            # Scoped run with nothing in scope (e.g. a docs-only PR): no work.
            printf '\n\n' > shard.txt
          fi
          {
            echo "patterns=$(sed -n '1p' shard.txt)"
            echo "paths=$(sed -n '2p' shard.txt)"
          } >> "$GITHUB_OUTPUT"
          cat shard.txt

      - name: Run mutation testing (this shard's modules)
        if: steps.shard.outputs.patterns != ''
        run: |
          uv run mutmut run ${{ steps.shard.outputs.patterns }}
          uv run mutmut-ratchet stats --paths ${{ steps.shard.outputs.paths }} > mutation-stats.json

      - name: Enforce per-file floor vs baseline
        if: steps.shard.outputs.patterns != ''
        run: uv run mutmut-ratchet ratchet --mode floor --stats mutation-stats.json

      - name: No modules in this shard
        if: steps.shard.outputs.patterns == ''
        run: echo "No modules in scope for this shard; nothing to check."

  mutation-gate:
    name: "Mutation floor"
    needs: [mutation]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Aggregate shard results
        run: |
          if [ "${{ needs.mutation.result }}" != "success" ]; then
            echo "One or more mutation shards failed."; exit 1
          fi
          echo "All mutation shards passed."
```

A simpler single-job form, for a package small enough not to need a matrix:

```yaml
      - name: Mutation test (per-module floor ratchet)
        run: |
          uv run mutmut run --max-children 4
          uv run mutmut-ratchet stats > mutation-stats.json
          uv run mutmut-ratchet ratchet --mode floor --stats mutation-stats.json
```

Never cache the `mutants/` working tree: a stale cache can report a higher score
than reality and let a regression through. Caching uv's download cache is fine.

## Bootstrapping a baseline

```bash
uv run mutmut run                                   # a FULL run
uv run mutmut-ratchet stats > mutation-stats.json
uv run mutmut-ratchet ratchet --mode floor --stats mutation-stats.json --update
uv run mutmut-ratchet timings                       # writes the shard weights
```

Commit both JSON files. Refresh the timings profile periodically; a stale one
degrades to a slightly suboptimal — never incorrect — split.

## Licence

Apache-2.0. See `LICENSE` and `NOTICE`.
