Metadata-Version: 2.4
Name: pytest-flakeradar
Version: 0.1.0
Summary: Flaky-test radar for pytest: per-test flake rate across runs, with a FLAKY.md leaderboard your CI commits back
Project-URL: Homepage, https://github.com/sophie-nguyenthuthuy/flakeradar
Author-email: "Sophie Nguyen (Nguyễn Thu Thuỷ)" <sophie.nguyenthuthuy@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: ci,flaky,flaky-tests,pytest,test-reliability
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Requires-Dist: pytest>=7.0
Description-Content-Type: text/markdown

# flakeradar 🎯

**Flaky-test radar for pytest.** Tracks per-test flake rate across runs and writes a
`FLAKY.md` leaderboard your CI commits back — so flaky tests are named, ranked, and
impossible to ignore.

- Zero dependencies beyond pytest. Pure stdlib.
- History is a plain JSONL file (`.flakeradar/history.jsonl`) you commit to the repo —
  that's what makes "across runs" work on stateless CI runners.
- Scores *flakiness*, not failure: a test that always fails scores **0** (it's broken,
  not flaky). A test that alternates pass/fail scores near **1**.

## Install

```bash
pip install pytest-flakeradar
```

## Use

```bash
pytest --flakeradar
```

Every run appends one record to `.flakeradar/history.jsonl`, rescores the window, and
rewrites `FLAKY.md`:

| # | Test | Flake score | Recoveries | Flips | Fail / Pass | Last outcome |
|--:|------|------------:|-----------:|------:|:-----------:|:------------:|
| 1 | `tests/test_api.py::test_ws_reconnect` | 0.75 | 1 | 2 | 2 / 2 | passed |

**Flake score** = (intra-run recoveries + inter-run flips) / runs seen, over the last
`--flakeradar-window` runs (default 50).

- *Recovery*: failed then passed **within one run** (e.g. pytest-rerunfailures retries).
  The strongest flake evidence there is.
- *Flip*: final outcome changed between consecutive runs.

### Options

| Flag | Default | |
|------|---------|---|
| `--flakeradar` | off | enable recording + leaderboard |
| `--flakeradar-dir` | `.flakeradar` | history directory (relative to rootdir) |
| `--flakeradar-md` | `FLAKY.md` | leaderboard path; `""` disables |
| `--flakeradar-window` | `50` | runs to score over |
| `--flakeradar-keep` | `200` | max runs kept in history |
| `--flakeradar-top` | `20` | max leaderboard rows |
| `--flakeradar-fail-above SCORE` | off | exit non-zero if any score ≥ SCORE (CI gate) |

### CLI (no pytest run needed)

```bash
flakeradar report            # print the leaderboard
flakeradar md --out FLAKY.md # regenerate the markdown
```

## CI setup (the commit-back loop)

The runner is stateless, so persist history by committing it back. GitHub Actions:
see [examples/github-actions.yml](examples/github-actions.yml) — run pytest with
`--flakeradar`, then commit `.flakeradar/history.jsonl` + `FLAKY.md` with
`[skip ci]`. GitLab and Buildkite equivalents live in [examples/](examples/) too.

Prefer not to commit? Restore/save `.flakeradar/` with your CI cache instead — the
leaderboard still works, it's just not visible in the repo.

## CI providers

flakeradar stamps each run with provider metadata (run id/URL/branch/commit), detected
from environment variables. Built-in adapters: **GitHub Actions, GitLab CI, Buildkite,
CircleCI, Jenkins**, with a local fallback.

**Your provider missing? That's a ~20-line PR.** See
[CONTRIBUTING.md](CONTRIBUTING.md) — one small class with `detect(env)` and
`run_meta(env)`. Third-party packages can also register adapters via the
`flakeradar.adapters` entry-point group without touching this repo.

## License

MIT © Sophie Nguyen (Nguyễn Thu Thuỷ)
