Metadata-Version: 2.4
Name: pytest-perfguard
Version: 0.1.0
Summary: Automatic, low-overhead performance regression checks for pytest
Author: community-of-python
License-Expression: MIT
License-File: LICENSE
Classifier: Framework :: Pytest
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: pytest>=7
Requires-Python: >=3.10
Project-URL: Homepage, https://github.com/community-of-python/pytest-perfguard
Project-URL: Issues, https://github.com/community-of-python/pytest-perfguard/issues
Description-Content-Type: text/markdown

# pytest-perfguard

`pytest-perfguard` records CPU and memory metrics for ordinary pytest tests and
compares them with compatible historical results. It does not require benchmark
fixtures, decorators, or markers.

This repository currently contains a proof of concept. Its goals are:

- measure the existing test run instead of repeating tests;
- use Linux hardware instruction counters when permitted;
- fall back explicitly to process CPU time when counters are unavailable;
- collect process RSS with no sampling thread or binary dependency;
- keep history in GitLab job artifacts rather than an external service;
- catch both single-test jumps and suite-wide accumulation;
- retain a fixed anchor so gradual drift remains visible after the rolling
  baseline adapts;
- keep performance regressions non-blocking without masking functional failures.

## Install

Once the proof of concept is released, add the package to the same environment
as pytest:

```console
uv add --dev pytest-perfguard==0.1.0
```

The plugin is automatically discovered through pytest's `pytest11` entry point.
CI enables collection with:

```console
PERFGUARD=1 pytest
```

It writes one artifact per GitLab test job under
`.perfguard/current/<CI_JOB_NAME_SLUG>-<CI_JOB_ID>.json`, avoiding collisions
when GitLab downloads several jobs' artifacts. With pytest-xdist, workers send
their results to the controller and the controller writes one job-level file.

## Non-blocking GitLab design

Do not put `allow_failure: true` on the normal test job: that would also make
functional test failures non-blocking.

Instead, retain the normal blocking test job and add small report jobs. The
default-branch job keeps the rolling database, while merge-request output
expires:

```yaml
.perfguard-report:
  stage: report
  needs:
    - job: test
      artifacts: true
  script:
    - uvx --from "pytest-perfguard==${PERFGUARD_VERSION}" pytest-perfguard gitlab
  when: always
  allow_failure: true
  artifacts:
    when: always
    paths:
      - .perfguard/results.json
      - .perfguard/junit.xml
    reports:
      junit: .perfguard/junit.xml

perfguard:
  extends: .perfguard-report
  artifacts:
    expire_in: never
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: always

perfguard-mr:
  extends: .perfguard-report
  artifacts:
    expire_in: 30 days
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: always
```

The complete example is in [`examples/gitlab-ci.yml`](examples/gitlab-ci.yml).
Projects with several test jobs list each producer under `needs`.
On a merge request, the command downloads the latest successful target-branch
`perfguard` artifact using `CI_JOB_TOKEN`. On the default branch it records the
current samples into rolling history while preserving the fixed anchor. No
GitLab credentials enter the application test container.

The result job returns:

- `0` when no confirmed regression is found, including warm-up and persistent
  drift;
- `1` when a regression is found;
- `2` when artifact preparation or comparison fails.

Because only this reporting job has `allow_failure: true`, all three outcomes
remain visible without weakening the test gate.

The command prints the largest regressions and persistent drifts directly in the
job log and writes a JUnit report. Confirmed regressions are failures; drift is
a non-failing skipped testcase. Node IDs, metrics, baselines, and limits
therefore appear in GitLab without downloading JSON.

## Baseline policy

Results are compared only when architecture, CPU model, measurement backends,
xdist worker count, and runner key match. On GitLab the runner key defaults to
`CI_RUNNER_DESCRIPTION`, then `CI_RUNNER_ID`; `PERFGUARD_RUNNER_KEY` can group a
known-homogeneous pool or separate another CI provider's machines. This
prevents different shared-runner hosts from teaching one noisy baseline.
Python version, lock hash, and commit SHA are recorded as diagnostic context
rather than compatibility keys, so interpreter and dependency upgrades are
compared.

Each compatible series keeps two baselines with different purposes:

- a rolling history estimates current noise with median and MAD;
- a fixed anchor is created automatically after three successful default-branch
  samples and never moves during normal recording.

A measurement that exceeds the rolling limit is a confirmed regression. If it
exceeds only the fixed-anchor limit, it is reported as persistent drift without
a failing exit code. Requiring rolling corroboration prevents a short, unusually
fast anchor from making every later stable run flaky, while the fixed anchor
still prevents gradual degradation from disappearing silently as history moves.
To intentionally rebase after an understood change, run one successful
default-branch pipeline with either:

```console
pytest-perfguard gitlab --accept-baseline
```

or `PERFGUARD_ACCEPT_BASELINE=1`. Acceptance is rejected on merge-request and
other compare-only pipelines. Do not leave the variable enabled permanently.

The initial thresholds require both a relative increase and an absolute floor:

| Metric | Relative increase | Absolute floor |
| --- | ---: | ---: |
| CPU instructions (per test and summed job) | 15% | 1,000,000 instructions |
| Process CPU time (summed job and session) | 15% | 5 ms |
| RSS boundary growth (per test and summed positive growth) | 20% | 16 MiB |
| Process end and peak RSS | 20% | 16 MiB |

Three compatible historical samples are required before gating. Thresholds and
history length can be changed by CLI options.

Job aggregates are compared only when the set of passed test node IDs matches.
They catch regressions that are individually below the absolute floor, such as
1,000 tests each becoming 2 ms slower or several tests each retaining 8 MiB.
Individual baselines use the source of the test function, so editing a different
test in the same file does not reset them.

Per-test process CPU remains in the raw artifact, but becomes a regression
attribution only when the corresponding job aggregate also exceeds its rolling
limit. Shared runners can move unrelated process work, garbage collection, and
background activity between individual pytest calls; allowing an isolated call
spike to fail the report produced false positives in real xdist trials. Per-test
instruction counts and RSS growth remain independently guarded.

## Measurement scope and limitations

- Linux instruction counting uses `perf_event_open`. Rootless or restricted
  runners commonly deny it; the artifact then says `process_time`, never
  `instructions`.
- CPU time is process CPU consumed during each pytest call. Hardware
  instructions count only the thread running that call. The artifact records
  these scopes explicitly. Summed job and session CPU remain guarded so
  threaded work is not silently ignored; individual CPU-time outliers are
  reported only as attribution for a corroborated job regression.
- RSS uses `/proc/self/statm` on Linux, working set on Windows, and peak RSS as a
  fallback. With xdist, process RSS is summed across the controller and workers,
  and per-test CPU is summed across workers; the artifact records the process
  count.
- RSS growth is a regression signal, not proof of a memory leak. Allocation
  tracing and repeated workloads remain diagnostic follow-ups.
- The proof of concept measures the pytest process. Separate application
  services require a service- or cgroup-level collector.
- Shared runners with different CPU models build separate history buckets.
- The default-branch `.perfguard/results.json` is the baseline database and uses
  `expire_in: never`; deleting it restarts warm-up. Merge-request comparison
  artifacts expire after 30 days in the example.
