Metadata-Version: 2.4
Name: pytest-inspect-evals
Version: 0.2.0
Summary: pytest plugin with shared test gates, fixtures and helpers for Inspect AI evaluations
Author: Matt Fisher
Author-email: Matt Fisher <m@ttfisher.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Framework :: Pytest
Requires-Dist: inspect-ai>=0.3.262
Requires-Dist: numpy>=1.26.0
Requires-Dist: pydantic>=2.10.0
Requires-Dist: pytest>=8.0
Requires-Dist: pytest-rerunfailures>=15
Requires-Dist: requests>=2.32.0
Requires-Dist: urllib3>=2.0
Requires-Dist: datasets>=4.8.5 ; extra == 'huggingface'
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/Generality-Labs/pytest-inspect-evals
Project-URL: Repository, https://github.com/Generality-Labs/pytest-inspect-evals
Project-URL: Issues, https://github.com/Generality-Labs/pytest-inspect-evals/issues
Provides-Extra: huggingface
Description-Content-Type: text/markdown

# pytest-inspect-evals

A pytest plugin with the test gates, fixtures and helpers shared by [Inspect AI](https://inspect.aisi.org.uk/) evaluation repos. It started as the shared test code in [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals). Installing it replaces a copied `conftest.py` and `tests/utils/` with one pinned dependency.

## Install

```bash
uv add --dev pytest-inspect-evals
# or, to use the Hugging Face dataset helpers:
uv add --dev "pytest-inspect-evals[huggingface]"
```

The plugin loads automatically. There is nothing to add to `conftest.py`.

## Test gates

Four markers mark tests that are skipped unless you switch them on.

| Marker             | CLI flag             | Env var                      | Pytest config option             | Default |
| ------------------ | -------------------- | ---------------------------- | -------------------------------- | ------- |
| `slow`             | `--runslow`          | `RUN_SLOW_TESTS`             | `inspect_evals_slow`             | off     |
| `dataset_download` | `--dataset-download` | `RUN_DATASET_DOWNLOAD_TESTS` | `inspect_evals_dataset_download` | off     |
| `k8s`              | `--runk8s`           | `RUN_K8S_TESTS`              | `inspect_evals_k8s`              | off     |
| `gpu`              | `--rungpu`           | `RUN_GPU_TESTS`              | `inspect_evals_gpu`              | off     |

The env var wins when it is set to a non-empty value: `1`, `true`, `yes` or `on` switch the gate on, anything else switches it off. Otherwise the CLI flag switches it on. Otherwise the config option decides, and the default is off. An empty env var counts as unset.

To run a gate by default in your repo, set its option in `pyproject.toml`:

```toml
[tool.pytest.ini_options]
inspect_evals_dataset_download = true
```

Three more markers have fixed behaviour:

- `huggingface`: skipped when `HF_TOKEN` is unset or blank. Otherwise the test is retried up to twice, 60 seconds apart, except for a gated-dataset error (`GatedRepoError`, or `DatasetNotFoundError` mentioning a gated dataset), which is not retried and is reported as a skip. See [Gated Hugging Face datasets and strict mode](#gated-hugging-face-datasets-and-strict-mode) to fail on these errors instead.
- `docker` and `posix_only`: skipped on Windows.

The plugin registers all seven markers, so they work under `--strict-markers` without any config.

## Gated Hugging Face datasets and strict mode

By default, a `huggingface` test is skipped when `HF_TOKEN` is unset, and a gated-dataset error is reported as a skip. This keeps local runs green for contributors who have not accepted a dataset's terms.

Strict mode is for CI jobs that are expected to have access. It is off by default and can be turned on in three ways:

- Environment variable: `REQUIRE_HF_ACCESS=1`
- Command-line flag: `pytest --require-hf-access`
- Ini option: `require_hf_access = true` under `[tool.pytest.ini_options]`

A non-empty `REQUIRE_HF_ACCESS` value wins over the flag, and the flag wins over the ini option. For example, `REQUIRE_HF_ACCESS=0` turns strict mode off even when the ini option is true.

In strict mode:

- A gated-dataset error fails the test instead of being skipped.
- Collecting `huggingface` tests with `HF_TOKEN` unset or blank stops the run with a usage error. Under pytest-xdist, each `huggingface` test fails at setup with the same message instead, because a usage error on a worker would crash the run.
- The terminal output ends with a "Hugging Face access denied" section that lists the dataset page to visit for each gated dataset, such as `https://huggingface.co/datasets/<repo_id>`. When the repo id is unknown, it lists the test's node id followed by "(dataset unknown)".

CI tooling can find these failures in the report. Each failed report has a `("hf_access_denied", "<repo_id>")` entry in its `user_properties`, and the value is an empty string when the repo id is unknown. `--report-log` writes it to the JSON lines file under `user_properties`.

Gated errors are never retried, in either mode. Other errors are still retried twice, 60 seconds apart.

The dataset viewer helpers in `pytest_inspect_evals.huggingface` check access when the viewer API returns a non-200 response. If the dataset is gated, they raise the Hub's `GatedRepoError`, which is handled as described above. If the dataset is missing or private, they raise `RepositoryNotFoundError`, which is an ordinary failure. For any other non-200 response, or if the access check itself fails, they raise an `AssertionError` as before.

## Adding your own gate

`skip_if_marker_present` is the function the plugin uses. Call it from your `conftest.py` for repo-specific gates:

```python
from pytest_inspect_evals.gates import skip_if_marker_present


def pytest_addoption(parser):
    parser.addoption("--custom-smoke", action="store_true", default=False)


def pytest_configure(config):
    config.addinivalue_line("markers", "custom_smoke: tests that call a live service")


def pytest_collection_modifyitems(config, items):
    skip_if_marker_present(
        config,
        items,
        marker="custom_smoke",
        cli_flag="--custom-smoke",
        env_var="RUN_CUSTOM_SMOKE_TESTS",
        reason="custom_smoke tests disabled (set RUN_CUSTOM_SMOKE_TESTS=1 or pass --custom-smoke)",
    )
```

## Fixtures

`set_model_roles` returns a context manager that sets model roles, for testing scorers that call a grader model:

```python
def test_judge_scorer(set_model_roles):
    with set_model_roles(grader=get_model("mockllm/model")):
        ...
```

`mock_docker_sandbox` stubs the Docker sandbox lifecycle, so an eval with `sandbox="docker"` runs without Docker. Combine it with mocked `sandbox().exec` results to test a scorer end to end:

```python
def test_e2e(mock_docker_sandbox):
    [log] = eval(my_task(), model="mockllm/model")
    assert log.status == "success"
```

## Helpers

- `pytest_inspect_evals.assertions`: `assert_task_structure`, `assert_eval_success`, `run_single_sample_eval`, `get_metric_value`.
- `pytest_inspect_evals.solvers`: `mock_solver_with_output`, `run_command`.
- `pytest_inspect_evals.sandbox`: `MockExecResult`, `create_sandbox_tool_task`, `assert_sandbox_test_passed`.
- `pytest_inspect_evals.metrics`: `run_metrics`, `assert_agreeing_epochs_change_nothing`, `MetricEvalError`. These run custom metrics through the real epoch reducer, the way an eval does.
- `pytest_inspect_evals.huggingface` (needs the `huggingface` extra): `assert_huggingface_dataset_structure`, `get_dataset_infos_dict`, `assert_huggingface_dataset_is_valid`, `assert_dataset_contains_subsets`, `assert_dataset_has_columns`. These check a dataset's schema through the Hugging Face dataset viewer API without downloading it.

## Migrating from a copied conftest

If your `conftest.py` came from inspect_evals or inspect-evals-template, remove what the plugin now provides:

- the `pytest_addoption` lines for `--runslow`, `--dataset-download`, `--runk8s` and `--rungpu`;
- the matching skip logic in `pytest_collection_modifyitems`;
- the seven markers above from your `markers =` config (keeping them is harmless).

If you keep the options, pytest stops at startup with `argparse.ArgumentError: argument --runslow: conflicting option string: --runslow`.

## Private inspect_ai APIs

Two fixtures use inspect_ai internals that have no public equivalent: `set_model_roles` calls `inspect_ai.model._model.init_model_roles`, and `mock_docker_sandbox` patches `inspect_ai.util._sandbox.docker.docker.DockerSandboxEnvironment`. A future inspect_ai release could break them.

## Development

```bash
uv sync
uv run pre-commit install   # optional: run the lint stack on every commit
uv run pytest
uv run basedpyright src
```

Each pull request adds a changelog fragment rather than editing `CHANGELOG.md`, so concurrent PRs don't conflict. Run `uv run scriv create`, uncomment the sections that apply in the new file under `changelog.d/`, and commit it with the change, or delete it if the change needs no entry. To release, run the **Prepare release** workflow from the Actions tab. It bumps `version` in `pyproject.toml`, collects the fragments into `CHANGELOG.md`, and opens a release pull request, whose CI starts once you click **Approve workflows to run** on it; merging it tags the release. It needs _Settings → Actions → General_ → **Allow GitHub Actions to create and approve pull requests**. By hand, the same is `uv version --bump minor` and `uv run scriv collect`.

Linting (ruff, [zizmor](https://docs.zizmor.sh/), mdformat) runs via [pre-commit](https://pre-commit.com); CI runs the same stack plus basedpyright and pytest via the shared [`python-ci`](https://github.com/Generality-Labs/python-project-template) reusable workflow.

## Releasing

See [RELEASING.md](RELEASING.md).
