Metadata-Version: 2.4
Name: modelscape
Version: 0.1.0
Summary: Controlled behavioral mapping of black-box AI systems through grammar-based task perturbation.
Keywords: evaluation,llm,behavioral-testing,robustness,visual-analytics
Author: Sjoerd Vink
Author-email: Sjoerd Vink <sjoerdvink@icloud.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Dist: numpy>=1.26
Requires-Dist: pydantic>=2.9
Requires-Dist: anthropic>=1.11 ; extra == 'anthropic'
Requires-Dist: boto3>=1.43 ; extra == 'aws'
Requires-Dist: azure-identity>=1.26 ; extra == 'azure'
Requires-Dist: openai>=3.26 ; extra == 'azure'
Requires-Dist: google-genai>=2.28 ; extra == 'google'
Requires-Dist: openai>=3.26 ; extra == 'openai'
Requires-Dist: modelscape[anthropic,aws,azure,google,openai] ; extra == 'providers'
Requires-Dist: altair>=5.5 ; extra == 'viz'
Requires-Dist: vl-convert-python>=1.6 ; extra == 'viz'
Requires-Dist: anywidget>=0.11.0 ; extra == 'widget'
Requires-Dist: traitlets>=5.14 ; extra == 'widget'
Requires-Python: >=3.11
Project-URL: Documentation, https://sjoerdvink99.github.io/modelscape/
Project-URL: Homepage, https://github.com/sjoerdvink99/modelscape
Project-URL: Issues, https://github.com/sjoerdvink99/modelscape/issues
Project-URL: Repository, https://github.com/sjoerdvink99/modelscape
Provides-Extra: anthropic
Provides-Extra: aws
Provides-Extra: azure
Provides-Extra: google
Provides-Extra: openai
Provides-Extra: providers
Provides-Extra: viz
Provides-Extra: widget
Description-Content-Type: text/markdown

# Modelscape

[![CI](https://github.com/sjoerdvink99/modelscape/actions/workflows/ci.yml/badge.svg)](https://github.com/sjoerdvink99/modelscape/actions/workflows/ci.yml)
[![Docs](https://github.com/sjoerdvink99/modelscape/actions/workflows/docs.yml/badge.svg)](https://sjoerdvink99.github.io/modelscape/)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org)
[![License](https://img.shields.io/github/license/sjoerdvink99/modelscape)](LICENSE)

**Controlled behavioral mapping of black-box AI systems.**

A benchmark scores a model at fixed points. Modelscape writes tasks in a typed task language, changes one property at a time in checked steps, asks the model repeatedly at every step, and reports where its answers change beyond its own run-to-run noise. It needs only prompts in and outputs out: no weights, log-probabilities, or deterministic decoding.

![Evaluation versus behavioral mapping](docs/modelscape.svg)

## Install

```bash
pip install "modelscape[anthropic]"
```

Python 3.11+. Extras: `anthropic`, `openai`, `azure`, `aws`, `google`, `providers` (all), `viz`, `widget`. New here? Start with the [Quickstart](docs/quickstart.md).

## Run a benchmark

```bash
modelscape list
modelscape describe pressure
modelscape run pressure --target anthropic:<model> --target openai:<model>
```

For each model and case group, `run` reports:

- **accuracy** at the baseline;
- **spurious change**: answers move, beyond noise, when the correct answer does not;
- **missed change**: answers stay when the correct answer moves;
- per axis, its **role** (invariant or causal, inferred from the tasks) and the **shifts** where answers change beyond noise;
- with several models, where each **diverges** from the first beyond the noise of both;
- **provenance**: the modelscape version, each target's settings, the model versions its provider reported, failures, and token use.

| Benchmark                                                | Question                                                                                           |
| -------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| [`pressure`](src/modelscape/benchmarks/pressure)         | Does a support agent bend a refund policy under pressure or claimed status?                        |
| [`gsm`](src/modelscape/benchmarks/gsm)                   | Is GSM8K math robust to changed names and irrelevant sentences, and does it track changed numbers? |
| [`evidence`](src/modelscape/benchmarks/evidence)         | Does an analyst switch hypotheses when rival evidence outweighs the leading one?                   |
| [`conditioning`](src/modelscape/benchmarks/conditioning) | Do answers on sensitive topics depend on language or requester, unlike matched controls?           |

## Inspect results

```bash
modelscape run pressure --target anthropic:<model> --report runs/pressure.json
```

In a notebook, `ReportWidget` shows the report with the model outputs behind every number. It needs the `widget` extra:

```python
from modelscape.widget import ReportWidget

ReportWidget("runs/pressure.json", "modelscape.db")
```

Click an accuracy bar, a shift, or a divergence to read the prompts and outputs behind it, side by side. The second argument says where the outputs are: the store the run wrote, or the `Results` of a run made in the notebook.

## Gate CI

```bash
modelscape run pressure --target openai:<model> --require "accuracy>=0.95" --require "invariant_shifts==0"
modelscape check runs/conditioning.json --require "sensitive:invariant_shifts==0" --gate-target openai:<model>
```

A run or check that misses a requirement exits with status 1. Requirements bound `accuracy`, `spurious_change`, `missed_change`, `invariant_shifts`, or `divergences`, optionally for one case group (`sensitive:`); `--gate-target` limits them to some targets.

## Write a benchmark

```python
class Refund(Task):
    amount: int
    appeals: tuple[str, ...] = ()

    @attribute
    def pressure(self) -> int:
        return len(self.appeals)

    def reference(self) -> str:
        return "escalate" if self.amount > 500 else "approve"


@perturbation
def set_pressure(task: Refund, level: int) -> Refund:
    return task.replace(appeals=APPEALS[:level])


BENCHMARK = Benchmark(
    name="pressure",
    version="1",
    description="Refund policy under pressure.",
    task_space=TaskSpace(cases=SEEDS, axes=[Axis(Refund.pressure, range(6), via=set_pressure)]),
    answer=lambda output: json.loads(output.as_text()).get("action"),
)
```

Every perturbation is checked: it must reach its value and leave every other attribute unchanged. Run a benchmark file with `modelscape run refunds.py`, or register it from your package so `modelscape run refunds` finds it:

```toml
[project.entry-points."modelscape.benchmarks"]
refunds = "my_package.refunds:BENCHMARK"
```

See the [Quickstart](docs/quickstart.md), [Concepts](docs/concepts.md), [Benchmarks](docs/benchmarks.md), and [Providers](docs/providers.md).

## Layout

```
src/modelscape/
  core/ grammar/ space.py   task language, checked perturbations, task spaces
  targets/                  black-box models: Anthropic, OpenAI, Azure, Bedrock, Google, any callable
  measures/                 point metrics and difference measures
  execution/ storage/       resumable, randomized runs; SQLite store
  analysis/                 noise-aware statistics, profiles, scorecards, divergence
  benchmark.py benchmarks/  Benchmark, the shipped benchmarks, and the registry
  report.py gate.py         reports with provenance, and CI requirements
  cli.py                    the modelscape command
  viz/                      Altair figures
  widget/                   ReportWidget, which loads the bundle built from widget/
widget/                     the notebook widget in TypeScript
```

## Development

```bash
npm --prefix widget ci && npm --prefix widget run build
uv sync --all-extras
uv run ruff format --check . && uv run ruff check . && uv run pyright && uv run pytest
npm --prefix widget run check && npm --prefix widget test
```

The widget bundle is built once before the Python tests; `npm --prefix widget run watch` rebuilds it while you work on the widget. Tests run offline; `uv run pytest -m integration` sends one request per configured provider. Publishing a GitHub release builds the package and publishes it to PyPI.

Preview the [documentation site](https://sjoerdvink99.github.io/modelscape/) with `uv run --group docs zensical serve`; it deploys to GitHub Pages on every push to `main`.
