Metadata-Version: 2.4
Name: vcti-benchmark
Version: 2.0.0
Summary: Software performance measured on the vcti-measure model — calibrated timing, traced allocation, and the convention for running a benchmark
Author: Visual Collaboration Technologies Inc.
License-Expression: LicenseRef-Proprietary
Project-URL: Repository, https://github.com/vcollab/vcti-python-benchmark
Project-URL: Changelog, https://github.com/vcollab/vcti-python-benchmark/blob/main/CHANGELOG.md
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: <3.15,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: vcti-measure>=1.1
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: numpy; extra == "test"
Provides-Extra: lint
Requires-Dist: ruff; extra == "lint"
Provides-Extra: typecheck
Requires-Dist: mypy; extra == "typecheck"
Dynamic: license-file

# vcti-benchmark

Benchmarking for the vcti Python packages: the quantities worth measuring
about running code, the instruments that take them, and the convention for
running one from a script.

The model is [`vcti-measure`](https://github.com/vcollab/vcti-python-measure)'s
— measurements, instruments, observations, studies and one JSON artifact per
run. This package is what makes that model about **software performance**. It
adds no concepts.

## Installation

```bash
pip install vcti-benchmark
```

### In `pyproject.toml` dependencies

```toml
[project.optional-dependencies]
bench = ["vcti-benchmark>=2.0.0"]
```

---

## Quick Start

A benchmark is a study whose items are callables. `benchmark()` declares one:

```python
from vcti.benchmark import ALLOCATION, UNITS, WALL_CLOCK_TIME, benchmark, run_script
from vcti.measure.core import Axis, Manifest

MANIFEST = Manifest(
    "vcti.nputils",                       # your namespace, so results never collide
    [
        benchmark(
            "build",
            summary="Building an array of n elements.",
            subjects={"growable": lambda values, n: build_growable(values)},
            references={"list": lambda values, n: list(values)},
            axes=[Axis(name="n", values=(1_000, 100_000), labels=("1K", "100K"))],
            setup=lambda n: range(n),
            measurements=[WALL_CLOCK_TIME, ALLOCATION],
            follow=["wall-clock-time.median", "allocation.net"],
        )
    ],
    units=list(UNITS),
)

if __name__ == "__main__":
    run_script(MANIFEST, package="vcti-nputils")
```

That is the whole of `benchmarks/bench_build.py`. Running it prints each
observation as it lands and writes one JSON artifact into
`benchmarks/results/`, named for the study, the commit and the artifact's own
identifier — so repeated runs of the same commit accumulate rather than
overwrite, and a name that somehow already exists is reported rather than
replaced:

```
  build/growable n=1000 [wall-clock-time]  median=32.4 µs  stddev=8.54 µs  ops=27.9 K/s
  build/growable n=1000 [allocation]       peak=31.2 KiB  net=31.1 KiB
  ...
  8 measured
  wrote benchmarks/results/build-af7822d-5d748a3e91c4402fb0d7f1c88e2a6b35.json
```

Rendering it is a separate step, in a separate package — so last month's run
can be re-rendered without measuring anything again.

---

## What you get

| | |
|---|---|
| `WALL_CLOCK_TIME` | Calibrated per-call time: min, max, mean, median, stddev, iqr, rounds, outliers, ops |
| `ELAPSED_TIME` | One call, measured once — for work that does not survive repetition |
| `ALLOCATION` | Transient `peak` and `net` bytes, from one act of measuring |
| `RESIDENT_SET` | Declared; nothing here provides it |
| `CalibratedTimer`, `SingleShotTimer`, `TracedAllocation` | The instruments, standard library only |
| `UNITS` | ns, bytes, count, ratio, per-second, with their display ladders |
| `benchmark()` | Declares a study whose items are callables |
| `run_script()` | Describe the run, run it, write the artifact, say where |
| `conditions()`, `source()`, `environment()` | Which build, on which machine |

---

## A few things worth knowing

- **`setup` cannot see which implementation is about to run.** It takes the
  axis values and nothing else, so every implementation is handed equivalent
  input by construction rather than by care.
- **Each measurement is taken separately**, with its own preparation. Traced
  allocation inside a timing loop would distort the timings and report a peak
  accumulated across thousands of calls.
- **An attempt that produced nothing still appears**, carrying a status and a
  reason. Asking for `RESIDENT_SET` gives you observations marked
  *unavailable* saying so, not silence.
- **`net` is not `retained`.** It is allocated minus freed, which is not a
  reachability analysis — an operation returning a view over existing memory
  allocates nothing.
- **The timer records how it ran** — whether collection was suspended, how
  many warmup rounds it discarded, and how many calls each round performed —
  as factors beside the numbers. A median averaged over one call and one
  averaged over four thousand are not equally trustworthy.
- **Which instrument answered is provenance, never identity**, so replacing
  one does not orphan the series it fed.

---

## Dependencies

`vcti-measure`. The instruments themselves need only the standard library.

---

## Documentation

| If you want to… | Read |
|---|---|
| Understand the methodology and what is still unsettled | [docs/design.md](docs/design.md) |
| Understand the measurement model itself | `vcti-measure`'s design document |
