Metadata-Version: 2.5
Name: robostats
Version: 0.1.0
Summary: Statistics and uncertainty quantification for robot policy evaluation
Project-URL: Homepage, https://github.com/jamesgyc/robostats
Project-URL: Repository, https://github.com/jamesgyc/robostats
Project-URL: Issues, https://github.com/jamesgyc/robostats/issues
Author-email: James Gong <jamesgongyecheng2024@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: confidence-intervals,evaluation,mcnemar,robotics,statistics,uncertainty-quantification
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.11
Description-Content-Type: text/markdown

# robostats

Statistics and uncertainty quantification for robot policy evaluation.

Robot policy results are usually reported as a single success rate. `robostats`
takes the episode-level records behind that number and returns what the number
actually supports: an interval, a comparison against another policy that respects
the shared evaluation scenarios, and a report that states what each run declared
about how it was collected.

**Status: 0.1.0.** Intervals, paired, unpaired and combined comparison,
ingestion, recording and reporting all work and are validated by enumeration.
Power analysis and comparison of more than two policies do not exist yet. See
the roadmap.

---

## Why this exists

Four things motivated it. Three are measured and reproducible from this
repository.

**1. A widely used library reports maximal significance for two identical
policies.**

`statsmodels.stats.contingency_tables.mcnemar` with the continuity correction, on
a table with zero discordant pairs, computes `(|0 - 0| - 1)^2 / 0`. It returns
`statistic=inf, pvalue=0.0`. Two policies that agreed on every single scenario
are reported as maximally different. Without the correction the same table
returns `nan`. `robostats` returns `p = 1.0`.

Details and a minimal reproduction: [`results/statsmodels-mcnemar-m0.md`](results/statsmodels-mcnemar-m0.md).

**2. The default interval method misses its nominal coverage where robot
policies actually operate.**

The Wilson score interval is the common recommendation for binomial proportions,
and it is a good one across most of the parameter space. But policy success rates
cluster near the ceiling, and that is where it degrades. Measured by exact
enumeration at n = 50 and 95% nominal:

| true success rate | Wilson | Clopper-Pearson |
|---|---|---|
| 0.92 | 0.9407 | 0.9679 |
| 0.98 | 0.9216 | 0.9822 |
| 0.99 | 0.9106 | 0.9862 |

Across the whole region `p` in [0.85, 0.99], Wilson's coverage falls to 0.8951.
A nominal 95% interval delivering 90% is not a rounding issue.

**3. The guarantee is cheap.**

Exactness costs width, and the usual objection to Clopper-Pearson is that it is
too conservative to be useful. In this operating region that objection does not
hold up. Expected interval width at n = 50, 95%:

| method | mean width | mean coverage | min coverage |
|---|---|---|---|
| Wilson | 0.1488 | 0.9549 | 0.8951 |
| Clopper-Pearson | 0.1602 | 0.9773 | 0.9509 |
| Agresti-Coull | 0.1593 | 0.9684 | 0.9469 |

Clopper-Pearson buys a guarantee that never drops below nominal for about 7.7%
extra width. Agresti-Coull costs nearly the same width without the guarantee.

**4. Nothing tells you how the two runs were configured.**

Evaluation protocol details, the execution horizon in particular, move reported
success rates substantially. Two runs at different horizons can differ for
reasons that have nothing to do with the policies, and most published numbers do
not say which horizon produced them. Nothing in the usual tooling asks.

`robostats` does not refuse those comparisons. Refusing would be the wrong call:
an execution horizon is an action-chunk deployment choice, not a property of the
measurement apparatus, and different policies have different natural chunk sizes,
so demanding a shared horizon handicaps whichever policy it suits less. Most real
harness output records none of these fields anyway, so a package that insists on
them refuses most real data on first use.

What it does instead is record what each side declared and put it in the report,
every time. Two runs at horizon 8 and horizon 16 fingerprint differently, and the
report says so without ruling on it:

```
Protocol:      pi_zero  protocol 0ce610a4e6e5
               octo     protocol a43826bc58d4
```

and where a run recorded nothing:

```
Protocol:      pi_zero  (not recorded)
               octo     (not recorded)
```

The line is never omitted, and the blank is the point: it is the case worth
seeing, and an empty field argues for recording it better than an exception does.
`ComparisonResult.protocol_mismatch` says whether the two sides declared the same
thing, and both fingerprint sets are on the result.

**This makes an incomparable comparison visible. It does not prevent one.**
Whether two runs are comparable is a judgement about the experiment, and this
package does not make it for you. The one thing it does insist on is scenario
identity: `pair()` raises without a join key, because that is arithmetic rather
than protocol. Without matched scenarios there is no paired comparison to
compute.

---

## Install

Not on PyPI yet. Install from source:

```
git clone https://github.com/jamesgyc/robostats
cd robostats
uv sync
```

or into an existing environment, from a checkout:

```
pip install .
```

Requires Python 3.11 or later. Runtime dependencies are numpy and scipy, and
nothing else.

## Use

Every output below is pasted from a real run of the code above it.

```python
from robostats import Protocol, compare, load_jsonl, pair, report

protocol = Protocol(execution_horizon=8, reset_mode="hard", max_steps=520)
fields = dict(
    policy_id_field="policy",
    task_id_field="task",
    success_field="success",
    scenario_fields=("suite", "task", "init_state_id"),
)

a = load_jsonl("runs/pi_zero.jsonl", protocol=protocol, **fields)
b = load_jsonl("runs/octo.jsonl", protocol=protocol, **fields)

print(report(compare(pair(a, b))))
```

```
Paired comparison: pi_zero vs octo

Estimand:      delta = p_A - p_B, the difference in true success rate between pi_zero (A) and octo (B) on the same scenarios.
Estimate:      delta = 0.1000  95% CI [-0.1241, 0.3264]  (tango)
Test:          p = 0.6250  (McNemar, exact)
Pairs:         20 matched, 4 discordant
Dropped:       0 from pi_zero, 0 from octo
Joined on:     suite / task / init_state_id
Protocol:      pi_zero  protocol 0ce610a4e6e5
               octo     protocol 0ce610a4e6e5
```

The protocol block is always present, one line per side, including when both
sides declared the same thing and including when neither declared anything. A
line that only appears when something is wrong trains readers to stop looking for
it. `Joined on:` is there for the same reason: it says which columns composed the
join key, so a missing configuration field is visible rather than silently folded
into a key that matches anyway.

### When the two runs do not cover the same scenarios

Real runs overlap partially. `align()` keeps every scenario either policy was
evaluated on, `overlap()` says how much they share before any statistic is
computed, and `mode="all"` runs every comparison that applies rather than picking
one for you.

```python
from robostats import align, compare, overlap, report

alignment = align(a, b)
print(report(overlap(alignment)))
print(report(compare(alignment, mode="all")))
```

```
Overlap of 2 policies

Scenarios:     40 total, 20 observed by all 2
Coverage:      1 policy: 20, 2 policies: 20
Overlap:       pi_zero/octo 20
Comparable:    all 2 policies connected (weakest link 20)
Absences:      pi_zero  unknown=8
               octo     unknown=12

Sensitivity: pi_zero vs octo

Mode      n          delta   95% CI             p
paired    20 shared  0.1000  [-0.1241, 0.3264]  0.6250
unpaired  32 / 28    0.1518  [-0.0956, 0.3850]  0.2316
combined  20 + 12/8  0.1259  [-0.0515, 0.3232]  0.1581

The combined row uses the shared scenarios and the singly-observed ones together, assuming which scenarios each policy ran is unrelated to how they would have gone. Its n column reads shared + only-A/only-B.
The paired row uses the 20 scenarios both policies were evaluated on. The unpaired row uses everything each observed, 32 and 28, and discards the pairing.
The two modes agree: neither rejects at 0.05.
The combined row does not reject at 0.05. It is kept out of the agreement above because it assumes something the other two do not.
The paired row could not have rejected at 0.05 on this table. Its exact test conditions on the 4 discordant scenarios, so the smallest p-value available to it is 0.1250, however they fall.
The combined row's 0.1581 comes from an asymptotic statistic, which has no such floor. The gap between them is that difference and not a disagreement about the data.
```

Three modes, three different n, three different answers, and the report says why
they differ rather than leaving a reader to guess which line to quote. The last
two lines are the kind of thing that is invisible in a single-mode output: with
four discordant scenarios the exact test cannot reach 0.05 whichever way they
fall, so its large p-value is a statement about how little it had to condition on.

`mode="combined"` is never selected for you. It assumes which scenarios each
policy ran is unrelated to how they would have gone, and no file the package
reads records whether that holds, so asking for it is a decision the caller makes
in writing. **Its coverage is weak over a handful of shared scenarios** — see the
`combined_difference` docstring and `results/combined-coverage/`.

`mode="auto"` chooses between paired and unpaired from the observation mask
alone, never from the outcomes, and says what it chose:

```
Mode:          paired (auto: 20 shared scenarios, at or above the threshold of 20)
```

Conditioning on the mask is what makes selecting legitimate: the mask is
ancillary to the outcomes, so the error rate of whichever test follows is
unaffected. A rule that peeked at the successes would make the reported p-value
conditional on a choice the data drove. **The threshold of 20 is a placeholder
and is not calibrated** — the constant's own comment and `CHANGELOG.md` say so.

### The loader guesses nothing

Column names are taken literally. There is no fuzzy matching, no fallback when a
named column is missing, and no inference of the evaluation protocol from file
contents. A loader that decides `succ` and `is_success` both mean `success` is
making a silent choice about your data, which is the behavior this package exists
to surface rather than commit.

### Scenario identity, not episode order

Pairing joins on `scenario_id`, composed from columns you name. It never joins on
episode index or row position. Episode indices wrap when a run requests more
episodes than there are distinct initial states, so joining on them produces
mismatched pairs silently, with no error and a plausible-looking result.

---

## What it does

- **Intervals**: Wilson, Clopper-Pearson, Agresti-Coull, with exact endpoints at
  the boundaries.
- **Paired comparison**: McNemar's test, defaulting to the exact conditional
  binomial rather than the chi-square approximation, because discordant counts in
  this setting are routinely small.
- **Unpaired comparison**: the score test with the Miettinen-Nurminen interval,
  for runs whose scenarios do not match.
- **Combined comparison**: a signed likelihood-ratio statistic over the
  partially-overlapping design, using the shared scenarios and the singly
  observed ones together. Never selected automatically.
- **Sensitivity table**: `mode="all"` runs every applicable comparison together
  and names any that does not apply and why.
- **Overlap diagnostic**: `overlap()` reports what two or more runs share,
  reading only which scenarios were observed and never the outcomes.
- **Recording**: `EpisodeRecorder` writes episode records during an evaluation
  that would otherwise discard them. It writes; it never runs anything.
- **Effect size, always**: no function returns a p-value without a point estimate
  and an interval for it. Tango's score interval is used for the paired
  difference.
- **Protocol disclosure**: every comparison records the protocol fingerprints
  each side carried and reports them per side, including blanks. Nothing about
  the protocol blocks a comparison; scenario identity is the only requirement,
  and it is enforced at the join.
- **Reporting**: deterministic plain text, with p-values below 1e-4 rendered as
  `< 0.0001` rather than as `0.0000`.

## What it does not do

It does not run rollouts, load policies, wrap simulators, or orchestrate
evaluation. It reads episode records and returns statistics. Reading another
tool's output is in scope; producing it is not.

---

## Validation

Every method's coverage is measured by exact enumeration, not simulation. For
fixed `n` and `p` the outcome space is `x = 0..n`, so coverage is a finite sum
with no sampling error, no seed, and no confidence band around the estimate
itself. Rerunning produces byte-identical artifacts, which means a diff in
`results/coverage/` is always a change in the code.

```
uv run python validation/coverage.py
uv run python validation/coherence.py
uv run python validation/combined_coverage.py
uv run python validation/partial_overlap.py
```

Curves and a hash manifest are committed under `results/coverage/`,
`results/coherence/`, `results/combined-coverage/` and
`results/partial-overlap/`. The last of those is partly Monte Carlo and says so
per table; everything else is enumerated. Clopper-
Pearson is asserted to hold at or above nominal at every grid point, which is a
theorem rather than a tolerance. Wilson and Agresti-Coull are approximate and dip
below nominal by construction, so their curves are pinned as regressions instead.

This study is not decorative. It found a defect in the Tango endpoint handling at
the parameter boundary that the unit test suite could not see.

## Design principles

- **Never judge whether a number is trustworthy.** Compute it, report it, stop.
- **No silent method selection.** Nothing switches between exact and approximate
  based on the data. The caller chooses, or gets the conservative default.
- **Require only what changes the answer.** A field that changes the meaning of
  the p-value or the interval is required; everything else is recorded when
  supplied, displayed always, and never demanded. Scenario identity is required.
  The protocol is disclosed.
- **Test against definitions, not against other implementations.** Agreement with
  another library only shows that two things agree. The interval endpoints are
  checked as roots of their defining equations; the exact McNemar p-value is
  checked in integer arithmetic.

## Roadmap

- Power and minimum detectable effect
- Comparison of more than two policies, and the set that cannot be separated
- A calibrated threshold for `mode="auto"`, which this release does not have
- A command line interface

## License

Apache-2.0.
