Metadata-Version: 2.5
Name: rivaldefs
Version: 0.1.0
Summary: Direction and implementation robustness of disagreement between rival binary labeling rules
Project-URL: Homepage, https://github.com/elliottower/rivaldefs
Author: Elliot Tower
License: MIT License
        
        Copyright (c) 2026 Elliot Tower
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: agreement,case definitions,loevinger h,multiverse,nestedness,reproducibility
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Description-Content-Type: text/markdown

# rivaldefs

**Which of two rival labeling rules sits inside the other, and is the gap between the rules
larger than the gap between their implementations?**

[![Tests](https://github.com/elliottower/rivaldefs/actions/workflows/test.yml/badge.svg)](https://github.com/elliottower/rivaldefs/actions/workflows/test.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

Two rules applied to the same units split them into four groups: positive under both, under
the first only, under the second only, under neither. Agreement statistics such as κ reduce
the four to one number and lose the direction of the disagreement. Two pairs of rules can share
κ while one pair is nested (one rule is a stricter cut of the other) and the other crosses
(each rule picks out units the other misses).

`rivaldefs` reports the direction as a quantity, with its uncertainty, and compares the
disagreement between two rules with the disagreement among admissible implementations of each
rule. It implements the indices of a frozen preregistration (criteria-direction, PREREG_v2,
OSF https://osf.io/bpw6e/).

## Terms

The package was written before the manuscript settled its vocabulary. The code that ran the
analyses is released unchanged, so the identifiers keep the earlier words.

| package | manuscript | meaning |
|---|---|---|
| rule (`Rule`) | case definition | a family of admissible implementations |
| variant | implementation | one way of coding a definition |
| labeler | an implementation's classifications | one 0/1 column of the units × labelers matrix |
| direction | orientation | which of two definitions has fewer positives: `a smaller`, `b smaller` or `equal` |
| % inside, `C` | containment | share of the smaller positive set that is also positive under the other definition |
| `H` | H | 1 − m/E0 = κ/κmax where defined |

## Version and scope

The analyses of the criteria-direction paper (simulation, MIMIC-IV, NHANES) ran on commit
`76cd25a684614de93e6ae96b5b58527782825d64`. The SHA-256 of `src/rivaldefs`, taken as the
analysis code takes it (sha256 over `name\0<sha256 of the file>\n` for each `*.py` file in
name order), is `76c690d8363f95f569f6c9349cfa2296e1d70cd1d9801aff0bfac9921809df7b`, the value
recorded in the result files. Version 0.1.0 has the same `src/`.

`rivaldefs` is the statistical core. The simulation generators and the MIMIC-IV and NHANES
classification pipelines are in the criteria-direction repository.

## Known limitations in 0.1.0

None of the three affects the recorded analyses; the two checks that establish this are in
criteria-direction,
`papers/paper1_bmc_mrm_rival_definitions/experiments/PREREG_v2_rival_definitions/results/rivaldefs_release_preflight_2026-10-07.json`.

- `qualifying_cells` reads `orientation_error=None` as no adverse evidence, and `OCRecord`
  does not require that `None` go with an undefined population orientation. A record with a
  defined orientation and no orientation error would not block its cell. Pass `None` only for
  pairs of equal population prevalence.
- `ladder` orders weighted prevalences by their floating values and does not apply
  `TIE_RTOL`. Two prevalences that tie under the pairwise rule but differ in the last bits can
  be ordered either way, which can change the on-ladder share.
- `gram` and `grid_from_gram` trust their input. `grid_from_gram` checks shape, symmetry and
  N, and does not check that the matrix is a feasible Gram matrix of binary columns; `gram`
  does not validate weights. `grid` validates before calling them.

## The problem, concretely

N = 1,000, two pairs of rules:

| | positives (x, y) | both | only first | only second | κ | H | % inside |
|---|---|---:|---:|---:|---:|---:|---:|
| pair 1 | 100, 300 | 95 | 5 | 205 | 0.38 | 0.93 | 95% |
| pair 2 | 200, 200 | 101 | 99 | 99 | 0.38 | 0.38 | 51% |

κ is 13/34 for the first pair and 61/160 for the second, 0.38 both. In the first pair five
units break the nesting where independence would put seventy; in the second each rule has 99
units the other misses.

## Install

```bash
git clone https://github.com/elliottower/rivaldefs && cd rivaldefs && pip install -e ".[test]"
```

Python 3.10+, NumPy.

## Use

```python
import numpy as np
import rivaldefs as rd

t = rd.PairTable(n11=95, n10=5, n01=205, n00=695)
t.orientation   # 'a smaller'
t.h             # 0.928...  = 1 - m/E0 = 13/14, Loevinger's H (kappa / kappa_max)
t.c             # 0.95      share of the smaller set inside the larger
t.kappa         # 0.382...

# Many labelers at once: a units x labelers 0/1 matrix.
rng = np.random.default_rng()
theta = rng.normal(size=2000)
labels = np.column_stack([
    theta > 1.0,                                          # rule A, implementation 0
    theta > 0.9,                                          # rule A, implementation 1
    theta + rng.normal(scale=0.8, size=2000) > 0.6,       # rule B, implementation 0
    theta + rng.normal(scale=0.8, size=2000) > 0.7,       # rule B, implementation 1
]).astype(int)

g = rd.grid(labels)          # K x K grids of H, C, kappa, discordance, NODF, orientation
g.h[0, 2]

a = rd.Rule("A", columns=(0, 1), levels=((0,), (1,)))
b = rd.Rule("B", columns=(2, 3), levels=((0,), (1,)))
cmp = rd.compare_rules(g, a, b)      # factor-balanced implementation distribution
cmp.delta                    # D_between - max(D_within), percentage points
cmp.s                        # probability a random implementation pair keeps the reference orientation

res = rd.bootstrap(labels, pairs=[(0, 2)], rule_pairs=[rd.RulePair(a, b)], n_boot=2000)
res.pairs[0].h                         # Interval(estimate, lo, hi, fraction_undefined, reported)
res.pairs[0].orientation_probabilities
res.rule_pairs[0].delta
```

## What it computes

- **Pairs.** The 2×2 table (counts or weights), orientation (the smaller set, or `equal`),
  the minority cell m and its independence value E0, H = 1 − m/E0, % inside C, κ, κmax,
  discordance d and the NODF pair score. Undefined values are NaN: H when a prevalence is 0 or 1,
  C when the smaller set is empty.
- **Many labelers.** K × K grids, Mokken's scale H, and the prevalence ladder (on-ladder share,
  its independence value, ladder H).
- **Rules and implementations.** A `Rule` is a family of labelers with factor levels. Its
  implementation distribution is factor-balanced by default (uniform on a full factorial),
  reference-weighted as an alternative, and the within-rule draws can be conditioned on being
  distinct. `compare_rules` returns D_between, D_within, Δ, R, S and the weighted quantiles of H
  over implementation pairs, computed exactly over every variant pair.
- **Uncertainty.** `bootstrap` resamples the counts of observed label patterns (the
  participant bootstrap, from pattern counts) and recomputes every labeler, orientation, H and
  rule comparison per replicate. `replicate_weights` does the same from survey replicate weights.
  A replicate where H or R is undefined is excluded; an interval is reported only when at least
  90% of replicates are defined, and the fraction undefined is always returned. The result
  objects apply the registered gates: `PairResult.h_delta_interval` (the delta-method interval)
  is None unless the modal orientation has bootstrap probability ≥ 0.95 and 0 < m < E0, and
  `RulePairResult.r` is withheld (NaN, `reported=False`) unless the lower percentile of R's
  denominator exceeds 0.5 percentage points. The ungated values sit in
  `h_delta_interval_ungated` and `r_ungated`, for diagnostics.
- **Ties and undefined terms.** Two positive counts are equal when they differ by at most
  `TIE_RTOL` × N (2^20 machine epsilons), so floating survey weights that sum to the same
  prevalence tie; integer counts never merge. Under the conditional-distinct-pair law a rule
  with one implementation has no within term, and Δ and R are then NaN whichever side it is on.
- **Orientation rule.** `qualifying_cells` reads a simulation's operating characteristics by
  E0 band × prevalence-ratio band; `orientation_status` marks a real pair resolved when its
  cells qualify and its modal orientation has bootstrap probability ≥ 0.95. A pair with E0 = 0
  (a prevalence of 0 or 1) is never resolved, and malformed records or probabilities are
  refused rather than read.
- **Missing inputs.** `delta_bounds` gives conservative outer bounds on Δ from each unit's
  possible completions.

## Tests

```bash
make test
```

The tests check exact values on hand-worked tables, H against an independent count of
Guttman errors, H = κ/κmax on random tables, the invariance of Δ, R and S to duplicated
implementations, the equivalence of the pattern-count and unit bootstrap (for H, Δ, R and S),
survey replicate intervals against a hand computation, the registered gates on both sides of
their thresholds, and that every completion of a small table with missing inputs falls inside
the bounds. Randomized tests draw a new seed each run and print it when they fail; replay one
with `RIVALDEFS_TEST_SEED=<seed> make test`.

## License

MIT
