Metadata-Version: 2.4
Name: leakfence
Version: 0.5.0
Summary: Framework-agnostic spatial (subject/session) and temporal (overlapping-window) leakage auditing for windowed signal ML pipelines -- EEG and beyond.
Project-URL: Homepage, https://github.com/ptapal/leakfence
Project-URL: Repository, https://github.com/ptapal/leakfence
Project-URL: Issues, https://github.com/ptapal/leakfence/issues
Project-URL: Changelog, https://github.com/ptapal/leakfence/blob/master/CHANGELOG.md
Author: Polina Tapal
License-Expression: MIT
License-File: LICENSE
Keywords: bci,biosignal,cross-validation,data-leakage,eeg,machine-learning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: numpy>=1.20
Provides-Extra: plot
Requires-Dist: matplotlib>=3.5; extra == 'plot'
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == 'test'
Description-Content-Type: text/markdown

# leakfence

Audits a train/test split for subject/session and overlapping-window leakage. Plain arrays, no framework.

## Why

Overlapping sliding windows plus a naive shuffle-split gives inflated, irreproducible accuracy. Two flavors:

- **Temporal**: adjacent windows from one recording land on both sides. Not generalizing, recognizing a near-duplicate.
- **Spatial**: same subject (or subject+session) in both partitions. The model learns *who*, not the task.

MNE/Braindecode/MOABB/EEGDash give correct splitting primitives if you build on their objects. None of them audit a split you already have.

## Install

```bash
pip install leakfence
```

numpy is the only runtime dependency. Plots need `pip install leakfence[plot]`.

## Use

```python
from leakfence import audit_split

report = audit_split(
    train_idx=train_idx,
    test_idx=test_idx,
    subject=subject_ids,
    session=session_ids,          # needed for paradigm="cross-session"
    sample_range=sample_ranges,   # (start, end) per window, optional
    recording=recording_ids,      # scopes the temporal check
    paradigm="cross-subject",     # within-session | cross-session | cross-subject | None
)

print(report.summary())
report.raise_if_failed()          # or strict=True on audit_split()
```

`paradigm` matters: the same subject in both partitions is a bug for cross-subject, correct for cross-session. Pick wrong, get false positives.

One check runs unconditionally, without `subject` or `sample_range`: the same row index in both partitions, wrong under every paradigm. See `report.index_overlap_count`. Sampling with replacement makes shared indices intentional, so `allow_index_reuse=True` skips it.

Indices are normalized before anything is compared. Negative indices resolve against the dataset length, so `-1` and `n-1` are one row. Boolean masks expand to positions. Out-of-range, float, or wrong-length arguments raise. Length comes from `subject`/`sample_range`/`recording`; pass `n_rows` for negative indices with none of those.

Severity, not just pass/fail: `report.subject_overlap_count`, `report.temporal_overlap_rate`, `report.max_window_overlap`.

Is that severity surprising, or what chance gives you anyway?

```python
from leakfence import permutation_test_group_overlap, permutation_test_temporal_overlap

permutation_test_group_overlap(train_idx, test_idx, subject_ids, n_permutations=999)
# PermutationResult(observed=10.0, null_mean=8.9, null_std=1.0, p_value=1.0, n_permutations=999)
```

Null: random train/test relabeling of the same items, same sizes. `p_value` is P(null <= observed), one-sided. A plain shuffle-split scores near 1, as leaky as chance because it is chance. A split-construction test, not a classifier-accuracy test (cf. Ojala & Garriga 2010).

**Mind the direction.** Small `p_value` here is *good news*, the opposite of "p < 0.05, therefore an effect is present". Writing "p = 0.03, confirming leakage" would be backwards. To avoid reporting the raw number:

```python
result.beats_chance()          # True: better structured than a random split
result.percentile              # 97.0: beats 97% of random splits
```

`.to_dict()` carries `p_value_meaning` and `beats_chance_at_0.05` for the same reason.

The statistic follows `paradigm`. Default `"cross-subject"` tests `subject_overlap_count`, low is good. `"cross-session"` tests `session_pairs` instead, since subject overlap there is expected and would score a correct split like a leaky one. `"within-session"` raises: no group-overlap invariant applies, use `permutation_test_temporal_overlap`.

Same data appearing twice anywhere, not just across train/test:

```python
from leakfence import check_duplicates

groups, violations = check_duplicates(X, train_idx=train_idx, test_idx=test_idx)
# groups: {fingerprint: (row indices)} for every exact content match
```

SHA256 per row, the same primitive as `guard_fit`'s `test_fingerprints`. A group straddling train/test is `severity="error"`, an exact leak. A duplicate inside one partition is `severity="warning"`: worth knowing about, not a split problem.

`report.clean_test_idx()` returns `test_idx` with contaminated windows dropped.

`from leakfence.plot import plot_temporal_leakage` (needs `leakfence[plot]`) draws per-window severity against time, one row per recording. Flat is clean, a spike shows where and how bad. Returns the axes; `show=True` also calls `plt.show()`.

Not every overlap is equal:

```python
from leakfence import pair_variance_ratio

pair_variance_ratio(stats.pairs, sample_range, recording, signal)
```

Overlap-region variance over recording baseline, per pair. Above 1 means an artifact or outlier, worse than a leak on plain background. Not a stats test.

Sample ranges are half-open, `[start, end)`, so `(0, 100)` and `(100, 200)` do not overlap. If your epoching reports `end` as the last sample *in* the window, add 1 before passing ranges in.

See [`examples/quickstart.py`](https://github.com/ptapal/leakfence/blob/master/examples/quickstart.py).

## Global-preprocessing leaks

Fitting a scaler/PCA/ICA on the whole dataset before splitting. Wrap the estimator, not the array:

```python
from leakfence import guard_fit

scaler, violations = guard_fit(StandardScaler(), n_train=len(train_idx))
scaler.fit(X_full)             # 200 rows, 150 are train -> flagged
scaler.fit(X_full[train_idx])  # clean
```

A row-count check cannot catch the right *number* of rows drawn from the wrong ones: an off-by-one slice, a resample that swapped a row in and out. Fingerprint content instead:

```python
from leakfence import fingerprint_rows, guard_fit

test_fps = fingerprint_rows(X_full[test_idx])
scaler, violations = guard_fit(StandardScaler(), test_fingerprints=test_fps)
scaler.fit(X_candidate)  # flagged if any row matches held-out content, whatever the row count
```

The two checks carry different confidence. `n_train` is a heuristic (`severity="warning"`, a resampled train set could trip it innocently); `test_fingerprints` is a definitive content match (`severity="error"`). `strict=True` raises only on the error case. Works on anything with `.fit(X, ...)`.

Fingerprints normalize integer width, `-0.0`/`0.0` and NaN payloads. They do not normalize float precision: `float32(0.1)` and `float64(0.1)` are different numbers, so fingerprint after any cast your pipeline performs. Object and string dtypes raise instead of hashing, since `ndarray(dtype=object).tobytes()` serializes pointers and equal content would hash differently.

`guard_fit` patches the instance, which makes it unpicklable on purpose: silently dropping a check during a `joblib` round-trip would be worse. Give it a lifetime:

```python
from leakfence import guarded, unguard

with guarded(scaler, n_train=len(train_idx)) as (scaler, violations):
    scaler.fit(X_train)
# restored on exit: picklable, cloneable, joblib-safe

scaler, violations = guard_fit(scaler, n_train=len(train_idx))
unguard(scaler)   # or undo it by hand
```

This catches your own mistakes, not evasion: `type(est).fit(est, X)`, cloning and re-instantiation all bypass the patch.

## Auditing a pipeline you didn't write

A collaborator's code, a paper's supplementary repo, something AutoML-generated, where the nested `Pipeline`/`ColumnTransformer` structure isn't already in your head. An unfitted pipeline has no rows and no train/test labels, so it cannot detect a leak, but it can say what is stateful and where, with no data:

```python
from leakfence import lint_pipeline

lint_pipeline(pipe)
# [{'path': 'prep.num', 'stateful': True}, {'path': 'prep.pass', 'stateful': False}, {'path': 'clf', 'stateful': True}]
```

An inventory, not a detector. It cannot know whether `pipe` is the whole object passed to `cross_val_score`; that is a fact about the call site.

## How much data do you actually have

Not a leak, not a model judgment. Window overlap inflates the nominal count, 75% overlap by about 4x:

```python
from leakfence import independent_capacity

independent_capacity(sample_range, recording)
# {'sub-01': {'nominal_windows': 77, 'independent_capacity': 20, 'inflation_factor': 3.85,
#             'window_length': 384, 'window_length_varies': False}, ...}
```

`independent_capacity` counts how many windows would tile the recording without sharing a sample. It is an upper bound, not an estimate: EEG independence is set by autocorrelation time, and two adjacent non-overlapping windows of resting-state alpha are still correlated. Read it as "at most this many".

Real example: COG-BCI resting-state, 5 subjects, 75% overlap, 385 nominal windows, at most 100 independent. Same 385 [`validation/`](https://github.com/ptapal/leakfence/blob/master/validation/test_all_datasets.py) reports.

Enough data for *what*, though. Covariance-based (Riemannian/SPD) pipelines estimate a channel by channel covariance: `n(n+1)/2` free parameters, quadratic in channel count.

```python
from leakfence import dimensionality_report

dimensionality_report(n_channels=14, sample_range=sample_range, recording=recording)
# {0: {'p': 105, 'window_length': 40, 'independent_capacity': 5, 'nominal_windows': 20,
#      'per_trial':     {'n_samples': 40, 'mp_ratio': 0.35, 'rank_deficient': False, ...},
#      'across_trials': {'n_samples': 5,  'mp_ratio': 2.8,  'rank_deficient': True, 'param_ratio': 21.0, ...}}}
```

Two questions live here, and conflating them is how you end up calling a full-rank matrix singular:

- **`per_trial`**: can one window's channel covariance be full rank at all? Channels vs. time samples inside the window. Marchenko-Pastur: `mp_ratio = C/N`, rank-deficient exactly when `C > N`. 14 channels from a 40-sample window is full rank.
- **`across_trials`**: enough independent windows for `p = C(C+1)/2` parameters? `param_ratio = p/N`, a parameter-counting rule of thumb. `param_ratio >= 1` (`ill_conditioned`) means shrink or regularize. Not a claim about rank.

`across_trials` uses `independent_capacity`, not nominal windows: overlap adds no independent covariance information either.

## Paste-into-appendix report

Everything above, concatenated. No new numbers, just formatting:

```python
from leakfence import experiment_report

print(experiment_report(report, sizes=sizes, dims=dims, dataset_name="toy resting-state", paradigm="cross-subject"))
```

```
Dataset: toy resting-state
Paradigm: cross-subject

Leakage audit:
  subject overlap: 0
  temporal overlap rate: 0.0%
  max window overlap: 0.0%
  verdict: no leakage detected

Effective sample size (upper bound: tiling, not autocorrelation):
  sub-01: 77 nominal, at most 20 independent (3.85x inflation)
  sub-02: 77 nominal, at most 20 independent (3.85x inflation)

Covariance estimability:
  sub-01: per-trial covariance full rank (14 ch / 384 samples, Marchenko-Pastur C/N=0.04)
       across trials under-determined (p=105 params / 20 independent windows, p/N=5.25)
  sub-02: per-trial covariance full rank (14 ch / 384 samples, Marchenko-Pastur C/N=0.04)
       across trials under-determined (p=105 params / 20 independent windows, p/N=5.25)
```

`sizes` and `dims` are optional; omit either and that section is skipped. These numbers come from running [`examples/quickstart.py`](https://github.com/ptapal/leakfence/blob/master/examples/quickstart.py)'s toy dataset, not from invention.

Same inputs, JSON instead of text, for logging runs or plotting the null distribution yourself:

```python
from leakfence import experiment_report_dict

experiment_report_dict(report, dataset_name="toy resting-state", paradigm="cross-subject")
# {"dataset_name": "toy resting-state", "paradigm": "cross-subject",
#  "audit": {"ok": True, "subject_overlap_count": 0, ..., "group_stats": {...}, "temporal_stats": None},
#  "sizes": None, "dims": None}
```

`report.to_dict()` works the same way, capping the temporal pair list at 1000 entries; pass `max_pairs=None` for all of it. `PermutationResult.to_dict()` includes the full `null_distribution`. `GroupOverlapStats`, `TemporalOverlapStats` and `Violation` all have `to_dict()`. JSON object keys are always strings, so an int recording id round-trips as `"0"`.

## Building a split instead of auditing one

`audit_split` checks a split you already have. `split_windowed` constructs one that is leak-free by construction, Roth (2026)-style (`arXiv:2603.10742`), adapted for EEG since Roth's `split_group`/`split_temporal` don't know windows exist:

```python
from leakfence import split_windowed, assess_once

part = split_windowed(subject, session=session_ids, paradigm="cross-session", test_frac=0.2)
part.train_idx, part.test_idx     # whole (subject, session) groups, never split
assess_once(part)                 # raises LeakageError on a second assessment
```

`cross-subject` and `cross-session` assign whole groups to one side, which rules out temporal overlap for free. `within-session` cuts each recording chronologically and drops any train window straddling the boundary. `assess_once` is process-scoped: it resets on restart, the same limitation as Roth's version.

`reset_assessments()` forgets every partition assessed so far, so re-running a notebook cell is not indistinguishable from reusing a test set. Calling it in a script defeats the point.

Covers the three MOABB paradigms only, not general-purpose splitting.

## Not doing (yet)

- General-purpose split construction. For anything outside the three paradigms, use Braindecode/MOABB.
- No CLI.

## Adjacent tools

- **timefence**: temporal leak in tabular feature/label joins. No windows.
- **DataSAIL**: leakage-reduced splits for biomolecular similarity data. Not time-series.
- **EEGDash / Braindecode**: correct splitters, but only on their own data objects.

## Citation

`split_windowed` and `assess_once` adapt the split-then-assess-once discipline from:

Roth, S. (2026). A Grammar of Machine Learning Workflows: Rejecting Data Leakage at Call Time. *arXiv:2603.10742*.

## License

MIT
