Metadata-Version: 2.4
Name: sleep-kit-psg
Version: 2.1.0
Summary: Config-driven, auditable preprocessing for sleep-staging PSG datasets
Author-email: Li Jinyang <jinyang03702@163.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/lijinyang439-arch/PSGPrep
Project-URL: Repository, https://github.com/lijinyang439-arch/PSGPrep
Project-URL: Documentation, https://github.com/lijinyang439-arch/PSGPrep#readme
Project-URL: Issues, https://github.com/lijinyang439-arch/PSGPrep/issues
Project-URL: PyPI, https://pypi.org/project/sleep-kit-psg/
Keywords: polysomnography,sleep staging,EDF,PSG,preprocessing,reproducibility
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: numpy>=1.23
Requires-Dist: scipy>=1.9
Requires-Dist: mne>=1.5
Requires-Dist: PyYAML>=6.0
Provides-Extra: hdf5
Requires-Dist: h5py>=3.8; extra == "hdf5"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: check-wheel-contents>=0.6; extra == "dev"
Requires-Dist: h5py>=3.8; extra == "dev"
Requires-Dist: mypy>=1.11; extra == "dev"
Requires-Dist: ruff>=0.9; extra == "dev"
Requires-Dist: twine>=6.0; extra == "dev"
Requires-Dist: types-PyYAML>=6.0; extra == "dev"
Dynamic: license-file

<p align="center">
  <img src="https://raw.githubusercontent.com/lijinyang439-arch/PSGPrep/main/assets/sleepkit-psg-social-preview.png" width="100%" alt="SleepKit PSG: auditable PSG preprocessing for sleep-staging research">
</p>

<h1 align="center">SleepKit PSG</h1>

<p align="center"><strong>Prepare heterogeneous polysomnography data with an explicit, reproducible contract.</strong></p>

<p align="center">
  <a href="https://pypi.org/project/sleep-kit-psg/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sleep-kit-psg.svg"></a>
  <a href="https://github.com/lijinyang439-arch/PSGPrep/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/lijinyang439-arch/PSGPrep/actions/workflows/ci.yml/badge.svg"></a>
  <img alt="Python 3.10+" src="https://img.shields.io/badge/Python-3.10%2B-3776AB">
  <a href="https://github.com/lijinyang439-arch/PSGPrep/blob/main/LICENSE"><img alt="Apache-2.0" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
</p>

<p align="center">
  <a href="https://github.com/lijinyang439-arch/PSGPrep/blob/main/README.zh-CN.md">简体中文</a> ·
  <a href="https://github.com/lijinyang439-arch/PSGPrep/blob/main/MIGRATION.md">Migration</a> ·
  <a href="https://github.com/lijinyang439-arch/PSGPrep/blob/main/docs/dataset_profiles.md">Dataset profiles</a> ·
  <a href="https://github.com/lijinyang439-arch/PSGPrep/blob/main/docs/output_format.md">Output contract</a>
</p>

SleepKit PSG converts PSG recordings and sleep-stage annotations into consistent NumPy arrays for research and model development. Dataset assumptions live in validated YAML profiles; discovery, pairing, channel derivation, filtering, resampling, alignment, quality control, and provenance use the same Python API and CLI.

This is the maintained PyPI distribution of the PSGPrep engineering rewrite. Version 2.1 unifies the historical `sleep-kit-psg` package and PSGPrep 2.0 implementation under the canonical `sleep_kit` Python namespace.

> [!IMPORTANT]
> SleepKit PSG is a research preprocessing tool, not a medical device or automated diagnostic system. It contains no PSG recordings, participant data, credentials, or dataset licenses.

## Core capabilities

- 22 packaged dataset profiles with explicit validation status.
- Deterministic regular-expression pairing and subject grouping; unmatched files remain visible.
- Auditable channel matching, reference subtraction, units, canonical sleep stages, and QC.
- Configurable filtering, resampling, epoch alignment, normalization, and sequence generation.
- Atomic NPZ/NPY writes, resumable records, per-record failure isolation, and validation.
- Processing-contract hashes that prevent incompatible runs from sharing an output directory.
- Optional HMAC pseudonymization of record and subject IDs.
- Typed Python API and matching CLI with stable exit codes and structured JSON reports.

## Installation

Python 3.10 or newer is required. The base install includes NumPy, SciPy, MNE, and PyYAML so EDF workflows work immediately:

```bash
python -m pip install sleep-kit-psg
```

Install HDF5 support for DOD and PhysioNet 2018 profiles when needed:

```bash
python -m pip install 'sleep-kit-psg[hdf5]'
```

For development from this repository:

```bash
git clone https://github.com/lijinyang439-arch/PSGPrep.git
cd PSGPrep
python -m pip install -e '.[dev]'
```

## Minimal runnable example

The repository includes a deterministic synthetic PSG record. This exercises the same public API, filtering, epoching, output, and validation path without downloading controlled data:

```bash
python examples/synthetic_quickstart.py --output ./demo-output
```

The command exits `0` only when preprocessing and artifact validation both pass. Remove or choose another output directory before rerunning because unrelated or differently configured output is never overwritten silently.

## Typical CLI workflow

First inspect the packaged contract and scan pairing without reading signal samples:

```bash
sleepkit-psg profiles
sleepkit-psg show-profile shhs1
sleepkit-psg scan \
  --profile shhs1 \
  --input-root /path/to/shhs1/edfs \
  --annotation-root /path/to/shhs1/annotations \
  --details
```

Then process selected canonical channels and validate the result:

```bash
sleepkit-psg preprocess \
  --profile shhs1 \
  --input-root /path/to/shhs1/edfs \
  --annotation-root /path/to/shhs1/annotations \
  --output-root ./outputs/shhs1 \
  --channels C4 E1 \
  --target-sfreq 100 \
  --workers 4

sleepkit-psg validate --output-root ./outputs/shhs1
```

Use `--require-complete-pairing` when any unmatched source must fail the run. Use `--fail-fast` only with one worker. `--log-file` appends a log outside the output directory; progress always goes to standard error and the JSON result goes to standard output.

Exit codes are consistent across commands:

| Code | Meaning |
|---:|---|
| `0` | Command and all selected records/artifacts succeeded |
| `1` | Scan was incomplete when required, or a record/artifact failed |
| `2` | Invalid arguments, profile, paths, configuration, or I/O setup |

## Python API

The high-level API accepts both `str` and `pathlib.Path`:

```python
from pathlib import Path

from sleep_kit import preprocess, scan, validate_output

pairing = scan(
    "shhs1",
    input_root=Path("/path/to/shhs1/edfs"),
    annotation_root=Path("/path/to/shhs1/annotations"),
)
print(f"paired={len(pairing.records)} unmatched={len(pairing.unmatched_signals)}")

summary = preprocess(
    "shhs1",
    input_root=Path("/path/to/shhs1/edfs"),
    annotation_root=Path("/path/to/shhs1/annotations"),
    output_root=Path("outputs/shhs1"),
    channels=("C4", "E1"),
    target_sfreq=100,
    workers=4,
)

validation = validate_output("outputs/shhs1")
if not summary.succeeded or not validation.valid:
    raise RuntimeError((summary.as_dict(), validation.as_dict()))
```

The main public surface is:

- `list_profiles() -> tuple[str, ...]`
- `load_profile(name_or_path) -> DatasetProfile`
- `scan(profile, input_root, annotation_root=None) -> DiscoveryReport`
- `preprocess(profile, input_root, output_root, **options) -> RunSummary`
- `validate_output(output_root) -> ValidationReport`

`run_pipeline(..., options=RunOptions(...))` remains available for advanced and PSGPrep 2.0-compatible usage.

## Inputs and supported scope

Signal readers currently cover EDF/BDF/EDF-compatible REC through MNE, explicit NPZ contracts, MATLAB arrays used by PhysioNet 2018, and DOD HDF5. Annotation readers cover NSRR XML, EDF annotations, epoch text/tables, event tables, HDF5 hypnograms, and PhysioNet 2018 files.

Packaged profiles include ABC, CCSHS, CFS, DCSM, DOD, HMC, HomePAP, ISRUC, MASS (`mass13`), MESA, MNC, MrOS visits 1/2, NCHSDB, PhysioNet 2018, SHHS visits 1/2, Sleep-EDF cassette/telemetry, SOF, STAGES, and WSC. A profile marked `migrated-unverified` is a starting contract, not a claim that every dataset release or record has been validated. See [dataset profiles](https://github.com/lijinyang439-arch/PSGPrep/blob/main/docs/dataset_profiles.md) and the bounded [validation report](https://github.com/lijinyang439-arch/PSGPrep/blob/main/VALIDATION_REPORT.md).

Canonical labels are `W=0`, `N1=1`, `N2=2`, `N3/N4=3`, `REM=4`, and `UNKNOWN=5`. The default epoch is 30 seconds, target sampling rate is profile-defined (normally 100 Hz), and normalization defaults to `none`. Scientific overrides are recorded in provenance.

## Output and overwrite behavior

The recommended NPZ layout is:

```text
output-root/
├── records/<record_id>.npz
├── sequences/<record_id>.npz
├── completion/<record_id>.json
├── manifest.jsonl
├── errors.jsonl
├── provenance.json
└── summary.json
```

Record `x` is `float32 [epoch, channel, sample]`; `y` is `int8 [epoch]`. Sequence arrays are `float32 [sequence, epoch, channel, sample]` and `int8 [sequence, epoch]`. See the complete [output contract](https://github.com/lijinyang439-arch/PSGPrep/blob/main/docs/output_format.md).

- A completed record is resumed by default and reported as `skipped_existing`.
- `overwrite=True` or `--overwrite` reprocesses only records under the same contract.
- A changed profile, version, channel order, sampling rate, normalization, sequence length, format, or ID salt requires a new output directory.
- A non-empty directory without SleepKit provenance is rejected and left untouched.
- Every record failure is written to `errors.jsonl`; no exception is silently discarded.

## Privacy and reproducibility

Source paths in manifests are relative. For private datasets, enable `hash_ids` with a high-entropy secret salt:

```bash
sleepkit-psg preprocess \
  --profile /path/to/private-profile.yaml \
  --input-root /path/to/private-signals \
  --output-root ./outputs/private \
  --hash-ids \
  --id-salt-file /secure/path/id-salt.bin
```

The salt is never copied; only its SHA-256 fingerprint prevents resuming with a different key. Pseudonymization is not anonymization. Split model data by `subject_id`, not by epoch, sequence, or visit.

## FAQ and limitations

**Why were zero records processed?** `scan --details` reports whether globs found files, regular expressions extracted IDs, and annotations paired. Point roots at the levels expected by the profile instead of relying on fuzzy filename matching.

**Why did a record fail?** Inspect `errors.jsonl`, then its profile's channel candidates, raw label map, sampling rate, and alignment policy. Failures are intentionally explicit.

**Can I add a dataset?** Copy [examples/custom_dataset.yaml](https://github.com/lijinyang439-arch/PSGPrep/blob/main/examples/custom_dataset.yaml), use relative globs and named `record_id` groups, then add synthetic parser/pairing tests before claiming real-data validation.

**Does SleepKit score sleep automatically?** No. It prepares recordings and existing sleep-stage annotations for downstream analysis or models.

**Are all 22 profiles fully validated?** No. Validation status is per profile and bounded. Dataset releases, headers, channel semantics, scoring standards, and unusual records still require study-specific review.

## Migration from 1.x and PSGPrep 2.0

Version 2.1 is a breaking product upgrade from `sleep-kit-psg` 1.x. The canonical import is still `sleep_kit`, but output defaults, pairing, exceptions, and configuration are stricter. `fast_preprocess()` and `sleepkit-process` remain as deprecated compatibility wrappers. The `psgprep` import and command from PSGPrep 2.0 also remain temporarily available. See [MIGRATION.md](https://github.com/lijinyang439-arch/PSGPrep/blob/main/MIGRATION.md) for exact mappings and behavioral changes.

## Development and contribution

```bash
python -m pip install -e '.[dev]'
ruff format --no-cache --check .
ruff check --no-cache .
mypy --no-incremental --cache-dir=/tmp/sleepkit-mypy-cache
python -m unittest discover -s tests -v
python scripts/verify_examples.py
python scripts/release_check.py
python -m build
twine check dist/*
check-wheel-contents dist/*.whl
```

Contributions must not contain recordings, annotations, clinical tables, credentials, private IDs, or local absolute paths. Scientific changes need a profile update, regression test, and clear validation boundary. See [CONTRIBUTING.md](https://github.com/lijinyang439-arch/PSGPrep/blob/main/CONTRIBUTING.md), [SECURITY.md](https://github.com/lijinyang439-arch/PSGPrep/blob/main/SECURITY.md), and [CHANGELOG.md](https://github.com/lijinyang439-arch/PSGPrep/blob/main/CHANGELOG.md).

## Citation and license

Use [CITATION.cff](https://github.com/lijinyang439-arch/PSGPrep/blob/main/CITATION.cff), cite each source dataset separately, and report the package version plus profile digest stored in `provenance.json`.

SleepKit PSG is licensed under Apache-2.0. The license covers this software, not any input or generated dataset.
