Metadata-Version: 2.4
Name: sumeds
Version: 0.3.0
Summary: Lazy, privacy-aware code occurrence summaries for MEDS datasets
Keywords: MEDS,healthcare,Polars,privacy
Author: Robin P. van de Water
Author-email: Robin P. van de Water <robin.vandewater@hpi.de>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Dist: meds>=0.4.1,<0.5
Requires-Dist: polars>=1.30
Requires-Dist: pyarrow>=15
Requires-Dist: pyyaml>=6
Requires-Dist: tqdm>=4.67
Requires-Dist: jupyter>=1.0 ; extra == 'notebook'
Requires-Dist: ipykernel>=6.30 ; extra == 'notebook'
Requires-Dist: polars[rt64]>=1.34 ; extra == 'rt64'
Requires-Python: >=3.10, <3.15
Provides-Extra: notebook
Provides-Extra: rt64
Description-Content-Type: text/markdown

# suMEDS

[![PyPI - Version](https://img.shields.io/pypi/v/suMEDS)](https://pypi.org/project/suMEDS/)
[![codecov](https://codecov.io/gh/rvandewater/suMEDS/graph/badge.svg?token=RW6JXHNT0W)](https://codecov.io/gh/rvandewater/suMEDS)
[![tests](https://github.com/rvandewater/suMEDS/actions/workflows/tests.yaml/badge.svg)](https://github.com/rvandewater/suMEDS/actions/workflows/tests.yml)
[![code-quality](https://github.com/rvandewater/suMEDS/actions/workflows/code-quality.yaml/badge.svg)](https://github.com/rvandewater/suMEDS/actions/workflows/code-quality-main.yaml)
[![Python Version](https://img.shields.io/pypi/pyversions/suMEDS.svg)](https://pypi.python.org/pypi/suMEDS/)
[![license](https://img.shields.io/badge/License-MIT-green.svg?labelColor=gray)](https://github.com/rvandewater/suMEDS#license)
[![PRs](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/rvandewater/suMEDS/pulls)
[![contributors](https://img.shields.io/github/contributors/rvandewater/suMEDS.svg)](https://github.com/rvandewater/suMEDS/graphs/contributors)
[![DOI](https://zenodo.org/badge/1307476724.svg)](https://doi.org/10.5281/zenodo.21647050)
[![MEDS v0.4](https://img.shields.io/badge/MEDS-0.4-blue)](https://medical-event-data-standard.github.io/)

`suMEDS` (pronounced "summed-s", past tense of "sum" + s) creates a compact code catalog for performing tasks
[Medical Event Data Standard (MEDS)](https://github.com/Medical-Event-Data-Standard/meds)
datasets. MEDS data should be "summed" in this way in a storage-efficient format, without the need to access the full dataset.


It can be seen as a form of "extended metadata," as we sometimes need more information than vanilla MEDS metadata provides, but don't want to expose our complete dataset (either because of privacy concerns, storage capacity, or token use efficiency for LLMs).
It joins canonical descriptions and configurable parent, child, and sibling
codes to event and unique-subject counts, then masks or removes rare codes
before writing Parquet, CSV, or JSON.

> [!WARNING]
> This package is in early development and is human-guided, but AI-agent-generated. It is not yet reviewed for production use. Please report issues and suggestions.
>


Polars scans, aggregations, joins, and writes stay lazy. This should mean that it could run with little resources on large MEDS datasets.

## Install and run
Normal install:

```bash
pip install sumeds
```

Datasets with more than 2³² rows require Polars' 64-bit runtime:

```bash
pip install "sumeds[rt64]"
```

### Development
```bash
uv sync
uv run suMEDS tests/resources/MIMICIV_DEMO/MEDS_cohort \
  --output code-summary.parquet \
  --min-subjects 20 \
  --split-columns
```

Or use YAML:

```yaml
summary:
  per_split: false
  split_columns: true
  partitions: 256
privacy:
  min_subjects: 20
  min_split_subjects: 1
  rare_code_action: bucket
  rare_code_label: __RARE__
  round_counts_to: 5
```

```bash
uv run suMEDS /path/to/MEDS -o summary.parquet -c examples/summary.yaml
# The suffix selects CSV or JSON instead:
uv run suMEDS /path/to/MEDS -o summary.json -c examples/summary.yaml
```

CLI flags override YAML. The defaults bucket codes seen in fewer than 20 unique
subjects and retain exact counts.

Optionally fill missing descriptions, all ancestors, OMOP concept IDs, domains,
and standard-concept markers from OHDSI Athena. Child and sibling expansion are
opt-in:

```bash
uv run suMEDS /path/to/MEDS -o summary.parquet \
  --athena-csv /path/to/athena --child-codes --child-depth 3 --sibling-codes
# Or use PostgreSQL through the installed psql client:
uv run suMEDS /path/to/MEDS -o summary.parquet \
  --athena-postgres postgresql://postgres@localhost/omop
```

Standalone enrichment avoids rescanning event data:

```bash
uv run suMEDS-enrich metadata/codes.parquet -o codes-enriched.parquet \
  --athena-csv /path/to/athena
```

The standalone command shows phase progress and reports Athena matches and
before/after metadata coverage.

Both `VOCABULARY//CODE//...` and
`VOCABULARY//VOCABULARY_VERSION//CODE` layouts are resolved. See the
[Athena enrichment guide](docs/enrichment.md).

## Python API

```python
from sumeds import EnrichmentConfig, SummaryConfig, summarize

path = summarize(
    "/path/to/MEDS",
    "code-summary.parquet",
    SummaryConfig(
        min_subjects=20,
        rare_code_action="bucket",
        enrichment=EnrichmentConfig(csv_dir="/path/to/athena"),
    ),
)
```

The output format is inferred from `.parquet`, `.csv`, or `.json` (with
`.jsonl`/`.ndjson` also supported). It preserves all code-metadata extension columns and adds
`event_count`, `subject_count`, and `is_masked`. `per_split=True` emits one row
per split. `split_columns=True` keeps the total row and adds columns such as
`event_count_train` and `subject_count_train` from `subject_splits.parquet`.
Set `min_split_subjects` above 1 to suppress rare split-level cells.

## Documentation

```bash
uv run mkdocs serve
uv run mkdocs build --strict
```

See the [documentation site](docs/index.md) for the data flow, complete CLI and
YAML reference, output schema, Python API, and privacy limitations.

## Development

```bash
uv run pre-commit run --all-files
uv run pytest
uv run mkdocs build --strict
```

The included integration test uses `tests/resources/MIMICIV_DEMO/MEDS_cohort` when present.
