Metadata-Version: 2.4
Name: meddeid-eval
Version: 0.2.0
Summary: Canonical metrics and stability analysis for MedDeID
Author: Stig Hellemans
License-Expression: AGPL-3.0-only
Project-URL: Homepage, https://meddeid.github.io/
Project-URL: Documentation, https://meddeid.github.io/workflows/train-and-evaluate/
Project-URL: Source, https://github.com/stighellemans/meddeid-eval
Project-URL: Issues, https://github.com/stighellemans/meddeid-eval/issues
Project-URL: MedDeID on Hugging Face, https://huggingface.co/collections/stighellemans/meddeid
Keywords: de-identification,clinical-nlp,evaluation,privacy
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: meddeid-core>=0.1.0
Requires-Dist: meddeid-language-nl>=0.1.0
Requires-Dist: PyYAML>=6
Provides-Extra: infer
Requires-Dist: meddeid>=0.1.0; extra == "infer"
Provides-Extra: plots
Requires-Dist: matplotlib>=3.7; extra == "plots"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: matplotlib>=3.7; extra == "dev"
Dynamic: license-file

# meddeid-eval

Reproducible evaluation for clinical de-identification. `meddeid-eval` computes
exact-span metrics, character-level recall, core-PII recall, non-PII redaction
rate, and stability results from canonical MedDeID JSONL files.

See the [suite evaluation workflow](https://meddeid.github.io/workflows/train-and-evaluate/#evaluate-predictions)
for the end-to-end handoff. This repository remains authoritative for metric
definitions, commands, stability configuration, and plotting support.

## Installation

```bash
python -m pip install meddeid-eval
```

Optional extras add model inference and plotting support:

```bash
python -m pip install 'meddeid-eval[infer,plots]'
```

## Usage

```bash
meddeid-eval score --gold meddeid-dutch-synthetic-benchmark.jsonl --predictions predictions.jsonl
meddeid-eval stability expand --config stability.yaml
```

Gold and prediction files are matched by `document_id` and use half-open
`[begin, end)` Unicode-code-point offsets. The score command reports exact
precision, recall, and F1 together with character coverage and redaction
metrics.

Core-PII recall is the label-agnostic fraction of protocol-defined core PII
characters covered by any predicted redaction. Each primary gold span owns a
nested `subannotations` list. `formatting`, `additional_info`, `medical_info`,
`title`, and `time` segments are excluded from the denominator.

Stability perturbations use the complete `nl-BE` resources from
`meddeid-language-nl`; incomplete resources are reported as installation
errors.

## External comparators

Comparison systems run in their own environments. Export their predictions in
the canonical MedDeID JSONL schema and evaluate them with the same `score`
command. Belgian DEDUCE is not installed by `meddeid-eval`.

## Development

```bash
pip install -e '.[dev]'
pytest
```

## Licence

AGPL-3.0-only. External comparison systems retain their own licence terms.
