Metadata-Version: 2.4
Name: fiducio
Version: 0.3.1
Summary: Model-agnostic post-hoc calibration for 2D and 3D semantic segmentation
Author: Fiducio maintainers (fiducio-ai)
Maintainer: fiducio-ai
License-Expression: Apache-2.0
Project-URL: Project page, https://fiducio-ai.github.io/Fiducio/paper/
Project-URL: Homepage, https://github.com/fiducio-ai/Fiducio
Project-URL: Documentation, https://fiducio-ai.github.io/Fiducio/
Project-URL: Repository, https://github.com/fiducio-ai/Fiducio
Project-URL: Issues, https://github.com/fiducio-ai/Fiducio/issues
Project-URL: Changelog, https://github.com/fiducio-ai/Fiducio/blob/main/CHANGELOG.md
Keywords: calibration,post-hoc calibration,semantic segmentation,uncertainty,temperature scaling,matrix scaling,dirichlet calibration,pytorch
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.22
Requires-Dist: torch>=2.10
Provides-Extra: plots
Requires-Dist: matplotlib>=3.5; extra == "plots"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.6; extra == "docs"
Requires-Dist: mkdocs-material>=9.5; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.30; extra == "docs"
Provides-Extra: dev
Requires-Dist: fiducio[plots]; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: ruff>=0.4.0; extra == "dev"
Requires-Dist: mypy>=1.11; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Provides-Extra: all
Requires-Dist: fiducio[plots]; extra == "all"
Requires-Dist: fiducio[docs]; extra == "all"
Requires-Dist: fiducio[dev]; extra == "all"
Dynamic: license-file

# Fiducio

[![Tests](https://github.com/fiducio-ai/Fiducio/actions/workflows/tests.yml/badge.svg)](https://github.com/fiducio-ai/Fiducio/actions/workflows/tests.yml)
[![Docs](https://github.com/fiducio-ai/Fiducio/actions/workflows/docs.yml/badge.svg)](https://fiducio-ai.github.io/Fiducio/)
[![Python](https://img.shields.io/badge/python-3.10%E2%80%933.14-blue)](https://www.python.org/)
[![License](https://img.shields.io/badge/license-Apache--2.0-green)](https://github.com/fiducio-ai/Fiducio/blob/main/LICENSE)

**Fiducio** is a model-agnostic Python library of **post-hoc calibrators for 2D
and 3D semantic segmentation**. Give it the logits or probabilities of any
segmentation model and a labelled calibration set to fit a probability
transform without retraining the segmentation model.
It implements temperature, vector, matrix and Dirichlet scaling, their
translation-invariant variants, and the class-conditional (CDC, CMSap, CMSop)
calibrators introduced in our paper
[*Rethinking Post-Hoc Calibration in Semantic Segmentation*](https://arxiv.org/abs/2607.01902),
accepted at TMLR. Whether the segmentation is preserved depends on the
calibrator; see the guarantees below.

It works with PyTorch U-Net, nnU-Net, SegFormer or any model that produces
logits: Fiducio only ever sees `(B, C, *spatial)` tensors and never needs to know
the architecture.

![Reliability diagrams of an over-confident model before and after calibration with TemperatureScaling and CMSop](https://raw.githubusercontent.com/fiducio-ai/Fiducio/main/docs/assets/reliability_before_after.png)

*Top-1 reliability diagrams on held-out synthetic data (regenerate with
[`examples/readme_figure.py`](https://github.com/fiducio-ai/Fiducio/blob/main/examples/readme_figure.py)).*

## Install

Fiducio is available on [PyPI](https://pypi.org/project/fiducio/):

```bash
pip install fiducio
```

or, in a uv application:

```bash
uv add fiducio
```

Requires Python ≥ 3.10 and PyTorch ≥ 2.10. The core install depends only on `numpy` and `torch`. Optional extras:
`fiducio[plots]` (matplotlib reliability diagrams), `fiducio[docs]`,
`fiducio[dev]`, `fiducio[all]`.

To install the development version from GitHub:

```bash
pip install "git+https://github.com/fiducio-ai/Fiducio.git"
```

For development from a checkout, use the committed uv lockfile:

```bash
uv sync --locked
uv run --no-sync pytest
```

This uses CPU PyTorch for development. See [CONTRIBUTING.md](https://github.com/fiducio-ai/Fiducio/blob/main/CONTRIBUTING.md)
for the pinned uv version, documentation and build commands.

## Quick start

```python
import torch
from fiducio import TemperatureScaling, load_calibrator

# From a held-out, labelled calibration set:
torch.manual_seed(0)
logits = torch.randn(8, 4, 64, 64)  # (B, C, H, W) model logits
labels = torch.randint(0, 4, (8, 64, 64))  # (B, H, W) labels

calibrator = TemperatureScaling(input_type="logits", device="cpu").fit(logits, labels)

# Apply to new predictions (returns calibrated probabilities):
probs = calibrator.transform(torch.randn(2, 4, 64, 64))

# Save and reload (loads on CPU by default):
calibrator.save("calibrator.pt")
calibrator = load_calibrator("calibrator.pt")
```

The random tensors demonstrate the API. Measure calibration on separate test
cases; see the [Quickstart](https://fiducio-ai.github.io/Fiducio/quickstart/).

Every calibrator exposes the same API:
`fit(predictions, targets, mask=None)`, `transform`, `predict_proba`,
`fit_transform`, `save`, and the top-level `load_calibrator`.

## Accepted tensor shapes

The class axis is **always** dimension 1.

| Use case | `predictions` | `targets` |
|----------|---------------|-----------|
| 2D segmentation | `(B, C, H, W)` | `(B, H, W)` |
| 3D segmentation | `(B, C, D, H, W)` | `(B, D, H, W)` |
| general n-D | `(B, C, *spatial)` | `(B, *spatial)` |
| per-pixel table | `(N, C)` | `(N,)` |

Inputs may be `logits` or `probs` (set `input_type`). Binary segmentation is two
channels (`C = 2`). Floating class-distribution targets, including MHR, have the
same shape as predictions. Use `mask` to exclude soft targets; `ignore_index`
applies to hard labels.

## Calibrators

| Class | Alias | Translation-invariant | Decision preservation |
|-------|-------|:---------------------:|-----------------------|
| `TemperatureScaling` | `TS` | yes | argmax and full order |
| `EnsembleTemperatureScaling` | `ETS` | yes | argmax and full order¹ |
| `VectorScaling` | `VS` | no | — |
| `MatrixScaling` | `MS` | no | — |
| `TranslationInvariantMatrixScaling` | `MSc` | yes | — |
| `DirichletCalibration` | `DC` | yes | — |
| `ClassConditionalMatrixScaling` | `CMS` / `CDC` | yes | — |
| `ArgmaxPreservingMatrixScaling` | `CMSAP` / `CMSap` | yes | argmax |
| `OrderPreservingMatrixScaling` | `CMSOP` / `CMSop` | yes | argmax and full order |
| `BinaryIsotonicCalibration` | — | yes | none; may change the decision at 0.5 |

¹ In exact arithmetic; in `float32` the mixture can resolve near-ties differently
once it collapses towards the uniform component.

Translation-invariant means the output is unchanged when the same constant is
added to every input logit of a voxel.

Every calibrator exposes `decision_function` (calibrated logits, or
log-probabilities for isotonic calibration). The nine parametric methods have
a configurable `optimizer` (`"adam"` default, or `"lbfgs"`); isotonic fitting
uses weighted PAVA instead.

### Early stopping

Adam-fitted calibrators support the paper's recipe of early stopping on a
validation set (with an optional learning-rate decay on plateaus):

```python
from fiducio import OrderPreservingMatrixScaling

calibrator = OrderPreservingMatrixScaling(max_iter=2000, patience=20, lr_patience=10)
calibrator.fit(
    logits,
    labels,
    val_predictions=val_logits,
    val_targets=val_labels,  # held out from both
)
```

The iterate with the best validation NLL is kept. The default fixed budget
(`max_iter`) is short, so it can underfit expressive calibrators; prefer a larger
`max_iter` together with `patience` for CDC, CMSap and CMSop. This reproduces
only the Adam + validation-NLL early-stopping recipe; the paper's case batching,
class weighting, schedules, clipping and hyperparameter search are not part of the
library (see the [implementation guide](https://github.com/fiducio-ai/Fiducio/blob/main/reproducibility/README.md)).

### Paper method mapping

| Paper method | Class |
|--------------|-------|
| TS | `TemperatureScaling` |
| MS | `MatrixScaling` |
| MSc | `TranslationInvariantMatrixScaling` |
| CDC | `ClassConditionalMatrixScaling` |
| CMSap | `ArgmaxPreservingMatrixScaling` |
| CMSop | `OrderPreservingMatrixScaling` |

CDC, CMSap and CMSop are class-conditional: they fit one affine map per
uncalibrated top class, optimized **jointly** in one cross-entropy loss.
Regularization applies to each expert's induced affine map in the common
logit space. Set `independent_experts=True` to fit each expert separately.
See [Calibrators](https://fiducio-ai.github.io/Fiducio/calibrators/) for the
regularization formula and method-specific limits.

Calibration metrics are included: `negative_log_likelihood`,
`expected_calibration_error` (ECE, population-weighted), `average_calibration_error`
(ACE, unweighted over the same bins — doesn't let a sparsely populated bin get
drowned out by a large one), `brier_score`, `reliability_curve`, plus an
optional reliability-diagram plot (`fiducio.plots.reliability_diagram`, needs
`fiducio[plots]`).

BA-ECE (boundary-aware ECE), per-case metric aggregation, the DSC/flip-rate
audit and the paper's paired hierarchical-bootstrap tests are not included; the
shipped metrics are top-1 pooled point estimates. `average_calibration_error` is the
unweighted mean over non-empty uniform bins (not the adaptive ACE of other work)
and is also available as `unweighted_calibration_error`.

Decision and ordering guarantees hold in exact arithmetic and are subject to
floating-point ties and rounding. They do not cover thresholded or post-processed
segmentations.

## Multi-rater calibration

[TwinTrack](https://arxiv.org/abs/2604.15950v2) (Kirscher et al., MIDL 2026,
short paper) motivates binary isotonic calibration towards the mean human
response (MHR). Fiducio also extends its nine parametric methods to accept
class distributions. Missing annotations are excluded from the vote counts.

```python
from fiducio import BinaryIsotonicCalibration, mean_human_response

mhr, annotated = mean_human_response(annotations, num_classes=2)  # (B,C,*spatial)
cal = BinaryIsotonicCalibration(input_type="probs").fit(probabilities, mhr, mask=annotated)
calibrated = cal.transform(test_probabilities)
```

The isotonic map is binary, with at most 250 bins and weighted PAVA. It may
change a segmentation at threshold 0.5. Its tie handling and interpolation are
Fiducio conventions; exact TwinTrack reproduction is not claimed. All fits
remain in memory. See the [multi-rater guide](https://fiducio-ai.github.io/Fiducio/multi_rater/)
and `examples/multi_rater_calibration.py` for validation and the distinction
between ECE against MHR and mean ECE against individual raters.

## Documentation

The [paper project page](https://fiducio-ai.github.io/Fiducio/paper/) presents the
methods, interactive illustrations and published results.

The [paper implementation guide](https://github.com/fiducio-ai/Fiducio/blob/main/reproducibility/README.md) records method
mapping, numerical-reference coverage, fitting limits and the remaining steps
needed to reproduce the paper's experiments. For ensemble inputs, see
[`examples/ensemble_pooling.py`](https://github.com/fiducio-ai/Fiducio/blob/main/examples/ensemble_pooling.py).

### Results in the paper

The paper compares these calibrators on real segmentation benchmarks. The
reliability diagrams below (paper appendix) show the uncalibrated ensemble
pooling (`p0`, `p̄`) against the best calibrator per dataset; the exact
protocol, datasets and metrics are in the paper. This repository ships the
calibrators, not the fitted models or benchmark pipelines, so it does not
regenerate these figures (see the
[implementation guide](https://github.com/fiducio-ai/Fiducio/blob/main/reproducibility/README.md)).

![Reliability diagrams from the paper on three datasets](https://raw.githubusercontent.com/fiducio-ai/Fiducio/main/docs/assets/paper_reliability.png)

Full guide and API reference: **https://fiducio-ai.github.io/Fiducio/**

- [Installation](https://fiducio-ai.github.io/Fiducio/installation/)
- [Quickstart](https://fiducio-ai.github.io/Fiducio/quickstart/)
- [Concepts](https://fiducio-ai.github.io/Fiducio/concepts/)
- [Calibrators](https://fiducio-ai.github.io/Fiducio/calibrators/)
- [Examples](https://fiducio-ai.github.io/Fiducio/examples/)

## Citation

Fiducio accompanies the paper *Rethinking Post-Hoc Calibration in Semantic
Segmentation* (Kirscher et al., Transactions on Machine Learning Research, 2026;
[OpenReview](https://openreview.net/forum?id=xwNoSNxgxV), preprint: [arXiv:2607.01902](https://arxiv.org/abs/2607.01902)).
If you use Fiducio in your research, please cite it using the metadata in
[`CITATION.cff`](https://github.com/fiducio-ai/Fiducio/blob/main/CITATION.cff), or:

```bibtex
@article{kirscher2026rethinking,
  title   = {Rethinking Post-Hoc Calibration in Semantic Segmentation},
  author  = {Kirscher, Tristan and Kahl, Kim-Celine and Kovacs, Balint and
             Rokuss, Maximilian and Maier-Hein, Klaus and Coubez, Xavier and
             Meyer, Philippe and Faisan, Sylvain},
  journal = {Transactions on Machine Learning Research},
  issn    = {2835-8856},
  year    = {2026},
  url     = {https://openreview.net/forum?id=xwNoSNxgxV}
}
```

For work using calibration to MHR or the isotonic approach, also cite
*TwinTrack: Post-hoc Multi-Rater Calibration for Medical Image Segmentation*
(Kirscher et al., MIDL 2026, short paper):

```bibtex
@misc{kirscher2026twintrack,
  title         = {TwinTrack: Post-hoc Multi-Rater Calibration for Medical Image Segmentation},
  author        = {Kirscher, Tristan and Ertl, Alexandra and Maier-Hein, Klaus and
                   Coubez, Xavier and Meyer, Philippe and Faisan, Sylvain},
  year          = {2026},
  eprint        = {2604.15950},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  note          = {Accepted at MIDL 2026, short paper},
  url           = {https://arxiv.org/abs/2604.15950v2}
}
```

## Status

Fiducio is **beta** (`0.x`): usable and tested, with an API that may still change
before `1.0`. It is not certified for safety-critical or clinical use.

## License

[Apache-2.0](https://github.com/fiducio-ai/Fiducio/blob/main/LICENSE).
