Metadata-Version: 2.4
Name: mote_error
Version: 0.1.0
Summary: MOTe error (Missing, Overlap, Trespass, and Excess) for Document Layout Analysis, and the Character Error Vector for page-level OCR
Author: Jonathan Bourne, Mwiza Simbeye
Project-URL: Homepage, https://github.com/THE-3TC/mote_error
Project-URL: Repository, https://github.com/THE-3TC/mote_error
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<2.0,>=1.21.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: pillow>=9.0.0
Requires-Dist: datasets>=2.0.0
Requires-Dist: matplotlib>=3.5.0
Requires-Dist: pycocotools>=2.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: flake8>=6.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: isort>=5.12.0; extra == "dev"
Provides-Extra: benchmarks
Requires-Dist: torch>=2.5.1; extra == "benchmarks"
Requires-Dist: torchvision>=0.20.1; extra == "benchmarks"
Requires-Dist: transformers>=4.46.3; extra == "benchmarks"
Requires-Dist: datasets>=2.0.0; extra == "benchmarks"
Requires-Dist: huggingface-hub>=0.16.0; extra == "benchmarks"
Requires-Dist: evaluate>=0.4.0; extra == "benchmarks"
Requires-Dist: tqdm>=4.65.0; extra == "benchmarks"
Requires-Dist: marimo>=0.8.22; extra == "benchmarks"
Requires-Dist: requests>=2.32.4; extra == "benchmarks"
Requires-Dist: jinja2>=3.1.6; extra == "benchmarks"
Provides-Extra: models
Requires-Dist: transformers>=4.30.0; extra == "models"
Requires-Dist: torch>=2.0.0; extra == "models"
Requires-Dist: torchvision>=0.15.0; extra == "models"
Provides-Extra: pipeline
Requires-Dist: apache-beam>=2.69.0; extra == "pipeline"
Requires-Dist: pyarrow>=12.0.0; extra == "pipeline"
Requires-Dist: pyyaml>=6.0; extra == "pipeline"
Requires-Dist: pytesseract>=0.3.10; extra == "pipeline"
Requires-Dist: torch>=2.0.0; extra == "pipeline"
Requires-Dist: torchvision>=0.15.0; extra == "pipeline"
Requires-Dist: transformers>=4.30.0; extra == "pipeline"
Requires-Dist: easyocr>=1.7.0; extra == "pipeline"
Provides-Extra: paddle-benchmark
Requires-Dist: datasets>=2.0.0; extra == "paddle-benchmark"
Requires-Dist: huggingface-hub>=0.16.0; extra == "paddle-benchmark"
Requires-Dist: evaluate>=0.4.0; extra == "paddle-benchmark"
Requires-Dist: tqdm>=4.65.0; extra == "paddle-benchmark"
Requires-Dist: marimo>=0.8.22; extra == "paddle-benchmark"
Requires-Dist: requests>=2.32.4; extra == "paddle-benchmark"
Requires-Dist: paddleocr>=2.9.0; extra == "paddle-benchmark"
Dynamic: license-file

<p align="center">
  <img src="https://raw.githubusercontent.com/THE-3TC/mote_error/main/docs/mote_error_banner.svg"
       alt="mote_error" width="700">
</p>


**Decomposable evaluation metrics for document understanding pipelines — the MOTe error for layout parsing, and the Character Error Vector for page-level OCR**

[![PyPI version](https://img.shields.io/pypi/v/mote_error)](https://pypi.org/project/mote_error/)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)

> **mote_error is the successor to [`cotescore`](https://pypi.org/project/cotescore/).** The layout metric has been reframed from the COTe score into the MOTe error; see [Coming from the COTe score?](#layout-parsing--the-mote-error) below.


## Overview

Document Layout Analysis (DLA) is the process of parsing a page into meaningful elements, typically using machine learning models. Traditional evaluation metrics such as IoU, F1, and mAP were designed for 3D-to-2D image projections (e.g. photographs) and can give misleading results for natively 2D printed documents.

Document understanding pipelines generally have two stages. The first stage parses the page into individual visual elements or regions, effectively focusing on answering **where is the text?**. The second stage is the Optical Character Recognition, where the text is transcribed in to machine readable format, answering **what does the text say?**. When the output text quality is poor conventional metrics can struggle to identify the root cause. 

The **mote_error** library provides two decomposable metrics one for each stage of the pipeline.  

For a conceptual introduction to these metrics there is a blog post on the [COTe score](https://the-3tc.github.io/blog/cote-score/) (the original framing that MOTe reframes as an error), a blog post on the [CEV](https://the-3tc.github.io/blog/cev-score/), and two [tutorials](https://lightning.ai/the3tc/templates/cotescore-library-tutorial) available to run for free on a lightning ai studio (written for `cotescore`).

### Layout parsing — the MOTe error


- **Structural Semantic Units (SSUs)** — a relational labelling approach that shifts focus from the physical bounding boxes of regions to the semantic structure of the content.
- **MOTe error** — a decomposable error that breaks page-parsing quality into interpretable components:
  - **M**issing — ground-truth area that no prediction covers (the complement of Coverage)
  - **O**verlap — redundant predictions covering the same ground-truth area
  - **T**respass — predictions that cross semantic boundaries into a different SSU
  - **e**xcess — a support metric for predictions that fall on background or white-space

$$\mathrm{MOTe} = \mathcal{M} + \mathcal{O} + \mathcal{T}$$

0 is a perfectly parsed page and 1 is a page with no predictions. Values above 1 are possible, and mean that Overlap plus Trespass outweigh the area the predictions cover. Excess is reported alongside but is not part of the sum.

MOTe is more informative than traditional metrics, reveals distinct model failure modes, and remains useful even when explicit SSU labels are unavailable. Because it works on per-instance masks it also applies to instance and panoptic segmentation (where Overlap is zero by construction).

> **Coming from the COTe score?** MOTe is the same decomposition reframed as an error. With default weights `MOTe = 1 − COTe` and `Missing = 1 − Coverage`, so previously computed results convert directly.

### Page-level OCR — the Character Error Vector

Character Error Rate assumes the text was parsed perfectly. When it was not, the reading order is wrong, the alignment CER depends on becomes meaningless, and the metric breaks down precisely when a diagnosis is most needed.

- **Character Error Vector (CEV)** — a bag-of-characters evaluator that needs no alignment, and decomposes page-level OCR error into **parsing**, **OCR** and **interaction** components.
- **Two instantiations** ship with the library: **SpACER** (Spatially Aware Character Error Rate), a count-based metric analogous to CER, and **CDD**, a character-distribution distance built on the Jensen–Shannon distance.

Because the components separate the stages, the CEV can say whether the transcription was let down by the parser or by the OCR engine. This library is the reference implementation for the CEV paper.

## Installation

```bash
pip install mote_error
```

That is all you need to compute MOTe and CEV scores on predictions you already have. Running the layout models themselves requires extra dependencies — see [Optional extras](#optional-extras--running-the-benchmark-models) at the end.

## Quick Start

The library ships with a bundled limerick case study that you can use to try it out immediately.

```python
from mote_error import mote, load_limerick_example, extract_ssu_boxes
from mote_error.adapters import boxes_to_gt_ssu_map, boxes_to_pred_masks, compute_canvas

# Load the bundled example: ground-truth dict, document image, and example predictions
ground_truth, image, pred_boxes = load_limerick_example()

h, w = image.shape[:2]
canvas_w, canvas_h = compute_canvas(w, h)

# Build tagged SSU-level GT boxes and rasterize to a 2-D SSU id map
gt_boxes = extract_ssu_boxes(ground_truth)
gt_ssu_map = boxes_to_gt_ssu_map(gt_boxes, w, h, canvas_w, canvas_h)
preds = boxes_to_pred_masks(pred_boxes, w, h, canvas_w, canvas_h)

# Compute the MOTe error; the result is a named tuple
result = mote(gt_ssu_map, preds)
print(f"MOTe={result.mote:.3f}  M={result.missing:.3f}  O={result.overlap:.3f}  "
      f"T={result.trespass:.3f}  e={result.excess:.3f}")
```
---

![MOTe pixel-state visualisation showing Covered, Missing, Overlap, Trespass and Excess regions](docs/example_mote_components.png)

*MOTe pixel-state visualisation: each pixel in the document is classified as Covered (green), Overlap (yellow), Trespass (red), Trespass AND Overlap (purple), Missing (grey), or Excess (blue). Produced by `notebooks/limerick_analysis.py`.*

---

See [`notebooks/limerick_analysis.py`](notebooks/limerick_analysis.py) for a full worked example comparing MOTe against F1 and mean IoU at different granularity levels.

## Core Metrics — MOTe

| Function | Description |
|---|---|
| `mote(gt_ssu_map, preds)` | Returns a `MOTeResult` named tuple `(mote, missing, overlap, trespass, excess)` `[0, ∞)` |
| `missing(gt_ssu_map, preds)` | Fraction of GT area no prediction covers `[0, 1]` |
| `coverage(gt_ssu_map, preds)` | Fraction of GT area covered, `1 − missing` `[0, 1]` |
| `overlap(gt_ssu_map, preds)` | Redundant prediction area within GT `[0, ∞)` |
| `trespass(gt_ssu_map, preds)` | GT area covered by predictions assigned to a different SSU `[0, ∞)` |
| `excess(gt_ssu_map, preds)` | Predicted area falling on background, over total background `[0, 1]` |
| `mote_class(gt_ssu_map, ssu_to_class, preds)` | Per-class interaction matrices and per-GT-class Missing (`ClassMOTeResult`) |

All functions are importable directly from `mote_error`:

```python
from mote_error import mote, missing, coverage, overlap, trespass, excess, mote_class
```

`mote` accepts `weight_missing`, `weight_overlap` and `weight_trespass` to reweight the sum; equal weights are the default because each component measures the same spatial area. When ground truth is available as boxes, pass a `GTBoxes` and an `(M, 4)` array of predicted boxes to `mote` to use the analytic fast path, which never rasterises. A page with no SSUs has Missing, Overlap and Trespass of 0.

The conventional instance-matching baselines MOTe is compared against in the paper — `iou`, `mean_iou` and `f1` — remain available from `mote_error.layout` if you want to reproduce those comparisons.

For an alternative visual overview of what the different elements of the MOTe error mean please see [`notebooks/metrics_exploration.py`](notebooks/metrics_exploration.py)

## Core Metrics — the CEV

The CEV compares four bags of characters. Which ones you can build determines which components you get:

| Symbol | Key | What it is |
|---|---|---|
| $Q$ | `gt` | all ground-truth characters on the page |
| $R$ | `parsing` | ground-truth characters falling inside **predicted** regions |
| $S^*$ | `ocr` | OCR run on **ground-truth** regions |
| $S$ | `total` | OCR run on **predicted** regions |

From those come the four components — `d_pars` (parsing alone), `d_ocr` (OCR given perfect regions), `d_int` (their interaction), and `d_total` (the whole pipeline).

| Function | Description |
|---|---|
| `spacer(reference, prediction)` | Macro SpACER between two token Counters `[0, ∞)`: $(\hat{E} + D + I) / 2C$ with net deletions $D$ and insertions $I$ at page level |
| `spacer_micro(ref_boxes, pred_boxes)` | Per-box SpACER: $D$ and $I$ accumulated per box, so a deletion in one box is not cancelled by an insertion in another |
| `spacer_decomp(named_dict)` | Four-way SpACER decomposition (macro and micro) from page or per-box text |
| `cdd_decomp(named_dict)` | Four-way CDD decomposition; any `(Counter, Counter) -> float` metric can be substituted |
| `spacer_decomp_spatial(gt_chars, pred_regions, ...)` | As above, building $R$ by joining character positions to predicted regions |
| `cdd_decomp_spatial(gt_chars, pred_regions, ...)` | Spatial CDD equivalent |
| `jsd_distance(p, q)` | Square root of the Jensen–Shannon divergence `[0, 1]` — the default CDD metric |
| `text_to_counter(text, mode)` | Build a character (or word, for SpAWER) frequency Counter |

```python
from mote_error import spacer_decomp, cdd_decomp

bags = {"gt": gt_text, "ocr": ocr_of_gt_regions, "total": ocr_of_predicted_regions}
sp = spacer_decomp(bags)
cd = cdd_decomp(bags)
print(sp.d_ocr_macro, sp.d_total_macro)
```

Components whose inputs are absent come back as `None` rather than failing. Building $R$ needs character-level positions, so a dataset with only region-level text yields `d_ocr` and `d_total` while `d_pars` and `d_int` are `None`. That pair is still enough to triage: combining MOTe with the ratio `d_ocr / d_total` predicts the dominant error source with an F1 of 0.91, using only values that are cheap to obtain. The `*_spatial` functions recover the full vector when character boxes are available.

## Visualisation

```python
from mote_error import compute_mote_masks, visualize_mote_states
import matplotlib.pyplot as plt

masks = compute_mote_masks(gt_ssu_map, preds)

fig, ax = plt.subplots()
visualize_mote_states(image, masks, ax=ax)
plt.show()
```

## Datasets

The library includes loaders for the datasets used in the two papers, all sharing one annotation format:

```python
from mote_error.dataset import (
    NCSEDataset, DocLayNetDataset, HNLA2013Dataset,
    SpiritualistDataset, HierTextDataset, DocBankDataset,
)

# Bundled toy example (no download required)
from mote_error import load_limerick_example
chars_df, image, pred_boxes = load_limerick_example()
```

## Examples

The primary interactive example is the Marimo notebook at [`notebooks/limerick_analysis.py`](notebooks/limerick_analysis.py). It demonstrates:

- Visualising SSU, line, and character-level bounding boxes
- The granularity-mismatch problem and how MOTe handles it
- Side-by-side comparison of MOTe, F1, and mean IoU
- MOTe pixel-state visualisation

To run the notebook (requires [Marimo](https://marimo.io)):

```bash
pip install marimo
marimo edit notebooks/limerick_analysis.py
```

### End-to-end walkthrough on a real dataset

There are two tutorial notebooks, covering MOTe and CEV that provide hands on experience of the metrics. The tutorials use the `marimo` library, the NCSEv2 test set and require a GPU. A 1-click reproducible lightning.ai studio is available to run for free [here](https://lightning.ai/the3tc/templates/cotescore-library-tutorial)

1. [`notebooks/Tutorial_1_the_MOTe.py`](notebooks/Tutorial_1_the_MOTe.py) — scores two layout
   models (DocLayout-YOLO and Docling Heron) with MOTe, compares it against F1 and mean IoU,
   and writes a folder of MOTe pixel-state diagnostics per model.
2. [`notebooks/Tutorial_2_the_CEV.py`](notebooks/Tutorial_2_the_CEV.py) — runs EasyOCR and
   Tesseract over both models' regions and scores them with the CEV (SpACER and CDD), then uses
   MOTe together with `d_ocr / d_total` to attribute each page's error to parsing or to OCR.

Both notebooks take the images folder and the ground-truth CSV as plain string paths at the top,
so they can be pointed at your own dataset. The layout models need a GPU and the
`doclayout-yolo`, `easyocr` and `pytesseract` packages (plus the `tesseract` binary).

```bash
marimo edit notebooks/Tutorial_1_the_MOTe.py
marimo edit notebooks/Tutorial_2_the_CEV.py
```

## Optional extras — running the benchmark models

Only needed if you want to run the document layout models yourself, for example to reproduce the papers' benchmarks. Scoring existing predictions needs none of this.

The benchmarking extras include either PyTorch or PaddlePaddle — **do not install both in the same environment**. These frameworks require different CUDA versions and will conflict:

| Extra | Use case | GPU framework |
|---|---|---|
| `mote_error[benchmarks]` | Torch-based DLA models (DocLayout-YOLO, Heron) | PyTorch |
| `mote_error[paddle-benchmark]` | PaddleOCR / PP-DocLayout | PaddlePaddle |

```bash
# Torch-based benchmarks
pip install "mote_error[benchmarks]"

# PaddlePaddle benchmarks (install PaddlePaddle first, separately)
# See: https://www.paddlepaddle.org.cn/install/quick
pip install "mote_error[paddle-benchmark]"
```

> **Note:** PaddlePaddle must be installed before `mote_error[paddle-benchmark]` — it is not listed as a pip dependency because the correct wheel depends on your CUDA version. Follow the [PaddlePaddle installation guide](https://www.paddlepaddle.org.cn/install/quick) to get the right version.

## Questions and Bug Reports

If you have questions, find a bug, or want to request a feature, please [open an issue](https://github.com/THE-3TC/mote_error/issues) on GitHub.

## Citation

If you use mote_error in your research, please cite the paper for the metric you used.

**MOTe** — layout parsing. MOTe reframes the COTe score from this paper as an error (`MOTe = 1 − COTe`); until the MOTe paper is out, please cite:

Bourne, Jonathan, Mwiza Simbeye, and Ishtar Govia. “The COTe Score: A Decomposable Framework for Evaluating Document Layout Analysis Models.” arXiv:2603.12718. Preprint, arXiv, March 13, 2026. https://doi.org/10.48550/arXiv.2603.12718.


```bibtex
@misc{bourne2026cote,
  title         = {The {COTe} Score: A Decomposable Framework for Evaluating {Document Layout Analysis} Models},
  author        = {Bourne, Jonathan and Simbeye, Mwiza and Govia, Ishtar},
  year          = {2026},
  month         = mar,
  publisher     = {arXiv},
  doi           = {10.48550/arXiv.2603.12718},
}
```

**CEV / SpACER** — page-level OCR:

Bourne, Jonathan, Mwiza Simbeye, and Joseph Nockels. "The Character Error Vector: Decomposable Errors for Page-Level OCR Evaluation." arXiv:2604.06160. Preprint, arXiv, April 2026. https://doi.org/10.48550/arXiv.2604.06160.

```bibtex
@misc{bourne2026charactererrorvector,
  title         = {The {Character Error Vector}: Decomposable errors for page-level {OCR} evaluation},
  author        = {Bourne, Jonathan and Simbeye, Mwiza and Nockels, Joseph},
  year          = {2026},
  month         = apr,
  eprint        = {2604.06160},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2604.06160},
}
```
