Metadata-Version: 2.4
Name: ocr-util
Version: 3.0.1
Summary: OCR Utils
Author-email: Universitäts- und Landesbibliothek Sachsen-Anhalt <development@bibliothek.uni-halle.de>
License-Expression: MIT
Project-URL: Homepage, https://github.com/ulb-sachsen-anhalt/ocr-util
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rapidfuzz>3
Requires-Dist: nltk
Requires-Dist: requests
Requires-Dist: docker
Requires-Dist: numpy
Requires-Dist: shapely
Requires-Dist: lxml
Requires-Dist: opencv-python-headless
Requires-Dist: exifread
Requires-Dist: Pillow
Provides-Extra: dev
Requires-Dist: black; extra == "dev"
Requires-Dist: pylint; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: coverage-badge; extra == "test"
Requires-Dist: pytest-xdist; extra == "test"
Requires-Dist: flake8; extra == "test"
Requires-Dist: lxml-stubs; extra == "test"
Requires-Dist: opencv-stubs; extra == "test"
Dynamic: license-file

# OCR Util

![python-app](https://github.com/ulb-sachsen-anhalt/ocr-util/actions/workflows/python-app.yml/badge.svg) [![Coverage](https://raw.githubusercontent.com/ulb-sachsen-anhalt/ocr-util/main/coverage.svg)](https://github.com/ulb-sachsen-anhalt/ocr-util/actions/workflows/python-app.yml) [![PyPi version](https://badgen.net/pypi/v/ocr-util/)](https://pypi.org/project/ocr-util) ![PyPI - Downloads](https://img.shields.io/pypi/dm/ocr-util) ![PyPI - License](https://img.shields.io/pypi/l/ocr-util) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/ocr-util)


Collection of utils for 
* evaluation of OCR data for the masses
* generation of pair-wise Trainingdata for OCR-Backends

## Requirements

* recent *nix-OS
* Python3.10+ Environment

## Usage

Each section contains detailed usage help instructions:

```bash
# evaluation
ocr-util eval --help

# slice image by image + input OCR
ocr-util slice --help

# render input OCR (regions, lines, words) on given image
ocr-util show --help
```

### Data problems

Inconsistent OCR Groundtruth with empty texts (ALTO String elements missing CONTENT or PAGE without TextEquiv) or invalid geometrical coordinates (less than 3 points or even empty) will lead to evaluation errors if geometry must be respected.

_Please note_:  
Invalid data files are excluded and reported where possible from evaluation.  
The term 'invalid' refers to malformed or structurally invalid XML, i.e. syntax errors and further if included geometrical information includes inconsistencies like missing points or missmatching shapes.

### Evaluation Filter-Then-Aggregate

The evaluation CLI supports a single pre-aggregation filter using metadata extractors.

Example: keep only entries where MODS language is exactly German, then aggregate by publication century:

```bash
ocr-util eval <candidates> \
	--reference <groundtruth> \
	--mets-file <mets.xml> \
	--filter-by "mods:language=ger" \
	--aggregate-by "mods:dateIssued:century"
```

The dimensions can also be reversed: filter by publication century or decade,
then aggregate by language. For example, keep only publications from the 17th
century (1601-1700 inclusive), then group their evaluation results by MODS language:

```bash
ocr-util eval <candidates> \
	--reference <groundtruth> \
	--mets-file <mets.xml> \
	--filter-by "mods:dateIssued:century=17th" \
	--aggregate-by "mods:language"
```

Filtering still happens before evaluation and aggregation; only the metadata
dimensions exchange roles. To select a decade instead, replace the filter above
with `--filter-by "mods:dateIssued:decade=1800s"` (1800-1809 inclusive).

Multi-language filter values are interpreted as sets:

```bash
ocr-util eval <candidates> \
	--reference <groundtruth> \
	--mets-file <mets.xml> \
	--filter-by "mods:language=ger+eng" \
	--aggregate-by "mods:dateIssued:century"
```

Behavior:
* centuries follow calendar numbering: the 18th century is 1701-1800,
	and the 19th century is 1801-1900; decades retain ten-year buckets such as 1800-1809
* single filter value -> exact match (e.g. `ger` does not match `ger+eng`)
* multi-value filter -> all filter values must be present in any order
* entries missing the filter criterion are reported as WARNING and discarded
* METS coverage reports how many evaluation pairs map to full-text references
	and how many METS references have no evaluation pair
* aggregation coverage reports how many evaluated pairs provide each dimension;
	reported `items` count candidate/ground-truth pairs, not MODS elements

## Development

Platform: Intel(R) Core(TM) i5-6500 CPU@3.20GHz, 16GB RAM, Ubuntu 22.04 LTS, Python 3.10+

```bash
# clone local
git clone <repository-url> <local-dir>
cd <local-dir>

# enable virtual python 3 environment (linux)
# and update pip itself
python3.10 -m venv venv
. venv/bin/activate
python -m pip install -U pip

# install with dev dependencies
python -m pip install -e ".[dev,test]"

# run tests with coverage
python -m pytest --cov=src

# run tests faster (parallel, auto worker count)
python -m pytest -q -n auto
```

## Contribution

Contributions, suggestions and proposals welcome!

## License

Under terms of the [MIT license](https://opensource.org/licenses/MIT).

**NOTE**: This software depends on packages that _might_ be licensed under different terms.
