Metadata-Version: 2.5
Name: pyconverters-paddleocr
Version: 1.6.59
Summary: Convert PDF to structured text using PaddleOCR
Author-email: Olivier Terrier <olivier.terrier@kairntech.com>
License: Proprietary
License-File: AUTHORS.md
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: beautifulsoup4
Requires-Dist: filetype>=1.2.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: markdownify
Requires-Dist: pylatexenc
Requires-Dist: pymultirole-plugins<1.7.0,>=1.6.0
Requires-Dist: pypdf>=5.0.0
Requires-Dist: requests
Provides-Extra: dev
Requires-Dist: bump2version; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Provides-Extra: docs
Requires-Dist: lxml-html-clean; extra == 'docs'
Requires-Dist: m2r2; extra == 'docs'
Requires-Dist: sphinx; extra == 'docs'
Requires-Dist: sphinx-rtd-theme; extra == 'docs'
Requires-Dist: sphinxcontrib-apidoc; extra == 'docs'
Provides-Extra: sbom
Requires-Dist: cyclonedx-bom; extra == 'sbom'
Requires-Dist: pip-audit; extra == 'sbom'
Provides-Extra: test
Requires-Dist: langdetect; extra == 'test'
Requires-Dist: pandas; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: ruff; extra == 'test'
Description-Content-Type: text/markdown

# pyconverters_paddleocr

[![license](https://img.shields.io/badge/license-proprietary-red)](LICENSE)
[![version](https://img.shields.io/pypi/v/pyconverters-paddleocr)](https://pypi.org/project/pyconverters-paddleocr/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pyconverters-paddleocr)](https://pypi.org/project/pyconverters-paddleocr/)

Convert PDF to structured text using [PaddleOCR](https://github.com/kermitt2/paddleocr)

## Installation

You can simply `pip install pyconverters_paddleocr`.

## OCR service on CPU

The converter calls the `/layout-parsing` API of a PaddleX server. Besides PaddleOCR-VL on
GPU (`deploy/paddleocr-vl`), `deploy/paddleocr-cpu` builds an image serving PP-StructureV3
with the PP-OCRv6 medium models on CPU, on the same API: the converter works with both.

### Build and publish the image

```bash
cd deploy/paddleocr-cpu
docker compose build         # image paddleocr-structure-cpu:3.7.0
docker tag paddleocr-structure-cpu:3.7.0 kairntech/paddleocr-structure-cpu:3.7.0-ocrv6
docker push kairntech/paddleocr-structure-cpu:3.7.0-ocrv6
```

The build patches PaddleX so that words are no longer glued after an accented letter or a
digit, and generates the PP-OCRv6 pipeline configuration; it fails if a PaddleX upgrade
changed the code or the model names it relies on. `docker compose up -d` in the same
directory runs the service locally on port 9093.

The models (~1 GB) are not in the image: they are downloaded at the first start, which needs
an Internet access, and kept in a volume. The first request after a start is slow.

### Add it to the Sherpa docker-compose

Under `services:`, at the indentation of the other services:

```yaml
    sherpa-paddleocr-cpu:
        container_name: "sherpa-paddleocr-cpu"
        image: kairntech/paddleocr-structure-cpu:3.7.0-ocrv6
        volumes:
            # models (~1 GB) downloaded at the first start
            - paddleocr-cpu-models:/root/.paddlex
        # Paddle takes every core: keep some for the rest of Sherpa
        cpus: 8
        healthcheck:
            test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')"]
            interval: 30s
            start_period: 300s
        networks:
            - sherpanet
        restart: unless-stopped
```

and under the top-level `volumes:`:

```yaml
    paddleocr-cpu-models:
        driver: local
        name: sherpa_paddleocr-cpu-models
```

No port is published: in Sherpa, set the `base_url` of the converter to
`http://sherpa-paddleocr-cpu:8080`. Tune `cpus` to the machine: expect roughly 10 to 25
seconds per page with 24 threads.

PP-StructureV3 wraps its tables in an HTML envelope that only recent versions of the
converter remove: with an older one, the tables show up as a single line of text in Sherpa.

## OCR service on GPU (PaddleOCR-VL)

`deploy/paddleocr-vl` runs PaddleOCR-VL with two containers: `paddleocr-vl-api`, the PaddleX
server called by the converter, and `paddleocr-vlm-server`, the vision-language model served
by vLLM. Both use the images published by PaddlePaddle on the Baidu registry: nothing to
build. Locally, `docker compose up -d` in that directory serves it on port 9093.

The tags are in `deploy/paddleocr-vl/.env`: `latest-gpu-sm120-offline` targets the Blackwell
GPUs (RTX 50xx) and embeds the models, so no Internet access is needed at the first start.
For another GPU, pick the matching suffix (`latest-nvidia-gpu-offline`, …) among the tags of
`ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/paddleocr-vl`.

### Add it to the Sherpa docker-compose

The PaddleX server finds the model server through the URL in its pipeline configuration.
Copy that configuration next to the Sherpa `docker-compose.yml`, pointing to the container
name used in Sherpa:

```bash
mkdir -p sherpa-paddleocr-vl
sed 's#http://paddleocr-vlm-server:8080#http://sherpa-paddleocr-vlm-server:8080#' \
    <this repository>/deploy/paddleocr-vl/pipeline_config_vllm.yaml \
    > sherpa-paddleocr-vl/pipeline_config_vllm.yaml
```

Then, under `services:`, at the indentation of the other services:

```yaml
    sherpa-paddleocr-vl-api:
        container_name: "sherpa-paddleocr-vl-api"
        image: ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/paddleocr-vl:latest-gpu-sm120-offline
        user: root
        command: paddlex --serve --pipeline /home/paddleocr/pipeline_config_vllm.yaml
        volumes:
            - ./sherpa-paddleocr-vl/pipeline_config_vllm.yaml:/home/paddleocr/pipeline_config_vllm.yaml
        depends_on:
            sherpa-paddleocr-vlm-server:
                condition: service_healthy
        deploy:
            resources:
                reservations:
                    devices:
                        - driver: nvidia
                          device_ids: ["0"]
                          capabilities: [gpu]
        healthcheck:
            test: ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]
        networks:
            - sherpanet
        restart: unless-stopped
    sherpa-paddleocr-vlm-server:
        container_name: "sherpa-paddleocr-vlm-server"
        image: ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/paddleocr-genai-vllm-server:latest-gpu-sm120-offline
        user: root
        deploy:
            resources:
                reservations:
                    devices:
                        - driver: nvidia
                          device_ids: ["0"]
                          capabilities: [gpu]
        healthcheck:
            test: ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]
            start_period: 300s
        networks:
            - sherpanet
        restart: unless-stopped
```

No port is published: in Sherpa, set the `base_url` of the converter to
`http://sherpa-paddleocr-vl-api:8080`. `device_ids` selects the GPU; vLLM reserves most of
its memory, so do not share it with another GPU service of Sherpa.

## Development

The build is driven by [Task](https://taskfile.dev) and [uv](https://docs.astral.sh/uv/),
with the shared stages coming from the `python-archetype` submodule.

### Getting started

The stages live in a Git submodule, so **clone with `--recurse-submodules`**:

```bash
git clone --recurse-submodules git@bitbucket.org:kairntech/pyconverters_paddleocr.git
cd pyconverters_paddleocr
sh -c "$(curl -sSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
task
```

Already cloned without it? The submodule directory is empty, and `task` fails on:

```
task: No Taskfile found at ".../submodules/python-archetype/resources/Taskfile.yml"
```

which means exactly that, and nothing worse:

```bash
git submodule update --init
```

**Task is the only manual prerequisite.** An archetype cannot bootstrap itself: uv and the
Python interpreter install themselves on demand (every task that runs uv depends on an
internal `install-python` task), but the thing that runs them does not. Make sure
`~/.local/bin` is on your `PATH` — that is where `task` and `uv` both land.

### Running the pipeline

```bash
task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- py:lint # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs
```

`task` with no argument is safe by construction: it runs every stage but the last, and that
bound is computed from the `STAGES` list rather than written down. The last stage is the
only one with an effect outside your machine.

`STAGES`, declared once in `Taskfile.yml`, is the single definition of the pipeline order —
so what you run locally is what Jenkins runs.

### Individual stages

| Task                            | Description                                           |
|---------------------------------|-------------------------------------------------------|
| `task py:sync`                  | Install the project and its dependencies (uv sync)    |
| `task py:lint`                  | `ruff check` and `ruff format --check`                |
| `task py:format`                | Reformat the code with ruff                           |
| `task py:test`                  | Run the test suite                                    |
| `task py:test-marker -- <m>`    | Run the tests carrying one pytest marker              |
| `task py:sbom`                  | Generate a CycloneDX SBOM of the resolved environment |
| `task py:check-vulnerabilities` | Check for known CVEs                                  |
| `task py:check-updates`         | Check for dependency updates                          |
| `task py:build`                 | Build the wheel and sdist (uv build)                  |
| `task py:publish`               | Publish the distributions (uv publish)                |
| `task py:version-file`          | Print the path of the file carrying `__version__`     |
| `task py:set-version VERSION=x` | Write that version into it                            |

`uv.lock` is not versioned here, so `py:sync` always resolves from scratch (`--upgrade`):
a stale lock lying around on a machine would otherwise make you test and audit versions the
CI never sees.

### SBOM & vulnerability check

`task py:sbom` and `task py:check-vulnerabilities` wrap the underlying tools. To run them
by hand:

```bash
uv sync --extra test --extra sbom
uv run cyclonedx-py environment -o sbom.cdx.json --output-format json
uv run pip-audit --skip-editable --format json --output audit-report.json
```

Ask for **all** the extras, not just `sbom`: `uv sync` synchronises rather than adds, so
whatever the requested extras do not pull in gets removed. `--extra sbom` on its own
uninstalls `ruff` and `pytest`, leaving an environment that can no longer lint or test.

`--skip-editable` skips the project itself, which no advisory database can know about. The
tasks do the same, and pip-audit still exits non-zero when it finds a real vulnerability —
which is the contract the CI relies on. Avoid `--strict`: it turns "dependency not found on
PyPI" into a fatal error, so it breaks as soon as the local version is not a published one.
