Metadata-Version: 2.5
Name: dot-parser
Version: 3.2.3
Summary: Document-to-markdown parser and chunker for RAG pipelines
Project-URL: Homepage, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser
Project-URL: Repository, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser
Project-URL: Issues, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/issues
Author-email: Kannon For Deep Tech <louis.letarnec@deepika.ai>
License-Expression: MIT
License-File: LICENSE.md
Keywords: chunking,deepika,markdown,open-toolbox,parser,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.12
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: mammoth>=1.11
Requires-Dist: markitdown[all]>=0.1
Requires-Dist: pdfplumber>=0.11
Requires-Dist: pillow>=10.0
Requires-Dist: pydantic>=2
Requires-Dist: pypdfium2>=5
Requires-Dist: python-docx>=1.2.0
Requires-Dist: semchunk>=3.0
Requires-Dist: typing-extensions>=4.16
Provides-Extra: all
Requires-Dist: docling>=2.0; extra == 'all'
Requires-Dist: dot-inference<5,>=3.1; extra == 'all'
Requires-Dist: llama-cloud>=2.4; extra == 'all'
Requires-Dist: mistralai>=2.0; extra == 'all'
Provides-Extra: docling
Requires-Dist: docling>=2.0; extra == 'docling'
Provides-Extra: inference
Requires-Dist: dot-inference<5,>=3.1; extra == 'inference'
Provides-Extra: llama
Requires-Dist: llama-cloud>=2.4; extra == 'llama'
Provides-Extra: mistral
Requires-Dist: mistralai>=2.0; extra == 'mistral'
Description-Content-Type: text/markdown

# dot-parser

[![PyPI](https://img.shields.io/pypi/v/dot-parser)](https://pypi.org/project/dot-parser/)
![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)
[![Licence: MIT](https://img.shields.io/badge/licence-MIT-blue)](LICENSE.md)
[![Pipeline](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/badges/main/pipeline.svg)](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/pipelines)

**Turn documents into clean Markdown, then into retrieval-ready chunks.**

```python
from dot_parser import parse, chunk

markdown = parse("report.pdf")
chunks = chunk(markdown)

for c in chunks:
    print(c.section_path, len(c.content))
    # ['# Report', '## Methods', '### Analysis'] 842
```

## Why dot-parser

A RAG pipeline needs two things from a document: faithful text, and chunks that
keep their place in the document's structure. Most tools give you one or the
other — parsers stop at raw text, splitters assume you already have Markdown and
cut it blind to headings.

dot-parser does both in one step. It converts PDF, DOCX, PPTX, HTML, XLSX, CSV,
Markdown and plain text into clean Markdown, then splits that Markdown into
chunks carrying their heading hierarchy in `section_path` — so a chunk still
knows it came from *Report › Methods › Analysis*.

The default install stays light: heavy backends are optional extras, and you pick
per document whether to run locally or through a cloud OCR. See
[docs/DESIGN.md](docs/DESIGN.md) for how it compares to Docling, MarkItDown and
LangChain splitters.

## Features

- One `parse()` call for PDF, DOCX, PPTX, HTML, XLSX, CSV, Markdown and text
- Swappable PDF backends: local and fast, local and layout-aware, or cloud OCR
- Heading-aware chunking with `section_path` metadata
- Image extraction with per-image descriptions, from a VLM or from OCR annotations
- Batch PDF parsing through the Mistral Batch API (50% cheaper)
- Cost estimation before you send anything
- Light by default — heavy backends live behind extras

## Installation

```bash
pip install dot-parser

# With optional PDF backends (each ~50 MB - 1 GB):
pip install 'dot-parser[docling]'
pip install 'dot-parser[mistral]'
pip install 'dot-parser[llama]'
pip install 'dot-parser[all]'
```

Requires Python 3.12+.

## Quick start

```python
from dot_parser import parse, chunk

markdown: str = parse("report.pdf")
chunks: list[Chunk] = chunk(markdown)

for c in chunks:
    print(c.section_path, c.heading, len(c.content))
    # ["# Report", "## Methods", "### Analysis"], "### Analysis", 842
```

For a different PDF backend:

```python
from dot_parser import parse, parse_pdfs, Mistral, Docling

# single PDF, cloud OCR
md = parse("scanned.pdf", backend=Mistral())

# many PDFs, batched (50% off via Mistral Batch API)
mds = parse_pdfs(["a.pdf", "b.pdf", "c.pdf"], backend=Mistral())

# local layout-aware ML pipeline
md = parse("paper.pdf", backend=Docling())
```

## API

### `parse(source, format=None, backend=None) -> str`

Converts a document to Markdown.

- **source** -- file path (`str` or `Path`) or raw `bytes`
- **format** -- explicit format hint (e.g. `"pdf"`, `"docx"`). Required when source is `bytes`, otherwise inferred from the file extension.
- **backend** -- optional PDF backend (`Plumber`, `Docling`, `Mistral`, `Llama`). Defaults to `Plumber()`. Backends only apply to PDF input.

Supported formats: `.pdf`, `.md`, `.txt`, `.docx`, `.pptx`, `.html`, `.xhtml`, `.htm`, `.xlsx`, `.csv`

Raises `ParseError` on unsupported formats or conversion failures.

### `parse_pdfs(sources, backend=None) -> list[str | None]`

Converts a list of PDFs to Markdown, in input order. Backends that implement a `parse_pdfs` method (e.g. `Mistral` via the Batch API, 50% off) handle the whole list in a single optimized call. Otherwise falls back to looping `parse()` per source. Failed items are returned as `None` (no exception raised).

### `parse_with_images(source, format=None, *, backend=None, vlm=None, annotate_images=True, include_tables=True) -> ParseResult`

Converts a `.pdf`, `.docx` or `.pptx` and extracts its images alongside the
Markdown, keeping `![name](name)` anchors at each image's position. Each image is
described either by a VLM or by OCR annotations from the same call. See
[docs/IMAGE_EXTRACTION.md](docs/IMAGE_EXTRACTION.md) for the available paths per format.

Everything runs locally when you pass your own models: the `VLM`,
`PageTranscriber` and `ZoneTranscriber` protocols each take one method, so any
model fits (a local server such as Ollama or vLLM, or a cloud API).
`dot_parser.inference` implements them on a [dot-inference](https://pypi.org/project/dot-inference/)
client (`pip install 'dot-parser[inference]'`):

```python
from dot_parser import Plumber, parse_with_images
from dot_parser.inference import InferenceReader, InferenceVLM, local_client

result = parse_with_images(
    "spec.pdf",
    backend=Plumber(
        ocr=InferenceReader(local_client("http://gpu-host:8000/v1", "ocr-model")),
        ocr_scope="zone",  # the scanned pages, and the passages the text can't carry
        max_concurrency=8,
    ),
    vlm=InferenceVLM(local_client("http://gpu-host:8001/v1", "vision-model")),
    max_concurrency=8,  # images described at once
)
```

`ocr_scope` says what the `ocr` model reads, the rest being read from the
PDF's text layer: `"unreadable"` (default) the scanned and unreadable pages;
`"page"` those, and each page with a passage its text layer can't carry (an
equation in glyphs without Unicode mapping, a fraction, a block in a broken
font), whole; `"zone"` those passages alone, rendered and read one by one;
`"all"` every page, the text layer unused (images are still extracted).
An OCR model that locates its blocks (LightOnOCR with the prompt `grounding`:
`InferenceReader(client, page_prompt="grounding", zone_prompt="")`) gives each
figure its place in the text, scanned pages included: the figure is cut from
the page there, and the `vlm` describes it.

What the `ocr` model failed to read (the server down, a timeout) or had no time
for (`deadline`) is listed in `result.unread`, one `UnreadPart` per page or
passage; a page left empty says so in the Markdown (`*[Page 3 not read: ...]*`),
a passage keeps the text the PDF has.

### Backends

- **`Plumber(use_outline=True, ocr=None, ocr_scope="unreadable", max_concurrency=1, deadline=None)`** -- pdfplumber, default. Local, CPU-only. Pages set in columns are read column by column; headings come from the PDF's outline (bookmarks) when it has one, otherwise from the fonts and the section numbers; ruled tables become Markdown tables (a chart's grid is read as text); sub- and superscripts are written in `<sub>`/`<sup>`; running headers/footers and table-of-contents lines are dropped. Set `use_outline=False` to always guess headings from the fonts. A scanned page (no text layer), or an unreadable one (a font whose encoding is broken), comes back empty, with a warning, unless `ocr` is given: a `PageTranscriber` that reads the page rendered as an image (`PAGE_TRANSCRIPTION_PROMPT` is a prompt written for a VLM doing it), and with `ocr_scope` more of the document (see above); `max_concurrency` calls run at once, `deadline` bounds them; a page the model doesn't read gets a placeholder, and `ParseResult.unread` lists what it didn't. With `parse_with_images()`, images are rendered from the page as PNG; logos repeated on most pages are dropped.
- **`Docling(force_full_page_ocr=True)`** -- IBM Docling, layout-aware ML pipeline + Tesseract OCR. Robust on image-only and broken-encoding PDFs. Install with `pip install dot-parser[docling]`.
- **`Mistral(api_key=None, model="mistral-ocr-4-1")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
- **`Llama(api_key=None, tier="cost_effective")`** -- LlamaCloud parsing API. Reads `LLAMA_CLOUD_API_KEY` from env. Tiers: `fast`, `cost_effective`, `agentic`, `agentic_plus`. Install with `pip install dot-parser[llama]`.

#### Mistral OCR enrichments

Three optional extras on the `Mistral(...)` constructor, applying to the `*_with_images` methods. All default to off, so existing callers see no change:

```python
from dot_parser import parse_with_images, Mistral

result = parse_with_images(
    "spec.pdf",
    backend=Mistral(
        extract_headers_footers=True,  # page furniture -> result.pages[i].header/.footer
        confidence_scores="page",  # or "word"   -> result.pages[i].*_confidence
        extract_captions=True,  # figure captions -> image.original_caption
    ),
)

for page in result.pages:
    print(page.page, page.average_confidence, page.minimum_confidence)
```

- `extract_headers_footers` moves running headers/footers out of the markdown and into `ParseResult.pages`, so repeated "page 4 of 27" noise stays out of retrieval chunks. It **removes that text from the markdown** -- the only one of the three that changes `result.markdown`.
- `confidence_scores` surfaces OCR's self-reported quality. **Calibrate any threshold on your own corpus -- absolute cutoffs do not transfer.** Measured over the 39 pages of the benchmark corpus (`native_simple`, `image_only`, `broken_encoding`), `average_confidence` sat at 0.985-0.989 and `minimum_confidence` at 0.23-0.46 on *every* document regardless of quality. `minimum_confidence` is the single worst region on the page, so it is low even on clean pages and is not a page-quality alarm by itself.
- `extract_captions` fills `ExtractedImage.original_caption` from the figure caption printed beside each image. Needs OCR 4+ **and** `mistralai>=2.8`; on older SDKs it logs a warning and leaves captions `None` rather than failing.

If the OCR model rejects any of these parameters, the call is retried once without them -- you get the markdown and images, minus the enrichments.

#### Caption-grounded image interpretation

Mistral OCR's inline annotations (`annotate_images=True`) are produced from the cropped image alone -- the annotating model sees no caption and no page text, which on domain-specific figures yields confidently wrong descriptions. `interpret_images()` replaces that pass with client-side VLM calls grounded in each figure's own caption and surrounding markdown:

```python
from dot_parser import parse_with_images, interpret_images, Mistral, MistralVLM

result = parse_with_images(
    "paper.pdf",
    backend=Mistral(extract_captions=True),
    annotate_images=False,  # skip the ungrounded inline pass
)
result = interpret_images(result, MistralVLM())
```

Same number of VLM calls as `annotate_images=True`, but each one carries the caption. On a figure-heavy biomechanics paper this turned "magnetic field components" / "fluid flow through a curved pipe" into accurate descriptions of the actual rib diagrams.

### `chunk(markdown, max_tokens=None, merge=True) -> list[Chunk]`

Splits Markdown into chunks with heading-hierarchy metadata.

- **markdown** -- Markdown string (typically output of `parse()`)
- **max_tokens** -- maximum tokens per chunk. Auto-computed from the 95th percentile of section sizes (clamped 64-512) when `None`.
- **merge** -- when `True` (default), merges small adjacent sections sharing the same parent heading to reduce fragmentation.

The chunking algorithm is described in [docs/DESIGN.md](docs/DESIGN.md).

### `Chunk`

```python
@dataclass(frozen=True)
class Chunk:
    content: str  # full text of the chunk (heading + body)
    heading: str | None  # the heading line, if any
    section_path: list[str]  # hierarchy of headings leading to this chunk
```

### `ParseError`

Raised by `parse()` when a document cannot be converted.

### Cost estimation

`cost_per_1k_pages(backend)` and `estimate_cost(backend, pages, batch=False)`
report what a cloud backend will cost, without instantiating it or needing an
API key.

## Stability

`dot-parser` follows semantic versioning: everything exported from the top-level
package is covered, anything underscore-prefixed is internal and may change in
any release. Public names are never removed without a deprecation period.

```toml
dependencies = ["dot-parser>=3.2,<4"]
```

See [docs/VERSIONING.md](docs/VERSIONING.md) for the full policy.

## Roadmap

- [ ] Table extraction as structured data, not just Markdown
- [ ] Streaming parse for very large documents
- [ ] Additional cloud OCR backends
- [ ] Small images filtered by default for `Plumber`
- [ ] Image anchors placed from the OCR model's grounding when it gives no figure (a chart read as a table only)

## Documentation

| Document | Contents |
|---|---|
| [docs/DESIGN.md](docs/DESIGN.md) | Design rationale, backend choices, chunking algorithm |
| [docs/IMAGE_EXTRACTION.md](docs/IMAGE_EXTRACTION.md) | Image-extraction paths per format |
| [docs/PDF_READING.md](docs/PDF_READING.md) | How a PDF is read: what goes to the OCR model and to the VLM |
| [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) | Environment setup, tests, code style |
| [docs/VERSIONING.md](docs/VERSIONING.md) | Versioning, deprecation policy, how to depend on this package |
| [docs/PUBLISHING.md](docs/PUBLISHING.md) | Cutting a release |
| [CHANGELOG.md](CHANGELOG.md) | Release history |

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the
licensing terms that apply to contributions and how to submit a change.

## Licence

Copyright (C) 2026 Kannon For Deep Tech (deepika)

This software is distributed under the MIT License — see
[LICENSE.md](LICENSE.md).
