Metadata-Version: 2.4
Name: mag-parse
Version: 0.2.0
Summary: One parse() for every document - PDFs via mag-pdf, everything else via mag-file-handler
Author-email: Magure <aman.p@magureinc.com>
Maintainer-email: Magure <aman.p@magureinc.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/magurelabs/magoneai-file-handler
Project-URL: Source, https://github.com/magurelabs/magoneai-file-handler
Project-URL: Issues, https://github.com/magurelabs/magoneai-file-handler/issues
Keywords: pdf,docx,xlsx,ocr,extraction,markdown,parsing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: pdf
Requires-Dist: mag-pdf<1,>=0.3; extra == "pdf"
Provides-Extra: tables
Requires-Dist: mag-pdf[tables]<1,>=0.3; extra == "tables"
Provides-Extra: office
Requires-Dist: mag-file-handler<1,>=0.3; extra == "office"
Provides-Extra: all
Requires-Dist: mag-parse[office,pdf]; extra == "all"
Provides-Extra: everything
Requires-Dist: mag-parse[office,tables]; extra == "everything"
Provides-Extra: serve
Requires-Dist: grpcio<2,>=1.60; extra == "serve"
Requires-Dist: grpcio-health-checking<2,>=1.60; extra == "serve"
Requires-Dist: grpcio-reflection<2,>=1.60; extra == "serve"
Requires-Dist: protobuf<7,>=4.25; extra == "serve"
Provides-Extra: serve-test
Requires-Dist: mag-parse[serve]; extra == "serve-test"
Requires-Dist: pytest>=7; extra == "serve-test"
Requires-Dist: pytest-timeout>=2.1; extra == "serve-test"
Provides-Extra: proto
Requires-Dist: grpcio-tools==1.62.3; extra == "proto"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: pytest-timeout>=2.1; extra == "test"

# mag-parse

One `parse()` for every document. PDFs go to [`mag-pdf`](../magpdf);
everything else goes to [`mag-file-handler`](../file_handler).

```python
from mag_parse import parse

r = parse("invoice.pdf")                      # quality="fast" (default)
r = parse("report.pdf", quality="accurate")   # table structure
r = parse("deck.pptx")                        # quality does not apply
r = parse(pdf_bytes, filename="report.pdf")   # bytes work too

r.markdown        # always populated
r.text            # always populated
r.format          # "pdf" | "docx" | "xlsx" | "eml" | ...
r.engine          # which engine actually ran
r.ok, r.error, r.warnings
```

The facade itself has **zero required dependencies**. Both engines are extras,
and that is not tidiness — see [Install](#install).

## Routing

Decided by **magic bytes, never the extension**. A `.docx` renamed to `.pdf`
does not reach the PDF engine, and a PDF with no extension does.

```
parse(file, quality=...)
        │
   sniff %PDF
        │
   ┌────┴─────┐
 PDF        not PDF
   │            │
mag-pdf   mag-file-handler
   │            │
   └────┬───────┘
        ▼
    one Result
```

## `quality` is a PDF-only knob

| | What runs | Install |
|---|---|---|
| `"fast"` (default) | LiteParse behind an OCR gate; OCRs only pages that need it | `mag-parse[pdf]` |
| `"accurate"` | DocLayout-YOLO + TableFormer for real table structure | `mag-parse[tables]` |

For every other format `quality` is **accepted and ignored**, so a caller never
has to branch on file type. `Result.quality` is `""` there — the knob did not
default, it does not exist.

> `"accurate"` selects mag-pdf's `profile="tables"`, leaving `table_mode` at
> its default. Those are two different fast/accurate axes: the profile picks
> the *pipeline*, `table_mode` picks TableFormer's *weights*. The heavier
> weights measured 27.6 s against 16.1 s for one additional table row, so they
> are not what "accurate" buys — the table pipeline is.

## Markdown, and how honest it is

Both `.markdown` and `.text` are always populated, on every path including
failure. They are not equally structured, and `structured_markdown` says which
you got:

| Source | `.markdown` | `structured_markdown` |
|---|---|---|
| PDF | real GFM — pipe tables, headings | `True` |
| md / txt / csv | the text, which is already faithful | `False` |
| docx / xlsx / pptx / html / eml | the text, tables flattened | `False` + a warning |

A docx table currently arrives as tab-separated fragments rather than a pipe
table. That is `mag-file-handler`'s plain-text output passed through verbatim,
and the warning says so rather than letting it pass as structured markdown.
Teaching the engines real markdown per format is the planned follow-up; the
API does not change when it lands.

## Errors

A **bad document is a Result**, never an exception — check `.ok` / `.error`.
Only two things raise, and both are the caller's to fix:

| Exception | Means |
|---|---|
| `UnknownQuality` | `quality=` was not `"fast"` or `"accurate"`. Raised *before* any I/O, so a typo cannot run the wrong pipeline on a large file first. |
| `EngineNotAvailable` | The extra for this file type is not installed. Names the exact `pip install`. |

`Result.status` is `"ok"`, `"empty"` or `"error"`. **`empty` is a failure** — an
extraction that returns nothing while reporting success is the silent failure
both engines guard against, and the facade does not launder it.

## gRPC server

For running the facade as a sidecar. One RPC, because there is one `parse()`.

```bash
pip install 'mag-parse[all,serve]'
mag-parse-serve --host 0.0.0.0 --port 50051
```

```bash
# reflection is on, so no .proto needed to poke it
grpcurl -plaintext localhost:50051 list

# `content` is bytes, so JSON carries it base64-encoded
grpcurl -plaintext -d "{\"content\":\"$(base64 -w0 invoice.pdf)\",\"quality\":\"accurate\",\"filename\":\"invoice.pdf\"}" localhost:50051 mag_parse.v1.Parser/Parse

grpcurl -plaintext localhost:50051 mag_parse.v1.Parser/GetStatus
grpc_health_probe -addr localhost:50051
```

| | |
|---|---|
| `mag_parse.v1.Parser/Parse` | `content`, `quality`, `password`, `filename` -> a `ParseReply` |
| `mag_parse.v1.Parser/GetStatus` | per-engine readiness, for a human |
| `grpc.health.v1.Health/Check` | the standard probe, for a machine |

The schema is [`v1/parse.proto`](src/mag_parse/v1/parse.proto), which ships
inside the wheel so a client in another language does not have to chase a
GitHub URL for the version it happens to have. `ParseReply` carries the whole
`Result`: `markdown`, `text`, `status`, `ok`, `error`, `format`, `engine`,
`quality`, `structured_markdown`, `page_count`, `warnings`, and `extra` as a
`google.protobuf.Struct`. `pages` is absent, because those are engine-specific
page objects with no serialisation contract.

Five behaviours worth knowing before you deploy it:

- **A bad document is an OK response with `ok = false`.** An encrypted PDF is a
  fact about the document, not an outage; only a bad *server* returns a non-OK
  status.
- **A missing engine is not a broken server.** Both engines are extras, so a
  `[pdf]`-only install fails office calls `FAILED_PRECONDITION` with the exact
  pip command while the health service still says `SERVING`. What does turn it
  `NOT_SERVING` is an engine that is installed and failed to warm - for
  `accurate`, a missing model does not raise, it silently degrades to plain
  text-layer markdown.
- **`accurate` runs one document at a time** (mag-pdf holds a process-scope
  lock; pypdfium2 is not thread-safe). Run **one** process and scale with
  replicas - a second server in the same container loads its own copy of the
  models for no extra throughput. A full lane returns `RESOURCE_EXHAUSTED`
  after `MAGPARSE_QUEUE_TIMEOUT_SECONDS` rather than holding the stream open.
- **Raise your client's receive limit.** The server accepts up to
  `MAGPARSE_MAX_MESSAGE_BYTES` and sends replies of any size, but a stock gRPC
  *client* still refuses anything over 4 MB - which a large document's markdown
  will exceed. Pass `("grpc.max_receive_message_length", -1)` in the channel
  options.
- **No vision.** There is no way to hand an LLM client across a wire, so the
  server is offline by construction and images come back as an error `Result`.

Server-level knobs only, from the environment: `MAGPARSE_FAST_CONCURRENCY` (2),
`MAGPARSE_OFFICE_CONCURRENCY` (4), `MAGPARSE_MAX_MESSAGE_BYTES` (200 MB),
`MAGPARSE_QUEUE_TIMEOUT_SECONDS` (30), `MAGPARSE_SHUTDOWN_GRACE_SECONDS` (30).
Engine knobs are deliberately absent from both the environment and the wire:
`parse()` hides `num_workers`, `device` and `table_mode`, and exposing them
would mean bypassing the facade.

### Timing

Latency is reported in the **logs**, not on the wire, in the same shape the
rest of the platform logs it - an event name, then `key=value`, with
`duration_ms` spelled the way every Temporal activity spells it:

```
parse_complete name=invoice.pdf lane=accurate format=pdf engine=mag-pdf/0.3.0 (tables)
  status=ok ok=True pages=12 chars=48311 bytes=2203114 queue_ms=0.1 duration_ms=27604.3
parse_failed   name=broken.pdf lane=fast bytes=914 queue_ms=0.0 duration_ms=12.4 error=RuntimeError
warmup_complete lane=accurate ready=True duration_ms=4182.9
```

`queue_ms` is separate from `duration_ms` on purpose. Rolled together, a
saturated sidecar reads as a slow engine - and the obvious response to that
reading, tuning the engine, is the wrong one. A caller that wants its own
number measures the round trip, which is the one that includes the transport.

### Container

One image, all three routes, no network at runtime. Built from the **repo
root**, because it installs all three packages from this tree rather than from
PyPI:

```bash
docker build -f mag_parse/Dockerfile -t magureai/mag-parse:0.1.0 .
docker run --rm -p 50051:50051 magureai/mag-parse:0.1.0
```

```yaml
  magparse:
    image: magureai/mag-parse:0.1.0
    command: ["mag-parse-serve", "--host", "0.0.0.0", "--port", "50051"]
    environment:
      MAGPARSE_FAST_CONCURRENCY: 2
      MAGPARSE_OFFICE_CONCURRENCY: 4
      LOG_LEVEL: INFO
      # Deliberately NOT set: HF_HUB_CACHE / HF_HOME. The image bakes both
      # (/opt/models/hub, offline). Overriding either points the loader at an
      # empty cache, and for the accurate route that failure is SILENT - the
      # document degrades to plain text-layer markdown and still reports ok.
    # torch + TableFormer + the ONNX YOLO session resident is ~1.5-2.5g, plus a
    # few hundred MB per concurrent document.
    mem_limit: 4g
```

Weights are baked and `HF_HUB_OFFLINE=1`, so `docker run --network none`
works. The image ships a `HEALTHCHECK` that speaks `grpc.health.v1` - the same
question `grpc_health_probe` asks - so compose's `condition: service_healthy`
works without adding a probe binary. **linux/amd64 only**: neither `liteparse`
nor `extractous` publishes an ARM64 wheel, so an arm64 image would lose both
the PDF and the office routes rather than one of them.

The build gates are the image's test suite. They run every route for real -
the OCR canary, TableFormer predicting, extractous on Linux, and all three
routes driven over an actual gRPC socket - so a broken image fails
`docker build` instead of quietly serving worse markdown.

The stubs under `src/mag_parse/v1/` are **committed**, because a user running
`pip install` has no protoc. After editing the .proto, regenerate them from
`mag_parse/src`:

```bash
pip install 'mag-parse[proto]'   # grpcio-tools, pinned EXACTLY
python -m grpc_tools.protoc -I . --python_out=. --pyi_out=. --grpc_python_out=. mag_parse/v1/parse.proto
```

Use that pin rather than whatever pip resolves. protoc stamps its gencode
version into the output and protobuf's rule is one-directional - the runtime
must be newer than the gencode - so a current grpcio-tools emits gencode 7.x
that refuses to import on the `protobuf<7` runtime this package supports.

## Install

```bash
pip install 'mag-parse[pdf]'        # PDFs; one dependency, Tesseract bundled
pip install 'mag-parse[office]'     # docx, xlsx, pptx, eml, html, md, txt, csv
pip install 'mag-parse[all]'        # both of the above
pip install 'mag-parse[tables]'     # adds quality="accurate" (~2 GB: torch)
pip install 'mag-parse[all,serve]'  # + the gRPC server (`mag-parse-serve`)
```

`serve` carries no engine on purpose - it is composed with whichever of the
above you need, so a PDF-only sidecar does not inherit the office stack's
platform limits.

The engines are extras rather than dependencies because **`extractous`** (under
`mag-file-handler`) publishes no Linux ARM64 wheel and declares
`requires-python <3.14`, while **`liteparse`** (under `mag-pdf`) has neither
limit. Making the office side required would impose both constraints on
PDF-only users:

| Platform | `[pdf]` | `[office]` |
|---|---|---|
| Linux x86-64 | ✅ | ✅ |
| Linux ARM64 | ✅ | ❌ no wheel |
| macOS ARM64 / x86-64 | ✅ | ✅ |
| Windows x86-64 | ✅ | ✅ |
| Python 3.14 | ✅ | ❌ `<3.14` |

Python 3.10+. Apache-2.0.
