Metadata-Version: 2.4
Name: estravon-backend
Version: 0.2.2
Summary: Self-hosted PDF extraction backend for the Estravon Zotero plugin
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: python-fasthtml<1.0,>=0.12.0
Requires-Dist: replicate<2.0,>=0.34.0
Requires-Dist: httpx<1.0,>=0.27.0
Requires-Dist: python-multipart<1.0,>=0.0.9
Requires-Dist: python-dotenv<2.0,>=1.0.0
Requires-Dist: pypdf[cryptography]>=4.0
Requires-Dist: mistralai<3.0,>=2.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Provides-Extra: nlp
Requires-Dist: spacy>=3.7; extra == "nlp"
Provides-Extra: mineru
Requires-Dist: mineru[pipeline]<4.0,>=3.0; extra == "mineru"
Requires-Dist: psutil>=5.9; extra == "mineru"
Dynamic: license-file

# estravon-backend

Self-hosted PDF extraction backend for the
[Estravon Zotero plugin](https://github.com/tiberavonltd/estravon-plugin).

> **Independent project.**
> Estravon is not affiliated with, endorsed by, or in any way connected to the
> [Zotero project](https://www.zotero.org/) or the Corporation for Digital Scholarship.
> Zotero is a registered trademark of the Corporation for Digital Scholarship.

---

Extracts nominated sections of a book PDF to Markdown and attaches the result
directly to the Zotero item — synced, versioned, always co-located with the source.

**Just want to run it?** Skip this page — follow the step-by-step guide at
[estravon.com/install](https://estravon.com/install) instead. It covers `pip install`,
virtual environments, and `.env` configuration without requiring a clone.

This README is for people who want to read the source, modify the backend,
or run in editable mode.

---

## Developer setup

```bash
git clone https://github.com/tiberavonltd/estravon-backend.git
cd estravon-backend

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

pip install -e ".[dev]"
```

Create a `.env` file in the repo root and add your API key:

```
MISTRAL_API_KEY=your_key_here
```

Get a key at [console.mistral.ai](https://console.mistral.ai/) (~$0.002/page).

Start the backend:

```bash
estravon --port 7766
```

Run the test suite:

```bash
pytest
```

---

## Supported extraction backends

| Backend | Pricing | `.env` config |
|---|---|---|
| [Mistral OCR](https://console.mistral.ai/) | ~$0.002/page (Jul.'26), pay-as-you-go, see pricing at mistral | `MISTRAL_API_KEY=...` (default) |
| [Datalab](https://www.datalab.to/) | $25/month (Jul.'26), subscription,  see pricing at datalab | `DATALAB_API_KEY=...` + `_ZM_BACKEND=datalab` |
| [Replicate](https://replicate.com/) | a datalab model is available, Pay-as-you-go | `REPLICATE_API_TOKEN=...` + `_ZM_BACKEND=replicate` |
| [MinerU](https://github.com/opendatalab/MinerU) | Free — runs locally, no API key, no per-page cost | none — `_ZM_BACKEND=mineru` (see below) |

Only one token needs to be set. If the selected backend's token is missing,
`get_backend()` automatically falls back to whichever of the other two *is*
configured (fallback order: Mistral → Replicate → Datalab, Replicate →
Mistral → Datalab, Datalab → Replicate → Mistral) rather than refusing to
start — a warning is printed to stdout when this happens. **MinerU is never
part of this fallback chain** — local inference is much slower than a hosted
API, so it only runs when explicitly selected.

### Running fully offline with MinerU (no API key)

MinerU processes the PDF entirely on your own machine — nothing is uploaded
anywhere. It's not installed by default: `mineru[core]` pulls in a real ML
stack (PyTorch, onnxruntime, ~5.5 GB installed), so it lives behind an
optional extra.

You don't need a clone of this repo to use it — a plain virtual environment
is enough. From any empty folder (PowerShell on Windows, or a terminal on
macOS/Linux):

```powershell
# Windows (PowerShell)
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install "estravon-backend[mineru]"
estravon --port 7766 --backend mineru
```

```bash
# macOS / Linux
python3 -m venv .venv
source .venv/bin/activate
pip install "estravon-backend[mineru]"
estravon --port 7766 --backend mineru
```

That's the whole sequence — no `.env` file, no API key, no repo checkout.
Leave that terminal window open; it's the running server the plugin talks to
on `localhost:7766`. First run downloads MinerU's model weights (~1 GB,
cached under `~/.cache/huggingface` on macOS/Linux, or
`C:\Users\<you>\.cache\huggingface` on Windows, so later runs skip the
download).

**Known limitations (CPU `pipeline` mode):**
- Slow — roughly 12 seconds per page on a modern CPU, versus seconds for a
  hosted API. Fine for a handful of chapters, not ideal for whole books.
- Formula rendering has a known defect: some LaTeX output has extra spacing
  between characters (e.g. `\mathrm{s i n}` instead of `\mathrm{sin}`),
  which is visibly wrong for text-mode math. Plain text and table extraction
  are not affected by this.
- Needs a machine with a few GB of free RAM (and, on Linux, swap configured
  is recommended — the pre-flight check in `MinerUBackend` estimates the
  requirement and refuses to start rather than risking an out-of-memory
  crash, but the estimate is a rough one, not a guarantee).

---

## Architecture

```
Zotero plugin  →  POST /process  →  process_section()
                                          ↓
              MistralBackend | DatalabBackend | ReplicateBackend | MinerUBackend
                                          ↓
                                 result .md + images returned
```

The backend is a single-process [FastHTML](https://fastht.ml) server, pinned to one engine
per running instance. `/process` is **synchronous** — one job runs at a time and the HTTP
response doesn't return until it's done or failed; a concurrent call gets `409`, there is no
job queue or `GET /jobs/{id}` polling endpoint on this server (that pattern exists on
Estravon's separate hosted SaaS deployment, not in this package).
`GET /status` exposes the current server state (`idle` / `running` / `error`).

**Full reference, including the engine-decision table, the two-step Markdown-fetch gotcha,
honest per-engine behaviour differences, and the engine-implementer contract:
[`docs/API.md`](docs/API.md)** (+ machine-readable [`docs/openapi.json`](docs/openapi.json)).
The table below is a quick-reference summary, not the canonical source.

---

## API reference

| Route | Method | Purpose |
|---|---|---|
| `/ping` | GET | Liveness check — returns `{"status":"ok","state":...,"backend":...}` |
| `/status` | GET | Current server state (`idle`/`running`/`error`), time in that state, and the last completed job's summary |
| `/schema-registry` | GET | Serves `schema_registry.json` so downstream tooling can introspect the extraction output format and `SCHEMA_VERSION` |
| `/process` | POST | Runs one extraction (see below). Rejects a second request with `409` while a job is already running — one job at a time |
| `/files/{job_id}/{filename}` | GET | Downloads a result `.md` or image file from a completed job |

`POST /process` accepts `multipart/form-data` with:

| Field | Required | Notes |
|---|---|---|
| `section_name` | yes | Human-readable label, slugified into the output filename |
| `page_range` | yes | 1-based inclusive range, e.g. `"14-93"` |
| `pdf_file` **or** `pdf_path` | yes (one of) | Upload the PDF as bytes, or point at a file already on disk — the latter is useful for scripting/agent use without a network round trip |
| `chunk_size` | no (default `80`) | Pages per API call — see Chunking below |
| `mode` | no (default `balanced`) | `fast` / `balanced` / `accurate` — controls both extraction quality and how much content-statistics metadata is computed (see below). Only Replicate and Datalab actually change behaviour on this; Mistral and MinerU accept and ignore it |
| `force_ocr` | no (default `false`) | Discards the PDF's existing text layer and re-OCRs from scratch. Useful for patents and scans with a broken/garbled embedded text layer. Ignored by Mistral OCR (always OCR-native — there's no "re-OCR" to force) |
| `source_item_key` / `page_offset` | no | Passed through to the traceability footer; `page_offset` lets a caller record the *original* page numbers when it has already trimmed the PDF before sending it |

⚠️ **The response's `files[].md_url` is a path to fetch, not the Markdown text itself** — see
[`docs/API.md`](docs/API.md#-the-markdown-text-is-not-in-the-process-response--two-step-fetch-required)
for the full two-step-fetch explanation.

---

## Chunking

Sections longer than `chunk_size` pages are split automatically
(`compute_chunks()` in `chunking.py`) into multiple labelled sub-ranges —
e.g. a 200-page section at `chunk_size=80` becomes three chunks,
`chapter_01_a.md` (pages 1–80), `chapter_01_b.md` (81–160),
`chapter_01_c.md` (161–200). If the whole section fits in one chunk, the
output file has no suffix at all (`chapter_01.md`). Section names are
slugified before use as filenames, so spaces and punctuation in the name
you type never break file retrieval.

---

## Content statistics

Every extraction embeds a `content_stats` block in the `.md` footer (and in
`state.json`), computed at a tier driven by `mode`:

| `mode` | Tier | What's computed |
|---|---|---|
| `fast` | `basic` | Word/sentence/paragraph counts, structure counts |
| `balanced` (default) | `vocab` | Basic + a vocabulary profile (keyword extraction, type-token ratio) |
| `accurate` | `full` | Vocab + named-entity extraction (requires `spaCy`; silently skipped if not installed) |

This is metadata about the extracted text, not the extraction itself — it's
there so downstream tooling (search, agents, dashboards) can reason about a
section without re-parsing the whole markdown file.

---

## Large PDF handling

`MistralBackend` and `DatalabBackend` both pre-split an oversized source PDF
*before* uploading, rather than sending the whole file and letting the API
reject or silently mishandle it:

| Backend | Threshold | What happens above it |
|---|---|---|
| Mistral | 35 MB | Requested page range is split out locally (`split_pdf_pages()`, pure-Python via `pypdf`) and only that chunk is uploaded |
| Datalab | 150 MB | Same pre-split, uploaded via multipart instead of relying on Datalab's native `page_range` param |

This is defense-in-depth — the Zotero plugin already trims client-side
before sending anything, but the same backend is also usable standalone
(scripts, other clients) where that isn't guaranteed.

---

## Health check

```bash
curl http://localhost:7766/ping
# {"status":"ok","state":"idle","backend":"mistral"}

curl http://localhost:7766/status
# {"state":"idle","state_since_s":4.1,"backend":"mistral","last_job":{}}
```

---

## Links

- [Plugin repository](https://github.com/tiberavonltd/estravon-plugin)
- [End-user install guide](https://estravon.com/install)
- [estravon.com](https://estravon.com)
- [Report an issue](https://github.com/tiberavonltd/estravon-backend/issues)

---

## License

[AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.html) — the same license as Zotero itself.
