Metadata-Version: 2.4
Name: estravon-backend-benchmarks
Version: 0.2.2
Summary: Apples-to-apples PDF-to-Markdown engine comparison for estravon-backend
Author-email: tiberavonltd <info@estravon.com>
License: AGPL-3.0
Project-URL: Repository, https://github.com/tiberavonltd/estravon-backend-benchmarks
Project-URL: Documentation, https://tiberavonltd.github.io/estravon-backend-benchmarks/
Keywords: nbdev
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx
Requires-Dist: fastdownload
Provides-Extra: dev
Requires-Dist: nbdev; extra == "dev"
Dynamic: license-file

# estravon-backend-benchmarks


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

**Prerequisite:** you need `estravon-backend` installed and reachable with the
engines you want to compare configured – MinerU installed locally
(`pip install "estravon-backend[mineru]"`), and/or `MISTRAL_API_KEY` /
`DATALAB_API_KEY` / `REPLICATE_API_TOKEN` set **as environment variables in
the shell/process that calls `compare()`** – e.g. `export MISTRAL_API_KEY=...`
before running your script, or `os.environ["MISTRAL_API_KEY"] = ...` before
calling `compare()`. This package orchestrates `estravon-backend`; it never
extracts anything on its own, and it never reads or passes engine credentials
itself – see “A note on API keys” below for exactly how they reach the
engine. See `estravon-backend`’s `docs/API.md` for the engine-decision table
and how to get each engine’s key.

**Picking up the `.env` file from `estravon-backend`’s own setup:**
`estravon-backend`’s README has you create a `.env` file (e.g.
`MISTRAL_API_KEY=your_key_here`) in *that* repo’s root, and `estravon` loads
it automatically – but only from its own process’s current working
directory at startup. This package spawns `estravon` as a subprocess from
wherever *your* notebook’s kernel happens to be running, which usually isn’t
that directory – so the reliable way to make the same key visible here is
to load that `.env` explicitly, once, before calling `compare()`:

``` python
from pathlib import Path
from dotenv import load_dotenv

load_dotenv(Path("~/path/to/estravon-backend/.env").expanduser())
```

`python-dotenv` ships as a dependency of `estravon-backend` itself, so if
you’ve already installed that (this package’s own prerequisite, above),
it’s already available – no separate install. `.expanduser()` resolves `~`
portably on Linux/macOS/Windows; for a literal `$HOME`-style variable
instead, use `os.path.expandvars(...)`. Once loaded, the variables are in
your notebook process’s own environment, and `LocalEngineProcess` (Mode A)
inherits them when it spawns `estravon` – it doesn’t matter what that
subprocess’s own working directory ends up being.

## Install

``` python
!pip install estravon-backend-benchmarks
```

Also installable via git, e.g. for the latest unreleased commit on `main`, or for contributing (editing a notebook under `nbs/`):

``` sh
pip install git+https://github.com/tiberavonltd/estravon-backend-benchmarks.git
```

To contribute, clone and install in dev mode instead:

``` sh
git clone https://github.com/tiberavonltd/estravon-backend-benchmarks.git
cd estravon-backend-benchmarks
pip install -e ".[dev]"
nbdev-install-hooks   # keeps notebook diffs clean on commit
```

See `CONTRIBUTING.md` for the edit-notebook -\> `nbdev-export` -\> `nbdev-test` loop
(nbdev 3.x only ships hyphenated console scripts – `nbdev_install_hooks` with an
underscore will not be found).

## Quickstart

`compare()` has two modes (see `estravon-backend`’s `docs/API.md` section
“Engine selection” for why there’s no single-URL multi-engine mode – engine
choice is fixed per running instance, on purpose):

- **Mode A (below, the common case):** pass `engines=[...]` and this package
  spawns one pinned `estravon --backend <engine> --port <N>` subprocess per
  engine for you – nothing to configure beyond having `estravon-backend`
  installed with the relevant keys/local models available.
- **Mode B:** pass `engine_urls={"mistral": "http://host:port", ...}` instead,
  if you already have separately-running single-engine instances.

**A note on API keys – two different things share the name “api_key”:**

- **Engine credentials** (`MISTRAL_API_KEY`, `DATALAB_API_KEY`,
  `REPLICATE_API_TOKEN`) are read by `estravon-backend` itself from the
  environment – this package never touches them. In **Mode A**,
  `LocalEngineProcess` spawns `estravon --backend <engine> ...` as a plain
  subprocess that inherits your calling process’s environment, so setting
  them before you call `compare()` is enough; there’s no parameter for them
  here because there doesn’t need to be one.
- **`compare()`’s own `api_key=` parameter is a different thing entirely**:
  it’s only used in **Mode B**, forwarded as the `X-API-Key` header to
  authenticate to a *hosted* `estravon-backend` instance (the same header
  the hosted service’s own billing/auth uses) – it has nothing to do with
  which engine that instance runs. In Mode A it’s hardcoded to `None`:
  locally spawned instances have no such auth layer.

The cell below is marked non-executing in this rendered copy of the notebook
(no backend/engines are available in the environment that builds these
docs) – otherwise it’s exactly what you’d run locally, unmodified: it uses
`get_artusi()` (a real, public-domain 1891 cookbook PDF, downloaded and
cached on first call – see `estravon_bench.io`) instead of a placeholder
path, so there’s nothing to swap in before trying it.

``` python
from estravon_bench.compare import compare
from estravon_bench.io import get_artusi

result = compare(
    pdf_path=str(get_artusi()),   # public-domain 1891 cookbook -- downloaded/cached on first call
    page_range="1-4",
    engines=["mineru", "mistral"],   # whichever engines you have configured
    mode="balanced",
)
print(result.to_markdown_table())
```

This prints a side-by-side table:

    | engine | time (s) | cost (usd) | local | pages | status |
    |---|---|---|---|---|---|
    | mineru | 34.10 | free (local) | yes | 4 | ok |
    | mistral | 2.30 | $0.0080 | no | 4 | ok |

Then look at each engine’s actual output:

``` python
for r in result:
    print(f"--- {r.engine} ---")
    print(r.markdown[:500] if r.ok else f"ERROR: {r.error}")

print(result.diff("mineru", "mistral"))   # optional -- line diff for eyeballing
```

**⚠️ The two-step fetch is handled for you** – `Client.fetch_markdown()`
already does the `md_url` → actual text round trip described in
`estravon-backend`’s `docs/API.md`. If you’re extending `client.py`
yourself, that’s the detail to know about; `compare()`’s own callers never
see a bare URL.

**Honest-cost labelling:** `local=True` engines (MinerU today) show
`cost_usd=0.0` and the table renders “free (local)” – that means *zero
dollars*, not *zero effort or best value*. A free engine that’s ten times
slower is not automatically the right choice.

**Scope:** this is an evaluation aid for comparing engines on your own PDFs
apples-to-apples – not a production layer, and not a leaderboard. It
compares *your* PDF on *your* configured engines; it makes no claim about
which engine is best in general.

## Images

Engines that extract images (Datalab and Mistral, real-verified; any future engine automatically once it returns `image_urls`) have them fetched for you – `result.images` is a `{filename: bytes}` dict per engine, populated by `compare()` the same way `result.markdown` is. `to_markdown_table()`’s `images` column shows the count per engine at a glance.

No automated image comparison here – engines encode/crop/caption images differently even for “the same” figure, so this package limits itself to presence/count and letting you look:

``` python
from IPython.display import Image

row = result.get("mistral")
for filename, data in row.images.items():
    print(filename, f"{len(data)} bytes")
    display(Image(data=data))
```

## Combining results from separate `compare()` calls

Running one engine at a time (different session, different machine, resource constraints) instead of passing `engines=[...]` all at once? Merge the results back into one `ComparisonResultList` afterwards – `+` and `.merge()` both return a new list without touching the originals; `.extend()` mutates the first list in place if you’d rather accumulate into one object as you go:

``` python
from estravon_bench.compare import compare
from estravon_bench.io import get_artusi
from estravon_bench.report import ComparisonResultList

pdf = str(get_artusi())
mineru_result  = compare(pdf, "1-4", engines=["mineru"])
mistral_result = compare(pdf, "1-4", engines=["mistral"])

combined = mineru_result + mistral_result                      # new list, both inputs untouched
# equivalent, and reads better for more than two lists:
combined = ComparisonResultList.merge(mineru_result, mistral_result)

mineru_result.extend(mistral_result)   # or: mutate mineru_result in place instead

print(combined.to_markdown_table())
```

## Saving and loading a comparison run

`ComparisonResultList.save(dir_path)` writes the whole list to disk as plain files – `manifest.json` plus one subdirectory per engine holding `result.md` and an `images/` folder – not a database or a binary format, so any file browser, `git diff`, or image viewer can inspect a saved run directly. `ComparisonResultList.load(dir_path)` reads it back into real `ComparisonResult` objects, markdown/images/metadata all restored exactly:

``` python
from estravon_bench.report import ComparisonResultList

result.save("runs/mineru_vs_mistral")

# ...later, or in a different session:
reloaded = ComparisonResultList.load("runs/mineru_vs_mistral")
print(reloaded.to_markdown_table())
```

## Worked examples

Real, live-run engine comparisons – actual findings, not illustrative examples – live in a separate notebook: [`06_worked_examples.ipynb`](https://github.com/tiberavonltd/estravon-backend-benchmarks/blob/main/nbs/06_worked_examples.ipynb).
