raw2md
Copyright (c) the raw2md authors
License: GPL-3.0-or-later (see LICENSE)

The package `raw2md_engine` in this distribution is a modified copy of the
software below. Datalab wrote the original packages. Datalab does not
endorse or support raw2md.

Original packages
-----------------

marker-pdf (package `marker`)
  Author:     Datalab
  Repository: https://github.com/datalab-to/marker
  Tag:        v1.10.2
  Commit:     5a41cdbac6232a10aaf4fad29f8e6c1a0c2a0808
  License:    GPL-3.0-or-later
  Copy:       src/raw2md_engine/marker/ (license text in LICENSE there)

surya-ocr (package `surya`)
  Author:     Datalab
  Repository: https://github.com/datalab-to/surya
  Tag:        v0.17.1
  Commit:     411fb1895a02ed629928d5ae59df96f44ac9dc41
  License:    GPL-3.0-or-later
  Copy:       src/raw2md_engine/surya/ (license text in LICENSE there)

pdftext
  Author:     Datalab
  Repository: https://github.com/datalab-to/pdftext
  Tag:        v0.6.3
  Commit:     d76a57535d33e9467fd92c5d3c42d80d8c84f686
  License:    Apache-2.0
  Copy:       src/raw2md_engine/pdftext/ (license text in LICENSE there)

Changes
-------

The copy differs from the original packages in the ways below. The history
of the repository holds each change with its diff.

- The import paths `marker.`, `surya.`, and `pdftext.` are rewritten to
  `raw2md_engine.marker.`, `raw2md_engine.surya.`, and
  `raw2md_engine.pdftext.`.
- The parts that raw2md does not call are removed: the LLM pass, the
  extraction converter, the config parser, the input formats other than
  PDF, the output formats other than Markdown, and the command-line and
  browser applications.
- The PDF converter builds its provider and its Markdown renderer
  directly.
- The settings are fixed values. They do not load from environment
  variables or from a `local.env` file.
- The copy downloads nothing: no font when the converter starts and no
  model weights. `marker/models.py` takes the folder of the weights from
  its caller.
- The models build their non-persistent buffers on the CPU, and the
  loaders build them again after the load of the weights, before the move
  to the device (2026-10-19). Changed files, under
  src/raw2md_engine/surya/: `common/load.py`, `common/adetr/decoder.py`,
  `common/surya/decoder/__init__.py`, `common/surya/encoder/__init__.py`,
  `ocr_error/model/encoder.py`, `foundation/loader.py`,
  `table_rec/loader.py`, and `ocr_error/loader.py`.
- The copy follows the API of transformers 5 (2026-10-19): the tied
  weights as a mapping, `post_init` in every model, the default rotary
  frequencies and the head mask in the copy itself, the tokenizer helpers
  from their new module, and no head pruning or ONNX configuration.
  Changed files, under src/raw2md_engine/surya/: `common/pretrained.py`,
  `common/adetr/decoder.py`, `common/donut/encoder.py`,
  `common/surya/__init__.py`, `common/surya/decoder/__init__.py`,
  `common/surya/decoder/config.py`, `ocr_error/model/config.py`,
  `ocr_error/model/encoder.py`, `ocr_error/tokenizer.py`,
  `table_rec/model/encoder.py`, and `table_rec/model/encoderdecoder.py`.
- The copy fixes defects of its text extraction and its Markdown output,
  drops code that raw2md never runs, and follows the formatting and lint
  rules of raw2md (2026-10-24). Changed files, under src/raw2md_engine/:
  - Text: `pdftext/pdf/pages.py` ignores the blank spans of a line when it
    marks a superscript or a subscript. `marker/schema/text/line.py` keeps
    a dash at the end of a line. `marker/schema/blocks/listitem.py` removes
    only the leading bullet of a list item.
  - Links: `pdftext/pdf/links.py` drops a link whose target is inside the
    document. The page references, the reference processor and block, and
    the span anchors go with it, in `pdftext/schema.py`,
    `pdftext/extraction.py`, `marker/builders/document.py`,
    `marker/providers/`, `marker/schema/`, and
    `marker/renderers/markdown.py`.
  - Tables: `pdftext/tables.py` measures the gap between characters along
    the line and keeps the character boxes of a table run.
    `marker/processors/table.py` splits a run of numbers at the cell
    boundaries and removes dot leaders only outside the values.
    `marker/renderers/markdown.py` escapes a vertical bar in a cell and
    writes one separator line for a table of one row.
  - Formats and math: `marker/schema/text/span.py` applies every format of
    a span, with math innermost and bold outermost, and keeps superscript
    markup out of math. `marker/renderers/__init__.py` keeps the minus when
    two inline math spans merge. `marker/renderers/markdown.py` keeps the
    escaped space at the end of math and removes bold from a heading.
  - Layout: `marker/processors/sectionheader.py` sorts the heading heights
    together with their cluster labels, gives one level to neighboring
    clusters of close mean heights, and keeps headers without lines out of
    the clusters. `marker/processors/ignoretext.py`
    counts the repeats of a running header on the even and the odd pages
    apart.
  - Dead code: `marker/converters/pdf.py` drops the line merge, blank page,
    and block relabel processors, and their files go.
  - Imports: `marker/settings.py` imports torch only when the device is
    read. `surya/common/surya/processor/tokenizer.py` imports no torch and
    builds without a special token file.
  - Style: ruff formats every Python file of the copy and applies its safe
    fixes. `surya/detection/heatmap.py` takes its result class from
    `surya/detection/schema.py`, which breaks an import cycle.
- The Markdown renderer returns the HTML that it converts to Markdown
  (2026-11-01). Changed file, under src/raw2md_engine/:
  `marker/renderers/markdown.py`.
- The PDF converter names each of its steps to an optional phase callback
  before the step runs (2026-11-01). Changed file, under
  src/raw2md_engine/: `marker/converters/pdf.py`.

Each changed file of `pdftext/` carries a notice at its top.

Last change: 2026-11-01.
