Metadata-Version: 2.4
Name: myform
Version: 1.1.1
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: POSIX :: Linux
Classifier: Framework :: Pytest
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Dist: mdformat>=1.0,<2
Requires-Dist: mdformat-deflist>=0.1.4
Requires-Dist: mdformat-footnote>=0.1.2
Requires-Dist: mdformat-front-matters>=2.0.0
Requires-Dist: mdformat-gfm>=1.0.0
Requires-Dist: mdit-py-plugins>=0.5.0
Requires-Dist: more-itertools>=10.7.0
Requires-Dist: my-basis>=1.0.0
Requires-Dist: pydantic>=2.11.7
Requires-Dist: regex>=2024.4.28
Requires-Dist: srsly>=2.5.3
Requires-Dist: wikiparse>=1.0.2,<2 ; extra == 'wikitext'
Provides-Extra: wikitext
License-File: LICENSE
Summary: A typed translator that reads and writes MyST, GFM, Sphinx rST, and Typst.
Author-email: Robb Doering <robb@doering.ai>
License-Expression: MPL-2.0
Requires-Python: >=3.13
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://myform.docs.doering.ai/
Project-URL: Homepage, https://gitlab.com/doering-ai/libs/myform
Project-URL: Issues, https://gitlab.com/doering-ai/libs/myform/-/issues
Project-URL: Repository, https://gitlab.com/doering-ai/libs/myform

# myform

![logo](assets/logo_512.png)

> **Status:** `myform 1.0.0` is public on PyPI.
> The current development branch prepares the next release; expect breaking changes there.

![Pipeline Status](https://img.shields.io/gitlab/pipeline-status/doering-ai/libs/myform?branch=main) ![Test Coverage](https://img.shields.io/gitlab/pipeline-coverage/doering-ai/libs/myform?branch=main) [![License](https://img.shields.io/gitlab/license/doering-ai/libs/myform)](LICENSE) [![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit) [![pyrefly](https://img.shields.io/endpoint?url=https://pyrefly.org/badge.json)](https://github.com/facebook/pyrefly) [![ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

`myform` is a universal docs-format translator built around one absolute, typed document tree.
It reads MyST Markdown, GitHub Flavored Markdown (GFM), Obsidian-Flavored Markdown, Sphinx
reStructuredText, and Typst, then writes idiomatic MyST, GFM, Obsidian, Sphinx reStructuredText,
Typst, or a bounded MediaWiki wikitext subset.
Formatting is the identity case: `myform.convert(text, format, format)` runs through the same reader, typed tree, and writer as a cross-format conversion.

| Format                     | Read | Write |
| :------------------------- | :--: | :---: |
| MyST Markdown              | Yes  |  Yes  |
| GFM                        | Yes  |  Yes  |
| Obsidian-Flavored Markdown | Yes  |  Yes  |
| Sphinx reStructuredText    | Yes  |  Yes  |
| Typst                      | Yes  |  Yes  |
| MediaWiki wikitext         | Yes  |  Yes  |

The two Markdown identity conversions remain the compatibility rail.
They produce byte-for-byte the same text as the existing mdformat pipelines, including the MyST opinions developed in this repository.

A sixth registered dialect, `plane` (`myform.convert(text, src, 'plane')`), is a different kind of target: it emits and reads Plane's stored `description_html` -- an empirically reverse-engineered ProseMirror/Tiptap-schema HTML subset -- for exact wire fidelity to one vendor's storage, not idiomatic semantic HTML. It is intentionally absent from the table above.

**Documentation:** [myform.docs.doering.ai](https://myform.docs.doering.ai/)

## Install

Install the public release from PyPI:

```sh
pip install myform
```

To develop from the checked-out source with `uv`:

```sh
uv sync --all-extras
uv run myform --help
```

`myform 1.0.0` includes the native Typst reader, so Typst reads as well as writes.
That reader is backed by the `_typst_native` ABI3 extension over `typst-syntax`, and its source builds require Rust 1.92.
The release pipeline intentionally produces one audited `cp313-abi3-manylinux_2_28_x86_64` wheel; macOS, Windows, non-x86_64 Linux, and older-glibc Linux systems must build from source until a verified wheel matrix is added.
`import myform` itself remains pure Python and loads the extension only when a Typst document is parsed.

## Translate documents

```sh
uv run myform convert guide.md --from myst --to rst -o guide.rst
printf '# Hello\n' | uv run myform convert - --from gfm --to typst
uv run myform convert guide.md --from myst --to gfm --report json >guide.gfm
uv run myform convert guide.md --from myst --to wikitext -o guide.wiki
```

Use `-` as the input path to read stdin.
Output goes to stdout unless `-o/--output` names a file.
Conversion reports always go to stderr, keeping document output safe for pipelines.
Choose `--report summary` (the default), `--report json`, or `--report none`.

## Format documents

The `format` verb performs an identity conversion:

```sh
uv run myform format README.md
uv run myform format README.md --format myst -o README.formatted.md
```

When no myform config exists, a myst identity takes the mdformat-facade fast path, which honors a discovered `.mdformat.toml` exactly as the `mdformat` CLI would for the same target path: the nearest file wins on the upward walk, `number`, `wrap`, and `end_of_line` set the facade options, `[plugin.*]` tables reach their plugins, paths matched by `exclude` pass through unchanged, and invalid configuration fails loudly. A discovered `myform.yaml`, `.myform.yaml`, or `[tool.myform]` table still routes the target through the translator pipeline instead.

The Python API exposes the same contract:

```python
import myform

rst = myform.convert(source, 'myst', 'rst')
formatted = myform.convert(source, 'myst', 'myst')
wikitext = myform.convert(source, 'myst', 'wikitext')
```

Extensions that preserve unsupported nodes out of band use
`myform.convert_artifact(...)`. Its `ConversionArtifact` keeps target text, the
`ConversionReport`, and deterministic typed `EnclaveRecord` carriers together. A carrier path is
`enclaves/<full-sha256>.json`; every `ENCLOSED` report row binds its exact source occurrence to
one carrier, and the target writer must recover a final typed link to that exact path. The legacy
`convert()` and `convert_with_report()` shapes raise `ArtifactRequiredError` rather than silently
discarding a real carrier. Built-in writers do not yet claim `ENCLAVE`; each adoption requires a
target-specific marker/carrier contract and tests.

Wikitext reading uses the optional [`wikiparse`](https://gitlab.com/doering-ai/apps/wikiparse) adapter (`myform[wikitext]` extra) and requires the `pandoc` executable on `PATH`.
The adapter translates wikitext through wikiparse's document profile into Markdown, then feeds that through the existing MyST reader to produce the typed `Doc`.
Templates are extracted as standalone citation records by the upstream parser; non-citation templates are converted to text or carried.
Install the extra and pandoc to read wikitext:

```sh
pip install myform[wikitext]
```

## Configure translation

`myform` discovers the nearest `myform.yaml`, `.myform.yaml`, or `[tool.myform]` table by walking upward from the target path.
Within one directory, that order is also the precedence order.
The nearest directory wins rather than merging several files.

```yaml
degrade:
  mode: annotate
  comments: true
assist:
  enabled: false
typst:
  postformat: off
extensions: []
```

### Fold-aware tables

MyST and GFM table writers size columns from visible inline content by default. Only typically-hidden
material -- link destinations, image URLs, and MyST role targets -- is folded out of the measurement;
rendered markup such as emphasis, code, math, and HTML still counts at its serialized width. Markdown
source remains unchanged apart from padding, so pipe positions can vary when a folded cell's serialized
form is longer than its measured width.

Disable the policy per target when source-character alignment is required:

```yaml
myst:
  fold_aware_tables: false
gfm:
  fold_aware_tables: false
```

The MyST mdformat facade exposes the same default as `[plugin.myst] fold_aware_tables = true`;
`--no-fold-aware-tables` disables it for one invocation. Bare `mdformat --extensions gfm` retains
the upstream GFM formatter, while `myform.format(..., 'gfm')` uses this policy.

### Safe Setext headings

MyST, GFM, and Obsidian formatting rewrite ordinary Setext headings to ATX form by default. A
list label followed by a single nested dash is preserved as an empty nested list rather than being
mistaken for an H2 underline. Set `myst.setext_headings: false` (or use
`--no-setext-headings` in the MyST facade) to retain the stock CommonMark precedence for MyST;
the universal API accepts `options={"setext_headings": false}` for any CommonMark-family reader.

Every non-native conversion is represented in the conversion report.
The default `annotate` degradation mode also leaves a provenance comment when the target supports comments; `silent` suppresses comments but not reports, while `strict` refuses carrying, dropping, or enclosing unsupported constructs.

The optional LM assist is degradation-only and disabled by default, which makes zero network calls.
When enabled, it uses the configured local OpenAI-compatible gateway and retains a deterministic fallback.

The committed [`myform.schema.json`](myform.schema.json) and [`myst-options.schema.json`](myst-options.schema.json) schemas describe the translator and mdformat-facade configuration surfaces.
Refresh or verify both with:

```sh
uv run myform-write-schema
uv run myform-write-schema --check
```

## Use the MyST mdformat facade

The original plugin remains available through mdformat's `myst` extension entry point:

```sh
uv run mdformat README.md
```

This repository began as a ground-up rewrite of [`mdformat-myst`](https://github.com/executablebooks/mdformat-myst), written to make uncompiled MyST documents look closer to the hand-formatted sources maintained by the MyST team.
That facade remains useful on its own:

1. [Targets](https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html#creating-explicit-targets) hug the headings or blocks they identify instead of being separated by an extra blank line.

2. [Directive options](https://myst-parser.readthedocs.io/en/latest/syntax/roles-and-directives.html) prefer the readable `:key: value` prefix form and escalate to a YAML block only when necessary.

3. Directive contents are formatted as first-class Markdown rather than treated as opaque text.

4. Parser and renderer rules are explicit, decorator-registered examples of a modern mdformat extension.

5. YAML values use `srsly`'s maintained `ruamel.yaml` integration.

6. The optional `sembr` mode reflows paragraphs to [one sentence per line](https://sembr.org/) without changing their rendered output:

   ```toml
   [plugin.myst]
   sembr = true
   ```

   Enable it per invocation with `mdformat --sembr <filename>`; `--no-sembr` overrides the TOML setting.

   The splitter holds back wherever only a citation can start or continue: abbreviations (`cf.`, `nn.`, `vgl.`), labels with a number (`Prop. 17`, `art. 3`), roman numerals (`II. Def. vi`), and locators (`chap. 7 and 8`, `12:10`).
   List items keep their prose on one line, so a short bullet stays compact, unless `sembr_list_items = true` is set as well; each continuation line then sits under its item's marker. An item inside a block quote always stays on one line.

7. Ordered lists can keep their numbers.
   mdformat numbers a list from its first item alone, so `2. 3. 4.` comes back as `2. 1. 1.`, and its `number` option renumbers consecutively and zero-pads (`08. 09. 10.`).
   When the numbers carry meaning (the objections and replies of a medieval question, a list that continues across an interruption, a deliberate gap), ask for the source's own:

   ```toml
   [plugin.myst]
   list_numbering = "preserve"
   ```

   Every item then returns with the number it had, gaps and repeats included, and nested blocks indent under the width of their own marker (`9.` and `10.` differ).
   `mdformat --list-numbering preserve <filename>` does the same for one invocation; the default, `mdformat`, leaves the numbering to mdformat.

8. MyST's `attrs_block` and `amsmath` extensions are read as a site that enables them reads them.
   A line holding only an attribute list (`{.glossary}`, `{#id}`, `{attribution="Immanuel Kant"}`) is a block of its own.
   It stays directly above the block it describes (a definition list, heading, quote, fence, table) and stays apart from it when it began apart.
   A bare `\begin{align*}` ... `\end{align*}` environment is display math and comes back exactly as written, indentation included, instead of escaped as prose.
   Braces that do not hold a valid attribute list (`{name}`, `{u = 0}`, `{Footnote 1: See § xviii.}`) stay text, and the line never interrupts a paragraph.

The facade also preserves established skip-formatting comments:

```md
<!-- myform-ignore -->

This next block is left exactly as written.
```

```md
<!-- myform-ignore-start -->

Every block in this range is preserved.

<!-- myform-ignore-end -->
```

```md
<!-- myform-ignore-file -->
```

The markers act at the document top level.
Their strings remain configurable through `[plugin.myst]`.

### Format a library of scholarly files

OCR-derived scholarly chapters are long, citation-dense, and full of numbered prose.
This `.mdformat.toml` formats a whole file of that kind without renumbering lists or re-wrapping text:

```toml
wrap = "keep"                # line breaks stay as written; sembr alone decides them
# number is deliberately unset: list_numbering owns ordered lists

[plugin.myst]
list_numbering = "preserve"  # 2. 3. 4. stay 2. 3. 4.
sembr = true                 # optional: one sentence per line
sembr_list_items = true      # optional: numbered prose items too
```

Formatting is meant to change layout, not text, but mdformat and the CommonMark rules behind it make a few exceptions that matter for scanned books.
Check these before a library-wide run:

- A table row with more cells than its header loses the extra cells. GFM drops them when the table is parsed, so the formatter never sees them. Widen the header, or fence the table off with `<!-- myform-ignore -->`.
- Footnote definitions are written in the order of their first reference, not the order of the source.
- A link destination is percent-encoded when it holds a non-ASCII character or a backslash.
- An entity such as `&lt;` comes back as the character it names.
- A `$$...$$` pair inside a paragraph reads as inline math between two literal dollar signs, and the dollar signs come back escaped (`\$$...$\$`).
- A tangle of `*` runs (OCR'd bold and italic fragments) can re-nest differently on the first format and is not stable until the second. Clean such passages in the source.
- A role cannot hold a `{` in its name or attribute list. A stray `{` in prose therefore stays text rather than swallowing everything up to the next role.
- `sembr` chooses its line breaks after mdformat has decided where to escape, so a second format can still add or drop a backslash (`\[`, `\]`, `10\.`) and never changes a word. With `sembr` on, 248 of the 1,592 lit chapters change that way on the second format, and one of them again on the third. Without it, 16 chapters change on the second format (13 in table padding, 3 in the `*` tangle above). Format twice, or leave the line breaks to `Sembr.reflow` and format with `sembr = false`.

## Documentation

- [Documentation front door](https://myform.docs.doering.ai/)
- [API reference](https://myform.docs.doering.ai/api.html)
- [Universal translator specification](https://myform.docs.doering.ai/spec/myform-spec.html)
- [Typst ecosystem memo](https://myform.docs.doering.ai/spec/typst-ecosystem-memo.html)
- [Release procedure](RELEASING.md)

