# antlrope

> Fast, C++-accelerated ANTLR runtime for Python. Parse text with an ANTLR grammar at C++ speed (typically 10–20× faster than the official pure-Python runtime) while you write only plain Python listener callbacks, with no C or C++ to write or compile.

**Antlrope** drives the official ANTLR4 C++ runtime's ATN interpreter directly from the serialized ATN that the stock `-Dlanguage=Python3` ANTLR tool already emits, so no C++ code is generated per grammar and there is nothing to compile. After parsing in C++, a single, filtered stream of events crosses into Python in one transfer (instead of one foreign-function call per parse-tree node) and drives named callbacks. No Python parse tree is built.

When to use it: for throughput on large inputs or inputs with many records. When not to use it (use the official `antlr4-python3-runtime` instead): when the grammar relies on semantic predicates (`{...}?`) or embedded actions (`{...}` target-language code), which the interpreted ATN cannot execute; or when you need a parse tree you can keep and navigate freely rather than a single streaming pass.

## Install

- `pip install antlrope` (or `conda install -c conda-forge antlrope`): pre-built wheels and conda packages for CPython 3.12 and newer on Linux, macOS, and Windows. Installs `antlr4-python3-runtime` automatically.
- `pip install antlr4-tools`: the ANTLR tool (Java) used to generate parsers from a `.g4` grammar. Needed only to generate parsers, not to run them.

## Workflow

1. Generate a parser with the stock ANTLR tool (ordinary ANTLR, nothing special):
   `antlr4 -Dlanguage=Python3 JSON.g4 -o generated`  → `generated/JSONLexer.py`, `generated/JSONParser.py`
2. Generate a facade from the parser module (an importable dotted path) and a grammar-name prefix:
   `antlrope gen generated.JSONParser JSON -o json_listener.py`
   → a `JsonEventListener` base class (named `<Grammar>.capitalize() + "EventListener"`) with an `enter<Rule>`/`exit<Rule>` pair per grammar rule, `visitTerminal`/`visitError`, token-type constants, and a `walk(text)` method (inherited from `FacadeListener`). The facade imports the lexer and parser and stores them, so `walk` and `walk_parallel` need no class arguments. The lexer module is derived from the parser's (`<Grammar>Parser` → `<Grammar>Lexer`); pass `--lexer` to override. You subclass it; you do not edit it.
3. Subclass the facade, override only the callbacks you need, and call `walk`:

```python
from json_listener import JsonEventListener        # generated in step 2

class StringCollector(JsonEventListener):
    def __init__(self):
        self.strings = []
    def visitTerminal(self, token_type: int, text: str) -> None:
        if token_type == self.STRING:                # STRING is a generated constant on the class
            self.strings.append(text)

c = StringCollector()
c.walk('{"name": "ada", "tags": ["x", "y"]}')   # optional: start_rule="value"
print(c.strings)
```

Key model facts for writing listeners:
- Rule callbacks take no arguments (no node or context object). Reconstruct state from the order of the enter, exit, and terminal events, usually with a small stack (push on `enter<Rule>`, pop on `exit<Rule>`).
- `visitTerminal(self, token_type, text)` gives the token's type (compare against the class's token-type constants, e.g. `self.STRING`) and its source text (already sliced for you).
- Override only what you need: callbacks you do not define are filtered out in C++ and never cross into Python (faster).
- Token-type constants for anonymous string literals use ANTLR's positional names (`T__0`, `T__1`, …, matching the generated lexer), not the token-type value.
- Source position: inside any callback, `self.line_col()` returns the current event's `LineCol(line, column)` (or `None`), `self.span()` its raw `(start, stop)` char offsets, and `self.sourcename()` the source name (e.g. filename) if one was set. After `walk`, `self.syntax_errors` is a list of `ParseError`; it is reset on every `walk`. ANTLR's console error listener is suppressed.

## Public API (`from antlrope import ...`)

- `FacadeListener`: base of every generated `<Grammar>EventListener`. Callbacks `enter<Rule>()` / `exit<Rule>()` / `visitTerminal(token_type, text)` / `visitError(token_type, text)`; helpers `line_col()`, `span()`, `sourcename()`; attribute `syntax_errors`. Methods `walk(text, *, start_rule=None, filtered=True)` (on the generated subclass) and classmethod `walk_parallel(chunks, *, start_rule=None, max_workers=None, filtered=True, factory=None)`. Both use the facade's lexer and parser. `walk_parallel` parses independent chunks on multiple CPU cores and returns a lazy iterator that yields listeners in input order, with at most `max_workers` parses running at once (default `os.cpu_count()`); `factory` is an optional zero-argument callable that creates each listener.
- Chunking: `classmethod`s on the generated `<Grammar>EventListener` (call as `MyListener.split_on_token(...)`), all yielding positioned `Chunk`s for `walk_parallel`. The token and rule chunkers use the facade's lexer and parser and the class's token-type constants, so they take no lexer or parser argument:
  - `split_on_token(text, token_types, *, where="before"|"after", channel=0, sourcename="")` and `split_between_tokens(text, pairs, *, nested=False, ...)`: token-aware, so a delimiter inside a string or comment never causes a split.
  - `split_on_pattern(text, pattern, *, where=..., flags=0, sourcename="")` and `chunk_by_pattern(text, pattern, ...)`: regex; faster but not token-aware.
  - `chunk_by_rule(text, rule, *, start_rule=None, outermost=True, ...)`: split on grammar structure (parses in C++).
  - `stream_on_token(path, token_types, *, where=..., encoding="utf-8", sourcename="", ...)` and `stream_on_pattern(source, pattern, *, where=..., window_chars=65536, window_lines=None, encoding="utf-8", sourcename="")`: streaming with bounded memory (they read a file or stream incrementally).
  - `lex(text, *, keep=None)`: the parser-free token pass the splitters build on; yields `LexToken(type, channel, start, stop)`.
- `Chunk(text, offset=0, line=1, column=0, sourcename="")`: a source piece plus its position and optional source name; a bare `str` may be passed to `walk_parallel` instead (positioned contiguously).
- `SourceMap(text)`: `.line_col(offset)` and `.offset(line, column)` convert between char offsets and `(line, column)`.
- `LineCol(line, column)`: the named tuple returned by `line_col()`.
- `ParseError`: a parse diagnostic; an `Exception` subclass (so it can be raised): `line`, `column`, `start`, `stop`, `message`.
- CLI (`antlrope` is a command group): `antlrope gen <parser-module> <name> [-o <file>]` generates a facade (and, when `-o` is used, writes an origin header recording the version, command, run directory, and input SHA256s); `antlrope regen <file>` regenerates it from that header; `antlrope up-to-date <file>` exits non-zero if its inputs changed; `antlrope check <parser-module>` exits non-zero if the grammar uses semantic predicates or embedded actions (which antlrope cannot run); `antlrope rules <parser-module>` / `antlrope tokens <parser-module>` list the rule names and token-type constants (`--json` for tooling).

Character offsets throughout are 0-based **codepoint** indices; `line` is 1-based, `column` 0-based (matching ANTLR's `line:column`).

## Documentation

- [Getting started](https://zuzukin.github.io/antlrope/getting-started/): a complete walkthrough from generating a parser to writing and running a listener.
- [Installation](https://zuzukin.github.io/antlrope/installation/): pip and conda, the ANTLR tool, supported Python versions and platforms.
- [Chunking](https://zuzukin.github.io/antlrope/chunking/): split large inputs, or inputs with many records, for parallel or streaming parsing.
- [Parallel parsing](https://zuzukin.github.io/antlrope/parallel-parsing/): `walk_parallel` on multiple CPU cores.
- [Recipe: streaming records](https://zuzukin.github.io/antlrope/streaming-records/): parse a large file record by record with bounded memory.
- [Migrating from antlr4-python3-runtime](https://zuzukin.github.io/antlrope/migrating/): map a `ParseTreeListener` onto the facade.
- [Examples](https://zuzukin.github.io/antlrope/examples/): complete programs (JSON value reconstruction, parallel schema indexing).
- [API reference](https://zuzukin.github.io/antlrope/reference/api/): every symbol, generated from docstrings.
- [Command line](https://zuzukin.github.io/antlrope/reference/cli/): the `antlrope` command group (`gen`, `regen`, `up-to-date`, `check`, `rules`, `tokens`).
- [Glossary](https://zuzukin.github.io/antlrope/glossary/): ATN, DFA, facade, FFI, GIL, and other terms.
- [How it works](https://zuzukin.github.io/antlrope/concepts/): the bulk, filtered event stream and native masking.
- [Performance & limitations](https://zuzukin.github.io/antlrope/performance/): the limits on speed, and which grammars (those with predicates or actions) are not supported.
- [Source](https://github.com/zuzukin/antlrope) · [PyPI](https://pypi.org/project/antlrope/)
