Metadata-Version: 2.2
Name: koine
Version: 0.11.2
Summary: Native Koine runtime: runtime grammars, parsing and AST transpilation over the C API.
Author-Email: Chris Bates <1314008+chrsbats@users.noreply.github.com>
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Compilers
Classifier: Topic :: Text Processing :: General
Project-URL: Homepage, https://github.com/chrsbats/koine
Project-URL: Bug Tracker, https://github.com/chrsbats/koine/issues
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# Native Koine: runtime parser and transpiler

The C++ library builds and runs on the tested Linux x86-64 environment. It uses
actual runtime cpp-peglib definitions, PCRE2 regex terminals, native YAML/JSON
loading, a C interface, and Python `ctypes` / Nim `importc` packages named `koine`.

**Status: parser, lexer, transpiler and shared SLIP grammar checks pass on Linux x86-64.**
The loader uses rapidyaml 0.16.0 with explicit YAML 1.2 core scalar resolution.
The Python package exposes `Parser`, `PlaceholderParser` and `Transpiler`.
Nim exposes the same capabilities as ordinary procedures over `JsonNode`.

## Build and check

Requirements: C/C++17 compiler (GCC 15 recommended, GCC 12+ required for GCC builds),
CMake 3.22+, and Python for tests. Conformance checks also
need `pytest`, `pytest-subtests`, `parsimonious`, `regex`, `safer-yaml` and `pyyaml`
from the development environment.

From the repository root:

```bash
CC=gcc-15 CXX=g++-15 cmake -S native -B native/build -DCMAKE_BUILD_TYPE=Debug
cmake --build native/build --target koine_c_test koine_lexer_c_test koine_transpiler_c_test koine_document_test -j 4
ctest --test-dir native/build --output-on-failure
```

Set CMake's `Python3_EXECUTABLE` to an existing development interpreter if needed.
Dependencies are fetched into the build tree at configure time. cpp-peglib is
pinned to v1.22.0's commit, rapidyaml to the 0.16.0 source release and SHA256,
and nlohmann/json to v3.11.3. rapidyaml and PCRE2 10.39 are linked privately and
statically into `libkoine`; no system PCRE2 development package is needed.

Use a fresh build directory when changing compilers. GCC versions before 12 are
rejected because [GCC bug 66139](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66139)
leaks temporaries when AST conversion throws. GCC 15.2 passes the native suites
and C ASan/UBSan/leak checks, including numeric overflow and recovery. Both Nim
backends also pass eight API cases under ASan/UBSan with leak detection. Wrapper
references are constructed only after native construction succeeds, avoiding
Nim 2.2.12's C++ failed-result cleanup defect through the approved construction
order. No special exception mode or runtime patch is needed.

The library and Python package are staged together in `native/build/python/koine`.
The test scripts import that package using `PYTHONPATH`. The shipping pure-Python
implementation under `python/koine` also supplies the behavioral reference; its
off-root location prevents it from shadowing an installed native package:

```bash
PYTHONPATH=native/build/python python3 native/tests/test_native.py
```

An ordinary consumer uses:

```python
from koine import Parser

parser = Parser({
    "start_rule": "word",
    "rules": {"word": {"regex": "[a-z]+", "ast": {"leaf": True}}},
})
assert parser.validate("hello") == (True, "success")
ast = parser.parse("hello")["ast"]
```

`Parser.from_file(path)` accepts YAML 1.2 / JSON and resolves subgrammars relative
to each declaring file. Input beginning with `{` or `[` is parsed as strict JSON
(duplicate keys rejected) before falling back to YAML flow syntax. For in-memory
grammars, use `Parser(grammar, base_path=directory)` when relative dependency
paths need a specific directory.
`PlaceholderParser.from_file(path)` and `PlaceholderParser(grammar)` replace
subgrammars with their local placeholders without opening dependency files.

`parse(text, start_rule=...)` selects another rule. Construction prepares the
grammar once; `validate(text)` returns `(True, "success")` or `(False, message)`.
When the caller only needs generated text, `parse_transpile(text, transpiler,
start_rule=...)` performs both steps behind one native call and avoids returning
and re-encoding the AST. It returns the same success/error dictionaries as
`parse`. Native objects and result buffers are released automatically by the
wrapper. C consumers use [include/koine.h](include/koine.h), initialize outputs
to zero, and call the declared release functions. `koine_parser_prepare` and
`koine_parser_prepare_file` expose base-path and placeholder construction;
`koine_parser_parse_transpile` is the combined C entry point.

Parsing validates UTF-8 once per input, indexes source lines and code-point
columns once, and reuses match state, so cost scales linearly with work rather
than with input size per attempt.

## Python and Nim packages

From the checkout, the root project builds the pure-Python package; build this
native subproject explicitly:

```bash
CC=gcc-15 CXX=g++-15 python3 -m pip install ./native
```

Published `pip install koine` prefers a native platform wheel and falls back to the
pure package elsewhere. `pip install --no-binary=koine koine` forces the pure
package from source. The native package uses ordinary dictionaries/lists/scalars
and has no runtime dependency on Parsimonious, `regex` or a Python YAML library.
Wheels bundle the shared core and dependency notices. Source builds fetch the
pinned native dependencies; loading new grammars after installation never invokes
a compiler.

Build release artifacts from the repository with `tools/build.py` (pure
sdist/universal wheel at the root, portable native wheel from `native/`). Pass a
Windows wheel built on Windows with `--windows-wheel`. The native wheel is repaired
with auditwheel and receives the `manylinux_2_27`/`manylinux_2_28` tags. Validate
and publish saved artifacts with `tools/publish.py`; see the root README for the
release commands.

For Nim 2 with ARC/ORC, install once from the repository root:

```bash
CC=gcc-15 CXX=g++-15 nimble install
```

Nimble builds the shared core and C header during installation. Programs then use
`import koine`, without library-search or linker flags:

```nim
import std/json
import koine

let parser = newParser(%* {"start_rule": "word", "rules": {
  "word": {"regex": "[a-z]+", "ast": {"leaf": true}}
}})
doAssert parser.validate("hello") == (true, "success")
let ast = parser.parse("hello")["ast"]
doAssert newTranspiler().transpile(ast) == "hello"
```

Both `nim c app.nim` and `nim cpp app.nim` work with an installed package. If GCC
15 is installed alongside an older default compiler, select it explicitly:

```bash
nim c --gcc.exe:gcc-15 --gcc.linkerexe:gcc-15 app.nim
nim cpp --gcc.cpp.exe:g++-15 --gcc.cpp.linkerexe:g++-15 app.nim
```

The public procedures are:

| Capability | Nim API |
| --- | --- |
| In-memory grammar | `newParser(grammar, basePath = "")` |
| YAML/JSON grammar file | `parserFromFile(path)` |
| Isolated placeholders | `newPlaceholderParser(grammar)`, `placeholderParserFromFile(path)` |
| Parse / validate | `parser.parse(source, startRule = "")`, `parser.validate(source)` |
| Parse and generate text | `parser.parseTranspile(transpiler, source, startRule = "")` |
| Output rules | `newTranspiler(grammar = nil)`, `transpilerFromFile(path)` |
| Generate text / inspect state | `transpiler.transpile(ast)`, `transpiler.state` |

Parser/transpiler references share automatic native ownership. Returned ASTs and
state are independent `JsonNode` values; state inspection returns a snapshot.
Syntax errors return a JSON error result, while invalid construction/API calls
raise `ValueError`. Standard `std/json` preserves uint64 values above int64 as raw
JSON numbers: use `ast["value"].to(uint64)` to retrieve them and `$ast` to serialize
them without quotes or precision loss.

On Linux, deploy `libkoine.so` beside the executable (with its bundled dependency
notices). The binding provides `$ORIGIN` and installed-package rpaths. On Windows,
deploy `koine.dll` beside the executable. The MSVC DLL statically links the C++
and C runtimes, so it needs no Visual C++ redistributable. Windows Nim consumers
must use the MSVC backend (`--cc:vcc`); the binding then links `koine.lib`. MinGW
on Windows is not supported. Linux x86-64 with GCC 15.2 and Windows x86-64 with
MSVC 2022/Nim 2.2.12 are tested; other architectures are not.

### Portable Linux wheels

The verified x86-64 wheel has `py3-none-manylinux_2_27_x86_64` and
`py3-none-manylinux_2_28_x86_64` tags. It installs normally on Ubuntu 22.04; clean
Python 3.9, 3.12 and 3.13 environments pass the dependency-free API checks.
Python 3.9 also passes those checks with the older glibc 2.28 loader and C/C++
runtimes.

Release wheels use GCC 15 against an older glibc 2.28 sysroot, with the released
AlmaLinux 8 GCC 15 toolset. This was built directly, without containers or system
RPM installation. The toolset supplies its standard C++ runtime compatibility
support; no dependency or runtime source was patched. Ordinary source installs
use the host toolchain shown above and do not need this release SDK.

Use standard CMake compiler/sysroot settings in a release toolchain file, then
build from a saved source distribution and check the resulting binary with
auditwheel. For an already prepared SDK, a toolchain file looks like:

```cmake
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSROOT "/path/to/sysroot")
set(toolset "${CMAKE_SYSROOT}/opt/rh/gcc-toolset-15/root/usr")
set(CMAKE_C_COMPILER "${toolset}/bin/gcc")
set(CMAKE_CXX_COMPILER "${toolset}/bin/g++")
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
```

The repository ships a provisioning script and toolchain file, so no system RPMs
or containers are needed:

```bash
uv run --with libarchive-c python native/tools/prepare_linux_sdk.py build/sdk-alma8-gcc15
export KOINE_SDK_ROOT=$PWD/build/sdk-alma8-gcc15
export CMAKE_TOOLCHAIN_FILE=$PWD/native/tools/linux-gcc15.cmake
export LD_LIBRARY_PATH=$KOINE_SDK_ROOT/compiler-runtime/usr/lib64:$KOINE_SDK_ROOT/sysroot/opt/rh/gcc-toolset-15/root/usr/lib64
uv build native/ --wheel --out-dir build/portable-dist
uvx auditwheel repair --plat manylinux_2_28_x86_64 --wheel-dir build/wheelhouse \
  build/portable-dist/koine-*-linux_x86_64.whl
```

`patchelf` must be on `PATH` for auditwheel. Do not relabel a host-built wheel to
an older manylinux tag: the compiler's linked runtime symbols determine the actual
compatibility. Koine and dependency notices are included in both the Python wheel
and Nim runtime component.

### Windows x64

Release builds use MSVC (Visual Studio 2022) on Windows. The repository uses a
Windows virtual environment at `.venvs/win`, created by Windows `uv`; from WSL the
Windows interpreter can be invoked directly as `.venvs/win/Scripts/python.exe`.

```powershell
uv venv .venvs/win
.venvs\win\Scripts\python.exe -m pip install build scikit-build-core cmake
$env:CMAKE_GENERATOR = "Visual Studio 17 2022"
.venvs\win\Scripts\python.exe -m build native --wheel --no-isolation --outdir build\wheels
```

The build uses `/Zc:__cplusplus`, `/permissive-` and a static MSVC runtime; the
resulting `koine.dll` imports only `KERNEL32.dll`. The `win_amd64` wheel contains
`koine/koine.dll`, its notices, and no Python dependencies.

For the full native suite:

```powershell
cmake -S native -B build\native -G "Visual Studio 17 2022" -A x64 `
  -DBUILD_TESTING=ON -DPython3_EXECUTABLE=.venvs\win\Scripts\python.exe
cmake --build build\native --config Release --parallel
ctest --test-dir build\native -C Release --output-on-failure
```

Nim consumers use the same Windows toolchain:

```powershell
nimble --nimbleDir:build\nimble install -y
nim c --cc:vcc --path:build\nimble\pkgs2\koine-0.11.2-* app.nim
nim cpp --cc:vcc --path:build\nimble\pkgs2\koine-0.11.2-* app.nim
```

The Nimble install builds `koine.dll` and `koine.lib` with the Visual Studio
generator and installs both beside the binding. Compiling without `--cc:vcc` on
Windows raises an explicit error; MinGW is not a supported Windows toolchain.

### Measured release performance

Milliseconds per ordinary API call, medians of three samples on Linux x86-64:

| Case | Python reference | Native Python wheel | Native Nim package |
| --- | ---: | ---: | ---: |
| Math parse + transpile | 0.096 | 0.111 | 0.078 |
| Tokenized Python parse + transpile | 0.295 | 0.227 | 0.179 |
| Complete SLIP `root.slip` parse | 268.820 | 75.205 | 80.308 |

Combined `parse_transpile` / `parseTranspile`, milliseconds per call:

| Case | Native Python wheel | Native Nim package |
| --- | ---: | ---: |
| Math parse + transpile | 0.072 | 0.054 |
| Tokenized Python parse + transpile | 0.143 | 0.139 |

Inputs and raw outputs are equal across all paths. Timings include C JSON
emission and host conversion; grammar preparation is excluded. Samples use 200,
100 and 3 calls respectively. Python uses the installed portable wheel; Nim uses
the source-installed Release core. Both are GCC 15 builds. Results are workload-
and environment-dependent, with no general speedup guarantee. The benchmark also
reports standalone host JSON decoding (about 2.08 ms Python / 2.85 ms Nim for the
complete root result); this is not an attribution of all serialization overhead.

## Supported parser behavior

- Literal and regex matching, rule references, sequence, ordered choice,
  optional/repeated terms, lookahead, and non-left-recursive grammar construction.
- `leaf`, `tag`, `discard`, `promote`, typed leaves, named children, custom
  structures, associative operators, and Unicode-code-point source locations.
- Inline AST normalization, subgrammar discovery/namespaces/cycles, qualified
  sibling and parent references, reachability/empty-AST checks, directive
  conflicts, and left-recursion checks (including zero-width regex terminals).
- PCRE2 matching at a byte offset in the **complete subject**, preserving
  lookbehind/anchors. The adapter translates Python's absolute-end `\Z` to
  PCRE2's `\z`; escaped literal `\\Z` remains literal.
  Unicode character classes are not identical to Python's `regex`: for example,
  `\w+` accepts decomposed `e` + U+0301 in the pure scannerless parser but not in
  PCRE2 10.39. Use an explicit class such as `[\p{L}\p{M}\p{N}_]+` when those
  combining marks are part of an identifier.
- Independent per-call results, serialized access to one parser from threads,
  C error reporting, and buffers that outlive source and parser storage.
- Optional token-based parsing using the existing `lexer` and `token` grammar
  keys, longest-match/first-specification tie selection, skip actions, and token
  discard/type directives. Native `handle_indent` supports INDENT/DEDENT,
  blank-line handling, dedent validation and EOF dedents.

Both `Parser` and `PlaceholderParser` accept lexer grammars through their normal
constructors and `from_file()`. The implementation normalizes CRLF/CR to LF and
expands tabs at eight-code-point tab stops, matching the reference. Lexical,
indentation and token syntax failures return ordinary parse errors with source
locations. Zero-progress token matches produce explicit errors.

Raw token-mode ASTs retain the reference representation: branch `text` contains
token names; token leaves contain normalized source values and source positions.
Parent positions use the first direct AST child with a location, or default to
1:1. Leaf wrappers concatenate child text. Unicode decimal digits are converted
without changing token text or the exact int64/uint64 numeric policy.

AST shaping runs after successful matching, so rejected backtracking branches
cannot trigger numeric conversions. Alias/single-choice boundaries, promoted
lists and cleanup follow the reference behavior. Extra AST metadata is ignored
as in the reference, including malformed keys already present in shared fixtures.

Native `type: number` preserves exact int64/uint64 integer tokens and uses float64
for decimal/exponent forms. Whole-valued floats such as `1.0` and `1e3` remain
floats, and negative zero retains its sign through JSON and source round trips.
Overflow is explicit and token `text` is preserved. Integer conversion intentionally
differs from the Python reference's float-first rounding for large integers:
`9007199254740993` remains exact in native output.

Native parsing, tokenization and transpilation do not
call the Python implementations. The conformance runner loads no reference
implementation. It supplies a failing import-only stub for a backend-specific
grammar-string helper; any accidental call fails the native run.

## Transpilation

```python
from koine import Transpiler

transpiler = Transpiler({
    "rules": {"greeting": {"template": "Hello, {name}!"}},
})
assert transpiler.transpile({
    "tag": "greeting", "children": {"name": {"text": "world"}},
}) == "Hello, world!"
```

`Transpiler.from_file(path)` loads YAML/JSON output rules natively, without a host
YAML dependency. `Transpiler()` uses the documented value/text fallbacks. Output
rules are prepared once; `transpile(ast)` returns ordinary text. Named and list children, binary-op
fields, `cases`, path conditions, `state_set`, indentation and global indentation
configuration follow [TRANSPILING.md](../TRANSPILING.md). Boolean/null values use
the reference's `True` / `False` / `None` spelling when converted to text.

Each top-level call resets state and indentation. Updates occur after a node's
output and are visible to subsequent siblings and parents. `transpiler.state`
returns a detached snapshot of the most recent call's state. Exceptions cannot
carry indentation or state into the next call. Calls on a handle are serialized.

Approved native template scope is named substitutions (`{name}`) and escaped
braces (`{{` / `}}`). Broader Python `str.format` specifications, conversions and
field-access expressions raise explicit native errors. Substituted child text
is inserted literally rather than interpreted as another template.

C callers use `koine_transpiler_create`, `koine_transpiler_from_file`, `koine_transpiler_transpile`,
`koine_transpiler_state` and `koine_transpiler_destroy`. Creation accepts runtime
YAML/JSON grammar data; transpilation borrows a length-delimited JSON AST and
returns owned UTF-8 text. State snapshots are owned JSON buffers. Release both
through `koine_buffer_free`; outputs survive input and handle destruction.

The runtime math example uses the existing calculator parser grammar and
[math_to_slip.yaml](tests/math_to_slip.yaml). For example:

```text
10 + 5 * 2  -> (10 + (5 * 2))  -> 20
2 ^ 3 ^ 2   -> (2 ** (3 ** 2)) -> 512
```

With the sibling SLIP project environment, run from its checkout with an absolute
native staging path and absolute test-script path:

```bash
PYTHONPATH=/path/to/koine/native/build/python uv run --no-sync python3 /path/to/koine/native/tests/check_math_slip.py
```

This saves/reloads generated programs and evaluates them using the existing SLIP
runtime. Ordinary SLIP evaluation remains left-to-right.

## YAML loading and verification

rapidyaml parses syntax, escaping and scalar styles; `src/yaml.cpp` converts its
tree directly into owned `ordered_json` data. Quoted and block scalars remain
strings; plain scalars use YAML 1.2 core types. Duplicate keys and values outside
the JSON-compatible model are rejected. Exceptions cross the C boundary as errors.
There is no intermediate YAML-to-JSON text emission step.

The former libfyaml beta generic adapter was removed after independent checks
exposed invalid NUL escaping in its optimized emitter and quoted-string type loss
in its event emitter. The replacement uses supported rapidyaml APIs without
dependency patches.

All eleven CTest suites pass on Linux and Windows.
**122 public behavioral cases and 2 subtests run
through native entry points**, including every lexer-dependent case and both
complete Python↔JavaScript parse/transpile tests. Four Parsimonious grammar-string
tests remain separate Python-backend tests and pass in that backend. These are
not counted as native behavior tests. The pure backend passes the full 127-case
suite plus 2 subtests, including the reference-only checks.

Additional checks include 18 parser, 9 lexer and 12 transpiler Python API cases
and C ownership/API checks. Default CTest runs 82 document comparisons, 9 raw
AST/status comparisons and 3,996 regex comparisons over 10 Koine grammar files.
The 0.11.2 candidate also passes all 22 SLIP source-printing tests with each backend.

Earlier expanded SLIP corpus runs passed 85 document comparisons over all 13
Koine/SLIP grammar files, 7,437 regex comparisons, 169 parser/placeholder/integration
checks and 141 raw AST/status comparisons, including complete `root.slip`, text,
scalar types and locations. See [PLAN.md](../PLAN.md) for the historical evidence.

Lexer checks compare raw token values/locations, normalized source, ASTs and error
positions against the reference, including Unicode, NUL, tabs, blank lines,
lookbehind/end assertions, aliases and placeholders. Python/JavaScript fixture
checks now generate ASTs natively and compare raw ASTs, unnormalized output and
state with the independent reference results. Earlier checks also verified six
generated math programs through actual SLIP evaluation, including saved-artifact round trips,
precedence, associativity and exact large integers.

With a sibling SLIP checkout, reproduce the additional corpus checks explicitly:

```bash
PYTHONPATH=native/build/python python3 native/tests/compare_asts.py --slip-root ../slip
python3 native/tests/compare_documents.py native/build/koine_document_test tests/*.yaml ../slip/grammar/*.yaml
PYTHONPATH=native/build/python python3 native/tests/compare_regex.py ../slip/grammar/*.yaml
```

C parser/lexer/transpiler ownership and document checks also pass under ASan/UBSan
with leak detection. Nim C/C++ checks exercise native construction failures,
numeric overflow, UTF-8/NUL, shared aliases, detached results/state and repeated
cleanup under the same sanitizers.

Install development dependencies with `python3 -m pip install '.[dev]'` (select
GCC 15 through `CC`/`CXX` as above). Native behavior and reference-only grammar-string
checks are separate:

```bash
PYTHONPATH=native/build/python python3 native/tests/conformance.py
PYTHONPATH="$PWD/python" python3 -m pytest -q tests/test_edges.py -k test_transpile_grammar
```

The distribution inspection/clean-install tools are `tests/check_distribution.py`
and `tests/check_installed.py`. `tests/check_nim_asan.py` honors `CC`/`CXX`, uses an
instrumented CMake build and tests both ordinary Nim backends. End-to-end release
measurements use `tests/benchmark.py` with `tests/benchmark_nim.nim`; JSON emission
and host conversion are included, and output equality is asserted before timing.
These are bounded conformance results; broader regex dialect equivalence and
other platforms remain explicit gaps. See [PLAN.md](../PLAN.md) for saved-artifact
verification, timings and the active platform step.
