Metadata-Version: 2.4
Name: loaderx
Version: 2.0.0
Summary: Rebuildable high-performance record containers
Author-email: Ben0i0d <ben0i0d@foxmail.com>
License-Expression: MIT
Project-URL: Homepage, https://codeberg.org/eoelab/loaderx
Project-URL: Documentation, https://codeberg.org/eoelab/loaderx
Project-URL: Source, https://codeberg.org/eoelab/loaderx
Project-URL: Bug Tracker, https://codeberg.org/eoelab/loaderx
Keywords: flax,python,dataloader
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: cffi
Requires-Dist: msgpack
Dynamic: license-file

# Loaderx
Zrecord is a rebuildable high-performance on-disk record container built from
authoritative source data and scripts. ``create`` is append-only, ``close``
publishes the container Header, and ``open`` is read-only. To change
content, rebuild it at a new path.

Zrecord is the typed on-disk container; Loaderx is the sampler and prefetch
loader that consumes Zrecord streams. They currently ship together while both
layers mature, but their public responsibilities remain separate.

```
pip install loaderx
```

Wheels are published for Linux (glibc ≥ 2.17 and musl, x86-64 and arm64), macOS
(≥ 11.0, Intel and Apple Silicon) and Windows (x64 and arm64). The bindings use
cffi in ABI mode, so nothing links against the CPython ABI and one wheel per
platform serves every supported Python.

## Design Philosophy

loaderx is built around several core principles:

1. A pragmatic approach that prioritizes minimal memory overhead and minimal dependencies.
2. A strong focus on single-machine training workflows.
3. We implement based on NumPy semantics, persisted by the private native store engine.
4. An **immortal (endless) step-based data loader**, rather than the traditional epoch-based design—better aligned with modern ML training practices.
5. **Dense and ragged are separate contracts, and the loader serves both.** A
   dense stream stacks into one array per batch; a ragged one comes back as a
   list. Neither is padded, and equal length is never treated as a special case
   of variable length.

## 设计文档

- [性能优化与工程决策记录](docs/性能优化.md)
- [分布式架构草案](docs/分布式架构.md)

## Quick Start
```python
import numpy as np

from loaderx.zrecord import Dense
from loaderx.dataloader import DataLoader

data = np.load('data.npy', mmap_mode='r')
label = np.load('label.npy', mmap_mode='r')
with Dense.create('train_data', data.dtype, data.shape[1:]) as ds:
    ds.append(data)
with Dense.create('train_label', label.dtype, label.shape[1:]) as ds:
    ds.append(label)

data_store = Dense.open('train_data')
label_store = Dense.open('train_label')
loader = DataLoader({'data': data_store, 'label': label_store},
                    transform=lambda batch: batch)
for i, batch in enumerate(loader):
    if i >= 256:
        break

print(batch['data'].shape)
print(batch['label'].shape)

loader.close()
data_store.close()
label_store.close()
```

A batch is a dict ``{name: values}``: each value is the stacked
``(batch_size, *item_shape)`` array for that stream.  Every stream is gathered
at the same indices, so record ``i`` lines up across them.  The ``transform``
callback is the collate step — reshape, cast, stack — where ``values`` is the
plain dense batch ready for the model.

### Creating a dense store
```python
import numpy as np
from loaderx.zrecord import Dense

data = np.load('data.npy', mmap_mode='r')
with Dense.create('train_data', data.dtype, data.shape[1:]) as ds:
    ds.append(data)
```
One record per slice along axis 0; a 1-D array (the usual shape of a label set)
becomes a store of scalar records. The caller controls each append batch and can
slice a large or mmapped array to set its own memory bound. Creation and opening
are explicit: ``create`` requires a new path and returns an append-only writer;
leaving its context calls ``close()``, which publishes the result.
``open`` requires an existing container produced by a successful build and returns a read-only
reader. Open the result with ``Dense.open``:

    ds = Dense.open('train_data')
    batch = ds[:]
    ds.close()

Python defines the exact record schema: ``dtype`` plus the dense ``item_shape``,
or only ``dtype`` for ragged stores. The MsgPack bytes live opaquely in the
static page at the front of ``meta.zr``; Zig persists them but never interprets
them. The schema accepts no user metadata. Python selects the geometry and gives
the private native engine only the runtime record boundaries it needs. Each Ragged record
carries its shape in an inline little-endian u64 prefix. One native physical
engine consumes the trusted Dense stride or Ragged offsets. Append inputs are
strictly NumPy arrays: Dense takes one batched ndarray and Ragged takes an
iterable of ndarrays. Raw bytes and pre-encoded images are made explicit with
``np.frombuffer(raw, dtype=np.uint8)`` and stored in a
``Ragged.create(path, dtype=np.uint8)`` rather than creating a second
public storage API.

### Records
One persistent format, two native execution contracts. ``Dense`` is the **dense** contract
where every record is exactly one row of the recorded ``item_shape``; reads
are fixed-stride gathers and the batch shape follows from the schema, so no
per-record metadata is touched:

```python
import numpy as np
from loaderx.zrecord import Dense

data = np.arange(64, dtype=np.float32).reshape(8, 2, 4)
with Dense.create('data', data.dtype, data.shape[1:]) as ds:
    ds.append(data)
ds = Dense.open('data')
ds[0, 5, 2]                          # (3, 2, 4) — shape from the persisted schema
ds.close()
```

A store is a collection of records, not an ndarray, so ``ds[0, 5, 2]`` is a
record set — never ``ds[0][5][2]``. A scalar selects one record; indices must be
in ``0..len(ds)-1``. The ragged example below reads the same way.

``Ragged`` is the **ragged** contract for variable-length records. It is a
separate contract: :class:`Ragged` hands back a list of arrays, so a
loader never has to carry ``row_splits`` around. ``dtype`` is unified and
explicit; each record keeps its **own shape**, recorded per record as it is
written and restored exactly on read — so records may differ in shape
arbitrarily, and nothing is ever inferred from the source (an iterator can't
tell you what its later records look like). Scalar ``shape == ()`` is preserved;
zero-byte arrays are rejected because physical records are nonempty. Densifying a list into a dense
batch is the model's call — a plain numpy loop, wherever you need it:

```python
from loaderx.zrecord import Ragged

seqs = [np.arange(L, dtype=np.int32) for L in (3, 1, 4, 1, 5)]
with Ragged.create('tokens', np.int32) as rs:
    rs.append(seqs)                       # dtype explicit; each record keeps its shape
rs = Ragged.open('tokens')

records = rs[0, 2, 4]                # list of ndarray — one per record, exact shapes
rs.close()
lengths = np.array([len(r) for r in records])
padded = np.zeros((len(records), lengths.max()), dtype=records[0].dtype)
for i, r in enumerate(records):
    padded[i, :len(r)] = r             # (B, max_len) — your policy, your loop
```

A ``DataLoader`` dynamically composes a dict of dense and ragged streams.
Collation is the ``transform`` — a batch dict in, a batch dict out:

```python
def collate(batch):
    return {'input_ids': batch['tokens'], 'label': batch['label']}

loader = DataLoader({'tokens': dense_tokens, 'label': labelset}, batch_size=32,
                    transform=collate)
batch = next(loader)
loader.close()
```

The transform runs once for each gathered batch on a loader transform worker.
Dense values are independent writable contiguous arrays; Ragged records share
one backing allocation per batch. The return value is handed to the consumer
unchanged. Calls may run concurrently and complete out of order, so the callback
must be thread-safe; exceptions are propagated to the consumer. Keep shared
mutable state and nested parallel runtimes out of the callback.

Numba can optionally accelerate a CPU-heavy Dense transform while releasing the
GIL. Compile it before timing, then call it from the ordinary transform:

```python
import numba
import numpy as np

@numba.njit(nogil=True, parallel=False)
def normalize_u8(x):
    out = np.empty(x.shape, dtype=np.float32)
    for i in range(x.size):
        out.flat[i] = x.flat[i] / 255.0
    return out

normalize_u8(np.zeros((1, 3, 224, 224), dtype=np.uint8))  # compile warmup

def transform(batch):
    batch["image"] = normalize_u8(batch["image"])
    return batch
```

Numba is neither a loaderx dependency nor a loader backend. Its compilation
warmup belongs outside build or loader benchmark timing.

### Creating containers

``Dense.create`` and ``Ragged.create`` return append-only writers. ``append`` is
explicit — one container of records becomes one native batch.
Dense append is synchronous and borrows an already-contiguous ndarray without a
snapshot copy. Ragged append consumes its iterable once into one owned packed
buffer, then completes the native append before returning. Nothing is inferred.

```python
from loaderx.zrecord import Dense, Ragged

ds = Dense.create('mnist/x', dtype=np.uint8, item_shape=(28, 28))
ds.append(images[i:i + 1024])        # synchronous native batch; returns None
ds.append(single_image[None])        # one sample is batch_size 1 — add the axis yourself
ds.close()                            # publish before opening

tok = Ragged.create('tokens', dtype=np.int32)
tok.append([seq_a, seq_b, seq_c])
tok.close()

with Dense.open('mnist/x') as ds:
    first_four = ds[:4]               # opened containers are read-only
```

Dense and Ragged append both report native errors in the current call and return
only after accepting the batch. A writer cannot be read, and a reader cannot be
appended to. ``close()`` on a writer publishes the container Header;
``close()`` on a reader releases it. Dense, Ragged, and DataLoader support
``with`` for scoped lifetimes. Content is never changed in place: rerun the
authoritative build at a new path, validate it, then switch consumers to it.

The exact schema is declared at creation and encoded by Python as MsgPack. Dense
schema contains only ``dtype`` and ``item_shape``; Ragged schema contains only
``dtype``. Structured, subarray, object, and metadata-bearing dtypes are not
supported: their semantics do not round-trip through one canonical NumPy dtype
string. The encoded schema has 4064 bytes available in the fixed 4096-byte
metadata page. Ragged schema size is effectively fixed; Dense schema size grows
only with the integer ``item_shape``, so the physical limit is far above any
practical NumPy array rank.
Unexpected fields are rejected. ``append`` validates dtype and shape in Python,
then passes the derived byte width to the private Dense operation. It does not
coerce Python lists, tuples, bytes, bytearrays, or other array-like objects;
callers convert them with ``np.asarray`` or ``np.frombuffer`` first.
Dictionary training stays explicit and separate; creating a
``zstd_dict`` store requires the completed dictionary, so no valid store is
published without one.

### Codec notes

`"zstd"` compresses each record independently with plain zstd (level 3). Use it
for general-purpose compression — it is fast and the default.

`"zstd_dict"` trains a shared dictionary on a sample of the data before writing
any record, then compresses every record against it at level 19. The dictionary
captures structure shared across records that per-record compression cannot see —
a large win for many small, similar records (image tiles, token sequences).

The dictionary's cost is a cache footprint: every record's decompression
references the shared dictionary window, so a larger dictionary means more
cache misses on gather — the path a loader pays forever. Three named tiers
(`loaderx.zrecord.DICT_TIERS`) preset the whole tradeoff — the dictionary size
and how much data trains it — so a caller picks a tier, never a number:

| tier | dict | sample | tradeoff |
|------|------|--------|----------|
| `"fast"`     | 32 KiB  | 4 MiB  | fastest gather and training; ratio barely above `zstd` |
| `"balanced"` | 128 KiB | 16 MiB | **default** — most of the ratio at a fraction of the gather cost |
| `"max"`      | 1 MiB   | 64 MiB | best ratio; slowest gather and training |

The sample is the byte budget the dictionary trains on (a strided subset of the
records), so each tier costs the same training time whatever the record size.
At realistic image scale (768 KiB records) the tiers converge — on the earlier
measurement box's structured data `"balanced"` and `"max"` both gather
~1.6 GiB/s at a 1.74x ratio — because a dictionary is a small fraction of a
large frame. The tiers still matter at small record sizes, where the dict is
most of a record and `"max"` trades gather throughput for ratio.

`"zstd_dict"` records can only be read from a store that has the dictionary
(`dict.zr`). The dictionary is loaded on open and shared, lock-free, across all
reader threads.

A dictionary must train on the settled, complete data. :func:`train_dict` is the
standalone, **manual** training step — a numpy array, an iterable of records, or
a typed container all train the same way, sized by a tier — and its bytes are handed to
a container-writing path via ``dict_bytes``. ``zstd_dict`` never trains by itself: a
write without a dictionary is an error. A stream cannot train its own
dictionary, but it can write with one trained on the settled data:

```python
from loaderx.utils import train_dict
from loaderx.zrecord import Dense, Ragged

# train once on the settled data — a standalone, reusable artifact
d = train_dict(settled_array, tier="balanced")

# then any new store can install it and append explicitly
with Ragged.create('tokens', np.int32, codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(token_generator)
with Dense.create('data', data.dtype, data.shape[1:],
                       codec='zstd_dict', dict_bytes=d) as ds:
    ds.append(data)
```

Changing an existing store's codec is a rewrite, not a store mutation: read the
records as numpy values and append them to a new store with the new codec. A
whole-store slice preserves index order, which keeps multi-stream alignment.
There is no dedicated ``recode`` or store-to-store path because the ordinary
read and append contracts already express the operation:

```python
from loaderx.utils import train_dict
from loaderx.zrecord import Dense

with Dense.open("src") as s, \
     Dense.create("dst", dtype=s.dtype, item_shape=s.item_shape,
                       codec="zstd_dict", dict_bytes=train_dict(s)) as d:
    d.append(s[:])                             # ndarray -> dense append
```

``Ragged`` is the same shape: ``s[:]`` returns ``list[np.ndarray]``, which
is exactly its append input, and ``dtype`` is all the destination needs. The
native compression path bounds its own working memory; there is no public chunk
parameter. ``dst`` must not already hold a store.

**Important:** Train the dictionary from settled authoritative input before
building the container. Training it before preprocessing is complete wastes
compression and does not describe the final records.

### Multi-stream stores

A zrecord store is one stream; a training sample is usually several named
streams (skeleton + label + id, tokens + label, ...). Composition is a plain
Python ``dict`` passed to ``DataLoader``. There is no persistent wrapper,
manifest, directory convention or bundle mutation API. ``DataLoader`` verifies
that all streams have the same length, then gathers every stream with the same
indices.

```python
from loaderx.zrecord import Dense, Ragged
from loaderx.dataloader import DataLoader

root = "xsub/train"
with Dense.create(root + "/joint", joint.dtype, joint.shape[1:]) as s:
    s.append(joint)
with Dense.create(root + "/label", label.dtype, label.shape[1:]) as s:
    s.append(label)
with Ragged.create(root + "/token", np.int32) as s:
    s.append(seqs)

streams = {
    "joint": Dense.open(root + "/joint"),
    "label": Dense.open(root + "/label"),
    "token": Ragged.open(root + "/token"),
}
streams["joint"][0, 5, 2]            # each stream keeps its own index API
loader = DataLoader(streams, batch_size=256)
batch = next(loader)                  # {name: values}, index-aligned

loader.close()
for stream in streams.values():
    stream.close()
```

Equal-length and variable-length are both just stores — `Dense` (one
fixed-shape record per sample) and `Ragged` (variable row count per
record). The loader does not interpret either: it fetches by index and packs a
dict, so a dense stream's batch value is the stacked `(B, *item_shape)` array
and a ragged stream's is a list of per-record arrays. No padding is imposed —
densify to a fixed shape however the model needs (a plain numpy loop), or
reshape/stack in the `transform` collate.

### CPU → GPU transfer

loaderx hands over CPU batches; getting them to the accelerator is the
`transform`'s job — the one place your framework is already imported. The batch
dict is a plain `{name: numpy array}`, zero-copy on the way out, so a device
transfer is one call per stream:

```python
import torch

device = "cuda:0"
def to_device(batch):
    return {k: torch.from_numpy(v).to(device, non_blocking=True)
            for k, v in batch.items()}

streams = {
    "joint": Dense.open(root + "/joint"),
    "label": Dense.open(root + "/label"),
    "token": Ragged.open(root + "/token"),
}
loader = DataLoader(streams, transform=to_device)
for batch in loader:
    model(batch)                        # already on device
```

Call `loader.close()` when the training loop exits. The streams remain
caller-owned and should be closed at the application lifecycle boundary.

A `non_blocking=True` copy is genuinely asynchronous only when its source is
pinned. loaderx does not pin memory for you — pinning is framework-owned
(torch's `.pin_memory()`, CUDA's `cudaHostAlloc`), and a vendor-free core stops
exactly at the CPU batch. Pin in the transform what you copy:

```python
def to_device(batch):
    return {k: torch.from_numpy(v).pin_memory().to(device, non_blocking=True)
            for k, v in batch.items()}
```

JAX is the same shape — `jax.device_put` is already an asynchronous handoff on
GPU:

```python
import jax

def to_device(batch):
    return {k: jax.device_put(v) for k, v in batch.items()}
```

The transfer runs on the transform stage and never touches loaderx internals:
the copy overlaps the next batch's gather/transform, and any pinned pool is the
caller's to own and reuse. This is the entire H2D answer — there is no `pin=`
hook or device backend, because the only unified thing a multi-framework loader
can own is the CPU batch.

For practical integration examples, please refer to the **[Data2Latent](https://codeberg.org/eoelab/Data2Latent)** repository

## Benchmarks

Dense and Ragged are measured separately because they expose different
contracts, but every store table uses the same columns. `scripts/bench_dense.py`
measures fixed-shape random gather, `scripts/bench_ragged.py` measures
variable-shape records, and `scripts/bench.py` covers machine, sampler, and the
end-to-end loader comparison. Every path runs through the public Python binding,
so CFFI, NumPy allocation, and Ragged list/shape reconstruction are timed.

### Methodology

The results below are one complete qualitative pass from the same checkout on a
warm page cache. They are not three-run medians: an unexpected result is traced
separately instead of being hidden by repeated aggregation. Within each store workload every backend
receives identical source records and random index plans. Loader backends receive
the same source and seed but use their own shipped samplers, so their exact
permutations differ. Before gather timing, the store benchmark sweeps every record and
validates every planned result for exact
dtype, shape, order, and values. `logical write` and `logical gather` divide
uncompressed NumPy payload bytes by elapsed time; they measure bytes accepted or
returned by the public API, not physical storage bandwidth. Writable containers
are created before timing. `logical write` times only append/write/assignment
calls and stops when the final call returns; close, commit, format finalization
and Zrecord Header publication happen afterward. No backend requests `fsync`.
Reusable source preparation is outside that timer; in particular, ``zstd_dict``
trains its standalone dictionary first and byte-record adapters encode input
before their timed write calls.
Disk size is allocated blocks, not sparse apparent size. `krecords/s` is gather
record throughput and `p95` is the 95th-percentile latency of one random gather
batch. Results are comparable within one workload table, not across payload
distributions or geometries.
The finalized output is opened read-only before timing. Every random gather plan
is timed exactly once; output allocation, reads, decompression and reconstruction
are included, while open and close are not.
Every backend name states its actual codec; the full default set is required
rather than silently skipped when a package is missing.
Dense and Ragged use the same CHW RGB image generator, record count, batch plan
and seed. Dense fixes every image at `(3, 224, 224)`; Ragged changes only H and W.

**Machine** — one local workstation (AMD Ryzen AI 9 HX PRO 370, 12 cores / 24
threads):

| machine   | value |
|-----------|-------|
| CPU       | AMD Ryzen AI 9 HX PRO 370 w/ Radeon 890M, 1 socket, 12 cores / 24 threads |
| frequency | 605–5158 MHz |
| caches    | L1d 576 KiB, L1i 384 KiB, L2 12 MiB, L3 24 MiB |
| NUMA      | 1 node |
| memory    | 31 GiB (not limited by cgroup) |
| shared memory | 16 GiB `/dev/shm` |
| OS        | Debian GNU/Linux forky/sid, kernel 7.1.3+deb13-amd64, x86_64 |
| python    | CPython 3.14.7 (standard GIL build), numpy 2.5.2 |

The benchmark process sees all 24 threads and is not memory-limited by cgroup.
The 16 GiB shared-memory mount accommodates the four-worker, 36.8 MiB-batch torch
pipeline. Store reads run on the ordinary page cache.

### Large Vision Records

#### Fixed-Shape Dense

Zrecord against array-store alternatives: random batch gather, 2,500 CHW RGB
records, batch 256. Every image is `(3, 224, 224)` and comes from the same
spatial model used by the variable-shape benchmark.

Fixed-resolution vision records — 147 KiB per record, 36.8 MiB per batch:

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-zstd | 7206 MiB/s | 7395 MiB/s | 51.5 | 5.82 ms | 24.4 MiB | 14.69x |
| zrecord-zstdict | 30 MiB/s | 7138 MiB/s | 49.7 | 6.05 ms | 15.8 MiB | 22.74x |
| zrecord-raw | 3117 MiB/s | 9590 MiB/s | 66.8 | 4.70 ms | 358.9 MiB | 1.00x |
| npy-mmap-raw | 2314 MiB/s | 4152 MiB/s | 28.9 | 10.74 ms | 358.9 MiB | 1.00x |
| hdf5-raw | 2499 MiB/s | 1810 MiB/s | 12.6 | 25.49 ms | 359.0 MiB | 1.00x |
| hdf5-gzip | 257 MiB/s | 592 MiB/s | 4.1 | 68.37 ms | 26.1 MiB | 13.73x |
| lmdb-raw | 4291 MiB/s | 3800 MiB/s | 26.5 | 10.44 ms | 361.4 MiB | 0.99x |
| arrow-ipc-raw | 3070 MiB/s | 2965 MiB/s | 20.7 | 16.28 ms | 358.9 MiB | 1.00x |
| arrow-ipc-zstd | 700 MiB/s | 171 MiB/s | 1.2 | 254.38 ms | 22.6 MiB | 15.86x |
| parquet-raw | 1700 MiB/s | 470 MiB/s | 3.3 | 86.56 ms | 358.9 MiB | 1.00x |
| parquet-zstd | 613 MiB/s | 150 MiB/s | 1.0 | 289.02 ms | 22.6 MiB | 15.86x |
| arrayrecord-raw | 2012 MiB/s | 1855 MiB/s | 12.9 | 24.12 ms | 359.2 MiB | 1.00x |
| arrayrecord-zstd | 1024 MiB/s | 1531 MiB/s | 10.7 | 34.26 ms | 25.1 MiB | 14.32x |
| tiledb-raw | 692 MiB/s | 668 MiB/s | 4.7 | 59.75 ms | 359.0 MiB | 1.00x |
| tiledb-zstd | 1253 MiB/s | 1478 MiB/s | 10.3 | 27.34 ms | 26.9 MiB | 13.36x |

At 147 KiB per record, Zrecord-raw reaches 9.4 GiB/s and is 2.3x npy-mmap-raw;
plain zstd gathers at 7.2 GiB/s while reducing the corpus 14.69x. LMDB and Arrow
IPC are competitive raw record
stores, while codecs tied to whole IPC batches or Parquet row groups pay read
amplification on random gathers. Dense demonstrates that
typed record ownership and per-record compression do not turn fixed tensors
into an object-store slow path.

#### Variable-Shape Ragged

This workload contains 2,500 variable-resolution CHW RGB images. Height and
width are independently lognormal and clipped to 64..512 (observed medians
223 and 224), totaling 417.1 MiB of logical uint8 payload. Each of 50 random
batches contains 256 records. Every backend persists payload plus exact shape
and must return an ordered `list[np.ndarray]` of `(3, H, W)` arrays; a flat byte
list or a one-dimensional variable-length abstraction is not enough.

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-zstd | 2220 MiB/s | 5666 MiB/s | 34.2 | 8.41 ms | 26.9 MiB | 15.52x |
| zrecord-zstdict | 28 MiB/s | 5681 MiB/s | 34.3 | 8.55 ms | 17.7 MiB | 23.58x |
| zrecord-raw | 1460 MiB/s | 6806 MiB/s | 41.1 | 7.58 ms | 417.2 MiB | 1.00x |
| hdf5-raw | 1610 MiB/s | 1108 MiB/s | 6.7 | 45.80 ms | 418.0 MiB | 1.00x |
| hdf5-gzip | 229 MiB/s | 114 MiB/s | 0.7 | 415.95 ms | 29.9 MiB | 13.94x |
| lmdb-raw | 4126 MiB/s | 6837 MiB/s | 41.2 | 8.14 ms | 422.2 MiB | 0.99x |
| arrow-ipc-raw | 3213 MiB/s | 4251 MiB/s | 25.6 | 12.90 ms | 417.2 MiB | 1.00x |
| arrow-ipc-zstd | 706 MiB/s | 177 MiB/s | 1.1 | 266.37 ms | 25.9 MiB | 16.10x |
| parquet-raw | 2219 MiB/s | 505 MiB/s | 3.0 | 92.79 ms | 417.2 MiB | 1.00x |
| parquet-zstd | 728 MiB/s | 158 MiB/s | 1.0 | 296.60 ms | 25.9 MiB | 16.10x |
| arrayrecord-raw | 1650 MiB/s | 2571 MiB/s | 15.5 | 18.34 ms | 417.6 MiB | 1.00x |
| arrayrecord-zstd | 941 MiB/s | 1420 MiB/s | 8.6 | 37.14 ms | 27.2 MiB | 15.31x |
| tiledb-raw | 443 MiB/s | 49 MiB/s | 0.3 | 920.09 ms | 417.2 MiB | 1.00x |
| tiledb-zstd | 727 MiB/s | 124 MiB/s | 0.7 | 375.98 ms | 27.0 MiB | 15.46x |

Zrecord-raw and LMDB are effectively tied in this pass; Arrow IPC is
the strongest raw typed-file alternative. Zrecord-zstd delivers 5.5 GiB/s of
logical payload while reducing the corpus to 26.9 MiB. HDF5, Arrow IPC, Parquet,
ArrayRecord and TileDB
show the same framework/codec tradeoffs in both tables; compressed batch, chunk
and row-group formats pay read amplification on random records.

The shared generator makes compression ratios directly comparable across
contracts: Zrecord zstd is 14.69x Dense versus 15.52x Ragged, and zstdict is
22.74x versus 23.58x. The remaining difference comes from the H/W distribution
and Ragged shape metadata, not a different image entropy model.

### Small Token Records

Both token workloads come from the same real corpus: WikiText-103 raw train,
tokenized with GPT-2 and stored as `int32` IDs. Preparation is outside every
measurement. `scripts/prepare_tokens.py` preserves nonempty text boundaries,
combines fragments shorter than 16 tokens, and splits records at 512 tokens into
`tokens.npy` plus `offsets.npy`; both benchmark scripts mmap those files.

#### Fixed Token Blocks

The Dense workload ignores text boundaries and packs the stream into 200,000
fixed `int32[512]` records: 2 KiB per record and 390.6 MiB logical payload.
Each of 100 random batches gathers 256 records.

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-zstd | 346 MiB/s | 1681 MiB/s | 860.9 | 0.38 ms | 193.2 MiB | 2.02x |
| zrecord-zstdict | 41 MiB/s | 1892 MiB/s | 968.8 | 0.33 ms | 153.8 MiB | 2.54x |
| zrecord-raw | 2477 MiB/s | 6261 MiB/s | 3205.5 | 0.11 ms | 393.7 MiB | 0.99x |
| npy-mmap-raw | 2024 MiB/s | 7897 MiB/s | 4043.3 | 0.08 ms | 390.6 MiB | 1.00x |
| lmdb-raw | 1022 MiB/s | 1066 MiB/s | 545.9 | 0.53 ms | 786.3 MiB | 0.50x |
| arrow-ipc-raw | 2744 MiB/s | 207 MiB/s | 105.8 | 2.60 ms | 390.8 MiB | 1.00x |
| arrayrecord-raw | 905 MiB/s | 149 MiB/s | 76.4 | 6.33 ms | 401.4 MiB | 0.97x |
| arrayrecord-zstd | 114 MiB/s | 160 MiB/s | 82.0 | 4.83 ms | 201.3 MiB | 1.94x |

The contiguous NumPy baseline is strongest when the whole corpus is one fixed
typed matrix. Zrecord-raw reaches 3.21 Mrecords/s while retaining independent
record semantics; the per-record zstd codecs halve disk and still return
0.86–0.97 Mrecords/s. LMDB's B-tree/page overhead is visible in both throughput
and disk.

#### Variable Token Sequences

The Ragged workload keeps 200,000 real text records of 16..512 tokens: p10 28,
median 129, mean 138.8, p90 254, totaling 105.9 MiB. It uses the same 256-record,
100-batch random plan and every backend must return ordered `list[np.ndarray]`
with exact `int32` values and original one-dimensional shapes.

| backend | logical write | logical gather | krecords/s | p95 | disk | ratio |
|---|---:|---:|---:|---:|---:|---:|
| zrecord-zstd | 105 MiB/s | 225 MiB/s | 426.5 | 0.76 ms | 67.8 MiB | 1.56x |
| zrecord-zstdict | 37 MiB/s | 277 MiB/s | 526.3 | 0.63 ms | 49.4 MiB | 2.14x |
| zrecord-raw | 457 MiB/s | 325 MiB/s | 616.1 | 0.46 ms | 112.0 MiB | 0.95x |
| lmdb-raw | 887 MiB/s | 106 MiB/s | 200.7 | 1.34 ms | 153.0 MiB | 0.69x |
| arrow-ipc-raw | 2230 MiB/s | 39 MiB/s | 74.1 | 4.48 ms | 109.9 MiB | 0.96x |
| arrayrecord-raw | 399 MiB/s | 24 MiB/s | 45.7 | 8.34 ms | 118.8 MiB | 0.89x |
| arrayrecord-zstd | 59 MiB/s | 27 MiB/s | 51.1 | 7.79 ms | 76.2 MiB | 1.39x |

Here the record contract, not bulk byte bandwidth, is the useful scale.
Zrecord's three codecs return 427–616 krecords/s with less than 0.8 ms p95;
the dictionary gives the best disk ratio and is slightly ahead of plain zstd in
this pass.

### Sampler

Index generation on its own, IID (with replacement), 1M index space, against
NumPy's modern API. The µs-scale figures fluctuate with box load; this pass shows
an approximately 2.1–2.3x margin across batch sizes.

| sampler | batch | per batch | vs default_rng |
|---|---:|---:|---:|
| numpy default_rng | 256 | 4.6 µs | 1.00x |
| **zsampler** | 256 | 2.0 µs | **2.34x** |
| numpy default_rng | 1024 | 5.5 µs | 1.00x |
| **zsampler** | 1024 | 2.5 µs | **2.25x** |
| numpy default_rng | 8192 | 20.6 µs | 1.00x |
| **zsampler** | 8192 | 9.9 µs | **2.09x** |

### End-to-End DataLoader

The full input pipeline comparison (sample, fetch, collate, hand over a batch)
uses exactly the Dense vision source above: 2,500 `(3,224,224)` uint8
records from `make_vision_records`, 4 high-level workers, and batch 256
(36.8 MiB). After warmup, throughput and memory are collected during one
200-batch qualitative pass. `peak PSS` sums proportional
set size across the process tree, apportioning mapped shared and copy-on-write
pages instead of counting each once per worker. It does not include ordinary
kernel page-cache pages used by pread, while resident mmap pages are attributed
to the mapping process, so it is a process-mapping diagnostic rather than total
pipeline physical memory. `aggregate RSS` deliberately sums
each process's full resident set: on Linux it double-counts shared/COW pages,
which explains process-tree RSS inflation but is neither physical memory nor a
projection of Windows committed memory. The explicit spawn row is the relevant
no-fork control; Windows itself still requires a native run. Torch fork is kept
because it is the normal Linux mode, while spawn exposes the ownership model
used on platforms without fork.
Grain setup remains optional through `--only grain`; the published command
selects it explicitly in the same workload matrix.

`storage` is the actual backing store used by each pipeline. This is an
end-to-end systems comparison, not a scheduler-only comparison over one shared
storage layer: Torch reads read-only NumPy mmap files, Loaderx reads Zrecord,
and Grain reads ArrayRecord.

| loader | model | storage | batches/s | p95 | steady PSS | peak PSS | peak RSS |
|---|---|---|---:|---:|---:|---:|---:|
| **loaderx** | threads | zrecord-zstd | 147.0 | 15.35 ms | 985 MiB | 986 MiB | 990 MiB |
| loaderx-raw | threads | zrecord-raw | 210.3 | 13.69 ms | 987 MiB | 987 MiB | 991 MiB |
| torch | fork | npy-mmap-raw | 100.1 | 32.96 ms | 1748 MiB | 1841 MiB | 6387 MiB |
| torch-spawn | spawn | npy-mmap-raw | 106.8 | 32.14 ms | 2833 MiB | 2890 MiB | 4869 MiB |
| grain | processes | arrayrecord-zstd | 40.3 | 109.40 ms | 1834 MiB | 1917 MiB | 2036 MiB |

At 36.8 MiB per batch the per-batch gather dominates the tiny sampler cost, and
the transform threads overlap Python-side collation with the next gather. The
memory is the source, Zrecord container and bounded in-flight batches. loaderx prefetches in
threads inside one process, so workers share one interpreter, one NumPy runtime
and one set of gather buffers. With source geometry and entropy held constant,
raw is 1.43x compressed loaderx; compressed loaderx is 1.47x Torch fork, 1.38x
Torch spawn and 3.65x Grain, while raw is 2.10x, 1.97x and 5.22x faster.
Torch's aggregate RSS is high because
Linux fork mappings are counted repeatedly; it is not a total-memory ratio
against Zrecord's unaccounted page cache. The explicit `torch-spawn` row removes
fork/COW dependence. Because this Dataset keeps only mmap paths, spawn does
not copy the full corpus into every worker; a Windows Dataset holding Python
lists or in-memory arrays would be a different, deliberately harsher workload.

The Torch-only worker sweep runs each count once. Worker 0 is an in-process
baseline (145.5–153.8 batches/s and about 1485 MiB peak PSS in the two equivalent
rows), so the process-context comparison starts at one worker:

| workers | fork batches/s | fork peak PSS | fork peak RSS | spawn batches/s | spawn peak PSS | spawn peak RSS |
|---:|---:|---:|---:|---:|---:|---:|
| 1 | 42.5 | 1678 MiB | 2483 MiB | 44.9 | 1891 MiB | 2128 MiB |
| 2 | 69.4 | 1723 MiB | 3792 MiB | 71.8 | 2241 MiB | 3058 MiB |
| 4 | 104.4 | 1866 MiB | 6365 MiB | 102.0 | 2926 MiB | 4893 MiB |
| 8 | 117.8 | 2016 MiB | 11320 MiB | 103.6 | 4218 MiB | 8351 MiB |

Spawn peak PSS grows from 1891 to 4218 MiB as workers rise from one to eight,
while fork grows from 1678 to 2016 MiB because it retains COW sharing. At eight
workers spawn uses 2.09x fork's peak PSS and throughput has already flattened.
This demonstrates no-fork memory pressure; it is not labeled OOM because this
31 GiB machine completed the run. Fork aggregate RSS grows faster because Linux
counts shared/COW mappings in every process, so RSS is diagnostic rather than
physical memory.

**CPU-heavy transform solutions.** The same Python per-sample transform exposes
the GIL bottleneck on standard CPython. Numba is an explicit solution, not the
default: `--transform numba-nogil` compiles the equivalent batch transform with
`nogil=True`. The other explicit solution runs the Python transform on
free-threaded CPython. Numba compilation is warmed before timing. Store
microbenchmarks are not repeated under free-threaded Python because Loaderx has
no separate no-GIL Store implementation; the Store table above applies to both.

| loader | GIL Python | GIL + Numba nogil | free-threaded Python | Numba gain | free-threaded gain |
|---|---:|---:|---:|---:|---:|
| **loaderx** | 44.5 batches/s | 107.0 batches/s | 73.7 batches/s | **2.40x** | **1.66x** |
| loaderx-raw | 47.2 batches/s | 123.5 batches/s | 82.1 batches/s | **2.62x** | **1.74x** |

Peak PSS for compressed/raw was 1204/1189 MiB with GIL Python,
1290/1262 MiB with Numba, and 1190/1190 MiB with free-threaded Python. These are
two deployment solutions to the transform bottleneck, not claims that Store
itself was optimized for either runtime.

**Conclusion** — why the numbers look like this.

**Every hot path is batched natively.** Zsampler draws a whole batch of indices;
Dense gathers and decompresses a whole fixed-shape batch in one CFFI call;
Ragged gathers each shape-prefixed record once before Python validates its
prefix and reconstructs the exact arrays. The speedup is not
bought with sampling shortcuts: the IID draw is unbiased like NumPy's (Lemire
with rejection, so uniformity costs nothing over a real index space).

**The layouts match what a training loader does.** Zrecord is built for random
record access: Dense gathers fixed-width records directly into one ndarray;
Ragged restores independently shaped records from inline shape/payload
entries. Array stores are built primarily for contiguous scans, so a scattered
batch fights their layout. A dense raw gather fans out across the shared
Executor budget where NumPy fancy indexing is one thread; Ragged instead trades some raw
specialization for compression and a complete variable-shape persistence model.

**Compression is in the storage kernel, and there is one codec.** `zrecord-zstd` is not
"storage plus a codec": the layout, the multi-core decompress and the GIL-free
copy are one path, so turning compression on costs part of a margin, not an
order of magnitude. The ratio is the data, not the
format: in the current Dense structured-vision workload, plain zstd reaches
14.69x and the balanced dictionary reaches 22.74x.

**Loader results combine architecture and storage.** loaderx uses threads and
never ends an epoch, so a step pays no IPC and never waits on an epoch boundary;
torch uses finite shuffled epochs, worker processes and shared-memory handoff.
Here compressed loaderx is 1.47x Torch fork, 1.38x Torch spawn and 3.65x Grain;
raw loaderx is 2.10x, 1.97x and 5.22x faster, respectively.
Storage also differs per loader — each reads from what it was
built for — so the loader table is a different comparison from either store
table, not a rerun.
The one crack in the thread model is a CPU-heavy Python transform, which the GIL
serializes. Numba `nogil=True` and free-threaded Python are measured as two
explicit solutions rather than silently changing the default transform.

**What these numbers do not claim.** Everything runs with a warm page cache: this
measures the access path, not cold storage or disk. `disk` is allocated
blocks, and zrecord files grow to their written frontier. `logical write`
measures append/write calls after writer setup and before close/finalization; it
does not benchmark durability, stronger transactional guarantees, or reusable
dictionary training. Arrow IPC and Parquet use 256-record groups, so random
batches pay their real group-level read amplification. Ragged Zrecord arrays are
views into one batch allocation, while most byte-store adapters return
independent copies. This is a single
qualitative pass —
the µs-scale sampler timings and the loader `batches/s` fluctuate with box load
(on these 12 cores the compressed loader trails the raw one by about 25%), so
treat the absolute numbers as ballpark and the cross-backend margins as the
signal.

### Reproduction

```bash
.venv/bin/python scripts/bench_dense.py
.venv/bin/python scripts/bench_ragged.py

.venv/bin/python scripts/prepare_tokens.py wiki.train.tokens /tmp/wikitext-gpt2
.venv/bin/python scripts/bench_dense.py --workload tokens \
  --token-corpus /tmp/wikitext-gpt2 --records 200000 --batches 100
.venv/bin/python scripts/bench_ragged.py --workload tokens \
  --token-corpus /tmp/wikitext-gpt2 --records 200000 --batches 100

.venv/bin/python scripts/bench.py sampler
.venv/bin/python scripts/bench.py loader \
  --only loaderx,loaderx-raw,torch,torch-spawn,grain --workers 4
.venv/bin/python scripts/bench.py loader --only torch,torch-spawn \
  --workers 0,1,2,4,8
.venv/bin/python scripts/bench.py loader \
  --only loaderx,loaderx-raw --workers 4 --transform python-loop
.venv/bin/python scripts/bench.py loader --only loaderx,loaderx-raw \
  --workers 4 --transform numba-nogil

.venv-t/bin/python scripts/bench.py loader --only loaderx,loaderx-raw \
  --workers 4 --transform python-loop
```

`prepare_tokens.py` also accepts Hugging Face WikiText Parquet shards directly.
The published run used `Salesforce/wikitext`, config `wikitext-103-raw-v1`,
revision `refs/convert/parquet`, train shards `0000.parquet` then
`0001.parquet`; their SHA-256 values are respectively
`74da360f23826045b3e6ac6375411fdb15f003030aa74f2596ed08b857cb9212` and
`ba090ac30dbf5461e8dcbdd1a1b8e6f3cf9c2c756d64f0c1220450acd514f720`.
The focused token defaults omit formats that are already represented in the
larger vision matrix; `--only` can select any registered backend explicitly.
The loader rows used Numba 0.67.0, Torch 2.13.0 and Grain 0.2.18; all benchmark dependencies
are pinned in `scripts/requirements-bench.txt`. Temporary stores used the ordinary
disk-backed `/tmp` filesystem, not `/dev/shm`.

## Real-data verification: NTU RGB-D skeletons

The vision tables above are synthetic; the token tables use real WikiText-103.
As a separate historical ground-truth check,
loaderx was run end to end on **NTU RGB-D skeleton data** — 114,480 raw `.skeleton` files,
120 action classes, 25 joints — processed into the ST-GCN `N C T V M` layout
(per-sample `(3, 300, 25, 2)` float32) for the `xsub`/`xview` protocols. The
`.npy` outputs of the standard preprocessing pipeline were treated as ground
truth. This verification was not rerun with the synthetic benchmarks above;
its throughput is retained as a separate historical 12-core result.

**Correctness — the read path is bit-exact against the ground truth:**

* Full scan of all **228,356 records** (joint float32 + label int64, all four
  splits) through `Dense`: byte-for-byte identical to the reference npy.
* A `DataLoader` over joint + label + an index stream, run under all three
  sampler modes (`sequential`, `iid`, `cyclic`): every received batch is
  bit-exact to the ground truth at its own declared indices, and the streams
  stay index-aligned.
* Sampler semantics hold on real index spaces: `sequential` walks in order,
  `cyclic` draws a full cycle without replacement, `iid` is deterministic per
  seed.

**Storage — zstd on this data:**

| store                 | on disk | ratio |
|-----------------------|---------|-------|
| npy (raw float32)     | 6.4 GB  | 1.00x |
| zrecord raw           | 6.86 GB | 1.00x |
| zrecord zstd          | 0.79 GB | 8.66x |
| zrecord zstd_dict     | 0.75 GB | 9.13x |

Sizes above are for one split (`xview/val`, 38,132 records); across all four
splits the zstd `joint` stores total 4.97 GB against 41 GB of raw npy (~8x).

**Throughput** (180 KB per record, warm page cache, 12 physical cores):

| path                                | throughput |
|-------------------------------------|-----------|
| random-batch gather, zstd store     | 4.1–4.5 GiB/s |
| same, npy-mmap fancy indexing       | 0.6–1.3 GiB/s |
| DataLoader, 4 prefetch threads      | 5.7–6.6 GiB/s (123–144 batches/s) |

zstd decompression reads ~8x fewer bytes than raw storage, so the compressed
store gathers *faster* than the raw one (zstd 4587 MiB/s vs raw 1792 MiB/s on
the same split).

**The npy intermediate is optional.** Parse the skeleton files in parallel and
feed each fixed-shape ndarray produced by the parser directly to
`Dense.append`. This writes the store in one pass with no npy staging or
second read; zrecord bounds its compression working memory independently.

## Current Limitations
* Single-host only; multi-host training is not supported.
* A single sample must be at most 2 GiB (2^31 bytes). There is no fixed record
  count: `length` is a u64 and the record table grows on demand. Practical store
  size is bounded by disk and the platform's positional file-offset range.
* Metadata is read and written as the host's struct layout, so a store carries
  the host's byte order and is not portable to a machine of the opposite
  endianness. Every published platform is little-endian, so this only matters
  if you build for
  one yourself.

## Build
```
zig build                       # host shared objects, into loaderx/lib/
zig build test                  # native store suite, in both Debug and ReleaseFast
python3 scripts/test_loaderx.py # Python integration suite against the real build
uv pip install --python .venv/bin/python -r scripts/requirements-bench.txt
.venv/bin/python scripts/bench_dense.py  # fixed-shape store comparison
.venv/bin/python scripts/bench_ragged.py # variable-length store comparison
.venv/bin/python scripts/bench.py        # machine, sampler, and dense loader layers
```

The Zig side is tested for behaviour only; throughput is measured from Python,
through the binding a client actually uses. Optional benchmark contenders are
skipped when not installed. See [Benchmarks](#benchmarks).

### Publishing
Zig cross-compiles every target from one machine, so releases need no CI matrix:

```
zig build dist                    # every platform, into zig-out/dist/<wheel tag>/
python3 scripts/build_wheels.py   # one wheel per platform, plus the sdist
```

The dist directories are named after their Python wheel platform tag, so the tag
mapping lives in exactly one place (`dist_targets` in `build.zig`). glibc and
macOS minimums are pinned in the target triple, which is what makes
`manylinux_2_17` and `macosx_11_0` honest rather than aspirational. Each wheel is
checked after packing: it must carry this platform's libraries and no others.

The sdist ships sources only. Installing from it runs `zig build` through
`setup.py`, so it needs the Zig compiler; wheel users never hit that path.

---

# Zsampler
Index Generator: a high-performance sampler implemented in Zig. Every mode is a
pure function of `(seed, step)`, so a run resumes exactly by seeking to a step —
there is no epoch to track, in keeping with the endless step-based loader.

```python
from loaderx.zsampler import Sampler

sampler = Sampler(1_000_000, 256, Sampler.Mode.IID, seed=42)
indices = sampler.next()       # borrowed until this sampler's next draw
saved = indices.copy()         # retain across draws only when needed
```

`next()` and iteration return a view of one reusable `uint64` batch buffer.
The contents stay unchanged until the next explicit draw from that Sampler;
copy only plans that must outlive it. DataLoader consumes each view synchronously
before drawing again.

1. **Sequential** — traverse the index space in order through a fixed-size sliding
   window, treating the space as a circular queue so the tail never truncates.
2. **IID** — draw each index uniformly at random with replacement. Unbiased
   (Lemire with rejection), matching NumPy. Simplest, but coverage is uneven over
   any short run.
3. **Cyclic** — without replacement, round-robin. Each cycle traverses a fresh
   permutation of the whole index space, so within a cycle every record appears
   exactly once and no batch repeats an index — coverage is even by construction,
   which keeps how often each sample is seen uniform. The permutation is a
   stateless bijection (a small Feistel network over the index space, brought
   into range by cycle-walking), so a million-record shuffle materializes nothing
   the size of the dataset and reshuffling each cycle is free. Every batch is
   exactly `batch_size`: an endless step-based loader has no final partial batch
   to special-case, so when `batch_size` does not divide the length the cycle's
   remainder is dropped — a different remainder each cycle, since the permutation
   changes, so every record is still reached over time.

---

# Zrecord
Zrecord is loaderx's rebuildable typed record container. Its private native
runtime is the byte-oriented engine beneath the public `Dense` and `Ragged`
contracts.

The private Python CFFI surface is kept together in `loaderx/_store.py` because
both contracts share one `libstore`, error model, lifecycle and opaque Store
handle. Python owns all schema semantics; Zig stores the opaque schema bytes in
``meta.zr``. In Zig,
`src/store.zig` adapts each Dense stride or Ragged offsets call to the shared
physical engine and is the sole C ABI export/composition root. `zrecord.py`
remains the unified Python-facing API.

**Trust boundary.** Python and Zig are one zrecord implementation, not two
independently supported products. `loaderx/_store.py`, the C ABI in
`src/store.zig`, and the native handles are private implementation details; no
defensive-validation contract is provided to code that calls them directly.
Public inputs are normalized and validated once, in whichever
half can express the rule most simply, and the internal Python→CFFI→Zig call
then trusts that contract instead of repeating it at every layer. Python owns
the creator/reader capability model and chooses append, gather, creator Header
sync, or reader handle release; the Zig engine stores no writable/reader mode. Native
create/open still select read-write/exclusive or read-only/shared file handles,
because those are OS access and locking mechanics rather than API capabilities.
This is not a
license to trust storage or the operating system: native code still validates
    persisted addresses and lengths, buffer bounds and integer overflow, short or
    failed I/O, codec output, locking, and commit ordering. Those checks protect
    normal I/O behavior, basic malformed-store rejection and native memory safety; checks that only
defend against bypassing the public Python API do not belong in zrecord.

1. `RecordEngine` is an unordered physical store made of N records. Records are
   independent and carry no ordering, so every index and slice operation is
   equivalent to a gather.
2. It hands the container layer a dense index space: records are exactly
   `0..N`. Named streams are composed dynamically by a plain Python dict;
   `DataLoader` validates that the independent containers have equal lengths.
3. The engine reads and writes byte ranges. Python owns the record schema and
   selects Dense or Ragged geometry; the private ABI receives only the runtime
   byte geometry. Dense stores persist
   one fixed-width physical record per logical record. Ragged stores
   persist one variable-width physical record per logical record:
   `[u64le ndim][u64le dims...][payload]`. Shape and payload therefore share one
   location, codec frame, and append publication.
4. The IO model (`append | read`) is batch-oriented and
   shape-agnostic. Per-call adapters expose record boundaries through a compile-time
   source interface; the engine has one append operation and carries no Dense,
   Ragged, dtype, or array-shape semantics.
   A single-record operation is just the `batch_size == 1` case.
5. The engine owns its temporary memory internally — allocation and release are explicit.
6. A store has exactly **one** codec in its native physical header, fixed at
   creation and immutable afterwards. Every record is compressed and decompressed
   independently:

```
| tag   |  name     | algorithm                              |
|-------|-----------|----------------------------------------|
|   0   |  raw      | none                                   |
|   1   |  zstd     | zstd (plain, level 3)                  |
|   2   |  zstdict  | zstd with a trained dictionary (level 19) |
```

7. Compression is transparent to the client:
    * Compression runs concurrently across the shared Executor budget. A compressed store never
      falls back to raw: each record is stored as the codec's output, even when
      an incompressible record's frame is larger than its input — write `raw`
      if the data does not compress.
    * Decompression writes straight into the caller's destination memory
      (`gather`), with no intermediate buffer and no extra copy.
    * zstd is the one transparent codec — faster than Deflate at both ends and a
      better ratio, so there is no reason to carry a second. It is vendored C,
      built for every platform by Zig, so the one-wheel-per-platform story is
      unchanged.
    * `zstd_dict` additionally trains one dictionary on a sample of the data
      (stored as `dict.zr`) and compresses every record against it. Because each
      record is still independent, random access is unchanged — but the
      dictionary carries the structure shared across records, which per-record
      compression cannot see. On many small, similar records (image tiles, token
      sequences) this is a large win: the current Dense structured-vision set
      is 14.69x with plain zstd and 22.74x with the balanced dictionary. The dictionary is loaded once on
      open and shared, lock-free, across all reader threads. The dictionary size
      is chosen from the `DICT_TIERS` presets (see Codec notes).
    * A `zstd_dict` store needs its dictionary to read every record; a `raw` or
      `zstd` store rejects an unexpected dictionary as malformed state.

## Persistence format
The 2.0 clean break does not change the current on-disk format. That format is
the current implementation only: there is no compatibility layer, migration,
version dispatch, checksum, or recovery facility. Zrecord is not the authority
for irreplaceable data. Keep authoritative source data and reproducible build
scripts; after an interrupted build, storage failure, implementation change, or
content change, rebuild a complete container at a new path.

Native storage uses a fixed file set:
```
store/
  ├── meta.zr      4096-byte static Header/schema page + RecordLoc table
  ├── data.zr      payload stream
  └── dict.zr      zstd dictionary (only in dict stores)
```

### Metadata (meta.zr)

Files are read and written **positionally** — pread/pwrite at computed offsets,
no mmap. `meta.zr` starts with one fixed 4096-byte static page: a naturally
aligned 32-byte Header, then the opaque MsgPack schema and unused zero padding.
An array of 16-byte `RecordLoc`s starts at offset 4096. Record `i` is one
pread/pwrite at `4096 + i * 16`; there is no variable table base, segment
mapping, or rollover fd table.

**1. Python schema** — bytes `32..32+schema_length` are exactly one immutable
MsgPack object. Dense stores contain only ``dtype`` and ``item_shape``; Ragged
stores contain only ``dtype``. Native create persists these bytes together with
the physical container but does not decode them. Open acquires the native lifetime
lock before copying the schema to Python for validation, so schema and physical
metadata are one locked snapshot. Dense record width is derived once from
dtype/item_shape and passed to the native handle as runtime geometry; it is not
independently persisted as a second authority. There is no format version or
legacy kind dispatch.

**2. Physical header** — the first 32 bytes of `meta.zr`. The format
deliberately carries no payload or metadata checksum.

* `codec` is the store's one compression method, stamped at creation and
  immutable — there is no per-record tag anywhere.
* `length` (u64) is the physical record count; it equals logical length for both
  dense stores and inline ragged stores.
* `tail_offset` (u64) is the absolute committed frontier in `data.zr`.
* `schema_length` (u32) is the occupied prefix of the static schema area and
  must be in `1..4064`.

```zig
const Codec = enum(u8) { raw = 0, zstd = 1, zstdict = 2, _ };
const Header = extern struct {
    length: u64,
    tail_offset: u64,
    schema_length: u32,
    reserved: [11]u8,
    codec: u8,
};
```

**3. Record table** — contiguous 16-byte entries start at offset 4096 in
`meta.zr` and grow as location runs are written. `offset` is an absolute byte offset in `data.zr`;
`phys_length`/`logic_length` are the stored and original sizes. The
codec is not here: it is the header's, so a record is stored exactly the way the
store is declared.

```zig
const RecordLoc = extern struct {
    offset: u64,
    phys_length: u32,
    logic_length: u32,
};
```

There is no liveness flag. Every entry below `length` is a record.

**4. No fixed record-count cap.** The table and payload stream grow naturally in
their fixed files. The practical bounds are the u64 count, supported positional
file offsets, 2 GiB per record, and disk.

## Executor

**1. Write.** Writes are append-only; everything else is offset redirection.
The codec is immutable store state, so `append` and `gather` dispatch once at
their entry points into separate raw, zstd, or zstd-dictionary implementations.
Their contexts and workers are deliberately not unified: only validation,
location bounds, locking, and final publication are shared.
Geometry is equally explicit across the whole stack: Python derives Dense
record width from its schema and passes it to each fixed-stride operation,
while `Ragged` supplies offsets for its shape-prefixed records.
The private ABI turns those inputs into compile-time record sources and
destinations; the engine has one append and one gather operation. Its shared
opaque handle remains private and carries no typed-store geometry.
* Compressed append: workers claim individual records and reuse one operation-local
  frame buffer per task, reserve physical offsets in completion order through a short frontier
  lock, and issue positional payload writes in parallel. Logical IDs remain in
  the loc table, so physical completion order does not change random gather.
  The caller waits for every payload before publishing the new locations and
  in-process length. Append does not overwrite the on-disk Header and is
  intentionally lazy. Writer ``close()`` writes the current Header and closes
  the files, making the page-cache state available for read-only open.
  Every compressed frame owns one absolute range in `data.zr`; positional writes extend the file to the
  current frontier. Python budgets the process-wide executor at three quarters
  of the logical CPUs available to the process, leaving headroom for packing,
  transforms, and the caller without encoding a platform-specific thread count.
  Each producer configures its CCtx or shared immutable CDict once, then starts
  every independent record frame with `ZSTD_compress2`.
* Raw append preserves the stronger invariant already supplied by Python: every
  record in one Dense or packed Ragged batch is a boundary inside one contiguous
  source buffer. After locating the records, the engine consumes that buffer in
  plan order and passes each contiguous planned range directly to
  `writePositionalAll` on `data.zr`. There is no per-record iovec construction,
  byte-budget flush, payload copy, or platform-specific syscall path. Zig's
  `std.Io` handles short writes and maps the same positional operation to POSIX
  and Windows implementations.
**2. Read.** Fill the destination memory concurrently, in place from the Python
side (executed on async threads).
* Committed records are immutable and `length` is published through an atomic.
  The fixed `meta.zr` and `data.zr` handles require no rollover fd-table
  synchronization; record I/O itself stays lock free.
* Every record is read at the offset its table entry records — the record table
  is addressed by pure arithmetic, so random access is one pread for the
  location and one for the bytes, with no batching assumptions about layout.
  Each lane reads one location and immediately reads/decompresses that record;
  there is no separate metadata phase or sequential-run special case. Compressed
  records use a per-lane staging buffer and decode in place into the destination.

**3. Internal fan-out.** `Io.Group.async` fans work out up to the executor lane
budget; lanes for which the runtime cannot reserve concurrency run inline on the
calling thread. Python configures the process-level budget as
`max(physical cores, logical cores * 3 / 4)`, using platform topology where
available.
* Lanes receive contiguous blocks rather than a strided subset, keeping each
  worker's reads and writes sequential.
* Each lane creates one zstd context (`ZSTD_CCtx` to write, `ZSTD_DCtx` to
  read) and reuses it across every record it handles, rather than paying that
  setup per record. The dictionary (`ZSTD_CDict`/`ZSTD_DDict`) is immutable, so
  all lanes share one, lock-free.
* Decompression writes straight into the caller's destination buffer, so there
  is no intermediate copy.

**4. File access.**
* Metadata: one naturally growing `meta.zr`, containing the fixed Header/schema
  page and loc table.
* Payload: one naturally growing `data.zr`, accessed concurrently through
  `readPositionalAll`/`writePositionalAll`. No path depends on filesystem
  sparse-file support.

**Execution model.** Opened readers are immutable, so calls on the same reader may
gather concurrently. Creator appends are synchronous and native Storage
serialization protects their physical commit. `close()` requires a quiescent
handle; it is not concurrent with append or gather. Each native append or gather
fans out internally across the shared Executor. A creator holds a lifetime, nonblocking
exclusive lock on `meta.zr`; opened readers hold shared locks, so multiple
handles and processes may consume one completed container concurrently.
