Metadata-Version: 2.5
Name: deltacat-io-core
Version: 0.1.29a1
Summary: Shared local IO execution layer for DeltaCAT read/write clients.
Author: Ray Team
License-Expression: Apache-2.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: <3.13,>=3.10
Requires-Dist: boto3<2,>=1.41.0
Requires-Dist: cryptography<47,>=46
Requires-Dist: fsspec
Requires-Dist: msgpack~=1.0.7
Requires-Dist: numpy<2,>=1.23
Requires-Dist: pyarrow<24,>=17
Requires-Dist: typing-extensions>=4.6.1
Provides-Extra: all
Requires-Dist: av==15.0.0; extra == 'all'
Requires-Dist: daft==0.5.22; extra == 'all'
Requires-Dist: fastavro; extra == 'all'
Requires-Dist: pandas==2.2.3; extra == 'all'
Requires-Dist: polars==1.29.0; extra == 'all'
Requires-Dist: pylance>=0.37.0; extra == 'all'
Requires-Dist: pyyaml<7,>=6; extra == 'all'
Requires-Dist: zarr<4,>=3; extra == 'all'
Provides-Extra: daft
Requires-Dist: daft==0.5.22; extra == 'daft'
Provides-Extra: dataloader
Requires-Dist: av==15.0.0; extra == 'dataloader'
Requires-Dist: polars==1.29.0; extra == 'dataloader'
Provides-Extra: io
Requires-Dist: fastavro; extra == 'io'
Provides-Extra: lance
Requires-Dist: pylance>=0.37.0; extra == 'lance'
Provides-Extra: lerobot
Requires-Dist: av==15.0.0; extra == 'lerobot'
Requires-Dist: polars==1.29.0; extra == 'lerobot'
Provides-Extra: packds
Requires-Dist: av==15.0.0; extra == 'packds'
Provides-Extra: pandas
Requires-Dist: pandas==2.2.3; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: polars==1.29.0; extra == 'polars'
Provides-Extra: umi
Requires-Dist: av==15.0.0; extra == 'umi'
Requires-Dist: polars==1.29.0; extra == 'umi'
Requires-Dist: pyyaml<7,>=6; extra == 'umi'
Requires-Dist: zarr<4,>=3; extra == 'umi'
Description-Content-Type: text/markdown

# deltacat-io-core

`deltacat-io-core` is the shared local execution layer for DeltaCAT reads and
writes.

It is used by both:

- `deltacat-client` for direct thin-plan execution
- `deltacat` for shared local execution and compatibility wrappers

## Naming

- distribution/package name: `deltacat-io-core`
- Python import module: `deltacat_io_core`

The distribution uses dashes for consistency with `deltacat-client`. The import
module keeps underscores because Python module names cannot contain `-`.

## Scope

`deltacat-io-core` owns the code that should behave the same regardless of
whether the caller is using the thin client or the thick DeltaCAT package.

Today that includes:

- direct execution of thin `Plan` objects
- MOR execution for thin and thick paths
- local file materialization and manifest building
- bounded, authority-bound recovery-frontier index construction and scanning
- schema alignment and table conversion helpers
- sort-aware file ordering and manifest handling
- shared compaction/MOR helper layers and model types
- format-specific local readers/writers

## Non-Goals

`deltacat-io-core` does not own:

- server routes or REST/MCP request handling
- authoritative catalog/storage mutations
- native Ray job orchestration surfaces
- public end-user API shape for `deltacat` or `deltacat-client`

It is a shared implementation layer, not the top-level user product.

## Architecture

The current read architecture is:

1. The server resolves a thin `Plan`.
2. `client.catalog.read(plan=...)` executes that plan directly through
   `deltacat-io-core`.
3. `dc.read_table(plan=...)` for thin plans also executes through the same
   shared path.

There is no longer a runtime bridge back into thick DeltaCAT for thin plan
execution. The plan contract is expected to carry the metadata required for
direct execution.

The current write architecture is:

1. The client stages local files or materializes local data through shared
   helpers.
2. The authoritative commit still happens through DeltaCAT server/native
   boundaries.
3. Shared write-preparation and manifest logic lives in `deltacat-io-core`.

## Installation

Base install:

```bash
uv pip install deltacat-io-core
```

Optional extras:

- `deltacat-io-core[io]` for local file readers/writers (`pyarrow`, `fastavro`)
- `deltacat-io-core[pandas]` for Pandas conversions
- `deltacat-io-core[polars]` for Polars conversions and lazy scan helpers
- `deltacat-io-core[daft]` for Daft conversions and lazy scan helpers
- `deltacat-io-core[lance]` for Lance dataset support
- `deltacat-io-core[umi]` for aligned-Zarr UMI input and LeRobotV2 video output
- `deltacat-io-core[all]` for the full local IO stack

## Read Capabilities

The shared read executor currently handles:

- schema-table reads
- schemaless manifest-table reads
- MOR reads
- direct `pyarrow`, `pandas`, `polars`, `numpy`, `daft`, and `ray_dataset`
  outputs where supported
- lazy `pyarrow_parquet`
- lazy `lance`

It also enforces direct validation for unsupported combinations, for example:

- schemaless + `pyarrow_parquet`
- schemaless + `lance`
- mixed-content lazy plans for format-specific readers
- unknown content types in the shared path

### Polars / Daft Capability Matrix

The shared executor applies the same capability decision in thin
`execute_read_plan(...)` execution and in thick reads that delegate into
that shared path.

| Engine | Content | v1 behavior |
| --- | --- | --- |
| Polars | Parquet | Lazy scan via `pl.scan_parquet(...)` when the existing local preconditions hold |
| Polars | Lance | Explicit eager fallback; no reader-level Lance row-filter pushdown |
| Polars | PackDS | Same as Lance; PackDS plans stay on the explicit eager Lance fallback |
| Daft | Parquet | Lazy scan via shared `build_daft_lazy_scan(...)` when the group is local/shared-eligible |
| Daft | Lance | Lazy only for a single dataset on the shared local path; multi-dataset falls back eagerly |
| Daft | PackDS | Same as Lance under PackDS v5: a single pruned episode dataset can use native lazy Lance scanning; multi-episode plans fall back eagerly |

Notes:

- Mixed-schema lazy eligibility on the shared path requires per-file
  `schema_id` lookups plus top-level schema information with resolvable
  field types, whether that comes from `schema_serialized` or a typed
  top-level `schema` summary.
- JSON schema summaries and requests share the canonical datatype codec in
  `schema_types.py`. Readable scalar names follow PyArrow (`float64`, not
  PyArrow's `double` display text); parameter-rich types use a lossless
  one-field Arrow IPC token. Request parsing is strict, while response/plan
  decoding narrowly tolerates the five scalar spellings emitted by older
  DeltaCAT servers so rolling upgrades and durable plans remain readable.
- On the shared Daft path, non-identity Parquet content encodings (for
  example `.parquet.gz`) stay on the eager PyArrow path.
- When the process is pinned to `DAFT_RUNNER=ray`, the shared local Daft
  lazy path declines and falls back to the eager shared path instead of
  spawning a Ray-backed local lazy scan.

## Write Capabilities

The shared write layer currently covers:

- write input normalization
- local data materialization
- manifest construction for existing files and datasets
- schema/read compatibility helpers
- standard catalog write orchestration slices

Authoritative catalog mutation, commit, retention, and compaction boundaries
still remain on the native/server side where they belong.

### Long-running Lance writes

`write_lance_batch_stream` accepts an optional `refresh_access_ref` callback from
an existing scoped credential owner. The shared credential factory checks the
original binding/scope/routing and provider intent on each refresh and emits an
engine refresh deadline before actual expiry. The owner's existing lease cache
keeps vending independent of fragment count. A failed refresh is retained for
that write attempt so the engine cannot amplify it into repeated credential
requests; job-level retry supplies a fresh stream and output path.

The writer uses Lance's public namespace interface only as a fixed-path
credential adapter (`namespace` on pylance3, `namespace_client` on pylance11).
It does not create catalog
namespaces, implement a second writer, or retry/delete partially consumed
streams. Unsupported writers refuse before stream consumption. Nonrenewing and
local callers retain the ordinary write path.

`open_prepared_lance_dataset` accepts the same scoped `access_ref` and optional
`refresh_access_ref` pair for native in-flight reads. It supplies initial routing
options as well as the native provider: the callback alone is insufficient for
Lance's bootstrap. Pylance11 uses the same fixed-path namespace adapter for
reads instead of3's direct provider argument. Both generated-model bootstrap
requests and mapping-shaped native refresh requests validate the fixed ID/path.
Unsupported reader interfaces refuse before opening storage. Earlier source
qualification described the3.0.0 environment as11.0.0 incorrectly; actual engine
versions and installed wheel qualification must be recorded independently.

`storage_read_support.EntryReadAuthority` reuses `RenewingStorageAuthority` for
one job's manifest reads. Path/revision validation precedes lease reuse; at most
128 scoped bindings are retained. The owner keeps an independent copy of each
original binding policy and refuses conflicting policies for that same scope.
Credential refresh uses the original policy, not a mutable current revision.
Fresh reads share one acquisition per binding/scope; a failed acquisition is
retained for the attempt, including concurrent native callbacks. No catalog RPC
or global credential cache is introduced. Training inventory, native payload
and media copies share that owner. Local HTTP controls cover 8/32 independent
datasets, in-flight fragment rotation, and zero extra storage requests while a
lease remains fresh. This is not a claim of in-flight renewal for every engine
or already-open Parquet/stream handles, nor live AWS/Alpha qualification.
An already-open Parquet handle retains its original credentials. Real HTTP tests
over 8/32 row groups deliberately expire those credentials after the footer GET:
the scan fails, and reopening the retained file with a fresh read owner recovers
the exact rows/schema using the original binding revision. A changed current
revision is not substituted. This is read-attempt recovery, not transparent
mid-handle renewal or proof of whole-job recovery for every consumer.

## Recovery-frontier indexes

`AuthoritativeFrontierIndexWriter` consumes exact, authenticated state-census
Arrow batches and produces one immutable identity index. The index is bound to
the census checkpoint URI, VersionId, payload digest/size/content type, manifest
digest, catalog root, protocol, and hold fence. It rejects marker filenames,
malformed or stale state identities, duplicate/out-of-order census rows, and
mismatched cardinality.

`ExactVersionFrontierIndexScan` implements the `FrontierScan` contract over
that index. It requires a binding-scoped `StorageAccessRef`, reads the exact S3
VersionId once, verifies the byte count and SHA-256 before exposing rows, and
streams Arrow batches from bounded local disk. Its logical remote-call count is
one regardless of census or frontier cardinality. Callers must continue to run
`LeaseBatchPrecondition` before every remote mutation/publication batch.

## Retirement-cohort authorities

`write_authoritative_retirement_cohort` streams a sorted exact membership set
and a complete frozen transaction-success census into one immutable Arrow
file. `RetirementCohortIndex` memory-maps that file and performs exact binary
search over bounded record batches; a lookup is `O(log N)` and never scans or
materializes the full cohort as Python rows. Its monotonic-successor proof
streams both sorted sections in fixed Arrow chunks, proving member and success
set inclusion in `O(M + S)` time and bounded memory.

`RetirementStateEvidence` carries only state already authenticated by the
caller: member identity, normalized status, explicit commit scope, and at most
one direct transaction parent. Resolution is pure and depth-one. Unbound
sessions resolve to reversible abort; applied state with complete success
identity resolves commit-forward. Unknown, contradictory, self-referential,
or unresolved state fails closed.

## Relationship To Other Packages

Use `deltacat-client` when you want the public thin client.

Use `deltacat` when you want the thick/native package.

Use `deltacat-io-core` directly only if you are intentionally building against
the shared execution layer itself.
