Metadata-Version: 2.4
Name: o41-checkpoint
Version: 0.1.0
Requires-Dist: pytest>=8.0.0 ; extra == 'dev'
Requires-Dist: ruff>=0.15.12 ; extra == 'dev'
Requires-Dist: ty>=0.0.34 ; extra == 'dev'
Provides-Extra: dev
License-File: LICENSE
Summary: Python bindings for 041's Rust checkpoint engine
License-Expression: Apache-2.0
Requires-Python: >=3.14, <3.15
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# 041 Checkpoint for Python

Save, resume, and fork training checkpoints with Python bindings to the
`o41-checkpoint` Rust engine.

The Rust core handles multipart uploads, parallel ranged reads, retries, signed
URL refresh, placement, and writer fencing. The Python interface accepts bytes
and paths and returns typed checkpoint objects. Blocking calls release the GIL.

## Install

```sh
uv add o41-checkpoint
```

Requires Python 3.14. Wheels target Linux x86-64, Linux arm64, and macOS arm64.
Building from source requires Rust.

## Resume and save

```python
from pathlib import Path
from o41_checkpoint import Checkpoints, step_name

checkpoints = Checkpoints("o41://experiments/run-1")
writer = checkpoints.resume()

if writer.resumed is not None:
    writer.resumed.download("./restored")

saved = writer.save(
    step_name(600),
    {
        "weights.safetensors": Path("weights.safetensors"),
        "state.json": b'{"step": 600}',
    },
    meta={"step": 600, "loss": 0.42},
)
print(saved.path, saved.size)
```

`resume()` returns a writer whose `resumed` checkpoint is the latest, or `None`
when the location is empty. The same writer can save many checkpoints.
`writer.latest` is the name that the next save must exceed. Use `start()` to
require an empty location.

Each file value is bytes or a filesystem path. Strings are paths. Rust reads
paths one upload part at a time. Keep source files unchanged until `save()`
returns. The core commits the manifest last, so interrupted saves do not appear
in `load()` or `list()`.

`save()` returns a `Saved` receipt with `name`, `path`, `committed`, `size`,
`placement`, and `warning`. It returns after commit, without a follow-up read.
Names must ascend in byte order. `step_name(600)` produces `step-00000600`. Use
your own naming scheme for runs beyond 99,999,999 steps.

## Read a checkpoint

```python
latest = checkpoints.load()  # None when no checkpoint is committed
at_step = checkpoints.load(step_name(600))

if at_step is not None:
    state = at_step["state.json"].read()
    header = at_step["weights.safetensors"].read(start=0, stop=8)
    at_step["weights.safetensors"].download("./weights.safetensors")
    at_step.download("./checkpoint")
    print(at_step.meta, at_step.committed)

for checkpoint in checkpoints.list():
    print(checkpoint.name, checkpoint.size)
```

Reading and listing do not open a writer. `load()` and `load("latest")` return
the latest checkpoint. A missing checkpoint returns `None`.

`checkpoint.files` maps names to files with `name` and `size`. Checkpoints also
expose `path`, `name`, `committed`, `size`, `meta`, `source`, `location`, and
`transfer`. Commit times are timezone-aware `datetime` values. `location` and
`transfer` are `None` for local directories and raw buckets.

`read()` returns bytes. Ranges use Python's exclusive `stop` convention. Large
reads use the core's parallel range reader. `download()` transfers files in
bounded windows and returns a `Path`. It stages downloads beside the target and
replaces files after successful transfer. A checkpoint download stages all files
before moving them into the destination, one file at a time. Other files in the
destination remain in place. It downloads the checkpoint's data files; it does
not create a checkpoint catalog at the destination.

## Fork a run

```python
source = checkpoints.load(step_name(600))
if source is not None:
    fork = Checkpoints("o41://experiments/run-2").fork(source)
    assert fork.resumed is not None
    fork.resumed.download("./fork-source")
    fork.save(step_name(700), {"state.json": b'{"step": 700}'})
```

`fork()` requires an empty destination. It accepts a `Checkpoint` or a concrete
checkpoint path, such as `o41://experiments/run-1/step-00000600`. Load the
latest checkpoint first to fork it. The source stays in its existing location.
The first save records `checkpoint.source`; subsequent saves do not.

## Choose storage

The same API supports these locations:

| Location                          | Storage                                 | Writer fencing |
| --------------------------------- | --------------------------------------- | -------------- |
| `./checkpoints/run-1` or a `Path` | Local directory                         | No             |
| `s3://bucket/run-1`               | S3-compatible bucket                    | No             |
| `gs://bucket/run-1`               | Google Cloud Storage                    | No             |
| `az://account/container/run-1`    | Azure Blob Storage                      | No             |
| `o41://experiments/run-1`         | 041 Artifacts placement and signed URLs | Yes            |

Local directories need no credentials. Raw buckets use the Rust `object_store`
credential configuration. Pass overrides through
`store_options={"aws_region": "us-east-1"}`, using `object_store` option names.

For `o41://`, set `O41_ARTIFACTS_API_KEY` to an organization API key.
`O41_ARTIFACTS_URL` overrides `https://artifacts.041.io`. `O41_ARTIFACTS_CLOUD`
and `O41_ARTIFACTS_REGION` override placement together. Otherwise the core
detects placement through SkyPilot or VM metadata and uses the organization's
fallback when unknown.

`Checkpoints(location, api_key=..., url=..., organization=..., cloud=..., region=..., retry_for=..., store_options=...)`
overrides the core's environment configuration. `retry_for` is a duration in
seconds. Omitted options use the Rust defaults, including its retry rules and
longer begin/commit deadlines.

## Notes and failures

```python
checkpoints.write_note("evaluated.json", b'{"step": 600}')
note = checkpoints.note("evaluated.json")  # bytes, or None when absent
```

Notes hold up to 64 KiB beside the checkpoints and do not require a writer. A
checkpoint holds 1 to 1000 files and up to 16 KiB of JSON metadata. File names
cannot contain directories.

`CheckpointError` reports failures from the Rust engine. `NotEmpty` and
`Superseded` derive from it. On `o41://`, another writer opening the artifact
fences the old writer. Catch `Superseded` to stop that run. Exceptions expose
`code`, `status`, `epoch`, and `location` when the core supplies them. Python
argument conversion errors use `TypeError` or `ValueError`, and local staging
failures can raise `OSError`.

Handles retain the Rust runtime for as long as they are referenced. Create them
inside each worker process rather than sharing them across a process fork.

