Metadata-Version: 2.4
Name: piper-offload
Version: 0.10.0rc12
Summary: Model-agnostic GPU/CPU memory management for PyTorch — host offloading, block streaming, and LRU model caching.
Keywords: inference,offloading,pytorch,quantization,streaming
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: torch>=2.14,<2.15
Requires-Dist: piper-kernels>=0.7.9,<0.8
Requires-Dist: piper-offload[bnb,gguf,quanto,torchao,triton] ; extra == 'all'
Requires-Dist: bitsandbytes>=0.49,<0.51 ; extra == 'bnb'
Requires-Dist: gguf>=0.18,<0.20 ; extra == 'gguf'
Requires-Dist: piper-offload[torchao,triton] ; extra == 'gguf'
Requires-Dist: optimum-quanto>=0.2.7,<0.3 ; extra == 'quanto'
Requires-Dist: numpy>=2.3.2,<3 ; extra == 'torchao'
Requires-Dist: torchao>=0.18,<0.19 ; extra == 'torchao'
Requires-Dist: triton>=3.8,<3.9 ; sys_platform == 'linux' and extra == 'triton'
Requires-Dist: triton-windows>=3.8,<3.9 ; sys_platform == 'win32' and extra == 'triton'
Requires-Python: >=3.14
Project-URL: Homepage, https://github.com/Boffee/piper-offload
Project-URL: Repository, https://github.com/Boffee/piper-offload.git
Project-URL: Issues, https://github.com/Boffee/piper-offload/issues
Project-URL: Changelog, https://github.com/Boffee/piper-offload/blob/main/CHANGELOG.md
Provides-Extra: all
Provides-Extra: bnb
Provides-Extra: gguf
Provides-Extra: quanto
Provides-Extra: torchao
Provides-Extra: triton
Description-Content-Type: text/markdown

# Piper Offload

Keep reusable PyTorch models in host memory and move their weights onto a GPU
when needed. Piper Offload manages whole-model uploads when a model fits in
VRAM, and block streaming when it does not. Models and adapters stay cached
between uses, including compatible checkpoint mappings.

Use it for workloads that run several models in sequence or repeatedly reuse
large models. `ModelCache` handles caching and activation; `ModelOffloader`
provides direct control over one model's lifecycle.

## Installation

Requires Python 3.14 or newer and PyTorch 2.14.

```bash
pip install piper-offload
```

The base dependencies are `torch` and `piper-kernels`. Optional integrations
are available through the `bnb`, `quanto`, `gguf`, and `torchao` extras;
`triton` adds accelerated kernels, and `all` includes every integration.
See [supported formats](https://github.com/Boffee/piper-offload/blob/main/docs/formats.md) for backend requirements.

## Quick start

This example runs on CUDA when available and on CPU otherwise. The factory
builds a fresh model; the cache retains it for subsequent uses.

```python
import torch
from torch import nn
from piper_offload import ModelCache, ModelSpec


def build_model():
    return nn.Sequential(
        nn.Linear(8, 16), nn.ReLU(), nn.Linear(16, 4),
    ).eval().requires_grad_(False)


cache = ModelCache()
spec = ModelSpec(key="example", factory=build_model)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

for _ in range(2):
    inputs = torch.randn(1, 8, device=device)
    with cache.use(spec, device=device) as model, torch.inference_mode():
        output = model(inputs)

print(output.shape)  # torch.Size([1, 4])
del model  # release the reference returned by the last use
cache.clear()
```

Each `use()` activates the model and deactivates it on exit. Place inputs on
the compute device yourself. For a model larger than VRAM, select its block
lists with `ModelSpec(..., block_paths=("transformer_blocks",))`; see
[streaming](https://github.com/Boffee/piper-offload/blob/main/docs/streaming.md) for the required model structure.

## Fit and limitations

- Each cached model has one runtime and supports sequential use. Concurrent
  replicas need separately constructed models under distinct keys.
- `ModelCache` retains host state until explicit eviction. Pinned memory has
  a separate budget; cache accounting is not a limit on process RAM.
- Factories transfer ownership of compatible CPU storage. Set `requires_grad`
  before capture and do not mutate factory-produced tensors afterwards.
- Buffer changes made on CUDA are discarded at deactivation. Models that
  need persistent buffer state across calls require a different arrangement.
- Compilation supports declared block forwards. Streamed training requires
  activation checkpointing and explicit handling of optimizer updates.

## Documentation

Both application developers and coding agents can start with the relevant
topic. These links point to `main`; select the matching Git tag when integrating
a pinned release.

| Task | Read |
|---|---|
| Cache models, manage activation, inspect or release resources | [Models](https://github.com/Boffee/piper-offload/blob/main/docs/models.md) |
| Run models larger than VRAM, compile blocks, or train streamed weights | [Streaming](https://github.com/Boffee/piper-offload/blob/main/docs/streaming.md) |
| Apply LoRA, parameter deltas, or parameter values | [Adapters](https://github.com/Boffee/piper-offload/blob/main/docs/adapters.md) |
| Configure checkpoint backing, pinning, and Windows copy retention | [Memory](https://github.com/Boffee/piper-offload/blob/main/docs/memory.md) |
| Check tensor formats, optional dependencies, or DTensor support | [Formats](https://github.com/Boffee/piper-offload/blob/main/docs/formats.md) |
| Understand the implementation or contribute changes | [Development](https://github.com/Boffee/piper-offload/blob/main/docs/development.md) |

Contributor instructions are in [AGENTS.md](https://github.com/Boffee/piper-offload/blob/main/AGENTS.md). See
[versioning and releases](https://github.com/Boffee/piper-offload/blob/main/VERSIONING.md) for compatibility policy and
[the changelog](https://github.com/Boffee/piper-offload/blob/main/CHANGELOG.md) for release history.

Licensed under the [Apache License, Version 2.0](https://github.com/Boffee/piper-offload/blob/main/LICENSE).
