Metadata-Version: 2.2
Name: compress-utils
Version: 0.10.1
Summary: Simple & high-performance compression utilities for Python
Keywords: compression,zlib,brotli,zstd,xz,lzma,snappy,utility,performance
Author-Email: Nicolas Dupont <nicolas@dupontcyborg.com>
License: MIT
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Free Threading :: 2 - Beta
Classifier: Programming Language :: C++
Classifier: Operating System :: OS Independent
Classifier: Topic :: System :: Archiving :: Compression
Classifier: Topic :: Software Development :: Libraries
Project-URL: Homepage, https://github.com/dupontcyborg/compress-utils
Project-URL: Documentation, https://github.com/dupontcyborg/compress-utils#readme
Project-URL: Repository, https://github.com/dupontcyborg/compress-utils
Project-URL: Bug Tracker, https://github.com/dupontcyborg/compress-utils/issues
Requires-Python: >=3.10
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: zstandard; extra == "test"
Requires-Dist: brotli; extra == "test"
Requires-Dist: lz4; extra == "test"
Requires-Dist: python-snappy; extra == "test"
Description-Content-Type: text/markdown

# compress-utils

[![PyPI version](https://badge.fury.io/py/compress-utils.svg)](https://badge.fury.io/py/compress-utils)
[![PyPI Downloads](https://static.pepy.tech/badge/compress-utils)](https://pepy.tech/projects/compress-utils)
[![Python Versions](https://img.shields.io/pypi/pyversions/compress-utils.svg)](https://pypi.org/project/compress-utils/)
[![Python build](https://img.shields.io/github/actions/workflow/status/dupontcyborg/compress-utils/build_and_test_python.yml?label=python)](https://github.com/dupontcyborg/compress-utils/actions/workflows/build_and_test_python.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Unified Python interface for eight compression algorithms — Zstandard, Brotli, zlib, gzip, bzip2, LZ4, XZ/LZMA, and Snappy — over a single high-performance C core. Same API for every algorithm.

## Installation

```bash
pip install compress-utils
```

The wheel is self-contained: no system codec libraries are required at runtime. Type stubs (`.pyi`) and a `py.typed` marker ship with the wheel, so `mypy`, `pyright`, and IDE autocomplete work out of the box.

## Quick start

```python
import compress_utils as cu

data = b"the quick brown fox jumps over the lazy dog" * 100

compressed = cu.compress(data, "zstd", level=5)
restored   = cu.decompress(compressed, "zstd")
assert restored == data
```

Algorithm names accept both lowercase strings (`"zstd"`, `"brotli"`, …) and the typed enum:

```python
cu.compress(data, cu.Algorithm.zstd, level=5)
```

## Streaming

For inputs that don't fit in memory or arrive incrementally:

```python
from compress_utils import CompressStream, DecompressStream

cs = CompressStream("zstd", level=5)
chunks = [cs.compress(chunk) for chunk in iter_chunks(some_file)]
chunks.append(cs.finish())
compressed = b"".join(chunks)

ds = DecompressStream("zstd")
restored = ds.decompress(compressed) + ds.finish()
```

Streaming uses chunked buffers with an internal drain protocol, and output is yielded in 64 KB pages, so streaming a multi-GB file does not hold the whole compressed result in memory. For applications that round-trip many small payloads with the same algorithm, prefer the functional API over creating a new `CompressStream` per payload.

## Supported algorithms

<!-- BEGIN:algorithms -->
<!-- Generated by tools/check-readmes.py from docs/readme/manifest.json. Edit the manifest and run `tools/check-readmes.py --fix`, not this block. -->

| Algorithm | Spelling               | Wire format                                                                              |
|-----------|------------------------|------------------------------------------------------------------------------------------|
| Zstandard | `"zstd"`               | ZSTD frame with content size                                                             |
| Brotli    | `"brotli"`             | Raw Brotli stream                                                                        |
| zlib      | `"zlib"`               | zlib wrapper (RFC 1950)                                                                  |
| gzip      | `"gzip"`               | gzip stream (RFC 1952)                                                                   |
| bzip2     | `"bz2"` (also "bzip2") | bzip2 stream                                                                             |
| LZ4       | `"lz4"`                | LZ4 frame (interoperable with the `lz4` CLI and `.lz4` files)                            |
| XZ / LZMA | `"xz"` (also "lzma")   | XZ stream with CRC64                                                                     |
| Snappy    | `"snappy"`             | Raw Snappy block (interoperable with reference snappy and python-snappy)                 |
| DEFLATE   | `"deflate"`            | Raw DEFLATE (RFC 1951), no wrapper                                                       |
| LZ4 block | `"lz4_raw"`            | Raw LZ4 block, no frame or size prefix (one-shot compression, incremental decompression) |
| LZO1X     | `"lzo"`                | Raw LZO1X block (decompression only)                                                     |
<!-- END:algorithms -->

`cu.is_available(name_or_enum)` returns `True` for algorithms compiled into the installed wheel.

### Raw blocks and `expected_size`

Raw block formats such as those inside Parquet and ORC pages don't record their decompressed size. Pass `expected_size`, an upper bound on the output, and the result holds only the bytes actually decoded:

```python
page = cu.decompress(block, "lz4_raw", expected_size=uncompressed_page_size)
```

Without `expected_size`, `lz4_raw` blocks are decoded with the incremental stream decoder, which is slower, so pass it when you know the size. Every other algorithm accepts it too, which skips the size probe and decodes straight into a buffer of that size.

## Compression levels

Every algorithm accepts a `level` from **1 (fastest) to 10 (smallest)**. The binding maps this to each algorithm's native range automatically, so you don't need to know that Zstandard goes 1–22 and zlib goes 1–9. It defaults to `5` if omitted.

## Errors and limits

```python
cu.version()                                  # e.g. "0.9.0"
cu.set_max_decompressed_size(256 * 1024**2)   # bound one-shot decompression
                                              # (default: 1 GiB; 0 = unbounded)

try:
    cu.decompress(garbage, "zstd")
except cu.CompressError as e:
    print(e)
```

### Type checking

The package ships PEP 561 type information:

```python
import compress_utils as cu
reveal_type(cu.compress)           # → (data: Buffer, algorithm: object, level: int = 5) -> bytes
reveal_type(cu.Algorithm.zstd)     # → Algorithm
```

Stubs are regenerated from the compiled module at build time (via `pybind11-stubgen`), so they cannot drift from the binding signatures.

## Concurrency

Wheels are also published for free-threaded Python 3.14 (`python3.14t`), where the module runs without the GIL and calls on different threads run in parallel whatever their size.

On regular Python, calls with at least 256 KiB of input (or of recorded output, for formats that store their size) release the GIL, so other Python threads keep running and several threads can compress or decompress in parallel. Smaller calls keep the GIL, since getting it back can cost more than the call itself. When streaming, feed chunks of 256 KiB or more to get this.

A `CompressStream` or `DecompressStream` can be shared between threads: calls on one stream run one at a time. Don't resize a `bytearray` while it's being passed to a call; Python raises `BufferError` if you try.

While another thread runs pure Python, each call that released the GIL can wait up to the interpreter's switch interval (5 ms by default) to get it back, as with the standard library's `zlib`. Larger chunks (1 MiB or more) make that wait a small share of each call, and `sys.setswitchinterval(0.0005)` shortens it.

## Performance

Overhead versus native C is **0.89–1.07× on compress and 0.74–0.98× on decompress**. The binding is the C core via pybind11, so overhead only shows on the fastest decoders (zlib and gzip). Compression ratio is byte-identical to every other binding. See the [language comparison](https://github.com/dupontcyborg/compress-utils/blob/main/benchmarks/README.md#language-comparison) for methodology and the full chart, and [Concurrency](#concurrency) for when calls release the GIL.

## License

MIT. See [LICENSE](https://github.com/dupontcyborg/compress-utils/blob/main/LICENSE).

---

Part of [compress-utils](https://github.com/dupontcyborg/compress-utils). Built by [Nico Dupont](https://nico.codes).
