Metadata-Version: 2.2
Name: compress-utils
Version: 0.12.0
Summary: Simple & high-performance compression utilities for Python
Keywords: compression,zlib,brotli,zstd,xz,lzma,snappy,utility,performance
Author: Nicolas Dupont
License: MIT
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Programming Language :: Python :: Free Threading :: 3 - Stable
Classifier: Programming Language :: C++
Classifier: Operating System :: OS Independent
Classifier: Topic :: System :: Archiving :: Compression
Classifier: Topic :: Software Development :: Libraries
Project-URL: Homepage, https://github.com/dupontcyborg/compress-utils
Project-URL: Documentation, https://github.com/dupontcyborg/compress-utils#readme
Project-URL: Repository, https://github.com/dupontcyborg/compress-utils
Project-URL: Bug Tracker, https://github.com/dupontcyborg/compress-utils/issues
Requires-Python: >=3.11
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: zstandard; extra == "test"
Requires-Dist: brotli; extra == "test"
Requires-Dist: lz4; extra == "test"
Requires-Dist: python-snappy; extra == "test"
Description-Content-Type: text/markdown

# compress-utils

[![PyPI version](https://img.shields.io/pypi/v/compress-utils.svg)](https://pypi.org/project/compress-utils/)
[![PyPI Downloads](https://static.pepy.tech/badge/compress-utils)](https://pepy.tech/projects/compress-utils)
[![Python Versions](https://img.shields.io/pypi/pyversions/compress-utils.svg)](https://pypi.org/project/compress-utils/)
[![Python build](https://img.shields.io/github/actions/workflow/status/dupontcyborg/compress-utils/build_and_test_python.yml?label=python)](https://github.com/dupontcyborg/compress-utils/actions/workflows/build_and_test_python.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

One API for Zstandard, Brotli, zlib, gzip, bzip2, LZ4, XZ/LZMA and Snappy, plus raw DEFLATE, LZ4 blocks and LZO decoding, with performance on par with the dedicated Python packages for each format. Self-contained wheels with no system libraries, free-threaded Python supported, all backed by one C core.

## Installation

```bash
pip install compress-utils
```

Self-contained wheels (no system codec libraries) with PEP 561 type stubs, for CPython 3.11–3.15 including free-threaded 3.14t and 3.15t, on Linux (manylinux, x86-64 and aarch64), macOS (Apple silicon) and Windows (x86-64); other platforms such as musl Linux, Intel macOS and Windows on Arm build from the source distribution, which needs a C and C++ toolchain.

## Quick start

```python
import compress_utils as cu

data = b"the quick brown fox jumps over the lazy dog" * 100

compressed = cu.compress(data, "zstd", level=5)
restored   = cu.decompress(compressed, "zstd")
assert restored == data
```

Algorithm names accept both lowercase strings (`"zstd"`, `"brotli"`, …) and the typed enum:

```python
cu.compress(data, cu.Algorithm.zstd, level=5)
```

## Performance

<!-- bench:start -->
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/dupontcyborg/compress-utils/main/docs/benchmarks/img/readme-python-dark.png">
  <img src="https://raw.githubusercontent.com/dupontcyborg/compress-utils/main/docs/benchmarks/img/readme-python-light.png" alt="Compression ratio against throughput in MB/s on a log axis for compress-utils for Python and brotli (PyPI), cramjam, lz4 (PyPI), Python stdlib and zstandard, one row per algorithm (zstd, brotli, zlib, xz, lz4, snappy, bz2) with compress on the left and decompress on the right, dickens one-shot on AWS c8g.xlarge (Graviton4, arm64), 2026-10-29" width="830">
</picture>

compress-utils for Python against other Python compression libraries: speed against compression ratio.

Measured on AWS c8g.xlarge (Graviton4), dickens (10.2 MB of text), one-shot, 2026-10-29.

Full results [here](https://github.com/dupontcyborg/compress-utils/blob/main/docs/benchmarks/python.md).
<!-- bench:end -->

## Supported algorithms

<!-- BEGIN:algorithms -->
<!-- Generated by tools/check-readmes.py from docs/readme/manifest.json. Edit the manifest and run `tools/check-readmes.py --fix`, not this block. -->

| Algorithm | Spelling               | Wire format                                                                              |
|-----------|------------------------|------------------------------------------------------------------------------------------|
| Zstandard | `"zstd"`               | ZSTD frame with content size                                                             |
| Brotli    | `"brotli"`             | Raw Brotli stream                                                                        |
| zlib      | `"zlib"`               | zlib wrapper (RFC 1950)                                                                  |
| gzip      | `"gzip"`               | gzip stream (RFC 1952)                                                                   |
| bzip2     | `"bz2"` (also "bzip2") | bzip2 stream                                                                             |
| LZ4       | `"lz4"`                | LZ4 frame (interoperable with the `lz4` CLI and `.lz4` files)                            |
| XZ / LZMA | `"xz"` (also "lzma")   | XZ stream with CRC64                                                                     |
| Snappy    | `"snappy"`             | Raw Snappy block (interoperable with reference snappy and python-snappy)                 |
| DEFLATE   | `"deflate"`            | Raw DEFLATE (RFC 1951), no wrapper                                                       |
| LZ4 block | `"lz4_raw"`            | Raw LZ4 block, no frame or size prefix (one-shot compression, incremental decompression) |
| LZO1X     | `"lzo"`                | Raw LZO1X block (decompression only)                                                     |
<!-- END:algorithms -->

`cu.is_available(name_or_enum)` returns `True` for algorithms compiled into the installed wheel.

### Raw blocks and `expected_size`

Raw block formats such as those inside Parquet and ORC pages don't record their decompressed size, so pass `expected_size`, an upper bound on the output; the result holds only the bytes actually decoded. More in the [advanced guide](https://github.com/dupontcyborg/compress-utils/blob/main/docs/bindings/python-advanced.md#raw-blocks-and-expected_size).

```python
page = cu.decompress(block, "lz4_raw", expected_size=uncompressed_page_size)
```

## Streaming

For inputs that don't fit in memory or arrive incrementally:

```python
from compress_utils import CompressStream, DecompressStream

cs = CompressStream("zstd", level=5)
chunks = [cs.compress(chunk) for chunk in iter_chunks(some_file)]
chunks.append(cs.finish())
compressed = b"".join(chunks)

ds = DecompressStream("zstd")
restored = ds.decompress(compressed) + ds.finish()
```

Streaming does not hold the whole result in memory, so a multi-GB file is fine. To stop at the end of one frame or member and get the bytes after it, pass `single_stream=True`; see [stopping at the end of a stream](https://github.com/dupontcyborg/compress-utils/blob/main/docs/bindings/python-advanced.md#stopping-at-the-end-of-a-stream).

## Compression levels

Every algorithm accepts a `level` from **1 (fastest) to 10 (smallest)**. The binding maps this to each algorithm's native range automatically, so you don't need to know that Zstandard goes 1–22 and zlib goes 1–9. It defaults to `5` if omitted.

## Errors and limits

Decompression is bounded by default: output is capped at 1 GiB, so a hostile or corrupt input cannot exhaust memory. Change the cap with `set_max_decompressed_size`, and see the [untrusted input guidance](https://github.com/dupontcyborg/compress-utils#untrusted-input) in the project README for what else to set when you decompress data you don't control.

```python
cu.version()                                  # e.g. "0.12.0"
cu.set_max_decompressed_size(256 * 1024**2)   # cap decompress() for every algorithm
                                              # (default: 1 GiB; 0 = unbounded)
ds = cu.DecompressStream("gzip", max_output=256 * 1024**2)  # streams: no limit unless set

try:
    cu.decompress(garbage, "zstd")
except cu.CompressError as e:
    print(e)
```

The package ships PEP 561 type stubs; [type checking](https://github.com/dupontcyborg/compress-utils/blob/main/docs/bindings/python-advanced.md#type-checking) shows what they report.

## Concurrency

Wheels for free-threaded Python 3.14 and 3.15 (`python3.14t`, `python3.15t`) run without the GIL, so calls on different threads run in parallel. On regular Python, large calls (256 KiB or more of input) release the GIL so other threads keep running; when streaming, feed chunks of 256 KiB or more.

A `CompressStream` or `DecompressStream` can be shared between threads: calls on one stream run one at a time. More in the [advanced guide](https://github.com/dupontcyborg/compress-utils/blob/main/docs/bindings/python-advanced.md#concurrency-details).

## License

MIT. See [LICENSE](https://github.com/dupontcyborg/compress-utils/blob/main/LICENSE).

---

Part of [compress-utils](https://github.com/dupontcyborg/compress-utils). Built by [Nico Dupont](https://nico.codes).
