Metadata-Version: 2.5
Name: wuti-py-util
Version: 0.7.7
Summary: Small Python toolkit: structured files, validation, filesystem, and optional tabular readers.
Project-URL: Homepage, https://github.com/wuti-labs/wuti-py-util
Project-URL: Repository, https://github.com/wuti-labs/wuti-py-util
Author-email: Alex Qin <qinqinghai@hotmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: csv,excel,validation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: jsonschema>=4.25
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: zstandard>=0.23
Provides-Extra: all
Requires-Dist: openpyxl>=3.1.5; extra == 'all'
Requires-Dist: pandas>=2.3; extra == 'all'
Requires-Dist: tabulate>=0.10.0; extra == 'all'
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1.5; extra == 'excel'
Requires-Dist: pandas>=2.3; extra == 'excel'
Requires-Dist: tabulate>=0.10.0; extra == 'excel'
Provides-Extra: pandas
Requires-Dist: pandas>=2.3; extra == 'pandas'
Requires-Dist: tabulate>=0.10.0; extra == 'pandas'
Description-Content-Type: text/markdown

# wuti-py-util

[![codecov](https://codecov.io/github/wuti-labs/wuti-py-util/graph/badge.svg?token=DTNQBHFGM5)](https://codecov.io/github/wuti-labs/wuti-py-util)

A small Python toolkit for internal tools: read and write structured files,
validate inputs, touch the filesystem safely, handle common archive formats,
and optionally read CSV/Excel.

Requirements: Python >= 3.12. [uv](https://docs.astral.sh/uv/) is recommended
for local development.

## Install

```sh
pip install wuti-py-util
```

The core package depends on `jsonschema`, `PyYAML`, and `zstandard`. DataFrame
and Excel features are opt-in:

| Extra    | Enables                                | Install                              |
| -------- | -------------------------------------- | ------------------------------------ |
| `pandas` | `conversion_stats`, `csv` (CsvReader)  | `pip install "wuti-py-util[pandas]"` |
| `excel`  | `excel` (ExcelReader), includes pandas | `pip install "wuti-py-util[excel]"`  |
| `all`    | Everything above                       | `pip install "wuti-py-util[all]"`    |

## What this library is

Core modules: `serialization`, `validation`, `filesystem`, `log`, `archive`,
`csv` / `excel` (extras), `check`, plus small helpers (`filelist`, `filehash`,
`datetime`, `string`, `formatting`, `environment`, `process`).

Frozen (kept for compatibility, no new features): `database`, `web`,
`dataclass_utils`.

## Formatting (`wuti.util.formatting`)

Small human-readable helpers:

```python
from wuti.util.formatting import format_bytes_human

format_bytes_human(1536)  # '1.50 KB'
format_bytes_human(3 * 1024**3)  # '3.00 GB'

result = filesystem.prune_empty_dirs("build")
    print(failure)  # '/path/to/dir: os_error (denied)'
```

`wuti.util.filesystem` also provides path containment helpers:
`validate_relative_path` rejects absolute paths, NUL bytes, and `..`
traversal; `resolve_contained_path` resolves a relative path under a root
and rejects symlink escape via `realpath` + `commonpath`.

## Archives (`wuti.util.archive`)

Create, extract, list, and test common archive formats:

- Archives: `.tar.gz`, `.tar.xz`, `.tar.bz2`, `.tar.lz`, `.tar.zst`, `.tgz`,
  `.tbz2`, `.zip`, `.tar`
- Single-file compressed data (extract, list, and test only): `.zst`,
  `.zstd`, `.xz`, `.lz`

`.tar.lz` and `.lz` are implemented in pure Python on top of the standard
library `lzma` module (lzip container around a raw LZMA1 stream), so no system
lzip tools are needed. `.zst` / `.tar.zst` rely on the `zstandard` package,
which is a core dependency.

```python
import shutil
import wuti.util.archive as archive

archive.create_archive("my-folder", "backup.tar.zst")
archive.list_archive("backup.tar.zst")  # [ArchiveEntry(name=..., size=..., is_dir=...)]
archive.extract_archive("backup.tar.zst", "out-folder")

# Verify integrity without writing anything to disk (full decode).
archive.test_archive("backup.tar.zst")

# Teach shutil.unpack_archive about .zst/.zstd/.lz/.xz single-file data.
archive.register_unpack_formats()  # idempotent; safe to call again
shutil.unpack_archive("payload.xz", "payload-folder")
```

### Transactional extraction

`uncompress_archive` extracts like a build system would: the output is always
one member inside the destination folder (a folder named after the archive's
top-level member, or the archive stem when the members have no single
top-level name; a file named after the stem for single-file data), the
extraction happens in a staging folder and is committed with one atomic
rename, and an already-extracted output is skipped, so repeated runs are
idempotent and a failed run leaves the destination unchanged.

Creation is transactional too: `create_archive` writes a temporary file in
the destination folder and commits with one atomic rename, so a failed run
never leaves a partial archive and an existing archive stays intact.

```python
target = archive.uncompress_archive("downloads/gettext-0.21.tar.xz", "build")
# -> "build/gettext-0.21"

archive.uncompressed_folderpath("downloads/gettext-0.21.tar.xz", "build")
# -> "build/gettext-0.21" (predicts the path without extracting)

archive.uncompressed_name("downloads/gettext-0.21.tar.xz")
# -> "gettext-0.21" (the member name placed under the destination folder)
```

### Reproducible archives

Pass `mtime=` to `create_archive` (or set the standard `SOURCE_DATE_EPOCH`
environment variable to an integer in \[0, 4294967295\]; the argument wins when
both are present) to build byte-identical archives from identical content:
member mtimes, member order (name-sorted), uid/gid, and file modes are
normalized, and the gzip container header of `.tar.gz`/`.tgz` carries the
epoch instead of the wall clock. Zip timestamps clamp to the DOS floor of
1980-01-01 and carry 2-second granularity, so pick an even epoch when a
byte-identical zip must read back the exact time.

```python
archive.create_archive("my-folder", "release.tar.zst", mtime=1700000000)
# or: SOURCE_DATE_EPOCH=1700000000 create ...
```

Extraction is defensive: member names are validated before extraction (NUL
bytes, absolute paths, `..` parts, and Windows drive roots are rejected), tar
members are extracted with the standard library `filter="data"` filter, zip
symlinks are rejected, and the extraction result is scanned for links escaping
the destination folder. Total output size is not limited, so extracting
untrusted archives remains a denial-of-service risk.

When a file's name carries no recognized suffix (or a wrong one),
`extract_archive`, `list_archive`, and `test_archive` fall back to
magic-number sniffing: the leading bytes identify zip/gzip/bzip2/xz/lzip/zstd
containers, and a decoded prefix distinguishes a wrapped tar from single-file
data. `create_archive` still selects the format from the target suffix only.

Zip files written by many tools store UTF-8 member names without the EFS flag
bit (Python then decodes them as cp437, producing mangled non-ASCII names).
`list_archive` and `extract_archive` detect this and restore the proper UTF-8
spelling when the bytes round-trip cleanly.

## Development

Development dependencies (pytest, coverage, ruff, mypy, hatchling, etc.) live in
`[dependency-groups].dev` in `pyproject.toml` — not as a published PyPI extra.
Use [uv](https://docs.astral.sh/uv/) from the repository root:

```sh
uv sync --group dev
```

Run checks through that environment:

```sh
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run mypy
```

If you intentionally use an already-active shared venv, use `--inexact` so uv
does not remove packages installed manually with `uv pip install`:

```sh
uv sync --active --inexact --group dev
uv run --active pytest
uv run --active ruff check .
uv run --active ruff format --check .
uv run --active mypy
```

### Test data (`tests/data`)

Integration tests use fixtures from the
[`wuti-py-testdata`](https://github.com/wuti-labs/wuti-py-testdata) Git submodule
(Git LFS). After cloning, initialize the submodule **and** pull LFS objects
([Git LFS](https://git-lfs.com/) must be installed):

```sh
git submodule update --init
git -C tests/data lfs pull
```

CI runs the same LFS pull after checkout with `submodules: true`.

Light fixtures live under `tests/data/samples/` (`sample.json`, `sample.yaml`,
`filelist.lst`). ABS and GEO datasets stay in their own folders. Path constants
in `tests/testdata.py` (`SAMPLE_JSON`, `SAMPLE_YAML`, `SAMPLE_FILELIST`, …)
point at that layout.

`tests/testdata.py` sets `HAS_TEST_DATA` when `tests/data` exists and is
non-empty. That check is intentionally shallow: it does **not** verify that
files are real payloads rather than [LFS pointer
files](https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md). If you only
run `git submodule update --init` without `lfs pull`, the directory still
contains pointer stubs, `HAS_TEST_DATA` is true, and pytest will **run** those
tests instead of skipping them — then fail when code reads corrupt or tiny
pointer content. Any future “do we have test data?” logic should not rely on
presence alone; detect LFS pointers or validate expected file sizes/checksums.

Tests marked with `@requires_test_data` skip when the submodule is missing or
empty; they do not skip on unfetched LFS.

## Build and publish

Version comes from `__version__` in `src/wuti/util/__init__.py`. The GitHub
Release workflow requires the git tag (`vX.Y.Z`) to match that version. PyPI
rejects duplicate versions.

```sh
uv build
uv pip install dist/wuti_py_util-<version>.tar.gz
```

Publish to PyPI (`https://upload.pypi.org/legacy/`) with a [PyPI API
token](https://pypi.org/manage/account/token/) via `UV_PUBLISH_TOKEN` or
`--token`:

```sh
uv publish
```
