Metadata-Version: 2.5
Name: hf-dataset-slow-disk
Version: 0.2.0
Summary: Hugging Face-style lazy Parquet datasets optimized for slow disks
Project-URL: Repository, https://github.com/giangndm/hf-dataset-slow-disk
Project-URL: Issues, https://github.com/giangndm/hf-dataset-slow-disk/issues
Author: Giang Nguyen
License-Expression: MIT
License-File: LICENSE
Keywords: dataset,huggingface,machine-learning,parquet,slow-disk
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: filelock>=3.16
Requires-Dist: huggingface-hub>=0.28
Requires-Dist: numpy>=1.26
Requires-Dist: pyarrow>=17
Requires-Dist: soundfile>=0.12
Description-Content-Type: text/markdown

# hf-dataset-slow-disk

A Hugging Face-style Parquet dataset library for HDDs, network filesystems, and
other slow storage.

## Features

- Sequential row-group reads with bounded memory
- Lazy background download and audio decoding
- Standard Hugging Face Hub Parquet cache
- Resumable general metadata cache with visible progress logs
- Deterministic shuffle, epochs, and exact batch resume
- Sample-mixed concatenation with one shared worker pipeline

## Install

```bash
pip install hf-dataset-slow-disk
```

## Quick start

```python
from hf_dataset_slow_disk import load_dataset

dataset = load_dataset(
    "capleaf/viVoice",
    split="train",
    batch_size=8,
    num_workers=2,
).shuffle(seed=42)

dataset.set_epoch(0)

try:
    for batch in dataset:
        train_step(batch)
finally:
    dataset.close()
```

Workers start lazily on the first `get()` or when dataset iteration begins.
Defaults are one worker, an eight-batch decoded buffer, and a
two-file Parquet buffer.

An optional process-pickleable `collate_fn(batch)` runs on decoded column-oriented
batches inside each background worker. It must preserve column names, order, and
row count, allowing CPU preprocessing to overlap model execution.

## Core API

```python
from hf_dataset_slow_disk import (
    Audio,
    ConcatDataset,
    Dataset,
    DatasetDict,
    concatenate_datasets,
    load_dataset,
)
```

- `load_dataset()` loads Hub or local Parquet data.
- `shuffle(seed)` and `set_epoch(epoch)` select deterministic order.
- `set_batch_idx(batch_idx)` restores the next training batch.
- `for batch in dataset`, `get()`, and `iter_batches()` consume batches.
- `iter_rows()` performs synchronous sample-by-sample iteration.
- `repeat=N` intentionally repeats each chunk N times before shuffle, useful
  for balancing an underrepresented source before concatenation.
- `concatenate_datasets()` mixes compatible sources with their configured
  weights.
- `cast_column()` configures lazy audio decoding.

## Documentation

- [Slow-disk runtime and caching](docs/slow_disk.md)
- [Deterministic shuffle and resume](docs/shuffle.md)
- [Dataset concatenation](docs/concat.md)

## Examples

```bash
uv run python examples/basic.py
uv run python examples/benchmark.py
uv run python examples/concat.py
```

The benchmark and concat examples use batch size 8 and stop after 10 batches.

## Disclaimer

This project was developed with the assistance of AI coding tools. Its source
code is publicly available at
[giangndm/hf-dataset-slow-disk](https://github.com/giangndm/hf-dataset-slow-disk).
Please review and validate it for your own workloads before production use.

## Development

```bash
uv sync
uv run pytest -q
uv build
```
