Metadata-Version: 2.5
Name: hf-ark
Version: 0.1.0
Summary: Snapshot the Hugging Face models and datasets your code depends on into a portable, checksummed, offline-usable archive.
Project-URL: Homepage, https://github.com/Dnakitare/hf-ark
Author-email: Daniel Nakitare <dnakitare@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=1.0.0
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Description-Content-Type: text/markdown

# hf-ark

The models your code depends on live on servers Nvidia now owns. Snapshot them before the terms change.

`hf-ark` finds the Hugging Face models and datasets a codebase (or machine) depends on and snapshots them into a portable, checksummed, offline-usable archive: an "ark". It's deliberately boring under the hood. An ark is a valid `huggingface_hub` cache directory plus a manifest with checksums and license tags. No custom format. Anything that reads an HF cache can read an ark.

## Quickstart

```
pip install hf-ark
hf-ark snapshot --from . --include-cache
hf-ark verify ./hf-ark
```

Later, on any machine:

```
hf-ark restore ./hf-ark
HF_HUB_OFFLINE=1 python your_script.py
```

## What an ark is

An ark is just a `huggingface_hub` cache directory (`models--org--name/`, `blobs/`, `snapshots/`) plus one `ark.json` file recording, per repo: the resolved commit sha, license tag, gated status, and a sha256 for every file. `hf-ark` computes those checksums itself; it doesn't trust the hub's metadata.

Because an ark is a real HF cache, you don't need to restore it anywhere to use it. Point `HF_HUB_CACHE` straight at it:

```
HF_HUB_CACHE=/path/to/ark HF_HUB_OFFLINE=1 python your_script.py
```

`hf-ark restore` exists for when you'd rather merge an ark into your normal cache location instead.

## Commands

- `hf-ark scan [PATHS...]` finds HF repo references in your source (`from_pretrained(...)`, `hf_hub_download(...)`, huggingface.co URLs, and an optional `hf-ark.txt` file). Add `--include-cache` to also list what's already in your local HF cache, `--check` to confirm each repo exists on the hub, `--json` for machine-readable output.
- `hf-ark snapshot [REPO_IDS...] [-o ARK]` downloads repos into an ark and records them in `ark.json`. Takes explicit repo IDs, `--from PATHS` (runs the scanner), and/or `--include-cache`. `--dry-run` prints the download size without writing anything.
- `hf-ark verify ARK` recomputes every file's sha256 and checks it against the manifest.
- `hf-ark list ARK` prints a table of what's in an ark: repo, type, sha, license, size. No network.
- `hf-ark restore ARK [--target DIR]` merges an ark into a local HF cache (default: wherever `huggingface_hub` already looks).

Run `hf-ark <command> --help` for the full flag list.

## FAQ

**Does this redistribute model weights?** No. It downloads what you already have access to and stores it under your control. Nothing gets uploaded anywhere.

**What about licenses?** The manifest records each model's license tag as reported by the hub. The contents stay under their own license. Snapshotting a repo doesn't change what you're allowed to do with it.

**Gated models?** You still need a token with access (`HF_TOKEN`, same as `huggingface_hub` always used). An ark of a gated repo doesn't grant access to anyone else; it's just your already-authorized download saved locally.

**Private hub / mirror?** `hf-ark` respects `HF_ENDPOINT`, same as `huggingface_hub`.

**Is Hugging Face actually going to break?** Nobody knows. That's the point. Snapshots are cheap insurance either way.

## Not this

`hf-ark` isn't a mirror server (see [olah](https://github.com/vtuber-plan/olah) for that), a torrent client, or a package manager. It snapshots and restores. That's it.

## License

MIT
