Metadata-Version: 2.5
Name: fs_explorer
Version: 0.1.0
Summary: List and read disk images, filesystems, and archives using only the Python standard library
Author-email: Arthur Gouhier <ajgouhier@gmail.com>
License-Expression: BSD-3-Clause
License-File: LICENSE
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# fs-explorer

List and read the contents of disk images, filesystems, and archives using only the Python standard
library.

```
pip install fs-explorer        # distribution
import fs_explorer             # package
fs-explorer ls disk.dmg        # command (also: python -m fs_explorer)
```

## Guarantees

- **No dependencies.** Python 3.10+, standard library only. System libraries (`libzstd`, `liblz4`,
  `liblzo2`, `libcrypto`, Apple's `libcompression`/CommonCrypto) are used through `ctypes` when
  present, always with a pure-Python fallback.
- **Listing reads metadata only.** Directory records, inode tables, B-tree nodes, central directories,
  and headers. File data is not read. Formats that can only be listed by decompressing everything
  (e.g. `.tar.gz`) are detected and refused with an explicit reason, unless you opt in with
  `full_read_limit`.
- **Reading touches only what the file needs.** Only the blocks, chunks, extents, or solid-block prefix
  that the requested file depends on are read and decoded.
- **Streaming.** Entries are yielded as they are parsed. Memory is bounded by the largest single
  directory plus the cache budget (default 64 MiB), not by image size.
- **Read-only.** No mounting, no subprocesses, no writes to the source. `extract()` writes only under its
  destination directory.

## Supported formats

### Containers and partition maps

| Format | Notes | What is read to list |
|---|---|---|
| Raw images (`.img`, `.iso`, `.bin`) | detection by content | superblocks |
| UDIF / DMG | raw, zero, zlib, bzip2, LZMA, ADC, LZFSE chunks | koly trailer + plist chunk table |
| Encrypted DMG v1 / v2 | AES-128/256, password | header, key blobs; data decrypted per 4 KiB chunk |
| `.sparseimage`, `.sparsebundle` | bands read lazily; missing bands are zeros | header / `Info.plist` |
| VHD (fixed, dynamic, differencing), VHDX (incl. log replay, differencing) | parents via sibling files | footer/headers, BAT |
| VMDK (monolithicSparse, streamOptimized, split, flat, snapshots) | | grain directory/tables |
| QCOW2 (v2/v3, compressed deflate/zstd, backing files, extended L2) | | L1/L2 tables |
| VDI, Android sparse, BIN/CUE (Mode 1, Mode 2 Form 1) | | block map / chunk list / cue sheet |
| GPT, APM, MBR with EBR chains | volumes named `p1`…`pN` | partition tables |

### Filesystems

| Filesystem | Reading extras |
|---|---|
| HFS+ / HFSX | decmpfs compression (zlib, LZVN, LZFSE), hard links, resource forks, xattrs |
| APFS | decmpfs, clones, sparse files, **native encryption** (password or recovery key); volumes `v1`…`vN` |
| ISO 9660 + Joliet + Rock Ridge | multi-extent files, zisofs |
| UDF (1.02–2.60, sparable and metadata partitions) | embedded data, all allocation descriptor kinds |
| FAT12/16/32, exFAT | long names, NoFatChain |
| NTFS | LZNT1 compression, sparse files, named streams, reparse symlinks/junctions, WOF (XPRESS/LZX) |
| ext2/3/4 | extents, indirect blocks, inline data, holes, xattrs |
| SquashFS 4 | gzip, lzma, lzo, xz, lz4, zstd; fragments; xattrs |
| Btrfs (single, DUP, RAID1, RAID1C3/4) | subvolumes, compressed extents (zlib, zstd, LZO), holes |

### Archives and streams

| Format | Listing | Reading |
|---|---|---|
| zip (Zip64) | central directory only | stored, deflate, Deflate64, bzip2, LZMA, xz, zstd, PPMd; ZipCrypto, WinZip AES |
| tar (ustar, GNU, PAX, sparse) | headers only | direct views; compressed tar readable forward-only |
| 7z | header (incl. encrypted header) | LZMA, LZMA2, BCJ/ARM/PPC/SPARC/IA64, Delta, BCJ2, bzip2, Deflate, Deflate64, PPMd, 7zAES |
| RAR5, RAR4 | headers (incl. encrypted headers) | **stored members only** (with decryption); compressed members raise `MissingCodecError` |
| xar and `.pkg` (via Bom) | TOC + Bom `Paths` tree | zlib/bzip2/xz members; Bom-backed payloads through pbzx + cpio |
| cpio (newc, crc, odc, binary), ar / `.deb`, RPM, CAB | headers | CAB: stored, MSZIP, LZX; RPM: payload (gzip, bzip2, xz, zstd) forward-only |
| gzip, xz, zstd, bzip2, lz4 streams | one entry: original name and size when stored | decompressed contents |

### Codec tiers

| Tier | Codecs | Implementation |
|---|---|---|
| core | zlib/deflate, bzip2, LZMA/xz, BCJ, Delta | standard library |
| core | zstd | `compression.zstd` (3.14+) → `libzstd` → pure Python |
| core | LZFSE, LZVN, ADC, LZ4, pbzx, CRC32C | `libcompression`/`liblz4` where available → pure Python |
| common | LZO1X, LZNT1, MSZIP, BCJ2, Deflate64, zisofs, ZipCrypto, WinZip AES | `liblzo2` → pure Python |
| heavy | PPMd H and I, LZX, XPRESS Huffman | pure Python (slow: PPMd ≈ 0.1 MB/s) |
| crypto | AES (ECB, CBC, CTR, XTS), 3DES, RFC 3394, PBKDF2, 7z/RAR KDFs | `libcrypto` / CommonCrypto → pure Python |

### Refused or out of scope

- **Refused for listing** (detected; allowed with `full_read_limit`): compressed tar and cpio, `.deb` data
  payloads, `.pkg` payloads without a Bom, disk images wrapped in a compression stream (`.img.xz`).
  Reading individual members still works by streaming.
- **Refused for reading:** NTFS EFS, ext4 fscrypt, APFS per-file (hardware-bound) keys, QCOW2
  encryption, VDI differencing images, RAR compressed members.
- **Not detected:** NDIF, Disk Copy 4.2, E01, WIM/ESD, XFS, f2fs, multi-device Btrfs striping,
  multi-volume archives, BitLocker, LUKS, StuffIt, plain (non-wrapped) HFS.

## Command line

```
fs-explorer ls      [OPTIONS] SOURCE [TARGET]
fs-explorer cat     [OPTIONS] SOURCE PATH
fs-explorer extract [OPTIONS] SOURCE [PATH] -o DEST
fs-explorer probe   [OPTIONS] SOURCE
```

```console
$ fs-explorer probe Install.dmg
UDIF (UDZO) > GPT (512-byte sectors) > p2 "Installer" > HFS+

$ fs-explorer ls -l disk.img /p2/Users -r 0
drwxr-xr-x      501       20              2024-05-01 10:12 /p2/Users/alice
...

$ fs-explorer ls -n 1 backup.tar /old          # descend into nested containers
$ fs-explorer cat disk.vhdx /p1/Windows/win.ini
$ fs-explorer cat --offset 1M --length 4096 image.qcow2 /p1/big.bin | xxd
$ fs-explorer extract release.7z /docs -o out/ -v
$ fs-explorer ls --json encrypted.dmg --ask-password
$ fs-explorer ls outer.zip /disk.dmg/p1/Applications     # paths cross containers
```

Common options: `--password PW`, `--password-file PATH`, `--ask-password` (prompts once per
encrypted layer, showing the APFS hint), `$FS_EXPLORER_PASSWORD`, `--format NAME`,
`--backend auto|pure|system`, `--cache-size 64M`, `--full-read-limit SIZE`,
`--errors raise|yield|skip` (default `yield`: errors go to stderr, listing continues), `--stats`.

`ls`: `-r N` recursion depth (default unlimited, 0 = direct children), `-n N` nested container
descent, `--no-stat`, `-i/-x PATTERN` include/exclude (excluded directories are pruned),
`-T f,d,l,v` entry types, `-a` system files, `--no-flatten`, `--sort`, `-l`, `--json` (JSON Lines),
`-0`, `--relative`.

`cat`: `--offset`, `--length`, `--no-follow`, `--verify`.
`extract`: `-o DEST`, `-r/-i/-x/-T`, `--preserve mode,mtime,owner,symlinks,hardlinks,xattrs,resource_forks`,
`--on-conflict error|skip|overwrite|rename`, `--max-size`, `--max-ratio`, `--no-verify`, `--no-sparse`, `-v`.

Exit codes: 0 success; 1 some entries had errors; 2 usage error; 3 unsupported format or missing
codec; 4 missing or wrong password; 5 target not found; 6 unsafe path or limit exceeded.

## Python API

```python
import fs_explorer as fx

for e in fx.iter_entries("disk.dmg", target="/p2/Users", recursive=1):
    print(e.path, e.type.name, e.size, e.mtime)

data = fx.read_file("backup.7z", "/docs/report.pdf", password="secret")

with fx.open_file("image.vhdx", "/p1/pagefile.sys") as f:   # io.RawIOBase
    f.seek(1 << 30)
    chunk = f.read(4096)

for e in fx.extract("archive.zip", "/", "out/", on_conflict="rename"):
    print("wrote", e.path)

print(" > ".join(map(str, fx.probe("Install.dmg"))))

# Reuse parsed layers across calls:
with fx.open_source("disk.img", password=lambda req: ask_user(req.description, req.hint)) as src:
    names = [e.name for e in src.entries(stat=False, recursive=0)]
    blob = src.read_file("/p1/etc/hostname")
    print(src.stats.as_dict())      # bytes read, cache hits, decompressed, decrypted, backends
```

`iter_entries(source, *, recursive=-1, target="/", password=None, nested=0, stat=True, include=(),
exclude=(), types=None, format_hint=None, errors="raise", system_files=False, flatten=True,
sort=False, full_read_limit=0, resolver=None, cache_size=64<<20, backend="auto")`.
A source is a path, a binary file object (seekable), `bytes`, or a directory (`.sparsebundle`).

`Entry` fields: `path`, `name`, `type` (`FILE`, `DIR`, `SYMLINK`, `HARDLINK`, `CHAR_DEVICE`,
`BLOCK_DEVICE`, `FIFO`, `SOCKET`, `VOLUME`, `OTHER`), `depth`, `size`, `stored_size`, `mode`, `uid`,
`gid`, `owner`, `group`, `mtime_ns`/`ctime_ns`/`atime_ns`/`btime_ns` (ns since the Unix epoch, UTC;
`.mtime` etc. give `datetime`s), `nlink`, `inode`, `link_target`, `format`, `container`, `extra`,
`error`. `entry.to_dict()` is JSON-ready.

Errors derive from `FsExplorerError`: `UnsupportedFormatError`, `PasswordRequiredError`,
`WrongPasswordError`, `CorruptDataError`, `MissingCodecError`, `TargetNotFoundError`,
`AmbiguousTargetError`, `NotAFileError`, `ChecksumError`, `UnsafePathError`, `LimitExceededError`.

## Paths, volumes, and depth

- Paths are `/`-separated from the source root. Containers are unwrapped until a filesystem or archive
  is reached. When a partition map, APFS container, or multi-volume layer has several listable
  volumes, the root lists them as `VOLUME` entries with canonical names: `p1`…`pN` for partitions
  (1-based, like `disk2s1`/`sda1`; MBR logical partitions start at `p5`) and `v1`…`vN` for APFS
  volumes. Single-child layers are collapsed (`flatten=True`).
- Labels (filesystem volume name, else the GPT/APM partition name) are in `extra["label"]` and are
  accepted as aliases in targets: `/Macintosh HD/Users`. Exact match first, then case-insensitive;
  canonical names always win; a label shared by several volumes raises `AmbiguousTargetError`.
  Labels are read lazily (only for `stat=True`, i.e. `ls -l`/`--json`, and alias lookups).
- Unrecognized partitions (swap, empty) appear as `VOLUME` entries with `error` set.
- A path segment that names a container file descends into it: `/Downloads/x.dmg/p1/Applications`.
  As a listing target, `/x.dmg` yields the file entry itself; `/x.dmg/` lists its contents.
- Btrfs subvolumes appear as directories where they are linked; the root is the default subvolume.
- `depth` is relative to the target (0 = direct children). Each directory, volume boundary, and
  (with `nested > 0`) container boundary adds one level.
- HFS+ names are converted from NFD to NFC (`keep_raw_names` keeps them); targets are normalized
  before lookup. Symlinks inside images are followed within the image (loop limit 40); a link that
  escapes the image root raises `TargetNotFoundError`.

## Passwords and crypto

`password` may be a `str`/`bytes`, or a callable receiving a `PasswordRequest(description, hint,
attempt)` and returning a candidate or `None` to give up. Supported: encrypted DMG (v1/v2),
APFS native encryption (user passwords and the recovery key; the stored hint is passed to the
callable), 7z (content and header), RAR5 and RAR4 (stored members and encrypted headers), zip
(ZipCrypto, WinZip AES-128/192/256).

AES and 3DES use `libcrypto` (OpenSSL 1.1/3, including the copy shipped with CPython on Windows) or
CommonCrypto on macOS when available; otherwise a pure-Python T-table AES (~1 MB/s). For encrypted
APFS volumes and bulk reads from encrypted DMGs the native backend matters more than anything else;
`--stats` shows which backend served each codec and cipher. Setting `FS_EXPLORER_NO_NATIVE=1` disables
every system library, as if none were installed.

## `full_read_limit` and spooling

Some members have no random access: a DMG inside a deflated zip member, a zip inside a `.tar.gz`, an
`.img.xz`. When such a member's stored size is within `full_read_limit`, it is spooled to a
`SpooledTemporaryFile` and opened normally; otherwise listing inside it raises `LimitExceededError`
or `UnsupportedFormatError` with the reason. The same limit enables listing formats that need full
decompression (compressed tar/cpio). Reading a single member of a compressed tar never needs it.

## Extraction safety

- `..` components, absolute paths, drive letters, and NUL bytes are rejected (`UnsafePathError`);
  member names that start with `/` are extracted relative to the destination.
- Symlinks are created last, and a symlink whose target is absolute or resolves outside the
  destination is refused; files are never written through a symlink, pre-existing or created during
  the same extraction.
- `max_size` (total bytes) and `max_ratio` (per member, default 1000×) are enforced while writing.
- Case-insensitive collisions, Windows-invalid characters, and reserved names are handled
  (`on_conflict`); holes are skipped with `seek` (`sparse=True`); checksums are verified by default.
- Metadata: mode and mtime by default; `owner` (root only), `xattrs`, `resource_forks` (the named fork
  on macOS, AppleDouble `._` files elsewhere), `symlinks` and `hardlinks` within the extraction set.
- Solid blocks (7z, CAB folders, RAR solid, pbzx, RPM payloads) are processed in storage order, so
  extraction is linear.

## Performance notes

Listing is dominated by B-tree walking and header parsing and typically runs at thousands to tens of
thousands of entries per second. Reading throughput depends on the codec:

| Path | Throughput (Python 3.11) |
|---|---|
| stdlib zlib, bzip2, xz; native zstd/lz4/lzo; libcrypto AES | hundreds of MB/s |
| pure LZ4, LZO | 15–25 MB/s |
| pure zstd, LZFSE, LZVN, Deflate64 | 8–15 MB/s |
| pure LZX, XPRESS | 5–7 MB/s |
| pure AES-CBC / XTS | ~1 MB/s |
| pure PPMd | 0.03–0.15 MB/s |

`tools/bench.py IMAGE --read-largest 3 --backend pure|system` measures entries/s and MB/s on your
own images.

## Limitations

- RAR decompression is not implemented (stored members only).
- APFS: live tree only (no snapshot selection); sealed volumes are untested.
- HFS+: no journal replay; case-insensitive comparison approximates Apple's FastUnicodeCompare.
- Btrfs: no log-tree replay, no RAID0/10/5/6, no seed devices or zoned mode.
- UDF: VAT (virtual) partitions are refused.
- FAT timestamps are local time and reported as if UTC.
- Encrypted DMG handling was validated against images built by our fixture generator and reference
  decryptors, not Apple-made images.

## Development

```
pip install -e .[dev]
pytest -q                      # ~950 tests, about a minute
ruff check src tests tools
mypy
```

Fixtures live in `tests/fixtures/` (xz-compressed images decompressed on first use into
`tests/fixtures/_build/`, with JSON oracles recording paths, types, sizes, and SHA-256 of every
file). `tools/make_fixtures_linux.sh` regenerates them from the standard tree
(`tools/fixtures/tree.py`) with `mkfs.*`, `mksquashfs`, `xorriso`, `qemu-img`, `7z`, `rar`, `gcab`,
`rpmbuild`, and Python builders for Apple formats (HFS+, APFS, UDIF, encrypted DMG, sparse images,
pkg) that Linux tools cannot write. `tools/make_fixtures_macos.sh` builds Apple-native images with
`hdiutil`. Some reference images come from the dfVFS project (Apache 2.0, see
`tests/fixtures/dfvfs/NOTICE`).
