Metadata-Version: 2.5
Name: fs_explorer
Version: 0.2.0
Summary: List and read disk images, filesystems, and archives using only the Python standard library
Project-URL: Repository, https://github.com/ajgouhier/fs_explorer
Author-email: Arthur Gouhier <ajgouhier@gmail.com>
License-Expression: BSD-3-Clause
License-File: LICENSE
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# fs-explorer

List and read the contents of disk images, filesystems, and archives using only the Python standard
library.

```
pip install fs-explorer        # distribution
import fs_explorer             # package
fs-explorer ls disk.dmg        # command (also: python -m fs_explorer)
```

## Guarantees

- **No dependencies.** Python 3.10+, standard library only. System libraries (`libzstd`, `liblz4`,
  `liblzo2`, `libcrypto`, Apple's `libcompression`/CommonCrypto) are used through `ctypes` when
  present, always with a pure-Python fallback.
- **Listing reads metadata only.** Directory records, inode tables, B-tree nodes, central directories,
  and headers. File data is not read. Formats that can only be listed by decompressing everything
  (e.g. `.tar.gz`) are detected and refused with an explicit reason, unless you opt in with
  `full_read_limit`.
- **Reading touches only what the file needs.** Only the blocks, chunks, extents, or solid-block prefix
  that the requested file depends on are read and decoded.
- **Streaming.** Entries are yielded as they are parsed. Memory is bounded by the largest single
  directory plus the cache budget (default 64 MiB), not by image size.
- **Read-only.** No mounting, no subprocesses, no writes to the source. `extract()` writes only under its
  destination directory.

## Supported formats

### Containers and partition maps

| Format | Notes | What is read to list |
|---|---|---|
| Raw images (`.img`, `.iso`, `.bin`) | detection by content | superblocks |
| UDIF / DMG | raw, zero, zlib, bzip2, LZMA, ADC, LZFSE chunks | koly trailer + plist chunk table |
| Encrypted DMG v1 / v2 | AES-128/256, password | header, key blobs; data decrypted per 4 KiB chunk |
| `.sparseimage`, `.sparsebundle` | bands read lazily; missing bands are zeros | header / `Info.plist` |
| VHD (fixed, dynamic, differencing), VHDX (incl. log replay, differencing) | parents via sibling files | footer/headers, BAT |
| VMDK (monolithicSparse, streamOptimized, split, flat, snapshots) | | grain directory/tables |
| QCOW2 (v2/v3, compressed deflate/zstd, backing files, extended L2) | | L1/L2 tables |
| VDI, Android sparse, BIN/CUE (Mode 1, Mode 2 Form 1) | | block map / chunk list / cue sheet |
| GPT, APM, MBR with EBR chains | volumes named `p1`…`pN` | partition tables |

### Filesystems

| Filesystem | Reading extras |
|---|---|
| HFS+ / HFSX | decmpfs compression (zlib, LZVN, LZFSE), hard links, resource forks, xattrs |
| APFS | decmpfs, clones, sparse files, **native encryption** (password or recovery key); volumes `v1`…`vN` |
| ISO 9660 + Joliet + Rock Ridge | multi-extent files, zisofs |
| UDF (1.02–2.60, sparable and metadata partitions) | embedded data, all allocation descriptor kinds |
| FAT12/16/32, exFAT | long names, NoFatChain |
| NTFS | LZNT1 compression, sparse files, named streams, reparse symlinks/junctions, WOF (XPRESS/LZX) |
| ext2/3/4 | extents, indirect blocks, inline data, holes, xattrs |
| SquashFS 4 | gzip, lzma, lzo, xz, lz4, zstd; fragments; xattrs |
| Btrfs (single, DUP, RAID1, RAID1C3/4) | subvolumes, compressed extents (zlib, zstd, LZO), holes |

### Archives and streams

| Format | Listing | Reading |
|---|---|---|
| zip (Zip64) | central directory only | stored, deflate, Deflate64, bzip2, LZMA, xz, zstd, PPMd; ZipCrypto, WinZip AES |
| tar (ustar, GNU, PAX, sparse) | headers only | direct views; compressed tar readable forward-only |
| 7z | header (incl. encrypted header) | LZMA, LZMA2, BCJ/ARM/PPC/SPARC/IA64, Delta, BCJ2, bzip2, Deflate, Deflate64, PPMd, 7zAES |
| RAR5, RAR4 | headers (incl. encrypted headers) | **stored members only** (with decryption); compressed members raise `MissingCodecError` |
| xar and `.pkg` (via Bom) | TOC + Bom `Paths` tree | zlib/bzip2/xz members; Bom-backed payloads through pbzx + cpio |
| cpio (newc, crc, odc, binary), ar / `.deb`, RPM, CAB | headers | CAB: stored, MSZIP, LZX; RPM: payload (gzip, bzip2, xz, zstd) forward-only |
| gzip, xz, zstd, bzip2, lz4 streams | one entry: original name and size when stored | decompressed contents |

### Codec tiers

| Tier | Codecs | Implementation |
|---|---|---|
| core | zlib/deflate, bzip2, LZMA/xz, BCJ, Delta | standard library |
| core | zstd | `compression.zstd` (3.14+) → `libzstd` → pure Python |
| core | LZFSE, LZVN, ADC, LZ4, pbzx, CRC32C | `libcompression`/`liblz4` where available → pure Python |
| common | LZO1X, LZNT1, MSZIP, BCJ2, Deflate64, zisofs, ZipCrypto, WinZip AES | `liblzo2` → pure Python |
| heavy | PPMd H and I, LZX, XPRESS Huffman | pure Python (slow: PPMd ≈ 0.1 MB/s) |
| crypto | AES (ECB, CBC, CTR, XTS), 3DES, RFC 3394, PBKDF2, 7z/RAR KDFs | `libcrypto` / CommonCrypto → pure Python |

### Refused or out of scope

- **Refused for listing** (detected; allowed with `full_read_limit`): compressed tar and cpio, `.deb` data
  payloads, `.pkg` payloads without a Bom, disk images wrapped in a compression stream (`.img.xz`).
  Reading individual members still works by streaming.
- **Refused for reading:** NTFS EFS, ext4 fscrypt, APFS per-file (hardware-bound) keys, QCOW2
  encryption, VDI differencing images, RAR compressed members.
- **Not detected:** NDIF, Disk Copy 4.2, E01, WIM/ESD, XFS, f2fs, multi-device Btrfs striping,
  multi-volume archives, BitLocker, LUKS, StuffIt, plain (non-wrapped) HFS.

## Command line

```
fs-explorer ls      [OPTIONS] SOURCE [TARGET]
fs-explorer stat    [OPTIONS] SOURCE [PATH]
fs-explorer cat     [OPTIONS] SOURCE PATH
fs-explorer extract [OPTIONS] SOURCE [PATH...] -o DEST
fs-explorer probe   [OPTIONS] SOURCE
```

```console
$ fs-explorer probe Install.dmg
UDIF (UDZO) > GPT (512-byte sectors) > p2 "Installer" > HFS+

$ fs-explorer ls -l disk.img /p2/Users -r 0
drwxr-xr-x      501       20              2024-05-01 10:12 /p2/Users/alice
...

$ fs-explorer ls backup.tar /old/disk.dmg          # a container lists its contents
$ fs-explorer stat --summary backup.tar /old       # one entry, with totals below a folder
$ fs-explorer cat disk.vhdx /p1/Windows/win.ini
$ fs-explorer cat --offset 1M --length 4096 image.qcow2 /p1/big.bin | xxd
$ fs-explorer extract release.7z /docs /README -o out/ -v
$ fs-explorer ls --json encrypted.dmg --ask-password
```

`ls` lists a folder, volume or container; naming a plain file is a usage error (exit 2) — use
`stat` for one entry.

Common options: `--password PW`, `--password-file PATH`, `$FS_EXPLORER_PASSWORD` (stored for every
layer), `--ask-password` (prompts per encrypted layer, showing its path and the APFS hint),
`--format NAME`, `--backend auto|pure|system`, `--cache-size 64M`, `--full-read-limit SIZE`,
`--errors raise|yield|skip` (default `yield`: errors go to stderr, listing continues), `--stats`.

`ls`: `-r N` recursion depth (default unlimited, 0 = direct children), `-n N` nested container
descent, `--no-stat`, `-i/-x PATTERN` include/exclude (excluded directories are pruned),
`-T f,d,l,v` entry types, `-a` system files, `--no-flatten`, `--sort`, `-l`, `--json` (JSON Lines),
`-0`, `--relative`.

`stat`: `-L` follow a final symlink, `--container` detect a container file, `--summary` totals below
a folder, `--json`.

`cat`: `--offset`, `--length`, `--no-follow`, `--verify`.

`extract`: several paths (default `/`), `-o DEST`, `--base PATH` (names are relative to it; default:
the paths' common parent), `-r N`, `-n N` (containers inside are written as folders of their
contents), `--preserve mode,mtime,owner,symlinks,hardlinks,xattrs,resource_forks`,
`--on-conflict error|skip|overwrite|rename`, `--links skip_escaping|refuse|keep` (`keep` needs
`--unsafe`), `--volumes canonical|labels`, `--max-size`, `--max-ratio`, `--no-verify`, `--no-sparse`,
`-v`. Skipped links, devices and unreadable volumes are counted on stderr.

Exit codes: 0 success; 1 some entries had errors; 2 usage error (including `ls` on a file);
3 unsupported format or missing codec; 4 missing or wrong password; 5 target not found; 6 unsafe path
or limit exceeded.

## Python API

```python
import fs_explorer as fx

with fx.Source("disk.dmg") as src:                         # parsed layers are reused across calls
    src.unlock("/", "secret")                              # passwords are stored per layer (see below)
    for e in src.entries("/p2/Users", recursive=1):        # folders, volumes and containers
        print(e.path, e.type.name, e.size, e.mtime)
    e = src.stat("/p2/Users/alice/notes.txt")              # any node: file, folder, volume, "/"
    s = src.summary("/p2/Users")                           # counts, bytes, newest mtime, per child
    with src.open_file("/p2/big.iso") as f:                # io.RawIOBase; seekable for most sources
        f.seek(1 << 30)
        chunk = f.read(4096)
    for w in src.walk(["/p2/Users/alice", "/p2/etc/hosts"], base="/p2"):   # storage order
        if w.open:
            with w.open() as f:
                upload(w.rel, f)
    res = src.extract(["/p2/Applications"], "out/", links="skip_escaping", progress=print)
    print(res.count, res.bytes, res.skipped, res.errors)
    print(src.stats.as_dict())                             # bytes read, cache hits, decompressed, backends

print(fx.detect("mystery.bin"))                            # cheap: head and tail only
data = fx.read_file("backup.7z", "/docs/report.pdf", password="secret")
```

Module-level shortcuts open a `Source` for one call: `iter_entries(source, target="/", **opts)`,
`stat`, `summary`, `read_file`, `open_file`, `extract(source, paths, dest, **opts)`, `probe`,
`detect`, `open_source`. These (unlike `Source`) accept a plain-string `password`, stored for every
layer. A source is a path, a seekable binary file object, `bytes`, or a directory (`.sparsebundle`).

- **`Source(source, *, password=None, cancel=None, progress=None, mounts=8, spool_dir=None,
  spool_limit=None, resolver=None, cache_size=64<<20, backend="auto", full_read_limit=0,
  format_hint=None, flatten=True, system_files=False, keep_raw_names=False)`.** `password` is a
  provider callable or None; `cancel`/`progress` are defaults for every call.
- **`entries(target="/", *, recursive=-1, nested=0, stat=True, include=(), exclude=(), types=None,
  errors="raise", sort=False, containers=False, container_limit=2000, cancel=None)`** lists below a
  folder, volume or container file; a plain file raises `NotAFolderError`. `containers=True` fills
  `Entry.container` by probing each file's head and tail, without descending (for at most
  `container_limit` files per listing).
- **`stat(path, *, follow_symlinks=False, container=False)`** returns one `Entry`.
- **`summary(path="/")`** returns `Summary(count, files, folders, bytes, stored, newest_ns, children,
  exact)`; `children` maps each direct child to its rolled-up `ChildSummary`. Flat archives answer
  from an index built once; filesystems are walked on demand and the result is cached.
- **`walk(paths, *, base="/", recursive=-1, order="storage", nested=0, cancel=None, progress=None,
  errors="raise")`** yields `Walked(entry, rel, open)` for a selection (overlaps are merged). In
  storage order folders come first, then files in the order their data is stored (each solid block
  is decoded once, front to back), then links and devices.
- **`extract(paths, dest, *, base="/", recursive=-1, preserve=..., on_conflict="error",
  links="skip_escaping", unsafe=False, volumes="canonical", nested=0, max_size=None,
  max_ratio=1000.0, verify=True, sparse=True, measure=False, errors="raise", cancel=None,
  progress=None)`** returns `ExtractResult(count, bytes, skipped, errors)`; `skipped` counts
  `devices`, `links`, `hardlinks`, `system`, `unreadable_volumes` and `resource_forks`.
- **`detect(source)`** returns `Detected(format, layer, listable, encrypted, reason)` or None,
  reading at most the first and last 64 KiB.
- **`unlock(path, password)`, `locked()`, `release(path)`, `probe()`, `close()`**: see below.

**Cancellation and progress.** `cancel` is a callable returning True to stop (a
`threading.Event().is_set` works); it is polled in every hot loop (layer opens, B-tree and catalog
walks, archive header decoding, index builds, spooling, copying, skip-forward in solid blocks) and
raises `Cancelled`. A per-call `cancel` overrides the source's, and applies to reads of an
`ImageFile` the call returned. Cancelling leaves the `Source` usable: half-built indexes and partial
spools are discarded, and a file being extracted is removed. `progress` receives
`ProgressEvent(phase, path, done, total, entries)` with phase `measure`, `spool` or `copy`, at most
every 64 KiB and once per entry; `extract(measure=True)` computes `total` first.

**Seeking in compressed members.** `ImageFile.seekable()` is True for stored data and for
compressed data with restart points: deflate (zip, gzip, tar.gz; a decoder snapshot every 4 MiB of
output, at most 64 per stream), bzip2 (block boundaries), multi-block xz (the stream index), zstd
(frames or the seekable-format seek table), pbzx chunks and CAB MSZIP blocks, including AES- and
ZipCrypto-encrypted zip members. A seek then costs at most about one interval of decoding. Raw
LZMA/LZMA2 (7z folders, zip method 14), lz4, Deflate64 and PPMd still re-decode from the start on a
backward seek and report `seekable() == False`.

**Threads.** Every public `Source` method may be called from several threads at once; an
`ImageFile` handle belongs to one thread. Readers of members of one solid block get their own
decoders (a small pool per block), so they neither corrupt nor block each other.

`Entry` fields: `path`, `name`, `type` (`FILE`, `DIR`, `SYMLINK`, `HARDLINK`, `CHAR_DEVICE`,
`BLOCK_DEVICE`, `FIFO`, `SOCKET`, `VOLUME`, `OTHER`), `depth`, `size`, `stored_size`, `mode`, `uid`,
`gid`, `owner`, `group`, `mtime_ns`/`ctime_ns`/`atime_ns`/`btime_ns` (ns since the Unix epoch, UTC;
`.mtime` etc. give `datetime`s), `nlink`, `inode`, `link_target`, `format`, `container`, `extra`,
`error`, `error_code`. `entry.to_dict()` is JSON-ready.

Errors derive from `FsExplorerError` and carry a stable `code`: `UnsupportedFormatError`
(`unsupported_format`), `PasswordRequiredError` (`password_required`), `WrongPasswordError`
(`wrong_password`), `CorruptDataError` (`corrupt`), `ChecksumError` (`checksum`), `MissingCodecError`
(`missing_codec`), `TargetNotFoundError` (`not_found`), `AmbiguousTargetError` (`ambiguous`),
`NotAFileError` (`not_a_file`), `NotAFolderError` (`not_a_folder`), `UnsafePathError`
(`unsafe_path`), `LimitExceededError` (`limit`), `Cancelled` (`cancelled`) and `ExtractConflictError`
(`conflict`). Raw parser exceptions (`struct.error`, `IndexError`, …) never escape: they come out as
`CorruptDataError`.

## Paths, volumes, and depth

- Paths are `/`-separated from the source root. Containers are unwrapped until a filesystem or archive
  is reached. When a partition map, APFS container, or multi-volume layer has several listable
  volumes, the root lists them as `VOLUME` entries with canonical names: `p1`…`pN` for partitions
  (1-based, like `disk2s1`/`sda1`; MBR logical partitions start at `p5`) and `v1`…`vN` for APFS
  volumes. Single-child layers are collapsed (`flatten=True`).
- Labels (filesystem volume name, else the GPT/APM partition name) are in `extra["label"]`;
  `extra["display"]` is the label when it is unique among the volumes and not itself a canonical
  name, else the canonical name. Labels are accepted as aliases in paths: `/Macintosh HD/Users`.
  Exact match first, then case-insensitive; canonical names always win; a label shared by several
  volumes raises `AmbiguousTargetError`. Filesystem labels are read lazily (only for `stat=True`,
  i.e. `ls -l`/`--json`, and alias lookups). `extract(volumes="labels")` names volume folders by
  their display names.
- Unrecognized partitions (swap, empty) appear as `VOLUME` entries with `error` set.
- A path segment that names a container file descends into it: `/Downloads/x.dmg/p1/Applications`;
  `entries("/x.dmg")` lists inside the DMG and `stat("/x.dmg")` gives the file.
- Btrfs subvolumes appear as directories where they are linked; the root is the default subvolume.
- `depth` is relative to the target (0 = direct children). Each directory, volume boundary, and
  (with `nested > 0`) container boundary adds one level.
- HFS+ names are converted from NFD to NFC (`keep_raw_names` keeps them); targets are normalized
  before lookup. Symlinks inside images are followed within the image (loop limit 40); a link that
  escapes the image root raises `TargetNotFoundError`.

## Passwords and crypto

Passwords are stored per layer on an open `Source`, so unlocking one layer re-parses nothing else:

```python
src = fx.Source("disk.dmg", password=my_provider)   # a provider, or None
src.unlock("/", "outer-secret")          # the DMG itself; "/" also applies to every layer below
src.unlock("/v2", "volume-secret")       # one APFS volume; the others stay locked
src.locked()                             # [LockedLayer(path, kind, description, hint), ...]
```

`unlock` tries the password at once where the format allows it (DMG key blob, APFS keybag, 7z
header) and raises `WrongPasswordError` for a wrong one. A provider receives
`PasswordRequest(description, hint, attempt, path, kind)`: `path` is the virtual path of the layer
asking (`"/"`, `"/v2"`, `"/inner.7z"`, a zip member's own path) and `kind` one of `udif`,
`apfs_volume`, `7z_header`, `7z_content`, `zip`, `rar_header`, `rar_content`; returning None gives
up. Supported: encrypted DMG (v1/v2), APFS native encryption (user passwords and the recovery key;
the stored hint is passed along), 7z (content and header), RAR5 and RAR4 (stored members and
encrypted headers), zip (ZipCrypto, WinZip AES-128/192/256).

AES and 3DES use `libcrypto` (OpenSSL 1.1/3, including the copy shipped with CPython on Windows) or
CommonCrypto on macOS when available; otherwise a pure-Python T-table AES (~1 MB/s). For encrypted
APFS volumes and bulk reads from encrypted DMGs the native backend matters more than anything else;
`--stats` shows which backend served each codec and cipher. Setting `FS_EXPLORER_NO_NATIVE=1` disables
every system library, as if none were installed.

## `full_read_limit`, spooling and mounts

Some members have no random access: a DMG inside a deflated zip member, a zip inside a `.tar.gz`, an
`.img.xz`. When such a member's stored size is within `full_read_limit`, it is spooled to a
temporary file (under `spool_dir` when given) and opened normally; otherwise listing inside it raises
`LimitExceededError` or `UnsupportedFormatError` with the reason. The same limit enables listing
formats that need full decompression (compressed tar/cpio). Reading a single member of a compressed
tar never needs it.

Nested mounts (containers inside containers, volumes) are kept in an LRU of `mounts` entries (the
outer stack is never evicted), and `spool_limit` bounds the total size of live spools.
`release(path)` closes the mount at `path` and everything under it.

## Extraction safety

- `..` components, absolute paths, drive letters, and NUL bytes are rejected (`UnsafePathError`);
  member names that start with `/` are extracted relative to the destination.
- Symlinks are created last. A symlink whose target is absolute or resolves outside the destination
  is left out and counted (`links="skip_escaping"`, the default — disk images routinely hold links
  like `Applications -> /Applications`), refused (`links="refuse"`, raises `UnsafePathError`), or
  written as is (`links="keep"`, only with `unsafe=True`). Files are never written through a
  symlink, pre-existing or created during the same extraction.
- `max_size` (total bytes) and `max_ratio` (per member, default 1000×) are enforced while writing.
- Case-insensitive collisions, Windows-invalid characters, and reserved names are handled
  (`on_conflict`); holes are skipped with `seek` (`sparse=True`); checksums are verified by default.
- Metadata: mode and mtime by default; `owner` (root only), `xattrs`, `resource_forks` (the named fork
  on macOS, AppleDouble `._` files elsewhere), `symlinks` and `hardlinks` within the extraction set.
- Files are written in storage order (see `walk`), so solid blocks (7z, CAB folders, RAR solid, pbzx,
  RPM payloads) are decoded once, front to back.

## Performance notes

Listing is dominated by B-tree walking and header parsing and typically runs at thousands to tens of
thousands of entries per second. Reading throughput depends on the codec:

| Path | Throughput (Python 3.11) |
|---|---|
| stdlib zlib, bzip2, xz; native zstd/lz4/lzo; libcrypto AES | hundreds of MB/s |
| pure LZ4, LZO | 15–25 MB/s |
| pure zstd, LZFSE, LZVN, Deflate64 | 8–15 MB/s |
| pure LZX, XPRESS | 5–7 MB/s |
| pure AES-CBC / XTS | ~1 MB/s |
| pure PPMd | 0.03–0.15 MB/s |

`tools/bench.py IMAGE --read-largest 3 --backend pure|system` measures entries/s and MB/s on your
own images.

## Limitations

- RAR decompression is not implemented (stored members only).
- APFS: live tree only (no snapshot selection); sealed volumes are untested.
- HFS+: no journal replay; case-insensitive comparison approximates Apple's FastUnicodeCompare.
- Btrfs: no log-tree replay, no RAID0/10/5/6, no seed devices or zoned mode.
- UDF: VAT (virtual) partitions are refused.
- FAT timestamps are local time and reported as if UTC.
- Encrypted DMG handling was validated against images built by our fixture generator and reference
  decryptors, not Apple-made images.

## Development

```
pip install -e .[dev]
pytest -q                      # ~1100 tests, two to three minutes; run locally (no hosted CI)
ruff check src tests tools
mypy
```

Fixtures live in `tests/fixtures/` (xz-compressed images decompressed on first use into
`tests/fixtures/_build/`, with JSON oracles recording paths, types, sizes, and SHA-256 of every
file). `tools/make_fixtures_linux.sh` regenerates them from the standard tree
(`tools/fixtures/tree.py`) with `mkfs.*`, `mksquashfs`, `xorriso`, `qemu-img`, `7z`, `rar`, `gcab`,
`rpmbuild`, and Python builders for Apple formats (HFS+, APFS, UDIF, encrypted DMG, sparse images,
pkg) that Linux tools cannot write. `tools/make_fixtures_macos.sh` builds Apple-native images with
`hdiutil`. Some reference images come from the dfVFS project (Apache 2.0, see
`tests/fixtures/dfvfs/NOTICE`).
