Metadata-Version: 2.5
Name: fs_explorer
Version: 0.5.0
Summary: List and read disk images, filesystems, and archives using only the Python standard library
Project-URL: Repository, https://github.com/ajgouhier/fs_explorer
Author-email: Arthur Gouhier <ajgouhier@gmail.com>
License-Expression: BSD-3-Clause
License-File: LICENSE
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: ziglang==0.15.1; extra == 'dev'
Description-Content-Type: text/markdown

# fs-explorer

List and read the contents of disk images, filesystems, and archives using only the Python standard
library, and create new DMG (APFS, exFAT), SquashFS and ISO 9660 images.

```
pip install fs-explorer        # distribution
import fs_explorer             # package
fs-explorer ls disk.dmg        # command (also: python -m fs_explorer)
```

## Guarantees

- **No dependencies.** Python 3.10+, standard library only. System libraries (`libzstd`, `liblz4`,
  `liblzo2`, `libcrypto`, Apple's `libcompression`/CommonCrypto) are used through `ctypes` when
  present. The package also ships `fsx`, a small library of codec and checksum kernels prebuilt for
  Linux, macOS and Windows (see [Performance](#performance)). It is loaded through `ctypes` too:
  nothing is compiled at install time, and every kernel has a pure-Python fallback.
- **Listing reads metadata only.** Directory records, inode tables, B-tree nodes, central directories,
  and headers. File data is not read. Formats that can only be listed by decompressing everything
  (e.g. `.tar.gz`) are detected and refused with an explicit reason, unless you opt in with
  `full_read_limit`.
- **Reading touches only what the file needs.** Only the blocks, chunks, extents, or solid-block prefix
  that the requested file depends on are read and decoded.
- **Streaming.** Entries are yielded as they are parsed. Memory is bounded by the largest single
  directory plus the cache budget (default 64 MiB), not by image size.
- **Sources are never written.** No mounting, no subprocesses. `extract()` writes only under its
  destination directory; `create` writes one new output file (through a temporary file in the same
  folder, renamed into place when complete) and nothing else.

## Supported formats

### Containers and partition maps

| Format | Notes | What is read to list |
|---|---|---|
| Raw images (`.img`, `.iso`, `.bin`) | detection by content | superblocks |
| UDIF / DMG | raw, zero, zlib, bzip2, LZMA, ADC, LZFSE chunks | koly trailer + plist chunk table |
| Encrypted DMG v1 / v2 | AES-128/256, password | header, key blobs; data decrypted per 4 KiB chunk |
| `.sparseimage`, `.sparsebundle` | bands read lazily; missing bands are zeros | header / `Info.plist` |
| ASIF (`.asif`, macOS 26+), plain or encrypted | whole and partial chunks; unmapped ranges are holes | header, directories, chunk tables |
| VHD (fixed, dynamic, differencing), VHDX (incl. log replay, differencing) | parents via sibling files | footer/headers, BAT |
| VMDK (monolithicSparse, streamOptimized, split, flat, snapshots) | | grain directory/tables |
| QCOW2 (v2/v3, compressed deflate/zstd, backing files, extended L2) | | L1/L2 tables |
| VDI, Android sparse, BIN/CUE (Mode 1, Mode 2 Form 1) | | block map / chunk list / cue sheet |
| GPT, APM, MBR with EBR chains | volumes named `p1`…`pN` | partition tables |

### Filesystems

| Filesystem | Reading extras |
|---|---|
| HFS+ / HFSX | decmpfs compression (zlib, LZVN, LZFSE), hard links, resource forks, xattrs |
| APFS | decmpfs, clones, sparse files, **native encryption** (password or recovery key); volumes `v1`…`vN` |
| ISO 9660 + Joliet + Rock Ridge | multi-extent files, zisofs; multi-session images (the newest session, or `iso_session`); El Torito boot images as `/[BOOT]/Boot-*.img` (with `system_files`, `ls -a`) |
| UDF (1.02–2.60, sparable and metadata partitions) | embedded data, all allocation descriptor kinds |
| FAT12/16/32, exFAT | long names, NoFatChain |
| NTFS | LZNT1 compression, sparse files, named streams, reparse symlinks/junctions, WOF (XPRESS/LZX) |
| ext2/3/4 | extents, indirect blocks, inline data, holes, xattrs |
| SquashFS 4 | gzip, lzma, lzo, xz, lz4, zstd; fragments; xattrs |
| Btrfs (single, DUP, RAID1, RAID1C3/4) | subvolumes, compressed extents (zlib, zstd, LZO), holes |

### Archives and streams

| Format | Listing | Reading |
|---|---|---|
| zip (Zip64) | central directory only | stored, deflate, Deflate64, bzip2, LZMA, xz, zstd, PPMd; ZipCrypto, WinZip AES |
| tar (ustar, GNU, PAX, sparse) | headers only | direct views; compressed tar readable forward-only |
| 7z | header (incl. encrypted header) | LZMA, LZMA2, BCJ/ARM/PPC/SPARC/IA64, Delta, BCJ2, bzip2, Deflate, Deflate64, PPMd, 7zAES |
| RAR5, RAR4 | headers (incl. encrypted headers) | **stored members only** (with decryption); compressed members raise `MissingCodecError` |
| xar and `.pkg` (via Bom) | TOC + Bom `Paths` tree | zlib/bzip2/xz members; Bom-backed payloads through pbzx + cpio |
| cpio (newc, crc, odc, binary), ar / `.deb`, RPM, CAB | headers | CAB: stored, MSZIP, LZX; RPM: payload (gzip, bzip2, xz, zstd) forward-only |
| gzip, xz, zstd, bzip2, lz4 streams | one entry: original name and size when stored | decompressed contents |

### Codec tiers

| Tier | Codecs | Implementation |
|---|---|---|
| core | zlib/deflate, bzip2, LZMA/xz, BCJ, Delta | standard library |
| core | zstd | `compression.zstd` (3.14+) → `libzstd` → `fsx` (decoding) → pure Python |
| core | LZFSE, LZVN, ADC, LZ4, pbzx, CRC32C | `libcompression`/`liblz4` where available → `fsx` → pure Python |
| common | LZO1X, LZNT1, MSZIP, BCJ2, Deflate64, zisofs, ZipCrypto, WinZip AES | `liblzo2` → `fsx` → pure Python |
| heavy | PPMd H and I, LZX, XPRESS Huffman | `fsx` → pure Python (slow: PPMd ≈ 0.3 MB/s) |
| crypto | AES (ECB, CBC, CTR, XTS), 3DES, RFC 3394, PBKDF2, 7z/RAR KDFs | `libcrypto` / CommonCrypto → `fsx` (AES, SHA, unit loops, RAR 3 KDF, ZipCrypto) → pure Python |

### Refused or out of scope

- **Refused for listing** (detected; allowed with `full_read_limit`): compressed tar and cpio, `.deb` data
  payloads, `.pkg` payloads without a Bom, disk images wrapped in a compression stream (`.img.xz`).
  Reading individual members still works by streaming.
- **Refused for reading:** NTFS EFS, ext4 fscrypt, APFS per-file (hardware-bound) keys, QCOW2
  encryption, VDI differencing images, RAR compressed members.
- **Not detected:** NDIF, Disk Copy 4.2, E01, WIM/ESD, XFS, f2fs, multi-device Btrfs striping,
  multi-volume archives, BitLocker, LUKS, StuffIt, plain (non-wrapped) HFS.

## Command line

```
fs-explorer ls      [OPTIONS] SOURCE [TARGET]
fs-explorer stat    [OPTIONS] SOURCE [PATH]
fs-explorer cat     [OPTIONS] SOURCE PATH
fs-explorer extract [OPTIONS] SOURCE [PATH...] -o DEST
fs-explorer probe   [OPTIONS] SOURCE
fs-explorer create  dmg|squashfs|iso [OPTIONS] OUT [SRC...]
```

```console
$ fs-explorer probe Install.dmg
UDIF (UDZO) > GPT (512-byte sectors) > p2 "Installer" > HFS+

$ fs-explorer ls -l disk.img /p2/Users -r 0
drwxr-xr-x      501       20              2024-05-01 10:12 /p2/Users/alice
...

$ fs-explorer ls backup.tar /old/disk.dmg          # a container lists its contents
$ fs-explorer stat --summary backup.tar /old       # one entry, with totals below a folder
$ fs-explorer cat disk.vhdx /p1/Windows/win.ini
$ fs-explorer cat --offset 1M --length 4096 image.qcow2 /p1/big.bin | xxd
$ fs-explorer extract release.7z /docs /README -o out/ -v
$ fs-explorer ls --json encrypted.dmg --ask-password
```

`ls` lists a folder, volume or container; naming a plain file is a usage error (exit 2) — use
`stat` for one entry.

Common options: `--password PW`, `--password-file PATH`, `$FS_EXPLORER_PASSWORD` (stored for every
layer), `--ask-password` (prompts per encrypted layer, showing its path and the APFS hint),
`--format NAME`, `--backend auto|pure|system`, `--cache-size 64M`, `--full-read-limit SIZE`,
`--errors raise|yield|skip` (default `yield`: errors go to stderr, listing continues), `--stats`,
`--iso-session N` (multi-session ISO: read session N instead of the newest), `--threads N` (threads
that decode ahead and extract big files; default 0: one per CPU, up to 16; 1: none).

`ls`: `-r N` recursion depth (default unlimited, 0 = direct children), `-n N` nested container
descent, `--no-stat`, `-i/-x PATTERN` include/exclude (excluded directories are pruned),
`-T f,d,l,v` entry types, `-a` system files, `--no-flatten`, `--sort`, `-l`, `--json` (JSON Lines),
`-0`, `--relative`.

`stat`: `-L` follow a final symlink, `--container` detect a container file, `--summary` totals below
a folder, `--json`.

`cat`: `--offset`, `--length`, `--no-follow`, `--verify`.

`extract`: several paths (default `/`), `-o DEST`, `--base PATH` (names are relative to it; default:
the paths' common parent), `-r N`, `-n N` (containers inside are written as folders of their
contents), `--preserve mode,mtime,owner,symlinks,hardlinks,xattrs,resource_forks`,
`--on-conflict error|skip|overwrite|rename`, `--links skip_escaping|refuse|keep` (`keep` needs
`--unsafe`), `--volumes canonical|labels`, `--max-size`, `--max-ratio`, `--no-verify`, `--no-sparse`,
`-v`. Skipped links, devices and unreadable volumes are counted on stderr.

`create`: see [Creating images](#creating-images).

Exit codes: 0 success; 1 some entries had errors; 2 usage error (including `ls` on a file);
3 unsupported format or missing codec; 4 missing or wrong password; 5 target not found; 6 unsafe path
or limit exceeded.

## Python API

```python
import fs_explorer as fx

with fx.Source("disk.dmg") as src:                         # parsed layers are reused across calls
    src.unlock("/", "secret")                              # passwords are stored per layer (see below)
    for e in src.entries("/p2/Users", recursive=1):        # folders, volumes and containers
        print(e.path, e.type.name, e.size, e.mtime)
    e = src.stat("/p2/Users/alice/notes.txt")              # any node: file, folder, volume, "/"
    s = src.summary("/p2/Users")                           # counts, bytes, newest mtime, per child
    with src.open_file("/p2/big.iso") as f:                # io.RawIOBase; seekable for most sources
        f.seek(1 << 30)
        chunk = f.read(4096)
    for w in src.walk(["/p2/Users/alice", "/p2/etc/hosts"], base="/p2"):   # storage order
        if w.open:
            with w.open() as f:
                upload(w.rel, f)
    res = src.extract(["/p2/Applications"], "out/", links="skip_escaping", progress=print)
    print(res.count, res.bytes, res.skipped, res.errors)
    print(src.stats.as_dict())                             # bytes read, cache hits, decompressed, backends

print(fx.detect("mystery.bin"))                            # cheap: head and tail only
data = fx.read_file("backup.7z", "/docs/report.pdf", password="secret")
```

Module-level shortcuts open a `Source` for one call: `iter_entries(source, target="/", **opts)`,
`stat`, `summary`, `read_file`, `open_file`, `extract(source, paths, dest, **opts)`, `probe`,
`detect`, `open_source`. These (unlike `Source`) accept a plain-string `password`, stored for every
layer. A source is a path, a seekable binary file object, `bytes`, or a directory (`.sparsebundle`).

- **`Source(source, *, password=None, cancel=None, progress=None, mounts=8, spool_dir=None,
  spool_limit=None, resolver=None, cache_size=64<<20, backend="auto", full_read_limit=0,
  format_hint=None, flatten=True, system_files=False, keep_raw_names=False, iso_session=None,
  threads=0)`.**
  `password` is a provider callable or None; `cancel`/`progress` are defaults for every call.
  `system_files=True` also lists an ISO's El Torito boot images under `/[BOOT]`. `iso_session=N`
  reads session N (1-based) of a multi-session ISO instead of the newest. `threads` sizes the pool
  that decodes chunks ahead of reads and extracts big files (0: one per CPU, up to 16; 1: none).
- **`entries(target="/", *, recursive=-1, nested=0, stat=True, include=(), exclude=(), types=None,
  errors="raise", sort=False, containers=False, container_limit=2000, cancel=None)`** lists below a
  folder, volume or container file; a plain file raises `NotAFolderError`. `errors` applies to
  entries under the target; an unresolvable target always raises, on the first iteration.
  `containers=True` fills
  `Entry.container` by probing each file's head and tail, without descending (for at most
  `container_limit` files per listing).
- **`stat(path, *, follow_symlinks=False, container=False)`** returns one `Entry`.
- **`summary(path="/")`** returns `Summary(count, files, folders, bytes, stored, newest_ns, children,
  exact)`; `children` maps each direct child to its rolled-up `ChildSummary`. Flat archives answer
  from an index built once; filesystems are walked on demand and the result is cached.
- **`walk(paths, *, base="/", recursive=-1, order="auto", nested=0, measure=False, cancel=None,
  progress=None, errors="raise")`** yields `Walked(entry, rel, open)` for a selection (overlaps are
  merged). `order="tree"` yields in tree order. `order="storage"` yields folders as they are met,
  then files in the order their data is stored (each solid block is decoded once, front to back),
  then links and devices. `order="auto"` (the default) uses storage order for solid archives (7z,
  CAB, solid RAR, pbzx, compressed tar/cpio, RPM, XAR) and tree order elsewhere, so a large
  filesystem folder streams without a full walk first. Hard links always come after the files.
- **`extract(paths, dest, *, base="/", recursive=-1, preserve=..., on_conflict="error",
  links="skip_escaping", unsafe=False, volumes="canonical", nested=0, order="auto", max_size=None,
  max_ratio=1000.0, verify=True, sparse=True, measure=False, errors="raise", cancel=None,
  progress=None)`** returns `ExtractResult(count, bytes, skipped, errors)`; `skipped` counts
  `devices`, `links`, `hardlinks`, `system` (system files left out: NTFS `$` files, HFS+ private
  folders), `unreadable_volumes` (swap, empty or unrecognized partitions, skipped rather than
  raised) and `resource_forks`.
- **`detect(source)`** returns `Detected(format, layer, listable, encrypted, reason)` or None,
  reading at most the first and last 64 KiB.
- **`unlock(path, password)`, `locked()`, `release(path)`, `probe()`, `close()`**: see below.

**Cancellation and progress.** `cancel` is a callable returning True to stop (a
`threading.Event().is_set` works); it is polled in every hot loop (layer opens, B-tree and catalog
walks, archive header decoding, index builds, spooling, copying, skip-forward in solid blocks) and
raises `Cancelled`. A per-call `cancel` overrides the source's, and applies to reads of an
`ImageFile` the call returned. Cancelling leaves the `Source` usable: half-built indexes and partial
spools are discarded, and a file being extracted is removed. `progress` receives
`ProgressEvent(phase, path, done, total, entries)` with phase `measure` (once, with the total),
`spool`, `walk` (once per item `walk` hands out) or `copy` (bytes read or written, at most every
64 KiB); `create.dmg` adds `image` and `encrypt` (see below). `walk` reports real bytes read from the files it returned, not entry sizes; `measure=True`
on `walk` or `extract` computes `total` first.

**Seeking in compressed members.** `ImageFile.seekable()` is True for stored data and for
compressed data with restart points: deflate (zip, gzip, tar.gz; a decoder snapshot every 4 MiB of
output, at most 64 per stream), bzip2 (block boundaries), multi-block xz (the stream index), zstd
(frames or the seekable-format seek table), pbzx chunks and CAB MSZIP blocks, including AES- and
ZipCrypto-encrypted zip members. A seek then costs at most about one interval of decoding. Raw
LZMA/LZMA2 (7z folders, zip method 14), lz4, Deflate64 and PPMd still re-decode from the start on a
backward seek and report `seekable() == False`.

**Threads.** Every public `Source` method may be called from several threads at once; an
`ImageFile` handle belongs to one thread. Readers of members of one solid block get their own
decoders (a small pool per block), so they neither corrupt nor block each other. `extract` copies
files of 1–64 MiB on the source's thread pool while it goes on with the rest, when their codec runs
outside the GIL; results, errors and progress events still come in walk order, from the calling
thread.

`Entry` fields: `path`, `name`, `type` (`FILE`, `DIR`, `SYMLINK`, `HARDLINK`, `CHAR_DEVICE`,
`BLOCK_DEVICE`, `FIFO`, `SOCKET`, `VOLUME`, `OTHER`), `depth`, `size`, `stored_size`, `mode`, `uid`,
`gid`, `owner`, `group`, `mtime_ns`/`ctime_ns`/`atime_ns`/`btime_ns` (ns since the Unix epoch, UTC;
`.mtime` etc. give `datetime`s), `nlink`, `inode`, `link_target`, `format`, `container`, `extra`,
`error`, `error_code`. `entry.to_dict()` is JSON-ready.

Errors derive from `FsExplorerError` and carry a stable `code`: `UnsupportedFormatError`
(`unsupported_format`), `PasswordRequiredError` (`password_required`), `WrongPasswordError`
(`wrong_password`), `CorruptDataError` (`corrupt`), `ChecksumError` (`checksum`), `MissingCodecError`
(`missing_codec`), `TargetNotFoundError` (`not_found`), `AmbiguousTargetError` (`ambiguous`),
`NotAFileError` (`not_a_file`), `NotAFolderError` (`not_a_folder`), `UnsafePathError`
(`unsafe_path`), `LimitExceededError` (`limit`), `Cancelled` (`cancelled`), `ExtractConflictError`
(`conflict`) and `CreateError` (`create`, with a `reason`). Raw parser exceptions (`struct.error`, `IndexError`, …) never escape: they come out as
`CorruptDataError`.

## Creating images

```console
$ fs-explorer create dmg out.dmg folder/ --fs apfs --format ULFO --volname Backup
$ fs-explorer create dmg secret.dmg folder/ --encryption AES-256 --ask-password
$ fs-explorer create dmg blank.dmg --fs exfat --size 2G --volname Stick      # empty volume
$ fs-explorer create squashfs root.sqfs rootfs/ --comp zstd -b 1M -e '*.pyc'
$ fs-explorer create squashfs docs.sqfs --from-source backup.tar:/home/alice/docs
$ fs-explorer create iso disc.iso folder/ --volume-id BACKUP --boot isolinux/isolinux.bin --boot-info-table
```

```python
import fs_explorer as fx
from fs_explorer import create

res = create.dmg("out.dmg", ["folder"], fs="apfs", format="ULFO", encryption="AES-256", password="pw")
res = create.squashfs("out.sqfs", ["rootfs"], compression="zstd", block_size=1 << 20)
res = create.squashfs("docs.sqfs", [create.FromSource(fx.Source("backup.tar"), "/home/alice/docs")])
res = create.iso("disc.iso", ["folder"], volume_id="BACKUP", boot=[create.BootImage("efi.img", platform="efi")])
print(res.size, res.files, res.duplicates, res.skipped)      # CreateResult
```

**Inputs.** Host files and folders, or `FromSource(source, path)`: a folder inside anything
fs-explorer reads (an archive, a partition of another image), so converting a tar or a DMG into a
SquashFS is ordinary input. One source folder becomes the image root; a single file or several
sources become top-level entries. `exclude` takes globs (a pattern with `/` matches the relative
path, otherwise any name; `**` crosses folders); `one_file_system` stays on the source's device.
The tree is scanned first (counts and sizes), then each file is read once; a file that changes size
in between raises `CreateError` (`source_changed`). Every call takes `cancel`, `progress` (phases
`measure` and `copy`, in source bytes; then, for `create.dmg`, `image` while the disk image is written
or compressed (a UDIF image is encrypted as it is written; a bare `UDRW` disk is built in place and
has no such pass) and `encrypt` while an encrypted `UDRW` or ASIF image is encrypted, each from 0 to
the bytes of that pass, at most every MiB), `processors` (a thread pool for block compression;
pure-Python encoders such as LZFSE run in up to that many worker processes instead, at most one per
96 MB of available memory, falling back to threads where processes cannot start; output does not
depend on it) and `overwrite` (an existing destination is refused otherwise). `cancel` is
polled in every phase (scan, layout, file data, metadata, image writing, encryption); a cancelled or
interrupted call (Ctrl-C in the CLI, exit status 130) raises `Cancelled` and leaves nothing behind.

**DMG.** `create.dmg(dest, sources=(), *, fs="apfs"|"exfat", format="UDRW", level=None, size=None,
volname="untitled", layout="GPT"|"MBR"|"NONE", case_sensitive=False, encryption=None|"AES-128"|"AES-256",
password=None, symlinks="skip", appledouble=False, kdf_iterations=149253, ...)`.

| Option | Values |
|---|---|
| `format` | `UDRW` (the bare disk, read-write, no UDIF trailer, as hdiutil writes it; holes stay sparse), `UDRO` (uncompressed UDIF, read-only: data stored packed, free space takes none), `UDZO` (zlib), `UDBZ` (bzip2), `ULFO` (LZFSE), `ULMO` (LZMA); CRC32 checksums for all but `UDRW`. `ASIF` (Apple Sparse Image Format, macOS 26+; read-write, 1 MiB chunks stored whole, empty chunks take no space; conventionally `.asif`) |
| `level` | compression level (`--level`): `UDZO` and `UDBZ` 1–9 (default 1 and 9, as hdiutil), `ULMO` 0–9 (xz preset, default 6); refused for `UDRW`, `UDRO`, `ULFO` (LZFSE has no levels) and `ASIF` |
| `size` | volume size (`"512m"`, `"2G"`); default: the content plus free space (10 %, at least 8 MiB for exFAT and 16 MiB for APFS); required for an empty volume |
| `encryption` | `encrcdsa` v2 as `hdiutil -encryption`: PBKDF2-HMAC-SHA1, AES-wrapped key blob, AES-CBC per 512-byte block with HMAC-SHA1 IVs, over the raw disk for `UDRW` and over the whole UDIF or ASIF file otherwise (an ASIF's header records no data size, as diskutil writes it) |
| `layout` | `GPT` (hdiutil `GPTSPUD`), `MBR` (exFAT only: macOS does not mount APFS on MBR, and neither hdiutil nor diskutil makes it) or `NONE` (the volume is the whole disk) |
| APFS | one volume, case-insensitive (or `case_sensitive=True`); xattrs (but not macOS's host-local `com.apple.provenance`), resource forks, symlinks, hard links, devices, sparse files; no snapshots, no native encryption, no compression |
| exFAT | macOS's layout (cluster size, FAT and cluster-heap alignment as `newfs_exfat`); contiguous files; symlinks `skip` (default, reported), `follow` or `error`; devices and hard-link sharing are not representable; xattrs dropped and reported (read by name only, not value) unless `appledouble=True` writes `._name` files (Linux `security.`/`system.`/`trusted.` and macOS `com.apple.provenance` are ignored silently) |

**SquashFS 4.0.** `create.squashfs(dest, sources, *, compression="zstd"|"gzip"|"lz4"|"xz",
block_size=128 KiB, level=None, lz4_hc=False, gzip_window=15, duplicates=True, exportable=True,
fragments=True, always_fragments=False, sparse=True, xattrs=True, hard_links=True,
compress_inodes/dirs/data/fragments/xattrs/ids=True, all_root=False, uid=None, gid=None,
root_mode=None, all_time=None, mkfs_time=None, pad=True, ...)`. These mirror `mksquashfs`
(`-comp`, `-b`, `-Xcompression-level`, `-Xhc`, `-no-duplicates`, `-no-exports`, `-no-fragments`,
`-always-use-fragments`, `-no-sparse`, `-no-xattrs`, `-no-hardlinks`, `-noI -noD -noF -noX -noId`,
`-all-root`, `-force-uid/-gid`, `-root-mode`, `-all-time`, `-mkfs-time`, `-nopad`); the CLI spells
them `--comp`, `--block-size`, `--level`, `--no-duplicates`, `--uncompressed-inodes`, … Duplicate
files share blocks only after a byte-for-byte comparison. Times default to `$SOURCE_DATE_EPOCH` when
set, and output is byte-identical for identical inputs and options. Images are always new (no append
mode).

**ISO 9660.** `create.iso(dest, sources, *, volume_id="CDROM", rock_ridge=True, joliet=True,
level=None, udf=False, zisofs=False, boot=None, duplicates=False, publisher=None, preparer=None,
application="fs-explorer", system_id=None, all_time=None, mkfs_time=None, boot_catalog="boot.catalog",
...)`, laid out as `genisoimage -R -J`: level 1 primary names (8.3, collisions mangled as
genisoimage does; `level=2|3` allows 31 characters), Rock Ridge (RRIP 1.12: POSIX names, modes,
owners, times, symlinks, hard links, devices; folders deeper than 8 levels moved to `rr_moved`) and
Joliet (UCS-2 names of at most 64 units; shortened or changed names are listed in `skipped`).
Files over 4 GiB get multi-extent records and level 3. `zisofs` stores files that shrink in zisofs
format (Rock Ridge `ZF`, read transparently by Linux); `duplicates` stores identical files once.
`boot` takes `BootImage(path, platform="bios"|"efi", emulation="none"|"floppy", load_size=None,
boot_info_table=False)` entries naming files of the image: an El Torito catalog (`boot.catalog`)
with no-emulation BIOS and EFI entries, without isohybrid. Times default to `$SOURCE_DATE_EPOCH`;
output is byte-identical for identical inputs and options. `udf=True` adds a UDF 1.02 bridge laid out
as `genisoimage -udf` (one partition from sector 257, File Entries pointing at the same data as the ISO
trees; modes, owners, times and hard links carried; symlinks and devices left out and listed in
`skipped`); it is added on its own, with a note in `skipped`, when a file is 4 GiB or more (unless
`zisofs`, which it excludes), so macOS and Windows can read such files.

**Verification.** SquashFS images are checked with `unsquashfs`, ISO images with `isovfy`, `isoinfo`,
libcdio's `iso-info`, 7-Zip, `udfinfo` and against `genisoimage`, exFAT volumes with `fsck.exfat`,
APFS containers with apfsprogs' `fsck.apfs` and fs-explorer's own readers; UDIF output is accepted
by 7-Zip. Acceptance by macOS itself (`hdiutil verify`, attach, `fsck_apfs`/`fsck_exfat`, writing to
the mounted volume) is checked by `tools/check_created_macos.py`, run on a Mac; the writers' conventions
are compared with images made by macOS 27's `diskutil` in `tests/fixtures/macos`.

**Encoders.** Compression follows the reading side: standard library (zlib, bzip2, xz, and zstd on
Python 3.14+), then `libzstd`, `liblz4`, `libcompression` through `ctypes`, then `fsx`, then pure
Python. `fsx` writes the same bytes as the pure LZFSE/LZVN encoders and as `liblz4`, so an
image does not depend on which of them ran (Apple's `libcompression` writes its own LZFSE). The pure
encoders are slow (LZ4 about 7 MB/s, zstd about 3 MB/s, LZFSE about 2 MB/s, less at high zstd
levels); when they are the only choice, `processors` runs them in worker processes.

## Paths, volumes, and depth

- Paths are `/`-separated from the source root. Containers are unwrapped until a filesystem or archive
  is reached. When a partition map, APFS container, or multi-volume layer has several listable
  volumes, the root lists them as `VOLUME` entries with canonical names: `p1`…`pN` for partitions
  (1-based, like `disk2s1`/`sda1`; MBR logical partitions start at `p5`) and `v1`…`vN` for APFS
  volumes. Single-child layers are collapsed (`flatten=True`).
- Labels (filesystem volume name, else the GPT/APM partition name) are in `extra["label"]`;
  `extra["display"]` is the label when it is unique among the volumes and not itself a canonical
  name, else the canonical name. Labels are accepted as aliases in paths: `/Macintosh HD/Users`.
  Exact match first, then case-insensitive; canonical names always win; a label shared by several
  volumes raises `AmbiguousTargetError`. Filesystem labels are read lazily (only for `stat=True`,
  i.e. `ls -l`/`--json`, and alias lookups). `extract(volumes="labels")` names volume folders by
  their display names.
- Unrecognized partitions (swap, empty) appear as `VOLUME` entries with `error` set.
- A path segment that names a container file descends into it: `/Downloads/x.dmg/p1/Applications`;
  `entries("/x.dmg")` lists inside the DMG and `stat("/x.dmg")` gives the file.
- Btrfs subvolumes appear as directories where they are linked; the root is the default subvolume.
- `depth` is relative to the target (0 = direct children). Each directory, volume boundary, and
  (with `nested > 0`) container boundary adds one level.
- HFS+ names are converted from NFD to NFC (`keep_raw_names` keeps them); targets are normalized
  before lookup. Symlinks inside images are followed within the image (loop limit 40); a link that
  escapes the image root raises `TargetNotFoundError`.

## Passwords and crypto

Passwords are stored per layer on an open `Source`, so unlocking one layer re-parses nothing else:

```python
src = fx.Source("disk.dmg", password=my_provider)   # a provider, or None
src.unlock("/", "outer-secret")          # the DMG itself; "/" also applies to every layer below
src.unlock("/v2", "volume-secret")       # one APFS volume; the others stay locked
src.locked()                             # [LockedLayer(path, kind, description, hint), ...]
```

`unlock` tries the password at once where the format allows it (DMG key blob, APFS keybag, 7z or
RAR encrypted headers; for an archive already open with encrypted content, it decodes the first
encrypted block) and raises `WrongPasswordError` for a wrong one, which is not kept. Zip passwords
are per member and checked when a member is read. A provider receives
`PasswordRequest(description, hint, attempt, path, kind)`: `path` is the virtual path of the layer
asking (`"/"`, `"/v2"`, `"/inner.7z"`, a zip member's own path) and `kind` one of `udif`,
`apfs_volume`, `7z_header`, `7z_content`, `zip`, `rar_header`, `rar_content`; returning None gives
up. Supported: encrypted DMG (v1/v2), APFS native encryption (user passwords and the recovery key;
the stored hint is passed along), 7z (content and header), RAR5 and RAR4 (stored members and
encrypted headers), zip (ZipCrypto, WinZip AES-128/192/256).

AES and 3DES use `libcrypto` (OpenSSL 1.1/3, including the copy shipped with CPython on Windows) or
CommonCrypto on macOS when available; otherwise `fsx`'s AES (hundreds of MB/s), and a pure-Python
T-table AES (~1 MB/s) when neither loads. Encrypted DMG units and APFS XTS units are processed in
batches by an `fsx` loop around the system cipher. `--stats` shows which backend served each codec
and cipher. Setting `FS_EXPLORER_NO_NATIVE=1` disables every system library and `fsx`, as if none
were installed.

## `full_read_limit`, spooling and mounts

Some members have no random access: a DMG inside a deflated zip member, a zip inside a `.tar.gz`, an
`.img.xz`. When such a member's stored size is within `full_read_limit`, it is spooled to a
temporary file (under `spool_dir` when given) and opened normally; otherwise listing inside it raises
`LimitExceededError` or `UnsupportedFormatError` with the reason. The same limit enables listing
formats that need full decompression (compressed tar/cpio). Reading a single member of a compressed
tar never needs it.

Nested mounts (containers inside containers, volumes) are kept in an LRU of `mounts` entries (the
outer stack is never evicted), and `spool_limit` bounds the total size of live spools. An evicted
mount is closed, spool included, as soon as nothing uses it any more (a call in progress, a walk
listing it, an open file read from it, a mount nested in it); a `Walked.open` whose mount was
closed meanwhile reopens the file by path. `release(path)` closes the mount at `path` and
everything under it at once.

## Extraction safety

- An archive member whose name has a `..` component (or is empty) is left out of the listing: no
  `..` folder is synthesized. It is reported as one error entry (`error_code="unsafe_path"`,
  `name` the raw member name, `path` the archive's root) that `errors="skip"` drops, and `extract`
  raises `UnsafePathError` for it (or collects it with `errors="yield"`). A leading `/` is dropped:
  `/abs.txt` lists at the root.
- `..` components, absolute paths, drive letters, and NUL bytes are rejected (`UnsafePathError`);
  member names that start with `/` are extracted relative to the destination.
- Symlinks are created last. A symlink whose target is absolute or resolves outside the destination
  is left out and counted (`links="skip_escaping"`, the default — disk images routinely hold links
  like `Applications -> /Applications`), refused (`links="refuse"`, raises `UnsafePathError`), or
  written as is (`links="keep"`, only with `unsafe=True`). Files are never written through a
  symlink, pre-existing or created during the same extraction.
- `max_size` (total bytes) and `max_ratio` (per member, default 1000×) are enforced while writing.
- Case-insensitive collisions, Windows-invalid characters, and reserved names are handled
  (`on_conflict`); holes are skipped with `seek` (`sparse=True`); checksums are verified by default.
- Metadata: mode and mtime by default (folders always keep owner read, write and enter permission, so an
  extracted tree stays usable and deletable); `owner` (root only), `xattrs`, `resource_forks` (the named fork
  on macOS, AppleDouble `._` files elsewhere), `symlinks` and `hardlinks` within the extraction set.
- Files are written in storage order (see `walk`), so solid blocks (7z, CAB folders, RAR solid, pbzx,
  RPM payloads) are decoded once, front to back.

## Performance

**Backends.** `backend="auto"` (the default) takes, per codec and cipher, the standard library, then a
system library, then `fsx`, then pure Python. `"system"` is the same without pure Python
(`MissingCodecError` instead); `"pure"` never loads a native library. `--stats` lists what served
each codec.

**`fsx`.** The bundled library (`fs_explorer/_native/`) holds LZFSE/LZVN (both directions), PPMd H
and I, LZX, XPRESS, LZNT1, ADC, Deflate64, BCJ2, the zstd decoder and LZ4 (fallbacks for hosts
without the system libraries), AES and SHA kernels with batch loops for DMG and APFS units, and the
Fletcher-64, CRC-32C, exFAT and FAT-chain loops of the filesystem readers. It is built for Linux
(x86_64, aarch64; no libc is used, so glibc and musl alike), Windows (x86_64, arm64) and macOS
(arm64), imports nothing, allocates nothing, and checks every access against the lengths it is
given. Other platforms (Intel Macs among them) use system libraries and pure Python. Each kernel
passes a known-answer test before first use; one that fails is disabled with a warning.
`fs_explorer.native.info()` reports where it was loaded from and which kernels are on.

| Variable | Effect |
|---|---|
| `FS_EXPLORER_NO_NATIVE=1` | no system library and no `fsx` |
| `FS_EXPLORER_NO_BUNDLED=1` | no `fsx` (system libraries still used) |
| `FS_EXPLORER_NATIVE_LIB=PATH` | load `fsx` from this file instead of the packaged one |

An embedder that has no files, such as an agent run from memory, passes the library's bytes to
`fs_explorer.native.provide(blob)` before first use. On Linux it is loaded through `memfd_create`
and never touches the disk; where memory-backed files may not be executed, `allow_tempfile=True`
permits a temporary file instead (unlinked at once), and without it `fsx` stays off. On macOS and
Windows a provided blob goes through a temporary file deleted after loading.

**Threads.** zlib, bzip2, xz, `libcrypto` and every `fsx` call run outside the GIL. Reads that run
on through a compressed image (UDIF, encrypted DMG, qcow2, VMDK, ASIF) decode the next chunks ahead
on the source's thread pool, as do indexed xz and zstd streams, bzip2 blocks and pbzx chunks; and
`extract` copies files of 1–64 MiB in parallel. `threads=1` turns all of it off. Pure-Python codecs
gain nothing from threads, so nothing is read ahead for them.

| Path (x86_64, Python 3.14, one thread) | Throughput |
|---|---|
| stdlib zlib, bzip2, xz; zstd (`compression.zstd`, `libzstd`, or `fsx` for decoding); LZ4 (`liblz4` or `fsx`); `libcrypto` AES | hundreds of MB/s and more |
| `fsx` LZFSE/LZVN decode | 0.4–0.9 GB/s |
| `fsx` LZX, XPRESS, LZNT1, ADC, Deflate64, BCJ2 | 0.2–0.8 GB/s |
| `fsx` LZFSE encode | 30–90 MB/s |
| `fsx` PPMd | 13–24 MB/s |
| pure LZ4, LZO | 15–40 MB/s |
| pure zstd, LZFSE, LZVN, Deflate64 | 8–20 MB/s |
| pure LZX, XPRESS | 5–15 MB/s |
| pure AES-CBC / XTS | ~1 MB/s |
| pure PPMd | 0.1–0.35 MB/s |

`tools/bench.py IMAGE --read-largest 3 --backend pure|system` measures entries/s and MB/s on your
own images; `tools/bench.py --suite` builds a set of images under `local/bench/` and times listing,
lookups, reads, extraction and creation for each backend (`--src DIR` to compare two trees,
`--python 3.10,…` to run under container images of other Python versions, `--compare A.json B.json`).

## Limitations

- RAR decompression is not implemented (stored members only).
- APFS: live tree only (no snapshot selection); sealed volumes are untested.
- HFS+: no journal replay; case-insensitive comparison approximates Apple's FastUnicodeCompare.
- Btrfs: no log-tree replay, no RAID0/10/5/6, no seed devices or zoned mode.
- UDF: VAT (virtual) partitions are refused.
- FAT timestamps (and exFAT ones without a UTC offset) are local time and reported as if UTC.
  Zip and CAB DOS times are converted as local time in the reader's zone, as `zipfile`, Info-ZIP
  and 7-Zip do; a zip member's extended (`UT`) or NTFS timestamp, which is UTC, takes precedence.

## Development

```
pip install -e .[dev]
pytest -q                      # ~2200 tests, two to three minutes; run locally (no hosted CI)
FS_EXPLORER_NO_BUNDLED=1 pytest -q   # the same without fsx
ruff check src tests tools
mypy
python tools/build_native.py   # rebuild fsx for every target (zig from the dev extra)
python tools/build_native.py --check   # rebuild into a temporary folder and compare with the manifest
```

`src/fs_explorer/_native/README.md` describes the C sources, the third-party code they include
(the zstd decoder, LZ4, LZFSE; BSD), the fuzzing harness (`tests/fuzz_native*.py`, also under ASan/UBSan) and how
to add a kernel group.

Fixtures live in `tests/fixtures/`, every one committed as `NAME.xz` (even archives and streams
that are already compressed; `stream-*.xz` are the xz subjects themselves). Tests ask for `NAME`,
and `fixture_path` decompresses it on first use into `tests/fixtures/_build/`, an ignored local
cache that can be deleted at any time. JSON oracles record paths, types, sizes, and SHA-256 of every
file. Tests run with `TZ=UTC` (zip and CAB DOS times depend on the reader's zone). `tools/make_fixtures_linux.sh` regenerates them from the standard tree
(`tools/fixtures/tree.py`) with `mkfs.*`, `mksquashfs`, `xorriso`, `qemu-img`, `7z`, `rar`, `gcab`,
`rpmbuild`, and Python builders for Apple formats (HFS+, APFS, UDIF, encrypted DMG, sparse images,
pkg) that Linux tools cannot write. `tools/make_fixtures_macos.sh` builds Apple-native images with
`hdiutil`, and `tools/check_created_macos.py` checks `create.dmg` output with `diskutil`, `hdiutil` and
`fsck_apfs`/`fsck_exfat` (and captures reference images); `tests/fixtures/macos/` holds images made by
macOS itself: 27 made by macOS 27's `diskutil`/`hdiutil` (DMG and ASIF, see its README) and a raw MBR
image with HFS+ made by Disk Utility. Some reference images come from the dfVFS project (Apache 2.0, see
`tests/fixtures/dfvfs/NOTICE`).
