Metadata-Version: 2.4
Name: spatialdata-js-util
Version: 0.1.0
Summary: Read/write utilities for browser-oriented SpatialData stores (image codecs, Morton-indexed points, CSC tables)
Author: SpatialData.js contributors
License: MIT
Project-URL: Homepage, https://github.com/Taylor-CCB-Group/SpatialData.js
Project-URL: Repository, https://github.com/Taylor-CCB-Group/SpatialData.js
Keywords: spatialdata,zarr,ome-zarr,htj2k,jpeg2000,spatial-omics
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: imagecodecs>=2025.8.2
Requires-Dist: numcodecs>=0.16
Requires-Dist: numpy>=2.0
Requires-Dist: pandas>=2.2
Requires-Dist: pyarrow>=18
Requires-Dist: zarr>=3.3.0
Provides-Extra: write
Requires-Dist: anndata>=0.12; extra == "write"
Requires-Dist: dask[array]>=2026.7.1; extra == "write"
Requires-Dist: ome-zarr>=0.16; extra == "write"
Requires-Dist: scipy>=1.11; extra == "write"
Requires-Dist: spatialdata>=0.8.0; extra == "write"
Provides-Extra: tui
Requires-Dist: textual>=1.0; extra == "tui"
Provides-Extra: all
Requires-Dist: spatialdata-js-util[tui,write]; extra == "all"

# spatialdata-js-util

Python utilities for producing — and reading back — SpatialData stores tuned for
incremental reading in the browser by
[SpatialData.js](https://github.com/Taylor-CCB-Group/SpatialData.js).

Three optimizations, all aimed at making a viewport cost a few small ranged reads
instead of a whole-element download:

| Area | What it does | Why |
|------|--------------|-----|
| **images** | Recompress rasters as JPEG 2000 or HTJ2K, and add [multiscale pyramids](#multiscale-pyramids) where they are missing | Tiles decode in the browser; HTJ2K is markedly cheaper to decode than JPEG 2000. Without a pyramid, a zoomed-out view still costs full-resolution chunks |
| **points** | Morton-sort Points Parquet with sentinel bbox rows and controlled row groups | A viewport maps to a few Parquet row-group reads |
| **tables** | Convert `X`/layers from CSR to CSC | Reading one gene becomes one contiguous range read, not a scan of every row |

Installing the package also **registers the image codecs with zarr**, so stores
written here open in ordinary `spatialdata.read_zarr` — see
[Reading stores back in Python](#reading-stores-back-in-python).

Every command is available three ways: an [interactive TUI](#interactive-tui)
(start here), the `spatialdata-js-util` CLI, and the [Python API](#python-api).

> **Status: experimental.** The HTJ2K codec identifier is not a standard
> OME-Zarr codec. Stores using it are readable by the `SpatialData.js` runtime
> and by Python with this package installed. It is published in the hope that,
> if useful, the approach can be adopted more broadly.

## Install

```bash
pip install spatialdata-js-util
```

That base install is enough to **read** stores. To **write** them, add the
`write` extra, which pulls in `spatialdata`, `anndata`, `dask` and `scipy`:

```bash
pip install 'spatialdata-js-util[write]'
```

Requires Python 3.12+ and `spatialdata >= 0.8.0`.

## Interactive TUI

If you would rather not memorise flags, everything below is also available as a
terminal UI. It is the easiest way to get started.

```bash
pip install 'spatialdata-js-util[tui]'
spatialdata-js-util tui

# Or pre-fill the store path so you skip the first prompt
spatialdata-js-util tui ~/data/xenium.zarr
```

Pick a command from the home menu and the UI walks you through it:

```text
IMAGES   Recompress rasters (JPEG 2000 / HTJ2K)
POINTS   List Points elements in a Zarr store
POINTS   Morton-sort Points from Zarr
POINTS   Morton-sort CSV/Parquet file
POINTS   Write multiscale Points Parquet
POINTS   Write index permutations derivative store
TABLES   List table elements in a Zarr store
TABLES   Convert table matrices to CSC
CODECS   Show HTJ2K backend availability
```

The flow is the same for every command: **guided form → confirm → run → report.**

- **Forms** are pre-filled with sensible defaults, and with the store path you
  already used this session. `Enter` advances between fields and submits on the
  last one; `Escape` goes back; `q` quits.
- **Anything that overwrites asks first.** Rewriting `points/<key>/points.parquet`
  in place, converting tables in place, or replacing an existing output store all
  stop on a confirmation screen naming the exact path before touching anything.
- **Runs stream their JSON output** to a log pane, so a long recompression shows
  progress rather than appearing to hang.
- The recompress form carries the [pyramid](#multiscale-pyramids) controls, so
  single-resolution input can be made browser-ready without leaving the UI.
- **Writes are verified afterwards** and the report screen shows a pass/fail
  table. For Morton writes that means:

  | Check | Meaning |
  |-------|---------|
  | `column_present` | `morton_code_2d` column exists |
  | `sentinel_prefix` | First 2–4 rows have `morton_code_2d == 0` |
  | `sentinel_bbox` | Sentinel rows encode the full dataset x/y bounds |
  | `morton_monotonic` | Morton codes non-decreasing after sentinels |
  | `row_group_sentinels` | Row group 0 contains only sentinel rows |
  | `no_uint_intermediates` | No persisted `*_uint` staging columns |

  Multiscale and index-permutation runs show schema/manifest checks instead.

`CODECS → Show HTJ2K backend availability` is a good first stop: it tells you
which codec backend your environment resolved to, and whether it passed the
multi-component probe described [below](#htj2k-backends-and-why-there-is-a-probe).

### Reading stores back in Python

```python
import spatialdata as sd

sdata = sd.read_zarr("store-htj2k.zarr")   # HTJ2K/JP2K images decode transparently
```

No import of this package and no `register_codecs()` call is needed — the codecs
are advertised through `zarr.codecs` entry points, so zarr finds them as soon as
the distribution is installed. (`spatialdata_js_util.codecs.register_codecs()`
exists only for running from a source tree with no installed metadata.)

### HTJ2K backends, and why there is a probe

HTJ2K encode/decode can go through either of two OpenJPH builds:

- **`imagecodecs`** — native OpenJPH, pure `pip install`, no Node.js. Preferred.
- **`openjph-wasm`** — the vendored WASM the JS reader uses, driven by Node.js
  worker processes. Requires `node` on `PATH`.

A WASM build previously used in this project silently decoded *every* component
of a multi-component codestream as component 0 — losing volumetric planes with
no error (see [`docs/multi-component-codec-findings.md`](docs/multi-component-codec-findings.md)).
Because that failure is silent, a backend is not trusted on the strength of its
library name. `imagecodecs` is admitted only if it decodes a committed
multi-component codestream — produced by the WASM encoder — to exactly the
expected samples. Any mismatch, and it is rejected in favour of the WASM path.

Check what your environment resolved to:

```bash
spatialdata-js-util codecs info
```

Pin a backend explicitly with `SPATIALDATA_JS_UTIL_HTJ2K_BACKEND=imagecodecs`
or `=openjph-wasm` (the probe still gates `imagecodecs`).

## Images

Recompress a store, preserving everything that is not an image. Path sources are
copied first, so tables, shapes, points and unconfigured rasters survive without
the whole object being loaded.

```bash
spatialdata-js-util images recompress input.zarr output-htj2k.zarr \
  --codec experimental.openjph_htj2k --preset balanced --chunks auto --overwrite
```

With `--sibling`, each original image is kept and a compressed copy added
alongside it (`morphology_focus:htj2k_balanced`), so tools without the codec keep
working.

### Presets and `--quality`

`--quality` is the OpenJPH quantization step: **lower means higher fidelity and a
larger codestream** — not a JPEG-style 0–100 scale. The step is relative to the
dtype's full dynamic range, so one input LSB is `1/256` for 8-bit data and
`1/65536` for 16-bit.

HTJ2K presets are therefore expressed as a **multiple of the input's LSB**, and
resolve to a concrete step from the raster's dtype:

| Preset | JPEG 2000 | HTJ2K | `uint8` | `uint16` |
|--------|-----------|-------|---------|----------|
| `lossless` | reversible, round-trip verified | `reversible=True` | — | — |
| `balanced` | `level=100` | 2 LSB | `0.0078125` | `0.000030518` |
| `small` | `level=75` | 5 LSB | `0.01953125` | `0.000076294` |

A step **below one input LSB is strictly worse than lossless**: it cannot buy
fidelity the data does not have, so the irreversible transform returns a
bit-identical image while encoding *larger* than `--preset lossless`. Passing
such a `--quality` warns and names the floor. The resolved step is recorded in
the manifest under `encode_options`.

An explicit `--quality` is an absolute step, not an LSB multiple, so pick it for
the raster's dtype — `0.0005` below is ~33 LSB of `uint16` morphology, but would
be 0.13 LSB (worse than lossless) on 8-bit input:

```bash
spatialdata-js-util images recompress input.zarr out.zarr \
  --codec experimental.openjph_htj2k --quality 0.0005 --sibling --overwrite
```

### Multiscale pyramids

Acquisition output is often written at a single resolution. A browser then has no
choice but to fetch full-resolution chunks however far out the user is zoomed —
which defeats the image codecs rather than complementing them. `--pyramid` gives
any single-level image a pyramid before recompressing it, so one command turns
single-resolution input into a browser-ready store:

```bash
spatialdata-js-util images recompress input.zarr output.zarr \
  --codec experimental.openjph_htj2k --preset balanced --pyramid --overwrite
```

To add pyramids without touching codecs:

```bash
spatialdata-js-util images add-pyramid input.zarr output.zarr --overwrite
```

By default levels are chosen automatically: keep halving until the largest
spatial axis is `--pyramid-min-size` (1024) or smaller, so the coarsest level is
roughly one chunk. Override with `--pyramid-levels N` (a *total* count including
full resolution) and `--pyramid-downscale K`.

Images that already have more than one level are **skipped**, and the manifest
says so; `--pyramid-force` rebuilds them anyway. Pass `--labels` to
`add-pyramid` to include label elements — they are downsampled by the label
model, so ids are never averaged into values that identify no object.

Full resolution is copied through unchanged; only the added levels are new.
Levels are generated through SpatialData's own parsers, which get the per-level
scale *and* half-pixel translation right — hand-written `multiscales` metadata
tends to misalign every level against the full-resolution image.

Pyramids are always written to a **different** store than the source. SpatialData
refuses to delete the files backing a live element, so there is no in-place mode.

### Other image notes

Labels are written with Blosc/zstd. Browser image codecs support `uint8`,
`int8`, `uint16` and `int16` only. A sidecar manifest records the expanded
config, per-raster stats, the codec backend used, and decoded checksums:

```bash
spatialdata-js-util images inspect output-htj2k.manifest.json
```

For repeatable multi-image runs, use a JSON config:

```json
{
  "default_image": { "codec": "imagecodecs_jpeg2k", "preset": "lossless", "chunks": "auto" },
  "images": {
    "morphology_focus": { "preset": "balanced" },
    "he_image": { "codec": "experimental.openjph_htj2k", "quality": 0.001 }
  },
  "default_labels": { "codec": "blosc", "clevel": 5 }
}
```

## Points

Morton-sorted Points Parquet, Vitessce-compatible:

- `morton_code_2d` added using 16 bits per axis
- the first 2–4 rows are sentinels with `morton_code_2d == 0` encoding the full
  point bounding box, so readers can get bounds without a scan
- `{feature_key}_codes` added when the element declares a `feature_key`
- string/categorical columns moved to the right of the table
- row-group size controlled, so a viewport maps to a few row-group reads

```bash
# List Points elements
spatialdata-js-util points list ~/data/xenium.zarr

# Morton-sort transcripts in place on points/<key>/points.parquet
spatialdata-js-util points morton-from-zarr ~/data/xenium.zarr --points-key transcripts

# Morton-sort a CSV or Parquet file
spatialdata-js-util points morton input.csv output.parquet --feature-key feature_name
```

Morton v1 belongs on the **canonical** element path `points/<key>/points.parquet`.
Use `--experimental` only for layouts standard readers cannot consume (see
[ADR 0002](../../docs/adr/0002-spatially-aware-vector-loading.md)).

`points index-permutations` builds a derivative store with several sort layouts
side by side for benchmarking. All of these are also available from the
[TUI](#interactive-tui), which verifies the output after each write.

### Multiscale Zarr points (experimental)

`points zarr` writes the layout of merfisheyes'
[points-zarr spec draft 0.1](https://github.com/kresnajenie/merfisheyes/blob/main/docs/points-zarr-spec.md):
levels of uniform random samples (each a quarter of the one below), each stored in
tile order and in feature order, with a row index. A viewer can then draw any zoom
from a bounded read, and read only the rows of the genes selected.

```bash
# A new store: a standard points element, with the copy beside it
spatialdata-js-util points zarr transcripts.parquet transcripts.zarr \
  --feature-key feature_name --coordinates x_location,y_location

# Add the copy to an existing store's points element, from its own points.parquet
spatialdata-js-util points add-zarr ~/data/xenium.zarr --points-key transcripts
```

The copy is the element's `experimental.points-zarr` subgroup; the element itself stays
an ordinary SpatialData points element, so spatialdata and other standard readers read
the store as before ([ADR 0006](../../docs/adr/0006-experimental-formats-beside-the-standard.md)).
Writing works out of core: Parquet input is read in batches, spilled to temporary files
beside the output (about 18 bytes a point, plus one level's sorted copy) and sorted one
band of tile rows at a time.

On exFAT (and other filesystems without extended attributes) macOS writes a `._name`
file beside each file that carries one, which doubles the file count and breaks
readers that list a directory (pyarrow reads `._part.0.parquet` as a part). Delete
them after writing: `find STORE -name '._*' -delete`.

## Tables

```bash
# Rewrite every table's X (and named layers) as CSC, in place
spatialdata-js-util tables to-csc ~/data/xenium.zarr

# Or write a converted copy, leaving the source alone
spatialdata-js-util tables to-csc input.zarr output-csc.zarr --overwrite
```

The conversion changes only the sparse layout, not the values, and the result
stays readable by any AnnData client. Dense matrices are left alone unless you
pass `--densify`, since sparsifying a dense matrix can make it bigger.

Only the matrices are rewritten — `obs`, `var`, `uns` and the rest are left
byte-for-byte alone, and what is written is pinned to the store's existing
encodings (no sharding, no `nullable-string-array` index). Re-serialising a whole
table would otherwise silently re-encode it with whatever conventions the
installed AnnData currently prefers, which can leave readers unable to find the
variable names even though every value is still present.

```python
from spatialdata_js_util import convert_store_tables_to_csc, to_csc

convert_store_tables_to_csc("store.zarr")     # whole store
csc = to_csc(adata.X)                         # or just a matrix
```

## Python API

```python
from spatialdata_js_util import (
    recompress_spatialdata,
    add_pyramids,
    has_pyramid,
    write_morton_points_parquet,
    convert_store_tables_to_csc,
    backend_report,
)

result = recompress_spatialdata(
    "input.zarr", "output-htj2k.zarr",
    codec="experimental.openjph_htj2k",
    preset="balanced",
    pyramid=True,          # add levels to single-resolution images first
    overwrite=True,
)
print(result.manifest_path)

# Or pyramids on their own
has_pyramid("input.zarr", "images", "morphology")   # -> False
add_pyramids("input.zarr", "output.zarr", levels=4)
```

## Monorepo development

From the repository root:

```bash
pnpm test:python
```

The vendored `openjph-wasm` assets under
`src/spatialdata_js_util/codecs/vendor/openjph/` are gitignored and produced by
`node scripts/vendor-openjph-for-python.mjs`, which `pnpm test:python` runs
first. They are only needed for the WASM backend; the `imagecodecs` backend works
without them.

Fixture-generation scripts under `scripts/` are repo-local and not shipped in the
wheel. To regenerate the backend probe fixture (requires Node.js):

```bash
uv run --directory python/spatialdata-js-util python scripts/build_htj2k_probe.py
```

TypeScript callers can use `encodeHtj2kPlane()` from `zarrextra` directly; see
[`docs/htj2k-wasm-encode-design.md`](docs/htj2k-wasm-encode-design.md).
