Metadata-Version: 2.4
Name: osl-pathfinder
Version: 1.0.0
Summary: A tiny two-way id<->fields<->path mapping for neuroimaging datasets (OSL ecosystem).
Author: Jize Zhang
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/OHBA-analysis/pathfinder
Project-URL: Repository, https://github.com/OHBA-analysis/pathfinder
Keywords: neuroimaging,filename,path,dataset,osl,bids
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: POSIX :: Linux
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: parse>=1.19
Requires-Dist: numpy>=1.20
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

# osl-pathfinder

A tiny two-way mapping between a short **id** and the **files** it owns.

```bash
pip install osl-pathfinder==1.0.0
```

You write the filename of each *kind* of file once, as a template with
`{field}` placeholders. `Pathfinder` then gives you clean conversions in **both
directions** — and groups every kind under one short id:

```
fields  <->  id (a short label)  <->  path (per file kind)
```

So you stop hand-writing the code that slices an id out of a filename, or
rebuilds a filename from an id, in every script. One source of truth.

```python
from osl_pathfinder import Pathfinder

pf = Pathfinder(templates={
    "eeg":  "/data/sub-{subject}/ses-{session}/sub-{subject}_ses-{session}_run-{run}_block-{block}_eeg.fif",
    "fmri": "/data/sub-{subject}/ses-{session}/sub-{subject}_ses-{session}_run-{run}_block-{block}_bold.nii.gz",
    "t1w":  "/data/sub-{subject}/anat/sub-{subject}_T1w.nii.gz",
}, anchor="eeg")

pf.field2id(subject="007", session="1", run="2", block="3")  # -> "007-1-2-3"
pf.id2path("007-1-2-3", "eeg")                              # -> Path(".../_eeg.fif")  (must exist)
pf.id2path("007-1-2-3", "t1w")                              # -> shared T1w (subject only)
pf.path2id(some_path, "eeg")                               # -> "007-1-2-3"
pf.id2path("007-1-2-3")                                    # all kinds -> {"eeg": Path, "fmri": Path, "t1w": Path}
```

Or, if you are as lazy as I am, use a compact ID and `{foo}` for filename text
you do not care about:

```python
from osl_pathfinder import Pathfinder

pf = Pathfinder(
    id="{subject:d}{session:1d}{run:1d}{block:1d}",  # -> 7111, 10121
    templates={
        "eeg": (
            "/data/sub-{subject}/ses-{session}/"
            "{foo}_run-{run}_block-{block}_eeg.fif"
        ),
        # The EEG and fMRI filename prefixes are intentionally untracked.
        "fmri": (
            "/data/sub-{subject}/ses-{session}/"
            "{foo}un-{run}_block-{block}_bold.nii.gz"
        ),
        "t1w": "/data/sub-{subject}/anat/sub-{subject}_T1w.nii.gz",
    },
    anchor="eeg",
)

pf.field2id(subject="7", session="1", run="2", block="3")  # -> "7123"
pf.id2path("7123", "eeg")     # existing EEG Path
pf.id2path("7123", "t1w")     # shared subject-level T1w
pf.path2id(some_path, "eeg")  # -> "7123"
pf.id2path("7123")             # -> {"eeg": Path, "fmri": Path, "t1w": Path}
```

Run `python example.py` for a complete, self-contained tour.

## Why an ID?

The id is a compact, readable label for one entity (subject/session/run/block).
Use it as a plot title, a dict key, a CSV column — without dragging a full path
around, and without re-deriving it from a filename each time.

### Custom and compact IDs

Pass `id="..."` to control exactly how IDs look. The compact quick start above
uses `id="{subject:d}{session:1d}{run:1d}{block:1d}"`: subject 7/session 1/run
1/block 1 becomes `"7111"`, while subject 10/session 1/run 2/block 1 becomes
`"10121"`. The trailing fixed-width `:1d` fields make the separator-free ID
unambiguous, so it still round-trips through `id2field()`.

### Numeric fields and canonical values

Numeric path format specifications also define how a real path is parsed. For
example, `{subject:03d}` parses the text `"007"` as the integer `7`, after
which Pathfinder exposes the value as the string `"7"`. The complete
conversion is:

```text
path component "sub-007"
    -> parse {subject:03d}
integer 7
    -> Pathfinder string field
{"subject": "7"}
    -> render id="{subject:d}{session:1d}{run:1d}{block:1d}"
ID "7111"
```

Thus a path ending in
`sub-007_ses-01_run-01_block-01_eeg.fif` maps to `"7111"`. This behavior is
specific to numeric specifications such as `:03d`; a string-alignment
specification such as `:0>3` preserves the parsed text `"007"` instead.
All values returned by `id2field()` and `path2field()` are strings, and IDs such
as `"1111"` are strings rather than integers.

Use the same kind of format specification for a field everywhere it appears,
both within one template and across file kinds. For example, do not casually
mix numeric `{subject:03d}` with string-alignment `{subject:0>3}`: the former
parses `"007"` to the canonical field value `"7"`, while the latter preserves
`"007"`. Pathfinder does not reject this difference across kinds. Mix field
specifications only when you deliberately understand and want the resulting
canonical values.

### ID ambiguity protection

The *bare* default joins fields with `-` (`"007-1-2-3"`) only because, without a
separator or fixed widths, an id can't be split back unambiguously — so supply
an explicit `id=` (as above) whenever you want the compact form.

**Ambiguity check.** The constructor rejects an id template where two
variable-width fields touch with no separator, because `id2field` couldn't
recover them — e.g. `"{subject:d}{session:d}{run:1d}{block:1d}"` raises
`ValueError` (`subject`/`session` are both variable-width and adjacent). Fix it
by adding a separator or giving all but one a fixed width (`{session:1d}`).

## Mixed granularity (one T1w for many blocks)

A template only uses the fields it names. `t1w` mentions just `{subject}`, so
every block of a subject resolves to the **same** T1w file. No special-casing.

## Independent path wildcards: `{foo}`

Ordinary placeholders are consistent fields: if `{subject}` appears twice in a
pattern, both occurrences must match the same string. Use the reserved
placeholder `{foo}` for unimportant text that is only a garbage bin, not an
entity field. Each `{foo}` occurrence is independent, so repeated occurrences
may match different strings:

```python
pf = Pathfinder(
    templates={
        "raw": "/data/{foo}b-{subject:02d}/{foo}s-{session:02d}/"
               "eeg/{foo}-checker_eeg.set",
    },
    id="{subject:d}{session:1d}",
)
pf.id2path("12", "raw")  # each {foo} becomes its own `*` glob
```

`{foo}` is omitted from `path2field` and never contributes to the default ID.
`path2id` and `scan` therefore work even when its occurrences match different
strings. A `{foo}` kind can only resolve an existing file:
`require_existence=False` raises because inventing a wildcard would make an
output path non-reproducible. Define build-output kinds with explicit fields.

## Keep dataset fields consistently named

Literal path labels may differ while the underlying Pathfinder field stays the
same. For example, if fMRI calls `run` a `scan` and `block` a `part`, keep using
the same `{run}` and `{block}` fields:

```python
templates = {
    "fmri": (
        "/data/sub-{subject:03d}_ses-{session:02d}_"
        "scan-{run:02d}_part-{block:02d}/bold.nii.gz"
    ),
    "eeg": (
        "/data/sub-{subject:03d}_ses-{session:02d}_"
        "run-{run:02d}_block-{block:02d}/eeg.fif"
    ),
}
```

If modalities genuinely use different entity values—for example, EEG
`run2block1` corresponds to fMRI `run1block3`—Pathfinder does not hide the mismatch.
Correct the dataset names, or create a soft-link layer with consistent names
when the source cannot be changed.

## API at a glance

### Constructor

| API | purpose | filesystem behavior |
|---|---|---|
| `Pathfinder(templates, id=None, anchor=None, sep="-")` | define every file-kind template and the ID format | discovers and freezes the ID cohort from existing anchor files |

- `templates` is the required `{kind: template}` mapping. Path templates may be strings or `PathLike` values and are stored as strings; every concrete path returned by Pathfinder is a `pathlib.Path`.
- `id` selects the ID fields and formatting; if omitted, the anchor fields are joined with `sep`.
- `anchor` selects the file kind used to discover IDs and defaults to the first kind in `templates` insertion order. Use the finest-grained file kind as the anchor.


### Properties

These are attributes, not methods: use `pf.ids`, not `pf.ids()`.

| Property | value | lifecycle |
|---|---|---|
| `pf.templates` | read-only `{kind: template}` mapping | frozen at construction |
| `pf.anchor` | kind used for ID discovery | frozen at construction |
| `pf.id_template` | format template used to build and parse IDs | frozen at construction |
| `pf.id_fields` | ordered tuple of fields encoded by the ID | frozen at construction |
| `pf.ids` | list copy of the anchor-derived ID cohort | cohort frozen at construction |
| `pf.full_table` | latest table returned by `scan()` | `None` before scanning; replaced by each scan |

### Conversion and lookup methods `pf.A2B(A, ...)`

The defaults are `fuzzy=False` and `require_existence=True`.

Every `field2B()` method rejects `foo` in its supplied fields: `{foo}` is only
an independent wildcard written inside a path template; it is never a field
value that a caller can bind.

| Method | input → output | filesystem behavior |
|---|---|---|
| `pf.field2id(fields, *, fuzzy=False)` **(default)** | complete fields → one ID | no disk access |
| `pf.field2id(constraints, *, fuzzy=True)` | partial fields → matching IDs | filters fixed `pf.ids`; no rescan |
| `pf.id2field(id)` | one ID → field dictionary | no disk access |
| `pf.id2path(id, kind, *, require_existence=True)` **(default)** | one ID → one existing kind's `Path` | globs missing non-ID fields; exactly one match required |
| `pf.id2path(id, kind, *, require_existence=False)` | one ID → one output `Path` | no disk access; ID plus extras must supply every template field |
| `pf.id2path(id, *, require_existence=...)` | one ID → `{kind: Path}` for every kind | applies the selected existence mode to every kind |
| `pf.field2path(partial_fields, kind, *, require_existence=True)` **(default)** | partial or complete fields → one existing `Path` | missing fields become `*`; exactly one match required |
| `pf.field2path(complete_fields, kind, *, require_existence=False)` | fields complete for this kind → one output `Path` | no disk access; every ordinary placeholder in this kind required |
| `pf.path2field(path, kind)` | supplied path → field dictionary | parses the path; does not test existence |
| `pf.path2id(path, kind)` | supplied path → ID | parses the path; does not test existence |

Exact `field2id(..., fuzzy=False)` requires one scalar value per field and
rejects lists or other non-string iterables. Iterable values express OR
matching only in fuzzy mode: `session=["1", "2"]` means session 1 or session 2
when `fuzzy=True`.

Unlike `field2id(..., fuzzy=True)`, a partial `field2path()` lookup never
returns a list. It returns the single matching path or raises if the pattern is
missing or ambiguous.

“Complete fields” means complete for the selected kind, not complete for
the whole Pathfinder: a subject-level `t1w` template containing only
`{subject}` does not require session, run, or block fields.

### Path lookup modes

`id2path` and `field2path` take a keyword-only `require_existence` boolean:

- **`require_existence=True` (default) finds an existing file.** Missing fields
  become glob wildcards. The pattern must match exactly one path; zero or
  multiple matches raise. The all-kinds `id2path(id)` form checks every kind
  and reports all missing kinds together.

- **`require_existence=False` builds an output path.** It does not inspect the
  filesystem. Every ordinary placeholder used by the selected kind must be
  supplied, and the returned `Path` need not exist. A template containing
  `{foo}` is rejected because an independent wildcard has no build-time value.

The all-kinds build form succeeds only when every kind can be rendered exactly
and no selected template contains `{foo}`.

`id2path` accepts extra keyword fields for ordinary placeholders outside the
ID, but they cannot override a field encoded by the ID. `field2path` instead
takes every constraint in its required first mapping argument. Neither method
accepts `foo`; unknown fields and misspelled flags such as `req_exist=False`
raise `TypeError`.

### Validation method

| Method | input → output | filesystem behavior |
|---|---|---|
| `pf.scan(csv_path=None)` | fixed IDs × kinds → validation table | checks current disk state; prints the table and optionally writes CSV |

`scan()` validates every configured kind for the fixed IDs discovered during
construction; it does not discover new IDs or change the anchor. Missing files
are normal validation rows, while patterns matching multiple files are
collected and raised together. Existing rows contain a concrete `Path`;
missing or ambiguous rows contain `path=None` and retain the expected pattern
with unresolved portions shown as `*`.

The latest table is available as `pf.full_table`. It is only a scan cache:
path-lookup methods always inspect the filesystem directly.

### Module-level split helpers

These functions are not methods on `pf`, but functions using it.

| Function | input → output | filesystem behavior |
|---|---|---|
| `split_pf(pf, proportions, unit=None, seed=42, json_path=None)` | fixed IDs → reproducible subset lists | optionally saves an atomic JSON replay record |
| `split_pf_from_file(pf, json_path)` | saved record + current fixed cohort → the same subset lists | reads and validates the JSON record |

## Pipelines: define once, read *and* write

Construction discovers the fixed ID cohort from the anchor, so create the
Pathfinder after the anchor files are available. Define every kind once in one
shared module and import that `pf` in each pipeline step. Read inputs with the
default existence check; build outputs with `require_existence=False`:

```python
# pathfinder.py
pf = Pathfinder(templates={"raw": ..., "src": ..., "power": ...}, id="...")

# 1.py: src does not exist yet
for file_id in pf.ids:
    raw_path = pf.id2path(file_id, "raw")
    src_path = pf.id2path(file_id, "src", require_existence=False)
    save(transform(raw_path), src_path)

# 2.py: src now exists
for file_id in pf.ids:
    raw_path = pf.id2path(file_id, "raw")
    src_path = pf.id2path(file_id, "src")
```

## Reproducible development/evaluation subsets

The helper returns one sorted ID list per requested proportion and does not
modify `pf`. Pass `json_path=` when you also want a replayable JSON split
record:

```python
from osl_pathfinder import split_pf, split_pf_from_file

split_path = "results/natview_dev_eval_split.json"
dev_ids, eval_ids = split_pf(
    pf,
    proportions=[0.8, 0.2],
    unit=["subject"],       # keep every session/run/block of a subject together
    seed=42,                # fixed by default; record it with your results
    json_path=split_path,
)

# Later, or on another machine with the same Pathfinder cohort:
dev_ids, eval_ids = split_pf_from_file(pf, split_path)
```

- **Per-file splitting:** `unit=None` or `unit=[]` makes every discovered file
  ID independently splittable.

- **Grouped splitting:** `unit=["subject", "block"]` keeps every session and
  run belonging to the same subject/block together. Unit names must exactly
  match fields in `pf.id_fields`.

- **Percentages apply to units:** an indivisible group is assigned as a whole,
  so grouped subsets may contain slightly different file counts when groups
  have different sizes.

- **Random sampling is isolated:** IDs and grouping keys are sorted before a
  local NumPy `RandomState` is used. Splitting does not change NumPy's global
  random state.

- **Exact replay needs the saved result:** a seed and unchanged sorted IDs are
  normally repeatable, but NumPy documents machine/build caveats. Pin NumPy and
  save the JSON split record or returned ID lists for cross-machine replay.
  NumPy is installed as a normal Pathfinder dependency.

The saved JSON is a small replay record, not a second path manifest. Version
1.0 freezes the following split-record schema:

| key | required value |
|---|---|
| `format` | string, exactly `"osl-pathfinder.split"` |
| `version` | integer, exactly `1` |
| `id_fields` | non-empty array of unique strings |
| `unit` | array of unique strings; empty means per-ID splitting |
| `proportions` | non-empty array of finite, non-negative numbers summing to 1 |
| `seed` | integer from 0 to `2**32 - 1`, or `null` |
| `input_ids` | sorted array of unique string IDs in the input cohort |
| `shuffled_keys` | array of unique, non-empty string arrays |
| `counts` | non-empty array of non-negative integers, one per subset |

Replay uses `input_ids`, `shuffled_keys`, and `counts` directly, then checks the
current Pathfinder's cohort and grouping fields. It does not depend on rerunning
the random generator. A changed cohort or incompatible ID definition raises an
error instead of silently changing the split. Required keys and meanings are
strict. Unknown additional keys are allowed and ignored during replay, so a
project or later compatible writer can attach optional provenance metadata.

Split records are written through a temporary file in the target directory and
atomically replaced into place. The parent directory must already exist.

## For developers

```bash
pip install -e '.[test]'  # editable checkout, for local development + pytest
pytest -q
```
