Metadata-Version: 2.4
Name: sfm-analysis
Version: 0.1.3
Summary: Analysis and printable HTML behavior reports for SFM feeder session logs
Author: Neurotech Hub
License: MIT
Project-URL: Homepage, https://github.com/Neurotech-Hub/SFM
Project-URL: Source, https://github.com/Neurotech-Hub/SFM/tree/main/packages/sfm-analysis
Project-URL: Issues, https://github.com/Neurotech-Hub/SFM/issues
Keywords: neuroscience,behavior,operant,feeder,CAN,reports
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: pandas
Requires-Dist: pandas>=1.3; extra == "pandas"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Provides-Extra: all
Requires-Dist: sfm-analysis[pandas]; extra == "all"
Dynamic: license-file

# sfm-analysis

Cross-platform analysis and printable HTML behavior reports for
[SFM](https://github.com/Neurotech-Hub/SFM) feeder session logs.
Runs on Windows, macOS, and Linux — no Raspberry Pi, no hardware
libraries. Pure standard library: no jinja2, no pandas, no matplotlib. A
report is one self-contained HTML file with inline SVG charts, printable
via Ctrl+P.

This package is carved out of the SFM repository's `packages/dev_gui/`
(the Raspberry Pi base station that records the logs) precisely so that
analysis doesn't have to happen on the Pi. The base station's own
`run_report.py` is a thin wrapper around the CLI documented below.

## Install

```bash
pip install sfm-analysis
```

To track `main` instead of a released version:

```bash
pip install "git+https://github.com/Neurotech-Hub/SFM.git#subdirectory=packages/sfm-analysis"
```

## Quick start

```bash
# Try it right now, with no rig, no log directory, and no real data:
sfm-report --demo --open

# List every session found in the default log directory
# ($SFM_LOG_DIR, else an external drive's sfm_logs/, else ~/sfm_logs):
sfm-report --list

# One session, opened in your browser when done:
sfm-report EXP-Test-02 --open

# Same report, but the actogram ticks pellet takes instead of presence:
sfm-report EXP-Test-02 --design actogram_takes --open

# Every session for a cohort, combined into one comparative report:
sfm-report "cohortA_*" --combine -o /tmp/cohortA.html
```

`--demo` renders a real, bundled two_armed_bandit session
(`sfm_analysis.report.demo`) — the fastest way to see what a report looks
like on a fresh install.

## The `sfm-report` CLI

```
sfm-report --help

  target...         Session name, shell glob, or path to a .csv (repeatable)
  --log-dir         Default: $SFM_LOG_DIR, else an external drive, else ~/sfm_logs
  --list            List discovered sessions and exit
  --list-designs    List report designs and exit
  --check-names     Show how each session name parses (subject/cohort/day) and exit
  --combine         One comparative report over all targets
  --all             Every session in --log-dir
  --demo            Render the bundled demo session (no rig or log dir needed)
  --since, --until  Filter by date (YYYY-MM-DD)
  --run             Only this run_id (a file can hold several)
  --design          Force a report design: bundled name, or path to a .json file
  --align           Combined-report time alignment: relative | wall | trial | event:<name>
  --out, -o         Output file (single report) or directory (multiple)
  --no-explorer     Skip the embedded interactive timeline (guaranteed script-free, leaner output)
  --open            Open the result in your default browser when done
```

A **session** is one CSV written by the base station's unified 15-column
log format (`timestamp_iso, timestamp_ms, ..., fields_json, details`).
Files in an older schema are skipped with a warning rather than crashing
the run — see `report.loader.sniff_schema`.

## The interactive timeline

```bash
sfm-report EXP-Test-02 --open
```

is enough — every design embeds a real pan/zoom/brush-zoom timeline
directly in the one report file by default, right where the static
raster panels used to sit alone. It's the answer to "I don't want 12
stacked panels, I want to point at the timeline and zoom in there", and
it's the whole report you get from one self-contained `.html` you can
email or drop somewhere — never a pair of files that has to travel
together:

- **Drag on the main view** to brush-select a range and zoom into it.
- **Drag the overview strip** (the thin bar above the main view) to pan.
- **Scroll** to zoom in/out centered on the cursor.
- **Reset zoom** returns to the full run.
- Hovering shows the cursor's wall-clock time, time-of-day, and time
  into the run, all three always visible — the wall-clock and time-of-
  day readouts use the *recording rig's* local clock (the same
  `timestamp_iso`-based reasoning as `report.timezones`), never the
  browser's, regardless of what machine you're viewing it on.

It's drawn on `<canvas>`, not SVG — the same design decision as the
static report's panels would make a bad interactive widget at real data
volumes (a raster with tens of thousands of DOM nodes is what makes a
browser tab hang). The data itself — lanes, spans, marks — comes from
exactly the same `report.timeline_data.panel_marks_spans` the static
raster panels use, just for the whole run's `[0, duration]` window
instead of one paginated slice, so the two can never disagree about
what a "presence bout" or a "dispense cycle" is. Those static panels
don't disappear — they go **print-only**: hidden on screen (the
interactive view replaces them there), but still present for `Ctrl+P`,
since the printed PDF can't run the JS that draws the canvas.

`build_session_report(..., include_explorer=False)` — or `--no-explorer`
on the CLI — drops the embedded timeline (and its one inline `<script>`)
entirely, for a caller that wants the guaranteed-script-free,
leaner document. It's the only place in this package that ships
JavaScript — everything else, including the rest of the printed report,
is deliberately script-free. Self-containment is still the invariant
either way: no `http(s)://` reference anywhere, nothing that needs the
network to render or print.

## How a report is put together

A report **design** (`report/designs/<name>.json`) declares which
**sections** to render and in what order; sections are plain Python
functions registered under `report/sections/`. This mirrors the SFM
experiment engine's own JSON-schema/Python split (`base_station/experiment/`
in the parent repo — `matches: []` in a design plays the same role as
`build(**kwargs)` in an experiment template).

```
src/sfm_analysis/report/
├── designs/<name>.json     # "matches": ["<experiment>"] + ordered refs
├── analyses/<name>.py      # numbers (no HTML)
└── sections/<name>.py      # HTML; SECTIONS = {"func": ...}
```

Copy-from starter: [`examples/report_design/`](examples/report_design/).
Full walkthrough: [docs/ANALYSIS_GUIDE.md](docs/ANALYSIS_GUIDE.md#9-turning-an-analysis-into-a-report-section).

The design is picked automatically from the session's `experiment` field
(the fields_json on its `session_start` row), falling back to
`default.json` for anything without a matching design or with no
`session_start` at all — see `resolve_design`. Force one explicitly with
`--design <name>` or `build_session_report(..., design=...)`.

Metrics (retrieval latency, presence bouts, dispense cycles, bandit
choice/WSLS/reversal, …) live in `report/analyses/` and
`report/metrics.py`, separate from the HTML that renders them — the
metrics are plain dataclasses, computed once per run
(`compute_run_metrics`) and handed to every section read-only. This is
what makes the numbers themselves testable without parsing HTML (see
`tests/test_report_metrics.py`, which never touches `render_report_html`
at all).

Add a new comparative view or per-experiment section by writing a
function that returns a `SectionResult` (`report/schema.py`) and adding
its `module.function` ref to the relevant `report/designs/*.json` — no
changes to the CLI or the render pipeline needed. A section that raises
never kills the whole report; it renders a visible error block with the
traceback instead (`schema.run_section`), so one broken section can't take
the rest of the report down with it.

### Long and multi-day sessions

`timeline.session_raster`'s detail-panel width (`window_s`) scales with
run duration by default — `max(600, duration/12)` — so a 24h run gets
about 12 panels instead of 144 stuck at a fixed 600s. Pin a fixed width
regardless of duration with `"options": {"window_s": ...}`; raise only
the short-run floor (e.g. free_feeding's slower cadence) with
`"options": {"min_window_s": ...}`. When `timeline.explorer` is also
active (the default — see "The interactive timeline" above), these
panels go print-only: hidden on screen since the embedded explorer
covers the same ground with real zoom, still present in the printed PDF.

`timeline.actogram` — one row per calendar day, time-of-day on the
x-axis — renders automatically whenever a run spans 2 or more distinct
days; it's the figure that matters for a multi-day experiment, where
session_raster's panels (even adaptively sized) stop being the right
tool. Absent entirely for shorter runs.

It deliberately draws **no light/dark shading**. The rig doesn't record
the facility's light schedule, so any shading would be a fixed
clock-time assumption rendered as though it were measured data. Time of
day is on the axis; apply your own light cycle to it if you need one.

The section heading and figure caption both name the plotted event
(`Actogram — MousePresence Detected` by default) so a printed page is
unambiguous about what each tick is. Ticks default to presence onsets.
You can remap them **without editing the installed package** — see
[Customize the actogram](#customize-the-actogram) below.

## Customize the actogram

After `pip install sfm-analysis` on a laptop, two knobs change which CAN
events become actogram ticks. Names must match `event_name` on
`frame_type == "EVENT"` rows — the same strings as the GUI log
(`Pellet Taken`, `Dome Opened`, `Loaded`, `Fault: Jam`, …). Several
names overlay as **one** series (union of timestamps), not separate
colours. Nothing in `site-packages` needs to be edited.

### HTML report (no file copy)

`actogram_takes` ships in the wheel — presence stays the default; this
name is opt-in only (`sfm-report --list-designs`):

```bash
sfm-report MySession --design actogram_takes --open
```

### Python (any session CSV)

```python
from sfm_analysis.analysis import load_session
from sfm_analysis.report.metrics import activity_by_day

s = load_session("MySession")          # name, glob, or path to the CSV
days = activity_by_day(s.run(), event_names=("Pellet Taken",))
for day in days:
    print(day.date, len(day.times), day.times[:3])
```

Compare presence vs takes vs dome+takes against the bundled demo, or
pass your own CSV. This is the same recipe after a pip install (no git
checkout):

```bash
python -m sfm_analysis.examples.actogram_by_event
python -m sfm_analysis.examples.actogram_by_event /path/to/MySession.csv
```

### A different event, or several

Copy the shipped design next to your logs and edit `event_names`, then
pass the **path** (so you still do not patch the install):

```json
{ "ref": "timeline.actogram",
  "options": { "event_names": ["Dome Opened", "Pellet Taken"] } }
```

```bash
sfm-report MySession --design ./my_actogram.json --open
```

A starting file lives at
[`examples/report_design/designs/actogram_takes.json`](examples/report_design/designs/actogram_takes.json)
(identical to the bundled design). Or from Python:

```python
from pathlib import Path
from sfm_analysis.report import build_session_report

build_session_report(
    Path("MySession.csv"),
    design="actogram_takes",   # bundled name, or a path to your .json
)
```

## Python API

For the full reference — every log column, every event name, every
`RunMetrics` field and its `None` semantics, every tidy-table column, and a
worked "question to tested answer" walkthrough — see
[docs/ANALYSIS_GUIDE.md](docs/ANALYSIS_GUIDE.md). What follows here is the
quick-start version.

### Custom analysis: `sfm_analysis.analysis`

This is the entry point for asking your own question of a session — one
import, tidy `list[dict]` tables, and a small generic interval algebra
for "was X happening while Y" questions, rather than five imports across
four modules or a bespoke query language.

```python
from sfm_analysis.analysis import load_session

s = load_session("EXP-Test-02")      # name, glob, or path — same resolution as the CLI
cycles = s.cycles_table()            # list[dict], one row per dispense cycle
bouts = s.bouts_table()              # presence / dome / fault intervals, unified

import pandas as pd
df = pd.DataFrame(cycles)            # or: from sfm_analysis.analysis import to_dataframe
df.groupby("node")["retrieval_latency"].median()
```

`load_session` resolves a target exactly the way `sfm-report` does
(`$SFM_LOG_DIR`, then an external drive, then `~/sfm_logs`, or pass
`log_dir=...`), loads the CSV, splits it into runs, and precomputes
metrics for every run once. If the file holds more than one run — a
9-row aborted restart followed by the real session, say — `s.run()`
picks the run with the most rows by default; pass `run_id=` anywhere to
be explicit instead.

**Tidy tables** (`s.cycles_table()`, `s.bouts_table()`, `s.events_table()`,
`s.trials_table()`) are the fastest way to get real freedom: every row is
a plain dict with the same keys, ready for `pandas.DataFrame`, `csv.DictWriter`,
or a plain list comprehension. `to_dataframe(rows)` is the explicit,
optional pandas handoff (`pip install sfm-analysis[pandas]`) — the core
package stays pandas-free.

**Interval algebra** (`sfm_analysis.analysis.intervals`) answers the
class of question tidy tables alone don't: `overlap`, `subtract`,
`merge`, `around`, `count_in`, `rate_in`, all plain functions over
`(t0, t1)` second-pairs.

| Question | One-liner |
|---|---|
| Windows where dome-open overlapped presence | `overlap(dome_iv, presence_iv)` |
| Takes within 30s after a fault started | `count_in(take_times, around(fault_starts, (0, 30)))` |
| Pellet-take rate during a time window | `rate_in(take_times, [(t0, t1)])` (takes/second) |
| Time NOT covered by any dome-open bout | `subtract([(0, run.duration_s)], dome_iv)` |

```python
from sfm_analysis.analysis import intervals as iv

run = s.run()
m = s.metrics_for()
presence_iv = iv.as_intervals(m.presence[0], run_end=run.duration_s)
dome_iv = iv.as_intervals(m.dome[0], run_end=run.duration_s)
len(iv.overlap(dome_iv, presence_iv))     # dome-open bouts that overlapped presence
```

`as_intervals` is the one deliberately strict step: a bout still open
when the run ended (`censored=True`) has `t1=None`, and this refuses to
guess whether that means zero-length or unbounded — pass `run_end=` to
resolve it explicitly, or filter censored bouts out yourself first.

### Lower-level: `sfm_analysis.report`

`sfm_analysis.analysis` is built entirely on public functions in
`sfm_analysis.report` — nothing is hidden from you, and dropping down is
one import away if you need something the curated API doesn't expose yet
(a report design's own analysis module, e.g. `report.analyses.bandit`,
for instance). This is also how the report itself gets built
programmatically:

```python
from pathlib import Path
from sfm_analysis.report import build_session_report, build_combined_report, load_runs

build_session_report(Path("EXP-Test-02.csv"), out_path=Path("report.html"))
build_combined_report([Path("A.csv"), Path("B.csv")], design="two_armed_bandit")

runs = load_runs(Path("EXP-Test-02.csv"))   # what load_session uses internally
```

### Five things to know before trusting a number

Each of these is enforced or worked around somewhere in this codebase,
but none of it is obvious from the CSV alone — `sfm_analysis.analysis`
handles #1, #3, and #4 for you (via `Session`/`load_runs`,
`bouts_table`/`dome_bouts`, and `as_intervals`'s required `run_end`,
respectively), but they're worth understanding regardless of which layer
you work at:

1. **Scope everything to `(session, run_id)`.** A single CSV can hold
   several runs — reopening a session name appends to the same file and
   increments `run_id`. A cumulative count or an inter-trial interval that
   crosses a run boundary is meaningless. `split_runs` is the one place
   that groups rows this way; always work from its output, not the raw
   `LogRow` list.
2. **Never read `elapsed_s` directly.** It's *run-relative* and resets to
   0 every time a session is reopened. Use `RunData` rows' `.t`
   (run-relative seconds recomputed from `timestamp_ms` by `split_runs`)
   instead.
3. **`"Dome Opened"` is two different wire events.** The
   `CanEvent.DomeOpened` milestone (raw byte 0 = `0x03`, carries a pellet
   count) and a renamed `InputChanged` dome edge (raw byte 0 = `0x06`)
   both render as the same `event_name`. Pairing opens/closes by name
   alone double-counts every opening. Use `LogRow.can_event` to
   disambiguate (see `metrics.dome_bouts`, which already does this dedup —
   prefer it over rolling your own).
4. **A bout still open when the run ends is `censored`, not zero-length.**
   `Bout.dur` is `None` for a censored bout on purpose. Filter deliberately
   rather than letting a `None` silently become a `0` in an average.
   Bout, censored, cycle, ready, and the other analysis nouns are defined in
   [ANALYSIS_GUIDE — Vocabulary](docs/ANALYSIS_GUIDE.md#vocabulary).
5. **Two different "pellets presented" counts exist, on purpose.**
   `PelletAccounting.presented` counts rows actually seen in the log;
   `.presented_total` (what you almost always want) uses the base-station
   ledger's gap-corrected total when available, which also counts pellets
   whose own event frame never arrived. `.presented_total` /
   `.taken_total` are the ledger-corrected numbers; the bare `.presented`
   / `.taken` fields are what a naive `by_event("Loaded")` count gets you.

## Recipes

`examples/analysis/` has runnable, commented scripts for each cookbook
question above — copy one as a starting point rather than writing from
scratch:

| Script | Question |
|---|---|
| [`dome_openings_during_presence.py`](examples/analysis/dome_openings_during_presence.py) | Windows where the dome was open while the mouse was present |
| [`retrieval_latency_by_node.py`](examples/analysis/retrieval_latency_by_node.py) | Per-node summary stats, stdlib-only and pandas paths side by side |
| [`takes_after_fault.py`](examples/analysis/takes_after_fault.py) | Pellet takes within a window after each fault interval started |
| [`exp_events_by_name.py`](examples/analysis/exp_events_by_name.py) | Inventory a session's experiment-engine (`source=EXP`) events — the starting point for analysing a custom template's own log rows |
| [`actogram_by_event.py`](examples/analysis/actogram_by_event.py) | Remap actogram ticks (presence vs pellet takes vs dome+takes). After pip install: `python -m sfm_analysis.examples.actogram_by_event` |

Every one of them runs standalone against the bundled demo session, no
rig or `--log-dir` needed:

```bash
python examples/analysis/dome_openings_during_presence.py
```

They're also executed in CI (`tests/test_examples.py`), with their
output pinned against the demo session's known ground truth — a change
that breaks a documented recipe fails the build instead of being found
by the next person who copies one.

To ship a **custom report design** for your own experiment (JSON design +
Python analysis + HTML section), copy the three-file starter under
[`examples/report_design/`](examples/report_design/) into
`src/sfm_analysis/report/{designs,analyses,sections}/`. File layout and a
worked walkthrough: [docs/ANALYSIS_GUIDE.md](docs/ANALYSIS_GUIDE.md#9-turning-an-analysis-into-a-report-section).

## Testing your own analysis code

The test suite's own fixture builders are the fastest way to get a
synthetic session without a real rig — `tests/report_fixtures.py`:

```python
from report_fixtures import bandit_run, write_session
from sfm_analysis.report.loader import load_rows
from sfm_analysis.report.session import split_runs

def test_my_analysis(tmp_path):
    rows = bandit_run(n_trials=5, session="synthetic")
    csv_path = write_session(tmp_path, rows, session="synthetic")
    loaded, _, _ = load_rows(csv_path)
    runs = split_runs(loaded, [], csv_path)
    # ... assert against my_analysis_fn(runs[0])
```

There's also a real field-recorded fixture bundled with the package
itself, `sfm_analysis.report.demo` (the same session `--demo` renders) —
a genuine two_armed_bandit session with a fault interval, a dome
milestone/edge dedup case, and post-`session_end` rows, independently
verified against the raw CSV (see `test_report_smoke.py`'s docstring for
the exact numbers):

```python
from sfm_analysis.analysis import load_session
from sfm_analysis.report.demo import DEMO_SESSION_PATH

s = load_session(str(DEMO_SESSION_PATH))
```

Point the smoke test at a different real file with
`SFM_SMOKE_CSV=/path/to/session.csv`.

```bash
cd packages/sfm-analysis
pip install -e ".[dev]"
pytest
```

## Design notes worth knowing

- **Self-contained is a hard invariant.** No `http(s)://` reference, no
  `<script src=>`, nothing that needs the network to render or print
  (`report/render.py`'s module docstring; enforced by
  `tests/test_report_render.py`). The interactive timeline is the one
  exception to "no script": at most one inline `<script>` in the whole
  document. `--no-explorer` drops it for a guaranteed script-free file.
- **Every chart binds color + dash pattern + marker glyph + hatch texture
  to the same ordinal "key"** (`report/style.py`), so a series never
  depends on hue alone — this matters most exactly when a report is
  printed in grayscale.
- **A section function must not touch the filesystem.** `SectionContext`
  is built once and handed to every section read-only; all the data a
  section could need is already on `ctx.runs` / `ctx.metrics`.

## Firmware/SDK version skew

`sfm_analysis.protocol` is a Python mirror of the SFM firmware's
`firmware/src/services/ServiceTypes.h` and `firmware/src/services/CanService.h` (parent
repo). Because this package now versions and ships independently of the
firmware, it's possible for a base station running old firmware to be
paired with a newer `sfm-analysis`, or vice versa. There's no version
check yet — if you're seeing CAN events fail to decode, check that the
firmware and this package were built from compatible commits.
