Metadata-Version: 2.4
Name: headway
Version: 0.11.0
Summary: Compare computational graphs and benchmarking results for methods development
Author-email: Ben Skubi <bskubi@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/bskubi/headway
Project-URL: Guide, https://github.com/bskubi/headway/blob/main/src/headway/GUIDE.md
Project-URL: Issues, https://github.com/bskubi/headway/issues
Keywords: computation graph,notation,equations,scientific computing,benchmarking,reproducibility
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Documentation
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Requires-Dist: tree-sitter-language-pack>=1.22
Requires-Dist: beniget>=0.4
Requires-Dist: mypy>=1.10
Requires-Dist: docstring_parser>=0.16
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: jaxtyping>=0.2.30; extra == "test"
Requires-Dist: scipy>=1.11; extra == "test"
Requires-Dist: polars>=1.0; extra == "test"
Requires-Dist: statsmodels>=0.14; extra == "test"
Dynamic: license-file

# headway

See how a method works, and how much headway each change to it makes. A spec module declares a method's indices,
symbols (with their dimensions), steps (with their equations) and groups, each pointing into the code. headway
checks it against the code, so the picture cannot quietly go stale, and draws it as an interactive computation
graph, with a legend of indices, symbols and equations and the code behind every step. Versions of the method
(commits, experiments, competing methods) open side by side, and their benchmark results show as a heatmap, so
you can see what each change gained.

The code can be in any language: the spec is Python, and a reference like `"R/score.R:score_cells"`,
`"src/lib.rs:Model.score"` or `"Snakefile:align"` finds the definition with tree-sitter (`"main.nf#process TRIM"`
anchors on text, and `"config.yaml:2"` names lines, for files it cannot parse). `examples/pipeline` is a headway
of a pipeline in Nextflow, Snakemake, bash, R and Rust.

**[GUIDE.md](https://github.com/bskubi/headway/blob/main/src/headway/GUIDE.md)** (also `headway guide`) is the full description: how headway works and how
to write a headway for a codebase. It is written to be handed to a coding agent working in another repository.

```
pip install headway                             # into the environment that imports the method
headway init my_pkg.method:run -o my_pkg/graphspec.py   # a first draft, read from the code (Python)
headway redraft my_pkg.graphspec               # draft it again from the code, with init's arguments
headway tex my_pkg.model:residuals              # a function's numpy code, written as math
headway check my_pkg.graphspec                  # exit 1 if the spec and the code disagree
headway build my_pkg.graphspec -o headway.html  # one HTML file; open it in a browser
    [--collapse "round 0"] [--hide setup] [--json headway.json] [--no-verify] [--offline]
    [--name v3] [--compare v1.json --compare v2.html] [--results results.json]
headway serve [--open]                          # the library at http://127.0.0.1:8765/: open, compare, serve
headway serve my_pkg.graphspec [--open]         # the same, opened on the spec, rebuilt live
    [--compare old.json]
headway diff v1.json v2.json my_pkg.graphspec   # what changed from each version to the next
headway attach v3.json results.json             # put benchmark results into a built headway (or build --results)
headway results v1.json v2.json v3.json         # the versions' results on their common benchmark suite
headway guide                                   # the guide
```

From Python, e.g. at the end of a benchmark script in the method's own repository:

```python
import headway
headway.build("my_pkg.graphspec", "headways/v5.html", json="headways/v5.json", name="v5",
              results=[{"suite": "my_bench.suite", "values": values, "replicates": ["seed0", "seed1", "seed2"]}],
              compare=["headways/v4.json"])
```

`serve` watches the spec's package and every file its code references point to. On a save it rebuilds in a fresh
Python process (about a second, including the equation checks) and the open page takes the new graph without
losing its view: the selection, the zoom, the open groups. Steps that changed pulse, and a note says how many
changed. A spec that fails to build leaves the last good graph up, with the error over it. Each step, value and
group in the side panel links to its declaration in the spec ("✎ declared at …").

`headway serve` serves the **library**: one place to open headways from. It lists the headways opened recently
(in any directory, kept in `~/.local/state/headway/recent.json`) and what is in a folder: the specs (modules
that make a `Headway` or `Suite` at their top level; test files are skipped) and the built headways (pages and
`--json` files), with when each was built, its results and its keys, sortable by any column. It starts in the
directory the server was started in; browse to any other (the path bar, a folder's subfolders, **launch** (the folder it was
launched in), **home**, or type a path). **Open** a headway, tick several to open them side by side (the most recently built in
the page, the others in lanes beside it), or **Serve live** a spec, as `headway serve SPEC` does. Built files
open in the current viewer. Hidden directories, virtual environments, `node_modules`, `build` and `dist` are not
looked in.

With something open, **Open…** shows the same library over the page, to **Add beside** what is open (a lane of
its own) or **Open** something instead; **←** closes everything and goes back to the library. The address says
what is open (`/?open=v4.json&open=v3.json`), so a reload, a bookmark or the browser's Back button returns to it.
The server answers only requests to 127.0.0.1 or localhost.

## Editing from the page

The page stays a static view; edits happen in the spec, and the page takes you there.

- **✎ links** open the spec in your editor at the exact argument: the equation, the description, the inputs, the
  code references. They are in the side panel, and on the right of each legend row when you hover over it. For an
  object declared inside a helper function of the spec, the panel says so: editing the helper changes everything
  it declares.
- **`--editor`** (on `build` and `serve`, or `$HEADWAY_EDITOR`): `vscode` (the default), `cursor`, `pycharm`,
  `file`, or a URL template with `{file}`, `{line}` and `{col}`.
- **Ask an AI**, in the side panel, writes a prompt to paste into any assistant that can read the repository. It
  carries the object's declaration and location, its inputs and outputs, the code it points to and its equation
  check; with nothing selected, it is about the spec as a whole. It has two modes:
  - **Change the spec**: the request, with instructions to keep the references resolving, keep the `@G.verify`
    function in line with any changed equation, flag needed code changes, and run `headway check`. Preset requests
    fill in the box, worded for what is selected (a step, the steps around a symbol, a group, or the whole spec):
    **define local symbols** (put the symbols an equation uses but the graph does not define in `where=`), **add
    explanations** (a sentence or two in the spec on what each computation and output is, and what it is for),
    **check descriptions** (against the code), **reconcile with the code** (after the code changed), **standardize
    notation** (the same operation written the same way; fixed numbers as constants) and **disambiguate symbols**
    (one meaning per symbol across the spec).
  - **Explain**: a question, answered as a separate product; the assistant is told to change nothing, to take the
    code as the ground truth and to say which reasons are documented and which are its own reading. Presets: what is
    it for, read the equation, why this way, how it fits in, code vs. math, check my understanding. **Answer as**
    sets how elaborate the answer is: a quick answer, a full explanation, a tutoring session, a worked example,
    study notes to keep, or an interactive page (shown the way the assistant's environment shows pages, or as one
    HTML file).

  The mode and the answer format are remembered per browser; what is typed is kept per object and per mode.

The built page is a static file: open it from disk, send it, or put it on any web server. It loads KaTeX, elkjs,
highlight.js and html-to-image from CDNs; with `--offline` they are written into the file instead (about 3 MB), and the page needs no
network at all. `--offline` downloads them once, into `~/.cache/headway`. The ✎ links point to the spec on the
machine that built the page.

The viewer fills the window: the graph, the side panel on the right and
the legend below. Drag a divider to resize; drag it nearly shut, double-click it or use its arrow to collapse a
pane. The sizes are remembered per browser. Click a step or a value for its equation, inputs, outputs and
code (with a link that opens your editor at the line); double-click a group to collapse or expand it; show only what is
upstream or downstream of the selection; hide groups; show the settings as nodes. Steps done once per fold or per
cell are drawn as stacked cards; ✗ marks a step whose equation check failed. **Copy image** (above the graph,
and in the Results view) copies a PNG of the whole graph, or of the results table, to the clipboard; each equation
and each piece of code in the side panel has a small copy button of its own (on hover). Where the browser takes no
image on the clipboard, the PNG is downloaded instead.

**Find** (the box in the graph bar, or press `/`) boxes every step, value and group whose name or key holds what you
type. Case is ignored, and `_` and spaces count as the same.

- An **exact** match (a name is what you typed) is boxed solid; a **partial** one (a name holds it) is dashed.
  Tick **Exact** to keep only the exact ones.
- Tick **Text** to search the descriptions too: a step, value or group whose description holds what you typed is
  boxed dashed (with **Exact**, only where the description has it as whole words). A step with no description is
  searched by its code's docstring, as its panel shows it.
- The rest of the graph fades. A match inside a collapsed group boxes the group.
- Enter or ↓ jumps to the next match, and Shift+Enter or ↑ to the previous one, in the layout's reading order.
  Each jump centres the match and selects it, so its panel opens. The ‹ › buttons do the same, and ⤢ puts all
  the matches in view. Esc clears.

**The keyboard** walks the graph as it is drawn (when the focus is not in a text box):

- The arrows along the flow (→ and ← when the graph runs right, ↓ and ↑ when it runs down) move downstream or
  upstream, along an edge, to the node at its other end. With nothing selected, the first one starts at the
  beginning of the graph.
- The arrows across the flow aim at another of the node's edges, its inputs and then its outputs (drawn thick).
  The next move along the flow takes that edge. After a move, the aim is the edge you came by, so the way back is
  one key. Otherwise a move takes the edge most nearly straight ahead.
- A collapsed group is one node, and stays collapsed when you walk into it. A hidden group is not drawn, so it is
  not walked into either: hiding a group is how to take it out of the way, and the Groups menu shows it again.
- `c` collapses the group the node is in, or expands the collapsed group it is on, back on the node it was
  collapsed from.
- `g` goes from a node to its group, and from the group back to the node.
- Tab and Shift+Tab step through the legend's entries linked to the selection (its symbols and equations), and
  then back to the node. Enter selects the entry, and Esc goes back to the node.

The page's header says which version of headway built it. The button at the top right switches the page between Auto (light or dark as the system is), Light and Dark; the choice is kept in the browser.

## A spec

```python
from headway import Headway
G = Headway("my method")
j = G.index("j", "j", "bin")
g = G.index("g", "g", "group")
m = G.symbol("m", "m_{jg}", "calls", dims=(j, g), role="input")
R = G.symbol("R", "R_{jg}", "residuals", dims=(j, g))
kr = G.setting("kr", r"\kappa_r", "pseudocounts", value=5, code="my_pkg.settings:Settings.pseudo")
G.op("residuals", "residuals", r"R_{jg} = \frac{m_{jg} + \kappa_r q}{n_{jg} + \kappa_r} - q",
     inputs=[m, kr], outputs=[R], group="model",
     code="my_pkg.model:residuals",                 # or "my_pkg.model:fit#step 5" / "#from text..to text"
     bind={"methylated": m, "return": R},           # for the shape trace
     where={"q": "the genome-wide fraction"})       # symbols used only in this step's equation

@G.verify("residuals")                              # the equation, written plainly, against the code
def residuals_as_written(): ...
```

Each step's side panel also shows its code written as math ("From the code"). This covers base Python and
numpy, and is read from the code without running it. Give `names={"methylated": m, "pseudo": r"\kappa_r"}` to
write the code's names as symbols: a symbol's indices then carry into the math, so `x.sum(0)` on $x_{jg}$ is
$\sum_j x_{jg}$. The code and the equation sit side by side, and a mismatch between them is easy to see.

A step with no description yet shows its code's docstring or the comment above it, in any language, marked as
coming from the code.

`headway init my_pkg.method:run` drafts this from the code, starting at the method's entry function:

- every call to the package's own functions is a step, with its code references;
- a step's inputs and outputs are the variables the code binds to and from its calls, traced by Python's scoping
  rules; a value made by other code between steps is shown as an input "made by the code" at its line, not
  guessed at, and a comment lists where values may change in place;
- a call that changes a variable in place (stores into it) makes the variable's next value one of its outputs;
- functions that call others become groups, and loops become `per=`;
- with `--hints`, the code's type hints are taken as assumptions, with what the mypy type checker infers from
  them: a settings dataclass's fields become settings, methods of the package's classes become steps, arrays and
  lists changed in place (by `+=`, `.append`, or through a view) are followed, and shapes in the type hints
  (jaxtyping's `Float[np.ndarray, "cells genes"]`) or docstrings become `dims=`;
- docstrings (numpydoc, Google) describe the inputs, outputs, settings and groups.

It reads the source without running it, and draws no edge the code does not state. The TeX, the descriptions and
the dimensions are left as TODOs (see the guide).

`where` is for symbols an equation uses that are not values of the graph: a step's own intermediates, functions
like the logistic σ. The viewer shows them under the equation, as a paper would ("where q is …"). A value another
step also uses should be a symbol, listed in both steps' inputs or outputs, so it shows as a node and an edge.
headway does not read the TeX, so it cannot tell you which symbols lack a definition; it only checks that a
where-entry is not written exactly as a declared symbol the step should list instead.

### Documents, figures and media

A step, a value or a group can show images, tables, sounds, videos and excerpts of documents in its side panel.
Several documents can feed one headway (a paper, its supplement, a protocol), so a headway can describe an analysis
as a paper does, with its figures and tables beside the steps:

```python
paper = G.document("smith2024", "Smith et al. 2024", path="papers/smith2024.pdf",
                   url="https://doi.org/10.1000/xyz")
lister = G.document("lister2013", "Lister et al. 2013", url="https://doi.org/10.1126/science.1237905")
supp = G.document("smith2024s", "Smith et al. 2024, supplement", path="papers/smith2024_supp.pdf")
G.op("cluster", "clustering", r"...", inputs=[...], outputs=[...],
     media=[G.excerpt(paper, 4, panel="a", caption="Fig. 2a: the clusters"),          # found by its letter
            G.excerpt(paper, between=("We clustered", "into 12 groups."), caption="Results"),  # by its text
            G.cite(lister, note="the clusters it compares with"),                     # a source, linked
            G.table("tables/s3.csv", "Table S3", doc=supp, page=12),
            G.image("figs/umap.png", "UMAP of the clusters"),
            G.video("movies/trajectory.mp4", "The trajectory over time"),
            G.audio("https://example.org/sonified.mp3")])
```

- **An excerpt** is a region of a PDF page, rendered into the page when the headway is built (with poppler).
  It can be found by what it says: `between=("Successful identification", "(Fig. 3a–c).")` takes the lines of
  that passage, in its column, wherever it is in the document; `panel="c"` takes panel c of the figure on a page,
  bounded by its neighbors' letters and the gutters between them. Or give a box: (left, top, right, bottom) as
  fractions of the page. `headway excerpt PAPER.pdf --page 4 --panels` lists a page's panels, and
  `headway excerpt ... -o x.png` previews one. Images, tables and sounds up to 8 MB are written into the page
  too, so it stands alone. Larger files, and any with `embed=False`, are linked:
  the page loads them from their files, or through `headway serve` and the library.
- **Opening a document:** a medium that comes from a document says so, with a link that opens it at its page.
  With nothing selected, the side panel lists the documents and what each is shown with, as does the legend.
- **A document that has moved:** `headway build` looks for it in the project, by name, and by its content when
  `sha256=` is given, and warns. In the viewer, **locate…** next to a document asks for its file, checks that it
  is the same document, shows it in the page, and keeps a copy in the browser, which opens from then on. That is
  how a page opened on another machine reaches the documents it was built from.
- A step that a document describes needs no code reference.
- Find, with **Text**, searches the captions too.

## Comparing versions

A version is a built headway: the JSON from `build --json`, or a page from `build`. Keep one wherever it is useful
(at a tag, before an experiment, next to a set of results) and compare any of them, with each other or with the
spec as it is now. headway does not manage the versions or need git; it records the spec's file and, if it is in
a git repository, the commit, branch and whether the work tree had changes, and shows them with the version.

- **In the viewer**, `--compare` (on `build` and `serve`) or **Open…** → **Add beside** opens other versions, each
  in a lane of its own (served by `headway serve`, from the library; a page opened as a file uses the browser's
  file picker). The lanes share one layout, so a step sits in the same place in every lane: stacked one above
  another when the graph flows right, side by side when it flows down. A marker on each node says what changed
  against the lane before it (or against a lane pinned as the baseline): **+** added, **Δ** changed, **−** gone
  (drawn dashed where it was), **⋔** split or merged. The lanes start in the order the versions were built (a
  version opened later takes its place in that order); drag them to reorder, × to close one, any number of them.
  The eye on a version (in the version bar, and on its row in the Results view) hides it and keeps it open: it is
  left out of the graph and the results, and the objectives are scored over the versions shown (a rank objective
  ranks among them; a mean objective does not depend on the others). Press on one eye and drag across its
  neighbours, or shift-click, to hide or show several at once; **Show all** brings them back. A version is
  compared with the shown version before it, and a hidden baseline is set aside until it is shown again. The
  browser remembers which versions were hidden.
  Opening a version that is open already (the same build, or one given the same `--name`) reloads it. **One at a
  time** draws only the version you are reading, with its markers, and keeps the others open: click another to
  switch. Click a lane to read that version in the side panel and the legend. Selecting an object selects it in every
  lane; the panel lists it version by version and shows what changed, with the equations one above the other
  and a diff of the code. Ask-AI prompts say which version they are about and what changed in it; Explain has
  **What changed?**. **Collapse unchanged** (in the **Groups** menu) leaves open only the groups where something
  changed.
- **`headway diff`** prints the same comparison as text, version after version (`--wording` for labels and
  descriptions too). A spec module is built as it is now.

Objects correspond by key. When a key changes, say so with `was=`, on symbols, settings, constants, steps and
groups:

```python
G.op("weighted_score", "module score", ..., was="score")                    # renamed
G.op("sizefactor", "size factors", ..., was="normalize")                    # split: both halves name
G.op("scale_counts", "scale counts", ..., was="normalize")                  # the step they came from
G.symbol("S", "S_i", "module score", ..., was=["score", "cell_score"])      # renamed twice
```

A merged object lists every key it replaces. `was` is matched across all the open versions at once, so a rename two
versions back still lines up; `headway check` rejects a `was` naming a key the spec still declares. What counts
as a change is read from the built versions, never from the TeX: inputs and outputs, group, dimensions (structure),
the equation's text, the code the references resolve to (so a code change under an unchanged equation shows), a
setting's value, the equation check, and the wording (labels, descriptions; hidden unless asked for). Code no
reference points to is not seen.

`examples/cellscore` is a small method developed over four versions (gene centering; size factors, splitting a
step; gene weights, renaming the score), with `build.sh` building each version and the comparison page.

## Benchmarks

A benchmark suite is declared like a method, as a `headway.Suite` (a Headway): each metric is a step, with its
notation, equation, description, code reference and `@verify` check, so it is annotated and checked the same way.
A suite adds its datasets (some can be held out), each metric's direction and range, and objectives, which combine
metrics into one number to rank by and are declared apart from them.

```python
B = Suite("my benchmark", version="1")
S = B.symbol("S", "S_i", "score", dims=(i,), role="input")
B.dataset("deep", "deep sequencing", "...", values={"depth_sd": 0.3}, code="my_bench.data:simulate")
B.dataset("atlas", "cell atlas", "...", source="GSE123456", facts={"cells": "12,000", "platform": "10x v3"},
          heldout=True, code="my_bench.data:load_atlas")
B.metric("auroc", r"\mathrm{AUROC}", "separation", "How often ...", equation=r"...", inputs=[S, y],
         better="higher", range=(0, 1), code="my_bench.metrics:auroc", source="Mann-Whitney U, scaled")
B.objective("overall", "overall", metrics={"auroc": 2, "depth_dep": 1})        # how="mean" (absolute) or "rank"
```

Datasets are annotated too. When the suite makes its data (a simulation, a preprocessing), declare those steps in
the suite like any other, with their equations, code and checks, and give each dataset its `values` of their
settings: the steps are written once, and the datasets differ in their values (`headway check` rejects a value
for anything but a declared setting). A dataset can also have a `source`, `facts` (anything else worth knowing)
and `code` (where it is made or loaded).

headway computes no metric. Your benchmark writes a results file, `{"suite": "my_bench.suite", "values":
{dataset: {metric: [one value per replicate]}}, "replicates": [ids], "figures": {...}, "run": {...}}`, and `build
--results` or `attach` puts it into a version's headway with the suite, built and checked. Versions with results on
the same suite (title and version) are compared, wherever their files are. `replicates` names the values (seeds,
folds) so that versions pair up. `figures` gives the images the benchmark made for each dataset and figure the
suite declares (`B.figure`); they are written into the headway and shown as columns beside the metrics, a
thumbnail per version, compared side by side when clicked: to look at, never ranked (a judgment of them is a
metric of its own). `--key NAME=VALUE` (or `keys=` in Python) labels a version with a hyperparameter, a variant or an option.
Any value can carry an interval, `{"value": 0.81, "lo": 0.78, "hi": 0.84}`, of a kind the metric declares
(`interval=`), and `differences` gives a version's difference from another with its own interval, computed by the
benchmark on the same samples for both. The viewer pairs versions by replicate: under each value, `+0.026 5/5 ●`
is the mean per-replicate difference, how many replicates agree, and whether the benchmark's intervals of the
difference exclude 0 (● all, ◐ some, ○ none); it is colored only when supported.

In the viewer, **Graph | Results** switches between the computation graph and the results of the same open
versions: a heatmap of versions × metrics × datasets (grouped by metric or by dataset), each value the mean over
replicates ± their s.d., shaded from worst to best among the versions (or across the metric's range), with its
difference from the row above it (or the baseline) and its place among the versions shown (**Ranks**: `#1` the best,
`#2=` a shared place). Missing results are hatched, never filled in; an objective missing
any value it needs is "incomplete". **+ Objective** adds one of your own, kept in the browser: the sum,
over the metrics and the datasets shown, of a transform of each version's mean that you type per metric (`-x`,
`x/0.002`, `log(x)`, ...). Click a column for what the metric or objective is (its equation, code, check
and source) and the replicates of every version, plotted. A column's header sorts by it (clicked again,
unsorted; a metric's header sorts by the mean place over the datasets shown); drag a version's row to reorder the
versions (the graph's lanes too; a sorted order is kept as the new order), or a column's header to reorder the
columns. **Columns** shows, hides and reorders the objectives, metrics, datasets and figures (kept in the browser,
per suite; hiding a column changes no objective). Each value's difference is from the row
above it as shown, or from the baseline. Click a dataset (under
the table, or its column when grouped by dataset) for what it is: its settings, how it is made, its facts and code,
and every version's results on it. In the suite's own page, the legend lists the datasets, and a setting shows
its value in each. Click a version for what
changed in it, and to go to it in the graph. The Ask-AI prompts there carry the results, the metrics' definitions
and what changed between the versions: **Why do they differ?** and **What to try next?**. `headway results`
prints the same table as text. The objectives are the only computation headway does on results (bench.py; the
viewer computes them the same way).

## What is checked

| check | catches | where |
|---|---|---|
| references | a renamed, moved or deleted function (in any language tree-sitter parses); an anchor that is gone | `headway check` |
| structure | undeclared symbols or indices, two producers of one value, cycles, unused values | `headway check` |
| equations (`@G.verify`) | code that no longer computes the equation as written | `headway check`; a failure is marked ✗ in the viewer |
| shapes (`headway.trace`) | arrays whose axes do not match the declared indices, an index with two sizes | your test suite |

```python
with headway.trace(G) as t:
    run_the_method_on_small_data()
t.assert_ok()
```

Not checked: that the graph's edges are all the data flow there is (a step could use a value the spec does not
list), or that an equation without a `@verify` matches its code.
