Roadmap · Testing & CI · P1

Cover the compiler's type rules

A build-it-yourself guide. This pins down the link-validation stage of compile_graph — the code that checks a wire's endpoints, its ports, and whether the two types actually match. It's ModuLearn's core promise and currently its least-tested code.

Done when: a new tests/test_type_rules.py proves valid wires compile, family and subtype mismatches raise, unknown ports raise, and multiple problems are reported together — and you've watched each test fail when the rule is deliberately broken.

0 / 0 steps

Where the logic lives

Everything you're testing is in modulearn/compiler.py. Read these lines before writing anything — the tests should mirror what's actually there, not what you assume:

Lines	What happens
177–200	The link loop. Every wire is validated here, and each failure is appended to `errors` rather than raised.
187–192	Port existence: `from_port` must be a real output, `to_port` a real input.
193–196	The type check. A plain string compare: `src_out[sp] != dst_in[dp]`.
197–199	An input wired more than once.
210–211	The collected raise — every link error surfaces together here.

The five behaviours to cover

#	Behaviour	How to trigger it
1	Valid wires compile	A correct dataset → model → train graph returns a `CompiledGraph`, no raise.
2	Family mismatch raises	Wire a `dataset` output into an input expecting `model`.
3	Subtype mismatch raises	Wire the `lr` hyperparameter into the `epochs` port — `scalar/lr` ≠ `scalar/epochs`.
4	Unknown port raises	Reference a `from_port`/`to_port` name that doesn't exist.
5	Errors are collected	Two bad wires in one graph → `GraphError.errors` contains both.

#3 is the headline. Field-specific scalar subtypes are why a learning rate can't land in an epochs slot. #5 is the most under-tested — the all-or-nothing promise only holds if two independent mistakes both appear.

Steps

Every block below is verified working against 0.2.0 — the suite passes, and each type test was confirmed to fail when the rule is broken (step 9). Build the file top to bottom.

Create the file with its docstring and imports

Make tests/test_type_rules.py. The __main__ block at the very bottom (step 8) lets it run without pytest, matching the other test files.

"""Type-rule tests: the compiler's link-validation stage. Valid wires compile,
family and scalar-subtype mismatches are rejected, unknown ports are caught, and
independent problems are reported together rather than one at a time.

Dependency-free — pure compiler, no web server or ML deps. Runs under pytest, or
standalone with ``python tests/test_type_rules.py``.
"""
from modulearn import compile_graph, GraphError
from modulearn.demo import build_registry

Add the baseline graph + a wire-removal helper

The baseline is a known-good graph; every failing test is this with exactly one wire broken. _drop_link matters because re-wiring an input that's already connected would also trip the "wired more than once" rule and muddy your assertion.

def _base_graph():
    """A minimal valid iris → mlp → train graph. Every failing case below is this
    graph with exactly one wire broken."""
    return {
        "nodes": [
            {"id": "d", "type": "dataset.iris", "params": {}},
            {"id": "m", "type": "model.mlp", "params": {"hidden": [8]}},
            {"id": "t", "type": "train.train", "params": {}},
        ],
        "links": [
            {"from": "d", "from_port": "dataset", "to": "m", "to_port": "dataset"},
            {"from": "m", "from_port": "model", "to": "t", "to_port": "model"},
        ],
    }


def _drop_link(g, to_id, to_port):
    """Remove a wire so we can re-wire that input without also tripping the
    'input wired more than once' rule."""
    g["links"] = [l for l in g["links"]
                  if not (l["to"] == to_id and l["to_port"] == to_port)]
    return g

Add catalog + error helpers

_port_types reads the declared types from the registry instead of hardcoding strings, so a rename can't silently make a mismatch test vacuous. _errors_from is the try/except/raise shape used across the suite.

def _port_types():
    """Read the declared port types out of the catalog rather than hardcoding
    them, so a future rename can't quietly make these tests vacuous."""
    types = {}
    for n in build_registry().catalog()["nodes"]:
        for p in n["inputs"]:
            types[(n["id"], "in", p["name"])] = p["type"]
        for p in n["outputs"]:
            types[(n["id"], "out", p["name"])] = p["type"]
    return types


def _errors_from(graph):
    """Compile and return the collected error list, failing if it compiled."""
    try:
        compile_graph(graph, build_registry())
    except GraphError as e:
        return e.errors
    raise AssertionError("expected GraphError, but the graph compiled")

Test 1 — a correct graph compiles

Write this first. If the baseline can't compile, every failing test below would pass for the wrong reason.

def test_valid_wires_compile():
    c = compile_graph(_base_graph(), build_registry(), run_id="r")
    assert c.dataset == "iris"
    assert c.model == "mlp"

Test 2 — family mismatch (dataset → model)

Drop the good m→t model wire, then feed the Train node's model input from a dataset output instead.

def test_family_mismatch_is_rejected():
    t = _port_types()
    # the whole point: these two families really are different
    assert t[("dataset.iris", "out", "dataset")] != t[("train.train", "in", "model")]

    g = _drop_link(_base_graph(), "t", "model")
    g["links"].append({"from": "d", "from_port": "dataset",
                       "to": "t", "to_port": "model"})
    errs = _errors_from(g)
    # name the exact wire: a loose "any type mismatch" would also match an error
    # from a different link and pass for the wrong reason.
    assert any("type mismatch" in m and "d.dataset" in m and "t.model" in m
               for m in errs), errs

Note the assertion names the wire. That isn't fussiness — see step 9, where a looser version was caught passing against a deliberately broken compiler.

Test 3 — scalar subtype mismatch (the headline)

Both ends are scalars; only the subtype differs. This is the test that proves a learning rate can't land in an epochs slot.

def test_scalar_subtype_mismatch_is_rejected():
    t = _port_types()
    lr_out = t[("hyperparameter.lr", "out", "value")]
    epochs_in = t[("train.train", "in", "epochs")]
    # both are scalars, but field-specific — that distinction is the feature
    assert lr_out.startswith("scalar/") and epochs_in.startswith("scalar/")
    assert lr_out != epochs_in

    g = _base_graph()
    g["nodes"].append({"id": "h", "type": "hyperparameter.lr", "params": {}})
    g["links"].append({"from": "h", "from_port": "value",
                       "to": "t", "to_port": "epochs"})
    errs = _errors_from(g)
    assert any("type mismatch" in m and "h.value" in m and "t.epochs" in m
               for m in errs), errs

Test 4 — ports that don't exist

Two small tests, one per direction. These hit lines 187–192 and short-circuit before the type compare ever runs.

def test_unknown_port_is_rejected():
    g = _base_graph()
    g["links"].append({"from": "d", "from_port": "dataset",
                       "to": "t", "to_port": "bogus"})
    errs = _errors_from(g)
    assert any("no input port" in m for m in errs), errs


def test_unknown_output_port_is_rejected():
    g = _base_graph()
    g["links"].append({"from": "d", "from_port": "nope",
                       "to": "t", "to_port": "loss"})
    errs = _errors_from(g)
    assert any("no output port" in m for m in errs), errs

Test 5 — every problem reported together

Two mismatches in one graph. Both live in the link loop, so both survive to the collected raise. Assert the count, not just presence — that's what proves nothing was dropped. Finish the file with the standalone runner.

def test_all_link_errors_are_collected():
    g = _drop_link(_base_graph(), "t", "model")
    g["links"].append({"from": "d", "from_port": "dataset",     # mismatch 1
                       "to": "t", "to_port": "model"})
    g["nodes"].append({"id": "h", "type": "hyperparameter.lr", "params": {}})
    g["links"].append({"from": "h", "from_port": "value",       # mismatch 2
                       "to": "t", "to_port": "epochs"})
    errs = _errors_from(g)
    mismatches = [m for m in errs if "type mismatch" in m]
    assert len(mismatches) == 2, errs
    assert any("t.model" in m for m in mismatches), errs
    assert any("t.epochs" in m for m in mismatches), errs


if __name__ == "__main__":
    for name, fn in sorted(globals().items()):
        if name.startswith("test_") and callable(fn):
            fn()
            print(f"ok  {name}")
    print("all type-rule tests passed")

Run them, then prove they bite
```
pytest tests/test_type_rules.py -v   # expect 6 passed
pytest -q                            # whole suite: 18 → 24
```
Now the step that actually matters. Temporarily invert the type check in modulearn/compiler.py line 193 and re-run:
```
# line 193 — change this:
        if src_out[sp] != dst_in[dp]:
# ...to this, run pytest, then change it back:
        if src_out[sp] == dst_in[dp]:
```
You should see 4 failed, 2 passed. Tests 1, 2, 3 and 5 must go red; the two unknown-port tests correctly stay green because they never reach the type compare. Then restore the !=.

This caught a real bug while writing this guide. The first draft asserted only any("type mismatch" in m). Under the inverted compiler that still passed — because the now-broken d→m wire produced its own "type mismatch" message, and the loose assertion happily matched it. The fix is the wire-naming assertions in steps 5–6. A green test you've never watched go red proves nothing.

Commit, push, confirm CI is green

git add tests/test_type_rules.py
git commit -m "Cover the compiler's type rules"
git push
gh run list --branch main --limit 1   # expect: success

CI now runs these across Python 3.10–3.13.

Mark it shipped on the roadmap

In docs/roadmap.html, find the item titled "Cover the compiler's type rules" and add done:true, after its eff: field — the same edit pattern used for the Robustness items. The board will show Testing at 1/5 and the bar will tick up.

Three traps

Loose assertions pass for the wrong reason. Checking only that some error says "type mismatch" will match a message from a completely different wire. Always name the endpoints you expect ("d.dataset" in m and "t.model" in m). This is the one that actually bit during writing — see step 9.

Duplicate-wire noise. If a test wires two things into one input, the compiler also flags "input wired more than once" (line 197). Use _drop_link first so each failing test exercises a single rule and your assertion stays unambiguous.

Errors short-circuit at line 210. The link loop's collected errors raise before the Train-count and model-resolution checks ever run. So a graph with a bad wire will never report a missing-Train error too — that's by design, and your tests should expect it rather than fight it.

ModuLearn · docs/type-rules-guide.html · companion to roadmap.html · line numbers refer to modulearn/compiler.py as of 0.2.0 · step state is local to this browser.