Metadata-Version: 2.5
Name: phigrade
Version: 3.1.0
Summary: A lightweight autograder where tests run locally and data is stored remotely
Project-URL: Repository, https://github.com/mld-instructors/phigrade
Author-email: Matt Gormley <mgormley@cs.cmu.edu>, Jacob Rast <jrast@andrew.cmu.edu>
License-Expression: MIT
License-File: LICENSE
Keywords: autograder,education,grading
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: <4.0,>=3.12
Requires-Dist: fastapi<0.116.0,>=0.115.13
Requires-Dist: httpx<0.29.0,>=0.28.1
Requires-Dist: numpy<3.0.0,>=2.0.0
Requires-Dist: omegaconf<3.0.0,>=2.3.0
Requires-Dist: pydantic<3.0.0,>=2.11.7
Requires-Dist: requests<3.0.0,>=2.32.4
Requires-Dist: tinydb<5.0.0,>=4.8.2
Requires-Dist: uvicorn<0.35.0,>=0.34.3
Description-Content-Type: text/markdown

# phigrade

The PhiGrade client library. Instructors use it to publish an answer key by running
their reference solution; students use it to check their own code against that answer
key and get immediate, per-checkpoint feedback.

The defining property is that **student code never leaves the machine it runs on.**
Tests execute locally; what travels to the server is the *value* a function produced
at a checkpoint, which the server compares against the value the instructor's solution
produced at the same checkpoint. See the [top-level README](../README.md) for the full
rationale and the platform's data model.

This package works against a hosted PhiGrade backend or entirely offline against a
local file, using the same test code either way.

> **A word on vocabulary.** This document says **test** for the pytest function you
> write, and **unit check** for the record the server stores about it — the
> `UnitCheck` row that carries its point value and aggregation policy. The web
> application displays that same record to students as a **"Unit Test"**, and a
> `SubUnitCheck` as a **"Comparison"**. The code avoids the name `UnitTest`
> because pytest collects classes matching `Test*`, and these models are imported
> into test modules.

## Installation

```bash
pip install phigrade
```

For development inside this repository, the Python components form a single `uv`
workspace rooted at the repo:

```bash
uv sync --all-packages        # from the repository root
```

Requires Python 3.12+.

## Quick Start

**1. Write tests.** A PhiGrade test is an ordinary function decorated with `@weight`,
which declares its point value. Inside it, call a comparison helper at each point you
want checked.

```python
import numpy as np
import phigrade
from phigrade import weight

from mysolution import my_add, my_normalize


@weight(1.0)
def test_my_add():
    phigrade.is_equal(my_add(2, 3))


@weight(3.0)
def test_my_normalize():
    result = my_normalize(np.array([3.0, 4.0]))
    phigrade.all_close(result, rtol=1e-5, atol=1e-8)
```

Note what is absent: no expected values. The instructor's reference run supplies them.
The same file is the assignment's test suite *and* its answer key generator.

**2. Add a `phigrade.yaml`** next to the test file:

```yaml
course_id: "your-course-uuid"
assessment_id: "your-assessment-uuid"
use_local_server: false
server_url: "https://phigrade.example.org"
teacher_mode: false
timeout_seconds: 5
submission_files:
  - mysolution.py
```

Your API key is **never** written to `phigrade.yaml`. It is read from the
`PHIGRADE_API_KEY` environment variable, so a config file can be committed and shared
without leaking a credential:

```bash
export PHIGRADE_API_KEY="your-api-key"   # mint this on the Account Settings page
```

An `api_key` key in `phigrade.yaml` is a hard error pointing you at the environment
variable. Local mode needs no key at all.

**3. Run the tests.** They are named `test_*`, so `pytest` collects them normally:

```bash
pytest test_mysolution.py
```

Passing checkpoints print a confirmation; a failing checkpoint raises `AssertionError`
naming the checkpoint and showing what your code produced.

**4. Publish the answer key** (instructors). Run the same tests with
`PHIGRADE_TEACHER_MODE=true`, from a directory holding your reference solution. Each
checkpoint's value is recorded as the expected value, and the whole run publishes as
one definitive answer key when it finishes clean — see
[Teacher mode](#teacher-mode-teacher_mode-true) for what that means. The committed
`phigrade.yaml` stays `teacher_mode: false`, so there is nothing to set back:

```bash
PHIGRADE_TEACHER_MODE=true pytest test_mysolution.py
```

## Authoring API

Everything students and instructors need is exported at the package top level.

```python
import phigrade
from phigrade import weight, load_phigrade_config
```

The comparison helpers (`all_close`, `is_equal`, `row_match`, `key_value`,
`always_pass`) are called
qualified — `phigrade.all_close(...)` — so the module they come from is visible at the
call site. `weight` and `load_phigrade_config` are not comparison functions and are
imported bare.

### `@weight(w: float, aggregation: str = "fail_fast")`

Marks a function as a graded test and declares its point value. Attaches
`is_utest`, `autograder_function`, `autograder_module`, `weight`, and `call_count` to
the wrapped function, and installs the output-capture and Gradescope-finalization
wrappers.

`aggregation` decides how the test's checkpoints combine into its score, and is
validated at decoration time against `fail_fast` (the default), `even_weight`, and
`weighted`. See [Multiple checkpoints per test](#multiple-checkpoints-per-test).

> **The decorated function must be a module-level global.** The comparison helpers
> locate their calling test by walking the stack and looking the function up in the
> frame's globals. Nested functions and methods are not found, and raise
> `phigrade function must be called from within a function decorated with @weight`.

### `phigrade.is_equal(system_output: Any, weight: float = 1.0) -> None`

Records `system_output` as a checkpoint, compared by exact equality (`==`). The value
must be JSON-serializable.

### `phigrade.all_close(system_output: np.ndarray, rtol=1e-05, atol=1e-08, equal_nan=False, weight=1.0) -> None`

Records a NumPy array checkpoint, compared with `np.allclose` at the given tolerances,
which are stored alongside the reference so the student run uses the same ones. Raises
`TypeError` if `system_output` is not an `ndarray`.

### `phigrade.always_pass(weight: float = 1.0) -> None`

A checkpoint that simply records `True`, equivalent to `phigrade.is_equal(True)`. Useful
when reaching a line of code is itself the thing being graded — for example, after an
in-test assertion that would have raised.

### `phigrade.row_match(system_output: str, weight: float = 1.0) -> None`

A **partial-credit** checkpoint for multi-line string output. Both sides are split into
lines and compared position by position, ignoring trailing whitespace; the fraction
awarded is `matched / max(len(reference_lines), len(produced_lines))`, so missing and
extra lines are both penalized. Raises `TypeError` if `system_output` is not a `str`.

> A partial score still *fails* the unit test: like every other helper, this one raises
> `AssertionError` unless the checkpoint earns full credit. The partial points are
> recorded on the server and appear in the student's score, and the `AssertionError`
> names the points earned so partial credit is not mistaken for zero.

### `phigrade.key_value(system_output: str, threshold=1e-3, pattern=r"^(\S+)\s+(\S+)$", weight=1.0) -> None`

A **partial-credit** checkpoint for a file of `key value` lines — a metrics file, say.
Each non-blank line is parsed with `pattern` into a key and a numeric value, and a key
matches when it is present on both sides and the two values differ by less than
`threshold`, an **absolute** tolerance. The fraction awarded is
`matched / max(reference_keys, produced_keys)`, so a missing key, an extra key and a
malformed line each cost one key rather than the whole checkpoint. `threshold` and
`pattern` are recorded with the reference, so the student run reuses the instructor's
settings. Raises `TypeError` if `system_output` is not a `str`.

Unlike `row_match`, keys are matched **by name, not by position**, and the
`AssertionError` names the keys that cost the points along with the values your code
produced for them:

```
1 of 2 keys matched. Values differed for: error(test) (yours: 0.55).
```

> It never names the reference value or the size of the difference. Either would hand
> back the answer key, which is the one thing that must stay on the server.

The `weight` argument on every helper is that checkpoint's relative weight, used only
by the `weighted` aggregation. It is recorded when the instructor publishes the answer
key, so students do not need to pass it.

All helpers set `__tracebackhide__`, so pytest hides the phigrade frames and the
failure points at your test.

### Multiple checkpoints per test

A test may call the helpers as many times as it likes. Each call becomes a separate
checkpoint named `<function>@<n>` — `test_my_concat@0`, `test_my_concat@1`, and so on —
so a single test can award partial credit.

How the checkpoints combine into the test's score is chosen with `@weight(...,
aggregation=...)` and recorded on the unit check when the answer key is published:

Each checkpoint scores a fraction between 0 and 1; the combined fraction is then
scaled to the test's declared point value, so a `@weight(3.0)` test with every
checkpoint correct earns 3.0.

| Aggregation | Test score |
| --- | --- |
| `fail_fast` (default) | `w ×` the smallest checkpoint fraction; correct only if every checkpoint is correct |
| `even_weight` | `w ×` the mean checkpoint fraction |
| `weighted` | `w ×` the mean weighted by each checkpoint's `weight` argument |

> The counter lives on the function object and increments across calls within a
> process. Invoking the same test twice in one interpreter produces `@0, @1` on the
> first run and `@2, @3` on the second, which will not match the reference. Let pytest
> run each test once.

### `load_phigrade_config(test_func) -> PhiGradeConfig`

Returns the resolved configuration, letting a test branch on mode — useful when the
reference run and the student run should exercise different code:

```python
config = load_phigrade_config(test_my_add)

@weight(3.0)
def test_handles_edge_case():
    if config.teacher_mode:
        phigrade.is_equal(reference_impl(...))
    else:
        phigrade.is_equal(student_impl(...))
```

## The Two Modes

The mode comes from `teacher_mode` in `phigrade.yaml`, or from the
`PHIGRADE_TEACHER_MODE` environment variable, which overrides it. There is no CLI flag.

The override is what lets the committed config stay `teacher_mode: false` forever:
publishing is then a property of the command, not an edit-and-revert dance on a file
that is easy to commit in the wrong state. It announces itself on stdout — see
[Environment overrides](#environment-overrides).

### Teacher mode (`teacher_mode: true`)

1. Creates a **staged reference attempt**. `submission_files` are sent as
   *paths with empty contents*, so the reference solution's source is never
   uploaded. The attempt is not the answer key yet — the previous key stays in
   force until the run completes.
2. Registers each `@weight` test as a unit check, carrying its point value.
   Re-running is safe; an already-existing unit check is tolerated.
3. Records each checkpoint's value and comparison settings as the expected
   answer.
4. **Publishes at the end of a clean run**, and only then. A publish is
   definitive: the answer key in force becomes exactly the tests and
   checkpoints that run recorded. A test you deleted stops being graded; a
   test whose module you renamed is a new test, and the old one retires.
   Earlier references are kept for the record and never graded against.
5. **Never asserts.** A reference run cannot fail a comparison.
6. Writes no Gradescope output and does not capture stdout.

> **A publish is definitive about *which* tests are in the key, not about what
> they are worth.** A test's point value is fixed at its first-ever publish:
> step 2 tolerates an already-existing unit check rather than updating it, so
> changing a test's `@weight` and republishing silently leaves the old value in
> force — and still reports a clean publish. Renaming the test, or its module,
> is the only way to change a point value today, because that is a new test as
> far as the key is concerned.

> **A run that dies partway publishes nothing.** A collection error, a raise
> in the reference solution or a `^C` leaves the staged attempt unpublished
> and the previous answer key in force, and the run says so at exit. Fix the
> problem and run it again — there is no half-published state to undo.
>
> Publishing happens through a pytest plugin that ships with the package (the
> `pytest11` entry point in `pyproject.toml`), so a plain
> `PHIGRADE_TEACHER_MODE=true pytest` needs nothing extra — as long as the
> installed package knows about the entry point. An environment set up before
> this shipped needs `uv sync --all-packages` (or a plain reinstall) once, so
> pytest actually discovers the plugin; run it before the plugin is installed
> and the exit warning above still catches you — every test recorded cleanly,
> but you get "this teacher-mode run did not complete" anyway, because nothing
> ever called the publish. If you drive the tests yourself rather than through
> pytest — a custom runner, a notebook — call `phigrade.finalize_reference_run()`
> yourself once they have all run; it is the same idempotent publish the
> plugin calls, safe to call again on a run that already published.

> **Teacher mode refuses to run under `pytest-xdist` at all.** Each `-n`
> worker is a separate process running only part of the suite, so no single
> process ever holds the whole answer key; publishing from one would drop
> every test the others ran. `PHIGRADE_TEACHER_MODE=true pytest -n auto` fails
> immediately, on the first test, before anything is staged. `-n auto` is
> still fine — and, for a large class, wanted — for grading *student* runs
> and for the `pytest -n auto` case mentioned under
> [Submission Slots](#submission-slots); it is only a teacher-mode publish
> that must run single-process. Drop `-n` (or add `-p no:xdist`) to publish.

In local mode, a teacher run **rebuilds the assessment it is publishing** from
scratch: every row belonging to that `assessment_id` is dropped when the
reference attempt arrives, so a checkpoint or a test you deleted cannot
survive in the file. Rows belonging to *other* assessments are untouched,
which is what lets one answer-key file hold several
[submission slots](#submission-slots). This is deliberately unlike the
server, which supersedes rather than deletes: offline there is no gradebook,
and nothing to keep history for. Offline still follows the same grading rule
— a key that was never published does not grade — so a local run that dies
partway leaves a file that reports a missing reference rather than one that
grades against half a key. An answer-key file written by an older `phigrade`
has no record of which reference attempt wrote each reference and will not
grade at all; the 404 it produces says so and points at republishing, since
there is deliberately no legacy fallback.

### Student mode (`teacher_mode: false`)

1. Creates a normal attempt, uploading the **actual contents** of every path in
   `submission_files`. A missing file raises `FileNotFoundError`.
2. Submits each checkpoint's produced value and reads back the verdict.
3. Prints a line per checkpoint; raises `AssertionError` on the first failure within a
   test.
4. Writes Gradescope output if configured.

Students cannot define unit checks. A checkpoint with no published reference raises,
which is the expected signal that the instructor has not published the answer key yet.

## Configuration

The only configuration source is a YAML file named `phigrade.yaml`, located **in the
directory containing the test module** that defines the `@weight` function — not the
current working directory. It is loaded once and cached for the life of the process.

Four keys can be overridden from the environment — see
[Environment overrides](#environment-overrides) below. No other key can.

| Key | Type | Default | Notes |
| --- | --- | --- | --- |
| `course_id` | str | **required** | Required in both local and remote mode |
| `assessment_id` | str or mapping | **required** | The assessment's UUID, or a submission-slot block — see [Submission Slots](#submission-slots) |
| `use_local_server` | bool | `false` | Run against a local file instead of a server; `PHIGRADE_USE_LOCAL_SERVER` overrides |
| `server_url` | str | — | Required when remote; overwritten in local mode |
| `teacher_mode` | bool | `false` | Reference run vs. student run; `PHIGRADE_TEACHER_MODE` overrides |
| `timeout_seconds` | int | `2` | Per-request HTTP timeout |
| `local_server_db_file` | str | `phigrade_db.json` beside the config | The local answer key; `PHIGRADE_LOCAL_DB_FILE` overrides |
| `gradescope_json_file` | str | — | Enables Gradescope output (student mode only) |
| `submission_files` | list[str] | — | Paths **relative to the working directory** |

The API key is deliberately absent from this table: it is read from the
`PHIGRADE_API_KEY` environment variable, and an `api_key` key in the file is rejected.

Remote mode requires `server_url`, `course_id`, `assessment_id`, and a non-empty
`PHIGRADE_API_KEY`; local mode requires `course_id` and `assessment_id`. Note that
`submission_files` resolve
against the CWD while `phigrade.yaml` resolves against the test module's directory —
the usual cause of a `FileNotFoundError` is running pytest from the wrong directory.

### Environment overrides

| Variable | Overrides | Values |
| --- | --- | --- |
| `PHIGRADE_TEACHER_MODE` | `teacher_mode` | `1`/`true`/`yes`, `0`/`false`/`no` |
| `PHIGRADE_USE_LOCAL_SERVER` | `use_local_server` | the same |
| `PHIGRADE_LOCAL_DB_FILE` | `local_server_db_file` | a path |
| `PHIGRADE_SLOT` | which slot `assessment_id` resolves to | a `slot_name` |

Between them, one committed `phigrade.yaml` serves online publishing, offline
development, and a separate offline answer key per slot.

The two booleans are matched case-insensitively and ignore surrounding whitespace. Any
*other* value is an error naming the variable, rather than a silent default: quietly
reading a misspelt `ture` as false would fail to publish an answer key with no
indication of why. An unset or blank variable is simply not an override.

`PHIGRADE_LOCAL_DB_FILE` is used exactly as given, so a relative path resolves against
the **working directory** — unlike the config key it overrides, which defaults to a path
beside `phigrade.yaml`.

Every override announces itself on stdout when the config loads:

```
Teacher mode: enabled (PHIGRADE_TEACHER_MODE)
Local answer key: slot_a_key.json (PHIGRADE_LOCAL_DB_FILE)
```

That line is the whole point of announcing: a variable exported once in a shell profile
must not silently republish an answer key, or redirect a submission, weeks later. The
config is loaded once per process, so each line appears at most once per `pytest` run.

### Submission Slots

`assessment_id` can be a mapping instead of a single UUID string, to offer several
assessments as named **submission slots** the student picks between — for example a
"human work only" slot beside a "human or AI work" slot, both backed by their own
assessment on the server:

```yaml
course_id: "your-course-uuid"
assessment_id:
  message: "Select the appropriate submission slot."
  slots:
    - slot_name: "Human only"
      slot_description: "This submission contains human work only."
      slot_assessment_id: "human-only-assessment-uuid"
    - slot_name: "Human or AI"
      slot_description: "This submission may contain AI-assisted work."
      slot_assessment_id: "human-or-ai-assessment-uuid"
```

`message` is optional and defaults to `"Select the appropriate submission slot."`.
Each entry of `slots` requires all three keys: `slot_name`, `slot_description`, and
`slot_assessment_id` (the assessment UUID that slot resolves to).

The slot is resolved once, at config-load time, in this order:

1. The `PHIGRADE_SLOT` environment variable, if set to a non-empty value. The match
   against `slot_name` ignores surrounding whitespace and case.
2. Otherwise, an interactive prompt listing each slot's name and description,
   numbered.
3. Otherwise, a `ValueError` naming the valid slot names.

Either way the resolved slot is announced — `Submitting to: Human only` — so a
`PHIGRADE_SLOT` exported once in a shell profile cannot silently redirect later
submissions. On the environment-variable path that line goes to stdout, which
pytest captures and prints on failure (or always, under `-s`).

The prompt reads and writes `/dev/tty` directly rather than stdin/stdout, so it
works under a plain `pytest` run **with no `-s` flag** — pytest's output capture
never sees it.

Some runs have no terminal that can be prompted on, and they **must** set
`PHIGRADE_SLOT` to one of the slot names:

```bash
export PHIGRADE_SLOT="Human only"
```

That covers non-interactive environments — a Gradescope autograder, CI — and two
cases that would otherwise hang: `pytest -n auto` (each xdist worker is its own
process, so N workers would interleave N prompts on one terminal) and any
background job (`nohup pytest &` under a job-control shell, where reading the
terminal raises `SIGTTIN` and suspends the process). Both are detected and turned
into the same error naming the valid slot names.

Because the config is loaded once and cached for the life of the process (see
above), a `pytest` run resolves the slot — and therefore prompts, if it prompts at
all — at most once, no matter how many tests run.

#### Teacher mode publishes to one slot only

A reference run (`teacher_mode: true`) resolves the slot exactly as a student run
does, and then publishes the answer key to that **one** slot's assessment —
references are per-assessment on the server. A student who picks any other slot
gets a `Sub-unit-check reference not found` error on every checkpoint.

So an instructor must publish once per slot. Because the config is cached per
process, that means one process per slot:

```bash
PHIGRADE_TEACHER_MODE=true PHIGRADE_SLOT="Human only" pytest test_mysolution.py
PHIGRADE_TEACHER_MODE=true PHIGRADE_SLOT="Human or AI" pytest test_mysolution.py
```

Run those before releasing the assignment, and re-run both whenever the reference
solution changes.

In local mode both publishes can share one answer-key file: a reference run clears only
the rows of the assessment it is publishing. Giving each slot its own file with
`PHIGRADE_LOCAL_DB_FILE` also works, and is what you want when the two keys are
distributed separately.

## Authenticating

Remote mode needs an API key. There are two ways to provide one:

**`PHIGRADE_API_KEY`** — set it in the shell before running tests, as shown above.
Good for CI and for one-off overrides.

**`phigrade auth login`** — a saved credential, so you don't need the environment
variable in every shell:

```bash
phigrade auth login --server https://phigrade.example.org
```

It prompts for an API key (mint one in the web app's Account Settings page),
validates it against the server, and on success saves it to
`$XDG_CONFIG_HOME/phigrade/credentials.json` (`~/.config/phigrade/credentials.json`
if `XDG_CONFIG_HOME` is unset), created with file mode `0600` so it is never
world-readable. `phigrade auth whoami --server <url>` reports which account a saved
(or `--api-key`-supplied) credential belongs to. `--token` is still accepted as an
alias for `--api-key`, for scripts written before the terminology settled.

A remote-mode `phigrade.yaml` run resolves its key in this order: `PHIGRADE_API_KEY`
first, then the credentials file for that `phigrade.yaml`'s `server_url`. The
environment variable always wins, so it can still override a saved credential without
disturbing it.

## Local Mode

Set `use_local_server: true` and no backend is needed. The client starts a local server
in a daemon subprocess on an automatically chosen port, backed by a TinyDB JSON file.

This is not a mock: the local server implements the same HTTP API as the hosted
backend, and both use the *same* comparison code (`phigrade/compare.py`, which the
backend imports directly). A suite developed locally behaves identically against a real
course. Local mode has no authentication, so `PHIGRADE_API_KEY` need not be set.

Two distinct uses:

* **Drafting an assignment** before a course exists on the server.
* **Distributing a self-contained assignment** — ship the tests plus the answer-key
  JSON, and students get immediate feedback with no account, network, or credentials.

**The invariant that makes distribution safe:** a student-mode run never writes to the
answer-key file. It requires the file to exist, copies it to a temporary file, and
serves from the copy. Running the tests cannot corrupt or reveal the key, accidentally
or otherwise. (`tests/test_student_mode_no_persistence.py` exists to enforce this.)

## Gradescope Integration

Set `gradescope_json_file` in student mode and the client writes a Gradescope
`results.json`:

```yaml
gradescope_json_file: "results.json"
```

Behavior worth knowing:

* The file is rewritten after **every** test, so it is valid even if the run is killed
  partway through.
* `stdout` and `stderr` inside a test body are tee-captured — still shown live — and
  included in that test's output. Python streams only; subprocess output is not caught.
* A test that raises before making any comparison is still reported, with an
  explanatory message and a score of zero.
* Unit checks the student never attempted appear with score zero, since the test list
  comes from the published unit-check set rather than from what ran.
* A test is `passed` only if it made *all* its expected comparisons and all passed —
  skipping comparisons cannot yield a pass.
* Individual outputs longer than 500 characters are truncated in the middle.

No `run_autograder` / `setup.sh` scaffolding ships with this package; wire it into your
own Gradescope container.

## Package Layout

```
phigrade/
├── phigrade/
│   ├── __init__.py            Public exports: weight, is_equal,
│   │                          all_close, always_pass,
│   │                          row_match, key_value, PhiGradeConfig,
│   │                          load_phigrade_config, __version__
│   ├── phigrade.py            The client: config loading, the @weight decorator and
│   │                          its wrappers, comparison helpers, submission creation,
│   │                          HTTP calls, Gradescope finalization, finalize_reference_run
│   ├── pytest_plugin.py       The pytest11 plugin: publishes a staged reference
│   │                          attempt from pytest_sessionfinish on a clean run
│   ├── app.py                 create_app() — the TinyDB-backed local server, mirroring
│   │                          the backend's REST API
│   ├── server.py              Server lifecycle: port discovery, subprocess spawn,
│   │                          readiness polling
│   ├── compare.py             Scoring and aggregation — shared with the backend
│   ├── gradescope_output.py   results.json writer
│   ├── cli.py                 The `phigrade` console script: `auth login`, `auth whoami`
│   └── credentials.py         The CLI's credentials file (server-keyed API keys)
├── examples/                  A runnable end-to-end example (see below)
└── tests/                     The suite (see below)
```

### Examples

`examples/` is both documentation and a regression test — its tests run as part of the
default suite.

| File | What it shows |
| --- | --- |
| `simplefns.py` | The code under test, including a deliberately wrong variant |
| `test_simplefns.py` | The canonical autograder file: single and multiple comparisons per test, branching on `teacher_mode`, and the missing-reference error |
| `phigrade.yaml` | A local, student-mode configuration |
| `phigrade_db.json` | A checked-in answer key from a teacher run — this is what makes the example runnable out of the box |
| `run_teacher_mode.py` | The reference workflow, including the global-state reset ritual |

Run them from the `phigrade/` directory (not from `examples/`, since paths resolve
against the CWD):

```bash
uv run --all-packages pytest examples/test_simplefns.py
```

## Contributing

### Tests

```bash
cd phigrade
uv run --all-packages pytest
uv run --all-packages pytest tests/test_local_server.py
```

There is no `conftest.py` and no pytest configuration — discovery is the default from
the `phigrade/` directory.

| Test file | Covers |
| --- | --- |
| `test_local_server.py` | The local server's HTTP endpoints directly |
| `test_is_equal_local.py` | The largest suite: end-to-end runs against generated test modules, call-count tracking, Gradescope output, error paths |
| `test_all_close_local.py` | Tolerances, non-array inputs, mixed comparison types |
| `test_compare_rowmatch.py` | `rowmatch` partial-credit fractions and aggregation validation, at the unit level |
| `test_compare_keyvalue.py` | `keyvalue` partial-credit fractions and its per-key feedback, at the unit level |
| `test_phigrade_aggregation_local.py` | Each aggregation type end-to-end, plus `always_pass`, `rowmatch` and `keyvalue` |
| `test_phigrade_config.py` | Config loading and the environment overrides (the invalid-config paths are **not** covered) |
| `test_student_mode_no_persistence.py` | The answer-key isolation invariant, and the per-assessment reset |
| `test_teacher_mode_endpoints.py` | Which endpoints each mode calls, with `requests.post` mocked |
| `test_real_backend_integration.py` | Full stack against a real backend instance |

The last of these boots the actual backend, which works because the `uv` workspace puts
both packages in one environment.

### Global state

The client keeps process-global mutable state: `_config`, `_local_server`,
`_submission_id`, `_unit_check_attempt_ids`, `_unit_check_definition_cache`,
`_gradescope_test_errors`, and `_gradescope_test_output`. There is no public reset API,
so anything that runs more than one logical session in a single interpreter must clear
these by hand — see the `clear_phigrade_state` autouse fixture in
`test_real_backend_integration.py` and the reset in `examples/run_teacher_mode.py`.
Removing this global state in favor of an explicit session object would be a welcome
improvement.

### Lint, format, and types

```bash
make python-tools-phigrade      # from the repo root: ruff check, ruff format, mypy
```

Ruff handles both linting and formatting (it replaced black). Line length 88, target
`py312`, rules `E,F,I,B,UP`. Type hints on everything; docstrings on public functions.

### Releasing

`make upload` from the repository root — it refuses to run with a dirty working tree,
then builds and uploads to PyPI:

```bash
make upload      # uv build --package phigrade && twine upload dist/*
```

Bump `version` in `pyproject.toml` first; `phigrade.__version__` reads it from package
metadata at runtime.

## Known Limitations

* **The CLI is auth-only.** `phigrade auth login` and `phigrade auth whoami` are the
  only `phigrade` console-script commands; there is no `phigrade run` or similar, and
  tests are still run with `pytest`.
* **Only four keys have environment-variable overrides** — see
  [Environment overrides](#environment-overrides). Everything else is YAML only.
* **Checkpoint maximums are always `1.0`** when references are published. That is by
  design — a checkpoint scores a fraction, and the fraction is scaled to the test's
  `@weight` — but it does mean a checkpoint cannot carry its own point value. Relative
  weighting between checkpoints is settable, via each helper's `weight` argument under
  `weighted` aggregation.
* **`Server` has no `stop()`.** The subprocess is a daemon and dies with its parent.
* **Student-mode temporary answer-key copies are never cleaned up.**
* **The comparison set is small** — exact equality, `np.allclose`, `row_match` and
  `key_value`. Adding a comparison type means changing `compare.py`, which the backend
  imports, so both components must be updated and released together.

## License

MIT. See [LICENSE](./LICENSE).
