Metadata-Version: 2.5
Name: screamingface
Version: 0.1.11
Summary: Evaluate Models and Fusions on URL4-native research benchmarks
Project-URL: Homepage, https://screamingface.ai
Project-URL: Repository, https://github.com/ScreamingFace/screamingface
Project-URL: Issues, https://github.com/ScreamingFace/screamingface/issues
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai,benchmarks,evaluation,llm,url4
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: httpx>=0.27
Requires-Dist: ijson<4,>=3.4
Requires-Dist: pynacl>=1.5
Requires-Dist: pyyaml>=6.0
Requires-Dist: websockets<18,>=17.0.1
Provides-Extra: inspect
Requires-Dist: inspect-ai==0.3.263; extra == 'inspect'
Provides-Extra: notebook
Requires-Dist: ipyevents<3,>=2.0.2; extra == 'notebook'
Requires-Dist: ipywidgets<9,>=7.7.1; extra == 'notebook'
Provides-Extra: runtime
Requires-Dist: aiosqlite==0.22.1; extra == 'runtime'
Requires-Dist: bcrypt==5.0.0; extra == 'runtime'
Requires-Dist: cryptography==50.0.2; extra == 'runtime'
Requires-Dist: datasets==5.1.0; extra == 'runtime'
Requires-Dist: fastapi>=0.141.1; extra == 'runtime'
Requires-Dist: google-re2<2,>=1.1.20240702; extra == 'runtime'
Requires-Dist: immutabledict>=4.2.0; extra == 'runtime'
Requires-Dist: langdetect>=1.0.9; extra == 'runtime'
Requires-Dist: litellm==1.104.2; extra == 'runtime'
Requires-Dist: nats-py>=2.9.0; extra == 'runtime'
Requires-Dist: nltk>=3.9.1; extra == 'runtime'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.30.0; extra == 'runtime'
Requires-Dist: opentelemetry-sdk>=1.30.0; extra == 'runtime'
Requires-Dist: pdfplumber==0.11.10; extra == 'runtime'
Requires-Dist: prometheus-client>=0.26.0; extra == 'runtime'
Requires-Dist: pydantic-settings>=2.14.2; extra == 'runtime'
Requires-Dist: pyjwt==2.15.1; extra == 'runtime'
Requires-Dist: python-docx==1.2.0; extra == 'runtime'
Requires-Dist: python-multipart>=0.0.20; extra == 'runtime'
Requires-Dist: tortoise-orm[asyncpg]==1.1.8; extra == 'runtime'
Requires-Dist: uvicorn[standard]>=0.52.3; extra == 'runtime'
Description-Content-Type: text/markdown

# screamingface

Evaluate composable Candidate Recipes against URL4-native research Benchmarks.

> **Development status:** immutable Model, Fanout and Pipeline authoring, Engine-backed discovery,
> the direct evaluation API, and the confirmed `url4-cloud` lifecycle are implemented. The current MVP
> Engine publishes canonical `draco`, canonical `ifeval`, and the separately named
> `healthbench-worst30` challenge as complete URL4 Benchmark resources. There is no fixture,
> embedded benchmark runtime, or Client-side execution fallback.

## Local notebook runtime

Local Engine result artifacts live in the private `<data_dir>/artifacts` directory
rather than temporary storage. Set `URL4_CLOUD_ARTIFACTS_DIR` to override it;
`screamingface status` shows the absolute serving directory and its current size.
The Engine's retention policy still applies. This configuration is independent of SDK
report rendering, export and recovery APIs.

Install the SDK, notebook tools, and local services together:

```bash
pip install "screamingface[runtime,notebook]"
screamingface prepare --all  # first run only
screamingface up
```

`draco`, the Benchmark the examples below use, is a local inspect Task, and so is `ifeval`: the plain
`screamingface up` Engine lists neither. From a checkout, `just local-stack-notebooks` serves an
Engine with the `inspect` extra beside the stack; otherwise point `SCREAMINGFACE_ENGINE_URL` at such
an Engine and `SCREAMINGFACE_SCOREBOARD_URL` at its Scoreboard (a non-default Engine without a
Scoreboard is refused).

`screamingface up` starts AI Gateway, Scoreboard, and the Engine in the background. Use
`screamingface status`, `screamingface doctor`, `screamingface logs`, `screamingface restart`, and
`screamingface down` to manage them. Runtime state is stored under `~/.screamingface` by default;
set `SCREAMINGFACE_DATA_DIR` or pass `--data-dir` to override it.

Set `URL4_RUNNER_CONFIG` before `up` or `restart` to supply a custom runner TOML as a full
runner-world override (routes, capabilities, timeouts, providers). Relative paths and `~` resolve
at startup; neither this variable nor the paired Gateway variable below is saved in runtime state, so
re-export both on each `up`/`restart`. Route variants (such as `:free`) are declared
with `~` in TOML (e.g. `openrouter/vendor/model~free`) and paired with their decoded `:` ID in
Gateway settings (`AIGW_OPENROUTER_DEFAULT_MODELS`, whose entries must start with `openrouter/`),
which replace rather than append to the model inventory:

```bash
URL4_RUNNER_CONFIG=custom.toml AIGW_OPENROUTER_DEFAULT_MODELS='["openrouter/vendor/model:free"]' screamingface up
```

The default service ports are Gateway `9105`, Scoreboard `9106`, and Engine `9108`. Override them
with `--gateway-port`, `--scoreboard-port`, and `--engine-port`, or the corresponding
`SCREAMINGFACE_GATEWAY_PORT`, `SCREAMINGFACE_SCOREBOARD_PORT`, and `SCREAMINGFACE_ENGINE_PORT`
environment variables. `screamingface up` prints the resolved SDK environment variables; the SDK
does not switch away from its hosted defaults automatically.

Benchmark datasets live in `<data-dir>/benchmark-assets`, where `screamingface prepare` writes
them. To read them from another folder instead — for example a read-only copy bundled with an
app — start the runtime with `screamingface --data-dir <dir> up --benchmark-assets-dir <folder>`.
The folder must exist, or `up` fails before any service starts. The Engine reads datasets from it,
`status --json` and `doctor` report bundle status from it (and name it), and `restart` keeps it.
`prepare` still writes only to the data directory and refuses `--benchmark-assets-dir`.

The local gateway caches model responses by default (`AIGW_REQUEST_CACHE_ENABLED=false` before
`screamingface up` turns it off). To start from the community cache instead of an empty one, pull
it into the local gateway — from a notebook or a shell:

```python
import screamingface as sf

sf.cache.pull_from_remote()  # everything
sf.cache.pull_from_remote(["openrouter/anthropic/claude-opus-5.5"])  # only these models
```

```bash
screamingface cache pull --model openrouter/anthropic/claude-opus-5.5
```

The pull signs in to the hosted Engine (a browser opens once per process), shows its progress, and
keeps rows you already have. Models are named exactly as the hosted cache names them, and a run
can only hit rows of the models its candidates and judge use, so a filtered pull loses no hits.
If it stops, calling it again in the same process continues where it left off; a new process
starts over, re-reading but not re-adding what is already local. One evaluation opts out of the
cache with `evaluate(..., use_cache=False)`.

A pulled row answers only the exact request it was stored for: the same model, prompt and
parameters, the same Benchmark revision, and the same `answer_seed` if the run used one. Change
any of them (even `temperature`) and every call misses and is paid. The rows hold a one-way hash
of the request, not the request, so a pull cannot tell you which configurations it covers. Two
ways to land on one that does:

- Run a standard Benchmark with a bare `sf.Model("…")`: the Benchmark fixes the prompts and
  parameters, so this is the most common configuration.
- Re-run a published leaderboard entry with `sf.evaluate(entry.url4)`. Only runs made on the
  hosted Engine are in the hosted cache, so an entry that was run locally and submitted gives no
  hits; and a leaderboard entry does not carry its `answer_seed`, so a seeded run re-run from its
  `url4` alone misses.

A filtered pull that matches no rows says so (a model name with a typo matches nothing).

For scripts and troubleshooting:

```bash
screamingface status --json
screamingface doctor
screamingface logs --service engine --tail 100 --no-follow
screamingface prepare --list
```

Logs are timestamped, tagged by service, rotated at 10 MiB, and retain five backups. The log and
its backups are readable only by you. Versions before `OME-990` wrote prompts into this log;
`screamingface logs --purge` deletes the backups and empties the live log (it does not securely
erase the disk blocks). Three structural prompt carriers are redacted before anything is
written: url4 `q=` query values, litellm's `Messages:` exception suffix, and litellm's debug curl
`-d` body. The runtime also forces `LITELLM_LOG=WARNING`. Prompt text that appears without one
of those markers is not detected, so review a log before sharing it. Benchmark
preparation records a versioned manifest, skips current assets, and supports `--force` when a
fresh download is required.

## Target v1 workflow

The approved Python workflow is:

```python
import screamingface as sf

sf.connect()  # notebook panel; connect any providers enabled by this Engine

opus = sf.Model("openrouter/anthropic/claude-opus-4.8")
gpt = sf.Model("openrouter/openai/gpt-5.5")

frontier_pair = sf.Fanout([opus, gpt], name="frontier-pair") >> sf.Synthesize(
    "openrouter/anthropic/claude-opus-4.8"
)

report = sf.evaluate(
    [opus, gpt, frontier_pair],
    benchmark="draco",
    limit=1,
)
```

`evaluate(...)` requires an explicit Benchmark id, fetches its Candidate-independent URL4
expression once, compiles and structurally links every Candidate, executes those complete URL4s
concurrently, and returns one immutable `Report` in declared order. There is no implicit default
Benchmark selection. All no-spend validation finishes before the first paid Run starts. Execution
requires a Benchmark Runner image containing the expression's referenced data, grading, and
Aggregation routes.

A complete evaluation URL4 is already linked to its Benchmark and can be evaluated directly. The
Client validates the embedded Candidate projection, executes that exact expression, and recovers
the pinned Benchmark identity from the Engine result:

```python
score = sf.leaderboards.get_score(score_id)
editable_python = score.url4.to_python()
replayed_report = sf.evaluate(score.url4)
```

`Url4.to_python()` is local and no-spend: it returns a `str` of Python source, not live objects —
the `sf.Model`, the `sf.Fanout(...) >> ...` chain, the `sf.Pipeline`, the `sf.CorrectiveLoop`, or
the `sf.SelfCorrective` written back out as editable code, plus the recovered Benchmark call. Raw
URL4 evaluation does not accept `benchmark=`, `limit=` or `cases=` because any of them would imply
recompiling an already-complete expression.
Replay starts a new, potentially paid Run; identical URL4 does not guarantee identical model
output.

Every Client-compiled Candidate URL4 contains exactly one inert `_sf_recipe` source with the
versioned `screamingface.recipe.v1` descriptor. It preserves the public Recipe structure and names
for exact replay reporting and `to_python()` reconstruction. These operations require that
descriptor; the Client does not guess authoring structure from an executable call graph.

The installed `draco` definition always refers to the complete official 100-task Benchmark;
`limit=1` merely runs one Case. Grading uses five independent Judge passes per criterion. The
current executable Judge is `openrouter/google/gemini-3.1-pro-preview`, Google's official
replacement for the paper's retired `Gemini-3-Pro Preview`. Reports should disclose that Judge
version difference when comparing scores with the paper.

Benchmark IDs are flat, complete identities. Bounded development runs use `limit` without changing
the named protocol: `benchmark="draco", limit=1` still uses canonical DRACO's full rubric and five
Judge passes. The non-canonical HealthBench challenge is therefore named independently as
`healthbench-worst30`, rather than presented as a canonical HealthBench variant.

A run can also name a fixed list of Case ids instead of the first N: `sf.evaluate(model,
benchmark="lab_bench_seqqa", cases=[3, 17, 42])`, where a Case id is the Case's position in the
Benchmark, 1 to N. This is for a development set and a held-out set that are scattered rather
than a prefix. The SDK posts the list to the Engine and refuses before any model call unless the
Engine echoes exactly that list back, so an Engine without the selection route never runs and
bills the whole Benchmark. `report.selected_cases` names the Cases scored, ascending, and
`report.selection_fingerprint` is the `sha256:` of the sorted ids, to cite together with the
Benchmark id and revision. A selected run can't be submitted to the leaderboard, `cases=` and
`limit=` can't be combined, at most 1,000 ids are accepted, and a list equal to Cases 1 to N is
the same run as `limit=N`.

### How a recipe reads

A recipe reads from left to right, and `>>` joins its steps. A block starts with `sf.Fanout` and
has four parts:

```text
sf.Fanout(members) >> [sf.Judge] >> [sf.Rank | sf.Filter | sf.Concat]* >> reducer or sf.Synthesize
       map               score                  narrow                        reduce / write
```

- **Map.** `sf.Fanout(members, signals=None, quorum="all", name=None, prompt=None)` sends the input
  to every member at the same time. `signals=` declares the numbers that the run measures from
  each answer (see "Signal sources").
- **Score.** `sf.Judge(model, rubric=...)` is optional. One judge call scores every member, and
  the later steps of the block read the score as `"score:judge"`. The judge does not choose a
  member.
- **Narrow.** `sf.Rank`, `sf.Filter` and `sf.Concat` are optional. `sf.Rank` and `sf.Filter` keep
  some members, and `sf.Concat` joins all the member texts. A narrow step does not give the answer.
- **Reduce or write.** One reducer gives the answer from the member answers: `sf.Vote`,
  `sf.Median`, `sf.Mean`, `sf.ArgMax`, `sf.ArgMin`, `sf.First`, `sf.Merge`, `sf.JudgePick` or
  `sf.Gate`. Or one `sf.Synthesize(model)` writer writes a new answer.

The chain rules:

- A `sf.Fanout` block ends in one reducer or in one `sf.Synthesize`. Nothing else ends it.
- A block has at most one `sf.Judge`, directly after the `sf.Fanout`. A `sf.Judge` needs a later
  step in its block that reads `"score:judge"`, and a step that reads `"score:judge"` needs a
  `sf.Judge` in its block. `sf.Judge` and `sf.JudgePick` use the same judge signal, so a block
  with a `sf.Judge` cannot end in `sf.JudgePick`.
- A narrow step never ends a recipe. After `sf.Rank` or `sf.Filter`, put another narrow step, a
  reducer or `sf.Synthesize`. After `sf.Concat`, put `sf.Synthesize`. A block has at most 4 narrow
  steps.
- `sf.Merge`, `sf.JudgePick` and a `sf.Gate` with a judge or writer fallback read every member
  (`sf.JudgePick` and the fallback in their own model call), so they cannot follow a narrow step.
- More stages can follow a closed block: `>> sf.Model(...)`, a new `sf.Fanout(...)` block, or a
  `sf.Pipeline`. A closed block is a recipe, so it can also be a `sf.Fanout` member or the model of
  a `sf.Synthesize`.
- A step in the wrong place raises `ValueError` at the `>>` that adds it, or at the `>>` that
  closes its block. A chain that ends inside a block is a value that you can extend, but the
  compile, `sf.evaluate(...)` and `Url4` rendering refuse it before any call.
- A named `sf.Pipeline` keeps its stages as one group, so a later `>>` cannot reach into it. It
  cannot end in an open block: close the `sf.Fanout` block inside the named Pipeline.

`sf.Synthesize` reads one of three inputs. The input depends on the step before it:

1. Directly after `sf.Fanout`: JSON with the block input (`input`) and the member answers
   (`outputs`; see "Quorum and failed members").
2. After `sf.Rank` or `sf.Filter`: JSON with the block input and the kept member texts,
   `{"input":"...","selected":[{"id":"member_2","text":"..."}]}` (see "Rank and Concat
   selections").
3. After `sf.Concat`: the joined text only. Put the task in the prompt of the writer.

A narrow step can also come before a reducer. Then the narrow steps and the reducer run in one
combine call, and the reducer reads only the members that the narrow steps kept:

```python
import screamingface as sf

routes = ["provider/alpha", "provider/beta", "provider/gamma"]
reply_json = (
    "Answer the request. Reply with only one JSON object: "
    '{"answer": <a number>, "confidence": <a number from 0 to 1>}.'
)
json_members = [sf.Model(route, prompt=reply_json) for route in routes]
judge = sf.Judge("provider/judge", rubric="Give a higher number to a more correct answer.")

# The mean of the answers of the members with a confidence above 0.7.
confident_mean = (
    sf.Fanout(json_members)
    >> sf.Filter(where=sf.signal("field:confidence") > 0.7)
    >> sf.Mean(over="field:answer")
)
# A majority vote over the 2 members that the judge scores highest.
best_two_vote = sf.Fanout(routes) >> judge >> sf.Rank(2, by="score:judge") >> sf.Vote()
```

A member that a narrow step leaves out has the status `"excluded"` and the reason `not_selected`
(or `no_signal`, when it has no value for the signal that the step reads). The evidence has one
record per narrow step in `CombineEvidence.narrow` (a `sf.report.CombineNarrow`: the select
operator, the signal, the kept members and, for a Rank, the cut, the scores and the ranks). When a
narrow step keeps no member, the Case abstains with `no_votes`. When a Rank finds fewer than `k`
members with a value, the Case abstains with `cut_not_met`. This needs an Engine and an SDK from
the same E4b release.

`sf.Fusion(members, synthesizer=...)` is a shortcut for one block. A route, an `sf.Model` or a
recipe as `synthesizer=` becomes `sf.Synthesize(...)`, and a reducer comes after the `sf.Fanout`
as it is (see "Fusion as a named recipe"). `optional=True` wraps the block as
`sf.Pipeline([...], optional=True)`; it needs a closed block, so `sf.Fusion` refuses
`optional=True` with a narrow step as `synthesizer=`, and it refuses a bare `sf.Judge`, which only
scores. The examples in this README use the `sf.Fanout(...) >> ...` form.

### Candidate policy

Models work without prompt configuration. When a `sf.Fanout` block ends in a writer, name the
writer model explicitly with `sf.Synthesize(model)`; neither the Engine catalogue nor a Benchmark
silently chooses one. The SDK supplies general answer and constraint-aware synthesis prompts, then
embeds the explicit Model routes and resolved prompt defaults in the final URL4:

```python
plain = sf.Model("openrouter/openai/gpt-5.5")
pair = sf.Fanout([opus, gpt]) >> sf.Synthesize("openrouter/anthropic/claude-opus-4.8")
```

Researchers can override only Candidate-owned policy when an experiment needs it:

```python
careful = sf.Model(
    "openrouter/openai/gpt-5.5",
    prompt="Answer from primary evidence and follow every requested output constraint.",
    params={"reasoning_effort": "high"},
)

constraint_aware = sf.Fanout([opus, gpt]) >> sf.Synthesize(
    sf.Model(
        "openrouter/openai/gpt-5.5",
        prompt="Produce one final answer that preserves every constraint in the original request.",
        params={"reasoning_effort": "high"},
    )
)
```

These overrides never alter Benchmark-owned Cases, fixed Judge models or prompts, Grading, or
Aggregation. Prompt defaults and explicit overrides are embedded in each final URL4. Every
`sf.Fanout` block ends in one reducer or one `sf.Synthesize` writer, and it always produces one
answer. Model-call parameters are never
invented by the SDK: a parameter-free Model emits no sampling or retrieval parameters and therefore
uses the Engine's configured defaults. Benchmarks may still impose explicit execution policy in
their own URL4 protocol. Transport, routing, tool, and Benchmark-policy fields remain unavailable
through Candidate `params`.

### Quorum and failed members

A `sf.Fanout` requires every member by default (`quorum="all"`). Mark individual members
with `optional=True` to tolerate their failures:

```python
resilient = sf.Fanout(
    [
        sf.Model("provider/a", optional=True),
        sf.Model("provider/b", optional=True),
        "provider/c",  # required
    ],
    quorum=2,
) >> sf.Synthesize("provider/synth")
```

The compiler wraps optional members with URL4's `;optional` and gates the member-only
expression with `;quorum=2`, before the writer runs. Required members always
have to succeed, even if other successes already meet the quorum. Strings and Recipes
without `optional=True` remain required. `sf.Pipeline([...], optional=True)` guards a complete
composite (a chain or a closed block) when it is a member of another `sf.Fanout`; its
intermediate answers do not count toward the quorum of that `sf.Fanout`. `sf.Fanout` has no
`optional=`, so wrap the block: `sf.Pipeline([sf.Fanout(members) >> sf.Vote()], optional=True)`.
Optional Recipes are only valid as `sf.Fanout` members, not standalone Candidates, pipeline
stages, or writers.

The `sf.Synthesize` writer directly after the `sf.Fanout` receives JSON containing the original
`input` and an `outputs` string with ordered `member_N: answer` sections for successful members
only. Blocks with the default quorum and only required members retain their existing outputs
object.

Quorum is a success floor checked after members finish, not an early response race.
`quorum="all"` still requires every member to succeed, including optional members;
`quorum=0` permits synthesis with no successful members when every member is optional.
`None`, booleans, negative numbers, and numbers above the member count are rejected.
Quorum and per-member optional settings survive `Url4.to_python()` and replay.
A block that ends in a reducer (`sf.Vote`, `sf.JudgePick` and the other reduce steps) takes
quorum and optional members too. It reads only the members that answered, and the evidence lists
each failed optional member with the reason `missing`; a judge (`sf.Judge` or `sf.JudgePick`)
sees only the members that answered. `sf.Concat` joins only the members that answered, so set
`label=True` when the writer must know which member wrote each part. With `quorum=0` and no member
answering, the reducer abstains and the Case scores 0.0.

**Deployment requirement:** before using composed quorum Recipes (such as a Pipeline
feeding a quorum block or a quorum block used as a writer), deploy an Engine
containing the URL4 nested-expression scope fix shipped with this change. Older runtimes
can pass literal binding references instead of resolved upstream answers. Deploy the
Engine operation-reporting update as well to retain member outputs and accounting.
URL4 already supports quorum and optional sources; the runtime change fixes scope
propagation for these compositions.

### Independent samples of one model

`sf.samples(model, n)` builds `n` Models of one route that differ only in an integer `seed`, so a
vote over them counts `n` real answers and not one cached answer `n` times:

```python
candidate = sf.Fanout(sf.samples("openrouter/openai/gpt-5.5", 5, temperature=0.7)) >> sf.Vote(
    over="last_number", normalize="numeric"
)
```

Seeds default to `0` through `n - 1`; pass `seeds=[...]` (unique ints) to choose them. `prompts=[...]`
gives member `i` its own prompt (for example one per language), `temperature=` and `params=` apply
to every member, and a `sf.Model` as the first argument is the base whose prompt and params the
samples inherit. A model whose provider does not accept `seed` fails with `sf.PlanningError` (code
`unsupported_model_parameter`) before the run starts and before any model is called. The SDK emits
`sf.DuplicateMembersWarning` when a block that ends in a reducer has two members that send the
same request, because the gateway may serve them from one cached answer. The warning fires for
every reducer step.

A call that sets its own `seed` keeps it, and a run seed (`answer_seed=`, one "sitting" of an
evaluation) only applies to calls without one. So `sf.samples()` members, and any
`sf.Model(params={"seed": ...})`, send the same requests in every sitting, and a mean and
confidence interval across sittings would look tighter than the model really is. When a run seed is
set and a Recipe or a replayed URL4 has such calls, the SDK emits `sf.FixedSeedSittingWarning`
before the run starts, naming the calls, and then runs as usual. If you already pass different `seeds=` to `sf.samples()` in
each sitting, the warning is expected, and you can silence it:

```python
import warnings

warnings.filterwarnings("ignore", category=sf.FixedSeedSittingWarning)
```

Otherwise drop the explicit seed. The check covers Recipes and raw URL4 replays
(`client.evaluate("<url4>", answer_seed=...)`), with the same text.

### Serial and recursive composition

Every complete `Recipe` accepts one input and returns one final answer. `Model` is atomic,
a `sf.Fanout` block runs members in parallel and passes their answers to a reducer or a writer,
and `Pipeline` passes one answer through ordered serial stages:

```python
draft = sf.Model("openrouter/openai/gpt-5.5")
review = sf.Model(
    "openrouter/anthropic/claude-opus-4.8",
    prompt="Review the previous answer and return a corrected answer.",
)
final = sf.Model(
    "openrouter/openai/gpt-5.5",
    prompt="Polish the previous answer without adding unsupported claims.",
)

review_chain = sf.Pipeline([draft, review, final], name="review-chain")
same_stages = draft >> review >> final  # unnamed: gpt-5.5->claude-opus-4.8->gpt-5.5
```

The first Pipeline stage receives the Candidate input; each later stage receives only the previous
stage's answer. No original input or accumulated history is injected implicitly. The one exception
is a `sf.Synthesize` writer after a `sf.Rank` or a `sf.Filter`: its selection JSON also holds the
input of that block (see "Rank and Concat selections"). Pipelines and closed blocks compose
recursively, also as the model of a writer:

```python
judge = sf.Model("openrouter/anthropic/claude-opus-4.8")
writer = sf.Model("openrouter/openai/gpt-5.5")

candidate = sf.Fanout(
    [
        review_chain,
        sf.Model("openrouter/google/gemini-3.1-pro-preview"),
    ]
) >> sf.Synthesize(sf.Pipeline([judge, writer]))
```

`Pipeline([...])` is the canonical serial representation; `a >> b` is immutable shorthand for
`sf.Pipeline([a, b])` and never executes work. An unnamed nested Pipeline is flattened, and a
named one keeps its grouping. Every Recipe-valued position also accepts a model-route string as
shorthand for `sf.Model(...)`. Shape mismatches, cycles, invalid models, and invalid parameters
fail before spend.

Each flat Benchmark resource uses `screamingface.benchmark.v1` and carries one canonical `url4`
plus an opaque immutable `revision`. The SDK fetches the requested id, compiles a Recipe
into an expression accepting `$input`, and links it through the single universal `$candidate`
binding. A Benchmark invokes that bound expression through `/candidate`; that route evaluates it
inside the same Engine job, not through an additional Client or control-plane request. Old
structural member/synthesizer bindings fail with a typed planning error rather than invoking a
second Client compilation path.

The Candidate input is normally plain text. Engine-owned Benchmarks that require native chat
history wrap structured turns in the versioned Candidate-input envelope; the Runner preserves
their roles while the SDK continues to treat `$input` as opaque. This supports multi-turn and
stateful protocols without a Client-interpreted workflow language.

### Deterministic synthesizers

A block can end in a rule (a reducer) instead of a model:

```python
routes = [
    "openrouter/openai/gpt-5.5",
    "openrouter/anthropic/claude-opus-4.8",
    "openrouter/google/gemini-3.1-pro-preview",
]

by_vote = sf.Fanout(routes) >> sf.Vote(over="last_number", normalize="numeric")
by_median = sf.Fanout(routes) >> sf.Median(over="last_number", select="value")
```

`over=` names the rule that reads each answer, for example `"last_line"`, `"last_number"`,
`"boxed"`, `"whole"`, `"json"`, `"benchmark"`, `"field:<path>"` (a field of a JSON reply), or
`sf.Extract(rule, ...)` for a rule with arguments.

The combine step makes no model call. The answer is one member's text byte for byte, or the median
number itself for `select="value"`. A tie goes to the first member in author order and sets `tie`.
A member with no usable value does not vote. When no member votes, the Case abstains: its output is
`""`, its score is 0.0 and `case.abstained` is true. It does not fail.
On HealthBench a harmful answer can score below 0, so an abstain can rank above a bad answer; this
is intended. Only the final step of a Candidate can mark a Case as abstained: a combine step that
feeds another stage (a Rank or a Filter before `sf.Synthesize`) never does.

**Weighted median.** `sf.Median(weights=(1, 1, 3))` takes one positive number per member, or
`sf.signal(key)`, as `sf.Vote` does. The members with a usable weight are sorted by value. The
answer is the first value where the running weight reaches half the total weight. At exactly
half, the answer is the mean of that value and the next one, so equal weights give the plain
median. A member with a missing or non-positive weight is dropped. It is never the nearest pick.
The evidence shows that member with the status `"excluded"` and the reason `no_weight`.
When no member is left, the Case abstains. For a number in a JSON reply, read the field with
`over=`: `sf.Median(over="field:answer", weights="field:confidence", select="value")` (see
"Combine recipes"). `select="value"` returns the median number, and the default
`select="nearest"` returns the text of the nearest member. A `sf.signal("score:judge")` weight
raises `ValueError`: a Median does not read a judge's scores as weights; use
`sf.Vote(weights="score:judge")` or `sf.ArgMax("score:judge")`.

`sf.Mean` returns the mean of the members' numbers (`select="value"`, the default) or the text of
the member nearest the mean (`select="nearest"`). Give it weights to make some members count more:

```python
by_mean = sf.Fanout(routes) >> sf.Mean(weights=[1, 1, 2])
```

The sums are exact, and the division is rounded to 34 significant digits. A member with no usable
weight is dropped: it is not in the mean and it is never the nearest member. For a number in a
JSON reply, use `sf.Mean(over="field:answer", weights="field:confidence")` (see "Combine
recipes"). `select="nearest"` returns the text of the member nearest the mean.
The evidence shows a member with no usable weight with the status `"excluded"` and the reason
`no_weight`.

To group answers the way the benchmark grades them, let the benchmark choose the extract rule. The
SDK reads the rule from the benchmark before any spend:

```python
by_grader = sf.Fanout(routes) >> sf.Vote(over="benchmark")
report = client.evaluate(by_grader, benchmark="inspect-gsm8k")
```

A benchmark with no answer rule raises `ValueError` before any spend. For a rule with arguments, use
`sf.Vote(over=sf.Extract("regex", pattern=...))`.

**Regex patterns.** `sf.Extract("regex", pattern=...)` and `sf.matches(pattern)` run on RE2 in the
Engine. RE2 has no lookarounds and no backreferences. `\d` and `\w` match ASCII only: use `\p{L}` and
`\p{N}` for Unicode letters and digits. `$` is the end of the text, so write `\s*$` to allow a
trailing newline. The repeat limit is 1000. The Engine refuses a compiled program over 5000
instructions. The SDK checks the pattern before any spend, and it is a little stricter than RE2 for
`\Q...\E`, `\C`, `\x{...}` and POSIX `[[:class:]]`. The Engine compile is authoritative: a pattern
that the SDK accepts and RE2 refuses fails the Case with `combine_spec_invalid`.

The evidence (the Answer row per member, the tally, the result, the agreement and the chosen member)
is in the `combine` field of each case's `operations` entry (`op_synthesis_1` for a single reducer).
`case.answers()` joins each member's text to its Answer. `case.trace()` and `report.trace_dict()`
show what each block did, in order. `report.trace_dict()` is `screamingface.report-trace.v1`.
`report.to_dict()` and `report.export()` stay `report.v1`, and each Case with a combine step also
has a `trace` key with the same entries (see "Combine steps in the report"). This needs an Engine
that serves `GET /v1/operators`.

Give a vote weights to make some members count more. A list gives each member a fixed weight, one
positive number per member in author order. A `sf.signal` weighs each member by a number on its own
Answer:

```python
fixed = sf.Fanout(routes) >> sf.Vote(weights=[2, 1, 1])

# Each member replies with JSON such as {"answer": "B", "confidence": 0.4}.
by_confidence = sf.Fanout(routes) >> sf.Vote(over="field:answer", weights="field:confidence")

# The cost of each member's own model calls as the weight.
by_cost = sf.Fanout(routes) >> sf.Vote(weights="cost_usd")
```

A weight key is a string, or the same key as `sf.signal(key)`: `weights="cost_usd"` and
`weights=sf.signal("cost_usd")` are one rule.

Each answer scores the sum of the weights of the members who gave it. The answer is the text of the
heaviest member who gave the winning answer (the first in author order on equal weights). A member
with no weight does not vote, and when no member is left the Case abstains. The report shows each
member's `weight` and each answer's summed `score`.
A member whose signal weight is not usable has the status `"excluded"` and the reason `no_weight`
in the evidence.

A weight that is zero, negative, not finite or above `sys.float_info.max / 256` raises `ValueError`
at `sf.Vote(...)`, and a weight that is not a number raises `TypeError`. A wrong number of weights
raises `ValueError` at the `>>` that closes the block. All of these raise before any call. Weights
apply to `sf.Vote`, `sf.Mean` and `sf.Median` (see "Weighted median" above).
`weights="score:judge"` needs a `sf.Judge` step in the same block (see "Judge synthesizer"
below).

The accounting keys are `cost_usd`, `latency_ms`, `tokens_in` and `tokens_out`. For a member they
are the sum over the model calls that member made, which the Engine finds by each call's route and
params. When two members send the same route and params (for example a repeated model, or a model
that a later stage or another block also calls), the Engine cannot tell the calls apart, so those
members have no accounting value and do not vote. This also holds in a corrective loop with
`max_rounds` of 2 or more, where every round sends the member's calls again. To give each member
its own calls, use `sf.samples()` (a different `seed` for each member).

A cached replay reads each call's original cost, latency and tokens, so it gives the same signals
and the same tally. A fresh live rerun can give a different `latency_ms` (and a different cost, if
prices changed). Only `cost_usd` is proven end to end by the paid replay lane. `latency_ms`,
`tokens_in` and `tokens_out` are carried by the cache metadata and covered by unit tests only.

**Members left out.** A step that reads a signal or a signal weight leaves out a member that has
no value for it. The evidence row of that member has `status: "excluded"` and the reason
`no_signal` (no value for the signal that the step reads: `sf.Rank(k, by=...)`,
`sf.ArgMax(...)`, `sf.ArgMin(...)` or `sf.Filter(where=sf.signal(...) ...)`) or `no_weight` (no
usable signal weight in a weighted `sf.Vote`, `sf.Median` or `sf.Mean`: missing, zero, negative or
too large). `case.answers()` gives that member `sf.Answer(status="excluded", reason="no_signal")` (or
`reason="no_weight"`), with the value it read, and the trace `extract` entry shows the same status
and reason. The row is written also when the step abstains because no member is left. A member
that failed keeps `status: "fail"` and its own reason, also when its signal is missing. Fixed
weights (a list) never leave a member out. An excluded member is not a failure: `dropped` and the
quorum count only failed members. The answer of the Case does not change. Only the evidence says
more. So `sf.Answer.status` can now also be `"excluded"`. This needs an Engine and an SDK from the
same release: an older SDK cannot read evidence with an excluded member.

### Judge synthesizer

A judge is a model that reads the members. There are two judge steps, and neither one writes the
answer:

- `sf.Judge(model, rubric=..., params=None)` scores the members. It is the step after `sf.Fanout`,
  and a later step of the block reads the scores as `"score:judge"`: `sf.ArgMax`, `sf.ArgMin`,
  `sf.Rank`, `sf.Filter` or a weighted `sf.Vote`.
- `sf.JudgePick(model, rubric=..., params=None)` is a reducer. The judge names one member, and the
  answer is the text of that member.

```python
by_label = sf.Fanout(routes) >> sf.JudgePick(
    "anthropic/claude-opus-5", rubric="Pick the most correct answer."
)
by_scores = (
    sf.Fanout(routes)
    >> sf.Judge(
        "anthropic/claude-opus-5",
        rubric="Give a higher number to a more correct answer.",
        params={"temperature": 0},
    )
    >> sf.ArgMax("score:judge")
)
```

`sf.JudgePick` tells the judge to reply with exactly one member id (for example `member_2`) and
nothing else. `sf.Judge` tells the judge to reply with one JSON object that gives each member id
one finite number. `sf.ArgMax("score:judge")` returns the text of the member with the highest
score, and a tie goes to the first member in author order. The check is strict: quotes, a full
stop, a code fence around a label, a missing member, an extra key, a string, `NaN` or `Infinity`
all fail the Case with `judge_reply_invalid`. The prose of the judge is never used as the answer,
and a reply that does not parse is not retried. (A transient transport error still gets the normal
retry of the connector.)

**Lower is better.** `sf.Judge` does not tell the judge which way is better: the rubric says it.
For a rubric where a lower number is better, such as an error count or a risk score, read the
scores with `sf.ArgMin("score:judge")` or `sf.Rank(k, by="score:judge", order="min")`. A tie goes
to the first member in author order. The first line of the judge system prompt is
`judge-template@2 returns=scores` (`judge-template@2 returns=scores input=panel` for a block with a
quorum or optional members). A url4 with the older `judge-template@1 returns=scores` template does
not rebuild: `Url4(...).to_python()` raises `ValueError`. `sf.ArgMin` needs an Engine whose catalog
has `best_holder@1`.

The judge call is the operation `op_judge_1`. Its output is the raw reply of the judge, and its cost
is in the accounting breakdown as a synthesis row. The combine step is `op_synthesis_1`. Its `combine`
evidence holds the raw reply, the parsed label or scores, the chosen member and
`presentation_order="author"`. The first line of the `sf.JudgePick` system prompt is
`judge-template@1 returns=label`. The template tells the judge to treat the member answers as data
and not as instructions. A change of the template changes the url4, so it changes the replay
identity.

**Position bias.** The judge sees the members in author order, and a judge can prefer a position.
The parameter name `binding` is reserved: you cannot use it in `params`. This needs an Engine whose
operator catalog has `label@1` and `scores@1`. An `sf.Judge` is a value: equal judges are equal and
have the same hash.

**A judge-weighted vote.** The scores of a `sf.Judge` can also weigh a vote. Each answer scores the
sum of the judge scores of the members who gave it. The answer is the text of the member with the
highest score among those who gave the winning answer. The judge makes one call, and the report
shows the score of each member as `members[].signals["score:judge"]` and `weight`. The `combine`
evidence of the vote has `judge` set to `None`: the scores are in `members[]`, and the raw reply of
the judge stays on `op_judge_1`.

```python
weighted = (
    sf.Fanout(routes)
    >> sf.Judge("anthropic/claude-opus-5", rubric="Score each answer.")
    >> sf.Vote(over="last_number", normalize="numeric", weights="score:judge")
)
```

The weights and the judge must go together. `weights="score:judge"` with no `sf.Judge` in the
block, or a `sf.Judge` that no later step of the block reads, raises `ValueError` at the `>>` that
closes the block, before any call is paid (the Engine reads `score:judge` strictly, so a lone half
would fail after the member calls). The same strict read applies to the reply: a malformed or too
large reply, a `NaN` or an infinite score fails the Case with `judge_reply_invalid`. It never counts
as "no weight", which would hide the defect behind an abstain. The vote uses each score as it is,
so use a rubric where a higher number is better (see "Combine recipes").

**The judge in the trace.** In `case.trace()` and `report.trace_dict()`, the `reduce` entry of a
step that reads a judge names the judge call: `judge_operation_id` is the operation of the judge
(for example `op_judge_1`), and `judge_model` is the `model` of that operation's `model_call`
entry. This holds for `sf.JudgePick`, for every step that reads `"score:judge"` (`sf.ArgMax`,
`sf.ArgMin`, `sf.Rank`, `sf.Filter`, a weighted `sf.Vote`), and for a gate with a judge fallback.
For any other step, both keys are `None`. The SDK reads the link from the operation graph of the
Case and never guesses: both keys are `None` for a Case result with no operation graph, and for a
step with no judge operation or with more than one. `judge_model` is `None` when the judge call has
no accounting. The link does not change the url4. The `combine` evidence itself has no judge model.

**The judge and the writer.** The model that scores the members should not also write the
answer. The SDK emits `sf.JudgeWriterWarning` (an `sf.EvaluationWarning`) when it compiles the
recipe, before any call, in two cases: a stage that reads a judge is followed directly by a model
stage (an `sf.Model` or a route string) with the same model as the judge, or a block has a
`sf.Judge` and a `sf.Synthesize` writer of the same model (`sf.Fanout(m) >> sf.Judge(j, ...) >>
sf.Rank(...) >> sf.Synthesize(j)`). The warning names the Pipeline (when you named it), the stage
positions and the route. It does not stop the run and does not change the url4. Two spellings of
one route (with and without the provider prefix that the SDK adds) are one model, as for
`sf.DuplicateMembersWarning`. The SDK sends the warning once per Pipeline object, and not again
when it rebuilds a url4. Only a writer that is one `sf.Model` (or a route) is checked: a writer
that is a block or a nested Pipeline is not. If one model in both roles is what you want, silence
the warning:

```python
import warnings

warnings.filterwarnings("ignore", category=sf.JudgeWriterWarning)
```

### Pick and merge JSON fields

If the members reply with JSON, a rule can choose or build the answer. No model writes it.

```python
by_confidence = sf.Fanout(routes) >> sf.ArgMax("field:confidence")
by_answer = sf.Fanout(routes) >> sf.First(where=sf.field("answer", normalize="text") == "yes")
# the answer is only the "answer" field of the most confident member's JSON reply
answer_only = sf.Fanout(routes) >> sf.ArgMax("field:confidence", output="answer")

# the member with the lowest value of a named signal (here, the cheapest member)
cheapest = sf.Fanout(routes) >> sf.ArgMin("cost_usd")

merged = sf.Fanout(routes) >> sf.Merge(
    fields={
        "answer": sf.FieldRule.majority(normalize="text"),
        "score": sf.FieldRule.max(),
        "source": sf.FieldRule.from_member(2),
        "synonyms": sf.FieldRule.union(normalize="text"),
        "summary": sf.FieldRule.model("anthropic/claude-sonnet-5", prompt="Write one summary."),
    }
)
```

`sf.ArgMax(by)` and `sf.ArgMin(by)` return the text of the member with the highest or the lowest
value of the signal `by`, byte for byte. `by` is a signal key: `field:<path>` (a number in the JSON
reply, with a path such as `confidence`, `a.b` or `a[0].b`), an accounting key (`cost_usd`,
`latency_ms`, `tokens_in`, `tokens_out`), `score:judge` after a `sf.Judge`, or a name that the
`sf.Fanout` declares in `signals=`. A field path that is longer than 256 characters raises
`ValueError` before any call. A member with no value for the signal is left out: a field that is
missing or is not a number has no value. When no member has a value, the Case abstains. A tie goes
to the first member in author order and sets `tie`. For an accounting key, the SDK adds `sources`
to the combine context (the model calls of each member), as it does for a weighted `sf.Vote`; a
member with a call that appears twice in the Candidate gets no value. In the evidence, a member
with no value for the signal has the status `"excluded"` and the reason `no_signal`.

`sf.First(where=sf.field(path, normalize=None) == value)` returns the first member, in author
order, whose JSON field is equal to `value`. With `normalize="text"` or `normalize="numeric"`, the
value of each member and `value` itself go through the same normalizer before the compare. A
number or a boolean in `value` is compared in its JSON spelling (`1` is not `1.0`), unless you use
`normalize="numeric"`. Because `json_field@1` returns the contents of a string without its quotes,
`== 7` also matches a member whose value is the string `"7"`. An empty `value`, or `== True` with
`normalize="numeric"`, raises `ValueError` before any call. `sf.field` compares with `==` only; for
a number, use `sf.signal("field:<path>")` in a `sf.Filter`, or `sf.ArgMax`.

`output="answer"` (on `sf.ArgMax`, `sf.ArgMin` or `sf.First`) makes the answer one top-level field
of the winner's JSON reply, not its whole text. A string is returned without its quotes, and any
other value as compact JSON (`42`, `true`, `{"a":1}`). When the winner's field is missing, `null`
or blank, or its reply is not a JSON object, the Case abstains with the reason `output_missing`.
The next member is never used. `output` is a field name (a letter or `_`, then letters, digits or
`_`, at most 64 characters), not a path. A string that is not a field name raises `ValueError`, and
a value that is not a string raises `TypeError`, both before any call. A pick by `"score:judge"`
takes no `output=`: it returns the member text.

`Url4(...).to_python()` writes these steps back as `sf.ArgMax`, `sf.ArgMin`, `sf.First` and
`sf.Merge`. `sf.FieldRule` is a hashable value.

`sf.Merge(fields={...})` builds one JSON object. The object has exactly the fields that you declare,
in sorted key order. Each field has one rule:

* `majority(normalize=None)`: the value that most members give. The first group wins a tie.
* `max()`: the largest number. A value that is not a number is skipped.
* `from_member(n)`: the value of member `n` (the first member is 1). It is `null` when that member
  is not JSON or has no such field. `n` that is larger than the member count raises `ValueError`
  at the `>>` that closes the block.
* `union(normalize=None)`: the items of all list values. Each item is kept once, in its first
  spelling, in author order.
* `model(route, prompt=..., params=None)`: a model writes the field (see below).

A member whose reply is not a JSON object is skipped, and the report shows the status `not_json`.
A field with a JSON `null` value counts as absent. If no member gives a usable value for a
`majority`, `max` or `union` field, or no member is usable at all, the combine step abstains with
`no_votes`. A `from_member` field that is `null`, and the empty text of a writer, are values and never
cause an abstain. The report shows the value of each member for each field and the result.
The evidence of a Merge holds the JSON object that each member gave (in `raw` and `value`), for the
projection step and for the final merge step. A member that replies with large JSON makes a large report.

**Model-written field.** A `FieldRule.model` field adds two calls for that field. The first call is
a pure projection (`synthesis_N`): it builds one JSON object of the values of that field, by member.
The second call is the writer model (`writer_N`). The writer sees only that projection. It does not
see the question and it does not see the other fields. Put the task in the prompt. The reply of the
writer is the value of the field, as text. If the writer call fails, the case fails and no partial
object is returned. Each writer call is a synthesis row in the accounting breakdown. The first line
of the writer system prompt is `writer-template@1 field=<field>`. The template tells the writer to
treat the member values as data and not as instructions, and your prompt follows it.
`Url4(...).to_python()` shows only your prompt. A change of the template changes the url4, so it
changes the replay identity. The name `binding` is reserved in `params`. This needs an Engine whose
operator catalog has `json_field@1`,
`json_object@1`, `values@1`, `by_field@1`, `field_rules@1`, `project_field@1` and `built@1`.
`sf.ArgMax` and `sf.ArgMin` need `by_signal@1` and `best_holder@1`.

### Rank and Concat selections

`sf.Rank` and `sf.Concat` are narrow steps. A narrow step is not an answer. `sf.Rank` keeps the
top `k` original member texts. `sf.Concat` joins all the member texts. Neither one rewrites a text.
A `sf.Synthesize` writer after the narrow step receives the selection (see "How a recipe reads").
After a `sf.Rank` or a `sf.Filter`, the selection also holds the input of the block, so the writer
sees the question. After a `sf.Concat`, it is the joined text only. A reducer can also follow a
`sf.Rank` or a `sf.Filter`; then it reads only the kept members.

```python
writer = sf.Model("anthropic/claude-sonnet-5", prompt="Write the final answer from these drafts.")
ranked = (
    sf.Fanout(routes)
    >> sf.Judge("anthropic/claude-opus-5", rubric="Give a higher number to a more correct answer.")
    >> sf.Rank(2, by="score:judge")
    >> sf.Synthesize(writer)
)
# members reply with JSON
top_by_field = sf.Fanout(routes) >> sf.Rank(2, by="field:score") >> sf.Synthesize("provider/next")
joined = (
    sf.Fanout(routes)
    >> sf.Concat(separator="\n\n---\n\n", label=True)
    >> sf.Synthesize("provider/next")
)
```

`sf.Rank(k, by=..., order="max")` keeps the `k` members with the highest value of the signal `by`.
With `order="min"` it keeps the members with the lowest values, for example the cheapest members.
`by` is a signal key: `score:judge` after a `sf.Judge` (one judge call scores every member),
`field:<path>` (a number in the JSON reply, with a path like `score` or `result.score[0]`), an
accounting key such as `cost_usd`, or a declared name. A member with no value for the signal is not
ranked. In the evidence, a member with no value for the signal has the status `"excluded"` and the
reason `no_signal`.

The writer receives this JSON, best member first:
`{"input":"...","selected":[{"id":"member_2","text":"..."},{"id":"member_4","text":"..."}]}`.
`input` is the input of this block, character for character: the Case input when the block is the
first stage, or the answer of the previous stage when it is not. A conversation input arrives as its
JSON text. The input is never cut: a selection of more than 8 MiB fails with
`combine_input_too_large`. A tie at the cut goes to the earlier member, and the evidence sets
`tie`. `k` is from 1 to 256 and must not be more than the number of members. If fewer than `k`
members have a usable score, the combine step abstains with `cut_not_met`, and the writer gets the
input and no member, `{"input":"...","selected":[]}`. The judge call and the scores of all the
members stay in the report (`op_judge_1` and the `combine` evidence), also for the members that do
not continue. A judge reply that is not valid fails the Case with `judge_reply_invalid`, as for
`sf.ArgMax("score:judge")`.

`sf.Concat(separator="\n\n---\n\n", label=True)` makes no model call. With `label=True`, each part
starts with `[member_1]` on its own line. The parts are in author order, and an empty member text
keeps its place. The separator is 1 to 1024 characters. If a member text contains the separator,
the text is joined unchanged and the evidence sets `separator_collision`, because then the
boundaries are not unique. A joined text of more than 8 MiB fails with `combine_input_too_large`.

```python
cheapest_two = (
    sf.Fanout(routes)
    >> sf.Rank(2, by="cost_usd", order="min")
    >> sf.Synthesize(
        sf.Model("anthropic/claude-sonnet-5", prompt="Write the final answer from these drafts.")
    )
)
```

**Rank and cut in the evidence.** The `combine` evidence of a Rank gives each member with a score
its `rank` (`CombineAnswerRow.rank`): 1 is the best, and equal scores take author order, so no two
members share a rank. A member with no score (it failed or was excluded) has no rank. The evidence
also has the `cut` (`CombineEvidence.cut`, a `sf.report.CombineCut`) with `k` and `order`. The
default order is written as `"max"`. The chosen members are exactly the members with a `rank` of
`cut.k` or less, best first. For `cheapest_two` with four members and the costs member_1 0.03,
member_2 0.01, member_3 0.02 and member_4 0.01, the ranks are member_2 1, member_4 2, member_3 3
and member_1 4, the chosen members are member_2 and member_4, and the cut is
`{"k": 2, "order": "min"}`. When the step abstains with `cut_not_met`, the ranks and the cut are
still written. Only a Rank has a rank and a cut. A Filter, an ArgMax and an ArgMin have neither,
and their evidence keeps its keys. A Rank before a reducer writes its ranks, its scores and its cut
in its `narrow` record instead (see "How a recipe reads").

The chain checks the place of a narrow step when you build it (see "How a recipe reads"). A
narrow step never ends a recipe, a `sf.Concat` must be followed by `sf.Synthesize`, and a selection
goes only to `sf.Synthesize`, a reducer or another narrow step. Each case raises `ValueError`
before any call. After a Rank or a Filter, the writer receives the input of the block and the kept
member texts, never the texts that were cut. After a Concat, it receives only the joined text, so
put the task in the prompt of the writer. This needs an Engine whose operator catalog has
`scores@1` and `top_k@1` for `Rank` by a judge, `by_signal@1` and `top_k@1` for `Rank` by a field
or another signal, or `join@1` and `all@1` for `Concat`.

### Filter selection

`sf.Filter` keeps the members that pass a test. A `sf.Synthesize` writer after it gets the input of
the block and only those members, as the same JSON that `sf.Rank` writes
(`{"input":"...","selected":[...]}`), in author order. `where=` is a regex or a signal comparison:

```python
confident = (
    sf.Fanout(routes)  # Each member replies with JSON such as {"answer": "B", "confidence": 0.4}.
    >> sf.Filter(where=sf.signal("field:confidence") >= 0.6)
    >> sf.Synthesize(writer)
)
formatted = sf.Filter(where=sf.matches(r"ANSWER: \d+", ignore_case=True))
checked = sf.Filter(where=sf.matches("yes"), over="field:verdict")
```

`sf.signal(key)` compares with `>`, `>=`, `<`, `<=` and `==`. The number must be finite. There is
no `!=`, and no `and` or `or`: a Filter has one test. `sf.signal(key) == sf.signal(key)` stays a
plain `bool`, and a comparison is not a `bool` (`if sf.signal("cost_usd") < 1:` raises
`TypeError`). A threshold is stored as a float, so `1` and `1.0` are one test.
`sf.signal("field:x") == True` also passes a member whose number is `1.0`, because a signal holds a
float. A member with no value for the signal, or that failed to answer, is dropped.
An accounting key (`cost_usd`, `latency_ms`, `tokens_in`, `tokens_out`) reads the model calls of
each member, and `sf.signal("score:judge")` needs a `sf.Judge` before the Filter in the same
block. Without `over=`, a regex reads the original text of the member. `over=` (only with
`sf.matches`) reads each answer first, and the regex reads that answer: a member whose answer
cannot be read is dropped, and the report shows the value it read. `over=` with a signal
comparison raises `ValueError`.

A Filter can keep the members that a judge scores high enough. One judge call scores every
member, and the Filter keeps each member whose score passes the test:

```python
judged = (
    sf.Fanout(routes)
    >> sf.Judge("openai/gpt-5.5", rubric="Score each draft from 0 to 10.")
    >> sf.Filter(where=sf.signal("score:judge") >= 7)
    >> sf.Synthesize(writer)
)
```

The rubric says which way is better, so write the test to match it: `>=` when a higher number is
better, `<=` when a lower number is better. A judge reply that is not one finite score for each
member fails the Case with `judge_reply_invalid`.

A Filter that keeps no member abstains: its `combine` evidence has the result `abstain` with the
reason `no_votes`. A writer after it still runs, and it gets the input and no member:
`{"input":"...","selected":[]}`. So a writer can say that no answer passed. (The Case is not marked
`abstained`, because the writer is the final step.) When a reducer follows the Filter, the Case
abstains with `no_votes` (see "How a recipe reads"). The report shows a pass mark in
`members[].score` of the `combine` evidence: `1.0` when the member is kept, `0.0` when it is
dropped, and `None` when the member failed to answer. The place rules of "Rank and Concat
selections" apply to a Filter too. This needs an Engine whose operator catalog has `filter@1` and
`kept@1`.

A member with no value for the signal of `where=` keeps the mark `0.0`. In the evidence, it has
the status `"excluded"` and the reason `no_signal`.

### Combine recipes

Each row is one combine scenario with a recipe that you can copy. Each recipe runs offline in the
Engine's tests, from the url4 that the SDK compiles. A recipe is a chain: `sf.Fanout(members)` sends
the input to every member, an optional `sf.Judge` scores the answers, optional narrow steps
(`sf.Rank`, `sf.Filter`, `sf.Concat`) keep some of them, and one reducer or one `sf.Synthesize`
writer gives the answer. `>>` joins the steps. A scenario that needs two values from each member
(an answer and a confidence) asks each member for one JSON reply. Ask every member for the same
confidence scale, for example 0 to 1. The Engine cannot see that one member writes 80 and another
member writes 0.8.

| S | Scenario | Recipe |
|---|---|---|
| S1 | Read a number from each answer and return the median number | `s1_median_number` |
| S2 | Read a number from each answer and return the answer nearest the median | `s2_median_member` |
| S3 | Read a number from each answer and return the mean, with fixed weights | `s3_mean_static` |
| S4 | Read the answer and the confidence of each JSON reply and return the mean weighted by confidence | `s4_mean_by_confidence` |
| S4, median form | Read the answer and the confidence of each JSON reply and return the median weighted by confidence | `s4_median_by_confidence` |
| S5 | Return the answer with the highest confidence | `s5_max_confidence` or `sf.ArgMax("field:confidence", output="answer")` |
| S7 | Keep the 2 answers with the highest confidence, then a writer combines them | `s7_top_confidence` |
| S8 | Return the answer with the highest confidence, read as a signal | `s8_argmax_confidence` |
| S9 | Keep the answers with a confidence above 0.7, then a writer combines them | `s9_confident_only` |
| S10 | A judge scores each answer, keep the best 2, then a writer combines them | `s10_judge_top` |
| S10, threshold form | A judge scores each answer, keep the answers with a score of 7 or more (see "Filter selection") | `sf.Fanout(m) >> sf.Judge(j, rubric=R) >> sf.Filter(where=sf.signal("score:judge") >= 7) >> sf.Synthesize(w)` |
| S11 | A judge scores each answer and returns the best answer | `s11_judge_best` |
| S11, min form | A judge scores each answer with a lower number as better, and returns the answer with the lowest score | `sf.Fanout(m) >> sf.Judge(j, rubric=R) >> sf.ArgMin("score:judge")` |

```python
import screamingface as sf

routes = ["provider/alpha", "provider/beta", "provider/gamma"]
# Ask every member for the same JSON reply and the same confidence scale (0 to 1).
reply_json = (
    "Answer the request. Reply with only one JSON object: "
    '{"answer": "<your answer>", "confidence": <a number from 0 to 1>}.'
)
json_members = [sf.Model(route, prompt=reply_json) for route in routes]
writer = sf.Model(
    "provider/writer", prompt="Write the final answer to the input from the selected answers."
)
judge = sf.Judge("provider/judge", rubric="Give a higher number to a more correct answer.")

s1_median_number = sf.Fanout(routes) >> sf.Median(over="last_number", select="value")
s2_median_member = sf.Fanout(routes) >> sf.Median(over="last_number")
s3_mean_static = sf.Fanout(routes) >> sf.Mean(weights=(1, 1, 2))
s4_mean_by_confidence = sf.Fanout(json_members) >> sf.Mean(
    over="field:answer", weights="field:confidence"
)
s4_median_by_confidence = sf.Fanout(json_members) >> sf.Median(
    over="field:answer", weights="field:confidence", select="value"
)
s5_max_confidence = sf.Fanout(json_members) >> sf.ArgMax("field:confidence")
s7_top_confidence = (
    sf.Fanout(json_members) >> sf.Rank(2, by="field:confidence") >> sf.Synthesize(writer)
)
s8_argmax_confidence = sf.Fanout(json_members) >> sf.ArgMax("field:confidence")
s9_confident_only = (
    sf.Fanout(json_members)
    >> sf.Filter(where=sf.signal("field:confidence") > 0.7)
    >> sf.Synthesize(writer)
)
s10_judge_top = sf.Fanout(routes) >> judge >> sf.Rank(2, by="score:judge") >> sf.Synthesize(writer)
s11_judge_best = sf.Fanout(routes) >> judge >> sf.ArgMax("score:judge")
```

The writer after a Rank or a Filter gets one JSON object: the Case input and the kept member
texts (see "Rank and Concat selections"). `sf.Rank` gives the kept members best first, and
`sf.Filter` gives them in author order. The `>` test of `sf.Filter` is strict: a confidence of
exactly 0.7 is not kept. When no member passes, the Filter abstains with `no_votes` and the writer
gets `{"input":"...","selected":[]}` (see "Filter selection"). A member whose reply is not JSON has
no confidence, so `sf.ArgMax` never returns it. When no member has a confidence, the Case abstains.
A judge reply that is not one JSON object with a finite score for each member fails the Case with
`judge_reply_invalid`.

`sf.Judge` asks the judge to score each member as the rubric says; the step after it reads the
scores as `"score:judge"`. Say in the rubric which way is better, and pick the reader to match:
`sf.ArgMax` or `sf.Rank(..., order="max")` when a higher number is better, `sf.ArgMin` or
`order="min"` when a lower number is better. `sf.Vote(weights="score:judge")` uses the scores as
given, as weights: with a rubric where a lower number is better, the worst answers get the most
weight. Use a higher-is-better rubric for a judge-weighted vote.

### Signal sources

A `sf.Fanout` can declare named signals: numbers that the run measures from each member answer.
Give `signals={name: source}` to `sf.Fanout`, and read a name in a later step of the same block:
by its key (`weights="conf"`, `sf.ArgMax("conf")`, `sf.Rank(2, by="conf")`) or with
`sf.signal(name)` in a condition. The combine evidence records the value of every declared name
for every member, also when no step reads it. A step that reads a name that its `sf.Fanout` does
not declare raises `ValueError` before any call.

- `sf.Verbalized(pattern)` reads a number that the member writes in its text, for example
  `Confidence: 90`. The pattern is an RE2 pattern with exactly one group. The last match counts,
  and the number stays as the member wrote it. When nothing matches, or the group is not a decimal
  number, the member has no value.
- `sf.Logprob("mean" | "sum" | "prob")` reads the token log-probabilities of the reply, over all
  reply tokens. `mean` is the mean, `sum` is the sum, and `prob` is `exp(mean)`, a number above 0
  and at most 1.
- `sf.Entropy(top_logprobs=5)` is the mean token entropy of the reply, in nats, from the first
  `top_logprobs` alternatives of each token (1 to 20, default 5). A lower entropy means a more
  certain model.
- `sf.Sum({name: coefficient, ...})` adds other declared signals of the same member, each
  multiplied by a fixed coefficient (see "Sum signals and the abstain band").
- `sf.Consistency(n, temperature=1.0)` is the share of the member's `n` extra answers that equal
  its answer (failed extra calls do not count; see "Consistency samples").

Each row is one recipe that you can copy. Each recipe runs offline in the Engine's tests, from the
url4 that the SDK compiles.

| S | Scenario | Recipe |
|---|---|---|
| S12 | A vote weighted by the confidence that each member states | `s12_verbalized_vote` |
| S13 | A vote weighted by the token probability of each answer | `s13_logprob_vote` |
| S14 | Return the answer with the highest mean token log-probability | `s14_logprob_best` |
| S15 | Return the answer with the lowest mean token entropy | `s15_entropy_best` |
| S16 | Record a small model's mean log-probability against a threshold, then a larger writer | `s16_logprob_threshold` |

```python
import screamingface as sf

routes = [
    "openrouter/openai/gpt-4o-mini",
    "openrouter/openai/gpt-4.1-mini",
    "openrouter/openai/gpt-4.1-nano",
]
# Ask every member for the same confidence scale (0 to 100), and for the answer on the last line.
stated = (
    "Answer the request. On the first line write Confidence: and a whole number from 0 to 100. "
    "On the last line write only your final answer."
)

s12_verbalized_vote = sf.Fanout(
    [sf.Model(route, prompt=stated) for route in routes],
    signals={"conf": sf.Verbalized(r"Confidence: ([0-9]+)")},
) >> sf.Vote(weights="conf")
s13_logprob_vote = sf.Fanout(routes, signals={"p": sf.Logprob("prob")}) >> sf.Vote(weights="p")
s14_logprob_best = sf.Fanout(routes, signals={"lp": sf.Logprob("mean")}) >> sf.ArgMax("lp")
s15_entropy_best = sf.Fanout(routes, signals={"ent": sf.Entropy()}) >> sf.ArgMin("ent")
s16_logprob_threshold = (
    sf.Fanout(["openrouter/openai/gpt-4o-mini"], signals={"lp": sf.Logprob("mean")})
    >> sf.Filter(where=sf.signal("lp") >= -0.3)
    >> sf.Synthesize(
        sf.Model(
            "openrouter/openai/gpt-4.1",
            prompt="Write the final answer to the input from the selected answers.",
        )
    )
)
```

Ask every member for the same confidence scale. The Engine does not change the numbers: a member
that writes 90 and a member that writes 0.9 get very different weights. The Engine reads only the
last 64 KiB (65,536 characters) of each reply, so put the confidence line in that part. A line
before a longer text is not found, and that member has no value.

`sum` favors short replies: each token adds a number of 0 or less, so a longer reply has a lower
sum. `mean` and `prob` do not depend on the reply length, so use them to compare members.

A vote weight must be above 0. A `mean` or a `sum` is 0 or less, so `sf.Vote`, `sf.Median` and
`sf.Mean` refuse `sf.Logprob("mean")` and `sf.Logprob("sum")` as weights when `>>` closes the
block. They also refuse `sf.Entropy`, because a higher entropy means less confidence: use
`sf.ArgMin` or `sf.Rank(..., order="min")` for entropy. Use `sf.Logprob("prob")` as a vote weight. The error
says so:

```python
try:
    sf.Fanout(routes, signals={"lp": sf.Logprob("mean")}) >> sf.Vote(weights="lp")
except ValueError as error:
    print(error)  # names the fix: sf.Logprob("prob")
```

`sf.Logprob` and `sf.Entropy` work with OpenRouter models only, and each member must be one model
call (not a nested block or Pipeline). The SDK asks each member call for `logprobs`, and also for
`top_logprobs` when the `sf.Fanout` declares `sf.Entropy`. A model that the gateway parameter catalog
marks without logprobs is refused before any call. A model that cannot return logprobs at run time
fails with `logprobs_unsupported` and the Case fails; under a quorum, that member is left out and
shows as `missing`. A live test verifies one model: `openrouter/openai/gpt-4o-mini`.

Logprobs make each reply larger. The gateway cache does not keep a reply over 1,000,000 bytes, so a
rerun calls the model again, and the numbers can change. A frozen copy does not keep an entry over
2,000,000 bytes (request plus reply), and the capture reports `failed`. Keep `top_logprobs` small
when you need a replay that gives the same numbers.

S16 records the small model's value and whether it passed the Filter. The larger model still runs
on every Case: a cascade that skips it comes with E4c. When the small model does not pass, the
writer gets `{"input":"...","selected":[]}`.

### Gate the majority

A gate decides when a majority vote is strong enough to keep. It compares one named number with
one threshold. The gate is data, not code, so a reader can replay it from the recipe:
`sf.Fanout(members) >> sf.Gate(when=<number> <cmp> <threshold>, then=sf.Vote(...), otherwise=...)`.
The number is `sf.agreement()`, `sf.signal(name).chosen()` (a declared signal of the chosen
member) or `sf.signal(name).mean()` (the mean of a declared signal over the members). `name` must
be a name that the `sf.Fanout` declares in `signals=` (see "Signal sources"); any other name raises
`ValueError` before any call. `<cmp>` is `>=`, `>`, `<=` or `<`, and it is the condition that keeps
the majority: when `observed cmp threshold` is true, the answer is the majority answer. `>=` keeps
the majority at equality. An agreement gate takes only `>=` or `>`, and a threshold from 0 to 1.
The agreement is the winner's share of the votes; for a weighted vote, it is the winner's share of
the weight. When the comparison is false, the fallback gives the answer:

- `otherwise="abstain"` (the default): the Case abstains with the reason `gate_not_met`. This is a
  scored decline, not a wrong answer and not a failure, and coverage counts it with the other
  abstains.
- `otherwise=sf.JudgePick(...)`: the judge names one member, and the answer is that member's text.
- `otherwise=sf.Synthesize(...)`: the model writes the answer.

In this version the fallback call runs on every Case, also when the gate keeps the majority, and
you pay for it. The gate chooses and records; a cascade that skips the call comes with E4c. If the
fallback call fails, the Case fails, also when the gate keeps the majority.

When there is no number to compare (the chosen member has no value for the name, or no member has
a value for a mean), the gate takes the fallback. A vote with no answer does not run the gate: the
Case abstains with `no_votes`, as it does without a gate.

Each recipe below runs offline in the Engine's tests, from the url4 that the SDK compiles. The
routes are placeholders: put your own model routes in their place.

```python
import screamingface as sf

panel = ["prov/alpha", "prov/beta", "prov/gamma"]

# Keep the majority when at least 70% of the votes agree; else the Case abstains.
g1_agreement_abstain = sf.Fanout(panel, name="vote") >> sf.Gate(
    when=sf.agreement() >= 0.7, then=sf.Vote()
)
# Below 70%, a judge names one member, and the answer is that member's text.
g2_agreement_judge = sf.Fanout(panel, name="vote") >> sf.Gate(
    when=sf.agreement() >= 0.7,
    then=sf.Vote(),
    otherwise=sf.JudgePick("prov/judge", rubric="Pick the best answer."),
)
# Below 70%, a model writes the answer.
g3_agreement_writer = sf.Fanout(panel, name="vote") >> sf.Gate(
    when=sf.agreement() >= 0.7,
    then=sf.Vote(),
    otherwise=sf.Synthesize("prov/writer", prompt="Write one answer."),
)
# Keep the majority when the chosen member states a confidence of 80 or more.
# Each member must write a line such as "Confidence: 90" (see "Signal sources").
g4_confidence_gate = sf.Fanout(
    panel, name="vote", signals={"conf": sf.Verbalized(r"Confidence: ([0-9]+)")}
) >> sf.Gate(when=sf.signal("conf").chosen() >= 80, then=sf.Vote())
```

For a logprob cascade, declare `signals={"lp": sf.Logprob("prob")}` and gate on
`sf.signal("lp").chosen()`: the majority stays when the token probability of the chosen member's
answer is high enough.

`sf.Vote(verify=None)` is a reserved place for a later verifier; any other value raises
`ValueError`.

This gate is not the per-round gate of a corrective loop.

The Case pane, the run summary and the export show the gate (see "Combine steps in the report").

### Fusion as a named recipe

`sf.Fanout(members) >> sf.Synthesize(model)` is the named recipe: Fan-out, then one model writes
the answer from the member answers. The shortcut `sf.Fusion(members, synthesizer=...)` builds the
same chain: a route, an `sf.Model` or a recipe as `synthesizer=` becomes `sf.Synthesize(...)`, and
a step (a reducer, `sf.Synthesize`, or a narrow step that you then close with `>> sf.Synthesize(...)`)
comes after the `sf.Fanout` as it is. The two give the same value and the same url4, byte for byte,
and the value shows in the `sf.Fanout(...) >> ...` form. `sf.Fusion` also takes `signals=`,
`quorum=`, `name=` and `prompt=` (it gives them to the `sf.Fanout`) and `optional=True` (it gives
`sf.Pipeline([chain], optional=True)`). `optional=True` with a narrow step as `synthesizer=` raises
`ValueError`, because an optional chain must be a closed block, and a bare `sf.Judge` as
`synthesizer=` raises `ValueError`, because a judge only scores.

```python
writer = sf.Model("openrouter/openai/gpt-5.5", prompt="Write the final answer from the drafts.")
chain = sf.Fanout(routes) >> sf.Synthesize(writer)
same = sf.Fusion(routes, synthesizer=writer)
```

`sf.Synthesize(model, prompt=...)` sets the writer's prompt, and `sf.Fanout(members, prompt=...)`
sets the prompt of each member. The writer can be any recipe: `sf.Synthesize(sf.Pipeline([j, w]))`
runs `j`, then `w`. A chain goes on after the writer: `sf.Fanout(m) >> sf.Synthesize(w) >>
sf.Model(p)` gives the written answer to `p`. The candidate is named from the block: `a+b`, or the
`name=` of the `sf.Fanout`.

### Sum signals and the abstain band

A sum is a declared signal that adds other declared signals of the same member, each multiplied
by a fixed coefficient: `signals={"risk": sf.Sum({"sev_a": 0.5, "sev_b": 0.3}), ...}` gives each
member `risk = 0.5 × sev_a + 0.3 × sev_b`. Each component is another name that the same
`sf.Fanout` declares in `signals=`, never an E4a key (such as `cost_usd` or `score:judge`) and never another
`sf.Sum`; any other component raises `ValueError` before any call. A sum has 1 to 7 components.
Each coefficient is a finite number that is not zero; a negative coefficient is allowed, and the
Engine does not normalize the coefficients. The sum counts toward the 8 names that a `sf.Fanout`
can declare.

When a component is missing, the sum is missing, not zero. A rule leaves the member out, as for
any declared signal. The Engine adds the exact products and rounds once, so the order of the
components does not change the value. A sum that is too large for a float is missing too.

A step or a gate reads the sum by its name: `sf.ArgMax`, `sf.ArgMin`, `sf.Rank`, `sf.Filter`, a
vote weight, and a gate on `sf.signal("risk").chosen()` or `sf.signal("risk").mean()`. The evidence
records the sum of each member like any declared signal. A sum that adds `sf.Logprob("mean")`,
`sf.Logprob("sum")` or `sf.Entropy(...)` cannot be a vote weight: the `>>` that closes the block
raises `ValueError` before any call. A member whose sum is
zero or less, or larger than `7.022238808055921e+305` (the largest float divided by 256), gets
no weight, and the vote leaves it out with `no_weight`.

The abstain band is a range of the gate's number where the Case abstains and gives no verdict:
`sf.Gate(..., abstain_if=<number>.between(lo, hi))`, on the same number as `when=`. `lo` and `hi`
are finite numbers, and `lo <= hi`; a band of
one point is valid. For an agreement gate, `lo` and `hi` must be from 0 to 1. The band is closed
(both ends are in it). The Engine checks the band before the comparison. When the observed number
is in the band, the Case abstains with the reason `in_band` for every `otherwise`, also a judge
or a writer. When the number is outside the band, the gate compares as before. A missing number
skips the band, and the gate takes the fallback as before. The gate record shows
`"branch": "band"` and `"passed": null`. Coverage counts the Case with the other abstains.

A judge or writer fallback still runs on every Case, also when the number is in the band; its
reply is not read.

A sum adds the signals of one member. To compare the panel, gate on `sf.signal(name).mean()`.

Each recipe below runs offline in the Engine's tests, from the url4 that the SDK compiles. The
routes are placeholders: put your own model routes in their place.

```python
import screamingface as sf

panel = ["prov/alpha", "prov/beta", "prov/gamma"]
# Each member writes two severity lines, such as "A: 2" and "B: 5" (see "Signal sources").
severity = {
    "risk": sf.Sum({"sev_a": 0.5, "sev_b": 0.3}),
    "sev_a": sf.Verbalized(r"A: ([0-9]+)"),
    "sev_b": sf.Verbalized(r"B: ([0-9]+)"),
}

# The member with the lowest risk gives the answer.
s1_sum_pick = sf.Fanout(panel, name="pick", signals=severity) >> sf.ArgMin("risk")
# The Case abstains with in_band when the mean risk is from 2.5 to 3.5. Else the vote keeps
# its majority when the mean risk is 3.0 or less, and abstains with gate_not_met when not.
risk = sf.signal("risk").mean()
s2_band_gate = sf.Fanout(panel, name="vote", signals=severity) >> sf.Gate(
    when=risk <= 3.0, then=sf.Vote(), abstain_if=risk.between(2.5, 3.5)
)
```

The Case pane, the run summary and the export show the sum and the band (see "Combine steps in the
report").

### Consistency samples

`sf.Consistency(n, temperature=1.0)` is a declared signal that measures how often a member gives
its answer again: `signals={"cons": sf.Consistency(n=5)}`. Each member makes `n` more calls with
distinct seeds at the stated `temperature`; its other params do not change. The value of a member
is the share of the extra answers that equal the member's answer after the step's own extract and
normalize, a number from 0 to 1. A step that has no extract, such as `sf.ArgMax("cons")`,
compares the last line with the `text` normalizer. The extra answers never vote: they feed only
this number, and the step reads the members' answers as before.

An extra answer with nothing to extract is a successful extra call that does not match. The value
is also missing when the member has no answer to compare (it failed, or nothing was extracted). A
rule that reads a missing value leaves the member out, as for any declared signal.

Each member makes n more calls on every Case: a `sf.Fanout` of M members makes M × n extra calls,
and you pay for them. The extra calls run also when the member call fails. A failed extra call does
not fail the Case: the value is the share of matches over the extra calls that answered, and when
every extra call fails, the value is missing. The extra answers go into the combine context
with the member answers, so many long extra answers can pass the 64 MiB context limit; the step
then fails with `combine_input_too_large`. The extra calls use fixed seeds that the recipe
sets, so a replay returns the same samples; they do not change with the run seed. The free
preflight checks that each member's model takes `seed` and `temperature` before any call.

The rules: one `sf.Consistency` per `sf.Fanout`; each member is one `sf.Model` or a route, not a
Pipeline or a block; `n` from 1 to 32, with no default; `temperature` a finite number of 0 or
more; at most 1024 extra calls per `sf.Fanout` (members × n); the block ends in a reducer, not in
`sf.Synthesize`. A block that breaks a rule raises `ValueError` before any call. `sf.Consistency` is not
allowed inside a corrective loop in this version.

The SDK tests compile each recipe below to a fixed url4, and the Engine's tests run the first one
offline. The routes are placeholders: put your own model routes in their place.

```python
import screamingface as sf

panel = ["prov/alpha", "prov/beta", "prov/gamma"]
# Each member answers 3 more times at temperature 1.0. The answer comes from the member whose
# extra answers agree most often with its own answer.
c1_pick_most_consistent = sf.Fanout(
    panel, name="pick", signals={"cons": sf.Consistency(n=3)}
) >> sf.ArgMax("cons")
# Each member answers 2 more times at temperature 0.7. The median compares the last number of
# each answer, and the evidence records each member's value.
c2_median_consistency = (
    sf.Fanout(panel, name="median", signals={"cons": sf.Consistency(n=2, temperature=0.7)})
    >> sf.Median()
)
```

`sf.Vote(weights="cons")` weights each member by its value; a member with the value 0
has no weight and does not vote.

The Case pane, the run summary and the export show the value and the failed extra calls (see
"Combine steps in the report").

## Install

```bash
pip install screamingface
```

Python 3.12 or newer is required. This command installs the hosted client. The local stack
(`screamingface up`) also needs the runtime services and notebook tools:

```bash
pip install "screamingface[runtime,notebook]"
```

### Troubleshooting

`SCREAMINGFACE_RUNTIME_ERROR No module named 'X'` from `screamingface up` — common on
Colab, which preinstalls some of the runtime dependencies: the `[runtime]` extra is not
installed. Run `pip install "screamingface[runtime,notebook]"` and start again with
`screamingface up`. `screamingface doctor` names the missing modules.

## Client configuration

The module-level interface constructs one process-wide Client lazily against the hosted development
Engine and public Scoreboard. Configure either origin explicitly when selecting local or alternate
deployments:

```python
import screamingface as sf

sf.configure(
    engine_url="http://127.0.0.1:9108",
    scoreboard_url="http://127.0.0.1:9106",
)
report = sf.evaluate(candidates, benchmark="draco", limit=1)
sf.close()
```

`sf.configure(...)` replaces and closes any existing default Client. A non-default `engine_url`
must come with its `scoreboard_url`; the call raises otherwise, so a local or preview Engine can
never publish to the hosted leaderboard by accident. The same holds for the environment:
`SCREAMINGFACE_ENGINE_URL` needs `SCREAMINGFACE_SCOREBOARD_URL`. `sf.close()` releases the
default Client so the next module-level operation constructs a fresh one.

The Client hides Benchmark fetching, URL4 compilation, REST/WebSocket transport, Event replay,
and Report decoding behind `sf.evaluate(...)`.

### Hosted caller authentication

Hosted Engines may be protected by Cloudflare Access. The Client logs in through the browser on
first use. For an unattended run that a person started (for example the e2e workbench), set
`SCREAMINGFACE_ACCESS_TOKEN` to an Access application token (`cloudflared access token -app=<engine
url>`); the Client uses it for every origin whose Access audience matches the token's `aud`, and
falls back to the browser login otherwise. The token acts as the person it was issued to.

```python
client = sf.Client(engine_url="https://fusion.dev.screamingface.ai")
client.login()  # optional: the first protected request also starts login
```

The Client discovers the Access application audience from the Engine redirect, creates an
ephemeral encryption keypair, and opens Cloudflare Access login in the user's browser. It polls
Cloudflare's encrypted transfer service and decrypts the returned application token locally. The
token is held only in process memory and sent as `Cf-Access-Token` on REST requests and WebSocket
handshakes. `client.logout()` forgets it and opens the Engine's Cloudflare Access logout endpoint
in the browser. Concurrent callers share one browser login, a server-rejected token starts one fresh
login even before its local expiry, and an Access-specific WebSocket rejection is retried once after
reauthentication.

The login URL is always printed. Desktop Python also attempts to open it automatically; in Jupyter
or Colab, click the displayed URL and complete the configured Access login. This flow does not use
a localhost callback and does not require dynamic client registration or **Allow loopback
clients**. The user's email or identity must be allowed by the Cloudflare Access policy for the
hosted Engine. The Client prints a confirmation after it receives and validates the transferred
token; `client.authenticated` then returns `True`. Local Engines that do not advertise Access
continue to work without authentication.

The public authentication boundary is the URL4 Cloud origin, not AI Gateway. After Cloudflare
Access authenticates the caller, the deployment passes the verified identity to URL4 Cloud as
`X-User-Email`. URL4 Cloud forwards that identity—not the Access token—to the internal AI Gateway.
Consequently, the Python Client never calls AI Gateway or a model provider directly:

```text
Python Client -- Cf-Access-Token --> Cloudflare Access --> URL4 Cloud
                                                          -- X-User-Email --> AI Gateway
                                                                             -- provider key --> Provider
```

These credentials have deliberately different lifetimes and owners:

- The Cloudflare Access token exists only in Client memory and authenticates calls to URL4 Cloud.
- `X-User-Email` is derived from the edge-verified identity and selects the AI Gateway account.
- A provider key entered through `sf.connect()` travels through URL4 Cloud once for AI Gateway to
  validate and store; URL4 Cloud does not retain it.
- No shared Cloudflare key, provider key, or administrator key is configured on the Client.

If the hosted application later adopts Cloudflare Managed OAuth, the Client can migrate to OAuth
discovery and authorization code + PKCE. That is a possible future protocol, not an additional
authentication mode implemented by this package today.

In a notebook, `sf.connect()` displays the connection panel bound to the lazy default Client. For a
remote Engine, the panel first checks noninteractively whether Cloudflare Access is present. An
unprotected remote Engine loads its providers normally; a protected Engine shows the Engine login
row and loads provider rows only after login succeeds. Login waits in the background, so the
notebook remains usable and the row becomes
**Cancel** while the encrypted transfer is pending. Opening `sf.connect()` again reflects that same
in-progress login, and all open panels follow its eventual login/logout state. Cancel stops only the
pending transfer without opening another browser page;
Log out clears the completed token and opens Cloudflare Access logout. An explicit Access rejection
is shown in the panel, while an abandoned browser flow remains cancelable until its timeout. Local
Engines omit this row. This one-time transfer polling is separate from the authenticated WebSocket
used for model execution.
The Engine derives its connection catalogue from AI Gateway's enabled provider plugins. API keys
are supported for any provider that advertises `api_key`, and the panel can start OAuth for any
provider that advertises `oauth`. A key is sent only to the SF Engine,
which asks AI Gateway to validate and store it; the Python Client does not persist it and never
calls AI Gateway or a provider directly. One OpenRouter key covers every enabled `openrouter/...`
route, but does not authorize direct routes owned by other providers.

```python
sf.connect()
flow = sf.connect("codex", method="oauth")
flow.authorize_url
connection = flow.wait()  # or flow.cancel()
sf.connections.list()
sf.connections.get("openrouter")
sf.disconnect("openrouter")
```

The same panel retains OAuth, pending, authorization, cancellation, and reauthentication states.
The Engine catalogue remains authoritative about which methods each provider supports.

Local and hosted Engines use the same Client contract. Local mode may run the URL4 executor
in-process with an in-memory event bus; hosted mode may use the REST/WebSocket control plane with
NATS and scheduled workers. Engine execution is selected through `engine_url`; public Leaderboard
discovery is configured independently through `scoreboard_url`. Generic URL4 execution remains
benchmark-agnostic; installed Engine definitions own Benchmark semantics.

## Progress and Reports

`evaluate` consumes the Engine's REST and WebSocket lifecycle internally. An optional callback
receives typed CloudEvents views in sequence:

```python
def observe(event: sf.Event) -> None:
    print(event.kind)


report = sf.evaluate(
    candidates,
    benchmark="draco",
    limit=1,
    on_event=observe,
)
```

If a callback raises, the Client attempts to cancel all active Candidate Runs and re-raises the
original exception.

One `Report` shape covers one or many Candidates:

```python
report.ok
report.benchmark
report.case_count
report.candidates[0]
report.candidates["frontier-pair"]
report.candidates["frontier-pair"].run_id
report.candidates["frontier-pair"].url4
report.candidates["frontier-pair"].operations
report.candidates.only
report.failures
report.usage
report.started_at
report.completed_at
report.to_dict()
report.to_json()
artifact_path = report.export()  # Path("report.json")
report.export("runs/draco.json")
```

`Report.export(...)` writes the exact complete JSON document returned by `to_json()`, creates
parent directories, replaces an existing selected file for deterministic reruns, and returns its
`Path`. A Report remains one JSON document even when it contains multiple Candidates; JSONL is
reserved for a future collection of independent Reports.

JSON export serializes one Case at a time and replaces the destination only after the complete
file is flushed and fsynced. Existing output symlinks and ordinary file permissions are
preserved; new files honor the process umask. A failure before replacement keeps the previous
file. On POSIX, failure during the final directory fsync means replacement happened but its
durability is unconfirmed. `to_json()` uses the same serializer, but its returned string
still occupies memory; use `export()` for large reports.

`CandidateResult.client_version` records the caller-reported Client version retained by the
Engine for that run. It is also present in each candidate in `report.json`; missing or ambiguous
evidence is `null`. It is not the version of the Client later exporting the report, nor the
version that originally generated cached model answers.

Each entry in `CandidateResult.operations` is a public immutable `sf.OperationInfo` value.

Authentication, validation, transport, execution, protocol, and invalid-result failures raise
typed exceptions.

Expected SDK failures inherit from `ScreamingFaceError` and always carry a stable error code, plus
an optional HTTP status, structured details, remediation hint, and `permanent`/`retryable`
classification. The public classes reflect distinct recovery actions:

- `EngineUnavailableError`: start, reconfigure, or retry the Engine.
- `AuthenticationError`: authenticate the caller again.
- `PlanningError`: change the Candidate, Benchmark, Model, or evaluation configuration.
- `ExecutionError`: inspect or retry a Run that failed after reaching the Engine.
- `ProviderConnectionError`: change a provider credential or provider connection.

These classes arrive directly when the failure happens before any Candidate runs — for example
while the Client loads the Benchmark or the Model catalogue — and when an Evaluation has one
Candidate. A multi-Candidate Evaluation is different once its Candidates run: one failed
Candidate does not stop the others. They run to their end, the progress output marks the failed
row at once, and then the Evaluation raises `ExecutionError` with `code="candidates_failed"`:

- `error.details["failed"]` maps each failed Candidate's name to its error code, for example
  `websocket_disconnected`, `engine_at_capacity`, `authentication_failed` or
  `engine_unreachable` (`unexpected_error` when the failure had no SDK code);
- `error.__cause__` is the first failure itself, so an `AuthenticationError` or
  `EngineUnavailableError` from a running Candidate is found there, not raised directly;
- `error.partial_report` is a `Report` of the Candidates that succeeded, or `None` when none
  did. It holds nothing for a failed Candidate.

To keep the paid results, export the Partial Report; `details["failed"]` names the Candidates to run again:

```python
try:
    report = sf.evaluate(candidates, benchmark="draco")
except sf.ExecutionError as error:
    if error.code != "candidates_failed":
        raise
    if error.partial_report is not None:
        error.partial_report.export("partial-report.json")
    failed = error.details["failed"]  # {candidate name: code}
```

Ctrl-C, task cancellation, and an exception raised by your own `on_event` callback still stop
every Run of the Evaluation and re-raise that exception unchanged.

IPython and Jupyter render these failures as a concise message, hint, and code instead of exposing
dependency tracebacks. Notebook panels render the same safe text inline. Programmatic callers can
catch a specific recovery class or catch `ScreamingFaceError` for every expected SDK failure;
translated low-level failures remain attached through `error.__cause__` for debugging. Programmer
errors such as invalid Python argument types retain their normal tracebacks.

Every `CandidateResult` exposes the Engine-owned top-level `coverage` ratio. A partial score remains
available alongside the Cases that could not be graded, and the notebook Report panel labels the
result as partial rather than silently presenting it as a complete evaluation.

### Browse completed Case results

Install `screamingface[notebook]` and display a completed `report` in a live notebook to
browse 25 Case results per page. Use Previous/Next, enter an exact Case ID in Go to case
number, or filter by Candidate using the dropdown or a Candidate name. The answer/grading
and cost views remain available for every result. Numeric and string Case IDs retain their
original identity.

Download prepares the complete report JSON on disk, including every Candidate and Case,
then provides a Jupyter file link (or the saved path when no local server serves it).
In Colab it uses the native browser download API; the button can download the saved file
again without re-exporting it. Jupyter prepares exports and pages asynchronously. Colab
publishes bounded page changes and streaming exports inside the click callback, as its
widget transport requires. Failures leave the current page available and allow retry.
`report.export('report.json')` also writes the complete portable artifact directly.

Colab's custom widget manager is enabled automatically for this browser. The notebook extra
accepts ipywidgets 7.7.1 and 8, preserving Colab's compatible preinstalled version. If you
previously upgraded Colab to ipywidgets 8 and see a blank report, install
`ipywidgets==7.7.1` and restart the Colab session before displaying the report. Ordinary
Jupyter supports ipywidgets 8. Case clicks use Colab’s native callback API there and
ipywidgets DOM events in Jupyter. This compatibility verification covers completed reports;
older-widget live-progress accessibility has separate existing limitations.

Static HTML, including environments without the notebook extra, previews the first 25
results and shortens oversized Case panes. Small JSON downloads remain inline; larger
ones use explicit export. The complete retained data remains available through the Report
and JSON export.

### Combine steps in the report

The HTML report and the JSON export show what each combine step did. They show the same evidence
as `case.answers()` and `case.trace()`, and you do not have to write Python code to see it.

**The Case pane.** Every reducer step of a Case has a Combine section in the **Answer & grading**
view. The section shows the step facts: the reducer, the rule, the result, the cut, the signal that
the step reads, the judge model, the agreement, the tie and the dropped count. It has one row per
member: the status and the reason, the value, the weight, the score, the rank, each signal, and a
mark for a chosen member. A plain vote has its section too, with its tally. A declared signal
column names its source on hover. A member value longer than 80 characters ends with `…`. The full
value is in the export and in `case.answers()`. A step with no evidence record shows only
`evidence` and an em dash.

**The run summary.** Each Candidate card has one table per reducer step. The table shows how often
each member was chosen and how often it was excluded, for example `2 of 3`. It also shows the mean
of each declared signal with `n`, the number of Cases that gave a value. The card has no table when
no Case of the Candidate has combine evidence.

**The export.** `report.export()` and `report.to_dict()` add a `trace` key to each Case with a
combine step. It holds the same entries as `report.trace_dict()` for that Case. The trace holds no
member output text except each member's extracted `raw` and `value`. For `whole@1` and
`json_object@1` these are the whole reply, so the export holds that text two times. The other keys
do not change, and the schema stays `report.v1`. A Case with no combine evidence has no `trace`
key.

**The gate.** A gated vote has one more chip in its Combine section, right after the agreement
chip, for example `gate · agreement >= 0.7 · observed 0.6667 · fallback · judge`. The chip shows
the number that the gate read, the comparison, the threshold, the observed value and the branch:
`majority`, `fallback` with its fallback (`abstain`, `judge` or `writer`), or `not run` when the
vote found no answer. The run summary of a gated step has one line before its table, for example
`gate · majority 2 of 3 · fallback 1 of 3`, with `not run` added when a Case did not run the gate.
The export holds the gate record in `operations[].combine.gate`, with the keys `observe`,
`observed`, `cmp`, `threshold`, `passed`, `branch` and `otherwise`. When the step read a judge
fallback, the judge chip and the trace `judge_model` name that judge.

**Sums and the band.** A sum is one more signal column in the Combine section. Hover its header
to see the formula, for example `source: sum of 0.5 × sev_a + 0.3 × sev_b`. The run summary
shows the mean of a sum as it shows the mean of any declared signal. A gate with a band shows the
band in its chip, for example
`gate · agreement >= 0.7 · band [0.6, 0.7] · observed 0.6667 · in band · abstain`.
The result chip of that Case is `abstain · in_band`. The run summary line counts the Cases in the
band, for example `gate · majority 1 of 3 · fallback 1 of 3 · in band 1 of 3`; the count shows
only when a Case is in the band. The export holds the band in
`operations[].combine.gate.abstain_band` as `[lo, hi]`, and the record of a Case in the band has
`"branch": "band"` and `"passed": null`.

**Consistency.** A step that declares `sf.Consistency` has one more chip in its Combine section,
for example `samples · cons · 4 per member · failed 6 of 12`: the declared name, `n`, and how
many extra calls of the step failed, out of all of them. The `cons` cell of a member shows its
value, and a note such as `2 failed` when some of its extra calls failed. A missing value names
why on hover, for example `every sample failed` or `no answer to compare`. The run summary of the
step has one line before its table, for example `samples · cons · failed 12 of 24`. The export
holds the counts in `operations[].combine.consistency`: for each member, `member_id`, `matched`,
`succeeded` and `failed`.

Numbers show 4 significant digits; hover a number to see the exact stored value. A missing value
is an em dash; hover it to see why. The combine steps of an Attempt (`case.attempts`) are not in
the pane, the summary or the export.

### Completed accounting breakdown

Each Candidate exposes derived views of the Engine's retained operation and grading records:

```python
breakdown = report.candidates[0].accounting
breakdown.by_stage  # generation, synthesis, grading
breakdown.by_operation  # (stage, operation/check id)
breakdown.by_member  # direct model-member operation ids
breakdown.by_model  # request model or reliable declared identity; None when unknown
breakdown.by_case  # original Case ids
breakdown.unattributed_cost_usd
```

Each group has `usage`, `calls`, `cache`, `provider_latency_ms`, and `provider_attempts`.
These summarize retained observations; they do not replace the authoritative run total.
Missing observations remain unknown. A missing record stays in its declared model group
when that identity is unambiguous and agrees with retained requests. If any row's model
cannot be identified, all named `by_model` summaries are unknown: the unidentified work
could belong to any of them. The `None` bucket summarizes only its anonymous observations.
Calls count consumed responses, not provider retries.
Provider time sums attempt latencies and is not wall time. Unknown or unpriced costs prevent
an exact remainder; inconsistent records disable the breakdown (`consistent=False`).
Loop internals and composite-member ownership remain unattributed where the retained contract
cannot prove their scope. Derived group views are not added to Report JSON.

Each completed Report Case has **Answer & grading** and **Cost & usage** views.
The latter shows labelled activity blocks; whole-run totals remain above the Cases.
Try [the offline review notebook](examples/14_report_accounting.ipynb) without credentials or
paid calls. Its synthetic figures are explicitly labelled; actual evaluations use the same UI.

## Ownership boundary

```text
Researcher or SF App
        ↓
ScreamingFace Python Client
  Recipe authoring · URL4 compilation · Events · Reports · Leaderboard reads
        ├─ REST + WebSocket → SF Engine → AI Gateway
        └─ public HTTPS GET → Scoreboard
```

Evaluation, discovery, and provider-connection operations call the configured SF Engine.
Leaderboard discovery calls the configured public Scoreboard. The Client never calls AI Gateway,
model providers, Tavily, or Benchmark datasets directly. Local and hosted Engines expose the same
Client-visible contract; in-memory channels, NATS, workers, and deployment topology are Engine
details.

Models, `sf.Fanout` blocks, and Pipelines are immutable, structurally comparable,
Client-independent, and network-free. Models select routes and optional answer policy; a
`sf.Fanout` block declares parallel topology and ends in an explicit reducer or writer; Pipelines
declare serial topology. Every placement compiles to a
distinct logical invocation, even when two Recipe values are equal or reused. The SDK compiles the
complete Recipe into one Candidate expression. Benchmarks are immutable Engine protocols
that own Cases, Candidate Invocation order, fixed Judge configuration, Grading, Aggregation,
and execution policy. Reports record the exact Engine-pinned Benchmark revision.

Durable reuse across graph positions, Candidates, retries, and resumed Evaluations belongs to the
Engine's provenance-aware response cache; Client compilation never merges authored positions.

## Discovery

An Engine implementing the provisional catalogue contract exposes typed discovery:

```python
models = sf.models.list()
gpt_details = sf.models.get("openrouter/openai/gpt-5.5")
benchmarks = sf.benchmarks.list()
boards = sf.leaderboards.list()
draco_board = sf.leaderboards.get("draco", top=50)

# After evaluating, with capture=True, a Benchmark whose Scoreboard accepts submissions:
submission = sf.leaderboards.submit(
    report.candidates.only,
    authors=["alice@example.com", "bob@example.org"],
)
same_submission = sf.leaderboards.get_score(submission.id)

# Add or fix the paper link and authors of a score you submitted, then read the edit log:
sf.leaderboards.edit(submission.id, paper_url="https://arxiv.org/abs/2610.01234")
edit_log = sf.leaderboards.metadata_events(submission.id)

# Attach open metadata about the authors (keyed by email) and the paper:
sf.leaderboards.edit(
    submission.id,
    authors_metadata={"alice@example.com": {"full_name": "Alice Smith", "institution": "Uni"}},
    paper_metadata={"title": "Our ensemble", "url": "https://arxiv.org/abs/2610.01234"},
)

# Replay the published run at $0 from its pinned cache (None unless it ran with capture=True):
replay = sf.evaluate(pinned_cache=same_submission.pinned_cache)
reproduction = replay.candidates.only.reproduction
if reproduction is not None:
    verdict = reproduction.outcome  # "exact" or "failed"; failures include .reason
editable_python = same_submission.url4.to_python()
replayed_report = sf.evaluate(same_submission.url4)
```

Explicit Clients provide the same interface through `client.models.list()` and
`client.models.get(model_id)` alongside `client.benchmarks.list()`; asynchronous Clients use the
same names with `await`. `ModelInfo` rows are lightweight summaries containing the supported
parameter and tool names. `ModelDetails` is the Connection-specific contract for one Model, including
typed parameter schemas, gateway policy, provider evidence, tools, transport, and freshness.

`sf.leaderboards` uses the separate public Scoreboard: `list()` returns its registered benchmark
summaries, while `get(benchmark_id, top=...)` returns one immutable `Leaderboard` containing ranked
best-per-spec entries and imported single-Model baselines. `submit(candidate_result)` publishes an
already-evaluated result — the Benchmark-native score exactly as the Engine graded it, fractional
or negative included — without asking the caller to repeat its Benchmark, URL4,
models, or run identity; `get_score(id)` retrieves the resulting immutable
`LeaderboardScore`. Omit `authors` to credit the authenticated submitter by default. When supplied,
the list is the exact credit line—the submitter is not added automatically—and the public
`LeaderboardScore.authors` and `LeaderboardEntry.authors` values contain the Scoreboard's
privacy-trimmed author identifiers. A score's `.url4` property is a string-compatible `Url4` value:
`.to_python()` produces an editable fork, while passing the value to `sf.evaluate(...)` replays it
through the configured Engine. Submitting a limited or incompletely graded Candidate surfaces a
`Partial submission` advisory because its score is not directly comparable with a full run. In a
notebook the Client displays a branded notice even when the score is assigned to a variable;
headless callers receive `sf.EvaluationWarning` attributed to their submission line. If that
warning category is configured as an error, submission stops before the Scoreboard is changed.
`submit(..., paper_url=...)` links a paper you have already published (an `http` or `https` link
with a host and no user info or whitespace). Later, the submitter can call
`edit(score_id, authors=..., paper_url=...)`: an argument you leave out stays unchanged,
`paper_url=None` clears the link, and `authors=None` is refused (pass `[submitter]` to go back to
the default). `edit` returns the updated `LeaderboardScore`, which carries `paper_url` and
`metadata_updated_at`. `metadata_events(score_id)` returns the owner-only edit log as
`ScoreMetadataEvent` rows with the old and new values. The SDK does not check that the paper exists
or that the authors wrote it. A Scoreboard deployment may
keep writes closed, in which case `submit()` raises a typed `LeaderboardError`. Explicit Clients
expose the same interface at `client.leaderboards`; asynchronous Clients use `await`. The
Scoreboard is the deployed data system, while a Leaderboard is the ranked domain resource returned
to callers.

Provider access and explicit Candidate parameters are preflighted before execution. The SDK
fetches one detail document per distinct required Candidate Model, including parameter-free
Models, and reuses any document fetched during model admission. Authoritative missing provider
access raises `ProviderConnectionError` before any Candidate dispatch or progress events, with a
hint to connect the provider with `sf.connect()` or to ask the operator of a hosted Engine to
connect it in AI Gateway.
`ModelDetails.execution_access` is `"configured"`, `"missing"`, or `None` when an older Gateway
omits the field. Unknown access preserves existing behavior; the early missing-access guarantee
requires a Gateway that publishes the field. Configured access is not credential validation.
Missing, disabled, wrong-type, or out-of-range parameters also fail before any paid Run begins.
Access and capability data come from Engine/AI Gateway discovery, with no provider-specific
credential or parameter rules in the SDK. Sync and async Clients use the same checks.

The returned catalogues are immutable ordered sequences: iteration, indexing, slicing, and
`len()` work normally in scripts and sidecars. Evaluating one in Jupyter automatically renders a
searchable catalogue when the `notebook` extra is installed, with escaped static HTML and compact
terminal representations as fallbacks. Notebook rendering does not change the underlying values
or introduce a separate discovery operation.

### Replay a published run

`evaluate(..., capture=True)` captures a **pinned cache**: every model and web-tool answer of the
run, stored with it (`CandidateResult.pinned_cache`). Capture is off by default. After the result is submitted,
`sf.evaluate(pinned_cache="<id>")` replays that run at $0: the recipe and answer seed come from the
public board, and every call is answered from the pinned cache. `CandidateResult.reproduction` then
says whether the replay gave the published score: `exact`, or `failed` with a `reason` and the
cases the pinned cache could not answer. An exact replay is recorded on the board (a leaderboard
login is asked for only then) unless `record_reproduction=False`.

- `score_id="<id>"` replays a score on a private board that you submitted.
- `url4=` is optional; when passed it must be the same parsed recipe as the published one. A passed
  `answer_seed` must equal the captured run's; a run captured without a seed replays without one.
- `limit=n` replays only the first n cases, for debugging. It is never judged or recorded.
  `cases=` is refused on a replay: a replay runs the published Case set or its first n, never a
  different list.
- A capture that was never submitted replays with `url4=` (and `answer_seed=` if it had one) and
  has no verdict.
- A replay that cannot start (a partial capture, or an Engine that does not replay) raises
  `sf.ReproductionError`. Every other mismatch fails before any run.

## Examples

From a checkout, `just local-stack-notebooks` runs every notebook below: it bakes the frozen
benchmark assets, brings the local stack up, serves an Engine carrying the imported
inspect_evals boards alongside ours, and opens JupyterLab against it.

- [`examples/00_quickstart.ipynb`](examples/00_quickstart.ipynb): one Candidate through the
  first canonical `draco` Case, from discovery through Report evidence.
- [`examples/01_client_tour.ipynb`](examples/01_client_tour.ipynb): a no-spend tour of Client
  lifecycle, hosted authentication, discovery, connections, authoring, events, errors, Reports,
  and the asynchronous API.
- [`examples/06_draco_full_e2e.ipynb`](examples/06_draco_full_e2e.ipynb): the complete seven-solo,
  nine-fusion canonical DRACO experiment and audit workflow, with execution disabled by default.
- [`examples/07_ifeval_e2e.ipynb`](examples/07_ifeval_e2e.ipynb): canonical deterministic
  IFEval across the solo/panel × plain/corrective Recipe grid.
- [`examples/08_healthbench.ipynb`](examples/08_healthbench.ipynb): both HealthBench
  boards — the worst-30% open-fusion challenge and the full 525-case exam with the
  official score — rehearsed cheaply with `limit=1` first.
- [`examples/12_inspect_evals_benchmarks.ipynb`](examples/12_inspect_evals_benchmarks.ipynb): the
  catalogue as two origin groups — ours and the imported inspect_evals boards — then a
  fusion against one of the imported boards.
- [`examples/18_picture_benchmarks.ipynb`](examples/18_picture_benchmarks.ipynb): two Cases of
  each picture Benchmark (LAB-Bench FigQA and TableQA, MMMU, MathVista, HLE, LABBench2 FigQA2 and
  TableQA2) on the local stack or dev, chosen by one variable, with paid runs off by default.
- [`examples/19_map_select_reduce.ipynb`](examples/19_map_select_reduce.ipynb): the combine
  recipes in the map → score → narrow → reduce form, built and compiled offline (no model
  call).

All notebooks are deterministic outputs of `scripts/build_notebooks.py`.

## Development

```bash
uv run ruff check .
uv run ruff format --check .
uv run pyright
uv run pytest --cov=screamingface --cov-fail-under=95 -q
uv run --extra notebook python scripts/check_notebooks.py
uv build
uv run python scripts/check_distribution.py
```


## Reopen completed results

Public `Client` and `AsyncClient` instances save completed-run metadata before downloading
results. Artifact downloads are streamed and checked against the Engine ticket's size and
SHA-256. Cases are validated incrementally into a disk index and loaded on demand, preserving
prompts, answers, grading, operations, accounting, and the SDK's run provenance.

```python
import screamingface as sf

entries = sf.reports.list()
report = sf.reports.get(entries[0].id)
# In async code: report = await sf.reports.get_async(entries[0].id)
# Explicit removal: sf.reports.delete(entries[0].id)
```

The default location is `~/.screamingface/results`; `SCREAMINGFACE_RESULTS_DIR` overrides it,
or use `SCREAMINGFACE_DATA_DIR` to choose the containing data directory. `list()` reads
lightweight metadata and groups Candidates from the same Evaluation. `get()` and
`get_async()` use local copies when available and obtain fresh authentication when a missing
artifact needs downloading. They never start model runs. `destination='/another/disk'`
lets recovery copy retained results to another location. Incomplete recovery raises the
existing `candidates_failed` error with a `partial_report` and named Candidate failures.

New saved evaluations track whether evaluation, notebook display or JSON export is in progress.
When a POSIX notebook process exits unexpectedly, the next persisted Client for the same Engine
shows an informational notice with an explicit `sf.reports.get(...)` command for retained completed
results. It does not rerun models or recover automatically. Healthy, active and legacy results
stay quiet; headless Clients do not scan results at startup. Owners that cannot be verified dead
(including non-POSIX environments) are treated conservatively. Process-ID reuse can also suppress
a notice; saved results remain available through `sf.reports.list()`. On POSIX, overlapping
notebooks sharing the same directory preserve one another's unfinished work, including when
a saved Candidate is recovered, displayed or exported from another process.

Local copies remain until explicit deletion. Deleting a report also makes existing
Reports backed by its files unreadable. Recovery requires a saved completion record and
either a valid local raw result or an unexpired Engine artifact. An unavailable artifact
reports `result_expired` with its age. Death before completion is recorded cannot be recovered.
Disk failures report `result_storage_failed`; collection never silently falls back to RAM.
Only completion/result metadata is saved, never authentication credentials.

Set `Client(save_results=False)` or `AsyncClient(save_results=False)` for the previous
in-memory collection path. These options do not change a custom supplied `run_transport`.
Calling `to_dict()`, `to_json()`, converting Cases to a list, or requesting a Case slice
still materializes the requested data. `CandidateResult.accounting` also materializes its
full accounting breakdown. Notebook cost disclosures instead derive a compact context by
streaming Cases and cache that context beside a saved Case index on first use. Warm run-level
disclosures reuse the cache without decoding Cases; missing, stale or malformed caches
rebuild, and a read-only cache location still returns derived values. Per-Case HTML still
materializes its rendered output. Automatic lifecycle discovery remains a separate follow-up.

## Usage analytics

Analytics is enabled by default in Python scripts, headless jobs and notebooks.
The SDK makes a best-effort attempt to show its disclosure; redirected, missing
or closed output does not prevent enrollment. Existing opt-outs remain off.
In local Python or Jupyter:

```python
import screamingface as sf

sf.analytics.enable()  # Explicitly enable again after opting out
sf.analytics.status()  # Inspect consent and effective state
sf.analytics.disable()  # Stop sending and remove the local identifier
sf.analytics.reset_identifier()  # Start a new identifier without linking it
```

The equivalent commands are `screamingface analytics status`, `enable`, `disable`
and `reset-id`. They do not start local runtime services. `sf.analytics.enable(persist=False)`
accepts for this process only when you do not want persistent preferences.
`sf.analytics.disable(process_only=True)` stops this process without changing the saved choice.
A truthy `DO_NOT_TRACK` overrides acceptance, including in headless jobs.
Set `DO_NOT_TRACK=1` before import to prevent automatic enrollment entirely.
Noninteractive use never asks for input.

Studio inherits this default when its sidecar bundles this SDK version. Starting
it may enroll a fresh installation; the notice goes to runtime logs. Startup alone
emits no event. Studio and Python share the SDK preference file and opt-out controls,
independent of Studio's runtime data directory. Studio currently has no dedicated
analytics notice or settings control; no UI follow-up is planned for now.

Import, connection setup, and the first evaluation can attempt the notice, whichever
runs first. Python notices go to stderr, including redirected logs, so stdout
remains usable for data. Local notebooks show inline text. The notice never asks
for input and is attempted at most once per process when enrolling a new installation.
Saved choices have no scheduled re-prompt.

> ScreamingFace collects anonymous usage analytics (API usage, SDK versioning, and your local-or-hosted engine preference) to help us build a better API. It does not store any fusion, prompts, outputs, scores, or keys. It's on by default; turn it off anytime with `sf.analytics.disable()` (or set `DO_NOT_TRACK=1`).

Preferences live in `~/.screamingface/analytics.json`, or the explicit
`SCREAMINGFACE_ANALYTICS_CONFIG` path. This is independent of runtime `--data-dir`,
virtual environments and project directories. Writes are atomic and owner-only;
unreadable or malformed preferences disable collection and are never automatically
replaced. Failed automatic enrollment leaves analytics off without interrupting
product work. An explicit `enable()` still raises if preferences cannot be saved.

The four events are `evaluation_started`, `evaluation_finished`,
`submission_started` and `submission_finished`. They contain random installation,
session and operation IDs, SDK version, Python/Jupyter origin, sync/async interface,
recipe/raw-URL4/submission workflow, outcome and coarse duration. For the current
release, local engines map to BYOK and remote engines to hosted. No prompts,
responses, scores, provider/model names, execution IDs, credentials or email are sent.

An installation ID is shared by processes using the same preference file; a session
ID is created lazily per process/kernel. Opt-out or identifier reset also invalidates
the active session so replacement identifiers cannot be joined through it. These
identifiers measure consenting installations/sessions, not verified people.

The SDK defaults to `https://analytics.screamingface.ai`. Set
`SCREAMINGFACE_ANALYTICS_ENDPOINT` to an HTTPS origin **before first analytics use**
to select another deployment, for example:

```bash
export SCREAMINGFACE_ANALYTICS_ENDPOINT=https://analytics.dev.screamingface.ai
# For staging: https://analytics.staging.screamingface.ai
```

Restart the process/kernel to change destinations. This setting is independent of
the engine URL; there is no automatic detection or fallback to another deployment.
The selected service owns the PostHog project configuration; the SDK needs no
PostHog token. Local consent remains installation-wide, so use a separate
`SCREAMINGFACE_ANALYTICS_CONFIG` file for isolated test consent/identifiers.
Native delivery posts to `/v1/events`, without product credentials or cookies. Delivery has a memory-only queue capped at 100 pending events, at most two
attempts per event and a two-second total delivery budget. It never waits for HTTP
in evaluation, never waits for delivery at shutdown, and may lose events. Consent
is rechecked before dispatch and retry; already dispatched requests can complete
after opt-out. There is no durable event spool.

In Colab, import or first interactive use starts a consent-only browser bridge in
a background worker. Unknown consent attempts the same inline disclosure, then
conditionally enables analytics without overriding a saved decline. The first
evaluation waits up to two seconds for pending setup; asynchronous evaluations keep
the event loop free during that wait. A timeout lets evaluation proceed untracked
and invalidates late setup replies. There is no automatic retry or replay of the
missed operation. Use
`sf.analytics.enable()` to explicitly enable: it waits up to three seconds for confirmed browser
consent and a usable ID, then prints `Analytics enabled.` or
`Couldn't enable analytics; tracking remains off.` It returns `None`, so notebooks show only the confirmation message. Use
`sf.analytics.status()` for programmatic checks and detailed state. After successful return, the next evaluation
can be tracked without a separate polling cell. A timed-out acceptance is cancelled
locally so a late response cannot silently enable tracking.

Import/setup initiation and event delivery remain nonblocking. `sf.analytics.disable()`
stops local tracking immediately and returns `None`. It prints `Analytics disabled.`
and, while browser persistence is pending, `Saving your choice in the browser.`
Use `sf.analytics.status()` to inspect `pending` until acknowledged. Operations begun before consent is
confirmed are skipped, never replayed retroactively.

Colab uses the selected service's `/bridge/consent` iframe and cookie-checked
`/bridge/id/events` route. The runtime never receives the cookie itself or sends
events directly to `/v1/events`. The status has `scope="browser"` and an anonymous
`persistent_id`; `installation_id` is absent in meaning (`None`). No local consent
file is used. `persist=False` is unsupported in Colab because the server requires
browser consent for every batch. Process-only disable does not alter remembered
browser consent; normal disable requests a persisted decline. DNT suppresses all
requests, including decline persistence, so it cannot clear existing browser cookies.

The Colab queue holds at most 100 events, with one daemon worker and no retry.
Each initialization or control sequence shares a two-second Python budget, with
a shorter browser deadline for each command. Consent, DNT and cancellation are
rechecked before follow-on commands, including ID retrieval.
Blocked storage or browser errors fail closed; there is no runtime HTTP fallback.
Late replies cannot re-enable the kernel after opt-out. Already-dispatched requests
may complete; successful persisted decline prevents subsequent bridge delivery.
Cookie continuity is per browser profile/Colab partition, not cross-browser identity.
Safari may show a first-run Colab JavaScript-output warning, also reproduced without
this bridge. Real Colab background-worker verification remains a release gate.

The service must support `POST /bridge/consent/enroll` before this SDK's default
Colab enrollment can work. Older services leave new users off; explicit controls
remain compatible. Enabled preferences alone do not prove events reached reporting.
Failure diagnostics and automatic recovery are not part of this default change.

There is no identity linking. Events follow PostHog-managed retention under the
selected plan; ScreamingFace does not promise a fixed raw-event deletion deadline.
A query retention window is not a guarantee of physical deletion, and session
recording retention does not control these events. Disabling or resetting an
identifier does not delete previously accepted events. Production rollout still
requires verified IP handling and a historical-deletion procedure.
