Metadata-Version: 2.5
Name: firedrill-cloud
Version: 0.1.15
Summary: Firedrill Python SDK — authenticated client for managed test worlds. Your agent stays in your runner.
Project-URL: Homepage, https://firedrill.run
Project-URL: Documentation, https://docs.firedrill.run/sdk/python
Project-URL: Repository, https://github.com/firedrill-tools/firedrill-sdk-python
Author: Reload Tech Inc.
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: keyring<26,>=25
Requires-Dist: pydantic>=2
Requires-Dist: questionary<3,>=2.1
Provides-Extra: aiohttp
Requires-Dist: httpx-aiohttp>=0.2.0; extra == 'aiohttp'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: twine>=6; extra == 'dev'
Description-Content-Type: text/markdown

# Firedrill Python SDK and CLI

Firedrill gives your agent stateful synthetic Tools to work against, so you can
simulate tasks and check what actually changed. Tools run in Firedrill. Your agent,
application and model credentials stay in your own runner.

[Documentation](https://docs.firedrill.run/) · [Detailed guide](USER_GUIDE.md) ·
[Browser testing](CUSTOMER_BROWSER.md) · [API reference](https://docs.firedrill.run/api-reference/overview)

## Install

```sh
python -m pip install firedrill-cloud
firedrill init
```

Requires Python 3.10 or later. The package includes the `firedrill` CLI; Node is not required. `python -m firedrill_cloud` invokes the same client if another installation owns the command.

## Choose Tools

In an interactive terminal, `init` guides you through sign-in, project selection,
the Tool library and starting data. Then choose how to use them:

Use arrow keys and Enter for single-choice menus. In the Tool list, Space checks
or unchecks a Tool; Enter confirms one to eight Tools. You do not type Tool
numbers. Project names remain text input. If sign-in is needed, init asks
**Open browser to sign in** or **Cancel** before opening browser approval. A valid
login for this API is reused, and init shows your account email before setup.
Development also requires authentication. JSON, CI and non-terminal commands
never open menus.

- **Use normally:** get a reusable Tool environment and connect your existing agent.
- **Automated tests:** save Tools and data, then add tasks and checks. Each independent
  case gets isolated Tool state when you run it.

Neither choice uploads or launches your agent. Add more Tools later:

```sh
firedrill library list
firedrill tools add --tool LIBRARY_ID --initial-state empty --use reusable
```

Use an actual library ID; repeat `--tool` for more Tools. Choose `starter` for
available synthetic starting data. The command prints readiness and connection
links. Acceptance is not readiness: `tools setup wait --setup ID` observes it.

## Connect your agent

Inspect the ready setup's environment and its actual actor/copy choices first:

```sh
firedrill connect --environment YOUR_ENVIRONMENT_ID --json
firedrill connect --environment YOUR_ENVIRONMENT_ID --actor YOUR_ACTOR_ID --issue --format env
```

Plain `connect`, `--json` and `--format json` only inspect; they never wake Tools
or issue credentials. `--issue` explicitly permits wake and returns short-lived
HTTP or MCP connection values for your agent. If ready actor metadata is unavailable,
the client reports it as unknown; it does not guess permissions. With several copies,
select the exact `--actor-instance ID`. Keep
credential output private. Change your test configuration or endpoint variables,
not production agent logic. Only use interfaces supported by the selected Tool.
The web app is for viewing Tool apps, data, history and results; it does not run
your local agent.

Use Tools normally, inspect data, reset to starting state, or save tests.
Reusable environments keep evolving state; independent automated cases do not.
More than one named copy of the same Tool is supported.

Starting data can also be saved as versioned datasets and reused per Tool copy
or scenario. See [the detailed guide](USER_GUIDE.md#save-and-reuse-starting-data)
for selection and reset; saving data does not change a running connection.

## Run saved tests

Use your existing Tool/data-only setup, actual executable and actor, and a JSON
array of nonempty behavioral assertions you wrote for that Tool. No example agent
or passing check is supplied:

```sh
firedrill test inspect --from-setup YOUR_PARENT_SETUP_ID --json
firedrill test init --from-setup YOUR_PARENT_SETUP_ID --command python \
  --argument tests/run_agent.py --actor YOUR_ACTOR_ID \
  --instruction 'YOUR_REVIEWED_TASK' --checks checks.json
firedrill test validate --config firedrill.config.json
firedrill run preview --config firedrill.config.json
firedrill run --config firedrill.config.json
```

`test inspect` makes read-only authenticated setup/build reads. It returns exact
baseline actors and their scopes, named Tool copies, operation input/output schemas,
state schemas, routes, faults and declarative connection mappings. Baseline grants
are authored definitions, not live authorization. Records, actor attributes, tasks,
source code, credentials and recipe instructions are excluded. Missing copy metadata
is `null` (unknown), not an inferred default. An empty actor list requires explicit
scenario authoring; Firedrill never invents an operator.
For a named actor scope, init accepts explicit `--actor-instance ID` and verifies
the exact actor/scope pair before writing. This is not a default Tool-copy selector.

The native SDK offers `inspect_test_setup(client, project_id=..., tool_setup_id=...)`
and `await inspect_test_setup_async(client, ...)`, returning the same versioned JSON
projection. `TestInspectionError.code` is `invalid_request`, `invalid_response` or
`scope_changed`; errors do not expose a full world body or arbitrary transport cause.

For native authoring, `scaffold_test_definitions(id=..., target_id=..., actor_id=...,
instruction=..., bindings=["http"], checks=your_checks)` builds the same editable
source offline. Optional `actor_instance_id` retains an exact named actor scope.
It supplies neither checks nor an agent, and leaves authored assertion defaults
unchanged. Follow with `ToolSetupsClient.prepare_tests(parent_setup_id, test_definitions=definitions,
idempotency_key=...)` (or the async client) for authoritative selected-build
validation; retain and submit that exact intent before waiting for the saved setup.

Use `--argument=--model` for executable arguments starting with `--`. Init validates
and exclusively writes editable source/config without invoking your executable or
creating a saved setup. The named `starting-point` scenario inherits the selected
Tool/data baseline; edit it for your own cases. Existing files are never overwritten.
Preview saves the same reviewed source if needed, then reviews a service plan and
observed allowance/capacity without launching cases or your agent. Admission is not
a reservation. `test save` explicitly saves that source without running it.

The generated config keeps the original immutable `baseSetupId` plus `testSource`.
Edit the same logical test IDs and repeat validate/preview/run. Each revision derives
from that original Tool/data setup; an already test-bearing base is refused rather
than silently discarding its tests. Your ordinary connection is unchanged.

You can also run an already-saved test setup with a targets-only selection config:

```json
{
  "schemaVersion": 1,
  "drillIds": ["YOUR_SAVED_DRILL_ID"],
  "targets": {
    "YOUR_SAVED_TARGET_ID": {
      "command": "python",
      "arguments": ["tests/run_agent.py"],
      "bindings": ["mcp"]
    }
  }
}
```

Replace both IDs with actual saved test IDs and use your own executable.
The target receives task JSON on stdin and a scoped Tool connection in its
environment. Init uses `output: "text"` to retain real stdout; no TargetResult wrapper
is fabricated. For the explicit adapter above, JSON output must be an actual
TargetResult; logs go to stderr. Your executable is run directly, without a shell.
The [guide](USER_GUIDE.md) covers this test adapter,
endpoint-variable mapping and SDK callback alternatives.

```sh
firedrill run preview --setup YOUR_READY_SETUP_ID --drill YOUR_SAVED_DRILL_ID
firedrill run --setup YOUR_READY_SETUP_ID --drill YOUR_SAVED_DRILL_ID
```

Repeat `--drill ID` for several saved tests. Flags fill absent selection fields or
exact-confirm them; they never replace different IDs in a config. Alternatively pair
`--environment ID --build HASH`; this is exclusive of `--setup`.
The command prints the Results link and succeeds only when the simulation passes.
Agent completion or a browser check passing cannot override a failed Tool-state
check. Parallel cases and continuing sequences use the same Simulator and Results;
continuing steps deliberately retain evolving state.

For reviewed scenario/task/actor/check/fault choices, add `matrixSource` to the same
source-bearing config. A version1 matrix selects `drillIds` templates and named
`axes`/`choices`, each with explicit canonical `drill` substitutions. Optional partial
`exclude` selections remove combinations; `maxVariants` defaults to 100. Different
axes cannot replace the same field, and checks cannot be emptied. Stable hashed IDs
are identical across Python and TypeScript. Inspect it offline with
`firedrill test matrix --json`; `--output FILE` exclusively writes expanded definitions.
Validation, save, preview and run all use that same expansion.

## Inspect or cancel a shared simulation

Set `PROJECT` to your exact project ID and `BATCH` to the `cisuite_...` ID
printed by `run`:

```sh
firedrill simulation list --project "$PROJECT" --limit 20 --json
firedrill simulation get --project "$PROJECT" --batch "$BATCH" --json
firedrill simulation wait --project "$PROJECT" --batch "$BATCH" --timeout-ms 120000 --json
firedrill simulation cancel --project "$PROJECT" --batch "$BATCH" --confirm "$BATCH" \
  --reason "Stop this shared batch" --idempotency-key cancel-reviewed-batch-001 --json
```

List returns one bounded page; reuse `nextCursor` with unchanged filters.
Get and wait need no config or runner and never invoke your agent. Wait exits
0 when the batch is terminal, **not necessarily passed**; inspect
`batch.conclusion`. Timeout exits 2 and local interruption exits 130; neither
cancels remote work. `cleanupComplete: null` means unknown, and `false` means
cleanup is pending, even if the batch is terminal.

Cancel explicitly requests stopping the shared batch, unlike `run cancel` for
one native run. It privately saves the exact version, reason and key before sending.
An omitted version is captured once; retry never substitutes a newer version
under that key. After uncertainty, use the printed
`simulation cancel --resume PRIVATE_FILE` command with the original project
and API. Keep that file private.
CI requires an explicit project and cancellation key, or the exact resume file.
A `cancelling` response is acceptance, not completed cancellation.

## Use your existing Python test runner

`run_simulation` and `run_simulation_async` accept native typed options and actual
customer callbacks. Your adapter receives `SimulationInteractionContext`: the exact
task, read-only case/attempt assignment, scoped `binding`, cancellation event,
deadline and optional capture handles. Do not put the project credential in your
agent's Tool client. Select only protocols the saved target supports; optional
`target_bindings` is checked against reviewed case metadata before launch.

For pytest or unittest, keep your actual adapter in your own project:

```python
import os
from firedrill_cloud import Firedrill, run_simulation
from your_adapter import run_agent  # your callback, returning its actual TargetResult

def test_saved_behaviors():
    batch = run_simulation(
        Firedrill(token=os.environ["FIREDRILL_CREDENTIAL"]),
        project_id=os.environ["FIREDRILL_PROJECT_ID"],
        selection={"tool_setup_id": os.environ["FIREDRILL_TEST_SETUP_ID"],
                   "drill_ids": ["first-test"], "seeds": ["13"]},
        targets={"agent": run_agent},
        target_bindings={"agent": ["http"]},
    )
    assert batch.conclusion == "passed", f"{batch.conclusion}: {batch.results_url}"
```

An async test uses `AsyncFiredrill`, an async customer callback, and
`await run_simulation_async(...)` with the same native options. `unittest` can use
`self.assertEqual(batch.conclusion, "passed", batch.results_url)`;
`unittest.IsolatedAsyncioTestCase` supports the async form. Target completion is not
the verdict: assert the retained service conclusion and link the actual results.
Persist `on_checkpoint` snapshots privately if you need durable recovery; on
`SimulationRecoveryError`, retain `error.checkpoint` and resume that exact request.
Never blindly rerun a possibly started callback. Sync callbacks must cooperate with
the cancellation event/deadline; Python cannot safely hard-terminate arbitrary code.

## Browser tests and evidence

Use your browser driver with `customerBrowserTarget` (TypeScript) or
`customer_browser_target` / `customer_browser_target_async` (Python). The adapter
provides the exact case's connection, declared UI observations and optional logs,
screenshots, files and recordings. Your browser and app stay in your runner.
See [Browser testing](CUSTOMER_BROWSER.md) for preparation, cleanup and captures.

Results distinguish Firedrill's Tool-state checks from your runner's observations.
Evidence and captures are optional; choose what to record.

## Optional customer measurements

Your adapter can attach actual model-call usage, latency and customer evaluations
to its existing result. These measurements do not change Firedrill's native verdict.
Keep prompts, responses, credentials and raw traces out of the measurement fields.

```python
from firedrill_cloud import execution_metrics, model_call_metrics, with_execution_metrics

# Populate this only after your actual adapter has obtained its result and usage.
measured_result = with_execution_metrics(actual_target_result, execution_metrics(
    wall_time_ms=measured_elapsed_ms,
    model_calls=[model_call_metrics(
        id="actual-call-1", requested_model=requested_model,
        usage={"inputTokens": reported_input_tokens, "outputTokens": reported_output_tokens},
    )],
))
```

Missing measurements remain `None`, including cost; they are never interpreted as
zero spend. An empty `model_calls` list means no model calls were recorded, not that
the agent made none. Supply cost only when you have an explicit reported amount or
an estimate with its currency and `pricingDescription`; Firedrill does not infer
provider prices. Cached-input counts are subsets of input tokens, not extra tokens.

`import_opentelemetry_genai(otlp_json)` is an optional, bounded version1 projection
of current GenAI model-inference spans. It returns only allowlisted measurements,
call identities and value-free diagnostics; raw events, content and resource
attributes are not uploaded. Review its `issues` and `skippedSpans` before attaching
`modelCalls` with `execution_metrics(model_calls=...)`. This is not trace ingestion
or a provider billing attestation.

## Compare your actual model adapters

Comparison helpers return the versioned camelCase `SimulationComparison` projection
used by native benchmark checkpoints. The generated model remains available as
`firedrill_cloud.generated.types.SimulationComparison`; generated sync/async
`client.simulations_compare(project_id, batch_ids="ID,ID")` expose its native
snake_case attributes. Use `model_dump(by_alias=True, exclude_unset=True)` for the
same camelCase report with explicit unknown token/cost measurements preserved as
`None`, not zero. `get_simulation_comparison` / `_async` additionally validate exact
report scope, retained measurement coverage and the isolated 4 MiB response bound.

`preview_model_benchmark` and `run_model_benchmark` use the same saved build,
tests, explicit nonempty seeds and policy for 1–8 customer variants. Supply real
callbacks under the same target IDs; each callback selects and calls its model
in your own runner. Variants run in reviewed order, with ordinary per-case
concurrency inside each variant. Async forms have the same native options.

```python
from firedrill_cloud import preview_model_benchmark, run_model_benchmark
from your_adapter import run_model_a, run_model_b

options = dict(
    project_id=project_id, campaign_id="reviewed-models",
    selection={"tool_setup_id": test_setup_id, "drill_ids": ["first-test"], "seeds": ["13", "29"]},
    variants=[
        {"id": "model-a", "label": "Model A", "targets": {"agent": run_model_a}},
        {"id": "model-b", "label": "Model B", "targets": {"agent": run_model_b}},
    ],
)
review = preview_model_benchmark(client, **options)  # never invokes either callback
result = run_model_benchmark(client, **options)
print(result["comparisonUrl"])
assert all(item["batch"]["conclusion"] == "passed" for item in result["batches"])
```

Preview reports the whole campaign's actual required drill allowance and potential
maximum runner/model attempts. These are observations, not reservations. Run honors
service blockers and known total required allowance before its first remaining
launch; each launch still has ordinary service admission. Missing snapshots remain
unknown, and maximum attempts are not a client-side pricing cap.

Persist `on_checkpoint` snapshots privately. `ModelBenchmarkRecoveryError.checkpoint`
retains the exact campaign, per-variant preview intents and ordinary runner recovery.
Resume only with those original options and adapters; a possibly started callback
is not automatically invoked again. The SDK cannot serialize or inspect your callback
implementation: keep it unchanged yourself, and optionally identify reviewed settings
with a `configuration_digest`. The CLI additionally pins source and command settings
with an internal digest and refuses changed inputs before authentication.

For commands, put `schemaVersion:1`, `config:"firedrill.config.json"`, `campaignId`
and `variants` in `firedrill.benchmark.json`. Each variant supplies `id`, `label` and
real `targets` commands using the ordinary runner schema; optional `environment`
contains only nonsecret local settings such as `APP_MODEL`. Provider keys stay in
your customer environment and are never benchmark source or uploaded metadata.

```sh
firedrill benchmark preview --project "$PROJECT" --config firedrill.benchmark.json --json
firedrill benchmark run --project "$PROJECT" --config firedrill.benchmark.json --json
```

The retained comparison shows native verdicts separately from measured wall time,
recorded model calls, nullable usage/cost and explicitly supplied customer evaluations.
Firedrill neither ranks models nor runs a proprietary judge. `get_simulation_comparison`
and its async form read 1–8 exact batches without executing anything; missing/truncated
measurement coverage stays visible. The bounded comparison response limit is 4 MiB,
isolated to that exact read-only endpoint.

## Project API keys

Create or revoke a project key in **Settings → Credentials** in the web app, or
from the CLI after `firedrill login`:

```sh
firedrill credentials create --project "$PROJECT" --name "CI" --kind service --access automation
firedrill credentials list --project "$PROJECT"
firedrill credentials revoke --project "$PROJECT" --id cred_... --confirm cred_...
```

Creation shows the key once. Store it in your CI secret manager; never put it in
the repository. Use `--json` for non-interactive creation. Revocation takes
effect immediately and does not change completed results. Use `--access run`
instead when CI only executes already-saved tests; saving new Tool or test setups
also needs project-automation access.

## CI and coding agents

Use the same commands and configuration in CI. Put `FIREDRILL_CREDENTIAL` in its
secret manager and pass the project explicitly:

```sh
firedrill test validate --project "$PROJECT" --config firedrill.config.json --json
firedrill run preview --project "$PROJECT" --config firedrill.config.json --json
firedrill run --project "$PROJECT" --config firedrill.config.json --json
```

Use project-automation access when saving source; run-only access executes an
already-saved setup. Keep test definitions and
your runner config in the agent repository. CI never prompts or opens a browser.
Run tests on pull requests, pushes, schedules or manually.
The [guide](USER_GUIDE.md) covers the optional CI client and GitHub integration.

Interactive operations also have explicit arguments and JSON output for coding
agents. Help needs no login: `firedrill init --help`.

Optionally generate a reviewable ordinary CI file with
`firedrill test workflow --project ID --runtime python --client-version VERSION
--install-command 'YOUR_DEPENDENCY_COMMAND'`. Supply the exact client version you
tested; no floating install is guessed. The same config/source is used in CI.
`--schedule` is explicit opt-in, fork PRs skip secret-bearing execution, existing
files are never overwritten, and private recovery/session files are never uploaded.
The command only writes source; review it and wire your customer model keys privately.

## Recovery and advanced controls

Interrupted setups or runs print a recovery file. Resume the exact request instead
of starting another one; project, Tool versions and request key stay unchanged.
Local files under `.firedrill/cloud/` are private and Git-ignored. Credentials
belong in the OS keychain or your secret manager.

The runner reads authoritative work before retrying a retained pre-start action.
Expired or withdrawn work releases its local slot only after service cleanup;
the same batch can then assign another case. A possibly started agent invocation
is never automatically replayed. Stopping your runner does not cancel the shared
batch. Keep the checkpoint; another runner can explicitly join the known batch
with `firedrill run --batch BATCH_ID --config runner.json --project PROJECT_ID`.
That config contains only the real target commands, capacity and capture choices;
joining never selects new cases or changes the original batch.

`cancel_simulation` / `cancel_simulation_async` explicitly cancel the shared batch
using its observed `expected_version`, reason and exact `idempotency_key`. A
`SimulationCancellationError` exposes a detached private `request`, including the
original API origin. Retain and retry those exact inputs after an uncertain reply;
do not create another cancellation intent or print the private reason.

Dataset SDK errors distinguish a definitive `DatasetRejectedError` from an
uncertain `DatasetRecoveryError`. The rejection exposes `code`, `status_code`,
`issues` (field paths and repair suggestions) and `to_dict()` for diagnostics.
Correct a rejected request before preparing a new intent. For an uncertain
response, submit the original retained `intent` and idempotency key. Do not print
the intent: it contains your submitted records, although credentials are omitted.

The [detailed guide](USER_GUIDE.md) covers SDK callbacks, custom Tool publication,
scenario overrides, time advancement, full/selective resets, snapshots,
cancellation, history, evidence, concurrency and exact-request recovery.
