Metadata-Version: 2.4
Name: crossgl-dyson
Version: 1.0.1
Summary: CrossGL Dyson authenticated CPU and GPU compute routing SDK
Home-page: https://github.com/CrossGL/dyson
Author: CrossGL team
Author-email: nripesh@crossgl.net
License: MIT
Project-URL: Documentation, https://crossgl.net/docs/dyson
Project-URL: Product, https://crossgl.net/products/dyson
Project-URL: Source, https://github.com/CrossGL/dyson
Keywords: crossgl,compute,serverless,gpu,routing
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: dill<0.5,>=0.4
Requires-Dist: numpy<3,>=2.0
Requires-Dist: requests<3,>=2.32
Provides-Extra: ml
Requires-Dist: torch>=2.0; extra == "ml"
Requires-Dist: tensorflow>=2.13; extra == "ml"
Provides-Extra: service
Requires-Dist: fastapi<1,>=0.116; extra == "service"
Requires-Dist: google-auth<3,>=2.40; extra == "service"
Requires-Dist: google-cloud-firestore<3,>=2.21; extra == "service"
Requires-Dist: google-cloud-storage<4,>=3.2; extra == "service"
Requires-Dist: pydantic<3,>=2.11; extra == "service"
Requires-Dist: uvicorn[standard]<1,>=0.35; extra == "service"
Provides-Extra: dev
Requires-Dist: pre-commit>=3.5; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-mock>=3.14; extra == "dev"
Requires-Dist: httpx<1,>=0.28; extra == "dev"
Requires-Dist: ruff>=0.12; extra == "dev"
Provides-Extra: docs
Requires-Dist: breathe>=4.35; extra == "docs"
Requires-Dist: furo>=2024.1.29; extra == "docs"
Requires-Dist: myst-parser>=2.0; extra == "docs"
Requires-Dist: sphinx>=7.2; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=2.0; extra == "docs"
Requires-Dist: sphinx-copybutton>=0.5.2; extra == "docs"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Dyson

Dyson is CrossGL's authenticated compute-routing SDK and service. It accepts a bounded
Python workload, selects an explicit CPU or NVIDIA L4 profile, runs one isolated Cloud
Run Job, and settles measured usage against the user's shared CrossGL Ledger account.

## Install the SDK

The published distribution is **`crossgl-dyson`** and the Python import remains
**`dyson`**:

```bash
python3 -m pip install "crossgl-dyson==1.0.1"
```

The first public GA package is `1.0.1`. The immutable `v1.0.0` tag identifies an
earlier reviewed release candidate that was never published; never move or reuse that tag.

Do not install the unprefixed `dyson` distribution from PyPI. That name is owned by an
unrelated Selenium framework and is not a CrossGL artifact. A CrossGL wheel must report
`Name: crossgl-dyson`, contain the `dyson` import package, and pass
`scripts/verify_distribution.py` before publication.

For development from a reviewed checkout:

```bash
python3 -m pip install -e .
python3 -m pip install -e ".[ml]"       # local ML workload dependencies
python3 -m pip install -e ".[service]"  # API, repository, and control plane
python3 -m pip install -e ".[dev]"      # tests and lint tooling
python3 -m pip install -e ".[service,dev]"
```

### Package publication

`.github/workflows/publish-package.yml` is the only reviewed PyPI publication path. It is
manual-only and must be dispatched from protected `release/dyson-ga`; the exact version
must match `dyson/_version.py`, an existing immutable `v<version>` tag must resolve to the
event SHA, and the operator must enter `PUBLISH_CROSSGL_DYSON_<version>`. The unprivileged
build job runs the full test/lint/format suite, uses exact build tools, produces one wheel
and one source distribution, runs strict Twine and offline archive/RECORD/license checks,
and uploads their SHA-256-bound private artifact. A separate `dyson-pypi` environment job
downloads that artifact and receives OIDC only for the pinned PyPA publishing action; no
source checkout, package code, API token, `skip-existing`, or long-lived credential is
present in the publish job. Publication is not accepted when upload returns: a third
credential-free, no-OIDC job waits at most five minutes for public-index convergence,
redownloads the exact wheel and sdist, reruns the offline archive gate, validates both
PyPI Integrity API attestations against the exact `CrossGL/dyson` workflow/environment
publisher, and performs a dependency-complete install/import in a new virtual environment
through `https://pypi.org/simple`. It archives a sanitized
`package-publication.json` artifact for 30 days; release evidence is incomplete unless
that job passes.

Before the first publication, configure a PyPI pending Trusted Publisher for project
`crossgl-dyson`, owner `CrossGL`, repository `dyson`, workflow
`publish-package.yml`, environment `dyson-pypi`. Protect that GitHub environment with
required reviewers and permit only `release/dyson-ga`. Build locally without publishing:

```bash
rm -rf build dist
python3 -m build
python3 -m twine check --strict dist/*
python3 scripts/verify_distribution.py --dist-dir dist --expected-version 1.0.1
```

Never upload `dist/*` with a developer token or rename an artifact to the occupied
`dyson` distribution. PyPI and GitHub publication remain external mutations and require
separate operator approval.

## SDK configuration

Dyson never requires credentials at import time. The production SDK uses these settings:

| Variable | Purpose |
| --- | --- |
| `CROSSGL_API_KEY` | A CrossGL API key created in the shared account dashboard. |
| `DYSON_SERVERLESS_API_URL` | Dyson API origin. Defaults to the production Cloud Run service. |

`DYSON_API_KEY`, `DYSON_API`, and `CGL_KEY` remain accepted as credential aliases for
older clients. Client-side model-provider credentials, API-key verification, and credit
deduction are not part of the GA contract: the authenticated API routes and meters every
job server-side.

## Submit a workload

```python
import os

import dyson

os.environ["CROSSGL_API_KEY"] = "cgl_example_replace_me"


def workload(x, y):
    return x + y


submission_key = dyson.create_idempotency_key()
# Persist submission_key with your operation before the request if the process may restart.
job = dyson.submit_job(
    workload,
    2,
    3,
    profile="cpu-balanced",
    idempotency_key=submission_key,
)
terminal = dyson.wait_for_job(job["job_id"], timeout=600)
result = dyson.get_job_result(terminal["job_id"])
print(dyson.decode_result(result["result"]))
```

For Python-version-stable payloads, submit source and JSON arguments:

```python
job = dyson.submit_source_job(
    "def workload(x, y):\n    return x + y\n",
    "workload",
    2,
    3,
    profile="cpu-small",
    idempotency_key="billing-run-2026-08-29-001",
)
```

Every `POST /v1/jobs` requires an 8–128 character `Idempotency-Key`; the Python
SDK generates one when omitted, but callers that need process-restart recovery should create
and durably store it before submission with `dyson.create_idempotency_key()`. If a transport,
HTTP, or response-decoding failure makes the outcome ambiguous, `DysonClientError.idempotency_key`
retains the exact generated identity without including it in the error message. Retry the exact
payload with that value rather than creating another logical submission.

Reuse an identity only for retries of the same logical payload. Dyson derives an account-scoped
stable job ID, persists the request fingerprint, a high-entropy private launch identity, and whether
that job was key-protected, and returns 409 if the key is later paired with different content. A
private staged submission record lets a retry reconstruct an exact already-uploaded payload and
atomically claim a job that was persisted before Cloud Run was requested. The private identity is
included in the execution override, never in public job responses, and is removed from the worker
environment before user code starts. Once the claim is durable, another request may recover the
one execution but never issue a second run.

A persisted submission remains retryable in `preparing` for exactly five minutes. The create
transaction checks the account-scoped job identity before enforcing the concurrency counter, so an
exact simultaneous replay that observes the winner in the final slot returns that job and cannot
cancel their shared idempotent Ledger hold. If the create-only payload already exists, replay verifies
its exact bytes under a caller-owned 20-second deadline with provider retries disabled. A definitive
metadata/download disappearance or generation-precondition race, API/auth/transport/checksum failure,
or deadline exhaustion returns retryable 503 with `Retry-After: 60` without launching compute, claiming
preparation failure, cancelling billing, or releasing capacity; only authoritative different bytes are
a conflicting artifact. A successful Firestore create with a lost response is
recovered by an authoritative read before Dyson can cancel the bounded hold; if that read is
unavailable, the hold is preserved for an exact replay instead of risking unbilled compute. If
that path exists, its embedded account and job identities must exactly match the account-scoped
Firestore collection and document requested by the authenticated caller. Every account-scoped
private-job read binds those identities before returning state, and every transactional launch,
cancellation, grace, settlement, pending-reconciliation, conflict, and terminal write revalidates
them. A drifted status/result/cancellation read therefore fails before opportunistic API
reconciliation can call Cloud Run or Ledger, and terminal settlement cannot delete a pointer or
decrement capacity from a substituted job path. Drift is retained for incident investigation; it is
never repaired by copying embedded identities or moving billing state.

If payload preparation or URL signing fails, cleanup must first
atomically claim `preparation_failed` against `launch_requested`. A concurrent launch winner keeps its
hold and active pointer, while a killed failure cleanup resumes hold cancellation and slot release from
the durable claim. If the caller never retries, scheduled reconciliation similarly claims
`preparation_expired` against the launch transaction, then releases its hold and concurrency slot; a
killed cleanup resumes from that durable state. The same idempotency key then truthfully returns the
terminal failure, and a new logical submission needs a new key. Once launch is claimed, recovery follows
Cloud Run page tokens through at most ten 100-execution pages per scan and accepts only the exact
worker parent, job ID, private launch identity, user-code timeout, persisted task timeout, and a
creation time consistent with the durable claim. The task timeout is the user-code timeout plus a
five-minute controller reserve for payload download, child-process teardown, serialization, result
upload, and clean exit. Normal operation responses and already-persisted execution names receive
the same identity proof as ambiguous-history recovery. `None` means the list was exhausted with no
match; an API outage, malformed or cyclic page, still-present token at the bound,
conflicting identity/timeout, or multiple match remains `pending_reconciliation` and
cannot release or capture the reservation. Exhaustive absence after the launch grace
period is the only no-execution release path. Ledger returns an authoritative `200`
reservation for already `cancelled` or `expired` holds and a `409` for captured or any
other cancellation conflict. A raw `409` can never relabel billing or remove capacity.
Dyson instead performs the existing account/job-scoped, read-only Ledger settlement
audit and requires exact reservation identities, funding, amounts, chronology, totals,
and status derivation. An exact all-released audit converges to `billing_status=cancelled`;
only the scheduled reconciler may apply an exact all-captured or captured-plus-released audit, and it
terminalizes the job as `billing_status=conflicted`, preserves every authoritative nested status and
captured total, emits the aggregate managed incident, and removes the active pointer only in that
terminal transaction. API reads keep even an audit-proven terminal conflict active until that
scheduled attempt. This makes an irreversible billing incident visible without falsely calling it
cancellation or permanently consuming an account slot. A pending, malformed, unavailable, or
substituted audit keeps the durable job and active pointer in `pending_reconciliation`. Its private
schema-v1 conflict record preserves first/last observation, monotonic attempt count, and
released/total aggregate counts without reservation or account identities. The exact
immutable reconciler setting is 900 seconds: a terminal conflict alerts immediately,
while an unresolved conflict receives the dedicated billing-integrity alert once its
preserved age reaches that bound. The same transaction that ages a live conflict maintains its exact
private `dysonCancellationConflicts` index record. Before any scheduled external reconciliation,
the immutable reconciler also checks a pre-existing nested conflict against that index and atomically
creates only a missing exact row; malformed, substituted, terminal, or drifted state fails the run.
This closes legacy jobs created before transactional indexing without calling Ledger or changing job,
billing, or capacity truth. Terminal convergence atomically deletes that open
record, creates one immutable `dysonCancellationConflictHistory` record, records the authoritative
Ledger-audit observation plus API-versus-scheduler source on the private job, and only then releases
the active pointer. Operators use the bounded read-only inspector below; they must never scan through
a mutating reconciliation helper, delete the active pointer, or rewrite reservation state. Repository
validation rejects any terminal write whose job status is nonterminal, whose settled
billing fields are absent or pending, whose per-reservation states disagree with the
top-level captured/cancelled truth, or whose conflicted state lacks the exact audit marker.
Both the exhaustive-absence branch and a definitively failed Cloud Run launch operation
persist terminal billing in the same transaction that removes the active pointer.
Every successful
`POST /v1/jobs` response includes `idempotency_protected` and request-specific
`submission_replayed`; the latter is `false` for the one creation path and `true` when an existing
job is returned or resumed, including a transaction race. `GET` and list responses retain
`idempotency_protected` but omit the request-specific replay field. Raw idempotency keys,
fingerprints, launch identities, private submission stages, settlement intents, and result digests are
never public. Cancellation first commits an exact intent against the live execution; only then does
Dyson call Cloud Run. A lost response is retried by reconciliation, while a concurrently terminal
or already-settling job cannot be overwritten back to `cancelling`. Cloud Run's single-task terminal
counters are provisional until one unambiguous counter is paired with complete `createTime`,
`startTime`, and `completionTime`; missing, contradictory, or backwards timing remains
`pending_reconciliation` and can never fall back to an invented billable duration. Authenticated job
responses expose `execution_started_at` only when that exact provider timestamp has become
authoritative. Because Cloud Run may withhold the active projection until terminal convergence, the
release runner observes it for a bounded 30 seconds, then retries cancellation only until the exact
execution is schedulable and finally requires the terminal provider start timestamp to precede the
accepted cancellation request (within the five-second clock-skew bound). A delayed projection can no
longer create a false acceptance failure, while a cancellation that actually preceded compute start
still fails the metered-cancellation proof.
The worker's one
schema-v1 result is launch-bound and create-only. A success envelope is accepted only with the exact
bounded key set, canonical `python-dill-base64` payload, finite reported duration, bounded text, and
an authoritative Cloud Run `succeeded` task; a failure envelope requires its exact bounded error form
and a Cloud Run `failed` task. Because the exact worker exits zero only after publishing success and
raises after publishing failure, either cross-state contradiction is definitively invalid. A started
Cloud Run cancellation is the separate metered race and may retain either already-published envelope.
Duplicate keys, extra fields, malformed base64, a nominal `ok: true` without an encoded result, or a
success/failure bit that contradicts terminal task truth are treated as no valid result and cannot
authorize capture. On a first terminal observation without a valid result, Dyson durably records
that observation no earlier than authoritative completion and an expiry exactly 120 seconds later.
Every started resultless settlement must retain that complete first-observation state and claim no
earlier than its expiry; an exact no-start cancellation is the only resultless decision that carries no
publication grace. A result generation may settle after the window only when Cloud Storage's
provider-authored creation time proves that exact immutable generation existed no later than the
expiry; an object created even fractionally after the deadline is treated as absent and can never
authorize capture. The persisted observation remains immutable. Dyson then commits one immutable
schema-v2 decision before Ledger: the exact key set, execution and profile identities,
completion/claim chronology, result digest, canonical GCS generation, provider creation time, and
success/workload-failure/infrastructure-release/cancellation combination must agree. Resultless
outcomes carry explicit null artifact provenance. The same dependency-free contract validates
creation, replay, every persisted active or terminal read, and release evidence. A contradictory old
intent produces retryable operator-review state rather than another capture. Process-exit replay can
therefore perform only the frozen money movement. `/result` serves bytes only when the terminal
record says a result was present. It selects that frozen GCS generation directly instead of resolving the
live object name, and bounds its metadata plus payload work under one shared 20-second deadline before
revalidating the account/job URI, digest, generation, provider creation time, and envelope. A late or
substituted object is never exposed. A metadata-to-payload deletion or generation-precondition race is
treated as an absent exact generation. `/result` starts one caller-owned monotonic deadline before
opportunistic reconciliation; metadata and payload reuse only its remainder and disable client retries.
Google API/auth failures, Requests/urllib3/stdlib transport failures,
raw socket/SSL/DNS errors, checksum corruption, and deadline exhaustion take the typed unavailable
path; local `ValueError`/`TypeError` and artifact metadata violations are never reclassified as transient
I/O. Reconciliation keeps the reservation pending without starting result grace, release inspection
records an unreadable generation and cannot clear, and a committed result request returns retryable 502
with `Retry-After: 60`; no raw Storage exception becomes an untyped API 500. If the committed
generation is absent before its provider-derived age-31 UTC lifecycle-eligibility boundary, `/result` returns
that retryable 502 so the API server-error policy can alert rather than misreporting an ordinary missing
result. Once the first midnight UTC at provider creation date plus 31 calendar days has passed, absence is expected
and returns 410.
A caller that loses a response can therefore repeat the exact request, distinguish replay from creation, and
continue polling without reserving or launching a second job.

The Python SDK keeps result lifecycle failures distinct. A retention-eligible committed generation that is
unexpectedly unreadable raises `DysonResultTemporarilyUnavailableError`; its raw `retry_after` and parsed
`retry_after_seconds` preserve the server's retry instruction. Once the provider-derived lifecycle has
expired, the same request raises non-retryable `DysonResultExpiredError`. Both inherit
`DysonResultError` and `DysonClientError`, so existing broad handlers remain compatible:

```python
try:
    result = dyson.get_job_result(terminal["job_id"])
except dyson.DysonResultTemporarilyUnavailableError as exc:
    # Schedule a bounded retry after the server's delay; keep the same job ID.
    retry_in = exc.retry_after_seconds or 60
except dyson.DysonResultExpiredError:
    # This job's artifact is permanently unavailable. Do not retry this result URL.
    result = None
```

Dyson guarantees lifecycle protection for at least 30 elapsed days after provider creation.
The exact provider deletion-eligibility boundary is midnight UTC on the creation date plus 31
calendar days; retrieval is not promised after that boundary. Download and retain any result your application needs for longer under your own data policy;
submitting a replacement workload is a new billable job, not a recovery of the expired artifact.

## Routing compatibility

`DysonRouter` remains available for existing applications, but it no longer performs
client-side LLM voting or billing. `route_hardware()` calls the authenticated `/v1/estimate`
contract and returns the server profile, price ceiling, and routing reason. Historical
`hardware_type` and `spec` keys are retained as aliases.

```python
router = dyson.DysonRouter()
decision = router.route_hardware(workload, complexity="low")
print(decision["profile"]["name"], decision["maximum_credits"])
print(router.run(workload, 2, 3, hardware=decision))
```

`dyson.run()` is the functional compatibility wrapper over the same serverless API:

```python
compiled = dyson.run(workload, target_device="c4cpu")
print(compiled(2, 3))
```

The CPU worker image supports Python and NumPy. The GPU image supports PyTorch on CUDA.
C++, TensorFlow, JAX, multi-device jobs, and managed training fail before submission until
dedicated tested worker images exist; they must not be advertised as GA runtimes.

## Production topology

The deployment creates separate immutable images and identities for:

- `dyson-api`: authenticated public control plane; owns Firestore metadata, signed
  artifact URLs, job launch/cancellation, and Ledger calls.
- `dyson-worker-cpu-small`, `dyson-worker-cpu-balanced`, and
  `dyson-worker-cpu-highmem`: credential-free CPU jobs with profile-specific limits.
- `dyson-worker-gpu-l4`: credential-free CUDA/PyTorch worker with one NVIDIA L4. Cloud Run caps
  GPU tasks at 3,600 seconds, so Dyson exposes a 3,300-second user-code maximum and preserves the
  final 300 seconds for startup, immutable result publication, and settlement-safe exit.
- `dyson-reconciler`: private scheduled job that derives terminal state from Cloud Run
  and idempotently captures or releases Ledger reservations. It scans at most 100 pointers on a
  private document-ID ring under a 75-second crash lease, a 45-second monotonic work budget, and a
  60-second task hard timeout. The final five seconds are reserved for committing only the fully
  processed cursor prefix, so deferred work resumes next minute instead of losing a whole page.
  Structured output reports checked/scanned/deferred counts, deadline and cursor progress, pending
  settlement, exact-orphan cleanup, and identity-free durable lease-contention streaks/totals. A
  zero-progress deadline or second consecutive `lease_busy` run fails for release alerting; neither
  the cursor nor lease owner is exposed.

Workers receive only short-lived signed Cloud Storage URLs. Result PUT signatures require generation
zero, and the worker envelope includes the exact private launch identity; an execution therefore
cannot overwrite or substitute a settled result. Worker, repository, and parser import one immutable
result contract: a 6 MiB envelope, 4 MiB decoded payload, 64 KiB captured stdout/stderr plus the exact
truncation suffix, and an 8 KiB UTF-8 error. Runtime environment variables cannot drift those trust
boundaries. Repository reads first inspect object metadata, reject a proven oversized object without
downloading it, and bind every download to the observed generation; missing-size reads are range
capped at one byte beyond the limit. A definitively oversized or malformed immutable result enters
the same persisted 120-second invalid-result grace and then releases the hold, while a transient
storage or precondition failure remains pending and cannot authorize billing.

Each URL covers the exact 30-minute launch grace, user-code timeout, and five-minute controller
reserve; the Cloud Run task has that same persisted controller reserve while the child process still
enforces the user-visible timeout. Started executions require complete ordered create/start/completion
timing. The sole exception is an exact one-task cancellation with ordered create/completion metadata
and no `startTime`: it proves zero compute started, bypasses result lookup, records zero duration, and
releases the hold as `cancelled_before_start`. A started cancellation remains metered with the
one-minute minimum. Ledger holds cover the complete submission/start/task window, the exact 120-second
result-publication grace, the one-hour post-execution settlement window, the 120-second reconciler
recovery allowance, at most 60 seconds of bounded reservation work before Firestore persistence, and
a final 30-second Ledger settlement dispatch. The API readiness gate rejects any configured lifetime
that exceeds Ledger's 24-hour maximum.
Reported actual credits are capped to the amount reserved even when trusted Cloud Run wall time
includes controller overhead. Workers use a service account with no project roles and direct VPC
egress through a dedicated network that permits only restricted Google APIs; general internet
egress is denied. Payloads and results remain lifecycle-protected for at least 30 elapsed days and become
delete-eligible only at the age-31 UTC boundary. See
[`docs/SERVERLESS_ARCHITECTURE.md`](docs/SERVERLESS_ARCHITECTURE.md).

The deployment workflow is intentionally manual. Select `sandbox` or `production` and pass a
positive, immutable Secret Manager version; `latest` is rejected. The channels are isolated:

| Boundary | Sandbox | Production |
| --- | --- | --- |
| API | `dyson-api-sandbox` | `dyson-api` |
| Ledger | `crossgl-ledger-sandbox` | `crossgl-ledger` |
| Billing secret | `CROSSGL_BILLING_SERVICE_KEY_SANDBOX` | `CROSSGL_BILLING_SERVICE_KEY` |
| Firestore | `dyson-sandbox` named database | `(default)` |
| Artifacts | `dyson-backend-dyson-jobs-sandbox` | `dyson-backend-dyson-jobs-production` |
| Browser origins | exact website preview + loopback | `crossgl.net` + `www.crossgl.net` |

Cloud Run cannot create a service with `--no-traffic`. On a channel's first release, the workflow
therefore proves the canonical service is absent, creates one private non-ready holding revision with
no billing secret or worker bindings, and then deploys the real tagged candidate at zero traffic. The
holding revision exposes no usable Dyson release; any failure deletes the current-run service, paused
schedule, and immutable jobs. Existing services never take this bootstrap path, and an inability to
distinguish absence from a permission or API failure stops before mutation.

Each release creates commit-addressed worker and reconciler jobs, a paused commit-addressed
scheduler, and a digest-pinned API revision with no traffic. The candidate must pass health,
readiness, authentication, exact CORS, IAM, secret, image, resource, network, DNS, scheduler,
Firestore, and at-least-30-day age-31 UTC artifact-lifecycle checks before that exact revision receives 100% traffic.
Schema-v4 candidate and active topology evidence records the API revision's exact 300-second
submission grace, 1,800-second start grace, 300-second worker overhead, 120-second result-publication
grace, and 3,600-second settlement grace alongside the observed bucket's exact Delete action,
age 31, `jobs/` prefix, UTC/30-day interpretation, and disabled retention-policy, versioning, and
soft-delete state. Candidate and active records must carry the same policy; downstream recovery,
alert, conflict, restore, and identity gates reject a legacy or drifted topology before doing release
work. Topology inspection also requires an authoritative provider creation time and positive bucket
metageneration. A failed post-promotion check restores the previous
revision and scheduler. Sanitized candidate and active evidence is retained as a GitHub Actions
artifact for 30 days.
External actions are pinned to reviewed full commit SHAs, hosted jobs use Ubuntu 24.04, and the
release Cloud SDK is pinned to `576.0.0`; update those pins and their contract tests together.

Production additionally requires `confirm_production=DEPLOY_PRODUCTION` and refuses to begin if
the identical full Git commit is not already the sole healthy sandbox revision. Record the
sandbox API URL after its first deployment and configure the website preview's public
`DYSON_SANDBOX_API_URL` repository variable to that exact HTTPS origin. Do not reuse the
production Ledger key in the sandbox secret.

Static cloud infrastructure is a separately authorized one-time operation. The routine release
workflow is intentionally unable to enable APIs, create service accounts/VPC/DNS/firewall/storage/
Firestore resources, or edit project, secret, bucket, or service-account IAM. Before its first
image build it runs `scripts/verify_release_prerequisites.py`, which proves the WIF pool contains
only the exact reviewed sandbox/production providers and claims, keyless release/runtime identities,
release-role allowlists, runtime IAM, repository/bucket/subnet access, pinned secret version,
network/DNS/firewall, data stores, and required APIs. Its sanitized, mode-0600 result is archived
beside candidate and active topology evidence.

`scripts/provision_release_infrastructure.py` is the reviewed operator path for those static GCP
resources. It is plan-only by default and makes no external calls. It never accepts a billing key;
create and review `CROSSGL_BILLING_SERVICE_KEY_SANDBOX` and
`CROSSGL_BILLING_SERVICE_KEY` out of band first. Inspect the plan before using the explicit apply
mode (these commands mutate GCP and require a separately authorized administrator):

```bash
python scripts/provision_release_infrastructure.py \
  --channel sandbox \
  --billing-secret-version 1 \
  --notification-channel \
  'projects/dyson-backend/notificationChannels/<deliverable-on-call-channel-id>'

python scripts/provision_release_infrastructure.py \
  --channel sandbox \
  --billing-secret-version 1 \
  --notification-channel \
  'projects/dyson-backend/notificationChannels/<deliverable-on-call-channel-id>' \
  --apply \
  --confirm PROVISION_DYSON_SANDBOX_INFRASTRUCTURE \
  --confirm-active-account '<active-admin-account>'
```

Provision production only after reviewing the sandbox infrastructure and use the distinct
`PROVISION_DYSON_PRODUCTION_INFRASTRUCTURE` confirmation. Apply is idempotent for absent resources
and additive grants, refuses to reinterpret permission/API errors as absence, validates existing
provider/service-account/key/IAM state before adding any sensitive grant, removes only the known
legacy API `roles/run.developer` grant, and finishes with the same exact verifier used by CI.
Unknown IAM or monitoring drift is never deleted or rewritten automatically; the verifier fails for
human review. Keep the operator identity out of GitHub Actions and remove its elevated session after
provisioning.

Before running even the plan, an operator must create and test one real Cloud Monitoring notification
channel out of band. Its display name must be `Dyson release on-call`; it must be enabled, report
`VERIFIED`, use an approved human on-call route (email, Google Chat, PagerDuty, Slack, SMS, or an
authenticated webhook), and have exactly the user labels `product=dyson,purpose=release_alerts`. The
provisioner accepts only that channel's full `projects/dyson-backend/notificationChannels/...`
resource name. It never accepts the destination address/token and never creates, updates, disables,
or deletes a notification channel. Verify delivery with the selected provider before approving a
release; an API-visible `VERIFIED` state does not replace a human delivery test.

For each channel, apply creates only missing resources after proving that every existing reserved
resource is exact: a one-minute multi-region HTTPS `/readyz` check that requires HTTP 200 and the
exact ready response, plus enabled `ERROR` policies for five-minute readiness loss, any five-minute
Cloud Run API 5xx count, any failed immutable reconciler execution, a structured consecutive
lease-contention streak of at least two, billing cancellation conflicts from only the exact
channel reconciler name, and persisted-job identity incidents from only that channel's API or
reconciler. The billing-conflict policy alerts immediately for an exact audit-proven
terminal `conflicted` job or once an unresolved active conflict reaches the immutable 900-second age
bound. The cancellation-conflict and persisted-identity policies direct operators to bounded private
inspectors; the identity policy matches both the exact channel API service and immutable reconciler.
All three matched-log policies are rate-limited to one notification per five minutes and expose only
aggregate state. Every incident notifies the one reviewed on-call channel and auto-closes only
after 30 minutes. The service name, canonical host,
reconciler-name regex, matcher, thresholds, runbook text, regions, labels, notification routing, and
incident strategy are all fail-closed. Routine releases receive only `roles/monitoring.viewer`; they
list and validate these controls before the first image build and archive a sanitized schema-v3
prerequisite record containing names/counts and only a SHA-256 of the channel resource. They cannot
create or edit monitoring resources.

### Persisted-job and active-pointer integrity incident discovery

Every runtime read binds the account/job fields inside a private job to its exact
`dysonUsers/{account-hash}/jobs/{job-id}` path. It then validates the reverse lifecycle relation:
`queued`, `running`, `cancelling`, and `settling` jobs must have the exact canonical
`dysonActiveJobs/{job-id}` pointer for the same account; terminal jobs must have no pointer; and an
unknown status is never trusted. Status membership alone is not active-job proof. A queued row must
be in exactly one prelaunch phase (`preparing`, `launch_requested`, `preparation_expired`, or
`preparation_failed`) with no Cloud Run execution, cancellation, result-grace, or settlement lineage;
a launch-requested row must carry its durable positive claim time. A running row must be `launched`,
retain that launch claim, and carry a nonempty operation or execution identity. A cancelling row must
also carry the exact execution plus a positive cancellation-request time and monotonic optional
dispatch time. A settling row must carry the launch and execution identity plus either complete,
ordered immutable result-grace evidence or one exact immutable schema-v3 settlement intent. No
status-active row may carry terminal completion, result, duration, failure, or settlement-completion
fields. Any impossible combination raises a fixed, identity-free phase incident while retaining the
job, pointer, billing state, and capacity.

Job creation replay, direct get/list reads, and transactional launch,
preparation, cancellation, result-grace, settlement, conflict, pending, and terminal updates all make
this check before a lifecycle write. Direct API reconciliation therefore fails before Cloud Run or
Ledger work, and terminal settlement cannot hide a missing pointer while leaving account capacity
stuck.

Every active-pointer read independently binds its document ID to its embedded job/account fields
before loading a job, calling Cloud Run or Ledger, deleting a pointer, decrementing capacity, or
advancing a release completeness cycle. A missing, non-string, substituted, or otherwise malformed
pointer is retained and raises the same value-free integrity incident; it is never treated as cleanup.
An exact pointer must reference a job whose lifecycle is still `queued`, `running`, `cancelling`, or
`settling` **and** whose phase lineage satisfies the same prelaunch, launch, execution, cancellation,
result-grace, or settlement contract. A terminal, unknown-status, or structurally impossible active
target is retained, fails the immutable reconciler before external work or completeness advancement,
and is surfaced by the bounded audit instead of silently consuming capacity. Only an exact pointer whose
referenced job is authoritatively absent may use the
transactional orphan repair. Admission accepts only an absent-or-canonical nonnegative integer
`active_job_count`; booleans, strings, fractions, negatives, and values above the bounded audit limit
fail with the same value-free integrity incident. Terminal settlement and orphan cleanup require that
an existing pointer can decrement at least one slot, and retain every job/pointer/billing record if the
counter would underflow.

API reads return a retryable 503 with `Retry-After: 60`; API and reconciler processes emit the same
identity-free structured event. The exact channel-scoped managed policy alerts on path/payload drift,
missing reverse pointers, invalid lifecycle state, structurally impossible active phases, and
non-active pointer targets without placing a path, account, job, cursor, or embedded value in Cloud
Logging. An integrity failure abandons the immutable reconciler lease without cursor or full-ring
advancement. A persisted active job with no
pointer remains outside that ring, while a terminal row with no pointer cannot be discovered there;
release approval therefore requires the complete read-only bidirectional audit below.

Wait through the 120-second retired-writer boundary after active-topology verification, then run the
auditor from the exact reviewed Dyson source:

```bash
python scripts/inspect_job_identities.py \
  --channel sandbox \
  --source-commit '<full-reviewed-dyson-sha>' \
  --active-topology evidence/active.json \
  --page-size 100 \
  --max-jobs 100000 \
  --max-active-pointers 100000 \
  --max-users 100000 \
  --max-incidents 100 \
  --max-result-artifact-reads 200000 \
  --private-output operator/job-identities-private.json \
  --evidence-output evidence/job-identities.json
```

The inspector first requires schema-v4 active topology, hashes that exact file, and rejects any drift
from the release's exact 300/1,800/300/120/3,600-second job-lifecycle policy before deriving its expected
bucket, location, minimum-retention, age, UTC, retention-policy, versioning, and soft-delete values.
It then describes the live bucket before any customer-state scan and after all three scans. Both
observations must retain enforced public-access prevention, uniform bucket-level access, the exact
active-topology lifecycle, one canonical provider creation time that predates active verification,
and the same positive metageneration. Persistent drift therefore fails before a customer record is
read; a recreated bucket fails on chronology; and a lifecycle/security change that is reverted during
the scan still increments metageneration and fails the final comparison. The release identity uses
bucket-scoped `roles/storage.legacyBucketReader` for these metadata reads and
`roles/storage.objectViewer` for exact-generation probes, with no artifact write or delete authority.

`ReadOnlyPersistedJobStore` has no mutation methods. Before any customer-state read it spends one
bounded marker read to obtain a server-issued Firestore `read_time`; every collection query page,
exhaustion look-ahead, reverse-pointer read, and pointer-target read uses that exact selector. It then
walks the full `jobs` collection group, `dysonActiveJobs`, and top-level `dysonUsers` collection in
stable document-name order. It builds private canonical account/job identity sets independently from
the persisted-job and active-pointer directions and requires their exact symmetric difference to be
empty. For every identity present in both active directions it also requires the exact Firestore
document `update_time` generation to match, so a queued/running/cancelling/settling phase mutation of
the same job cannot combine two individually valid generations into clear evidence. Every persisted
active or terminal lifecycle event must also be no later than both that document generation plus the
exact five-second API/provider clock-skew allowance and the shared server snapshot. Planned
`submission_expires_at` and `result_grace_expires_at` deadlines are excluded from the event set, but
their derivation and phase chronology remain exact. At that same snapshot, both directions
independently require every active row's exact creation, update, submission-expiry, workload-timeout,
and worker-overhead anchors. `preparing` may exceed its submission
deadline by at most the exact 120-second recovery allowance; preparation cleanup, ambiguous execution
discovery, running/cancelling compute, result-grace settlement, and immutable Ledger settlement each
have a phase-specific deadline derived from the topology's exact 300/1,800/300/120/3,600-second policy,
the persisted task timeout, and that same two-scheduler-interval allowance. Only aggregate
set/generation/deadline coverage and mismatch counts enter sanitized evidence. The hard maximum is one snapshot
marker read, 100,001 persisted-job scan reads, 100,000 persisted-job-to-pointer reads, 100,001
active-pointer scan reads, 100,000 pointer-to-job reads, and 100,001 user-counter scan reads (500,004
documents total), plus two bounded bucket-metadata reads and at most 200,000 Cloud Storage generation
probes. Every persisted row must retain one canonical 1-36 character private Ledger user ID whose
stable lowercase SHA-256 prefix exactly equals both its embedded account hash and account-scoped
Firestore path; every pointer-referenced active job independently proves the same ownership. Missing,
overlength, padded, or cross-account Ledger identities fail every runtime read and lifecycle transaction
before Cloud Run, Ledger, pointer, capacity, or job mutation. Every persisted row also retains one
exact launch context: its profile maps to the current channel/profile worker namespace, its immutable
worker name carries one strict ten-hex release suffix, its 32-character launch token is canonical,
payload and result URIs equal the channel bucket's exact `jobs/<account>/<job>/` paths, an operation
belongs to the exact project and region, and any execution belongs to that exact persisted worker.
A deployment therefore continues to read and reconcile jobs launched by an older immutable release,
but never accepts a worker from another profile, channel, or naming namespace. New rows additionally
bind the full lowercase 40-character release source to the worker suffix; malformed or contradictory
source provenance fails closed, while pre-provenance rows retain the strict namespace/suffix contract.
Direct repository reads, reconciliation, and release evidence all use the same dependency-free
contract; drift raises a value-free typed integrity incident before provider or billing calls. A failed
Cloud Run long-running operation is not absence authority: reconciliation first
performs the bounded, launch-token/job-ID/timeout-bound execution search, preserves billing throughout
the full 1,800-second start window, resumes a matching execution, and releases reservations only after
the search exhausts and the grace boundary has elapsed. Every persisted row—including terminal
history—and every pointer-referenced active target must independently preserve
`submission_expires_at == created_at + 300` from the topology-bound submission policy. Repository
reads and direct reconciliation reject a missing, shortened, extended, malformed, or type-coerced
deadline with one value-free typed incident before Cloud Run, Ledger, pointer, counter, or job
mutation; no legacy fallback reconstructs or silently blesses lost authority.
Each identity-exact active persisted job
must satisfy the exact phase-lineage and snapshot-deadline contracts and have one exact pointer; every
pointer must have one identity-, lifecycle-, phase-, and
deadline-validated active job; each canonical user counter must equal the exact number of that user's
active persisted jobs; every immutable settlement intent found in active or terminal history must
satisfy the same exact execution/result/profile/decision contract used before Ledger, including
Cloud Run provider start/completion authority and a six-decimal duration exactly equal to
`max(0.1, completion - start)` for started work; evidence binds intent schema v3 explicitly. The
launch request cannot predate job creation, execution start and completion cannot predate that launch
by more than the reviewed five-second provider allowance, and an available result's provider creation
time must fall from execution start minus five seconds through completion plus five seconds. Optional
cancellation request/dispatch markers must remain ordered between launch and settlement claim. Every
available result must also bind its account/job-derived channel URI, canonical immutable GCS generation,
and provider creation time. For each retention-eligible result, the auditor addresses that frozen
generation directly, revalidates metadata and the 6 MiB maximum, and downloads exactly one byte with
an `ifGenerationMatch` precondition. Settlement already obtained the full bounded payload and froze its
SHA-256; an existing immutable generation cannot change bytes, so this bounded probe proves the
committed object still exists and is readable without re-exporting private result content. Terminal
history is classified as legitimately expired only from the first midnight UTC at provider creation date plus 31 calendar days, matching the deployed
GCS `age: 31` rule and guaranteeing at least 30 elapsed days; active rows are never
exempt. Persisted grace state must also prove the required, optional, or forbidden exact
post-completion 120-second window—including that a generation first created after expiry cannot
authorize capture. The settlement-claim transaction also requires its start boolean/timestamp to
equal any authority already persisted while Cloud Run was running, so later provider drift cannot
overwrite durable timing evidence. Every terminal row must have one exact shared projection. Execution-backed
history must project the immutable intent's exact status, failure, provider start timestamp,
provider-derived duration, execution-start truth, result availability/digest/generation/creation time,
completion time, capture-or-release billing
direction, and canonical non-secret Ledger reservation identity, funding, requested/reserved/captured
amounts, job reference, creation/expiry, terminal provider event, and audit truth. Dyson validates the
Ledger creation and settlement response schemas before persistence, then terminal history retains only
the canonical provider projection. Schema v30 requires reservation order to be exactly included then paid, forbids a paid fallback beside unlimited access, re-derives the maximum hold from the reviewed profile rate, tier, and timeout, and re-derives a captured terminal row's exact charge from the same rate and tier plus its frozen execution duration. It also re-derives the complete hold TTL from the persisted
profile timeout and reviewed lifecycle policy for every terminal row, requires reservation creation
within the same `job_created_at - 60` through `job_created_at + 5` window, and rejects capture,
cancellation, or expiration after `expires_at` or beyond the persisted settlement authority plus the
five-second provider allowance. Execution-backed provider events must also be no earlier than the
durable settlement claim minus that allowance; pre-execution releases use their persisted expiry,
failure-claim, or launch authority as the matching lower bound. Reservation creation, capture, and cancellation are completed events
bound to the same Firestore generation-plus-five-second and shared-snapshot authorities; reservation
expiry remains a future deadline rather than a completed event. The only terminal history permitted
without an execution settlement intent is a failed pre-execution release: it must retain one of the
four exact private submission/launch failure reasons, matching phase and chronology, no execution,
result, grace, cancellation, or duration fields, and release-only nested billing truth. An active job
whose parent user
document is absent is an incident; an absent counter on a present zero-job user normalizes to zero.
Unknown persisted statuses, impossible active phase combinations, contradictory settlement intents or
terminal projections, stale low/high or malformed counters, more jobs/pointers/users/incidents than the reviewed bounds,
cross-scan count disagreement, exact active-identity-set disagreement—including an equal-count
A-to-B lifecycle swap between directional scans—same-identity active state-generation disagreement,
or incomplete exhaustion fail closed; the auditor never invokes the mutating reconciliation scan.

The non-overwriting mode-0600 private report contains only trusted path identities, hashes of
untrusted embedded values, mismatch field names, status/pointer classes, fixed phase/deadline/snapshot-
timing issue classes, private timing authorities, and exact incident kinds. Keep it in restricted
operator records and never print or upload it. The companion schema-v30 record contains only stable SHA-256
incident handles, classes, the exact
server-issued Firestore snapshot read time, one marker-read count, the exact five-second event-clock
skew, aggregate state-generation and persisted/pointer-referenced snapshot-time
coverage/match/mismatch counts, complete all-persisted and pointer-referenced active submission-
deadline validation against the exact 300-second policy with zero dedicated deadline incidents, the
exact settlement-intent schema-v3 authority, complete
persisted and pointer-referenced settlement-chronology coverage and zero dedicated chronology incidents,
live bucket creation/metageneration/observation chronology and exact non-secret lifecycle/security
state, active/terminal/unknown, phase-lineage, persisted/pointer-referenced settlement-intent, exact
grace-state, provider artifact-provenance, retention-eligibility/expiry, exact-generation readability,
and bounded probe validation, all-terminal projection validation plus exact terminal included-first allocation and reservation-
lifetime coverage with zero dedicated allocation and lifetime incidents, exact pre-execution
projection coverage, exact persisted/pointer-referenced active-deadline coverage with a 120-second
recovery allowance, exact 60-second pre-persistence reservation and 30-second final-settlement
dispatch allowances, complete persisted and pointer-referenced active Ledger-hold validation plus
zero dedicated billing-hold incident counts and the reviewed aggregate profile-rate table, forward/reverse validation, zero exact active-identity-set mismatch, user/counter totals,
exact source/topology/database provenance, scan bounds, independent exhaustion, and chronology. It
never carries a Ledger user ID, reservation ID, idempotency key, job ID, API-key ID, or monthly-period
timestamp.

A release passes only with `result=clear`, exact 100/100000/100000/100000/100/200000 bounds,
exactly one server snapshot marker read, one canonical Firestore snapshot read time at or after the
120-second retired-writer boundary, independently exhausted persisted-job, active-pointer, and
user-counter scans, no artifact-read-bound
exhaustion, exact private Ledger-account hash validation for every persisted job and every
pointer-referenced active job with zero dedicated ownership incidents, exact profile-to-worker,
32-character launch-token, canonical payload/result URI, project/region operation, and worker-parent
execution validation for every persisted job and every pointer-referenced active job with zero
dedicated Cloud Run resource-identity incidents, an exact complete hardware profile projection,
workload timeout no greater than that profile's specific maximum, and task timeout equal to workload
plus the topology-bound 300-second worker overhead for every persisted job and every
pointer-referenced active job with zero dedicated execution-contract incidents, exact active Ledger
hold validation in both directions using the topology-bound profile rates, reviewed tier multiplier,
rate-derived estimate, canonical included-before-paid split, no paid fallback beside unlimited access,
account-scoped reservation identity, shared monthly period, metadata, funding semantics, creation no earlier than the reviewed 60-second preparation bound or
more than five seconds after job creation, and exact reservation lifetime through every valid active
deadline plus 30-second settlement dispatch, with zero dedicated billing-hold incidents,
exact snapshot-causal event-time
validation for every persisted job and every pointer-referenced active job, exact phase-lineage and
phase-specific lifecycle-deadline validation
plus reverse-pointer validation for every persisted active job, matching deadline validation for
every pointer-referenced active job, exact validation of every
persisted and pointer-referenced schema-v3 settlement intent—including launch-causal provider
start/completion, exact derived duration, result-creation bounds, and ordered optional cancellation—
with zero dedicated chronology incidents, exact first-observation grace-state validation for
every such intent, exact provider generation/creation-time provenance validation for every available
result in both scan directions, exact canonical result URI plus bounded exact-generation readability
for every retention-eligible result, exact provider-derived lifecycle-expiry accounting for older
terminal history, exact terminal projection, included-first captured-credit allocation, and profile/timeout-derived
reservation-lifetime validation for every persisted terminal job with zero dedicated allocation and lifetime incidents, and exact
reason/phase/chronology/release-billing coverage for every pre-execution terminal failure, forward
identity/lifecycle/phase validation for every pointer, an exact match between the
persisted-active and pointer-target account/job identity sets, exact matching Firestore `update_time`
state generations for every active identity in both directions, every user counter validated, the
summed counter exactly equal to both active-job and pointer totals, zero unknown
statuses, zero phase, active-deadline, snapshot-timing, settlement-intent, settlement-grace,
settlement-chronology, artifact-provenance, result-durability, state-generation, settlement-projection, terminal-billing-
lifetime, or pre-execution
terminal-projection incidents, zero other
incidents, zero incident entries, exact pre-scan and post-scan live bucket observations
with one unchanged positive metageneration, and a start at least 120 seconds after topology
verification and after the channel's final acceptance completion. A substituted worker/profile, forged hardware profile,
profile-over-limit workload timeout, non-exact task timeout, malformed launch token, noncanonical
payload or result URI, cross-project/cross-region operation, execution outside
the persisted worker parent, or an active hold with a wrong account/job/profile/timeout, rate,
funding order or split, unlimited-plus-paid mix, scoped idempotency key, monthly period, metadata,
stale/future reservation creation, legacy-short lifetime, pre-claim provider event, pre-launch
execution/result history, out-of-bounds result creation, unordered cancellation, or forged
start/completion/duration relation fails closed before Cloud Run, Ledger, or Firestore mutation. A missing or mismatched reverse
pointer, dormant unknown state, terminal pointer, malformed pointer, orphan pointer, non-active pointer
target, running row
without launch/operation/execution lineage, cancelling row without execution and cancellation intent,
settling row without result-grace or settlement intent, started resultless intent with missing,
partial, short, backdated, or not-yet-expired grace, result artifact with missing/malformed generation or
provider creation time, noncanonical account/job result URI, retention-eligible frozen generation that
is missing, oversized, metadata-drifted, or unreadable, artifact created after its exact grace expiry,
immutable intent whose execution
state contradicts its final result decision, terminal status/failure/duration/execution-start/result/
digest/generation/creation-time/completion/billing output that differs from its frozen intent, a
terminal row without an intent whose exact pre-execution reason, phase, chronology, forbidden
execution/result fields, or release-only billing projection differs, active
row with terminal fields, malformed active creation/submission/timeout anchors, preparing or cleanup
state beyond its exact bound, ambiguous execution discovery, running/cancelling compute, result
settlement, or Ledger settlement beyond its phase-specific deadline, any active, terminal, or nested
Ledger reservation creation/capture/cancellation event later than its Firestore `update_time` plus five
seconds or later than the shared snapshot, a state
generation later than that snapshot, the same active identity observed at different Firestore update
generations, missing active user's account document, stale low/high or
malformed counter, or path/payload drift is
therefore
visible without exposing its identity or allowing equal totals for different active jobs to produce
false-clear evidence. Keep
the 503 and alert open, preserve the private report, and use a separately approved backup/recovery
procedure. Operators must never rewrite an embedded identity, active pointer, billing
object, capacity counter, or Ledger reservation to silence the incident; the auditor cannot authorize
or perform remediation. After reviewed recovery, rerun the entire audit and require a fresh clear
record.

### Cancellation-conflict incident discovery and convergence

The alert payload is deliberately aggregate-only; never add a job, account, reservation, cursor, or
incident identifier to Cloud Logging or Monitoring. Conflict aging instead maintains an exact private
open index in Firestore, and terminal settlement atomically moves that record to immutable history in
the same transaction that records scheduler/API provenance and removes capacity. A routine protected
release identity already has `roles/datastore.viewer`, so an authorized operator can use Application
Default Credentials to inspect that index without receiving the reconciler's write or Ledger-secret
permissions. The private schema-v3 reconciliation control record binds each lease to the exact
40-character deployed source SHA and continuously renews a conflict-index completeness cycle. Each
fully processed pointer must bind its Firestore document ID, embedded job/account identities, and the
loaded private job exactly before it can count toward the cycle or perform external work. Any drift
fails the scheduled execution without deleting the pointer, decrementing capacity, settling billing,
or advancing completeness. Exact pointers are then checked for legacy nested conflict state before
external work. A bounded page containing the entire ring completes immediately; larger or exactly
full pages require two committed document-order wraps, conservatively proving at least one complete
ring despite pointer churn. A source change resets the in-progress cycle. `discover` refuses even an
empty index unless the latest completed cycle uses the topology's exact source and began at least 120
seconds after active-topology verification—longer than the retired API and reconciler hard timeouts—so
an in-flight pre-index writer cannot appear behind an already-scanned cursor:

```bash
python scripts/inspect_cancellation_conflicts.py discover \
  --channel sandbox \
  --source-commit '<full-reviewed-dyson-sha>' \
  --active-topology evidence/active.json \
  --limit 100 \
  --private-output operator/cancellation-conflicts-private.json \
  --evidence-output evidence/cancellation-conflicts.json
```

`discover` reads one reconciliation control document plus at most 101 index documents to return at
most 100 incidents; it never calls `list_active_jobs`, Ledger, Cloud Run, or a write API. If `has_more`
is true, use only the exact `next_cursor` from the private report in a new bounded invocation and keep
each output path unique. The mode-0600 private report contains internal job, hashed-account, and
Ledger-account identities needed for an authorized support investigation. Keep it in restricted
operator records, never print it, upload it as a workflow artifact, or place it in the cross-repository
bundle. The schema-v2 companion evidence contains only stable SHA-256 incident handles, aggregate
counts/ages, exact active topology provenance, hashed cursors, and the full-ring cycle's source,
start/completion times, completion reason, verified-job count, backfill count, and wrap count. A
release-clear observation requires that post-quiescence cycle plus exactly `result=clear`, zero
incidents, and no next page; any missing/stale cycle, found incident, or unread page prevents approval.

Do not invoke a customer API read to force captured/mixed convergence. Leave the immutable one-minute
scheduler enabled. After it runs, prove that every incident from the exact private baseline moved to
history, retained monotonic chronology/counts, recorded `scheduled-reconciler` for captured or mixed
truth, removed its active pointer only with terminal billing, and still contains no identity in the
sanitized output:

```bash
python scripts/inspect_cancellation_conflicts.py verify \
  --channel sandbox \
  --source-commit '<full-reviewed-dyson-sha>' \
  --active-topology evidence/active.json \
  --baseline-private operator/cancellation-conflicts-private.json \
  --private-output operator/cancellation-conflicts-converged-private.json \
  --evidence-output evidence/cancellation-conflict-convergence.json
```

The verifier has no mutation methods and fails if an incident disappears, remains open, changes
account, regresses attempts/reservation counts, retains capacity, lacks exact job/history evidence, or
claims API resolution for captured/mixed billing. All-released truth may record either exact API or
scheduled reconciliation; captured/mixed truth is scheduler-only. Preserve failed outputs privately,
investigate Ledger availability or malformed state, and wait for authoritative reconciliation. Never
repair an incident by writing either index collection, a nested job, an active pointer, or a Ledger
reservation. After convergence, run a fresh first-page discovery and require `clear` before assembling
the release bundle.

The same prerequisite record also proves the channel Firestore database is Standard Native mode in
`nam5`, uses pessimistic concurrency, has deletion protection and seven-day point-in-time recovery
enabled, and has exactly one daily managed backup schedule retained for fourteen days. New databases
are created with those controls atomically. For a recognized legacy database with only protection
disabled, the dual-confirmation provisioner may add deletion protection and PITR; it never disables
protection, deletes a backup/database, or rewrites/deletes a drifted schedule. Routine release
identities receive only the narrow backup-schedule and backup viewer roles in addition to their
existing database viewer role.

After active promotion, wait for the next scheduled backup and create recovery evidence with the
same active topology artifact (a backup created before promotion is rejected):

```bash
python scripts/verify_firestore_recovery.py \
  --channel sandbox \
  --source-commit '<full-reviewed-dyson-sha>' \
  --active-topology evidence/active.json \
  --output evidence/data-recovery.json
```

Use `--channel production` for production. This read-only gate requires a READY backup no more than
36 hours old, created after active-topology verification, with the exact fourteen-day expiry. It
revalidates live PITR/delete protection/scheduling, binds the active revision and topology SHA-256,
and writes a non-overwriting mode-0600 record containing only hashes of the backup resource and
database UID. Keep that record with the cross-repository release bundle; schedule presence alone is
not proof that a current release is recoverable.

### Completed recovery and alert-delivery drills

A READY backup is necessary but does not prove that Firestore can restore it. Before release approval,
perform one separately authorized restore into a new disposable database whose ID is exactly
`dyson-restore-<channel>-<source-sha-prefix>-<UTC YYYYMMDDhhmm>`. Never use the active channel database
as the destination. Capture the full backup and restore-operation resource names from the reviewed
GCP operation, then run the read-only observation while the disposable database is active:

```bash
python scripts/verify_firestore_restore_drill.py observe \
  --channel sandbox \
  --source-commit '<full-reviewed-dyson-sha>' \
  --active-topology evidence/active.json \
  --data-recovery evidence/data-recovery.json \
  --source-backup 'projects/dyson-backend/locations/nam5/backups/<backup-id>' \
  --restore-operation 'projects/dyson-backend/databases/<restore-db>/operations/<operation-id>' \
  --restored-database 'dyson-restore-sandbox-<10-char-source>-<UTC timestamp>' \
  --output evidence/restore-observation.json
```

The observer accepts only the backup hashed by `data-recovery.json`, a successful restore operation
started after that recovery check, exact destination `sourceInfo`, Standard Native `nam5` metadata,
and a bounded read against the `dysonUsers` application path. It records no database, operation,
document, or backup identifier. Review the observation, then use a separately approved administrator
session to delete **only** the disposable database, with its current etag. If the restored database
inherited deletion protection, separately review and disable protection only on that exact disposable
destination immediately before deletion; never alter the active channel database. The verifier has no
restore, update, disable-protection, or delete code path. Within four hours, prove cleanup and emit the
bundle record:

```bash
python scripts/verify_firestore_restore_drill.py finalize \
  --channel sandbox \
  --source-commit '<full-reviewed-dyson-sha>' \
  --active-topology evidence/active.json \
  --data-recovery evidence/data-recovery.json \
  --observation evidence/restore-observation.json \
  --restored-database 'dyson-restore-sandbox-<10-char-source>-<UTC timestamp>' \
  --output evidence/restore-drill.json
```

Finalization lists active and deleted databases read-only, requires the disposable database to be
absent from the active inventory and present once with valid delete/purge chronology, and writes only
hashes plus timestamps. Keep `restore-observation.json` private beside operator records; only the
sanitized `restore-drill.json` enters the release bundle. If observation, data-plane reading, or cleanup
fails, the release is not recoverable and must not be approved.

A Monitoring channel that reports `VERIFIED` likewise does not prove provider delivery or human
response. For each channel and active release, prepare a fresh two-hour challenge after active topology
and prerequisite verification:

```bash
python scripts/verify_alert_delivery.py prepare \
  --channel sandbox \
  --source-commit '<full-reviewed-dyson-sha>' \
  --active-topology evidence/active.json \
  --prerequisites evidence/prerequisites.json \
  --output evidence/alert-delivery-challenge.json
```

The prepare step revalidates the exact live channel, uptime check, and all six policies. An authorized
operator must send the exact challenge through that configured provider's authenticated test path (or
capture it from a real Monitoring alert), then have a human on-call acknowledge the received message.
Create a new mode-0600 `evidence/provider-receipt.json` from actual provider metadata with exactly this
shape; hash identifiers and identities locally and never record an address, token, phone number, or
message body:

```json
{
  "schema_version": 1,
  "status": "received-and-acknowledged",
  "channel": "sandbox",
  "source_commit": "<full-reviewed-dyson-sha>",
  "active_revision": "<exact-active-revision>",
  "notification_channel_resource_sha256": "<64-hex-from-prerequisites>",
  "delivery_route_type": "<exact-live-channel-type>",
  "delivery_method": "provider-test-message",
  "challenge": "<exact-received-challenge>",
  "provider_message_id_sha256": "<64-hex>",
  "recipient_identity_sha256": "<64-hex>",
  "acknowledger_identity_sha256": "<64-hex>",
  "sent_at": "<UTC timestamp>",
  "received_at": "<UTC timestamp>",
  "acknowledged_at": "<UTC timestamp>"
}
```

Use `delivery_method: monitoring-alert` only for a real incident notification. Verification requires
delivery within fifteen minutes, human acknowledgment within thirty minutes, the unexpired exact
challenge, and unchanged live observability before writing sanitized evidence:

```bash
python scripts/verify_alert_delivery.py verify \
  --challenge evidence/alert-delivery-challenge.json \
  --receipt evidence/provider-receipt.json \
  --active-topology evidence/active.json \
  --prerequisites evidence/prerequisites.json \
  --output evidence/alert-delivery.json
```

The final record contains only SHA-256 identities, route type, policy names, and timestamps; it never
contains the challenge or destination. A human-authored receipt without an actual provider message is
not evidence. Archive the private challenge/receipt with restricted operator records and put only
`alert-delivery.json` in the release bundle. The verifier itself performs read-only GCP inventory and
cannot send a test, acknowledge an incident, or edit Monitoring.

The routine release service account has only Cloud Run and Scheduler administration plus read-only
network, DNS, Firestore, IAM, Monitoring, Secret Manager, and service-usage inspection at project scope; network
use is granted only on the channel subnet. Artifact Registry writer and bucket-metadata reader are
resource-scoped, and `roles/iam.serviceAccountUser` is granted only on the three channel runtime
identities. It has no
project IAM administration, service-account administration, network/firewall administration,
storage administration, Secret Accessor, or user-managed key. Update the allowlist in
`scripts/verify_release_prerequisites.py`, its tests, and this documentation together.
Both channels must be dispatched from the protected `release/dyson-ga` branch; the workflow rejects
any other ref before checkout and checks out the exact event SHA without persisting GitHub credentials.
Configure the `dyson-sandbox` and `dyson-production` GitHub environments to allow only that branch,
keep production reviewer approval enabled, and define `GCP_WORKLOAD_IDENTITY_PROVIDER` plus
`GCP_DEPLOY_SERVICE_ACCOUNT` as environment secrets. The values are not interchangeable:

| environment | exact provider | exact deploy identity |
|---|---|---|
| `dyson-sandbox` | `projects/1017725801616/locations/global/workloadIdentityPools/github-release/providers/dyson-sandbox` | `dyson-sandbox-release@dyson-backend.iam.gserviceaccount.com` |
| `dyson-production` | `projects/1017725801616/locations/global/workloadIdentityPools/github-release/providers/dyson-production` | `dyson-production-release@dyson-backend.iam.gserviceaccount.com` |

Deployment is WIF-only: JSON service-account keys are not accepted, and the workflow rejects a
wrong-channel provider/account or any API, reconciler, or worker runtime identity before checkout.
Map `google.subject=assertion.sub`; each provider must require GitHub owner ID `170319640`, repository
ID `849206125`, `refs/heads/release/dyson-ga`, its exact environment claim, and
`CrossGL/dyson/.github/workflows/deploy-serverless.yml@refs/heads/release/dyson-ga`. Grant each
service account `roles/iam.workloadIdentityUser` only to the exact subject
`repo:CrossGL/dyson:environment:<environment>` in the `github-release` pool. Both billing secrets
must match active CrossGL Ledger internal service keys in their respective channels.

## Superseded release lifecycle planning

Successful releases intentionally keep immutable workers, reconciler, scheduler, and image tags so
an immediate rollback remains executable. They must not accumulate without a reviewed bound.
`scripts/plan_superseded_releases.py` performs the read-only inventory and writes a private JSON
plan; it has no apply mode and its gcloud allowlist contains only describe/list operations.
Production plans enforce a 30-day minimum age, sandbox plans enforce seven days, and neither floor
can be lowered. Every plan retains at least the two newest ready rollback sources **per channel** in
addition to all traffic-bearing, tagged, latest-created, latest-ready, enabled-scheduler, and
nonterminal-execution sources. Selection is oldest-first and atomic, with defaults of at most two
source releases and 20 resources; hard parser limits prevent more than five releases or 50
resources in one plan.

Run it only after the exact GA revision is active, using the active full source label rather than an
assumed branch head:

```bash
python scripts/plan_superseded_releases.py \
  --channel sandbox \
  --expected-active-source '<40-character-active-source-sha>' \
  --output "$HOME/dyson-sandbox-lifecycle-plan.json"

python scripts/plan_superseded_releases.py \
  --channel production \
  --expected-active-source '<40-character-active-source-sha>' \
  --output "$HOME/dyson-production-lifecycle-plan.json"
```

The evidence file is created once with mode `0600`; an existing file is never overwritten. Exact
commit-addressed jobs must have matching product/component/channel/source labels, schedulers must
target that source's reconciler, every revision-backed release must still have its complete
resource set, and all timestamps must be trustworthy. Split traffic, an unexpected active source,
malformed managed names/labels, ambiguous scheduler ownership, or malformed/truncated inventory
blocks the entire plan. An incomplete revision-backed release or missing timestamp defers that
whole source and can never yield a partial candidate. Unlabelled legacy resources outside the
immutable naming contract are reported as unmanaged and can never become candidates by prefix
alone.

Artifact Registry candidates are deliberately limited to the exact full-SHA tags. The plan records
the digest for review but never authorizes digest deletion, so a digest-pinned revision or job
cannot be broken by this tool. Removing image versions, Cloud Run revisions, jobs, schedulers, or
tags remains a separately authorized operator action: regenerate the plan immediately beforehand,
re-prove active topology and zero nonterminal work, preserve the whole atomic source group, and
never translate unmanaged prefixes into bulk deletion commands. An offline `--inventory` plus
`--as-of` mode exists only for review/tests and applies the same safety rules.

## Release settlement acceptance

After Ledger and Dyson are deployed to a channel, use
[`scripts/run_release_acceptance.py`](scripts/run_release_acceptance.py) to prove the live
CPU and billing path. It is dry-run by default and performs no network or billing mutation
without `--execute` plus the exact channel confirmation. The four `cpu-small` probes cover
successful execution, an expected workload failure, cancellation, and an isolated worker
supervisor interrupt that deliberately leaves no result artifact. The probe uses Python's handled
`SIGINT` because Linux protects namespace PID 1 from default-fatal signals such as `SIGKILL`; it
therefore exits the supervisor before result upload. The interrupt runs only inside its own
credential-free Cloud Run task and proves the infrastructure-failure path rather than adding a
public fault-injection endpoint. All estimates must fit the default 0.2-credit reservation ceiling
before any job is submitted.

The runner deliberately polls Ledger's service-authenticated, read-only settlement audit
before reading terminal Dyson state. A pass therefore proves that the one-minute scheduler
and reconciler independently captured billable execution and explicitly cancelled—not merely
expired—the infrastructure-failure reservations at zero charge. It then proves that the failed
job has no result artifact. Evidence contains job and reservation IDs, funding class, captured
and released amounts, terminal states, source SHAs supplied by the operator, and a hash of the
Ledger user ID. It never contains either
credential, the raw user ID, result values, logs, or failure tracebacks, and it creates a new
mode-0600 file rather than overwriting evidence.

First inspect the exact plan without credentials:

```bash
python scripts/run_release_acceptance.py \
  --channel sandbox \
  --dyson-api-url "${DYSON_SANDBOX_API_URL}" \
  --ledger-api-url https://crossgl-ledger-sandbox-z6rzv36noq-uc.a.run.app \
  --dyson-source-commit "$(git rev-parse HEAD)" \
  --ledger-source-commit "${LEDGER_RELEASE_COMMIT}" \
  --expected-tier dyson_developer \
  --expected-funding included
```

Use a dedicated, non-unlimited acceptance account with an active Dyson subscription. Load
its CrossGL API key and the matching channel's Ledger internal service key into environment
variables without placing values on the command line, then explicitly execute:

```bash
read -r -s DYSON_ACCEPTANCE_API_KEY && export DYSON_ACCEPTANCE_API_KEY
read -r -s DYSON_ACCEPTANCE_LEDGER_SERVICE_KEY && export DYSON_ACCEPTANCE_LEDGER_SERVICE_KEY

python scripts/run_release_acceptance.py \
  --channel sandbox \
  --dyson-api-url "${DYSON_SANDBOX_API_URL}" \
  --ledger-api-url https://crossgl-ledger-sandbox-z6rzv36noq-uc.a.run.app \
  --dyson-source-commit "$(git rev-parse HEAD)" \
  --ledger-source-commit "${LEDGER_RELEASE_COMMIT}" \
  --expected-tier dyson_developer \
  --expected-funding included \
  --execute \
  --confirm RUN_DYSON_SANDBOX_ACCEPTANCE \
  --output "${RELEASE_EVIDENCE_DIR}/dyson-sandbox-included.json"

unset DYSON_ACCEPTANCE_API_KEY DYSON_ACCEPTANCE_LEDGER_SERVICE_KEY
```

Run a second sandbox acceptance with `--expected-funding overage` using a plan account whose
monthly Dyson allowance is exhausted and whose shared paid balance is positive. That mode
requires billable probes to capture a positive paid amount and the infrastructure probe to
release a positive paid reservation; merely creating a zero-value paid hold does not pass.
`--expected-funding split` is available when every probe must reserve both funding sources,
with billable probes capturing both and the infrastructure probe releasing both. Pair these
acceptance files with the Ledger and
Dyson topology artifacts that independently prove the supplied source SHAs, revisions, and
image digests. Production uses the exact canonical production origins and the separate
`RUN_DYSON_PRODUCTION_ACCEPTANCE` confirmation; do not run it without production approval.

## Development gates

```bash
python3 -m pip install -r requirements-dev.txt
python3 -m pip install --no-deps -e .
python3 -m pytest
ruff check .
ruff format --check .
python3 -m pip wheel --no-deps --wheel-dir /tmp/dyson-wheel .
actionlint -no-color
```

The package supports Python 3.11 through 3.14. Production images currently use Python
3.12.

## Retired VM controller

The historical shared Compute Engine controller is not a GA transport and has no optional
installation extra or service unit. `dyson.start_instance` is now an import-safe tombstone:
known legacy symbols raise `LegacyVMRetiredError` with migration guidance. Browser and SDK
traffic must use the authenticated serverless API.
