Synthetic worlds, visible records and accountable evaluation
HAEnv generates synthetic longitudinal patient records and evaluates an agent's clinical decision at an index time T. This page explains the boundaries. The product specification defines required behavior and the acceptance checklist defines its verification.
0Scope and task contracts
The current packs ask different clinical questions. Each declares its visible view, answer contract, derived gold, supported interaction and emission checks.
| Pack | Decision | Decision evidence |
|---|---|---|
| M2 | Differential diagnosis and initial workup | Persisted diagnostic atoms and the registered evidence-purchase contract. |
| P2 | Acute disposition | Visible triage record and the printed ordered rules; an accepted disposition set. |
| P3 | Chronic medication action | The identified drug, accepted action set and applicable clinician-review requirement. |
| P4 | Interpretation of follow-up change | Observed change, measurement context, saved source class and accepted next steps. |
Earlier diagnosis, forecasting and retrospective-review tasks remain supported where registered. Their geometries and scoring conventions retain their own scope. They do not define today's four-pack overall score.
1Responsibilities
| Boundary | Responsibility |
|---|---|
| Application and task adapters | Load jobs, compose the pipeline and register task-specific gold, prompts, views and checks. |
| Patient world and simulation kernel | Render coherent histories, measurements, medication courses and events from explicit premises. |
| Interaction | Deliver authorized records and purchased evidence, retaining tool calls and returns. |
| Judging | Read saved decision atoms, semantic verdicts, management criteria and gates under an identified policy. |
| Runtime | Resolve request settings, dispatch under budgets, persist receipts and recover matching completed attempts. |
| Provenance and reporting | Identify source and criterion definitions, enforce comparability and aggregate saved measurements. |
HAEnv composes the simulation kernel with task-specific registries and adapters.
verifier_core supplies domain-independent verification primitives. Clinical
rules belong in sourced registry entries, not in a generic transport or plugin contract.
Task registration declares behavior explicitly; unsupported combinations must fail visibly.
The task and plugin contracts constrain extensions. Shared machinery belongs in the core when tasks need the same mechanism; a pack-specific decision stays with that pack's policy and adapter.
2Data flow and visibility
The solver sees authorized records through T, inclusive. Future observations, latent premises, adjudication snapshots and measurement truth remain verifier-only. Observed values and true values have different roles: noise, method changes and pre-analytical variation are part of the task, not errors to erase from a prompt.
A documented diagnosis in the permitted record is allowed. Hidden diagnostic answers are not. Unavailable measurements remain unavailable rather than normal or negative. Static prompt checks and dynamic result guards block registered leaks before delivery. Public source material can contain answers; runtime isolation is not a secrecy guarantee against distribution browsing.
3Synthetic world and emission
Facts, gold and generation controls
Patient facts, gold adjudication and generation controls are separate inputs. Narrative extraction records what the text says; it must not invent a gold answer or pretend a sampled control was extracted. The offline extraction example keeps its synthetic placeholder status and original replay identity.
Histories, measurements and events
A hidden world drives the longitudinal record. The generation layer combines clinical trajectories, medication courses, background comorbidity, events and observation processes. Deterministic generation keys draws by case identity; pinned facts remain pinned. Model-assisted prose and saved caches have a separate provenance from deterministic numeric rendering.
Preserve units, dates, method changes, device availability, missingness and the distinction between an acting factor and a merely present one. Background complexity is sampled from its registered policy, not chosen to disclose the gold class. Each item must remain answerable from its permitted view.
See skeleton and residual generation, world composition and history and distractor conditioning for the corresponding mechanisms.
Sourced clinical resources
Clinical thresholds, acceptance sets, effects and measurement assumptions are registered with their sources and review status. Authored assumptions remain identified as such. Shared wearable reference measurements retain their population and measurement convention. A reference for daily differences cannot silently become a reference for daily levels.
Emission and batch checks
Build, leak probing, solving and grading occur in that order. An item must pass applicable premise, gold re-derivation, verification and leak checks before emission. Blocking findings retain a reason; an empty emitted batch is not success. Batch audits additionally check coverage, declared diversity, nested inputs and nonclinical shortcuts. Passing a finite set of checks does not prove universal clinical consistency.
Persisted inputs and source declarations
Save jobs, case and payload hashes, world identity and audit outcomes alongside their runs. The generated freeze and provenance resources identify source segments and pack objects. Read those declarations directly; stable filenames and dates do not determine whether a pack is current or whether an artifact can be discarded.
4Interaction and execution
Query, full context and registered legacy geometries
Current query mode starts with the encounter header and retrieves permitted profile, series, labs, medications, notes, sections and protocol-library records. Full context delivers the same authorized domain through its registered view. Query limits, truncation boundaries and the final answer turn are recorded answer conditions. At a limit, further tools are closed; an unanswered request is not evidence seen.
Legacy single, gated, slices and multi
remain separate geometries where registered. The geometry
contract explains their episode boundaries. A geometry name does not make an unsupported
task/view combination valid.
M2 evidence purchase
The visible menu supplies target, kind, price and budget state. Menu identity determines billing; self-reported kinds cannot change it. Over-budget purchases are refused before charge or revelation. Purchased tests count as investigations already ordered. Full context does not make withheld evidence free. Record the actual returned result, including unavailability.
Core purchase, query limits and paid USD budgets are distinct. M2's off-menu execution remains a limitation; the menu guarantee is not a claim that every arbitrary target is budget-bounded.
Saved evidence and accountable requests
Persist the raw answer, query/purchase trace, effective request settings and usage status. Judges consume only their declared question records, gold and answer fields. A saved reasoning trace is an audit artifact, not permission for an undeclared judge to read additional truth.
Budget, stopping and receipt replay
Paid execution requires an approved cap and shared ledger. Admit attempts against available budget, retain request IDs and settle from billing evidence. Unknown usage retains its explicit status and applicable conservative bound. Configuration, route choice and concurrency are recorded; secrets are masked. No price, token floor or scheduling count is copied into this page.
Durable receipts precede derived result writes. Resume replays matching saved completions before any new purchase. A possibly sent request retains its bound. Resends are limited to the classified infrastructure/authentication or harness failures allowed by the specification. Empty output, malformed model JSON, truncation, spent tool budgets and deterministic leak verdicts remain terminal under their original settings. Operator drain stops new dispatch while retaining completed work and accounting for in-flight attempts.
Offline controls
Offline stubs, replay fixtures and deterministic builds exercise wiring without model calls. They are identified separately from measured paid answers. A missing replay is an explicit error, not an invitation to send a request. Synthetic controls and prepared clinician materials do not establish clinical validity or clinician sign-off.
5Current and legacy scoring
Gates, core correctness and management quality
Emission gates decide whether a question may exist. Answer hard gates affect its measured outcome. Registered propensity gates are reported separately. Their meanings and denominators must not be collapsed into a single generic pass/fail.
Declared judge inputs
Decision correctness reads persisted task-specific atoms and accepted answer sets. Semantic diagnosis/test matching uses saved atomic verdicts under its identified consensus policy. Management quality reads the saved management snapshot, applicable source facts, registered adapter rules and complete submitted answer. It cannot invent an unstated clinical indication or replace the saved core decision.
Missing, unresolved, not applicable and false are distinct. Invalid output or missing required truth remains an error or unscored state. Historical votes and answers are not silently replaced when a report is read. The current semantic and management policies, including their different voting conventions, are defined in specification sections 7–8.
Current four-pack overall score
S = G * D * (1 + Q) / 2
G records abort/hard-gate eligibility, D the pack's decision correctness, and Q the uncapped positive management-quality score. If G * D is zero, S is zero for any Q. Otherwise a missing Q leaves S uncomputed. The capped management rubric score is a separate reading.
P2 D reads accepted disposition membership; P3 D combines drug and action correctness; P4 D requires both source and next-step correctness; M2 D reads persisted diagnostic correctness. Unknown M2 core correctness is mapped to zero by board reading, while management preparation preserves it as unscored. Propensity gates are outside G. The overall-score contract is authoritative.
The current k3 convention averages saved answer scores per common item; pk1 reads the first answer separately. Packs average their item scores and the overall result weights the packs equally. Missing required packs or required computed scores prevent an overall rank. Paired item-cluster bootstrap intervals quantify item sampling, not all judge uncertainty.
Legacy scores and profiles
Earlier rank_ddx/rank_models composites use their registered
scoring dimensions, applicability, pairing and gate multiplier. Kernel tracks and process
profiles also retain their own roles. A legacy mean of scoring dimensions is not today's S;
standalone pack composites are diagnostic outputs distinct from the overall score.
Read the scoring convention, population, saved repetitions, coverage and source identities before comparing values. The legacy tracks guide and historical validity evidence describe those earlier conventions. Their anchor agreement does not clinically validate the current rubric.
6Provenance and verification
Generation, world, criterion definitions, solver answer conditions and execution routes have distinct identities. A changed route alone does not establish equivalent verdicts. Criterion-aware publication rejects missing, stale or mixed required identities; a legacy whole-judging stamp retains its original vintage semantics.
Relocation preserves resource bytes and historical source labels. Replaying saved artifacts keeps their original identities. An explicit recomputation creates an identified new measurement; it does not stamp every unmeasured old column as current. Responses absent from storage cannot be reconstructed from scores. Reporting does not authorize new model or judge requests.
The acceptance checklist covers normal, boundary and refusal paths, negative controls, plugin registration, installed resource access, offline behavior and publication. Generated inventories and resource declarations supply machine counts directly; this page maintains no competing inventory. Inherited failures and missing artifacts remain visible rather than becoming a claim that the full suite passes.
7Interpretation limits
- Clinical registries and management policies retain their stated provisional/review status. Synthetic consistency and internal controls do not establish clinical correctness or real-world validity.
- Semantic and quality judgements have unresolved judge-noise, cross-vendor and self-preference questions. Historical calibration does not validate a later policy.
- Protocol-library policy is authored evaluation material. FHIR-style views do not imply a complete FHIR-server implementation. Some requested tests have no interpretable simulated result.
- Registered emission/leak controls and mutation sets are finite. P4's question-blind RCV baseline constrains claims about modest performance.
- A clean checkout does not include every historical answer, judge receipt or paid run input. Exact historical reproduction requires the corresponding saved artifacts; item-sampling intervals exclude other sources of uncertainty.
See current limitations, the data card and ethics and use boundaries.