# evalshift-sdk — complete reference for AI tools

Canonical hosted copy: https://www.evalshift.dev/sdk-llms-full.txt
Package: evalshift-sdk (PyPI) | import: evalshift | version: 0.3.0 | schema: 2.1.0
Python: >=3.10 | runtime deps: none (stdlib only) | typed (py.typed) | license: MIT
Install: pip install evalshift-sdk   (or: uv add evalshift-sdk)
Optional extras: pip install "evalshift-sdk[langchain]" -> langchain-core>=0.2;
  "[openai]" -> openai>=1.40; "[anthropic]" -> anthropic>=0.40; "[google-genai]" -> google-genai>=1.0

Purpose: in-process capture SDK for AI agents. Wraps agent invocations, tool calls, and model
calls; records each run as a span tree; serializes to a JSON capture envelope written to
<base>/captures/<suite>/cap_<hex>.json on local disk. No network code anywhere in the SDK.
Captures are consumed by the separate evalshift CLI
(https://github.com/babaliauskas/evalshift-cli); disk is the only interface between SDK and CLI.

Quickstart (minimum working setup):
1. pip install evalshift-sdk
2. Decorate the agent entry point with @capture.agent(suite="...", redact=..., tools=...); both
   redact= and tools= are REQUIRED. redact=: True=mask emails/API keys, False=verbatim, or a
   callable. tools=: the toolset this session's model calls are offered — [] to assert none, or a
   real toolset (Anthropic/OpenAI/Gemini shape). Record model calls inside with
   record_model_call(..., tools=...) or capture.model_call(..., tools=...) — pass tools=None on
   either to inherit the decorator's value instead of asserting a per-call one; decorate tools with
   @capture.tool.
3. Run with EVALSHIFT_CAPTURE=1 set (capture is OFF by default — without it every wrapper is a
   pure pass-through and no file is written).
4. Result: .evalshift/captures/<suite>/cap_<hex>.json appears in the CWD, ready for the
   evalshift CLI. Full runnable code in "Minimal examples" below.

Safety model (three distinct rules):
1. CAPTURE PATH IS FAIL-OPEN: every piece of SDK bookkeeping is guarded; a fault drops the
   capture and logs one debug line, never raises into the host. The user's function call is the
   only unwrapped statement — its return value and exceptions always propagate. When the user
   function raises, an `error` event is recorded, the partial capture is still written, then the
   original exception re-raises.
2. REDACTION IS FAIL-CLOSED: a redactor that raises drops the whole capture (never written
   half-masked). The host agent is unaffected.
3. READ PATH RAISES: load_capture/load_envelope raise typed MigrationError subclasses; they do
   not fail open.

## Environment variables

| Var | Default | Meaning | When read |
|---|---|---|---|
| EVALSHIFT_CAPTURE | unset (OFF) | Master gate. Truthy set (case-insensitive, stripped): {"1","true","yes","on"}. Anything else = off. | Live, every call |
| EVALSHIFT_DIR | ".evalshift" | Capture root dir. Relative paths resolve against CWD (no repo-root walk). | Live, each write |
| EVALSHIFT_MAX_CAPTURES | 200 | Keep newest N *.json per suite dir (GC after each disk write, ordered by file mtime). | Config construction (import / reset_config) |
| EVALSHIFT_CAPTURE_TTL | off | Evict capture files older than N seconds. | Config construction |
| EVALSHIFT_DEDUP | on | Per-process dedup keyed (suite, input_hash). | Config construction |
| EVALSHIFT_SAMPLE_RATE | off (capture all) | Fraction of runs to capture (0.0-1.0), decided once per agent invocation. | Config construction |

"Uncapped/off" literal set for numeric knobs: {"0","none","unlimited","off"} (also empty string).
Precedence: explicit configure(...) > env var > built-in default. Malformed env values fail open
to the default. Negative or zero ints -> uncapped (None).

## Full API

All imports `from evalshift import ...` unless noted. `capture` is a module-level singleton
facade instance.

capture.agent(*, suite: str, redact: Redactor | bool, tools: Any, code_version: str = "",
              conversation_id: str | None = None, turn_index: int | None = None,
              parent_capture_id: str | None = None) -> decorator
  redact= is REQUIRED (no default): True -> default_redactor (masks emails, sk-..., AKIA...,
  Bearer ... and NOTHING else); False -> capture payloads verbatim; a (value)->value callable ->
  custom redactor. Any other value, None INCLUDED, raises TypeError at decoration time, whether
  or not EVALSHIFT_CAPTURE is set. There is no process-wide fallback.
  tools= is ALSO REQUIRED (no default) -- this session's toolset; see record_model_call below for
  the accepted shapes and the tools=[] / tools=None distinction. Unlike redact=, an unrecognised
  VALUE never raises (degrades to no session toolset, debug log) -- only omitting the keyword
  itself raises TypeError. Inherited by any record_model_call / capture.model_call inside this
  decorated function that passes its own tools=None; a call's own non-None value always overrides
  it (per-call authority — a real agent can switch toolsets mid-run).
  Captures one agent invocation per call. Auto-detects async def. No-op when gate off (gate is
  re-read at every call, not frozen at decoration — enabling EVALSHIFT_CAPTURE after import
  works). Agent input auto-derived by binding call args to the signature ({param: value});
  binding failure falls back to {"args": [...], "kwargs": {...}}. Conversation kwargs are STATIC
  (fixed at decoration time) — use agent_session for per-turn values.

capture.agent_session(*, suite: str, redact: Redactor | bool, tools: Any, code_version: str = "",
                      agent_input: Any = None, conversation_id: str | None = None,
                      turn_index: int | None = None, parent_capture_id: str | None = None)
  Sync context manager; yields SpanTree | None (None when gate off / not sampled — treat the
  yielded SpanTree as opaque, its only public use is the None-check). One capture per `with`
  block. The session is contextvar-scoped, NOT lexical: @capture.tool calls, capture.model_call
  recorders, and record_model_call attach to it from any function called (directly or
  transitively) inside the block. redact= and tools= are REQUIRED, same contract as the
  decorator's, and are validated at the `with` statement. ALWAYS pass agent_input (see behavior
  rules): any JSON-able value; it is identity only — hashed as-is into the envelope input_hash
  (dedup key), never stored raw, and redaction does not apply to it. Recommended value: the full
  messages list. Recommended primitive for multi-turn conversations (fresh turn_index per with).

capture.agent_session_async(...)
  Identical params/behavior; `async with` form.

capture.model_call(*, model_id: str, tools: Any, input: Any = None,
                   generation_config: dict[str, Any] | None = None) -> recorder
  Streaming model-call recorder; usable as `with` or `async with`. Methods:
    rec.add_text(text: str) -> None            # append streamed chunk
    rec.set_usage(*, input_tokens: int = 0, output_tokens: int = 0, cost_usd: float = 0.0,
                  latency_ms: int | None = None) -> None
    rec.set_generation_config(config: dict[str, Any]) -> None   # last write wins; mid-stream OK
    rec.set_requested_tool_calls(calls: Any) -> None   # last write wins; before or during block
  Records exactly one model_call event on exit, output = "".join(chunks). latency_ms auto-derived
  from block duration (round((end-start)*1000)) unless set via set_usage. Inert without an active
  session (no toolset resolution/sidecar write either, for a no-op). Recorder faults never break
  the host stream loop. tools= is REQUIRED, same contract as record_model_call below.
  generation_config as in record_model_call below. set_requested_tool_calls records what the model
  ASKED to call -- same contract as record_model_call's requested_tool_calls= below; call it before
  or during the block (a streamed tool call is complete only once its argument deltas arrive).

capture.tool  /  capture.tool(name: str | None = None)
  Decorator (bare or with name; name defaults to fn.__name__). Auto-detects async def. Records a
  tool span -> serialized as TWO events: tool_call (at start; name, arguments, call_id,
  parent_call_id) + tool_result (at close; result, error). No-op when no agent session active.
  A raising tool records error=str(exc), result=None, and the exception propagates.
  NOT related to record_model_call's tools= -- @capture.tool decorates a function your agent
  actually CALLS (EXECUTED); tools= records what the model was OFFERED on a given call, called or
  not; requested_tool_calls= records what the model ASKED to call. See the three-way table under
  record_model_call below -- the three diverge routinely and none is derivable from another.

record_model_call(*, model_id: str, tools: Any, input: Any = None, output: Any = None,
                  requested_tool_calls: Any = None,
                  input_tokens: int = 0, output_tokens: int = 0, cost_usd: float = 0.0,
                  latency_ms: int | None = None,
                  generation_config: dict[str, Any] | None = None) -> None
  Records an already-complete (atomic) model call into the active session. No-op outside one.
  Zero-duration event: latency_ms is 0 unless passed explicitly.
  tools= is REQUIRED (no default) -- the toolset this call was offered, in Anthropic
  ({name, description, input_schema}), OpenAI ({type: "function", function: {name, description,
  parameters}}), or Gemini types.Tool shape, or a bare list mixing any of those
  (evalshift.capture.toolset.normalize_tools). Three distinct values:
    tools=[...]  own toolset for this call -- always wins over the session's, even mid-session
    tools=[]     REAL assertion "no tools on this call" -- not a default, a value
    tools=None   defer to the enclosing session's own tools= (capture.agent / agent_session /
                 agent_session_async) -- the ONE place None has meaning; anywhere else it is just
                 another value that fails to normalise
  Recorded as tools_offered (tool-name list, always stamped once the effective value normalises)
  and toolset_ref (sha256:<hex> pointer to the full schema, written once per distinct toolset by
  ToolsetSink -- <base>/toolsets/<hex>.json -- stamped only if that write succeeds). A value
  matching no recognised shape normalises to None: NEITHER field is stamped (debug log only,
  never raised) -- structurally invalid for that event on purpose, so the CLI refuses it at
  promotion rather than trusting an unstamped toolset. NOT allow-listed (unlike
  generation_config): input_schema is arbitrary user JSON needed whole to dispatch; normalisation
  only recognises/rejects tool SHAPES, never prunes keys within one. Canonical shape is
  {name, description, input_schema} plus ONE optional key: "strict": true, carried from OpenAI's
  function.strict / Anthropic's top-level strict when truthy, omitted entirely when false or
  absent (so every fingerprint written before this key existed is byte-identical). It is kept
  because a replay that drops it runs the target under a weaker schema constraint than the source.
  Gemini FunctionDeclarations have no strict flag and never gain one. NOT redacted, config not
  payload like generation_config, but by a different mechanism worth knowing: generation_config
  lives in span.metadata (redact_tree never walks metadata); tools_offered/toolset_ref are
  top-level span.data fields, safe only because model_call's redactable-field tuple names its
  fields one by one and neither toolset field is among them (requested_tool_calls, below, is the
  counter-example: same kind of field, but it IS in that tuple). Full reasoning:
  docs/DECISIONS.md D-toolset.
  requested_tool_calls: OPTIONAL (default None) -- what the MODEL ASKED FOR in this response.
  Three different facts live on one model_call event and are never interchangeable:
    OFFERED   what the model COULD call      tools=            -> tools_offered / toolset_ref
    REQUESTED what the model ASKED to call   requested_tool_calls= -> requested_tool_calls
    EXECUTED  what the app ACTUALLY RAN      @capture.tool     -> tool_call / tool_result events
  They diverge routinely (a guard rejects a requested call; the app runs one the model never asked
  for; the process dies before dispatch) and each divergence is the signal an eval wants, so all
  three are recorded, never one inferred from another. Pass a list of
  {"name": str, "arguments": dict, "call_id": str | None} items -- build it with
  evalshift.capture.requested.extract_requested_tool_calls(response_dict), a stdlib helper that
  reads a raw Anthropic/OpenAI/Gemini response. Each item is normalised to EXACTLY those three
  keys (extra provider keys dropped, arguments -> {}, call_id -> None) because the CLI's
  RequestedToolCall model is extra="forbid". None/omitted = "not recorded" (field null); [] = a
  real assertion "the model requested no tools" -- the CLI's fallback to executed calls turns on
  that distinction, so they are never conflated. Malformed (not a list, or no item with a usable
  name) -> dropped fail-open, debug log, event still recorded. UNLIKE tools=, these arguments ARE
  redacted: model-generated payload, not config. Full reasoning: docs/DECISIONS.md D-requested.
  generation_config: ALLOW-LISTED, then recorded under the event's metadata["generation_config"]
  so the CLI can replay promoted cases with the same settings. Recorded keys, and ONLY these:
  temperature, top_p, response_mime_type, response_schema, response_format, max_output_tokens,
  max_tokens, tool_choice, parallel_tool_calls, tool_config (Gemini's spelling of tool_choice).
  Every other key is dropped — debug log only, never a warning or raise. Reason:
  metadata is config, not payload, and the redactor walks payload fields only, so an unlisted key
  (system_instruction above all, safety_settings) would enter the capture unmasked.
  The three tool-use keys exist so a replay can re-impose the constraint the source ran under
  instead of silently dropping it; parallel_tool_calls: False survives (the filter is `is not
  None`, never truthiness).
  Values are JSON-coerced: primitives/dict/list pass through; an object exposing model_dump (a
  google.genai.types.ToolConfig, say -- duck-typed via getattr, never imported, same pattern as
  toolset._coerce_schema) is dumped to a dict; anything else (e.g. a Pydantic response_schema
  class) is stored as str(value), so a non-JSON-able setting no longer kills the
  whole capture at write time. Non-dict generation_config dropped fail-open; a config the
  allow-list empties writes no generation_config key at all. The LangChain adapter applies the
  same allow-list to invocation_params (flat keys + a Gemini-style nested generation_config dict;
  flat wins), so bind_tools(tool_choice=..., parallel_tool_calls=False) is recorded automatically.

configure(*, sink: Sink | None = UNSET, sample_rate: float | None = UNSET, dedup: bool = UNSET,
          max_captures: int | None = UNSET, capture_ttl: float | None = UNSET,
          require_model_call: bool = UNSET) -> None
  Process-wide options, MERGE semantics: only passed kwargs change. None = disabled/unset for
  sink/sample_rate/max_captures/capture_ttl. There is NO redact knob — configure(redact=...) was
  removed in 0.3.0; masking is required per capture point instead. require_model_call=True drops captures with
  no model_call event (persistence gate for eval-grade capture; off by default).

evalshift.config.reset_config() -> None
  NOT a top-level export. Resets config to defaults, re-reads hygiene env vars, clears the dedup
  registry. Test-isolation utility.

Redactor (protocol, @runtime_checkable): __call__(value: Any) -> Any

default_redactor(value: Any) -> Any
  Recursive; walks str/dict/list/tuple, returns copies (never mutates), other types pass
  through. Masks: emails -> "[REDACTED_EMAIL]"; sk- keys (16+ chars) and AKIA+16 AWS keys ->
  "[REDACTED_KEY]"; "Bearer <token>" -> "Bearer [REDACTED_KEY]".

Sink (protocol, @runtime_checkable): write(envelope: CaptureEnvelope) -> Path | None

FileSink(base: str | os.PathLike[str] | None = None)
  .write(envelope) -> Path | None. Writes <base>/captures/<suite>/<capture_id>.json (UTF-8
  JSON). Base resolution AT WRITE TIME: constructor arg > EVALSHIFT_DIR > ".evalshift" relative
  to CWD. Returns absolute Path, or None on OSError (capture dropped, debug log). Suite segment
  sanitized against path traversal (separators and ".." replaced).

MemorySink()
  .write(envelope) -> None (buffers in memory; nothing touches disk)
  .flush() -> list[CaptureEnvelope]            # drains and clears, write order
  .captures -> tuple[CaptureEnvelope, ...]     # non-draining snapshot
  Thread-safe. Use on read-only filesystems (Lambda) and in tests.
  EXCEPTION: toolset sidecars (ToolsetSink, behind model_call's toolset_ref) are always
  file-based, MemorySink or not -- point EVALSHIFT_DIR at a writable mount (e.g. /tmp) too, or
  every capture's toolset_ref stays unstamped and the CLI refuses to promote it.

load_capture(raw: str | bytes, *, target: str | None = None,
             default_version: str | None = None) -> dict[str, Any]
  Parse capture JSON + migrate (upgrade-on-read) to current (or target) schema version. Raises
  MigrationError subclasses. default_version opts into a version for captures missing
  schema_version (otherwise MissingSchemaVersionError).

load_envelope(raw: str | bytes, *, target: str | None = None,
              default_version: str | None = None) -> CaptureEnvelope
  parse -> upgrade -> reconstruct typed dataclasses. Unknown event type = hard error
  (UnknownEventTypeError); unknown FIELDS from newer-minor captures dropped tolerantly.

register_migration(from_version: str, to_version: str, apply: Callable[[dict], dict],
                   *, description: str = "") -> None
  Register a single-step forward-only schema upgrade. apply must be pure dict->dict, no
  mutation, no I/O. ValueError on backward/same step or duplicate outgoing edge.

MigrationError — base of all read errors. Subclasses (import from evalshift.trace.migrate):
  UnreadableCaptureError        bad UTF-8/JSON, non-object top level, bad timestamp
  MissingSchemaVersionError     no schema_version key and no default_version given
  InvalidSchemaVersionError     schema_version not "MAJOR.MINOR.PATCH"
  UnsupportedSchemaVersionError capture major newer than supported -> refused
  NoMigrationPathError          no registered chain reaches the target version
  UnknownEventTypeError         event "type" not a known discriminator

SCHEMA_VERSION = "2.1.0" (envelope schema this SDK writes; supported: "2.0.0", "2.1.0" -- 2.1.0
  registers an identity migration 2.0.0 -> 2.1.0 for its additive requested_tool_calls field, and
  2.0.0 registers no migration from any 1.x capture: ObsoleteSchemaVersionError on read, by
  design, see docs/SCHEMA.md)
__version__ = "0.3.0"

evalshift.adapters.langchain.EvalShiftCallbackHandler(*, suite: str, redact: Redactor | bool,
                                                      tools: Any, code_version: str = "")
  LangChain BaseCallbackHandler; drop into callbacks=[...]. One capture per root run. Gate,
  sampling, configure(...), dedup, GC apply identically; gate+sampling decided per root run.
  Keyword-only ctor; NO conversation_id/turn_index/parent_capture_id kwargs. One instance
  reusable across invocations and threads. redact= and tools= are BOTH REQUIRED. tools= has no
  per-call override surface here (LangChain callbacks carry no user-supplied tools kwarg): it is
  normalised ONCE at construction and that one resolved value is stamped onto every model_call
  span this handler ever opens (on_llm_start / on_chat_model_start), for the handler's whole
  lifetime -- the handler itself plays the "session" role. Records model calls (with token usage
  extracted from LLMResult), tool calls, retriever calls (retrieval events), and the chain's
  final output (final_output event). requested_tool_calls is captured with NO extra wiring:
  on_llm_end reads generations[0][0].message.tool_calls (LangChain's already provider-normalised
  AIMessage.tool_calls), maps args->arguments and id->call_id, and runs it through the same
  normaliser record_model_call uses. Chat message present but no tool calls -> [] ("asked for
  nothing"); plain non-chat Generation (no .message) -> field unset (None). invalid_tool_calls
  are NOT included (parse failures, not requests). Streaming needs no special case: the
  aggregated message arrives on on_llm_end. Payloads coerced to JSON-able primitives. Import is
  guarded: importing the module without langchain-core installed does not fail. DO NOT mix with
  @capture.tool on the same code path (double-record risk; the handler keeps its own
  run_id-based bookkeeping and does not bind the contextvar session).

evalshift.adapters.openai.wrap_openai(client: C) -> C            # [openai] extra
evalshift.adapters.anthropic.wrap_anthropic(client: C) -> C      # [anthropic] extra
evalshift.adapters.genai.wrap_genai(client: C) -> C              # [google-genai] extra
  Provider client wrappers (D-wrappers). Each returns a drop-in proxy over a client INSTANCE the
  user already built (openai.OpenAI/AsyncOpenAI; anthropic.Anthropic/AsyncAnthropic;
  google.genai.Client). Nothing is monkeypatched. Inside an active capture session
  (@capture.agent / agent_session) every intercepted call records ONE model_call via
  record_model_call: model_id, tools (the call's own tools kwarg or [] when absent -- NEVER the
  session's; Gemini reads config.tools, callables declared via the SDK's converter), input
  (ALWAYS a messages-style list: Anthropic system / Responses instructions / Gemini
  system_instruction -> leading {"role": "system"}; Gemini Content/Part -> role-tagged messages,
  function calls -> tool_calls, function responses -> "tool" messages), output (text),
  requested_tool_calls (via extract_requested_tool_calls; stream deltas reassembled; [] when the
  model asked for nothing), input/output tokens, latency_ms, generation_config (raw kwargs /
  config dict, allow-listed). cost_usd stays 0 (CLI prices at promote). Outside a session: inert.
  Intercepted: openai chat.completions.create + responses.create; anthropic messages.create +
  messages.stream (manager records on __exit__ from get_final_message()); genai
  models.generate_content + models.generate_content_stream + both under client.aio -- sync,
  async and stream=True forms. Everything else (parse/beta, with_raw_response, embeddings,
  chats, count_tokens, batches, files, live ...) is forwarded untouched and NOT recorded.
  Streaming: the returned stream is a proxy forwarding every attribute; records once when
  exhausted, closed or failed (partial output kept); usage from the final chunk (OpenAI chat
  needs stream_options={"include_usage": True}, else 0); an abandoned stream records nothing.
  Fail-open: the real call is never guarded (provider errors propagate; a failed request records
  nothing); a wrapper fault = "not recorded". Proxy is NOT an isinstance of the client class;
  evalshift.adapters._wrap.unwrap(proxy) returns the real client. Modules import-guard their SDK
  (import without the extra does not fail). OpenAI-compatible servers (Ollama, vLLM, llama.cpp,
  LM Studio, TGI, Together, Groq, Fireworks, OpenRouter) need no wrapper of their own:
  wrap_openai(OpenAI(base_url=...)); a server omitting usage records 0 tokens. Do NOT also call
  record_model_call for the same request (double record). Pairs with @capture.tool as intended.

evalshift.capture.requested.extract_requested_tool_calls(response: Any) -> list[dict] | None
  NOT a top-level export (full import path above, like evalshift.capture.toolset). Derives the
  tool calls the model REQUESTED in its response -- distinct from tools= (what it was OFFERED)
  and @capture.tool (what the app EXECUTED) -- for record_model_call(requested_tool_calls=...).
  Accepts an already-serialised response dict, an object with model_dump()/to_dict(), or the
  provider response object itself (attribute-walked; no provider SDK is ever imported, D-deps).
  Shapes: OpenAI Chat Completions choices[0].message.tool_calls[*] {id, type: "function",
  function: {name, arguments (JSON STRING)}} + the deprecated message.function_call single-call
  form; OpenAI Responses output[*] items with type == "function_call" {name, arguments (JSON
  string), call_id}; Anthropic content[*] blocks with type == "tool_use" {id, name, input};
  Gemini candidates[0].content.parts[*].functionCall (REST/camelCase) or .function_call
  (to_dict()/snake_case) {name, args, id?}. First choice/candidate only (n>1 alternatives are
  not concatenated).
  Returns items of EXACTLY {"name": str, "arguments": dict, "call_id": str | None}, response
  order; call_id is None where the provider has none (Gemini REST, legacy function_call).
  [] vs None are NOT interchangeable: [] = recognised response, model requested nothing (a real
  value); None = not a recognised response, OR one of its tool calls had no usable name -- the
  WHOLE response is then refused rather than reported one call short (same refusal rule as
  normalize_tools). Never substitute [] for None.
  Never raises. Degradations (debug log only, arguments -> {} for that one call): unparseable
  JSON arguments, JSON parsing to a non-object, non-object input/args. arguments is JSON-coerced
  via capture.generation.jsonable -> a deep copy, always sink-serialisable.

## Behavior rules (invariants)

- Gate off (EVALSHIFT_CAPTURE not truthy) -> every wrapper is a pure pass-through; nothing
  recorded, no tree built.
- No active agent session -> record_model_call, @capture.tool-wrapped calls, and
  capture.model_call recorders are inert no-ops (no toolset resolution or sidecar write either).
- Failed agent runs ARE captured: error event recorded (message=str(exc) or exception type name
  if empty; category=exception class name), partial capture written, original exception
  re-raises.
- Dedup: per-process registry keyed (suite, input_hash). Duplicate -> sink write returns None,
  no file. Registry clears on process restart or reset_config().
- agent_session with agent_input=None: input_hash is the hash of None (constant) -> with dedup
  on (the default), every session after the first for that suite is silently dropped. ALWAYS
  pass agent_input, or set conversation_id.
- conversation_id set -> input_hash = hash({agent_input, conversation_id, turn_index}), so
  repeated short turns ("yes", "1pm") don't dedup-collapse. conversation_id=None -> input_hash =
  hash(agent_input), byte-identical to pre-1.1.0.
- The SDK never returns the written capture_id to the caller; parent_capture_id must be
  user-managed or omitted.
- @capture.agent conversation kwargs are static per decoration; per-turn values require
  agent_session / agent_session_async.
- Tool span -> 2 events (tool_call + tool_result). model_call/error spans -> 1 event each.
- Event ordering: sort by (timestamp, monotonic op-order) -> dense sequence_index; deterministic
  under concurrency. Span timing/parentage stored under event.metadata["evalshift"].
- Streaming model_call latency auto-derived from with-block duration; record_model_call latency
  is 0 unless passed.
- Session scope is DYNAMIC (contextvar), not lexical: once an agent wrapper/session is active,
  tool and model-call recording works in any function called from it, however deep — no need to
  place recording calls lexically inside the decorated function or `with` block.
- Async: decorators auto-detect async def; contextvars propagate across await and into
  asyncio.gather child tasks (correct parentage for concurrent tools). Threaded tools
  (asyncio.to_thread / run_in_executor) are lock-safe.
- The agent has no user-recordable "final output" field: the manual surface writes no
  final_output event (LangChain adapter only). Persist the agent's answer as the last
  model_call's output.
- Hygiene defaults: dedup ON, max_captures 200/suite, TTL off, sampling off. Escape hatch:
  EVALSHIFT_MAX_CAPTURES=0 EVALSHIFT_DEDUP=off. GC runs only after a real disk write (Path
  returned), orders by file mtime, never recurses, never raises.
- require_model_call=True (opt-in): captures with no model_call event are dropped before
  redaction/serialization (debug log only).
- Redaction is REQUIRED at every capture point (agent decorator / agent_session /
  agent_session_async / handler ctor); there is no process-wide setter and no default. Runs in
  memory before serialization; masked values flow into trace events AND the derived tool
  input_hash. Raising redactor -> capture dropped (fail-closed).
- Redactable fields per span kind: tool: arguments,result,error |
  model_call: input,output,requested_tool_calls |
  retrieval: query,documents | guardrail: reason | final_output: text | error: message.
  NOT scrubbed: names, model_id, token counts, costs, timestamps, call ids,
  metadata["evalshift"], envelope fields (capture_id, suite, code_version, input_hash).
- All drops are silent except one logging.getLogger("evalshift") debug line per drop.
- retrieval / guardrail / final_output event types exist in the schema but have NO public
  recording API in the manual surface; only the LangChain adapter (retrieval, final_output) and
  internals emit them. Do not document them as user-recordable.
- Toolset sidecars (<base>/toolsets/<hex>.json) are content-addressed and written once per
  distinct fingerprint by ToolsetSink; a repeat write of the same toolset is a cheap no-op
  existence check, not a second write. GC (hygiene/gc.py) never touches this directory: its
  caller always passes <base>/captures/<suite>/, a sibling, and evict() never recurses into
  subdirectories regardless. No orphan sweeping exists for toolsets in this SDK.

## Capture file format

Path: <base>/captures/<suite>/<capture_id>.json  (capture_id = "cap_" + uuid4 hex)
Toolset sidecars: <base>/toolsets/<fingerprint-hex-no-prefix>.json (see ToolsetSink above)

Envelope keys (unchanged since schema 1.1.0, in order): schema_version, capture_id, suite,
input_hash, code_version, created_at (ISO-8601 UTC), trace, conversation_id, turn_index,
parent_capture_id (last three optional, null for standalone captures; added in 1.1.0). Schemas
2.0.0 and 2.1.0 (current) each changed a TRACE-internal field set (model_call gained
toolset_ref/tools_offered, then requested_tool_calls, below), not this envelope key list -- a
schema_version bump is still owed for a trace-internal field addition even when ENVELOPE_KEYS
itself is untouched (docs/SCHEMA.md).

trace (AgentTrace, the frozen CLI contract): run_id, prompt_id, example_id, role
("source"|"target"), events. Capture-time defaults: run_id=example_id=capture_id,
prompt_id=suite, role="source".

Event types and own fields (all events also carry: type, sequence_index, timestamp, metadata):
  model_call:  model_id, input, output, input_tokens, output_tokens, cost_usd, latency_ms,
               toolset_ref (str|None, "sha256:<hex>"), tools_offered (list[str]|None) -- both
               added in schema 2.0.0; None on a capture written before per-call toolset capture
               existed, real values (tools_offered=[] included) on every capture written after;
               requested_tool_calls (list[{name, arguments, call_id}]|None) -- added in schema
               2.1.0, what the MODEL ASKED FOR in its response, as opposed to tools_offered (what
               it was allowed to ask for) and the tool_call events (what the app actually ran).
               None = not recorded; [] = the model requested no tools.
  tool_call:   name, arguments, call_id, parent_call_id
  tool_result: name, call_id, result, error
  retrieval:   source, query, documents
  guardrail:   name, verdict ("pass"|"fail"|"warn"|"skipped"), reason
  final_output: text
  error:       message, category

metadata["evalshift"] block: span_id, start_ts, end_ts, [parent_call_id], and on tool_result:
input_hash (SHA-256 of the redacted tool arguments; feeds the (call_id, input_hash) -> result
replay fixture table).

Forward compat on read: older, same major -> migrate up chain; older, different (older) major
with no registered bridge -> refuse (ObsoleteSchemaVersionError). Both cases are concrete: a
2.0.0 capture upgrades to 2.1.0 through a registered identity edge (requested_tool_calls stays
absent -> None, never fabricated as []), while 2.0.0 registers NO migration from any 1.x capture
(there is no honest value tools_offered can take for one), so a 1.x capture raises
ObsoleteSchemaVersionError, not a silent upgrade; same
version -> as-is; newer minor/patch (same major) -> warn + best-effort read, unknown fields
dropped; newer major -> refuse (UnsupportedSchemaVersionError).

## Minimal examples

# 1. Decorated agent + tool + atomic model call
from evalshift import capture, record_model_call
from evalshift.capture.requested import extract_requested_tool_calls

@capture.tool(name="search_orders")
def search_orders(customer_id: str) -> dict:
    return {"orders": []}

ROUTER_TOOLS = [{"name": "search_orders", "description": "Look up a customer's orders.",
                  "input_schema": {"type": "object",
                                    "properties": {"customer_id": {"type": "string"}},
                                    "required": ["customer_id"]}}]

@capture.agent(suite="support", redact=True, tools=[])   # this agent never switches toolsets
def handle_ticket(query: str) -> str:
    record_model_call(model_id="claude-sonnet-5", tools=ROUTER_TOOLS,     # OFFERED
                      input={"query": query}, output="On it.",
                      # REQUESTED: what the model asked for, from the raw provider response.
                      # extract_requested_tool_calls returns None when it recognises nothing.
                      requested_tool_calls=extract_requested_tool_calls(response_dict))
    search_orders(customer_id="c42")                                      # EXECUTED
    return "done"
# Run: EVALSHIFT_CAPTURE=1 python agent.py -> .evalshift/captures/support/cap_<hex>.json
# Sidecar written once: .evalshift/toolsets/<fingerprint-hex>.json ("tools": ROUTER_TOOLS)

# 2. Streaming model call — MUST run inside an active agent session (a @capture.agent call or
#    an agent_session block); bare capture.model_call outside one is an inert no-op.
from evalshift import capture

@capture.agent(suite="support", redact=True, tools=[])
def answer(messages: list) -> str:
    with capture.model_call(model_id="claude-sonnet-5", tools=[], input=messages) as rec:
        for chunk in stream:
            rec.add_text(chunk.text)
        rec.set_usage(input_tokens=812, output_tokens=204, cost_usd=0.0031)
    return "done"

# 3. Multi-turn conversation (one capture per turn)
import uuid
from evalshift import capture, record_model_call
conv = f"conv_{uuid.uuid4().hex}"
messages = [{"role": "system", "content": "You are a scheduling assistant."}]
for i, user_text in enumerate(turns):
    messages.append({"role": "user", "content": user_text})
    with capture.agent_session(suite="scheduler", redact=True, tools=[], agent_input=messages,
                               conversation_id=conv, turn_index=i):
        reply = run_model(messages)
        record_model_call(model_id="claude-sonnet-5", tools=[], input=messages, output=reply)
    messages.append({"role": "assistant", "content": reply})

# 4. LangChain (pip install "evalshift-sdk[langchain]")
from evalshift.adapters.langchain import EvalShiftCallbackHandler
handler = EvalShiftCallbackHandler(suite="rag_agent", redact=True, tools=bound_tools)
chain.invoke({"question": q}, config={"callbacks": [handler]})

# 4b. Provider client wrapper (pip install "evalshift-sdk[openai]"; same shape for
#     evalshift.adapters.anthropic.wrap_anthropic / evalshift.adapters.genai.wrap_genai)
from openai import OpenAI
from evalshift.adapters.openai import wrap_openai
client = wrap_openai(OpenAI())          # OpenAI(base_url=...) for Ollama / vLLM / Groq ...

@capture.agent(suite="support", redact=True, tools=[])
def answer(q: str) -> str:              # each create() inside records one model_call
    r = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": q}])
    return r.choices[0].message.content or ""

# 5. Redaction — redact= is REQUIRED at every capture point; there is no global setter.
from evalshift import capture, default_redactor

@capture.agent(suite="support", redact=True, tools=[])     # mask emails/API keys
def handle(query: str) -> str: ...

@capture.agent(suite="fixtures", redact=False, tools=[])   # verbatim, on purpose
def replay(case: dict) -> str: ...

def my_redactor(value):                     # custom: any callable (value: Any) -> Any
    return default_redactor(value)          # must return a copy, never mutate

@capture.agent(suite="clinical", redact=my_redactor, tools=[])
def handle_case(record: dict) -> str: ...

# 6. Toolset switching + session inheritance (D-toolset) — tools= is REQUIRED at every capture
#    point; there is no global setter (same shape as rule 5's redact= example).
from evalshift import capture, record_model_call
TOOLSET_A = [{"name": "search", "description": "", "input_schema": {}}]
TOOLSET_B = [{"name": "refund", "description": "", "input_schema": {}}]

@capture.agent(suite="router", redact=False, tools=TOOLSET_A)  # this session's DEFAULT toolset
def agent(mode: str) -> str:
    record_model_call(model_id="m", tools=None, output="uses TOOLSET_A")   # inherits the session
    if mode == "refund":
        record_model_call(model_id="m", tools=TOOLSET_B, output="uses TOOLSET_B")  # own, overrides
    return "done"
# tools=[] asserts "no tools" for real (a value, not a default); tools=None means "same as the
# session"; any other value is this call's own and always wins, even one that fails to normalise
# (which does NOT fall back to the session — see docs/DECISIONS.md D-toolset).

# 7. Reading a written capture back (tooling / tests)
from pathlib import Path
from evalshift import load_envelope, MigrationError
try:
    env = load_envelope(Path(".evalshift/captures/support/cap_abc123.json").read_bytes())
    print(env.suite, env.capture_id, [e.type for e in env.trace.events])
except MigrationError as e:                 # read path raises; it does NOT fail open
    print(f"unreadable capture: {e}")
    # A 1.x capture lands here too: schema 2.0.0 registers no migration from the 1.x major
    # (ObsoleteSchemaVersionError, a MigrationError subclass) — re-run the agent to re-capture.

# model_call input convention (multi-turn): full per-turn context, role-tagged:
# [{"role": "system", ...}, ...prior turns..., {"role": "user", "content": current}]

## Troubleshooting: no file written

Check in order: (1) EVALSHIFT_CAPTURE not truthy; (2) wrong CWD — default .evalshift is
CWD-relative, set EVALSHIFT_DIR; (3) dedup collapsed it (classic: agent_session without
agent_input); (4) sampling skipped it; (5) require_model_call dropped it; (6) redactor raised
(fail-closed); (7) filesystem OSError (use MemorySink on read-only mounts). Enable
logging.getLogger("evalshift").setLevel(logging.DEBUG) to see which branch fired.

Read-only filesystem / Lambda: MemorySink (or EVALSHIFT_DIR at a writable mount, e.g. /tmp) fixes
the capture envelope, but do the EVALSHIFT_DIR part regardless -- toolset sidecars are always
file-based, so with no writable mount at all, toolset_ref stays unstamped on every capture and
the CLI refuses to promote it.
