Metadata-Version: 2.5
Name: runtime-labs-sdk
Version: 0.7.0
Summary: OpenTelemetry tracing with eval payloads and verified source identities
Project-URL: Repository, https://github.com/arpandesai0/runtime-labs
Project-URL: Documentation, https://github.com/arpandesai0/runtime-labs/blob/main/packages/sdk/README.md
Requires-Python: <3.14,>=3.11
Requires-Dist: httpx<1,>=0.28
Requires-Dist: opentelemetry-api<2,>=1.45.0
Requires-Dist: opentelemetry-exporter-otlp-proto-http<2,>=1.45.0
Requires-Dist: opentelemetry-sdk<2,>=1.45.0
Requires-Dist: python-dotenv<2,>=1.1
Requires-Dist: runtime-labs-contracts==0.7.0
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.30; extra == 'anthropic'
Provides-Extra: cerebras
Requires-Dist: cerebras-cloud-sdk; extra == 'cerebras'
Provides-Extra: cohere
Requires-Dist: cohere>=5; extra == 'cohere'
Provides-Extra: fireworks
Requires-Dist: fireworks-ai>=1; extra == 'fireworks'
Provides-Extra: google
Requires-Dist: google-genai<4,>=2.27; extra == 'google'
Provides-Extra: groq
Requires-Dist: groq>=0.4; extra == 'groq'
Provides-Extra: huggingface
Requires-Dist: huggingface-hub>=0.22; extra == 'huggingface'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: langgraph
Requires-Dist: langgraph==1.2.18; extra == 'langgraph'
Provides-Extra: litellm
Requires-Dist: litellm>=1; extra == 'litellm'
Provides-Extra: mcp
Requires-Dist: mcp<3,>=1.28; extra == 'mcp'
Provides-Extra: mistral
Requires-Dist: mistralai>=1; extra == 'mistral'
Provides-Extra: ollama
Requires-Dist: ollama>=0.3; extra == 'ollama'
Provides-Extra: openai
Requires-Dist: openai>=1; extra == 'openai'
Provides-Extra: openai-agents
Requires-Dist: openai-agents>=0.1; extra == 'openai-agents'
Provides-Extra: otlp
Requires-Dist: opentelemetry-exporter-otlp-proto-http<2,>=1.45.0; extra == 'otlp'
Provides-Extra: together
Requires-Dist: together>=1; extra == 'together'
Description-Content-Type: text/markdown

# RuntimeLabs local Python SDK

The SDK emits OpenTelemetry spans with full eval payloads for LangGraph execution, the
OpenAI Agents SDK, LangChain models, tools and retrievers, and model calls through the
OpenAI, Anthropic and Google SDKs and the clients individual developers use for Groq,
OpenRouter, Ollama, Mistral, LiteLLM and others (streaming included). The scanner's strict `DeploymentManifest`
adds `runtime_labs.node_id` when running code matches a manifest node. Tracing also
works with a missing manifest: those operations remain unattributed and the handle
reports fixed diagnostic codes.

## Install and initialize

Install or upgrade the published SDK (contracts install at the matching version):

```bash
pip install -U 'runtime-labs-sdk[google,langgraph]'
python -c 'import importlib.metadata as m; print(m.version("runtime-labs-sdk"))'
```

Releases are published automatically from `main`; see the repository's GitHub
Releases for what changed in each version.

From this workspace:

```bash
uv sync
uv run python examples/sdk_offline/demo.py
```

For an application with its own environment, install both local distributions
(substitute the checkout directory):

```bash
uv pip install /path/to/runtime-labs/packages/contracts \
  '/path/to/runtime-labs/packages/sdk[google,langgraph]'
```

`runtime-labs-sdk` is independently packaged and has no scanner dependency. The
Google and LangGraph adapters are optional and loaded at initialization; importing
`runtime_labs` does not import either integration. Unsupported dependency versions
are diagnosed and their adapters are skipped.

Connect and scan the repository in the web UI first. No manifest file is downloaded
or configured by the customer: the SDK asks the backend for the latest successful
scan's manifest at startup. Inside a Git checkout it identifies the repository by its
`origin` remote; SSH and HTTPS remotes normalize to the same host/owner/repository
identity. It also sends the Git blob hash of each application `.py` file (contents
only, never paths). Where the remote is missing or not connected, such as in a Docker
image, the backend picks the workspace repository by those hashes. It then finds the
commit with the most matching files and returns that commit's manifest, or the closest
scanned one, so a deployment can run behind the latest scan.

A node binds by its file's repository path, which the backend supplies for recognized
files, or else its path suffix, and its symbol, so it stays bound when its body
changes, its lines move, or the image lays out the repository differently. Newly added
code is attributed after the next scan.

Set the key from Settings › API key in the web app, in the environment or an
application `.env`. Releases know the hosted API, so nothing else is needed:

```dotenv
RUNTIME_LABS_API_KEY=rl_…
```

To point the SDK at another backend, set `RUNTIME_LABS_API_URL` or pass `api_url=`.
Keys starting `rl_dev_` come from a development backend and use `http://127.0.0.1:8000`.

At startup, before constructing provider clients or building/executing graphs:

```python
import runtime_labs

sdk = runtime_labs.init()  # or init(api_key="rl_…"), which takes precedence

try:
    # Normal application startup and execution.
    ...
finally:
    sdk.flush()
    sdk.shutdown()
```

The API key is the workspace's only SDK credential: each workspace has exactly one,
created (and rotated) by an admin under Settings › API key. It is sent as an
Authorization bearer header for manifest lookup and telemetry delivery, and it can
do nothing else: it cannot read traces, graphs or repositories. The backend serves
the manifest only for repositories connected to the key's workspace and accepts
spans only for them. The key is hidden from configuration representations and is
never included in span payloads or provider credentials.

Configuration rules:

- `api_url=` overrides process `RUNTIME_LABS_API_URL`, then the nearest `.env` value,
  then the local default above. SDK settings are read without interpolation or
  mutating the application's environment.
- By default the SDK resolves the checkout's `origin` and HEAD, requests
  `POST /api/sdk/manifest`, validates the strict deployment manifest, and holds it
  in memory. No manifest cache file or customer download step is required.
- The backend resolves the complete canonical Git identity, reads the successful
  scan's saved artifact, and returns the manifest and repository ingestion route.
  Duplicate matching checkouts and unavailable artifacts are rejected rather than
  guessed. The manifest need not be from the running commit.
- The API key and `api_url` are the only settings. No environment variable or
  `.env` entry supplies a manifest; tests and offline demos load one in-process with
  `runtime_labs.testing.use_offline_manifest(path)`, and their spans go only to the
  exporter they pass.
- A failed lookup never breaks application execution. Inspect `sdk.diagnostics`
  for `api_key_missing`, `api_key_rejected`, `repository_not_connected`,
  `repository_identity_unavailable`, or `manifest_fetch_failed`. Unsupported or
  unmatched spans stay unattributed.
- Initialization fetches once with a five-second HTTP timeout; retry after fixing
  backend/scan availability by shutting down and initializing again.

## Startup logs

`init` writes a short startup report to stderr through the standard `runtime_labs`
logger (stdout stays reserved for local span output):

```text
****** RuntimeLabs - Trust in every outcome ******
Verifying API key...
Downloading manifest file...
Setup completed
***************************************************
```

The lines are collected while `init` runs and printed together as one block, so
other startup output cannot interleave with them. Failures appear in the block, for
example `API key verification failed`. The API
key is never logged. A handler is added only if the application has not attached
one to `runtime_labs`; silence the report with
`logging.getLogger("runtime_labs").setLevel(logging.WARNING)`.

## Public API and local output

`init(*, api_key=None, exporter=None, tracer_provider=None, capture_payloads=True, api_url=None)`
returns an SDK handle with:

- `diagnostics`: a sorted tuple of sanitized, deduplicated diagnostic codes.
- `sources.manifest`: a detached strict-contract snapshot; editing it cannot change
  the immutable attribution indexes. Repository/revision metadata stay here.
- `flush(timeout_millis=30000) -> bool`: flush only this SDK's exporter.
- `shutdown()`: flush, disable instrumentation, restore owned hooks, and release
  SDK-owned resources. It is safe to call repeatedly.

Identical initialization returns the same handle. A different key, resolved path,
exporter, explicit provider or payload setting returns that handle with
`initialization_conflict`,
without installing more hooks. Shut down and initialize again to change settings;
construct new graphs after reinitialization. Shutdown should follow completion or
cancellation of application work; it does not await running requests.

After successful runtime bootstrap, the SDK automatically exports OTLP/HTTP spans
to the repository route supplied by the backend. The OTLP exporter is included in
the base SDK dependency set. Explicit exporters and standard `OTEL_*` settings
remain supported; `OTEL_TRACES_EXPORTER=none` disables delivery. Offline manifest
mode retains the console default. Console records contain trace/span/parent IDs,
start/end timestamps, duration, status, and allowed attributes including application
payloads. HTTP authentication, environment dumps, and source bodies are not captured.
Errors use only `error.type` values `cancelled`, `timeout` or `_OTHER`; automatic
exception events and status descriptions are disabled. Application code and other
instrumentation retain their own logging/export policies.

## Payloads for evals

Full payload capture is enabled by default. The MVP performs **no redaction or SDK
size limiting**. Each payload attribute is a JSON string, supported by standard
OTel exporters. Decode it with `json.loads(span.attributes[attribute_name])`.

| Span | Payload attributes |
| --- | --- |
| Workflow / agent | `runtime_labs.input`, `runtime_labs.output`: input state and returned value |
| Google model call | `runtime_labs.request`: original model, contents and supplied generation config; `runtime_labs.response`: returned response body, candidates, usage, parsed output and automatic tool-calling history |
| Google model call | `gen_ai.input.messages`, `gen_ai.output.messages`: standard message/part representation; `gen_ai.system_instructions` when supplied |
| Tool execution | `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`: bound arguments and returned value |

The `runtime_labs.*` payload attributes carry application data, not additional
source identity fields. `runtime_labs.node_id` remains the only custom source ID.
Prompts, completions, all returned candidates and tool payloads are stored in full.
Raw provider request/response attributes retain provider-specific body fields;
HTTP options and `sdk_http_response` transport envelopes are not body payloads.
The SDK does not inspect client credentials or automatically export its API key.

Request/input snapshots are taken before execution, so in-place application or
Google automatic function-calling mutations cannot rewrite the original input.
The returned automatic function-calling history retains its intermediate tool
calls/results. Workflow outputs contain the final graph state, including the
application's answer, enabling complete input/output evaluation examples. Runtime
trace and span IDs join these payloads to the manifest's source node IDs.

JSON-compatible values, Pydantic models, dataclasses, enums, dates and bytes are
supported; bytes are encoded as base64. Callable tool definitions store callable
names and qualified names rather than executable code. Tool arguments are serialized
independently: unsupported or cyclic argument values (such as database sessions
and ORM models) are omitted while the other arguments, including defaults, remain
captured. `runtime_labs.tool.arguments.omitted` lists the omitted parameter names,
and the handle reports `tool_arguments_omitted`. No object internals or string
representations are inspected as a fallback. Unsupported or cyclic values in other
payloads produce `payload_capture_failed` and omit that payload attribute while
preserving application results/exceptions and other span data. A failed or
cancelled operation keeps its input; it does not invent a response.

To select metadata-only output explicitly:

```python
sdk = runtime_labs.init(api_key="rl_…", capture_payloads=False)
```

OTel/application processors and downstream collectors can have their own limits;
the RuntimeLabs SDK does not introduce payload truncation. Local JSON and OTLP
export receive the same payload attributes. Evaluation execution/dataset storage
are separate later components; this change supplies their runtime examples.

For assertions, supply the standard OTel in-memory exporter:

```python
from runtime_labs.export import InMemorySpanExporter

exporter = InMemorySpanExporter()
sdk = runtime_labs.init(api_key="rl_…", exporter=exporter)
# Execute work; then inspect exporter.get_finished_spans().
```

The explicitly selected in-memory exporter retains spans until `clear()` or
`shutdown()`; applications should bound its use. Custom exporters use the standard
OTel `SpanExporter` interface. Export/flush/shutdown exceptions become diagnostic
codes and do not replace application results or exceptions. Local export is
synchronous, so exporter work contributes latency.

An application-owned global `TracerProvider` is respected. An optional explicit
`tracer_provider` also works. The SDK adds one reusable processor scoped to
`runtime_labs`, never shuts down an application provider, and does not export its
other instrumentation scopes through the local exporter. If no application provider
exists, the SDK creates a private OTel provider without taking the global slot.
Active OTel context still establishes parentage across both providers. Application
processors on a shared provider can also observe SDK spans.

Remote OTLP/HTTP export requires **both** explicit standard settings:

```bash
uv pip install '/path/to/runtime-labs/packages/sdk[otlp]'
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
```

`OTEL_EXPORTER_OTLP_ENDPOINT`, generic protocol/timeout/header settings and the
trace-specific settings are handled by the standard exporter. HTTP/protobuf is the
supported protocol. An endpoint alone never enables remote export; missing extras,
missing endpoint or unsupported protocol produce diagnostics and local fallback.
The pinned exporter's per-instance transport logger is silenced so server reasons
and transport exceptions cannot echo private text; failures use sanitized SDK
diagnostics. Its optional internal metrics use a no-op provider.
`OTEL_TRACES_EXPORTER=none` discards SDK output. No collector/server is included.

## Correlation identifiers and trace context

The SDK records the identifiers a framework uses to tie runs together, verbatim,
and never decides what they mean. On every LangGraph `invoke`/`ainvoke` it copies
allowlisted keys from the run config onto the workflow span as
`runtime_labs.ctx.<framework>.<key>`, for example
`runtime_labs.ctx.langgraph.thread_id`. The defaults are LangGraph's `thread_id`,
`checkpoint_id`, `checkpoint_ns` and `run_id`. Only allowlisted keys leave the
process, because applications often keep user IDs and secrets in the same config;
only non-empty strings, integers and UUIDs are copied, truncated to 256 characters.

Extend the allowlist without code changes:

```bash
export RUNTIME_LABS_CORRELATION_KEYS=langgraph.tenant,langgraph.session
```

or with `runtime_labs.init(api_key=..., correlation_keys={"langgraph": ["tenant"]})`.
RuntimeLabs pipelines map these raw identifiers to concepts such as conversations
with rules that are data, so new frameworks need a rule, not an SDK release.

Within one process, span parents follow OpenTelemetry context, which LangGraph and
asyncio carry into their own threads and tasks. Work that crosses a process or
service boundary (an HTTP call, a queue message, an outbox row) must carry the
context itself, as a standard W3C `traceparent`:

```python
headers = runtime_labs.inject_context()        # producer
with runtime_labs.continue_trace(headers):     # consumer
    handle(message)
```

Code that starts raw threads should run work with
`contextvars.copy_context().run(...)` to keep the parent.

## Supported interfaces and conventions

| Component | Pinned/tested version | Coverage |
| --- | --- | --- |
| CPython | 3.11–3.13 | Code identity and UTF-8 call positions |
| Google Gen AI | 2.27–3.x (verified 2.27.0, 3.0.0) | `Models`/`AsyncModels` `generate_content`, `generate_content_stream`, `embed_content`; chats send through these |
| Google Generative AI (legacy `google-generativeai`) | 0.5+ (verified 0.8.6) | `GenerativeModel.generate_content` and `generate_content_async`, which chat `send_message` uses; a stream is traced until iterated to the end or dropped |
| OpenAI | 1+ (verified 3.46.0) | Chat Completions, Responses and legacy Completions `create` (sync, async, `stream=True`), `parse`, and embeddings. The `.stream()` helpers send through `create`. Azure OpenAI uses the same classes |
| Anthropic | 0.30+ (verified 2.6.0) | `messages.create` (also `stream=True`) and `messages.stream`, sync and async, including the beta namespace; Bedrock and Vertex clients are the same classes |
| OpenAI-compatible services | through the `openai` client | OpenRouter, Groq, Together, Fireworks, DeepSeek, xAI, Perplexity, Mistral, Cerebras, SambaNova, NVIDIA, Hugging Face, GitHub Models, Gemini's compatibility endpoint, Ollama and LM Studio are the OpenAI adapter, and the span's `gen_ai.provider.name` names the service by the client's `base_url` |
| Groq, Cerebras, Together, Fireworks | `groq` 0.4+, `cerebras-cloud-sdk`, `together` 1+, `fireworks-ai` 1+ (verified groq 1.7.0, cerebras 1.91.0, together 3.1.1, fireworks 1.2.33) | `chat.completions.create` (and `completions.create`, Together's `embeddings.create`), sync, async and `stream=True`; these clients are generated like OpenAI's |
| Mistral | `mistralai` 1+ (verified 3.7.0) | `chat.complete`, `chat.stream` and their `_async` forms, and `embeddings.create` |
| Ollama | `ollama` 0.3+ (verified 0.6.3) | `Client` and `AsyncClient` `chat`, `generate` (also streaming) and `embed`, and the module-level `ollama.chat` and friends |
| Cohere | `cohere` 5+ (verified 7.3.1) | `ClientV2` and `Client` `chat`, `chat_stream` and `embed`, sync and async |
| LiteLLM | `litellm` 1+ (verified 1.108.3) | `completion`, `acompletion`, `text_completion`, `embedding` (and `a` forms); the provider is read from the model prefix (`groq/llama-3.3-70b`), and the requests LiteLLM makes through a provider's SDK are part of its span, not spans of their own |
| Hugging Face | `huggingface-hub` 0.22+ (verified 1.33.0) | `InferenceClient` and `AsyncInferenceClient` `chat_completion`, also reached as `chat.completions.create` |
| OpenAI Agents SDK | 0.1+ (verified 0.24.0) | A trace processor turns the framework's agent, tool, handoff and guardrail spans into spans, nested as the framework nested them. Its model spans are left to the OpenAI adapter |
| LangChain callbacks | `langchain-core` 0.3+ (verified 1.7.0) | One handler for every chat model, LLM, tool and retriever. A model whose SDK has an adapter above is recorded there once; any other (Bedrock, Vertex, ...) is recorded from what LangChain reports |
| LangGraph | 1.2.18 | `StateGraph` construction, `add_node`, `compile`; compiled `invoke`/`ainvoke` |
| LangChain Core | 1.7.0 (workspace lock) | LangGraph callback signature and context propagation |
| MCP (optional `mcp` extra) | 2.3.0 | `ClientSession.initialize`, `list_tools`, `call_tool`: one CLIENT span per request ([MCP conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md)) with the server's name, the tools it offered (`runtime_labs.mcp.tools`), arguments and result; trace context is injected into `params._meta` |
| OTel API/SDK | 1.45.1 | Context, tracing, processors and standard exporters |
| OTel OTLP/HTTP exporter | 1.45.1 (optional) | Explicit HTTP/protobuf export |
| OTel Python semantic-conventions package | 0.66b1 (SDK dependency) | SDK resource conventions |
| GenAI conventions | `51a0f580f8a6ad804ca4de5e033683c6fbbe5906` | Development conventions, mapping in `semconv.py` |

Verified official references: [OTel Python instrumentation](https://opentelemetry.io/docs/languages/python/instrumentation/),
[GenAI inference conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/51a0f580f8a6ad804ca4de5e033683c6fbbe5906/docs/gen-ai/client-inference.md),
[agent/workflow/tool conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/51a0f580f8a6ad804ca4de5e033683c6fbbe5906/docs/gen-ai/gen-ai-agent-spans.md),
[Google SDK](https://github.com/googleapis/python-genai/tree/v2.27.0),
[LangGraph](https://docs.langchain.com/oss/python/langgraph/overview), and
[streaming boundaries](https://docs.langchain.com/oss/python/langgraph/streaming).
GenAI conventions remain in development; their revision is explicit rather than
silently adopting future changes.

Provider spans use `generate_content`, `CLIENT`, and `gcp.vertex_ai`, `gcp.gemini`
or `gcp.gen_ai` according to runtime backend metadata. Request/response models come
from actual request/response metadata. Tokens come only from provider usage:
input = prompt count, output = candidate count plus reasoning count when reported;
reasoning and cache-read counts are separately reported when available. Missing
counts remain absent. For automatic function calling, the final SDK response's
reported usage is recorded; counts from hidden HTTP rounds are not estimated or
aggregated. One span covers the whole public logical call, including these rounds.
Return identity, arguments, exceptions and async cancellation are preserved.

Workflow, agent and tool spans use `INTERNAL` and standard `invoke_workflow`,
`invoke_agent` and `execute_tool` operations and names. The only custom source
attribute is `runtime_labs.node_id`. No extra source IDs or static parent IDs are
attached. The Google adapter honors and sets standard OTel instrumentation
suppression, and deduplicates reentry. Uncooperative third-party instrumentors that
ignore suppression cannot be deduplicated reliably.

## Source-binding policy

The SDK consumes `DeploymentManifest` directly and validates all references using
the shared strict contract. It neither scans nor executes repository source, calls
an LLM, generates IDs, or imports modules selected by manifest data.

Binding finds the manifest node with the running function's name in the manifest file
that shares the longest path suffix with the running file. Installed packages never
bind. Paths are compared from the end because an image may lay out the repository
differently from the checkout. Names at one location are aliases of one node.

In a file identical to the scanned one, the executed CALL instruction must match the
exact manifest range, including UTF-8 byte columns, so same-line sync/async calls
remain distinct, and workflow construction must fall within its constructor range. In
an edited file those positions have moved, so the nearest node with that name binds.
Registered callback objects identify nested agents. Known module-level Python
tools and their already-loaded aliases receive wrappers; registration by itself
never emits a tool span. Ordinary functions such as `finalize_response` receive
no span. Code with no matching node, or several equally good ones, produces generic
spans and diagnostics.

Runtime parents always come from OTel context. A direct shared-tool call is under
the executing agent. A Google automatic tool call is under that logical provider
span, itself under the executing agent. Static registration/containment edges do
not override these parents. LangGraph's public runtime events primarily describe
streaming and do not expose this example's exact Python callback identities, so
this adapter uses narrowly tested construction/registration/execution wrappers
without callback arguments or customer decorators.

## Offline acceptance harness

`examples/sdk_offline` contains an unchanged source copy of the sibling simple
example and its saved verified manifest. The sibling application's source and
configuration are untouched. The demo denies socket connections and uses real
pinned clients/graphs with HTTPX mock transports and fake Vertex credentials.

Each of billing, technical and general, through both sync and async graph entry
points, asserts this six-span tree (the selected agent varies):

```text
invoke_workflow build_workflow
  invoke_agent triage_agent
    generate_content offline-request-model
  invoke_agent billing_agent | technical_agent | general_agent
    execute_tool lookup_policy
    generate_content offline-request-model
```

All six spans in each run must have IDs in the loaded manifest. Across the three
branches, the harness observes all ten distinct nodes, including all four model
sites. It checks the actual tool parent on every run and prints node IDs with the
tree. This is an observation of the offline runs, not a source coverage estimate.
Tests additionally exercise true async callbacks/provider calls using a newly
compiled scanner manifest, automatic tool execution, parallel requests, stale
loaded code and failure/privacy/lifecycle cases. Tests deny network globally.

## Limitations

Streaming graph execution has no workflow-lifetime adapter; registered callbacks
may still emit leaf spans. The OpenAI Agents SDK's `generation` and `response` spans
are not read, so a model reached some other way than an adapter above has an agent span
but no model span. Chains and runnables from LangChain are
not spans. Tracing disabled in the Agents SDK (`set_tracing_disabled`) disables its
adapter too. Provider-driven tools are covered when they are already-loaded, manifest-matched
Python functions passed to Google; arbitrary callable objects, local tool closures,
cached pre-init callable references, decorators with replaced code, packaged/zip or
unavailable source, and other framework/provider versions are not guaranteed.
Graphs must be constructed after initialization. No whole-program tracing or
instrumentation of ordinary application functions is performed. Hosted lookup,
authentication, collectors, providers not listed above (AWS Bedrock through `boto3`, xAI's
gRPC SDK, Replicate and others without an adapter report `model_call_unobserved`), evaluation execution and cross-commit
reconciliation are outside this milestone.
