Metadata-Version: 2.5
Name: infinity-observability-sdk
Version: 0.9.0
Summary: Unified observability SDK for Infinity Constellation services — tracing, structured logging, error tracking.
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: opentelemetry-api>=1.39.1
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.39.1
Requires-Dist: opentelemetry-sdk>=1.39.1
Requires-Dist: structlog>=25.1.0
Provides-Extra: django
Requires-Dist: opentelemetry-instrumentation-django>=0.48b0; extra == 'django'
Provides-Extra: logfire
Requires-Dist: logfire>=4.18.0; extra == 'logfire'
Provides-Extra: sentry
Requires-Dist: sentry-sdk>=2.0.0; extra == 'sentry'
Description-Content-Type: text/markdown

# Infinity Observability SDK

[![CI](https://github.com/infinity-constellation/infinity-observability-sdk/actions/workflows/ci.yml/badge.svg)](https://github.com/infinity-constellation/infinity-observability-sdk/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/infinity-observability-sdk)](https://pypi.org/project/infinity-observability-sdk/)
[![Python](https://img.shields.io/pypi/pyversions/infinity-observability-sdk)](https://pypi.org/project/infinity-observability-sdk/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![basedpyright](https://img.shields.io/badge/type_checker-basedpyright-blue)](https://github.com/DetachHead/basedpyright)

Unified observability for Infinity Constellation services — OTel tracing, OTLP
metrics, and structured logging from a single `configure_observability()` call.

This README is the **usage guide**: what to call, when, and the conventions
every service is expected to follow. It applies to any service in any
organisation; nothing below depends on a particular tenant, account or cluster.

Companion documents:

| Document | Covers |
|---|---|
| [`CONSUMING.md`](CONSUMING.md) | **Start here to adopt.** The four calls, what the service supplies (`service_name`) vs. what the deployment supplies (tenant, endpoint, credential), the consumer rules, healthy-vs-degraded, common mistakes. Short; written to be vendored into a service repo. |
| [`docs/REFERENCE.md`](docs/REFERENCE.md) | Every exported symbol, split into "the four you need" and "advanced"; fail-soft degradation catalogue; troubleshooting |
| [`docs/UPGRADING.md`](docs/UPGRADING.md) | 0.7 → 0.8 upgrade (symptoms + verification) and the compatibility policy |
| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | How the SDK fits the telemetry pipeline — collectors, VictoriaLogs/Grafana, agent-evidence joins |
| [`docs/redaction.md`](docs/redaction.md) | Full redaction reference (what is scrubbed, and why) |
| [`AGENTS.md`](AGENTS.md) | Working *on* the SDK, plus a condensed consumer guide for coding agents |

## What it installs

| Concern | Behaviour |
|---|---|
| Tracing | OTel `TracerProvider` + OTLP HTTP span exporter (`BatchSpanProcessor`) |
| Metrics | OTel `MeterProvider` + OTLP HTTP metric exporter, periodic export (60 s default) |
| Logging | structlog → one JSON object per line on stdout, with `trace_id`/`span_id` correlation |
| Error tracking | Sentry, when `sentry_dsn` is set (needs `[sentry]`) |
| Auto-instrumentation | httpx / pydantic-ai via Logfire (needs `[logfire]`); Django via `instrument_django()` (needs `[django]`) |
| Redaction | Credential/PII scrubbing on every log value and on SDK-set span attributes |

This is a **library**: it has no server, no transport of its own, and no
deployment topology of its own. It writes to the collector and to stdout; what
happens after that is described in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).

## Installation

```bash
pip install infinity-observability-sdk
```

Optional extras:

```bash
pip install infinity-observability-sdk[sentry]    # Sentry error tracking
pip install infinity-observability-sdk[logfire]   # pydantic-ai / httpx auto-instrumentation
pip install infinity-observability-sdk[django]    # Django OTel auto-instrumentation
```

Supports Python 3.11+ (the dev environment pins 3.12).

## Quick start — the four calls

A consumer needs four things and nothing else; everything further down this
README is advanced.

```python
from infinity_observability_sdk import (
    ObservabilityConfig,
    agent_run_context,
    configure_observability,
    get_logger,
)

configure_observability(ObservabilityConfig(service_name="my_service"))  # 1. once, at startup

logger = get_logger(__name__)                                             # 2. all logging
logger.info("service.started", port=8000)

with agent_run_context(run_id, task_id=task_id, actor_id=actor_id):       # 3. around agent work
    ...
```

```bash
python -m infinity_observability_sdk.conformance --root .                # 4. in CI
```

Call `configure_observability()` **once, at process startup**, before any code
creates a tracer, meter, or logger. `service_name` is the only value the
service decides; the business unit (tenant) comes from the deployment via
`INFINITY_OBSERVABILITY_BUSINESS_UNIT` (fallbacks: `OBSERVABILITY_BUSINESS_UNIT`,
then `local` in development; the winning source is logged as
`observability.tenant_resolved`), and the service is then identified
everywhere as `<business_unit>.<service_name>` (`service.name`). Scaffold all
four with `python -m infinity_observability_sdk.scaffold --service my_service --package my_package`.

## Where telemetry goes

You normally do **not** configure an endpoint. The SDK resolves one, in order:

1. `INFINITY_OBSERVABILITY_ENDPOINT`, if set explicitly;
2. in-cluster: the platform's default collector for the shared-platform profile
   (detected by the presence of `/var/run/secrets/kubernetes.io/serviceaccount`);
3. local development: `http://localhost:4318`.

Traces go to `<endpoint>/v1/traces` and metrics to `<endpoint>/v1/metrics`; the
API key, when present, is sent as the `X-Api-Key` header.

Both variables are the **deployment binding**: the platform (or the business
unit operating its own federated gateway) supplies them at deploy time; the
service never hard-codes them. The in-cluster default in step 2 is a heuristic
for the shared-platform profile only; federated deployments should always set
the endpoint explicitly. See [`CONSUMING.md`](CONSUMING.md#what-the-service-supplies-vs-what-the-deployment-supplies).

### Environment variables

| Variable | Required | Description |
|---|---|---|
| `INFINITY_OBSERVABILITY_BUSINESS_UNIT` | Yes, in any real deployment | Tenant / business unit. Becomes the `<bu>.` prefix of `service.name` and `service.namespace`. Fallbacks: `OBSERVABILITY_BUSINESS_UNIT`, then `local` when `ENVIRONMENT=development`. An explicit `ObservabilityConfig(business_unit_name=...)` is legacy (deprecated, supported until at least 1.0): it still wins, logs `observability.business_unit_name_deprecated`, and `observability.tenant_conflict` if the deployment disagrees. Nothing resolved: `observability.tenant_unresolved` warning, unqualified `service.name`. Metadata only — the gateway stamps the real tenant. |
| `INFINITY_OBSERVABILITY_ENDPOINT` | No | Collector base URL. Auto-detected when unset. |
| `INFINITY_OBSERVABILITY_API_KEY` | Only if the endpoint is set explicitly | Collector API key. When the endpoint is auto-detected, no key is needed — local and in-cluster collectors accept unauthenticated traffic. Setting the endpoint explicitly *without* a key disables trace/metric export and records a `traces+metrics` degradation (or raises `RuntimeError` with `fail_soft=False`). |
| `ENVIRONMENT` | No | Deployment environment. Read **once** at configure time and cached. `development` (case-insensitive) selects human-readable console logs; anything else — including unset or a typo — renders JSON. Defaults to `production`. |

## Configuration

```python
ObservabilityConfig(
    service_name="my_service",        # required, non-blank — the only required field
    business_unit_name=None,          # legacy/deprecated; leave None, the deployment resolves the tenant

    # Integrations
    instrument_pydantic_ai=True,      # default: True
    instrument_httpx=True,            # default: True
    sentry_dsn=None,                  # optional — enables Sentry if set
    sentry_environment=None,          # falls back to ENVIRONMENT
    sentry_traces_sample_rate=0.0,    # 0.0–1.0, default: 0.0
    trace_sample_rate=1.0,             # non-agent root spans, 0.0–1.0

    # Metrics
    enable_metrics=True,              # default: True — installs a MeterProvider
    metric_export_interval_millis=60000,

    # PII — opt-in, off by default (SOC2)
    sentry_send_default_pii=False,    # request/user PII in Sentry events
    httpx_capture_all=False,          # full httpx headers/bodies on spans

    # Logging
    log_level="INFO",                 # DEBUG | INFO | WARNING | ERROR | CRITICAL
    fail_soft=True,                   # keep local telemetry alive on startup failures

    # Advanced
    additional_resource_attributes={},  # extra OTel resource attributes
)
```

`ObservabilityConfig` is a frozen dataclass and validates on construction:
`service_name` (and `business_unit_name` when given) must not be blank,
`sentry_traces_sample_rate` and `trace_sample_rate` must be within `0.0–1.0`, and
`metric_export_interval_millis` must be positive.

`configure_observability()` is **idempotent** — safe to call more than once;
subsequent calls log at debug and return without reconfiguring. Logfire
auto-instrumentation is configured by the SDK when either
`instrument_httpx=True` or `instrument_pydantic_ai=True`; services must not call
`logfire.configure()` themselves.

## Sampling

Agent-run spans are identified by the boolean
`infinity_observability_sdk.agent_run` span attribute, which is set by both
`agent_run_context()` and the legacy `agent_context()` API. They are
head-sampled at 100% so every agent-run subtree is retained, regardless of its
parent decision or a custom `span_name=` override. Other root spans use
`ParentBased(TraceIdRatioBased(trace_sample_rate))`, while children follow
their parent.

When `OTEL_TRACES_SAMPLER` or `OTEL_TRACES_SAMPLER_ARG` is set, those
environment variables define the base sampler. `ObservabilityConfig.trace_sample_rate`
is ignored and a warning is emitted. The SDK always wraps that base sampler to
force agent-run spans to be sampled. Otherwise, `trace_sample_rate` defaults to
`1.0` and must be between `0.0` and `1.0`.

> **Upgrade note (0.8.0):** the SDK now installs an explicit sampler; services
> relying on `OTEL_TRACES_SAMPLER` keep their configured ratio, services without
> it get `trace_sample_rate` (default 1.0 — same as the previous default).

## Fail-soft startup

`ObservabilityConfig.fail_soft` defaults to `True`. Missing credentials,
exporter construction failures, foreign metric providers, and optional
Sentry/Logfire failures are recorded without preventing local tracing and
logging from working. Every degradation also emits an ERROR log event named
`observability.degraded`, with `component` and `reason`, on the standard stdout
path; this is the monitored signal for a telemetry-dead service. Call
`get_degradation_reasons()` to inspect the component/reason pairs. Set
`fail_soft=False` to restore strict startup errors.

`configure_logging()` deliberately runs before and outside the fail-soft guard:
local stdout logging is the floor everything else degrades to.

## Tracing

### Agent runs — `agent_run_context()` (preferred)

Wrap each agent run. This emits the standard `agent.run` span *and* binds
`run_id` / `task_id` / `actor_id` into the logging context, so every log line
written inside the block carries them as join keys:

```python
from infinity_observability_sdk import agent_run_context, get_logger

logger = get_logger(__name__)

with agent_run_context(
    run.id,
    task_id=task.id,
    actor_id=actor.id,
    backend=run.backend_type,
    systems_context_hash=snapshot.systems_context_hash,
    context_hash=pack.content_hash,
):
    logger.info("run.launched")   # carries run_id / task_id / actor_id
```

- Span attributes: `agent.run_id`, `env` and `infinity_observability_sdk.version`
  always; `agent.task_id`, `agent.actor_id`, `agent.backend`,
  `agent.systems_context_hash`, `agent.context_hash`,
  `agent.prompt_template_version` when supplied.
- Optional identifiers are **omitted** rather than emitted as `None`.
- Extra keyword arguments become span attributes verbatim (after redaction).
- `span_name=` overrides the default `agent.run` for a more specific name.
- Previous contextvar values are restored on exit.

The `run_id` / `task_id` / `actor_id` join keys are **independent of
`trace_id`** — that is deliberate: it is the correlation path that survives
sampling. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).

### Manual spans

```python
from infinity_observability_sdk import get_tracer

with get_tracer(__name__).start_as_current_span("thing.happened") as span:
    span.set_attribute("thing.id", thing.id)
```

### Legacy — `agent_context()` (frozen)

The AI-wattage span API. It emits an `agent:<name>` span carrying
`infinity_observability_sdk.*` cost attributes (including
`ai_wattage = approximate_person_hours * hourly_rate`) and sets OTel baggage for
downstream propagation.

**Its signature is backward-compatible and frozen — do not remove or rename
positional parameters. New code should use `agent_run_context()` instead.**

```python
from infinity_observability_sdk import agent_context

with agent_context(
    agent_employee_equivalent="data_engineer",
    hourly_rate=75.0,
    agent_name="my_agent",
    task_description="summarise quarterly report",
    task_instance_identifier="run-abc-123",
    approximate_person_hours=2.0,
    business_unit_name="my_bu",
    service_name="my_service",
) as span:
    ...
```

## Metrics

`configure_observability()` installs a real `MeterProvider`. Without it, every
instrument resolves against the no-op global meter and measurements are silently
dropped.

```python
from infinity_observability_sdk import AGENT_RUNS_STARTED_TOTAL, get_meter

runs_started = get_meter(__name__).create_counter(AGENT_RUNS_STARTED_TOTAL)
runs_started.add(1, {"backend": "devin"})
```

Use the exported constants, never string literals, so services cannot typo a
metric name apart:

| Constant | Metric |
|---|---|
| `AGENT_RUNS_STARTED_TOTAL` | `agent.runs.started_total` |
| `AGENT_RUNS_COMPLETED_TOTAL` | `agent.runs.completed_total` |
| `AGENT_RUNS_FAILED_TOTAL` | `agent.runs.failed_total` |
| `AGENT_RUN_WALL_TIME_SECONDS` | `agent.runs.wall_time_seconds` |
| `AGENT_RUN_COST_USD` | `agent.runs.cost_usd` |
| `AGENT_RUN_TOKENS_TOTAL` | `agent.runs.tokens_total` |
| `EVAL_CASES_PASSED_TOTAL` | `eval.cases_passed_total` |
| `EVAL_CASES_FAILED_TOTAL` | `eval.cases_failed_total` |
| `EVAL_SUITE_DURATION_SECONDS` | `eval.suite_duration_seconds` |
| `CONTEXT_PACK_FREEZE_DURATION_SECONDS` | `context.pack.freeze_duration_seconds` |
| `CONTEXT_PACK_FREEZE_BYTES` | `context.pack.freeze_bytes` |

`METRIC_NAMES` holds the full set, for validation.

`configure_metrics()` can also be called standalone — pass
`metric_readers=[InMemoryMetricReader()]` to collect measurements in tests
instead of exporting them.

OTel allows a meter provider to be installed only once. If something else got
there first, `configure_metrics()` does **not** pretend to have won: it reuses an
already-installed OTel SDK `MeterProvider` (logging a warning, since the export
configuration is then the other caller's) and raises `RuntimeError` for any
other provider type. Pass `enable_metrics=False` when a service installs its own
provider on purpose.

## Logging

`get_logger()` is the **only** application-facing log API. The stdlib `logging`
bridge exists so third-party libraries (Django, uvicorn, httpx) land in the same
pipeline — not so application code can bypass `get_logger()`. Do not log via
`logfire.*`: that emits span events and never reaches stdout. Services must not
call `logfire.configure()`; set `instrument_httpx` /
`instrument_pydantic_ai` on `ObservabilityConfig` instead.

```python
from infinity_observability_sdk import get_logger

logger = get_logger(__name__)
logger.info("run.started", run_id=run.id, backend="devin")
```

Records go to **stdout, one JSON object per line**. The SDK ships no log
transport: Vector tails stdout and forwards to VictoriaLogs.

### Record schema

The shape below is a versioned contract (`LOG_SCHEMA_VERSION`, currently `"1"`);
`tests/test_log_contract.py` fails if a processor change alters it.

| Key | Present | Format | Notes |
|---|---|---|---|
| `timestamp` | always | ISO-8601 UTC | always UTC |
| `level` | always | lowercase | |
| `event` | always | short dotted string, e.g. `run.started` | first positional arg |
| `service` | always | `<business_unit>.<service>` | set by the SDK |
| `logger` | always | logger name | |
| `trace_id` | inside a span | 32 lowercase hex | Grafana derived field |
| `span_id` | inside a span | 16 lowercase hex | Grafana derived field |
| `exception` | on `.exception()` / `exc_info` | whole traceback as one string | |
| `run_id`, `task_id`, `actor_id` | inside `agent_run_context()` | string | agent-evidence join keys |
| *your keys* | — | JSON scalars | `snake_case` |

Rules:

- Keys are `snake_case`; event names are dotted and lowercase.
- `REQUIRED_LOG_KEYS`, `SPAN_LOG_KEYS` and `exception` are **reserved** — the
  processor chain owns them and overwrites any application-supplied value.
- Tracebacks collapse into the single `exception` string, so a record never
  spans more than one line and no multiline stitching is needed downstream.
- The trace/span ID hex widths are fixed; Grafana derived fields and LogsQL
  queries match on them.
- `ConsoleRenderer` is selected **only** when `ENVIRONMENT` is exactly
  `development` (case-insensitive). Every other value renders JSON.
- The traceback is rendered to text by the SDK in *every* environment, including
  development, so the redactor sees it. Development therefore prints plain
  tracebacks rather than rich/better-exceptions ones.

## PII and redaction

PII capture is **opt-in**: `sentry_send_default_pii` and `httpx_capture_all` both
default to `False`. Enable them per service only where the data is needed and
permitted.

Everything the SDK emits is scrubbed before it leaves the process — log values
(including tracebacks) and the span attributes set through `agent_context()` /
`agent_run_context()`. Matches are replaced with `[REDACTED]` (exported as
`REDACTED`), and `redact_text()` / `redact_attributes()` are exported for
services building their own payloads.

In short: credential-shaped strings (`devinkey_*`, `gh[pousr]_*`,
`github_pat_*`, `AKIA*`/`ASIA*`, `xox[baprs]-*`, `sk-*`, JWTs, `?token=`,
`Authorization:` headers) are replaced anywhere they appear, and any key whose
name reads as a credential has its whole value replaced. Cookies are redacted
wholesale. Token *counting* keys (`max_tokens`, `prompt_tokens`, `tokens_total`)
and routing fields (`cache_key`, `idempotency_key`) are deliberately left alone.

Redaction is best-effort defence in depth — **do not deliberately log secrets and
rely on it**. The full rule set, its normalisation and its edge cases are in
[`docs/redaction.md`](docs/redaction.md).

## Django

Call `instrument_django()` **before** the ASGI/WSGI app is loaded — typically at
the top of `core/asgi.py`:

```python
from infinity_observability_sdk import configure_observability, ObservabilityConfig, instrument_django

configure_observability(ObservabilityConfig(service_name="my_api"))
instrument_django()
```

Requires `infinity-observability-sdk[django]`.

## Conformance check

The SDK ships a static consumer check:

```bash
uv run python -m infinity_observability_sdk.conformance \
  --root /path/to/service \
  --min-version 0.8.0 \
  --governed-metrics-file service-metrics.txt
```

It reports `stale_sdk_pin`, `missing_startup_configuration`,
`bypassed_log_path`, and `non_governed_metric_name` findings. `--help` states
the full consumer contract (including the rules the check cannot see) and
`--print-rules` emits it as JSON. A service may commit the same document as
`o11y-conformance.json` at its root — the CLI reads `source_dirs`,
`min_version`, `expected_version`, `exclude` and `governed_metrics` from it —
so the check runs with no flags. Services may also pass governed metric names
with repeated `--governed-metric` flags or use `check_service()` from a pytest.

`python -m infinity_observability_sdk.scaffold --service <svc> --package <pkg>`
emits the startup call (`service_name` only), a conformance pytest,
`o11y-conformance.json` and a condensed `OBSERVABILITY.md` into any service
repository; see [`CONSUMING.md`](CONSUMING.md). A line ending in
`# o11y-conformance: allow` is the documented escape hatch for intentional
exceptions. The check covers the SDK pin, startup bootstrap, log-path
bypasses, and metric-name vocabulary. It does not validate emitted log-envelope
keys or resource identity; those are covered at runtime by the SDK's
`tests/test_log_contract.py` and by the `service.name` set from
`ObservabilityConfig`.

## Standalone logging

For services that only need structured logging, without OTel tracing:

```python
from infinity_observability_sdk import configure_logging, get_logger

configure_logging(service_name="my_bu.my_service", log_level="INFO")
logger = get_logger(__name__)
```

Records follow the same logging contract. `configure_logging()` reads
`ENVIRONMENT` itself when called standalone.

## Working on the SDK

Development, testing, commit conventions, and the release pipeline are documented
in [`AGENTS.md`](AGENTS.md). In short:

```bash
uv sync --dev   # install deps
make test       # run tests
make ci         # full CI suite (install, lint, format-check, tests, build, package check)
```

Releases are automated by python-semantic-release on merge to `main` — the
version lives in the git tag, and `CHANGELOG.md` is frozen at v0.5.2 (release
notes live on the [Releases](https://github.com/infinity-constellation/infinity-observability-sdk/releases)
page). Do not bump the version or create tags manually.
