Metadata-Version: 2.4
Name: livekit-evals
Version: 0.3.4
Summary: Track and evaluate LiveKit agent sessions with automatic metrics, transcripts, and usage analytics
Author-email: superbryndev <support@superbryndev.com>
License: MIT
Project-URL: Homepage, https://github.com/superbryndev/livekit-evals
Project-URL: Documentation, https://github.com/superbryndev/livekit-evals#readme
Project-URL: Repository, https://github.com/superbryndev/livekit-evals
Project-URL: Issues, https://github.com/superbryndev/livekit-evals/issues
Keywords: livekit,agents,analytics,metrics,evaluation,voice-ai
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Communications :: Telephony
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: livekit-agents>=0.8.0
Requires-Dist: aiohttp>=3.9.0
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23; extra == "test"
Dynamic: license-file

# LiveKit Evals

[![PyPI version](https://badge.fury.io/py/livekit-evals.svg)](https://badge.fury.io/py/livekit-evals)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Track and evaluate your LiveKit voice AI agents with just 3 lines of code.**

Automatically capture transcripts, usage metrics, latency data, and session analytics from your LiveKit agents. Perfect for monitoring, debugging, and optimizing your voice AI applications.

## ✨ Features

- 🎯 **3-Line Integration** - Add to any LiveKit agent in seconds
- 📝 **Precise Transcripts** - Accurate timing using VAD state change events
- 📊 **Usage Metrics** - Track LLM tokens, STT duration, TTS characters
- ⚡ **Latency Tracking** - Monitor LLM, STT, and TTS performance
- 🔍 **Auto-Detection** - Automatically extracts models, providers, and configuration
- 📞 **SIP Support** - Detects SIP trunking and phone numbers
- 🎥 **Recording URLs** - Captures egress recording links
- 🎙️ **Call Recordings** - Automatic call recording to S3 (MP3 format, enabled by default, no S3 config needed)
- 🔊 **Stereo Recording** - Dual-channel recording with agent on left, caller on right (one param)
- 🗂️ **Custom Data** - Attach arbitrary JSON to every webhook payload via `custom_data`
- 🔐 **Secure** - API key authentication; temporary S3 credentials fetched per-session
- ⏱️ **Turn-Detection Latency** - End-of-utterance (EOU) + transcription delays per turn, the hidden part of response latency
- 🌐 **Network Telemetry** - Connection quality, reconnects, and WebRTC stats (jitter, RTT, packet loss, audio dropouts)
- 🚨 **Provider Errors** - Typed STT/LLM/TTS failures with recoverability, model, and provider
- 🗣️ **Conversation Analytics** - Talk ratio, silence, interruptions, backchannels, DTMF keypresses, full SIP attributes

## 🚀 Quick Start

### Prerequisites

1. **Get your API key** from [https://app.superbryn.com/api-keys](https://app.superbryn.com/api-keys)
2. **Set environment variable:**
   ```bash
   export SUPERBRYN_API_KEY=your_api_key_here
   ```

### Installation

```bash
pip install -U "livekit-evals>=0.3.0"
```

### Integration (3 Lines)

Add these lines to your LiveKit agent:

```python
from livekit_evals import create_webhook_handler

async def entrypoint(ctx: JobContext):
    # ... your existing setup code ...
    
    # 1. Create webhook handler (recording enabled by default)
    webhook_handler = create_webhook_handler(
        room=ctx.room,
        is_deployed_on_lk_cloud=True,  # Set to False if self-hosting
        # disable_recording=True  # Uncomment to disable call recording
    )
    
    # ... create your session ...
    session = AgentSession(
        llm=openai.LLM(model="gpt-4o-mini"),
        stt=deepgram.STT(model="nova-3"),
        tts=cartesia.TTS(voice="..."),
    )
    
    # ... your session setup ...
    await session.start(agent=YourAgent(), room=ctx.room)
    
    # 2. Attach to session (MUST be after session.start)
    if webhook_handler:
        webhook_handler.attach_to_session(session)
        # 3. Send webhook on shutdown
        ctx.add_shutdown_callback(webhook_handler.send_webhook)
    
    await ctx.connect()
```

**That's it!** 🎉 Your agent will now automatically track all session data and send it to your webhook endpoint.

## 📖 Full Example

Here's a complete working example:

```python
import logging
from dotenv import load_dotenv
from livekit.agents import (
    Agent,
    AgentSession,
    JobContext,
    WorkerOptions,
    cli,
)
from livekit.plugins import cartesia, deepgram, openai, silero

# Import livekit-evals
from livekit_evals import create_webhook_handler

logger = logging.getLogger("agent")
load_dotenv()


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="""You are a helpful voice AI assistant.
            You eagerly assist users with their questions.""",
        )


async def entrypoint(ctx: JobContext):
    # Logging setup
    ctx.log_context_fields = {"room": ctx.room.name}

    # Initialize webhook handler (auto-detects all metadata)
    webhook_handler = create_webhook_handler(
        room=ctx.room,
        is_deployed_on_lk_cloud=True  # Set to False if self-hosting
    )

    # Set up voice AI pipeline
    session = AgentSession(
        llm=openai.LLM(model="gpt-4o-mini"),
        stt=deepgram.STT(model="nova-3", language="en"),
        tts=cartesia.TTS(voice="your-voice-id"),
        vad=silero.VAD.load(),
    )

    # Start the session
    await session.start(agent=Assistant(), room=ctx.room)

    # Attach webhook handler to capture events
    # IMPORTANT: Must be after session.start()
    if webhook_handler:
        webhook_handler.attach_to_session(session)
        ctx.add_shutdown_callback(webhook_handler.send_webhook)

    # Connect to room
    await ctx.connect()


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```

## 🔧 Configuration

### Environment Variables

| Variable | Required | Description | Default |
|----------|----------|-------------|---------|
| `SUPERBRYN_API_KEY` | ✅ Yes | API key for webhook authentication and call recording | - |
| `LIVEKIT_PROJECT_ID` | ⚪ Optional | LiveKit project ID | Auto-detected from `LIVEKIT_URL` |
| `AGENT_ID` | ⚪ Optional | Unique agent identifier | `"livekit-agent"` |
| `VERSION_ID` | ⚪ Optional | Agent version identifier | `"v1"` |

**Note:** Call recording is **enabled by default**. Temporary S3 credentials are fetched automatically using your `SUPERBRYN_API_KEY` -- no S3 configuration needed.

### Setting Environment Variables

**Linux/Mac:**
```bash
export SUPERBRYN_API_KEY=your_api_key_here
```

**Windows (CMD):**
```cmd
set SUPERBRYN_API_KEY=your_api_key_here
```

**Windows (PowerShell):**
```powershell
$env:SUPERBRYN_API_KEY="your_api_key_here"
```

**Docker:**
```bash
docker run -e SUPERBRYN_API_KEY=your_api_key_here ...
```

**.env file:**
```env
SUPERBRYN_API_KEY=your_api_key_here
LIVEKIT_PROJECT_ID=my-project-id
AGENT_ID=my-agent
VERSION_ID=v1.0.0
```

## 📊 What Gets Tracked

### Transcript Data
- **Precise timing** using VAD state change events
- Speaker turns (user/assistant)
- Start/end timestamps (ISO 8601)
- Start/end times in milliseconds (relative to call start)
- Response delays between turns
- Interruption detection
- Confidence scores (when available)
- Language detection
- Speaker IDs

### Tool Calls
- **Function name** and raw JSON arguments for every tool invoked by the agent
- **Tool result** (output string) paired with each call
- **Error flag** (`is_error`) when the tool execution raised an exception
- **Timing** — `start_ms` (call dispatched) and `end_ms` (result received), both as ms offsets from call start

### Usage Metrics
- **LLM:** Input tokens, output tokens, total tokens, model, provider
- **STT:** Audio duration, model, provider
- **TTS:** Character count, audio duration, model, provider, voice ID

### Latency Metrics
- **LLM:** Time to first token (TTFT), total duration
- **STT:** Processing duration
- **TTS:** Time to first byte (TTFB), total duration
- **Aggregated:** Average latencies per component

### Session Metadata
- Agent ID and version
- LiveKit project ID
- System prompt
- Call duration
- Phone number (if SIP call)
- SIP trunking detection
- Egress recording URLs
- LiveKit Cloud deployment status

### Extended Telemetry (on by default)

Everything below ships as **additive** sections in the same payload — all previously existing
fields are unchanged. Disable with `extended_capture=False` for the legacy shape.
Older livekit-agents versions simply produce fewer fields (each capture is
feature-detected; nothing breaks).

| Section | What's inside |
|---------|---------------|
| `call.turn_detection` | Per-turn end-of-utterance delay, transcription delay, end-of-turn model inference stats |
| `call.latency` (new keys) | `eou_ms`, `transcription_ms`, `e2e_ms` (EOU + LLM TTFT + TTS TTFB — true perceived latency) |
| `call.speech_stats` | User/agent talk seconds, turn counts, talk ratio, silence, longest gap, response delays; realtime models also `speech_created_count` and `cancelled_responses` |
| `call.transcript.turns[]` (realtime models) | Agent turns carry `provider_request_id`, and `turn_latency.llm_ms` is that reply's own time to first token (0 when not measured) |
| `call.interruptions` | Interrupted agent turns, false interruptions, detected interruptions + backchannels (SDK ≥1.6) |
| `call.network` | Connection quality timeline, reconnects, disconnect reasons, WebRTC stats polled every 10s (jitter, RTT, packet loss %, concealed samples, NACKs, bytes) |
| `call.errors` | Typed provider failures (`stt_error`/`llm_error`/`tts_error`), message, `recoverable`, model, provider |
| `call.close` | LiveKit's own close reason + terminal error (separate from your `call_end_reason`) |
| `call.sip` | Full `sip.*` attribute snapshot + DTMF keypresses with timestamps |
| `call.vad` | VAD inference count/latency and idle time |
| `call.environment` | livekit-agents / livekit / livekit-evals / Python versions, RTC stats support flag |
| `call.usage` (new keys) | LLM cached prompt tokens + tokens/sec, STT/TTS connection acquire times, realtime-model audio/text/cached token splits |

Example of the extended sections:

```json
{
  "call": {
    "latency": { "llm_ms": 450.5, "stt_ms": 120.3, "tts_ms": 180.7, "total_ms": 751.5,
                 "eou_ms": 420.0, "transcription_ms": 95.0, "e2e_ms": 1051.2 },
    "turn_detection": {
      "avg_eou_delay_ms": 420.0, "max_eou_delay_ms": 610.2, "avg_transcription_delay_ms": 95.0,
      "eot_inference_count": 14, "avg_eot_prediction_ms": 81.5,
      "eou_events": [ { "timestamp_ms": 5210, "eou_delay_ms": 402.1, "transcription_delay_ms": 88.0, "on_user_turn_completed_delay_ms": 1.2 } ]
    },
    "speech_stats": {
      "user_talk_seconds": 41.2, "agent_talk_seconds": 88.6, "user_turn_count": 12, "agent_turn_count": 13,
      "avg_user_turn_seconds": 3.4, "avg_agent_turn_seconds": 6.8, "agent_talk_ratio": 0.683,
      "silence_seconds": 20.2, "longest_silence_ms": 4100,
      "avg_response_delay_ms": 910.4, "max_response_delay_ms": 2210.0
    },
    "interruptions": { "agent_turns_interrupted": 2, "false_interruptions": 0,
                       "detected_interruptions": 2, "backchannels": 5, "detection_delay_ms": 150.0 },
    "network": {
      "worst_connection_quality": "poor",
      "connection_quality_events": [ { "timestamp_ms": 61200, "participant": "+12025551234", "quality": "poor" } ],
      "reconnect_count": 1,
      "connection_events": [ { "type": "reconnecting", "timestamp_ms": 63400 }, { "type": "reconnected", "timestamp_ms": 64100 } ],
      "track_subscription_failures": 0,
      "room_disconnect_reason": "client_initiated",
      "rtc": {
        "summary": { "samples_collected": 15, "avg_jitter_ms": 12.0, "max_jitter_ms": 41.5,
                     "avg_rtt_ms": 70.0, "max_rtt_ms": 180.2, "packets_received": 14200, "packets_lost": 36,
                     "packet_loss_pct": 0.25, "concealed_samples": 4800, "silent_concealed_samples": 1200,
                     "nack_count": 6, "bytes_received": 2400000, "bytes_sent": 2300000,
                     "avg_jitter_buffer_delay_ms": 42.1, "remote_fraction_lost_max": 0.02 },
        "samples": [ { "t_ms": 10000, "jitter_ms": 9.5, "rtt_ms": 65.0, "packets_lost": 4 } ]
      }
    },
    "errors": [ { "timestamp_ms": 84200, "error_type": "stt_error", "message": "deepgram websocket closed: 1011",
                  "recoverable": true, "label": "deepgram.STT", "source_model": "nova-3", "source_provider": "deepgram" } ],
    "close": { "reason": "participant_disconnected", "error_type": null, "error_message": null },
    "sip": { "attributes": { "sip.callID": "abc-123", "sip.callStatus": "hangup", "sip.phoneNumber": "+12025551234" },
             "dtmf": [ { "timestamp_ms": 32000, "digit": "1", "code": 1 } ] },
    "vad": { "inference_count": 4210, "avg_inference_ms": 1.9, "total_inference_seconds": 8.1, "idle_time_seconds": 3.2 },
    "environment": { "livekit_evals_version": "0.2.14", "livekit_agents_version": "1.6.5",
                     "livekit_rtc_version": "1.1.13", "python_version": "3.12.12", "rtc_stats_supported": true }
  }
}
```

## 🔍 How It Works

1. **Event Listening:** Attaches to LiveKit session events (`user_state_changed`, `agent_state_changed`, `conversation_item_added`) and to the per-plugin `metrics_collected` events on STT/LLM/TTS (the non-deprecated metrics surface), with `session_usage_updated` as a fallback for realtime models. With extended capture (default) it also listens to the session `error` / `agent_false_interruption` events, the session-level `metrics_collected` re-emit (the only surface carrying VAD/EOU/interruption metrics — a one-line deprecation warning in logs is expected), room events (`connection_quality_changed`, `reconnecting`/`reconnected`, `sip_dtmf_received`, `participant_attributes_changed`, ...), and polls `room.get_rtc_stats()` every 10s
2. **Data Aggregation:** Collects and processes events during the session
3. **Auto-Detection:** Extracts configuration from session objects
4. **Webhook Delivery:** Sends comprehensive payload to webhook endpoint when session ends

### Webhook Payload Format

```json
{
  "event": "call.ended",
  "call": {
    "id": "room-name",
    "room_name": "room-name",
    "participant_identity": "user-123",
    "started_at": "2025-10-19T12:00:00.000Z",
    "ended_at": "2025-10-19T12:05:30.000Z",
    "duration_seconds": 330,
    "transcript": {
      "turns": [
        {
          "speaker": "user",
          "text": "Hello, how are you?",
          "timestamp": "2025-10-19T12:00:05.000Z",
          "start_timestamp": "2025-10-19T12:00:05.000Z",
          "end_timestamp": "2025-10-19T12:00:07.000Z",
          "start_time_ms": 5000,
          "end_time_ms": 7000,
          "interrupted": false,
          "confidence_score": 0.98,
          "language": "en"
        },
        {
          "speaker": "assistant",
          "text": "I'm doing great, thanks for asking!",
          "timestamp": "2025-10-19T12:00:08.000Z",
          "start_timestamp": "2025-10-19T12:00:08.000Z",
          "end_timestamp": "2025-10-19T12:00:11.000Z",
          "start_time_ms": 8000,
          "end_time_ms": 11000,
          "response_delay_ms": 1000,
          "interrupted": false
        }
      ]
    },
    "tool_calls": [
      {
        "id": "call_abc123",
        "function_name": "book_appointment",
        "arguments": "{\"date\": \"2026-06-20\", \"time\": \"10:00\"}",
        "result": "{\"confirmation_id\": \"APT-9001\", \"status\": \"confirmed\"}",
        "is_error": false,
        "start_ms": 12500,
        "end_ms": 13800,
        "timestamp_ms": 12500
      }
    ],
    "recording_url": "https://...",
    "stereo_recording_url": "https://...",
    "metadata": {
      "agent_id": "my-agent",
      "livekit_project_id": "my-project",
      "llm_model": "gpt-4o-mini",
      "llm_provider": "openai",
      "stt_model": "nova-3",
      "stt_provider": "deepgram",
      "tts_model": "sonic-english",
      "tts_provider": "cartesia",
      "tts_voice_id": "...",
      "system_prompt": "You are a helpful assistant...",
      "sip_trunking_enabled": false,
      "egress_enabled": true,
      "lk_agent_enabled": true,
      "phone_number": null
    },
    "usage": {
      "llm_model": "gpt-4o-mini",
      "llm_provider": "openai",
      "llm_input_tokens": 1250,
      "llm_output_tokens": 850,
      "llm_total_tokens": 2100,
      "stt_provider": "deepgram",
      "stt_model": "nova-3",
      "stt_duration_seconds": 45.2,
      "audio_duration_seconds": 45.2,
      "tts_provider": "cartesia",
      "tts_model": "sonic-english",
      "tts_characters": 1200,
      "tts_audio_duration_seconds": 42.5
    },
    "latency": {
      "llm_ms": 450.5,
      "stt_ms": 120.3,
      "tts_ms": 180.7,
      "total_ms": 751.5
    },
    "custom_data": {
      "ticket_id": "TKT-9001",
      "customer_tier": "enterprise"
    }
  }
}
```

## 🛠️ Advanced Usage

### Prompt Sync

Keeps SuperBryn in step with the prompt your agent runs. Pass the prompt template (placeholders unfilled):

```python
PROMPT = "You are a helpful voice AI assistant for {{company}}."

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    prompt=PROMPT,            # the template your agent runs
    prompt_version="v13",     # optional: your label for this version
)
```

- Every call carries `metadata.prompt_ref = {"hash": "sha256:...", "version": "v13"}`, and the template's text is pushed to SuperBryn once per process per version (`POST /public-api/v1/prompts`).
- When you deploy a changed prompt, the agent's settings in SuperBryn show the change and offer to make it a new version. A prompt saved but never deployed never shows up, because no call names it.
- Prompt sync never holds the webhook back: if the push fails, the call is still sent and is matched once the prompt text arrives.
- The API key needs the `obs:write` scope. Keys generated in Monitor > Configure have it; with an older key a `SUPERBRYN_PROMPT_SYNC_FORBIDDEN` warning is logged once.
- Without `prompt=`, the agent's instructions are sent as `metadata.system_prompt` (as before) and read as rendered text, so a change shows up after a few calls instead of one.
- `first_message=` and `prompt_tools=[{"name", "description", "parameters"}]` are pushed with the prompt. `sync_prompt=False` turns it off.

### Disabling Extended Capture

Extended telemetry (turn detection, network stats, errors, SIP detail, ...) is
on by default and adds no meaningful overhead — network stats are polled once
every 10 seconds and everything else is passive event listening. To emit the
exact legacy payload instead:

```python
webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    extended_capture=False,  # legacy payload shape only
)
```

### Custom Data

Attach any JSON-serializable fields to the webhook payload using `custom_data`. These are forwarded verbatim in `payload["call"]["custom_data"]` and are never interpreted by the package — they're purely for your own downstream use.

**At creation time** (data known at session startup):

```python
webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    custom_data={
        "ticket_id": "TKT-9001",
        "customer_tier": "enterprise",
        "lead_source": "website",
    },
)
```

**During the session** (data discovered at runtime, e.g. after a tool call):

```python
# Replace the entire dict
webhook_handler.set_custom_data({"resolved": True, "resolution_code": "answered"})

# Or merge additional keys while keeping existing ones
webhook_handler.update_custom_data({"appointment_booked": True, "slot": "2026-06-05T10:00"})
```

Both methods can be called at any point before `send_webhook()` fires on shutdown.

### Custom API Key

Pass API key directly instead of using environment variable:

```python
webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    api_key="your_api_key_here"
)
```

### Custom LiveKit Project ID

```python
webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    livekit_project_id="my-custom-project-id"
)
```

### Self-Hosted Agents

If you're self-hosting your LiveKit agents (not using LiveKit Cloud):

```python
webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=False  # Important for cost calculation
)
```

### Custom Telephony Rates

If you're using custom telephony providers (Twilio, Vonage, etc.) with specific per-minute rates:

```python
webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    call_rate_usd=0.015  # Your custom rate per minute ($/min)
)
```

This overrides default provider costs and ensures accurate cost tracking for your telephony usage.

### Call Recording (Enabled by Default)

Call recording is **automatically enabled**. Recordings are:
- ✅ MP3 format (universal compatibility)
- ✅ Publicly accessible via direct URL
- ✅ Secured with short-lived credentials (30-minute expiry, scoped per session)
- ✅ Automatically included in webhook payload

No S3 keys, buckets, or regions need to be configured -- the package fetches
temporary upload credentials from SuperBryn's credentials service using your
`SUPERBRYN_API_KEY`.

**Recording URLs** are included in the webhook payload:

```json
{
  "call": {
    "recording_url": "https://superbryn-call-recordings.s3.ap-south-1.amazonaws.com/call_recordings/+12025551234/20250106-153045/call.mp3"
  }
}
```

**To disable recording:**

```python
webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    disable_recording=True  # Disable call recording
)
```

### Bring Your Own Egress (External Recording URL)

If you already run your **own** egress, you can disable SuperBryn's recording and
supply your own recording URL instead. Disable our egress with
`disable_recording=True`, then call `set_external_recording_url()` once your
recording is ready (any time before the webhook fires on shutdown):

```python
webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    disable_recording=True,  # don't start SuperBryn's egress
)

# ... later, once your own egress has produced a file ...
reachable = await webhook_handler.set_external_recording_url(
    "https://my-bucket.s3.amazonaws.com/calls/room-123/call.mp3?X-Amz-Signature=...",
    # probe=False,  # skip the call-time reachability check
)
```

**Only public or pre-signed URLs are supported.** When you call this method the
URL is validated at call time with a lightweight ranged GET (`Range: bytes=0-0`):

- `200`/`206` → reachable; the URL is marked usable for mirroring.
- `401`/`403` → private object or bad/expired signature — a clear error is logged
  and mirroring should be skipped.
- `404` → object not uploaded yet or wrong path (a timing/path issue).

> A ranged **GET** is used instead of `HEAD` because S3 pre-signed URLs are signed
> for a single HTTP method — a `HEAD` on a GET-signed URL returns `403` and would
> falsely look private. Note the check reflects reachability *at call time*; a
> pre-signed URL can still expire before a later (e.g. server-side) mirror runs,
> so sign for a long-enough TTL or mirror promptly.

The webhook payload gains two fields so the consumer can decide whether to mirror:

```json
{
  "call": {
    "recording_url": "https://my-bucket.s3.amazonaws.com/.../call.mp3?...",
    "recording_url_source": "external",     // "superbryn_s3" for the managed flow
    "recording_url_reachable": true          // call-time probe result (null if not probed)
  }
}
```

For the default managed flow these are `"superbryn_s3"` / `true` — the file is
already in SuperBryn's bucket and needs no mirroring.

### Stereo Recording (Dual-Channel)

Record in dual-channel stereo where the **agent is on the left channel** and **all other participants (caller/SIP) are on the right channel**. This is useful for separate-speaker transcription and analysis.

Stereo is **opt-in** — the default is mono. Enabling it is a **one-line change**: add `stereo_recording=True`. There is nothing else to configure or install — it reuses the exact same recording pipeline (egress + S3 credentials), just switching the egress to `DUAL_CHANNEL_AGENT` mixing instead of a mono mix.

```python
webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    stereo_recording=True  # L=agent, R=caller
)
```

**Prerequisites:** the only requirement is that recording already works for you (i.e. you currently get a `recording_url` in your webhook payload). That requires:
- `SUPERBRYN_API_KEY` set (used to fetch temporary S3 credentials — no S3 config needed)
- LiveKit Egress available on your deployment (enabled by default on LiveKit Cloud; the egress service must be running if self-hosted)
- Standard LiveKit env vars (`LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET`)

If recording works today, `stereo_recording=True` is the **only** change needed — no version bump, no new dependency. `stereo_recording=True` also implies recording is enabled (it overrides `disable_recording`).

When stereo is enabled, both `recording_url` and `stereo_recording_url` are populated in the webhook payload. **Note:** a single MP3 file is written — the stereo separation lives in the file's two channels, so both fields point to the *same* URL:

```json
{
  "call": {
    "recording_url": "https://...call.mp3",
    "stereo_recording_url": "https://...call.mp3"
  }
}
```

If you need to stop the recording before deleting the room (e.g. in a graceful shutdown), call `stop_egress()` to ensure the file is finalized on S3:

```python
await webhook_handler.stop_egress()  # finalize recording before room deletion
```

### Semantic Call End Reasons

If your agent knows the **business reason** for ending a call, set it explicitly on
the `WebhookHandler` before closing the room. This helps preserve reasons such as
`transfer_to_human`, `conversation_complete`, `caller_hung_up`, or
`no_answer_timeout` in the final webhook payload.

Without this, LiveKit may only emit a generic close reason like "participant left"
or "session closed".

```python
# Example: transfer to human
webhook_handler.set_call_end_reason("transfer_to_human")
await webhook_handler.stop_egress()
await ctx.api.room.delete_room(...)
```

You can use any short snake_case reason string that fits your application.

Common examples:

- `conversation_complete`
- `purpose_achieved`
- `transfer_to_human`
- `caller_hung_up`
- `main_agent_hung_up`
- `no_answer_timeout`
- `silence_timeout`
- `duration_limit`

`set_call_end_reason()` is most useful when **your application logic** decides why
the call is ending. For example:

- an `end_call` tool is invoked by the agent
- your app triggers a transfer to a human
- you enforce a silence timeout or no-answer timeout
- you intentionally delete the room during graceful shutdown

## 🧪 Testing data (eval mode)

When SuperBryn tests your agent, eval mode sends your agent's own record of each test call to SuperBryn, so you can see it next to the test: its transcript and tool calls, the latency of each reply (transcript, end of turn, first LLM token, first TTS audio, end to end), its models, turn-taking and interruption settings, tools, token and audio usage, every LLM request with what became of its answer (spoken, cut off, not spoken, stopped or only a tool call; not for models such as Nova Sonic that keep one request ID for the whole session), and every STT, LLM or TTS failure. It runs only for SuperBryn test calls, in the background: it adds no latency and records no audio. Every other call sends nothing about the call. To spot phone tests, a SIP call without a test ID checks SuperBryn's caller numbers; they are cached for an hour in a file in the temp dir that every job on the host shares (a failed fetch is retried after a minute).

It needs livekit-evals 0.3.0 or later:

```bash
pip install -U "livekit-evals>=0.3.0"
```

Add two lines to your entrypoint, once per job, before `session.start()`:

```python
from livekit_evals import attach_test_data

async def entrypoint(ctx: JobContext):
    session = AgentSession(...)
    attach_test_data(ctx, session)
    await session.start(agent=YourAgent(), room=ctx.room)
```

| Variable | Required | Description |
|----------|----------|-------------|
| `SUPERBRYN_TEST_API_KEY` | ✅ Yes | Organization API key with the `test_data:write` scope. Falls back to `SUPERBRYN_API_KEY` |
| `SUPERBRYN_AGENT_ID` | ✅ Yes | Your SuperBryn agent ID (a UUID); without it nothing is sent. Leaves `AGENT_ID`, which Observability uses, untouched |
| `SUPERBRYN_BASE_URL` | ⚪ Optional | Defaults to `https://api.superbryn.com` |

`attach_test_data` also takes `api_key=`, `agent_id=` and `base_url=`.

How a test call is recognised:
- **Phone tests** need nothing more. They are matched by the number SuperBryn calls from.
- **SIP endpoint tests** carry an `X-SuperBryn-Call-Id` header. Set your inbound trunk's `include_headers` to X headers (`SIP_X_HEADERS`), or map `X-SuperBryn-Call-Id` with `headers_to_attributes` to an attribute named `superbryn_call_id` (any name ending in `superbryn_call_id` or `x-superbryn-call-id` works; other names are not read).
- **Webhook and Custom cURL triggers**: copy `superbryn_call_id` from SuperBryn's trigger request into the job (dispatch) or room metadata as `{"superbryn_call_id": "..."}`.

Its info, warning and error log lines (logger `test_data`) start with `SUPERBRYN_TEST_DATA_`: `SUPERBRYN_TEST_DATA_SENT` when SuperBryn accepted an event, `SUPERBRYN_TEST_DATA_DISABLED` when the API key or agent ID is missing, `SUPERBRYN_TEST_DATA_RETRY`, `SUPERBRYN_TEST_DATA_REJECTED` and `SUPERBRYN_TEST_DATA_FAILED` for delivery problems, and `SUPERBRYN_TEST_DATA_ERROR` for anything else. None of them affects the call.

## 🐛 Troubleshooting

### Webhook Not Sending

**Check API Key:**
```bash
echo $SUPERBRYN_API_KEY
```

**Enable Debug Logging:**
```python
import logging
logging.basicConfig(level=logging.DEBUG)
```

**Look for these log messages:**
- `SUPERBRYN_WEBHOOK_HANDLER_CREATED` - Handler initialized
- `SUPERBRYN_WEBHOOK_SENT` - Webhook delivered successfully
- `SUPERBRYN_WEBHOOK_UNAUTHORIZED` - Invalid API key
- `SUPERBRYN_WEBHOOK_FAILED` - Delivery failed
- `SUPERBRYN_WEBHOOK_ERROR` - Exception occurred

### Common Errors

| Error | Cause | Solution |
|-------|-------|----------|
| `SUPERBRYN_API_KEY not configured` | Missing API key | Set `SUPERBRYN_API_KEY` environment variable |
| `SUPERBRYN_WEBHOOK_UNAUTHORIZED` | Invalid API key | Verify your API key is correct |
| `SUPERBRYN_WEBHOOK_FORBIDDEN` | Expired/disabled key | Generate a new API key |
| `No empty turn found to fill` | State change timing issue | Usually harmless, check logs for patterns |

### Missing Transcript Data

Ensure `webhook_handler.attach_to_session(session)` is called:
- ✅ **After** `await session.start()`
- ✅ At the **end** of your entrypoint (no early returns)

### Provider Detection Issues

The package auto-detects providers from model names. Supported providers (25+):

**LLM Providers:**
- OpenAI (gpt, whisper, tts-1, o1, o3)
- Anthropic (claude)
- Google (gemini, palm, bard, gemma)
- Meta (llama, meta-llama)
- Mistral (mistral, mixtral)
- Cohere (cohere, command)
- Perplexity (perplexity, pplx)
- Groq
- Together AI (together, togethercomputer)
- Replicate
- Hugging Face (huggingface, hf-)

**TTS Providers:**
- ElevenLabs (eleven, elevenlabs)
- Cartesia (cartesia, sonic)
- PlayHT (playht, play.ht)
- Resemble AI (resemble, resembleai)
- Murf (murf, murf.ai)
- WellSaid Labs (wellsaid, wellsaidlabs)
- Speechify
- Sarvam (saarika, sarvam, bulbul)
- Azure/Microsoft (azure, microsoft)
- AWS Polly (aws, polly, amazon)
- Google Cloud (gcloud, google-cloud)

**STT Providers:**
- Deepgram (deepgram, nova, aura)
- AssemblyAI (assemblyai, assembly)
- Rev.ai (rev.ai, revai)
- Speechmatics
- Gladia

**Realtime/Multi-modal:**
- LiveKit
- Twilio
- Vonage

If your provider isn't detected, it will show as `"unknown"` but won't affect functionality.

## 📝 Migration Guide

If you're currently using the standalone `webhook_handler.py`:

**Before:**
```python
from webhook_handler import create_webhook_handler
```

**After:**
```python
from livekit_evals import create_webhook_handler
```

Everything else stays the same! The API is identical.

## 🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🔗 Links

- [LiveKit Agents Documentation](https://docs.livekit.io/agents/)
- [GitHub Repository](https://github.com/superbryndev/livekit-evals)
- [Issue Tracker](https://github.com/superbryndev/livekit-evals/issues)
- [Get API Key](https://app.superbryn.com/api-keys)

## 💡 Support

- 📧 Email: support@superbryn.com
- 💬 GitHub Issues: [Report a bug](https://github.com/superbryndev/livekit-evals/issues)
- 📚 Documentation: [README](https://github.com/superbryndev/livekit-evals#readme)

---

Made with ❤️ by [SuperBryn](https://www.superbryn.com)
