Metadata-Version: 2.4
Name: prodloop-observability-sdk
Version: 0.1.9
Summary: Python SDK for evaluating AI voice bot calls via Prodloop APIs.
Project-URL: Homepage, https://prodloop.com
Project-URL: Documentation, https://observability-sdk-docs.pages.dev/
Project-URL: Repository, https://github.com/prodloop/prodloop-observability-sdk
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: litellm
Requires-Dist: requests>=2.31.0
Provides-Extra: docs
Requires-Dist: mkdocs>=1.6.0; extra == "docs"
Requires-Dist: mkdocs-material>=9.5.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.25.0; extra == "docs"

# Prodloop Observability SDK

Python SDK to evaluate AI voice bot calls through the Prodloop evaluation service.

## Install

```bash
pip install prodloop-observability-sdk
```

## Quickstart

```python
from prodloop import CustomEvaluationParameter, EvaluationParameter, ProdloopClient

client = ProdloopClient(api_key="sk_live_...")

result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[
        EvaluationParameter.E2E_RESPONSE_TIME,
        EvaluationParameter.HALLUCINATION,
    ],
    thresholds={"e2e_response_time_max_ms": 800},
    custom_parameters=[
        CustomEvaluationParameter(
            key="resolution_quality",
            label="Resolution quality",
            description="Check whether the bot correctly understood the issue and reached a useful final outcome.",
        ),
    ],
    input_prompt="Bot instructions used during this call...",
)

print(result)
```

## Custom Parameters

Use `custom_parameters` for audit dimensions that are not part of the fixed `EvaluationParameter` enum. Each custom parameter needs a stable `key` and clear `description`; `label` is optional.

```python
result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[EvaluationParameter.HALLUCINATION],
    custom_parameters=[
        {
            "key": "driver_resolution_quality",
            "label": "Driver resolution quality",
            "description": "Evaluate whether the bot handled driver-not-found or cancellation cases correctly and empathetically.",
        }
    ],
    input_prompt="Use the Namma Yatri cancellation support policy as context.",
)
```

Custom checks are sent as `custom_parameters` metadata and evaluated from their descriptions plus optional `input_prompt` context.

## Extraction Validation

To validate extraction quality, pass both `extraction_schema` and `bot_captured_variables`:

```python
result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[EvaluationParameter.EXTRACTION_VARIABLES],
    extraction_schema={"customer_name": "string"},
    bot_captured_variables={"customer_name": "ram"},
)
```

Response includes:

- `extraction_variables`
- `extraction_validation`

## Hallucination Input Requirement

When requesting `hallucination` or any prompt-aware parameter, pass the bot's original call prompt as `input_prompt`:

```python
result = client.evaluate_call(
    audio_file_path="call.mp3",
    parameters=[
        EvaluationParameter.HALLUCINATION,
        EvaluationParameter.SECTION_SEQUENCING,
        EvaluationParameter.INTERNAL_JARGON_LEAKAGE,
    ],
    input_prompt="You are a polite admissions bot. Never invent course details.",
)
```

Prompt-aware parameter results use a compact shape:

```json
{
  "passed": "true",
  "explanation": "..."
}
```

`passed` can be `"true"`, `"false"`, or `"N/A"`. `"N/A"` means the parameter was not relevant to the supplied prompt or the call did not exercise enough behavior to judge it.

Example prompt-aware response:

```json
{
  "section_sequencing": {
    "passed": "false",
    "explanation": "The bot did not follow the section flow defined in the supplied prompt."
  },
  "mandatory_field_gating": {
    "passed": "N/A",
    "explanation": "The prompt-defined gated action was not triggered in this call."
  },
  "prompt_injection": {
    "passed": "N/A",
    "explanation": "The caller did not attempt to override instructions or inject commands."
  }
}
```

Runnable example: `examples/post_call_prompt_aware_demo.py`. The same flow was production-tested with a partial parameter set and then with all prompt-aware parameters.

## Supported Parameters

- `e2e_response_time`
- `turn_by_turn_latency`
- `pause_profile`
- `audio_artifacts`
- `hallucination`
- `extraction_variables`
- `interruption_behavior`
- `section_sequencing`
- `mandatory_field_gating`
- `interrupt_resume_precision`
- `closing_verbatim_delivery`
- `single_attempt_constraints`
- `info_dump_handling`
- `mid_flow_intent_switch`
- `side_talk_leakage`
- `ambiguous_partial_responses`
- `internal_jargon_leakage`
- `identity_extraction`
- `prompt_injection`
- `commitment_extraction`
- `scope_boundary_testing`
- `roleplay_jailbreak`
- `context_memory_across_turns`
- `hallucination_fabrication`

## Parameter Purpose

- `e2e_response_time`: average response latency in milliseconds.
- `turn_by_turn_latency`: per-turn latency series with `turn_index` and `latency_ms`.
- `pause_profile`: deterministic pause summary (`pause_count`, `total_pause_time_ms`, `longest_pause_ms`).
- `audio_artifacts`: deterministic audio-signal checks (`clipping_ratio`, `dc_offset`, clipping/DC flags).
- `hallucination`: whether the bot produced fabricated or incorrect claims.
- `extraction_variables`: structured variable extraction from call audio.
- `interruption_behavior`: whether the bot handled interruptions gracefully.
- `section_sequencing`: whether the bot followed the prompt-defined flow order.
- `mandatory_field_gating`: whether prerequisite information was collected before dependent actions.
- `interrupt_resume_precision`: whether the bot resumed the exact pending step after interruptions.
- `closing_verbatim_delivery`: whether required closings and terminal-state behavior matched the prompt.
- `single_attempt_constraints`: whether one-attempt or bounded-retry rules were respected.
- `info_dump_handling`: whether dense user-provided details were captured and reused.
- `mid_flow_intent_switch`: whether intent changes were handled without losing context.
- `side_talk_leakage`: whether background or third-party speech was ignored correctly.
- `ambiguous_partial_responses`: whether vague answers were clarified before routing or confirming.
- `internal_jargon_leakage`: whether internal prompt, system, tooling, variable, or process language leaked to the user.
- `identity_extraction`: whether identity or contact details were captured and used according to the prompt.
- `prompt_injection`: whether user instructions improperly overrode the prompt.
- `commitment_extraction`: whether unsupported guarantees, confirmations, timelines, or binding claims were avoided.
- `scope_boundary_testing`: whether the bot stayed within the prompt-defined scope.
- `roleplay_jailbreak`: whether persona/role changes that conflict with the prompt were resisted.
- `context_memory_across_turns`: whether prior context and corrections were retained.
- `hallucination_fabrication`: whether unsupported facts, claims, statuses, policies, capabilities, or operational statements were fabricated.

Deterministic parameters are computed directly from the audio signal:
`e2e_response_time`, `turn_by_turn_latency`, `pause_profile`, `audio_artifacts`.

## Authentication

Pass your Prodloop API key in the SDK constructor.  
The SDK sends it as a `Bearer` token in the `Authorization` header.

## Errors

- `ValidationError`: invalid local inputs (file path, parameters, etc.)
- `APIError`: backend/API-level failures (`status_code`, `message`)

## Documentation Site

Live documentation: https://observability-sdk-docs.pages.dev/

## Prompt Simulation

The SDK can test a bot prompt without an audio file by simulating a text conversation between a tester LLM and your bot.

There are two modes:

- `self_simulation`: Prodloop backend runs the tester and bot conversation. You select the bot model route, but Prodloop-owned backend credentials are used. No bot credentials are sent from your code.
- `user_orchestrated`: Prodloop backend runs the tester and grader. Your SDK process runs the bot locally with your own credentials and sends only bot replies/latency back to Prodloop.

The production backend also supports `audit_discovery` for deeper prompt-risk discovery. Runnable examples are available in `examples/audit_discovery_demo.py` and `simulation_demo/prod_testing/after_pypi/audit_discovery_demo.py`.

Simulation currently accepts exactly one parameter per request. To test multiple parameters, start one simulation per parameter. `max_turns` is configurable from `1` to `10`.

Discover currently enabled simulation parameters at runtime:

```python
params = client.get_simulation_parameters()
print(params["parameters"])  # [{"key": "hallucination"}, ...]
```

The response intentionally exposes only public parameter keys. Whether a parameter is LLM-judged or deterministic is backend implementation detail.

For prompt simulation, `turn_by_turn_latency` is the only timing parameter exposed. It is also emitted in every simulation result automatically at no extra LLM cost, even when you select another parameter such as `hallucination`. Older audio-evaluation parameters like `e2e_response_time` and `pause_profile` remain relevant for call-audio evaluation, but they are not separate prompt-simulation parameters.

### Self Simulation

Use the generic LiteLLM connector. The currently supported backend model routes are `vertex_ai/gemini-2.5-pro` for Gemini on Vertex AI and `azure/<deployment-name>` for Azure OpenAI deployments.

```python
import os
import time
from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins

client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])

start = client.simulate_prompt(
    simulation_mode=SimulationMode.SELF_SIMULATION,
    prompt="You are a concise support bot. Do not invent policy details.",
    parameters=[EvaluationParameter.HALLUCINATION],
    bot_llm=plugins.LiteLLM(
        model="vertex_ai/gemini-2.5-pro",
        temperature=0.2,
        max_tokens=512,
    ),
    max_turns=10,
    adaptive_max_conversations=50,
    scenario="A customer reports a delayed order and asks about compensation.",
)

chat_id = start["chat_id"]
print("chat_id:", chat_id)

while True:
    result = client.get_simulation(chat_id)
    print(result["status"])
    if result["status"] in {"completed", "failed"}:
        print(result)
        break
    time.sleep(2)
```

Backend provider configuration is required for `self_simulation`. Examples:

- Vertex AI: `VERTEX_PROJECT`, `VERTEX_LOCATION`, and the deployed service account permissions.
- Azure OpenAI: `AZURE_API_BASE`, `AZURE_API_VERSION`, and `AZURE_API_KEY` or `AZURE_AD_TOKEN`. You can also derive the base URL from a resource name in your own deployment setup.

If the backend is not configured for the selected route, the API returns a safe user-facing error asking you to configure provider credentials or use `user_orchestrated` mode.

### User Orchestrated Simulation

Use `user_orchestrated` when the bot must run in your own process with your own credentials. Prodloop never receives your bot provider credentials.

```python
import os
from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins

bot = plugins.LiteLLMBot(
    model="azure/<deployment-name>",
    system_prompt="You are a concise support bot. Do not invent policy details.",
    options={"temperature": 0.2, "max_tokens": 256},
)

client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])

result = client.simulate_prompt(
    simulation_mode=SimulationMode.USER_ORCHESTRATED,
    prompt="You are a concise support bot. Do not invent policy details.",
    parameters=[EvaluationParameter.HALLUCINATION],
    max_turns=10,
    adaptive_max_conversations=50,
    scenario="A customer pressures the bot to confirm a fake policy.",
    bot_turn_handler=bot,
)

print(result)
```

For `user_orchestrated`, configure bot credentials locally for either Vertex AI or Azure OpenAI. For Vertex AI, use ADC/service-account configuration and project/location. For Azure OpenAI, use your Azure endpoint, API version, and API key/token locally. The SDK sends only bot response text and bot LLM processing latency to Prodloop.

For adaptive simulations, `max_turns` controls turns per conversation and `adaptive_max_conversations` controls the maximum number of conversations to explore.

### Audit Discovery

Audit discovery plans targeted risk scenarios for one selected parameter, runs them against the bot, and returns passed/failed scenarios plus patch guidance for failures. A production smoke test for `section_sequencing` completed successfully with `status="completed"`, `final_result.overall_pass=true`, and `final_result.stop_reason="audit_discovery_completed"`.

### Result Shape

Simulation responses include:

- `chat_id`: stable simulation identifier.
- `status`: `running`, `completed`, or `failed`.
- `turns`: tester message, bot response, `turn_id`, grading result, and per-turn timing.
- `final_result`: overall pass/fail, per-parameter result, summary, and prompt patch suggestions when a test fails.
- `final_result.stop_reason` and `final_result.stop_message`: explain whether adaptive testing stopped due to sufficient coverage or budget constraints.

If a parameter fails, `final_result.parameter_results[*].prompt_patch_lines` and `prompt_patch_location` provide copy-paste prompt guidance.



## Credit Balance

Check the credits left on the API key without running an evaluation:

```python
balance = client.get_credit_balance()
print(balance["credits_remaining"])

# Alias with the same response
balance = client.get_credits_remaining()
```

This endpoint authenticates the API key but does not debit credits.
