Metadata-Version: 2.5
Name: pulse-eval
Version: 0.1.0
Summary: Open-source voice + chat agent testing framework — compliance and quality evaluation before deployment
Project-URL: Homepage, https://github.com/pulse-eval/pulse-eval
Project-URL: Bug Tracker, https://github.com/pulse-eval/pulse-eval/issues
Project-URL: Documentation, https://github.com/pulse-eval/pulse-eval#readme
Author: PULSE Contributors
License: MIT
License-File: LICENSE
Keywords: compliance,dpdp,evaluation,hipaa,llm,pytest,rbi,testing,voice-agent
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: jiwer>=3.0
Requires-Dist: litellm>=1.40
Requires-Dist: pydantic>=2.0
Requires-Dist: pydub>=0.25
Requires-Dist: pyyaml>=6
Requires-Dist: rich>=13
Requires-Dist: typer>=0.9
Requires-Dist: websockets>=12
Provides-Extra: dev
Requires-Dist: pip-audit>=2.7; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-mock>=3.12; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Provides-Extra: livekit
Requires-Dist: livekit>=0.11; extra == 'livekit'
Provides-Extra: pipecat
Requires-Dist: pipecat-ai>=0.0.30; extra == 'pipecat'
Description-Content-Type: text/markdown

# PULSE

**Open-source pytest-style compliance testing and evaluation framework for AI voice and chat agents.**

[![CI](https://github.com/pulse-eval/pulse-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/pulse-eval/pulse-eval/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/pulse-eval)](https://pypi.org/project/pulse-eval/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)

> ⚠️ **Technical signal only. Not legal certification. Consult qualified counsel before relying on PULSE results for regulatory purposes.**

PULSE is for teams building **custom-coded** AI voice and chat agents (LiveKit, Pipecat, raw Python/Node) who need to validate compliance (DPDP, RBI KYC, TRAI DLT, HIPAA) and quality (WER, TTFA, barge-in) before shipping. It is not a SaaS tool. Zero cloud, zero dashboard, zero account. Like pytest is for Python testing, PULSE is for agent compliance testing.

---

## Quickstart — pytest-native (primary path)

```python
# test_my_agent.py
from pulse_eval.testing import AgentTestCase, assert_compliant, assert_quality

def test_kyc_flow_hindi():
    """KYC verification in Hindi must pass DPDP + RBI KYC + TRAI DLT."""
    case = AgentTestCase(
        agent=my_livekit_agent,
        scenario="kyc_verification_hindi",  # loaded from tests/generated/
    )
    assert_compliant(case, packs=["dpdp", "rbi_kyc", "trai_dlt"])

def test_voice_quality_hindi():
    """Voice quality metrics must meet minimum thresholds."""
    case = AgentTestCase(agent=my_livekit_agent, scenario="kyc_verification_hindi")
    assert_quality(case, metrics=["VOICE-013", "VOICE-021"], min_score=0.7)
```

Run it exactly like any other pytest:

```bash
pytest tests/test_my_agent.py -v
```

Failures include full judge reasoning and the compliance disclaimer, right in the pytest output.

---

## CLI Batch Workflow

For full compliance reports (CI, pre-release audits):

```bash
pip install pulse-eval

# 1. Configure PULSE — enter your LiteLLM model strings and API keys
pulse init

# 2. Generate test scenarios from your agent's system prompt
pulse generate \
  --prompt interview_agent_prompt.txt \
  --domain general \
  --language en-IN \
  --count 11

# 3. Run the full evaluation
pulse run config.yaml

# 4. Open the HTML report in your browser
# Reports saved to .pulse/runs/<run_id>/report.html
```

### Example terminal output

```
PULSE Run  config.yaml
─────────────────────────────────────────────────────────
  Agent:    my_agent.handler:respond (generic)
  Judge:    claude-sonnet-4-6
  Scenarios:11

  Running 11 scenarios... ████████████████ 100%  12.3s

┌──────────────────────────────────────────────────────┐
│  Pass Rate: 81.8%  │  Scenarios: 11  │  Passed: 9   │
│  Failed: 2         │  Critical: 1    │              │
└──────────────────────────────────────────────────────┘

  Latency  p50=520ms  p75=620ms  p95=990ms  p99=1450ms

⚠️  Critical Failures:
  [CRITICAL] DPDP-001 (DPDP Act 2023 §6(1))
  Reason: Agent collected phone number before obtaining explicit consent
  Turn: 2 | Confidence: 0.91
  ⚠️  Technical signal only. Not legal certification...

Cohort Breakdown:
  hi-IN  north-indian  6 scenarios  93.3%  WER: 6.2%
  en-IN  neutral       5 scenarios  90.0%  WER: 3.1%

📄 Reports → .pulse/runs/a3f12b9c/
```

---

## Judge Calibration

PULSE is the first framework in this category to publish a judge agreement rate. Before launching your compliance program, calibrate your judge against human-labeled transcripts:

```bash
pulse calibrate labeled_cases/
```

Output:
```
  Agreement Rate: 87.3%  (42 cases) | Cohen's κ: 0.812

  Per-Rule Agreement:
  DPDP-001   91.2%
  RBI-001    85.7%
  TRAI-002   88.1%
  HIPAA-001  83.9%

  📝 README snippet: .pulse/calibration/summary.md
```

Put the agreement rate in your README. It's your credential — nobody else in this category publishes it.

> **Calibration note**: Agreement rate measures whether the judge's verdict matches human reviewer verdicts. Cohen's kappa corrects for chance agreement. This is not a measure of legal accuracy.

---

## PULSE vs. DeepEval vs. Promptfoo

| | **PULSE** | **DeepEval** | **Promptfoo** |
|---|---|---|---|
| Voice-native metrics (WER, TTFA, barge-in) | ✅ | ❌ | ❌ |
| India compliance (DPDP, RBI KYC, TRAI DLT) | ✅ | ❌ | ❌ |
| HIPAA compliance pack | ✅ | Partial | Partial |
| Fully air-gapped (local models via Ollama) | ✅ | Partial | Partial |
| pytest-native API | ✅ | ✅ | ❌ |
| Zero cloud, zero accounts | ✅ | ❌ (cloud dashboard) | ❌ |
| Judge calibration rate published | ✅ | ❌ | ❌ |
| Open-source, MIT, no paid tier | ✅ | Partial | ✅ |

**If your agent is text-only and you don't need India compliance, use DeepEval or Promptfoo — excellent tools.** PULSE exists specifically for:
1. The **voice layer** — metrics that require audio timestamps, WER, STT confidence
2. **India-specific compliance** — DPDP Act 2023, RBI KYC, TRAI DLT/DND — none of the others touch this

---

## What PULSE covers

### Compliance Assertion Packs

| Pack | Regulations | Key Assertions |
|------|-------------|----------------|
| `dpdp` | DPDP Act 2023 | Consent before PII, purpose stated, right to withdraw, no third-party without disclosure |
| `rbi_kyc` | RBI Master Circular on KYC | Identity before account info, consent to record, interest rate disclosure, no guaranteed returns |
| `trai_dlt` | TRAI DLT/DND 2018 | Registered sender ID, consent for outbound, DND registry respected, opt-out honored |
| `hipaa` | HIPAA Privacy Rule | Identity before PHI, minimum necessary, no PHI in tool calls, no unauthorized diagnosis |

### Quality Metrics

**Universal (voice + chat):** Task Completion, Context Retention, Factual Accuracy, Entity Extraction, Out-of-Scope Handling, Escalation Appropriateness, Tone, Repetition Rate, Persona Consistency, Harmful Advice Detection, PII Leakage

**Voice-only:** TTFA, Interruption Recovery, Filler Word Rate, Dead Air, Barge-in Handling, Speech Rate, WER (via jiwer), Domain Term Accuracy, Noisy Environment Robustness, Disfluency Handling, Intent Classification Accuracy & Confidence, Multi-Intent Handling, Clarifying Questions, Voice Response Length, Latency Consistency (p50/p75/p95/p99), Tail Latency, Avg Turns to Completion

**Chat-only:** Response Length, Message Coherence, Formatting Appropriateness, Chat Response Latency

---

## Configuration

### config.yaml

```yaml
agent:
  type: voice
  adapter: generic          # generic (default) | livekit | pipecat
  entrypoint: "my_agent.handler:respond"
  # For LiveKit:
  # adapter: livekit
  # livekit_url: "ws://localhost:7880"
  # room_name: "test-room"
  # api_key: "devkey"
  # api_secret: "secret"

judge:
  primary_model: "claude-sonnet-4-6"       # critical/high assertions
  fast_model: "gemini/gemini-3-flash"       # medium assertions
  # Fully local/air-gapped:
  # primary_model: "ollama/llama3.1"
  # fast_model: "ollama/llama3.1"

simulator:
  model: "gemini/gemini-3-flash"
  persona_style: realistic  # realistic | adversarial | polite

metrics:
  enabled: true
  voice:
    ttfa_warn_ms: 1500
    ttfa_fail_ms: 3000
    wer_warn: 0.15
    wer_fail: 0.30

scenarios:
  - name: "kyc_verification_hindi"
    language: hi-IN
    accent: north-indian
    domain: fsi
    max_turns: 9
    expected_intent: "home_loan_eligibility_inquiry"
    domain_terms: [EMI, CIBIL, NACH, KYC, PAN]
    persona: "first time home loan applicant, nervous, speaks Hindi, salaried employee"
    assertions:
      - rbi_kyc.identity_before_disclosure
      - rbi_kyc.interest_rate_disclosed
      - dpdp.consent_before_pii
      - dpdp.purpose_stated
      - trai_dlt.consent_for_outbound
    metrics:
      - VOICE-013    # WER
      - VOICE-021    # intent accuracy
      - VOICE-031    # latency consistency
```

### Bring Your Own Model (BYO)

PULSE uses [LiteLLM](https://github.com/BerriAI/litellm) to route ALL LLM calls. Any LiteLLM-supported model string works:

```yaml
# Anthropic
primary_model: "claude-sonnet-4-6"

# Google
primary_model: "gemini/gemini-2.5-pro"

# OpenAI
primary_model: "gpt-4o"

# Fully local — air-gapped mode (customer transcripts never leave your infra)
primary_model: "ollama/llama3.1"
fast_model: "ollama/llama3.1"
```

Air-gapped mode (Ollama/vLLM) is directly relevant to DPDP and RBI data-localization requirements — customer transcript data never leaves your infrastructure.

---

## Adapters

| Adapter | Use Case | Dependency |
|---------|----------|-----------|
| `generic` (default) | Any custom-coded agent | None (always available) |
| `livekit` | LiveKit voice agents | `pip install 'pulse-eval[livekit]'` |
| `pipecat` | Pipecat pipeline agents | `pip install 'pulse-eval[pipecat]'` |

### Generic Adapter — Mode A (function call)

```python
# my_agent/handler.py
def respond(text: str) -> str:
    return your_agent_logic(text)
```

```yaml
# config.yaml
agent:
  adapter: generic
  entrypoint: "my_agent.handler:respond"
```

### Generic Adapter — Mode B (HTTP/WebSocket)

```yaml
agent:
  adapter: generic
  entrypoint: "ws://localhost:8000/agent"  # WebSocket
  # or: entrypoint: "http://localhost:8000/agent"  # HTTP POST
```

Protocol: `{"role": "user", "content": "..."}` → `{"role": "agent", "content": "..."}`

---

## Dogfood: Mock Interview Voice Agent (LiveKit)

PULSE was built alongside a Mock Interview voice agent running on LiveKit. This is the primary internal validation target:

```python
# tests/test_interview_agent.py
from pulse_eval.testing import AgentTestCase, assert_compliant, assert_quality
from my_agents.interview_agent import InterviewLiveKitAgent

def test_interview_agent_compliance():
    agent = InterviewLiveKitAgent(room="test-room")
    case = AgentTestCase(
        agent=agent,
        scenario={
            "name": "mock_interview_en_in",
            "language": "en-IN",
            "domain": "general",
            "max_turns": 12,
            "persona": "fresh graduate applying for software engineer role",
        }
    )
    # Checks tone, context retention, no harmful advice
    assert_compliant(case, packs=["dpdp"])
    assert_quality(case, metrics=["VOICE-013", "VOICE-005", "VOICE-001"], min_score=0.7)
```

---

## Convert Production Failures to Tests

```bash
pulse convert-failure failed_call_2026_07_31.json
# ✅ Regression test created: tests/regression/failed_consent_flow_fsi.yaml
# It will run automatically on every future pulse run.
```

---

## Local Storage

```
.pulse/                    ← NEVER commit (in .gitignore)
  config.toml              ← API keys
  runs/
    <run_id>/
      report.json
      report.html
      transcripts/
        <scenario_name>.json
  calibration/
    report.json
    summary.md
```

No database. No SQLite. No cloud. Pure local files.

---

## What's NOT in PULSE

- ❌ **Production monitoring** — PULSE is pre-deployment testing only. Monitoring is a separate product with different infrastructure requirements.
- ❌ **PDF reports** — Planned for v2 once calibration data exists to back "audit-ready" claims.
- ❌ **Vapi / Retell / Bland.ai** — Low-code platforms, wrong side of the "coded, not low-code" filter.
- ❌ **Web dashboard, accounts, payments** — Zero, forever, MIT.

---

## License

MIT. No gated packs, no license key, no paid tier. PULSE is free, forever.

The business model is trust — built here, monetized elsewhere in a separate downstream product if at all.

---

> ⚠️ **Disclaimer**: Every compliance assertion result carries the label "Technical signal only. Not legal certification. Consult qualified counsel before relying on this result for regulatory purposes." This is not a footnote — it appears on every result in every output format.
