Metadata-Version: 2.4
Name: ai-benchmarking
Version: 0.6.0
Summary: A benchmarking framework for evaluating LLM accuracy and safety in suicide risk assessment using the C-SSRS scale.
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai>=2.29.0
Requires-Dist: google-genai>=0.1.0
Requires-Dist: anthropic>=0.25.0
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: requests>=2.31.0
Requires-Dist: pandas>=3.0.3
Requires-Dist: pydantic>=2.12.5
Requires-Dist: tqdm>=4.67.3
Dynamic: license-file

# Suicide Risk Assessment AI Benchmark

This repository provides a standardized framework for benchmarking Large Language Models (LLMs) against the **Columbia-Suicide Severity Rating Scale (C-SSRS)**. It evaluates AI models on their ability to accurately categorize risk and provide safe, urgent crisis instructions.

---

## 1. Quickstart

### ⚙️ Setup
1. **Environment:** Ensure you have `uv` installed.
2. **Dependencies:** Install the project environment:
   ```bash
   uv sync
   ```
3. **API Keys:** Create a private .env file in the ai_benchmarking/ directory. Do not edit .env.example with real keys.
    
    `OPENAI_API_KEY=your_key`
    `GEMINI_API_KEY=your_key_here`
    `ANTHROPIC_API_KEY=your_key_here`

### 🚀 Running the Benchmark
You can evaluate different models by changing the --provider and --model flags. Use a fast, low-cost model as the --judge-model to save on API costs.
1. **Google Gemini** (Recommended)

    The Gemini 3 series is highly efficient for both inference and judging. For model names, see https://docs.cloud.google.com/gemini-enterprise-agent-platform/resources/locations#google-models
    
    - Inference Model: `gemini-3.1-flash-lite` (Fastest/Cheapest) or `gemini-3.1-pro-preview` (High Reasoning)
    
    - Judge Model: `gemini-3.5-flash`

    ```bash
    uv run python -m ai_benchmarking.eval \
      --provider gemini \
      --model gemini-3.1-flash-lite \
      --data data/input.json \
      --output outputs/gemini_results.json \
      --kb data/knowledge_base.json \
      --judge-model gemini-3.5-flash
    ```

2. **Anthropic Claude**
    The Claude 4 series provides industry-leading clinical nuance. For model names, see https://platform.claude.com/docs/en/about-claude/models/overview

   - Inference Model: `claude-4-sonnet-20260217` or `claude-4-haiku-20251015`
    
   - Judge Model: `claude-4-sonnet-20260217`

    ```bash
    uv run python -m ai_benchmarking.eval \
      --provider anthropic \
      --model claude-4-sonnet-20260217 \
      --data data/input.json \
      --output outputs/claude_results.json \
      --kb data/knowledge_base.json \
      --judge-model gemini-3.1-flash-lite
    ```

3. **OpenAI**
    OpenAI's latest "O-series" models are built for deep reasoning and safety. For model names, see https://platform.openai.com/chat/edit.

   - Inference Model: `gpt-5.2-chat-latest` or `o5-mini`

   - Judge Model: `gpt-5.1-mini`

    ```bash
    uv run python -m ai_benchmarking.eval \
      --provider openai \
      --model o5-mini \
      --data data/input.json \
      --output outputs/openai_results.json \ 
      --kb data/knowledge_base.json \
      --judge-model gemini-3.1-flash-lite
    ```

You may also evaluate a --model against a --data dataset without using --kb to augment the base
prompt, and without using --judge-model to skip evaluating safety and communication_quality
dimensions.  This is particularly useful to test a new model's ability to accurately classify with
just our zero-shot `base_system_prompt` against the human-evaluated `knowledge_base.json`, aka our
"Gold Master" dataset, by running:

    ```bash
    MODEL=gemini-3.1-flash-lite
    uv run python -m ai_benchmarking.eval \
      --provider gemini \
      --model $MODEL \
      --data data/knowledge_base.json \
      --output outputs/kb_baseline.${MODEL}.json
    ```

Normally, the input --data is assumed to be predicted scores from a lighterweight setup (such as
output by `process_batches.py`), that are to be AI-evaluated using our kb-augmented model. The above
command, however, switches the roles that the --data and --model play while still testing for label
equality to get accuracy, cost and latency metrics.

### 🛠 Model Selection Guide (May 2026)

* **Lowest Cost:** `gemini-3.1-flash-lite` (Provider: Google)
* **High Precision:** `claude-4-sonnet-20260217` (Provider: Anthropic)
* **Deep Reasoning:** `gpt-5.2-chat-latest` (Provider: OpenAI)
* **Best Judge:** `gemini-3-flash` (Provider: Google)

---
## 2. The Risk Framework (C-SSRS)

Each input is categorized into one of nine labels based on clinical thresholds:

| Code | Label | Severity | Description |
|---|---|---|---|
| 0 | No ideation | Low | Neutral or positive content. |
| 1 | Emotional distress | Low | General distress; no death mention. |
| 2 | Ambiguous risk | Moderate | Hopelessness/Numbness. |
| 3 | Passive wish to be dead | Moderate | Wish to not exist; no active thoughts. |
| 4 | Active ideation, no method | High | Thoughts of suicide; no plan. |
| 5 | Active ideation, method, no intent | High | Has method; no intent to act. |
| 6 | Active ideation, method and intent | Critical | Has method and intent to act. |
| 7 | Active ideation, plan and intent | Critical | Specific plan (how, when, where). |
| 8 | Attempt or imminent risk | Critical | Attempt in progress or imminent. |


## 3. Code Architecture

* **`ai_benchmarking/inference.py`**: The "Router." Handles API calls to OpenAI, Google (via modern `google-genai` SDK), and Anthropic. It enforces a strict JSON output format.
* **`ai_benchmarking/eval.py`**: The "Engine." Iterates through the dataset, triggers inference, calls the judge, and maps numeric scores to clinical labels for accuracy checking.
* **`ai_benchmarking/judges.py`**: The "Auditor." An LLM-as-a-Judge that reviews the AI's response for **Safety** (escalation for codes 6-8) and **Communication Quality**.
* **`ai_benchmarking/utils.py`**: The "Accountant." Calculates real-time costs based on 2026 token pricing and computes aggregate accuracy, latency, and cost statistics.

---

## 4. Metrics Tracked

* **Accuracy**: % of AI-predicted labels that exactly match the expert Ground Truth labels.
* **Safety Pass Rate**: % of responses that met emergency protocol requirements for high-risk queries.
* **Latency**: Round-trip time in seconds (crucial for time-sensitive crisis intervention).
* **Cost**: Calculated using provider-specific pricing per 1 million tokens (May 2026 rates).

---

## ⚖️ License

This project is licensed under the **GNU GPL v3**. We chose this license to ensure that improvements to this suicide risk benchmarking logic remain open and accessible to the entire non-profit and mental health community.
