Metadata-Version: 2.4
Name: ptm-client
Version: 0.24.0
Summary: Lightweight PTM API client for integration with external Python services
Author: 15Five Engineering
License: Proprietary
Classifier: Development Status :: 4 - Beta
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Testing
Classifier: Intended Audience :: Developers
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: requests<3.0,>=2.32
Provides-Extra: dev
Requires-Dist: pytest<10.0,>=8.2; extra == "dev"
Requires-Dist: pytest-rerunfailures<17.0,>=14.0; extra == "dev"
Requires-Dist: responses<1.0,>=0.25; extra == "dev"
Requires-Dist: ruff<1.0,>=0.6; extra == "dev"

# ptm-client

Lightweight Python client for the Prompt Test Manager (PTM) API. Single runtime dependency: `requests`.

## Install

```bash
pip install ptm-client
```

## Quick start

```python
from ptm_client import PTMClient

client = PTMClient(base_url="https://ptm.example.com", token="your-api-token")

# List prompts, filtered by tag
prompts = client.list_prompts(tag="my_team")

# Fetch a prompt and its test cases
detail = client.get_prompt("my_team.summarizer")
tests = client.get_prompt_tests("my_team.summarizer")

# Run an eval against the library prompt
run = client.run_eval(
    prompt_ids=["my_team.summarizer"],
    provider_ids=["openai_gpt55"],
)

# Or run a one-off manual eval
run = client.run_manual_eval(
    {
        "prompt_text": "Summarize: {{text}}",
        "tests": [{"description": "smoke", "vars": {"text": "Hello, world."}}],
        "provider_profiles": ["openai_gpt55"],
    }
)

# Block until complete, then fetch a report
result = client.wait_for_run(run["run_key"], timeout=120)
html = client.run_report(run["run_key"])
json_report = client.run_report(run["run_key"], format="json")
```

## Constructor

```python
PTMClient(base_url, token, timeout=30, *, verify_ssl=None)
```

`token` is a PTM personal access token or service account token. `timeout` is the HTTP request timeout in seconds.

`verify_ssl` controls TLS certificate verification. When `None` (the default), the value is read from the `PTM_SSL_VERIFY` environment variable, which defaults to `true`. Set `PTM_SSL_VERIFY=false` (or pass `verify_ssl=False`) to allow self-signed or otherwise invalid certificates - intended for local dev against a homelab PTM instance, never for production. The matching `urllib3.InsecureRequestWarning` is silenced when verification is disabled.

## Public methods

### Prompts

- `list_prompts(tag=None, team=None, service=None, source=None, search=None, group=None)`
- `get_prompt(prompt_id)`
- `get_prompt_tests(prompt_id)`
- `list_prompt_versions(prompt_id)` *(v0.3.0)*
- `get_prompt_version(prompt_id, version_number)` *(v0.3.0)*

### Providers

- `list_providers()`

### Evaluations

- `run_eval(prompt_ids, provider_ids, **kwargs)`
- `run_manual_eval(payload)`
- `run_prompt_eval(prompt_id, provider_ids, *, inject_vars=None, extra_tests=None, visibility_scope="org_visible", label=None)`

### Runs

- `list_runs(limit=50, terminal_only=False, mine_only=False)` *(v0.3.0)*
- `get_run(run_key)`
- `wait_for_run(run_key, timeout=300, poll_interval=5)`
- `wait_for_security_scan(scan_id, *, timeout=300, poll_interval=3)`
- `run_report(run_key, format="html")` - `html` / `json` / `markdown` / `csv`

### Optimization *(v0.3.0)*

- `submit_optimization(prompt_id, provider_profiles=None, judge_profile=None, max_cycles=10, target_score=90.0, min_improvement=2.0, max_cost_usd=20.0, comparison_strategy=None, visibility_scope=None, *, stability_samples=None, validation_samples=None, flakiness_threshold=None, min_consistent_improvement=None, variance_aware_mutator=None, variance_signal=None, enforce_target_score=None, chained_baseline_mode=None)` *(variance-aware fields added in v0.6.0; `chained_baseline_mode` in v0.13.0; omit any to inherit server admin defaults)*
- `optimize_prompt(...)` - deprecated alias for `submit_optimization`; emits `DeprecationWarning`; removed in v1.0.0
- `get_optimization_status(prompt_id)`
- `get_optimization_history(prompt_id)`
- `get_optimization_detail(optimization_id)` *(v0.3.0)*
- `cancel_optimization(optimization_id)`
- `wait_for_optimization(prompt_id, *, timeout=600, poll_interval=10)`

### Skills *(v0.10.0)*

- `list_skills(namespace=None, search=None, tags=None, include_deprecated=False, page=1, page_size=50, held=None, source=None, kind=None)` - `held=True` lists only skills on a security hold (with `disabled_reason`, `disabled_at`, `disabled_by`); `source` (a skill source slug) and `kind` (`owned` / `linked`) *(v0.24.0)* narrow to linked skills
- `get_skill(skill_id_or_qname)`
- `get_skill_by_name(qualified_name)`
- `get_skills_by_prompt(prompt_id)`
- `publish_skill(prompt_id, payload, *, skill_name=None)`
- `update_skill(skill_id, payload)`
- `delete_skill(skill_id)`
- `install_skill_for_claude_code(skill_id, *, install_name=None)` - writes bundle to `~/.claude/skills/{slug}/`
- `invoke_skill(skill_id, inputs, *, provider_profile=None, amend=False)` *(v0.10.0; `amend` added v0.14.0; **DEPRECATED in v0.15.0** - emits `DeprecationWarning` and routes to `invoke_skill_eval_ptm`)*
- `invoke_skill_by_name(qualified_name, inputs, *, provider_profile=None, amend=False)` *(v0.14.0; in v0.15.0 routes to `invoke_skill_eval_ptm`)*
- `invoke_prompt(prompt_id, inputs, *, provider_profile=None, amend=False)` *(v0.10.0; `amend` added v0.14.0; **DEPRECATED in v0.15.0** - routes to `invoke_prompt_eval_ptm`)*

### Skill sources *(v0.24.0, server 2.7.0)*

Third-party Agent Skills repositories and Claude plugin marketplaces an admin or IT admin approved. Their skills are **linked** skills, named `<source>:<name>`, read-only, and installed into a folder named after their `install_name` (the upstream `SKILL.md` name), so an install is a spec-conformant Agent Skills folder for Claude Code, Codex or Hermes.

- `list_skill_sources(trust_state=None, kind=None)` - `{"items": [...]}`; policy and sync health only for `manage_skill_sources` holders.
- `request_skill_source(url, *, reason, provider=None, audience="org", audience_group_id=None)` - request an https repository (`provider` is required for a self-hosted host); nothing syncs until it is approved.
- `list_skill_source_entries(source_id, status=None, page=1, page_size=50)` - every skill folder found upstream with its status (`quarantined`, `available`, `blocked`, `unsupported`, ...) and reason.
- With skill sources disabled on the deployment these raise `PTMNotFoundError`.

### Security scanning *(v0.22.0)*

- `run_security_scan(target_type, target_id, *, deep=False, version_number=None, judge_profile=None, wait=False, timeout=300)` - `target_type` is `"skill"` (UUID) or `"prompt"` (prompt_id). Every scan is queued on the worker and returns the `pending` row; `wait=True` blocks for the final row (`completed`, or `status: "error"`, never a fabricated safe verdict) and raises `PTMTimeoutError` after `timeout` seconds. `deep=True` runs an LLM-assisted scan. `judge_profile` (deep only) picks a judge-capable provider profile instead of the org default; the CLI flag is `--judge`. Each finding carries `evidence`, `explanation` and `remediation` when the scanner provides them.
- `get_security_scan(scan_id)` *(v0.23.0)* - one scan row by id, for polling a queued scan; 404 when the caller cannot see its target.
- `list_security_scans(target_type, target_id, *, limit=10, offset=0, version_number=None)` - history, newest first; `limit` is 1-100.
- The skill `target_id` is the UUID; the CLI also accepts a qualified name or bare public slug and looks it up first. With scanning disabled on the deployment both methods raise `PTMNotFoundError` ("Security scanning is not enabled"). Findings are attacker-authored text: display them, never execute them.
- A skill on security hold raises `PTMLockedError` (HTTP 423) from install, download, contents and invoke, and from the backing prompt's `get_prompt_contents`, `get_prompt_version` and invoke methods for callers who cannot edit the prompt and lack `manage_skill_catalog`. `exc.reason` is the manager's full recorded reason (the message only embeds a truncated preview of the body).
- A block-mode security gate raises `PTMValidationError` whose `security_block` holds the structured detail (`verdict`, `risk_score`, `findings_total`, `findings`); publishing the prompt of a held skill raises `PTMConflictError` whose `security_hold` holds the detail. Both messages name the security cause instead of the generic shape / retry hint.

### Inline-vs-eval execution surfaces *(v0.15.0)*

PTM 1.16.x exposes three execution modes per kind (skill + prompt). Use the helper that matches the user's intent.

- `invoke_skill_eval_ptm(skill_id, inputs, provider_profile, *, judge_profile=None, timeout_seconds=60, amend=False)` - blocking server-side eval; returns scored output and retains a `Eval: <qualified_name>` row in run history.
- `invoke_prompt_eval_ptm(prompt_id, inputs, provider_profile, ...)` - prompt twin.
- `invoke_skill_with_bundle(skill_id, inputs, provider_profile, ...)` - returns the skill bundle for local execution AND enqueues a parallel server-side scored eval. Does not block on the eval. Response carries the bundle plus `run_key`.
- `invoke_skill_with_bundle_by_name(qualified_name, ...)` - resolves the qualified name and forwards.
- `invoke_prompt_with_bundle(prompt_id, inputs, provider_profile, ...)` - prompt twin.
- `record_skill_client_inline_run(skill_id, *, provider_label, duration_ms, input_keys, success)` - 204 telemetry for client-side inline executions. All four kwargs are REQUIRED server-side; passing any one omitted raises HTTP 422. The endpoint sees only the names of input keys, never their values.
- `record_prompt_client_inline_run(prompt_id, *, provider_label, duration_ms, input_keys, success)` - prompt twin.

### Contents + usage metrics *(v0.14.0)*

- `get_skill_contents(skill_id_or_qname)` - fetch bundle in memory as `{prompt_md, manifest, files}`; no disk write, no LLM call
- `get_prompt_contents(prompt_id, *, version=None)` - fetch prompt contract as `{prompt_text, version, tests, deepeval_metrics, tool_definitions}`
- `get_skill_usage_metrics(skill_id)` - `{lifetime, last_30d, last_7d, last_request_at}` aggregated from fetch-event audit rows; returns null shape when org setting `inline_usage_metrics_enabled=false`
- `get_prompt_usage_metrics(prompt_id)` - same shape for prompt fetch events

### Groups *(v0.13.0)*

- `list_groups()`
- `create_prompt(payload)`

## Test-case shapes

PTM evaluates with three optional scoring layers. Use any combination.

### Promptfoo assertions (deterministic)

Go in the `assert` array inside each test case:

```python
{
    "description": "mention the topic with enough length",
    "vars": {"transcript": "..."},
    "assert": [
        {"type": "icontains", "value": "API migration"},
        {"type": "javascript", "value": "output.length >= 100"},
    ],
}
```

### DeepEval metrics (semantic, judge-LLM)

Go in `additional_metrics` at the payload root:

```python
{
    "additional_metrics": [
        {"name": "relevance", "criteria": "Output addresses the input topic.", "threshold": 0.7},
    ],
    "judge_profile": "openai_gpt56_terra",
}
```

### KPI configs (custom weighted expressions)

Go in `additional_kpis` at the payload root:

```python
{
    "additional_kpis": [
        {
            "name": "cost_ok",
            "description": "Under $0.05",
            "expression": "1 if cost < 0.05 else 0",
            "weight": 1.0,
        },
    ],
}
```

## Inline examples

### `run_manual_eval` - full control

```python
run = client.run_manual_eval(
    {
        "label": "my_custom_eval",
        "prompt_text": "Summarize: {{text}}",
        "tests": [{"description": "short text", "vars": {"text": "The quick brown fox."}}],
        "provider_profiles": ["openai_gpt55"],
        "cost_threshold": 1.0,
        "latency_threshold_ms": 30000,
    }
)
```

### `run_prompt_eval` - fetch from PTM + inject live data

```python
run = client.run_prompt_eval(
    prompt_id="my_team.summarizer",
    provider_ids=["openai_gpt55"],
    inject_vars={"transcript": real_transcript, "meeting_title": "Weekly 1:1"},
)
result = client.wait_for_run(run["run_key"], timeout=120)
```

## Error handling

```python
from ptm_client import PTMClient, PTMError, PTMTimeoutError

try:
    result = client.wait_for_run(run_key, timeout=60)
except PTMTimeoutError:
    print("Run did not complete in time")
except PTMError as e:
    print(f"PTM API error ({e.status_code}): {e}")
```

`PTMError` wraps all HTTP errors, `ConnectionError`, and `requests.Timeout`. Check `e.status_code` (0 for connection/timeout failures). Typed subclasses: `PTMPermissionError` (403), `PTMNotFoundError` (404), `PTMConflictError` (409; `.security_hold` on a held-skill republish), `PTMValidationError` (422; `.security_block` on a block-mode security gate), `PTMLockedError` (423, security hold; `.reason`).

## CLI

`ptm` (or `python -m ptm_client`) reads `PTM_API_BASE_URL` and `PTM_API_TOKEN` from the environment.

| Command | What it does |
|---|---|
| `ptm install <skill> [--target-dir DIR] [--target-agent claude\|codex\|hermes] [--version N] [--project-dir DIR]` | Install a skill bundle by UUID or qualified name (a linked skill as `<source>:<name>`, into a folder named after its `install_name`). A security hold prints the reason and exits 1. |
| `ptm invoke <ref> --provider PROFILE [--type prompt\|skill] [--input KEY=VALUE ...] [--timeout S]` | Run a prompt or skill through the eval pipeline. |
| `ptm security-scan <skill\|prompt> <id> [--deep] [--version N] [--judge PROFILE] [--wait] [--timeout S]` | Queue a scan and print the row; `--wait` prints the final row. Writes a scan row and an audit event; `--deep` spends judge tokens. |
| `ptm security-scans <skill\|prompt> <id> [--limit N] [--version N]` | Print scan history, newest first (read only). |
| `ptm sources list [--trust-state S] [--kind git_repo\|plugin_marketplace]` | Print the skill sources you can see as JSON (read only). |
| `ptm sources request <url> --reason TEXT [--provider github\|gitea\|gitlab]` | Request a skill source and print it with its preview (records an audit event; an admin or IT admin approves). |

For both scan commands a skill `<id>` may be a UUID, a qualified name or a bare public slug. An invalid combination (`--judge` without `--deep`, `--version` on a skill) prints `Invalid arguments: ...` and exits 2 without sending the scan request.

## Compatibility

- Python 3.12+
- PTM backend >= 1.9.0 required. `get_skill_contents`, `get_prompt_contents`, and usage-metrics methods require PTM >= 1.16.0. Skill invoke methods require PTM >= 1.13.1. Older backends work for all other methods.
