Metadata-Version: 2.5
Name: python-yama
Version: 0.10.3
Summary: A reproducible evaluation runner for tool-using Agent skills
Project-URL: Repository, https://github.com/world-sim-dev/yama
Project-URL: Issues, https://github.com/world-sim-dev/yama/issues
Author-email: Farmer Sun <podpodiumapp@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agent,evaluation,llm,skill,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Requires-Dist: fastapi>=0.139.0
Requires-Dist: jinja2>=3.1
Requires-Dist: litellm<2.0,>=1.80
Requires-Dist: openai<3.0,>=2.0
Requires-Dist: orjson>=3.11.9
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich<15.0,>=14.1
Description-Content-Type: text/markdown

# yama

English | [简体中文](https://github.com/world-sim-dev/yama/blob/main/README.zh-CN.md)

`yama` is a reproducible evaluation framework for tool-using Agent Skills: a declarative YAML Case describes what the model sees — the system prompt, Skill metadata, tool schemas, and multi-turn user messages — then yama calls an OpenAI-compatible chat completions endpoint, drives the tool loop, and scores the transcript with deterministic hard checks or an LLM judge. See the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese) for the full Case schema and execution contract.

## Quick start

```bash
# Run from a plugin root; collects __evals__/cases/**/*.yaml by default
cd dingding-simple
YAMA_LLM_API_KEY=... uv run --project yama yama

# Or, from a workspace root (a directory with yama.toml), pick a plugin by name
YAMA_LLM_API_KEY=... uv run --project yama yama --plugin dingding-simple
```

Instead of prefixing every command, you can put credentials in a `.env` file: on startup the CLI loads the nearest `.env` found from the working directory upward (variables already set in the environment are not overridden). The default endpoint can be overridden with `YAMA_LLM_BASE_URL`.

### Judge routing

Evaluated Agent calls always use the configured OpenAI-compatible API. In an ordinary non-CI run, Judge routing recognizes OpenAI/GPT/o-family names as Codex, Claude/Anthropic names as Claude Code, and Grok/xAI names as Cursor. An exact per-agent map may translate an API model name to a local CLI model slug; without a mapping, a recognized family uses that Agent's default model. Other families become local-first only through an exact mapping. A local-first chain is always `primary Agent -> original Judge API -> end` and never enters the fallback Agent.

```toml
[defaults.judge.local.models.codex]
"openai/gpt-5.4" = "gpt-5.4-high"

[defaults.judge.local.models.claude]
"anthropic/claude-opus-4.1" = "claude-opus-4-1"

[defaults.judge.local.models.cursor]
"x-ai/grok-4" = "grok-4-fast"
"kimi.k3" = "kimi-k3"

[defaults.judge.local.agent]
name = "claude" # "claude", "codex", or "cursor"
model = "claude-sonnet-4-5"
```

An unmapped, unrecognized model is API-first. Only after its original API exhausts retryable failures may `[defaults.judge.local.agent]` run as the terminal fallback: configured local model, then the same Agent's default only when that model slug is explicitly rejected, then end. There is no fallback-model API call. The HTML report shows the complete Judge attempt chain, including mapped/default/fallback source, retries, failures, and the effective local model when the Agent reports it.

CI and `pr-run` are API-only. Local agents receive the mapped model slug when present and a compatible `reasoning_effort`, but never temperature, credentials, or endpoint settings. Cursor Judge passes `--trust` only for its per-attempt empty temporary workspace so non-interactive execution cannot stop at Workspace Trust; it still omits `--force`/`--yolo`, and Yama does not edit Cursor's global trust configuration. Evaluated Agent API attempts default to a 180-second transport timeout; Judge API attempts and local Judge processes default to 120 seconds. The whole-Step timeout remains independently configured by `execution.timeout_seconds_per_step` (default 300 seconds).

## Writing a Case

A Case is a YAML file organized as `context` (what the model sees) → `mocks` (how tool calls get executed) → `steps` (user turns sent in order, each with assertions) → `outcome` (the pass bar for the whole Case):

```yaml
description: The model reads the DSL before proposing creative directions

context:
  system_prompt: { default: true }     # use SYSTEM.md from the plugin root

  tools:
    - file: tools/read-dsl.yaml         # tool schema, resolved relative to __evals__

mocks:
  tools:
    read_dsl:
      respond:
        result:
          visualStyle: { name: Vintage Film }

steps:
  - id: request-directions
    user: Give me a few creative directions
    assert:
      hard:
        - tool_called: { name: read_dsl }
        - assistant_contains: creative direction

outcome:
  require: {}
```

- `description` (optional) is a one-line summary of what the Case tests; it is shown in the CLI output and the HTML report.
- `context.tools` is the tool schema sent to the model; `mocks.tools` is how the runner responds when a tool call arrives. The two sets of tool names must match exactly.
- To use a Skill, declare it by name under `context.skills` and also declare `{builtin: skill}` under `context.tools`; the model reads the Skill body via `skill(name, file?, call_reason?)`.
- Skills resolve from `<plugin-root>/skills/<name>/SKILL.md` first. If that local Skill is absent, Yama uses the sibling `.common-skills/<name>/versions/...` tree only when `<plugin-root>/config.json` explicitly declares the name in `skill_refs`; an undeclared missing Skill is a collection error.
- To simulate command-line tools, declare `{builtin: bash}` and configure per-command output under `mocks.cli`.
- `steps[].assert.hard` are deterministic checks (nine types, including `tool_called`, `tool_arguments`, and `assistant_contains`); untagged hard checks must all pass by default. Add `assert.judge` for LLM scoring.

For the full schema (every Skill/tool/mock form, the `bash` sandbox, message injection, judge configuration, and more), see the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese).

## CLI usage

```bash
uv run --project yama yama --plugin dingding-simple --report
```

| Flag | Meaning |
| --- | --- |
| `paths` (positional, repeatable) | Explicit Case YAML paths to run |
| `--plugin-root PATH` | Use the given path as the single plugin root |
| `--plugin NAME` (repeatable) | Select plugins by name from `yama.toml` |
| `--all-plugins` | Run every plugin configured in `yama.toml` |
| `-k, --filter PATTERN` (repeatable) | Only run cases whose case key (path relative to the plugin root) matches — a substring, or an fnmatch glob when the pattern contains `*` `?` `[`; with several patterns a case runs if any matches |
| `--result-dir PATH` | Override the artifact root (default `<plugin_root>/.yama/runs`) |
| `--concurrency N` | Maximum number of cases running in parallel (default 10; `1` runs cases one at a time) |
| `--run-concurrency N` | Maximum number of runs active across all cases (default 32) |
| `--report [PATH]` | Also generate a single-file HTML report. Without PATH it writes a timestamped `.yama/reports/report-<ts>.html` (history accumulates) and keeps `.yama/reports/latest.html` as a symlink to the newest one; an explicit PATH writes exactly that file |
| `--preview` | Don't run anything (no LLM calls): resolve the collected Cases and write an HTML case preview (default `.yama/reports/preview.html`; `--report PATH` overrides), or print a JSON preview to stdout when combined with `--json` |
| `--list` | Only print the collected Cases, without running them |
| `--no-artifacts` | Write no artifact files at all |
| `--json` | Emit machine-readable JSON instead of Rich tables |
| `pr-run --upload-report --commit-id SHA` | With `--report`, upload the push report and merge/upload affected Plugin latest reports; requires `ADMIN_HOST` and `YAMA_EVAL_REPORT_API_TOKEN` |

`--plugin`, `--all-plugins`, and `--plugin-root` are mutually exclusive. When none of them is given, yama walks up from the current directory to the nearest directory containing `yama.toml` and uses it as the workspace root (falling back to the cwd), running it as a single plugin root.

### Dashboard

```bash
uv run --project yama yama dashboard --plugin dingding-simple
```

`yama dashboard` starts a local HTTP server (default `http://127.0.0.1:8765/`, change with `--host`/`--port`; it also accepts the same plugin selection flags as the main command plus `-k/--filter` and `--result-dir`). The index page lists the historical HTML reports accumulated under `.yama/reports/` (newest first, with the `latest` one flagged), `/reports/<name>` serves each report, and `/preview` renders the same case preview as `--preview` — re-collecting and re-resolving the case files on every refresh (collection errors render as an error page), so you can edit a Case YAML and just reload the browser. No LLM calls are ever made by the dashboard.

### PR evals

```bash
uv run --project yama yama pr-plan --base origin/main --json
YAMA_LLM_API_KEY=... uv run --project yama yama pr-run --base origin/main --report --json-summary yama-pr-summary.json
```

`yama pr-plan` selects PR-scoped eval cases without model calls. `yama pr-run` executes the same selection and writes a JSON summary for external CI jobs to render or upload; both evaluated-Agent and Judge calls are API-only, and credentials stay in environment variables or `.env`, not CLI flags. `pr-run` always forces builtin bash execution to the Linux `linux-bubblewrap` sandbox, regardless of Case backend settings, so the runner must be Linux with `bwrap`/`bubblewrap` installed. See the [CI integration guide](https://github.com/world-sim-dev/yama/blob/main/docs/pr-evals-ci-integration.zh-CN.md) for workflow examples and the summary JSON contract.

For CI environments that publish reports to the Yama Eval Report service, add
`--upload-report --commit-id <sha>` together with `--report`. The command reads
`ADMIN_HOST` and `YAMA_EVAL_REPORT_API_TOKEN` from the environment, uploads the
current commit report, then updates each affected Plugin latest report by merging
the just-run cases into the previous remote Plugin report.

### Output

Cases run in parallel, up to 10 at a time by default (`--concurrency` changes the cap). Runs produced by one Case's `execution.repeat` also run in parallel: omitted `execution.repeat_concurrency` means all expanded runs, subject to the command-wide `--run-concurrency` cap (default 32); set it to `1` to restore serial runs for that Case. Case results keep collection order and run results keep `run_index` order regardless of completion order. Whenever Cases or runs overlap, a transient dashboard at the bottom of the terminal shows one row per Case and each completed Case's full run/step detail is printed above it as one block, so concurrent output never interleaves. Fully serial execution streams each Case, run, and assertion as it completes. Either way a per-Case summary table closes the output, and with `--json` the progress stream goes to stderr so stdout stays valid JSON.
