# Behavioral Evaluations & EDK Guide

This guide introduces the **Eval Development Kit (EDK)** and details how to
write, validate, run, and report on **behavioral evaluations** in the Gemini CLI
codebase.

---

## Overview

Behavioral evaluations are automated tests designed to assert on the
**behavior** of the Gemini CLI agent (e.g., verifying which tools are called,
checking call ordering, or avoiding destructive commands) rather than checking
the final prose output.

Evaluating agent behavior is critical because:

1. Model responses are non-deterministic, making exact prose matching highly
   fragile.
2. We must ensure the model utilizes the most efficient tools (e.g., batching
   files via `read_many_files` instead of sequential `read_file` calls).
3. We must enforce safety boundaries (e.g., preventing execution of raw shell
   commands when safe alternatives exist).

All behavioral evaluations are stored under the `evals/` directory.

---

## EDK Developer Commands

The EDK provides CLI tools under `scripts/` to help contributors audit, check,
and monitor evals.

### 1. `npm run eval:inventory`

Scans all eval files under `evals/`, statically parses them, and provides a
structured overview of what exists in the repository.

- **Usage:**
  ```bash
  npm run eval:inventory
  ```
- **JSON Output:** For CI integration or inventory indexing, generate a
  machine-readable JSON report:
  ```bash
  npm run eval:inventory -- --json
  ```
- **Custom Root:** Run against another directory or repository:
  ```bash
  npm run eval:inventory -- --root /path/to/other/repo
  ```

---

### 2. `npm run eval:validate`

A lint-like checker that validates eval source files against standard structural
guidelines and best practices.

- **Usage:**
  ```bash
  npm run eval:validate
  ```
- **Custom Scopes:** Validate a specific file:
  ```bash
  npm run eval:validate -- evals/my-test.eval.ts
  ```

#### Validation Rules & Severities

| Rule ID              | Severity    | Description                                                                                                            |
| :------------------- | :---------- | :--------------------------------------------------------------------------------------------------------------------- |
| `file-naming`        | **Error**   | File must match `*.eval.ts` or `*.eval.tsx` naming conventions.                                                        |
| `valid-policy`       | **Error**   | Policy must be one of `ALWAYS_PASSES`, `USUALLY_PASSES`, or `USUALLY_FAILS`.                                           |
| `suite-metadata`     | **Error**   | Both `suiteName` and `suiteType` must be present as static string literals.                                            |
| `prompt-presence`    | **Error**   | Every eval case must have a non-empty `prompt` string.                                                                 |
| `case-name-static`   | **Error**   | The case name must be a static string literal, not computed dynamically.                                               |
| `invalid-tool-refs`  | **Error**   | All tools referenced in assertions must match known built-in or legacy tools.                                          |
| `positive-assertion` | **Error**   | Evaluation cases must assert on at least one tool call (e.g., check `waitForToolCall` has been invoked).               |
| `workspace-setup`    | **Error**   | Workspace behaviors (like file-system edits/reads) must set up a `files` object.                                       |
| `new-evals-policy`   | **Warning** | New evals must not use `ALWAYS_PASSES` policy initially (they should be promoted after nightly data proves stability). |

Warnings (`new-evals-policy`) will be logged with `⚠` and will **not** cause
the CLI process to exit with status `1`. Errors (`✗`) will block CI builds and
return exit status `1`.

---

### 3. `npm run eval:report`

Aggregates local vitest `report.json` artifacts, maps them against inventory
policies, and summarizes the pass rates per model.

- **Usage:**
  ```bash
  npm run eval:report
  ```
  By default, it scans `evals/logs/` recursively for `report.json` files.
- **Specifying Directory:**
  ```bash
  npm run eval:report -- /path/to/logs
  ```
- **JSON Output:**
  ```bash
  npm run eval:report -- --json
  ```

---

## Contributor Workflow

When writing a new behavioral evaluation, adhere to this workflow to ensure
high-quality, non-flaky test runs.

### Step-by-Step Guide

1. **Identify the Target Behavior**: Determine which tool calls need
   verification (e.g., `web_fetch` must be called).
2. **Author the Eval File**: Create your file under `evals/<name>.eval.ts`
   naming it properly.
3. **Configure Workspace Files**: If the eval reads or edits files, define them
   inside the `files` metadata field.
4. **Assert Behavior, Not Prose**: Ensure the `assert` block checks tool
   interactions using `rig.waitForToolCall` or similar. Do not check final
   prose.
5. **Run Locally**:
   ```bash
   RUN_EVALS=true npx vitest run evals/my-test.eval.ts
   ```
6. **Deflake**: Run your eval at least 3 times locally to verify it does not
   fail due to model variance.
7. **Run Validation**: Run `npm run eval:validate` to ensure no linting errors
   are present.

### Acceptance Criteria Checklist

- [ ] **Naming**: File ends with `.eval.ts` or `.eval.tsx`.
- [ ] **Policy**: New evals start as `USUALLY_PASSES`.
- [ ] **Metadata**: Static `suiteName` and `suiteType` (e.g. `'behavioral'`) are
      specified.
- [ ] **Assertions**: Uses `rig.waitForToolCall` or asserts tool arguments
      explicitly.
- [ ] **Clean workspace**: Does not write to files outside `rig.testDir`.

### Common Anti-Patterns to Avoid

- **Restricting core tools**: Never override `settings.tools.core` to limit
  tools. Evals must run against the default toolset.
- **Checking model prose**: Avoid `expect(result).toContain('something')` since
  model wording is non-deterministic.
- **Integration-only testing**: Evals that only write files without checking
  realistic model prompts are integration tests and belong under
  `integration-tests/`.

---

## CI & Dashboard Integration

You can easily automate behavioral evaluations or compile dashboard data using
EDK's JSON reporters.

### CI Validation Block

Add a step in your PR checks or GitHub workflows to automatically lint new evals
and block pull requests containing validation errors:

```yaml
- name: Run Eval Validator
  run: npm run eval:validate
```

### Publishing to a Dashboard

To record nightly performance metrics across multiple models:

1. Configure your workflow to run evaluations with the JSON reporter:
   ```bash
   cross-env GEMINI_MODEL=gemini-2.5-pro npx vitest run --config evals/vitest.config.ts --reporter=json --outputFile="evals/logs/eval-logs-gemini-2.5-pro/report.json"
   ```
2. Aggregate all test runs using the reporting tool:
   ```bash
   npm run eval:report -- evals/logs --json > aggregated_report.json
   ```
3. Upload `aggregated_report.json` to your dashboard storage backend to
   visualize pass rates over time.
