TouchStone documentation

Regression tests for RL environments. You review the tests, TouchStone runs them on your machine or CI runner, and only approved tests can fail a build.

How it fits together

Suite

A folder with suite.json and your adapter.py.

Adapter

Python you own that drives your environment and returns what it observed.

Case

One test: an input, the requirement it checks, and assertions on the observations.

Approval

A reviewer's sign-off on the exact content of a case. Any edit clears it.

Run

One execution, with every observation and assertion result kept as evidence.

Quickstart

Needs Linux (or WSL) and Python 3.12+. No GPU, container engine or API key.

1. Install

   uv tool install touchstone-usCopy

2. Start the workspace

   touchstone serveCopy

   Opens on loopback only, at port 4474. Change it with --port.

3. Open the example

   Go to 127.0.0.1:4474/tests and choose Create the example suite.

Starter walkthrough

The example checkout evaluator has a planted bug: it pays reward for a success marker even when nothing was bought. You catch it, fix it and keep the test.

1. Create the suite

   Writes suite.json and adapter.py. Nothing runs yet.

2. Add starter tests

   forged-success expects reward 0. valid-checkout expects reward 1.

3. Review and approve both

   Your approval is tied to the exact content you read.

4. Run approved tests

   forged-success fails with reward 1 and no purchase. The control passes, so the failure is real.

5. Fix the evaluator

   In adapter.py, set REQUIRE_PURCHASE = True. No restart needed.

6. Run again and compare

   Both pass. Compared with the failing run, the forged case shows Fixed.

Connect your environment

Pick how your grader runs. init writes the adapter for the first three.

touchstone init ./touchstone-suite \
  --requirement "Reward requires a paid backend order" \
  --http http://grader.internal:8080/evaluate

touchstone init ./touchstone-suite \
  --requirement "Reward requires a paid backend order" \
  --command "python grade.py"

touchstone init ./touchstone-suite --from-harbor ./tasks/my-task
touchstone check ./touchstone-suite --experimental --timeout 300

def setup(case):
    return your_environment.start()

def exercise(context, inputs):
    context.perform(inputs)

def collect(context):
    # Independent evidence, not the agent's claim of success.
    return {"reward": context.grade(), "purchased": context.admin_order_exists()}

def cleanup(context):
    if context is not None:
        context.close()

Write tests

Cases are JSON assertions on what your adapter returns.

{
  "cases": [{
    "id": "forged-checkout",
    "title": "A confirmation without an order earns no reward",
    "requirement_id": "verified-purchase",
    "input": {"action": "forge-marker"},
    "assertions": [
      {"path": "purchased", "operator": "eq", "expected": false},
      {"path": "reward", "operator": "eq", "expected": 0}
    ],
    "provenance": {"kind": "imported", "source": "Owner-authored regression"}
  }]
}

Operators: eq ne lte gte contains not_contains. Paths use dots for nested fields. Missing fields and wrong types never pass, and there is no eval.

Reusing an ID edits that case and clears its approval.

Propose tests with a model

Nothing calls a hosted model by default. Use your own signed-in Codex CLI or any trusted command:

touchstone generate .local/my-suite --codex
touchstone generate .local/my-suite --command python my_author.py

Or run touchstone suggest DIR to rank twenty reward-hacking patterns against your requirements, with no model at all. Twelve come from published taxonomies and eight from game and robotics field reports.

Gate CI

Run touchstone check and branch on the exit code.

jobs:
  regression:
    runs-on: [self-hosted, linux]
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10
      - run: uv tool install touchstone-us
      - run: touchstone check touchstone-suite --evidence-dir /var/lib/touchstone/evidence

Share with a team

Optional and always explicit. The control plane is a separate app your organization runs; no hosted instance is open to customers yet.

touchstone sync .local/my-suite <run-id> \
  --url http://127.0.0.1:4480 --suite checkout

Commands

Every browser action has a CLI equivalent, and output is JSON.

All commands accept --evidence-dir PATH; use the same path for review, check and report. From Python, use SafetyStore in touchstone.safety.store and check in touchstone.safety.engine.

Other tools

Two earlier workflows ship in the same server. Both need rootless Podman.

What a pass means

Reviews are self-reported local decisions, not authenticated release approvals. Treat inconclusive runs as unfinished work.

What it does not detect

Tests check what the grader returns for a known input. These failures happen while the grader behaves exactly as approved.

Usage reports

Each finished check, grader audit or stack comparison sends one small record to the TouchStone control plane. A failed report never affects the run.

DO_NOT_TRACK=1, TOUCHSTONE_LOCAL_ONLY=1 or check --local-only also send nothing. TOUCHSTONE_TELEMETRY=log prints each record instead.

Licensing

TouchStone is proprietary. You may run it on infrastructure you control for a thirty-day evaluation; continued use needs a written agreement such as a pilot or subscription. No redistribution, modification or use in a competing product. The full text ships as LICENSE. The control plane is licensed separately under the Functional Source License.