Metadata-Version: 2.5
Name: upshift
Version: 0.5.0
Summary: Test whether a model upgrade breaks your AI agent, then fix it or prove you should stay pinned
Project-URL: Homepage, https://github.com/Mechanism-world/upshift
Project-URL: Repository, https://github.com/Mechanism-world/upshift
Project-URL: Issues, https://github.com/Mechanism-world/upshift/issues
Project-URL: Changelog, https://github.com/Mechanism-world/upshift/blob/main/CHANGELOG.md
Author: Atila Vahedian
License-Expression: MIT
License-File: LICENSE
Keywords: agents,evaluation,llm,model-upgrade,openai,regression
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Requires-Dist: anthropic>=1.3
Requires-Dist: openai>=2.0
Requires-Dist: rich>=13
Description-Content-Type: text/markdown

<p align="center">
  <a href="https://mechanism.world"><img src="docs/assets/hero-v2.svg" alt="upshift — test whether a model upgrade breaks your agent, then fix it or prove you should stay pinned" width="100%"></a>
</p>

<p align="center">
  <a href="https://github.com/Mechanism-world/upshift/actions/workflows/ci.yml"><img src="https://github.com/Mechanism-world/upshift/actions/workflows/ci.yml/badge.svg" alt="ci"></a>
  <a href="https://github.com/Mechanism-world/upshift/releases"><img src="https://img.shields.io/github/v/release/Mechanism-world/upshift?display_name=tag&color=2563eb" alt="release"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-16a34a" alt="MIT"></a>
  <img src="https://img.shields.io/badge/python-3.12%2B-3b82f6" alt="python 3.12+">
  <img src="https://img.shields.io/badge/providers-OpenAI%20%C2%B7%20Anthropic-0ea5e9" alt="providers">
  <img src="https://img.shields.io/badge/runs-on%20your%20machine-64748b" alt="runs locally">
</p>

> ## Paused — archived 2026-09
>
> **Mechanism/Upshift was an experiment in detecting and repairing AI-agent regressions
> across model upgrades. Development is paused while I pursue a new research direction.**
>
> The repository stays up as a technical artifact: the code is MIT-licensed and complete,
> and every run record behind every number below is committed here. Nothing is being
> maintained — issues and pull requests will not be triaged, and the CLI is pinned to
> whatever the provider SDKs looked like in September 2026.
>
> **The strongest result.** On the real `gpt-5.5` → `gpt-5.6-sol` upgrade, a 38-case
> booking agent regressed on **36 of 38** cases. Three stacked, full-suite-verified
> repairs restored **32**; **4** remained broken. The verdict was `STAY PINNED` — the bar
> was every regression repaired, and 88.9% is not every.
> [Full accounting.](runs/real-56sol/REPORT.md)
>
> **What the evidence did not support**, and why the work stopped: see
> [What it has found so far](#what-it-has-found-so-far). Three controlled A/B studies
> found that competent engineers close these migrations quickly and cheaply without the
> tool. That finding is recorded here rather than buried, because it is the most useful
> thing this repository knows.

**upshift** takes a tool-calling agent, runs its eval cases on two model versions, tells you
with a p-value what regressed, tries a small set of repairs, and either hands you a patch
it has verified against your whole suite — or tells you to stay pinned, and why. It is a
local CLI: it runs on your machine, with your own API keys, and calls no service of ours.
Every claim in this README is backed by run records committed in this repository.

- **Two providers**: OpenAI (chat/completions, responses) and Anthropic (messages).
- **Statistics, not vibes**: N runs per case (default 5), pass/fail as rates, Fisher exact
  p on every label, contested results re-tested on 2N. Deterministic checks only — no LLM
  judge.
- **Repairs that earn their place**: prompt edits, model params (effort included), tool
  schemas, endpoint routing. A repair is accepted only if it restores a broken case, breaks
  nothing, and survives a full-suite re-verification.
- **Three verdicts**: `SAFE`, `SAFE WITH PATCH`, `STAY PINNED`. The patch is `git apply`-able.
- **`upshift adapt`** reads your agent's codebase (Python, notebooks) and writes the adapter
  with a `file:line` citation for everything — anything it can't trace to source is omitted
  and reported, never invented.

## Install

Python 3.12+ and an environment with [uv](https://docs.astral.sh/uv/) or pipx:

```bash
uv tool install git+https://github.com/Mechanism-world/upshift
```

```bash
pipx install git+https://github.com/Mechanism-world/upshift
```

Verify: `upshift --version`. Nothing is installed globally beyond the `upshift` command; no
daemon, no account, no telemetry.

## Quick start

Sixty seconds, no API key. The built-in deterministic simulator runs the whole pipeline:

```bash
upshift init my-agent
upshift upgrade --agent my-agent --provider sim \
  --baseline-model sim-5.5 --candidate-model sim-5.6-sol --tag demo
```

You'll watch a baseline run, a candidate run that regresses 36/38 cases, a repair loop that
restores all of them, a `SAFE WITH PATCH` verdict, and a patch — in about a second, for $0.
(Simulator results validate the machinery, never a real model: every report is tagged with
its provider, and sim evidence can never produce a real verdict.)

The Anthropic-shaped simulator (`sim-fable-5` → `sim-fable-5-1`) reproduces the documented
Claude Fable 5.1 breaks the same way, for $0.

### A real upgrade

Point upshift at your own agent — generated by `adapt` or written by hand ([ADAPTER.md](ADAPTER.md)) —
put the key in your environment or a local `.env`, and run:

```bash
# OpenAI
upshift upgrade --agent my-agent --flex \
  --baseline-model gpt-5.5 --candidate-model gpt-5.6-sol --tag my-upgrade

# Anthropic
upshift upgrade --agent my-agent --provider anthropic \
  --baseline-model claude-fable-5 --candidate-model claude-fable-5-1 --tag my-upgrade
```

`--flex` uses OpenAI's flex tier (~50% cheaper, stacks with prompt caching); the Anthropic
path caches the prompt prefix automatically. `upshift cost` prints the exact recorded spend.
Runs are resumable — Ctrl-C exits immediately and a rerun picks up where it stopped.

`--max-cost-usd N` puts a ceiling on that spend. The priced cost of everything recorded
under the run id (or, for `upgrade`, the whole `--tag` family: baseline, candidate and every
repair screen and verification) is checked before each rep and between phases; on reaching
the ceiling the command stops before the next API call, writes a `COST_STOPPED.json` marker
instead of a verdict, and exits 3. Rerun the same command with a higher ceiling to resume.

### Onboarding your agent in minutes

```bash
upshift adapt https://github.com/you/your-agent --out my-agent --flex
```

`adapt` statically ranks the repo, uses the model as an extraction engine over cited
evidence, verifies every *verbatim* claim against the cited file mechanically, and writes
the five adapter files plus an `ADAPT_REPORT.md`: what it found, what it inferred, what it
could not determine, and exactly which lines to review. Measured on real repos — from
zero-edit on a simple agent to an honest refusal on schemas buried across files:
[the adapt reports](reports/).

### Framework agents (capture mode)

If the failing request is built inside pydantic-ai, litellm, LangChain, the Vercel AI SDK,
the Claude Agent SDK or opencode, there is nothing to lift into five adapter files. So don't
read the framework — record it. `upshift capture` stands between your agent and
`api.anthropic.com` and writes down the bytes it actually sends:

```bash
upshift capture --out cap                      # terminal 1: a local recorder on 127.0.0.1:8787
ANTHROPIC_BASE_URL=http://127.0.0.1:8787 ./run-my-agent    # terminal 2: your agent, unchanged
                                               # Ctrl-C the recorder when you're done
upshift adapt --from-capture cap --out my-agent
upshift upgrade --agent my-agent --provider anthropic \
  --baseline-model claude-fable-5 --candidate-model claude-fable-5-1 --tag my-upgrade
```

`adapt --from-capture` calls no model and reads no source: the prompt, the tools, the params
and the cases all come out of requests your agent really made. The recorder is loopback-only
by default, never writes a credential or an account identifier to disk, relays a 400 verbatim,
and reassembles SSE so a streaming agent adapts like a non-streaming one.

**Where to point each framework.** Every row was read from that framework's own source; the
citations and versions are in [docs/framework-mapping.md](docs/framework-mapping.md).

| framework | set this |
|---|---|
| `anthropic` Python / TypeScript SDK | `ANTHROPIC_BASE_URL=http://127.0.0.1:8787` |
| pydantic-ai | `ANTHROPIC_BASE_URL=http://127.0.0.1:8787` |
| litellm | `ANTHROPIC_API_BASE` or `ANTHROPIC_BASE_URL` — the **origin only**; litellm appends `/v1/messages` itself |
| langchain-anthropic | `ANTHROPIC_API_URL` (else `ANTHROPIC_BASE_URL`); it sends no header of its own, so add `--framework langchain-anthropic` |
| Vercel AI SDK | `ANTHROPIC_BASE_URL=http://127.0.0.1:8787/v1` — **with the `/v1`**: it requests `baseURL + "/messages"` |
| claude-agent-sdk | `ANTHROPIC_BASE_URL` in the process env, or `ClaudeAgentOptions(env=…)` |
| opencode | `provider.anthropic.options.baseURL` in `opencode.json` — **no env var exists**; same `/v1` rule as the AI SDK |

When a repair is accepted, the report and the patch say where that repair lives in *your*
framework — `AnthropicModelSettings(anthropic_effort=…)`, `drop_params=True`,
`providerOptions.anthropic.effort` — with the file and line each mapping was verified at. A
knob a framework does not have is reported as "not mapped", never guessed.

Live smoke on a real pydantic-ai agent, `claude-fable-5` → `claude-fable-5-1`:
[reports/capture-pydantic-ai-smoke.md](reports/capture-pydantic-ai-smoke.md).

### Let the application run itself (native runner)

If your agent already has a test or eval command, you don't have to rebuild it as a Python
backend. Add a `runner` block to `agent.json` and upshift invokes your command once per
case and rep, feeding the case on stdin and reading one JSON result line (protocol v1,
[ADAPTER.md](ADAPTER.md#native-runner)); the same checks, statistics and verdicts apply.
Running repository code needs `--allow-runner`; the child gets a minimal environment, a
timeout, bounded output and a fresh copy of its working directory per rep. Reference
runners for Python and Node are in [`examples/runners/`](examples/runners/). Results from a
native run are stamped `native_application` — the only scope that means "verified in the
application". Repairs are not generated in native mode; upshift measures regressions and
verifies the patch you supply.

### Verify the patch you actually ship

```bash
upshift verify-patch --agent my-agent --patch runs/my-upgrade/upgrade.patch --run runs/my-upgrade-final
```

Applies the exact exported patch to a clean copy, rebuilds every case's first request through
the same code path — from the patched files alone, so a patch that lost the endpoint routing or
a param rebuilds what it really says — and compares it with the run that verified the patch.
Exit 0 only when every case in the patched `cases.json` was compared and matched byte for byte
(cache keys and seeds excluded); a case the run never recorded, or a patched configuration that
disagrees with the run's, is exit 2 with the reason named. The candidate model is the one field
no patch can carry (it is a command-line argument), and the block says so explicitly. This
closes the gap between "the repair idea worked" and "the file we exported is what was
verified".

Operational notes: `--max-cost-usd` bounds spend for a whole `upgrade` (the default
`--budget` of 24 counts every repair candidate *screened*, since siblings are now screened
before one is accepted); `--retry-errored` re-runs reps that ended in a transient provider
error (rate limits, capacity, 5xx) when resuming — those never count as behaviour.

## What it has found so far

Every number below is reproducible from the committed records with `upshift cost` and
`upshift diff`.

**OpenAI, gpt-5.5 → gpt-5.6-sol.** Our 38-case booking agent regressed 36/38 (the
documented function-tools 400 on chat/completions, then behavioral regressions once that
was routed around). Three stacked, full-suite-verified repairs restored 32/36 with zero
confirmed collateral — and the verdict was still `STAY PINNED`, because the bar is *every*
regression repaired. [Full accounting.](runs/real-56sol/REPORT.md) Then
[shell_gpt](https://github.com/TheR1D/shell_gpt) (12k stars): 14/14 regressed on the same
400 with no workaround in its config; one one-line endpoint repair restored 14/14 —
`SAFE WITH PATCH`, $0.56. [Report](reports/shellgpt-upgrade.md) ·
[upstream issue](https://github.com/TheR1D/shell_gpt/issues/801).

**Anthropic, Claude Fable 5 → Fable 5.1, run on release day.** The cookbook SMS bot that
forces `tool_choice: any` broke 5/5 cases on the documented 400 and was fully restored —
drop the forced choice, add the documented instruction, one rung of effort —
`SAFE WITH PATCH`, 0 broken, $0.93. A quickstarts agent built around parallel tool calls
showed no regression at N=5. A third agent turned out to be broken *before* the
migration. [Report](reports/fable-5-1-upgrade.md) · upstream issues
[FACT#5](https://github.com/ruvnet/FACT/issues/5),
[claude-cookbooks#854](https://github.com/anthropics/claude-cookbooks/issues/854).

### What the controlled studies showed

The runs above say the machinery works. They do not say it was worth running. Three
controlled A/B studies asked that second question — a competent engineer with the official
docs and the project's own tests (Arm A) against the same engineer plus upshift (Arm B),
on a public repo neither arm had touched, with the protocol frozen and the patches judged
by a blinded evaluator before any result existed.

| Study | Target | Outcome |
|---|---|---|
| 2026-09-08 | [shell_gpt](https://github.com/TheR1D/shell_gpt) → GPT-6 Astra | Break was documented in the 400 itself; upshift supplied confidence, not diagnosis |
| 2026-09-11 | [lovia](https://github.com/cymoo/lovia) → GPT-6 Astra | **Tie.** Both arms found the same four defects and both patches were accepted at 42/42 live results. Arm B cost +6 min and ≈$0.55 and produced no run, no verdict and no flag. |
| 2026-09-14 | [gptme](https://github.com/gptme/gptme) → GPT-6 Astra | **Loss.** Arm A: 83 min, $0.80. Arm B: 112 min, $1.63. upshift took ~25 min and $1.42 for one finding that changed nothing, 0 accepted repairs, 2 false positives — and a `STAY PINNED` verdict that was **wrong**. Both arms passed blinded evaluation with zero regressions. |

The conclusion the three agree on: **these breaks are loud, documented and cheap to find.**
Engineers close them in about forty minutes for under a dollar without weakening a test.
A detection-and-repair tool priced in engineer-minutes has to beat that, and this one did
not. A confidence product that emits a confident wrong verdict has the worst failure mode
available to it, and that happened once in three studies.

What remains unfalsified is the thesis this tool was never pointed at: **silent behavioural
drift** — the upgrade that throws no error, passes the type checker, and quietly answers
differently. Every study above tested a loud break, because loud breaks are what the
release notes advertise. That is the open question this repository leaves behind.

Protocols, pre-registrations, sealed blinding mappings and both arms' patches are in the
private ops repository; the frozen protocol and results are committed on
[`experiment/astra-gptme-ab`](https://github.com/Mechanism-world/upshift/commits/experiment/astra-gptme-ab).

## How it fits together

- **The adapter** (`agent.json`, system prompt, tool schemas, `backend.py`, `cases/`) — five
  files that describe your agent to upshift. The repair loop may edit only the first three;
  your backend and your cases are the yardstick and are never touched.
- **The runner** executes every case N times per model and records everything: the manifest,
  each API request and response verbatim, each tool execution, each check — under
  `runs/<run_id>/`. Runs resume from disk.
- **The differ** labels each case (stable-pass, stable-fail, regressed, improved, flaky) with a
  Fisher exact p, and classifies failures into signatures that drive repair.
- **The repair loop** screens a candidate on the broken cases, verifies it on the full suite,
  adjudicates contested statuses on 2N reps, and stacks accepted repairs. Rejected on
  confirmed evidence means never retried.
- **The verdict** and the patch. `SAFE WITH PATCH` requires every regression restored and
  nothing broken; anything less is `STAY PINNED` with the evidence attached.
- **`adapt`** and the **simulators** get you to a first run without hand-writing the adapter
  or spending money.

Design decisions, in one file: [DESIGN.md](DESIGN.md). The adapter contract: [ADAPTER.md](ADAPTER.md).

## Why single runs lie

Agents are stochastic. A borderline case passing 4/5, 3/5, 3/5, 4/5 across four runs — with
or without a patch — is what we measured on our first real run, and a single-sample rule
would have vetoed every behavioral repair we had. upshift never decides on one run: outcomes
are rates against fixed thresholds (≥ 0.8 pass, ≤ 0.4 fail, otherwise flaky), contested
statuses get 2N reps at unchanged thresholds, symmetric for restorations and vetoes, and the
playbook contains no repair that dictates exact output phrasings — restoring an eval by
overfitting to its assertions would make the number a lie.

Anthropic's official migration skill (`/claude-api migrate`) edits your code for a target
model and "produces a checklist of items to verify manually." upshift is that verification:
it runs your suite on both versions, proves each repair against it, and hands you the patch
with the evidence — or tells you to stay pinned.

## Security and privacy

### What leaves your machine

upshift runs locally and sends nothing to Mechanism: there is no backend of ours, no
telemetry, no analytics, no account, and no license check. What does leave your machine is
what you asked upshift to send, to the provider you chose:

- **Provider API calls transmit your prompts and your credentials.** Every run sends your
  system prompt, tool schemas, eval-case messages and tool results to OpenAI or Anthropic (or
  to the base URL you set), authenticated with your key. That is the measurement; there is no
  way to test a model without talking to it.
- **`upshift adapt` sends cited slices of your code to the extraction model.** It reads the
  repository you point it at and puts the ranked excerpts in the prompt. Point it at a private
  repository and that repository's source reaches the model you configured.
- **Everything else stays on disk.** Run records, diffs, verdicts and patches are written to
  your local `runs/` directory. Nothing is uploaded, committed or pushed for you.
- **Your local transcripts can contain sensitive data.** A run record holds your prompts, the
  models' outputs and your tool results verbatim — that is what makes it evidence, and it is
  also why you should read a run directory before you publish it. An `adapt` record
  additionally quotes the source it read.

`git clone --depth 1` of a URL you pass to `adapt` is the only other outbound call.
Details, and the guards each claim rests on: [SECURITY.md](SECURITY.md).

### Code that gets executed

- **Backends you run are executed.** The `backend.py` in an adapter is code, and `upshift
  upgrade` imports and calls it in your process; the shell_gpt adapter runs model-generated
  commands inside Docker with `--network none`. Read a generated `backend.py` before you
  run it.
- **`adapt` reads; it does not execute.** It never imports, builds or tests the repository
  it analyses.
- Vulnerability reports: see [SECURITY.md](SECURITY.md).

### Verification scope — what a green result actually proves

Every run, diff, verdict and report carries one of three scopes, derived from how the run
executed (never declared by you):

- **`request_contract`** — upshift built the requests itself from your three patchable files
  and sent them, against a capture replay or a generated stub, so the result proves what the
  provider accepts or rejects about the request shape.
- **`adapted_agent`** — your adapter's `backend.py` executed real tool semantics, so the
  result proves the behaviour of the adapted reconstruction of your agent.
- **`native_application`** — your application's own entry point ran, with its own
  request-building code and the original configuration, so the result proves behaviour in the
  application itself.

No wording anywhere says "verified in the application" unless the scope is
`native_application`.

## Documentation

| Goal | Start here |
|---|---|
| Describe your agent to upshift | [ADAPTER.md](ADAPTER.md) |
| Understand every design decision and the statistics | [DESIGN.md](DESIGN.md) |
| Read the migration evidence | [shell_gpt on gpt-5.6](reports/shellgpt-upgrade.md) · [four Claude agents on Fable 5.1](reports/fable-5-1-upgrade.md) |
| See what `adapt` does on real repos | [adapt reports](reports/) |
| Capture a framework agent at the wire | [docs/framework-mapping.md](docs/framework-mapping.md) |
| What is actually implemented, and how far it is tested | [docs/capabilities.md](docs/capabilities.md) |
| What's out of scope, and why | [ROADMAP.md](ROADMAP.md) · [SCOPE.md](SCOPE.md) |
| What changed | [CHANGELOG.md](CHANGELOG.md) |

## Honest limits

- Tested on our synthetic agent and a handful of open-source agents (one on OpenAI, four on
  Anthropic). That is evidence, not a benchmark suite.
- Two providers. Plain API agents on both; framework agents on Anthropic only, through
  `upshift capture` — no Google, no local models, and still no code-level framework
  integration ([ROADMAP.md](ROADMAP.md)).
- Repairs are limited to prompts, params, tool schemas, and endpoint routing. Anthropic's
  thinking-block invalidation is detected and refused with the documented pointer, not
  repaired. The tool-schema repair covers one known tool shape.
- Tools that touch the world (shell, network, clusters) need a human-written deterministic
  backend; `adapt` stubs them with a TODO. Its generated eval cases are drafts.
- `adapt` doesn't yet chase identifiers across files, and reads notebooks as rendered cell
  text (cite lines accordingly).
- Capture mode has been exercised live against one framework (pydantic-ai, 3 cases at N=3 —
  a smoke, not evidence). The other seven mapping rows are verified from source, not from a
  live capture. A capture is a record of what your agent did, so the suite it produces is
  only as broad as the session you recorded.
- N=5 with Fisher exact tests tells you a 5/5 → 0/5 collapse is real (p ≈ 0.004). It will not
  resolve subtle single-case effects, and a clean result on one agent is not proof that a
  documented shift doesn't exist. Raise N if you need more power and can pay for it.

## Development

```bash
git clone https://github.com/Mechanism-world/upshift && cd upshift
uv sync --group dev
uv run ruff check src tests agents && uv run pytest -q
```

macOS note: uv's editable-install `.pth` file sometimes gets the `UF_HIDDEN` flag and CPython
skips it; tests self-heal via `tests/conftest.py`, and for the CLI entry point run
`chflags nohidden .venv/lib/python3.12/site-packages/*.pth`. See [CONTRIBUTING.md](CONTRIBUTING.md).

## Status

This repository is **archived and unmaintained** as of September 2026. Issues,
pull requests and discussions are not being triaged, and there will be no further
releases. The final tagged version is the last state the work reached.

You are welcome to fork it. The parts most likely to be useful on their own are the
model-comparison harness, the N-rep runner and recorder, the statistical differ, the
repair loop, and the eval/verification infrastructure — see
[What to take from this repository](#what-to-take-from-this-repository) below.

upshift was built by [Mechanism.world](https://github.com/Mechanism-world) and is released
under the MIT License.

## What to take from this repository

Everything here runs locally, holds no credentials, and is MIT-licensed. If you are
building something adjacent, these are the pieces that stand alone:

| Piece | Where | What it does |
|---|---|---|
| **Runner + recorder** | `src/upshift/runner.py`, `recorder.py` | Runs every eval case N times against two model versions, resumable, with every input, output, param and version written to disk so any diff stays inspectable later |
| **Statistical differ** | `src/upshift/differ.py`, `stats.py` | Fisher exact / Wilson intervals over pass rates — turns N reps into a claim about regression with a p-value, instead of a single-run anecdote |
| **Repair loop** | `src/upshift/repair/` | Signature-driven candidate generation, then accept-only-if: restores broken cases AND breaks nothing AND survives full-suite re-verification |
| **Provider layer** | `src/upshift/providers/` | OpenAI chat/completions, OpenAI responses, Anthropic messages behind one interface, with table-driven translation between the three request shapes |
| **Capture** | `src/upshift/capture/` | A loopback forwarding recorder that reconstructs an agent from its wire traffic — credential and account-id redaction, SSE reassembly, per-turn param derivation |
| **Verification scope** | `src/upshift/verify_patch.py`, `verdict.py` | The honest-verdict machinery: what a green result actually proves, what the patch cannot carry, and why a partial verification is never a passing one |

The test suite (2,018 tests, `uv run pytest`) is the real documentation for all of it.

## Contributors

<a href="https://github.com/Mechanism-world/upshift/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=Mechanism-world/upshift" alt="contributors">
</a>

## License

[MIT](LICENSE). This repository includes material derived from third-party projects under
their own licenses and quotes provider documentation for interoperability — see
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). OpenAI, GPT, Anthropic, and Claude are
trademarks of their respective owners; upshift is not affiliated with or endorsed by either.
