Metadata-Version: 2.5
Name: iterate-ai
Version: 0.5.0
Summary: Autonomous ML iteration agent: a Supervisor briefs experiments, a coding agent writes and runs real notebook cells against your data, and the winner ships as a runnable notebook. Multi-LLM backend (local Ollama to cloud), persistent memory, bounded autonomy.
Project-URL: Homepage, https://github.com/anthonyrodrigues443/Autonomous-ML-And-LLM-Iteration-Agent
Project-URL: Repository, https://github.com/anthonyrodrigues443/Autonomous-ML-And-LLM-Iteration-Agent
Project-URL: Issues, https://github.com/anthonyrodrigues443/Autonomous-ML-And-LLM-Iteration-Agent/issues
Author-email: Anthony Rodrigues <anthonyrodrigues443@users.noreply.github.com>
License: MIT
License-File: LICENSE
Keywords: agent,automl,autonomous-research,llm-eval,ml-iteration,prompt-engineering
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: e2b-code-interpreter>=1.0.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: ipykernel>=6.29.0
Requires-Dist: jupyter-client>=8.0.0
Requires-Dist: lightgbm>=4.5.0
Requires-Dist: nbformat>=5.10.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: openai>=1.50.0
Requires-Dist: pandas>=2.2.0
Requires-Dist: pydantic-settings>=2.5.0
Requires-Dist: pydantic>=2.9.0
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.9.0
Requires-Dist: scikit-learn>=1.5.0
Requires-Dist: tenacity>=9.0.0
Requires-Dist: textual>=1.0.0
Requires-Dist: threadpoolctl>=3.6.0
Requires-Dist: typer>=0.12.0
Requires-Dist: xgboost>=2.1.0
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40.0; extra == 'anthropic'
Provides-Extra: dashboard
Requires-Dist: plotly>=5.24.0; extra == 'dashboard'
Requires-Dist: streamlit>=1.40.0; extra == 'dashboard'
Provides-Extra: datasets
Requires-Dist: datasets>=3.0.0; extra == 'datasets'
Requires-Dist: kaggle>=1.6.0; extra == 'datasets'
Provides-Extra: dev
Requires-Dist: ipykernel>=6.29.0; extra == 'dev'
Requires-Dist: jupyter>=1.1.0; extra == 'dev'
Requires-Dist: mypy>=1.13.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
Requires-Dist: pytest-recording>=0.13.0; extra == 'dev'
Requires-Dist: pytest>=8.3.0; extra == 'dev'
Requires-Dist: ruff>=0.7.0; extra == 'dev'
Requires-Dist: vcrpy>=6.0.0; extra == 'dev'
Provides-Extra: mcp
Requires-Dist: mcp>=0.9.0; extra == 'mcp'
Provides-Extra: notion
Requires-Dist: notion-client>=2.2.0; extra == 'notion'
Description-Content-Type: text/markdown

# iterate

**Autonomous research-aware iteration agent for ML models and LLM prompts.**

[![PyPI](https://img.shields.io/pypi/v/iterate-ai)](https://pypi.org/project/iterate-ai/)
[![CI](https://github.com/anthonyrodrigues443/Autonomous-ML-And-LLM-Iteration-Agent/actions/workflows/ci.yml/badge.svg)](https://github.com/anthonyrodrigues443/Autonomous-ML-And-LLM-Iteration-Agent/actions/workflows/ci.yml)
[![Python](https://img.shields.io/pypi/pyversions/iterate-ai)](https://pypi.org/project/iterate-ai/)
[![License](https://img.shields.io/github/license/anthonyrodrigues443/Autonomous-ML-And-LLM-Iteration-Agent)](LICENSE)

```bash
pip install iterate-ai

# your CSV, your target column, your metric. LLM runs on local Ollama ($0)
# or any OpenAI-compatible endpoint. Full setup: Quick start below.
iterate run --data examples/churn_tabular/data.clean.csv --target Churn
# (--metric is optional now: omit it and the agent picks one from your data, and says why)

# the same loop on an LLM prompt: a labelled eval set plus one line saying what the job is
# (python examples/toxicity_jigsaw/prepare.py builds that eval set first, no account needed)
iterate run --data examples/toxicity_jigsaw/data.csv --target label \
            --task "decide whether this Wikipedia comment is toxic"
```

`iterate` runs an autonomous experiment loop on your ML problem. The agent **writes and runs its own training code**, cell by cell, in a live Jupyter kernel: a Supervisor reads the run history and briefs one experiment, a coding agent executes it against real cell outputs and real tracebacks, a Summarizer distills every finished notebook so the next one inherits what worked and what failed. In v0.3 you **talk to it while it runs**: a terminal UI streams the session as a live transcript (syntax-highlighted cells, scores, briefs) over a pinned input box, and anything you type in plain English becomes a question answered from the run's notebooks, a steer for the current experiment, or a standing rule every later experiment respects. In v0.5 the same loop iterates an **LLM prompt**: give it a labelled eval set and one line saying what the job is, and the agent writes a prompt, measures it, reads what it got wrong, and rewrites it. Every submission is scored on a sealed holdout, every attempt persists in memory, and the winner ships as a runnable notebook, or as `prompts.yaml` on a prompt run. 807 unit tests across 51 files run in CI on every push.

| v0.5 today | On the roadmap |
|---|---|
| **It iterates LLM prompts, not only models.** Pass `--task` with a labelled eval set and the agent writes a prompt, measures it on training rows, reads the misses, and rewrites it. Classification and regression, scored on the same sealed holdout a model gets, delivered as `prompts.yaml` | Vision transfer learning (v0.6) |
| **You no longer pick the metric.** Omit `--metric` and the agent reads your data, searches the literature, and chooses one, then tells you why. An explicit choice always wins | Cost-to-serve recommendations (v0.7) |
| **A Critic reviews every experiment for leakage**, so a pipeline that fits on the holdout does not get to bank its score; **a Researcher grounds proposals in real papers** (OpenAlex + arXiv, no API key), citing work it actually retrieved | Inferred inputs + MCP auto-discovery (v0.9), one-sentence input (v1.0) |
| A deterministic guard stack converts weak-model waste and outranks everything else: a user steer can shape a brief but never bypass a gate, and no agent can overturn a guard; talk to the run while it runs; winner ships as a runnable notebook | |

## Why I built this

I kept seeing the same failure mode on small AI teams. A model or a prompt ships, and under delivery pressure nobody iterates on it again, so it sits in production for months while baselines move on. Experiments get re-run because nobody wrote down why they failed the first time. Teams pay frontier-model prices because nobody checked whether a cheaper model with a better prompt would do the job. `iterate` is the institutional memory, research desk, and experiment runner those teams don't have time to build.

> **How this gets built:** [WORKFLOW.md](WORKFLOW.md) (the method) · [DECISIONS.md](DECISIONS.md) (every call I made against the AI's default) · [BUILD_LOG.md](BUILD_LOG.md) (the daily trail)

---

## The full pitch

> Every YC batch ships 200+ AI startups with 2-3 engineer teams. Under shipping pressure, two things break: nobody re-iterates models against new baselines, and LLM prompts sit in production for months untouched. Engineers re-run failed experiments because nobody logged why. Teams pay GPT-5 prices because nobody tested whether Haiku + better prompting would do the job at 1/50th the cost.
>
> AutoML brute-forces. Experiment trackers only log. Prompt evals only evaluate. AIDE iterates Kaggle problems once. `iterate` is being built as the system that runs an autonomous, literature-aware, memory-persistent improvement loop on **ML models, DL/vision models, AND LLM prompts** in production, optimizing for the best model you can actually **afford to serve**. That is the v1.0 vision; the releases below get there one dial at a time.

---

## Status

**v0.5 released: the same loop now iterates LLM prompts.** v0.1 proved the autonomous loop, v0.2 made the agent write and run its own code, v0.3 put you in the loop without stopping it, v0.4 made `--metric` optional and added a Researcher and a Critic, and v0.5 adds the second problem type. Pass `--task` and a labelled eval set, and the agent writes a prompt, reads what it got wrong, and rewrites it, for classification and for regression, scored on the same sealed holdout a model is. Behind every release there is now an eval suite with a measured ceiling per dataset, so a flat result reads as a miss or as an exhausted problem instead of a guess.

**Agent-first:** the autonomous loop landed at v0.1, not as a late-stage add-on. Two dials turn release to release: the inputs you must give *shrink* (toward one-sentence input) and the problem types *grow* (tabular, then prompts, then DL/vision).

| Release | Phase | Status |
|---|---|---|
| v0.1 | **The agentic loop**: Proposer + Orchestrator + Terminator + Memory + CLI, first autonomous tabular run | shipped |
| v0.2 | **Sandboxed code-gen + the multi-agent cell-by-cell system** (Supervisor, coding agent, Summarizer) + notebook deliverable + the deterministic guard stack | shipped |
| v0.3 | **Interactive runs**: terminal UI (live transcript + input box), plain-English chat with queued delivery, pause / resume / stop, notebook Q&A, standing rules | shipped |
| v0.4 | **Researcher + Critic specialists**: literature-grounded proposals with real citations, leak review before a score banks; agent picks the metric + starting model; probability metrics | shipped |
| v0.5 | **`PromptTarget`: agentic prompt iteration** — you give a labelled eval set and a one-line task, the agent writes a prompt, reads what it got wrong, and rewrites it. Classification **and** regression, scored on a sealed holdout | shipped |
| v0.6 | `DLModelTarget`: vision transfer learning, validated on a local RTX 4050 | planned |
| v0.7 | **Cost-constrained recommendation** + serving profile + `iterate cost` | planned |
| v0.9 | Infer features/target from the data + a description; **MCP discovery** of the data/code itself (absorbs the v0.8 milestone) | planned |
| v1.0 | One-sentence input + multi-backend benchmark + read-only dashboard + docs + launch (absorbs the v0.10 milestone) | planned |

---

## What it does

You give it a prepared CSV, the target column, and a metric. The agent does the rest: builds its own baseline, then runs one briefed experiment per iteration, cell by cell, against a sealed holdout it never sees.

What a live run looks like:

```
coder[iter-04]: cell 6 ok (1.7s, 4/300s budget)
coder[iter-04]: cell 7 error: NameError: name 'Xb_cat' is not defined
coder[iter-04]: cell 8 ok (0.1s, 5/300s budget)
agent loop: iteration 4 'Model Swap - XGBoost' -> f1=0.6312

                        Run summary
 iter   model                              f1   delta vs baseline
 base   baseline                       0.5676                   -
    1   Baseline Model                 0.6251             +0.0575
    2   Class Weight Balancing         0.6251             +0.0575
    3   Hyperparameter Tuning          0.6279             +0.0603
    4   Model Swap - XGBoost  <- best  0.6312             +0.0636
    ...
best: Model Swap - XGBoost (f1=0.6312, +0.0636 vs baseline)
```

Each iteration is a real R&D session, not a script dump:

1. The **Supervisor** compresses everything tried so far into a two-line brief: the banked best (exact config, threshold, components), the known dead ends, and exactly ONE new move to try.
2. The **coding agent** rebuilds the carried best, applies the brief's one change, measures it like-for-like on a validation split, and submits only what it can defend. Errors are debugged from real tracebacks, cell by cell.
3. The **Summarizer** digests the session (what helped, what hurt, the takeaway), so run 7 knows what run 3 learned.
4. The harness scores the submission on the sealed holdout and saves the notebook immediately. Ctrl-C keeps everything already earned.

**The harness is the moat, not the model.** Every fact in a brief is machine-derived from the actual banked code, never LLM recall. A stack of deterministic guards catches the failure modes weak models actually produce: briefs that re-commission already-banked work, submissions byte-identical to earlier ones, briefed changes that never reached a line of code, sessions that die without submitting (a floor submission banks automatically). Each guard exists because a live forensic run demonstrated the failure it prevents; the stack was validated across 21 instrumented runs on a local 12B model, which ties its all-time best score inside the guarded loop.

**Deliverables.** The winner is exported as `best.ipynb`: a runnable, annotated notebook of the actual winning session (hypothesis, staged cells with their real outputs, dead ends labeled, findings). `--notebooks all` keeps one notebook per iteration: the full research journey. Every experiment also persists in `.iterate/memory.db`, so the next run builds on this one.

---

## What v0.3 adds: you, in the loop

On a terminal, `iterate run` now opens an interactive session view: the run streams as a live transcript (each executed cell as a syntax-highlighted block with its status, seconds, and budget; briefs and scores as styled rows) above an input box that is always yours. Type anything, anytime, in plain English:

- **Questions** ("did we complete an iteration?", "why did iteration 3 fail?") get answered by the Supervisor from the run's actual recorded notebooks, not from model recall.
- **Instructions** ("try a smaller learning rate") reach the RUNNING session at its next cell, and the next brief sees them too. "next run, try catboost" waits for the next experiment.
- **Standing rules** ("from now on, never use lightgbm") are carried into every later experiment's planning, capped and lean.
- **`pause` / `resume`** park and continue the run at the next cell boundary, kernel kept alive (e2b leases included) and every clock suspended. **`/stop`** (or a double Ctrl-C) quits immediately and still prints the run summary of everything finished so far; a single Ctrl-C winds down gracefully first (the in-flight attempt banks its floor). Type **`/`** for the command palette: arrow keys move, Enter completes into the input box, a second Enter sends.

Messages queue while a cell or an LLM call is in flight; you get an instant ack saying when they will land. The message routing is decided by the Supervisor but EXECUTED by the harness, and the guard stack outranks chat: a steer can shape a brief, it can never re-commission banked work, unseal the holdout, or bypass a gate. `--plain` keeps the classic scrolling output (chat still works, line by line); piped, scripted, CI, and backgrounded runs behave exactly as before, non-interactive.

---

## What v0.5 adds: prompts, same loop

A prompt eval set is a CSV like any other: input columns plus one column holding the right answer. Pass `--task` and the run switches to prompt iteration. Nothing about the loop changes. The Supervisor still briefs one change per experiment, the coding agent still measures like for like, the Critic still reviews, the Summarizer still hands on what was learned. What changes is what a cell does: one model call per record instead of one fit.

- **The harness owns the model call.** Inside a session the agent writes the prompt and calls `ask(prompt, rows)`; it cannot change the model, the temperature or the endpoint between experiments, so two experiments differ by the prompt and nothing else. The allowed answers are a tool schema built from the label set, so an answer outside it cannot happen. `evaluate(answers, truth)` scores with the run's metric, and `submit(prompt)` runs it over the holdout and writes the predictions and the prompt together.
- **The holdout is sealed the same way.** Training rows are written with answers, holdout rows without, and holdout rows never enter the session. Few-shot examples can only come from training rows.
- **Classification and regression.** A closed set of labels is scored with f1, accuracy and the rest; a numeric answer (a rating, a score on a scale) with rmse, pearson, spearman or kendall. Free text is refused rather than scored, unless you pass `--allow-free-text` and accept exact-string matching.
- **Candidates are ranked on a fixed slice, the winner is re-scored on everything.** `--loop-holdout` (default 100) keeps the search cheap and paired; `best_score_on_full_holdout` in `prompts.yaml` is the number to quote.
- **`prompts.yaml` is the deliverable.** Every version, what changed, its score, and `best: true` on the one to put in production. Written by the harness, never by the agent.

```bash
iterate run --data eval.csv --target label --task "decide whether this ticket is urgent"
iterate run --data eval.csv --target label --task "..." --prompt-file current_prompt.txt   # start from the prompt you ship today
iterate run --data pairs.csv --target score --task "rate how similar the two sentences are, 0 to 5" --metric pearson
iterate run --data eval.csv --target label --task "..." --target-model gemma4:12b --target-backend ollama --model llama-3.3-70b-versatile --backend groq
```

The last form tunes a prompt for one model while a stronger model drives the run. The cost line is honest: a pass is one model call per record, so 100 records on a local 12B is minutes, not seconds. Every answer is cached, so re-measuring a prompt the run has already tried is free.

---

## Quick start

**Local-first. $0. No API keys required.**

```bash
# 1. Install Ollama + a local model (one-time)
brew install ollama
ollama pull gemma4:12b         # the model v0.2 was validated on
ollama serve                   # background server at localhost:11434

# 2. Install iterate (pulls scikit-learn / XGBoost / LightGBM)
pip install iterate-ai         # "iterate" was taken on PyPI; the command is still `iterate`

# 3. Prepare a tabular CSV (your standard ML data cleaning) and run
iterate run --data train.clean.csv --target churn --metric f1

# 3b. Or iterate a prompt: a labelled eval set + one line saying what the job is
python examples/toxicity_jigsaw/prepare.py     # builds examples/toxicity_jigsaw/data.csv, no account needed
iterate run --data examples/toxicity_jigsaw/data.csv --target label \
            --task "decide whether this Wikipedia comment is toxic" --metric f1
```

The first run offers a one-time setup wizard (backend, model, compute, install consent); after that, flags override saved defaults per run.

```bash
# Run the generated code in an isolated cloud sandbox instead of locally:
iterate run --data train.clean.csv --target churn --metric f1 --compute e2b

# Use a cloud LLM backend (aliases: groq, together, deepseek, openai):
iterate run --data train.clean.csv --target churn --metric f1 \
            --backend groq --model llama-3.3-70b-versatile --api-key "$GROQ_API_KEY"

# Seed the baseline from an existing notebook/script (read as text, never executed):
iterate run --data train.clean.csv --target churn --metric f1 \
            --source baseline_notebook.ipynb --baseline 0.78

# Bound the whole run; keep every iteration's notebook:
iterate run --data train.clean.csv --target churn --metric f1 \
            --until 30m --notebooks all
```

Useful flags: `--max-iterations`, `--patience`, `--until` (wall-clock bound), `--notebooks best|all|none`, `--compute local|e2b`, `--install/--no-install` (package-install consent), `--think` (reasoning mode for the coder, Ollama only), `--fresh` (archive memory, start a new chapter), `--plain` (classic output instead of the interactive UI), `--spec` (the v0.1 allow-list path, kept as the fast lane). Prompt runs: `--task` (switches to prompt iteration), `--prompt-file` (your current prompt as the baseline), `--target-model` / `--target-backend` (the model whose prompt is tuned, separate from the one driving the run), `--loop-holdout` (records per candidate during the search). Full reference: `iterate run --help`

**Where things land:** `.iterate/runs/<run_id>/best.ipynb` (the runnable winner), `notebooks/` (with `--notebooks all`), `best.json` (config + score sidecar), `prompts.yaml` on a prompt run (every version with its score, the best marked). Code-path winners ship as notebooks by design; `--spec` winners also save `best_model.joblib`.

**Safety boundaries:** your `--source` file is read as text, never executed. The generated code runs locally only with your consent (the setup wizard asks), or fully isolated with `--compute e2b`. The holdout labels never enter the kernel; scoring happens host-side.

> **Note on the one-line form.** The `iterate "improve our churn baseline"` experience,
> where the agent discovers the data, baseline, and metric itself, is the **v1.0 vision**,
> not v0.5. Today you pass `--data`/`--target` explicitly (and `--task` for a prompt run);
> the inputs shrink release by release (see the roadmap). Auto-discovery, vision targets,
> and cost-constrained serving are on the roadmap, not shipped yet.

---

## Three target families (the v1.0 shape)

| Target | What it iterates on | Status |
|---|---|---|
| `ModelTarget` | Trains a tabular model, scores it on a sealed holdout | **shipped (v0.1, code-gen in v0.2)** |
| `DLModelTarget` | Transfer-learns a vision model, scores it | planned (v0.6) |
| `PromptTarget` | Runs an LLM prompt against a labelled eval set, one model call per record, scored on a sealed holdout | **shipped (v0.5)** |

All inherit from `BenchmarkTarget`. Same iteration loop, different execution path. (LLMs are **prompt-iteration only**; we don't fine-tune foundation models.)

---

## Pluggable data + tools via MCP (v0.9)

`iterate` will use **Model Context Protocol (MCP)** servers as its discovery layer: filesystem, Postgres, Notion and friends, so adding a data source is config, not code. The discovery workflow (agent introspects your tables, past experiments, and notebooks, then pauses for your gap-fill) lands at v0.9. Today the data interface is a prepared CSV, deliberately: the loop had to be proven before the input surface grows.

---

## Architecture

```
src/iterate/
├── core/                 # the reasoning engine
│   ├── agent_loop        # v0.2 loop: Supervisor briefs -> coder runs -> Summarizer digests
│   ├── supervisor        # strategist: grounded briefs + deterministic no-op guards
│   ├── coder             # cell-by-cell coding agent on a live stateful kernel
│   ├── summarizer        # per-experiment digest (cross-notebook knowledge transfer)
│   ├── codegen           # code-gen contract, session preamble, floor submission
│   ├── orchestrator      # v0.1 spec-path loop (--spec)
│   ├── proposer          # spec-path proposer + dataset profiling
│   ├── reconstructor     # rebuild a baseline from --source (text only, never executed)
│   ├── memory            # persistent experiment store (sqlite)
│   ├── scoring           # sealed-holdout scoring, shared by both paths
│   └── terminator        # deadline / patience / max-iterations gates
├── targets/              # BenchmarkTarget protocol + tabular ModelTarget
├── adapters/
│   ├── data/             # csv loading + profiling
│   ├── models/           # estimator registry (spec path)
│   └── compute/          # LocalKernel + E2BKernel (Jupyter), runners, sandbox
├── deliver/              # runnable .ipynb rendering (sessions, leaderboards)
├── llm/                  # pluggable backends: native Ollama client + one
│                         #   OpenAI-compatible client (Groq/Together/Deepseek/OpenAI/vLLM)
├── prompts/              # every prompt in one yaml, versioned with the code
└── schemas/              # Pydantic types
```

**The LLM is plug-and-play; the harness does the lifting.** The same loop runs on a local 12B or a cloud 70B. The bet (the infra-over-model A/B is logged in [DECISIONS.md](DECISIONS.md) and [BUILD_LOG.md](BUILD_LOG.md)): a good enough harness makes weak local models perform like much bigger ones, and the guard stack is what closed that gap.

---

## Comparison with existing tools (the v1.0 target)

| Capability | AutoML (DataRobot/H2O) | W&B / MLflow | Braintrust / LangSmith | AIDE | **iterate** |
|---|---|---|---|---|---|
| Iterates ML models autonomously | ✓ | ✗ | ✗ | ✓ | **✓ shipped** |
| Agent writes its own training code | ✗ | ✗ | ✗ | ✓ | **✓ shipped** |
| Persistent memory across sessions | ✗ | log only | ✗ | ✗ | **✓ shipped** |
| Bounded autonomy (deadline / patience) | ✗ | ✗ | ✗ | partial | **✓ shipped** |
| Auditable reasoning trail (runnable notebooks) | ✗ | ✗ | ✗ | basic | **✓ shipped** |
| Iterates LLM prompts | ✗ | ✗ | eval only | ✗ | **✓ shipped** |
| Iterates DL / vision models | partial | ✗ | ✗ | partial | planned v0.6 |
| Literature-aware proposals | ✗ | ✗ | ✗ | partial | ✓ |
| Cost-to-serve-aware optimization | ✗ | ✗ | ✗ | ✗ | planned v0.7 |
| Auto-discovers data + context (MCP) | ✗ | ✗ | ✗ | partial | planned v0.9 |
| Open-source | mostly ✗ | MLflow yes | ✗ | ✓ | ✓ |

Known limits are documented honestly in [LIMITATIONS.md](LIMITATIONS.md); the evaluation trail lives in [BUILD_LOG.md](BUILD_LOG.md), and version-over-version measurements in [evals/RESULTS.md](evals/RESULTS.md) — same datasets, same floor model, same budget, with a brute-force ceiling per dataset so a flat result can be read as either a miss or an exhausted problem. That harness is internal development tooling and is not part of the installed package.

---

## License

MIT. The framework is open-source. Adapters for proprietary data sources can be built on top.

---

## Author

Anthony Rodrigues: [GitHub](https://github.com/anthonyrodrigues443)
