Metadata-Version: 2.5
Name: gxy-tool-bot
Version: 0.2.0
Summary: Agentic bot that generates Galaxy tool wrappers from user requests
Author: gxy-tool-bot contributors
License: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: click>=8.1
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest-mock>=3.12; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# gxy-tool-bot

Because once upon a time Björn said to Danielle: Wouldn't an Agentic Galaxy Tool Bot be cool? :smile:

So this is an agentic bot that generates [Galaxy](https://galaxyproject.org/) tool wrappers from user requests, powered by LLM APIs and orchestrated through GitHub Actions.

## Overview

`gxy-tool-bot` is a Python library + companion GitHub Actions workflows that automate the creation of Galaxy tool wrappers. It is **not a tool repo itself** — it's a library consumed by repos that house Galaxy tools.

## Current Status

This repository is under active development. The bot can plan tool wrappers, generate XML/macros/test data, and open PRs automatically. It has been tested end-to-end on real tool requests. Expect rough edges and iterate on configuration as needed.

## How it works

1. **Request:** Users file a GitHub issue using the "Tool Request" issue template (structured fields for tool name, description, links, contact).
2. **Plan:** The `tool-bot:tool-request` label triggers a workflow. An agent researches the tool (bioconda, GitHub, publications, web) and posts a plan as an issue comment with a `tool-bot:plan-ready` label.
3. **Review:** A maintainer reviews the plan and adds the `tool-bot:ready-to-implement` label.
4. **Generate:** An agent generates the tool XML, macros, and test data, then opens a PR with a `tool-bot:pr-opened` label on the issue. If the agent gives up or crashes, a `tool-bot:generation-failed` label is applied instead. If generation completes but validation finds issues, the PR is still created with the validation errors noted in the description — apply the `tool-bot:address-feedback` label to have the bot attempt fixes.

**Updating existing tools** follows the same plan → `tool-bot:ready-to-implement` → PR flow: a user files a "Tool Update" issue (auto-labeled `tool-bot:tool-update`) naming the tool directory and what to change, the bot posts an update plan, a maintainer approves, and the bot stages the existing `tools/<dir>` into its workspace, applies the plan, and opens a PR.

## Setup (for consuming repos)

### 1. Install the bot

The bot can be installed from PyPI or directly from GitHub:

```bash
pip install gxy-tool-bot              # from PyPI
pip install git+https://github.com/d-callan/gxy-tool-bot.git  # latest from GitHub
```

### 2. Create a config file

Create `.gxy-tool-bot.yml` in the repo root:

```yaml
api:
  base_url: https://openrouter.ai/api/v1   # or https://api.openai.com/v1
  model: z-ai/glm-5.2                       # or gpt-4o, etc.
  fallback_models:                          # optional; tried in order if primary fails
    - deepseek-ai/deepseek-r1
    - gpt-4o
  max_tool_iterations: 25
  # Optional: scale validation retry rounds by the number of tool XMLs in the plan.
  # The generator gets max_validation_retries + validation_retries_per_extra_tool_xml * (N - 1)
  # rounds, where N is the number of tool wrapper XMLs (macros.xml is not counted).
  # Each round gets the full max_tool_iterations budget. 0 (default) disables scaling.
  # e.g. with max_validation_retries=3 and validation_retries_per_extra_tool_xml=1,
  # a 4-XML tool family gets 3 + 1*(4-1) = 6 rounds of 25 iterations each.
  validation_retries_per_extra_tool_xml: 0
  temperature_plan: 0.4
  temperature_generate: 0.2
  max_context_chars: 100000
  max_validation_retries: 3
  read_timeout: 600                         # seconds; increase for slow models

exemplars:
  - url: https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/bcftools/bcftools_view.xml
    macros: https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/bcftools/macros.xml
    shed_yml: https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/bcftools/.shed.yml
  - url: https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/seqtk/seqtk_seq.xml
    macros: https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/seqtk/macros.xml
    shed_yml: https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/seqtk/.shed.yml

# repo is optional — falls back to GITHUB_REPOSITORY env var (set automatically in GitHub Actions)
repo: your-org/your-repo

# tool_owner is optional — defaults to 'iuc' for galaxyproject/tools-iuc,
# otherwise the GitHub repo owner (first part of owner/repo)
tool_owner: your-tool-shed-owner

allowed_maintainers:
  - your-github-handle

# Integrated self-review: runs a review agent after generate/feedback flows
# and feeds findings back to the original agent for fix rounds.
#   "never" (default) — no integrated review
#   "on-validation-pass" — review only after validation passes
#   "always" — review after completion regardless of validation status
integrated_review_mode: never
# Max review→fix rounds in integrated mode. Each round: review → feed findings → agent fixes.
# Default 1. 0 disables integrated review even if integrated_review_mode is set.
max_review_fix_rounds: 1

# Scheduled autoupdates (see "Scheduled autoupdates" below): detects newer
# versions of each tool's main conda requirement and opens a PR per tool.
# Disabled unless enabled: true.
autoupdate:
  enabled: false
  channels:                        # conda channels checked for latest versions
    - bioconda
    - conda-forge
  skip:                            # tool dirs (or XML paths) never to update
    - my_tool
  skip_file: autoupdate-skip.txt   # optional file with one entry per line
  max_tools_per_run: 10            # cap on tools updated per run (0 = no cap);
                                   # a first run on a big repo could otherwise
                                   # fire hundreds of agent jobs at once
```

### 3. Create GitHub labels

Run the helper script against your repo (requires the [`gh`](https://cli.github.com) CLI):

```bash
scripts/create-labels.sh owner/repo
```

It's idempotent (`gh label create --force`), so re-running is safe. It creates these labels:

| Label | Color | Purpose |
|-------|-------|---------|
| `tool-bot:tool-request` | `#0075ca` | Applied automatically by issue template; triggers planning |
| `tool-bot:tool-update` | `#0075ca` | Applied automatically by the Tool Update issue template; triggers update planning |
| `tool-bot:plan-ready` | `#a2eeef` | Applied by bot after plan is posted |
| `tool-bot:ready-to-implement` | `#0e8a16` | Applied by maintainer to approve plan; triggers generation |
| `tool-bot:pr-opened` | `#1d76db` | Applied by bot after PR is created |
| `tool-bot:generation-failed` | `#b60205` | Applied by bot if generation or planning fails |
| `tool-bot:retry-plan` | `#fbca04` | Applied by user/maintainer to re-trigger planning after a failure |
| `tool-bot:retry-generate` | `#fbca04` | Applied by user/maintainer to re-trigger generation after a failure |
| `tool-bot:address-feedback` | `#5319e7` | Applied to a PR to have the bot address review comments and CI failures |
| `tool-bot:review` | `#bfd4f2` | Applied to a PR to have the bot review tool files and post findings |

The prefixed names avoid collisions with generic labels on busy repos (e.g. a
plain `review` label applied for unrelated reasons would trigger a bot run).
For backward compatibility, the workflow templates and CLI also match the
original unprefixed names (`tool-request`, `review`, etc.) — existing
deployments keep working unchanged. If you know your repo only has the
prefixed labels, you can drop the unprefixed alternative from each `if:`
predicate.

> **Pinned-version caveat:** workflows `pip install gxy-tool-bot==<ver>`, and
> the *installed package* does its own label reads/writes. Prefixed-label
> support lands in releases after 0.1.2 — if your workflow pins 0.1.2 or
> earlier and your repo only has `tool-bot:*` labels, either use the legacy
> names or map them in `.gxy-tool-bot.yml`:
>
> ```yaml
> labels:
>   request: tool-bot:tool-request
>   tool_update: tool-bot:tool-update
>   plan_ready: tool-bot:plan-ready
>   ready_to_implement: tool-bot:ready-to-implement
>   pr_opened: tool-bot:pr-opened
>   generation_failed: tool-bot:generation-failed
>   review: tool-bot:review
> ```

### 4. Add the issue templates

Copy the templates from `examples/` in this repo into your repo's `.github/ISSUE_TEMPLATE/`:

- `issue-template-tool-request.yml` → `tool-request.yml` — auto-applies `tool-bot:tool-request`, triggers planning
- `issue-template-tool-update.yml` → `tool-update.yml` — auto-applies `tool-bot:tool-update`, triggers update planning

### 5. Add workflow files

Copy the workflow templates from the [`workflows/`](workflows/) directory in this repo into your repo's `.github/workflows/`:

- **`on-tool-request.yml`** → `.github/workflows/gxy-on-tool-request.yml` — triggers on new issues with `tool-bot:tool-request` or `tool-bot:tool-update` label or when `tool-bot:retry-plan` label is added; runs the planner (auto-detects update issues)
- **`on-ready-to-implement.yml`** → `.github/workflows/gxy-on-ready-to-implement.yml` — triggers when `tool-bot:ready-to-implement` or `tool-bot:retry-generate` label is added; runs the generator and opens a PR
- **`on-pr-feedback.yml`** → `.github/workflows/gxy-on-pr-feedback.yml` — triggers when `tool-bot:address-feedback` label is added to a PR; reads review comments and CI failures, pushes fixes as new commits
- **`on-pr-review.yml`** → `.github/workflows/gxy-on-pr-review.yml` — triggers when `tool-bot:review` label is added to a PR; reviews tool files and posts structured findings as a comment
- **`autoupdate.yml`** → `.github/workflows/gxy-autoupdate.yml` — runs on a cron schedule (edit the `cron:` line to set the frequency); detects outdated tool versions and opens one PR per tool. Requires `autoupdate.enabled: true` in `.gxy-tool-bot.yml`

> **CI artifact assumption:** The feedback workflow fetches CI failure details from GitHub Actions artifacts. This assumes the CI workflow uploads failure reports as artifacts (e.g. lint reports as `.txt` files, planemo test results as `.json`), following the same conventions as the [tools-iuc](https://github.com/galaxyproject/tools-iuc) repo's `pr.yaml` workflow. If your repo uses a different CI setup that doesn't upload artifacts on failure, the bot will not be able to include CI failure details in its feedback context.

> **Validation flag file:** When generation or feedback addressing completes but validation finds issues, the CLI writes a `.validation-failed` flag file to `GITHUB_WORKSPACE` (a built-in GitHub Actions env var). The workflows check for this file to post a comment on the PR noting that validation issues exist. The file is cleaned up after use and is never committed.

Both workflows install the bot from PyPI (pinned to a specific version):

```yaml
- run: pip install gxy-tool-bot==0.2.0
```

### 6. Add repo secrets

Go to Settings → Secrets and variables → Actions → New repository secret:

| Secret | Value | Used by |
|--------|-------|---------|
| `GXY_TOOL_BOT_API_KEY` | Your LLM API key (e.g. OpenRouter key) | Plan + generate workflows |

The `GITHUB_TOKEN` (automatically provided by GitHub Actions) is used for issue comments, labels, and PR creation — no extra secret needed.

### 7. Enable Issues

If the repo is a fork, Issues may be disabled by default. Enable them under Settings → General → Features → check "Issues".

### 8. Enable Actions

Make sure Actions are enabled: Settings → Actions → General → "Allow all actions and reusable workflows".

### 9. Test it

1. Go to Issues → New Issue → "Tool Request" template
2. Fill in the fields and submit
3. Check the Actions tab — the planning workflow should run
4. After the plan is posted, add the `ready-to-implement` label
5. The generation workflow should run and open a PR

## Scheduled autoupdates

`autoupdate.yml` runs on a cron schedule and is the one flow with no issue and no plan-approval step — the PR it opens *is* the review gate, like [planemo-autoupdate](https://github.com/planemo-autoupdate/autoupdate). On each run it:

1. Scans `tools/*/` for the main conda requirement — the `<requirement>` whose version is `@TOOL_VERSION@` (or `token_tool_version="..."`), falling back to the package matching the dir name.
2. Queries the configured conda channels for the latest version of that package.
3. For each outdated tool, runs the update agent to apply the bump *and* check upstream for breaking changes and new parameters — the thing a mechanical version bump can't do.
4. Opens one PR per tool titled `<tool_dir>: update tool wrapper to X.Y.Z`. The `<tool_dir>:` prefix means the `tool-bot:address-feedback` flow works on these PRs automatically.

Dedup rules (mirroring planemo-autoupdate, branch `tool-bot/autoupdate-<dir>`):

- An **open** PR on the tool's autoupdate branch → skipped, unless a newer version appeared since it opened — then new commits fold into the open PR and its title is updated (like planemo-autoupdate).
- A **closed, unmerged** autoupdate PR → a newer detected version reopens it; the same or an older version stays closed. So maintainers decline a specific version by closing the PR without deleting the branch.
- A branch whose last commit wasn't authored by the bot → never overwritten; the bot comments on the PR saying how to re-enable updates (delete the branch).
- Add tools to `autoupdate.skip` (or `autoupdate.skip_file`, one entry per line — tool dir names or `tools/<dir>/<file>.xml` paths like planemo-autoupdate's lists) to exclude them entirely.

The check frequency lives in the workflow's `cron:` line — GitHub reads schedules from the workflow file, not the bot config. Everything else (`enabled`, `channels`, `skip`, `skip_file`) is in `.gxy-tool-bot.yml`. Failed runs file or append to an "Autoupdate error: `<tool>`" issue.

## Philosophy & Design Decisions

### Run in GitHub Actions, not a black box

The bot runs entirely in GitHub Actions. This keeps agent traces, plans, generated files, and CI results publicly inspectable. Every run produces logs and artifacts you can review after the fact — to diagnose why a specific tool was written a certain way, or to identify patterns to improve the bot. The Actions logs capture tool calls, context size per iteration, and validation results; combined with file-based artifacts (generated files, test data) and summary comments posted to issues and PRs, this provides enough transparency to understand what the bot did and why.

### Validate files, don't over-prompt

Best practices don't go in the system prompt — they go in a [validation loop](gxy_tool_bot/validation.py) that inspects generated files for common mistakes (missing test data, bare output labels, Cheetah in macros, etc.) and instructs the agent to fix them in a retry loop. This keeps prompts concise (every line costs tokens on every LLM call) and prevents unnecessary bloat that might distract or confuse the agent, while still catching errors structurally. See [DEVELOPMENT.md](DEVELOPMENT.md) for the full tiered guidance on where to put conventions.

### Incremental feedback, not infinite loops

The validation loop only runs a limited number of times before forcing the result back to human hands — a commit is pushed to a PR that a maintainer can review and provide further direction on. The maintainer adds the `tool-bot:address-feedback` label to trigger another loop with CI failures and review comments incorporated. This keeps the bot autonomous for the common case but avoids burning tokens indefinitely on a problem it can't solve alone.

### Let planemo CI catch the rest

Things planemo already checks (XML well-formedness, shed metadata, duplicated output labels) are deliberately not duplicated in validation. The CI workflow reports these failures and the feedback flow picks them up on the next iteration. Only add a validation check if the bot is consistently making a specific mistake that wastes tokens and maintainer time.

### Direct CLI testing with run_in_conda

When `micromamba` or `conda` is available (installed automatically in CI), the bot has a `run_in_conda` tool that installs bioconda packages and runs commands directly — e.g. `samtools --help` to verify CLI flags, or `samtools view test-data/sample.bam | head` to inspect output format. This is useful when documentation is weak or planemo test output is hard to interpret. Environments are cached by package spec, so repeated calls with the same packages reuse the env. Only bioconda packages are supported; tools available exclusively as Docker/Singularity images cannot be test-run this way.

### Give the agent a way out

The agent has a `give_up` tool that lets it stop and explain why it can't proceed — open assumptions, unresolved questions, or fundamental issues with the request. A tool request might produce a recommendation not to make a tool rather than a plan to make one. A feedback request might result in push back with no commit until something the agent flags is resolved. This prevents forcing the agent into action when it doesn't have enough information, which would produce low-quality output. Better to surface the problem to a human than to generate a confident but wrong tool wrapper.

## Tips & Tricks

The bot is automated but not magic. It implements what's requested — it doesn't "think" about whether a request makes sense. If a request is vague or contradictory, the bot may not realize it needs to `give_up` and instead iterate on a wrong path and/or produce nonsense. The more detail you provide upfront, the better the results.

- **Edit the plans it generates.** The plan is your chance to course-correct before any code is written. Refine inputs, outputs, and parameters before adding the `ready-to-implement` label.
- **Give detailed review feedback.** Instead of "this is wrong", explain what's wrong and how to fix it. The bot follows instructions literally.
- **Resolve comments you don't need addressed.** During feedback loops, the bot sees all unresolved review comments. Stale comments it no longer needs to act on will confuse and distract it. Resolve them so the bot can focus on what still matters.
- **Help it with test failures.** Look at the CI failures yourself and tell the bot specifically how to fix them in a PR comment. The bot can struggle with planemo test failures that require domain knowledge or environment-specific context.
- **CI failures not uploaded as artifacts are invisible to the bot.** The feedback flow reads CI failure details from GitHub Actions artifacts. If a failure isn't uploaded as an artifact (e.g. a missing dependency, a runner error), the bot can't see it — you need to describe it in a PR comment explicitly.

## Tool Review

The bot can review Galaxy tool wrapper PRs for correctness, convention compliance, completeness, test coverage, and potential issues. It works in two modes:

### Standalone review

Add the `review` label to any PR with tool files. The bot:
1. Reads all tool files in the tool directory.
2. Fetches exemplar IUC tools for comparison.
3. Runs planemo lint/test if available.
4. Posts structured findings as a PR comment, grouped by severity (critical/warning/suggestion) and category (validation, conventions, completeness, test_coverage, security_bugs, brittleness).

This works on any PR — it doesn't matter whether the bot or a human created it.

### Integrated self-review

When `integrated_review_mode` is enabled in the config, the bot automatically reviews its own work after generate and feedback flows:

1. After the main agent loop + validation completes, a review agent (with fresh context) reviews the generated/updated files.
2. If the review finds critical or warning issues, the findings are fed back to the original agent as a user message (continuing from its conversation history — same pattern as validation retries).
3. The agent gets up to `max_review_fix_rounds` rounds to fix the issues.
4. Each fix round is followed by a re-review (fresh context) to check if the issues are resolved.

Config options:
```yaml
integrated_review_mode: "on-validation-pass"  # or "never" (default) or "always"
max_review_fix_rounds: 1  # number of review→fix rounds
```

## Development

See [DEVELOPMENT.md](DEVELOPMENT.md) for guidance on the codebase structure, where to add new conventions (prompts vs. validation vs. let CI catch it), and how the generation and feedback flows are organized.

## Eval Harness

The eval harness measures how well the bot generates tools and addresses feedback across a range of difficulty tiers. It runs real LLM calls against a set of eval cases (YAML + reference files) and reports validation pass rates, planemo lint/test results, iteration counts, and structural file correctness.

### Running locally

```bash
# Run all cases
gxy-tool-bot eval --config .gxy-tool-bot.yml --cases eval/cases/ --output eval-results.json

# Run only easy feedback cases
gxy-tool-bot eval --config .gxy-tool-bot.yml --cases eval/cases/ --filter difficulty=easy --filter type=feedback

# Run a specific case by name (glob)
gxy-tool-bot eval --config .gxy-tool-bot.yml --cases eval/cases/ --filter name=fb_fix_*

# Skip planemo checks (faster, no planemo install needed)
gxy-tool-bot eval --config .gxy-tool-bot.yml --cases eval/cases/ --no-planemo
```

The `--filter` option can be repeated to combine filters (different keys are AND'd, same key is OR'd). Supported keys: `name` (glob pattern), `type` (`generate`, `feedback`, or `update`), `difficulty` (`easy`, `medium`, `hard`).

Results are printed as a table to stdout and optionally written as JSON to `--output`.

### Running in CI

A manual dispatch workflow (`workflows/eval.yml`) runs the eval harness in GitHub Actions. Trigger it from the Actions tab with optional filter and planemo inputs. Results are uploaded as an artifact.

### Eval cases

Cases live in `eval/cases/<case_name>/` directories, each with a `case.yml` and supporting files:

- **Generate cases:** a `plan.md` fed to the generation pipeline, plus assertions on the output files.
- **Feedback cases:** "broken" tool files representing the PR state, plus simulated reviewer comments and CI failures, plus assertions on the expected fixes.
- **Update cases:** `type: update` — existing tool files plus a `plan.md` and an `update:` block (description, links) simulating the approved update plan; assertions check the updated files.

Cases are tiered by difficulty:
- **easy:** Cases the bot should consistently ace (simple tools, obvious fixes). Baseline for regression detection.
- **medium:** Multi-param tools, feedback with multiple comments. Should mostly pass.
- **hard:** Tool families (4+ XMLs), conflicting/naive reviewer suggestions, multi-issue feedback. Should struggle — improvements here are meaningful.

### Adding a new eval case

1. Create `eval/cases/<case_name>/case.yml` with metadata, assertions, and (for feedback) simulated feedback.
2. Add supporting files (`plan.md` for generate, broken tool files for feedback).
3. Run `gxy-tool-bot eval --cases eval/cases/ --filter name=<case_name>` to test it.

## License

MIT
