Metadata-Version: 2.4
Name: codi-api-agent
Version: 0.5.1
Summary: Read-only, spec-driven natural-language agent over any API (OpenAPI/Swagger, GraphQL, or auto-converted Postman/RAML/API Blueprint), with citations and a faithfulness check.
Author: API Agent
License: MIT
Keywords: openapi,graphql,llm,agent,api,postman,read-only,rag
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: openai>=1.30
Requires-Dist: requests>=2.31
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: graphql-core>=3.2
Provides-Extra: ui
Requires-Dist: streamlit>=1.30; extra == "ui"
Provides-Extra: embeddings
Requires-Dist: sentence-transformers>=3.0; extra == "embeddings"
Requires-Dist: numpy>=1.26; extra == "embeddings"
Provides-Extra: sql
Requires-Dist: psycopg[binary]>=3.1; extra == "sql"
Provides-Extra: all
Requires-Dist: streamlit>=1.30; extra == "all"
Requires-Dist: sentence-transformers>=3.0; extra == "all"
Requires-Dist: numpy>=1.26; extra == "all"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Provides-Extra: evals
Requires-Dist: evals>=3.0.1; extra == "evals"
Requires-Dist: pyyaml>=6; extra == "evals"

# API Agent

Load an **API description** — OpenAPI/Swagger or GraphQL natively, a Postman
collection / RAML / API Blueprint (auto-converted on load), or even a **prose API
reference doc** — ask a question in natural language, and the agent picks the
right operations, calls them (strictly read-only), and returns an **answer with
citations** plus a multi-signal **evaluation** (grounding, sufficiency,
responsiveness). Ships with a Streamlit UI with live steps, streaming answers,
and token/cost tracking.

```
Load spec(s) / reference doc ──▶ Catalog of operations
                     │
User question ──▶ Router + intent (narrow to relevant ops) ──▶ Executor (call operations)
                     ──▶ Synthesis + Citation ──▶ Self-review ──▶ Evaluator ──▶ Answer
```

The LLM provider is any **Chat Completions-compatible endpoint**, so the model is a config
value — hosted or local providers are swappable without code changes. The **generator**, **judge** and **router** roles are configured separately.
Full internals are documented in the source repository.

## Install as a pip package (share it / minimal setup)

```bash
python -m venv .venv && source .venv/bin/activate
pip install "codi-api-agent[all]"                          # UI + embeddings
# pip install "codi-api-agent[all,sql]"                    # …and the Postgres backend
# or from a wheel someone shared with you:
# pip install "codi_api_agent-0.3.1-py3-none-any.whl[all]"
export LLM_API_KEY="your-provider-key"
api-agent                                                  # opens the UI at http://localhost:8501
```

The distribution is `codi-api-agent`; the import name stays `api_agent`
(`from api_agent import Agent`).

Then add a free no-auth demo spec in the sidebar (Countries GraphQL or the SWAPI Postman
collection) and ask away. **Full step-by-step in [TUTORIAL.md](TUTORIAL.md).** Build the wheel
yourself from a checkout with `pip install build && python -m build` (→ `dist/`).

## Setup (from a source checkout)

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"       # editable install with UI + embeddings
# create .env with your provider + key, e.g.
printf 'LLM_BASE_URL=https://your-provider.example/v1\nLLM_API_KEY=your-key\n' > .env
```

Point `LLM_BASE_URL` at any Chat Completions-compatible endpoint and set `LLM_API_KEY`. A locally hosted
endpoint works the same way — use any non-empty string as the key if it does not check one.

Set `GENERATOR_MODEL` / `JUDGE_MODEL` to models your provider serves **that support
tool/function calling**.

**Rate-limit resilience:** give several keys (`LLM_API_KEYS=key1,key2,…`) and/or whole
backends (`LLM_POOL='[{"base_url":…,"api_key":…,"model":…}, …]'`) and the client rotates
over them per query and **fails over on a 429** with a cooldown — useful on free tiers.

**Cost controls (for paid, per-token providers):** set `MAX_RESPONSE_TOKENS` to cap tokens per
response — the agent stops early and returns a partial answer once the ceiling is hit (0 = unlimited).
Each response also shows an estimated **$ cost**; override the built-in per-model prices with
`MODEL_PRICING` (JSON per 1M tokens, e.g. `MODEL_PRICING='{"your-model":[2.5,10]}'`) or the sidebar
**💲 Budget & cost** fields. Unknown/free models simply show no cost.

## Run

```bash
api-agent            # console script installed with the package
```

It opens empty — add an API via the sidebar's **📚 Load an API** panel (set
`DEFAULT_SPEC=<url-or-path>` in `.env` to auto-load one on startup). Free, no-auth
demos to try: the **Countries GraphQL** endpoint `https://countries.trevorblades.com/`
(tick **GraphQL API**), or the Petstore spec
`https://petstore.swagger.io/v2/swagger.json` → *“fetch all pets that are sold”*.

## Load any API spec

Sidebar → **📚 Load an API** → paste a spec **URL or file path** (or upload files) →
**➕ Add spec**. OpenAPI/Swagger and GraphQL load natively; a Postman collection,
RAML 0.8, or API Blueprint file is **detected from its content and auto-converted**
to OpenAPI on load. You can add **several sources** (any mix of formats) — each keeps
its own base URL and auth, and one question can span all of them.

Every `GET`/`HEAD` operation (or GraphQL **query** field) becomes a callable tool
(write operations are excluded — the read-only guardrail; they remain *describable*
in documentation mode). Each operation maps: `operationId` → name,
`summary`/`description` → routing text, `parameters` → arguments, `servers`
(or Swagger-2.0 `host`+`basePath`) → base URL.

**Authenticated (private) APIs:** open the **Auth (optional)** expander and set a
header before loading — e.g. `Authorization` = `Bearer <token>`, or `X-API-Key` =
`<key>`. It's attached to every call. If an API needs auth and none is set, the
agent says so instead of guessing.

**No spec?** Write a small one by hand — describe just the `GET`
endpoints you care about (copy a block per endpoint), then load it as a file path. You don't
need to be an OpenAPI expert; the `summary`/`description` you write are what the agent routes on.

## Other formats (RAML / API Blueprint / Postman)

The agent's pipeline is format-agnostic — only the *loader* speaks OpenAPI — so these are
**converted to OpenAPI automatically when you load them** (in the UI or via `load_catalog`).
You can also convert ahead of time with the upstream npm tools directly
(`apib2swagger`, `api-spec-converter`, `postman-to-openapi`).

| Input | Converts to | Notes |
|---|---|---|
| **API Blueprint** (`.apib`) | Swagger 2.0 | read natively |
| **RAML 0.8** (`.raml`) | OpenAPI 3.0 | RAML **1.0** has no good free CLI converter — convert it to 0.8/OpenAPI first |
| **Postman collection** | OpenAPI 3.0 | converted on load via `postman-to-openapi` |

Requires Node/npx (the converters are npm tools, fetched on first use).

## No spec at all? Load a prose API reference doc

Sidebar → **📚 Load an API** → source type **API reference doc** → point it at a
reference page (URL, file, or upload) → **🔍 Extract endpoints**. The documented
`METHOD /path` lines, curl examples, path and query params are extracted from the
doc's **literal text** (free, deterministic — it cannot invent an endpoint); an
optional checkbox lets the LLM also enrich param types or handle prose-only docs.
You then **review and approve** the extracted endpoints before any become callable —
GETs load as tools, writes as documentation-only.

Inspect what a spec produces from Python:

```python
from api_agent.graphql_loader import load_catalog
catalog = load_catalog("your_spec.yaml")
print(len(catalog.tools), "callable operations")
```

## Connect a Postgres warehouse (SQL backend)

A data source does not have to be an HTTP API. Point the agent at a Postgres schema and every
function in it becomes a read-only tool — the same pipeline (route → execute → synthesize →
evaluate), minus the network hop.

```bash
pip install "codi-api-agent[sql]"          # adds psycopg; NOT included in [all]
export SQL_DSN="postgresql://readonly_user:pass@localhost:5432/warehouse"
```

Then Sidebar → **📚 Load an API** → **Postgres warehouse**, or set `SQL_DSN` and it loads on
start. Parameter names and types come from the database catalog itself, so there is no type
inference to get wrong, and descriptions come from each function's `COMMENT ON FUNCTION`.

**Read-only by construction, three independent walls:**

1. only introspected functions are callable — there is no tool that accepts a SQL string, so
   free-form SQL cannot be expressed;
2. every connection opens `default_transaction_read_only=on` with a `statement_timeout`;
3. point `SQL_DSN` at a role holding only `SELECT`/`EXECUTE`.

Arguments are bound server-side with an explicit `::type` cast taken from the catalog — never
interpolated into SQL text — and every call is `LIMIT`-capped.

| Setting | Default | Effect |
|---|---|---|
| `SQL_DSN` | — | connection string; use a SELECT-only role |
| `SQL_SCHEMA` | `gold` | schema to introspect |
| `SQL_FN_PREFIX` | `fn_` | only functions with this prefix are loaded |
| `SQL_TIMEOUT` | `30` | per-statement timeout, seconds |
| `SQL_PARAM_DOC_REF` | — | optional JSON documenting each parameter's allowed values; without it, parameters keep whatever value sets their own `COMMENT` declares |

Both backends can be loaded at once — a supervisor routes each question to exactly one of them.

## Verified working spec URLs (no auth)

| API | Spec URL |
|---|---|
| Petstore v2 (pets) — *default* | `https://petstore.swagger.io/v2/swagger.json` |
| APIs.guru (API directory) | `https://api.apis.guru/v2/specs/apis.guru/2.2.0/openapi.json` |
| ExchangeRate-API (FX rates) | `https://api.apis.guru/v2/specs/exchangerate-api.com/4/openapi.json` |
| Color Name API | `https://api.apis.guru/v2/specs/color.pizza/1.0.0/openapi.json` |

## Query your own GitHub repos

GitHub publishes its OpenAPI spec, so you can ask about *your* account:

1. **Load the public spec** (sidebar → **📚 Load an API** → *URL or file path*):
   `https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json`
   It has ~624 GET operations; the default **Max operations to load** (1000) loads them
   all and the router narrows per query.
2. **Set a token** in the **Auth** expander *before loading*: header `Authorization`,
   value `Bearer <your GitHub PAT>` (a token with `repo` / read scope).
3. **Ask:** *“show my repositories”* → routes to `GET /user/repos` and lists your repos.

More once loaded + authed: *“who am I on GitHub?”* (`GET /user`), *“list my open issues”*.

> The big GitHub spec takes a few seconds to fetch/parse on load. Without a token,
> authed endpoints return 401 and the agent tells you a key is needed.

## Routing (large specs)

When a spec has many operations, the **router** narrows them to the most relevant
per query before the agent runs: **hybrid** recall — lexical (idf-weighted, stemmed)
fused with **local embeddings** (a small local sentence-embedding model, free/offline; catches paraphrases
with zero shared words) via Reciprocal Rank Fusion — then a fast LLM makes the final
pick **and classifies the request intent** (write/doc/data) in the same call. Falls
back to lexical-only if embeddings are unavailable. Specs with ≤ `router_min_tools`
(default 6) operations skip routing. Configure in the sidebar or via
`ROUTER_ENABLED` / `ROUTER_TOP_K` / `ROUTER_MODEL` / `ROUTER_MIN_TOOLS` /
`EMBEDDING_MODEL`.

## Testing & performance report

The source repository carries a deterministic regression suite (no LLM, no network) and an
eval harness that reports guardrail reliability, per-category accuracy, faithfulness,
read-only/PII safety, abstention, latency and run-to-run consistency. Neither is part of the
installed package.

## Project layout

```
api_agent/
  config.py          # env-driven settings (provider, per-component models, router, budget)
  llm.py             # Chat Completions client + multi-LLM rotation + tool-call recovery
  openapi_loader.py  # OpenAPI/Swagger spec -> callable Tools; compaction + PII redaction
  graphql_loader.py  # GraphQL introspection/SDL + the load_catalog format dispatcher
  spec_convert.py    # Postman / RAML / API Blueprint -> OpenAPI (auto, via npx)
  doc_extract.py     # prose API reference doc -> draft OpenAPI (structural; LLM optional)
  catalog.py         # Tool + Catalog (operation registry + result cache + name recovery)
  router.py          # hybrid lexical+embedding routing per query
  schemas.py         # Evidence, Citation, ToolCall, Faithfulness, AgentResult, Usage
  agent.py           # pipeline: cache -> route -> execute -> synthesize -> review -> evaluate
  ui.py              # Streamlit app: live steps, streaming, Stop, token/cost, doc review gate
```

## Configuration reference

Everything is env-driven (`.env` or the environment). Only `LLM_BASE_URL` + `LLM_API_KEY` are
required; the rest have working defaults.

| Setting | Default | Effect |
|---|---|---|
| `GENERATOR_MODEL` / `JUDGE_MODEL` / `ROUTER_MODEL` | — / same / generator | the per-role models |
| `SYNTHESIS_MODEL` | generator | the **writer** model, used only where prose is produced |
| `LLM_API_KEYS` / `LLM_POOL` | — | rotation pools; fail over on a 429 |
| `EMBEDDING_MODEL` | a small local sentence-embedding model | hybrid semantic routing; blank = lexical only |
| `EMBEDDING_BASE_URL` / `EMBEDDING_API_KEY` | — | use a hosted `/embeddings` endpoint instead of the local model |
| `ROUTER_TOP_K` / `ROUTER_LLM_WINDOW` | 8 / 80 | operations reaching the executor / the router LLM |
| `ROUTER_MIN_TOOLS` | 6 | catalogs at or below this skip routing entirely |
| `MAX_TOOL_ITERATIONS` | 8 | executor loop ceiling |
| `MAX_SYNTHESIS_RETRIES` | 1 | self-correction retries when the validator flags a fixable claim |
| `LIST_DEFAULT` / `LIST_MAX` | 150 / 400 | rows shown by default / cap for "show all" |
| `PROFILE_ROWS` | 100000 | rows **scanned** to build the statistical profile (not shown) |
| `EVIDENCE_CHAR_LIMIT` / `EXECUTOR_CONTEXT_CHARS` | 12000 / 4000 | result size to synthesis / echoed back into the loop |
| `MAX_RESPONSE_TOKENS` | 0 | spending cap per response (0 = unlimited) |
| `MODEL_PRICING` | built-in | `{model: [in_per_1M, out_per_1M]}` for the $ estimate |
| `CACHE_SEMANTIC` | on | LLM confirm for reworded, value-matched cache repeats |
| `RENDER_EVIDENCE_TABLE` | off | render the evidence table and let the writer use tables |
| `RUBRIC_REPORT` | on | the per-answer quality scorecard |
| `SHARED_PROMPT_PREFIX` | on | share one prompt prefix across the full-evidence stages so the provider's cache can serve the repeats |
| `DATA_QUALITY_NOTES` | on | surface suspected load faults to the reader; off hides them (detection and logging continue) |
| `DEBUG_PANELS` | on | the **Sources** and **How this answer was produced** panels. **Set to 0 in production** — they name operations, arguments and sources |
| `JUDGE_ENABLED` / `ROUTER_ENABLED` | on | switch the judging / routing stages off |
| `HTTP_TIMEOUT` / `MAX_OPERATIONS` | 30 / 1000 | per-request timeout / cap on loaded operations |
| `LOG_LEVEL` / `LOG_FILE` / `LOG_PAYLOAD_CHARS` | `INFO` / — / 800 | `DEBUG` + `0` gives every step's full input and output |

Warehouse settings are in the SQL backend section above.

## Notes & limitations

- **Read-only:** only `GET`/`HEAD` operations are exposed; write endpoints are never called.
- **Honest failures:** if an operation needs a key (401/403), is unreachable, or the
  server returns 5xx, the agent reports that clearly instead of guessing.
- **Not production-hardened:** the loader fetches the given spec URL and calls
  endpoints as-is — SSRF egress controls and per-user credential scoping are
  follow-ups; fine for dev against trusted specs.
- Public demo servers (e.g. Petstore **v3**) are often flaky — prefer v2 / a spec
  whose server you control.
