Metadata-Version: 2.4
Name: ebaml
Version: 0.1.0
Summary: BAML for Explainable Boosting Machines: talk to your EBM.
Keywords: baml,llm,explainable-ai,explainable-boosting-machine,interpretml
Author: Johnny Godoy
Author-email: Johnny Godoy <johnny.godoy@ing.uchile.cl>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Dist: attrs>=26.1.0
Requires-Dist: baml-bridge>=0.23.0
Requires-Dist: beartype>=0.23.1
Requires-Dist: interpret>=0.7.8
Requires-Dist: orjson>=3.13.0
Requires-Dist: structlog>=26.1.0
Requires-Python: >=3.13
Project-URL: Documentation, https://johnny-godoy.github.io/EBaMl/
Project-URL: Source, https://github.com/johnny-godoy/EBaMl
Project-URL: Issues, https://github.com/johnny-godoy/EBaMl/issues
Project-URL: Changelog, https://github.com/johnny-godoy/EBaMl/blob/main/CHANGELOG.md
Description-Content-Type: text/markdown

# EBaMl

<!-- --8<-- [start:intro] -->
BAML for Explainable Boosting Machines. Python extracts the graphs from an EBM, and BAML holds the prompts, the LLM
clients and the chain-of-thought pipelines.

EBaMl is based on [TalkToEBM](https://github.com/interpretml/TalkToEBM) and adds local
explanations (why did the model predict this for this input?) computed exactly and put in words for a layperson or a
data scientist, a benchmark that scores answers automatically, a configurable LLM provider, token budgets and typed,
structured logging.

LLM calls go through [OpenRouter](https://openrouter.ai)'s free router (`openrouter/free`), so no paid API key is needed.

<!-- --8<-- [end:intro] -->
## Supported models

<!-- --8<-- [start:supported-models] -->
| Feature | Binary classifier | Regressor | Multiclass classifier | Interactions |
| --- | :---: | :---: | :---: | :---: |
| Graph descriptions (`describe_graph`) | ✅ | ✅ | ❌ | ⚠️ |
| Model summary (`describe_ebm`) | ✅ | ✅ | ❌ | ⚠️ |
| Graph text (`EBMGraph`) | ✅ | ✅ | ❌ | ⚠️ |
| Local explanations (`explain_instance`) | ✅ | ❌ | ❌ | ❌ |

✅ supported. ⚠️ partly supported: the main effects are described, and the interaction terms are skipped without a
warning. ❌ not supported yet. Full support is on the
[roadmap](https://github.com/johnny-godoy/EBaMl/blob/main/ROADMAP.md).
<!-- --8<-- [end:supported-models] -->
## Setup

<!-- --8<-- [start:setup] -->
### Requirements

- Python 3.13 or later and [uv](https://docs.astral.sh/uv/getting-started/installation/).
- The BAML toolchain 0.23 or later (step 1).
- An API key for an LLM provider. The default is [OpenRouter](https://openrouter.ai)'s free router, which needs no
  paid plan. You can use another provider instead, see "Choosing the model".

### 1. Install the BAML toolchain

```bash
curl -fsSL https://pkg.boundaryml.com/install.sh | sh -s
```

The installer prints a line to add to your shell profile. Add it, then open a new shell:

```bash
. "$HOME/.baml/env"
```

Check it:

```bash
baml --version   # toolchain 0.23 or later
```

If `baml` resolves to an older install (Homebrew `baml`, or `baml-py` in a virtualenv), remove it or put `~/.baml/bin`
first on `PATH`. The old CLIs have no `describe`, `generate add` or `agent` commands.

If a command stops with "the BAML agent skill is required", run `baml agent install` and retry.

### 2. Get the code and install the Python dependencies

```bash
git clone <repository-url> EBaMl
cd EBaMl
uv sync
```

### 3. Set the API key of your LLM provider

For the default provider, OpenRouter:

1. Create an account at <https://openrouter.ai> and open <https://openrouter.ai/settings/keys>.
2. Create a key. Keys start with `sk-or-`.
3. Export it in the shell you run the code from:

```bash
export OPENROUTER_API_KEY=sk-or-...
```

The key is read from the environment when a request is sent. It is never written to a `.baml` file.

Using another provider? Set its variable instead (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, ...) and pass `model=` as
described in "Choosing the model". Local models (Ollama) need no key.

### 4. Generate the Python SDK

```bash
baml generate
```

This writes the `src/baml_sdk/` package from `baml_src/`, next to `ebaml`, so it imports wherever the project is
installed. Rerun it every time a `.baml` file changes, including when you change the provider.

If `src/baml_sdk/` does not exist, `ebaml.llm.descriptors` runs `baml generate` when you import it. If the package was
generated by another BAML toolchain than the one that `baml-bridge` expects (a version skew), the import runs
`baml toolchain use <version>` and `baml generate`. If the agent skill check stops `baml generate`, the import runs it
again with `--agent-skill-check off`. If none of this works, `SDKUnavailableError` lists the steps that fix the cause.

### 5. Check the setup

```bash
OPENROUTER_API_KEY=unused baml test -x root::Live   # offline tests on a scripted client, no network
baml run main                         # prints "hello from baml"
```

These two do not call an LLM, so they pass without a working key or provider. They check the toolchain and the SDK.

### 6. Live test (optional)

Makes real requests to your provider (9 on the OpenRouter free router, so it can be slow or rate limited). It does
nothing without `OPENROUTER_API_KEY`, so with another provider run the example under "Usage" instead.

```bash
OPENROUTER_API_KEY=sk-or-... uv run python scripts/live_openrouter.py
```

It fits a small EBM, runs the three-turn chain directly and checks that the provider reports token usage, then runs
`describe_graph` and `describe_ebm`. It exits non-zero on the first failed assertion.

### Troubleshooting

| Symptom | Cause and fix |
| --- | --- |
| `baml: command not found` | The profile line from step 1 is not loaded. Open a new shell or run `. "$HOME/.baml/env"`. |
| `unknown command: describe` (or `generate add`, `agent`) | An older `baml` is first on `PATH`. See step 1. |
| `SDKUnavailableError`, or `ModuleNotFoundError: baml_sdk` | The SDK is not generated or has the wrong version, and `ebaml` could not fix it. The message of the error lists the steps. Run `baml generate` (step 4). |
| `MissingAPIKeyError` | The key of the provider is not exported in this shell. The message names the variable. See "Choosing the model" for the variable of each provider. |
| `APIKeyRejectedError` | The provider rejected the key. Check the key, its account and its credit. |
| `RateLimitedError` | The provider limited the rate. Lower the concurrency, `describe_ebm(ebm, max_concurrency=2)`, or pin a paid model. |
| `ProviderUnreachableError` | A request did not reach the provider. Check the network, `HTTPS_PROXY` and `base_url`. |

<!-- --8<-- [end:setup] -->
## Usage

<!-- --8<-- [start:usage] -->
```python
from interpret.glassbox import ExplainableBoostingClassifier

from ebaml.llm.descriptors import describe_ebm, describe_graph
from ebaml.llm.options import PromptOptions

ebm = ExplainableBoostingClassifier().fit(X, y)

print(describe_graph(ebm, feature_index=0))
print(describe_ebm(ebm, num_sentences=30, prompt=PromptOptions(dataset_description="...")))
```

A graph can be turned into text with `EBMGraph.to_text` and back with `EBMGraph.from_text`, which also reads a graph
embedded in a longer prompt. It needs the confidence bounds, and it only reads continuous features.

Three keyword-only arguments set the options. `graph=GraphOptions(...)` sets the options of `EBMGraph.from_ebm` and
`EBMGraph.to_text`, for example `max_tokens`. `prompt=PromptOptions(...)` sets the descriptions that the prompts use
(`expert_description`, `graph_description`, `dataset_description` and `task_description`). `model=` selects the LLM, see
"Choosing the model". The classes are in `ebaml.llm.options`. Token usage reported by the provider is logged at `INFO` on the `ebaml.llm.descriptors` logger.

### Explaining one prediction

```python
from ebaml.llm.descriptors import describe_ebm, explain_instance

ebm = ExplainableBoostingClassifier(interactions=0).fit(X, y)  # binary classifier, no interactions
global_explanation = describe_ebm(ebm)  # computed once, prepended to every local prompt

explanation = explain_instance(ebm, X.iloc[0].to_dict(), global_explanation, audience="layperson")
print(explanation.counterfactual)      # what would have to change to flip the prediction
print(explanation.sufficient_reason)   # which inputs settle the prediction on their own
print(explanation.completions)         # how certain the outcome is as inputs become known
```

The facts are computed exactly from the scores and confidence intervals of the EBM (`ebaml.local`); the LLM only puts
them in words. They are the three queries of [arXiv:2010.12265](https://arxiv.org/abs/2010.12265): minimum change
required, minimum sufficient reason and count completions. `audience="layperson"` (default) gives no model numbers;
`audience="data_scientist"` quotes scores, probabilities and confidence intervals. It supports binary classifiers
without interactions, and `explanation.facts` holds the numbers behind the text. The "Local explanations" page of the documentation explains what each explanation means, the options and the limits,
and `notebooks/local_explanations.py` walks through them on the Titanic data.

### Errors

Every error that `ebaml` raises on purpose derives from `ebaml.EBaMlError`. Use `except EBaMlError` to catch the errors
of `ebaml` apart from the errors of other code. Each error for a bad value also derives from `ValueError`, so an
`except ValueError` catches it too.

| Error | `ValueError` | Raised when |
| --- | --- | --- |
| `InvalidOptionError` | yes | An option is not one of its valid values, for example an unknown `Normalization`. |
| `OptionRangeError` | yes | A number option is outside its range, for example `confidence_level=1`. |
| `InvalidInstanceError` | yes | The instance to explain lacks a value, or has a category that the EBM did not see. |
| `UnsupportedModelError` | yes | A local explanation gets an EBM with interactions, or one that is not a binary classifier. |
| `GraphTextError` | yes | `EBMGraph.from_text` cannot read the text of a graph. |
| `EBMGraphError` | no | The base class of the errors of a graph. |
| `EBMGraphShapeError` | yes | The bins of a graph do not fit its scores, or a boolean format gets more than two bins. |
| `EBMGraphTokenLimitError` | no | The text of a graph is over the token budget, and its format cannot be simplified. |
| `EBMGraphSimplificationError` | no | The text of a graph is still over the token budget at the largest simplification. |
| `SDKUnavailableError` | no | `baml_sdk` is missing, and `ebaml` cannot run `baml generate`. The error also derives from `ImportError`. |
| `LLMRequestError` | no | The base class of the errors of a provider. Its message lists the steps that fix the cause. |
| `MissingAPIKeyError` | no | A provider needs an API key and the environment has none. |
| `APIKeyRejectedError` | no | A provider answers a request with "unauthorized" or "forbidden". |
| `RateLimitedError` | no | A provider limits the rate of the requests. |
| `ProviderUnreachableError` | no | A request does not reach the provider. |

`EBMGraphShapeError`, `EBMGraphTokenLimitError` and `EBMGraphSimplificationError` derive from `EBMGraphError`. The four provider errors derive from `LLMRequestError`. All classes are in `ebaml.core.exceptions`, and `ebaml` exports
them too.

<!-- --8<-- [end:usage] -->
## How it works

| Where | What |
| --- | --- |
| `src/ebaml/core/` | Shared kernel: types and errors. It imports no other part of `ebaml`. |
| `src/ebaml/graphs/` | Graph extraction and simplification (numpy, `interpret`), and the text the LLM reads. |
| `src/ebaml/llm/` | The edge: the calls into BAML (`descriptors.py`), their options and the retry after a cut-off reply. |
| `baml_src/talk_to_ebm.baml` | Prompts, the default LLM clients, the three-turn description chain, the summary, `describe_ebm`, token accounting. |
| `baml_src/model_spec.baml` | The model of a call: `ModelSpec` and one client builder for each provider. |
| `src/ebaml/local/`, `baml_src/local_explanations.baml` | Local explanations: the facts of one prediction computed deterministically, their rendering as JSON texts, and the prompts that put them in words. |
| `baml_src/talk_to_ebm_test.baml`, `baml_src/model_spec_test.baml` | Offline tests on a scripted client. |
| `baml.toml` | Generator config for the Python SDK. |
| `src/ebaml/benchmark.py`, `benchmark/` | The benchmark: loading, answer parsing and scoring, and the data and runner (see [Benchmark](#benchmark)). |
| `notebooks/` | Example notebooks, written with marimo. |
| `docs/`, `zensical.toml` | The documentation, built with Zensical (see [Development](#development)). |
| `tests/unit/` | Python unit tests (100% coverage required). `scripts/live_openrouter.py` is the optional live test. |
| `scripts/` | Maintenance scripts, see [scripts/README.md](scripts/README.md). |
| `.github/` | CI workflow and the scripts that turn failures into reports. |
| `src/baml_sdk/` | Generated. Do not edit. |

<!-- --8<-- [start:behaviour] -->
- **Model:** `openrouter/free` picks a free model for each request, so the model can change between calls and between
  the turns of one conversation. Quality and speed vary with it.
- **Retries:** transient failures (rate limits, network errors, 5xx) are retried up to 3 times. Requests the provider
  rejects are not retried.
- **Concurrency:** `describe_ebm` describes features concurrently, at most `max_concurrency` at a time (default 8).
  Free models are rate limited, so lower it (`describe_ebm(ebm, max_concurrency=2)`) if you see rate-limit errors.
- **Token budget:** there is no tokenizer before a call, so the check uses a rough estimate (about 3 characters per
  token). Going a little over `max_tokens` only warns. Going more than `overbudget_percent` (default 50) over makes
  `to_text` simplify the graph, or raise if it cannot. The provider's own error is the real limit. The functions in
  `ebaml.llm.descriptors` pass the BAML check. A direct call of `EBMGraph.to_text` checks nothing, unless you pass
  `check_budget`, for example `baml_sdk.check_token_budget`.

<!-- --8<-- [end:behaviour] -->
## Notebook

[`notebooks/titanic.py`](notebooks/titanic.py) is a [marimo](https://marimo.io) notebook that ports TalkToEBM's example.
It trains an EBM on the Titanic dataset (downloaded from OpenML), asks the LLM to describe a graph and the whole model,
and shows the graph and text API (`from_ebm`, `simplify`, `to_text`, `from_text`). The LLM cells run only when you press
their button. They use the default model, which needs `OPENROUTER_API_KEY`. See "Choosing the model" for other models.

```bash
uv run marimo edit notebooks/titanic.py   # interactive
uv run marimo run notebooks/titanic.py    # as an app
```

[`notebooks/local_explanations.py`](notebooks/local_explanations.py) is the place to learn the local explanations: pick a
passenger, see the facts with no API key, move the confidence slider, check the guarantees against random completions,
compare what the LLM reads for a layperson and a data scientist, and (with a key) get the explanations in words.

```bash
uv run marimo edit notebooks/local_explanations.py
```

## Benchmark

The benchmark follows [TalkToEBM's benchmark](https://github.com/interpretml/TalkToEBM/tree/main/benchmark) that checks how well a
model reads EBM graphs: function recognition (easy and hard), the largest jump, monotonicity, reading a value and the
widest confidence interval. It makes real requests, and it asks two LLM turns per question (6 tasks, 480 questions in
all). The default model needs `OPENROUTER_API_KEY`. The flag `--model provider/model` selects another model.

```bash
uv run python benchmark/run.py --task monotonicity --limit 10 --output results.json
```

See [benchmark/README.md](benchmark/README.md) for the tasks, the scoring and what it leaves out.

## Choosing the model

<!-- --8<-- [start:changing-model] -->
The default model is OpenRouter's free router, `openrouter/free`. It needs `OPENROUTER_API_KEY`. The `model` argument
of `describe_graph`, `describe_ebm` and `explain_instance` selects another model for one call. EBaMl sends every LLM
request of that call to that model. You do not change the package.

### Quick start

Pass the shorthand `"provider/model"`:

```python
from ebaml.llm.descriptors import describe_ebm

print(describe_ebm(ebm, model="anthropic/claude-haiku-4-5"))
```

Pass `ModelOptions` to set more:

```python
from ebaml.llm.options import ModelOptions

model = ModelOptions(provider="anthropic", model="claude-haiku-4-5", temperature=0.2, max_output_tokens=8000)
print(describe_ebm(ebm, model=model))
```

EBaMl splits the shorthand at the first `/`. The text `"openrouter/openai/gpt-4o-mini"` selects the provider
`openrouter` and the model `openai/gpt-4o-mini`. EBaMl checks the provider before the first paid request. It raises
`InvalidOptionError` if the provider is not in the table below.

### Providers

The provider is a BAML provider prefix. Set the environment variable of the key in the shell that runs Python.

| Provider | Default key variable | Notes |
| --- | --- | --- |
| `openrouter` | `OPENROUTER_API_KEY` | The model name is the OpenRouter name, for example `openai/gpt-4o-mini`. |
| `openai` | `OPENAI_API_KEY` | |
| `anthropic` | `ANTHROPIC_API_KEY` | |
| `google` | `GOOGLE_API_KEY` | The Gemini API. |
| `ollama` | None | A local server. `base_url` defaults to `http://localhost:11434/v1`. |
| `openai-generic` | None | Any OpenAI-compatible server. `base_url` is required. Set `api_key_env` if the server needs a key. |
| `azure` | `AZURE_OPENAI_API_KEY` | The model name is the name of the deployment. Set `base_url` to the deployment, or set `AZURE_OPENAI_ENDPOINT`. |
| `vertex` | None | Google Cloud credentials (`GOOGLE_APPLICATION_CREDENTIALS` or `gcloud auth`). The project comes from `GOOGLE_CLOUD_PROJECT`. |
| `bedrock` | None | The AWS credentials of the machine. The region comes from `AWS_REGION`. `api_key_env` has no effect. |

Run `baml describe <client>.new`, for example `baml describe aws.BedrockClient.new`, to see all settings of a client.

### Options

| Field | Meaning | Default |
| --- | --- | --- |
| `provider` | The BAML provider prefix from the table. | Required. |
| `model` | The name of the model at the provider. | Required. |
| `api_key_env` | The name of the environment variable that holds the key. | The variable of the provider. |
| `base_url` | The address of the server. | The address of the provider. |
| `temperature` | The sampling temperature. | 0.7 (0.0 in the benchmark). |
| `max_output_tokens` | The most tokens of one reply. | See below. |
| `headers` | Headers to add to each request. | None. |

A field that you set wins over the default of the call. The output limit has two defaults:

| Calls | `max_output_tokens` |
| --- | --- |
| Turn 1 of a graph description, the draft of the summary, and the two turns of the benchmark | 8000 |
| All other calls, including the local explanations | 6000 |

### Examples

```python
# OpenRouter, a pinned model (the free router picks a new model for each request)
describe_ebm(ebm, model="openrouter/openai/gpt-4o-mini")

# OpenAI
describe_ebm(ebm, model="openai/gpt-4o-mini")

# Google Gemini
describe_ebm(ebm, model="google/gemini-2.5-flash")

# A local model with Ollama. It needs no key.
describe_ebm(ebm, model="ollama/llama3.1")

# A company gateway with an OpenAI-compatible API
gateway = ModelOptions(
    provider="openai-generic",
    model="my-model",
    base_url="https://llm.example.com/v1",
    api_key_env="MY_GATEWAY_KEY",
    headers={"x-team": "ebm"},
)
describe_ebm(ebm, model=gateway)
```

The benchmark has the flag `--model`:

```bash
uv run python benchmark/run.py --model anthropic/claude-haiku-4-5 --limit 10
```

### Keys

Put the key in an environment variable, in the shell or with `os.environ`. `ModelOptions` holds the name of the
variable, never the key, so the key does not appear in a `repr`, a log line or a traceback. EBaMl reads the variable
only when it sends a request.

### Limits

- One call uses one model. The turns of a call cannot use different models.
- Python cannot set up fallback chains or round robin. A BAML client object cannot cross from Python to BAML, so
  Python sends plain data and BAML builds the client. A BAML client declaration can use `ai.clients.Fallback`: see
  "Change the default model" in `CONTRIBUTING.md`.
- A reasoning model counts its hidden tokens in `max_output_tokens`. Keep the limit large, or the reply can end before
  the answer.
- Quality, speed and cost depend on the model. Small models can ignore the number of sentences that you ask for.

Every client has the retry policy of the default clients: EBaMl retries a transient failure up to 3 times.
<!-- --8<-- [end:changing-model] -->

## Development

<!-- --8<-- [start:development] -->

```bash
baml fmt          # format .baml files
baml check        # compile check
baml test -x root::Live   # run the offline BAML tests
baml generate     # regenerate src/baml_sdk/
uv run pytest     # Python unit tests, fails under 100% coverage
uv run pytest README.md docs src --no-cov   # the code examples of the docs and the docstrings
prek run --all-files                       # every hook: formatters, ruff, vulture, pyrefly, baml fmt, ...
uv run python scripts/mutation.py          # mutation testing of all modules, fails if any mutant survives
uv run python scripts/mutation.py --base origin/main   # the same, but only for modules changed since main
uv run python scripts/mutation.py --module src/ebaml/local/facts.py   # the same, but only for one module
uv run python -m scripts.check_ignores     # every lint/type/test workaround needs a reason written next to it
uv run python -m scripts.compile_ignores   # all workarounds and their reasons as one Markdown report to audit
uv run --group docs python -m scripts.build_docs         # build the docs; fails on a broken link or a docstring warning
uv run --group docs python -m scripts.build_docs serve   # preview the docs (exports the notebooks, so the first run is slow)
```

CI (`.github/workflows/ci.yml`) runs prek, secret and security scans (trufflehog, semgrep, trivy), `baml check` and
`baml test`, pytest with 100% coverage, the code examples of the documentation, and mutation testing of the changed modules with a 100% score. A weekly run tests all modules, one job for each module. Lint and type workarounds
(`# noqa`, `# type: ignore`, skipped tests, ...) must carry a written reason next to them.

The `python` code blocks of `README.md` and `docs/*.md` run in order, and the blocks of one file share their variables.
The `Examples` sections of the docstrings run as doctests. `tests/docs/conftest.py` gives each Markdown file a small
data set as `X` and `y`, and it replaces the LLM calls with fakes, so the examples send no request. Put setup that the
reader does not need in an `<!-- invisible-code-block: python -->` comment. To skip a block, put
`<!-- skip: next -->` above it, and write the reason in the text.
<!-- --8<-- [end:development] -->
