Metadata-Version: 2.4
Name: ebaml
Version: 0.2.0
Summary: BAML for Explainable Boosting Machines: talk to your EBM.
Keywords: baml,llm,explainable-ai,explainable-boosting-machine,interpretml
Author: Johnny Godoy
Author-email: Johnny Godoy <johnny.godoy@ing.uchile.cl>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Dist: attrs>=26.1.0
Requires-Dist: baml-bridge>=0.23.0
Requires-Dist: beartype>=0.23.1
Requires-Dist: interpret>=0.7.8
Requires-Dist: orjson>=3.13.0
Requires-Dist: structlog>=26.1.0
Requires-Python: >=3.13
Project-URL: Documentation, https://johnny-godoy.github.io/EBaMl/
Project-URL: Source, https://github.com/johnny-godoy/EBaMl
Project-URL: Issues, https://github.com/johnny-godoy/EBaMl/issues
Project-URL: Changelog, https://github.com/johnny-godoy/EBaMl/blob/main/CHANGELOG.md
Description-Content-Type: text/markdown

# EBaMl

<!-- --8<-- [start:intro] -->
BAML for Explainable Boosting Machines. Python extracts the graphs from an EBM, and BAML holds the prompts, the LLM
clients and the chain-of-thought pipelines.

EBaMl is based on [TalkToEBM](https://github.com/interpretml/TalkToEBM) and adds local
explanations (why did the model predict this for this input?) computed exactly and put in words for a layperson or a
data scientist, a benchmark that scores answers automatically, a configurable LLM provider, token budgets and typed,
structured logging.

LLM calls go through [OpenRouter](https://openrouter.ai)'s free router (`openrouter/free`), so no paid API key is needed.

<!-- --8<-- [end:intro] -->
## Supported models

<!-- --8<-- [start:supported-models] -->
| Feature | Binary classifier | Regressor | Multiclass classifier | Interactions |
| --- | :---: | :---: | :---: | :---: |
| Graph descriptions (`describe_graph`) | ✅ | ✅ | ✅ | ⚠️ |
| Model summary (`describe_ebm`) | ✅ | ✅ | ✅ | ⚠️ |
| Graph text (`EBMGraph`) | ✅ | ✅ | ✅ | ⚠️ |
| Local explanations (`explain_instance`) | ✅ | ❌ | ❌ | ❌ |

✅ supported. ⚠️ partly supported: the main effects are described, and the interaction terms are skipped.
`describe_graph` and `describe_ebm` log an `interactions_skipped` warning that names them. ❌ not supported yet. Full support is on the
[roadmap](https://github.com/johnny-godoy/EBaMl/blob/main/ROADMAP.md).
<!-- --8<-- [end:supported-models] -->
## Setup

<!-- --8<-- [start:setup] -->
### Requirements

- Python 3.13 or later and [uv](https://docs.astral.sh/uv/getting-started/installation/).
- The BAML toolchain 0.23 or later (step 1).
- An API key for an LLM provider. The default is [OpenRouter](https://openrouter.ai)'s free router, which needs no
  paid plan. You can use another provider instead, see "Choosing the model".

### 1. Install the BAML toolchain

```bash
curl -fsSL https://pkg.boundaryml.com/install.sh | sh -s
```

The installer prints a line to add to your shell profile. Add it, then open a new shell:

```bash
. "$HOME/.baml/env"
```

Check it:

```bash
baml --version   # toolchain 0.23 or later
```

If `baml` resolves to an older install (Homebrew `baml`, or `baml-py` in a virtualenv), remove it or put `~/.baml/bin`
first on `PATH`. The old CLIs have no `describe`, `generate add` or `agent` commands.

If a command stops with "the BAML agent skill is required", run `baml agent install` and retry.

### 2. Get the code and install the Python dependencies

```bash
git clone <repository-url> EBaMl
cd EBaMl
uv sync
```

### 3. Set the API key of your LLM provider

For the default provider, OpenRouter:

1. Create an account at <https://openrouter.ai> and open <https://openrouter.ai/settings/keys>.
2. Create a key. Keys start with `sk-or-`.
3. Export it in the shell you run the code from:

```bash
export OPENROUTER_API_KEY=sk-or-...
```

The key is read from the environment when a request is sent. It is never written to a `.baml` file.

Using another provider? Set its variable instead (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, ...) and pass `model=` as
described in "Choosing the model". Local models (Ollama) need no key.

### 4. Generate the Python SDK

```bash
baml generate
```

This writes the `src/baml_sdk/` package from `baml_src/`, next to `ebaml`, so it imports wherever the project is
installed. Rerun it every time a `.baml` file changes, including when you change the provider.

If `src/baml_sdk/` does not exist, `ebaml.llm.descriptors` runs `baml generate` when you import it. If the package was
generated by another BAML toolchain than the one that `baml-bridge` expects (a version skew), the import runs
`baml toolchain use <version>` and `baml generate`. If the agent skill check stops `baml generate`, the import runs it
again with `--agent-skill-check off`. If none of this works, `SDKUnavailableError` lists the steps that fix the cause.

### 5. Check the setup

```bash
OPENROUTER_API_KEY=unused baml test -x root::Live   # offline tests on a scripted client, no network
baml run main                         # prints "hello from baml"
```

These two do not call an LLM, so they pass without a working key or provider. They check the toolchain and the SDK.

### 6. Live test (optional)

Makes real requests to your provider (9 on the OpenRouter free router, so it can be slow or rate limited). It does
nothing without `OPENROUTER_API_KEY`, so with another provider run the example under "Usage" instead.

```bash
OPENROUTER_API_KEY=sk-or-... uv run python scripts/live_openrouter.py
```

It fits a small EBM, runs the three-turn chain directly and checks that the provider reports token usage, then runs
`describe_graph` and `describe_ebm`. It exits non-zero on the first failed assertion.

### Troubleshooting

| Symptom | Cause and fix |
| --- | --- |
| `baml: command not found` | The profile line from step 1 is not loaded. Open a new shell or run `. "$HOME/.baml/env"`. |
| `unknown command: describe` (or `generate add`, `agent`) | An older `baml` is first on `PATH`. See step 1. |
| `SDKUnavailableError`, or `ModuleNotFoundError: baml_sdk` | The SDK is not generated or has the wrong version, and `ebaml` could not fix it. The message of the error lists the steps. Run `baml generate` (step 4). |
| `MissingAPIKeyError` | The key of the provider is not exported in this shell. The message names the variable. See "Choosing the model" for the variable of each provider. |
| `APIKeyRejectedError` | The provider rejected the key. Check the key, its account and its credit. |
| `RateLimitedError` | The provider limited the rate. Lower the concurrency, `describe_ebm(ebm, max_concurrency=2)`, or pin a paid model. |
| `ProviderUnreachableError` | A request did not reach the provider. Check the network, `HTTPS_PROXY` and `base_url`. |

<!-- --8<-- [end:setup] -->
## Usage

<!-- --8<-- [start:usage] -->
```python
from interpret.glassbox import ExplainableBoostingClassifier

from ebaml.llm.descriptors import describe_ebm, describe_graph
from ebaml.llm.options import PromptOptions

ebm = ExplainableBoostingClassifier().fit(X, y)

print(describe_graph(ebm, feature_index=0))
print(describe_ebm(ebm, num_sentences=30, prompt=PromptOptions(dataset_description="...")))
```

A graph can be turned into text with `EBMGraph.to_text` and back with `EBMGraph.from_text`, which also reads a graph
embedded in a longer prompt. It needs the confidence bounds, and it only reads continuous features.

Three keyword-only arguments set the options. `graph=GraphOptions(...)` sets the options of `EBMGraph.from_ebm` and
`EBMGraph.to_text`, for example `max_tokens`. `prompt=PromptOptions(...)` sets the descriptions that the prompts use
(`expert_description`, `graph_description`, `dataset_description` and `task_description`). `model=` selects the LLM, see
"Choosing the model". The classes are in `ebaml.llm.options`. Token usage reported by the provider is logged at `INFO` on the `ebaml.llm.descriptors` logger.

A multiclass classifier has one graph series per class. `GraphOptions(classes=[...])` selects the classes to describe,
because the text of a model with many classes is long. In every bin, the scores have the same mean over all the classes
of the model. Only the differences between the classes change the predicted probabilities.
`notebooks/multiclass.py` shows each option on the iris data.

<!-- invisible-code-block: python
import numpy as np

y_multiclass = np.select([X["age"] < 30, X["age"] < 60], ["young", "middle"], "old")
-->

```python
from ebaml.llm.options import GraphOptions

multiclass = ExplainableBoostingClassifier(interactions=0).fit(X, y_multiclass)
print(describe_graph(multiclass, feature_index=0, graph=GraphOptions(classes=["young", "old"])))
```

### Explaining one prediction

```python
from ebaml.llm.descriptors import describe_ebm, explain_instance

ebm = ExplainableBoostingClassifier(interactions=0).fit(X, y)  # binary classifier, no interactions
global_explanation = describe_ebm(ebm)  # computed once, prepended to every local prompt

explanation = explain_instance(ebm, X.iloc[0].to_dict(), global_explanation, audience="layperson")
print(explanation.counterfactual)      # what would have to change to flip the prediction
print(explanation.sufficient_reason)   # which inputs settle the prediction on their own
print(explanation.completions)         # how certain the outcome is as inputs become known
```

The facts are computed exactly from the scores and confidence intervals of the EBM (`ebaml.local`); the LLM only puts
them in words. They are the three queries of [arXiv:2010.12265](https://arxiv.org/abs/2010.12265): minimum change
required, minimum sufficient reason and count completions. `audience="layperson"` (default) gives no model numbers;
`audience="data_scientist"` quotes scores, probabilities and confidence intervals. It supports binary classifiers
without interactions, and `explanation.facts` holds the numbers behind the text. The "Local explanations" page of the documentation explains what each explanation means, the options and the limits,
and `notebooks/local_explanations.py` walks through them on the Titanic data.

### Errors

Every error that `ebaml` raises on purpose derives from `ebaml.EBaMlError`. Use `except EBaMlError` to catch the errors
of `ebaml` apart from the errors of other code. Each error for a bad value also derives from `ValueError`, so an
`except ValueError` catches it too.

| Error | `ValueError` | Raised when |
| --- | --- | --- |
| `InvalidOptionError` | yes | An option is not one of its valid values, for example an unknown `Normalization`. |
| `OptionRangeError` | yes | A number option is outside its range, for example `confidence_level=1`. |
| `InvalidInstanceError` | yes | The instance to explain lacks a value, or has a category that the EBM did not see. |
| `UnsupportedModelError` | yes | A local explanation gets an EBM with interactions, or one that is not a binary classifier. A graph gets `classes`, and the model has one output. `model_kind` gets an EBM that is not fitted. |
| `GraphTextError` | yes | `EBMGraph.from_text` cannot read the text of a graph. |
| `EBMGraphError` | no | The base class of the errors of a graph. |
| `EBMGraphShapeError` | yes | The bins of a graph do not fit its scores, the scores and the standard deviations have different shapes, or a boolean format gets more than two bins. |
| `EBMGraphTokenLimitError` | no | The text of a graph is over the token budget, and its format cannot be simplified. |
| `EBMGraphSimplificationError` | no | The text of a graph is still over the token budget at the largest simplification. |
| `SDKUnavailableError` | no | `baml_sdk` is missing, and `ebaml` cannot run `baml generate`. The error also derives from `ImportError`. |
| `LLMRequestError` | no | The base class of the errors of a provider. Its message lists the steps that fix the cause. |
| `MissingAPIKeyError` | no | A provider needs an API key and the environment has none. |
| `APIKeyRejectedError` | no | A provider answers a request with "unauthorized" or "forbidden". |
| `RateLimitedError` | no | A provider limits the rate of the requests. |
| `ProviderUnreachableError` | no | A request does not reach the provider. |

`EBMGraphShapeError`, `EBMGraphTokenLimitError` and `EBMGraphSimplificationError` derive from `EBMGraphError`. The four provider errors derive from `LLMRequestError`. All classes are in `ebaml.core.exceptions`, and `ebaml` exports
them too.

<!-- --8<-- [end:usage] -->
## How it works

| Where | What |
| --- | --- |
| `src/ebaml/core/` | Shared kernel: types and errors. It imports no other part of `ebaml`. |
| `src/ebaml/graphs/` | Graph extraction and simplification (numpy, `interpret`), and the text the LLM reads. |
| `src/ebaml/llm/` | The edge: the calls into BAML (`descriptors.py`), their options and the retry after a cut-off reply. |
| `baml_src/talk_to_ebm.baml` | Prompts, the default LLM clients, the three-turn description chain, the summary, `describe_ebm`, token accounting. |
| `baml_src/model_spec.baml` | The model of a call: `ModelSpec` and one client builder for each provider. |
| `src/ebaml/local/`, `baml_src/local_explanations.baml` | Local explanations: the facts of one prediction computed deterministically, their rendering as JSON texts, and the prompts that put them in words. |
| `baml_src/talk_to_ebm_test.baml`, `baml_src/model_spec_test.baml` | Offline tests on a scripted client. |
| `baml.toml` | Generator config for the Python SDK. |
| `src/ebaml/benchmark.py`, `benchmark/` | The benchmark: loading, answer parsing and scoring, and the data and runner (see [Benchmark](#benchmark)). |
| `notebooks/` | Example notebooks, written with marimo. |
| `docs/`, `zensical.toml` | The documentation, built with Zensical (see [Development](#development)). |
| `tests/unit/` | Python unit tests (100% coverage required). `scripts/live_openrouter.py` is the optional live test. |
| `scripts/` | Maintenance scripts, see [scripts/README.md](scripts/README.md). |
| `.github/` | CI workflow and the scripts that turn failures into reports. |
| `src/baml_sdk/` | Generated. Do not edit. |

<!-- --8<-- [start:behaviour] -->
- **Model:** `openrouter/free` picks a free model for each request, so the model can change between calls and between
  the turns of one conversation. Quality and speed vary with it.
- **Retries:** transient failures (rate limits, network errors, 5xx) are retried up to 3 times. Requests the provider
  rejects are not retried.
- **Concurrency:** `describe_ebm` describes features concurrently, at most `max_concurrency` at a time (default 8).
  Free models are rate limited, so lower it (`describe_ebm(ebm, max_concurrency=2)`) if you see rate-limit errors.
- **Token budget:** there is no tokenizer before a call, so the check uses a rough estimate (about 3 characters per
  token). Going a little over `max_tokens` only warns. Going more than `overbudget_percent` (default 50) over makes
  `to_text` simplify the graph, or raise if it cannot. The provider's own error is the real limit. The functions in
  `ebaml.llm.descriptors` pass the BAML check. A direct call of `EBMGraph.to_text` checks nothing, unless you pass
  `check_budget`, for example `baml_sdk.check_token_budget`.

<!-- --8<-- [end:behaviour] -->
## Notebook

[`notebooks/titanic.py`](notebooks/titanic.py) is a [marimo](https://marimo.io) notebook that ports TalkToEBM's example.
It trains an EBM on the Titanic dataset (downloaded from OpenML), asks the LLM to describe a graph and the whole model,
and shows the graph and text API (`from_ebm`, `simplify`, `to_text`, `from_text`). The LLM cells run only when you press
their button. They use the default model, which needs `OPENROUTER_API_KEY`. See "Choosing the model" for other models.

```bash
uv run marimo edit notebooks/titanic.py   # interactive
uv run marimo run notebooks/titanic.py    # as an app
```

[`notebooks/local_explanations.py`](notebooks/local_explanations.py) is the place to learn the local explanations: pick a
passenger, see the facts with no API key, move the confidence slider, check the guarantees against random completions,
compare what the LLM reads for a layperson and a data scientist, and (with a key) get the explanations in words.

```bash
uv run marimo edit notebooks/local_explanations.py
```

## Benchmark

The benchmark follows [TalkToEBM's benchmark](https://github.com/interpretml/TalkToEBM/tree/main/benchmark) that checks how well a
model reads EBM graphs: function recognition (easy and hard), the largest jump, monotonicity, reading a value and the
widest confidence interval. It makes real requests, and it asks two LLM turns per question (6 tasks, 480 questions in
all). The default model needs `OPENROUTER_API_KEY`. The flag `--model provider/model` selects another model.

```bash
uv run python benchmark/run.py --task monotonicity --limit 10 --output results.json
```

See [benchmark/README.md](benchmark/README.md) for the tasks, the scoring and what it leaves out.

## Choosing the model

<!-- --8<-- [start:changing-model] -->
The default model is OpenRouter's free router, `openrouter/free`. It needs `OPENROUTER_API_KEY`. The `model` argument
of `describe_graph`, `describe_ebm` and `explain_instance` selects another model for one call. EBaMl sends every LLM
request of that call to that model. You do not change the package.

### Quick start

Pass the shorthand `"provider/model"`:

```python
from ebaml.llm.descriptors import describe_ebm

print(describe_ebm(ebm, model="anthropic/claude-haiku-4-5"))
```

Pass `ModelOptions` to set more:

```python
from ebaml.llm.options import ModelOptions

model = ModelOptions(provider="anthropic", model="claude-haiku-4-5", temperature=0.2, max_output_tokens=8000)
print(describe_ebm(ebm, model=model))
```

EBaMl splits the shorthand at the first `/`. The text `"openrouter/openai/gpt-4o-mini"` selects the provider
`openrouter` and the model `openai/gpt-4o-mini`. EBaMl checks the provider before the first paid request. It raises
`InvalidOptionError` if the provider is not in the table below.

### Providers

The provider is a BAML provider prefix. Set the environment variable of the key in the shell that runs Python.

| Provider | Default key variable | Notes |
| --- | --- | --- |
| `openrouter` | `OPENROUTER_API_KEY` | The model name is the OpenRouter name, for example `openai/gpt-4o-mini`. |
| `openai` | `OPENAI_API_KEY` | |
| `anthropic` | `ANTHROPIC_API_KEY` | |
| `google` | `GOOGLE_API_KEY` | The Gemini API. |
| `ollama` | None | A local server. `base_url` defaults to `http://localhost:11434/v1`. |
| `openai-generic` | None | Any OpenAI-compatible server. `base_url` is required. Set `api_key_env` if the server needs a key. |
| `azure` | `AZURE_OPENAI_API_KEY` | The model name is the name of the deployment. Set `base_url` to the deployment, or set `AZURE_OPENAI_ENDPOINT`. |
| `vertex` | None | Google Cloud credentials (`GOOGLE_APPLICATION_CREDENTIALS` or `gcloud auth`). The project comes from `GOOGLE_CLOUD_PROJECT`. |
| `bedrock` | None | The AWS credentials of the machine. The region comes from `AWS_REGION`. `api_key_env` has no effect. |

Run `baml describe <client>.new`, for example `baml describe aws.BedrockClient.new`, to see all settings of a client.

### Options

| Field | Meaning | Default |
| --- | --- | --- |
| `provider` | The BAML provider prefix from the table. | Required. |
| `model` | The name of the model at the provider. | Required. |
| `api_key_env` | The name of the environment variable that holds the key. | The variable of the provider. |
| `base_url` | The address of the server. | The address of the provider. |
| `temperature` | The sampling temperature. | 0.7 (0.0 in the benchmark). |
| `max_output_tokens` | The most tokens of one reply. | See below. |
| `headers` | Headers to add to each request. | None. |

A field that you set wins over the default of the call. The output limit has two defaults:

| Calls | `max_output_tokens` |
| --- | --- |
| Turn 1 of a graph description, the draft of the summary, and the two turns of the benchmark | 8000 |
| All other calls, including the local explanations | 6000 |

### Examples

```python
# OpenRouter, a pinned model (the free router picks a new model for each request)
describe_ebm(ebm, model="openrouter/openai/gpt-4o-mini")

# OpenAI
describe_ebm(ebm, model="openai/gpt-4o-mini")

# Google Gemini
describe_ebm(ebm, model="google/gemini-2.5-flash")

# A local model with Ollama. It needs no key.
describe_ebm(ebm, model="ollama/llama3.1")

# A company gateway with an OpenAI-compatible API
gateway = ModelOptions(
    provider="openai-generic",
    model="my-model",
    base_url="https://llm.example.com/v1",
    api_key_env="MY_GATEWAY_KEY",
    headers={"x-team": "ebm"},
)
describe_ebm(ebm, model=gateway)
```

The benchmark has the flag `--model`:

```bash
uv run python benchmark/run.py --model anthropic/claude-haiku-4-5 --limit 10
```

### Keys

Put the key in an environment variable, in the shell or with `os.environ`. `ModelOptions` holds the name of the
variable, never the key, so the key does not appear in a `repr`, a log line or a traceback. EBaMl reads the variable
only when it sends a request.

### Limits

- One call uses one model. The turns of a call cannot use different models.
- Python cannot set up fallback chains or round robin. A BAML client object cannot cross from Python to BAML, so
  Python sends plain data and BAML builds the client. A BAML client declaration can use `ai.clients.Fallback`: see
  "Change the default model" in `CONTRIBUTING.md`.
- A reasoning model counts its hidden tokens in `max_output_tokens`. Keep the limit large, or the reply can end before
  the answer.
- Quality, speed and cost depend on the model. Small models can ignore the number of sentences that you ask for.

Every client has the retry policy of the default clients: EBaMl retries a transient failure up to 3 times.
<!-- --8<-- [end:changing-model] -->

## Development

<!-- --8<-- [start:development] -->

```bash
baml fmt          # format .baml files
baml check        # compile check
baml test -x root::Live   # run the offline BAML tests
baml generate     # regenerate src/baml_sdk/
uv run pytest     # Python unit tests, fails under 100% coverage
uv run pytest README.md docs src --no-cov   # the code examples of the docs and the docstrings
prek run --all-files                       # every hook: formatters, ruff, vulture, pyrefly, baml fmt, ...
uv run python scripts/mutation.py          # mutation testing of all modules, fails if any mutant survives
uv run python scripts/mutation.py --base origin/main   # the same, but only for the lines changed since main
uv run python scripts/mutation.py --module src/ebaml/local/facts.py   # the same, but only for one module
uv run python -m scripts.check_ignores     # every lint/type/test workaround needs a reason written next to it
uv run python -m scripts.compile_ignores   # all workarounds and their reasons as one Markdown report to audit
uv run --group docs python -m scripts.build_docs         # build the docs; fails on a broken link or a docstring warning
uv run --group docs python -m scripts.build_docs serve   # preview the docs (exports the notebooks, so the first run is slow)
```

CI (`.github/workflows/ci.yml`) runs prek, secret and security scans (trufflehog, semgrep, trivy), `baml check` and
`baml test`, pytest with 100% coverage, the code examples of the documentation, and, in pull requests, mutation testing of the changed lines with a 100% score. A weekly run tests all modules, one job for each module. Lint and type workarounds
(`# noqa`, `# type: ignore`, skipped tests, ...) must carry a written reason next to them.

The `python` code blocks of `README.md` and `docs/*.md` run in order, and the blocks of one file share their variables.
The `Examples` sections of the docstrings run as doctests. `tests/docs/conftest.py` gives each Markdown file a small
data set as `X` and `y`, and it replaces the LLM calls with fakes, so the examples send no request. Put setup that the
reader does not need in an `<!-- invisible-code-block: python -->` comment. To skip a block, put
`<!-- skip: next -->` above it, and write the reason in the text.
<!-- --8<-- [end:development] -->
