Metadata-Version: 2.5
Name: gqr
Version: 2.0.0
Summary: Package for Guarded Query Routing Benchmark (GQR-Bench)
Project-URL: repository, https://github.com/williambrach/gqr
Requires-Python: >=3.12
Requires-Dist: datasets>=3.5.1
Requires-Dist: pandas>=2.2.3
Requires-Dist: scikit-learn>=1.6.1
Description-Content-Type: text/markdown

# GQR-Bench (Guarded Query Routing Benchmark)

A benchmark and evaluation toolkit for developing and testing guarded query routing models for AI systems.

## Installation

```bash
uv add gqr
```

## Versions

| Version | Install | Evaluation suites | Training data |
|---|---|---|---|
| **0.0.5** (ECAI 2025 paper) | `uv add "gqr==0.0.5"` | original: ID test + 7 OOD test sets (`score`) | three ID domains |
| **1.0.0** | `uv add "gqr<2"` | original (`score`, `score_batch`) | `load_train_dataset(version="v1")` (three ID domains, default) and `version="v2"` (+ background class) |
| **2.0.0** | `uv add "gqr>=2.0.0"` | 1.0.0 **plus GQR-unseen**: nine unseen in-domain test sets scored with the same OOD test sets (`score_unseen`, `score_unseen_batch`) | unchanged |

`0.0.5` is the code used for the ECAI 2025 paper and the [leaderboard](https://gqr-bench.github.io/),
which report the original evaluation suite. It is kept for reference only: its finance source was
removed from the Hugging Face Hub, so `score`, `load_train_dataset` and `load_id_test_dataset` now
raise `DatasetNotFoundError`. To reproduce the GQR score, use `>=1.0.0`: it reads the same
instructions from the upstream dataset (see [Data sources](#data-sources)), so the benchmark splits
are unchanged and GQR scores stay comparable across versions.

Results from different versions should state which evaluation suite and which training data they
used.

## Quick Start

```python
import gqr

# Load development dataset for initial experimentation
dev_train_data, dev_eval_data = gqr.load_dev_dataset()

# Load training dataset for model development
train_data, eval_data = gqr.load_train_dataset()

# Load test datasets for final evaluation
domain_test_data = gqr.load_id_test_dataset()  # In-domain test data
ood_test_data = gqr.load_ood_test_dataset()    # Out-of-domain test data

# Score the model on gqr-bench
def scoring_function(text: str) -> int:
    # Scoring function takes text input (str) and returns predicted domain label (int)
    # Implement your classification logic here
    return 0  # Replace with actual domain prediction

# Evaluate model performance
score = gqr.score(scoring_function)
```

## GQR-Bench v2: background class for training

GQR-Bench v1 provides training data for the three in-distribution domains only.
v2 adds a fourth **background** class (label `3`, domain `"ood"`) to the train and
eval splits, so a router can learn rejection directly instead of only through a
confidence threshold. **Test sets are identical in v1 and v2**, so scores stay comparable.

```python
import gqr

train_data, eval_data = gqr.load_train_dataset(version="v2")  # adds a `source` column
background_train, background_eval = gqr.load_background_dataset()  # background rows only
```

The background class is sized like one in-distribution domain and drawn equally from
three general-purpose corpora at pinned revisions, none of which is used by any test
set: wikitext-103 prose (detokenized), dolly-15k instructions, and Yahoo Answers
questions from topics outside law, finance, and healthcare. Candidates are deduplicated on normalized
text (case, punctuation, and whitespace ignored) within and across sources before the
train/eval split, so no question appears in both splits. Each candidate must also pass
two filters:

1. no law, finance, or healthcare keyword;
2. no exact or 8-word-shingle overlap with the ID or OOD test sets.

Selection is content-addressed: each source keeps the eligible passages with the
smallest seeded hash. The build is therefore identical across machines and library
versions, and a fingerprint check warns if an upstream corpus ever changes.

The first call builds the corpus (one streamed pass over each source) and caches it
under `$GQR_CACHE_DIR` (default `~/.cache/gqr`), with per-source filter statistics.

## GQR-unseen: generalization to new in-domain sources

The GQR-Bench ID test set comes from the same three sources as the training data, so ID
accuracy there says little about how routing carries over to queries from other sources.
GQR-unseen adds nine in-domain test sets from sources GQR-Bench does not use, three per
domain, and pairs them with the existing OOD test sets:

| domain | sets |
|---|---|
| finance | `banking77`, `financial_qa_10k`, `reddit_finance` |
| healthcare | `icliniq`, `medquad`, `med_flashcards` |
| law | `legal_reddit`, `legal_qa_v1`, `mmlu_professional_law` |

Every dataset is pinned to a commit revision. Each text must pass a leakage screen (no
exact or 8-word-shingle overlap with GQR-Bench train, eval, ID test, OOD test, or the v2
background class) and is deduplicated across sets. Each set keeps at most 1,000 texts
with the smallest seeded content hash, so the build is identical across machines.

```python
import gqr

unseen_id = gqr.load_unseen_id_test_dataset()  # text, label (0/1/2), domain, dataset

scores = gqr.score_unseen_batch(batch_model_fn)  # or gqr.score_unseen(model_fn)
# {"unseen_id_accuracy", "ood_accuracy", "gqr_unseen_score", ...per-dataset breakdowns}
```

The **GQR-unseen score** is the harmonic mean of unseen ID accuracy (macro-averaged over
the nine sets, so every source weighs the same) and OOD accuracy on the existing GQR-Bench
OOD test sets (as in the GQR score). Report it next to the GQR score: a large gap indicates
weaker generalization to these unseen sources and warrants inspection of the per-dataset
results. A gap alone does not show *why* (writing style, different subtopics, difficulty, or
ambiguous source-derived labels can all contribute). The existing test sets are unchanged.

## Data sources

The original finance source `4DR1455/finance_questions` was removed from the Hugging
Face Hub. It was a re-upload of `DeividasM/financial-instruction-aq22`, and the
instructions GQR-Bench uses are identical, so the loader now reads the upstream
dataset and the benchmark splits are unchanged. `Stanford/web_questions` is loaded
under its current name, `stanfordnlp/web_questions`.

## Domain Labels

The repository provides mappings between numerical labels and domain names:

```python
# Get label mappings
print(gqr.label2domain)  # Maps numerical labels to domain names
print(gqr.domain2label)  # Maps domain names to numerical labels
```

## Score

```python
import gqr

def scoring_function(text: str) -> int:
    # Scoring function takes text input (str) and returns predicted domain label (int)
    # Implement your classification logic here
    return 0  # Replace with actual domain prediction

# Evaluate model performance
score = gqr.score(scoring_function)
```

## Contributing

```
git clone git@github.com:williambrach/gqr.git
```

```
uv venv --python 3.12
```

```
uv sync 
```

```
uv run --with pytest pytest tests
```

## Paper and Citations

If you use GQR-Bench in your research, please cite our paper:

```
@incollection{gqrbench2025,
      title={Guarded Query Routing for Large Language Models}, 
      author={Richard Šléher and William Brach and Tibor Sloboda and Kristián Košťál and Lukas Galke},
      booktitle={ECAI 2025},
      year={2025},
      pages={4129-4136},
      publisher={IOS Press}
}
```
