Metadata-Version: 2.5
Name: embed-parity
Version: 0.2.0
Summary: Compare SentenceTransformers embeddings with a TEI deployment
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: httpx>=0.25
Requires-Dist: numpy>=1.24
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.9; extra == 'dev'
Provides-Extra: sentence-transformers
Requires-Dist: sentence-transformers>=2.6; extra == 'sentence-transformers'
Provides-Extra: test
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# embed-parity

`embed-parity` answers one deliberately narrow question: does a Hugging Face Text
Embeddings Inference (TEI) deployment reproduce the embedding behavior of the
same model through SentenceTransformers?

It checks vector direction and norms, pairwise geometry, nearest neighbors,
batch consistency, model-aware query/document prefixes, and behavior at controlled token
lengths. When a length failure is found after shorter inputs pass, it performs a
binary search for the shortest failing token prefix.

## Installation

Python 3.10 or newer is required.

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e '.[sentence-transformers,test]'
```

## Usage

Start TEI with the same model used by the reference. For example, using the
official TEI container (choose an image tag appropriate to your environment):

```bash
docker run --gpus all -p 8080:80 \
  -v "$PWD/data:/data" \
  ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 \
  --model-id BAAI/bge-small-en-v1.5
```

Then compare the runtimes:

```bash
embed-parity compare \
  --model BAAI/bge-small-en-v1.5 \
  --revision MODEL_COMMIT_SHA \
  --tei http://localhost:8080
```

Write a JSON report and select batch sizes with:

```bash
embed-parity compare \
  --model BAAI/bge-small-en-v1.5 \
  --tei http://localhost:8080 \
  --batch-sizes 1,8,32 \
  --json report.json
```

The human report prints every default threshold. They are operational defaults,
not universal scientific constants, and every threshold has a CLI flag. Run
`embed-parity compare --help` for the full list. Use `--revision` to pin the
SentenceTransformers side and `--normalize` or `--no-normalize` to make the
reference normalization choice explicit.

To reproduce a failure with your own corpus, provide JSONL objects with a required
`text` and optional `id` and `category`:

```jsonl
{"id":"query-1","category":"query","text":"best waterproof boots"}
{"id":"doc-1","category":"document","text":"These boots use a sealed membrane."}
```

```bash
embed-parity compare --model org/model --tei http://localhost:8080 \
  --probe-file probes.jsonl --lengths 64,128,256,257,512
```

Use `--skip-length-analysis` for a fast corpus-only check. Reports include the
selected batch sizes and lengths plus a SHA-256 fingerprint of the exact corpus.
If `--json` is set, exit-code-2 execution failures also produce a structured JSON
error report.

### TEI transport controls

The CLI waits for `/health` before loading the reference model and retries
transient HTTP failures. Relevant options include:

```bash
embed-parity compare --model org/model --tei https://tei.example.com \
  --tei-api-key "$TEI_API_KEY" \
  --tei-header X-Tenant=search \
  --tei-retries 4 \
  --tei-retry-backoff 0.5 \
  --readiness-timeout 60 \
  --no-tei-truncate
```

Set `EMBED_PARITY_TEI_API_KEY` instead of passing the bearer token on the command
line when possible. `HF_TOKEN` is honored by Hugging Face libraries for private
or gated reference models. Authentication values are never written to reports.

Exit codes are:

- `0`: configured parity checks passed
- `1`: a meaningful parity check failed
- `2`: execution or configuration failed

## What is measured

The built-in deterministic corpus contains exactly 100 probes covering short and
paragraph text, semantic and unrelated pairs, queries and documents, code, SQL,
numbers, punctuation, whitespace, Unicode, accented text, six non-English
languages, and empty or near-empty strings. Long probes are separate: they are
created through the reference tokenizer at 32, 64, 128, 256, 384, 512, 768, and
1024 tokens. No LLM is used to generate any probe.

TEI rejects empty and whitespace-only strings. Those probes remain part of the
built-in corpus but are capability-filtered for TEI comparisons and listed under
`skipped_probes` in the report rather than causing an execution failure.

The comparison has four layers:

1. Structural checks validate counts, dimensions, finite values, zero vectors,
   and norm distributions. A dimension mismatch stops similarity analysis.
2. Vector checks report mean, median, minimum, p01, and p05 cross-runtime cosine,
   plus the ten worst probes.
3. Geometry checks compare the upper triangles of pairwise cosine matrices with
   Pearson, Spearman, mean absolute difference, and maximum difference.
4. Neighbor checks measure top-1 agreement and average top-5/top-10 overlap and
   identify the most changed neighborhoods.

Both runtimes are rerun at every requested batch size. For E5 and English BGE
models, query probes test model-family-recommended prefixes; E5 document probes
also test `passage: `. This is a diagnostic observation only: the tool says a
prefix *may* explain a difference; it does not claim an unobserved server
configuration as fact.

TEI metadata endpoints vary by server version. The adapter tries `/info` and `/`,
records the complete JSON object, and promotes recognized model, revision, dtype,
dimension, and input-length fields. Known model or revision conflicts fail the
comparison. Unknown fields remain unknown rather than being inferred; a known
dtype difference is diagnostic but does not fail otherwise-equivalent vectors.

## Tests and broken fixtures

```bash
pip install -e '.[dev]'
ruff format --check .
ruff check .
mypy embed_parity
pytest -m 'not integration' --cov
```

The deterministic fake backends and mocked HTTP adapter prove detection of all requested classes:
equivalent output, missing normalization, dimensional mismatch, small noise,
large perturbation, a 256-token truncation boundary, batch dependence, hidden
query prefix, NaN output, an incompatible embedding space, malformed/ragged TEI
responses, wrong response counts, metadata conflicts, and JSONL validation.

## Real-model experiment

The repository includes an opt-in three-model integration test for:

- `BAAI/bge-small-en-v1.5`
- `intfloat/e5-small-v2`
- `sentence-transformers/all-MiniLM-L6-v2`

Provide three live endpoints and run:

```bash
TEI_BGE_SMALL_URL=http://localhost:8081 \
TEI_E5_SMALL_URL=http://localhost:8082 \
TEI_MINILM_URL=http://localhost:8083 \
pytest -m integration -vv
```

The three models were also run against TEI 1.9.3 on Apple Metal on August 19,
2026 with identical pinned model commit SHAs on both runtimes. All baseline
configurations passed. A real MiniLM negative control with
TEI pooling deliberately changed from `mean` to `cls` failed and was diagnosed
as a known pooling mismatch. See [the experiment report](reports/README.md) and
the machine-readable JSON files in `reports/`.

The `live TEI parity` GitHub Actions workflow runs pinned MiniLM weekly and on
manual dispatch. Tagging `v0.2.0` builds a GitHub release and publishes to PyPI
through trusted publishing after the repository's `pypi` environment and PyPI
trusted publisher are configured.

## Scope and interpretation

This is not a general backend framework. It contains exactly two production
adapters: SentenceTransformers and TEI. A passing result means the selected probe
suite and thresholds found the runtimes operationally equivalent; it is not a
proof that every possible input will match. Diagnostics intentionally use phrases
such as “possible truncation mismatch” because measurements often narrow the
cause without proving the server's internal configuration.
