Metadata-Version: 2.4
Name: vlm-prior-probe
Version: 0.1.0
Summary: Black-box behavioral probes for measuring visual-evidence reliance and learned-prior persistence in Vision-Language Models.
Author-email: "Eduardo J. Barrios" <edujbarrios@outlook.com>
License-Expression: MPL-2.0
Project-URL: Homepage, https://github.com/edujbarrios/vlm-prior-probe
Project-URL: Source, https://github.com/edujbarrios/vlm-prior-probe
Project-URL: Issues, https://github.com/edujbarrios/vlm-prior-probe/issues
Keywords: vlm,vision-language-model,multimodal,evaluation,robustness,counterfactual,bias,interpretability,testing,vision,llm,ai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# VLM Prior Probe

[![PyPI version](https://img.shields.io/pypi/v/vlm-prior-probe.svg?label=PyPI&logo=pypi&cacheSeconds=300)](https://pypi.org/project/vlm-prior-probe/)
[![License: MPL 2.0](https://img.shields.io/badge/license-MPL%202.0-blue.svg)](https://github.com/edujbarrios/vlm-prior-probe/blob/main/LICENSE)

Measure when a vision-language model follows visual evidence versus learned priors.

VLM Prior Probe is a lightweight, provider-independent toolkit for black-box counterfactual evaluation of vision-language models.

**No model training. No model-specific integration. No hidden-state access. No LLM judge required.**

## Installation

```bash
pip install vlm-prior-probe
```

## Quick start

```python
from vlm_prior_probe import evaluate

samples = [{
    "id": "banana-001", "image": "blue_banana.png",
    "prompt": "What color is the banana?", "attribute": "color",
    "visual_value": "blue", "prior_value": "yellow",
}]

def target(image, prompt):
    return my_vlm(image=image, prompt=prompt)

report = evaluate(samples, target=target)
print(report["visual_adoption_rate"])
print(report["prior_persistence_rate"])
```

## Why VLM Prior Probe?

A plausible VLM answer may follow a canonical association instead of conflicting visual evidence. This package turns controlled interventions into repeatable behavioral measurements without a server, database, provider SDK, or runtime dependency.

## Core idea

Give the model a controlled disagreement—such as a blue banana against the canonical association *banana → yellow*—and classify its answer as `visual`, `prior`, `mixed`, `other`, or `unclear`. Unmatched answers are conservatively `unclear`; `other` requires sample-supplied `other_values`.

## Sample format

Required keys are `id`, `prompt`, `image`, `attribute`, `visual_value`, and `prior_value`. Optional `visual_aliases`, `prior_aliases`, `other_values`, `category`, `original_image`, `counterfactual_image`, `blank_image`, `unrelated_image`, and `prompts` add controlled vocabulary or interventions.

## Single-sample probing

```python
from vlm_prior_probe import probe

result = probe(samples[0], target=target)
print(result["executed_conditions"])
print(result["classification"])
```

The visual and text-only conditions run by default. Optional images run when present. Disable image-less execution with `text_only=False`. Target failures are recorded; use `on_error="raise"` for fail-fast behavior.

## Dataset evaluation

`evaluate(samples, target=target)` returns JSON-serializable results, overall rates, and breakdowns by `attribute` and optional `category`. A valid sample has a successful, non-empty string response in its primary visual condition.

## Prior Persistence Rate

`prior_persistence_rate` is the fraction of valid controlled cases whose primary response matches the canonical prior. It measures prior-consistent behavior, not its internal cause.

## Visual Adoption Rate

`visual_adoption_rate` is the fraction of valid controlled cases whose primary response matches the actual visual value. `visual_prior_balance` is visual adoption minus prior persistence, a descriptive—not universal—summary from -1 to +1.

## Counterfactual Sensitivity

For samples with `original_image` and `counterfactual_image`, sensitivity is true when the original response matches `prior_value` and the changed image response matches `visual_value`. The rate uses paired cases in which both calls are valid. Visual reliance similarly counts valid pairs where the primary condition is visual and text-only is prior.

## Model comparison

```python
from vlm_prior_probe import compare

comparison = compare(samples, targets={"base": base_target, "post-trained": post_target})
```

Differences are reported as right minus left. Comparing checkpoints can reveal whether prior persistence changed after a training stage; it does not attribute the behavior's origin.

## Confidence intervals

Use `evaluate(..., bootstrap=1000, seed=42)` for deterministic 95% percentile bootstrap intervals. Resampling uses `random.Random(seed)` and the standard library only. `compare` also returns paired bootstrap intervals for visual-adoption and prior-persistence differences.

## Supported attributes

The deterministic method is intended for controlled values such as color, count, shape, orientation, size, material, state, and spatial relation. Supply experiment-specific aliases; there is intentionally no large synonym database.

## Use cases

VLM evaluation, model or checkpoint comparison, robustness and regression testing, benchmark auditing, modality-reliance analysis, and controlled counterfactual research.

## Methodology

Matching applies Unicode, case, punctuation, and whitespace normalization followed by whole-phrase matching. Responses containing both controlled values are `mixed`; neither value is never forced into visual or prior. Library-side results are deterministic for fixed inputs, outputs, configuration, and seed, though provider calls may not be.

## Limitations

> VLM Prior Probe measures observable behavior under controlled interventions. It does not identify the internal mechanism responsible for a response and does not by itself determine whether a behavior originates from pretraining, fine-tuning, alignment, or another training stage.

A prior-consistent answer does not automatically prove that the image was ignored. Results depend on stimulus quality, prompts, aliases, target nondeterminism, and whether controlled values adequately represent the task. The package does not generate images, judge arbitrary semantics, or perform mechanistic interpretability.

## Issues

Report issues in the [GitHub issue tracker](https://github.com/edujbarrios/vlm-prior-probe/issues).

## Author

Eduardo J. Barrios — [edujbarrios@outlook.com](mailto:edujbarrios@outlook.com)

## License

[Mozilla Public License 2.0](LICENSE)
