Metadata-Version: 2.4
Name: openembedforge
Version: 0.2.0a1
Summary: Forge specialized retrieval models from your own data.
Author: OpenEmbedForge contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: embeddings,machine-learning,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: build<1.4,>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: numpy<2,>=1.23.5; extra == 'dev'
Requires-Dist: pillow>=10; extra == 'dev'
Requires-Dist: pkginfo>=1.12; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: torch>=2.0; extra == 'dev'
Requires-Dist: transformers>=4.40; extra == 'dev'
Requires-Dist: twine<7,>=6; extra == 'dev'
Provides-Extra: onnx
Provides-Extra: train
Requires-Dist: numpy<2,>=1.23.5; extra == 'train'
Requires-Dist: pillow>=10; extra == 'train'
Requires-Dist: torch>=2.0; extra == 'train'
Requires-Dist: transformers>=4.40; extra == 'train'
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/logo/openembedforge-logo.svg" alt="OpenEmbedForge geometric tiger logo" width="650">
</p>

<p align="center">
  <strong>Forge specialized retrieval models from your own data.</strong>
</p>

<p align="center">
  <a href="pyproject.toml"><img alt="Python 3.11+" src="https://img.shields.io/badge/Python-3.11%2B-111827?style=for-the-badge&labelColor=0f766e"></a>
  <a href=".github/workflows/ci.yml"><img alt="CI workflow" src="https://img.shields.io/badge/CI-configured-111827?style=for-the-badge&labelColor=f97316"></a>
  <a href="tests"><img alt="Pytest test suite" src="https://img.shields.io/badge/tests-pytest-111827?style=for-the-badge&labelColor=0f766e"></a>
  <a href="pyproject.toml"><img alt="Ruff linting" src="https://img.shields.io/badge/lint-Ruff-111827?style=for-the-badge&labelColor=f59e0b"></a>
  <a href="LICENSE"><img alt="Apache 2.0 license" src="https://img.shields.io/badge/license-Apache--2.0-111827?style=for-the-badge&labelColor=111827"></a>
  <a href="docs/branding.md"><img alt="Local documentation" src="https://img.shields.io/badge/docs-local-111827?style=for-the-badge&labelColor=0f766e"></a>
  <a href="CONTRIBUTING.md"><img alt="Contributions welcome" src="https://img.shields.io/badge/contributions-welcome-111827?style=for-the-badge&labelColor=0f766e"></a>
</p>

<p align="center">
  <a href="#quick-start"><img alt="Quick Start" src="https://img.shields.io/badge/Quick_Start-111827?style=flat-square&labelColor=f97316"></a>
  <a href="#architecture"><img alt="Architecture" src="https://img.shields.io/badge/Architecture-111827?style=flat-square&labelColor=0f766e"></a>
  <a href="#evaluation"><img alt="Evaluation" src="https://img.shields.io/badge/Evaluation-111827?style=flat-square&labelColor=f59e0b"></a>
  <a href="docs/research/questions.md"><img alt="Research" src="https://img.shields.io/badge/Research-111827?style=flat-square&labelColor=0f766e"></a>
  <a href="docs/ROADMAP.md"><img alt="Roadmap" src="https://img.shields.io/badge/Roadmap-111827?style=flat-square&labelColor=f97316"></a>
  <a href="CONTRIBUTING.md"><img alt="Contributing" src="https://img.shields.io/badge/Contributing-111827?style=flat-square&labelColor=111827"></a>
</p>

**OpenEmbedForge turns domain corpora into auditable retrieval-training datasets and specialized retrievers for RAG.**

Not another vector database. Not another RAG orchestration framework. Not just another embedding-model wrapper.

OpenEmbedForge focuses on:

```text
corpus -> training data -> specialized retrieval model -> evaluation -> deployment
```

<p align="center">
  <img src="docs/assets/openembedforge-pipeline.svg" alt="OpenEmbedForge pipeline from domain data to deployable retriever" width="900">
</p>

## Quick Start

Install the local development package:

```bash
pip install -e ".[dev]"
```

Use the v0.1 Python API:

```python
from openembedforge.corpus import WordChunker, chunk_documents, load_documents
from openembedforge.data import dataset_from_chunks
from openembedforge.mining import RandomNegativeMiner

documents = list(load_documents("./examples/data"))
chunks = chunk_documents(documents, WordChunker(max_words=64, overlap_words=8))
dataset = RandomNegativeMiner(negatives_per_example=1, seed=42).mine(dataset_from_chunks(chunks))

dataset.to_jsonl(".tmp/train.jsonl")
```

Or run the same MVP flow from the CLI:

```bash
openembedforge corpus prepare examples/data --output .tmp/corpus.jsonl
openembedforge dataset make .tmp/corpus.jsonl --output .tmp/train.jsonl
openembedforge train --dataset .tmp/train.jsonl --output .tmp/model
openembedforge evaluate --model .tmp/model --dataset .tmp/train.jsonl
```

For a raw corpus path:

```bash
openembedforge train ./documents --output ./domain-model
```

## Installation

Runtime is currently stdlib-only. Development tooling is optional:

```bash
pip install -e ".[dev]"
```

Future extras are reserved for heavier stacks:

```bash
pip install "openembedforge[train]"
pip install "openembedforge[onnx]"
```

`openembedforge[train]` enables optional Hugging Face Transformers backends, including
CLIP/SigLIP-style multimodal encoders:

```bash
openembedforge train \
  --dataset .tmp/train.jsonl \
  --backend transformer \
  --model-name-or-path sentence-transformers/all-MiniLM-L6-v2 \
  --output .tmp/transformer-model
```

The default test suite builds a tiny local Transformer model, so CI does not download models.

## Architecture

The current MVP includes:

- corpus loading for `.txt`, `.md`, `.json`, and `.jsonl`
- deterministic word-window chunking
- canonical `RetrievalExample` and JSONL datasets
- deterministic easy-negative mining
- a dependency-free dense hashing encoder for local tests and examples
- optional Transformer encoder training with an in-batch InfoNCE objective
- Recall@K, MRR, and nDCG@10
- CLI and Python APIs

Hard-negative mining, distillation, Matryoshka, ONNX, sparse, and late-interaction implementations are future work.

## Multimodal / Vision RAG

OpenEmbedForge now includes early multimodal foundations for image-text retrieval research.
The goal is to test whether adapting a retriever to a specialised domain corpus improves
evidence retrieval over generic embeddings, without making hosted APIs, CUDA, or vector
databases mandatory.

```mermaid
flowchart LR
  A[Domain image/text corpus] --> B[Training-data generation]
  B --> C[Multimodal contrastive training]
  C --> D[Specialised retriever]
  D --> E[Dense / multivector retrieval]
  E --> F[Context selection]
  F --> G[Vision-language model]
  G --> H[Grounded answer]
```

The current stable layer provides:

- `MultimodalCorpus.from_csv(...)` for paired image-text CSV files
- `MultimodalCorpus.from_huggingface(...)` for Dataset-like image-text rows
- `MultimodalSample` with metadata preservation
- deterministic train/validation/test splits, including group-aware splits
- in-memory dense retrieval for text->image, image->text, image->image, and text->text
- optional `TransformersMultimodalEncoder` for CLIP/SigLIP-style Hugging Face models
- Recall@K, MRR, mAP, and nDCG retrieval evaluation by direction
- random and metadata-aware negative miners that avoid obvious false negatives
- provider-neutral `MultimodalEncoder` and `VisionLanguageGenerator` protocols
- RAG context selectors: top-k, score threshold, and metadata diversity
- experimental pure-PyTorch MaxSim and patch-region utilities in
  `openembedforge.experimental.multivector`

Example:

```python
from openembedforge.multimodal import (
    DenseMultimodalRetriever,
    HashingMultimodalEncoder,
    MultimodalCorpus,
    TransformersMultimodalEncoder,
)

corpus = MultimodalCorpus.from_csv(
    "metadata.csv",
    image_column="image_path",
    text_column="report",
    group_column="patient_id",
)
retriever = DenseMultimodalRetriever(HashingMultimodalEncoder(), corpus)
results = retriever.search_text("right-sided pleural effusion", top_k=5)
```

CLI:

```bash
openembedforge multimodal inspect metadata.csv --text-column report --group-column patient_id
openembedforge multimodal train metadata.csv \
  --text-column report \
  --base-model hashing \
  --output ./outputs/domain-model
openembedforge multimodal evaluate metadata.csv --text-column report --model hashing --json
openembedforge multimodal benchmark metadata.csv \
  --text-column report \
  --baseline google/siglip-base-patch16-224 \
  --model ./outputs/domain-model
openembedforge multimodal search metadata.csv --text-column report --query "right effusion"
```

Research inspiration: the architecture is influenced by CLIP/SigLIP-style dual encoders,
domain-specific vision-language pretraining such as MedCLIP and BiomedCLIP, ColBERT-style
late interaction, ColPali-family visual retrieval, multimodal RAG, and region-level visual
retrieval. OpenEmbedForge does not claim to implement those systems unless a module says so.

Medical imaging is treated as an evaluation use case, not a hard-coded assumption. This
software is a research/development tool and is not a medical device or clinical diagnostic
system.

## Roadmap

v0.2: synthetic query generation, query quality filtering, BM25/dense hard negatives, iterative mining, full provenance.

v0.3: teacher/student distillation, domain-specialized small encoders, Matryoshka dimensions, ONNX, quantization.

v0.4: continued domain pretraining, sparse retrieval, late interaction, hybrid retrieval, vector-store adapters.

v0.5: AutoRetriever, deployment-aware strategy selection, RAG-aware evaluation, experimental scratch pretraining.

## CLI

```bash
openembedforge --help
openembedforge version
openembedforge corpus prepare ./documents --output ./data/corpus.jsonl
openembedforge dataset make ./data/corpus.jsonl --output ./data/train.jsonl
openembedforge dataset inspect ./data/train.jsonl
openembedforge train --dataset ./data/train.jsonl --output ./models/domain-retriever
openembedforge evaluate --model ./models/domain-retriever --dataset ./data/train.jsonl
```

## Evaluation

Evaluation is independent of training code and reports:

- Recall@1
- Recall@5
- Recall@10
- MRR
- mAP
- nDCG@10

JSON output is available:

```bash
openembedforge evaluate --model .tmp/model --dataset .tmp/train.jsonl --json
```

## Documentation

- [Branding](docs/branding.md)
- [Web corpus training notebook](examples/notebooks/web_corpus_training.ipynb)
- [Roadmap](docs/ROADMAP.md)
- [Research questions](docs/research/questions.md)

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

Apache-2.0.
