Metadata-Version: 2.4
Name: dd-embed
Version: 0.1.0
Summary: Shared embedding model abstraction layer for Digital Duck projects
Author-email: Digital Duck <p2p2learn@outlook.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/digital-duck/dd-embed
Project-URL: Repository, https://github.com/digital-duck/dd-embed
Project-URL: Issues, https://github.com/digital-duck/dd-embed/issues
Keywords: embedding,digital-duck,ai,sentence-transformers,nlp
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24.0
Provides-Extra: sentence-transformers
Requires-Dist: sentence-transformers>=2.0.0; extra == "sentence-transformers"
Provides-Extra: transformers
Requires-Dist: transformers>=4.30.0; extra == "transformers"
Requires-Dist: torch>=2.0.0; extra == "transformers"
Provides-Extra: openai
Requires-Dist: openai>=1.0.0; extra == "openai"
Provides-Extra: voyageai
Requires-Dist: voyageai>=0.1.0; extra == "voyageai"
Provides-Extra: gemini
Requires-Dist: google-generativeai>=0.3.0; extra == "gemini"
Provides-Extra: all
Requires-Dist: sentence-transformers>=2.0.0; extra == "all"
Requires-Dist: openai>=1.0.0; extra == "all"
Requires-Dist: voyageai>=0.1.0; extra == "all"
Requires-Dist: google-generativeai>=0.3.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# dd-embed

Shared embedding model abstraction layer for Digital Duck projects.

Extracted from semanscope and maniscope. Zero heavy deps in core (only numpy).
Adapters lazy-import their SDKs only when used.

## Install

```bash
pip install dd-embed                          # numpy only
pip install "dd-embed[sentence-transformers]" # + sentence-transformers
pip install "dd-embed[openai]"                # + OpenAI SDK (also covers openrouter)
pip install "dd-embed[voyageai]"              # + Voyage AI SDK
pip install "dd-embed[gemini]"                # + Google GenAI SDK
pip install "dd-embed[all]"                   # all provider SDKs
```

## Quick Start

```python
from dd_embed import embed

# Using sentence-transformers (local, free)
embeddings = embed(["hello", "world"], provider="sentence_transformers",
                   model_name="all-MiniLM-L6-v2")
print(embeddings.shape)  # (2, 384)

# Using OpenAI
embeddings = embed(["hello"], provider="openai", api_key="sk-...")

# Using Ollama (local)
embeddings = embed(["hello"], provider="ollama", model_name="bge-m3")
```

## Built-in Adapters

| Name | Class | SDK | Notes |
|------|-------|-----|-------|
| `sentence_transformers` | `SentenceTransformerAdapter` | `sentence-transformers` | Local, free, used by maniscope |
| `huggingface` | `HuggingFaceAdapter` | `transformers` + `torch` | AutoModel + mean pooling, E5/Qwen support |
| `ollama` | `OllamaEmbedAdapter` | `requests` | Local Ollama server |
| `openai` | `OpenAIEmbedAdapter` | `openai` | OpenAI embeddings API |
| `openrouter` | `OpenAIEmbedAdapter` (configured) | `openai` | OpenAI-compat endpoint |
| `gemini` | `GeminiEmbedAdapter` | `google-generativeai` | Google Gemini embeddings |
| `voyage` | `VoyageEmbedAdapter` | `voyageai` | Voyage AI embeddings |

## Embedding Cache

Disk-persistent, per-word granular cache (ported from semanscope):

```python
from dd_embed import EmbeddingCache, get_adapter

cache = EmbeddingCache()  # default: ~/projects/embedding_cache/dd_embed/master.pkl
adapter = get_adapter("sentence_transformers", model_name="all-MiniLM-L6-v2")

embeddings, cached, computed = cache.get_embeddings(
    texts=["apple", "banana", "cherry"],
    model_name="all-MiniLM-L6-v2",
    scope="en",
    embed_fn=lambda texts: adapter.embed(texts).embeddings,
)
print(f"Cached: {cached}, Computed: {computed}")
cache.save()
```

## Custom Adapters

```python
from dd_embed import EmbeddingAdapter, EmbeddingResult, register_adapter, embed
import numpy as np

class MyAdapter(EmbeddingAdapter):
    def embed(self, texts, **kwargs):
        vecs = np.random.randn(len(texts), 128)  # your logic here
        return EmbeddingResult(
            embeddings=vecs, success=True, provider="my_api",
            model="v1", dimensions=128, num_texts=len(texts),
        )

register_adapter("my_api", MyAdapter)
result = embed(["hello"], provider="my_api")
```

## Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `OPENAI_API_KEY` | OpenAI API key | -- |
| `OPENROUTER_API_KEY` | OpenRouter API key | -- |
| `GEMINI_API_KEY` | Google Gemini API key | -- |
| `VOYAGE_API_KEY` | Voyage AI API key | -- |
| `OLLAMA_HOST` | Ollama server URL | `http://localhost:11434` |

## License

MIT
