Metadata-Version: 2.4
Name: zerda
Version: 0.1.0
Summary: Advanced Retrieval-Augmented Generation (RAG) toolkit with first-class Arabic language support.
Author: Yousef Khalil
License: MIT
Project-URL: Homepage, https://zerdaco.vercel.app/
Keywords: rag,retrieval-augmented-generation,llm,nlp,arabic,embeddings,vector-database,langchain-alternative,agent,agents,agentic,retrieval,retrieval-augmented,retrieval-augmented-generation,retrieval-augmented-generation-toolkit,retrieval-augmented-generation-framework
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Natural Language :: Arabic
Classifier: Natural Language :: English
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Requires-Dist: tiktoken>=0.5; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.25; extra == "anthropic"
Provides-Extra: groq
Requires-Dist: groq>=0.4; extra == "groq"
Provides-Extra: mistral
Requires-Dist: mistralai>=0.1; extra == "mistral"
Provides-Extra: gemini
Requires-Dist: google-generativeai>=0.4; extra == "gemini"
Provides-Extra: ollama
Requires-Dist: requests>=2.28; extra == "ollama"
Requires-Dist: aiohttp>=3.9; extra == "ollama"
Provides-Extra: huggingface
Requires-Dist: sentence-transformers>=2.2; extra == "huggingface"
Requires-Dist: transformers>=4.30; extra == "huggingface"
Requires-Dist: torch>=2.0; extra == "huggingface"
Provides-Extra: arabic
Requires-Dist: sentence-transformers>=2.2; extra == "arabic"
Requires-Dist: transformers>=4.30; extra == "arabic"
Requires-Dist: torch>=2.0; extra == "arabic"
Requires-Dist: stanza>=1.7; extra == "arabic"
Provides-Extra: chroma
Requires-Dist: chromadb>=0.4; extra == "chroma"
Provides-Extra: faiss
Requires-Dist: faiss-cpu>=1.7; extra == "faiss"
Provides-Extra: pinecone
Requires-Dist: pinecone>=3.0; extra == "pinecone"
Provides-Extra: redis
Requires-Dist: redis>=5.0; extra == "redis"
Provides-Extra: pdf
Requires-Dist: pypdf>=3.0; extra == "pdf"
Requires-Dist: PyPDF2>=3.0; extra == "pdf"
Requires-Dist: pdfplumber>=0.10; extra == "pdf"
Requires-Dist: PyMuPDF>=1.23; extra == "pdf"
Provides-Extra: docx
Requires-Dist: python-docx>=1.0; extra == "docx"
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1; extra == "excel"
Requires-Dist: pandas>=2.0; extra == "excel"
Provides-Extra: html
Requires-Dist: beautifulsoup4>=4.12; extra == "html"
Provides-Extra: tables
Requires-Dist: camelot-py[cv]>=0.11; extra == "tables"
Provides-Extra: legacy-docs
Requires-Dist: textract>=1.6.5; extra == "legacy-docs"
Requires-Dist: chardet>=5.0; extra == "legacy-docs"
Provides-Extra: json-query
Requires-Dist: jq>=1.6; extra == "json-query"
Provides-Extra: documents
Requires-Dist: pypdf>=3.0; extra == "documents"
Requires-Dist: PyPDF2>=3.0; extra == "documents"
Requires-Dist: pdfplumber>=0.10; extra == "documents"
Requires-Dist: PyMuPDF>=1.23; extra == "documents"
Requires-Dist: python-docx>=1.0; extra == "documents"
Requires-Dist: openpyxl>=3.1; extra == "documents"
Requires-Dist: pandas>=2.0; extra == "documents"
Requires-Dist: beautifulsoup4>=4.12; extra == "documents"
Requires-Dist: chardet>=5.0; extra == "documents"
Requires-Dist: jq>=1.6; extra == "documents"
Provides-Extra: output
Requires-Dist: pydantic>=2.0; extra == "output"
Requires-Dist: PyYAML>=6.0; extra == "output"
Provides-Extra: observability
Requires-Dist: langsmith>=0.1; extra == "observability"
Provides-Extra: security
Requires-Dist: cryptography>=42.0; extra == "security"
Provides-Extra: dashboard
Requires-Dist: flask>=3.0; extra == "dashboard"
Provides-Extra: hot-reload
Requires-Dist: watchdog>=4.0; extra == "hot-reload"
Provides-Extra: progress
Requires-Dist: tqdm>=4.66; extra == "progress"
Provides-Extra: all
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: tiktoken>=0.5; extra == "all"
Requires-Dist: anthropic>=0.25; extra == "all"
Requires-Dist: groq>=0.4; extra == "all"
Requires-Dist: mistralai>=0.1; extra == "all"
Requires-Dist: google-generativeai>=0.4; extra == "all"
Requires-Dist: requests>=2.28; extra == "all"
Requires-Dist: aiohttp>=3.9; extra == "all"
Requires-Dist: sentence-transformers>=2.2; extra == "all"
Requires-Dist: transformers>=4.30; extra == "all"
Requires-Dist: torch>=2.0; extra == "all"
Requires-Dist: stanza>=1.7; extra == "all"
Requires-Dist: chromadb>=0.4; extra == "all"
Requires-Dist: faiss-cpu>=1.7; extra == "all"
Requires-Dist: pinecone>=3.0; extra == "all"
Requires-Dist: redis>=5.0; extra == "all"
Requires-Dist: pypdf>=3.0; extra == "all"
Requires-Dist: PyPDF2>=3.0; extra == "all"
Requires-Dist: pdfplumber>=0.10; extra == "all"
Requires-Dist: PyMuPDF>=1.23; extra == "all"
Requires-Dist: python-docx>=1.0; extra == "all"
Requires-Dist: openpyxl>=3.1; extra == "all"
Requires-Dist: pandas>=2.0; extra == "all"
Requires-Dist: beautifulsoup4>=4.12; extra == "all"
Requires-Dist: chardet>=5.0; extra == "all"
Requires-Dist: jq>=1.6; extra == "all"
Requires-Dist: pydantic>=2.0; extra == "all"
Requires-Dist: PyYAML>=6.0; extra == "all"
Requires-Dist: langsmith>=0.1; extra == "all"
Requires-Dist: cryptography>=42.0; extra == "all"
Requires-Dist: flask>=3.0; extra == "all"
Requires-Dist: watchdog>=4.0; extra == "all"
Requires-Dist: tqdm>=4.66; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pyflakes>=3.2; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

# zerda

**Advanced Retrieval-Augmented Generation (RAG) toolkit for Python, with first-class Arabic language support.**

`zerda` is a modular collection of building blocks for RAG applications —
embeddings, vector stores, document loaders, chunking, caching, prompt
construction, output parsing, routing, memory, and several ready-made RAG
system architectures (agentic, graph, hybrid, multi-hop, conversational,
federated, self-improving, streaming, domain-specific).

Every third-party integration (OpenAI, Anthropic, Groq, Mistral, Gemini,
Chroma, FAISS, Pinecone, Redis, ...) is **optional**. Installing `zerda`
itself only pulls in `numpy`; you install the extras you actually need.

> **Status:** Alpha. APIs may change between minor versions until `1.0`.

---

## Features

- **Multiple RAG architectures** out of the box: `RAGSystem` (core), plus
  `AgenticRAG`, `GraphRAG`, `HybridSearchRAG`, `MultiHopRAG`,
  `ConversationalRAG`, `MultiDocumentRAGSystem`, `FederatedRAG`,
  `SelfIRAG` / `RecursiveRAG` / `HyDERAG`, `StreamingRAG`, and
  `DomainSpecificRAG`.
- **Embeddings**: OpenAI, Gemini, Mistral, HuggingFace/local models, Ollama,
  and an Arabic-specialized embedder with text normalization.
- **LLM clients**: OpenAI, Anthropic, Groq, Mistral, Gemini, Ollama —
  sync/async, streaming, and tool-calling support.
- **Vector databases**: Chroma, FAISS, Pinecone behind a common interface.
- **Document loaders**: PDF, DOCX, HTML, CSV/Excel, JSON, plain text.
- **Chunking**: multiple strategies plus an Arabic-aware preprocessor.
- **Context engine**: query analysis, hybrid (vector + BM25) retrieval
  fusion, composite ranking, budget-aware context composition.
- **Multi-level semantic cache**: exact + semantic + persistent storage
  (memory, SQLite, Redis) with tenant isolation and a security guard layer
  (prompt-injection detection, PII scrubbing, HMAC integrity checks).
- **Memory**: short/long-term, episodic, semantic, and procedural
  conversation memory.
- **Prompt engine, structured output parsing, request router, plugin
  system, and tracing/observability exporters** (console, JSON, LangSmith).

## Installation

```bash
pip install zerda
```

This installs the core package (`numpy` only). Add extras for the pieces
you actually use:

```bash
# LLM providers
pip install "zerda[openai]"
pip install "zerda[anthropic]"
pip install "zerda[groq]"
pip install "zerda[mistral]"
pip install "zerda[gemini]"
pip install "zerda[ollama]"

# Local / HuggingFace embedding models (incl. the Arabic embedder)
pip install "zerda[huggingface]"
pip install "zerda[arabic]"

# Vector databases
pip install "zerda[chroma]"
pip install "zerda[faiss]"
pip install "zerda[pinecone]"

# Cache storage
pip install "zerda[redis]"

# Document loaders
pip install "zerda[documents]"   # pdf + docx + excel + html + json
pip install "zerda[pdf]"
pip install "zerda[tables]"      # camelot (needs system Ghostscript/OpenCV)
pip install "zerda[legacy-docs]" # textract for legacy formats

# Misc
pip install "zerda[output]"        # pydantic + PyYAML for output parsing
pip install "zerda[observability]" # LangSmith exporter
pip install "zerda[security]"      # cryptography
pip install "zerda[dashboard]"     # Flask memory dashboard
pip install "zerda[hot-reload]"    # watchdog for plugin hot-reload

# Everything
pip install "zerda[all]"

# Multiple extras at once
pip install "zerda[openai,faiss,documents]"
```

Every `zerda.*` sub-package can always be imported even without its
extras installed — only the specific classes that need a missing library
will be unavailable until you install the matching extra.

## Quickstart

```python
from zerda.llm import OpenAIInterface
from zerda.embeddings import OpenAIEmbedder
from zerda.vector_database import ChromaVectorDB   # name depends on the backend module
from zerda.chunks import TextSplitter               # example chunker
from zerda.context import ContextManager            # example context builder
from zerda.rag import RAGSystem, RAGConfig

llm = OpenAIInterface(api_key="sk-...")
embedder = OpenAIEmbedder(api_key="sk-...")
vector_db = ChromaVectorDB(embedder=embedder, collection_name="docs")
chunker = TextSplitter()
context_manager = ContextManager()

rag = RAGSystem(
    vector_db=vector_db,
    llm=llm,
    chunker=chunker,
    context_manager=context_manager,
    config=RAGConfig(),
)

answer = rag.query("ما هي فوائد الطاقة الشمسية؟")
print(answer)
```

`RAGSystem` is intentionally composable: it accepts any objects that
implement the small `vector_db` / `llm` / `chunker` / `context_manager`
interfaces described in its docstring, so you can mix and match `zerda`'s
own implementations with your own. For a fully worked example with a
specific vector store and embedder, check the docstrings in
`zerda.vector_database` and `zerda.embeddings`.

### Using a specialized RAG type

```python
from zerda.rag.types import GraphRAG, ConfigGraphRAG

graph_rag = GraphRAG(llm=llm, embedder=embedder, config=ConfigGraphRAG())
graph_rag.add_document("...")
result = graph_rag.query("How are these two entities related?")
```

## Module map

| Module                    | Purpose                                             |
| -------------------------- | ---------------------------------------------------- |
| `zerda.rag`                | Core `RAGSystem` + all specialized RAG types (`zerda.rag.types`) |
| `zerda.embeddings`         | Embedding provider clients                          |
| `zerda.llm`                | LLM chat-completion clients                          |
| `zerda.vector_database`    | Chroma / FAISS / Pinecone backends                   |
| `zerda.document_loaders`   | PDF / DOCX / HTML / CSV / JSON / text loaders        |
| `zerda.chunks`             | Text chunking & Arabic preprocessing                 |
| `zerda.context`            | Query analysis, retrieval fusion, context building   |
| `zerda.cache`               | Multi-level semantic cache with security guard       |
| `zerda.memory`              | Conversation & long-term memory                      |
| `zerda.prompt`              | Prompt templating & strategies                       |
| `zerda.output_parser`       | Structured output parsing & validation               |
| `zerda.router`              | Intent routing & handler dispatch                    |
| `zerda.observability`       | Tracing & metrics exporters                          |
| `zerda.plugins`             | Plugin system                                        |

## Contributing

Issues and pull requests are welcome. Please open an issue describing the
change before submitting a large PR.

## License

MIT — see [LICENSE](LICENSE).
