Metadata-Version: 2.4
Name: dastavej
Version: 0.1.0
Summary: A unified Python library for text and document processing
Author: Dastavej Contributors
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: pdf
Requires-Dist: pypdf>=5.0; extra == "pdf"
Provides-Extra: docx
Requires-Dist: python-docx>=1.1; extra == "docx"
Provides-Extra: html
Requires-Dist: beautifulsoup4>=4.12; extra == "html"
Provides-Extra: markdown
Requires-Dist: markdown>=3.7; extra == "markdown"
Requires-Dist: beautifulsoup4>=4.12; extra == "markdown"
Provides-Extra: nlp
Requires-Dist: spacy>=3.7; extra == "nlp"
Provides-Extra: sentiment
Requires-Dist: transformers>=4.40; extra == "sentiment"
Provides-Extra: embeddings
Requires-Dist: sentence-transformers>=3.0; extra == "embeddings"
Requires-Dist: numpy>=1.26; extra == "embeddings"
Provides-Extra: vectorstore
Requires-Dist: numpy>=1.26; extra == "vectorstore"
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: ollama
Requires-Dist: ollama>=0.4; extra == "ollama"
Provides-Extra: all
Requires-Dist: pypdf>=5.0; extra == "all"
Requires-Dist: python-docx>=1.1; extra == "all"
Requires-Dist: beautifulsoup4>=4.12; extra == "all"
Requires-Dist: markdown>=3.7; extra == "all"
Requires-Dist: spacy>=3.7; extra == "all"
Requires-Dist: transformers>=4.40; extra == "all"
Requires-Dist: sentence-transformers>=3.0; extra == "all"
Requires-Dist: numpy>=1.26; extra == "all"
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: ollama>=0.4; extra == "all"

# Dastavej

Dastavej is a Python document processing and Retrieval-Augmented Generation (RAG) library.

It provides a complete pipeline for:

- Document ingestion
- Text cleaning
- Text normalization
- Information extraction
- NLP processing
- Document chunking
- Text embeddings
- Vector storage
- Semantic search
- Retrieval
- RAG context construction
- Local LLM generation

---

## Architecture

```text
Document
   │
   ▼
Ingestion
   │
   ▼
TextDocument
   │
   ├── Cleaning
   ├── Normalization
   ├── Extraction
   └── NLP
          │
          ▼
       Chunking
          │
          ▼
     DocumentChunk
          │
          ▼
      Embeddings
          │
          ▼
     Vector Store
          │
          ▼
       Retriever
          │
          ▼
    Context Builder
          │
          ▼
      RAG Pipeline
          │
          ▼
       Local LLM
          │
          ▼
        Answer
