Metadata-Version: 2.4
Name: import-kb
Version: 0.2.4
Summary: A package for importing knowledge and embedding text.
Author-email: Coderskin <nebiyu.samuel@singularitynet.io>
License: MIT
Keywords: knowledge,embedding,openai,chromadb
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: openai>=1.0.0
Requires-Dist: chromadb>=0.3.21
Requires-Dist: python-dotenv>=0.20.0
Requires-Dist: tqdm>=4.64.0
Requires-Dist: sentence-transformers>=2.2.0
Requires-Dist: torch>=2.5.1
Provides-Extra: dev
Requires-Dist: pytest>=6.2.5; extra == "dev"
Dynamic: license-file

# Import Knowledge and Memory Portability (`import-kb`)

A utility package for importing distilled knowledge into a ChromaDB-based Long-Term Memory (LTM) system and backing up or restoring Omega user memory.

## Purpose
The `import-kb` package is designed to bridge the gap between static knowledge files (JSONL, MeTTa) and an active agent's memory. It processes structured knowledge, generates vector embeddings, and upserts them into a ChromaDB collection, enabling semantic search and retrieval for AI agents.

The package also provides `memory_portability`, a programmatic interface for exporting and restoring Omega conversation history and user LTM records. Omega Core remains responsible for its CLI, container lifecycle, transfer-directory mount, and user-facing decisions.

## Supported Embedding Models
This package supports the following embedding modes:

- **OpenAI (Cloud)**:
  - Default model: `text-embedding-3-large`
  - High accuracy but requires an internet connection and an API key.
- **ASI Cloud**:
  - Default model: `WhereIsAI/UAE-Large-V1`
  - Uses an OpenAI-compatible endpoint hosted by ASI Cloud, requires an API key.
- **SentenceTransformers (Local)**:
  - Default model: `intfloat/e5-large-v2`
  - Runs fully offline on your local machine.
  - Can be configured to use any model compatible with the `sentence-transformers` library (e.g., `all-MiniLM-L6-v2`).

## Installation

You can install the package directly from PyPI:

```bash
pip install import-kb
```

Or install it locally in editable mode:

```bash
git clone <repository-url>
cd import-knowledge-package
pip install -e .
```

## Setup
Create a `.env` file in your project root or set the following environment variables:

- `OPENAI_API_KEY`: Required if using OpenAI embeddings.
- `ASI_API_KEY`: Required if using ASI Cloud embeddings (`--provider asicloud`).
- `CHROMA_DB_PATH`: (Optional) Custom path to your Chroma database. Defaults to looking for `/PeTTa/chroma_db` or a local `chroma_db` folder.

## How to Run

### Command Line Interface (CLI)
After installation, you can run the import via the provided entry point:

```bash
# Use OpenAI embeddings (default)
import-knowledge

# Same as above, made explicit
import-knowledge --provider openai

# Override OpenAI model
import-knowledge --provider openai --model "text-embedding-3-small"

# Use ASI Cloud embeddings
import-knowledge --provider asicloud

# Override the ASI Cloud model
import-knowledge --provider asicloud --model "<model-name>"

# Use Local embeddings
import-knowledge --local

# Use a specific local model
import-knowledge --local --model "all-MiniLM-L6-v2"
```

Alternatively, run it as a module:
```bash
python3 -m import_knowledge.import_knowledge --local
```

### Programmatic Usage
You can initialize the embedding system and trigger the import programmatically from your Python scripts:

```python
from import_knowledge import initLocalEmbedding, main

# Initialize for local use
initLocalEmbedding(model_name="intfloat/e5-large-v2")

# Run the import process
main()
```

### Memory portability

The host application owns memory configuration and must pass the resolved paths
to the package explicitly:

```python
from pathlib import Path

from memory_portability import MemoryStore, MemoryTransfer

store = MemoryStore(
    memory_dir=Path("/path/to/omega/memory"),
    chroma_path=Path("/path/to/chroma_db"),
    collection_name="memories",
)
transfer = MemoryTransfer(
    transfer_dir=Path("/path/to/memory-transfer"),
    store=store,
)

transfer.export(component="both")
transfer.import_archive("omega-memory-<timestamp>.tar.gz")
transfer.recover()
```

`MemoryStore` does not infer Omega paths or read them from environment
variables. Resolve these values in the host application's configuration layer.

## Dependencies
- `openai`: For cloud-based embeddings.
- `sentence-transformers`: For local, offline embeddings.
- `chromadb`: Vector database for storage.
- `python-dotenv`: Management of environment variables.
- `tqdm`: Progress bars for batch processing.

---

## License

This project is licensed under the Apache License 2.0. See [LICENSE](LICENSE) file for details.
