# AIMLite (v2.0.0)
> The Django for AI & Machine Learning — an opinionated, convention-over-configuration Python framework with zero-path CLI execution, parameter-efficient LoRA fine-tuning, enterprise RAG knowledge bases, and multi-model serving.

AIMLite standardizes the entire machine learning lifecycle (Data, Model, Lifecycle) across three core paradigms (Scratch ML, RAG Knowledge Bases, LoRA Fine-Tuning). It eliminates repetitive boilerplate, fragile relative path configurations, and unstandardized project layouts.

---

## 1. Zero-Path CLI Reference

All commands run anywhere inside an AIMLite project directory with zero path arguments.

```bash
# Initialize a new project layout (<50ms, Vite/Django inspired)
aimlite init <project_name | .> [--type <scratch|rag|adapters>] [--clean] [--install] [-y]

# Manage project dependencies into managed .venv directly from aimlite.json
aimlite install [package_name ...]

# Validate dataset schema, file formats, and partition readiness
aimlite data validate

# Train a model class (auto-discovers class if omitted)
aimlite train [ModelName]

# Evaluate trained checkpoints against validation/test splits
aimlite evaluate [ModelName]

# Launch multi-model inference server with adaptive web UI, Swagger docs, and REST API
aimlite serve [ModelName] [--port 8000] [--host 127.0.0.1] [--api] [--frontend <dir>]

# Inspect Python runtime, accelerators, and directory permissions
aimlite doctor

# Execute any script with project root automatically placed on PYTHONPATH
aimlite benchmark [experiments/benchmark.py]
```

### CLI Flags & Options:
- `--clean`: Scaffolds pristine class skeletons with zero sample data or dummy code.
- `--api`: Runs the inference server in headless REST JSON mode (disables web UI, ideal for Docker/Kubernetes).
- `--frontend <dir>`: Serves a custom SPA build (React, Vue, Vite, Next.js static) on the same port with automatic fallback routing to `index.html`.
- `--type <scratch|rag|adapters>`: Selects project paradigm.

---

## 2. Standard Directory Layout

AIMLite enforces a clean, convention-based directory structure:

```
my_project/
├── aimlite.json             # Single source of truth for project metadata & dependencies
├── data/                    # Raw datasets (CSV, JSON, JSONL, Parquet, TXT, MD)
├── models/                  # Serialized weights (models/<ModelName>.pkl)
├── artifacts/               # Non-weight artifacts (indexes, tokenizers, configs)
├── experiments/             # Experiment logs, metrics snapshots, benchmarks
├── templates/               # (Optional) Developer override templates (app.html)
├── data.py                  # Dataset subclass & partitioning logic
├── model.py                 # Model subclass & forward pass
├── trainer.py               # BaseTrainer subclass (training loop)
├── evaluator.py             # BaseEvaluator subclass (metrics calculation)
└── inference.py             # BaseInference subclass (request handling & prediction)
```

---

## 3. The 3 Architectural Pillars

### Pillar 1: Data (`aimlite.data.Dataset`)
Defines dataset loading, validation, and deterministic partitioning.
```python
from aimlite import Dataset

class HousingDataset(Dataset):
    source = "data/housing.csv"       # Optional explicit file anchor
    # combine_all = True              # Concatenates all CSVs in data/
    # filename = "custom.csv"         # Targets specific filename in data/

dataset = HousingDataset()
records = dataset.load()              # Ingests rows as list of dicts

# Deterministic train/validation/test split (ratios must sum to 1.0)
train_data, val_data, test_data = dataset.split(
    train=0.8,
    validation=0.1,
    test=0.1,
    seed=42,                          # Deterministic random shuffle
    shuffle=True
)
```

### Pillar 2: Model (`aimlite.models.Model`)
Framework-agnostic model wrapper with built-in pickle persistence to `models/<name>.pkl`.
```python
from aimlite import Model

class HousePricePredictor(Model):
    def __init__(self, name="HousePricePredictor", **kwargs):
        super().__init__(name=name, **kwargs)
        self.weights = {}

    def predict(self, inputs, **kwargs):
        # inputs can be dict, list of dicts, or feature vector
        return {"price": 250000.0}

model = HousePricePredictor()
model.save()                          # Automatically saves to models/HousePricePredictor.pkl
model.load()                          # Loads from models/HousePricePredictor.pkl
```

### Pillar 3: Lifecycle (`aimlite.lifecycle`)
Separation of concerns across training, evaluation, and inference.
```python
from aimlite import BaseTrainer, BaseEvaluator, BaseInference

class HouseTrainer(BaseTrainer):
    def fit(self, model, dataset, **kwargs):
        train_data, val_data, _ = dataset.split(train=0.8, validation=0.2, test=0.0)
        # Real training loop...
        model.save()
        return {"status": "completed", "final_loss": 0.042}

class HouseEvaluator(BaseEvaluator):
    def evaluate(self, model, dataset, **kwargs):
        _, _, test_data = dataset.split(train=0.8, validation=0.1, test=0.1)
        # Compute real metrics (RMSE, R2, accuracy)...
        return {"rmse": 12400.5, "r2": 0.89}

class HouseInference(BaseInference):
    def run(self, model, raw_input, **kwargs):
        # Validate input and format output
        return model.predict(raw_input)
```

---

## 4. The 3 AI Paradigms

### Paradigm 1: Scratch Training (Classical ML & Deep Learning)
Use for custom Scikit-learn, PyTorch, XGBoost, or pure Python models.
- **Entrypoint**: `aimlite init my_model --type scratch`
- **Serving UI**: Renders an interactive Feature Form with auto-discovered dataset columns, preset buttons, and JSON payload toggle.

### Paradigm 2: Enterprise RAG (`aimlite.rag`)
Production knowledge retrieval with multi-provider LLM synthesis and vector indexing.
- **Entrypoint**: `aimlite init my_rag --type rag`
- **Core Components**:
  - `Document(content, metadata, id)`: Atomic text document container.
  - `SmartChunker(max_chunk_size=600, chunk_overlap=60)`: Hierarchical document chunker decomposing along Markdown headers (`#`, `##`), paragraphs, sentence boundaries, word tokens, and character windows.
  - `MemoryVectorStore(embedding_fn)`: In-memory cosine similarity store.
  - `PostgresVectorStore(connection_string)`: PostgreSQL `pgvector` store with ORM migration support.
  - `TfidfEmbedding()`: Zero-dependency pure Python TF-IDF embedding.
  - `SentenceTransformerEmbedding(model_name)`: Dense semantic embeddings (requires `sentence-transformers`; raises explicit `ImportError` if uninstalled).
  - `BaseChatProvider`: Supported providers:
    - `OpenAIChatProvider(api_key=..., model="gpt-4o")`
    - `AnthropicChatProvider(api_key=..., model="claude-3-5-sonnet-20241022")`
    - `GeminiChatProvider(api_key=..., model="gemini-1.5-flash")`
    - `OllamaChatProvider(model="llama3.2")`
    - `LocalChatProvider(model_or_fn=...)`
    - `MockChatProvider(fixed_response=...)` (for test harnesses only)
  - `KnowledgeModel`: Subclasses `RAGModel`, coordinates retriever, chunker, and provider. **Must be initialized with an explicit `chat_provider` in production.**
  - `RAGTrainer`: Ingests dataset documents, generates chunks, and builds persistent vector index.

```python
from aimlite.rag import (
    KnowledgeModel,
    SmartChunker,
    TfidfEmbedding,
    MemoryVectorStore,
    OpenAIChatProvider
)

chat_provider = OpenAIChatProvider(api_key="sk-...")
model = KnowledgeModel(
    name="SupportDocRAG",
    chat_provider=chat_provider,
    embedding_fn=TfidfEmbedding(),
    chunker=SmartChunker(max_chunk_size=500, chunk_overlap=50)
)
model.add_document("AIMLite uses zero-path execution.", title="Overview")
response = model.predict("How does AIMLite execute?")
# Returns: {"answer": "...", "sources": [{"content": "...", "score": 0.95}]}
```

### Paradigm 3: LoRA & PEFT Fine-Tuning (`aimlite.adapters`)
Parameter-efficient fine-tuning via low-rank matrix decomposition:
$$W = W_0 + \frac{\alpha}{r} (B \cdot A)$$
- **Entrypoint**: `aimlite init my_lora --type adapters`
- **Core Components**:
  - `LoRALayer(in_features, out_features, r=8, lora_alpha=16)`: Low-rank linear layer with forward ($h = W_0 x + \frac{\alpha}{r} B A x$) and analytical backward updates (`backward_B`, `backward_A`).
  - `AdapterModel(base_model, adapter_config)`: Base adapter wrapper with `merge_weights()`, `unmerge_weights()`, and `get_trainable_parameters()`.
  - `AdapterTrainer`: Executes true gradient backpropagation on LoRA matrices $A$ and $B$ with MSE loss and gradient clipping ($[-1.0, 1.0]$). Asserts non-zero parameter updates post-training.
  - `MultiAdapterManager`: Hot-swaps multiple adapter weights on top of a shared frozen base model without reloading.
  - `Hugging Face PEFT compatibility`: Saves standard `adapter_config.json` and `adapter_model.pkl`.

---

## 5. Multi-Model Serving & Endpoints

When `aimlite serve` is executed, it starts a threaded HTTP server (`ThreadingHTTPServer`) with non-blocking concurrent request handling:

| Method | Endpoint | Description |
| :--- | :--- | :--- |
| `GET` | `/` or `/app` | Paradigm-adaptive Web Application UI with live Model Switcher |
| `GET` | `/chat` | Dedicated Conversational Chat Playground |
| `GET` | `/models` | JSON inventory of all registered models and metadata |
| `GET` | `/health` | Server health, active model, and uptime status |
| `GET` | `/docs` | Interactive Swagger UI API documentation |
| `GET` | `/openapi.json` | Full OpenAPI 3.0 specification schema |
| `POST` | `/predict` | Primary inference dispatch (accepts optional `"model": "<name>"`) |
| `POST` | `/models/{name}/predict` | Direct inference target for a specific registered model |
| `POST` | `/models/{name}/search` | Semantic search endpoint for RAG models (retrieval only, no LLM) |

---

## 6. Coding Agent Rules & Guidelines

When writing or generating code for AIMLite:
1. **Never write relative path boilerplate**: Use `data/` for inputs and `models/` for checkpoints. Never hardcode absolute machine paths.
2. **Never mock in production paths**: Always provide real, working implementations. `KnowledgeModel` requires an explicit `chat_provider`.
3. **No silent fallbacks**: When a package like `sentence-transformers` is needed, raise an explicit `ImportError` rather than silently degrading to TF-IDF.
4. **Deterministic data splitting**: Always specify `seed` in `Dataset.split()` for reproducible evaluations. Ratios must sum to 1.0.
5. **No fake training results**: `AdapterTrainer.fit()` and custom trainers must compute real forward and backward passes.
6. **Threaded concurrency**: Custom inference servers should use `ThreadingHTTPServer` to avoid blocking concurrent requests.
7. **Zero TODO / FIXME comments**: Produce production-ready, clean, well-commented code.

---

## 7. Containerization & Production Docker Deployment

Deploying an AIMLite application to production uses this standardized `Dockerfile`:

```dockerfile
# syntax=docker/dockerfile:1
FROM python:3.12-slim

# Set environment configuration
ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=1 \
    PORT=8000 \
    HOST=0.0.0.0 \
    PATH="/app/.venv/bin:$PATH"

# Install system dependencies (curl for healthchecks & network tooling)
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    && rm -rf /var/lib/apt/lists/*

# 1. Install uv (blazing fast backend) and AIMLite framework
RUN pip install --no-cache-dir uv aimlite

# 2. Set project working directory
WORKDIR /app

# 3. Copy project manifest
COPY aimlite.json .

# 4. Install all project dependencies into managed .venv using aimlite CLI
RUN aimlite install

# 5. Copy the remaining application files, data, and frontend assets
COPY . .

# 6. Train and calibrate model weights for serving
RUN aimlite train

# 7. Expose serving port
EXPOSE 8000

# 8. Health check verifying inference server status
HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:${PORT:-8000}/health || exit 1

# 9. Host multi-model inference server with custom frontend (dynamically binds to $PORT on cloud hosts)
CMD ["sh", "-c", "aimlite serve --host 0.0.0.0 --port ${PORT:-8000} --frontend frontend"]
```

### Build & Run Commands:
```bash
docker build -t my-aimlite-app .
docker run -d -p 8000:8000 my-aimlite-app
curl http://localhost:8000/health
```
