Metadata-Version: 2.3
Name: cortex-engine
Version: 0.1.0
Summary: AI Operating System for Coding — routes queries across 30-50+ LLMs with GPU scheduling, LRU caching, evaluation, and feedback-driven routing.
Keywords: ai,llm,inference,routing,orchestration,gpu,fastapi,coding-assistant,model-serving
Author: Cortex-Engine Contributors
Author-email: Cortex-Engine Contributors <hello@cortex-engine.dev>
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
Classifier: Framework :: FastAPI
Classifier: Typing :: Typed
Requires-Dist: fastapi>=0.115,<1.0
Requires-Dist: uvicorn[standard]>=0.34,<1.0
Requires-Dist: redis>=5.2,<6.0
Requires-Dist: pydantic>=2.10,<3.0
Requires-Dist: pydantic-settings>=2.7,<3.0
Requires-Dist: httpx>=0.28,<1.0
Requires-Dist: sentence-transformers>=3.4,<4.0 ; extra == 'embeddings'
Requires-Dist: faiss-cpu>=1.10,<2.0 ; extra == 'embeddings'
Requires-Dist: cortex-engine[embeddings,qdrant,postgres] ; extra == 'full'
Requires-Dist: sqlalchemy[asyncio]>=2.0,<3.0 ; extra == 'postgres'
Requires-Dist: asyncpg>=0.30,<1.0 ; extra == 'postgres'
Requires-Dist: qdrant-client>=1.14,<2.0 ; extra == 'qdrant'
Requires-Python: >=3.11
Project-URL: Bug Tracker, https://github.com/imnotdev25/cortex-engine/issues
Project-URL: Changelog, https://github.com/imnotdev25/cortex-engine/blob/main/CHANGELOG.md
Project-URL: Homepage, https://github.com/imnotdev25/cortex-engine
Project-URL: Repository, https://github.com/imnotdev25/cortex-engine
Provides-Extra: embeddings
Provides-Extra: full
Provides-Extra: postgres
Provides-Extra: qdrant
Description-Content-Type: text/markdown

# 🧠 Cortex-Engine

> An AI Operating System for coding — routes queries across 30–50+ models, manages GPU scheduling, caching, evaluation, and continuous feedback.

```
models = processes   |   GPU = CPU   |   router = scheduler   |   kernel = control plane
```

---

## Architecture

```
User Query
    │
    ▼
┌─────────────────────────────────────────────────────────────────┐
│                        Cortex-Engine API  (FastAPI)                  │
│                                                                   │
│  POST /inference                                                  │
│       │                                                           │
│       ▼                                                           │
│  ┌─────────────────────────────────────────────────────────┐     │
│  │              Model Orchestrator (central brain)          │     │
│  │                                                          │     │
│  │  ① Cache Check ──────────► Redis LRU Cache              │     │
│  │       │ miss                                             │     │
│  │       ▼                                                  │     │
│  │  ② Router Engine ─────────► Cluster Detection           │     │
│  │       │                     Model Selection              │     │
│  │       ▼                     Confidence Score            │     │
│  │  ③ Scheduler ──────────────► GPU Assignment             │     │
│  │       │                     Priority Queue              │     │
│  │       ▼                                                  │     │
│  │  ④ Model Worker ──────────► vLLM / Triton (Phase 2)     │     │
│  │       │                                                  │     │
│  │       ▼                                                  │     │
│  │  ⑤ Evaluator ─────────────► Static Analysis             │     │
│  │       │                     LLM Grading (Phase 2)       │     │
│  │       ▼                                                  │     │
│  │  ⑥ Feedback Log ──────────► Redis (rolling 10k)         │     │
│  └─────────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────────┘
         │
         ▼
    Model Registry (Redis Hash)
    7 seed models → 50+ in Phase 3
```

---

## Latency Targets

| Component   | Target  |
|-------------|---------|
| Router      | < 50 ms |
| Cache check | < 5 ms  |
| Scheduling  | < 10 ms |
| Total overhead | < 100 ms |

---

## Project Structure

```
cortex_engine/
├── main.py                  ← FastAPI app, lifespan, middleware, health/metrics
├── config.py                ← Pydantic Settings (env-based)
├── dependencies.py          ← All Depends() providers
│
├── models/
│   └── schemas.py           ← All Pydantic request/response models + enums
│
├── services/
│   ├── registry.py          ← Redis-backed model registry (7 seed models)
│   ├── router.py            ← Keyword/heuristic router with tiebreak scoring
│   ├── cache_manager.py     ← LRU response cache + warm pool + eviction log
│   ├── scheduler.py         ← Priority queue scheduler (Redis sorted sets)
│   ├── evaluator.py         ← Static analysis + LLM-grade stub
│   ├── feedback.py          ← Rolling feedback log + accuracy stats
│   └── orchestrator.py      ← Central brain: routes→schedules→infers→evals
│
├── routers/
│   ├── inference.py         ← POST /inference, GET /inference/route-preview
│   └── api.py               ← /registry CRUD, /admin (queue/cache/feedback)
│
├── tests/
│   ├── conftest.py          ← Async fixtures with fakeredis (no real Redis needed)
│   ├── test_registry.py     ← 11 tests
│   ├── test_router.py       ← 11 tests
│   ├── test_cache.py        ← 7 tests
│   ├── test_evaluator.py    ← 6 tests
│   ├── test_scheduler.py    ← 7 tests
│   ├── test_orchestrator.py ← 11 tests (integration)
│   └── test_feedback.py     ← 4 tests  (57 total → all pass)
│
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── pytest.ini
└── .env.example
```

---

## Quickstart

### Option A — Local (needs Redis)

```bash
# 1. Clone & install
git clone <repo>
cd cortex_engine
pip install -r requirements.txt

# 2. Start Redis
docker run -d -p 6379:6379 redis:7.4-alpine

# 3. Configure
cp .env.example .env

# 4. Run
uvicorn main:app --reload --port 8000
```

### Option B — Docker Compose (recommended)

```bash
docker compose up --build
# Optional: include Redis Commander UI
docker compose --profile dev up --build
```

### Run Tests (no Redis needed)

```bash
pip install -r requirements.txt
pytest tests/ -v
```

---

## API Reference

### `POST /inference`

Route a query to the best model and get a response.

```json
// Request
{
  "query": "Write a pytest test for a function that reverses a string",
  "preferred_type": null,       // optional: debugging|testing|explanation|...
  "preferred_model": null,      // optional: override routing entirely
  "max_tokens": 2048,
  "temperature": 0.2,
  "evaluate": true
}

// Response
{
  "request_id": "a1b2c3d4e5f6",
  "output": "...",
  "model_used": "starcoder2-7b",
  "route": {
    "cluster": "testing",
    "selected_model": "starcoder2-7b",
    "confidence": 0.72,
    "fallback_models": ["phi-3-mini"],
    "routing_latency_ms": 1.4
  },
  "evaluation": {
    "success": true,
    "score": 0.8,
    "method": "static_analysis",
    "details": "has_content=✓; has_code=✓; no_apology=✓; syntax_ok=✓; keyword_coverage=✓"
  },
  "total_latency_ms": 14.2,
  "cached": false,
  "tokens_used": 87
}
```

### `GET /inference/route-preview?query=...`

Dry-run: see which model would be selected without running inference.

### `GET /registry/`

List all registered models with status and metadata.

### `POST /registry/`

Register a new model.

### `PATCH /registry/{model_name}/status`

Set model status: `available | loading | busy | offline`

### `GET /admin/queue`

Current GPU queue depth, running jobs, per-GPU load counts.

### `GET /admin/cache`

Cache hit rate, warm pool, recent evictions.

### `GET /admin/feedback`

Routing accuracy stats and recent feedback log.

### `GET /health`

Liveness check — Redis connectivity + model counts.

### `GET /metrics`

Full system metrics: routing accuracy, cache hit rate, GPU loads, queue depth.

---

## Cluster → Model Routing

| Cluster | Trigger Keywords | Models |
|---------|-----------------|--------|
| `debugging` | error, exception, traceback, crash, fix bug | codellama-7b |
| `testing` | pytest, jest, unit test, mock, assert | starcoder2-7b |
| `explanation` | explain, what is, how does, document | mistral-7b-instruct |
| `refactoring` | refactor, optimize, clean up, simplify | qwen-coder-14b |
| `python` | python, django, fastapi, flask, .py | qwen-coder-7b, qwen-coder-14b |
| `general_code` | code, class, implement, build | deepseek-coder-6.7b |
| `fallback` | everything else | phi-3-mini |

---

## Seed Models (Phase 1 MVP)

| Model | Cluster | Size | Latency | GPU |
|-------|---------|------|---------|-----|
| `qwen-coder-7b` | python | 7B | 250ms | gpu-0 |
| `qwen-coder-14b` | python | 14B | 400ms | gpu-1 |
| `deepseek-coder-6.7b` | general_code | 6.7B | 220ms | gpu-0 |
| `codellama-7b` | debugging | 7B | 270ms | gpu-1 |
| `mistral-7b-instruct` | explanation | 7B | 300ms | gpu-2 |
| `starcoder2-7b` | testing | 7B | 260ms | gpu-2 |
| `phi-3-mini` | fallback | 3.8B | 150ms | gpu-3 |

---

## Roadmap

### Phase 1 (now — MVP)
- [x] FastAPI kernel with all core services
- [x] Redis-backed model registry (7 models)
- [x] Keyword/heuristic router with cluster detection
- [x] LRU response cache + warm pool + eviction log
- [x] Priority queue GPU scheduler (Redis sorted sets)
- [x] Static analysis evaluator
- [x] Rolling feedback system + accuracy tracking
- [x] 59 tests (all passing, no real Redis needed)
- [x] Docker Compose stack

### Phase 2
- [ ] Swap inference stub → real vLLM HTTP calls
- [ ] Embedding-based routing (BGE/E5 + FAISS/Qdrant)
- [ ] 15 models across more language clusters
- [ ] Ray Serve for distributed model workers
- [ ] LLM judge model for evaluation (phi-3-mini)

### Phase 3
- [ ] 30–50+ models
- [ ] Self-improving router (daily retraining from feedback)
- [ ] Cost-aware routing (balance quality vs. $/token)
- [ ] Multi-model collaboration (chain models)
- [ ] PostgreSQL for persistent metadata
- [ ] Reinforcement learning router

---

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `REDIS_URL` | `redis://localhost:6379/0` | Redis connection URL |
| `REDIS_MAX_CONNECTIONS` | `50` | Connection pool size |
| `PORT` | `8000` | API server port |
| `WORKERS` | `1` | Uvicorn worker count |
| `CACHE_TTL_SECONDS` | `3600` | Response cache TTL |
| `ENABLE_EVALUATION` | `true` | Run evaluator on outputs |
| `ENABLE_FEEDBACK` | `true` | Log routing feedback |

---

## Tech Stack

| Layer | Tech |
|-------|------|
| API | FastAPI + Uvicorn |
| Cache / State | Redis 7 (asyncio) |
| Scheduling | Redis sorted sets |
| Config | Pydantic Settings |
| Testing | pytest-asyncio + fakeredis |
| Containers | Docker + Docker Compose |
| Phase 2 Serving | vLLM + Triton |
| Phase 2 Orchestration | Ray Serve + Kubernetes |
| Phase 2 Vector DB | FAISS / Qdrant |
| Phase 3 Metadata | PostgreSQL + SQLAlchemy |
