Metadata-Version: 2.4
Name: shadow-engine
Version: 0.1.0
Summary: A self-improving background agent with persistent codebase knowledge graph and parallel experimentation
Project-URL: Homepage, https://github.com/rudraneel93/shadow-engine
Project-URL: Documentation, https://github.com/rudraneel93/shadow-engine#readme
Project-URL: Repository, https://github.com/rudraneel93/shadow-engine
Project-URL: Issues, https://github.com/rudraneel93/shadow-engine/issues
Project-URL: Changelog, https://github.com/rudraneel93/shadow-engine/blob/main/CHANGELOG.md
Author-email: Rudraneel Das <rudraneel93@github.com>
License: MIT
License-File: LICENSE
Keywords: ai-agent,background-agent,code-indexing,codebase-intelligence,coding-agent,cross-session-learning,developer-tools,knowledge-graph,parallel-experimentation,semantic-search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Code Generators
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: chromadb>=0.5.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: networkx>=3.3
Requires-Dist: numpy>=2.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: tiktoken>=0.7.0
Requires-Dist: tree-sitter>=0.23.0
Requires-Dist: typer>=0.12.0
Requires-Dist: uvicorn>=0.30.0
Provides-Extra: dev
Requires-Dist: mypy>=1.10.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.5.0; extra == 'dev'
Provides-Extra: redis
Requires-Dist: redis>=5.0.0; extra == 'redis'
Description-Content-Type: text/markdown

<p align="center">
  <h1 align="center">🧠 Shadow Engineer</h1>
  <p align="center">
    <strong>A self-improving background agent with persistent codebase knowledge graph and parallel experimentation.</strong>
  </p>
  <p align="center">
    <a href="https://github.com/shadow-engine/shadow-engine/actions"><img src="https://img.shields.io/badge/CI-passing-brightgreen" alt="CI"></a>
    <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue" alt="License"></a>
    <a href="https://pypi.org/project/shadow-engine"><img src="https://img.shields.io/badge/python-3.12%2B-blue" alt="Python"></a>
    <a href="https://github.com/shadow-engine/shadow-engine/blob/main/API_DOCS.md"><img src="https://img.shields.io/badge/docs-API_DOCS.md-orange" alt="Docs"></a>
  </p>
</p>

---

> **No existing background agent framework does this.** Every agent today (Ramp's Inspect, Open-Inspect, Copilot, Claude Code) treats every session as a blank slate. Session 100 is no smarter than Session 1. Shadow Engineer remembers — and gets smarter with every session.

---

## Table of Contents

1. [What Is Shadow Engineer?](#what-is-shadow-engine)
2. [The Problem It Solves](#the-problem-it-solves)
3. [Architecture](#architecture)
4. [Installation](#installation)
5. [Quick Start (CLI)](#quick-start-cli)
6. [REST API Integration](#rest-api-integration)
7. [Python SDK](#python-sdk)
8. [Open-Inspect Integration](#open-inspect-integration)
9. [Deployment](#deployment)
10. [Configuration Reference](#configuration-reference)
11. [Supported Languages](#supported-languages)
12. [Project Structure](#project-structure)
13. [FAQ](#faq)
14. [License](#license)

---

## What Is Shadow Engineer?

Shadow Engineer is a **learning layer** that sits on top of any background coding agent (Open-Inspect, Claude Code, custom agents). It provides three capabilities that no other framework offers:

| Engine | What It Does | Unique Value |
|--------|-------------|--------------|
| **Knowledge Graph** | Indexes your codebase into a persistent, searchable semantic graph | Agents start with informed context — no more fumbling through code |
| **Laboratory** | Spawns N parallel agent sessions with different strategies and picks the winner | Not one attempt — choose from proven solutions |
| **Learning Engine** | Analyzes every session to extract patterns, track efficacy, and suggest approaches | Session 100 is smarter than Session 1 — compounding intelligence |

### The Compounding Moat

```
Session 1:   No context  | No patterns  | No approach data  | 40% success rate (guess)
Session 10:  10 symbols  | 3 patterns   | 2 approaches      | 55% success rate
Session 50:  50 symbols  | 8 patterns   | 5 approaches      | 70% success rate
Session 200: 100+ symbols | 20+ patterns | 10+ approaches   | 80%+ success rate
```

**Every session makes the next one smarter.** This is the defensible moat that no competitor ships.

---

## The Problem It Solves

Current background coding agents treat every session independently:

| Problem | Without Shadow Engineer | With Shadow Engineer |
|---------|------------------------|---------------------|
| **No memory** | Agent fumbles through codebase every time | Agent starts with relevant context from the knowledge graph |
| **No learning** | Same mistakes repeated across sessions | Pattern extraction + efficacy tracking prevents repeat failures |
| **Single attempt** | One approach, one model — if it fails, start over | N parallel experiments, winner picked automatically |
| **No codebase understanding** | "What file handles authentication?" every session | Semantic search: "authentication" → `auth/service.py` |

---

## Architecture

```
                    ┌──────────────────────────────────────────┐
                    │           SHADOW ENGINEER                 │
                    │                                           │
 User sends task    │  ┌──────────────┐  ┌──────────────────┐  │
        │           │  │  Knowledge   │  │    Laboratory    │  │
        ▼           │  │    Graph     │  │                  │  │
 ┌──────────┐       │  │              │  │  3 variants:     │  │
 │ Classify │───────┼──│ • 211 symbols│  │  • Targeted Fix  │  │
 │ problem  │       │  │ • 26 files   │  │  • Root Cause    │  │
 └────┬─────┘       │  │ • 3 patterns │  │  • Defense Depth │  │
      │             │  └──────┬───────┘  └────────┬─────────┘  │
 ┌────▼─────┐       │         │                   │            │
 │  Build   │       │         │                   ▼            │
 │ context  │───────┼─────────┼──▶  Agent Prompt + Approach    │
 └────┬─────┘       │         │                                │
      │             │         │     ┌──────────────────┐       │
      │             │         │     │     Learning     │       │
      ▼             │         │     │      Engine      │       │
 ┌──────────┐       │         │     │                  │       │
 │  Agent   │       │         │     │ • Pattern extract│       │
 │ executes │       │         └─────│ • Efficacy track │       │
 │  task    │       │               │ • Failure analyze│       │
 └────┬─────┘       │               └────────┬─────────┘       │
      │             │                        │                 │
      ▼             │                        ▼                 │
 ┌──────────┐       │           ┌──────────────────────┐      │
 │  Ingest  │───────┼──────────▶│  Next session is     │      │
 │  result  │       │           │  SMARTER than before  │      │
 └──────────┘       │           └──────────────────────┘      │
                    └──────────────────────────────────────────┘
```

### Three Engines

#### 1. Knowledge Graph — "Remember"
- **7 languages** indexed (Python, TypeScript, JavaScript, Go, Rust)
- **Semantic search** via ChromaDB vector embeddings
- **Dependency mapping** — "If I change `UserService.authenticate()`, what breaks?"
- **Impact analysis** — BFS up the dependency chain
- **Context injection** — relevant symbols, conventions, and approaches injected into agent prompts

#### 2. Laboratory — "Experiment"
- **12 strategy templates** — problem-type-aware (bug fix → "Targeted Fix" + "Root Cause + Guard" + "Defense in Depth")
- **Configurable scoring** — logistic curve normalization, no arbitrary cliffs
- **4 winner modes** — best performing, smallest change, fastest execution, first to pass
- **Concurrent execution** — semaphore-limited parallel spawning

#### 3. Learning Engine — "Improve"
- **Pattern extraction** — infers testing conventions, change scope, code review quality
- **Efficacy tracking** — running averages for success rate, duration, tokens
- **Failure analysis** — understands why approaches fail
- **Approach suggestion** — recommends historically-best approach and model
- **Confidence scores** — every classification returns `(type, 0.0–1.0)` not just a label

---

## Installation

### Prerequisites
- Python 3.12 or later
- (Optional) [Redis](https://redis.io) for production rate limiting
- (Optional) [Ollama](https://ollama.ai) for local LLM testing

### Option 1: pip (recommended)

```bash
pip install shadow-engine

# With optional Redis support
pip install "shadow-engine[redis]"
```

### Option 2: From source

```bash
git clone https://github.com/shadow-engine/shadow-engine.git
cd shadow-engine
pip install -e ".[dev]"
```

### Option 3: Docker

```bash
docker compose -f docker/docker-compose.yml up -d
```

This starts both Shadow Engineer and a Redis instance for rate limiting.

---

## Quick Start (CLI)

### 1. Index Your Codebase

```bash
cd /path/to/your/project
shadow-engine bootstrap
```

Output:
```
Bootstrapped: 347 symbols, 52 files indexed
```

### 2. Search for Symbols

```bash
shadow-engine search "authenticate"
```

Output:
```
[function] authenticate_user — src/auth/service.py
  Authenticates a user with email and password. Returns JWT token.
[method] authenticate — src/auth/middleware.py
  Middleware that checks the Authorization header for valid JWT.
```

### 3. Get AI-Ready Context

```bash
shadow-engine context "fix the login rate-limiting bug"
```

Output (injects into agent prompts):
```
## Codebase Knowledge Graph Context (Semantic)

### Semantically Relevant Symbols

- **authenticate_user** (`function`) in `src/auth/service.py` (relevance: 0.89)
  Authenticate a user from a JWT token.
  Depends on: UserModel, TokenService

- **login_handler** (`function`) in `src/auth/views.py` (relevance: 0.82)
  Handle login POST requests. Validates credentials and returns session token.
  Complexity: 6.5

### Learned Codebase Conventions
- **error_handling**: Auth errors return 401 with JSON body {error: string, code: string}
- **testing**: Agent writes tests alongside code changes.

### Historically Effective Approaches
- **Targeted Fix**: 85% success rate (17/20) — best model: claude-sonnet-4-6
```

### 4. Analyze Change Impact

```bash
shadow-engine impact "UserService"
```

Output:
```json
{
  "symbol": {"name": "UserService", "kind": "class", "file_path": "src/services/user.py"},
  "dependencies": ["Database", "CacheClient", "EmailService"],
  "direct_dependents": ["AuthController", "ProfileController", "AdminController"],
  "total_affected_symbols": 18
}
```

### 5. Get Approach Suggestion

```bash
shadow-engine suggest "add a search feature for products"
```

Output:
```json
{
  "problem_type": "feature",
  "classification_confidence": 0.85,
  "recommended_approach": "Extensible Design",
  "expected_success_rate": 0.78,
  "best_model": "claude-sonnet-4-6"
}
```

### 6. Create Parallel Experiments

```bash
shadow-engine experiment "refactor the billing module" --variants 3
```

### 7. Record Session Results

```bash
shadow-engine record \
  --session-id "session-abc123" \
  --outcome "success" \
  --prompt "fix the login rate-limiting bug" \
  --approach "Targeted Fix" \
  --model "claude-sonnet-4-6" \
  --files "src/auth/service.py" "tests/auth/test_service.py" \
  --tests-passed 12 --tests-failed 0 \
  --duration 45.2 --tokens 8500
```

### 8. View Improvement Report

```bash
shadow-engine report
```

---

## REST API Integration

Start the server:
```bash
# Development
uvicorn shadow_engine.api_server.server:app --reload

# Production
shadow-engine-server
# or
docker compose -f docker/docker-compose.yml up -d
```

### Authentication (optional)

```bash
export SHADOW_ENGINE_API_KEY="your-secret-key"
curl -H "X-API-Key: your-secret-key" http://localhost:8000/health
```

### Core Workflow

```bash
# 1. Index the codebase
curl -X POST http://localhost:8000/bootstrap

# 2. Get context for an agent prompt
curl "http://localhost:8000/context?task=fix+the+login+rate+limiting+bug"

# 3. Get approach suggestion
curl "http://localhost:8000/suggest?task=fix+the+login+rate+limiting+bug"

# 4. Create an experiment batch
curl -X POST "http://localhost:8000/experiment?task=refactor+auth&variants=3"

# 5. Record session result
curl -X POST http://localhost:8000/sessions/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "sess-001",
    "outcome": "success",
    "prompt": "fix the login bug",
    "approach": "Targeted Fix",
    "model": "claude-sonnet-4-6",
    "files_changed": ["src/auth.py", "tests/test_auth.py"],
    "tests_passed": 10,
    "tests_failed": 0,
    "duration_seconds": 30.0,
    "token_count": 5000
  }'

# 6. View improvement report
curl http://localhost:8000/report

# 7. Check operational metrics
curl http://localhost:8000/metrics
```

### Full API Reference

All 11 endpoints are documented in **[API_DOCS.md](API_DOCS.md)** with request/response schemas, field descriptions, and curl examples. Interactive Swagger UI at `http://localhost:8000/docs`.

---

## Python SDK

```python
import httpx

class ShadowEngineClient:
    """Minimal Python client for Shadow Engine REST API."""
    
    def __init__(self, base_url: str = "http://localhost:8000", api_key: str | None = None):
        self.base_url = base_url.rstrip("/")
        self.headers = {"X-API-Key": api_key} if api_key else {}

    def bootstrap(self, repo: str = ".") -> dict:
        r = httpx.post(f"{self.base_url}/bootstrap", params={"repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def get_context(self, task: str, repo: str = ".") -> str:
        r = httpx.get(f"{self.base_url}/context", params={"task": task, "repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()["context"]

    def search(self, query: str, kind: str | None = None, repo: str = ".") -> dict:
        params = {"query": query, "repo": repo}
        if kind: params["kind"] = kind
        r = httpx.get(f"{self.base_url}/search", params=params, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def suggest(self, task: str, repo: str = ".") -> dict:
        r = httpx.get(f"{self.base_url}/suggest", params={"task": task, "repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def create_experiment(self, task: str, variants: int = 3, repo: str = ".") -> dict:
        r = httpx.post(f"{self.base_url}/experiment", params={"task": task, "variants": variants, "repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def ingest_session(self, **kwargs) -> dict:
        r = httpx.post(f"{self.base_url}/sessions/ingest", json=kwargs, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def get_report(self, repo: str = ".") -> str:
        r = httpx.get(f"{self.base_url}/report", params={"repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.text

    def get_stats(self, repo: str = ".") -> dict:
        r = httpx.get(f"{self.base_url}/stats", params={"repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def get_metrics(self, repo: str = ".") -> dict:
        r = httpx.get(f"{self.base_url}/metrics", params={"repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def health(self) -> dict:
        r = httpx.get(f"{self.base_url}/health")
        r.raise_for_status()
        return r.json()
```

### Usage Example

```python
# Initialize
client = ShadowEngineClient(base_url="http://localhost:8000", api_key="your-secret-key")

# Bootstrap the codebase
result = client.bootstrap()
print(f"Indexed {result['symbols_indexed']} symbols")

# Get context for an agent
context = client.get_context("fix the login rate-limiting bug")

# Get approach suggestion
suggestion = client.suggest("fix the login rate-limiting bug")
print(f"Recommended: {suggestion['recommended_approach']} ({suggestion['expected_success_rate']:.0%} expected)")

# Record a completed session
client.ingest_session(
    session_id="sess-001",
    outcome="success",
    prompt="fix the login bug",
    approach="Targeted Fix",
    model="claude-sonnet-4-6",
    files_changed=["src/auth.py", "tests/test_auth.py"],
    tests_passed=10,
    tests_failed=0,
    duration_seconds=30.0,
    token_count=5000,
)

# After several sessions, view the improvement
print(client.get_report())
print(f"Overall success rate: {client.get_stats()['overall_success_rate']:.0%}")
```

---

## Open-Inspect Integration

Shadow Engineer provides an async bridge that plugs directly into Open-Inspect's session lifecycle:

```python
from shadow_engine.integrations.openinspect import OpenInspectBridge

# Initialize
bridge = OpenInspectBridge(repo_path="/path/to/your/repo")
bridge.bootstrap_if_needed()

# Before session spawn — enrich the prompt with knowledge graph context
config = {
    "prompt": "fix the login rate-limiting bug",
    "repository": "my-repo",
}
enriched = await bridge.enrich_session_config(config)
# enriched["prompt"] now contains knowledge graph context
# enriched["suggested_approach"] = "Targeted Fix"
# enriched["suggested_model"] = "claude-sonnet-4-6"
# enriched["problem_type"] = "bug_fix"
# enriched["classification_confidence"] = 0.95

# After session completes — ingest the result for learning
result = {
    "session_id": "sess-abc123",
    "outcome": "completed",
    "prompt": "fix the login rate-limiting bug",
    "approach": "Targeted Fix",
    "model": "claude-sonnet-4-6",
    "pr_url": "https://github.com/myorg/myrepo/pull/142",
    "files_changed": ["src/auth.py", "tests/test_auth.py"],
    "tests_passed": 12,
    "tests_failed": 0,
    "duration_seconds": 45.2,
    "token_count": 8500,
}
ingestion = await bridge.ingest_session_result(result)
```

---

## Deployment

### Docker (Production)

```bash
# With Redis for rate limiting
docker compose -f docker/docker-compose.yml up -d

# Without Redis (in-memory rate limiting)
docker build -t shadow-engine -f docker/Dockerfile .
docker run -p 8000:8000 \
  -e SHADOW_ENGINE_API_KEY=your-secret \
  -v /data/shadow-engine:/home/shadow/data \
  shadow-engine
```

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `SHADOW_ENGINE_API_KEY` | `""` (disabled) | API key for authentication. Set to enable |
| `SHADOW_ENGINE_REDIS_URL` | `redis://redis:6379` | Redis URL for rate limiting |
| `SHADOW_ENGINE_RATE_LIMIT` | `100` | Max requests per window |
| `SHADOW_ENGINE_RATE_WINDOW` | `60` | Rate limit window in seconds |
| `SHADOW_ENGINE_STORAGE_PATH` | `/home/shadow/data` | Persistent storage directory |
| `SHADOW_ENGINE_PORT` | `8000` | API server port |
| `SHADOW_ENGINE_HOST` | `0.0.0.0` | API server host |

### Production Checklist

See **[FINDINGS_REPORT.md](FINDINGS_REPORT.md)** §6.2 for a detailed 12-point production readiness checklist.

---

## Configuration Reference

### Scoring Configuration

Customize how experiment variants are scored:

```python
from shadow_engine.laboratory.experiment import ScoringConfig

config = ScoringConfig(
    test_pass_weight=0.50,      # Prioritize test passing
    change_size_weight=0.15,    # Less emphasis on change size
    speed_weight=0.20,          # More emphasis on speed
    token_efficiency_weight=0.05,
    file_scope_weight=0.10,
)
```

### Model Configuration

```python
from shadow_engine.main import ShadowEngine

engine = ShadowEngine(
    storage_path="./.shadow-engine",
    repo_path="./my-project",
    use_sqlite=True,    # Production: SQLite WAL mode (default)
    use_chroma=True,    # ChromaDB semantic search (default)
)
```

---

## Supported Languages

| Language | Extensions | Symbols Extracted |
|----------|-----------|-------------------|
| Python | `.py` | Functions, methods, classes |
| TypeScript | `.ts`, `.tsx` | Functions, classes, interfaces, enums, type aliases |
| JavaScript | `.js`, `.jsx` | Functions, classes, constants |
| Go | `.go` | Functions, methods, structs, interfaces |
| Rust | `.rs` | Functions, structs, traits, enums, type aliases |

---

## Project Structure

```
shadow-engine/
├── README.md                        # You are here
├── CHANGELOG.md                     # Version history
├── API_DOCS.md                      # Full REST API reference (1,070 lines)
├── FINDINGS_REPORT.md               # Research, efficiency, competitive analysis
├── pyproject.toml                   # Package dependencies + entry points
├── docker/
│   ├── Dockerfile                   # Production container
│   └── docker-compose.yml           # Redis + app stack
├── .github/workflows/ci.yml         # CI/CD pipeline
├── scripts/real_demo.py             # End-to-end real LLM demo
├── src/shadow_engine/
│   ├── main.py                      # Orchestrator + CLI (11 commands)
│   ├── knowledge_graph/
│   │   ├── models.py                # 10 Pydantic data models
│   │   ├── indexer.py               # 7-language AST parser
│   │   └── store.py                 # JSON backend (legacy)
│   ├── sqlite_store/db.py           # SQLite WAL backend (production default)
│   ├── chroma_store/vector_store.py # ChromaDB semantic search
│   ├── laboratory/experiment.py     # Experiment runner + scoring
│   ├── learning/engine.py           # Pattern extraction + efficacy tracking
│   ├── async_lab/executor.py        # Concurrent experiment execution
│   ├── api_server/server.py         # FastAPI REST server
│   ├── integrations/openinspect.py  # Open-Inspect async bridge
│   └── redis_limiter/               # Redis rate limiter
└── tests/
    ├── test_knowledge_graph.py       # 33 tests
    ├── test_laboratory.py            # 19 tests
    ├── test_learning.py              # 12 tests
    └── test_integration.py           # 16 tests
```

---

## FAQ

**Q: How is this different from Open-Inspect?**
A: Open-Inspect is a background agent framework — it spawns sandboxes and runs coding sessions. Shadow Engineer is a **learning layer** that sits on top. It adds cross-session memory, parallel experimentation, and compounding intelligence that Open-Inspect (and every other agent framework) lacks.

**Q: Can I use this without Open-Inspect?**
A: Yes. Shadow Engineer works with any background agent via its REST API or CLI. The Open-Inspect bridge is an optional integration.

**Q: Does it require a GPU?**
A: No. ChromaDB uses CPU embeddings by default. GPU is only needed if you want to use a custom embedding model.

**Q: What scale does this support?**
A: SQLite WAL mode supports 100K+ sessions. For larger scale, the findings report recommends PostgreSQL as a future backend.

**Q: Is this ready for production?**
A: See the [FINDINGS_REPORT.md](FINDINGS_REPORT.md) for a detailed assessment. Verdict: **A-** — production-grade for internal teams, beta-quality for public release.

---

## License

MIT — Build on it. Ship it. Make agents smarter.

---

*Inspired by [Ramp's Inspect](https://builders.ramp.com/post/why-we-built-our-background-agent) and [Open-Inspect](https://github.com/ColeMurray/background-agents).*