# Skills Tracking Integration with Session-Buddy

**Comprehensive AI agent metrics and intelligent skill recommendations for Crackerjack**

## Table of Contents

- [Overview](#overview)
- [Architecture](#architecture)
- [Installation](#installation)
- [Configuration](#configuration)
- [Backend Options](#backend-options)
- [Usage Guide](#usage-guide)
- [API Reference](#api-reference)
- [Data Migration](#data-migration)
- [Performance](#performance)
- [Troubleshooting](#troubleshooting)

______________________________________________________________________

## Overview

**Skills tracking** provides comprehensive metrics collection for all AI agent invocations in Crackerjack, enabling:

- **🎯 Intelligent Agent Selection**: Learn which agents work best for specific problems
- **📊 Performance Analytics**: Track success rates, duration, and effectiveness by context
- **🧠 Semantic Discovery**: Find agents using natural language queries
- **🔄 Continuous Learning**: System improves with every invocation
- **📈 Workflow Correlation**: Understand agent effectiveness by Oneiric phase

### What Gets Tracked?

Every agent invocation records:

```python
{
    "skill_name": "RefactoringAgent",
    "invoked_at": "2026-02-10T12:34:56Z",
    "session_id": "crackerjack-session-abc123",
    "completed": true,
    "duration_seconds": 45.2,
    "user_query": "Fix complexity issues in module X",
    "workflow_phase": "comprehensive_hooks",
    "alternatives_considered": ["PerformanceAgent", "DRYAgent"],
    "selection_rank": 1,  # First choice
    "error_type": null
}
```

### Why It Matters

**Before Skills Tracking**:

```python
# Which agent should I use for this problem?
agent = guess_best_agent(issue)  # Trial and error
```

**After Skills Tracking**:

```python
# Get data-driven recommendations
recommendations = tracker.get_recommendations(
    user_query="Fix complexity in async code",
    workflow_phase="comprehensive_hooks"
)
# Returns agents ranked by historical success rate for this context
```

______________________________________________________________________

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                         Crackerjack                                 │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                   Agent Orchestrator                          │   │
│  │  (Agent selection, execution coordination)                   │   │
│  └─────────────────────┬───────────────────────────────────────┘   │
│                        │                                           │
│  ┌─────────────────────▼───────────────────────────────────────┐   │
│  │                   Agent Context                              │   │
│  │  (skills_tracker: SkillsTrackerProtocol)                    │   │
│  └─────────────────────┬───────────────────────────────────────┘   │
│                        │                                           │
│  ┌─────────────────────▼───────────────────────────────────────┐   │
│  │              Skills Tracking Protocol                        │   │
│  │  • track_invocation() - Record agent invocation             │   │
│  │  • get_recommendations() - Get intelligent recommendations   │   │
│  │  • is_enabled() - Check if tracking active                  │   │
│  └─────────────────────┬───────────────────────────────────────┘   │
│                        │                                           │
└────────────────────────┼───────────────────────────────────────────┘
                         │
         ┌───────────────┼───────────────┬───────────────────┐
         │               │               │                   │
    ┌────▼─────┐   ┌────▼─────┐   ┌────▼─────┐       ┌──────▼──────┐
    │  Direct  │   │   MCP    │   │  Auto    │       │   No-Op     │
    │  Tracker │   │  Bridge  │   │ Selector │       │  (Disabled) │
    └────┬─────┘   └────┬─────┘   └────┬─────┘       └─────────────┘
         │              │              │
         └──────────────┼──────────────┘
                        │
         ┌──────────────▼──────────────┐
         │    Session-Buddy             │
         │  (Skills Metrics Storage)    │
         │  • Dhara (SQLite + WAL)     │
         │  • Vector embeddings (Akosha)│
         │  • Semantic search           │
         └──────────────────────────────┘
```

### Key Components

**1. SkillsTrackerProtocol**

Interface defining all tracking operations:

```python
@runtime_checkable
class SkillsTrackerProtocol(Protocol):
    def track_invocation(
        self,
        skill_name: str,
        user_query: str | None = None,
        alternatives_considered: list[str] | None = None,
        selection_rank: int | None = None,
        workflow_phase: str | None = None,
    ) -> Callable[..., None] | None:
        """Track skill invocation, returns completer function."""
        ...

    def get_recommendations(
        self,
        user_query: str,
        limit: int = 5,
        workflow_phase: str | None = None,
    ) -> list[dict[str, object]]:
        """Get skill recommendations based on query."""
        ...

    def is_enabled(self) -> bool:
        """Check if tracking is enabled."""
        ...

    def get_backend(self) -> str:
        """Get backend name."""
        ...
```

**2. Implementations**

- **`SessionBuddyDirectTracker`**: Direct API calls to session-buddy (fast, tight coupling)
- **`SessionBuddyMCPTracker`**: MCP bridge with fallback to direct (loose coupling)
- **`NoOpSkillsTracker`**: Disabled state (zero overhead)

**3. Integration Points**

- **`AgentContext.track_skill_invocation()`**: Manual tracking helper
- **`AgentContext.get_skill_recommendations()`**: Recommendation helper
- **`AgentOrchestrator._execute_crackerjack_agent()`**: Automatic tracking

______________________________________________________________________

## Installation

### Prerequisites

- **Python 3.13+**
- **Session-buddy** (auto-installed with crackerjack)

### Install Dependencies

```bash
# Session-buddy is included as a dependency
uv sync

# Or install manually for development
uv add session-buddy
```

### Verify Installation

```bash
# Test session-buddy import
python -c "from session_buddy.core.skills_tracker import get_session_tracker; print('✅ Session-buddy available')"

# Test crackerjack integration
python -c "from crackerjack.integration.skills_tracking import create_skills_tracker; print('✅ Crackerjack integration ready')"
```

______________________________________________________________________

## Configuration

### Settings File

**Location**: `settings/local.yaml` or `settings/crackerjack.yaml`

```yaml
skills:
  # Enable/disable skills tracking
  enabled: true

  # Backend: "direct", "mcp", "auto"
  backend: auto

  # Database path (null = default: .session-buddy/skills.db)
  db_path: null

  # MCP server URL
  mcp_server_url: "http://localhost:8678"

  # Recommendation settings
  min_similarity: 0.3          # Minimum similarity score (0.0-1.0)
  max_recommendations: 5       # Max agents to recommend
  enable_phase_aware: true     # Consider workflow phase
  phase_weight: 0.3           # Phase effectiveness weight (0.0-1.0)
```

### Environment Variables

```bash
# Disable skills tracking temporarily
export CRACKERJACK_SKILLS_ENABLED=false

# Use specific backend
export CRACKERJACK_SKILLS_BACKEND=direct

# Custom database path
export CRACKERJACK_SKILLS_DB_PATH=/path/to/skills.db
```

### Runtime Configuration

```python
from crackerjack.config import CrackerjackSettings

# Load settings
settings = CrackerjackSettings.load()

# Access skills configuration
print(f"Skills tracking enabled: {settings.skills.enabled}")
print(f"Backend: {settings.skills.backend}")
print(f"Min similarity: {settings.skills.min_similarity}")
```

______________________________________________________________________

## Backend Options

### Direct API (Tight Coupling)

**Implementation**: `SessionBuddyDirectTracker`

**Characteristics**:

- ✅ Fast (< 1ms latency)
- ✅ Simple setup (no networking)
- ✅ Low memory footprint
- ❌ Tight coupling (requires session-buddy in Python path)
- ❌ Single-machine only

**When to Use**:

- Local development
- Single-machine deployments
- Maximum performance required

**Example**:

```python
from crackerjack.integration.skills_tracking import create_skills_tracker

tracker = create_skills_tracker(
    session_id="my-session",
    backend="direct",
    db_path=Path(".session-buddy/skills.db")
)
```

### MCP Bridge (Loose Coupling)

**Implementation**: `SessionBuddyMCPTracker`

**Characteristics**:

- ✅ Loose coupling (MCP protocol)
- ✅ Remote deployment
- ✅ Easy testing (mock MCP server)
- ✅ Automatic fallback to direct tracking
- ❌ Higher latency (5-10ms)
- ❌ More complex setup

**When to Use**:

- Distributed systems
- Microservices architecture
- Multi-project setups
- Need remote monitoring

**Example**:

```python
tracker = create_skills_tracker(
    session_id="my-session",
    backend="mcp",
    mcp_server_url="http://localhost:8678"
)
```

### Auto Detection (Recommended)

**Implementation**: Tries MCP first, falls back to direct

**Characteristics**:

- ✅ Best of both worlds
- ✅ Automatic fallback
- ✅ Zero configuration
- ⚠️ Slightly slower initial connection (MCP probe)

**Example**:

```python
tracker = create_skills_tracker(
    session_id="my-session",
    backend="auto"  # Recommended!
)
```

______________________________________________________________________

## Usage Guide

### 1. Automatic Tracking (Default)

All agent executions via `AgentOrchestrator` are **automatically tracked**:

```python
# No code changes needed!
# Agent orchestrator handles tracking automatically

from crackerjack.intelligence.agent_orchestrator import AgentOrchestrator

orchestrator = AgentOrchestrator(...)
result = await orchestrator.execute_agent("RefactoringAgent", issue)
# ^^^ Automatically tracked with:
# - Agent name
# - User query
# - Success/failure
# - Duration
# - Workflow phase
```

### 2. Manual Tracking

For custom agent execution outside orchestrator:

```python
from crackerjack.agents.base import AgentContext
from crackerjack.integration.skills_tracking import create_skills_tracker

# Create tracker
tracker = create_skills_tracker(session_id="custom-session")

# Create context with tracker
context = AgentContext(
    project_path=Path("/my/project"),
    skills_tracker=tracker
)

# Track invocation
completer = context.track_skill_invocation(
    skill_name="MyCustomAgent",
    user_query="Fix memory leak in async code",
    workflow_phase="comprehensive_hooks",
    alternatives_considered=["PerformanceAgent", "TestSpecialistAgent"],
    selection_rank=1
)

# ... do work ...

# Complete tracking
completer(
    completed=True,
    follow_up_actions=["Run memory profiler"],
    error_type=None
)
```

### 3. Get Recommendations

Find the best agent for a problem:

```python
# Get recommendations
recommendations = context.get_skill_recommendations(
    user_query="I need to fix type errors in async functions",
    limit=5,
    workflow_phase="comprehensive_hooks"
)

# Process recommendations
for rec in recommendations:
    print(f"Agent: {rec['skill_name']}")
    print(f"  Similarity: {rec['similarity_score']:.2f}")
    print(f"  Success Rate: {rec['completed']}")
    print(f"  Avg Duration: {rec['duration_seconds']:.1f}s")
    print(f"  Phase: {rec.get('workflow_phase', 'N/A')}")
    print()
```

**Output**:

```
Agent: RefactoringAgent
  Similarity: 0.92
  Success Rate: True
  Avg Duration: 45.2s
  Phase: comprehensive_hooks

Agent: PerformanceAgent
  Similarity: 0.85
  Success Rate: True
  Avg Duration: 38.1s
  Phase: comprehensive_hooks

Agent: TypeAgent
  Similarity: 0.78
  Success Rate: True
  Avg Duration: 52.3s
  Phase: comprehensive_hooks
```

### 4. Workflow-Phase-Aware Recommendations

Get recommendations specific to Oneiric workflow phase:

```python
# Fast hooks phase (quick quality checks)
fast_recs = context.get_skill_recommendations(
    user_query="Fix import errors",
    workflow_phase="fast_hooks",
    limit=3
)

# Comprehensive hooks phase (deep analysis)
comp_recs = context.get_skill_recommendations(
    user_query="Fix complexity issues",
    workflow_phase="comprehensive_hooks",
    limit=3
)

# Only agents effective in this phase will be recommended
```

### 5. Semantic Discovery

Find agents using natural language:

```python
# Describe your problem in plain English
recommendations = context.get_skill_recommendations(
    user_query="My tests are flaky and I need to stabilize them",
    limit=5
)

# Semantic search finds:
# - TestSpecialistAgent (test stabilization)
# - TestCreationAgent (fixture issues)
# - DRYAgent (test code duplication)
```

______________________________________________________________________

## API Reference

### SkillsTrackerProtocol

Interface for all tracking implementations.

#### `track_invocation()`

Track a skill invocation.

```python
def track_invocation(
    self,
    skill_name: str,
    user_query: str | None = None,
    alternatives_considered: list[str] | None = None,
    selection_rank: int | None = None,
    workflow_phase: str | None = None,
) -> Callable[..., None] | None:
```

**Parameters**:

- `skill_name`: Name of the agent being invoked
- `user_query`: User's problem description
- `alternatives_considered`: Other agents shown to user
- `selection_rank`: Position in recommendation list (1 = first choice)
- `workflow_phase`: Current Oneiric workflow phase

**Returns**:

- Completer function or `None` if tracking disabled

**Completer Function**:

```python
def completer(
    *,
    completed: bool = True,
    follow_up_actions: list[str] | None = None,
    error_type: str | None = None,
) -> None:
    """Complete the skill invocation tracking."""
    ...

# Usage
completer(completed=True)
completer(completed=False, error_type="SyntaxError")
completer(completed=True, follow_up_actions=["Run pytest"])
```

#### `get_recommendations()`

Get skill recommendations based on query.

```python
def get_recommendations(
    self,
    user_query: str,
    limit: int = 5,
    workflow_phase: str | None = None,
) -> list[dict[str, object]]:
```

**Parameters**:

- `user_query`: User's problem description
- `limit`: Maximum recommendations to return
- `workflow_phase`: Current workflow phase (for phase-aware search)

**Returns**:

```python
[
    {
        "skill_name": str,           # Agent name
        "similarity_score": float,   # Semantic similarity (0.0-1.0)
        "completed": bool,           # Historical completion rate
        "duration_seconds": float,   # Average duration
        "workflow_phase": str,       # Most effective phase
    },
    ...
]
```

#### `is_enabled()`

Check if skills tracking is enabled.

```python
def is_enabled(self) -> bool:
```

**Returns**: `True` if tracking active, `False` if disabled

#### `get_backend()`

Get backend implementation name.

```python
def get_backend(self) -> str:
```

**Returns**: `"session-buddy-direct"`, `"session-buddy-mcp"`, or `"none"`

### AgentContext Helpers

Convenience methods on `AgentContext` for skills tracking.

#### `track_skill_invocation()`

```python
def track_skill_invocation(
    self,
    skill_name: str,
    user_query: str | None = None,
    alternatives_considered: list[str] | None = None,
    selection_rank: int | None = None,
    workflow_phase: str | None = None,
) -> t.Any | None:
    """Track a skill invocation with session-buddy.

    Returns:
        Completer function or None if tracking disabled.
    """
```

**Usage**:

```python
completer = context.track_skill_invocation(
    skill_name="RefactoringAgent",
    user_query="Fix complexity",
    workflow_phase="comprehensive_hooks"
)

if completer:
    completer(completed=True)
```

#### `get_skill_recommendations()`

```python
def get_skill_recommendations(
    self,
    user_query: str,
    limit: int = 5,
    workflow_phase: str | None = None,
) -> list[dict[str, t.Any]]:
    """Get skill recommendations from session-buddy.

    Args:
        user_query: User's problem description
        limit: Maximum recommendations to return
        workflow_phase: Current workflow phase

    Returns:
        List of recommendation dicts (empty list if tracking disabled)
    """
```

**Usage**:

```python
recommendations = context.get_skill_recommendations(
    user_query="Fix type errors",
    limit=5,
    workflow_phase="comprehensive_hooks"
)

for rec in recommendations:
    print(f"{rec['skill_name']}: {rec['similarity_score']:.2f}")
```

### Factory Function

#### `create_skills_tracker()`

Create a skills tracker instance.

```python
def create_skills_tracker(
    session_id: str,
    enabled: bool = True,
    backend: str = "direct",  # "direct", "mcp", "auto"
    db_path: Path | None = None,
    mcp_server_url: str = "http://localhost:8678",
) -> SkillsTrackerProtocol:
    """Create a skills tracker instance.

    Args:
        session_id: Session identifier for tracking
        enabled: Whether skills tracking is enabled
        backend: Which backend to use
        db_path: Path to skills database (for direct backend)
        mcp_server_url: URL of session-buddy MCP server (for MCP backend)

    Returns:
        Skills tracker implementation following SkillsTrackerProtocol
    """
```

**Usage**:

```python
# Disabled
tracker = create_skills_tracker(session_id="test", enabled=False)
# Returns: NoOpSkillsTracker()

# Direct backend
tracker = create_skills_tracker(
    session_id="my-session",
    backend="direct",
    db_path=Path(".session-buddy/skills.db")
)
# Returns: SessionBuddyDirectTracker()

# MCP backend
tracker = create_skills_tracker(
    session_id="my-session",
    backend="mcp",
    mcp_server_url="http://localhost:8678"
)
# Returns: SessionBuddyMCPTracker()

# Auto detection (recommended)
tracker = create_skills_tracker(
    session_id="my-session",
    backend="auto"
)
# Returns: SessionBuddyMCPTracker() if available, else SessionBuddyDirectTracker()
```

______________________________________________________________________

## Data Migration

### Migrate from JSON to Dhara

**Legacy JSON-based metrics** → **Dhara SQLite database**

#### Migration Script

```bash
# Navigate to project root
cd /path/to/crackerjack

# Run dry-run (preview)
python scripts/migrate_skills_to_sessionbuddy.py --dry-run

# Actual migration
python scripts/migrate_skills_to_sessionbuddy.py

# Validate migration
python scripts/validate_skills_migration.py
```

#### Features

- ✅ **Automatic backup** - Creates `.pre-migration.backup` before modifying
- ✅ **Dry-run mode** - Preview changes without modifying database
- ✅ **JSON validation** - Checks structure and required fields
- ✅ **Progress tracking** - Real-time migration status
- ✅ **Rollback support** - Restore from backup if issues occur
- ✅ **Comprehensive logging** - Detailed migration log

#### Rollback

```bash
# Rollback to backup
python scripts/rollback_skills_migration.py

# Forced rollback (bypass confirmation)
python scripts/rollback_skills_migration.py --force
```

#### Validation

```bash
# Validate JSON structure only
python scripts/validate_skills_migration.py --json-only

# Validate database integrity only
python scripts/validate_skills_migration.py --db-only

# Full validation (JSON + database)
python scripts/validate_skills_migration.py
```

### Manual Data Export

```python
from session_buddy.core.skills_tracker import get_session_tracker

# Get tracker
tracker = get_session_tracker(session_id="export-session")

# Export to JSON (for backup/analysis)
import json
from datetime import datetime

export_data = {
    "exported_at": datetime.now().isoformat(),
    "invocations": tracker.get_all_invocations(),  # Pseudo-code
    "skills": tracker.get_all_skills()  # Pseudo-code
}

with open("skills_export.json", "w") as f:
    json.dump(export_data, f, indent=2)
```

______________________________________________________________________

## Performance

### Benchmarks

**Direct API (Tight Coupling)**:

| Metric | Value |
|--------|-------|
| Latency per invocation | < 1ms |
| Throughput | 10,000+ invocations/second |
| Memory overhead | ~5MB per session |
| CPU overhead | < 0.5% |

**MCP Bridge (Loose Coupling)**:

| Metric | Value |
|--------|-------|
| Latency per invocation | 5-10ms (network round-trip) |
| Throughput | 1,000+ invocations/second |
| Memory overhead | ~10MB per session (includes client) |
| CPU overhead | ~2% |

**No-Op (Disabled)**:

| Metric | Value |
|--------|-------|
| Latency | ~0.001µs (single null check) |
| Throughput | Unlimited |
| Memory | 0 bytes |

### Optimization Tips

1. **Use direct backend** for maximum performance
1. **Batch recommendations** - Get multiple at once instead of one-by-one
1. **Cache recommendations** - Same query returns same results for 1 hour
1. **Use auto backend** - Lets system choose optimal backend

### Scaling

**Single-Session**:

- Up to 100,000 invocations per session
- Recommended: Create new session daily

**Multi-Session**:

- Unlimited sessions (use unique session IDs)
- Each session isolated (no interference)

**Database Size**:

- ~500 bytes per invocation
- 100K invocations ≈ 50MB
- Prune old data to keep database small

______________________________________________________________________

## Troubleshooting

### Import Errors

**Problem**: `ImportError: session_buddy not available`

**Solution**:

```bash
# Install session-buddy
uv add session-buddy

# Verify installation
python -c "from session_buddy.core.skills_tracker import get_session_tracker; print('OK')"
```

### MCP Connection Failures

**Problem**: `Failed to connect to MCP server`

**Solution**:

```bash
# Check if MCP server running
python -m crackerjack status

# Test MCP health
curl http://localhost:8678/health

# MCP failures automatically fall back to direct tracking
# Check logs for fallback message
grep "fallback" ~/.cache/crackerjack/logs/debug/latest.log
```

### Database Locks

**Problem**: `sqlite3.OperationalError: database is locked`

**Solution**:

```bash
# Check for other processes using database
lsof .session-buddy/skills.db

# Kill other processes if needed
kill -9 <PID>

# Or use copy-on-write mode
export CRACKERJACK_SKILLS_DB_PATH=/tmp/skills-$(date +%s).db
```

### Slow Performance

**Problem**: Tracking adds significant overhead

**Solution**:

```bash
# Check which backend is being used
python -c "from crackerjack.config import CrackerjackSettings; s = CrackerjackSettings.load(); print(s.skills.backend)"

# Try direct backend for better performance
# In settings/local.yaml:
skills:
  backend: direct

# Or disable tracking temporarily
export CRACKERJACK_SKILLS_ENABLED=false
```

### No Recommendations Returned

**Problem**: `get_recommendations()` returns empty list

**Solution**:

```bash
# Check if tracking is enabled
python -c "from crackerjack.config import CrackerjackSettings; s = CrackerjackSettings.load(); print(s.skills.enabled)"

# Check database has data
sqlite3 .session-buddy/skills.db "SELECT COUNT(*) FROM skill_invocation;"

# Verify Akosha embeddings available
ls -la .session-buddy/embeddings/

# Lower min_similarity threshold
# In settings/local.yaml:
skills:
  min_similarity: 0.1  # More permissive matching
```

### Migration Issues

**Problem**: Migration fails or produces errors

**Solution**:

```bash
# Validate JSON first
python scripts/validate_skills_migration.py --json-only

# Check backup exists
ls -la .session-buddy/skills.db.pre-migration.backup*

# Rollback if needed
python scripts/rollback_skills_migration.py --force

# Try with verbose output
python scripts/migrate_skills_to_sessionbuddy.py --verbose
```

### Debug Mode

```bash
# Enable verbose logging
export CRACKERJACK_VERBOSE=1

# Check logs
tail -f ~/.cache/crackerjack/logs/debug/latest.log

# Test tracking directly
python -c "
from crackerjack.integration.skills_tracking import create_skills_tracker

tracker = create_skills_tracker(session_id='debug-test')
print(f'Backend: {tracker.get_backend()}')
print(f'Enabled: {tracker.is_enabled()}')

# Test tracking
completer = tracker.track_invocation(
    skill_name='TestAgent',
    user_query='Test query'
)
print(f'Completer: {completer}')

if completer:
    completer(completed=True)
    print('✅ Tracking successful')
"
```

______________________________________________________________________

## See Also

- **CLAUDE.md**: Complete developer documentation
- **README.md**: Skills tracking overview and quick start
- **`scripts/migrate_skills_to_sessionbuddy.py`**: Migration tool source
- **`tests/integration/test_skills_tracking.py`**: Integration tests
