# ADR-002: Multi-Agent Quality Check Orchestration

## Status

**Accepted** - 2025-01-20

## Context

Crackerjack's AI auto-fix feature needed to handle diverse code quality issues (type errors, security vulnerabilities, performance problems, test failures, etc.). A single monolithic AI agent was insufficient due to the specialized knowledge required for each issue type.

### Problem Statement

How should Crackerjack orchestrate multiple specialized AI agents to:

1. **Intelligently route issues** to the most appropriate agent
1. **Avoid redundant work** when multiple agents could handle the same issue
1. **Handle complex cross-cutting concerns** requiring multiple agents
1. **Maintain performance** with async parallel execution
1. **Provide confidence scores** for routing decisions
1. **Support batch processing** of related issues

### Key Requirements

- Agent selection must be automatic and data-driven
- Support for single-agent, parallel, sequential, and consensus execution strategies
- Confidence scoring to ensure best-match routing (≥0.7 threshold)
- Batch grouping of related issues for efficiency
- Fallback to safe defaults when no agent is suitable
- Real-time progress tracking for multi-agent workflows

## Decision Drivers

| Driver | Importance | Rationale |
|--------|------------|-----------|
| **Accuracy** | Critical | Right agent must handle the right issue |
| **Performance** | High | Parallel execution where possible |
| **Transparency** | High | Users should see which agents are used |
| **Extensibility** | High | Adding new agents should be trivial |
| **Reliability** | Critical | Fallback when agents fail |

## Considered Options

### Option 1: Single Monolithic Agent (Rejected)

**Description**: Use one large AI agent with if/else logic for different issue types.

**Pros**:

- Simple implementation
- Single API call
- Easier to debug

**Cons**:

- Prompt size grows unbounded
- Context window issues
- No specialization (security issues treated same as formatting)
- Hard to maintain
- Slower (re-analyzes for each issue)

**Decision**: Rejected due to lack of specialization and scaling issues.

### Option 2: Manual Agent Assignment (Rejected)

**Description**: User specifies which agent to use for each issue type.

**Pros**:

- Full user control
- Predictable behavior

**Cons**:

- High cognitive load on users
- Requires deep knowledge of agent capabilities
- No intelligent routing
- Prone to errors

**Decision**: Rejected due to poor developer experience.

### Option 3: Confidence-Based Multi-Agent Orchestration (SELECTED)

**Description**: Automatic agent selection with confidence scoring, batch processing, and multiple execution strategies.

**Pros**:

- **Intelligent Routing**: Confidence scores ensure best-match agents
- **Parallel Execution**: Independent agents run concurrently
- **Batch Processing**: Related issues grouped for efficiency
- **Fallback**: Safe defaults when no agent is suitable
- **Extensible**: Adding agents requires no code changes
- **Transparent**: Detailed logging of agent selection

**Cons**:

- More complex architecture
- Requires agent registry and scoring system
- Potential for conflicting fixes (mitigated by sequential mode)

**Decision**: Selected as best balance of intelligence, performance, and usability.

## Decision Outcome

### Architecture Overview

```
┌─────────────────────────────────────────────────────────────┐
│                    Issue Parser Layer                       │
│  (Parses error messages, categorizes by type)               │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│                  Agent Registry (12 Agents)                 │
│  ┌──────────────┬──────────────┬────────────────────────┐  │
│  │ Security     │ Refactoring  │ Performance            │  │
│  │ Agent        │ Agent        │ Agent                  │  │
│  ├──────────────┼──────────────┼────────────────────────┤  │
│  │ Test Creation│ Formatting   │ Documentation          │  │
│  │ Agent        │ Agent        │ Agent                  │  │
│  ├──────────────┼──────────────┼────────────────────────┤  │
│  │ Import Opt.  │ DRY          │ Test Specialist        │  │
│  │ Agent        │ Agent        │ Agent                  │  │
│  ├──────────────┼──────────────┼────────────────────────┤  │
│  │ Semantic     │ Architect    │ Enhanced Proactive     │  │
│  │ Agent        │ Agent        │ Agent                  │  │
│  └──────────────┴──────────────┴────────────────────────┘  │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│                  Agent Selector Engine                       │
│  ┌──────────────┬──────────────┬────────────────────────┐  │
│  │ Confidence   │ Batch        │ Execution              │  │
│  │ Scoring (≥0.7)│ Grouping    │ Strategy Selection     │  │
│  └──────────────┴──────────────┴────────────────────────┘  │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│                 Agent Orchestrator                          │
│  ┌──────────────┬──────────────┬────────────────────────┐  │
│  │ Single Best  │ Parallel     │ Sequential             │  │
│  │ Execution    │ Execution    │ Execution              │  │
│  └──────────────┴──────────────┴────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
```

### Key Components

#### 1. Agent Registry

**File**: `crackerjack/intelligence/agent_registry.py`

```python
class AgentRegistry:
    """Central registry of all available AI agents."""

    def __init__(self) -> None:
        self._agents: dict[str, RegisteredAgent] = {}

    def register(self, agent: RegisteredAgent) -> None:
        """Register a new agent."""
        self._agents[agent.agent_id] = agent

    def get_agent(self, agent_id: str) -> RegisteredAgent | None:
        """Retrieve agent by ID."""

    def list_agents(self, capability: str | None = None) -> list[RegisteredAgent]:
        """List all agents, optionally filtered by capability."""
```

**Registered Agents** (12 total):

| Agent ID | Name | Capabilities | Confidence Threshold |
|----------|------|--------------|---------------------|
| `security` | SecurityAgent | shell_injection, weak_crypto, token_exposure | 0.7 |
| `refactoring` | RefactoringAgent | complexity_reduction, solid_principles, extraction | 0.7 |
| `performance` | PerformanceAgent | algorithm_optimization, string_building, o_n_squared | 0.7 |
| `documentation` | DocumentationAgent | changelog_generation, markdown_consistency | 0.7 |
| `dry` | DRYAgent | deduplication, pattern_extraction | 0.7 |
| `formatting` | FormattingAgent | code_style, import_organization | 0.7 |
| `test_creation` | TestCreationAgent | test_fixtures, import_errors, assertions | 0.7 |
| `import_optimization` | ImportOptimizationAgent | unused_imports, import_restructuring | 0.7 |
| `test_specialist` | TestSpecialistAgent | advanced_scenarios, fixture_management | 0.7 |
| `semantic` | SemanticAgent | business_logic, code_comprehension | 0.7 |
| `architect` | ArchitectAgent | design_patterns, system_optimization | 0.7 |
| `enhanced_proactive` | EnhancedProactiveAgent | predictive_quality, prevention | 0.7 |

#### 2. Agent Selector

**File**: `crackerjack/intelligence/agent_selector.py`

```python
class AgentSelector:
    """Select best agents for tasks using confidence scoring."""

    async def select_agents(
        self,
        task: TaskDescription,
        max_candidates: int = 3,
    ) -> list[AgentScore]:
        """
        Select and rank agents for a task.

        Returns:
            List of (agent, confidence_score) tuples, sorted by score.
            Only agents with confidence ≥ 0.7 are returned.
        """
        candidates = []

        for agent in self.registry.list_agents():
            score = await self._calculate_confidence(agent, task)
            if score >= 0.7:
                candidates.append(AgentScore(agent=agent, score=score))

        return sorted(candidates, key=lambda x: x.score, reverse=True)[:max_candidates]

    async def _calculate_confidence(
        self,
        agent: RegisteredAgent,
        task: TaskDescription,
    ) -> float:
        """Calculate confidence score (0.0 to 1.0) for agent-task match."""
        # Semantic similarity between task and agent capabilities
        # Past success rate for similar tasks
        # Agent specialization depth
        # Task complexity vs agent expertise level
```

**Confidence Scoring Factors**:

1. **Capability Match** (40%): Semantic similarity between task and agent capabilities
1. **Past Success Rate** (30%): Historical success rate for similar tasks
1. **Specialization Depth** (20%): How specialized the agent is for this issue type
1. **Task Complexity** (10%): Match between task complexity and agent expertise

#### 3. Agent Orchestrator

**File**: `crackerjack/intelligence/agent_orchestrator.py`

```python
class AgentOrchestrator:
    """Orchestrate multi-agent execution with multiple strategies."""

    async def execute(self, request: ExecutionRequest) -> ExecutionResult:
        """Execute task using specified strategy."""
        candidates = await self.selector.select_agents(
            request.task,
            max_candidates=request.max_agents,
        )

        if not candidates:
            return self._create_error_result("No suitable agents found")

        if request.strategy == ExecutionStrategy.SINGLE_BEST:
            return await self._execute_single_best(request, candidates)
        elif request.strategy == ExecutionStrategy.PARALLEL:
            return await self._execute_parallel(request, candidates)
        elif request.strategy == ExecutionStrategy.SEQUENTIAL:
            return await self._execute_sequential(request, candidates)
        elif request.strategy == ExecutionStrategy.CONSENSUS:
            return await self._execute_consensus(request, candidates)
```

**Execution Strategies**:

##### 1. Single Best (Default)

```python
async def _execute_single_best(
    self,
    request: ExecutionRequest,
    candidates: list[AgentScore],
) -> ExecutionResult:
    """Execute using single highest-confidence agent."""
    best_agent_score = candidates[0]
    agent = best_agent_score.agent

    result = await agent.execute(request.task, request.context)

    return ExecutionResult(
        success=result.success,
        primary_result=result,
        all_results=[(agent, result)],
        agents_used=[agent.agent_id],
        strategy_used=ExecutionStrategy.SINGLE_BEST,
    )
```

**Use Cases**:

- Clear issue type (e.g., type error → TestCreationAgent)
- Independent issues that don't affect each other
- Fastest execution (single agent)

##### 2. Parallel Execution

```python
async def _execute_parallel(
    self,
    request: ExecutionRequest,
    candidates: list[AgentScore],
) -> ExecutionResult:
    """Execute multiple agents concurrently."""
    tasks = [
        agent.execute(request.task, request.context)
        for agent_score in candidates
        for agent in [agent_score.agent]
    ]

    results = await asyncio.gather(*tasks, return_exceptions=True)

    return ExecutionResult(
        success=all(r.success for r in results if isinstance(r, BaseResult)),
        primary_result=results[0],  # First result as primary
        all_results=list(zip(candidates, results)),
        agents_used=[c.agent.agent_id for c in candidates],
        strategy_used=ExecutionStrategy.PARALLEL,
    )
```

**Use Cases**:

- Multiple independent fixes (e.g., formatting + linting)
- Exploratory fixes (try multiple approaches)
- Performance optimization (all agents run concurrently)

##### 3. Sequential Execution

```python
async def _execute_sequential(
    self,
    request: ExecutionRequest,
    candidates: list[AgentScore],
) -> ExecutionResult:
    """Execute agents in order of confidence."""
    results = []
    context = request.context

    for agent_score in candidates:
        agent = agent_score.agent
        result = await agent.execute(request.task, context)

        results.append((agent, result))

        # Update context for next agent
        if result.modified_files:
            context = context.with_modified_files(result.modified_files)

        # Stop if successful
        if result.success:
            break

    return ExecutionResult(
        success=results[-1][1].success,
        primary_result=results[-1][1],
        all_results=results,
        agents_used=[a.agent_id for a, _ in results],
        strategy_used=ExecutionStrategy.SEQUENTIAL,
    )
```

**Use Cases**:

- Cascading fixes (formatting → type checking → security)
- When earlier fixes enable later fixes
- Avoiding conflicting changes

##### 4. Consensus Execution

```python
async def _execute_consensus(
    self,
    request: ExecutionRequest,
    candidates: list[AgentScore],
) -> ExecutionResult:
    """Execute all agents and require consensus."""
    results = await asyncio.gather(*[
        agent.execute(request.task, request.context)
        for agent_score in candidates
        for agent in [agent_score.agent]
    ])

    # Check if all agents agree on the fix
    if len(set(r.fix_hash for r in results)) == 1:
        # Consensus reached
        return ExecutionResult(
            success=True,
            primary_result=results[0],
            all_results=list(zip(candidates, results)),
            agents_used=[c.agent.agent_id for c in candidates],
            strategy_used=ExecutionStrategy.CONSENSUS,
        )
    else:
        # No consensus - return all results for manual review
        return ExecutionResult(
            success=False,
            primary_result=None,
            all_results=list(zip(candidates, results)),
            agents_used=[c.agent.agent_id for c in candidates],
            strategy_used=ExecutionStrategy.CONSENSUS,
            error_message="No consensus reached",
        )
```

**Use Cases**:

- Critical security fixes requiring verification
- Architectural changes requiring multiple perspectives
- High-stakes refactoring

#### 4. Batch Processing

**File**: `crackerjack/intelligence/batch_processor.py`

```python
class BatchProcessor:
    """Group related issues for efficient batch processing."""

    async def group_issues(
        self,
        issues: list[Issue],
    ) -> dict[str, list[Issue]]:
        """
        Group related issues by file, issue type, and agent.

        Returns:
            Dictionary mapping agent_id to list of issues.
        """
        batches: dict[str, list[Issue]] = {}

        for issue in issues:
            # Select best agent for this issue
            agent = await self.selector.select_best_agent(issue)

            if agent.agent_id not in batches:
                batches[agent.agent_id] = []

            batches[agent.agent_id].append(issue)

        return batches

    async def execute_batches(
        self,
        batches: dict[str, list[Issue]],
    ) -> dict[str, list[BaseResult]]:
        """Execute each batch with its assigned agent."""
        results = {}

        for agent_id, issues in batches.items():
            agent = self.registry.get_agent(agent_id)

            # Execute all issues for this agent in one call
            result = await agent.execute_batch(issues)

            results[agent_id] = result

        return results
```

**Batching Strategies**:

1. **By File**: All issues in the same file processed together
1. **By Issue Type**: All type errors processed together
1. **By Agent**: All issues handled by the same agent processed together
1. **By Dependency**: Issues that depend on each other processed sequentially

### Configuration

**File**: `settings/agents.yml`

```yaml
# Agent orchestration configuration
agents:
  # Confidence threshold for agent selection
  min_confidence: 0.7

  # Maximum agents to consider for a task
  max_candidates: 3

  # Default execution strategy
  default_strategy: "single_best"  # single_best, parallel, sequential, consensus

  # Batch processing
  batch_enabled: true
  batch_size: 10  # Max issues per batch
  batch_strategy: "by_agent"  # by_file, by_type, by_agent, by_dependency

  # Timeout for agent execution
  timeout_seconds: 300

  # Fallback to safe defaults when no agent is suitable
  fallback_to_safe: true
```

### Usage Examples

#### Automatic Agent Selection

```python
from crackerjack.intelligence import AgentOrchestrator, TaskDescription

orchestrator = AgentOrchestrator()

# Define task
task = TaskDescription(
    description="Fix shell injection in subprocess.call",
    issue_type="security",
    file_path="src/utils.py",
    line_number=42,
)

# Execute with single best agent (default)
request = ExecutionRequest(
    task=task,
    strategy=ExecutionStrategy.SINGLE_BEST,
)

result = await orchestrator.execute(request)

# Result will use SecurityAgent with high confidence
print(f"Agent used: {result.agents_used}")  # ['security']
print(f"Success: {result.success}")
```

#### Parallel Execution

```python
# Multiple independent issues
request = ExecutionRequest(
    task=TaskDescription(
        description="Fix formatting and linting issues",
        issue_type="code_style",
    ),
    strategy=ExecutionStrategy.PARALLEL,
    max_agents=3,  # Use up to 3 agents concurrently
)

result = await orchestrator.execute(request)

# FormattingAgent and LintAgent run in parallel
print(f"Agents used: {result.agents_used}")  # ['formatting', 'lint']
```

#### Sequential Execution

```python
# Cascading fixes
request = ExecutionRequest(
    task=TaskDescription(
        description="Fix type errors after refactoring",
        issue_type="type_error",
    ),
    strategy=ExecutionStrategy.SEQUENTIAL,
)

result = await orchestrator.execute(request)

# RefactoringAgent → TestCreationAgent → TypeCheckAgent
print(f"Agents used: {result.agents_used}")
# ['refactoring', 'test_creation', 'type_check']
```

#### Batch Processing

```python
from crackerjack.intelligence import BatchProcessor

batch_processor = BatchProcessor()

# Group 50 issues by agent
issues = parse_error_file("errors.txt")
batches = await batch_processor.group_issues(issues)

# Execute batches
results = await batch_processor.execute_batches(batches)

# Output:
# SecurityAgent: 12 issues processed in 3.2s
# FormattingAgent: 18 issues processed in 2.1s
# TestCreationAgent: 20 issues processed in 5.4s
```

## Consequences

### Positive

1. **Intelligent Routing**: Confidence scores ensure best-match agents (≥0.7 threshold)
1. **Performance**: Parallel execution for independent issues (3x speedup)
1. **Efficiency**: Batch processing reduces API calls by 60%
1. **Transparency**: Detailed logging shows which agents were used and why
1. **Extensibility**: Adding new agents requires no code changes to orchestrator
1. **Flexibility**: 4 execution strategies for different use cases
1. **Reliability**: Fallback to safe defaults when no agent is suitable

### Negative

1. **Complexity**: More sophisticated architecture than single agent
1. **Debugging**: Harder to debug when multiple agents are involved
1. **Conflicting Fixes**: Parallel agents may make conflicting changes (mitigated by sequential mode)
1. **Confidence Calibration**: Requires ongoing tuning of confidence scoring
1. **Memory**: Batch processing can consume memory for large issue sets

### Risks

| Risk | Mitigation |
|------|------------|
| Confidence score is wrong | Log all decisions for analysis; continuously calibrate |
| Parallel agents conflict | Use sequential mode for dependent fixes |
| Batch processing OOM | Limit batch size to 10 issues per batch |
| Agent timeout | Set 300s timeout per agent; fallback to next agent |

## Performance Metrics

**Benchmark Results** (100 issues from real project):

| Strategy | Time | Speedup | API Calls |
|----------|------|---------|-----------|
| Single Agent (Baseline) | 450s | 1.0x | 100 |
| Single Best | 420s | 1.07x | 100 |
| Parallel (3 agents) | 150s | **3.0x** | 100 |
| Sequential | 380s | 1.18x | 95 |
| Consensus | 460s | 0.98x | 100 |
| **Batch Processing** | **180s** | **2.5x** | **40** |

**Key Findings**:

- Parallel execution provides 3x speedup for independent issues
- Batch processing reduces API calls by 60%
- Sequential mode is 18% faster than single agent due to cascading fixes
- Consensus is slower but provides verification for critical fixes

## Related Decisions

- **ADR-001**: MCP-first architecture with FastMCP
- **ADR-003**: Property-based testing with Hypothesis
- **ADR-004**: Quality gate threshold system
- **ADR-005**: Agent skill routing and selection

## References

- [Agent Coordination Architecture](../AGENT_COORDINATION_ARCHITECTURE_ANALYSIS.md)
- [Multi-Agent Review Summary](../archive/completion-reports/MULTI_AGENT_REVIEW_SUMMARY.md)
- [Agent System Implementation](../../crackerjack/agents/)

## Revision History

| Date | Author | Changes |
|------|--------|---------|
| 2025-01-20 | Les Leslie | Initial ADR creation |
| 2025-01-25 | Les Leslie | Added consensus execution strategy |
| 2025-02-01 | Les Leslie | Added performance metrics and benchmarks |
