# Dataknobs LLM - Best Practices Guide

**Package**: `dataknobs_llm`
**Version**: 0.1.0
**Last Updated**: 2025-10-29

---

## Table of Contents

1. [Prompt Design](#prompt-design)
2. [Template Organization](#template-organization)
3. [Validation Strategy](#validation-strategy)
4. [RAG Best Practices](#rag-best-practices)
5. [Conversation Management](#conversation-management)
6. [Middleware Patterns](#middleware-patterns)
7. [Storage and Performance](#storage-and-performance)
8. [Error Handling](#error-handling)
9. [Testing](#testing)
10. [Production Deployment](#production-deployment)
    - [Productionizing a Tool-Using Bot](#productionizing-a-tool-using-bot)
11. [Provider Lifecycle](#provider-lifecycle)
12. [Response Analysis Hooks](#response-analysis-hooks)

---

## Prompt Design

### Write Clear, Focused Prompts

**Good**:
```yaml
# system/code_reviewer.yaml
template: |
  You are an expert code reviewer specializing in {{language}}.

  Review code for:
  1. Correctness and bugs
  2. Best practices and style
  3. Security vulnerabilities
  4. Performance issues

  Provide specific, actionable feedback.

defaults:
  language: "Python"
```

**Avoid**:
```yaml
# Too vague, no clear instructions
template: "You are helpful. Do code stuff."
```

### Use Conditional Sections Wisely

**Good** - Optional context:
```yaml
template: |
  Analyze this {{language}} code((, following {{style_guide}})):

  {{code}}

  ((Focus on {{aspect}}.))
```

**Avoid** - Nested conditionals that are hard to read:
```yaml
# Too complex
template: "Hello (({{name}}((, age {{age}}((, from {{city}}((, in {{country}}))))))))."
```

**Recommendation**: Limit conditional nesting to 2 levels maximum.

### Provide Sensible Defaults

**Good**:
```yaml
template: "Analyze {{code}} for {{aspect}}"
defaults:
  aspect: "correctness and style"  # Reasonable default
validation:
  level: "warn"
  required_params: ["code"]  # Only truly required params
```

**Avoid**:
```yaml
# No defaults, everything required
validation:
  level: "error"
  required_params: ["code", "aspect", "language", "style", "format"]
```

**Recommendation**: Make parameters required only if they're truly essential.

### Use Descriptive Variable Names

**Good**:
```yaml
template: "Review {{source_code}} for {{security_issue_types}}"
```

**Avoid**:
```yaml
template: "Review {{x}} for {{y}}"  # Unclear what x and y are
```

### Structure Long Prompts with Sections

**Good**:
```yaml
template: |
  {{ROLE}}

  {{INSTRUCTIONS}}

  {{EXAMPLES}}

  {{INPUT}}

sections:
  ROLE: "You are an expert {{domain}} consultant."
  INSTRUCTIONS: |
    Provide analysis following these steps:
    1. Identify key patterns
    2. Assess quality
    3. Suggest improvements
  EXAMPLES: "((Example: {{example_text}}))"
  INPUT: "Input to analyze:\n{{content}}"
```

**Benefit**: Easier to read, maintain, and override specific sections.

---

## Template Organization

### Directory Structure

**Recommended structure**:
```
prompts/
├── system/           # System prompts (one per file)
│   ├── base_assistant.yaml
│   ├── code_reviewer.yaml
│   └── customer_support.yaml
├── user/             # User prompts (by name)
│   ├── initial_question.yaml
│   ├── detailed_question.yaml
│   ├── submit_code.yaml
│   └── follow_up.yaml
├── rag/              # RAG configurations
│   ├── docs_search.yaml
│   ├── code_search.yaml
│   └── faq_search.yaml
└── README.md         # Documentation
```

### Naming Conventions

**System prompts**: `{purpose}.yaml`
- `code_reviewer.yaml`
- `customer_support.yaml`
- `data_analyst.yaml`

**User prompts**: `{descriptive_name}.yaml`
- `initial_question.yaml` - Basic question
- `detailed_question.yaml` - Detailed question with context
- `submit_code.yaml` - Code submission prompt

**RAG configs**: `{source}_search.yaml`
- `docs_search.yaml`
- `code_search.yaml`

### Use Template Inheritance

**Base template** (system/base_assistant.yaml):
```yaml
template: |
  {{ROLE}}

  {{CAPABILITIES}}

  {{GUIDELINES}}

sections:
  ROLE: "You are a helpful AI assistant."
  CAPABILITIES: "You can answer questions and provide information."
  GUIDELINES: "Be accurate, concise, and helpful."
```

**Specialized template** (system/code_assistant.yaml):
```yaml
extends: "base_assistant"

sections:
  ROLE: "You are an expert software engineer and code reviewer."
  CAPABILITIES: |
    You can:
    - Review code for bugs and best practices
    - Explain programming concepts
    - Suggest improvements and optimizations
```

**Benefits**:
- DRY (Don't Repeat Yourself)
- Consistent structure across prompts
- Easy to update common elements
- Clear hierarchy

### Use Composite Libraries for Environments

**Pattern**:
```python
from dataknobs_llm.prompts import (
    FileSystemPromptLibrary,
    CompositePromptLibrary
)

# Environment-specific overrides
prod_library = FileSystemPromptLibrary("prompts/production/")
base_library = FileSystemPromptLibrary("prompts/base/")

# Production uses specific prompts, falls back to base
library = CompositePromptLibrary(
    libraries=[prod_library, base_library],
    names=["production", "base"]
)
```

**Use cases**:
- Development vs production prompts
- A/B testing variants
- Customer-specific customizations
- Feature flags

---

## Validation Strategy

### Choose Appropriate Validation Levels

**ERROR** - For critical parameters:
```yaml
# system/database_query.yaml
template: "Execute query on {{database}}: {{query}}"
validation:
  level: "error"  # Missing these would cause failures
  required_params: ["database", "query"]
```

**WARN** - For optional but recommended parameters:
```yaml
# system/analysis.yaml
template: "Analyze {{data}}((, focusing on {{aspect}}))"
validation:
  level: "warn"  # Missing aspect is okay, but user should know
  required_params: []
  optional_params: ["aspect"]
```

**IGNORE** - For always-optional parameters:
```yaml
# system/chat.yaml
template: "Chat about (({{topic}}))"
validation:
  level: "ignore"  # Truly optional, no warnings needed
```

### Validation Hierarchy

```python
# 1. Template defines defaults
# prompts/system/analyze.yaml
validation:
  level: "warn"  # Default for this template

# 2. Builder can override
builder = AsyncPromptBuilder(
    library=library,
    default_validation_level=ValidationLevel.ERROR  # Override for all prompts
)

# 3. Runtime can override
result = await builder.render_system_prompt(
    "analyze",
    params={...},
    validation_level=ValidationLevel.IGNORE  # Override for this call
)
```

**Recommendation**: Use template-level validation for most cases, runtime overrides sparingly.

### Document Required Parameters

```yaml
template: "Analyze {{code}} for {{issues}}"

# Clear documentation
validation:
  level: "error"
  required_params: ["code", "issues"]

# Add metadata for tools/docs
metadata:
  description: "Code analysis prompt"
  parameters:
    code:
      type: "string"
      description: "Source code to analyze"
    issues:
      type: "array"
      description: "List of issue types to check"
      examples: ["security", "performance", "style"]
```

---

## RAG Best Practices

### Separate RAG Configs from Prompts

**Good** - Reusable RAG config:
```yaml
# rag/code_search.yaml
adapter_name: "codebase"
query: "{{language}} {{topic}}"
k: 5
filters:
  type: "code"
score_threshold: 0.7

# system/code_helper.yaml
template: |
  Relevant code examples:
  {{RAG_CONTENT}}

  Help with: {{question}}

rag_config_refs: ["code_search"]  # Reference
```

**Benefits**:
- Reuse across multiple prompts
- Easy to test different RAG strategies
- Swap configs at runtime

### Parameterize RAG Queries

**Good**:
```yaml
# rag/docs_search.yaml
query: "{{topic}} {{language}} {{framework}}"
k: 5
```

**Avoid**:
```yaml
# Too specific, not reusable
query: "Python async programming with asyncio"
```

### Set Appropriate k and Thresholds

```yaml
# For broad context
k: 10
score_threshold: 0.5

# For precise results
k: 3
score_threshold: 0.8

# For maximum recall
k: 20
score_threshold: 0.3
```

**Recommendation**: Start with `k=5, score_threshold=0.7` and tune based on results.

### Handle RAG Failures Gracefully

```python
try:
    result = await builder.render_system_prompt(
        "prompt_with_rag",
        params={...},
        include_rag=True
    )
except Exception as e:
    logger.error(f"RAG search failed: {e}")
    # Fallback: render without RAG
    result = await builder.render_system_prompt(
        "prompt_with_rag",
        params={...},
        include_rag=False
    )
```

### Format RAG Content Clearly

**Prompt template**:
```yaml
template: |
  Use these reference documents:

  {{RAG_CONTENT}}

  ---

  Question: {{question}}

  Answer based on the documents above.
```

**Benefit**: Clear separation between RAG content and user input.

---

## Conversation Management

### Design Conversation State Carefully

**Good** - Meaningful metadata:
```python
manager = await ConversationManager.create(
    llm=llm,
    prompt_builder=builder,
    storage=storage,
    metadata={
        "user_id": user_id,
        "session_id": session_id,
        "conversation_type": "code_review",
        "language": "python",
        "started_at": datetime.now().isoformat()
    }
)
```

**Benefits**:
- Easy querying and filtering
- Analytics and tracking
- Context for resumption

### Use Branching for Exploration

**Pattern**:
```python
# Main conversation path
await manager.add_message(role="user", content="Explain lists")
response1 = await manager.complete()

# Try alternative explanation
await manager.switch_to_node("0")
response2 = await manager.complete(branch_name="simple-explanation")

# Compare approaches
branches = manager.get_branches()
# Pick best branch or show user options
```

**Use cases**:
- A/B testing responses
- Exploring different solutions
- Providing alternatives to users
- Debugging conversation flows

### Implement Conversation Limits

**Pattern**:
```python
async def safe_complete(manager, max_turns=50):
    messages = manager.state.get_current_messages()

    if len(messages) > max_turns:
        # Summarize or warn
        logger.warning(f"Conversation exceeding {max_turns} turns")
        # Consider summarization
        return await summarize_and_continue(manager)

    return await manager.complete()
```

**Recommendation**: Set limits based on:
- Token limits of your LLM
- Cost considerations
- User experience (very long conversations are confusing)

### Clean Up Old Conversations

**Pattern**:
```python
from datetime import datetime, timedelta

async def cleanup_old_conversations(storage, days=30):
    """Delete conversations older than specified days."""
    cutoff = datetime.now() - timedelta(days=days)

    conversations = await storage.list_conversations(
        sort_by="updated_at",
        sort_order="asc",
        limit=1000,
    )

    for conv in conversations:
        if conv.updated_at < cutoff:
            await storage.delete_conversation(conv.conversation_id)
            logger.info(f"Deleted old conversation: {conv.conversation_id}")
```

**Recommendation**: Implement regular cleanup or archival strategy.

### Save Important Checkpoints

**Pattern**:
```python
# After important milestones
await manager.add_message(role="user", content="Approve this solution")
response = await manager.complete()

# Save current state
conv_id = manager.conversation_id
checkpoint_node = manager.state.current_node_id

# Store for later resumption
save_checkpoint(user_id, conv_id, checkpoint_node)
```

---

## Middleware Patterns

### Order Middleware Thoughtfully

**Good order**:
```python
middleware = [
    LoggingMiddleware(logger),          # Log everything first
    ContentFilterMiddleware(...),        # Filter early
    ValidationMiddleware(...),           # Validate after filtering
    MetadataMiddleware(...)              # Add metadata last
]
```

**Reasoning**:
- Logging captures all requests/responses
- Filtering removes problematic content early
- Validation checks cleaned content
- Metadata added after all processing

### Use Validation Middleware Selectively

**Good** - Only for critical conversations:
```python
# Production customer support - validate responses
validation_mw = ValidationMiddleware(
    llm=validation_llm,
    prompt_builder=builder,
    validation_prompt="validate_support_response",
    auto_retry=True
)

# Internal testing - skip validation (save costs)
middleware = [LoggingMiddleware(logger)] if env == "prod" else []
```

**Costs**: ValidationMiddleware doubles LLM calls (one for response, one for validation).

### Implement Custom Middleware for Business Logic

**Example - Rate limiting**:

Use the built-in `RateLimitMiddleware` which delegates to `InMemoryRateLimiter` from `dataknobs-common`:

```python
from dataknobs_llm.conversations import RateLimitMiddleware

rate_limiter = RateLimitMiddleware(
    max_requests=10,
    window_seconds=60,
    scope="conversation",  # or "client" for per-client limiting
)

manager = ConversationManager(
    llm=provider,
    middleware=[rate_limiter],
)
```

When the limit is exceeded, `RateLimitError` is raised with a `retry_after` attribute indicating when to retry. See the [Rate Limiting guide](../../common/ratelimit.md) for the underlying `InMemoryRateLimiter` API.

**Example - Token counting**:
```python
class TokenBudgetMiddleware(ConversationMiddleware):
    def __init__(self, max_tokens=100000):
        self.max_tokens = max_tokens
        self.used_tokens = 0

    async def process_response(self, response, state):
        if response.usage:
            self.used_tokens += response.usage.get("total_tokens", 0)

            if self.used_tokens > self.max_tokens:
                logger.warning(f"Token budget exceeded: {self.used_tokens}/{self.max_tokens}")

            response.metadata["cumulative_tokens"] = self.used_tokens

        return response
```

### Handle Middleware Errors

**Pattern**:
```python
class SafeValidationMiddleware(ConversationMiddleware):
    async def process_response(self, response, state):
        try:
            # Validation logic
            is_valid = await self._validate(response)

            if not is_valid:
                response.metadata["validation_failed"] = True
                logger.warning("Validation failed")

        except Exception as e:
            # Don't fail entire request on validation error
            logger.error(f"Validation error: {e}")
            response.metadata["validation_error"] = str(e)

        return response
```

**Recommendation**: Decide per-middleware whether errors should:
- Fail the entire request (critical middleware)
- Log and continue (nice-to-have middleware)

---

## Storage and Performance

### Choose Storage Backend Based on Needs

**Development/Testing**:
```python
from dataknobs_data.backends import AsyncMemoryDatabase

storage = DataknobsConversationStorage(AsyncMemoryDatabase())
```
- Fast, no setup
- Data lost on restart
- Perfect for tests

**Local Development**:
```python
from dataknobs_data.backends import AsyncSQLiteDatabase

storage = DataknobsConversationStorage(
    AsyncSQLiteDatabase(db_path="conversations.db")
)
```
- Persistent across restarts
- No external dependencies
- Good for local development

**Production (Small Scale)**:
```python
from dataknobs_data.backends import AsyncSQLiteDatabase

storage = DataknobsConversationStorage(
    AsyncSQLiteDatabase(db_path="/var/data/conversations.db")
)
```
- Simple deployment
- Good for <100k conversations
- Single server setups

**Production (Large Scale)**:
```python
from dataknobs_data.backends import AsyncPostgresDatabase

storage = DataknobsConversationStorage(
    AsyncPostgresDatabase(
        connection_string="postgresql://user:pass@host:5432/db"
    )
)
```
- Scalable to millions of conversations
- ACID guarantees
- Advanced querying
- Suitable for distributed systems

### Implement Caching for Frequent Accesses

**Pattern**:
```python
from functools import lru_cache

class CachedConversationManager:
    def __init__(self, storage, cache_size=100):
        self.storage = storage
        self.cache = {}  # conversation_id -> state

    async def get_conversation(self, conversation_id):
        # Check cache first
        if conversation_id in self.cache:
            return self.cache[conversation_id]

        # Load from storage
        state = await self.storage.load_conversation(conversation_id)

        # Cache it
        if state:
            self.cache[conversation_id] = state

        return state

    async def save_conversation(self, state):
        # Update cache
        self.cache[state.conversation_id] = state

        # Persist to storage
        await self.storage.save_conversation(state)
```

**Benefits**:
- Reduces database load
- Faster conversation access
- Important for high-traffic applications

### Batch Operations When Possible

**Good**:
```python
# Process multiple conversations in parallel
async def process_batch(conversation_ids):
    tasks = [
        process_conversation(conv_id)
        for conv_id in conversation_ids
    ]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    return results
```

**Avoid**:
```python
# Sequential processing
for conv_id in conversation_ids:
    await process_conversation(conv_id)  # Slow!
```

### Monitor Performance

**Pattern**:
```python
import time

class PerformanceMiddleware(ConversationMiddleware):
    async def process_request(self, messages, state):
        state.metadata["request_start"] = time.time()
        return messages

    async def process_response(self, response, state):
        duration = time.time() - state.metadata.get("request_start", 0)

        response.metadata["llm_duration_seconds"] = duration

        if duration > 5.0:
            logger.warning(f"Slow LLM response: {duration}s")

        return response
```

---

## Error Handling

### Handle Missing Prompts Gracefully

**Pattern**:
```python
from dataknobs_llm.prompts import PromptNotFoundError

async def safe_render(builder, prompt_name, params, fallback_template=None):
    try:
        return await builder.render_user_prompt(prompt_name, params=params)
    except PromptNotFoundError:
        logger.warning(f"Prompt '{prompt_name}' not found")

        if fallback_template:
            # Use fallback
            from dataknobs_llm.prompts import render_template
            return render_template(fallback_template, params)

        raise  # Re-raise if no fallback
```

### Validate Parameters Early

**Good**:
```python
def validate_params(params, required):
    missing = [p for p in required if p not in params]
    if missing:
        raise ValueError(f"Missing required parameters: {missing}")

async def render_safely(builder, prompt_name, params):
    # Validate early
    validate_params(params, ["code", "language"])

    # Then render
    return await builder.render_user_prompt(prompt_name, params=params)
```

**Benefits**:
- Clear error messages
- Fail fast
- Easier debugging

### Handle LLM Errors with Retries

**Pattern**:
```python
import asyncio

async def complete_with_retry(manager, max_retries=3, backoff=1.0):
    for attempt in range(max_retries):
        try:
            return await manager.complete()
        except Exception as e:
            if attempt == max_retries - 1:
                # Final attempt failed
                logger.error(f"LLM failed after {max_retries} retries: {e}")
                raise

            # Exponential backoff
            wait = backoff * (2 ** attempt)
            logger.warning(f"LLM attempt {attempt+1} failed, retrying in {wait}s")
            await asyncio.sleep(wait)
```

### Log Errors with Context

**Good**:
```python
try:
    result = await manager.complete()
except Exception as e:
    logger.error(
        "LLM completion failed",
        extra={
            "conversation_id": manager.conversation_id,
            "message_count": len(manager.state.get_current_messages()),
            "error": str(e),
            "user_id": manager.state.metadata.get("user_id")
        }
    )
    raise
```

**Benefits**:
- Easier debugging
- Better monitoring
- Context for error analysis

---

## Testing

### Test Prompts in Isolation

**Pattern**:
```python
import pytest
from dataknobs_llm.prompts import render_template

def test_code_review_prompt():
    template = "Review {{language}} code:\n{{code}}"

    result = render_template(
        template,
        {"language": "Python", "code": "def test(): pass"}
    )

    assert "Review Python code:" in result.content
    assert "def test(): pass" in result.content

def test_optional_params():
    template = "Hello {{name}}((, age {{age}}))"

    # With optional param
    result = render_template(template, {"name": "Alice", "age": 30})
    assert "age 30" in result.content

    # Without optional param
    result = render_template(template, {"name": "Alice"})
    assert "age" not in result.content
```

### Use EchoProvider for Testing

**Pattern**:
```python
from dataknobs_llm.llm import EchoProvider

@pytest.mark.asyncio
async def test_conversation_flow():
    # Echo provider returns input for predictable testing
    llm = EchoProvider(config={"echo_prefix": "Echo: "})

    manager = await ConversationManager.create(
        llm=llm,
        prompt_builder=builder,
        storage=storage
    )

    await manager.add_message(role="user", content="Hello")
    response = await manager.complete()

    # Predictable response
    assert "Echo: " in response.content
    assert "Hello" in response.content
```

### Test Middleware Behavior

**Pattern**:
```python
@pytest.mark.asyncio
async def test_logging_middleware(caplog):
    logger = logging.getLogger("test")
    middleware = LoggingMiddleware(logger)

    manager = await ConversationManager.create(
        llm=echo_llm,
        prompt_builder=builder,
        storage=storage,
        middleware=[middleware]
    )

    with caplog.at_level(logging.INFO):
        await manager.add_message(role="user", content="Test")
        await manager.complete()

    # Check logs
    assert "Sending" in caplog.text
    assert "Received response" in caplog.text
```

### Test Error Conditions

**Pattern**:
```python
@pytest.mark.asyncio
async def test_missing_required_param():
    from dataknobs_llm.prompts import ValidationLevel

    with pytest.raises(ValueError, match="Missing required parameter"):
        result = await builder.render_user_prompt(
            "prompt_with_required_param",
            params={},  # Missing required param
            validation_level=ValidationLevel.ERROR
        )
```

### Mock External Dependencies

**Pattern**:
```python
from unittest.mock import AsyncMock, MagicMock

@pytest.mark.asyncio
async def test_with_mocked_adapter():
    # Mock adapter
    mock_adapter = AsyncMock()
    mock_adapter.search.return_value = [
        {"content": "Test result", "score": 0.9}
    ]

    builder = AsyncPromptBuilder(
        library=library,
        adapters={"docs": mock_adapter}
    )

    result = await builder.render_system_prompt(
        "prompt_with_rag",
        params={"topic": "testing"}
    )

    # Verify adapter was called
    mock_adapter.search.assert_called_once()
```

---

## Production Deployment

### Use Environment-Specific Configurations

**Pattern**:
```python
import os

def get_storage(env=None):
    env = env or os.getenv("ENV", "development")

    if env == "production":
        return DataknobsConversationStorage(
            AsyncPostgresDatabase(
                connection_string=os.getenv("DATABASE_URL")
            )
        )
    elif env == "staging":
        return DataknobsConversationStorage(
            AsyncPostgresDatabase(
                connection_string=os.getenv("STAGING_DATABASE_URL")
            )
        )
    else:  # development
        return DataknobsConversationStorage(
            AsyncSQLiteDatabase(db_path="dev_conversations.db")
        )
```

### Implement Health Checks

**Pattern**:
```python
async def health_check():
    """Check if conversation system is healthy."""
    try:
        # Test storage
        test_state = ConversationState(
            conversation_id="health-check",
            message_tree=Tree(ConversationNode(...))
        )
        await storage.save_conversation(test_state)
        loaded = await storage.load_conversation("health-check")
        await storage.delete_conversation("health-check")

        if not loaded:
            return {"status": "unhealthy", "reason": "storage failed"}

        # Test LLM
        response = await llm.complete([
            LLMMessage(role="user", content="health check")
        ])

        if not response:
            return {"status": "unhealthy", "reason": "llm failed"}

        return {"status": "healthy"}

    except Exception as e:
        return {"status": "unhealthy", "reason": str(e)}
```

### Monitor Token Usage

**Pattern**:
```python
class TokenMonitoringMiddleware(ConversationMiddleware):
    def __init__(self, alert_threshold=50000):
        self.alert_threshold = alert_threshold
        self.total_tokens = 0

    async def process_response(self, response, state):
        if response.usage:
            tokens = response.usage.get("total_tokens", 0)
            self.total_tokens += tokens

            # Log to metrics system
            metrics.increment("llm.tokens", tokens)

            # Alert if high usage
            if tokens > self.alert_threshold:
                logger.warning(f"High token usage: {tokens}")

            response.metadata["total_tokens"] = tokens

        return response
```

### Implement Rate Limiting

Use `InMemoryRateLimiter` from `dataknobs-common` instead of building custom rate limiters:

```python
from dataknobs_common.ratelimit import (
    InMemoryRateLimiter, RateLimit, RateLimiterConfig,
)
from dataknobs_common.exceptions import RateLimitError

# Configure per-category rates
config = RateLimiterConfig(
    default_rates=[RateLimit(limit=100, interval=60)],
    categories={
        "premium_user": [RateLimit(limit=200, interval=60)],
    },
)
limiter = InMemoryRateLimiter(config)

# Usage — non-blocking check
async def handle_request(user_id: str, message: str):
    category = get_user_tier(user_id)  # e.g. "premium_user" or "default"
    if not await limiter.try_acquire(category):
        status = await limiter.get_status(category)
        raise RateLimitError(
            "Rate limit exceeded", retry_after=status.reset_after,
        )
    # Process request...
```

For conversation middleware, use the built-in `RateLimitMiddleware` (see Middleware section above) which wraps `InMemoryRateLimiter` with conversation-aware scoping.

### Secure API Keys

**Good**:
```python
import os

# Load from environment
llm = OpenAIProvider(config={
    "api_key": os.getenv("OPENAI_API_KEY")
})
```

**Avoid**:
```python
# Hardcoded keys - NEVER DO THIS
llm = OpenAIProvider(config={
    "api_key": "sk-hardcoded-key-12345"
})
```

### Implement Graceful Shutdown

**Pattern**:
```python
import signal
import asyncio

class ConversationService:
    def __init__(self):
        self.shutdown_event = asyncio.Event()

    async def run(self):
        # Setup signal handlers
        signal.signal(signal.SIGINT, self._signal_handler)
        signal.signal(signal.SIGTERM, self._signal_handler)

        # Run until shutdown
        await self.shutdown_event.wait()

        # Cleanup
        await self.cleanup()

    def _signal_handler(self, signum, frame):
        logger.info("Shutdown signal received")
        self.shutdown_event.set()

    async def cleanup(self):
        # Close LLM connections
        await self.llm.close()

        # Flush any pending saves
        # ...

        logger.info("Shutdown complete")
```

### Productionizing a Tool-Using Bot

A bot that calls tools in a loop crosses more provider-boundary edge cases than a
plain chat bot: mid-conversation system messages, provider-specific request-shape
rules, response truncation, growing conversation history, and orphaned
tool-call/tool-result pairs. DataKnobs handles most of these **on by default** —
this checklist is what you already get for free, plus the two knobs worth opting
into for a long tool loop.

#### What you get for free (on by default)

On the default Anthropic + ReAct path these protections are active with zero
configuration — do not re-implement or over-configure them:

| Protection | On-by-default behavior |
|---|---|
| **Mid-conversation system messages** | Folded inline for Anthropic (the default `system_message_policy` is `"inline"`), keeping the request structurally valid. See [Mid-conversation system-message policy](providers.md#mid-conversation-system-message-policy-anthropic). |
| **Rejected sampling params** | A sampling parameter the model does not accept is dropped with a logged warning rather than sent (no `400`). |
| **Over-ceiling `max_tokens`** | Clamped down to the model's resolved output ceiling instead of being sent and rejected or silently truncated. |
| **Output / input ceilings** | Resolved live for the Claude family (Models API, TTL-cached) with a bundled fallback — no hard-coded token limits to maintain. See [Model constraints](providers.md#model-constraints-request-shape-rules). |
| **Tool-loop bounds** | Wall-clock-bounded by `tool_loop_timeout` (default `120.0` s) and iteration-capped by `max_tool_iterations` (default `5`), both on `DynaBotConfig`. |
| **Truncated tool call** | A tool call the provider truncated at the token budget (`LLMResponse.truncated`) is safely abandoned and the turn synthesized — never a hard failure. |
| **Vendor API errors** | Translated to typed `dataknobs` exceptions (including `ContextLengthExceededError`) at a single choke point in the base provider, so you catch `dataknobs_llm.exceptions` types, not raw SDK errors. |
| **Termination reason** | Written to the `reasoning_termination` conversation metadata on every ReAct turn, independent of `store_trace`. See [ReAct Reasoning](../../bots/guides/configuration.md#react-reasoning). |
| **Orphaned `tool_use` pairing** | Repaired in the shared DynaBot turn-finalize, so an unmatched `tool_use`/`tool_result` pair never reaches the provider as a `400`. This runs for **every** tool strategy (see the cross-strategy note below). |

#### Opt in for a long tool loop

Two protections are **off by default** — their defaults are byte-identical to
prior behavior — and live on the ReAct reasoning config. A bot that runs long
tool loops should consider both.

**`history_compaction` — strongly recommended for long loops.** Without it, a
tool loop that outgrows the model's input window fails the turn. Thanks to
vendor-error translation that failure is now a clean typed
`ContextLengthExceededError` rather than a raw SDK error — but it is still a turn
failure, not a recovery. `history_compaction` is what turns the failure into a
recovery: it bounds the history the current tool loop accumulates.

```yaml
reasoning:
  strategy: react
  history_compaction:
    enabled: true
    keep_recent_iterations: 3   # retain the 3 most-recent tool iterations verbatim
    budget_fraction: 0.75       # proactively compact at 75% of the resolved input ceiling
    # strategy: window          # default; "summarize" folds old iterations via an LLM call
```

It has two legs: a **proactive** leg that compacts before a completion once the
estimated history exceeds `budget_fraction` of the resolved input ceiling, and a
**reactive** backstop that catches a `ContextLengthExceededError`, compacts once,
and retries. The reactive backstop exists because the proactive estimate is a
character-ratio heuristic that can under-count — do not rely on the proactive leg
alone. See [ReAct Reasoning](../../bots/guides/configuration.md#react-reasoning)
for the full field reference.

**`truncation_retry_max_tokens` — optional.** The safe abandon-and-synthesize
behavior above is automatic; this knob only helps when the *truncated call was
the work* — e.g. a large structured tool payload you want to complete rather than
abandon. When set, a truncated tool-call iteration is retried **once** at this
`max_tokens` before falling back to the terminal path (structurally single-shot —
no retry loop). Set it comfortably above `max_tokens`: a provider with a known
output ceiling (the Claude family) clamps an oversized value, and a provider
without one passes it through, so a generous value is safe either way.

#### Cross-strategy coverage

Be precise about what applies to every tool strategy versus what is ReAct-only:

- **Every tool strategy** gets orphaned-`tool_use`/`tool_result` pairing repair,
  because it runs in the shared DynaBot turn-finalize — a custom or non-ReAct tool
  bot gets the same `400`-avoidance.
- **ReAct only:** `history_compaction`, `truncation_retry_max_tokens`, and the
  `reasoning_termination` metadata. A bot running a custom/non-ReAct tool strategy
  does **not** get these automatically.

#### Provider caveat

HuggingFace's text-generation inference path exposes no token-budget stop reason
(`finish_reason` is always `'stop'`), so `LLMResponse.truncated` is always `False`
for HuggingFace-backed bots. The truncation protections above
(abandon-and-synthesize, `truncation_retry_max_tokens`) are therefore inert for
HuggingFace — plan history bounds around `history_compaction` instead.

#### Recommended production config

A minimal, annotated starting point for an Anthropic tool-using bot, placing each
knob at its correct level (top-level `DynaBotConfig` vs nested `reasoning:`):

```yaml
llm:
  provider: anthropic
  model: claude-5-sonnet     # output/input ceilings resolve live; over-ceiling max_tokens is clamped
  max_tokens: 4096

max_tool_iterations: 8       # DynaBotConfig top level (default 5)
tool_loop_timeout: 180.0     # tool-loop wall-clock budget, seconds (default 120)
tool_timeout: 30.0           # per-tool timeout, seconds (default 30)

reasoning:
  strategy: react
  max_iterations: 8
  history_compaction:        # strongly recommended for long tool loops
    enabled: true
    keep_recent_iterations: 3
    budget_fraction: 0.75
  truncation_retry_max_tokens: 8192   # optional: complete (not just abandon) a truncated tool payload
```

This is a *recommended set*, not a full schema — the complete field references
live in the bots [Configuration guide](../../bots/guides/configuration.md#react-reasoning)
and the [Providers guide](providers.md). `ContextLengthExceededError` is importable
from `dataknobs_llm.exceptions`.

---

## Provider Lifecycle

### Context Manager Usage

LLM providers support context managers for automatic initialization and cleanup:

**Async providers** (recommended):
```python
async with OllamaProvider(config={"model": "llama3.2"}) as provider:
    response = await provider.complete([
        LLMMessage(role="user", content="Hello")
    ])
# Provider is automatically closed on exit
```

**Sync providers**:
```python
with SyncLLMProvider(config={"model": "llama3.2"}) as provider:
    response = provider.complete([
        LLMMessage(role="user", content="Hello")
    ])
```

Using `async with` on an `AsyncLLMProvider` calls `initialize()` on entry and `close()` on exit. Using `with` on an async provider raises `TypeError` — always use `async with`.

### Provider Close Behavior

`close()` is idempotent and safe to call multiple times. On close:

1. In-flight requests are cancelled via `asyncio.Task.cancel()`
2. The `_close_client()` hook is called (subclass extension point)
3. Provider state is reset (`_is_initialized = False`)

### Readiness Guard

All completion and embedding methods call `_check_ready()` before processing. This raises `ResourceError` if the provider is not initialized or is in the process of closing. You do not need to call this yourself — it is automatic.

### Implementing Custom Providers

When subclassing `AsyncLLMProvider`, override `_close_client()` to clean up provider-specific HTTP clients:

```python
class MyProvider(AsyncLLMProvider):
    async def initialize(self) -> None:
        await super().initialize()
        self._client = httpx.AsyncClient(timeout=30.0)

    async def _close_client(self) -> None:
        """Close the httpx client."""
        if hasattr(self, "_client"):
            await self._client.aclose()
```

The base `_close_client()` is a no-op — only override it if your provider manages an HTTP client or other closeable resource.

---

## Capability Detection

Each provider reports its capabilities via `get_capabilities()`, which returns a
list of `ModelCapability` values. This uses a template method pattern:

1. **`_detect_capabilities()`** (abstract, implemented by each provider) —
   auto-detects capabilities based on model name (e.g. `gpt-4o` gets
   `FUNCTION_CALLING` and `JSON_MODE`).
2. **`_resolve_capabilities()`** (base class) — applies `config.capabilities`
   overrides, allowing consumers to add or remove capabilities via configuration.
3. **`get_capabilities()`** (base class, concrete) — orchestrates the pipeline:
   calls `_detect_capabilities()`, filters out `None` values, then calls
   `_resolve_capabilities()`.

### Config Overrides

All providers respect capability overrides from `LLMConfig.capabilities`:

```python
config = LLMConfig(
    provider="ollama",
    model="custom-model",
    capabilities={
        "add": ["json_mode"],      # Add capabilities
        "remove": ["streaming"],   # Remove capabilities
    },
)
provider = OllamaProvider(config)
caps = provider.get_capabilities()  # json_mode included, streaming excluded
```

This works for all providers (Ollama, OpenAI, Anthropic, HuggingFace, Echo) since
the override logic is in the base class.

### Implementing Custom Providers

When subclassing `AsyncLLMProvider`, implement `_detect_capabilities()` to report
what your provider supports. Config overrides are handled automatically:

```python
class MyProvider(AsyncLLMProvider):
    def _detect_capabilities(self) -> list[ModelCapability]:
        caps = [ModelCapability.TEXT_GENERATION, ModelCapability.CHAT]
        if "json" in self.config.model:
            caps.append(ModelCapability.JSON_MODE)
        return caps
```

### Substrate-Bound Providers: `ProfileDetectionMixin`

If your provider draws its capabilities, constraints, and pricing from the
model-metadata substrate (`dataknobs_llm.llm.model_profile`) — the pattern the
built-in OpenAI, Anthropic, Bedrock, and Ollama providers use — you do **not**
hand-write the detection trio. Adopt `ProfileDetectionMixin` (listed **first**
among the bases) and supply a single hook, `_profile_resolver()`; the mixin then
implements `_detect_capabilities()`, `_detect_constraints()`, and
`_detect_pricing()` as reads off the resolved `ModelProfile`:

```python
from dataknobs_llm.llm.profile_detection import ProfileDetectionMixin

class MyBoundProvider(ProfileDetectionMixin, AsyncLLMProvider):
    def _profile_resolver(self, config: LLMConfig) -> LayeredModelProfileResolver:
        return LayeredModelProfileResolver([
            ConfigOverrideSource(getattr(config, "model_profile_overrides", None)),
            _MY_RESOURCE_SOURCE,   # bundled fallback facts
            _MY_HEURISTIC_SOURCE,  # last-resort family classification
        ])
```

Two optional hooks cover the only per-provider variances:

- Override `_profile_lookup_key(config)` when the resolve key is not
  `config.model` (the Bedrock provider strips a cross-region inference-profile
  prefix before the catalog lookup).
- Override `_detect_constraints(config)` — calling the mixin's
  `_resolve_profile(config)` helper so the lookup-key logic stays shared — when
  the family adds constraint rules beyond the four profile facets (the Anthropic
  provider bans inline system messages and unions a runtime-discovered
  rejected-param overlay).

The base `_detect_capabilities()` stays abstract, so an *unbound* provider (like
the hand-written `MyProvider` above) is unaffected — the mixin is opt-in by
inheritance.

---

## Response Analysis Hooks

### `_analyze_response()`

Every response built by a provider passes through `_analyze_response()` before being returned. The base implementation detects **thinking-only responses** — a pattern where reasoning models (e.g., o1, o3) consume significant completion tokens on internal `<think>` blocks but return empty visible content.

**Detection criteria** (all must be true):
- `response.content` is empty
- `response.tool_calls` is empty
- `response.usage["completion_tokens"]` exceeds 50

When detected, the response is annotated with `response.metadata["thinking_only"] = True` and a warning is logged.

### Handling Thinking-Only Responses

```python
response = await provider.complete(messages)

if response.metadata.get("thinking_only"):
    # Model consumed tokens but produced no visible output.
    # Common causes: reasoning model spent all tokens on internal
    # deliberation, or the prompt triggered a think-only path.
    logger.warning("Thinking-only response — consider adjusting prompt")
```

### Extending in Subclasses

Override `_analyze_response()` to add provider-specific analysis. Always call `super()` to preserve the base thinking-only detection:

```python
class MyProvider(AsyncLLMProvider):
    def _analyze_response(self, response: LLMResponse) -> LLMResponse:
        response = super()._analyze_response(response)
        # Add custom analysis
        if response.usage and response.usage.get("total_tokens", 0) > 10000:
            response.metadata["high_token_usage"] = True
        return response
```

---

## Summary

### Quick Reference

**Prompt Design**:
- Clear, focused prompts
- Sensible defaults
- Limited conditional nesting (max 2 levels)
- Use template inheritance

**Validation**:
- ERROR for critical params
- WARN for recommended params
- Document all parameters

**RAG**:
- Separate configs from prompts
- Parameterize queries
- Handle failures gracefully
- Start with k=5, threshold=0.7

**Conversations**:
- Meaningful metadata
- Implement limits
- Clean up old conversations
- Use branching for exploration

**Middleware**:
- Order thoughtfully (log first, metadata last)
- Use validation selectively (cost consideration)
- Handle errors appropriately
- Monitor performance

**Storage**:
- Choose backend for your scale
- Implement caching for high traffic
- Batch operations
- Monitor performance

**Production**:
- Environment-specific configs
- Health checks
- Rate limiting
- Monitor token usage
- Secure credentials
- Graceful shutdown
- Productionizing a tool-using bot: on-by-default protections + opt into `history_compaction` for long tool loops

**Provider Lifecycle**:
- Use `async with` for automatic init/close
- Override `_close_client()` for custom resource cleanup
- `close()` is idempotent and cancels in-flight requests

**Response Analysis**:
- `_analyze_response()` detects thinking-only responses automatically
- Check `response.metadata["thinking_only"]` for reasoning model edge cases
- Override with `super()` call for custom post-processing

---

**Follow these practices for robust, maintainable, and efficient LLM applications!**
