Metadata-Version: 2.4
Name: agent-sprint-testkit
Version: 0.1.1
Summary: AgentSprint TestKit - Universal AI agent benchmarking and testing framework
Home-page: https://github.com/your-org/astk
Author: ASTK Team
Author-email: ASTK Team <team@astk.dev>
Maintainer-email: ASTK Team <team@astk.dev>
Project-URL: Homepage, https://github.com/your-org/astk
Project-URL: Documentation, https://astk.readthedocs.io
Project-URL: Repository, https://github.com/your-org/astk.git
Project-URL: Issues, https://github.com/your-org/astk/issues
Project-URL: Changelog, https://github.com/your-org/astk/blob/main/CHANGELOG.md
Keywords: ai,agent,testing,benchmark,llm,chatbot
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: langchain>=0.1.0
Requires-Dist: langchain-openai>=0.0.2
Requires-Dist: langchain-core>=0.1.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: click>=8.0.0
Requires-Dist: opentelemetry-api>=1.20.0
Requires-Dist: opentelemetry-sdk>=1.20.0
Requires-Dist: psutil>=5.9.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: requests>=2.28.0
Requires-Dist: aiohttp>=3.8.0
Requires-Dist: pyyaml>=6.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: isort>=5.12.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Provides-Extra: docker
Requires-Dist: docker>=6.0.0; extra == "docker"
Provides-Extra: all
Requires-Dist: astk[dev,docker]; extra == "all"
Dynamic: author
Dynamic: home-page
Dynamic: requires-python

# AgentSprint TestKit (ASTK) 🚀

> Universal AI agent benchmarking and testing framework

ASTK is a comprehensive testing framework for AI agents that evaluates performance, intelligence, and capabilities through diverse scenarios. Test your agents against real-world tasks like file analysis, code comprehension, and complex reasoning.

## 🎯 Features

- **🧠 Intelligent Benchmarks**: 8 diverse scenarios testing different AI capabilities
- **📊 Performance Metrics**: Response time, success rate, and quality analysis
- **🔧 Easy Installation**: Simple pip install from PyPI
- **🌐 Universal Testing**: Works with CLI agents, REST APIs, Python modules, and more
- **🤖 Agent Ready**: Compatible with LangChain, OpenAI, and custom agents
- **📁 Built-in Examples**: File Q&A agent and project templates
- **⚙️ GitHub Actions**: Ready-to-use CI/CD workflow templates

## 📋 Quick Start

### 1. Install from PyPI

```bash
pip install agent-sprint-testkit
```

### 2. Verify Installation

```bash
astk --help
```

### 3. Set API Key

```bash
export OPENAI_API_KEY="your-api-key-here"
```

### 4. Initialize a Project

```bash
astk init my-agent-tests
cd my-agent-tests
```

### 5. Run Your First Benchmark

```bash
# Benchmark an example agent
astk benchmark examples/agents/file_qa_agent.py

# Or run directly from your project
python scripts/simple_benchmark.py examples/agents/file_qa_agent.py
```

## 🚀 Installation Options

### Option 1: Global Installation (Recommended)

```bash
pip install agent-sprint-testkit
astk --version
```

### Option 2: Development Setup

```bash
# Clone repository
git clone https://github.com/your-org/astk.git
cd astk

# Create virtual environment
python3.11 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install in development mode
pip install -e .
```

## 💻 CLI Commands

### Core Commands

```bash
# Initialize new project with templates
astk init <project-name>

# Run intelligent benchmarks
astk benchmark <agent-path>

# Generate detailed reports
astk report <results-dir>

# Show example usage
astk examples
```

### Legacy Script Commands (still supported)

```bash
# Run intelligent benchmark
python scripts/simple_benchmark.py <agent-path>

# Quick agent runner
python scripts/simple_run.py <agent-path>
```

## 🤖 Available Agents

### File Q&A Agent (`examples/agents/file_qa_agent.py`)

A LangChain-powered agent that can:

- **📁 List files** in directories
- **📖 Read file contents** and summarize
- **🔍 Answer questions** about file data
- **🧭 Navigate** directory structures

**Example Usage:**

```bash
# Direct agent usage
python examples/agents/file_qa_agent.py "What Python files are in this project?"

# Run with simple runner
python scripts/simple_run.py examples/agents/file_qa_agent.py

# Run intelligent benchmark
python scripts/simple_benchmark.py examples/agents/file_qa_agent.py
```

## 🧪 Benchmark Scenarios

The intelligent benchmark tests 8 diverse scenarios:

| Scenario                    | Test                               | Capability                 |
| --------------------------- | ---------------------------------- | -------------------------- |
| **📁 File Discovery**       | Find Python files and entry points | File system navigation     |
| **⚙️ Config Analysis**      | Analyze configuration files        | Technical comprehension    |
| **📖 README Comprehension** | Read and explain project           | Document analysis          |
| **🏗️ Code Structure**       | Analyze directory structure        | Architecture understanding |
| **📚 Documentation Search** | Explore documentation              | Information retrieval      |
| **🔗 Dependency Analysis**  | Analyze requirements/dependencies  | Technical analysis         |
| **💡 Example Exploration**  | Discover example code              | Code comprehension         |
| **🧪 Test Discovery**       | Find testing framework             | Development understanding  |

## 📊 Results & Metrics

Benchmarks generate comprehensive results:

```json
{
  "success_rate": 1.0,
  "total_duration_seconds": 25.4,
  "average_scenario_duration": 3.2,
  "average_response_length": 847,
  "scenarios": [...]
}
```

**Metrics Include:**

- ✅ **Success Rate**: Percentage of completed scenarios
- ⏱️ **Response Time**: Duration for each scenario
- 📝 **Response Quality**: Length and content analysis
- 🎯 **Scenario Details**: Individual query results

## 🛠️ Available Tools

### 🚀 ASTK CLI (Primary Interface)

```bash
# Initialize project with templates
astk init my-project

# Run intelligent benchmarks
astk benchmark <agent-path>

# Generate HTML/JSON reports
astk report <results-dir>

# View usage examples
astk examples
```

### 🧪 Legacy Script Runners (Still Supported)

```bash
# Direct benchmark execution
python scripts/simple_benchmark.py <agent-path>

# Basic agent runner
python scripts/simple_run.py <agent-path>
```

## 🏗️ Project Structure

```
ASTK/
├── 🤖 examples/agents/          # Example AI agents
│   └── file_qa_agent.py         # LangChain File Q&A agent
├── 📊 scripts/                  # Benchmark and utility scripts
│   ├── simple_benchmark.py      # Intelligent benchmark runner ⭐
│   ├── simple_run.py            # Basic agent runner
│   └── astk.py                  # Advanced CLI (WIP)
├── 🧠 astk/                     # Core ASTK framework
│   ├── benchmarks/              # Benchmark modules
│   ├── cli.py                   # Command-line interface
│   └── *.py                     # Core modules
├── 📁 benchmark_results/        # Generated benchmark results
├── ⚙️ config/                   # Configuration files
└── 📖 docs/                     # Documentation
```

## 🎮 Usage Examples

### Run Agent Directly

```bash
python examples/agents/file_qa_agent.py "Analyze the setup.py file"
```

### Quick Agent Test

```bash
python scripts/simple_run.py examples/agents/file_qa_agent.py
```

### Full Intelligence Benchmark

```bash
python scripts/simple_benchmark.py examples/agents/file_qa_agent.py
```

### Custom Queries

```bash
python examples/agents/file_qa_agent.py "What is the purpose of the astk directory?"
```

## 🔧 Troubleshooting

### Common Issues

**📦 Installation Problems**

```bash
# Update pip and reinstall
pip install --upgrade pip
pip install --upgrade agent-sprint-testkit

# Verify installation
astk --version
which astk
```

**🔑 OpenAI API Issues**

```bash
# Verify API key is set
echo $OPENAI_API_KEY

# Set API key
export OPENAI_API_KEY="sk-..."
```

**🐍 Development Environment Issues**

```bash
# For development setup
git clone https://github.com/your-org/astk.git
cd astk
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e .
```

**🤖 Agent Compatibility**

The framework supports multiple agent types:

- **CLI agents**: Accept queries as command-line arguments
- **Python modules**: Have a `chat()` method
- **REST APIs**: Expose `/chat` endpoint
- **Custom formats**: Use adapter patterns as needed

## 🚀 Creating Your Own Agent

Create a new agent that responds to command-line arguments:

```python
#!/usr/bin/env python3
import sys

async def main():
    if len(sys.argv) > 1:
        query = " ".join(sys.argv[1:])
        # Process query and return response
        print(f"Agent: {response}")
    else:
        # Default behavior
        print("Agent: Ready!")

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())
```

Then benchmark it:

```bash
python scripts/simple_benchmark.py path/to/your_agent.py
```

## 📈 Performance Tips

- **⚡ Faster Responses**: Use GPT-3.5-turbo for speed
- **🧠 Better Intelligence**: Use GPT-4 for complex reasoning
- **💰 Cost Optimization**: Monitor token usage in results
- **🔧 Custom Scenarios**: Modify `scripts/simple_benchmark.py` for specific tests

## 🤝 Contributing

1. Create new agents in `examples/agents/`
2. Add benchmark scenarios in `scripts/simple_benchmark.py`
3. Test with: `python scripts/simple_benchmark.py your_agent.py`

## 📄 License

Apache 2.0 License - See LICENSE file for details.

---

**🎯 Ready to benchmark your AI agents? Start with:**

```bash
# Install globally
pip install agent-sprint-testkit

# Run your first benchmark
astk benchmark examples/agents/file_qa_agent.py

# Or use the legacy script
python scripts/simple_benchmark.py examples/agents/file_qa_agent.py
```

**🚀 Get started in 3 commands:**

```bash
pip install agent-sprint-testkit
astk init my-tests
astk examples
```
