add brain
This commit is contained in:
@@ -0,0 +1,430 @@
|
||||
# Agent Designer - Multi-Agent System Architecture Toolkit
|
||||
|
||||
**Tier:** POWERFUL
|
||||
**Category:** Engineering
|
||||
**Tags:** AI agents, architecture, system design, orchestration, multi-agent systems
|
||||
|
||||
A comprehensive toolkit for designing, architecting, and evaluating multi-agent systems. Provides structured approaches to agent architecture patterns, tool design principles, communication strategies, and performance evaluation frameworks.
|
||||
|
||||
## Overview
|
||||
|
||||
The Agent Designer skill includes three core components:
|
||||
|
||||
1. **Agent Planner** (`agent_planner.py`) - Designs multi-agent system architectures
|
||||
2. **Tool Schema Generator** (`tool_schema_generator.py`) - Creates structured tool schemas
|
||||
3. **Agent Evaluator** (`agent_evaluator.py`) - Evaluates system performance and identifies optimizations
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Design a Multi-Agent Architecture
|
||||
|
||||
```bash
|
||||
# Use sample requirements or create your own
|
||||
python agent_planner.py assets/sample_system_requirements.json -o my_architecture
|
||||
|
||||
# This generates:
|
||||
# - my_architecture.json (complete architecture)
|
||||
# - my_architecture_diagram.mmd (Mermaid diagram)
|
||||
# - my_architecture_roadmap.json (implementation plan)
|
||||
```
|
||||
|
||||
### 2. Generate Tool Schemas
|
||||
|
||||
```bash
|
||||
# Use sample tool descriptions or create your own
|
||||
python tool_schema_generator.py assets/sample_tool_descriptions.json -o my_tools
|
||||
|
||||
# This generates:
|
||||
# - my_tools.json (complete schemas)
|
||||
# - my_tools_openai.json (OpenAI format)
|
||||
# - my_tools_anthropic.json (Anthropic format)
|
||||
# - my_tools_validation.json (validation rules)
|
||||
# - my_tools_examples.json (usage examples)
|
||||
```
|
||||
|
||||
### 3. Evaluate System Performance
|
||||
|
||||
```bash
|
||||
# Use sample execution logs or your own
|
||||
python agent_evaluator.py assets/sample_execution_logs.json -o evaluation
|
||||
|
||||
# This generates:
|
||||
# - evaluation.json (complete report)
|
||||
# - evaluation_summary.json (executive summary)
|
||||
# - evaluation_recommendations.json (optimization suggestions)
|
||||
# - evaluation_errors.json (error analysis)
|
||||
```
|
||||
|
||||
## Detailed Usage
|
||||
|
||||
### Agent Planner
|
||||
|
||||
The Agent Planner designs multi-agent architectures based on system requirements.
|
||||
|
||||
#### Input Format
|
||||
|
||||
Create a JSON file with system requirements:
|
||||
|
||||
```json
|
||||
{
|
||||
"goal": "Your system's primary objective",
|
||||
"description": "Detailed system description",
|
||||
"tasks": ["List", "of", "required", "tasks"],
|
||||
"constraints": {
|
||||
"max_response_time": 30000,
|
||||
"budget_per_task": 1.0,
|
||||
"quality_threshold": 0.9
|
||||
},
|
||||
"team_size": 6,
|
||||
"performance_requirements": {
|
||||
"high_throughput": true,
|
||||
"fault_tolerance": true,
|
||||
"low_latency": false
|
||||
},
|
||||
"safety_requirements": [
|
||||
"Input validation and sanitization",
|
||||
"Output content filtering"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Command Line Options
|
||||
|
||||
```bash
|
||||
python agent_planner.py <input_file> [OPTIONS]
|
||||
|
||||
Options:
|
||||
-o, --output PREFIX Output file prefix (default: agent_architecture)
|
||||
--format FORMAT Output format: json, both (default: both)
|
||||
```
|
||||
|
||||
#### Output Files
|
||||
|
||||
- **Architecture JSON**: Complete system design with agents, communication topology, and scaling strategy
|
||||
- **Mermaid Diagram**: Visual representation of the agent architecture
|
||||
- **Implementation Roadmap**: Phased implementation plan with timelines and risks
|
||||
|
||||
#### Architecture Patterns
|
||||
|
||||
The planner automatically selects from these patterns based on requirements:
|
||||
|
||||
- **Single Agent**: Simple, focused tasks (1 agent)
|
||||
- **Supervisor**: Hierarchical delegation (2-8 agents)
|
||||
- **Swarm**: Peer-to-peer collaboration (3-20 agents)
|
||||
- **Hierarchical**: Multi-level management (5-50 agents)
|
||||
- **Pipeline**: Sequential processing (3-15 agents)
|
||||
|
||||
### Tool Schema Generator
|
||||
|
||||
Generates structured tool schemas compatible with OpenAI and Anthropic formats.
|
||||
|
||||
#### Input Format
|
||||
|
||||
Create a JSON file with tool descriptions:
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": [
|
||||
{
|
||||
"name": "tool_name",
|
||||
"purpose": "What the tool does",
|
||||
"category": "Tool category (search, data, api, etc.)",
|
||||
"inputs": [
|
||||
{
|
||||
"name": "parameter_name",
|
||||
"type": "string",
|
||||
"description": "Parameter description",
|
||||
"required": true,
|
||||
"examples": ["example1", "example2"]
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "result_field",
|
||||
"type": "object",
|
||||
"description": "Output description"
|
||||
}
|
||||
],
|
||||
"error_conditions": ["List of possible errors"],
|
||||
"side_effects": ["List of side effects"],
|
||||
"idempotent": true,
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Command Line Options
|
||||
|
||||
```bash
|
||||
python tool_schema_generator.py <input_file> [OPTIONS]
|
||||
|
||||
Options:
|
||||
-o, --output PREFIX Output file prefix (default: tool_schemas)
|
||||
--format FORMAT Output format: json, both (default: both)
|
||||
--validate Validate generated schemas
|
||||
```
|
||||
|
||||
#### Output Files
|
||||
|
||||
- **Complete Schemas**: All schemas with validation and examples
|
||||
- **OpenAI Format**: Schemas compatible with OpenAI function calling
|
||||
- **Anthropic Format**: Schemas compatible with Anthropic tool use
|
||||
- **Validation Rules**: Input validation specifications
|
||||
- **Usage Examples**: Example calls and responses
|
||||
|
||||
#### Schema Features
|
||||
|
||||
- **Input Validation**: Comprehensive parameter validation rules
|
||||
- **Error Handling**: Structured error response formats
|
||||
- **Rate Limiting**: Configurable rate limit specifications
|
||||
- **Documentation**: Auto-generated usage examples
|
||||
- **Security**: Built-in security considerations
|
||||
|
||||
### Agent Evaluator
|
||||
|
||||
Analyzes agent execution logs to identify performance issues and optimization opportunities.
|
||||
|
||||
#### Input Format
|
||||
|
||||
Create a JSON file with execution logs:
|
||||
|
||||
```json
|
||||
{
|
||||
"execution_logs": [
|
||||
{
|
||||
"task_id": "unique_task_identifier",
|
||||
"agent_id": "agent_identifier",
|
||||
"task_type": "task_category",
|
||||
"start_time": "2024-01-15T09:00:00Z",
|
||||
"end_time": "2024-01-15T09:02:34Z",
|
||||
"duration_ms": 154000,
|
||||
"status": "success",
|
||||
"actions": [
|
||||
{
|
||||
"type": "tool_call",
|
||||
"tool_name": "web_search",
|
||||
"duration_ms": 2300,
|
||||
"success": true
|
||||
}
|
||||
],
|
||||
"results": {
|
||||
"summary": "Task results",
|
||||
"quality_score": 0.92
|
||||
},
|
||||
"tokens_used": {
|
||||
"input_tokens": 1250,
|
||||
"output_tokens": 2800,
|
||||
"total_tokens": 4050
|
||||
},
|
||||
"cost_usd": 0.081,
|
||||
"error_details": null,
|
||||
"tools_used": ["web_search"],
|
||||
"retry_count": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Command Line Options
|
||||
|
||||
```bash
|
||||
python agent_evaluator.py <input_file> [OPTIONS]
|
||||
|
||||
Options:
|
||||
-o, --output PREFIX Output file prefix (default: evaluation_report)
|
||||
--format FORMAT Output format: json, both (default: both)
|
||||
--detailed Include detailed analysis in output
|
||||
```
|
||||
|
||||
#### Output Files
|
||||
|
||||
- **Complete Report**: Comprehensive performance analysis
|
||||
- **Executive Summary**: High-level metrics and health assessment
|
||||
- **Optimization Recommendations**: Prioritized improvement suggestions
|
||||
- **Error Analysis**: Detailed error patterns and solutions
|
||||
|
||||
#### Evaluation Metrics
|
||||
|
||||
**Performance Metrics**:
|
||||
- Task success rate and completion times
|
||||
- Token usage and cost efficiency
|
||||
- Error rates and retry patterns
|
||||
- Throughput and latency distributions
|
||||
|
||||
**System Health**:
|
||||
- Overall health score (poor/fair/good/excellent)
|
||||
- SLA compliance tracking
|
||||
- Resource utilization analysis
|
||||
- Trend identification
|
||||
|
||||
**Bottleneck Analysis**:
|
||||
- Agent performance bottlenecks
|
||||
- Tool usage inefficiencies
|
||||
- Communication overhead
|
||||
- Resource constraints
|
||||
|
||||
## Architecture Patterns Guide
|
||||
|
||||
### When to Use Each Pattern
|
||||
|
||||
#### Single Agent
|
||||
- **Best for**: Simple, focused tasks with clear boundaries
|
||||
- **Team size**: 1 agent
|
||||
- **Complexity**: Low
|
||||
- **Examples**: Personal assistant, document summarizer, simple automation
|
||||
|
||||
#### Supervisor
|
||||
- **Best for**: Hierarchical task decomposition with quality control
|
||||
- **Team size**: 2-8 agents
|
||||
- **Complexity**: Medium
|
||||
- **Examples**: Research coordinator with specialists, content review workflow
|
||||
|
||||
#### Swarm
|
||||
- **Best for**: Distributed problem solving with high fault tolerance
|
||||
- **Team size**: 3-20 agents
|
||||
- **Complexity**: High
|
||||
- **Examples**: Parallel data processing, distributed research, competitive analysis
|
||||
|
||||
#### Hierarchical
|
||||
- **Best for**: Large-scale operations with organizational structure
|
||||
- **Team size**: 5-50 agents
|
||||
- **Complexity**: Very High
|
||||
- **Examples**: Enterprise workflows, complex business processes
|
||||
|
||||
#### Pipeline
|
||||
- **Best for**: Sequential processing with specialized stages
|
||||
- **Team size**: 3-15 agents
|
||||
- **Complexity**: Medium
|
||||
- **Examples**: Data ETL pipelines, content processing workflows
|
||||
|
||||
## Best Practices
|
||||
|
||||
### System Design
|
||||
|
||||
1. **Start Simple**: Begin with simpler patterns and evolve
|
||||
2. **Clear Responsibilities**: Define distinct roles for each agent
|
||||
3. **Robust Communication**: Design reliable message passing
|
||||
4. **Error Handling**: Plan for failures and recovery
|
||||
5. **Monitor Everything**: Implement comprehensive observability
|
||||
|
||||
### Tool Design
|
||||
|
||||
1. **Single Responsibility**: Each tool should have one clear purpose
|
||||
2. **Input Validation**: Validate all inputs thoroughly
|
||||
3. **Idempotency**: Design operations to be safely repeatable
|
||||
4. **Error Recovery**: Provide clear error messages and recovery paths
|
||||
5. **Documentation**: Include comprehensive usage examples
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
1. **Measure First**: Use the evaluator to identify actual bottlenecks
|
||||
2. **Optimize Bottlenecks**: Focus on highest-impact improvements
|
||||
3. **Cache Strategically**: Cache expensive operations and results
|
||||
4. **Parallel Processing**: Identify opportunities for parallelization
|
||||
5. **Resource Management**: Monitor and optimize resource usage
|
||||
|
||||
## Sample Files
|
||||
|
||||
The `assets/` directory contains sample files to help you get started:
|
||||
|
||||
- **`sample_system_requirements.json`**: Example system requirements for a research platform
|
||||
- **`sample_tool_descriptions.json`**: Example tool descriptions for common operations
|
||||
- **`sample_execution_logs.json`**: Example execution logs from a running system
|
||||
|
||||
The `expected_outputs/` directory shows expected results from processing these samples.
|
||||
|
||||
## References
|
||||
|
||||
See the `references/` directory for detailed documentation:
|
||||
|
||||
- **`agent_architecture_patterns.md`**: Comprehensive catalog of architecture patterns
|
||||
- **`tool_design_best_practices.md`**: Best practices for tool design and implementation
|
||||
- **`evaluation_methodology.md`**: Detailed methodology for system evaluation
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### With OpenAI
|
||||
|
||||
```python
|
||||
import json
|
||||
import openai
|
||||
|
||||
# Load generated OpenAI schemas
|
||||
with open('my_tools_openai.json') as f:
|
||||
schemas = json.load(f)
|
||||
|
||||
# Use with OpenAI function calling
|
||||
response = openai.ChatCompletion.create(
|
||||
model="gpt-4",
|
||||
messages=[{"role": "user", "content": "Search for AI news"}],
|
||||
functions=schemas['functions']
|
||||
)
|
||||
```
|
||||
|
||||
### With Anthropic Claude
|
||||
|
||||
```python
|
||||
import json
|
||||
import anthropic
|
||||
|
||||
# Load generated Anthropic schemas
|
||||
with open('my_tools_anthropic.json') as f:
|
||||
schemas = json.load(f)
|
||||
|
||||
# Use with Anthropic tool use
|
||||
client = anthropic.Anthropic()
|
||||
response = client.messages.create(
|
||||
model="claude-3-opus-20240229",
|
||||
messages=[{"role": "user", "content": "Search for AI news"}],
|
||||
tools=schemas['tools']
|
||||
)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**"No valid architecture pattern found"**
|
||||
- Check that team_size is reasonable (1-50)
|
||||
- Ensure tasks list is not empty
|
||||
- Verify performance_requirements are valid
|
||||
|
||||
**"Tool schema validation failed"**
|
||||
- Check that all required fields are present
|
||||
- Ensure parameter types are valid
|
||||
- Verify enum values are provided as arrays
|
||||
|
||||
**"Insufficient execution logs"**
|
||||
- Ensure logs contain required fields (task_id, agent_id, status)
|
||||
- Check that timestamps are in ISO 8601 format
|
||||
- Verify token usage fields are numeric
|
||||
|
||||
### Performance Tips
|
||||
|
||||
1. **Large Systems**: For systems with >20 agents, consider breaking into subsystems
|
||||
2. **Complex Tools**: Tools with >10 parameters may need simplification
|
||||
3. **Log Volume**: For >1000 log entries, consider sampling for faster analysis
|
||||
|
||||
## Contributing
|
||||
|
||||
This skill is part of the claude-skills repository. To contribute:
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch
|
||||
3. Make your changes
|
||||
4. Add tests and documentation
|
||||
5. Submit a pull request
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the MIT License - see the main repository for details.
|
||||
|
||||
## Support
|
||||
|
||||
For issues and questions:
|
||||
- Check the troubleshooting section above
|
||||
- Review the reference documentation in `references/`
|
||||
- Create an issue in the claude-skills repository
|
||||
Reference in New Issue
Block a user