Skip to content

testing.md

Latest commit

 

History

History
199 lines (159 loc) · 6.41 KB

testing.md

File metadata and controls

199 lines (159 loc) · 6.41 KB
Raw

Unit Tests Implementation Summary

Overview

Created a comprehensive unit test suite for the QueenBee agent orchestration system using pytest.

Test Files Created

1. tests/conftest.py - Shared Fixtures

  • Purpose: Pytest configuration and reusable test fixtures
  • Key Fixtures:
    • config_yaml_content: Sample configuration for tests
    • test_session_id: Generate test session IDs
    • test_agent_id: Generate test agent IDs
    • mock_ollama_response: Mock Ollama API responses
    • sample_discussion: Sample discussion data
    • temp_config_file: Temporary config file creation

2. tests/test_config.py - Configuration Tests (30+ tests)

  • TestDatabaseConfig: Database connection string formatting, defaults
  • TestOllamaConfig: Ollama configuration defaults and custom values
  • TestConsensusConfig: Consensus settings including timeout configuration
  • TestConfigLoader: YAML loading, environment variable overrides, validation
  • TestAgentPromptConfig: Agent configuration structure validation

Key Tests:

  • ✅ Connection string format validation
  • ✅ Default value verification
  • ✅ Environment variable overrides
  • ✅ Missing required fields (password) error handling
  • ✅ Specialist timeout configuration

3. tests/test_queen_complexity.py - Complexity Analysis (25+ tests)

  • Purpose: Test Queen's complexity detection logic
  • Coverage:
    • Simple queries (single words, short questions)
    • Complex keywords (analyze, compare, design, evaluate, explain, etc.)
    • Multiple questions detection
    • Long input detection (>50 words)
    • Case-insensitive matching
    • Word boundary detection
    • Real-world query examples

Key Tests:

  • ✅ Simple: "What is Python?" → False
  • ✅ Complex: "How to design a system?" → True
  • ✅ Complex: "Compare Python and JavaScript" → True
  • ✅ Edge case: Exactly 50 words → False, 51 words → True
  • ✅ Real-world scenarios for both simple and complex queries

4. tests/test_worker_logic.py - Contribution Logic (15+ tests)

  • TestContributionLogic: Agent contribution decision logic

    • First contribution always tries
    • Don't contribute twice in a row
    • Can contribute after another agent
    • Maximum 3 contributions per agent
    • Selective contribution in long discussions

  • TestDiscussionFormatting: Discussion formatting

    • Empty discussion handling
    • Single and multiple contributions
    • Order preservation
    • Proper formatting with separators

Key Tests:

  • ✅ First contribution always returns True
  • ✅ Blocks consecutive contributions from same agent
  • ✅ Enforces max 3 contributions limit
  • ✅ More selective after 6+ contributions
  • ✅ Proper formatting with contribution numbers

5. tests/test_database.py - Database Models (15+ tests)

  • TestTaskRepository: Task management operations

    • Create task returns UUID
    • Get pending tasks filters by session
    • Update task status
    • Set task result

  • TestChatRepository: Chat history operations

    • Add message with and without agent_id
    • Get session history with limit

  • TestEnums: Enum value validation

    • AgentType values
    • TaskStatus values
    • MessageRole values

Key Tests:

  • ✅ Task creation and ID generation
  • ✅ Session filtering for pending tasks
  • ✅ Status updates and result storage
  • ✅ Chat message insertion
  • ✅ Enum value correctness

Test Infrastructure

pytest.ini

  • Test discovery configuration
  • Coverage reporting settings
  • Test markers (unit, integration, slow)

scripts/run_tests.sh

  • Convenient test runner script
  • Checks for pytest installation
  • Runs full test suite with verbose output

tests/README.md

  • Comprehensive testing documentation
  • How to run tests
  • Writing new tests guide
  • Coverage goals and CI recommendations

Test Statistics

  • Total Test Files: 5 (4 test files + 1 conftest)
  • Total Tests: ~85+ individual test cases
  • Coverage Areas:
    • Configuration loading and validation
    • Complexity analysis (Queen agent)
    • Contribution logic (Worker manager)
    • Discussion formatting
    • Database operations (mocked)

Running Tests

Quick Start

# Install test dependencies
pip install pytest pytest-cov pytest-asyncio

# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=queenbee --cov-report=html

# Run specific test file
pytest tests/test_config.py -v

# Run specific test
pytest tests/test_queen_complexity.py::TestComplexityAnalysis::test_complex_keyword_analyze -v

Test Design Principles

  1. Isolation: Each test is independent with mocked dependencies
  2. Fast: No real database or LLM calls (all mocked)
  3. Readable: Clear test names describing what's being tested
  4. Comprehensive: Cover edge cases and error conditions
  5. Maintainable: Use fixtures to reduce code duplication

Mock Strategy

  • Database: Mock DatabaseManager and repositories
  • Ollama: Mock API responses (not tested yet, ready for integration)
  • File System: Use tmp_path fixture for temporary files
  • Config: Use in-memory YAML strings

Next Steps

Immediate

  1. Run Tests: Execute pytest tests/ -v to verify all tests pass
  2. Check Coverage: Run with --cov to see coverage metrics
  3. Fix Failures: Address any failing tests

Short-term

  • Add tests for agent base class
  • Add tests for rolling summary generation
  • Add integration tests with real database (using test DB)
  • Add performance benchmarks

Long-term

  • Integration tests with real Ollama
  • End-to-end CLI tests
  • Load testing with concurrent specialists
  • Snapshot testing for discussion outputs

Benefits

✅ Regression Prevention: Catch breaks when refactoring ✅ Documentation: Tests show how components should behave ✅ Confidence: Make changes knowing tests will catch issues ✅ Quality: Enforce expected behavior and edge cases ✅ CI/CD Ready: Can be integrated into automated pipelines

Test Coverage Goals

  • Unit Tests: Target >80% coverage for core logic
  • Critical Paths: Aim for 100% coverage:
    • ✅ Complexity analysis
    • ✅ Contribution logic
    • ✅ Configuration loading
    • ✅ Database operations (mocked)

Notes

  • All tests use mocks for external dependencies (DB, LLM)
  • Tests are designed to run fast (<5 seconds for full suite)
  • Integration tests should be added separately for real system testing
  • Consider adding pytest-xdist for parallel test execution