Skip to content

README.md

Latest commit

 

History

History
420 lines (268 loc) · 13.5 KB

README.md

File metadata and controls

420 lines (268 loc) · 13.5 KB
Raw

Go Development MCP Server Testing Framework

This directory contains our testing framework for the Go Development MCP Server. We use a Go-based testing approach as our primary testing methodology, which directly executes Go commands for more reliable and maintainable tests. The older PowerShell-based tests have been preserved in the legacy directory for reference purposes only but should not be used for new test development.

Directory Structure

testing/
├── main.go            # Primary Go-based test runner for execution strategies
├── direct_runner.go   # Standalone direct execution strategy runner
├── hybrid_runner.go   # Standalone hybrid execution strategy runner
├── MIGRATION_STATUS.md # Migration progress tracking (currently 60% complete)
├── legacy/            # Legacy PowerShell test scripts (for reference only)
│   ├── basic/         # Legacy basic test scripts
│   ├── core/          # Legacy core functionality test scripts
│   ├── strategies/    # Legacy execution strategy test scripts
│   └── utils/         # Legacy shared utility functions
├── GETTING_STARTED.md # Quick reference guide
└── README.md          # This documentation file

Go-based Testing Framework

We use a modern Go-based testing approach that offers several advantages:

Test Runners

  1. Main Test Runner (main.go)

    • Primary entry point for running all tests
    • Uses direct Go command execution without handlers
    • Simplified error handling and more predictable behavior
    • Runs with go run main.go or go run main.go -type=direct|hybrid|both

  2. Standalone Strategy Runners

    • direct_runner.go: Tests direct code execution strategy
    • hybrid_runner.go: Tests hybrid execution with project path and code
    • Both use build tags for selective compilation

Direct Execution Strategy

The Go-based tests execute Go commands directly using os/exec rather than going through handlers:

cmd := exec.Command("go", "run", mainGoPath)
output, err := cmd.CombinedOutput()

This approach eliminates dependencies on external packages and context parameters, resulting in simpler and more maintainable code.

Migration Status

Currently, approximately 60% of our tests have been migrated to the Go-based approach:

  • ✅ Core tool functionality (run, build) - Complete
  • ✅ Execution strategies - Complete
  • ✅ Input handling - Complete
  • 🔄 Integration between components - In Progress
  • ❌ Mock server for E2E tests - Not Started
  • ❌ API boundary testing - Not Started

For a detailed overview of the migration progress, see MIGRATION_STATUS.md.

Testing Approach

Our testing approach simulates real-world interactions with the server, focusing on:

  1. Testing all input modes:

    • Code-only: When only inline code is provided
    • Project path-only: When only a directory path is provided
    • Hybrid: When both code and project path are provided

  2. Testing all server tools:

    • go_build: Building Go code
    • go_run: Running Go code
    • go_fmt: Formatting Go code
    • go_test: Running tests for Go code
    • go_mod: Managing Go modules
    • go_analyze: Analyzing Go code for issues

  3. Verifying the Hybrid Strategy: The hybrid execution strategy combines:

    • Project structure and dependencies from the project_path
    • Modified code from the code parameter

    This strategy is critical for providing context-aware code execution while allowing modifications.

Running Go Tests

Running with Go Commands

You can run the Go-based tests directly using the Go command-line tools:

# Run the main test runner with both direct and hybrid tests

cd c:\Users\James\Documents\go-dev-mcp\scripts\testing
go run main.go

# Run only direct execution tests

go run main.go -type=direct

# Run only hybrid execution tests

go run main.go -type=hybrid

Using Standalone Test Runners

For specific tests with build tags:

# Run direct execution tests with the direct_runner

cd c:\Users\James\Documents\go-dev-mcp\scripts\testing
go run -tags=direct_test direct_runner.go

# Run hybrid execution tests with the hybrid_runner

go run -tags=hybrid_test hybrid_runner.go

Using the PowerShell Script Runner for Go Tests

We also provide a PowerShell script to run the Go tests with additional options:

# Run Go tests with verbose output

.\run_go_tests.ps1 -Verbose

# Run Go tests with race detection

.\run_go_tests.ps1 -Race

# Run Go tests with coverage

.\run_go_tests.ps1 -Cover

For more advanced testing with coverage reports:

# Generate coverage report

.\run_tests_with_coverage.ps1

Legacy Testing Framework Overview

Our testing approach simulates real-world interactions with the server, focusing on:

  1. Testing all input modes:

    • Code-only: When only inline code is provided
    • Project path-only: When only a directory path is provided
    • Hybrid: When both code and project path are provided

  2. Testing all server tools:

    • go_build: Building Go code
    • go_run: Running Go code
    • go_fmt: Formatting Go code
    • go_test: Running tests for Go code
    • go_mod: Managing Go modules
    • go_analyze: Analyzing Go code for issues

  3. Verifying the Hybrid Strategy: The hybrid execution strategy combines:

    • Project structure and dependencies from the project_path
    • Modified code from the code parameter

    This strategy is critical for providing context-aware code execution while allowing modifications.

Legacy PowerShell Testing

IMPORTANT: The following sections describe our legacy PowerShell testing approach. These scripts are kept for reference purposes only and should not be used for new test development. All new tests should use the Go-based approach described above.

Legacy Master Test Runner

The legacy run_tests.ps1 script provides a way to run legacy PowerShell tests by category:

# Run all tests

.\legacy\run_tests.ps1 -TestType all

# Run only basic tests

.\legacy\run_tests.ps1 -TestType basic

# Run only core tests

.\legacy\run_tests.ps1 -TestType core

# Run only strategy tests

.\legacy\run_tests.ps1 -TestType strategies

Additional parameters:

  • -VerboseOutput: Show detailed test information
  • -KeepTestDirs: Keep temporary test directories for inspection
  • -ServerExecutable <path>: Specify a custom server executable path

Legacy Core Test Scripts

legacy/core/all_tools_test.ps1

A comprehensive legacy test suite that tests all six tools provided by the Go Development MCP Server with all applicable input modes.

Usage:

.\legacy\core\all_tools_test.ps1 [-ServerExecutable <path>] [-KeepTestDirs] [-TestDir <path>] [-Verbose]

Parameters:

  • -ServerExecutable: Path to the MCP server executable (default: "....\build\server.exe")
  • -KeepTestDirs: If specified, test directories will not be deleted after testing
  • -TestDir: Custom directory to use for test files
  • -Verbose: Show detailed test information

legacy/core/e2e_test.ps1

An end-to-end legacy test script that verifies the server works correctly across multiple test cases.

Usage:

.\legacy\core\e2e_test.ps1 [-ServerUrl <url>] [-TempDir <path>] [-KeepTempFiles] [-Verbose]

Parameters:

  • -ServerUrl: URL of the running server (default: "http://localhost:8080")
  • -TempDir: Custom directory to use for test files
  • -KeepTempFiles: If specified, temporary files will not be deleted after the test
  • -Verbose: Show detailed test information

Legacy Strategy Test Scripts

legacy/strategies/hybrid_strategy_test.ps1

A detailed legacy test focused specifically on the hybrid execution strategy, verifying that modified code is correctly applied while maintaining project context.

Usage:

.\legacy\strategies\hybrid_strategy_test.ps1 [-ServerExecutable <path>] [-TestDir <path>] [-KeepTestDirs] [-Verbose]

Parameters:

  • -ServerExecutable: Path to the MCP server executable (default: "....\build\server.exe")
  • -TestDir: Base directory for test files
  • -KeepTestDirs: If specified, test directories won't be deleted after the test
  • -Verbose: Show detailed step-by-step execution information

This script creates test projects with varying complexity, applies code modifications, and verifies the hybrid strategy correctly combines project context with modified code.

legacy/strategies/hybrid_cli_test.ps1

A CLI-focused legacy test for the hybrid execution strategy, calling the server executable directly.

legacy/strategies/hybrid_strat_verify.ps1

A simplified legacy test for verifying hybrid strategy functionality without requiring the server to be running.

Legacy Basic Test Scripts

legacy/basic/simple_test.ps1

A minimal legacy test script that was formerly used as a starting point for quick tests or as a template for new test scripts.

Legacy Utilities

legacy/utils/test_utils.ps1

Legacy shared utility functions for PowerShell testing scripts. This file contains functions for:

  • Formatting and displaying test results
  • Creating test projects with various configurations
  • Running Go commands and capturing their output
  • Validating test results with custom assertions

legacy/utils/direct_test.go and legacy/utils/hybrid_test.go

Legacy Go packages for testing the direct and hybrid execution strategies. These files have been updated to use direct Go command execution instead of the handler-based approach, preserving backward compatibility while eliminating dependencies on missing packages.

legacy/utils/main.go

A simple entry point for running legacy tests that use the direct and hybrid execution strategies without requiring the handlers package.

Running Legacy PowerShell Tests

REMINDER: Legacy tests are maintained for reference only and should not be used for new test development.

Legacy Quick Test

For a quick sanity check with legacy tests:

cd c:\Users\James\Documents\go-dev-mcp\scripts\testing
.\legacy\basic\simple_test.ps1

Legacy Comprehensive Testing

To run comprehensive legacy tests with all tools and input modes:

cd c:\Users\James\Documents\go-dev-mcp\scripts\testing
.\legacy\core\all_tools_test.ps1 -Verbose

Legacy Strategy-Specific Testing

To run legacy tests for the hybrid execution strategy specifically:

cd c:\Users\James\Documents\go-dev-mcp\scripts\testing
.\legacy\strategies\hybrid_strategy_test.ps1 -Verbose

Test Results

The Go and legacy tests output results with color-coded status indicators:

  • ✅ PASS: Test completed successfully (green)
  • ❌ FAIL: Test failed with details about the failure (red)
  • ℹ️ INFO: Informational messages (white or cyan)

Each test provides timing information and a summary of passed and failed tests at the end.

Adding New Tests

To add a new Go-based test:

  1. Follow the patterns in main.go, direct_runner.go, or hybrid_runner.go
  2. Use the direct execution approach with os/exec to run Go commands
  3. Ensure your test covers specific scenarios or edge cases

Note: All new tests should be implemented using Go, not PowerShell.

Prerequisites

  • Go 1.16 or newer (primary requirement)
  • Go Development MCP Server (executable available in the build directory)
  • PowerShell 5.1 or newer (only required for legacy tests)

Troubleshooting

If Go-based tests fail:

  1. Check that the server executable path is correct
  2. Verify that Go is properly installed and in your PATH
  3. Check for any required dependencies
  4. Use the -v flag with go run for more verbose output
  5. Inspect test output for specific error messages

For legacy PowerShell tests:

  1. Use the -VerboseOutput flag for more detailed output
  2. Use the -KeepTestDirs flag to preserve test directories for inspection

Recent Improvements

May 22, 2025 Updates

Documentation Modernization

  • Restructured README to prioritize Go-based testing
  • Moved PowerShell test documentation to legacy section
  • Clarified that PowerShell tests are maintained for reference only
  • Aligned documentation with current testing practices

Simplified Dependency Structure

  • Removed dependency on handlers package and context parameters
  • Moved all PowerShell tests and test_utils.ps1 to the legacy directory
  • Updated standalone test files to use direct Go command execution
  • Eliminated all dependencies on old utility functions

Go-Native Testing Approach

  • Replaced MCP handler-based execution with direct Go command execution
  • Implemented simplified tool matching function directly in middleware
  • Removed context usage from test runner for simplicity
  • Built project with zero compilation errors

Improved Code Organization

  • Maintained backward compatibility for legacy tests
  • Ensured all test files follow the same pattern of direct command execution
  • Simplified code structure by removing unnecessary abstractions
  • Enhanced maintainability through reduced complexity

Previous Enhancements

Improved Test Result Tracking

  • Fixed test result counting in legacy scripts
  • Added consistent exit code handling in all test scripts
  • Proper scoping for test results collections

Enhanced Reliability

  • Added server availability checking to legacy E2E tests
  • Implemented file locking protection with retries
  • Improved cleanup procedures with garbage collection

For more information on the Go Development MCP Server, refer to the main README.md.