Test-Sage: Predictive Software Testing & Quality Analysis

TestSage is a research prototype designed to optimize regression testing cycles in large-scale Python repositories. It leverages Git history mining and Machine Learning to predict test failures and prioritize test execution, addressing the challenges of safeguarding quality in massive, rapidly-evolving codebases.

Research Goals

This project implements the core concepts of Predictive Test Selection:

1. Change Intent Analysis: Correlating file changes (Git diffs) with historical test failures.
2. Test Prioritization: Using Logistic Regression to rank tests by Probability of Failure, providing faster feedback to developers.
3. Quality Metrics: Going beyond code coverage by measuring Execution Cost and Flakiness (Duration Variance) to identify candidates for test refactoring.

Architecture

The pipeline consists of three stages:

1. Collector (src/collector.py): Mines the target repository (e.g., Textualize/rich) to build a dataset of commits, changed files, and fine-grained test outcomes (Pass/Fail/Duration).
2. Predictor (src/predictor.py):
   • Features: One-hot encoded file paths, historical failure rates, and test duration.
   • Model: A balanced LogisticRegression classifier trained to detect regression patterns.
   • Output: A prioritized list of tests for the current HEAD.
3. Quality Analyzer (src/quality.py): Computes a "Test Quality Index" to flag slow or unstable tests that degrade developer productivity.

Usage

1. Installation

git clone https://github.com/malihanawshin/Test-Sage.git
cd TestSage
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Mine History (Data Collection)

Run the collector on any local git repository with a pytest suite: python src/collector.py ../rich

Output: Generates test_history.csv containing commit-level test telemetry.

3. Predict & Prioritize

Train the model and get the prioritized test list for the latest commit: python src/predictor.py

Output:

--- Prediction for Latest Commit ---

Top 5 Tests Prioritized for Commit 494f83c:

• test_node failure_prob
• test_pad_left 0.770169
• test_span 0.759777
• test_assemble 0.731072
• test_assemble_meta 0.729539

4. Analyze Test Quality

Identify slow or unstable tests: python src/quality.py

Evaluation & Results

• Case Study: Evaluated on the Textualize/rich open-source repository (Top 100 Python project).
• Findings: The model successfully identifies high-risk tests based on file locality. The quality analyzer revealed that < 5% of tests account for > 20% of the total execution time, highlighting clear targets for optimization.

Future Work

• LLM Integration: Replacing keyword-based intent analysis with an LLM Agent to semantically understand why a file was changed.
• Agentic Refactoring: Building an agent that automatically refactors the "Top 5 Slowest Tests" identified by src/quality.py.