Automated Performance Regression Detection with Statistical Change Point Analysis

[JamesEjembi]

Problem Statement

Performance regressions are discovered in production after deployment. Implement a pre-deployment performance regression detection system that uses statistical change point analysis on benchmark results.

Technical Bounds

• Analysis: E-Divisive with Medians (EDM) change point detection
• Metrics: p50/p95/p99 latency, throughput, error rate
• Test duration: 5 minutes per benchmark scenario
• Alert: >2%% regression in any metric triggers CI block

Steps

1. Design benchmark scenarios for critical paths
2. Implement statistical analysis pipeline
3. Build baseline comparison with historical data
4. Add CI block on regression detection

[anilyagiz]

Hi, I’d like to work on this issue.

I have around 5 years of experience as a software developer, with hands-on experience in backend systems, performance-sensitive services, CI/CD workflows, automated testing, benchmarking, and observability. I have worked on projects where detecting latency changes, throughput drops, error rate increases, and deployment-related regressions early was important for reliability and user experience.

For this issue, I would start by identifying and designing benchmark scenarios for the most critical backend paths, focusing on realistic request patterns and measurable service behavior. Each benchmark scenario would run for the required 5-minute duration and collect p50, p95, p99 latency, throughput, and error rate metrics in a consistent format so results can be compared reliably across runs.

I would then implement a statistical analysis pipeline using E-Divisive with Medians change point detection. The goal would be to compare current benchmark results against historical baseline data and detect meaningful performance shifts rather than relying only on noisy single-run comparisons. I would also make sure the pipeline handles benchmark variability, outliers, and insufficient historical data gracefully.

For baseline comparison, I would structure the historical benchmark data so the system can evaluate recent results against previous stable runs. The implementation would calculate regression percentages per metric and apply the required threshold, where any regression greater than 2% in latency, throughput, or error rate would mark the run as failed.

I would integrate this into CI so that detected regressions block deployment before they reach production. The CI output would clearly report which benchmark scenario regressed, which metric triggered the block, the previous baseline value, the current value, and the percentage difference. This would make the result actionable for developers instead of just failing the pipeline without context.

I would also add tests around the statistical analysis logic, baseline comparison, regression threshold handling, and CI result generation. My goal would be to deliver a reliable and maintainable performance regression detection system that helps the VeriNode backend catch issues before deployment.

[Alqku]

Hello, Can I take this?