Spec: queue-drain-20260614T141231

docs/specs/queue-drain-20260614T141231.md

@@ -0,0 +1,64 @@
+<!-- generated_by_run: f011a0f7-c575-4a4e-9bdc-755fd146f9cd -->
+
+---
+campaign_id: 7c5e2a3f-b1d9-47f8-a4c2-e9f6d8a5b2c1
+slug: queue-drain-20260614T141231
+phases:
+  - implement
+  - test
+  - improve
+repos:
+  - OperationsCenter
+area_keywords:
+  - observer
+  - snapshot-validation
+  - ci-compatibility
+  - custodian-audit
+  - performance-testing
+status: active
+created_at: 2026-06-14T14:12:31Z
+---
+
+## Overview
+
+The observer snapshot validation CLI completed implementation in recent stages but now exhibits stability and audit issues requiring targeted hardening. Recent commits (46e53534, 5967fb5c, ff691fa9) reveal Python 3.11 compatibility gaps (ANSI code stripping, argument parsing), unresolved Custodian violations, and incomplete performance coverage for large metric sets. This campaign stabilizes the snapshot validation subsystem across CI platforms, clears audit findings, extends performance testing, and hardens error reporting for production readiness.
+
+## Goals
+
+1. **Resolve Python 3.11 and CI platform compatibility issues** — Fix ANSI code handling in test output (test_version_in_help strips codes on Python 3.11+), add is_eager=True to --version argument for compat, validate argparse behavior across Python 3.9–3.12 targets. Add integration tests for version output, help text, and error messages across target Python versions. Verify all CLI invocations emit clean output without platform-specific escape sequences. Success: All snapshot validation CLI tests green on Python 3.9, 3.10, 3.11, 3.12 in CI; zero ANSI code-related failures.
+
+2. **Clear Custodian audit violations in observer validation module** — Audit observer CLI and validation functions for Custodian gate violations (license headers, lint rules, docstring compliance, import organization). Fix identified violations in `src/operations_center/observer/cli.py` and validation layer modules. Re-run `custodian audit` and validate zero findings. Add pre-commit hooks or CI gates to prevent regressions. Success: Custodian audit returns clean result; pre-push validation includes observer module audit check.
+
+3. **Extend snapshot serialization performance testing to cover real-world patterns** — Add performance tests for mixed metric types (counters, histograms, gauges, distributions), realistic time-series ranges (1h, 24h, 7d windows), and error-injection scenarios (missing fields, malformed data). Validate sub-second latency targets for snapshots under 50K metrics. Add memory profiling for serialization peak usage. Success: ≥10 new performance tests; all latency assertions pass; memory profiling identifies peak usage patterns.
+
+4. **Improve snapshot validation error reporting and diagnostics** — Enhance validation failure messages with actionable guidance (e.g., "Layer 2 completeness failure: test_signal missing; add test execution step to pipeline"). Add structured JSON error logs with validation context (layer, metric, threshold, actual value, recommendation). Integrate error diagnostics into validation report output. Success: Users can diagnose validation failures without CLI source code inspection; error messages include specific remediation steps.
+
+## Constraints
+
+- **Scope:** Stability, audit, and testing only — no changes to validation logic, layer thresholds, or snapshot schema.
+
+- **Allowed paths:** Modify `src/operations_center/observer/cli.py` for argument/output compat; add tests under `tests/unit/observer/` for compat and performance; extend performance test fixtures; add structured logging to error reporting. Fix Custodian violations per audit guidance. Use existing test infrastructure (pytest, performance benchmarks).
+
+- **Avoid:** Refactoring validation layers, changing Layer 1–5 thresholds, modifying snapshot serialization core, or adding external dependencies. Do not introduce feature flags; fix issues outright.
+
+- **Cross-platform:** Tests must pass on Linux, macOS, and Windows CI runners. ANSI code handling must work on all platforms; platform-specific path handling is acceptable with guards.
+
+- **Performance targets:** Snapshot validation for snapshots ≤50K metrics must complete in <1s; serialization peak memory for 50K-metric snapshot ≤500MB; deserialization throughput ≥10K metrics/sec.
+
+- **Isolation:** All tests deterministic, parallel-safe, and independent of system locale or terminal settings. No external process spawns. Clock-injection for time-dependent tests.
+
+- **Custodian compliance:** All modified code must pass `custodian audit --strict` with zero findings. License headers, import ordering, docstring format, and line-length checks must be clean.
+
+- **Backward compatibility:** No breaking changes to validation report schema, exit codes, or CLI argument interface. Enhancements (error detail, structured logs) must be additive.
+
+## Success Criteria
+
+1. **CI platform compatibility validated:** Observer snapshot validation CLI tests pass on Python 3.9, 3.10, 3.11, 3.12 in GitHub Actions CI. Zero ANSI code or argparse failures. Help/version/error output is clean and consistent across platforms.
+
+2. **Custodian audit clean:** `custodian audit --strict` on observer module returns zero findings. License headers present and correct. Import ordering follows ruff standards. Docstrings meet project format. Line length and complexity checks pass.
+
+3. **Performance testing complete:** ≥10 new tests added for serialization patterns (mixed metric types, realistic time ranges, error injection). All latency assertions pass (<1s for ≤50K metrics). Memory profiling shows peak usage patterns. Performance test suite documents baseline and regression thresholds.
+
+4. **Error reporting actionable:** Validation failures include layer-specific remediation guidance. Structured JSON logs capture validation context (metric, threshold, actual value). Users can diagnose failures from error message and logs alone; zero "check source code" guidance needed.
+
+5. **Regression prevention:** Pre-push hook or CI gate enforces Custodian audit on observer module. Compat tests run in CI on all target Python versions. Performance tests include regression detection (latency/memory thresholds). All tests stable in 3 consecutive CI runs; zero flakiness.