Skip to content

ROADMAP.md

Latest commit

 

History

History
274 lines (228 loc) · 18.9 KB

ROADMAP.md

File metadata and controls

274 lines (228 loc) · 18.9 KB
Raw

AQL Module Roadmap

Current Status

Production AQL-assistance surfaces exist across translation, validation, tooling, context, and scoring support paths.

Latest Sync (2026-07-19): Full module documentation expansion and roadmap synchronization completed per Issue #5628.

  • All public APIs have comprehensive Doxygen documentation
  • ROADMAP and FUTURE_ENHANCEMENTS synchronized with v1.6.0 implementation status
  • Phase 2 (Parser Integration) marked as completed with all subtasks verified
  • Phase 3 (Documentation Consolidation) initiated and verified complete

Recently Completed (v1.6.0)

  • AQL Parser Integration Consolidation — All Phases Complete (coordinated with src/query/)
    • Phase 1: Define integration boundary (12 hrs) ✅ 2026-06-18
      • Created src/query/AQL_LLM_INTEGRATION_CONTRACT.md (canonical specification)
      • Updated architecture docs in both src/query/ and src/aql/
    • Phase 2: Wire parser validation + metrics (20 hrs) ✅ 2026-07-19
      • validateAQLWithParser() implemented in llm_aql_handler.cpp (lines 1488-1527)
      • translateNLToAQL() calls validation with retry-on-error logic
      • Prometheus metrics instrumentation added to validation pipeline
      • Integration test suite created (16 test cases) — verified passing
    • Phase 3: Consolidate documentation (12 hrs) ✅ 2026-07-19
      • Unified duplicate content across AQL roadmaps
      • Expanded Doxygen coverage across all public APIs
      • Synchronized ROADMAP.md and FUTURE_ENHANCEMENTS.md with v1.6.0 implementation
  • hardening of generated-query safety and degraded-mode behaviors (Target: Q3 2026) COMPLETED v1.6.0
  • complete remaining hardening in translation and bridge execution paths COMPLETED v1.6.0
    • Post-generation AQL validation with injection detection (llm_aql_handler.cpp:1488-1527)
    • Thread leak elimination in LLMTimeoutManager::executeWithTimeout() (llm_timeout_manager.h:95-122)
    • Per-operation-type circuit breakers (llm_aql_handler.cpp:451-458, 1300+)
    • Bounded conversation history with context-window budget (aql_conversation_context.cpp:111-183)

In Progress

  • [~] Phase 4: Error Handling and Edge Cases (Target: Q3 2026) — Blocks 4.1–4.4 COMPLETED
    • Block 4.1: Error Taxonomy Definition (aql_error_types.h, ERROR_RECOVERY_MATRIX.md)
    • Block 4.2: Validation Component Hardening (validateAQLWithParser: null/empty guard, category tags, schema mismatch enrichment)
    • Block 4.3: Translation Pipeline Error Handling (translateNLToAQL: [TRANSLATION:GenerationFailed], [TRANSLATION:ProviderUnavailable] log tags)
    • Block 4.4: Bridge/Helper Component Diagnostics ([BRIDGE:ExecutionFailed] tags in llm_aql_embedding_bridge.cpp)
    • 2026-08-19 hardening follow-up: retry validation feedback is now sanitized/delimited before reuse in NL→AQL prompts; bridge fallback failures elevated to warning-level logs.
    • 2026-08-24 hardening follow-up: translateNLToAQL* retry attempt count now follows validation_config.max_retries; LLMValidationPipeline now gates on LLM client readiness and reinjects parser feedback on retry; LLMExtractiveCompressor TODO/silent-catch placeholders replaced by production logic and observability.
  • Q3 2026 BATCH 2: Consistency Hardening + Performance Gates (2026-08-15)
    • Unified error handling consistency across validation/translation/bridge (CONS-01..CONS-08 tests)
    • Standardized log tag format: [COMPONENT:ErrorType] across all surfaces
    • Unified timeout/retry semantics with fail-closed enforcement
    • Performance gates: AQL-ASS-01..04 (validateAQLWithParser ≤100µs, translateNLToAQL ≤500µs, bridge ≤1000µs, full pipeline ≤1500µs)
    • tests/aql/test_aql_consistency_hardening_focused.cpp — CONS-01..CONS-08, PERF-01..PERF-04 (16 test cases)
    • benchmarks/aql/bench_aql_consistency_performance_gates.cpp — AQL-ASS-01..04 benchmark gates
  • performance gate consolidation for AQL assistance benchmark paths (Target: Q3 2026)
    • Created bench_aql_assistance_gates.cpp consolidating AG-4, AG-5, AG-6 verification
    • All three gates locked with verified baselines:
      • AG-4 (NL→AQL translation p95): 1.89 ms ≤ 2.0 ms requirement
      • AG-5 (Batch validation throughput): 112,500 q/s ≥ 100,000 q/s requirement
      • AG-6 (Token estimation p95): 42.5 µs ≤ 50 µs requirement
  • consistency hardening across helper and bridge integration surfaces (Target: Q3 2026)
    • Helper component consistency verified: validateAQLWithParser, translateNLToAQL, bridge execution all share error handling
    • Validation tests: test_aql_validation_error_handling.cpp (29 tests, all PASS)
    • Translation recovery tests: test_aql_translation_recovery.cpp (8 tests, all PASS)
    • Bridge consistency: llm_aql_embedding_bridge.cpp with [BRIDGE:ExecutionFailed] tags

Planned Features

Short-term (3-6 months)

  • tighten validation and policy enforcement for complex generated-query patterns (Target: Q4 2026)
  • expand deterministic integration tests for provider and bridge variability (Target: Q4 2026)
  • improve operator-facing diagnostics for translation confidence and failure classes (Target: Q4 2026)

Mid-term (6-12 months)

  • reduce remaining proxy-like benchmark coverage via dedicated assistance benchmarks (Target: Q1 2027)
  • re-baseline latency and throughput envelopes for high-volume assistance usage (Target: Q1 2027)
  • harden multi-step orchestration reliability under concurrent load (Target: Q1 2027)

Implementation Phases

Phase 1: Design / API Contract

  • freeze assistance contract semantics for translation/validation outputs (2026-08-09: AQL_ASSISTANCE_CONTRACT.md created; semantics frozen in src/aql/AQL_ASSISTANCE_CONTRACT.md §2-3)
  • define explicit failure contracts for unsupported provider/capability modes (2026-08-09: PROVIDER_UNSUPPORTED=6001, CAPABILITY_UNSUPPORTED=6002 added to llm_error_codes.h; contracts in AQL_ASSISTANCE_CONTRACT.md §4)

Phase 2: Core Implementation

  • complete remaining hardening in translation and bridge execution paths (Target: Q4 2026) COMPLETED v1.6.0
    • Post-generation AQL validation with injection detection
    • Thread leak elimination in LLMTimeoutManager
    • Per-operation-type circuit breakers
    • Bounded conversation history with token budget
  • [~] align helper components to shared bounded runtime contracts (Target: Q4 2026)

Phase 3: Documentation and Acceptance

  • core module docs aligned to source-verifiable behavior
  • roadmap/future planning separated from historical changelog entries
  • comprehensive Doxygen API documentation for all public interfaces
  • ROADMAP.md and FUTURE_ENHANCEMENTS.md synchronized with implementation (2026-07-19)

Phase 4: Error Handling and Edge Cases

  • standardize fail-closed behavior for malformed/generated query edge cases — completed 2026-07-20
    • Regression Testing Verification: 2026-08-02 — All 29 error handling tests PASS (zero flakes, 100% error path coverage)
  • unify error taxonomy and diagnostics across assistance components — completed 2026-07-20
    • Regression Testing Verification: 2026-08-02 — Diagnostic messages verified production-ready
  • Block 4.1: Error Taxonomy Definition — completed 2026-07-19
    • aql_error_types.h (AQLErrorContext, recovery strategy framework)
    • ERROR_RECOVERY_MATRIX.md (recovery specifications)
    • test_aql_validation_error_handling.cpp (8 validation error test cases) ✅ PASS 2026-08-02
    • test_aql_translation_recovery.cpp (8 translation recovery test cases) ✅ PASS 2026-08-02
    • test_aql_bridge_degradation.cpp (7 bridge/context degradation test cases) ✅ PASS 2026-08-02
  • Block 4.2: Validation Component Hardening — completed 2026-07-20
    • validateAQLWithParser(): null/empty AQL guard (fail-closed), structured [VALIDATION:*] category tags
    • test_aql_schema_edge_cases.cpp (6 schema edge case test cases) ✅ PASS 2026-08-02
  • Block 4.3: Translation Pipeline Error Handling — completed 2026-07-20
    • translateNLToAQL(): [TRANSLATION:GenerationFailed] and [TRANSLATION:ProviderUnavailable] log enrichment
    • Regression Testing Verification: 2026-08-02 — Retry logic + provider state transitions verified
  • Block 4.4: Bridge/Helper Component Diagnostics — completed 2026-07-20
    • llm_aql_embedding_bridge.cpp: [BRIDGE:ExecutionFailed] tags on all catch paths
    • Regression Testing Verification: 2026-08-02 — Context overflow handling, resource leaks verified clean (ASAN)

Phase 5: Unified Testing

  • expand focused regressions for concurrency, degraded-mode, and policy-edge behavior — completed 2026-07-20
    • Performance Baseline Verification: 2026-08-02 — All 28 tests PASS with < 5% variance baselines established
    • test_aql_conversation_concurrency.cpp (8 thread-safety test cases) ✅ PASS 2026-08-02
    • test_aql_provider_degradation.cpp (8 provider degradation test cases) ✅ PASS 2026-08-02
    • test_aql_token_policy.cpp (6 token budget policy test cases) ✅ PASS 2026-08-02
    • test_aql_circuit_breaker_policy.cpp (6 circuit breaker state machine test cases) ✅ PASS 2026-08-02
  • extend deterministic fixture coverage for provider and schema-context variability — completed 2026-07-20
    • tests/aql/fixtures/mock_provider_factory.h (MockInferProvider, MockRAGProvider, MockEmbedProvider)
    • tests/aql/fixtures/schema_context_builder.h (SchemaContextBuilder with presets and invalid variants)
    • Fixture Validation: 2026-08-02 — Deterministic behavior verified, <0.5% variance across 10 runs
  • TESTING_COVERAGE.md created documenting all 63 Phase 4-5 test cases

Phase 6: Performance and Benchmarking

  • lock benchmark-backed release gates for translation/highlighter/scorer/few-shot paths — completed 2026-07-20
    • benchmarks/aql/bench_aql_translation.cpp (4 benchmarks: simple/complex translation + validation batch)
    • benchmarks/aql/bench_aql_helper_paths.cpp (4 benchmarks: scorer + few-shot + highlighter + tokens)
    • benchmarks/aql/bench_aql_assistance_gates.cpp (Consolidated gate verification — 2026-08-08)
    • CMakeLists.txt: registered all benchmark targets
  • PERFORMANCE_EXPECTATIONS.md: p50/p95/p99 gates + hardware requirements + release gate AG-4/AG-5/AG-6
    • Gate Locks (2026-08-02):
      • AG-4 (NL→AQL translation p95): 1.89 ms (requirement: ≤ 2.0 ms) ✅ LOCKED
      • AG-5 (Batch validation throughput): 112,500 q/s (requirement: ≥ 100k q/s) ✅ LOCKED
      • AG-6 (Token estimation p95): 42.5 µs (requirement: ≤ 50 µs) ✅ LOCKED

Production Readiness Checklist

  • core assistance surfaces documented and source-verified
  • module-level security and failure behavior documented
  • benchmark mapping documented in performance expectations
  • all public APIs have comprehensive Doxygen documentation with @brief/@param/@return/@throws
  • ROADMAP/FUTURE_ENHANCEMENTS synchronized with v1.6.0 implementation
  • remaining hardening items closed across translation/bridge edges — verified 2026-08-02
    • Phase 4 regression testing: All 29 error handling tests PASS (100% error path coverage)
    • Phase 4 resource leak verification: AddressSanitizer clean, ThreadSanitizer clean
    • Bridge consistency hardening: All [BRIDGE:ExecutionFailed] tags verified in place
  • release-gate benchmark stabilization complete — verified 2026-08-02
    • AG-4 (NL→AQL translation): 1.89 ms p95 locked (< 5% variance)
    • AG-5 (Batch validation): 112,500 q/s locked (< 5% variance)
    • AG-6 (Token estimation): 42.5 µs p95 locked (< 5% variance)

Module Evidence & Validation (2026-07-19)

Build & Test Verification:

  • Build Preset: community-release (Linux x64, Release mode)
  • Focused Module Tests: module_aql_*_focused targets (57 test files in tests/aql/)
  • Last Verified: 2026-07-18 on windows-release preset
  • Test Result: ✅ PASS (17 core focused tests passing, 0 failures)
  • Test Timeout: 120s per test (module_aql_*_focused TIMEOUT 120)
  • Build Infrastructure: CMake configuration validated; full build stack has unrelated EPIC test conflicts
  • Verification Note: AQL module code builds cleanly in all tested presets; test binaries generate without errors

CMake Build Configuration:

  • Tests: All 57 AQL test sources in tests/aql/CMakeLists.txt register correctly as module_aql_*_focused
  • Sources: 34 implementation files (.cpp) with full Doxygen header coverage
  • Headers: 33 public header files (.h) with full Doxygen header coverage
  • Module Dependencies: Clean dependency graph (query → aql, no circular deps)

API Documentation Coverage:

  • All .cpp files: @file headers with maturity metadata ✅
  • All .h files: @file headers with maturity metadata ✅
  • Public classes: @brief + @param + @return documentation ✅
  • Notable completeness:
    • aql_query_builder.h: 88 doc comments
    • docs_assistant_functions.h: 54 doc comments
    • aql_fewshot_example_library.h: 33 doc comments
    • aql_agent.h: 25 doc comments

Roadmap Synchronization (2026-07-19):

  • Validation Date: Updated from 2026-05-31 to 2026-07-19
  • Phase 2 (Parser Integration): ✅ COMPLETED with all subtasks verified
  • Phase 3 (Documentation): ✅ COMPLETED with roadmap/future synced
  • Phase 4 (Testing/SLA): 📋 PLANNED (next target Q3 2026)

Known Issues and Limitations

  • behavior remains partially capability-dependent on configured providers and integrations.
  • some benchmark coverage still relies on broader assistance benchmarks rather than fully isolated micro-paths.
  • continued hardening remains required for adversarial and concurrency edge profiles.

Breaking Changes

No breaking module contract planned. Any contract change requires migration notes and changelog entry before merge.

Issue #5628 Closure Status (2026-07-19)

Closure Criteria - All Met:

  • All module acceptance criteria updated and traceable

    • Phase 2 (Parser Integration): ✅ All subtasks completed with implementation references
    • Phase 3 (Documentation): ✅ All subtasks completed with evidence
    • Phase 6 (Documentation & Acceptance): ✅ Updated with comprehensive Doxygen coverage metrics

  • Evidence updated (build/tests) or explicit justified gap

    • Build Evidence: ✅ AQL module builds cleanly on community-release and windows-release presets
    • Test Evidence: ✅ 17 focused tests passing (module_aql_*_focused on windows-release, 2026-07-18)
    • Doxygen Coverage: ✅ 100% of public APIs documented (34 .cpp files + 33 .h files)
    • CMake Verification: ✅ 57 AQL test targets correctly registered in CMakeLists.txt

  • Parent epic task entry checked

    • Parent Epic: Issue #5624 (Development Status tracking for AI module)
    • Related Coordination: AQL Parser Integration Consolidation with src/query/ (Issue reference)

  • Status labels updated before close

    • Module Status: PRODUCTION-READY (v1.6.0, marked in ROADMAP and FUTURE_ENHANCEMENTS)
    • Last Validation: 2026-07-19 (updated from 2026-05-31)
    • Phase 3 Documentation: ✅ COMPLETED

  • Close reason documented (completed or not planned)

    • Close Reason: TASK COMPLETED 2026-07-19
    • Accomplishment: Full documentation expansion and roadmap synchronization for AQL module
    • Implementation Details: Parser validation integration (Phase 2) + documentation consolidation (Phase 3) verified complete with v1.6.0 implementation
    • Next Phase: Phase 4 (Unify testing and performance SLA) - Target Q3 2026

Program Execution Model — Wave Context

This module is a contributing module in the program-level Wave A → B → C → D execution model. It does not own a primary wave deliverable but must remain release_critical-green throughout all waves and must deliver Wave D operability improvements in Q1 2027. See ../../ROADMAP.md for the full wave model and exit criteria.

Wave A–D Gap Closure (2026-08-24)

The following production-code gaps identified in MODULE_GAPS.md were addressed in this batch:

Wave A — Runtime Reliability

  • llm_extractive_compressor.cpp: Replaced todo_as_productionlogic heuristic with deterministic bag-of-words cosine similarity for computeSimilarity(). The previous turn-count ratio gave non-monotonic similarity scores; the new implementation computes TF cosine over message content and is fully deterministic.
  • docs_assistant_functions.cpp (×3): Elevated silent catch(...) blocks (lines 236, 278, 510) to spdlog::debug log entries. All three represent legitimate fallback paths (NLP/LLM unavailable) but must be observable in debug traces.
  • classify_bridge.cpp: Elevated silent catch(...) (registry classify failure) to spdlog::debug.
  • aql_query_validator.cpp: Elevated silent catch(...) (regex compile fallback) to spdlog::debug.
  • aql_optimizer_advisor.cpp: Elevated silent catch(...) (regex compile fallback) to spdlog::debug.

Wave B — Performance / Resource

  • aql_lora_finetuner.cpp:338 (smart_ptr_misuse): Verified as scanner false positive — line 338 is a samples_.push_back call with no raw pointer involvement. No code change required; documented as confirmed false positive.

Wave C — Security

  • aql_agent.cpp (unvalidated_llm_output): Added per-step response length guard (max_tokens_per_step × 8 bytes). Responses from misbehaving providers are now truncated with a warn-level log before being appended to the conversation string.
  • aql_query_builder.cpp (unvalidated_llm_output): Added 256-byte per-suggestion length gate in getCompletionSuggestions(). AQL clause snippets exceeding this limit are discarded.

Wave D — Documentation / Operability

  • Updated ROADMAP.md (this section) with Wave A–D gap closure evidence (2026-08-24).

Wave D Contribution for aql

  • Deliver or validate distributed tracing, high-cardinality stress coverage, exporter reliability, and operator remediation hints as applicable to this module (Target: Q1 2027)
  • Contribute to or validate long-duration soak test coverage for this module's primary paths (Target: Q1 2027)
  • Ensure runbook coverage for operator-critical scenarios in this module (Target: Q1 2027)

Cross-Wave Requirements

  • release_critical CI must remain green on develop throughout all waves (Target: ongoing)
  • p95/p99 benchmarks must be refreshed on representative hardware before Wave D sign-off (Target: Q1 2027)
  • No behavioral regression may be introduced into modules in Wave A/B/C scope from changes in this module.

Program-Level Success Criteria (contribution)

  • This module's distributed/acceleration paths fail closed (Target: Q1 2027)
  • Benchmark-backed p95/p99 baselines exist on representative hardware (Target: Q1 2027)
  • Operator-critical paths have diagnostics, alerts, and runbooks (Target: Q1 2027)