Skip to content

ROADMAP.md

Latest commit

 

History

History
237 lines (190 loc) · 13 KB

ROADMAP.md

File metadata and controls

237 lines (190 loc) · 13 KB
Raw

AI Module Roadmap

Current Status

Production runtime exists for prompt validation, endpoint invocation, JSON mapping, and structured fail-closed error handling.

Module Documentation Enhancements (2026-07-19)

  • Added comprehensive Doxygen documentation to all implementation functions
  • Documented AIPluginGenerator::validatePrompt() with detailed validation rules
  • Documented AIPluginGenerator::generatePlugin() with complete execution pipeline
  • Documented all CAI ethics integration functions with semantics and contracts
  • Added inline documentation for error handling, thread-safety, and retry policies
  • Lines added: ai_plugin_generator.cpp +133, cai_ethics_integration.cpp +130
  • Total module documentation lines: ~1,473 (up from 1,210)

In Progress

  • Validation hardening for non-description prompt fields (Target: Q3 2026)
  • Endpoint safety hardening (allow-list, response-size limits) (Target: Q3 2026)
  • Performance gate consolidation for AI generation proxy benchmarks (Target: Q3 2026)

Planned Features

Short-term (3-6 months)

  • Enforce schema-level validation for all generated payload fields (Target: Q4 2026)
  • Introduce deterministic retry/backoff policy for transient endpoint failures (Target: Q4 2026)
  • Add explicit redaction policy for diagnostic output fields (Target: Q4 2026)

Mid-term (6-12 months)

  • Integrate optional sandbox verification gate for generated code artifacts (artifact materialization + optional callback verification enforced in generator path) (Target: Q1 2027)
  • Add dedicated benchmark target for AI plugin generation path (Target: Q1 2027)
  • Expand observability counters for error classes and endpoint quality signals (Target: Q1 2027)

Long-term (Q3 2027+)

  • Wave C C1: Constitutional AI (CAI) safety module with 21 built-in principles, critic-revision loop, EthicsEvaluator integration, and CAI-01..15 + CAI-BENCH-01 coverage — include/ai/cai_ethics_integration.h, tests/test_cai_safety_module.cpp
  • Wave C C2: Federated learning coordinator with secure aggregation, Byzantine-robust averaging, DP tuning, and FEDERATED-01..15 + FEDERATED-BENCH-01 coverage — tests/test_federated_privacy_training.cpp
  • Human safety benchmark program for C1 (500 samples, 3 annotators) and convergence benchmark for C2 (10-node setup) — tests/test_cai_safety_module.cpp (CAI-BENCH-01), tests/test_federated_privacy_training.cpp (FEDERATED-BENCH-01)
  • Integrate optional sandbox verification gate for generated code artifacts (Target: Q1 2027)
  • Add dedicated benchmark target for AI plugin generation path (Target: Q1 2027)
  • Expand observability counters for error classes and endpoint quality signals (Target: Q1 2027)
  • Wave B B1: Self-RAG design/implementation/benchmark package (Target: Q1–Q2 2027) — core impl + IEE integration + ALCE acceptance-gate coverage done
  • Wave B B2: RotatE knowledge-graph completion integration package (Target: Q1–Q2 2027) — core impl + KGC-01..15 tests + TransE-baseline acceptance coverage done
  • Wave B B3: Multi-task LoRA fine-tuning package (Target: Q1–Q2 2027) — core impl + ablation/benchmark + acceptance-gate coverage done

Implementation Phases

Phase 1: Design / API Contract

  • Stable API for prompt/config/result types in public header
  • Validation-first behavior contract defined and implemented

Phase 2: Core Implementation

  • Endpoint invocation path implemented with configurable transport
  • JSON response mapping to GeneratedPlugin implemented

Phase 3: Error Handling and Edge Cases

  • Non-2xx, transport, and parse failures normalized to structured errors
  • Extended validation for capability/dependency fields (Target: Q3 2026)

Phase 4: Tests

  • Focused unit coverage for constructor, validation, and endpoint/error paths
  • Integration suite with deterministic endpoint fixtures (Target: Q3 2026)

Phase 5: Performance and Hardening

  • Add module-specific benchmark instead of proxy-only tracking (Target: Q1 2027)
  • Enforce endpoint allow-list and payload size bounds (Target: Q4 2026)

Phase 6: Documentation and Acceptance

  • Core module docs aligned with source-verifiable behavior
  • Completed work tracked in changelog; roadmap remains forward-looking

Production Readiness Checklist

  • Validation-first execution path documented and verified
  • Structured error handling for endpoint and parse failures verified
  • Proxy benchmark mapping documented in performance expectations
  • Dedicated benchmark target registered
  • Hardening follow-ups closed for endpoint safety controls

Known Issues and Limitations

  • Wave C C1/C2 production-runtime integration now covers AIPluginGenerator and LLMAQLHandler (executeInfer, executeInferStreaming, executeRAG, executeChat) via opt-in safety-gate and telemetry hooks.
  • Schema-level validation for all LLM output fields is enforced: code fields ≤ 1 MiB, security_report ≤ 64 KiB, version ≤ 64 chars (defaults to 0.1.0), manifest.description truncated at 8192 chars, oversized build_dependencies entries silently dropped.

Wave C Timeline and Dependencies

  • Start: Q3 2027 (early July)
  • Target: End Q4 2027 (mid-December)
  • Estimated Effort: 16–24 weeks total (depending on federated security requirements)

Dependencies

  • Wave A + Wave B stability checks tracked in focused regression suites and release-gate docs (CTEST.md, Wave A #5038, Wave B #5039)
  • Constitutional AI principles formalized in ethics framework (src/ai/cai_ethics_integration.cpp, tests/test_cai_safety_module.cpp)
  • Multi-node federated benchmark infra/security review tracking established (FEDERATED-BENCH-01 coverage + issue traceability #5040/#5039)

References

  • src/ai/FUTURE_ENHANCEMENTS.md#wave-c--strategic-ml-enhancements-q3-2027
  • docs/research/ml_enhancements_bibliography.md
  • #5040 (Wave C Issue)
  • #5038 (Wave A Issue)
  • #5039 (Wave B Issue)

Wave C Research Publication Opportunity

  • Joint paper: ThemisDB Integration of Research-Backed ML Features
  • Target venue window: MLSys 2028 / FAccT 2028
  • Dedicated benchmark coverage exists via benchmarks/bench_ai_plugin_generator.cpp / benchmarks/ai/bench_ai_plugin_generator.cpp.
  • Advanced field-level prompt validation remains incomplete.
  • Sandbox artifact materialization and optional callback verification are enforced when enable_sandbox_gate is enabled; external sandbox engines remain deployment-specific.
  • Wave B ML enhancement implementation and acceptance-gate coverage are complete; production promotion remains gated on Wave A deployment completion and latency prerequisites.

Wave B (Q1–Q2 2027) Tracking

B1: Self-RAG (Self-Retrieving, Auto-Critique)

  • Retrieval controller (binary classify: retrieve now?)
  • Critic model (Relevant/Partial/Irrelevant)
  • Iterative refinement loop (max 3 rounds)
  • Unit tests SELF_RAG-01..12
  • InferenceEngineEnhanced callback integration
  • ALCE benchmark vs vanilla RAG

B2: Knowledge Graph Completion (RotatE)

  • RotatE embedding model (relation-as-rotation)
  • Triple loss with negative sampling
  • Link-prediction head
  • Unit tests KGC-01..15
  • TransE baseline benchmark
  • KnowledgeGraphReasoner integration

B3: Multi-Task LoRA Fine-Tuning

  • Shared LoRA base + task-specific projections
  • Domain-gating mechanism
  • Joint loss with configurable task weighting
  • Unit tests MTL-01..10
  • Shared-vs-single-task baseline ablation
  • 3-task benchmark evaluation

Acceptance Gates

  • Hallucination rate reduction ≥ 20% vs standard RAG
  • Self-RAG latency increase ≤ 1.5× vs baseline
  • Precision@K retrieval ≥ 0.85 on golden-doc tests
  • RotatE MRR ≥ 0.35 and Hits@10 ≥ 0.55 on deterministic acceptance fixture
  • RotatE inference latency ≤ 50 ms for top-20 predictions
  • Multi-task LoRA average task performance ≥ +8% vs single-task
  • Multi-task LoRA training time increase ≤ 15%

Dependencies

  • Wave A deployment complete (Speculative Decoding, DPR, Fairness)
  • LLM inference P95 latency < 200 ms
  • KnowledgeGraphReasoner stable + benchmark suite passing

References

  • Research bibliography: ../../docs/research/ml_enhancements_bibliography.md
  • Future enhancements detail: FUTURE_ENHANCEMENTS.md
  • Issue scope: https://github.com/makr-code/ThemisDB/issues/5039

Module Validation Evidence (2026-07-19)

Documentation Status

  • @file Doxygen Headers: 100% coverage (4/4 files)

    • include/ai/ai_plugin_generator.h — hardened implementation metadata
    • include/ai/cai_ethics_integration.h — hardened implementation metadata
    • src/ai/ai_plugin_generator.cpp — hardened implementation metadata
    • src/ai/cai_ethics_integration.cpp — hardened implementation metadata

  • Function/Method Documentation: Enhanced 2026-07-19

    • Added comprehensive Doxygen comments to all public methods
    • Added parameter/return/error documentation to key functions
    • Documented thread-safety contracts and error handling semantics
    • Implementation files received the primary Doxygen expansion (+263 lines across 2 .cpp files)
    • Header files received follow-up contract clarifications for transport, validation, and latency semantics

Code Quality Metrics

  • Lines of Code:

    • Source: 946 lines (ai_plugin_generator.cpp: 634, cai_ethics_integration.cpp: 312)
    • Headers: 476 lines (ai_plugin_generator.h: 283, cai_ethics_integration.h: 193)
    • Total: 1,422 lines (module core)

  • API Stability:

    • Public API stable (AIPluginGenerator, CAIEthicsIntegration)
    • Config structures fully documented with field semantics
    • Callback types fully documented with signatures and contract details
    • Zero breaking changes in current implementation

Feature Coverage

  • ✅ Phase 1: Stable API for prompt/config/result types
  • ✅ Phase 2: Endpoint invocation with configurable transport
  • ✅ Phase 3: Structured error handling and edge cases
  • ✅ Phase 4: Focused unit test coverage (2 test files: test_ai_decision_auditor.cpp, test_ai_plugin_generator.cpp)
  • ✅ Phase 5: Observable counters implemented (Stats struct with 7 counters)
  • ✅ Phase 6: Documentation aligned with implementation

Hardening Completeness

  • ✅ Prompt validation (description length, token list sizes, format validation)
  • ✅ Request sanitization (ASCII control character stripping)
  • ✅ Endpoint allow-list enforcement
  • ✅ Request/response size limits (256 KiB / 8 MiB)
  • ✅ Retryable endpoint invocation (3 attempts, exponential backoff)
  • ✅ Response parsing with malformed input rejection
  • ✅ Output field validation (code size, manifest fields)
  • ✅ Optional C1 CAI safety gate (Wave C feature)
  • ✅ Optional sandbox artifact materialization + callback verification enforced in the generator path
  • ✅ Optional C2 federated telemetry (Wave C feature)
  • ✅ Observability counters for error classes

Production Readiness Status

  • Maturity: 🟡 Hardened implementation (all HIGH-severity gaps reviewed; full production validation still pending environment-complete build/test)
  • Configuration: Validatable via CMakeLists.txt, CMakePresets.json
  • Testing: Focused test targets auto-discovered (module_ai_*_focused.exe)
  • Error Handling: Fail-closed with structured Error result types
  • Logging: Redaction-aware with configurable truncation (120 chars max)
  • Thread Safety: Document-specified (not thread-safe for concurrent generatePlugin)
  • Memory Safety: RAII compliance, smart pointers for ownership
  • Security: Input validation, output bounds checking, endpoint allow-listing

Known Limitations (MODULE_GAPS.md reference)

  • HIGH-severity scanner findings were reviewed and either remediated or reclassified with source-backed explanations (2026-07-19)
  • MEDIUM-priority scope/documentation gaps remain tracked in MODULE_GAPS.md
  • Dedicated benchmark target for the AI plugin generator path is registered and tracked in benchmark docs
  • Sandbox verification now includes built-in artifact materialization, read-back verification, and optional callback enforcement

Acceptance Criteria Status

  • Validation-first execution path implemented and verified
  • Structured error handling for all failure points
  • Hardening follow-ups complete for endpoint safety, payload validation, observability counters, benchmark coverage, and sandbox artifact verification
  • API documentation complete and comprehensive
  • Implementation semantics documented at function level
  • Roadmap and Future Enhancements synchronized (2026-07-19)
  • All HIGH-severity gaps reviewed with either code remediation or source-backed disposition notes (2026-07-19)

Breaking Changes

No breaking API change planned. Any signature/semantic contract change requires explicit migration notes and changelog entry.