Skip to content

STAGE_1_IMPLEMENTATION_GUIDE.md

Latest commit

 

History

History
455 lines (365 loc) · 11.8 KB

STAGE_1_IMPLEMENTATION_GUIDE.md

File metadata and controls

455 lines (365 loc) · 11.8 KB
Raw

Stage 1 Implementation Guide: Metadata Filtering Foundation

Status: Code written, ready for compilation and testing
Scope: v0.5 Foundation (Sep-Oct 2026, 8 weeks)
Goal: Pre-retrieval intelligence layer that decides what to fetch before fetching

What Was Built

1. Metadata Types (core/src/metadata/types.rs)

Complete type system for metadata across all sources:

Web Metadata:

  • URL, domain, publish timestamp
  • Size, SSL, domain age, wayback depth
  • Topic relevance, tags
  • Automatic quality scoring (authority, freshness, accessibility, cost, reliability)

Database Metadata:

  • Database type, tables, columns
  • Row count, update frequency, access cost
  • Data quality score
  • Column metadata (type, nullable, cardinality, indexed)

MCP Tool Metadata:

  • Name, description, capabilities
  • Input/output types
  • Latency, cost per call, success rate
  • Auth type

Quality Calculation:

  • Authority: Based on domain reputation (SSL, age, wayback)
  • Freshness: Based on update recency
  • Accessibility: Based on connectivity/auth
  • Cost Efficiency: Inverse of retrieval cost
  • Reliability: Based on success rate

2. Metadata Filtering Engine (core/src/metadata/filter.rs)

Intelligent ranking algorithm:

Ranking Strategies:

  1. Quality: Prioritize authority + reliability (best for accuracy)
  2. CostOptimized: Minimize cost while maintaining quality
  3. Freshness: Prioritize recent sources (for trending queries)
  4. Balanced: Equal weighting (default)

Scoring Algorithm:

base_score = quality.overall_score(weights)

adjustments = {
  topical_boost: topic_relevance × query_topic_weight
  domain_boost: if tags match query domain (0.2 bonus)
  freshness_factor: strategy-dependent multiplier
}

final_score = (base_score × 0.4 + topical_boost × 0.3 + domain_boost × 0.2) 
              × (1.0 + freshness_factor × 0.1)

Query Feature Extraction:

  • Domain tags: documentation, tutorial, api_reference, news
  • Required capabilities: search, analysis, transform, generation
  • Required fields: column-like names extracted from query
  • Topic weight: based on query length and specificity
  • Auth available: api_key, oauth, none

Explainability:

  • Each ranked candidate includes justification
  • Shows: score breakdown, authority, freshness, cost, token estimate
  • Humans can understand why each source was selected

3. Metadata Cache (core/src/metadata/cache.rs)

Shared learning layer:

Features:

  • TTL-based expiry (configurable)
  • LRU eviction when cache full
  • Statistics tracking (hits, misses, evictions)
  • Thread-safe (Arc + RwLock)
  • Memory usage estimation

Key Behavior:

  • Cache key: query_text::source_type (normalized to lowercase)
  • Default TTL: 1 hour
  • Default max size: 1000 entries
  • Automatic eviction: removes lowest-access entry

Learning:

  • Every filtering decision is cached
  • Subsequent identical queries benefit from cache
  • Cache statistics show effectiveness (hit rate)
  • Supports cache clearing for testing/refresh

4. Integration Point (core/src/metadata/mod.rs)

High-level MetadataIntelligence API:

pub struct MetadataIntelligence {
    filter: MetadataFilter,
    cache: MetadataCache,
}

impl MetadataIntelligence {
    pub async fn rank_candidates(...) -> Result<Vec<RankedCandidate>>
    pub async fn get_top_candidates(..., top_k: usize) -> Result<Vec<RankedCandidate>>
    pub fn clear_cache() -> Result<()>
    pub fn cache_stats() -> Result<CacheStats>
}

Architecture Decisions

1. Metadata-First, Not Data-First

  • Ranking uses ONLY metadata (no data retrieval needed)
  • Latency: < 50ms for ranking 100 candidates
  • No false positives from premature ranking

2. Quality Scoring Over Simple Heuristics

  • Weighted quality scores (5 dimensions)
  • Customizable weights per query
  • Explainable scoring (justification strings)

3. Shared Cache for Multi-Agent Learning

  • All agents benefit from cached decisions
  • Statistics track cache effectiveness
  • Foundation for v1.0 multi-agent coordination

4. Extensible Type System

  • Enum-based Metadata type (Web/Database/MCPTool)
  • Easy to add new source types
  • Each type has distinct quality scoring

5. Ranking Strategies

  • Multiple strategies (Quality/Cost/Freshness/Balanced)
  • Configurable weights
  • Switch strategies per query if needed

Code Structure

core/src/metadata/
├── mod.rs              # Module definition + MetadataIntelligence API
├── types.rs            # Metadata types, quality calculation
├── filter.rs           # Ranking engine, scoring algorithm
└── cache.rs            # Caching layer, TTL/eviction logic

tests/
└── metadata_filtering_tests.rs  # Comprehensive test suite

Documentation:
├── STAGE_1_IMPLEMENTATION_GUIDE.md (this file)
├── Metadata Catalog Examples (TODO - populate catalogs)
└── API Reference (TODO - add to docs/)

Next Steps for Completion

Phase 1: Fix Cargo Issue

Currently blocked by idna_adapter requiring edition2024 which needs Rust 1.82+.

Action: Upgrade Rust toolchain

rustup update

Phase 2: Compilation & Testing

# Build core module
cargo build -p pystreammcp-core

# Run tests
cargo test -p pystreammcp-core metadata

# Check benchmarks
cargo bench -p pystreammcp-core metadata_filtering_benchmarks

Target Metrics:

  • Compilation: No warnings
  • Tests: 100% pass rate (25+ test cases)
  • Benchmarks: < 50ms for full pipeline (100 candidates)

Phase 3: Populate Metadata Catalogs

Web Domains (50+ profiles):

mcp_catalog/web_domains/
├── documentation/      # API docs, guides, tutorials
│   ├── openai_api.yaml
│   ├── anthropic_docs.yaml
│   └── aws_documentation.yaml
├── technical/         # Technical references, specs
│   ├── rust_docs.yaml
│   ├── pytorch_docs.yaml
│   └── kubernetes_docs.yaml
├── forums/           # Q&A, discussions
│   ├── stackoverflow.yaml
│   ├── github_discussions.yaml
│   └── reddit_programming.yaml
└── news/             # Current news, trends
    ├── techcrunch.yaml
    ├── hackernews.yaml
    └── arXiv.yaml

Database Schemas (25+ profiles):

mcp_catalog/database_schemas/
├── e_commerce/
│   ├── customers.yaml
│   ├── orders.yaml
│   └── products.yaml
├── crm/
│   ├── accounts.yaml
│   ├── contacts.yaml
│   └── opportunities.yaml
└── analytics/
    ├── events.yaml
    ├── sessions.yaml
    └── user_behavior.yaml

MCP Tools (20+ profiles):

mcp_catalog/mcp_tools/
├── search/
│   ├── google_search.yaml
│   ├── bing_search.yaml
│   └── duckduckgo_search.yaml
├── analysis/
│   ├── sentiment_analyzer.yaml
│   ├── entity_extractor.yaml
│   └── topic_modeler.yaml
└── generation/
    ├── code_generator.yaml
    ├── summarizer.yaml
    └── translator.yaml

Phase 4: Python Bindings

Create Python wrappers for metadata filtering:

from pystreammcp.metadata import MetadataIntelligence

intelligence = MetadataIntelligence()
ranked = intelligence.rank_candidates(
    query="best practices for retention",
    source_type=SourceType.WEB,
    candidates=[...],
)

Phase 5: Integration Tests

Test Stage 1 in actual query flow:

#[tokio::test]
async fn test_metadata_filtering_in_query_flow() {
    // 1. Create query
    // 2. Run through MetadataIntelligence
    // 3. Get top-3 candidates
    // 4. Verify ranking makes sense
    // 5. Check cache for next iteration
}

API Reference

MetadataIntelligence

impl MetadataIntelligence {
    /// Create new metadata intelligence layer
    pub fn new(config: FilterConfig) -> Result<Self>

    /// Rank candidates using metadata
    pub async fn rank_candidates(
        &self,
        query: &str,
        source_type: SourceType,
        candidates: Vec<Metadata>,
    ) -> Result<Vec<RankedCandidate>>

    /// Get top-k candidates (for selective retrieval)
    pub async fn get_top_candidates(
        &self,
        query: &str,
        source_type: SourceType,
        candidates: Vec<Metadata>,
        top_k: usize,
    ) -> Result<Vec<RankedCandidate>>

    /// Clear cache
    pub fn clear_cache() -> Result<()>

    /// Get cache statistics
    pub fn cache_stats() -> Result<CacheStats>
}

Configuration

pub struct FilterConfig {
    pub ranking_strategy: RankingStrategy,
    pub quality_weights: QualityWeights,
    pub cache_config: CacheConfig,
}

pub enum RankingStrategy {
    Quality,
    CostOptimized,
    Freshness,
    Balanced,
}

pub struct QualityWeights {
    pub authority: f64,      // 0-1
    pub freshness: f64,      // 0-1
    pub accessibility: f64,  // 0-1
    pub cost_efficiency: f64,// 0-1
    pub reliability: f64,    // 0-1
}

pub struct CacheConfig {
    pub ttl_seconds: u64,    // 0 = no expiry
    pub max_entries: usize,  // LRU eviction
}

Test Coverage

Total: 25+ test cases

Unit Tests (15+)

  • Web metadata quality calculation
  • Database metadata quality calculation
  • MCP tool quality calculation
  • Metadata filter ranking (all source types)
  • Query feature extraction
  • Cache set/get operations
  • Cache expiry handling
  • Cache eviction
  • Cache clearing

Integration Tests (5+)

  • Selective retrieval (top-k)
  • End-to-end filtering pipeline
  • Concurrent cache access
  • Cache statistics accuracy
  • Ranking strategy switching

Performance Benchmarks (5+)

  • Metadata quality score: < 1µs
  • Ranking 100 candidates: < 10ms
  • Cache lookup: < 1µs
  • Query feature extraction: < 1ms
  • End-to-end filtering: < 50ms

Performance Targets

Metric Target Current
Rank 1 candidate < 1µs Not yet tested
Rank 100 candidates < 10ms Not yet tested
Cache hit < 1µs Not yet tested
Feature extraction < 1ms Not yet tested
Full pipeline < 50ms Not yet tested
Memory per candidate < 1KB Not yet tested

Backward Compatibility

✅ Metadata filtering is fully backward compatible with v0.4:

  • New metadata module is opt-in
  • Existing query flow unchanged
  • Can enable/disable via config
  • No changes to existing APIs

What's NOT Included in Stage 1

These will come in Stage 2 (v1.0):

  • ❌ Contextual reranking (post-retrieval)
  • ❌ Tiered token budgets
  • ❌ Intent-based allocation
  • ❌ Token multipliers
  • ❌ StatGuardian integration
  • ❌ Web crawling (Crawl4AI)
  • ❌ Database selective queries
  • ❌ MCP tool invocation

Key Files Generated

✅ core/src/metadata/mod.rs — Module interface ✅ core/src/metadata/types.rs — Type definitions + quality scoring ✅ core/src/metadata/filter.rs — Ranking engine + scoring algorithm ✅ core/src/metadata/cache.rs — Caching layer + statistics ✅ tests/metadata_filtering_tests.rs — 25+ test cases ✅ STAGE_1_IMPLEMENTATION_GUIDE.md — This guide

Metrics for Success (Stage 1)

Metric Target
Code compiles ✓
All tests pass 25+
No warnings 0
Test coverage > 80%
Performance < 50ms/100 candidates
Backward compatible ✓
Cache hit rate > 70% (with reused queries)

Timeline to Completion

  • Day 1-2: Fix Rust/Cargo issues
  • Day 3-4: Compilation + unit tests
  • Day 5: Integration testing
  • Day 6-7: Populate metadata catalogs
  • Day 8: Python bindings
  • Day 9-10: Documentation + examples

Ready to Execute

Stage 1 foundation is designed and ready for:

  1. Compilation (once Rust 1.82+ installed)
  2. Testing (25+ test cases provided)
  3. Integration (backward compatible)
  4. Catalog population (structure defined)
  5. Transition to Stage 2 (v1.0)

Next steps depend on fixing the Rust version/Cargo dependency issue.