Proposed
LobeHub v2 operates within strict context window constraints:
| Model | Context Window | Practical Limit | Latency Threshold |
|---|---|---|---|
| GPT-4 | 128K tokens | ~40K tokens | >40K degrades |
| Claude 3 | 200K tokens | ~50K tokens | >50K degrades |
| Claude 3.5 | 200K tokens | ~60K tokens | >60K degrades |
Problems observed:
- Large CSV files (>100KB raw) quickly consume available context
- Loading full datasets into conversation history is wasteful
- Repeated queries on same data re-send entire datasets
- Complex multi-step analysis accumulates token usage exponentially
- Visualization data (base64 images) is token-expensive
Implement a tiered context management system with intelligent compression, prioritization, and incremental loading.
- Token Budgeting: Pre-allocate token budgets by component
- Progressive Disclosure: Load data incrementally based on relevance
- Compression Strategies: Multiple techniques to minimize token usage
- Caching at Context Level: Avoid re-sending unchanged information
- Smart Eviction: Remove low-priority content before high-priority
┌─────────────────────────────────────────────────────────────────────────────┐
│ Context Window (e.g., 50k tokens) │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TIER 1: IMMUTABLE (Always Present) │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ System Prompt + Core Instructions │ │ │
│ │ │ Budget: 500 tokens Priority: CRITICAL Eviction: NEVER │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TIER 2: CONVERSATION (Managed by LobeHub) │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Recent Messages + Tool Results │ │ │
│ │ │ Budget: 2000 tokens Priority: HIGH Eviction: LRU │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TIER 3: SCHEMA (Controlled by MCP) │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Active Dataset Schemas │ │ │
│ │ │ Budget: 300 tokens Priority: HIGH Eviction: LFU │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TIER 4: SAMPLES (On-Demand Loading) │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Representative Data Samples │ │ │
│ │ │ Budget: 800 tokens Priority: MEDIUM Eviction: FIFO │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TIER 5: RESULTS (Dynamic Content) │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Query Results + Aggregations + Visualizations │ │ │
│ │ │ Budget: 1500 tokens Priority: MEDIUM Eviction: AGE │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TIER 6: BUFFER (Reserved) │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Reserved for Response Generation │ │ │
│ │ │ Budget: 1000 tokens Priority: LOW Eviction: N/A │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TIER 7: EVICTABLE (Compress/Remove First) │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Old Results + Large Visualizations + Cached Queries │ │ │
│ │ │ Budget: Variable Priority: LOW Eviction: AGGRESSIVE│ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
| Tier | Component | Budget (tokens) | % of Total | Eviction Strategy |
|---|---|---|---|---|
| 1 | System Prompt | 500 | 10% | Never |
| 2 | Conversation | 2000 | 40% | LRU (Last Recent Use) |
| 3 | Schema | 300 | 6% | LFU (Least Frequently Used) |
| 4 | Samples | 800 | 16% | FIFO (First In, First Out) |
| 5 | Results | 1500 | 30% | Age-based |
| 6 | Buffer | 1000 | 20% | Reserved |
| Total Reserved | ~6100 | ~122% | Dynamic adjustment |
Note: Percentages sum >100% because tiers are dynamically balanced based on current needs
Before: Full column details with types, samples, stats
{
"columns": [
{"name": "customer_id", "type": "int64", "nullable": false, "unique": 15000},
{"name": "revenue", "type": "float64", "nullable": true, "min": 0.99, "max": 9999.99}
]
}After: Abbreviated schema
{"schema": "customer_id:int!,revenue:float[0.99-9999.99]"}Compression ratio: 5-10x
Before: Full JSON array
[
{"category": "Electronics", "revenue": 150000, "orders": 523},
{"category": "Clothing", "revenue": 89000, "orders": 1205}
]After: Markdown table
| Category | Revenue | Orders |
|----------|---------|--------|
| Electronics | 150K | 523 |
| Clothing | 89K | 1.2K |
Compression ratio: 2-3x
Before: Full precision
{"value": 1234567.8912345}After: Human-readable
{"value": "1.23M"}Compression ratio: Variable
Strategy: Reduce sample rows when context pressure increases
- Normal: 20 rows
- Pressure: 10 rows
- Critical: 5 rows
Strategy: Include only referenced columns
# User asks about "revenue by category"
# Only include: category, revenue (not: id, timestamp, metadata, etc.)interface ContextState {
// Current token usage by tier
usage: {
tier1_system: number;
tier2_conversation: number;
tier3_schema: number;
tier4_samples: number;
tier5_results: number;
tier6_buffer: number;
};
// Active datasets in context
activeDatasets: Map<string, DatasetContext>;
// Compression level (0-3)
compressionLevel: 0 | 1 | 2 | 3;
// Last access times for eviction
accessLog: Array<{
tier: number;
component: string;
timestamp: Date;
tokenCount: number;
}>;
}
interface DatasetContext {
filePath: string;
schema: Schema;
sample: any[];
lastQuery: string;
lastResult: any;
tokenCount: number;
priority: 'high' | 'medium' | 'low';
lastAccessed: Date;
}def manage_context_pressure(context_state):
"""
Adjust compression and eviction based on current token usage
"""
total_used = sum(context_state.usage.values())
window_size = get_context_window_size()
usage_ratio = total_used / window_size
if usage_ratio < 0.5:
# Low pressure: minimal compression
context_state.compression_level = 0
elif usage_ratio < 0.7:
# Medium pressure: light compression
context_state.compression_level = 1
compress_schemas(aggressive=False)
elif usage_ratio < 0.85:
# High pressure: aggressive compression
context_state.compression_level = 2
compress_schemas(aggressive=True)
reduce_samples(target_rows=10)
compress_numbers(precision=2)
else:
# Critical pressure: maximum compression + eviction
context_state.compression_level = 3
reduce_samples(target_rows=5)
compress_numbers(precision=1)
evict_low_priority_items()
truncate_old_results(max_age=5)User: "Analyze this large dataset"
│
▼
Step 1: Load Schema Only
├─ Send: Column names, types, row count
├─ Tokens: ~200
└─ Context: "Dataset has 5M rows, 25 columns..."
│
▼
Step 2: Load Sample (if requested)
├─ Send: 20 representative rows
├─ Tokens: ~800
└─ Context: "Sample data: [...]"
│
▼
Step 3: Execute Query
├─ Process: Full aggregation in Code-Server
├─ Send: Only aggregated results
├─ Tokens: ~500
└─ Context: "Results: {aggregates}"
│
▼
Step 4: Drill-down (if needed)
├─ Send: Filtered subset
├─ Tokens: ~600
└─ Context: "Filtered results: [...]"
When new dataset is loaded and space is needed:
eviction_priority = [
# Lowest priority: Remove samples first
lambda ds: (ds.lastAccessed > 5min_ago, ds.samples),
# Medium priority: Remove old results
lambda ds: (ds.lastQuery != current_query, ds.lastResult),
# High priority: Remove schema (rarely)
lambda ds: (ds.priority == 'low', ds.schema),
]def should_evict_result(result):
# Evict if:
# 1. Result is older than 10 minutes
# 2. Result size > 1000 tokens and newer results exist
# 3. Result not referenced in recent queries
return (
result.age > 600 or
(result.token_count > 1000 and newer_results_exist()) or
result.reference_count == 0
)All MCP tools return context_tokens_used field:
{
"result": {...},
"context_tokens_used": 450,
"context_metadata": {
"compression_applied": "schema_abbreviated",
"rows_included": 15,
"columns_included": 5,
"eviction_recommendation": "none"
}
}// In LobeHub agent logic
async function handleAnalyticsRequest(userQuery: string) {
const currentContext = getCurrentContextSize();
const availableTokens = CONTEXT_WINDOW - currentContext;
if (availableTokens < 2000) {
// Compress existing context
await compressContext();
}
// Determine optimal query strategy
const strategy = selectStrategy(userQuery, availableTokens);
switch(strategy) {
case 'schema_only':
return await mcp.profile_dataset({compress: 'maximum'});
case 'sample_and_query':
const sample = await mcp.stream_sample({rows: 10});
const result = await mcp.execute_query({return_limit: 20});
return {sample, result};
case 'direct_query':
return await mcp.execute_query({return_limit: 50});
}
}- 90%+ Token Reduction: Large datasets no longer overwhelm context
- Faster Responses: Less context = faster LLM processing
- Cost Savings: Reduced token usage = lower API costs
- Better UX: Users can work with large data iteratively
- Predictable Behavior: Clear budgets prevent context overflow
- Complexity: Additional layer of state management
- Information Loss: Compression may lose nuance
- Extra Round-trips: Incremental loading requires multiple MCP calls
- State Synchronization: Must keep LobeHub and MCP context in sync
- Learning Curve: Users need to understand "ask for schema first" pattern
- Tool Awareness: Agent must know when to request more/less data
- Implement token counting for all responses
- Add
context_tokens_usedto all tool outputs - Static budget allocation
- Simple compression (markdown tables)
- Dynamic compression based on pressure
- Smart eviction policies
- Incremental loading workflows
- Priority-based content retention
- Predictive loading (pre-load likely-needed data)
- Semantic compression (LLM-based summarization)
- Cross-query result deduplication
- Context usage analytics
| Metric | Before | Target | Measurement |
|---|---|---|---|
| Avg tokens/dataset | 50,000+ | <2,000 | Per-tool telemetry |
| Context overflow errors | Frequent | Zero | Error tracking |
| Query latency | Degrades | Stable | Timing logs |
| User satisfaction | Low | High | Feedback |
| Cost per session | High | Reduced 80% | Token usage logs |
- ADR-001: Analytics Architecture
- ADR-003: Caching and Query Optimization
- MCP-SPECIFICATION: Detailed tool specifications
- LobeHub Issue #3279: Context length performance
- OpenAI Tokenizer Guidelines
- Anthropic Context Management Best Practices
- "Attention Is All You Need" (Transformer architecture)
| Date | Author | Decision | Rationale |
|---|---|---|---|
| 2026-03-17 | Sisyphus | Adopt tiered context management with adaptive compression | Enables large-scale analytics within LobeHub constraints |
Status Legend:
- Proposed: Under review
- Accepted: Approved for implementation
- Deprecated: Replaced by newer ADR
- Superseded: See referenced ADR