Active Desktop Context Engine (ADCE)

High-Performance Desktop Context Graph, Time-Series Persistence & MCP Provider for Local AI Agents and Voice Interfaces

---

1. Overview

The Active Desktop Context Engine (ADCE) is a privacy-first Windows background daemon and Model Context Protocol (MCP) server.

Instead of relying on resource-heavy screenshot OCR, periodic screen polling, or cloud telemetry, ADCE integrates directly with native Win32 WinEvent hooks and UI Automation (UIA) caching to maintain a live, structured model of the user's active desktop environment with minimal resource overhead.

Key Capabilities:

• 100% Local Data Sovereignty: Complete on-device execution. Active window titles, document contents, and application states never leave localhost—ensuring enterprise privacy and zero telemetry leakage.
• Deterministic Focus & Window Topology: Event-driven tracking of foreground application envelopes, process metadata, window hierarchies, and focused UI controls via decoupled asynchronous channels.
• Virtual Desktop & Workspace Awareness: Extraction of active Virtual Desktop GUIDs, friendly names, and desktop indices via native COM interop.
• Container-Aware Tab Discovery: Targeted tab enumeration across modern browsers (Waterfox, Firefox, Chrome, Edge) and code editors (VS Code, Antigravity) without triggering unpruned DOM crawling stalls.
• Automated Credential & Privacy Sanitization: Built-in security firewall (ContextPrivacySanitizer) that detects IsPassword controls to redact passwords ([REDACTED_PASSWORD]), masks sensitive file buffers (.env, .pem, .kdbx, id_rsa), and strips OAuth tokens/query strings from browser address bars before context envelopes leave the extraction plane.
• Token-Efficient MCP Streaming: Compact, high-density JSON context snapshots over local MCP endpoints (get_desktop_context, desktop://current), giving local LLMs actionable workflow awareness without wasting tokens on raw screen dumps.
• Historical State Persistence: Embedded time-series storage (SQLite WAL) tracking focus transitions and tab history for temporal agent reasoning.

---

2. Core Philosophy & Architectural Vision

ADCE was designed to bridge low-level Windows accessibility infrastructure with high-level agentic AI and voice interfaces:

1. Unopinionated Accessibility Primitives (Core Vision): Rather than imposing rigid application-specific workflows, ADCE aims to surface clean, structured desktop primitives (window identity, container tabs, active editor buffers, workspace IDs, focused control types). Downstream speech recognition engines (Caster, Dragonfly, Talon) and local AI agent loops can consume these primitives to drive window switching, contextual grammar activation, or dynamic tool execution.
2. Dual-Consumer Architecture: A single high-performance engine serves both real-time local accessibility and voice tools (via direct .NET / IPC bindings) and local AI coding assistants (via Model Context Protocol JSON-RPC endpoints).
3. Non-Invasive Execution: 100% out-of-process execution using official Windows accessibility (FlaUI.UIA3), Win32 hooks (SetWinEventHook), and Virtual Desktop COM interfaces, requiring zero DLL injection or kernel drivers.
4. Targeted Zonal Anchoring: By scoping extractions to target class names and container bounding boxes, ADCE avoids the classic pitfalls of blind recursive DOM walking across modern Chromium and Gecko applications.

🚀 Verified Downstream Use Case: ADCE has been empirically integrated into Caster via caster-user-directory-and-notes, powering dynamic sub-window voice grammar activation (e.g. automatically activating CLI/Git commands only when focused in VS Code / Antigravity IDE integrated terminals without audio stutter). See the full First Real-World Use Case Guide.

---

3. Live Telemetry & DevTools HUD Demos

System Tray & Non-Activating Live HUD	Antigravity IDE (Monaco Editor & Zones)

Waterfox Browser (Gecko Tabs & Links)

SQLite Time-Series Transition Timeline

============================================================ # | TIME (UTC) | PROCESS | SEMANTIC ZONE ------------------------------------------------------------ 323 | 13:34:19.604 | Antigravity ID | [ChatAssistant] 324 | 13:36:26.553 | Antigravity ID | [EditorCodeBuffer] 329 | 13:36:31.583 | waterfox | [DocumentContent] 336 | 13:36:38.140 | Antigravity ID | [GitCommitBox] ============================================================

---

4. Architecture: Production Layered Model

ADCE is structured as a high-performance, unidirectional 4-tier pipeline designed for minimal resource overhead and zero cross-apartment COM deadlocks:

┌────────────────────────────────────────────────────────────────────────────────────────┐
│                              ADCE PRODUCTION ARCHITECTURE                              │
├────────────────────────────────────────────────────────────────────────────────────────┤
│  1. OS EVENT HOOK PLANE (Win32 Low-Level Event Hooks)                                  │
│  ├── SetWinEventHook listeners (EVENT_SYSTEM_FOREGROUND, EVENT_OBJECT_FOCUS)           │
│  └── Dedicated STA Message Pump Thread with Barrier Synchronization                     │
├────────────────────────────────────────────────────────────────────────────────────────┤
│  2. CONCURRENCY & EXTRACTION PLANE (FlaUI.UIA3 + Win32 Gating)                         │
│  ├── Win32 Shallow Filter (< 0.5 ms): Fast HWND, process & WS/WS_EX bitmask pre-gating │
│  ├── Async Channel Debouncer: 50ms trailing-edge window with 250ms burst delay clamp   │
│  ├── Monotonic Epoch Guard: Supersedes stale in-flight extractions                     │
│  ├── Scoped CacheRequest Batching: Single-roundtrip FlaUI.UIA3 COM queries (< 15 ms)   │
│  ├── Ancestor Chain Climber: Resolves leaf controls to typed DesktopSemanticZones      │
│  └── Privacy Sanitizer: Redacts passwords, secret files, and address bar tokens        │
├────────────────────────────────────────────────────────────────────────────────────────┤
│  3. PERSISTENCE & STORAGE PLANE (Dual-Tier In-Memory + SQLite WAL)                     │
│  ├── L1 In-Memory Atomic Cache: Lock-free instant query reads (< 15 ns)                │
│  └── L2 Time-Series WAL Store: Channel-decoupled asynchronous SQLite persistence       │
├────────────────────────────────────────────────────────────────────────────────────────┤
│  4. HOST DAEMON & CONSUMER INTERFACES (Tray Daemon & Model Context Protocol)           │
│  ├── System Tray Background Daemon: PerMonitorV2 DPI aware with dynamic status icon    │
│  ├── Non-Activating Live DevTools HUD: Floating diagnostic overlay (WS_EX_NOACTIVATE)  │
│  └── Model Context Protocol (MCP) Server: Universal JSON-RPC 2.0 (Stdio & HTTP/SSE)    │
└────────────────────────────────────────────────────────────────────────────────────────┘

---

5. Documentation Hub & Public Research Ledger

ADCE functions as both a production codebase and an evolving research ledger tracking low-level COM experiments, UI Automation benchmarks, and systems architecture.

📚 Central Documentation Entrypoint: Explore the full documentation suite in docs/CONTEXT.md. 📑 Architecture Specifications: Explore the normative architecture specifications in docs/architecture/.

🏛️ Architecture & System Specifications

Document	Description
📑 Core Domain Model Specification	Immutable records, 19 semantic zones, 6 application archetypes, and JSON serialization.
⚡ Extraction Pipeline Specification	Win32 shallow gating (< 0.5 ms), UIPI privilege checks, FlaUI caching, and dynamic rules.
💾 Dual-Tier Storage Architecture	Sub-microsecond L1 atomic memory cache and SQLite WAL time-series state persistence.
🖥️ Daemon Host & Consumer Integration	WinEvent STA message pump, tray host, HUD overlay, and MCP endpoints for Caster voice grammars.
📚 Application Layout Hierarchies Catalog	Per-application empirical profiles, container structures, and UIA selectors (Waterfox, Antigravity).
⚔️ Hostile Architecture & Systems Review	Adversarial review evaluating COM apartment deadlocks, GC allocation churn, and UIPI boundaries.

📘 Educational Guides & Consumer Integrations

Guide	Description
📘 Educational Refresher & Architecture Guide	Conceptual walkthrough of UI Automation, Win32 systems programming, and FlaUI caching.
🚀 First Real-World Use Case: Caster Dynamic Terminal Grammars	Production integration: Dynamic voice grammar activation in VS Code / Antigravity IDE via SSE streaming.
👁️ ADCE Focus & Zone Detection Explained	Practical visual guide to Windows focus mechanics, parent-chain traversal, and semantic zone detection.

🔬 Engineering Postmortems & Retrospectives

Ledger	Focus Area	Key Architectural Finding
📚 Master Postmortems Index	Master Ledger	Comprehensive catalog of development milestones and system retrospectives.
🔬 Claim Verifier Deprecation Postmortem	Verification	Retrospective on the self-confirmation loop anti-pattern and return to standard xUnit testing.
🔬 Hardware Acceleration & UIA Immunity	Graphics / UIA	DirectComposition clipping, PrintWindow capture modes, and UIA semantic immunity.
🔬 STA Threading & HUD Postmortem	Threading	STA WinForms message pump isolation and non-activating floating HUD window styles.

🗄️ Historical & Non-Normative Archive

Superseded test run outputs, exploratory research into unused third-party libraries, and deprecated custom claim verification matrices are archived in docs/archive/.

🧭 Navigation for AI Agents: Restrict architectural reasoning to docs/CONTEXT.md and docs/architecture/. Do not index or cite files in docs/archive/.

---

6. Engineering Roadmap & Milestone Status

Following our 4-Gate Epistemic Protocol, ADCE engineering is structured across clear progressive phases:

Phase / Milestone	Description	Status	Deliverables & Artifacts
Phase 1: Physical Observation	Identify DOM traversal traps and latency bottlenecks across real-world apps.	[x] Complete	• Doc 010: DOM Traversal Telemetry • Exposed 6,800-node DOM COM stall.
Phase 2: Adversarial Evaluation & Spikes	Gate 2 & Gate 3 empirical tests validating container targeting and Win32 gating.	[x] Complete	• Micro-Spike 1 Telemetry (FlaUI UIA3) • Micro-Spike 2 Telemetry (Win32 Shallow)
Phase 3: Ecosystem Audit & Synthesis	Deep-dive audits of leading open-source Windows/COM/UIA tooling catalogs.	[x] Complete	• Simon Mourier Ecosystem Suite • Roman Baeriswyl (Roemer / FlaUI) Deep Dive • Synthesis & Wheel Reinvention Audit
Phase 4: Architectural Specs & SSOT	Formalize ground-truth target zones, heuristic discovery archetypes, and MCP schemas.	[x] Complete	• UI Automation SSOT Reference • Dynamic Discovery & Requirements Spec • MCP JSON Schema Specification
Phase 5: Production Daemon Suite	Build modular multi-project solution (ADCE.slnx), event pipeline, storage, and MCP server.	[x] Complete	• Milestone 1: ADCE.Core domain models, events & serialization ([x] Complete) • Milestone 2: ADCE.Extraction standalone context grabber ([x] Complete) • Milestone 3: Low-overhead event pipeline (SetWinEventHook + channel debouncer) ([x] Complete) • Milestone 4: SQLite WAL store & in-memory live cache ([x] Complete) • Milestone 4.5: Ground-Truth Stimulus Test Harness ([x] Complete) • Milestone 5: High-Performance MCP Server (Stdio & SSE/HTTP) ([x] Complete) • Milestone 6: Windows System Tray Daemon & Live DevTools HUD ([x] Complete)
Phase 6: Application Layout Hierarchy Profiling & Viewport Boundary Hardening (Milestone 7)	Build verified empirical ground-truth profiles across desktop application archetypes, tracing leaf-to-root ancestor chains, isolating window chrome from client document viewports, and generating annotated visual proof.	[ ] Active	• WP 7.1: Known Limitations & Technical Gaps specification ([x] Complete) • WP 7.2: Application Hierarchy Catalog (docs/app_hierarchies/) & Profile Specification ([x] Complete) • WP 7.3: Waterfox (Gecko) Empirical Profile & Viewport Boundary Lock (01_waterfox.md) ([x] Complete) • WP 7.4: Antigravity IDE / VS Code (Monaco/Electron) Profile (02_antigravity_ide.md) ([x] Complete) • WP 7.5: Windows Terminal (Cascadia) & Shell Profiles ([ ] Scheduled)
Phase 7: Declarative App Definitions & Telemetry Self-Healing (Milestone 8)	Externalize layout rules into app_definitions.json and implement unmapped control logging to support dynamic voice and agent self-labeling overrides without modifying C# core code.	[ ] Scheduled	• WP 8.1: Declarative App Definitions Engine (app_definitions.json hot-reloader) • WP 8.2: Unmapped [Unknown] Subtree Telemetry Logger • WP 8.3: Dynamic Voice Labeling & Caster Semantic Alias Hooks
Phase 8: Advanced Context Primitives (Milestone 9)	Extract text selections, caret offsets, and on-demand full document buffers.	[ ] Scheduled	• WP 9.1: Caret position & active text selection extraction (TextPattern) • WP 9.2: Opt-in full document & editor buffer extraction (get_document_text) • WP 9.3: Multi-tier configurable privacy depth levels
Phase 9: External Voice & Agent Bindings (Milestone 10)	Connect Caster Dragonfly grammars and local AI assistants to live MCP streaming endpoints.	[ ] Deferred	• WP 10.1: Caster / Dragonfly dynamic voice grammar bindings • WP 10.2: Local AI coding assistant dynamic prompt injection loops

---

7. Known Limitations & Technical Gaps

Transparency regarding physical OS boundaries and framework constraints is a foundational tenet of ADCE:

1. Custom Canvas & Non-Accessible Toolkits: Applications built using pure immediate-mode custom graphics engines (e.g. Flutter, raw WebGL/HTML5 canvas, legacy Java AWT, Blender, Figma) do not construct standard Windows UI Automation trees unless explicitly run with accessibility flags enabled by the vendor. For these applications, ADCE accurately captures the top-level window envelope, process identity, and bounding coordinates, but falls back to DesktopSemanticZone.Unknown for intra-canvas sub-widgets.
2. Chromium AXTree Asynchronous IPC Latency: Unlike native Win32/WinUI controls whose vtables live in local memory ($\approx 1\text{–}5\text{ ms}$ response), Chromium/Electron applications marshal accessibility nodes on an asynchronous internal thread (AXTree). Deep queries across large Electron DOMs incur physical cross-process IPC delays ($\approx 40\text{–}75\text{ ms}$). ADCE compensates for this via 50ms trailing-edge debouncing and scoped container bounding box pruning.
3. Virtualized UI Element Trees: Modern controls utilizing virtualization (e.g. large file lists, virtualized tables, or long chat message streams) only instantiate UIA nodes for elements currently visible in the viewport. ADCE cannot inspect items that have been virtualized out of the active visual subtree without programmatic scrolling.
4. Single Global OS Keyboard Focus Pointer: Windows maintains a single global keyboard focus point at the OS level (GetFocus / GetGUIThreadInfo). When the user shifts focus between multiple non-foreground windows, background windows reflect their last-known captured state until brought to the foreground.
5. UIA3 vs. UIA2 Driver Boundary: ADCE runs exclusively on FlaUI.UIA3 over native Windows UIAutomationCore.dll (vtable COM). It does not maintain backwards compatibility shims for legacy UIA2 (UIAutomationClient.dll / MSAA wrappers), as UIA2 lacks batch CacheRequest support and suffers from high cross-apartment STA marshalling overhead.

---

8. Technology Stack

• Language & Framework: C# 14 / .NET 10 (LTS) (net10.0-windows)
• UI Automation Engine: FlaUI.UIA3 (v5.0.0+) over native UIAutomationCore.dll
• Concurrency: Native Win32 WinEvent hooks decoupled via System.Threading.Channels into MTA background workers
• Persistence: Embedded SQLite (WAL mode) with single-writer asynchronous queue & L1 cache
• Protocol: Model Context Protocol (MCP) JSON-RPC 2.0 (Stdio / SSE / HTTP Minimal API)
• DevTools HUD: WinForms Non-Activating overlay (WS_EX_NOACTIVATE | WS_EX_TOPMOST)

---

9. Building & Running

Running the System Tray Daemon & Live HUD

# Launch System Tray Daemon with live non-activating floating HUD overlay
dotnet run --project src/ADCE.Daemon -- --hud

# Launch System Tray Daemon with MCP server on HTTP/SSE port 8424
dotnet run --project src/ADCE.Daemon

# Launch as headless MCP server over Stdio (for IDE/Agent integration)
dotnet run --project src/ADCE.Daemon -- --stdio

Inspecting SQLite Time-Series History

# Visualize recent context transitions and application time distribution
dotnet run --project src/ADCE.Spikes -- --timeline 20

Running Test Suite & Spikes

# Run full automated unit test suite across all 5 projects (263 tests)
dotnet test --configuration Release

# Run Milestone 6 Daemon & End-to-End integration verification spike
dotnet run --project src/ADCE.Spikes -- --spike6

# Run Milestone 4 SQLite WAL store & L1 in-memory live cache verification spike
dotnet run --project src/ADCE.Spikes -- --storage

# Run Milestone 3 live event pipeline spike (listening for foreground & focus transitions)
dotnet run --project src/ADCE.Spikes -- --events -d 15

# Run Milestone 2 live standalone context grabber against active foreground window
dotnet run --project src/ADCE.Spikes -- --grab

10. Acknowledgments & Research Lineage

ADCE's technical architecture is informed by foundational research across the Windows systems and accessibility ecosystems:

• FlaUI by Roman Baeriswyl (Roemer) (MIT License): Powers ADCE's high-throughput UIA3 COM vtable automation and CacheRequest batch extraction.
• Microsoft.Data.Sqlite (MIT License) & SQLitePCLRaw by Eric Sink (Apache-2.0): Powers ADCE's embedded time-series state persistence.
• Simon Mourier (smourier): Open-source systems tools (HwndExplorer, UInspect) audited during exploratory research for low-overhead Win32 filtering techniques.
• Caster Upstream Lineage: Research documents 001–018 detailing the initial DOM traversal benchmarks and epistemic protocols that originated this engine.

---

11. License

Licensed under the Apache License, Version 2.0. See LICENSE for details.