asset-aware-mcp

Citation-ready document infrastructure for AI agents: turn PDFs, DOCX files, tables, figures, and evidence spans into reusable assets and Foam/LightRAG wikis.

🌐 繁體中文 · Docs Site · GitHub Wiki

v1.0.1 reliability refresh

• Large PDF text/table/figure results use a private, atomic, size-bounded MessagePack handoff instead of a multiprocessing pipe or executable pickle. This keeps multi-megabyte raster assets moving without pipe backpressure and fails closed on partial, oversized, malformed, or crashed worker output. Worker timeout environment values must be finite; NaN/infinities fall back to safe defaults. Finite values <=0 retain the historical explicit direct mode for compatibility and should not be used by managed production launchers.
• Codex-managed MCP configuration is validated as real TOML, preserves custom and unrelated tables, uses 180/900-second startup/tool timeouts, and never writes credential values. Its isolated working directory plus ASSET_AWARE_DISABLE_DOTENV=true prevents a managed server from silently reloading an unrelated workspace .env.
• Global Codex/Cline/Copilot config writes are workspace-trust gated and always use the exact published extension version plus isolated global storage. A lookalike repository cannot persist its local Python or .env values into a global agent launcher.
• MCP SDK 2 operational logs stay on stderr, empty/blank ingest requests are rejected before job persistence, and a true-stdio regression now verifies a large figure, a table, citation-ready evidence, complete bundle hashes, Foam notes, deterministic re-export, and an unchanged source PDF.
• GitHub Pages now provides a bilingual responsive Evidence Rail workflow, exact 30-tool explorer, install/development guidance, generated docs reader, and direct GitHub/Release/Issue links instead of stale raster architecture screenshots.

🎯 Why Asset-Aware MCP?

A server-local image path is not a portable multimodal payload. Whether an agent can dereference that path depends on its client, sandbox, and filesystem permissions.

Method	Can AI analyze image content?	Description
⚠️ Provide only a PNG path	Client-dependent	The client may be remote or sandboxed and cannot safely assume the server path exists locally
✅ Asset-Aware MCP	Yes, for compatible multimodal clients	Fetches bounded image bytes through MCP so the client can pass real image content to its vision model

Real-world Effect

# After retrieving the image via MCP, the AI can analyze it directly:

User: What is this figure about?

AI: This is the architecture diagram for Scaled Dot-Product Attention:
    1. Inputs: Q (Query), K (Key), V (Value)
    2. MatMul of Q and K
    3. Scale (1/√dₖ)
    4. Optional Mask (for decoder)
    5. SoftMax normalization
    6. Final MatMul with V to get the output

This is the value of Asset-Aware MCP - enabling AI Agents to truly "see" and understand charts and tables in your PDF literature.

---

✨ Features

• 📄 Asset-Aware ETL - PDF → Markdown with a pluggable multi-engine parser (ETL_ENGINE):
  • PyMuPDF (default) - Fast extraction (~50MB), no models required
  • PyMuPDF4LLM ([pdf-plus]) - Drop-in layout-aware upgrade, no GPU
  • Docling ([docling]) - MIT-licensed layout+table+formula+chart engine; bridges through an isolated .venv-docling interpreter when the main environment can't install it directly (see docs/docling-setup.md)
  • MinerU - Adapter retained, but the packaged extra is on security hold while MinerU pins a vulnerable transformers<5 chain
  • Marker - Adapter retained for evaluation, but production selection fails closed while upstream marker-pdf conflicts with the patched Pillow floor. The legacy use_marker parameter now means “prefer the configured structured extractor”; it does not bypass this hold.
• 🧩 Unified Segmentation Export - Normalized segmentation.json merges manifest, blocks, reading order, and persisted markdown line spans for downstream tools and extensions.
• 🩺 Safe PDF Preflight Router - document(op="preflight") classifies each page as native, sparse, image, scanned, or hybrid; returns 1-based top-left locators, source SHA-256, OCR reasons, and a bounded extraction-engine recommendation from a process-isolated inspector.
• 📦 Reusable Agent Asset Bundles - document(op="export_assets") writes deterministic manifest.json, assets.jsonl, copied media, and a portable Foam index.md/notes/** subtree while preserving stable IDs, hashes, locators, and citation refs.
• 🛡️ PDF Safety/Structure/Coverage/Accessibility Audits - OpenDataloader-inspired artifact-only reports flag suspicious hidden/off-page/prompt-injection text, native structure signals, segmentation coverage gaps, and accessibility/readability readiness via the existing document facade. document(op="prepare_ai") and document(op="auto") expose agent-ready status and next actions without adding public tools.
• 🧭 Structural Pointer Retrieval - Proxy-Pointer-inspired document(op="pointer_index"), document(op="structural_retrieve"), and document(op="compare") preserve section breadcrumbs, line/char/byte locators, source hashes, asset IDs, and evidence-span provenance without adding MCP tools.
• 🖼️ Layout Overlay Debugging - Render page overlays from original.pdf to inspect bbox, segment type, and reading order visually.
• 🔤 On-Demand OCR Preprocessing - Optional ocrmypdf preprocessing path for scanned PDFs before ETL.
• 🧭 Section Navigation - Dynamic hierarchy section tree through the section facade: browse, search, detail, content reading, and block extraction for any depth of headings.
• 🔄 Async Job Pipeline - Supports asynchronous ingest, configured structured parse, OCR, and conversion jobs with progress tracking.
• 🔀 Mixed-Format Batch Ingestion - document(op="auto", file_paths=[...]) auto-detects a batch mixing PDF with DOCX/DOC/ODT/ODS, ingests each file through its correct existing engine in one background job, isolates per-file failures so one bad file cannot abort the rest, and reports per-file progress — no new public tool required.
• 🗺️ Document Manifest - Provides a structured "map" of the document for precise data access by Agents.
• 🧠 LightRAG Integration - Knowledge Graph + Vector Index, supporting cross-document comparison and reasoning.
• 🧾 Verified Citation Bundles - citation_bundle, Foam evidence packs, citation health checks, table/figure evidence notes, and claim promotion export citation-ready spans with locator, quote/hash, context, CRAAP scaffold, and verification status.
• 📝 Docx Editing (DFM) - Edit .docx files in Markdown via Docx-Flavored Markdown format. Supports legacy .doc, .odt, and .ods ingest via LibreOffice auto-conversion. The balanced surface keeps 6 DOCX/DFM public entrypoints for ingest, read, save, validation, conversion, table edit planning, and Docx ↔ A2T bridges.
• 🛡️ DFM Integrity Checker - Automatic validation and auto-repair at every pipeline stage (post-ingest, pre-save, post-save). Catches orphan markers, column mismatches, and format inconsistencies.
• 📊 A2T (Anything to Table) - 7 operation-based tools for building professional tables from any source (PDF assets, Knowledge Graph, URLs, user input). Features: stable row IDs, row search/filter/paging, citation coverage, artifact-only large-table render, skipped-large-table UX, Citations (AssetRef), Audit Trail, Schema Evolution, Templates, Drafting, and Token-efficient resumption.
• 🖥️ VS Code Management Extension - Graphical interface for monitoring server status, ingested documents, document artifacts, citation spans, and A2T tables/drafts with one-click Excel export.
• 🔌 MCP SDK 2 Server - Uses the official Python SDK MCPServer API, runtime-injected context, and v2 clients. MCP SDK v1 is intentionally unsupported.
• 🔬 Research-ready, domain-neutral assets - Works with scholarly, technical, policy, and operational documents; bounded image bytes let compatible multimodal clients analyze figures instead of relying on server-local paths.

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                    AI Agent (Copilot)                   │
└─────────────────────┬───────────────────────────────────┘
                      │ MCP Protocol (Tools & Resources)
┌─────────────────────▼───────────────────────────────────┐
│            MCP Server (Modular Presentation)            │
│  ┌─────────────────────────────────────────────────┐   │
│  │ tools/: 30 public tools (balanced surface)                   │   │
│  │   17 facade tools + 13 high-frequency shortcuts       │   │
│  │   compact=17 │ legacy/direct compatibility=63 │
│  └─────────────────────────────────────────────────┘   │
│  ┌─────────────────────────────────────────────────┐   │
│  │ resources/: 13 resources in 2 modules           │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│                  ETL Pipeline (DDD)                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ PyMuPDF  │  │  Asset   │  │ LightRAG │              │
│  │ Adapter  │→ │  Parser  │→ │  Index   │              │
│  └──────────┘  └──────────┘  └──────────┘              │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│                   Local Storage                         │
│  ./data/                                                │
│  ├── {doc_id}/        # PDF document artifacts          │
│  ├── docx_{id}/       # Docx IR + DFM + Assets          │
│  ├── tables/          # A2T Tables (JSON/MD/XLSX)       │
│  │   └── drafts/      # Table Drafts (Persistence)      │
│  └── lightrag_db/     # Knowledge Graph                 │
└─────────────────────────────────────────────────────────┘

📁 Project Structure (DDD)

asset-aware-mcp/
├── src/
│   ├── domain/              # 🔵 Domain: Entities, Value Objects, Interfaces
│   ├── application/         # 🟢 Application: Doc Service, Table Service (A2T), Asset Service
│   ├── infrastructure/      # 🟠 Infrastructure: PyMuPDF, LightRAG, Excel Renderer
│   └── presentation/        # 🔴 Presentation: MCP SDK 2 MCPServer
├── data/                    # Document and Asset Storage
├── docs/
│   └── spec.md              # Technical Specification
├── tests/                   # Unit and Integration Tests
├── vscode-extension/        # VS Code Management Extension
└── pyproject.toml           # uv Project Config

📐 Architecture and workflows

The maintained, versioned references are the documentation site, architecture guide, PDF workflow, MCP tool catalog, and release checklist. They are generated and checked with the implementation so tool counts, engine holds, and release behavior do not drift inside obsolete screenshots.

🚀 Quick Start

# Install dependencies (using uv) — default install stays on the fast PyMuPDF backend
uv sync

# Optional high-fidelity PDF->asset engines:
# uv sync --extra pdf-plus   # PyMuPDF4LLM: drop-in layout-aware upgrade
# uv sync --extra docling    # Docling: MIT layout+table+formula+chart engine
# MinerU and Marker packaged extras are temporarily empty security holds.
# Then set ETL_ENGINE=pymupdf4llm|docling.

# Run MCP Server
uv run python -m src.presentation.server

# Or use the VS Code extension for graphical management

Runtime note: The VS Code extension prefers a managed Python 3.11 runtime when launching the MCP server via version-pinned uv tool run, with Python 3.10 fallback for older machines. This avoids native package builds on end-user machines, especially macOS systems without Xcode Command Line Tools, while keeping the project itself compatible with newer Python versions.

Installation scope note:

• The VS Code extension installs once per user. In a trusted workspace, the native VS Code MCP provider may use workspace-scoped DATA_DIR, cache, settings, and .env; local source is accepted only in Extension Development/Test mode (or a future explicit opt-in).
• Global Codex and Cline entries always launch asset-aware-mcp==<extension-version> from extension global storage. They do not inherit workspace-local source, workspace-scoped settings, or repository .env values. Restricted Mode skips external config writes and assistant-asset sync entirely.

Engine selection note: ETL_ENGINE picks the extraction backend (default pymupdf). The active packaged structured engines (pymupdf4llm, docling) lazy-load and gracefully fall back to PyMuPDF when their extra is not installed. Marker remains on hold because marker-pdf requires Pillow<11; MinerU is also on hold because MinerU 3.4.4 pins transformers<5 while current security fixes require transformers>=5.5. Both adapters remain in-tree, but this package will not install a known-vulnerable dependency chain. Use document(op="preflight", pdf_path="...") to choose between fast native extraction, OCR, and Docling before ingest.

Agent asset / Foam handoff:

document(op="preflight", pdf_path="/papers/source.pdf")
document(op="auto", file_paths=["/papers/source.pdf"])
document(op="export_assets", doc_id="doc_...", output_dir="agent-assets")

The exported directory is deterministic and portable: manifest.json is the bundle contract, assets.jsonl is the agent-readable inventory, and index.md plus notes/** can be mounted or copied into a Foam workspace.

🔌 MCP Tools

The default runtime surface is balanced: 30 public tools that keep the full document workflow available without overwhelming agents. It is made of 17 operation-based facade tools plus 13 high-frequency shortcuts. Set ASSET_AWARE_MCP_TOOL_SURFACE=compact for the 17 facade-only surface, or ASSET_AWARE_MCP_TOOL_SURFACE=legacy / ASSET_AWARE_MCP_ENABLE_LEGACY_TOOLS=true for the full 63-tool compatibility inventory.

Area	Balanced public tools
Documents, assets, evidence, conversion	document, document_asset, evidence, convert_document, ingest_documents, list_documents, parse_pdf_structure, fetch_document_asset, find_evidence_spans, verify_citation_ref, citation_bundle
DOCX / DFM	docx, docx_table, ingest_docx, get_docx_content, save_docx, docx_table_edit_plan
Sections, jobs, KG, ETL profiles	section, job, get_job_status, list_jobs, knowledge, etl_profile
A2T tables	plan_table, table_manage, table_data, table_cite, table_history, table_draft, discover_sources

See MCP Tools and Tool Consolidation for operation details, shortcut rationale, and legacy direct-tool mapping.

Agent handoff note: Use document(op="auto", file_paths=[...]) for new PDFs and document(op="auto", doc_id="...") or document(op="prepare_ai", doc_id="...") for existing documents. document(op="prepare_ai", output_format="json") returns the v2 readiness contract with status, blockers, warnings, capabilities, artifacts, missing_audits, invalid_audits, audit_artifacts, and next_actions. document(op="audit", doc_id="...") reuses current audit artifacts only when they are present and valid; pass refresh=true to rebuild safety, native-structure, coverage, and accessibility reports. Use document(op="pointer_index"), document(op="structural_retrieve", query="..."), and document(op="compare", doc_b_id="...", criteria="...") when an agent needs section-level structural retrieval or comparison without new public tools. Readiness and job-status artifact discovery are read-only, so status checks do not create document directories.

PDF audit caveat: The audit reports are inspired by OpenDataloader-style artifact workflows, but they are not a sanitizer, a PDF/UA certification, or an OpenDataloader compatibility layer. They preserve source artifacts and report conservative diagnostics for review.

🔧 Tech Stack

Category	Technology
Language	Python 3.10+
Package Manager	uv (all pip/setup-python removed)
ETL	PyMuPDF (default) + secure optional PyMuPDF4LLM / Docling engines; MinerU and Marker adapters are on dependency security hold
RAG	LightRAG (lightrag-hku)
MCP	Official Python MCP SDK 2 (MCPServer); SDK v1 unsupported
Storage	Local filesystem (JSON/Markdown/PNG)

📋 Documentation

Installation guidance:

• Default install: uv sync (slim ~227 MB; no LightRAG/KG dependencies).

• LightRAG / Knowledge Graph backend (optional, since v0.6.34): uv tool install --upgrade --python 3.11 'asset-aware-mcp[lightrag]' for uvx/published users, or uv sync --extra lightrag for local source checkouts. Required before setting ENABLE_LIGHTRAG=true.

• VS Code extension: run the command Asset-Aware MCP: Install LightRAG Backend from the Command Palette; it auto-detects source vs published mode and emits the matching install command.

• OpenRouter optional preset (since v0.6.35): set LLM_BACKEND=openrouter, OPENROUTER_API_KEY=..., and optionally OPENROUTER_MODEL=liquid/lfm-2.5-1.2b-instruct:free for fast low-cost summaries and draft RAG answers. LightRAG retrieval still uses the configured embedding backend.

• High-fidelity PDF engines: uv sync --extra pdf-plus (PyMuPDF4LLM) or uv sync --extra docling (Docling), then set ETL_ENGINE accordingly. Docling ships a cross-platform isolated installer; see docs/docling-setup.md.

• MinerU and Marker backends: their adapters remain available for upstream testing, but the packaged extras are empty security holds until their dependency caps permit patched transformers and Pillow releases.

• VS Code extension: assetAwareMcp.enableMarkerBackend is retained as a setting, but the launcher will not install marker-pdf while the security hold is active.

• Technical Spec - Detailed technical specification

• Architecture - System architecture

• Constitution - Project principles

• Competitive Analysis - MCP + DOCX ecosystem landscape

📄 License

Apache License 2.0