PHASE 3 v2: Ingestion Pipeline Overhaul

Mission

The agent's knowledge ingestion pipeline — from URL receipt to stored, queryable document — has critical failures that make scientific data unreliable. This phase replaces the fragmented fetch→parse→store chain with a single, observable, tested pipeline.

Success criteria: The Springer URL https://link.springer.com/article/10.1186/s12961-017-0235-3 ingests clean structured article text (paragraphs, headings, sections preserved), classifiable by ScientificSectionDetector, with verbatim quote retrieval returning real sentences — not minified JavaScript.

ROOT CAUSE ANALYSIS

Problem 1: Agent May Not Load Latest Plugin Code

autognostic-agent/package.json declares "@elizaos/plugin-autognostic": "file:../plugin-autognostic". The agent's build.ts marks the plugin as external, so Bun's bundler does NOT inline plugin code. At runtime, imports resolve from the symlinked node_modules/@elizaos/plugin-autognostic/dist/.

The gate: tsc -p tsconfig.json in the plugin directory writes to dist/. If you edit source files but don't run bun run build in the plugin directory, the agent loads STALE .js files from dist/. All our domToText() and PDF fallback fixes may never have been compiled.

Problem 2: HTML→Text Extraction Routing is Fragile

mirrorDocToKnowledge.ts decides how to fetch based on file extension:

const textExtensions = [".md", ".txt", ".json", ...];
const isLikelyText = textExtensions.some(ext => rawUrl.toLowerCase().endsWith(ext));

A Springer URL has no file extension. It falls into the else branch, which does a raw http.get(), gets HTML back, then detects HTML after the fact with a second copy of the same processing logic.

Problem 3: PDF Fallback Chain is Fragile

findBestPdfLink() correctly finds Springer's citation_pdf_url meta tag. But Springer's PDF URL returns a paywall page (HTTP 200, text/html) — not a PDF. The content-type guard catches this. But the fallback to HTML-extracted text only works if the domToText() rewrite was compiled.

Problem 4: The Pipeline is Duplicated

mirrorDocToKnowledge.ts contains the SAME PDF-fallback logic in TWO branches (~40 lines each). The isLikelyText branch and the else branch both instantiate WebPageProcessor, call extractFromHtml(), try PDF download with content-type guard, and fall back. Every bug fix must be applied in two places.

Problem 5: Callers Have Different Metadata Contracts

mirrorDocToKnowledge has two callers with different metadata conventions:

addUrlToKnowledgeAction passes metadata: { sourceId, versionId } — triggers verbatim document storage
ReconciliationService passes metadata: { autognosticSourceId, autognosticVersionId } — different keys, so verbatim storage is SKIPPED

The simplified mirrorDocToKnowledge must preserve this behavior exactly.

THE PLAN

WS-1: Build Verification Canary (30 min)

Problem it solves: No way to verify the agent loaded fresh plugin code.

Approach: Generate a buildmeta.ts file during build — avoids mutating source files.

File: src/config/buildmeta.ts (GENERATED, gitignored):

// Auto-generated by build script — do not edit
export const BUILD_META = {
  phase: "3",
  builtAt: "2026-02-17T22:15:00.000Z",
  version: "0.3.0",
};

File: src/config/buildmeta.template.ts (checked into git):

// Fallback when buildmeta.ts hasn't been generated yet
export const BUILD_META = {
  phase: "3",
  builtAt: "NOT_BUILT",
  version: "0.3.0",
};

Build integration: Add a prebuild step to package.json:

"scripts": {
  "prebuild": "node -e \"require('fs').writeFileSync('src/config/buildmeta.ts', '// Auto-generated by build — do not edit\\nexport const BUILD_META = { phase: \\\"3\\\", builtAt: \\\"' + new Date().toISOString() + '\\\", version: \\\"0.3.0\\\" };\\n')\"",
  "build": "tsc -p tsconfig.json"
}

File: .gitignore — add: src/config/buildmeta.ts

File: src/index.ts — add to initPlugin():

import { BUILD_META } from "./config/buildmeta";
// ... inside initPlugin():
console.log(`[autognostic] Plugin loaded — Phase ${BUILD_META.phase}, built ${BUILD_META.builtAt}`);

Fallback: If buildmeta.ts doesn't exist (first clone), tsc will fail. The template file provides a safe fallback. Use a conditional import or a try/catch dynamic import. SIMPLER: just make the prebuild step always run, so the file always exists.

Verification: After /rebuilt, agent terminal shows:

[autognostic] Plugin loaded — Phase 3, built 2026-02-17T22:15:00.000Z

If you see built NOT_BUILT, the prebuild step didn't run.

Tests: Add one test that imports BUILD_META and checks it has the expected shape.

WS-2: ContentResolver + PDF Magic Bytes (2-3 hours)

Problem it solves: Duplicated fetch→parse→PDF-fallback chain. (Folds former WS-4 into this.)

File: src/services/ContentResolver.ts

import { HttpService } from "./httpService";
import { WebPageProcessor, type ExtractedPage } from "./WebPageProcessor";
import { PdfExtractor } from "./PdfExtractor";
import { getScientificPaperDetector } from "./ScientificPaperDetector";
import { logger } from "../utils/logger";

export interface ResolvedContent {
  text: string;                       // Clean structured text
  contentType: string;                // Final MIME type
  source: "pdf" | "html" | "raw";    // How text was obtained
  title: string;                      // Page/document title
  resolvedUrl: string;                // URL actually fetched (after normalization)
  metadata: {
    doi?: string;
    authors?: string[];
    citationPdfUrl?: string;
    description?: string;
  };
  diagnostics: string[];              // Human-readable decision log
}

Key design decisions:

Route on RESPONSE content-type, never URL extension
PDF verification is dual-gate: content-type header application/pdf AND body starts with %PDF- magic bytes
resolvedUrl exposed so mirrorDocToKnowledge can do dual-storage (original URL + normalized URL)

Accept header strategy lives here, not in HttpService — keeps HttpService generic:

private getAcceptHeader(url: string): string {
  const detector = getScientificPaperDetector();
  if (detector.isLikelyScientificPaper(url)) {
    // Ask for PDF first — some publishers honor content negotiation
    return "application/pdf, text/html;q=0.9, */*;q=0.1";
  }
  return "text/html, application/xhtml+xml, */*;q=0.1";
}

Note: getScientificPaperDetector() returns a singleton with zero DB dependencies — this is safe for a "pure" service.

Diagnostics array records every routing decision for debugging
Max text length guard: If extracted text exceeds 500K chars, truncate and add diagnostic warning

The resolve() flow:

URL
 ↓
normalizeToRawUrl() → resolvedUrl
 ↓
HTTP GET with smart Accept headers (PDF-first for academic publishers)
 ↓
Check actual response:
 ├─ Response is PDF (content-type + magic bytes) → PdfExtractor → done
 ├─ Response is HTML (content-type or body detection) → WebPageProcessor:
 │   ├─ extractFromHtml() → structured text (this is the SAFETY NET)
 │   ├─ findBestPdfLink() → if found, try PDF download:
 │   │   ├─ HTTP GET pdf URL
 │   │   ├─ Gate 1: content-type includes "application/pdf"
 │   │   ├─ Gate 2: body starts with %PDF- (magic bytes)
 │   │   ├─ Both pass → PdfExtractor → use PDF text, source="pdf"
 │   │   └─ Either fails → use HTML extracted text, source="html"
 │   └─ No PDF link → use HTML extracted text, source="html"
 ├─ Response is text/plain or text/markdown → return as-is, source="raw"
 └─ Other → attempt text decode, add diagnostic warning, source="raw"

PDF magic byte check (built into ContentResolver):

private isPdfBytes(data: Uint8Array): boolean {
  return data.length >= 5 &&
    data[0] === 0x25 && data[1] === 0x50 &&
    data[2] === 0x44 && data[3] === 0x46 && data[4] === 0x2D;
}

normalizeToRawUrl(): Move from mirrorDocToKnowledge into ContentResolver as a private method. It stays unchanged (GitHub/GitLab/Gist transforms).

Tests for WS-2 (in tests/ContentResolver.test.ts):

Mock HttpService to return canned responses. Create MINIMAL synthetic HTML fixtures (not real Springer pages):

html-with-paywall-pdf — HTML with citation_pdf_url meta, mock PDF URL returns text/html → should use HTML extraction
html-with-real-pdf — HTML with PDF link, mock PDF URL returns application/pdf + valid %PDF- header → should use PDF extraction
direct-pdf — URL returns application/pdf directly → PdfExtractor
github-blob — GitHub blob URL → normalizes to raw.githubusercontent.com
html-no-article-tag — HTML without <article> → falls back to body with junk removal
fake-pdf-magic-bytes — Server claims application/pdf but body is HTML → rejects, falls back
plain-text — URL returns text/plain → returns as-is
academic-publisher-url — Known publisher URL → Accept header prefers PDF

WS-3: Simplify mirrorDocToKnowledge (1 hour)

Problem it solves: Delete ~120 lines of duplicated fetch/parse logic.

Critical contract to preserve: The function's CALLERS expect specific behavior:

Caller	metadata keys	Verbatim doc stored?	Notes
addUrlToKnowledgeAction	`{ sourceId, versionId }`	YES	Also stores profile
ReconciliationService	`{ autognosticSourceId, autognosticVersionId }`	NO (keys don't match)	Intentional — bulk reconciliation doesn't need verbatim store

The rewrite:

export async function mirrorDocToKnowledge(runtime, params) {
  const http = runtime.getService("http") ?? new HttpService(runtime);
  const resolver = new ContentResolver(http);
  const knowledge = runtime.getService("knowledge");

  // === CONTENT RESOLUTION (replaces all fetch/parse/PDF logic) ===
  const resolved = await resolver.resolve(params.url);

  // Log diagnostics at debug level (visible with LOG_LEVEL=debug)
  const log = logger.child({ operation: "mirrorDocToKnowledge", url: params.url });
  for (const d of resolved.diagnostics) {
    log.debug(d);
  }
  log.info("Content resolved", {
    source: resolved.source,
    textLength: resolved.text.length,
    resolvedUrl: resolved.resolvedUrl,
  });

  // === VERBATIM DOCUMENT STORAGE ===
  // Only store if caller provided sourceId + versionId
  // (addUrlToKnowledgeAction does; ReconciliationService does NOT)
  const sourceId = params.metadata?.sourceId as string | undefined;
  const versionId = params.metadata?.versionId as string | undefined;
  if (sourceId && versionId) {
    const contentHash = createHash("sha256").update(resolved.text).digest("hex");

    // Store under original URL (what user mentions in conversation)
    await autognosticDocumentsRepository.store(runtime, {
      sourceId,
      versionId,
      url: params.url,
      content: resolved.text,
      contentHash,
      mimeType: resolved.contentType,
      byteSize: Buffer.byteLength(resolved.text, "utf8"),
    });

    // Compute and store structural profile
    try {
      const profile = analyzeDocument(resolved.text);
      await autognosticDocumentsRepository.updateProfile(runtime, params.url, profile);
    } catch (err) {
      log.debug("Profile analysis failed (non-fatal)", {}, err);
    }

    // Dual storage: also store under resolved URL if different
    if (resolved.resolvedUrl !== params.url) {
      try {
        await autognosticDocumentsRepository.store(runtime, {
          sourceId: randomUUID(),
          versionId,
          url: resolved.resolvedUrl,
          content: resolved.text,
          contentHash,
          mimeType: resolved.contentType,
          byteSize: Buffer.byteLength(resolved.text, "utf8"),
        });
        const profile = analyzeDocument(resolved.text);
        await autognosticDocumentsRepository.updateProfile(runtime, resolved.resolvedUrl, profile);
      } catch { /* duplicate key or non-fatal */ }
    }
  }

  // === PUSH TO KNOWLEDGE SERVICE ===
  const clientDocumentId = randomUUID();
  const worldId = params.worldId ?? runtime.agentId;

  const result = await knowledge.addKnowledge({
    worldId,
    roomId: params.roomId,
    entityId: params.entityId,
    clientDocumentId,
    originalFilename: params.filename,
    contentType: resolved.contentType,
    content: resolved.text,
    metadata: {
      sourceUrl: params.url,
      rawUrl: resolved.resolvedUrl !== params.url ? resolved.resolvedUrl : undefined,
      autognostic: true,
      contentSource: resolved.source,  // "pdf", "html", or "raw"
      ...(params.metadata ?? {}),
    },
  });

  return {
    knowledgeDocumentId: result.storedDocumentMemoryId as string,
    clientDocumentId,
    worldId,
  };
}

What gets deleted:

normalizeToRawUrl() function (moved to ContentResolver)
textExtensions array and isLikelyText branching
Both copies of the WebPageProcessor/PDF fallback pipeline
~120 lines of duplicated code

What stays unchanged:

MirrorDocParams interface
Return type { knowledgeDocumentId, clientDocumentId, worldId }
Dual-storage pattern (original URL + resolved URL)
params.metadata passthrough to knowledge service

Compatibility check — addUrlToKnowledgeAction:

Calls mirrorDocToKnowledge() ✓ — return type unchanged
Then fetches stored content via docsRepo.getByUrl(url) ✓ — we still store under params.url
Runs paperHandler.process(url, content, id) ✓ — content is now clean text, which is what it expects

Compatibility check — ReconciliationService:

Calls mirrorDocToKnowledge() with metadata: { autognosticSourceId, autognosticVersionId } ✓
sourceId check is params.metadata?.sourceId → undefined → verbatim store skipped ✓ — same behavior as before
Uses res.knowledgeDocumentId ✓ — return type unchanged

WS-4: WebPageProcessor Hardening (1 hour)

Problem it solves: Publisher-specific HTML structures that the generic <article> selector misses.

Extended content root selectors:

const contentRoot =
  document.querySelector("article") ||
  document.querySelector("[class*='article-body']") ||
  document.querySelector("[class*='article-content']") ||
  document.querySelector("[class*='fulltext']") ||
  document.querySelector("[class*='paper-content']") ||
  document.querySelector("[role='main']") ||
  document.querySelector("main") ||
  document.body;

Reference section preservation: Currently the junk class filter might catch reference sections. Add a whitelist exception:

// In the junk removal loop, SKIP elements that look like reference lists
const isReferenceSection = /\breferences?\b|\bbibliography\b/i.test(cn) || /\breferences?\b|\bbibliography\b/i.test(elId);
if (isReferenceSection) continue; // Don't remove reference sections

Max text length guard in cleanExtractedText():

const MAX_EXTRACTED_CHARS = 500_000;
if (raw.length > MAX_EXTRACTED_CHARS) {
  console.warn(`[autognostic] Extracted text exceeds ${MAX_EXTRACTED_CHARS} chars (${raw.length}), truncating`);
  raw = raw.slice(0, MAX_EXTRACTED_CHARS);
}

Table extraction improvement: Current td/th uses tab separation, which DocumentAnalyzer handles. Keep as-is.

Tests: Add 2-3 tests to existing WebPageProcessor.test.ts:

HTML with class="c-article-body" instead of <article> tag
Reference section not stripped
Oversized text gets truncated

WS-5: Diagnostic Logging + LOG_LEVEL Note (30 min)

Problem it solves: Scattered console.debug/console.warn calls; diagnostics invisible at default log level.

Key insight from audit: The logger's default minLevel is "info". ContentResolver's diagnostic messages are debug-level. They're invisible unless LOG_LEVEL=debug is set. This is by design — production shouldn't be noisy — but the plan must document it.

Changes:

ContentResolver already uses diagnostics array (returned in ResolvedContent). These are always available regardless of log level.
mirrorDocToKnowledge logs diagnostics at debug level via logger.child()
Add to the agent's .env file (or document in README): LOG_LEVEL=debug for troubleshooting

Replace in mirrorDocToKnowledge: All console.debug() and console.warn() calls → use logger.child(). Already shown in WS-3 rewrite above.

WS-6: Integration Tests with Synthetic Fixtures (1-2 hours)

Problem it solves: No tests exercise the full pipeline; real HTML fixtures bloat the repo.

Approach: Create MINIMAL synthetic HTML strings inline in tests (not separate files). Each fixture is 20-50 lines of HTML that exercises a specific code path.

File: tests/ContentResolver.test.ts

import { describe, it, expect, vi } from "vitest";
import { ContentResolver } from "../src/services/ContentResolver";

// Mock HttpService
function createMockHttp(responses: Map<string, { status: number; headers: Record<string, string>; body: string | Uint8Array }>) {
  return {
    get: vi.fn(async (url: string, opts?: any) => {
      const resp = responses.get(url);
      if (!resp) throw new Error(`No mock for ${url}`);
      return {
        ok: resp.status >= 200 && resp.status < 300,
        status: resp.status,
        headers: new Headers(resp.headers),
        text: async () => typeof resp.body === "string" ? resp.body : new TextDecoder().decode(resp.body),
        arrayBuffer: async () => typeof resp.body === "string" ? new TextEncoder().encode(resp.body).buffer : resp.body.buffer,
      };
    }),
  } as any;
}

Test cases (8 total):

html-with-paywall-pdf → extracts clean article text
html-with-accessible-pdf → extracts from real PDF
direct-pdf-response → PdfExtractor path
github-blob-normalization → URL transform
html-body-fallback → no <article> tag
fake-pdf-rejected → magic byte check
plain-text-passthrough → text/plain
diagnostics-populated → verify diagnostics array

Note: Test 2 (real PDF) and 3 (direct PDF) need a real minimal PDF fixture. Create a tiny one-page PDF programmatically or use a ~1KB PDF bytes constant.

FILE CHANGE SUMMARY

File	Action	Description
`src/config/buildmeta.ts`	GENERATE	Auto-generated build timestamp (gitignored)
`src/config/buildmeta.template.ts`	CREATE	Fallback template (checked in)
`src/services/ContentResolver.ts`	CREATE	Unified URL→text pipeline with PDF magic bytes
`src/integration/mirrorDocToKnowledge.ts`	REWRITE	Simplify to use ContentResolver
`src/services/WebPageProcessor.ts`	EDIT	Publisher selectors, reference whitelist, length guard
`src/config/constants.ts`	NO CHANGE	Leave as-is
`src/services/httpService.ts`	NO CHANGE	Leave as-is (Accept headers owned by ContentResolver)
`src/services/PdfExtractor.ts`	NO CHANGE	Leave as-is (magic bytes check in ContentResolver)
`src/index.ts`	EDIT	Import BUILD_META, log on startup
`package.json`	EDIT	Add prebuild script for buildmeta generation
`.gitignore`	EDIT	Add `src/config/buildmeta.ts`
`tests/ContentResolver.test.ts`	CREATE	8 integration tests with mocked HTTP
`tests/WebPageProcessor.test.ts`	EDIT	Add 3 hardening tests

EXECUTION ORDER

WS-1 (canary)
 │  Create buildmeta.template.ts, prebuild script, index.ts import
 │  Run /replug — verify 272 tests pass
 │  Run /rebuilt — verify canary prints in agent terminal
 ↓
WS-2 (ContentResolver + PDF magic bytes)
 │  Create ContentResolver.ts with full resolve() flow
 │  Create tests/ContentResolver.test.ts
 │  Run /replug — verify 272 + new tests pass
 ↓
WS-3 (simplify mirrorDocToKnowledge)
 │  Rewrite to use ContentResolver
 │  Delete duplicated code
 │  Run /replug — verify all tests pass
 │  CRITICAL: Verify both callers still work:
 │    - addUrlToKnowledgeAction metadata { sourceId, versionId } → stores verbatim doc
 │    - ReconciliationService metadata { autognosticSourceId } → skips verbatim store
 ↓
WS-4 (WebPageProcessor hardening)
 │  Add publisher selectors, reference whitelist, length guard
 │  Add 3 tests to WebPageProcessor.test.ts
 │  Run /replug
 ↓
WS-5 (logging)
 │  Replace console.debug/warn in mirrorDocToKnowledge with logger
 │  Document LOG_LEVEL=debug for troubleshooting
 │  Run /replug
 ↓
WS-6 (integration tests)
 │  8 tests exercising full ContentResolver with mocked HTTP
 │  Run /replug — verify all new + existing tests pass

WS-1 is BLOCKING. Do it first and verify in the agent terminal before touching anything else.

GUARDRAILS

Run /replug after EVERY workstream — do not batch changes
All 272 existing tests must stay green — zero regressions
Do NOT modify these stable files:
- DocumentAnalyzer.ts / DocumentAnalyzer.types.ts
- ScientificPaperDetector.ts
- ScientificPaperHandler.ts
- getQuoteAction.ts
- GrammarEngine.ts
- ScientificSectionDetector.ts
- httpService.ts
- PdfExtractor.ts
ContentResolver is pure — depends only on HttpService, WebPageProcessor, PdfExtractor, and ScientificPaperDetector (all stateless/singletons). No database access, no IAgentRuntime dependency.
mirrorDocToKnowledge return type is frozen: { knowledgeDocumentId: string, clientDocumentId: string, worldId: UUID }. Both callers depend on this exact shape.
mirrorDocToKnowledge metadata contract is frozen: sourceId + versionId keys control verbatim storage. ReconciliationService's autognosticSourceId key intentionally does NOT match — this is correct behavior, preserve it.
Every new function must have at least one unit test
Git commit after each green workstream — git add -A && git commit -m "phase3: WS-N description"

VERIFICATION CHECKLIST

After all workstreams complete:

Agent terminal shows [autognostic] Plugin loaded — Phase 3, built <recent timestamp>
/replug reports all tests pass (272 + new ContentResolver + WebPageProcessor tests)
Springer URL ingestion logs show (with LOG_LEVEL=debug):
- Response content-type: text/html (routed to HTML pipeline)
- PDF link found: citation_pdf_url → <url>
- PDF link returned non-PDF content (paywall detected)
- Falling back to HTML extracted text (XXXX chars) where XXXX is 5K-30K
GET_EXACT_QUOTE on the Springer paper returns real sentences with paragraph structure
arXiv URL ingestion: Accept header requests PDF first; extracts from PDF when accessible
GitHub blob URL ingestion works unchanged (normalization preserved)
Build canary timestamp matches last bun run build time
ContentResolver.diagnostics array populated for every resolution

TROUBLESHOOTING

Canary shows NOT_BUILT: prebuild script didn't run. Check that package.json has the prebuild script and that bun run build triggers it.

Diagnostics not visible in logs: Set LOG_LEVEL=debug in agent's .env. Default minLevel is info.

Existing tests break after WS-3: Most likely the mirrorDocToKnowledge import path changed. Check that tests/integration.test.ts still resolves correctly.

ContentResolver tests fail on PDF extraction: The mock must return valid %PDF- magic bytes. Use new Uint8Array([0x25, 0x50, 0x44, 0x46, 0x2D, ...]) as the body.

WebPageProcessor returns empty text: Check that the content root selector matched. Add console.log(contentRoot?.tagName) temporarily to verify which element was selected.

FilesExpand file tree

PHASE3_PLAN.md

Latest commit

History