Fix evidence validation, debug diagnostics, and media bundling

README.md

@@ -4,8 +4,6 @@ Compile one Markdown article into one isolated, self-contained **OKF Bundle** (`
 
 OKF Compiler follows a subtraction-first rule: every document starts fresh. A compile reads only the current Markdown file and its relative assets. It never reads or mutates a long-lived knowledge base, performs no global concept merge, and materializes no global backlinks.
 
-The project was extracted from the experimental `openkb/okf` subsystem in `FFFFFFpy/OpenKB` and made independent of the OpenKB runtime.
-
 ## What it produces
 
 ```text
@@ -20,7 +18,8 @@ article.okf.zip
 │   └── article.md
 ├── sections/
 ├── assets/
-│   └── images/
+│   ├── images/
+│   └── media/
 ├── extracts/
 │   ├── summary.md
 │   ├── concepts/
@@ -31,7 +30,8 @@ article.okf.zip
 ```
 
 - `sources/original.md` preserves the input verbatim.
-- `sources/article.md` is the portable source with image links rewritten to bundled assets.
+- `sources/article.md` is the portable source with local asset links rewritten into the bundle.
+- Referenced Markdown images and local HTML `img`, `video`, `audio`, and `source` assets are copied.
 - `sections/*.md` are split conservatively from real ATX headings.
 - LLM output is document-local and evidence-anchored. It is proposed knowledge, not a global wiki mutation.
 
@@ -59,8 +59,6 @@ uv sync --extra dev --extra llm
 
 ## Configure the LLM with `.env`
 
-Copy the included template and fill in your provider values:
-
 ```powershell
 Copy-Item .env.example .env
 ```
@@ -72,72 +70,90 @@ OKF_LLM_API_KEY=your-key
 OKF_LLM_TIMEOUT=120
 ```
 
-Then test the connection:
-
-```powershell
-uv run okf test-llm
-```
-
 Configuration priority is:
 
 ```text
 CLI flags > process environment > .env > defaults
 ```
 
-Dotenv discovery order is:
+Dotenv discovery order is `--env-file`, `OKF_ENV_FILE`, `.env` in the current working directory, then `.env` beside the input.
 
-1. `--env-file PATH`.
-2. The path in `OKF_ENV_FILE`.
-3. `.env` in the current working directory.
-4. `.env` beside the input Markdown or input directory.
+## Compile one Markdown file
 
-Only one dotenv file is loaded. An explicitly selected file must exist; a typo does not silently fall back to some unrelated `.env` lurking elsewhere like a credential poltergeist.
+```powershell
+uv run okf compile article.md
+```
 
-Use a custom file when needed:
+Without LLM:
 
 ```powershell
-uv run okf test-llm --env-file "D:\secrets\okf.env"
+uv run okf compile article.md --no-llm
 ```
 
-`.env` is ignored by Git. `.env.example` is safe to commit and contains no real credentials.
+By default the output is `article.okf.zip` beside the source file.
 
-For migration, the old `OPENKB_LLM_*` names remain accepted as fallback values.
+## Debug diagnostics
 
-## Compile one Markdown file
-
-Without LLM:
+Normal bundle logs stay compact. For structured sidecar diagnostics:
 
 ```powershell
-uv run okf compile article.md --no-llm
+uv run okf compile article.md `
+  --debug-dir ".\okf-debug"
 ```
 
-With `.env` configured, no repeated LLM flags are needed:
+The run directory contains:
 
-```powershell
-uv run okf compile article.md
+```text
+okf-debug/<run>/
+├── run.json
+├── compiler.jsonl
+├── traceback.log              # only when an exception occurs
+└── validation/
+    ├── concepts.jsonl
+    ├── entities.jsonl
+    └── relations.jsonl
 ```
 
-CLI flags remain available for one-off overrides:
+To additionally save full LLM prompts, raw responses, and parsed JSON:
 
 ```powershell
 uv run okf compile article.md `
-  --base-url "https://api.example.com/v1" `
-  --model "your-model" `
-  --api-key "your-key"
+  --debug-dir ".\okf-debug" `
+  --debug-llm-payloads
 ```
 
-By default the output is `article.okf.zip` beside the source file.
+Full payload logs may contain the complete source article. API keys are not intentionally written, but debug directories should still be treated as private data.
 
-## Compile a wxarticle2md output directory
+Use strict mode when degraded extraction must fail CI or a batch job:
+
+```powershell
+uv run okf compile article.md --strict
+```
+
+The bundle is still written for inspection, but the command returns a non-zero exit code when an extraction stage rejected output.
+
+## Evidence model
 
-`wxarticle2md --md-mode advanced-lite` produces one article directory containing one Markdown file and `assets/`. OKF Compiler can enumerate that layout directly:
+New LLM extraction uses quote-based evidence:
+
+```json
+{
+  "section_id": "s0002",
+  "quote": "verbatim contiguous text copied from the document"
+}
+```
+
+The compiler locates the quote inside the selected section and calculates absolute `line_start` and `line_end`. This avoids asking an LLM to count Markdown lines. Legacy line-based evidence remains accepted for compatibility.
+
+## Compile a wxarticle2md output directory
 
 ```powershell
 uv run okf compile-dir `
   "D:\MyProjects\wxarticle2md\output" `
   --out "D:\MyProjects\okf-bundles" `
   --mode wechat `
-  --max-workers 3
+  --max-workers 3 `
+  --debug-dir "D:\MyProjects\okf-debug"
 ```
 
 Input selection priority in `wechat` mode:
@@ -155,8 +171,6 @@ Input selection priority in `wechat` mode:
 - Lower headings and conservative pseudo-heading matches become section anchors, not extra top-level sections.
 - If no suitable heading level exists, the document becomes one whole-document section.
 
-This deliberately favors stable, coarse sections over turning every bold sentence into a tiny knowledge fragment. Software has enough confetti already.
-
 ## Python API
 
 ```python
@@ -168,21 +182,13 @@ result = compile_one(
     Path("article.okf.zip"),
     CompileOptions(no_llm=True),
 )
-
-if not result.ok:
+if not result.successful:
     raise RuntimeError(result.error)
 ```
 
 ## Design boundaries
 
-OKF Compiler is a producer, not a knowledge-base runtime. Consumers such as OpenViking, WeKnora, or OpenKB may:
-
-- index sections, summaries, concepts, entities, and images separately;
-- resolve local concepts against global candidates;
-- accept or reject proposed edges;
-- materialize wiki pages or backlinks later.
-
-The compiler itself does none of those global operations.
+OKF Compiler is a producer, not a knowledge-base runtime. Consumers such as OpenViking, WeKnora, or OpenKB may index sections and extracts, resolve local concepts against global candidates, and accept or reject proposed edges. The compiler itself performs no global mutation.
 
 ## License and provenance
 

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "okf-compiler"
-version = "0.1.1"
+version = "0.2.0"
 description = "Compile Markdown articles into isolated, self-contained OKF Bundles."
 readme = "README.md"
 requires-python = ">=3.10"

src/okf_compiler/assets.py

@@ -1,4 +1,4 @@
-"""Collect relative Markdown images into a portable OKF bundle."""
+"""Collect relative Markdown and HTML media into a portable OKF bundle."""
 
 from __future__ import annotations
 
@@ -7,76 +7,145 @@
 import shutil
 from pathlib import Path
 
-from .schema import IMAGES_DIR, ImageRef
-
-_RELATIVE_RE = re.compile(r"!\[([^\]]*)\]\((?!https?://|data:)([^)]+)\)")
-_REWRITTEN_PREFIX = f"../{IMAGES_DIR}/"
-
-
-def collect_images(
-    section_bodies: list[str], source_markdown: str, source_dir: Path, images_dir: Path
-) -> tuple[list[str], str, list[ImageRef], list[str]]:
-    """Copy referenced images and rewrite both section and normalized source links."""
+from .schema import IMAGES_DIR, MEDIA_DIR, AssetRef
+
+_MARKDOWN_IMAGE_RE = re.compile(r"!\[([^\]]*)\]\((?!https?://|data:)([^)]+)\)", re.IGNORECASE)
+_HTML_TAG_RE = re.compile(r"<(?P<tag>video|source|audio|img)\b[^>]*>", re.IGNORECASE)
+_HTML_ATTR_RE = re.compile(
+    r"(?P<prefix>\b(?P<attr>src|poster)\s*=\s*)"
+    r"(?P<quote>[\"'])(?P<path>[^\"']+)(?P=quote)",
+    re.IGNORECASE,
+)
+_IMAGE_SUFFIXES = {".apng", ".avif", ".bmp", ".gif", ".ico", ".jpg", ".jpeg", ".png", ".svg", ".webp"}
+_AUDIO_SUFFIXES = {".aac", ".flac", ".m4a", ".mp3", ".ogg", ".opus", ".wav"}
+_VIDEO_SUFFIXES = {".avi", ".m4v", ".mkv", ".mov", ".mp4", ".webm"}
+
+
+def collect_assets(
+    section_bodies: list[str],
+    source_markdown: str,
+    source_dir: Path,
+    workdir: Path,
+) -> tuple[list[str], str, list[AssetRef], list[str]]:
     warnings: list[str] = []
-    refs: list[ImageRef] = []
-    assigned: dict[str, str] = {}
-    taken: set[str] = set()
+    refs: list[AssetRef] = []
+    assigned: dict[tuple[str, str], str] = {}
+    taken: dict[str, set[str]] = {IMAGES_DIR: set(), MEDIA_DIR: set()}
+    source_dir = Path(source_dir).resolve()
+    workdir = Path(workdir)
+
+    def register(rel_path: str, *, alt: str, kind: str, collect_ref: bool) -> str | None:
+        if _is_external_or_absolute(rel_path):
+            return None
+        norm = _normalize_rel(rel_path)
+        src = _resolve_under(rel_path, source_dir)
+        bundle_dir = IMAGES_DIR if kind == "image" else MEDIA_DIR
+        if src is None:
+            if collect_ref:
+                warnings.append(f"asset path escapes source dir: {rel_path}")
+            return None
+        if not src.is_file():
+            if collect_ref:
+                warnings.append(f"relative asset not found: {rel_path}")
+                refs.append(AssetRef(rel_path, "", str(src), False, alt, kind, bundle_dir))
+            return None
+
+        key = (bundle_dir, norm)
+        dest_name = assigned.get(key)
+        if dest_name is None:
+            dest_name = _unique_name(src.name, norm, taken[bundle_dir])
+            assigned[key] = dest_name
+            taken[bundle_dir].add(dest_name)
+            destination = workdir / bundle_dir / dest_name
+            destination.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy2(src, destination)
+        if collect_ref:
+            refs.append(AssetRef(rel_path, dest_name, str(src), True, alt, kind, bundle_dir))
+        return f"../{bundle_dir}/{dest_name}"
 
     def rewrite(text: str, *, collect_refs: bool) -> str:
-        parts: list[str] = []
-        cursor = 0
-        for match in _RELATIVE_RE.finditer(text):
-            parts.append(text[cursor : match.start()])
+        def markdown_repl(match: re.Match[str]) -> str:
             alt, rel_path = match.group(1), match.group(2)
-            norm = _normalize_rel(rel_path)
-            src = _resolve_under(rel_path, source_dir)
-            replacement = match.group(0)
-            if src is None:
-                if collect_refs:
-                    warnings.append(f"image path escapes source dir: {rel_path}")
-            elif not src.is_file():
-                if collect_refs:
-                    warnings.append(f"relative image not found: {rel_path}")
-                    refs.append(ImageRef(rel_path, "", str(src), False, alt))
-            else:
-                dest_name = assigned.get(norm)
-                if dest_name is None:
-                    dest_name = _unique_name(src.name, norm, taken)
-                    assigned[norm] = dest_name
-                    taken.add(dest_name)
-                    images_dir.mkdir(parents=True, exist_ok=True)
-                    shutil.copy2(src, images_dir / dest_name)
-                if collect_refs:
-                    refs.append(ImageRef(rel_path, dest_name, str(src), True, alt))
-                replacement = f"![{alt}]({_REWRITTEN_PREFIX}{dest_name})"
-            parts.append(replacement)
-            cursor = match.end()
-        parts.append(text[cursor:])
-        return "".join(parts)
+            replacement = register(rel_path, alt=alt, kind="image", collect_ref=collect_refs)
+            return f"![{alt}]({replacement})" if replacement else match.group(0)
+
+        text = _MARKDOWN_IMAGE_RE.sub(markdown_repl, text)
+
+        def html_tag_repl(tag_match: re.Match[str]) -> str:
+            tag = tag_match.group("tag").lower()
+
+            def html_attr_repl(attr_match: re.Match[str]) -> str:
+                rel_path = attr_match.group("path")
+                attr = attr_match.group("attr").lower()
+                kind = _asset_kind(tag, attr, rel_path)
+                replacement = register(rel_path, alt=attr, kind=kind, collect_ref=collect_refs)
+                if replacement is None:
+                    return attr_match.group(0)
+                quote = attr_match.group("quote")
+                return f"{attr_match.group('prefix')}{quote}{replacement}{quote}"
+
+            return _HTML_ATTR_RE.sub(html_attr_repl, tag_match.group(0))
+
+        return _HTML_TAG_RE.sub(html_tag_repl, text)
 
     rewritten_sections = [rewrite(body, collect_refs=True) for body in section_bodies]
     normalized_source = rewrite(source_markdown, collect_refs=False)
     return rewritten_sections, normalized_source, refs, warnings
 
 
-def count_missing(refs: list[ImageRef]) -> int:
+def collect_images(
+    section_bodies: list[str], source_markdown: str, source_dir: Path, images_dir: Path
+) -> tuple[list[str], str, list[AssetRef], list[str]]:
+    images_dir = Path(images_dir)
+    try:
+        workdir = images_dir.parents[1]
+    except IndexError:
+        workdir = images_dir.parent
+    return collect_assets(section_bodies, source_markdown, source_dir, workdir)
+
+
+def count_missing(refs: list[AssetRef]) -> int:
     return sum(1 for ref in refs if not ref.found)
 
 
+def _asset_kind(tag: str, attr: str, rel_path: str) -> str:
+    suffix = Path(rel_path.split("?", 1)[0].split("#", 1)[0]).suffix.lower()
+    if tag == "img" or attr == "poster" or suffix in _IMAGE_SUFFIXES:
+        return "image"
+    if tag == "audio" or suffix in _AUDIO_SUFFIXES:
+        return "audio"
+    if tag == "video" or suffix in _VIDEO_SUFFIXES:
+        return "video"
+    return "media"
+
+
+def _is_external_or_absolute(rel_path: str) -> bool:
+    value = rel_path.strip().strip('"').strip("'")
+    lower = value.lower()
+    return (
+        not value
+        or lower.startswith(("http://", "https://", "data:", "//", "#"))
+        or value.startswith(("/", "\\"))
+        or bool(re.match(r"^[A-Za-z]:", value))
+    )
+
+
 def _resolve_under(rel_path: str, source_dir: Path) -> Path | None:
     value = rel_path.strip().strip('"').strip("'")
-    if not value or value.startswith(("/", "\\")) or re.match(r"^[A-Za-z]:", value):
+    if _is_external_or_absolute(value):
         return None
+    value = value.split("?", 1)[0].split("#", 1)[0]
     candidate = (source_dir / value).resolve()
     try:
-        candidate.relative_to(source_dir.resolve())
+        candidate.relative_to(source_dir)
     except ValueError:
         return None
     return candidate
 
 
 def _normalize_rel(rel_path: str) -> str:
     value = rel_path.strip().strip('"').strip("'").replace("\\", "/")
+    value = value.split("?", 1)[0].split("#", 1)[0]
     return "/".join(part for part in value.split("/") if part and part != ".")