[codex] convert crawler to verification document indexing

[Nayab-23]

What changed

This repurposes the top-level crawler from GitHub lead discovery into a verification document crawler focused on SystemVerilog, SVA, protocol specifications, design specifications, formal artifacts, and related verification collateral.

Why it changed

The previous implementation was tightly coupled to GitHub commit and issue search. The new workflow needs to discover and rank verification documents from web sources and local internal document directories, and it needs Raspberry Pi-friendly continuous operation guidance.

Impact

• adds a document-centric crawl and ranking pipeline
• supports local indexing via LOCAL_DOC_DIRS
• updates merge and supervisor behavior to deduplicate document hits instead of GitHub leads
• refreshes the README and service docs for Raspberry Pi deployment and continuous runs

Validation

• python3 -m py_compile crawler.py merge_results.py coordinator.py supervisor.py dashboard.py