diff --git a/.env.example b/.env.example index 4dbc918..9ad3c68 100644 --- a/.env.example +++ b/.env.example @@ -1,6 +1,10 @@ PC_IPS=192.168.1.101,192.168.1.102,192.168.1.103,192.168.1.104 PC_USER=student -GITHUB_TOKENS=ghp_token1,ghp_token2,ghp_token3,ghp_token4 QUERIES_FILE=./queries.txt RESULTS_DIR=./results OUTPUT_FILE=./final_leads.csv +LOCAL_DOC_DIRS=/abs/path/specs,/abs/path/logs,/abs/path/assertions +MAX_WEB_RESULTS_PER_QUERY=8 +MAX_LOCAL_RESULTS_PER_QUERY=12 +SEARCH_DELAY_SECONDS=1.0 +MIN_PRIORITY_SCORE=6 diff --git a/README.md b/README.md index 7bc3acf..ffa5dde 100644 --- a/README.md +++ b/README.md @@ -1,127 +1,179 @@ -# GitHub Crawler +# Verification Document Crawler -Searches GitHub commits and issues for developers experiencing merge conflict pain, enriches each lead with GitHub profile and repo metadata, and outputs a prioritised CSV ranked by signal strength. Designed to run distributed across multiple lab PCs coordinated from a Raspberry Pi. +This repo crawls verification-related material instead of GitHub developer leads. ---- +Primary target categories: -## Requirements +- IEEE SystemVerilog references +- SVA tutorials and rulebooks +- Protocol specifications +- Design and microarchitecture specs +- Prior generated assertions and checker examples +- Formal logs, proofs, and counterexamples +- HIL correction notes +- RCA and postmortem reports +- Related verification plans, coverage reports, UVM references, and errata -- Python 3.8+ on the Raspberry Pi and each lab PC -- SSH access from the Pi to each lab PC (passwordless after setup) -- One GitHub personal access token per lab PC (classic tokens with `read:user` and `public_repo` scopes) +The crawler combines: ---- +- web discovery through Bing RSS search +- domain-focused queries for `ieeexplore.ieee.org`, `verificationacademy.com`, `accellera.org`, `github.com`, and PDF-heavy results +- optional local document indexing through `LOCAL_DOC_DIRS` for internal specs, logs, rulebooks, and reports -## Setup +## How It Works -### 1. Clone the repo onto the Raspberry Pi +For each query in `queries.txt`, the crawler: -```bash -git clone github-crawler -cd github-crawler -``` +1. searches the web across the configured source presets +2. scans local directories from `LOCAL_DOC_DIRS` +3. classifies each hit into a document type +4. scores and filters results +5. writes incremental CSV snapshots + +The main output is `final_leads.csv`, which is now a ranked document inventory rather than a contact-lead file. + +## Raspberry Pi Setup -### 2. Install dependencies +Recommended target: + +- Raspberry Pi 4 or newer +- Python 3.10+ +- always-on network connection +- enough local storage for logs and CSV history + +Clone and set up the repo: ```bash +git clone https://github.com/Nayab-23/GithubCrawler.git +cd GithubCrawler +python3 -m venv .venv +. .venv/bin/activate pip install -r requirements.txt +cp .env.example .env ``` -### 3. Configure environment +## Configuration + +Minimal `.env` for local continuous crawling: ```bash -cp .env.example .env +QUERIES_FILE=./queries.txt +RESULTS_DIR=./results +OUTPUT_FILE=./final_leads.csv +LOCAL_DOC_DIRS=/home/nayab/specs,/home/nayab/logs,/home/nayab/assertions,/home/nayab/rca +MAX_WEB_RESULTS_PER_QUERY=8 +MAX_LOCAL_RESULTS_PER_QUERY=12 +SEARCH_DELAY_SECONDS=1.0 +MIN_PRIORITY_SCORE=6 +POLL_INTERVAL_SECONDS=60 +STALL_TIMEOUT_SECONDS=600 +SLEEP_BETWEEN_SECONDS=3600 ``` -Edit `.env` and fill in: +Key variables: -| Variable | Description | -|---|---| -| `PC_IPS` | Comma-separated IP addresses of lab PCs | -| `PC_USER` | SSH username on each lab PC | -| `GITHUB_TOKENS` | One token per PC, in the same order as `PC_IPS` | -| `QUERIES_FILE` | Path to `queries.txt` (default: `./queries.txt`) | -| `RESULTS_DIR` | Directory where per-PC CSVs are collected (default: `./results`) | -| `OUTPUT_FILE` | Final merged output path (default: `./final_leads.csv`) | +- `LOCAL_DOC_DIRS`: comma-separated directories containing internal specs, rulebooks, logs, assertions, RCAs, and similar artifacts +- `SEARCH_DELAY_SECONDS`: delay between external web requests +- `MIN_PRIORITY_SCORE`: low-quality result cutoff +- `SLEEP_BETWEEN_SECONDS`: delay between full crawl cycles when using `supervisor.py` -### 4. Push SSH keys and deploy crawler to each PC +`LOCAL_DOC_DIRS` is the most important setting for internal verification material. -```bash -./setup_coordinator.sh -``` +## Running Once -This generates `~/.ssh/id_rsa` if needed, runs `ssh-copy-id` to each PC (prompts for password once per PC), and SCPs `crawler.py` to `/tmp/crawler.py` on each machine. +Run a single local crawl: ---- +```bash +./run_local.sh +``` -## Running +Run the multi-machine coordinator flow: ```bash ./run_crawl.sh ``` -Or directly: +The coordinator still splits `queries.txt` across machines, uploads `crawler.py`, runs it remotely, and merges the returned CSV files. + +## Running Continuously + +For Raspberry Pi use, the intended continuous entrypoint is: ```bash -python3 coordinator.py +. .venv/bin/activate +python supervisor.py ``` -The coordinator: -1. Splits `queries.txt` evenly across all PCs -2. SSHs into each PC in parallel and runs the crawler -3. Polls every 60 seconds until all crawlers finish (timeout: 4 hours) -4. SCPs result CSVs back to `RESULTS_DIR` -5. Runs `merge_results.py` to deduplicate, score, and produce the final output +Or install it as a systemd service: ---- - -## Output +```bash +chmod +x install_service.sh +./install_service.sh +``` -**`final_leads.csv`** contains one row per unique `(username, repo)` lead with the following fields: +Useful commands after installation: -| Field | Description | -|---|---| -| `query` | The search query that surfaced this lead | -| `source_type` | `commit`, `issue`, or `pr` | -| `repo` / `repo_name` | Full repo path and short name | -| `org` / `org_type` | Owner login and whether it's a `User` or `Organization` | -| `contributor_count` | Number of contributors to the repo | -| `language` | Primary repo language | -| `stars` | Repo star count | -| `username` | GitHub username | -| `display_name`, `email`, `company`, `bio`, `location` | GitHub profile fields | -| `github_profile`, `linkedin`, `twitter`, `blog` | Contact/social links | -| `commit_message` / `commit_url` / `commit_date` | Source commit or issue details | -| `priority_score` | Integer score (see below) | -| `priority` | `P1`, `P2`, or `P3` | +```bash +sudo systemctl status githubcrawler +sudo systemctl restart githubcrawler +tail -f supervisor.log +tail -f crawl.log +``` -### Priority tiers +## Should It Run 24/7? -| Label | Score | Meaning | -|---|---|---| -| **P1** | ≥ 8 | High-signal lead — strong repo fit, contact info present, relevant query | -| **P2** | 5–7 | Good lead — worth reaching out | -| **P3** | < 5 | Weak signal — low-fit repo or sparse profile | +Yes, it can run continuously on a Raspberry Pi, but it should not scrape aggressively in a tight loop. -Scoring bonuses include: contributor count in the 5–50 sweet spot (+3), email/LinkedIn present (+2 each), org account (+2), hardware language like VHDL/Verilog (+3), systems language like C/C++/Rust (+2), and high-pain queries like "merge conflict" or "synthesis error" (+2). +Recommended approach: -Leads are filtered before output: bots and CI accounts are removed, and repos with fewer than 3 or more than 500 contributors are excluded. +- run `supervisor.py` continuously +- set `SLEEP_BETWEEN_SECONDS` to something sensible like `1800` or `3600` +- keep `SEARCH_DELAY_SECONDS` at `1.0` or higher +- let local document crawling do most of the heavy lifting ---- +Practical guidance: -## Rate Limits +- `24/7` is fine for periodic refresh cycles +- `24/7 full-speed nonstop` is not a good idea +- external search quality will not improve much from hammering the web every minute +- local/internal document discovery benefits more from stable repeated indexing than from high request volume -GitHub enforces **30 search requests/minute** and **5,000 API calls/hour** per token. With 4 PCs each using a separate token: +Start with hourly cycles: -- **120 search requests/minute** combined -- **20,000 API calls/hour** combined +```bash +SLEEP_BETWEEN_SECONDS=3600 +``` -The crawler enforces a 2.1-second delay between every API call per token to stay safely within limits. If a 403 rate-limit response is received, the crawler reads the `X-RateLimit-Reset` header and sleeps until the window resets. +If you later need fresher results, reduce to: ---- +```bash +SLEEP_BETWEEN_SECONDS=900 +``` -## A Note on Language Filtering +I would not recommend a zero-sleep infinite loop on a Pi for this workload. -GitHub's `language:` filter **only works with code search** (`/search/code`), not with commit or issue search. Applying it to commit or issue queries silently returns zero results. +## Output -This crawler intentionally omits `language:` from all queries and instead filters by language **after enrichment**, using the primary language field from each repo's metadata. This means you get full result coverage and can filter to any language in post-processing. +The CSV contains document-centric fields such as: + +- `document_type` +- `title` +- `source_name` +- `source_domain` +- `url` +- `local_path` +- `file_type` +- `matched_keywords` +- `snippet` +- `published_hint` +- `priority_score` +- `priority` + +Higher scores are given to strong matches such as IEEE references, protocol specs, design specs, formal logs, assertion examples, and local internal documents. + +## Current Limitations + +- IEEE documents may resolve only to metadata or abstract pages when the full content is paywalled. +- Local content extraction is strongest for text files such as `.sv`, `.svh`, `.sva`, `.md`, `.txt`, `.log`, and `.rpt`. +- Binary formats such as `.pdf`, `.docx`, and `.pptx` are currently indexed mainly by path and filename unless you add a text extraction step. +- External web search is best-effort and should be treated as a discovery aid, not as a guaranteed structured corpus feed. diff --git a/coordinator.py b/coordinator.py index 3d20c93..f0d4e2c 100644 --- a/coordinator.py +++ b/coordinator.py @@ -23,11 +23,11 @@ def require_env(key): PC_IPS = [ip.strip() for ip in require_env("PC_IPS").split(",") if ip.strip()] -GITHUB_TOKENS = [t.strip() for t in require_env("GITHUB_TOKENS").split(",") if t.strip()] PC_USER = os.environ.get("PC_USER", "student").strip() QUERIES_FILE = os.environ.get("QUERIES_FILE", "./queries.txt").strip() RESULTS_DIR = os.environ.get("RESULTS_DIR", "./results").strip() OUTPUT_FILE = os.environ.get("OUTPUT_FILE", "./final_leads.csv").strip() +LOCAL_DOC_DIRS = os.environ.get("LOCAL_DOC_DIRS", "").strip() CRAWLER_LOCAL = os.path.join(os.path.dirname(os.path.abspath(__file__)), "crawler.py") CRAWLER_REMOTE = "/tmp/crawler.py" @@ -121,7 +121,7 @@ def split_queries(queries, n): # Per-PC worker # --------------------------------------------------------------------------- -def run_pc(ip, token, queries, results_dir, status): +def run_pc(ip, queries, results_dir, status): """ Full lifecycle for one PC: 1. SCP crawler.py @@ -133,6 +133,7 @@ def run_pc(ip, token, queries, results_dir, status): """ local_csv = os.path.join(results_dir, f"results_{ip}.csv") queries_json = json.dumps(queries) + local_doc_dirs_escaped = LOCAL_DOC_DIRS.replace("'", "'\"'\"'") def update(state, detail=""): status[ip] = {"state": state, "detail": detail} @@ -152,12 +153,8 @@ def update(state, detail=""): # Escape any single quotes inside the JSON so the shell single-quote # wrapper is never broken (replace ' with '"'"'). queries_json_escaped = queries_json.replace("'", "'\"'\"'") - remote_cmd = ( - f"GITHUB_TOKEN={token} " - f"python3 {CRAWLER_REMOTE} " - f"'{queries_json_escaped}' " - f"{REMOTE_CSV}" - ) + env_prefix = f"LOCAL_DOC_DIRS='{local_doc_dirs_escaped}' " if LOCAL_DOC_DIRS else "" + remote_cmd = f"{env_prefix}python3 {CRAWLER_REMOTE} '{queries_json_escaped}' {REMOTE_CSV}" proc = subprocess.Popen( ssh_cmd(ip, remote_cmd), stdout=subprocess.PIPE, @@ -207,17 +204,9 @@ def update(state, detail=""): # --------------------------------------------------------------------------- def main(): - # Validate token count matches PC count - if len(GITHUB_TOKENS) != len(PC_IPS): - log( - f"ERROR: GITHUB_TOKENS has {len(GITHUB_TOKENS)} entries but " - f"PC_IPS has {len(PC_IPS)}. They must match." - ) - sys.exit(1) - os.makedirs(RESULTS_DIR, exist_ok=True) - log("=== GitHub Crawler Coordinator ===") + log("=== Verification Document Crawler Coordinator ===") log(f" PCs : {', '.join(PC_IPS)}") log(f" SSH user : {PC_USER}") log(f" Queries : {QUERIES_FILE}") @@ -237,13 +226,13 @@ def main(): # Shared status dict, updated by threads (GIL protects simple dict writes) status = {ip: {"state": "pending", "detail": ""} for ip in PC_IPS} - def worker(ip, token, queries): - run_pc(ip, token, queries, RESULTS_DIR, status) + def worker(ip, queries): + run_pc(ip, queries, RESULTS_DIR, status) # Launch all PC workers in parallel threads = [] - for ip, token, chunk in zip(PC_IPS, GITHUB_TOKENS, chunks): - t = threading.Thread(target=worker, args=(ip, token, chunk), daemon=True) + for ip, chunk in zip(PC_IPS, chunks): + t = threading.Thread(target=worker, args=(ip, chunk), daemon=True) t.start() threads.append(t) diff --git a/crawler.py b/crawler.py index 21992fb..c947d0b 100644 --- a/crawler.py +++ b/crawler.py @@ -1,377 +1,518 @@ +import csv +import json import os +import re import sys -import json -import csv -import time import threading -import re -from datetime import datetime -from concurrent.futures import ThreadPoolExecutor, as_completed, TimeoutError as FutureTimeoutError +import time +import xml.etree.ElementTree as ET +from datetime import UTC, datetime +from html import unescape +from pathlib import Path +from typing import Iterable +from urllib.parse import urlparse import requests from dotenv import load_dotenv load_dotenv() -# --------------------------------------------------------------------------- -# Startup validation -# --------------------------------------------------------------------------- - -GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN", "").strip() -if not GITHUB_TOKEN: - print( - "ERROR: GITHUB_TOKEN environment variable is not set or is empty.\n" - " Export it before running: export GITHUB_TOKEN=ghp_...", - file=sys.stderr, - ) - sys.exit(1) - -HEADERS = { - "Authorization": f"token {GITHUB_TOKEN}", - "Accept": "application/vnd.github+json", -} - -RATE_LIMIT_DELAY = 2.1 # seconds between every API call -REQUEST_TIMEOUT = 30 # seconds before requests.get() gives up -ENRICH_TIMEOUT = 60 # seconds before a per-lead enrichment future is abandoned - -# --------------------------------------------------------------------------- -# Logging to crawl.log -# --------------------------------------------------------------------------- +USER_AGENT = os.environ.get( + "CRAWLER_USER_AGENT", + "Mozilla/5.0 (compatible; VerificationDocCrawler/1.0; +https://example.invalid)", +).strip() +REQUEST_TIMEOUT = int(os.environ.get("REQUEST_TIMEOUT_SECONDS", "30")) +SEARCH_DELAY_SECONDS = float(os.environ.get("SEARCH_DELAY_SECONDS", "1.0")) +MAX_WEB_RESULTS_PER_QUERY = int(os.environ.get("MAX_WEB_RESULTS_PER_QUERY", "8")) +MAX_LOCAL_RESULTS_PER_QUERY = int(os.environ.get("MAX_LOCAL_RESULTS_PER_QUERY", "12")) +MAX_LOCAL_FILE_BYTES = int(os.environ.get("MAX_LOCAL_FILE_BYTES", str(2 * 1024 * 1024))) +MIN_PRIORITY_SCORE = int(os.environ.get("MIN_PRIORITY_SCORE", "6")) +LOCAL_DOC_DIRS = [ + part.strip() + for part in os.environ.get("LOCAL_DOC_DIRS", "").split(",") + if part.strip() +] +HEADERS = {"User-Agent": USER_AGENT} LOG_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "crawl.log") _log_lock = threading.Lock() +_rate_lock = threading.Lock() +_next_call_at = 0.0 +CSV_FIELDS = [ + "query", + "source_type", + "document_type", + "title", + "source_name", + "source_domain", + "url", + "local_path", + "file_type", + "matched_keywords", + "snippet", + "published_hint", + "priority_score", + "priority", + "query_family", + "legacy_key", + "repo", + "repo_name", + "org", + "org_type", + "contributor_count", + "language", + "stars", + "username", + "display_name", + "email", + "company", + "bio", + "location", + "github_profile", + "linkedin", + "twitter", + "blog", + "commit_message", + "commit_url", + "commit_date", +] -def _ts(): - return datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ") +TEXT_EXTENSIONS = { + ".sv", + ".svh", + ".v", + ".vh", + ".sva", + ".txt", + ".md", + ".rst", + ".csv", + ".json", + ".yaml", + ".yml", + ".xml", + ".html", + ".htm", + ".log", + ".rpt", +} +SCAN_EXTENSIONS = TEXT_EXTENSIONS | {".pdf", ".doc", ".docx", ".ppt", ".pptx"} + +DOCUMENT_RULES = [ + ("ieee_systemverilog", (r"\bieee\b", r"\b1800\b", r"\bsystemverilog\b")), + ("sva_tutorial", (r"\bsva\b", r"\bassertion", r"\btutorial|\bguide|\btraining")), + ("internal_rulebook", (r"\brulebook\b", r"\bguideline\b|\bstyle guide\b|\bcoding standard\b")), + ("protocol_spec", (r"\bprotocol\b", r"\bspec\b|\bspecification\b|\bstandard\b")), + ("design_spec", (r"\bdesign\b", r"\bspec\b|\bspecification\b|\barchitecture\b")), + ("assertion_example", (r"\bassertion", r"\bproperty\b|\bsequence\b|\bchecker\b")), + ("formal_log", (r"\bformal\b", r"\blog\b|\bproof\b|\bcounterexample\b")), + ("hil_correction", (r"\bhil\b|\bhardware in the loop\b", r"\bfix\b|\bcorrection\b|\bpatch\b")), + ("rca_report", (r"\brca\b|\broot cause\b", r"\breport\b|\banalysis\b|\bpostmortem\b")), + ("coverage_report", (r"\bcoverage\b", r"\breport\b|\bclosure\b")), + ("uvm_reference", (r"\buvm\b", r"\bguide\b|\breference\b|\btutorial\b")), + ("verification_plan", (r"\bverification\b", r"\bplan\b|\bstrategy\b")), + ("errata", (r"\berrata\b|\bwaiver\b",)), +] +QUERY_FAMILIES = { + "ieee_systemverilog": ("ieee", "1800", "systemverilog"), + "sva_tutorial": ("sva", "assertion", "property", "sequence"), + "protocol_spec": ("protocol", "spec", "specification", "interface"), + "design_spec": ("design", "architecture", "microarchitecture", "block"), + "assertion_example": ("assertion", "checker", "property", "prior generated assertions"), + "formal_log": ("formal", "proof", "counterexample", "jasper", "vc formal"), + "hil_correction": ("hil", "hardware in the loop", "correction", "patch"), + "rca_report": ("rca", "root cause", "postmortem", "failure analysis"), +} -def log_line(msg): - """Append a timestamped line to crawl.log (thread-safe) and echo to stdout.""" - line = f"[{_ts()}] {msg}" - print(line, flush=True) - with _log_lock: - with open(LOG_PATH, "a", encoding="utf-8") as f: - f.write(line + "\n") +SEARCH_PRESETS = [ + {"source_name": "general-web", "site": None}, + {"source_name": "ieee", "site": "ieeexplore.ieee.org"}, + {"source_name": "verification-academy", "site": "verificationacademy.com"}, + {"source_name": "accellera", "site": "accellera.org"}, + {"source_name": "github", "site": "github.com"}, + {"source_name": "pdf-index", "site": None, "filetype": "pdf"}, +] +STOPWORDS = { + "and", + "the", + "for", + "with", + "pdf", + "guide", + "best", + "practices", + "report", + "tutorial", + "spec", + "specification", +} -# --------------------------------------------------------------------------- -# Thread-safe rate limiter -# --------------------------------------------------------------------------- -_rate_lock = threading.Lock() -_next_call_at = 0.0 # monotonic time before which no new call may fire +def _ts() -> str: + return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ") -def rate_limited_get(url, params=None, extra_headers=None): - """GET with a global 2.1 s floor between calls, handling 403 rate limits. +def log_line(message: str) -> None: + line = f"[{_ts()}] {message}" + print(line, flush=True) + with _log_lock: + with open(LOG_PATH, "a", encoding="utf-8") as handle: + handle.write(line + "\n") - Each thread atomically reserves the next available time slot and then - sleeps *outside* the lock so that threads don't pile up blocking one - another while sleeping. - Raises requests.exceptions.Timeout or requests.exceptions.ConnectionError - to callers — do not swallow them here so individual functions can decide - how to handle them. - """ +def rate_limited_get(url: str, params: dict[str, str] | None = None) -> requests.Response: global _next_call_at + with _rate_lock: + now = time.monotonic() + fire_at = max(now, _next_call_at) + _next_call_at = fire_at + SEARCH_DELAY_SECONDS + wait = fire_at - time.monotonic() + if wait > 0: + time.sleep(wait) + return requests.get(url, headers=HEADERS, params=params, timeout=REQUEST_TIMEOUT) - headers = dict(HEADERS) - if extra_headers: - headers.update(extra_headers) - - while True: - # Reserve a slot without sleeping under the lock. - with _rate_lock: - now = time.monotonic() - fire_at = max(now, _next_call_at) - _next_call_at = fire_at + RATE_LIMIT_DELAY - - wait = fire_at - time.monotonic() - if wait > 0: - time.sleep(wait) - - resp = requests.get(url, headers=headers, params=params, timeout=REQUEST_TIMEOUT) - - if resp.status_code == 403: - reset_ts = resp.headers.get("X-RateLimit-Reset") - if reset_ts: - sleep_secs = max(int(reset_ts) - int(time.time()), 0) + 5 - print(f" [rate limit] 403 received. Sleeping {sleep_secs}s until reset …") - time.sleep(sleep_secs) - continue - else: - print(f" [warn] 403 with no X-RateLimit-Reset header for {url}") - return resp - return resp +def text_to_words(text: str) -> list[str]: + return re.findall(r"[a-z0-9_+\-/.#]+", (text or "").lower()) -# --------------------------------------------------------------------------- -# Search helpers -# --------------------------------------------------------------------------- +def matched_keywords(text: str, query: str) -> list[str]: + haystack = f"{text} {query}".lower() + words = [] + for token in text_to_words(query): + if len(token) >= 3 and token in haystack and token not in words: + words.append(token) + return words[:8] -BOT_PATTERN = re.compile( - r"(bot|ci|auto|dependabot|renovate|github-actions)", - re.IGNORECASE, -) -CSV_FIELDS = [ - "query", "source_type", "repo", "repo_name", "org", "org_type", - "contributor_count", "language", "stars", "username", "display_name", - "email", "company", "bio", "location", "github_profile", "linkedin", - "twitter", "blog", "commit_message", "commit_url", "commit_date", -] +def query_terms(query: str) -> list[str]: + return [token for token in text_to_words(query) if len(token) >= 3 and token not in STOPWORDS] -def write_snapshot(leads, output_path): - """Persist an incremental CSV snapshot for live monitoring.""" - os.makedirs(os.path.dirname(output_path) if os.path.dirname(output_path) else ".", exist_ok=True) - with open(output_path, "w", newline="", encoding="utf-8") as f: - writer = csv.DictWriter(f, fieldnames=CSV_FIELDS, extrasaction="ignore") - writer.writeheader() - for lead in leads: - writer.writerow({field: lead.get(field, "") for field in CSV_FIELDS}) +def is_relevant_result(query: str, title: str, snippet: str, link: str, site: str | None) -> bool: + text = f"{title} {snippet} {link}".lower() + terms = query_terms(query) + hits = sum(1 for token in terms if token in text) + if site and site not in source_domain_from_url(link): + return False + if "systemverilog" in query.lower() and "systemverilog" not in text and "sva" not in text: + return False + if "formal" in query.lower() and "formal" not in text and "counterexample" not in text and "proof" not in text: + return False + return hits >= 2 -def search_commits(query): - """Return list of raw lead dicts from commit search.""" - leads = [] - url = "https://api.github.com/search/commits" - params = {"q": query, "per_page": 30, "page": 1} - extra = {"Accept": "application/vnd.github.cloak-preview+json"} +def classify_query(query: str) -> str: + lowered = query.lower() + for family, terms in QUERY_FAMILIES.items(): + if any(term in lowered for term in terms): + return family + return "verification_misc" - print(f" [commits] searching: {query}") - try: - resp = rate_limited_get(url, params=params, extra_headers=extra) - except requests.exceptions.Timeout: - print(f" [warn] commit search timed out for query: {query}") - return leads - except requests.exceptions.ConnectionError as exc: - print(f" [warn] commit search connection error for query '{query}': {exc}") - return leads - if not resp.ok: - print(f" [warn] commit search failed ({resp.status_code}): {resp.text[:200]}") - return leads - - items = resp.json().get("items", []) - print(f" [commits] got {len(items)} results") - - for item in items: - repo_full = item.get("repository", {}).get("full_name", "") - author = (item.get("author") or {}).get("login", "") - committer = (item.get("committer") or {}).get("login", "") - username = author or committer - if not username or not repo_full: - continue +def classify_document(text: str, domain: str, file_type: str) -> str: + lowered = f"{text} {domain} {file_type}".lower() + for doc_type, patterns in DOCUMENT_RULES: + if all(re.search(pattern, lowered) for pattern in patterns): + return doc_type + if "ieeexplore.ieee.org" in domain and "systemverilog" in lowered: + return "ieee_systemverilog" + if file_type in {"sv", "svh", "sva"} and re.search(r"\bassert", lowered): + return "assertion_example" + if file_type in {"log", "rpt"} and "formal" in lowered: + return "formal_log" + return "verification_misc" - commit_data = item.get("commit", {}) - message = commit_data.get("message", "")[:200] - commit_url = item.get("html_url", "") - commit_date = (commit_data.get("author") or {}).get("date", "") - leads.append({ - "query": query, - "source_type": "commit", - "repo": repo_full, - "username": username, - "commit_message": message, - "commit_url": commit_url, - "commit_date": commit_date, - }) +def normalize_title(raw: str, fallback: str) -> str: + title = re.sub(r"\s+", " ", unescape(raw or "")).strip() + return title or fallback - return leads +def file_type_from_name(path_or_url: str) -> str: + parsed = urlparse(path_or_url) + candidate = parsed.path if parsed.scheme else path_or_url + suffix = Path(candidate).suffix.lower().lstrip(".") + return suffix or "html" -def search_issues(query): - """Return list of raw lead dicts from issues/PR search.""" - leads = [] - url = "https://api.github.com/search/issues" - params = {"q": query, "per_page": 30, "page": 1} - print(f" [issues] searching: {query}") +def source_domain_from_url(url: str) -> str: try: - resp = rate_limited_get(url, params=params) - except requests.exceptions.Timeout: - print(f" [warn] issue search timed out for query: {query}") - return leads - except requests.exceptions.ConnectionError as exc: - print(f" [warn] issue search connection error for query '{query}': {exc}") - return leads + return (urlparse(url).netloc or "").lower() + except Exception: + return "" + + +def score_document(result: dict[str, str]) -> int: + score = 0 + doc_type = result.get("document_type", "") + file_type = result.get("file_type", "") + domain = result.get("source_domain", "") + keyword_count = len((result.get("matched_keywords") or "").split(", ")) if result.get("matched_keywords") else 0 + + doc_type_weights = { + "ieee_systemverilog": 5, + "protocol_spec": 5, + "design_spec": 4, + "formal_log": 4, + "assertion_example": 4, + "sva_tutorial": 3, + "internal_rulebook": 3, + "hil_correction": 3, + "rca_report": 3, + "coverage_report": 2, + "verification_plan": 2, + } + score += doc_type_weights.get(doc_type, 1) - if not resp.ok: - print(f" [warn] issue search failed ({resp.status_code}): {resp.text[:200]}") - return leads - - items = resp.json().get("items", []) - print(f" [issues] got {len(items)} results") - - for item in items: - html_url = item.get("html_url", "") - repo_url = item.get("repository_url", "") - repo_full = "/".join(repo_url.split("/")[-2:]) if repo_url else "" - username = (item.get("user") or {}).get("login", "") - if not username or not repo_full: - continue + if domain in {"ieeexplore.ieee.org", "standards.ieee.org"}: + score += 4 + elif domain in {"verificationacademy.com", "accellera.org", "github.com"}: + score += 2 - source_type = "pr" if "/pull/" in html_url else "issue" + if file_type in {"pdf", "sv", "svh", "sva", "log", "rpt"}: + score += 2 + if result.get("source_type") == "local_file": + score += 3 - leads.append({ - "query": query, - "source_type": source_type, - "repo": repo_full, - "username": username, - "commit_message": item.get("title", "")[:200], - "commit_url": html_url, - "commit_date": item.get("created_at", ""), - }) + score += min(keyword_count, 4) + return score - return leads +def priority_label(score: int) -> str: + if score >= 11: + return "P1" + if score >= 7: + return "P2" + return "P3" -# --------------------------------------------------------------------------- -# Enrichment helpers -# --------------------------------------------------------------------------- -def enrich_user(username): - """Fetch GitHub profile data for a username. Returns {} on any network error.""" - url = f"https://api.github.com/users/{username}" +def write_snapshot(rows: Iterable[dict[str, str]], output_path: str) -> None: + parent = os.path.dirname(output_path) + if parent: + os.makedirs(parent, exist_ok=True) + with open(output_path, "w", newline="", encoding="utf-8") as handle: + writer = csv.DictWriter(handle, fieldnames=CSV_FIELDS, extrasaction="ignore") + writer.writeheader() + for row in rows: + writer.writerow({field: row.get(field, "") for field in CSV_FIELDS}) + + +def to_row( + *, + query: str, + source_type: str, + title: str, + source_name: str, + source_domain: str, + url: str = "", + local_path: str = "", + file_type: str = "", + snippet: str = "", + published_hint: str = "", +) -> dict[str, str]: + doc_type = classify_document(f"{title} {snippet}", source_domain, file_type) + keywords = matched_keywords(f"{title} {snippet} {local_path} {url}", query) + family = classify_query(query) + legacy_key = url or local_path or f"{source_name}:{title}" + row = { + "query": query, + "source_type": source_type, + "document_type": doc_type, + "title": title, + "source_name": source_name, + "source_domain": source_domain, + "url": url, + "local_path": local_path, + "file_type": file_type, + "matched_keywords": ", ".join(keywords), + "snippet": snippet.strip(), + "published_hint": published_hint, + "query_family": family, + "legacy_key": legacy_key, + "repo": source_domain or source_name, + "repo_name": title, + "org": source_name, + "org_type": source_type, + "contributor_count": "", + "language": doc_type, + "stars": "", + "username": source_name, + "display_name": title, + "email": "", + "company": source_domain, + "bio": snippet[:280], + "location": local_path, + "github_profile": url if source_domain == "github.com" else "", + "linkedin": "", + "twitter": "", + "blog": local_path, + "commit_message": snippet[:200] or title[:200], + "commit_url": url, + "commit_date": published_hint or _ts(), + } + score = score_document(row) + row["priority_score"] = str(score) + row["priority"] = priority_label(score) + return row + + +def search_bing_rss(query: str, source_name: str, site: str | None, filetype: str | None) -> list[dict[str, str]]: + q = query + if site: + q = f"{q} site:{site}" + if filetype: + q = f"{q} filetype:{filetype}" + url = "https://www.bing.com/search" + params = {"q": q, "format": "rss"} try: - resp = rate_limited_get(url) - except requests.exceptions.Timeout: - print(f" [warn] user enrich timed out: {username}") - return {} - except requests.exceptions.ConnectionError as exc: - print(f" [warn] user enrich connection error for {username}: {exc}") - return {} - + resp = rate_limited_get(url, params=params) + except requests.RequestException as exc: + log_line(f"web_search_failed source={source_name} query={json.dumps(query)} error={exc}") + return [] if not resp.ok: - print(f" [warn] user enrichment failed for {username} ({resp.status_code})") - return {} - - data = resp.json() - blog = data.get("blog") or "" - linkedin = blog if "linkedin.com" in blog.lower() else "" - - return { - "display_name": data.get("name") or "", - "email": data.get("email") or "", - "company": data.get("company") or "", - "bio": data.get("bio") or "", - "location": data.get("location") or "", - "github_profile": data.get("html_url") or "", - "twitter": data.get("twitter_username") or "", - "blog": blog, - "linkedin": linkedin, - } - - -def _parse_contributor_count(resp): - """ - Use the Link header rel=last trick to get contributor count. - Falls back to counting the returned list if Link header is absent. - """ - link_header = resp.headers.get("Link", "") - if link_header: - match = re.search(r'page=(\d+)>;\s*rel="last"', link_header) - if match: - last_page = int(match.group(1)) - return last_page * 30 + log_line( + f"web_search_failed source={source_name} query={json.dumps(query)} status={resp.status_code}" + ) + return [] try: - return len(resp.json()) - except Exception: - return 0 - + root = ET.fromstring(resp.text) + except ET.ParseError as exc: + log_line(f"web_search_parse_failed source={source_name} query={json.dumps(query)} error={exc}") + return [] + + rows = [] + for item in root.findall("./channel/item"): + title = normalize_title(item.findtext("title", default=""), "Untitled result") + link = (item.findtext("link", default="") or "").strip() + snippet = normalize_title(item.findtext("description", default=""), "") + if not link: + continue + if not is_relevant_result(query, title, snippet, link, site): + continue + domain = source_domain_from_url(link) + rows.append( + to_row( + query=query, + source_type="web_result", + title=title, + source_name=source_name, + source_domain=domain, + url=link, + file_type=file_type_from_name(link), + snippet=snippet, + ) + ) + if len(rows) >= MAX_WEB_RESULTS_PER_QUERY: + break + log_line(f'web_search source="{source_name}" query="{query}" results={len(rows)}') + return rows -def enrich_repo(repo_full): - """Fetch repo metadata and contributor count. Returns {} on any network error.""" - repo_url = f"https://api.github.com/repos/{repo_full}" - try: - resp = rate_limited_get(repo_url) - except requests.exceptions.Timeout: - print(f" [warn] repo enrich timed out: {repo_full}") - return {} - except requests.exceptions.ConnectionError as exc: - print(f" [warn] repo enrich connection error for {repo_full}: {exc}") - return {} - if not resp.ok: - print(f" [warn] repo enrichment failed for {repo_full} ({resp.status_code})") - return {} - - data = resp.json() - owner = data.get("owner") or {} - org = owner.get("login") or "" - org_type = owner.get("type") or "" - repo_name = data.get("name") or "" - description = data.get("description") or "" - language = data.get("language") or "" - stars = data.get("stargazers_count") or 0 - - contributor_count = 0 - contrib_url = f"https://api.github.com/repos/{repo_full}/contributors" +def read_local_text(path: Path) -> str: + if path.suffix.lower() not in TEXT_EXTENSIONS: + return "" + if path.stat().st_size > MAX_LOCAL_FILE_BYTES: + return "" try: - contrib_resp = rate_limited_get(contrib_url, params={"per_page": 30, "anon": "true"}) - if contrib_resp.ok: - contributor_count = _parse_contributor_count(contrib_resp) - except requests.exceptions.Timeout: - print(f" [warn] contributor fetch timed out: {repo_full}") - except requests.exceptions.ConnectionError as exc: - print(f" [warn] contributor fetch connection error for {repo_full}: {exc}") - - return { - "repo_name": repo_name, - "org": org, - "org_type": org_type, - "language": language, - "stars": stars, - "description": description, - "contributor_count": contributor_count, - } - - -# --------------------------------------------------------------------------- -# Filtering -# --------------------------------------------------------------------------- - -def should_filter_out(lead): - username = lead.get("username", "") - contributor_count = lead.get("contributor_count", 0) + return path.read_text(encoding="utf-8", errors="ignore") + except OSError: + return "" - if BOT_PATTERN.search(username): - return True - if contributor_count < 3 or contributor_count > 500: - return True - return False +def search_local_docs(query: str) -> list[dict[str, str]]: + if not LOCAL_DOC_DIRS: + return [] -# --------------------------------------------------------------------------- -# Core enrichment worker -# --------------------------------------------------------------------------- + query_tokens = [token for token in text_to_words(query) if len(token) >= 3] + rows: list[dict[str, str]] = [] -def enrich_lead(lead): - """Enrich a single lead with user + repo data. Returns updated lead dict.""" - username = lead["username"] - repo = lead["repo"] - - print(f" [enrich] {username} @ {repo}") - - user_data = enrich_user(username) - repo_data = enrich_repo(repo) + for root_dir in LOCAL_DOC_DIRS: + root = Path(root_dir).expanduser() + if not root.exists(): + continue + for path in root.rglob("*"): + if not path.is_file(): + continue + if path.suffix.lower() not in SCAN_EXTENSIONS: + continue - lead.update(user_data) - lead.update(repo_data) - return lead + file_type = file_type_from_name(str(path)) + haystack_parts = [str(path.name), str(path)] + body = read_local_text(path) + if body: + haystack_parts.append(body[:20000]) + haystack = " ".join(haystack_parts).lower() + hits = [token for token in query_tokens if token in haystack] + if not hits: + continue + snippet = "" + if body: + for line in body.splitlines(): + candidate = line.strip() + if candidate and any(token in candidate.lower() for token in hits): + snippet = candidate[:240] + break + rows.append( + to_row( + query=query, + source_type="local_file", + title=path.name, + source_name="local-docs", + source_domain="local", + local_path=str(path), + file_type=file_type, + snippet=snippet, + published_hint=datetime.fromtimestamp(path.stat().st_mtime).strftime("%Y-%m-%d %H:%M:%S"), + ) + ) + + rows.sort(key=lambda row: int(row["priority_score"]), reverse=True) + limited = rows[:MAX_LOCAL_RESULTS_PER_QUERY] + log_line(f'local_search query="{query}" results={len(limited)} roots={len(LOCAL_DOC_DIRS)}') + return limited + + +def dedupe_rows(rows: list[dict[str, str]]) -> list[dict[str, str]]: + seen: set[tuple[str, str, str]] = set() + deduped: list[dict[str, str]] = [] + for row in rows: + key = ( + (row.get("query") or "").strip(), + (row.get("url") or "").strip(), + (row.get("local_path") or "").strip(), + ) + if key in seen: + continue + seen.add(key) + deduped.append(row) + return deduped + + +def crawl_query(query: str) -> list[dict[str, str]]: + rows = [] + for preset in SEARCH_PRESETS: + rows.extend( + search_bing_rss( + query=query, + source_name=preset["source_name"], + site=preset.get("site"), + filetype=preset.get("filetype"), + ) + ) + rows.extend(search_local_docs(query)) + rows = dedupe_rows(rows) + rows = [row for row in rows if int(row["priority_score"]) >= MIN_PRIORITY_SCORE] + rows.sort(key=lambda row: int(row["priority_score"]), reverse=True) + return rows -# --------------------------------------------------------------------------- -# Main -# --------------------------------------------------------------------------- -def main(): +def main() -> None: if len(sys.argv) != 3: print("Usage: python crawler.py '' ", file=sys.stderr) sys.exit(1) @@ -383,91 +524,34 @@ def main(): sys.exit(1) output_path = sys.argv[2] - - print(f"=== GitHub Crawler starting ===") - print(f" Queries : {len(queries)}") - print(f" Output : {output_path}") - print(f" Log : {LOG_PATH}") + print("=== Verification Document Crawler starting ===") + print(f" Queries : {len(queries)}") + print(f" Output : {output_path}") + print(f" Local doc dirs : {', '.join(LOCAL_DOC_DIRS) if LOCAL_DOC_DIRS else '(none)'}") + print(f" Log : {LOG_PATH}") print() - # ------------------------------------------------------------------ - # Phase 1: Search — log after every query completes - # ------------------------------------------------------------------ - raw_leads = [] + all_rows: list[dict[str, str]] = [] for query in queries: - commit_leads = search_commits(query) - issue_leads = search_issues(query) - raw_leads.extend(commit_leads) - raw_leads.extend(issue_leads) - write_snapshot(raw_leads, output_path) - log_line( - f'query="{query}" commits={len(commit_leads)} issues={len(issue_leads)}' - ) - - print(f"\n[phase 1] collected {len(raw_leads)} raw leads") - - # ------------------------------------------------------------------ - # Phase 2: Deduplicate by (username, repo) - # ------------------------------------------------------------------ - seen = {} - for lead in raw_leads: - key = (lead["username"], lead["repo"]) - if key not in seen: - seen[key] = lead - - unique_leads = list(seen.values()) - print(f"[phase 2] {len(unique_leads)} unique leads after deduplication\n") - - # ------------------------------------------------------------------ - # Phase 3: Enrich (threaded, rate-limited, per-future timeout) - # ------------------------------------------------------------------ - print("[phase 3] enriching leads …") - enriched = [] - skipped = 0 - - with ThreadPoolExecutor(max_workers=6) as executor: - futures = {executor.submit(enrich_lead, lead): lead for lead in unique_leads} - for future in as_completed(futures): - lead = futures[future] - try: - result = future.result(timeout=ENRICH_TIMEOUT) - enriched.append(result) - filtered_snapshot = [item for item in enriched if not should_filter_out(item)] - write_snapshot(filtered_snapshot, output_path) - except FutureTimeoutError: - skipped += 1 - print( - f" [warn] enrichment timed out after {ENRICH_TIMEOUT}s — " - f"skipping {lead.get('username')} @ {lead.get('repo')}" - ) - except Exception as exc: - skipped += 1 - print(f" [error] enrichment exception: {exc}") - - print(f"\n[phase 3] enriched {len(enriched)} leads, skipped {skipped}") - - # ------------------------------------------------------------------ - # Phase 4: Filter - # ------------------------------------------------------------------ - filtered = [l for l in enriched if not should_filter_out(l)] - print(f"[phase 4] {len(filtered)} leads after filtering " - f"(removed {len(enriched) - len(filtered)})\n") - - # ------------------------------------------------------------------ - # Phase 5: Write CSV - # ------------------------------------------------------------------ - write_snapshot(filtered, output_path) + rows = crawl_query(query) + all_rows.extend(rows) + unique_rows = dedupe_rows(all_rows) + write_snapshot(unique_rows, output_path) + log_line(f'query_complete query="{query}" rows={len(rows)} cumulative={len(unique_rows)}') - print(f"[done] wrote {len(filtered)} leads to {output_path}") + final_rows = dedupe_rows(all_rows) + final_rows.sort(key=lambda row: int(row["priority_score"]), reverse=True) + write_snapshot(final_rows, output_path) - # ------------------------------------------------------------------ - # Final log entry - # ------------------------------------------------------------------ - log_line(f"CRAWL COMPLETE total_leads={len(filtered)}") + print(f"[done] wrote {len(final_rows)} results to {output_path}") + log_line(f"CRAWL COMPLETE total_rows={len(final_rows)}") if __name__ == "__main__": if "--test" in sys.argv: - print("=== TEST MODE: single query 'fix merge conflict' → /tmp/test_output.csv ===") - sys.argv = [sys.argv[0], '["fix merge conflict"]', "/tmp/test_output.csv"] + sys.argv = [ + sys.argv[0], + json.dumps(["IEEE 1800 SystemVerilog SVA tutorial", "formal verification counterexample log"]), + "/tmp/test_output.csv", + ] main() diff --git a/githubcrawler.service b/githubcrawler.service index fa9be18..d5f50d0 100644 --- a/githubcrawler.service +++ b/githubcrawler.service @@ -1,5 +1,5 @@ [Unit] -Description=GitHub Crawler Supervisor +Description=Verification Document Crawler Supervisor Documentation=https://github.com/Nayab-23/GithubCrawler After=network-online.target Wants=network-online.target diff --git a/install_service.sh b/install_service.sh index 74b379b..269717d 100755 --- a/install_service.sh +++ b/install_service.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Installs and starts the githubcrawler systemd service. +# Installs and starts the verification document crawler systemd service. # Run once on the Pi from ~/GithubCrawler: # # chmod +x install_service.sh @@ -14,7 +14,7 @@ VENV="/home/nayab/GithubCrawler/.venv" ENV_FILE="/home/nayab/GithubCrawler/.env" SUPERVISOR="/home/nayab/GithubCrawler/supervisor.py" -echo "=== GitHub Crawler — systemd install ===" +echo "=== Verification Document Crawler — systemd install ===" echo "" # ── pre-flight checks ────────────────────────────────────────────────────── diff --git a/merge_results.py b/merge_results.py index ac83b83..c1602c9 100644 --- a/merge_results.py +++ b/merge_results.py @@ -1,174 +1,115 @@ -import os -import sys import csv import glob +import os +import sys -# --------------------------------------------------------------------------- -# Field definitions -# --------------------------------------------------------------------------- - -BASE_FIELDS = [ - "query", "source_type", "repo", "repo_name", "org", "org_type", - "contributor_count", "language", "stars", "username", "display_name", - "email", "company", "bio", "location", "github_profile", "linkedin", - "twitter", "blog", "commit_message", "commit_url", "commit_date", +OUTPUT_FIELDS = [ + "query", + "source_type", + "document_type", + "title", + "source_name", + "source_domain", + "url", + "local_path", + "file_type", + "matched_keywords", + "snippet", + "published_hint", + "priority_score", + "priority", + "query_family", + "legacy_key", + "repo", + "repo_name", + "org", + "org_type", + "contributor_count", + "language", + "stars", + "username", + "display_name", + "email", + "company", + "bio", + "location", + "github_profile", + "linkedin", + "twitter", + "blog", + "commit_message", + "commit_url", + "commit_date", ] -OUTPUT_FIELDS = BASE_FIELDS + ["priority_score", "priority"] - -# --------------------------------------------------------------------------- -# Scoring -# --------------------------------------------------------------------------- - -HARDWARE_LANG_KEYWORDS = {"vhdl", "verilog", "systemverilog"} -SYSTEMS_LANG_KEYWORDS = {"c", "c++", "cpp", "rust"} -MODERN_LANG_KEYWORDS = {"typescript", "python"} - -QUERY_BOOST_PHRASES = { - "merge conflict", - "breaking change", - "port mismatch", - "synthesis error", -} - - -def score_lead(lead): - score = 0 - - # contributor_count bands - try: - cc = int(lead.get("contributor_count") or 0) - except (ValueError, TypeError): - cc = 0 - - if 5 <= cc <= 50: - score += 3 - elif 3 <= cc <= 4: - score += 1 - elif 51 <= cc <= 200: - score += 2 - - # profile completeness - if lead.get("email", "").strip(): - score += 2 - if lead.get("linkedin", "").strip(): - score += 2 - if lead.get("company", "").strip(): - score += 1 - - # org type - if (lead.get("org_type") or "").strip() == "Organization": - score += 2 - - # language bonus - lang = (lead.get("language") or "").strip().lower() - if lang in HARDWARE_LANG_KEYWORDS: - score += 3 - elif lang in SYSTEMS_LANG_KEYWORDS: - score += 2 - elif lang in MODERN_LANG_KEYWORDS: - score += 1 - # query content bonus - query_lower = (lead.get("query") or "").lower() - if any(phrase in query_lower for phrase in QUERY_BOOST_PHRASES): - score += 2 +def row_key(row: dict[str, str]) -> tuple[str, str, str]: + return ( + (row.get("query") or "").strip(), + (row.get("url") or "").strip(), + (row.get("local_path") or "").strip(), + ) - return score - -def priority_label(score): - if score >= 8: - return "P1" - if score >= 5: - return "P2" - return "P3" - - -# --------------------------------------------------------------------------- -# Main -# --------------------------------------------------------------------------- - -def main(): +def main() -> None: if len(sys.argv) != 3: print("Usage: python merge_results.py ", file=sys.stderr) sys.exit(1) results_dir = sys.argv[1] output_path = sys.argv[2] - csv_files = glob.glob(os.path.join(results_dir, "*.csv")) if not csv_files: print(f"ERROR: no *.csv files found in {results_dir}", file=sys.stderr) sys.exit(1) - print(f"=== Merge Results ===") + seen: dict[tuple[str, str, str], dict[str, str]] = {} + total_read = 0 + + print("=== Merge Results ===") print(f" Input dir : {results_dir}") print(f" CSV files : {len(csv_files)}") print(f" Output : {output_path}") print() - # ------------------------------------------------------------------ - # Read & deduplicate - # ------------------------------------------------------------------ - seen = {} # (username, repo) -> row dict - total_read = 0 - for path in sorted(csv_files): file_count = 0 try: - with open(path, newline="", encoding="utf-8") as f: - reader = csv.DictReader(f) + with open(path, newline="", encoding="utf-8") as handle: + reader = csv.DictReader(handle) for row in reader: total_read += 1 file_count += 1 - key = (row.get("username", ""), row.get("repo", "")) - if key not in seen: - seen[key] = row + seen.setdefault(row_key(row), row) except Exception as exc: print(f" [warn] could not read {path}: {exc}") continue print(f" read {file_count:>5} rows from {os.path.basename(path)}") - unique_leads = list(seen.values()) - print(f"\n total read : {total_read}") - print(f" unique leads : {len(unique_leads)} " - f"(deduplicated {total_read - len(unique_leads)})\n") - - # ------------------------------------------------------------------ - # Score, label, sort - # ------------------------------------------------------------------ - for lead in unique_leads: - s = score_lead(lead) - lead["priority_score"] = s - lead["priority"] = priority_label(s) + merged_rows = list(seen.values()) + merged_rows.sort(key=lambda row: int(row.get("priority_score") or 0), reverse=True) - unique_leads.sort(key=lambda r: int(r["priority_score"]), reverse=True) - - # ------------------------------------------------------------------ - # Write output - # ------------------------------------------------------------------ - os.makedirs(os.path.dirname(output_path) if os.path.dirname(output_path) else ".", exist_ok=True) - - with open(output_path, "w", newline="", encoding="utf-8") as f: - writer = csv.DictWriter(f, fieldnames=OUTPUT_FIELDS, extrasaction="ignore") + parent = os.path.dirname(output_path) + if parent: + os.makedirs(parent, exist_ok=True) + with open(output_path, "w", newline="", encoding="utf-8") as handle: + writer = csv.DictWriter(handle, fieldnames=OUTPUT_FIELDS, extrasaction="ignore") writer.writeheader() - for lead in unique_leads: - writer.writerow({field: lead.get(field, "") for field in OUTPUT_FIELDS}) + for row in merged_rows: + writer.writerow({field: row.get(field, "") for field in OUTPUT_FIELDS}) - # ------------------------------------------------------------------ - # Summary - # ------------------------------------------------------------------ - p1 = sum(1 for l in unique_leads if l["priority"] == "P1") - p2 = sum(1 for l in unique_leads if l["priority"] == "P2") - p3 = sum(1 for l in unique_leads if l["priority"] == "P3") + p1 = sum(1 for row in merged_rows if row.get("priority") == "P1") + p2 = sum(1 for row in merged_rows if row.get("priority") == "P2") + p3 = sum(1 for row in merged_rows if row.get("priority") == "P3") - print(f"=== Summary ===") - print(f" Total leads : {len(unique_leads)}") - print(f" P1 (>= 8) : {p1}") - print(f" P2 (>= 5) : {p2}") - print(f" P3 (< 5) : {p3}") - print(f"\n Written to : {output_path}") + print(f"\n total read : {total_read}") + print(f" unique docs : {len(merged_rows)}") + print(f" deduplicated : {total_read - len(merged_rows)}") + print(f"\n=== Summary ===") + print(f" P1 : {p1}") + print(f" P2 : {p2}") + print(f" P3 : {p3}") + print(f"\n Written to : {output_path}") if __name__ == "__main__": diff --git a/queries.txt b/queries.txt index 9495456..ae2e8c3 100644 --- a/queries.txt +++ b/queries.txt @@ -1,44 +1,43 @@ -# TIER 1 — Direct Pain -"merge conflict" -"fix merge conflict" -"revert breaking change" -"api breaking change" -"port mismatch" -"synthesis error" -"linker script conflict" +# IEEE / language references +"IEEE 1800 SystemVerilog standard" +"IEEE SystemVerilog assertions tutorial" +"SVA property sequence implication tutorial" -# TIER 2 — Strong Signal -"fix function signature" -"broken import" -"fix broken import" -"dependency conflict" -"fix header" -"accidentally removed" -"forgot to update" -"out of sync" -"stale import" -"parameter mismatch" -"argument mismatch" -"wrong number of arguments" -"fix api contract" -"breaking api change" +# Assertion examples and rulebooks +"SystemVerilog assertion examples prior generated assertions" +"SVA checker library best practices" +"SystemVerilog coding standard rulebook assertions" +"formal verification assertion guideline pdf" -# TIER 3 — Moderate Signal -"fix build error" -"config conflict" -"version mismatch" -"fix linker error" -"regression introduced" -"conflicting changes" -"revert merge" -"fix driver conflict" -"fix register map" +# Protocol specifications +"AMBA AXI protocol specification pdf" +"APB protocol specification pdf" +"AHB protocol spec SystemVerilog" +"PCIe protocol verification assertions pdf" +"I2C protocol timing spec pdf" +"SPI protocol verification guide" +"UART protocol spec and assertions" +"USB protocol verification assertions" +"DDR controller protocol assertions" -# TIER 4 — Hardware/FPGA Deep Dive -"fix entity" -"fix module" -"signal width mismatch" -"fix port map" -"fix instantiation" -"synthesis failure" -"fix rtl" +# Design and verification specs +"project design spec SystemVerilog verification" +"microarchitecture design specification formal verification" +"block level verification plan SystemVerilog" +"interface design spec assertions" +"reset sequence verification specification" + +# Formal logs / debug / RCA / HIL +"formal verification log counterexample report" +"jaspergold proof log assertion failure" +"VC Formal counterexample log pdf" +"hardware in the loop correction report" +"HIL failure root cause analysis" +"assertion debug RCA report" +"postmortem protocol bug root cause SystemVerilog" +"coverage closure report formal verification" + +# Community and reference sources +"verificationacademy SystemVerilog assertions tutorial" +"accellera SystemVerilog assertions guide" +"github SystemVerilog assertion repository" diff --git a/run_crawl.sh b/run_crawl.sh index 20d39fb..6b0d84d 100755 --- a/run_crawl.sh +++ b/run_crawl.sh @@ -18,10 +18,10 @@ if [ ! -f "$SCRIPT_DIR/.env" ]; then echo "" echo " PC_IPS=192.168.1.101,192.168.1.102,..." echo " PC_USER=student" - echo " GITHUB_TOKENS=ghp_token1,ghp_token2,..." echo " QUERIES_FILE=./queries.txt" echo " RESULTS_DIR=./results" echo " OUTPUT_FILE=./final_leads.csv" + echo " LOCAL_DOC_DIRS=/abs/path/specs,/abs/path/logs" echo "" echo " Aborting." exit 1 diff --git a/run_local.sh b/run_local.sh index e3ca361..a156544 100755 --- a/run_local.sh +++ b/run_local.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Runs the crawler directly on this machine (no SSH, no coordinator). +# Runs the verification document crawler directly on this machine. # Output goes to /tmp/results_local_run.csv — supervisor.py merges it # into results_local.csv after each successful run. # @@ -19,11 +19,6 @@ else echo "[run_local.sh] WARNING: no .env found — relying on exported environment" fi -if [ -z "${GITHUB_TOKEN:-}" ]; then - echo "[run_local.sh] ERROR: GITHUB_TOKEN is not set. Aborting." - exit 1 -fi - QUERIES_FILE="${QUERIES_FILE:-$SCRIPT_DIR/queries.txt}" OUTPUT_TMP="/tmp/results_local_run.csv" diff --git a/supervisor.py b/supervisor.py index 5c0e90c..4032d1d 100644 --- a/supervisor.py +++ b/supervisor.py @@ -1,12 +1,12 @@ """ -supervisor.py — long-running watchdog for the GitHub crawler on the Pi. +supervisor.py — long-running watchdog for the verification document crawler. Lifecycle: 1. Launch run_local.sh as a subprocess. 2. Every 60 s check crawl.log — if the last line has not changed in 10 minutes, treat the crawler as stalled: kill it and restart. 3. After a clean exit, merge /tmp/results_local_run.csv into - results_local.csv (deduplicating on username+repo). + results_local.csv (deduplicating on query+url+local_path). 4. Sleep for SLEEP_BETWEEN seconds, then go back to step 1. Run with: @@ -82,17 +82,18 @@ def _last_log_line(): # --------------------------------------------------------------------------- def _load_seen_keys(path): - """Return set of (username, repo) tuples from an existing CSV.""" + """Return set of (query, url, local_path) tuples from an existing CSV.""" seen = set() if not os.path.exists(path): return seen try: with open(path, newline="", encoding="utf-8") as f: for row in csv.DictReader(f): - u = row.get("username", "").strip() - r = row.get("repo", "").strip() - if u and r: - seen.add((u, r)) + q = row.get("query", "").strip() + u = row.get("url", "").strip() + p = row.get("local_path", "").strip() + if q or u or p: + seen.add((q, u, p)) except Exception as exc: slog(f"[warn] could not read existing results: {exc}") return seen @@ -101,7 +102,7 @@ def _load_seen_keys(path): def merge_results(log_noop=False): """ Append rows from RESULTS_TMP into RESULTS_FINAL, skipping any - (username, repo) pair already present in RESULTS_FINAL. + (query, url, local_path) tuple already present in RESULTS_FINAL. Returns number of new rows written. """ if not os.path.exists(RESULTS_TMP): @@ -118,7 +119,11 @@ def merge_results(log_noop=False): reader = csv.DictReader(f) fieldnames = reader.fieldnames or [] for row in reader: - key = (row.get("username", "").strip(), row.get("repo", "").strip()) + key = ( + row.get("query", "").strip(), + row.get("url", "").strip(), + row.get("local_path", "").strip(), + ) if key not in seen: new_rows.append(row) seen.add(key)