Improve stable source quality summaries

Author: Pigbibi

.github/workflows/rss_source_pipeline.yml

@@ -88,6 +88,7 @@ jobs:
           python scripts/write_live_manifest.py \
             --base-dir . \
             --output data/live/source_manifest.json \
+            data/live/source_fetch_status.json \
             data/live/source_items.csv \
             data/live/source_events.csv \
             data/live/political_events.csv \

.github/workflows/source_event_pipeline.yml

@@ -71,13 +71,19 @@ jobs:
           cp data/output/source_event_pipeline/source_events.csv data/live/source_events.csv
           cp data/output/source_event_pipeline/source_events.csv data/live/political_events.csv
           cp data/output/source_event_pipeline/source_tracker.csv data/live/source_tracker.csv
+          MANIFEST_PATHS=(
+            data/live/source_items.csv
+            data/live/source_events.csv
+            data/live/political_events.csv
+            data/live/source_tracker.csv
+          )
+          if [ -f data/live/source_fetch_status.json ]; then
+            MANIFEST_PATHS+=(data/live/source_fetch_status.json)
+          fi
           python scripts/write_live_manifest.py \
             --base-dir . \
             --output data/live/source_manifest.json \
-            data/live/source_items.csv \
-            data/live/source_events.csv \
-            data/live/political_events.csv \
-            data/live/source_tracker.csv
+            "${MANIFEST_PATHS[@]}"
           git config user.name "github-actions[bot]"
           git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
           git add data/live/source_events.csv data/live/political_events.csv data/live/source_tracker.csv data/live/source_manifest.json

README.md

@@ -144,13 +144,14 @@ The price loader also accepts `as_of` instead of `date`.
 
 ## Live Pipeline Notes
 
-The RSS fetcher records per-feed health in `data/live/source_fetch_status.json` and keeps a small hash/row-count manifest in `data/live/source_manifest.json`. A single blocked or unavailable feed should not stop the rest of the official-source refresh.
+The RSS fetcher records per-feed health in `data/live/source_fetch_status.json` and keeps a small hash/row-count plus data-quality manifest in `data/live/source_manifest.json`. The manifest summarizes feed health, covered symbols, confidence counts, and event-type counts. A single blocked or unavailable feed should not stop the rest of the official-source refresh.
 
 If the RSS workflow downloads articles but `source_events.csv` is empty, the
 usual cause is deterministic alias coverage rather than a fetch failure: broad
-policy items often mention sectors such as grid infrastructure or crypto assets
-instead of company names. Alias updates should remain targeted, documented, and
-reviewable to avoid turning every broad policy article into a false positive.
+policy items often mention sectors such as grid infrastructure, HBM, foundry,
+AI servers, or crypto assets instead of company names. Alias updates should
+remain targeted, documented, and reviewable to avoid turning every broad policy
+article into a false positive.
 
 Local macOS Python installations can also fail HTTPS RSS fetches with a local CA
 certificate error. GitHub-hosted runners use a normal CA bundle; local operators

README.zh-CN.md

@@ -112,9 +112,9 @@ python -m pytest -q
 
 ## Live pipeline 说明
 
-RSS fetcher 会把每个源的成功/失败写入 `data/live/source_fetch_status.json`，并在 `data/live/source_manifest.json` 保留 live CSV 的 hash 和行数。单个源被屏蔽或临时不可用时，不应阻断其他官方源刷新。
+RSS fetcher 会把每个源的成功/失败写入 `data/live/source_fetch_status.json`，并在 `data/live/source_manifest.json` 保留 live CSV 的 hash、行数和数据质量摘要。manifest 会记录 feed 健康度、覆盖 symbol、置信度分布和事件类型分布。单个源被屏蔽或临时不可用时，不应阻断其他官方源刷新。
 
-如果 RSS workflow 能拉到 `source_items.csv`，但 `source_events.csv` 为空，通常不是新闻没跑，而是确定性 alias 覆盖不足：很多政策文件只写“grid infrastructure”“crypto assets”这类主题词，不直接写公司名。后续补 alias 要保持克制、可审计，避免把所有宽泛政策新闻都误映射成公司事件。
+如果 RSS workflow 能拉到 `source_items.csv`，但 `source_events.csv` 为空，通常不是新闻没跑，而是确定性 alias 覆盖不足：很多政策文件只写“grid infrastructure”“HBM”“foundry”“AI server”“crypto assets”这类主题词，不直接写公司名。后续补 alias 要保持克制、可审计，避免把所有宽泛政策新闻都误映射成公司事件。
 
 本机 macOS Python 也可能因为本地 CA 证书问题导致 HTTPS RSS 拉取失败；GitHub-hosted runner 使用正常 CA bundle。本地排查时可以先修 Python 证书，也可以直接从 GitHub Actions 下载 `source_items.csv` artifact 后验证抽取链路。
 

config/core_us_equity_aliases.csv

@@ -1,26 +1,26 @@
 symbol,name,aliases
 AAPL,Apple,Apple|AAPL|iPhone|iPad|Mac|Tim Cook
-AMD,Advanced Micro Devices,AMD|Advanced Micro Devices|Radeon|EPYC|Instinct
+AMD,Advanced Micro Devices,AMD|Advanced Micro Devices|Radeon|EPYC|Instinct|MI300|MI350
 AMZN,Amazon,Amazon|AMZN|AWS|Amazon Web Services|Project Kuiper
 AVGO,Broadcom,Broadcom|AVGO|VMware
 BA,Boeing,Boeing|BA|737 MAX|Starliner
 COIN,Coinbase,Coinbase|COIN|crypto exchange|crypto assets|digital assets|crypto asset transactions
 CRWD,CrowdStrike,CrowdStrike|CRWD|Falcon cybersecurity
 CVX,Chevron,Chevron|CVX|energy-related infrastructure|energy related infrastructure|large-scale energy|oil and gas
-DELL,Dell Technologies,Dell|Dell Technologies|DELL|AI server|PowerEdge
+DELL,Dell Technologies,Dell|Dell Technologies|DELL|AI server|PowerEdge|PowerScale|PowerStore|enterprise server|data center infrastructure
 ENPH,Enphase Energy,Enphase|ENPH|solar inverter
 F,Ford,Ford|F|F-150|Mustang Mach-E
 GE,GE Aerospace,GE Aerospace|General Electric|GE|jet engine
 GME,GameStop,GameStop|GME
 GOOGL,Alphabet,Alphabet|Google|GOOGL|GOOG|Gemini|Waymo
-INTC,Intel,Intel|INTC|foundry|CHIPS Act
+INTC,Intel,Intel|INTC|foundry|Intel Foundry|CHIPS Act|semiconductor manufacturing|advanced packaging|Gaudi
 IONQ,IonQ,IonQ|IONQ|quantum computing
 JPM,JPMorgan Chase,JPMorgan|JPMorgan Chase|JPM|Jamie Dimon
 LMT,Lockheed Martin,Lockheed Martin|LMT|F-35|missile defense
 META,Meta Platforms,Meta|META|Facebook|Instagram|WhatsApp|Llama
 MSFT,Microsoft,Microsoft|MSFT|Azure|OpenAI|Copilot
 MSTR,Strategy,Strategy|MicroStrategy|MSTR|Michael Saylor|Bitcoin treasury|crypto assets|Bitcoin
-MU,Micron Technology,Micron|MU|HBM|DRAM|NAND
+MU,Micron Technology,Micron|Micron Technology|MU|HBM|High Bandwidth Memory|DRAM|NAND|memory chip
 NEE,NextEra Energy,NextEra|NEE|renewable energy|grid infrastructure|energy-related infrastructure|energy related infrastructure|large-scale energy|renewable power
 NOC,Northrop Grumman,Northrop Grumman|NOC|B-21
 NVDA,NVIDIA,NVIDIA|Nvidia|NVDA|Blackwell|CUDA|GPU
@@ -37,5 +37,5 @@ SOFI,SoFi Technologies,SoFi|SOFI
 TSLA,Tesla,Tesla|TSLA|Elon Musk|Cybertruck|FSD|Optimus|xAI
 TSM,Taiwan Semiconductor,TSMC|Taiwan Semiconductor|TSM|foundry
 UNH,UnitedHealth Group,UnitedHealth|UNH|UnitedHealthcare
-VRT,Vertiv,Vertiv|VRT|data center power|liquid cooling|grid infrastructure|energy-related infrastructure|energy related infrastructure|large-scale energy|data center energy
+VRT,Vertiv,Vertiv|Vertiv Holdings|VRT|data center power|liquid cooling|grid infrastructure|energy-related infrastructure|energy related infrastructure|large-scale energy|data center energy|critical digital infrastructure|thermal management
 XOM,Exxon Mobil,Exxon|ExxonMobil|XOM|energy-related infrastructure|energy related infrastructure|large-scale energy|oil and gas

docs/source_ingestion_options.md

@@ -36,6 +36,7 @@ not part of this release. Reasons:
 - `scripts/fetch_rss_sources.py` fetches RSS/Atom feeds into `source_items.csv`.
 - `scripts/extract_source_mentions.py` converts `source_items.csv` into normalized event rows by deterministic alias matching.
 - `scripts/import_source_events.py` remains available when upstream records are already normalized.
+- `scripts/write_live_manifest.py` records hash, row-count, feed-health, covered-symbol, confidence, and event-type summaries for live artifacts.
 
 ## Operational Caveats
 

docs/source_registry.md

@@ -44,6 +44,26 @@ The stable release intentionally excludes X / Twitter, Truth Social, and
 Longbridge community/profile/following-list ingestion. They can be revisited
 later only if a stable official interface and clear operating policy exist.
 
+## Current Configured RSS Feeds
+
+`config/free_rss_feeds.csv` currently keeps the stable default set small:
+
+- White House presidential actions
+- SEC press releases
+- SEC speeches and statements
+- Federal Reserve press releases
+- Federal Reserve speeches and testimony
+
+`data/live/source_manifest.json` records source row counts, feed health, covered
+symbols, confidence counts, and event-type counts for every published live run.
+
+## Alias Coverage Notes
+
+`config/core_us_equity_aliases.csv` includes targeted aliases for the current AI
+and data-center names that often appear through products or infrastructure terms,
+including MU, INTC, AMD, DELL, and VRT. Alias updates should prefer company names,
+product names, or clear infrastructure phrases over very broad hype terms.
+
 ## Confidence Guide
 
 - `high`: official government source or direct filing with stable URL.

src/political_event_tracking_research/artifacts.py

@@ -21,11 +21,57 @@ def csv_row_count(path: str | Path) -> int:
         return sum(1 for _ in csv.DictReader(handle))
 
 
+def summarize_source_events(path: str | Path) -> dict[str, Any]:
+    symbols: set[str] = set()
+    confidence_counts: dict[str, int] = {}
+    event_type_counts: dict[str, int] = {}
+    with Path(path).open(newline="", encoding="utf-8") as handle:
+        for row in csv.DictReader(handle):
+            symbol = row.get("symbol", "").strip().upper()
+            if symbol:
+                symbols.add(symbol)
+            confidence = row.get("confidence", "").strip() or "unknown"
+            confidence_counts[confidence] = confidence_counts.get(confidence, 0) + 1
+            event_type = row.get("event_type", "").strip() or "unknown"
+            event_type_counts[event_type] = event_type_counts.get(event_type, 0) + 1
+    return {
+        "symbol_count": len(symbols),
+        "symbols": sorted(symbols),
+        "confidence_counts": dict(sorted(confidence_counts.items())),
+        "event_type_counts": dict(sorted(event_type_counts.items())),
+    }
+
+
+def summarize_feed_status(path: str | Path) -> dict[str, Any]:
+    payload = json.loads(Path(path).read_text(encoding="utf-8"))
+    feed_count = int(payload.get("feed_count") or 0)
+    successful_feed_count = int(payload.get("successful_feed_count") or 0)
+    failed_feed_count = int(payload.get("failed_feed_count") or 0)
+    return {
+        "feed_count": feed_count,
+        "successful_feed_count": successful_feed_count,
+        "failed_feed_count": failed_feed_count,
+        "feed_success_ratio": round(successful_feed_count / feed_count, 4) if feed_count else None,
+    }
+
+
+def build_data_quality_summary(paths: list[Path]) -> dict[str, Any]:
+    by_name = {path.name: path for path in paths}
+    summary: dict[str, Any] = {}
+    source_events_path = by_name.get("source_events.csv") or by_name.get("political_events.csv")
+    if source_events_path and source_events_path.exists():
+        summary["source_events"] = summarize_source_events(source_events_path)
+    feed_status_path = by_name.get("source_fetch_status.json")
+    if feed_status_path and feed_status_path.exists():
+        summary["feed_status"] = summarize_feed_status(feed_status_path)
+    return summary
+
+
 def build_live_manifest(paths: list[str | Path], *, base_dir: str | Path | None = None) -> dict[str, Any]:
     base = Path(base_dir).resolve() if base_dir else None
     artifacts: dict[str, Any] = {}
-    for raw_path in paths:
-        path = Path(raw_path)
+    resolved_paths = [Path(raw_path) for raw_path in paths]
+    for path in resolved_paths:
         key = path.name
         display_path = str(path)
         if base:
@@ -43,6 +89,7 @@ def build_live_manifest(paths: list[str | Path], *, base_dir: str | Path | None
         "generated_at": datetime.now(timezone.utc).isoformat(),
         "artifact_count": len(artifacts),
         "artifacts": artifacts,
+        "data_quality": build_data_quality_summary(resolved_paths),
     }
 
 

tests/test_artifacts.py

@@ -16,3 +16,29 @@ def test_build_live_manifest_records_hash_and_row_count(tmp_path: Path) -> None:
     assert item["path"] == "source_events.csv"
     assert item["row_count"] == 2
     assert len(item["sha256"]) == 64
+
+
+def test_build_live_manifest_includes_source_quality_summary(tmp_path: Path) -> None:
+    csv_path = tmp_path / "source_events.csv"
+    csv_path.write_text(
+        "\n".join(
+            [
+                "event_id,event_date,symbol,event_type,direction,confidence,source_url,notes",
+                "e1,2026-05-30,MU,public_mention,neutral,high,https://example.invalid/mu,MU mention",
+                "e2,2026-05-30,VRT,policy_capital,bullish,medium,https://example.invalid/vrt,VRT policy",
+            ]
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    status_path = tmp_path / "source_fetch_status.json"
+    status_path.write_text(
+        '{"feed_count": 2, "successful_feed_count": 1, "failed_feed_count": 1, "feeds": []}\n',
+        encoding="utf-8",
+    )
+
+    manifest = build_live_manifest([csv_path, status_path], base_dir=tmp_path)
+
+    assert manifest["data_quality"]["source_events"]["symbols"] == ["MU", "VRT"]
+    assert manifest["data_quality"]["source_events"]["confidence_counts"] == {"high": 1, "medium": 1}
+    assert manifest["data_quality"]["feed_status"]["feed_success_ratio"] == 0.5