Add PSCC source (Power Systems Computation Conference)

AGENTS.md

@@ -6,8 +6,8 @@ Guidance for coding agents (and humans) working in this repository. See
 ## What this is
 
 **ConfLens** — a NiceGUI web app that browses conference papers (ACL Anthology,
-EMNLP, NAACL, IJCAI, and OpenReview / ICLR · NeurIPS), classifies them against a
-theme with an LLM, discovers topics, and synthesises per-topic findings. It also
+EMNLP, NAACL, IJCAI, OpenReview / ICLR · NeurIPS, and PSCC), classifies them
+against a theme with an LLM, discovers topics, and synthesises per-topic findings. It also
 flags near-duplicate titles and offers a fully client-side results view
 (live confidence re-threshold, keyword search + highlight, sort/author facets,
 save/load a run). Python 3.13, managed with **uv**.
@@ -60,7 +60,7 @@ regexes over heavyweight parsers, small focused modules).
 |------|--------|
 | UI + exports | `app.py`, `pptx_export.py`, `bibtex.py` |
 | Orchestration | `pipeline.py` (`AnalysisConfig`, `run_analysis`) |
-| Sources | `sources.py` (registry + `make_source`; `IJCAISource`, `OpenReviewSource`), `scraper.py` (`AnthologyScraper` — also serves EMNLP/NAACL) |
+| Sources | `sources.py` (registry + `make_source`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`), `scraper.py` (`AnthologyScraper` — also serves EMNLP/NAACL) |
 | LLM providers | `llm.py` (`LLMClient`, `make_client`) |
 | Classify / topics | `classifier.py`, `topics.py` |
 | Near-duplicate detection | `dedup.py` (`annotate_duplicates`) |

README.md

@@ -18,9 +18,12 @@ A desktop-style web app (built with [NiceGUI](https://nicegui.io)) that:
    - the [ACL Anthology](https://aclanthology.org) (default; e.g. `acl-2026`),
    - **EMNLP** and **NAACL** proceedings (also on the ACL Anthology; e.g.
      `emnlp-2024`, `naacl-2024`),
-   - **IJCAI** accepted-paper pages (e.g. <https://2026.ijcai.org/accepted-papers/>), and
+   - **IJCAI** accepted-paper pages (e.g. <https://2026.ijcai.org/accepted-papers/>),
    - **OpenReview** venues — **ICLR / NeurIPS** and more — via the public JSON API,
-     by venue id (e.g. `ICLR.cc/2024/Conference`, `NeurIPS.cc/2024/Conference`).
+     by venue id (e.g. `ICLR.cc/2024/Conference`, `NeurIPS.cc/2024/Conference`), and
+   - **PSCC** (Power Systems Computation Conference) proceedings, by year
+     (e.g. `2024`) — titles + PDFs (abstracts aren't published, so classification
+     is title-based).
 
    The scraper is pluggable — adding another conference is one adapter in
    `conference_analyzer/sources.py`.
@@ -145,7 +148,7 @@ pluggable *LLM provider* (which model), with every expensive step cached on disk
 ```mermaid
 flowchart LR
     UI["NiceGUI UI<br/>app.py"] --> PIPE["pipeline.py"]
-    PIPE --> SRC["sources.py<br/>ACL · EMNLP · NAACL · IJCAI · OpenReview"]
+    PIPE --> SRC["sources.py<br/>ACL · EMNLP · NAACL · IJCAI · OpenReview · PSCC"]
     PIPE --> CLS["classifier.py"]
     PIPE --> TOP["topics.py<br/>model + summarize"]
     CLS --> LLM["llm.py<br/>Anthropic · OpenAI · LiteLLM"]
@@ -163,7 +166,7 @@ data-model and caching diagrams, plus extension points.
 
 | Stage | Module | Notes |
 |-------|--------|-------|
-| Sources | `conflens/sources.py` | Pluggable adapters (ACL Anthology, EMNLP, NAACL, IJCAI, OpenReview) behind one interface; registry + factory. |
+| Sources | `conflens/sources.py` | Pluggable adapters (ACL Anthology, EMNLP, NAACL, IJCAI, OpenReview, PSCC) behind one interface; registry + factory. |
 | Scrape listing | `conflens/scraper.py` | ACL Anthology adapter: parses the event page; abstracts + authors fetched per paper and cached. |
 | Near-duplicates | `conflens/dedup.py` | Flags near-identical titles (union-find + `difflib`); dependency-free. |
 | Classify | `conflens/classifier.py` | Batched, structured-output calls; relevance + confidence + a one-line reason per paper (cached). |
@@ -176,17 +179,17 @@ data-model and caching diagrams, plus extension points.
 ## Configuration (in the UI)
 
 - **Source** — `ACL Anthology`, `EMNLP (ACL Anthology)`, `NAACL (ACL Anthology)`,
-  `IJCAI`, or `OpenReview (ICLR / NeurIPS)`. Switching prefills the base URL and
-  target below and relabels them. (ACL Anthology, EMNLP and NAACL share one
-  adapter — any of them accepts any Anthology event slug, e.g. `acl-2024`,
+  `IJCAI`, `OpenReview (ICLR / NeurIPS)`, or `PSCC`. Switching prefills the base
+  URL and target below and relabels them. (ACL Anthology, EMNLP and NAACL share
+  one adapter — any of them accepts any Anthology event slug, e.g. `acl-2024`,
   `emnlp-2023`, `naacl-2024`.)
 - **Base URL** — the site/API root (e.g. `https://aclanthology.org`,
   `https://2026.ijcai.org`, `https://api2.openreview.net`); change it to point at
   a mirror, another year, or the v1 API host (`https://api.openreview.net`).
 - **Event / target** — for ACL, a slug (`acl-2024`, `emnlp-2023`, …) or full
   event URL; for IJCAI, the accepted-papers path or full URL; for OpenReview, the
   **venue id** (`ICLR.cc/2024/Conference`, `NeurIPS.cc/2024/Conference`) or a
-  venue group URL.
+  venue group URL; for PSCC, a **year** (`2024`, `2022`, …) or a full listing URL.
 
   > **OpenReview auth** — recent (API v2) venues challenge anonymous requests
   > from some networks. If a run returns an auth error, set `OPENREVIEW_TOKEN`,

conflens/pipeline.py

@@ -93,6 +93,10 @@ def run_analysis(
                     " Check the venue id (e.g. 'ICLR.cc/2024/Conference' or "
                     "'NeurIPS.cc/2024/Conference') — decisions may not be posted yet."
                 ),
+                "pscc": (
+                    " PSCC is biennial — try a year that was held, e.g. '2024' or "
+                    "'2022'."
+                ),
             }
             hint = hints.get(cfg.source, "")
             progress.set("listing", "No papers found on that page." + hint, 1.0)

conflens/sources.py

@@ -18,6 +18,9 @@
 * **openreview** — OpenReview venues (ICLR, NeurIPS, …) via the public JSON API;
   accepted papers are fetched by venue id (e.g. ``ICLR.cc/2024/Conference``),
   abstracts/authors/keywords/PDFs all inline.
+* **pscc** — Power Systems Computation Conference; a per-year HTML fragment from
+  the papers-repository endpoint gives titles, authors and PDF links (abstracts
+  aren't published on the site, so classification is title-based).
 
 Adding another conference is a matter of writing one more adapter and
 registering it in :data:`SOURCES`.
@@ -428,6 +431,131 @@ def enrich_abstracts(
         return papers
 
 
+# ---------------------------------------------------------------------------
+# PSCC source (Power Systems Computation Conference)
+# ---------------------------------------------------------------------------
+# The PSCC papers repository is served by a small PHP endpoint that returns an
+# HTML fragment of <div class='paper'> blocks for a given year. Titles, authors
+# and a PDF link are inline; abstracts live only inside the PDFs, so papers are
+# classified on their titles (the classifier already handles an empty abstract).
+_PSCC_BLOCK = "<div class='paper'>"
+_PSCC_TITLE = re.compile(r"<p class='title'>(.*?)</p>", re.IGNORECASE | re.DOTALL)
+_PSCC_AUTHORS = re.compile(r"<p class='authors'>(.*?)</p>", re.IGNORECASE | re.DOTALL)
+_PSCC_PDF = re.compile(r"href='(repo/papers/[^']+\.pdf)'", re.IGNORECASE)
+_PSCC_PID = re.compile(r"([0-9]{4}_[0-9]+)\.pdf", re.IGNORECASE)
+
+
+class PSCCSource:
+    """Fetch a PSCC edition's papers from the papers-repository endpoint.
+
+    ``target`` is a conference year (e.g. ``2024``) or a full listing URL. Only
+    titles, authors and PDF links are available on the site; abstracts are not,
+    so downstream classification is title-based.
+    """
+
+    name = "pscc"
+
+    def __init__(
+        self,
+        base_url: str = "https://pscc-central.epfl.ch",
+        cache_dir: Optional[str] = None,
+        timeout: int = 120,
+    ) -> None:
+        self.base_url = (base_url or "https://pscc-central.epfl.ch").rstrip("/")
+        self.timeout = timeout
+        self.cache_dir = cache_dir or os.path.join(
+            os.path.expanduser("~"), ".cache", "conflens"
+        )
+        os.makedirs(self.cache_dir, exist_ok=True)
+
+    @staticmethod
+    def _year(target: str) -> str:
+        m = re.search(r"(19|20)\d{2}", target or "")
+        return m.group(0) if m else (target or "").strip()
+
+    def _api_url(self, year: str) -> str:
+        return f"{self.base_url}/repo/make_table.php?authors=&year={year}&title="
+
+    def resolve_url(self, target: str) -> str:
+        t = (target or "").strip()
+        if t.startswith("http://") or t.startswith("https://"):
+            return t
+        return f"{self.base_url}/papers-repo"
+
+    def _cache_path(self, year: str) -> str:
+        digest = hashlib.sha1(f"{self.base_url}|{year}".encode("utf-8")).hexdigest()[:16]
+        return os.path.join(self.cache_dir, f"pscc_{digest}.json")
+
+    def list_papers(self, target: str, force_refresh: bool = False) -> list[Paper]:
+        year = self._year(target)
+        cache = self._cache_path(year)
+        if not force_refresh and os.path.exists(cache):
+            try:
+                with open(cache, "r", encoding="utf-8") as fh:
+                    data = json.load(fh)
+                return [Paper(**p) for p in data.get("papers", [])]
+            except (OSError, json.JSONDecodeError, TypeError):
+                pass
+
+        t = (target or "").strip()
+        url = t if t.startswith(("http://", "https://")) else self._api_url(year)
+        page = _robust_get(url, self.timeout, headers={"User-Agent": _BROWSER_UA})
+        papers = self.parse_papers(page, year)
+        try:
+            with open(cache, "w", encoding="utf-8") as fh:
+                json.dump({"year": year, "papers": [p.to_dict() for p in papers]}, fh)
+        except OSError:
+            pass
+        return papers
+
+    def parse_papers(self, page: str, year: str = "") -> list[Paper]:
+        """Parse a PSCC make_table.php fragment into papers (no network)."""
+        papers: list[Paper] = []
+        seen: set[str] = set()
+        for block in page.split(_PSCC_BLOCK)[1:]:
+            title_m = _PSCC_TITLE.search(block)
+            if not title_m:
+                continue
+            title = _clean(title_m.group(1))
+            if not title:
+                continue
+            pdf_m = _PSCC_PDF.search(block)
+            pdf_rel = pdf_m.group(1) if pdf_m else ""
+            pid_m = _PSCC_PID.search(pdf_rel)
+            pid = pid_m.group(1) if pid_m else f"{year or '0000'}-{len(papers) + 1}"
+            paper_id = f"pscc-{pid}"
+            if paper_id in seen:
+                continue
+            seen.add(paper_id)
+            authors_m = _PSCC_AUTHORS.search(block)
+            authors: list[str] = []
+            if authors_m:
+                authors = [a for a in (_clean(x) for x in authors_m.group(1).split(",")) if a]
+            pdf_url = f"{self.base_url}/{pdf_rel}" if pdf_rel else ""
+            papers.append(
+                Paper(
+                    paper_id=paper_id,
+                    title=title,
+                    url=pdf_url or f"{self.base_url}/papers-repo",
+                    pdf_url=pdf_url,
+                    authors=authors,
+                    abstract="",  # abstracts aren't published on the PSCC site
+                )
+            )
+        return papers
+
+    def enrich_abstracts(
+        self,
+        papers: list[Paper],
+        progress: Optional[Callable[[int, int], None]] = None,
+        force_refresh: bool = False,
+    ) -> list[Paper]:
+        # No abstract source on the site; titles + authors arrive with the listing.
+        if progress:
+            progress(len(papers), len(papers))
+        return papers
+
+
 # ---------------------------------------------------------------------------
 # Registry
 # ---------------------------------------------------------------------------
@@ -467,6 +595,13 @@ def enrich_abstracts(
         "base_label": "OpenReview API base",
         "target_label": "Venue ID (e.g. ICLR.cc/2024/Conference, NeurIPS.cc/2024/Conference)",
     },
+    "pscc": {
+        "label": "PSCC (Power Systems Computation Conf.)",
+        "base": "https://pscc-central.epfl.ch",
+        "target": "2024",
+        "base_label": "PSCC base URL",
+        "target_label": "Conference year (e.g. 2024, 2022) or full listing URL",
+    },
 }
 
 
@@ -480,4 +615,6 @@ def make_source(source: str, base_url: str, cache_dir: Optional[str] = None):
         return IJCAISource(base_url=base_url, cache_dir=cache_dir)
     if source == "openreview":
         return OpenReviewSource(base_url=base_url, cache_dir=cache_dir)
+    if source == "pscc":
+        return PSCCSource(base_url=base_url, cache_dir=cache_dir)
     raise ValueError(f"Unknown source: {source}")

docs/ARCHITECTURE.md

@@ -23,6 +23,7 @@ flowchart TB
       REG --> ACL["AnthologyScraper<br/>scraper.py<br/>(ACL · EMNLP · NAACL)"]
       REG --> IJ["IJCAISource"]
       REG --> OR["OpenReviewSource<br/>(ICLR · NeurIPS)"]
+      REG --> PS["PSCCSource"]
     end
 
     subgraph llm["LLM providers (llm.py · make_client)"]
@@ -47,7 +48,8 @@ flowchart TB
 
 EMNLP and NAACL are served by the same `AnthologyScraper` as the ACL Anthology
 (different default event); OpenReview talks to the public JSON API instead of
-scraping HTML.
+scraping HTML; PSCC reads a per-year HTML fragment from its papers-repository
+endpoint (titles + PDFs only — no abstracts, so classification is title-based).
 
 ## Pipeline stages
 
@@ -109,7 +111,7 @@ sequenceDiagram
 | `cli.py` | Console entry point (`conflens`); `--clear-cache`, `--host/--port`; loads `.env`.                                                                                                                                 |
 | `app.py` | NiceGUI UI: configuration form, input validation, progress, and the interactive results view (ECharts chart, per-topic findings + papers, live re-threshold / search / sort / facet, save + load a run), exports. |
 | `pipeline.py` | `AnalysisConfig` + `run_analysis()` orchestrating the stages with a `Progress` object (supports cooperative cancel).                                                                                              |
-| `sources.py` | Source interface + registry + `make_source()`; `IJCAISource`, `OpenReviewSource`, shared `_robust_get`.                                                                                                           |
+| `sources.py` | Source interface + registry + `make_source()`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`, shared `_robust_get`.                                                                                             |
 | `scraper.py` | `AnthologyScraper` (ACL Anthology adapter, also serving EMNLP / NAACL) + shared HTML helpers.                                                                                                                     |
 | `dedup.py` | `annotate_duplicates()` — dependency-free near-duplicate-title clustering (union-find + `difflib`).                                                                                                               |
 | `classifier.py` | Batched, structured-output relevance classification with on-disk cache.                                                                                                                                           |

tests/test_pscc_source.py

@@ -0,0 +1,66 @@
+from conflens.sources import SOURCES, PSCCSource, make_source
+
+# Two paper blocks in the shape returned by PSCC's make_table.php (single-quoted
+# attributes, a nested hidden citation div, entity-encoded characters).
+FRAGMENT = (
+    "Papers returned: 2."
+    "<div class='paper'><p class='title'>A Bayesian Approach &amp; Wind Farms</p>"
+    "<p class='authors'>Jos&eacute; Pessanha, Victor Almeida, Albert Melo</p>"
+    "<p class='year'>2024&nbsp;|&nbsp;<a href='repo/papers/2024/2024_965.pdf'>PDF</a>&nbsp;|&nbsp;"
+    "<a href='#' onclick='show_citation(\"p_repo/papers/2024/2024_965.pdf\");return false;'>Citation</a></p>"
+    "<div class='citation' id='p_repo/papers/2024/2024_965.pdf' style='display: none;'>"
+    "Jos&eacute; Pessanha et al., A Bayesian Approach, PSCC 2024.</div></div>"
+    "<div class='paper'><p class='title'>Synthetic Distribution Systems</p>"
+    "<p class='authors'>Henrique Caetano</p>"
+    "<p class='year'>2024&nbsp;|&nbsp;<a href='repo/papers/2024/2024_421.pdf'>PDF</a></p></div>"
+)
+
+
+def test_pscc_registered_with_all_keys():
+    assert "pscc" in SOURCES
+    entry = SOURCES["pscc"]
+    for key in ("label", "base", "target", "base_label", "target_label"):
+        assert entry.get(key), f"missing {key}"
+
+
+def test_make_source_returns_pscc_adapter(tmp_path):
+    src = make_source("pscc", base_url="https://pscc-central.epfl.ch", cache_dir=str(tmp_path))
+    assert isinstance(src, PSCCSource)
+
+
+def test_parse_papers(tmp_path):
+    src = PSCCSource(cache_dir=str(tmp_path))
+    papers = src.parse_papers(FRAGMENT, "2024")
+    assert [p.paper_id for p in papers] == ["pscc-2024_965", "pscc-2024_421"]
+
+    p0 = papers[0]
+    assert p0.title == "A Bayesian Approach & Wind Farms"          # entity unescaped
+    assert p0.authors == ["José Pessanha", "Victor Almeida", "Albert Melo"]
+    assert p0.pdf_url == "https://pscc-central.epfl.ch/repo/papers/2024/2024_965.pdf"
+    assert p0.url == p0.pdf_url                                     # title links to the PDF
+    assert p0.abstract == ""                                       # no abstract on the site
+    assert papers[1].authors == ["Henrique Caetano"]
+
+
+def test_year_and_url_helpers(tmp_path):
+    src = PSCCSource(cache_dir=str(tmp_path))
+    assert src._year("2024") == "2024"
+    assert src._year("23rd PSCC 2022") == "2022"
+    assert src._api_url("2024") == (
+        "https://pscc-central.epfl.ch/repo/make_table.php?authors=&year=2024&title="
+    )
+    # a full URL passes through resolve_url; a year maps to the human repo page
+    assert src.resolve_url("2024") == "https://pscc-central.epfl.ch/papers-repo"
+    full = "https://pscc-central.epfl.ch/repo/make_table.php?authors=&year=2022&title="
+    assert src.resolve_url(full) == full
+
+
+def test_parse_skips_untitled_and_dedupes(tmp_path):
+    src = PSCCSource(cache_dir=str(tmp_path))
+    dupe = FRAGMENT + (
+        "<div class='paper'><p class='title'>A Bayesian Approach &amp; Wind Farms</p>"
+        "<p class='year'><a href='repo/papers/2024/2024_965.pdf'>PDF</a></p></div>"
+        "<div class='paper'><p class='authors'>No Title Here</p></div>"
+    )
+    papers = src.parse_papers(dupe, "2024")
+    assert [p.paper_id for p in papers] == ["pscc-2024_965", "pscc-2024_421"]  # dup + untitled dropped