Add ISGT Europe source via the open DBLP API

AGENTS.md

@@ -6,8 +6,9 @@ Guidance for coding agents (and humans) working in this repository. See
 ## What this is
 
 **ConfLens** — a NiceGUI web app that browses conference papers (ACL Anthology,
-EMNLP, NAACL, IJCAI, OpenReview / ICLR · NeurIPS, and PSCC), classifies them
-against a theme with an LLM, discovers topics, and synthesises per-topic findings. It also
+EMNLP, NAACL, IJCAI, OpenReview / ICLR · NeurIPS, PSCC, and DBLP-indexed venues
+such as ISGT Europe), classifies them against a theme with an LLM, discovers
+topics, and synthesises per-topic findings. It also
 flags near-duplicate titles and offers a fully client-side results view
 (live confidence re-threshold, keyword search + highlight, sort/author facets,
 save/load a run). Python 3.13, managed with **uv**.
@@ -60,7 +61,7 @@ regexes over heavyweight parsers, small focused modules).
 |------|--------|
 | UI + exports | `app.py`, `pptx_export.py`, `bibtex.py` |
 | Orchestration | `pipeline.py` (`AnalysisConfig`, `run_analysis`) |
-| Sources | `sources.py` (registry + `make_source`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`), `scraper.py` (`AnthologyScraper` — also serves EMNLP/NAACL) |
+| Sources | `sources.py` (registry + `make_source`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`, `DBLPSource`), `scraper.py` (`AnthologyScraper` — also serves EMNLP/NAACL) |
 | LLM providers | `llm.py` (`LLMClient`, `make_client`) |
 | Classify / topics | `classifier.py`, `topics.py` |
 | Near-duplicate detection | `dedup.py` (`annotate_duplicates`) |

README.md

@@ -23,7 +23,10 @@ A desktop-style web app (built with [NiceGUI](https://nicegui.io)) that:
      by venue id (e.g. `ICLR.cc/2024/Conference`, `NeurIPS.cc/2024/Conference`), and
    - **PSCC** (Power Systems Computation Conference) proceedings, by year
      (e.g. `2024`) — titles + PDFs (abstracts aren't published, so classification
-     is title-based).
+     is title-based), and
+   - **ISGT Europe** (IEEE PES) via the open **DBLP** index, by venue + year
+     (e.g. `isgteurope 2024`) — titles + DOI links, title-based (the generic
+     DBLP adapter works for any DBLP-indexed conference).
 
    The scraper is pluggable — adding another conference is one adapter in
    `conference_analyzer/sources.py`.
@@ -148,7 +151,7 @@ pluggable *LLM provider* (which model), with every expensive step cached on disk
 ```mermaid
 flowchart LR
     UI["NiceGUI UI<br/>app.py"] --> PIPE["pipeline.py"]
-    PIPE --> SRC["sources.py<br/>ACL · EMNLP · NAACL · IJCAI · OpenReview · PSCC"]
+    PIPE --> SRC["sources.py<br/>ACL · EMNLP · NAACL · IJCAI · OpenReview · PSCC · ISGT Europe"]
     PIPE --> CLS["classifier.py"]
     PIPE --> TOP["topics.py<br/>model + summarize"]
     CLS --> LLM["llm.py<br/>Anthropic · OpenAI · LiteLLM"]
@@ -166,7 +169,7 @@ data-model and caching diagrams, plus extension points.
 
 | Stage | Module | Notes |
 |-------|--------|-------|
-| Sources | `conflens/sources.py` | Pluggable adapters (ACL Anthology, EMNLP, NAACL, IJCAI, OpenReview, PSCC) behind one interface; registry + factory. |
+| Sources | `conflens/sources.py` | Pluggable adapters (ACL Anthology, EMNLP, NAACL, IJCAI, OpenReview, PSCC, DBLP/ISGT Europe) behind one interface; registry + factory. |
 | Scrape listing | `conflens/scraper.py` | ACL Anthology adapter: parses the event page; abstracts + authors fetched per paper and cached. |
 | Near-duplicates | `conflens/dedup.py` | Flags near-identical titles (union-find + `difflib`); dependency-free. |
 | Classify | `conflens/classifier.py` | Batched, structured-output calls; relevance + confidence + a one-line reason per paper (cached). |
@@ -179,17 +182,25 @@ data-model and caching diagrams, plus extension points.
 ## Configuration (in the UI)
 
 - **Source** — `ACL Anthology`, `EMNLP (ACL Anthology)`, `NAACL (ACL Anthology)`,
-  `IJCAI`, `OpenReview (ICLR / NeurIPS)`, or `PSCC`. Switching prefills the base
-  URL and target below and relabels them. (ACL Anthology, EMNLP and NAACL share
-  one adapter — any of them accepts any Anthology event slug, e.g. `acl-2024`,
-  `emnlp-2023`, `naacl-2024`.)
+  `IJCAI`, `OpenReview (ICLR / NeurIPS)`, `PSCC`, or `ISGT Europe (via DBLP)`.
+  Switching prefills the base URL and target below and relabels them. (ACL
+  Anthology, EMNLP and NAACL share one adapter — any of them accepts any
+  Anthology event slug, e.g. `acl-2024`, `emnlp-2023`, `naacl-2024`.)
 - **Base URL** — the site/API root (e.g. `https://aclanthology.org`,
   `https://2026.ijcai.org`, `https://api2.openreview.net`); change it to point at
   a mirror, another year, or the v1 API host (`https://api.openreview.net`).
 - **Event / target** — for ACL, a slug (`acl-2024`, `emnlp-2023`, …) or full
   event URL; for IJCAI, the accepted-papers path or full URL; for OpenReview, the
   **venue id** (`ICLR.cc/2024/Conference`, `NeurIPS.cc/2024/Conference`) or a
-  venue group URL; for PSCC, a **year** (`2024`, `2022`, …) or a full listing URL.
+  venue group URL; for PSCC, a **year** (`2024`, `2022`, …) or a full listing URL;
+  for ISGT Europe, a DBLP **venue + year** (`isgteurope 2024`) or proceedings key.
+
+  > **Title-based sources** — PSCC and DBLP-backed venues (ISGT Europe) don't
+  > expose abstracts, so classification runs on titles alone (still useful, just
+  > coarser). The DBLP adapter is generic: any DBLP-indexed conference works by
+  > passing `<venue> <year>`. IEEE-Xplore-only venues that DBLP doesn't index
+  > (e.g. **PowerTech**, **PES General Meeting**) can't be added without an IEEE
+  > Xplore API key.
 
   > **OpenReview auth** — recent (API v2) venues challenge anonymous requests
   > from some networks. If a run returns an auth error, set `OPENREVIEW_TOKEN`,

conflens/pipeline.py

@@ -97,6 +97,10 @@ def run_analysis(
                     " PSCC is biennial — try a year that was held, e.g. '2024' or "
                     "'2022'."
                 ),
+                "isgteurope": (
+                    " Check the venue + year (e.g. 'isgteurope 2024') — DBLP may "
+                    "not have indexed that edition yet."
+                ),
             }
             hint = hints.get(cfg.source, "")
             progress.set("listing", "No papers found on that page." + hint, 1.0)

conflens/sources.py

@@ -21,6 +21,10 @@
 * **pscc** — Power Systems Computation Conference; a per-year HTML fragment from
   the papers-repository endpoint gives titles, authors and PDF links (abstracts
   aren't published on the site, so classification is title-based).
+* **isgteurope** — IEEE PES ISGT Europe, via the public DBLP search API (many
+  IEEE-Xplore-only venues are indexed there). Titles, authors and a DOI link per
+  paper; no abstracts, so classification is title-based. The generic
+  :class:`DBLPSource` behind it works for any DBLP-indexed conference.
 
 Adding another conference is a matter of writing one more adapter and
 registering it in :data:`SOURCES`.
@@ -81,7 +85,8 @@ def _robust_get(url: str, timeout: int = 120, headers: Optional[dict] = None) ->
             if e.code not in _RETRY_STATUS:
                 raise RuntimeError(f"Failed to fetch {url}: HTTP {e.code} {e.reason}") from e
             time.sleep(1.5 * (attempt + 1))
-        except (urllib.error.URLError, TimeoutError) as e:
+        except (urllib.error.URLError, TimeoutError, ConnectionError) as e:
+            # ConnectionError covers a peer reset mid-response (e.g. rate-limiting).
             last_err = e
             time.sleep(1.5 * (attempt + 1))
     if partial_best:
@@ -556,6 +561,165 @@ def enrich_abstracts(
         return papers
 
 
+# ---------------------------------------------------------------------------
+# DBLP source (open metadata index — e.g. IEEE PES ISGT Europe)
+# ---------------------------------------------------------------------------
+# DBLP indexes many conferences that are otherwise only on IEEE Xplore. Its
+# public search API returns, per proceedings, each paper's title, authors and a
+# link (usually a DOI) — but no abstract, so classification is title-based.
+_DBLP_DISAMBIG = re.compile(r"\s+\d{4}$")   # DBLP appends "0001"-style suffixes
+_DBLP_TRAIL_DOT = re.compile(r"\.\s*$")
+
+
+class DBLPSource:
+    """Fetch a conference edition's papers from the public DBLP search API.
+
+    ``target`` is a ``"venue year"`` pair (e.g. ``"isgteurope 2024"``), a DBLP
+    proceedings key (``conf/isgteurope/isgteurope2024``) or a DBLP URL. Only
+    titles, authors and a link (usually the DOI) are available — abstracts are
+    not, so downstream classification is title-based.
+    """
+
+    name = "dblp"
+    _PAGE = 100  # DBLP's public search API caps hits-per-request at 100
+
+    def __init__(
+        self,
+        base_url: str = "https://dblp.org",
+        cache_dir: Optional[str] = None,
+        timeout: int = 120,
+    ) -> None:
+        self.base_url = (base_url or "https://dblp.org").rstrip("/")
+        self.timeout = timeout
+        self.cache_dir = cache_dir or os.path.join(
+            os.path.expanduser("~"), ".cache", "conflens"
+        )
+        os.makedirs(self.cache_dir, exist_ok=True)
+
+    @staticmethod
+    def _proc_key(target: str) -> str:
+        """Normalise ``target`` to a DBLP proceedings key like ``conf/x/x2024``."""
+        t = (target or "").strip()
+        m = re.search(r"(conf/[^\s?#]+?)(?:\.html|\.bht)?(?:[?#].*)?$", t)
+        if m:
+            return m.group(1)
+        parts = t.split()
+        venue = parts[0].strip("/") if parts else ""
+        year = ""
+        for p in parts[1:]:
+            ym = re.search(r"(19|20)\d{2}", p)
+            if ym:
+                year = ym.group(0)
+                break
+        return f"conf/{venue}/{venue}{year}"
+
+    def resolve_url(self, target: str) -> str:
+        return f"{self.base_url}/db/{self._proc_key(target)}.html"
+
+    def _cache_path(self, key: str) -> str:
+        digest = hashlib.sha1(key.encode("utf-8")).hexdigest()[:16]
+        return os.path.join(self.cache_dir, f"dblp_{digest}.json")
+
+    def list_papers(self, target: str, force_refresh: bool = False) -> list[Paper]:
+        key = self._proc_key(target)
+        cache = self._cache_path(key)
+        if not force_refresh and os.path.exists(cache):
+            try:
+                with open(cache, "r", encoding="utf-8") as fh:
+                    data = json.load(fh)
+                return [Paper(**p) for p in data.get("papers", [])]
+            except (OSError, json.JSONDecodeError, TypeError):
+                pass
+
+        hits = self._fetch_hits(key)
+        papers = self.parse_hits(hits)
+        try:
+            with open(cache, "w", encoding="utf-8") as fh:
+                json.dump({"key": key, "papers": [p.to_dict() for p in papers]}, fh)
+        except OSError:
+            pass
+        return papers
+
+    def _fetch_hits(self, proc_key: str) -> list[dict]:
+        """Page through every DBLP record whose TOC is this proceedings."""
+        toc = f"db/{proc_key}.bht"
+        hits: list[dict] = []
+        first = 0
+        while True:
+            if first:  # be polite to DBLP between pages to avoid throttling
+                time.sleep(0.7)
+            q = urllib.parse.urlencode(
+                {"q": f"toc:{toc}:", "format": "json", "h": self._PAGE, "f": first, "c": 0}
+            )
+            raw = _robust_get(
+                f"{self.base_url}/search/publ/api?{q}",
+                self.timeout,
+                headers={"User-Agent": _BROWSER_UA, "Accept": "application/json"},
+            )
+            try:
+                result = json.loads(raw).get("result", {})
+            except json.JSONDecodeError:
+                break
+            batch = result.get("hits", {}).get("hit", []) or []
+            hits.extend(batch)
+            total = int(result.get("hits", {}).get("@total", len(hits)) or 0)
+            first += self._PAGE
+            if first >= total or not batch:
+                break
+        return hits
+
+    def parse_hits(self, hits: list[dict]) -> list[Paper]:
+        """Turn DBLP search hits into papers (no network)."""
+        papers: list[Paper] = []
+        seen: set[str] = set()
+        for hit in hits:
+            info = hit.get("info", {}) or {}
+            title = _DBLP_TRAIL_DOT.sub("", _clean(str(info.get("title", "") or "")))
+            if not title:
+                continue
+            key = info.get("key") or hit.get("@id") or f"{len(papers) + 1}"
+            paper_id = "dblp-" + str(key).replace("/", "-")
+            if paper_id in seen:
+                continue
+            seen.add(paper_id)
+
+            raw_authors = (info.get("authors", {}) or {}).get("author", [])
+            if isinstance(raw_authors, dict):
+                raw_authors = [raw_authors]
+            authors = []
+            for a in raw_authors:
+                name = a.get("text", "") if isinstance(a, dict) else str(a)
+                name = _DBLP_DISAMBIG.sub("", _clean(name))
+                if name:
+                    authors.append(name)
+
+            ee = info.get("ee") or info.get("url") or ""
+            if isinstance(ee, list):
+                ee = ee[0] if ee else ""
+            papers.append(
+                Paper(
+                    paper_id=paper_id,
+                    title=title,
+                    url=ee,          # DOI / landing page (full text is usually paywalled)
+                    pdf_url="",      # no open PDF via DBLP
+                    authors=authors,
+                    abstract="",     # DBLP has no abstracts → title-based classification
+                )
+            )
+        return papers
+
+    def enrich_abstracts(
+        self,
+        papers: list[Paper],
+        progress: Optional[Callable[[int, int], None]] = None,
+        force_refresh: bool = False,
+    ) -> list[Paper]:
+        # DBLP exposes no abstracts; titles + authors arrive with the listing.
+        if progress:
+            progress(len(papers), len(papers))
+        return papers
+
+
 # ---------------------------------------------------------------------------
 # Registry
 # ---------------------------------------------------------------------------
@@ -602,6 +766,13 @@ def enrich_abstracts(
         "base_label": "PSCC base URL",
         "target_label": "Conference year (e.g. 2024, 2022) or full listing URL",
     },
+    "isgteurope": {
+        "label": "ISGT Europe (IEEE PES, via DBLP)",
+        "base": "https://dblp.org",
+        "target": "isgteurope 2024",
+        "base_label": "DBLP base URL",
+        "target_label": "DBLP venue + year (e.g. isgteurope 2024) or proceedings key",
+    },
 }
 
 
@@ -617,4 +788,8 @@ def make_source(source: str, base_url: str, cache_dir: Optional[str] = None):
         return OpenReviewSource(base_url=base_url, cache_dir=cache_dir)
     if source == "pscc":
         return PSCCSource(base_url=base_url, cache_dir=cache_dir)
+    if source in ("isgteurope", "dblp"):
+        # DBLP-backed; ISGT Europe (IEEE PES) is the registered venue, but any
+        # DBLP-indexed conference works by passing "<venue> <year>" as the target.
+        return DBLPSource(base_url=base_url, cache_dir=cache_dir)
     raise ValueError(f"Unknown source: {source}")

docs/ARCHITECTURE.md

@@ -24,6 +24,7 @@ flowchart TB
       REG --> IJ["IJCAISource"]
       REG --> OR["OpenReviewSource<br/>(ICLR · NeurIPS)"]
       REG --> PS["PSCCSource"]
+      REG --> DB["DBLPSource<br/>(ISGT Europe · …)"]
     end
 
     subgraph llm["LLM providers (llm.py · make_client)"]
@@ -49,7 +50,10 @@ flowchart TB
 EMNLP and NAACL are served by the same `AnthologyScraper` as the ACL Anthology
 (different default event); OpenReview talks to the public JSON API instead of
 scraping HTML; PSCC reads a per-year HTML fragment from its papers-repository
-endpoint (titles + PDFs only — no abstracts, so classification is title-based).
+endpoint (titles + PDFs only — no abstracts, so classification is title-based);
+`DBLPSource` uses the open DBLP search API to reach IEEE-Xplore-only venues such
+as ISGT Europe (titles + DOI links, also title-based, and generic to any
+DBLP-indexed conference).
 
 ## Pipeline stages
 
@@ -111,7 +115,7 @@ sequenceDiagram
 | `cli.py` | Console entry point (`conflens`); `--clear-cache`, `--host/--port`; loads `.env`.                                                                                                                                 |
 | `app.py` | NiceGUI UI: configuration form, input validation, progress, and the interactive results view (ECharts chart, per-topic findings + papers, live re-threshold / search / sort / facet, save + load a run), exports. |
 | `pipeline.py` | `AnalysisConfig` + `run_analysis()` orchestrating the stages with a `Progress` object (supports cooperative cancel).                                                                                              |
-| `sources.py` | Source interface + registry + `make_source()`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`, shared `_robust_get`.                                                                                             |
+| `sources.py` | Source interface + registry + `make_source()`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`, `DBLPSource`, shared `_robust_get`.                                                                                |
 | `scraper.py` | `AnthologyScraper` (ACL Anthology adapter, also serving EMNLP / NAACL) + shared HTML helpers.                                                                                                                     |
 | `dedup.py` | `annotate_duplicates()` — dependency-free near-duplicate-title clustering (union-find + `difflib`).                                                                                                               |
 | `classifier.py` | Batched, structured-output relevance classification with on-disk cache.                                                                                                                                           |