diff --git a/AGENTS.md b/AGENTS.md index c56a0a7..301469e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,8 +6,9 @@ Guidance for coding agents (and humans) working in this repository. See ## What this is **ConfLens** — a NiceGUI web app that browses conference papers (ACL Anthology, -EMNLP, NAACL, IJCAI, OpenReview / ICLR · NeurIPS, and PSCC), classifies them -against a theme with an LLM, discovers topics, and synthesises per-topic findings. It also +EMNLP, NAACL, IJCAI, OpenReview / ICLR · NeurIPS, PSCC, and DBLP-indexed venues +such as ISGT Europe), classifies them against a theme with an LLM, discovers +topics, and synthesises per-topic findings. It also flags near-duplicate titles and offers a fully client-side results view (live confidence re-threshold, keyword search + highlight, sort/author facets, save/load a run). Python 3.13, managed with **uv**. @@ -60,7 +61,7 @@ regexes over heavyweight parsers, small focused modules). |------|--------| | UI + exports | `app.py`, `pptx_export.py`, `bibtex.py` | | Orchestration | `pipeline.py` (`AnalysisConfig`, `run_analysis`) | -| Sources | `sources.py` (registry + `make_source`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`), `scraper.py` (`AnthologyScraper` — also serves EMNLP/NAACL) | +| Sources | `sources.py` (registry + `make_source`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`, `DBLPSource`), `scraper.py` (`AnthologyScraper` — also serves EMNLP/NAACL) | | LLM providers | `llm.py` (`LLMClient`, `make_client`) | | Classify / topics | `classifier.py`, `topics.py` | | Near-duplicate detection | `dedup.py` (`annotate_duplicates`) | diff --git a/README.md b/README.md index fcef290..af811dc 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,10 @@ A desktop-style web app (built with [NiceGUI](https://nicegui.io)) that: by venue id (e.g. `ICLR.cc/2024/Conference`, `NeurIPS.cc/2024/Conference`), and - **PSCC** (Power Systems Computation Conference) proceedings, by year (e.g. `2024`) — titles + PDFs (abstracts aren't published, so classification - is title-based). + is title-based), and + - **ISGT Europe** (IEEE PES) via the open **DBLP** index, by venue + year + (e.g. `isgteurope 2024`) — titles + DOI links, title-based (the generic + DBLP adapter works for any DBLP-indexed conference). The scraper is pluggable — adding another conference is one adapter in `conference_analyzer/sources.py`. @@ -148,7 +151,7 @@ pluggable *LLM provider* (which model), with every expensive step cached on disk ```mermaid flowchart LR UI["NiceGUI UI
app.py"] --> PIPE["pipeline.py"] - PIPE --> SRC["sources.py
ACL · EMNLP · NAACL · IJCAI · OpenReview · PSCC"] + PIPE --> SRC["sources.py
ACL · EMNLP · NAACL · IJCAI · OpenReview · PSCC · ISGT Europe"] PIPE --> CLS["classifier.py"] PIPE --> TOP["topics.py
model + summarize"] CLS --> LLM["llm.py
Anthropic · OpenAI · LiteLLM"] @@ -166,7 +169,7 @@ data-model and caching diagrams, plus extension points. | Stage | Module | Notes | |-------|--------|-------| -| Sources | `conflens/sources.py` | Pluggable adapters (ACL Anthology, EMNLP, NAACL, IJCAI, OpenReview, PSCC) behind one interface; registry + factory. | +| Sources | `conflens/sources.py` | Pluggable adapters (ACL Anthology, EMNLP, NAACL, IJCAI, OpenReview, PSCC, DBLP/ISGT Europe) behind one interface; registry + factory. | | Scrape listing | `conflens/scraper.py` | ACL Anthology adapter: parses the event page; abstracts + authors fetched per paper and cached. | | Near-duplicates | `conflens/dedup.py` | Flags near-identical titles (union-find + `difflib`); dependency-free. | | Classify | `conflens/classifier.py` | Batched, structured-output calls; relevance + confidence + a one-line reason per paper (cached). | @@ -179,17 +182,25 @@ data-model and caching diagrams, plus extension points. ## Configuration (in the UI) - **Source** — `ACL Anthology`, `EMNLP (ACL Anthology)`, `NAACL (ACL Anthology)`, - `IJCAI`, `OpenReview (ICLR / NeurIPS)`, or `PSCC`. Switching prefills the base - URL and target below and relabels them. (ACL Anthology, EMNLP and NAACL share - one adapter — any of them accepts any Anthology event slug, e.g. `acl-2024`, - `emnlp-2023`, `naacl-2024`.) + `IJCAI`, `OpenReview (ICLR / NeurIPS)`, `PSCC`, or `ISGT Europe (via DBLP)`. + Switching prefills the base URL and target below and relabels them. (ACL + Anthology, EMNLP and NAACL share one adapter — any of them accepts any + Anthology event slug, e.g. `acl-2024`, `emnlp-2023`, `naacl-2024`.) - **Base URL** — the site/API root (e.g. `https://aclanthology.org`, `https://2026.ijcai.org`, `https://api2.openreview.net`); change it to point at a mirror, another year, or the v1 API host (`https://api.openreview.net`). - **Event / target** — for ACL, a slug (`acl-2024`, `emnlp-2023`, …) or full event URL; for IJCAI, the accepted-papers path or full URL; for OpenReview, the **venue id** (`ICLR.cc/2024/Conference`, `NeurIPS.cc/2024/Conference`) or a - venue group URL; for PSCC, a **year** (`2024`, `2022`, …) or a full listing URL. + venue group URL; for PSCC, a **year** (`2024`, `2022`, …) or a full listing URL; + for ISGT Europe, a DBLP **venue + year** (`isgteurope 2024`) or proceedings key. + + > **Title-based sources** — PSCC and DBLP-backed venues (ISGT Europe) don't + > expose abstracts, so classification runs on titles alone (still useful, just + > coarser). The DBLP adapter is generic: any DBLP-indexed conference works by + > passing ` `. IEEE-Xplore-only venues that DBLP doesn't index + > (e.g. **PowerTech**, **PES General Meeting**) can't be added without an IEEE + > Xplore API key. > **OpenReview auth** — recent (API v2) venues challenge anonymous requests > from some networks. If a run returns an auth error, set `OPENREVIEW_TOKEN`, diff --git a/conflens/pipeline.py b/conflens/pipeline.py index ef2c50c..450141d 100644 --- a/conflens/pipeline.py +++ b/conflens/pipeline.py @@ -97,6 +97,10 @@ def run_analysis( " PSCC is biennial — try a year that was held, e.g. '2024' or " "'2022'." ), + "isgteurope": ( + " Check the venue + year (e.g. 'isgteurope 2024') — DBLP may " + "not have indexed that edition yet." + ), } hint = hints.get(cfg.source, "") progress.set("listing", "No papers found on that page." + hint, 1.0) diff --git a/conflens/sources.py b/conflens/sources.py index 68a197f..92a2706 100644 --- a/conflens/sources.py +++ b/conflens/sources.py @@ -21,6 +21,10 @@ * **pscc** — Power Systems Computation Conference; a per-year HTML fragment from the papers-repository endpoint gives titles, authors and PDF links (abstracts aren't published on the site, so classification is title-based). +* **isgteurope** — IEEE PES ISGT Europe, via the public DBLP search API (many + IEEE-Xplore-only venues are indexed there). Titles, authors and a DOI link per + paper; no abstracts, so classification is title-based. The generic + :class:`DBLPSource` behind it works for any DBLP-indexed conference. Adding another conference is a matter of writing one more adapter and registering it in :data:`SOURCES`. @@ -81,7 +85,8 @@ def _robust_get(url: str, timeout: int = 120, headers: Optional[dict] = None) -> if e.code not in _RETRY_STATUS: raise RuntimeError(f"Failed to fetch {url}: HTTP {e.code} {e.reason}") from e time.sleep(1.5 * (attempt + 1)) - except (urllib.error.URLError, TimeoutError) as e: + except (urllib.error.URLError, TimeoutError, ConnectionError) as e: + # ConnectionError covers a peer reset mid-response (e.g. rate-limiting). last_err = e time.sleep(1.5 * (attempt + 1)) if partial_best: @@ -556,6 +561,165 @@ def enrich_abstracts( return papers +# --------------------------------------------------------------------------- +# DBLP source (open metadata index — e.g. IEEE PES ISGT Europe) +# --------------------------------------------------------------------------- +# DBLP indexes many conferences that are otherwise only on IEEE Xplore. Its +# public search API returns, per proceedings, each paper's title, authors and a +# link (usually a DOI) — but no abstract, so classification is title-based. +_DBLP_DISAMBIG = re.compile(r"\s+\d{4}$") # DBLP appends "0001"-style suffixes +_DBLP_TRAIL_DOT = re.compile(r"\.\s*$") + + +class DBLPSource: + """Fetch a conference edition's papers from the public DBLP search API. + + ``target`` is a ``"venue year"`` pair (e.g. ``"isgteurope 2024"``), a DBLP + proceedings key (``conf/isgteurope/isgteurope2024``) or a DBLP URL. Only + titles, authors and a link (usually the DOI) are available — abstracts are + not, so downstream classification is title-based. + """ + + name = "dblp" + _PAGE = 100 # DBLP's public search API caps hits-per-request at 100 + + def __init__( + self, + base_url: str = "https://dblp.org", + cache_dir: Optional[str] = None, + timeout: int = 120, + ) -> None: + self.base_url = (base_url or "https://dblp.org").rstrip("/") + self.timeout = timeout + self.cache_dir = cache_dir or os.path.join( + os.path.expanduser("~"), ".cache", "conflens" + ) + os.makedirs(self.cache_dir, exist_ok=True) + + @staticmethod + def _proc_key(target: str) -> str: + """Normalise ``target`` to a DBLP proceedings key like ``conf/x/x2024``.""" + t = (target or "").strip() + m = re.search(r"(conf/[^\s?#]+?)(?:\.html|\.bht)?(?:[?#].*)?$", t) + if m: + return m.group(1) + parts = t.split() + venue = parts[0].strip("/") if parts else "" + year = "" + for p in parts[1:]: + ym = re.search(r"(19|20)\d{2}", p) + if ym: + year = ym.group(0) + break + return f"conf/{venue}/{venue}{year}" + + def resolve_url(self, target: str) -> str: + return f"{self.base_url}/db/{self._proc_key(target)}.html" + + def _cache_path(self, key: str) -> str: + digest = hashlib.sha1(key.encode("utf-8")).hexdigest()[:16] + return os.path.join(self.cache_dir, f"dblp_{digest}.json") + + def list_papers(self, target: str, force_refresh: bool = False) -> list[Paper]: + key = self._proc_key(target) + cache = self._cache_path(key) + if not force_refresh and os.path.exists(cache): + try: + with open(cache, "r", encoding="utf-8") as fh: + data = json.load(fh) + return [Paper(**p) for p in data.get("papers", [])] + except (OSError, json.JSONDecodeError, TypeError): + pass + + hits = self._fetch_hits(key) + papers = self.parse_hits(hits) + try: + with open(cache, "w", encoding="utf-8") as fh: + json.dump({"key": key, "papers": [p.to_dict() for p in papers]}, fh) + except OSError: + pass + return papers + + def _fetch_hits(self, proc_key: str) -> list[dict]: + """Page through every DBLP record whose TOC is this proceedings.""" + toc = f"db/{proc_key}.bht" + hits: list[dict] = [] + first = 0 + while True: + if first: # be polite to DBLP between pages to avoid throttling + time.sleep(0.7) + q = urllib.parse.urlencode( + {"q": f"toc:{toc}:", "format": "json", "h": self._PAGE, "f": first, "c": 0} + ) + raw = _robust_get( + f"{self.base_url}/search/publ/api?{q}", + self.timeout, + headers={"User-Agent": _BROWSER_UA, "Accept": "application/json"}, + ) + try: + result = json.loads(raw).get("result", {}) + except json.JSONDecodeError: + break + batch = result.get("hits", {}).get("hit", []) or [] + hits.extend(batch) + total = int(result.get("hits", {}).get("@total", len(hits)) or 0) + first += self._PAGE + if first >= total or not batch: + break + return hits + + def parse_hits(self, hits: list[dict]) -> list[Paper]: + """Turn DBLP search hits into papers (no network).""" + papers: list[Paper] = [] + seen: set[str] = set() + for hit in hits: + info = hit.get("info", {}) or {} + title = _DBLP_TRAIL_DOT.sub("", _clean(str(info.get("title", "") or ""))) + if not title: + continue + key = info.get("key") or hit.get("@id") or f"{len(papers) + 1}" + paper_id = "dblp-" + str(key).replace("/", "-") + if paper_id in seen: + continue + seen.add(paper_id) + + raw_authors = (info.get("authors", {}) or {}).get("author", []) + if isinstance(raw_authors, dict): + raw_authors = [raw_authors] + authors = [] + for a in raw_authors: + name = a.get("text", "") if isinstance(a, dict) else str(a) + name = _DBLP_DISAMBIG.sub("", _clean(name)) + if name: + authors.append(name) + + ee = info.get("ee") or info.get("url") or "" + if isinstance(ee, list): + ee = ee[0] if ee else "" + papers.append( + Paper( + paper_id=paper_id, + title=title, + url=ee, # DOI / landing page (full text is usually paywalled) + pdf_url="", # no open PDF via DBLP + authors=authors, + abstract="", # DBLP has no abstracts → title-based classification + ) + ) + return papers + + def enrich_abstracts( + self, + papers: list[Paper], + progress: Optional[Callable[[int, int], None]] = None, + force_refresh: bool = False, + ) -> list[Paper]: + # DBLP exposes no abstracts; titles + authors arrive with the listing. + if progress: + progress(len(papers), len(papers)) + return papers + + # --------------------------------------------------------------------------- # Registry # --------------------------------------------------------------------------- @@ -602,6 +766,13 @@ def enrich_abstracts( "base_label": "PSCC base URL", "target_label": "Conference year (e.g. 2024, 2022) or full listing URL", }, + "isgteurope": { + "label": "ISGT Europe (IEEE PES, via DBLP)", + "base": "https://dblp.org", + "target": "isgteurope 2024", + "base_label": "DBLP base URL", + "target_label": "DBLP venue + year (e.g. isgteurope 2024) or proceedings key", + }, } @@ -617,4 +788,8 @@ def make_source(source: str, base_url: str, cache_dir: Optional[str] = None): return OpenReviewSource(base_url=base_url, cache_dir=cache_dir) if source == "pscc": return PSCCSource(base_url=base_url, cache_dir=cache_dir) + if source in ("isgteurope", "dblp"): + # DBLP-backed; ISGT Europe (IEEE PES) is the registered venue, but any + # DBLP-indexed conference works by passing " " as the target. + return DBLPSource(base_url=base_url, cache_dir=cache_dir) raise ValueError(f"Unknown source: {source}") diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 9dac888..cf985cf 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -24,6 +24,7 @@ flowchart TB REG --> IJ["IJCAISource"] REG --> OR["OpenReviewSource
(ICLR · NeurIPS)"] REG --> PS["PSCCSource"] + REG --> DB["DBLPSource
(ISGT Europe · …)"] end subgraph llm["LLM providers (llm.py · make_client)"] @@ -49,7 +50,10 @@ flowchart TB EMNLP and NAACL are served by the same `AnthologyScraper` as the ACL Anthology (different default event); OpenReview talks to the public JSON API instead of scraping HTML; PSCC reads a per-year HTML fragment from its papers-repository -endpoint (titles + PDFs only — no abstracts, so classification is title-based). +endpoint (titles + PDFs only — no abstracts, so classification is title-based); +`DBLPSource` uses the open DBLP search API to reach IEEE-Xplore-only venues such +as ISGT Europe (titles + DOI links, also title-based, and generic to any +DBLP-indexed conference). ## Pipeline stages @@ -111,7 +115,7 @@ sequenceDiagram | `cli.py` | Console entry point (`conflens`); `--clear-cache`, `--host/--port`; loads `.env`. | | `app.py` | NiceGUI UI: configuration form, input validation, progress, and the interactive results view (ECharts chart, per-topic findings + papers, live re-threshold / search / sort / facet, save + load a run), exports. | | `pipeline.py` | `AnalysisConfig` + `run_analysis()` orchestrating the stages with a `Progress` object (supports cooperative cancel). | -| `sources.py` | Source interface + registry + `make_source()`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`, shared `_robust_get`. | +| `sources.py` | Source interface + registry + `make_source()`; `IJCAISource`, `OpenReviewSource`, `PSCCSource`, `DBLPSource`, shared `_robust_get`. | | `scraper.py` | `AnthologyScraper` (ACL Anthology adapter, also serving EMNLP / NAACL) + shared HTML helpers. | | `dedup.py` | `annotate_duplicates()` — dependency-free near-duplicate-title clustering (union-find + `difflib`). | | `classifier.py` | Batched, structured-output relevance classification with on-disk cache. | diff --git a/tests/test_dblp_source.py b/tests/test_dblp_source.py new file mode 100644 index 0000000..98ac9c4 --- /dev/null +++ b/tests/test_dblp_source.py @@ -0,0 +1,81 @@ +from conflens.sources import SOURCES, DBLPSource, make_source + +# Two hits in the shape of DBLP's publ search API: authors as a list and as a +# single dict, disambiguation suffixes, a trailing period on the title. +HITS = [ + { + "info": { + "title": "Safety and Security Dependencies for Gridshield.", + "authors": {"author": [ + {"@pid": "195/3956-1", "text": "Reza Soltani 0001"}, + {"@pid": "191/1849", "text": "Baver Ozceylan"}, + ]}, + "year": "2024", + "key": "conf/isgteurope/0001OLKH24", + "ee": "https://doi.org/10.1109/ISGTEUROPE62998.2024.10863084", + "url": "https://dblp.org/rec/conf/isgteurope/0001OLKH24", + } + }, + { + "info": { + "title": "A Single-Author Paper", + "authors": {"author": {"@pid": "1/2", "text": "Ada Lovelace"}}, + "year": "2024", + "key": "conf/isgteurope/Lovelace24", + "ee": ["https://doi.org/10.1109/X", "https://example.org/alt"], + } + }, +] + + +def test_isgteurope_registered_and_uses_dblp(tmp_path): + assert "isgteurope" in SOURCES + for key in ("label", "base", "target", "base_label", "target_label"): + assert SOURCES["isgteurope"].get(key), f"missing {key}" + src = make_source("isgteurope", base_url="https://dblp.org", cache_dir=str(tmp_path)) + assert isinstance(src, DBLPSource) + + +def test_proc_key_normalisation(): + f = DBLPSource._proc_key + assert f("isgteurope 2024") == "conf/isgteurope/isgteurope2024" + assert f("conf/isgteurope/isgteurope2024") == "conf/isgteurope/isgteurope2024" + assert f("https://dblp.org/db/conf/isgteurope/isgteurope2023.html") == \ + "conf/isgteurope/isgteurope2023" + + +def test_resolve_url(tmp_path): + src = DBLPSource(cache_dir=str(tmp_path)) + assert src.resolve_url("isgteurope 2024") == \ + "https://dblp.org/db/conf/isgteurope/isgteurope2024.html" + + +def test_parse_hits(tmp_path): + src = DBLPSource(cache_dir=str(tmp_path)) + papers = src.parse_hits(HITS) + assert [p.paper_id for p in papers] == [ + "dblp-conf-isgteurope-0001OLKH24", + "dblp-conf-isgteurope-Lovelace24", + ] + p0 = papers[0] + assert p0.title == "Safety and Security Dependencies for Gridshield" # trailing dot dropped + assert p0.authors == ["Reza Soltani", "Baver Ozceylan"] # disambiguation stripped + assert p0.url == "https://doi.org/10.1109/ISGTEUROPE62998.2024.10863084" + assert p0.pdf_url == "" and p0.abstract == "" # title-based + + p1 = papers[1] + assert p1.authors == ["Ada Lovelace"] # single-author dict + assert p1.url == "https://doi.org/10.1109/X" # first ee link + + +def test_parse_hits_skips_untitled_and_dedupes(tmp_path): + src = DBLPSource(cache_dir=str(tmp_path)) + extra = HITS + [ + {"info": {"key": "conf/isgteurope/0001OLKH24", "title": "dup."}}, # duplicate key + {"info": {"key": "conf/x/y", "authors": {}}}, # no title + ] + papers = src.parse_hits(extra) + assert [p.paper_id for p in papers] == [ + "dblp-conf-isgteurope-0001OLKH24", + "dblp-conf-isgteurope-Lovelace24", + ]