BFG summer-2026 round readiness: source registry, off-peak scrapers, questions#52
Open
smodee wants to merge 2 commits into
Open
BFG summer-2026 round readiness: source registry, off-peak scrapers, questions#52smodee wants to merge 2 commits into
smodee wants to merge 2 commits into
Conversation
…questions Aligns the pipeline's curated-source registry and extraction to the BFG summer-2026 question set (25 questions), and authors the questions themselves. sources.yaml + source_lookup.py: - Add families enteric (cholera) and vaccine_preventable (measles, polio); populate vector_borne (dengue, chikungunya, oropouche). The route enum is derived from the YAML keys, so new families auto-register as routes. - Extend respiratory/hemorrhagic: WHO human-animal-interface, USDA APHIS livestock, WHO Ebola-DRC-2026 event + DON; add WHO PHEIC/IHR list and WHO SARS-CoV-2 variant tracking; fix the broken CDC-mpox URL. - Mirror H5N1 under animal_spillover (the source router classifies avian flu there rather than respiratory). - Pathogen aliases: bundibugyo/filovirus -> ebola, poliovirus/wpv1 -> polio, avian influenza/h5 -> h5n1. Off-peak extraction scrapers (landing pages are context-only, so these questions extracted 0 records; for off-peak pathogens the injected dashboard is the only on-topic survivor and must yield the number): - _who_hub_common + who_cholera: WHO situation hub -> latest dated sitrep PDF (Docling extracts the tables). - paho_dengue: PAHO alerts listing -> latest dengue epi-update PDF. - gpei_polio: clean get_text render of the JS-nav-heavy "polio this week" page. BFG question authoring (separate from the historic bioscancast_questions.csv): - bfg_summer_2026_questions.csv and bfg_summer_2026_options.csv. Pairs with the dashboard-bypass filter fix (separate PR); gpei_polio's survival through filtering depends on it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Gets the pipeline ready for the BFG summer-2026 round (25 questions): aligns the curated-source registry + extraction to the question set, and authors the questions.
Source registry (
sources.yaml,source_lookup.py)enteric(cholera) andvaccine_preventable(measles, polio); populatedvector_borne(dengue, chikungunya, oropouche). The route enum is derived from YAML keys, so new families auto-register as routes.respiratory/hemorrhagic: WHO human-animal-interface, USDA APHIS livestock, WHO Ebola-DRC-2026 event + DON; added WHO PHEIC/IHR list and WHO SARS-CoV-2 variant tracking; fixed the broken CDC-mpox URL.animal_spillover— the source router classifies avian flu there, notrespiratory.bundibugyo/filovirus→ebola,poliovirus/wpv1→polio,avian influenza/h5→h5n1.Off-peak extraction scrapers
For off-peak pathogens the news search returns mostly noise (correctly filtered), so the injected dashboard is the only on-topic survivor — and these landing pages are context-only, yielding 0 insight records. New scrapers surface the number:
_who_hub_common+who_cholera— WHO situation hub → latest dated sitrep PDF (Docling extracts the tables). Cholera: 0 → 15 records live.paho_dengue— PAHO alerts listing → latest dengue epi-update PDF. 0 → 14 records live.gpei_polio— cleanget_textrender of the JS-nav-heavy "polio this week" page (the default parser over-strips it). 0 → 2 records live.BFG questions
bfg_summer_2026_questions.csv+bfg_summer_2026_options.csv, kept separate from the historicbioscancast_questions.csv. All 25 route to the correct injected sources; dates parse exactly; options load via the existing forecasts-CSV channel (no loader changes).Notes
gpei_polio's survival through filtering depends on it.Full suite: 546 passed, 3 skipped.
🤖 Generated with Claude Code