SFAR Recommendations Scraper

Targeted crawler that discovers every SFAR (Société Française d'Anesthésie-Réanimation) clinical recommendation, downloads the document(s), and writes an XLSX manifest for human curation. Most documents are PDFs; the occasional Office document (.docx, .xlsx, …) is downloaded too. Loose image scans are not downloaded (they are reported as misses).

How it works

Discovery reads a local snapshot of the recommendations index (Recommandations - La SFAR.html) rather than fetching the live index. The snapshot is complete (includes 2026), reproducible, and sidesteps Cloudflare 403s. PDFs themselves are always fetched from the live site (which serves them fine with browser headers).

The index lists each recommendation across four tabs (Chronologie, Anesthésie, Réanimation, Urgences). Each row links to the document in one of three forms, all handled:

1. Direct PDF — /wp-content/uploads/YYYY/MM/<file>.pdf (SFAR- or HAS-hosted).
2. WP Download Manager — the landing page carries an <a class="wpdm-download-link" data-downloadurl=".../download/<slug>/?wpdmdl=<id>"> button. The real PDF URL lives in data-downloadurl, not in the visible href (which is just # or the package landing). This is the mechanism the previous version missed, which is why it downloaded almost nothing.
3. Landing page — /slug/ fetched live; all of its download buttons and any non-boilerplate direct documents are collected (a single reco can expose several files, e.g. main text + argumentaire + appendices). If the landing URL itself redirects straight to a document, that file is taken directly.

Footer boilerplate (CGV, règlement intérieur, certificats, chartes…) is filtered out. Documents are deduplicated by WP Download Manager id (or absolute file URL) and merged across the tabs they appear in.

Re-runs are idempotent: a file already recorded in discovery.json and present on disk is skipped with no network request, so a second run downloads only what is new or was missing.

Types

Best-effort from the row's CSS class: RFE, RPP, Préconisation. A document is tagged HAS (Haute Autorité de Santé) when it is hosted on has-sante.fr or its title carries an explicit HAS token.

Install / run (uv)

uv sync                       # create the venv from pyproject.toml + uv.lock

uv run scrape_sfar.py --dry-run     # discover + resolve + manifest, no downloads
uv run scrape_sfar.py               # discover + resolve + download + verify (default)
uv run scrape_sfar.py --verify-only # re-run the completeness check on existing discovery.json
uv run scrape_sfar.py --help        # all options

uv run make_feed.py                 # build feed.xml (Atom) from an existing discovery.json

Useful options: --index-html PATH (override the snapshot), --live-index (fetch the index live, falling back to the snapshot), --workers N (default 6), --delay SECONDS (default 0.5), --output DIR.

Outputs

• pdfs/<year>/<filename> — downloaded documents (mostly .pdf, plus the odd .docx/.xlsx); filename taken from the Content-Disposition header (falls back to the URL basename, with the extension inferred from the content-type); year-unknown for docs found only in discipline tabs.
• manifest.xlsx — formatted, filterable manifest (see columns below); include is left blank for curation.
• discovery.json — raw resolved documents plus the list of unresolved rows, for debugging.
• completeness_report.txt — see below.
• feed.xml — an Atom 1.0 feed of the downloaded recommendations, generated by make_feed.py from discovery.json. Each entry has the title, a link (SFAR landing page, or the direct document URL when there is no landing page), the year as its date, and type/discipline/year categories. Run uv run make_feed.py after a scrape; add --all to also include docs that failed to download.

Manifest columns

type · discipline · subdomain · year · title · landing_url · download_url · wpdmdl · filename · bytes · sha256 · is_new · include

Comprehensiveness check

After downloading, the script cross-checks that nothing was silently dropped and writes completeness_report.txt:

• MISS — index rows that resolved to zero documents (e.g. a HAS portal page with no direct file, a dead 404 link, or an image-only scan). These are surfaced, not hidden.
• ERROR — targets that failed to download: a genuine network/HTTP failure, an empty body, a non-document content-type, or a WP Download Manager Fichier non trouvé HTML page (an old version removed upstream — the current version is downloaded via a sibling link).
• File types — a count of what was downloaded, by extension.
• Coverage — per-year counts, a check that every year 2000-2026 is represented (and 2026 is non-empty), a diff of discovered titles against Recommandations - La SFAR.md, and a superset check against the previously curated ../pdfs/ set.
• External — documents hosted off sfar.org (e.g. HAS, Urofrance) listed for a manual check.

The process exits non-zero if there are unresolved rows or download errors, so an incomplete run fails visibly. Note that a few residual MISS/ERROR entries are dead upstream links (retired HAS portal pages, 404s, and superseded WPDM versions) that cannot be fetched from SFAR at all.