Add periodic R script to scrape Open Government ATIP requests by department

[charles-hebert]

Motivation

• Provide an automated, re-runnable scraper that extracts ATIP requests for a configurable department (default: Office of the Superintendent of Financial Institutions Canada) and writes the key fields to a CSV.
• Make the scraper suitable for periodic execution via CRON or n8n by exposing configuration through environment variables.

Description

• Added atip_requests_updater.R which constructs a search URL against https://open.canada.ca/en/search/ati and paginates results up to a configurable ATIP_MAX_PAGES.
• Parses HTML tables with flexible column-name matching and fallbacks to extract date, pages, description, and file_number.
• Normalizes dates and page counts, adds metadata columns (department, source_url, scraped_at_utc), and orders the output with the requested columns first.
• Implements incremental update behavior that appends only new requests (deduplicated by file_number) to the configured ATIP_OUTPUT_CSV, and includes a CRON/n8n usage example in the script header.

Testing

• Attempted syntax validation with Rscript -e "parse(file='atip_requests_updater.R')" but the check could not run because Rscript is not installed in this environment.
• Attempted an HTTP fetch to the Open Government search endpoint with curl which returned 403 Forbidden, so live scraping could not be validated in this environment.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f58643c329

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve existing rows when dedupe key is unavailable

When output_csv exists but the loaded data does not contain file_number (including the read-error fallback to an empty frame), this branch replaces the whole dataset with only the current scrape. In that case historical rows are silently discarded, which contradicts the append-only incremental behavior and can corrupt long-running exports; this should fail fast or keep existing rows instead of overwriting them.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Deduplicate scraped rows by file_number before append

The incremental filter only removes rows already present in existing, so duplicates within the same run (same file_number repeated across tables/pages) are all appended together. Because unique(scraped) includes source_url, duplicates with different page URLs are not collapsed, which can inflate counts and create duplicate request entries in the CSV.

Useful? React with 👍 / 👎.