Add OSFI speeches RAG ingestion pipeline (R script + docs)

[charles-hebert]

Motivation

• Provide an unattended R-based pipeline to crawl OSFI speech listings, persist HTML/text snapshots, and produce vector-ready chunks for RAG workflows.
• Support optional on-the-fly OpenAI embeddings and explicit tracking of removed speeches via deletion_date and removed_detected_on for downstream ingestion/upserts.

Description

• Add osfi_speeches_rag_pipeline.R, an Rscript that paginates the OSFI news index, extracts speech pages, saves raw HTML and cleaned text, chunks text into overlapping windows, and optionally requests OpenAI embeddings when OSFI_EMBED_PROVIDER=openai and OPENAI_API_KEY are set.
• The script writes a canonical registry at metadata/speech_table.csv, a chunk table at vector/speech_chunks.csv, a JSONL hand-off at vector/speech_chunks.jsonl, and an optional Parquet at vector/speech_chunks.parquet when the arrow package is installed.
• Implements robust fetching with retries/timeouts via httr2, speaker detection heuristics, sha256 checksums for text files, and status updates for speeches that disappear from the live index.
• Add osfi_speeches_rag_pipeline.md documenting outputs, deployment patterns (CRON/n8n), configuration flags/environment variables, and example usage.

Testing

• No automated tests were added or executed as part of this change.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dfe7571f1a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Restrict missing-content refetches to listed URLs

missing_content_urls is computed from all non-removed historical rows, but it is not intersected with current_urls. If a previously active speech disappears from the live listing and its local text file is missing, its URL gets into urls_to_fetch, listing_tbl |> filter(url == !!url) returns zero rows, and process_new_speech() then calls safe_response() with an empty URL, aborting the run before removal status can be updated. This breaks unattended runs in the exact recovery scenario this branch is trying to handle.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Validate digest package availability before hashing

The pipeline unconditionally calls digest::digest() when writing each speech record, but digest is neither loaded nor included in the declared required package list. On hosts where digest is not preinstalled, the job will fail at runtime after fetching content, causing partial outputs and making the script non-portable for unattended deployment.

Useful? React with 👍 / 👎.