Add OSFI speeches ingestion R script and README for recurring RAG pipeline

[charles-hebert]

Motivation

• Provide an automated, repeatable R workflow to crawl OSFI speeches and produce plain-text + vectorized chunks for a retrieval-augmented generation (RAG) pipeline.
• Only fetch and store speeches that are new or changed, and record when previously cataloged speeches disappear from the listing.
• Ensure speaker metadata is present (with explicit detection for Peter Routledge and Ben Gully) so chunks can be filtered by speaker.

Description

• Add osfi_speeches_ingest.R, an Rscript that discovers paginated OSFI speech listings, extracts listing metadata, downloads speech bodies, normalizes text, detects the speaker, chunks content and generates per-chunk embeddings using a deterministic hash backend.
• Implement cataloging logic that maintains speech_catalog.csv, speech_chunks.csv, per-chunk JSONL vectors at vectors/speech_chunk_vectors.jsonl, plain-text files under text/, and a run_log.csv; the catalog sets removed_noticed_date when a known URL is no longer present on the live listing.
• Improve HTML parsing robustness via updated helpers (extract_card_nodes, extract_listing_entry, extract_best_content_node, clean_body_text), speaker extraction logic (detect_speaker), and chunking/embedding helpers (split_into_chunks, hash_embedding).
• Make behavior configurable via environment variables such as OSFI_DATA_DIR, OSFI_VECTOR_BACKEND, OSFI_VECTOR_DIM, OSFI_CHUNK_SIZE, OSFI_CHUNK_OVERLAP, and OSFI_USER_AGENT.
• Add README_osfi_speeches.md documenting outputs, scheduling examples for cron and n8n, and recommended vectorization upgrade paths (OpenAI or local models).

Testing

• Ran git diff --check and repository static checks with no reported whitespace or diff errors.
• Executed Python-based static content checks to verify presence of key markers in the new script (e.g., removed_noticed_date, detect_speaker <- function, hash_embedding <- function, html_elements("main h3 a"), speech_chunk_vectors.jsonl, split_into_chunks <- function) and those checks passed.
• Could not run the script end-to-end because Rscript/R is not available in the execution environment, so runtime validation was not performed.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e12640a617

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Re-fetch speeches when existing URLs change

The selection for new_or_changed only keeps rows where content_hash is NA, so once a speech URL has been ingested once it is never fetched again even if OSFI edits the page content at that same URL. In that scenario, speech_catalog.csv, speech_chunks.csv, and speech_chunk_vectors.jsonl stay stale indefinitely, which breaks the script’s stated “new or changed” ingestion behavior.

Useful? React with 👍 / 👎.