Add GC Jobs scraper for OSFI and PSIC metadata extraction - #8
Add GC Jobs scraper for OSFI and PSIC metadata extraction#8charles-hebert wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d04adbcf3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| jobs_raw <- imap_dfr(department_ids, ~ { | ||
| fetch_department_jobs(department_id = .x, department_name = .y) |
There was a problem hiding this comment.
Pass full department names into fallback department field
imap_dfr() passes .y (the names OSFI/PSIC) as department_name, and parse_job_row() uses that value when department selectors are missing. The downstream filter only keeps rows matching full department names in target_departments_regex, so any row that relies on this fallback is discarded. In layouts where the department column/class is absent (the exact case this fallback is meant to handle), the scraper can return an empty result set even though jobs were parsed.
Useful? React with 👍 / 👎.
Motivation
Description
gc_jobs_scraper.R, anrvest+ tidyverse-based scraper that builds department-filtered URLs viadepartment_idsand extracts jobTitle,Date,Level,Location,Process_Type,Language_Req,Department, andDetail_URLinto a tibble.purrr::possibly()(safe_html_text,safe_attr) and fallback CSS selectors for both table-row and card-style layouts.stringr, extract classificationLevelusing regex pattern[A-Z]{2}-\d{2}, standardizeProcess_Typevalues, filter results to target departments viatarget_departments_regex, deduplicate, and export togc_jobs_output.csvwithreadr::write_csv().build_jobs_url()to allow looping department IDs as a fast alternative to JS-driven UI interactions and add a comment suggestingRSelenium/chromoteas a fallback if the portal becomes fully JavaScript rendered.Testing
Rscript -e "parse(file='gc_jobs_scraper.R'); cat('syntax ok\n')", which failed because the environment does not haveRscriptinstalled, so no runtime or syntax validation could be completed.Codex Task