Skip to content

Add GC Jobs scraper for OSFI and PSIC metadata extraction - #8

Open
charles-hebert wants to merge 1 commit into
mainfrom
codex/develop-r-script-for-job-scraping
Open

Add GC Jobs scraper for OSFI and PSIC metadata extraction#8
charles-hebert wants to merge 1 commit into
mainfrom
codex/develop-r-script-for-job-scraping

Conversation

@charles-hebert

Copy link
Copy Markdown
Owner

Motivation

  • Provide an automated R script to scrape job metadata from the GC Jobs portal for the Office of the Superintendent of Financial Institutions (OSFI) and the Office of the Public Sector Integrity Commissioner (PSIC) for scheduled runs (CRON / n8n).

Description

  • Add gc_jobs_scraper.R, an rvest + tidyverse-based scraper that builds department-filtered URLs via department_ids and extracts job Title, Date, Level, Location, Process_Type, Language_Req, Department, and Detail_URL into a tibble.
  • Implement resilient extraction helpers using purrr::possibly() (safe_html_text, safe_attr) and fallback CSS selectors for both table-row and card-style layouts.
  • Normalize and clean fields with stringr, extract classification Level using regex pattern [A-Z]{2}-\d{2}, standardize Process_Type values, filter results to target departments via target_departments_regex, deduplicate, and export to gc_jobs_output.csv with readr::write_csv().
  • Include notes and a simple URL builder build_jobs_url() to allow looping department IDs as a fast alternative to JS-driven UI interactions and add a comment suggesting RSelenium/chromote as a fallback if the portal becomes fully JavaScript rendered.

Testing

  • Attempted a syntax/parse check via Rscript -e "parse(file='gc_jobs_scraper.R'); cat('syntax ok\n')", which failed because the environment does not have Rscript installed, so no runtime or syntax validation could be completed.
  • No further automated tests were executed in this environment due to the missing R runtime.

Codex Task

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5d04adbcf3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gc_jobs_scraper.R
}

jobs_raw <- imap_dfr(department_ids, ~ {
fetch_department_jobs(department_id = .x, department_name = .y)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Pass full department names into fallback department field

imap_dfr() passes .y (the names OSFI/PSIC) as department_name, and parse_job_row() uses that value when department selectors are missing. The downstream filter only keeps rows matching full department names in target_departments_regex, so any row that relies on this fallback is discarded. In layouts where the department column/class is absent (the exact case this fallback is meant to handle), the scraper can return an empty result set even though jobs were parsed.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant