Skip to content

Add R project for sitemap-driven recipe scraping with y/n selection - #11

Open
charles-hebert wants to merge 1 commit into
mainfrom
codex/create-r-project-to-scrape-recipes
Open

Add R project for sitemap-driven recipe scraping with y/n selection#11
charles-hebert wants to merge 1 commit into
mainfrom
codex/create-r-project-to-scrape-recipes

Conversation

@charles-hebert

Copy link
Copy Markdown
Owner

Motivation

  • Provide a reusable two-step R workflow to extract recipe candidate URLs from a sitemap and then scrape only the user-selected recipes.
  • Support sitemaps that are sitemap indexes (recursive discovery) and make it easy to mark which pages to scrape via a selected y/n CSV column.
  • Keep the scraper generic so it can be adapted per-site by tuning selectors in one shared utility file.

Description

  • Add a new R project scaffold under recipes_scraper/ including recipes_scraper.Rproj and .gitignore to house the tooling and ignore R artifacts.
  • Implement shared utilities in R/sitemap_utils.R with safe_read_xml, extract_locs_from_sitemap, filter_recipe_urls, initial_recipe_table, and scrape_recipe_page to parse sitemaps and extract recipe fields.
  • Add scripts/01_extract_recipes_from_sitemap.R to produce a CSV of recipe_url + selected (defaults to n) from a sitemap and scripts/02_scrape_selected_recipes.R to read that CSV and scrape only rows with selected == 'y'.
  • Document setup and usage in README.md and make the CLI scripts executable (chmod +x), with example commands using Rscript.

Testing

  • Attempted an R syntax/parse check using Rscript -e "parse(file='recipes_scraper/R/sitemap_utils.R'); parse(file='recipes_scraper/scripts/01_extract_recipes_from_sitemap.R'); parse(file='recipes_scraper/scripts/02_scrape_selected_recipes.R')", but the check could not run because Rscript is not installed in the environment (test failed due to missing runtime).
  • No runtime scraping tests were executed in this environment; the scripts were marked executable and file contents were created as expected.

Codex Task

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fdb122bca6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


if (length(selected_urls) == 0) {
warning("No rows selected (selected='y'). Writing an empty output file.")
write_csv(tibble(recipe_url = character(), title = character(), ingredients = character(), instructions = character()), output_csv)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Ensure empty-output branch creates parent directory

When no URLs are selected, this branch writes output_csv immediately, but the parent directory is only created later in the non-empty branch. If a caller provides an output path in a directory that does not already exist, write_csv fails and the script exits without producing the expected empty CSV. Create the parent directory before this write (or before the branch) so both execution paths behave consistently.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant