Problem
Current workflow uses Quarto documents (.qmd) for STAC catalog creation. While functional for development, they present challenges for production automation:
Issues encountered:
- Logging: Python stdout not captured in logs during
quarto render - missing progress bars, timing, file counts
- Generated files: HTML, .rmarkdown, *_files/ directories clutter repo (now .gitignored)
- Complexity: Requires knitr + reticulate bridge for R↔Python interop
- Automation: More complex to run from cron/scripts than standalone files
- Debugging: Harder to test individual components in isolation
Current Structure
stac_create_collection.qmd # Mixed R/Python, uses reticulate
stac_create_item.qmd # Mixed R/Python, uses reticulate
Execution: quarto render file.qmd --execute
Proposed Structure
Option A: Language-appropriate scripts
scripts/
create_collection.R # Pure R (uses ngr::ngr_s3_keys_get)
create_items.py # Pure Python (uses stac-catalog env)
detect_changes.R # Pure R (uses ngr)
Option B: All Python (if we want consistency)
scripts/
create_collection.py
create_items.py
detect_changes.py
Benefits:
- Clean logging (redirect stdout properly)
- Simpler execution:
Rscript script.R or python script.py
- Better for cron jobs / VM automation
- No generated artifacts
- Easier testing and debugging
- Each script does one thing well
Migration Plan
When: Before Phase 3 (VM automation)
- Phase 2 completion = good breaking point
- Need clean scripts for production deployment
How:
- Extract code from .qmd files to standalone scripts
- Add proper logging (Python logging module, R logging)
- Test equivalence (same outputs)
- Update documentation (CLAUDE.md, README)
- Archive .qmd files (keep for reference)
- Update automation workflows
Trade-offs
Keep .qmd:
- Good for literate programming / exploration
- Nice for mixed R/Python notebooks
- Works in RStudio
Migrate to scripts:
- Better for production automation
- Cleaner logging and error handling
- Industry standard for data pipelines
- Easier maintenance
Recommendation
Migrate to standalone scripts before Phase 3 (VM automation). Quarto docs served us well for development, but production needs clean, simple, loggable scripts.
Related
Priority
Medium - Not blocking Phase 2, but should be done before Phase 3 (VM automation) for production readiness.
Problem
Current workflow uses Quarto documents (.qmd) for STAC catalog creation. While functional for development, they present challenges for production automation:
Issues encountered:
quarto render- missing progress bars, timing, file countsCurrent Structure
Execution:
quarto render file.qmd --executeProposed Structure
Option A: Language-appropriate scripts
Option B: All Python (if we want consistency)
Benefits:
Rscript script.Rorpython script.pyMigration Plan
When: Before Phase 3 (VM automation)
How:
Trade-offs
Keep .qmd:
Migrate to scripts:
Recommendation
Migrate to standalone scripts before Phase 3 (VM automation). Quarto docs served us well for development, but production needs clean, simple, loggable scripts.
Related
Priority
Medium - Not blocking Phase 2, but should be done before Phase 3 (VM automation) for production readiness.