Skip to content

Phase 2: Turn the notebook pipeline into a real batch job #17

Description

@ChantelRM

Suggested pace: ~2 sessions this week

  • Extract the core pipeline logic (Sections 0-6 of the notebook) into a script that runs non-interactively via Rscript
  • Swap flat-CSV outputs (unified_ledger.csv, pursuit_rankings.csv, etc.) for DuckDB tables as the actual storage layer -- reuses the DuckDB dependency already in place for querychat
  • Confirm the dashboard's app.R can read from the DuckDB tables instead of the CSVs, or documents clearly why it still reads CSVs if that's the final call
  • Update README to describe the pipeline as a script + database, not just a notebook

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions