From 7d1487a6a1c45df2fa330c9c38fc2d43d09239a1 Mon Sep 17 00:00:00 2001 From: Daniel Rakotomalala Date: Mon, 6 Apr 2026 22:03:52 +0200 Subject: [PATCH 1/2] correct build vignette in readme --- README.Rmd | 2 +- README.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.Rmd b/README.Rmd index 51fadbe..ad0641b 100644 --- a/README.Rmd +++ b/README.Rmd @@ -47,7 +47,7 @@ This is the Shiny implementation inherent to the project. You can install the development version with: ```{r, eval = FALSE} -devtools::install_github("danielrak/genprocShiny") +devtools::install_github("danielrak/genprocShiny", build_vignettes = TRUE) ``` --- diff --git a/README.md b/README.md index 19184b1..a0f49e3 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ This README was generated on: ``` r Sys.time() -#> [1] "2026-03-30 20:16:54 CEST" +#> [1] "2026-04-06 21:50:12 CEST" ``` ------------------------------------------------------------------------ @@ -38,7 +38,7 @@ This is the Shiny implementation inherent to the project. You can install the development version with: ``` r -devtools::install_github("danielrak/genprocShiny") +devtools::install_github("danielrak/genprocShiny", build_vignettes = TRUE) ``` ------------------------------------------------------------------------ From 5ccc4d5002454cbb45035b11de149d1cfc595055 Mon Sep 17 00:00:00 2001 From: Daniel Rakotomalala Date: Mon, 6 Apr 2026 22:16:49 +0200 Subject: [PATCH 2/2] New readme and vignette --- README.Rmd | 184 ++++++++++++++++++++--------- README.md | 215 ++++++++++++++++++++++++---------- vignettes/genprocShiny.Rmd | 229 ++++++++++++++++++++++++++----------- 3 files changed, 444 insertions(+), 184 deletions(-) diff --git a/README.Rmd b/README.Rmd index ad0641b..54554b6 100644 --- a/README.Rmd +++ b/README.Rmd @@ -31,15 +31,73 @@ Sys.time() --- - ## Overview -The project attempts to build a framework for turning single-case data processing scripts into **scalable, repeatable batch workflows**. +`genprocShiny` is a Shiny proof of concept for turning a **single working data-processing task** into a **batch process over many explicit input/output cases**. + +The concrete idea is simple: + +- you already know how to perform one transformation; +- you now need to run that transformation across many files, datasets, or parameterized cases; +- you want a structured way to define the cases, validate the function, run the process, and recover execution logs. + +In this PoC, the process is driven by a **mask**: a table in which each row defines one task to execute. + +--- + +## The concrete problem this project targets + +A common practical situation is the following: + +1. you successfully transform one input into one output; +2. you realize that the same operation must now be repeated dozens or hundreds of times; +3. each case has explicit inputs and outputs; +4. you need something more robust than copying code or writing an ad hoc loop. + +For example: + +- convert many files from one format to another; +- apply the same cleaning function to many datasets; +- generate many outputs whose paths are already listed in a control table; +- run a repeatable administrative or statistical transformation over many declared cases. + +This is the concrete narrative of `genprocShiny`: **scale input/output tasks from one validated transformation**. + +--- + +## What the current PoC already does + +The current app is intentionally minimal, but it already covers the full logic of a batch run: + +- **upload a mask** that declares the cases to run; +- **build a function from example code** or write the function directly; +- **map function arguments to mask column names**; +- **launch execution**; +- **retrieve row-wise logs** describing success or failure. + +This means the app is not just about iterating over rows. It is about making a transformation **operational**: + +- the cases are explicit; +- the interface between function and task table is explicit; +- execution is separated from result inspection; +- logging is part of the workflow. + +--- + +## Why this is not only a wrapper around `purrr` -It enables you to apply a function across many cases defined in a table, while ensuring **traceability, robustness, and operational control**. +Iteration tools are useful, but they do not by themselves define an execution framework. + +`genprocShiny` adds several layers around the actual iteration step: + +- a **mask-first** way to declare tasks; +- a **function-to-mask interface** through argument mapping; +- a **from-example-to-function** step for users who can demonstrate one task but do not want to parameterize everything by hand; +- **row-wise success/error logging**; +- **background execution** and parallel processing support. + +The goal is therefore not only to "map a function" but to help turn one transformation into a **repeatable batch process with observable execution**. -This is the Shiny implementation inherent to the project. - --- ## Installation @@ -52,90 +110,102 @@ devtools::install_github("danielrak/genprocShiny", build_vignettes = TRUE) --- -## A demo vignette +## Main entry points + +Launch the Shiny application with: ```{r, eval = FALSE} -vignette("genprocShiny", "genprocShiny") +genprocShiny::run_app() ``` ---- - -## Why? +Create a demo framework oriented toward repeated file processing with: -In many real-world workflows, processing starts from a single case: +```{r, eval = FALSE} +genprocShiny::create_demo_framework("path/to/your/folder") +``` -- one file -- one dataset -- one transformation +Browse available vignettes with: -Scaling this often leads to: +```{r, eval = FALSE} +browseVignettes("genprocShiny") +``` -- duplicated code -- fragile loops -- limited error handling -- no visibility on execution +--- -While tools like `purrr::pmap()` help iterate, they do not address **operational concerns**. +## A more concrete example of use -**genproc goes beyond iteration by structuring execution.** +Suppose you have one working transformation that turns one input file into one output file. ---- +You now want to run it on many cases. -## General features +A typical workflow is: -- **Standardized execution** - Functions are adapted to a consistent interface. +1. prepare a mask with columns such as input path, input file, output path, and output file; +2. define the transformation function; +3. map the function arguments to the mask column names; +4. run the process; +5. inspect the resulting logs. -- **Flexible parameter mapping** - Decouples function logic from input data structure. +This is exactly the kind of use case the current PoC is designed to make more explicit and more reusable. -- **Execution robustness** - Each run is isolated with controlled error handling. +--- -- **Traceable outputs** - Structured logs for each execution (success, failure, inputs). +## From example to function -- **Background and parallel execution** - Runs independently from the interactive session. +One important part of the current PoC is that the function can be obtained in two ways: ---- +- **directly**, by writing the function code yourself; +- **indirectly**, by starting from example code and transforming it into a function. -## Typical use cases +This matters for a practical reason: many users know how to perform one transformation in R, but the step from "working example" to "reusable function" is often the first barrier to scaling. -- Batch processing of files or datasets -- Reproducible data preparation pipelines -- Administrative or production workflows -- Scaling a prototype script to industrial execution +`genprocShiny` therefore includes a dedicated interface for building the function from example code and then validating or editing the resulting function before execution. --- ## Shiny interface -You can launch the application with: +The current interface is organized as four steps: -```{r, eval = FALSE} -genprocpoc::run_app() -``` +1. **Mask** +2. **Function** +3. **Execution** +4. **Results** -The interface allows you to: +This reflects the intended workflow of the PoC: -- upload an input table -- define a processing function -- map parameters -- launch execution -- inspect results +- declare the tasks; +- validate the transformation; +- run the batch process; +- inspect the logs. --- -## What’s next +## Current scope + +This repository should be read as a **proof of concept**. -Planned enhancements include: +Its value is not yet to cover every industrial case, but to make the core workflow tangible: -- stronger input validation and safer code handling -- improved monitoring and real-time feedback -- incremental logging during execution -- more polished user interface -- configuration management for reproducibility -- more scalable execution strategies +- a task table; +- a reusable function; +- an explicit interface between both; +- scalable execution; +- and structured logs. + +That scope is narrower and more concrete than a general-purpose workflow platform, and that is deliberate. --- + +## What comes next + +Planned improvements include: + +- stronger validation of user inputs and code; +- more informative logs, including richer execution metadata; +- better live monitoring during execution; +- easier editing and testing of functions and mappings; +- improved user experience in the Shiny interface; +- more robust support for broader classes of processing tasks. + +In other words, the current PoC already demonstrates the main idea, while the next iterations will make it safer, clearer, and more useful in real batch-processing contexts. diff --git a/README.md b/README.md index a0f49e3..32001f8 100644 --- a/README.md +++ b/README.md @@ -16,20 +16,93 @@ This README was generated on: ``` r Sys.time() -#> [1] "2026-04-06 21:50:12 CEST" +#> [1] "2026-04-06 22:04:31 CEST" ``` ------------------------------------------------------------------------ ## Overview -The project attempts to build a framework for turning single-case data -processing scripts into **scalable, repeatable batch workflows**. +`genprocShiny` is a Shiny proof of concept for turning a **single +working data-processing task** into a **batch process over many explicit +input/output cases**. -It enables you to apply a function across many cases defined in a table, -while ensuring **traceability, robustness, and operational control**. +The concrete idea is simple: -This is the Shiny implementation inherent to the project. +- you already know how to perform one transformation; +- you now need to run that transformation across many files, datasets, + or parameterized cases; +- you want a structured way to define the cases, validate the function, + run the process, and recover execution logs. + +In this PoC, the process is driven by a **mask**: a table in which each +row defines one task to execute. + +------------------------------------------------------------------------ + +## The concrete problem this project targets + +A common practical situation is the following: + +1. you successfully transform one input into one output; +2. you realize that the same operation must now be repeated dozens or + hundreds of times; +3. each case has explicit inputs and outputs; +4. you need something more robust than copying code or writing an ad + hoc loop. + +For example: + +- convert many files from one format to another; +- apply the same cleaning function to many datasets; +- generate many outputs whose paths are already listed in a control + table; +- run a repeatable administrative or statistical transformation over + many declared cases. + +This is the concrete narrative of `genprocShiny`: **scale input/output +tasks from one validated transformation**. + +------------------------------------------------------------------------ + +## What the current PoC already does + +The current app is intentionally minimal, but it already covers the full +logic of a batch run: + +- **upload a mask** that declares the cases to run; +- **build a function from example code** or write the function directly; +- **map function arguments to mask column names**; +- **launch execution**; +- **retrieve row-wise logs** describing success or failure. + +This means the app is not just about iterating over rows. It is about +making a transformation **operational**: + +- the cases are explicit; +- the interface between function and task table is explicit; +- execution is separated from result inspection; +- logging is part of the workflow. + +------------------------------------------------------------------------ + +## Why this is not only a wrapper around `purrr` + +Iteration tools are useful, but they do not by themselves define an +execution framework. + +`genprocShiny` adds several layers around the actual iteration step: + +- a **mask-first** way to declare tasks; +- a **function-to-mask interface** through argument mapping; +- a **from-example-to-function** step for users who can demonstrate one + task but do not want to parameterize everything by hand; +- **row-wise success/error logging**; +- **background execution** and parallel processing support. + +The goal is therefore not only to “map a function” but to help turn one +transformation into a **repeatable batch process with observable +execution**. ------------------------------------------------------------------------ @@ -43,91 +116,115 @@ devtools::install_github("danielrak/genprocShiny", build_vignettes = TRUE) ------------------------------------------------------------------------ -## A demo vignette +## Main entry points + +Launch the Shiny application with: ``` r -vignette("genprocShiny", "genprocShiny") +genprocShiny::run_app() ``` ------------------------------------------------------------------------- - -## Why? +Create a demo framework oriented toward repeated file processing with: -In many real-world workflows, processing starts from a single case: +``` r +genprocShiny::create_demo_framework("path/to/your/folder") +``` -- one file -- one dataset -- one transformation +Browse available vignettes with: -Scaling this often leads to: +``` r +browseVignettes("genprocShiny") +``` -- duplicated code -- fragile loops -- limited error handling -- no visibility on execution +------------------------------------------------------------------------ -While tools like `purrr::pmap()` help iterate, they do not address -**operational concerns**. +## A more concrete example of use -**genproc goes beyond iteration by structuring execution.** +Suppose you have one working transformation that turns one input file +into one output file. ------------------------------------------------------------------------- +You now want to run it on many cases. -## General features +A typical workflow is: -- **Standardized execution** - Functions are adapted to a consistent interface. +1. prepare a mask with columns such as input path, input file, output + path, and output file; +2. define the transformation function; +3. map the function arguments to the mask column names; +4. run the process; +5. inspect the resulting logs. -- **Flexible parameter mapping** - Decouples function logic from input data structure. +This is exactly the kind of use case the current PoC is designed to make +more explicit and more reusable. -- **Execution robustness** - Each run is isolated with controlled error handling. +------------------------------------------------------------------------ -- **Traceable outputs** - Structured logs for each execution (success, failure, inputs). +## From example to function -- **Background and parallel execution** - Runs independently from the interactive session. +One important part of the current PoC is that the function can be +obtained in two ways: ------------------------------------------------------------------------- +- **directly**, by writing the function code yourself; +- **indirectly**, by starting from example code and transforming it into + a function. -## Typical use cases +This matters for a practical reason: many users know how to perform one +transformation in R, but the step from “working example” to “reusable +function” is often the first barrier to scaling. -- Batch processing of files or datasets -- Reproducible data preparation pipelines -- Administrative or production workflows -- Scaling a prototype script to industrial execution +`genprocShiny` therefore includes a dedicated interface for building the +function from example code and then validating or editing the resulting +function before execution. ------------------------------------------------------------------------ ## Shiny interface -You can launch the application with: +The current interface is organized as four steps: -``` r -genprocpoc::run_app() -``` +1. **Mask** +2. **Function** +3. **Execution** +4. **Results** -The interface allows you to: +This reflects the intended workflow of the PoC: -- upload an input table -- define a processing function -- map parameters -- launch execution -- inspect results +- declare the tasks; +- validate the transformation; +- run the batch process; +- inspect the logs. ------------------------------------------------------------------------ -## What’s next +## Current scope + +This repository should be read as a **proof of concept**. -Planned enhancements include: +Its value is not yet to cover every industrial case, but to make the +core workflow tangible: -- stronger input validation and safer code handling -- improved monitoring and real-time feedback -- incremental logging during execution -- more polished user interface -- configuration management for reproducibility -- more scalable execution strategies +- a task table; +- a reusable function; +- an explicit interface between both; +- scalable execution; +- and structured logs. + +That scope is narrower and more concrete than a general-purpose workflow +platform, and that is deliberate. ------------------------------------------------------------------------ + +## What comes next + +Planned improvements include: + +- stronger validation of user inputs and code; +- more informative logs, including richer execution metadata; +- better live monitoring during execution; +- easier editing and testing of functions and mappings; +- improved user experience in the Shiny interface; +- more robust support for broader classes of processing tasks. + +In other words, the current PoC already demonstrates the main idea, +while the next iterations will make it safer, clearer, and more useful +in real batch-processing contexts. diff --git a/vignettes/genprocShiny.Rmd b/vignettes/genprocShiny.Rmd index e5e824f..10fc617 100644 --- a/vignettes/genprocShiny.Rmd +++ b/vignettes/genprocShiny.Rmd @@ -1,92 +1,185 @@ --- -title: "genproc: Shiny Proof of Concept" +title: "Getting started with genprocShiny" output: rmarkdown::html_vignette vignette: > - %\VignetteIndexEntry{genproc: Shiny Proof of Concept} + %\VignetteIndexEntry{Getting started with genprocShiny} %\VignetteEngine{knitr::rmarkdown} - \usepackage[utf8]{inputenc} + %\VignetteEncoding{UTF-8} --- -```{r setup, include = FALSE} +```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, - comment = "#>", - eval = FALSE + comment = "#>" ) -library(genprocShiny) ``` - -# Overview -`genproc` provides a minimal Shiny interface to run batch processes driven by a mask (a data frame where each row defines one execution case). +## Overview -The application is structured into four steps: +`genprocShiny` is a proof-of-concept Shiny app for scaling a repeated processing task from one case to many. -1. Upload a mask -2. Define a function -3. Configure and launch the process -4. Inspect logs +The core idea is simple: -The goal of this vignette is to demonstrate how to use the Shiny application through a reproducible demo workflow. +- define the cases to run in a **mask**; +- define the transformation as a **function**; +- map the function arguments to the mask columns; +- run the process and inspect the logs. -# Demo and how to use +The current app is intentionally minimal and is organized around four sections: -To get started, generate a ready-to-use demo framework: +1. Mask +2. Function +3. Execution +4. Results -```{r} -create_demo_framework("path/to/demo") +A practical way to think about the app is as a helper for **input/output tasks at scale**: for example, converting many files, exporting many objects, or applying the same transformation across a table of input/output definitions. + +## Launch the app + +```{r eval = FALSE} +run_app() +``` + +## Optional: create a demo framework + +The package provides a helper to create a demo folder containing example datasets, a mask, an output directory, and a function/mapping example. + +```{r eval = FALSE} +create_demo_framework("path/to/your/folder") ``` -This creates the following elements within `"path/to/demo"`, which we'll refer as `"./"` from now on: +This helper is useful for quickly creating material to test the app with a concrete batch-processing scenario. -- a mask file (`"./mask/mask.csv"`) -- a demo function and argument mapping (`"./func/func_and_args_mapping.R"`) -- input datasets (R built-in ones, `"./r_built_in_datasets/"`) -- an empty output directory (`"./out/"`) +## Step 1 - Upload a mask -Then, launch the application with: +In the **Mask** section, upload a `.csv` file describing the cases to run. -```{r} -run_app() +Each row of the mask corresponds to one execution case. + +For a file-processing use case, the mask will often contain columns such as: + +- `input_dir` +- `input_file` +- `output_dir` +- `output_file` + +The app displays a preview of the uploaded mask and performs basic validation. + +## Step 2 - Build the function + +The **Function** section supports two entry points: + +1. **Write a function directly** +2. **Generate a function from example code** + +Both routes feed the same function preview area. + +### Option A - Write the function directly + +You can write a standard R function in the function code box and validate it. + +For example: + +```r +function(input_dir, input_file, output_dir, output_file) { + data <- readRDS(file.path(input_dir, input_file)) + rio::export(data, file.path(output_dir, output_file)) +} +``` + +This route is the simplest one for the demo framework created by `create_demo_framework()` because the demo mask already separates directories and filenames. + +### Option B - Build the function from example code + +You can also start from a concrete code example and let the app convert it into a reusable function. + +For example, suppose you already know how to run the transformation once: + +```r +{ + data <- readRDS("/some/path/iris.rds") + rio::export(data, "/some/out/iris.csv") +} ``` -The interface contains four sections: - -- 1 - Mask -- 2 - Function -- 3 - Execution -- 4 - Results - -## Step 1 - Upload the mask - -Upload the file at `"./mask/mask.csv"`. - -The application then validates the file format, loads it as a data frame and displays a preview. - -Each row of the mask represents one execution case. - -## Step 2 - Define function (and its arguments mapping with the mask names) - -Open this file: `"./func/func_and_args_mapping.R"`. - -Copy the function part into the corresponding area in the app and click *Validate function*. - -Similarly, get the arguments mapping part in the app and click *Validate mapping*. - -The application ensures that mapping names match function arguments and mapping values match mask column names. - -## Step 3 - Configure and run - -- Set a process label -- Select a directory for upcoming logs (e.g. at `"./out/"`) -- Click *Run* - -The application will adapt the function to the mask, wrap execution with error handling and execute the tasks in parallel. For your information, the number of workers is set by default to `parallel::detectCores() / 2` (half of your available cores). - -## Step 4 - Retrieve logs - -Click *Get logs* - -The application reads the log file named `"[proc_label].rds"` within the previously defined logs directory, and displays it in a table. Its structure is basically that of the mask + a `success` flag and an `error message` if applicable. - - +Paste this code in the **Write your example code** box, then click **Validate example**. + +The app will: + +- turn the example into a function; +- display that function in the shared function preview area; +- copy the generated function into the function code box so you can edit it. + +This is useful when the user starts from a working script rather than from a formalized function. + +### Programmatic use + +The same feature is also available directly in R through `from_example_to_function()`. + +```{r eval = FALSE} +expr <- expression({ + data <- readRDS("/some/path/iris.rds") + rio::export(data, "/some/out/iris.csv") +}) + +from_example_to_function(expr) +``` + +In the current proof of concept, this function is especially helpful as a bridge between ad hoc code and reusable processing logic. + +## Step 3 - Map the function arguments to the mask + +In the **Arguments mapping** tab, provide a named character vector that links function arguments to mask column names. + +For example: + +```r +c( + "input_dir" = "input_dir", + "input_file" = "input_file", + "output_dir" = "output_dir", + "output_file" = "output_file" +) +``` + +The names correspond to the function arguments, and the values correspond to the mask columns. + +The app validates that: + +- the mapping is a named character vector; +- the values exist in the mask; +- the names match the function arguments. + +## Step 4 - Run the process + +In the **Execution** section: + +- choose a process label; +- select a logs directory; +- click **Run**. + +The app then launches the processing job and saves log files to the selected location. + +## Step 5 - Retrieve the logs + +In the **Results** section, click **Get logs** to load the resulting log table. + +The current proof of concept focuses on row-wise success/error feedback so the user can inspect which cases succeeded and which failed. + +## What this proof of concept already shows + +Even in its minimal form, the app already demonstrates a useful workflow: + +- explicit task definition through a mask; +- function creation from either direct coding or a worked example; +- argument mapping between function and mask; +- batch execution; +- per-case logs. + +This makes the project easier to understand when presented through a concrete use case such as file conversion or repeated input/output processing. + +## Current limitations and direction + +`genprocShiny` is still a proof of concept. + +The current app is intentionally lightweight and keeps the workflow simple. Future improvements may include richer validation, more ergonomic code editing, stronger live monitoring, and more advanced handling of failed cases and reruns.