From e2f156ec6620579d7c55b9456439e911325dd4ac Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 10 Jun 2026 20:44:17 +0000 Subject: [PATCH 1/2] docs: record assessment of 2009 design notebooks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Summarize the July and September 2009 notebooks (workflow-DAG chemometrics tool design): what is superseded by the immutable dataset + PreprocessingSpec + parent_id architecture, and what is worth building on — scoring excluded samples against the fitted model, named validation sets / predict endpoint, the tools-vs-transformations shape invariant, monitoring triggers, and the multiblock/classification model roadmap. https://claude.ai/code/session_01CErviAK6p4MuZiVWsQT3vR --- docs/IDEAS_FROM_2009_NOTES.md | 90 +++++++++++++++++++++++++++++++++++ pyproject.toml | 2 +- uv.lock | 2 +- 3 files changed, 92 insertions(+), 2 deletions(-) create mode 100644 docs/IDEAS_FROM_2009_NOTES.md diff --git a/docs/IDEAS_FROM_2009_NOTES.md b/docs/IDEAS_FROM_2009_NOTES.md new file mode 100644 index 0000000..b7c3950 --- /dev/null +++ b/docs/IDEAS_FROM_2009_NOTES.md @@ -0,0 +1,90 @@ +# Ideas extracted from the 2009 design notebooks + +Two scanned notebooks (July 2009 and 11–18 September 2009) contain the design +for an earlier workflow-based chemometrics tool: data tables organised as +blocks × stacks (train / test / check), transformation chains, lazily-copied +workflow DAGs, triggers/alarms, and multiblock models. This document records +what in those notes is superseded by ScorePilot's current architecture and +what is worth building on. + +## What the notes describe (condensed) + +- **Data-table (DT) model**: a table = blocks (X, Y, Z…) × stacks (train / + test / check) × arrays; a `split()` operation with rule dicts and wildcards + (``, ``, ``); variable-label range parsing (`Var001-099`). +- **Workflow DAG**: data → t₀…tₙ → model → plot nodes, where t₀ is always an + include/exclude filter; the rule "transformations never alter shape; tools + may (lags, batch features, DOE)"; coroutine/generator push execution over a + hand-rolled digraph. +- **Copy semantics**: many pages on cheap workflow copies — lazy/deferred deep + copy, copy-on-change, weakrefs, traits notification — so that + `wf2 = wf1.copy()` duplicates no data until something changes. +- **Excluded data**: stored, never discarded, in specially named stacks + (`excluded_0`, `Validation_0`); "move to validation rather than exclude"; + excluded rows re-offered as validation sets; track *included* rather than + excluded; missing-data masks distinguishing missing vs manually excluded vs + previously excluded. +- **Triggers / timers / notifiers**: condition functions over model outputs + (e.g. `T² > 99 % limit AND grade == 4279-A → alarm/email`), nested dict + AND/OR conditions, file-change notification. +- **Model kinds**: MB-PCA, MB-PLS, SIMCA, PLS-DA, L-shaped data; the rule + "all models handle missing data". + +## Superseded — do not reintroduce + +The copy-semantics machinery (lazy deepcopy, traits, weakrefs), the custom +digraph, the coroutine pipeline engine, masked-array exclusion bookkeeping, +and the storage layer notes. ScorePilot's **immutable dataset + per-variant +`PreprocessingSpec` + `parent_id` lineage** solves the same problem — cheap +variants, zero data duplication, rebuildable models — far more simply: a +variant *is* a recipe diff, so there is nothing to lazily copy. The 2009 +design needed extensive mechanism that `apply_spec(df, spec)` makes +unnecessary. The notes are a useful validation of the current architecture, +not a source of mechanism. + +## Worth building on (ranked) + +1. **Excluded samples become evidence, not waste.** The notes store excluded + rows in named stacks (`Validation_0`) and use "move to validation" rather + than plain exclusion. Today `excluded_rows` simply vanish from a fit; + there is no way to score them against the resulting model. Add a core + "apply model to data" function (project rows → scores, T², SPE, + predictions) and surface excluded rows as an automatic pseudo-validation + set on each variant, recorded in the Logbook. This is the strongest idea + in the notes and directly serves the Logbook's "validation history" + promise. + +2. **Named test sets + a predict endpoint** (the train/test/check "stacks"). + ScorePilot currently has no out-of-sample mechanism — only + cross-validation during fitting. Supporting one or more named validation + datasets per model, with results logged per variant, is the largest + functional gap the notes point at, and item 1 falls out of the same core + function. + +3. **The tools-vs-transformations invariant**: "transformations never alter + shape; tools may (lag, batch feature extraction, DOE)." A clean design + rule for evolving `PreprocessingSpec`: current operations are all + shape-preserving; shape-changing steps (lagged variables for dynamic PLS, + batch unfolding) should be a distinct, explicitly ordered stage. + +4. **Monitoring triggers/notifiers**: condition functions with nested AND/OR + over model outputs (`model.output['T2'][-1] > limit → notify`). Out of + scope for the core analysis loop, but a coherent future "monitoring mode" + that maps directly onto the T²/SPE limits already computed. + +5. **Model-kind roadmap**: MB-PCA / MB-PLS, SIMCA / PLS-DA, L-shaped data, + missing-data-tolerant fitting (today missing cells are only flagged in the + quality report). Per CLAUDE.md, the numerics belong upstream in + `process_improve`, not bespoke in `core/`. + +6. **Small UX wins**: variable-range selection by name pattern + (`Var001-099`), and ``/`` row splits (venetian-blind style test + sets) in the grid / variant-creation flow. + +## Design note worth recording + +July 6 entry: "easier to track *included* rather than excluded, because +future data sets are arbitrarily sourced." ScorePilot already implements the +right split: `x_columns` / `y_columns` are allow-lists (the model's contract +with any future data), while row exclusions are deny-lists (provenance: what +was removed and why). Keep it that way. diff --git a/pyproject.toml b/pyproject.toml index 8e22790..9ccca90 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "scorepilot" -version = "0.22.0" +version = "0.22.1" description = "Web-based tool for PCA/PLS model analysis in chemometrics." readme = "README.md" requires-python = ">=3.12" diff --git a/uv.lock b/uv.lock index c8ef9aa..fd0368a 100644 --- a/uv.lock +++ b/uv.lock @@ -1021,7 +1021,7 @@ wheels = [ [[package]] name = "scorepilot" -version = "0.22.0" +version = "0.22.1" source = { editable = "." } dependencies = [ { name = "alembic" }, From c84e2f26575e6f4e63cffa2fffec92f26eeb9a1e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 10 Jun 2026 21:17:44 +0000 Subject: [PATCH 2/2] docs: merge June 2009 notebook into the design-notes assessment The June notebook adds a worked treatment of latent-variable model inversion for product design (Jaeckle & MacGregor), a Y-space feasibility pre-check, null-space exploration, plus preprocessing breadth (MSC/SNV, winsorising), robust PCA, adaptive model update, and EDA plot ideas. The July upload was byte-identical to the notebook already covered. https://claude.ai/code/session_01CErviAK6p4MuZiVWsQT3vR --- docs/IDEAS_FROM_2009_NOTES.md | 80 ++++++++++++++++++++++++++++------- 1 file changed, 64 insertions(+), 16 deletions(-) diff --git a/docs/IDEAS_FROM_2009_NOTES.md b/docs/IDEAS_FROM_2009_NOTES.md index b7c3950..851adb4 100644 --- a/docs/IDEAS_FROM_2009_NOTES.md +++ b/docs/IDEAS_FROM_2009_NOTES.md @@ -1,15 +1,41 @@ # Ideas extracted from the 2009 design notebooks -Two scanned notebooks (July 2009 and 11–18 September 2009) contain the design -for an earlier workflow-based chemometrics tool: data tables organised as -blocks × stacks (train / test / check), transformation chains, lazily-copied -workflow DAGs, triggers/alarms, and multiblock models. This document records -what in those notes is superseded by ScorePilot's current architecture and -what is worth building on. +Three scanned notebooks (2 June 2009, July 2009, and 11–18 September 2009) +contain the design for an earlier workflow-based chemometrics tool: data +tables organised as blocks × stacks (train / test / check), transformation +chains, lazily-copied workflow DAGs, triggers/alarms, multiblock models, and a +worked treatment of latent-variable model inversion for product design. This +document records what in those notes is superseded by ScorePilot's current +architecture and what is worth building on. ## What the notes describe (condensed) -- **Data-table (DT) model**: a table = blocks (X, Y, Z…) × stacks (train / +### June 2009 notebook + +- **Model inversion for new product/process design** (after Jaeckle & + MacGregor, AIChE J 1998, v44) — nine pages of worked math. Direct OLS + inversion of `Y = XB` fails (collinearity, more variables than runs), so: + build PCA/PCR on X, regress quality variables Y onto the scores + (`B = (TᵀT)⁻¹TᵀY`), then invert a desired spec `y_des → t_des → x_new = + P_A t_des` — an "on-the-model-plane" inversion with SPE = 0. Three cases by + the count of quality specs K vs components A: K > A (least squares), + K = A (direct), K < A (generalised inverse; a **null space** of equivalent + solutions — move along it to enumerate candidate operating conditions that + all predict the same Y). Validity checks throughout: `t_des` inside the T² + limit; and a **Phase A feasibility pre-check** — PCA on the Y-space of + previous runs, requiring `y_des` to show low T² and low SPE there ("have we + made something like this before?") before attempting inversion. +- **v1 release checklist / course outlines**: preprocessing including MSC, + SNV, smoothing, trimming, winsorising; robust PCA; EDA aids (scatter-plot + matrix, correlation map, sparklines, colour-coded score plots); SIMCA / + PLS-DA; multiblock; "Tools: Inversion, Contributions, Optimization"; + adaptive model update / model maintenance; data import (Excel/CSV/XML, + OPC/PI connectivity). +- Website/app-store/business planning (not relevant here). + +### July and September 2009 notebooks + +- **DT data model**: a data table = blocks (X, Y, Z…) × stacks (train / test / check) × arrays; a `split()` operation with rule dicts and wildcards (``, ``, ``); variable-label range parsing (`Var001-099`). - **Workflow DAG**: data → t₀…tₙ → model → plot nodes, where t₀ is always an @@ -50,9 +76,9 @@ not a source of mechanism. there is no way to score them against the resulting model. Add a core "apply model to data" function (project rows → scores, T², SPE, predictions) and surface excluded rows as an automatic pseudo-validation - set on each variant, recorded in the Logbook. This is the strongest idea - in the notes and directly serves the Logbook's "validation history" - promise. + set on each variant, recorded in the Logbook. This is the strongest + workflow idea in the notes and directly serves the Logbook's "validation + history" promise. 2. **Named test sets + a predict endpoint** (the train/test/check "stacks"). ScorePilot currently has no out-of-sample mechanism — only @@ -61,25 +87,47 @@ not a source of mechanism. functional gap the notes point at, and item 1 falls out of the same core function. -3. **The tools-vs-transformations invariant**: "transformations never alter +3. **Model inversion for product/process design** (June notebook). Given a + desired quality spec `y_des`, invert a fitted PLS/PCR model to propose + operating conditions `x_new`, with the Phase A feasibility pre-check + (`y_des` must show low T²/SPE in a Y-space PCA of past products) and + null-space exploration when specs are fewer than components. A + distinctive, high-value analysis tool that builds directly on the + projection function from items 1–2. The numerics belong upstream in + `process_improve` per the CLAUDE.md rule. + +4. **The tools-vs-transformations invariant**: "transformations never alter shape; tools may (lag, batch feature extraction, DOE)." A clean design rule for evolving `PreprocessingSpec`: current operations are all shape-preserving; shape-changing steps (lagged variables for dynamic PLS, batch unfolding) should be a distinct, explicitly ordered stage. -4. **Monitoring triggers/notifiers**: condition functions with nested AND/OR +5. **Preprocessing breadth from the v1 checklist**: MSC and SNV (spectral + data), smoothing, trimming, winsorising. Natural `VariableTransform` / + spec extensions once spectral datasets matter; robust PCA sits behind the + same door as a model-kind option. + +6. **Monitoring triggers/notifiers**: condition functions with nested AND/OR over model outputs (`model.output['T2'][-1] > limit → notify`). Out of scope for the core analysis loop, but a coherent future "monitoring mode" that maps directly onto the T²/SPE limits already computed. -5. **Model-kind roadmap**: MB-PCA / MB-PLS, SIMCA / PLS-DA, L-shaped data, +7. **Adaptive model update / model maintenance** (June notebook). Refitting + an existing model on newly accumulated data is already expressible as "new + dataset + same `PreprocessingSpec` → child variant"; the missing piece is + a one-step "refresh this model on dataset D" action that preserves lineage + in the Hangar. + +8. **Model-kind roadmap**: MB-PCA / MB-PLS, SIMCA / PLS-DA, L-shaped data, missing-data-tolerant fitting (today missing cells are only flagged in the quality report). Per CLAUDE.md, the numerics belong upstream in `process_improve`, not bespoke in `core/`. -6. **Small UX wins**: variable-range selection by name pattern - (`Var001-099`), and ``/`` row splits (venetian-blind style test - sets) in the grid / variant-creation flow. +9. **Small UX wins**: variable-range selection by name pattern + (`Var001-099`), ``/`` row splits (venetian-blind style test + sets), and EDA aids from the course notes — scatter-plot matrix, + correlation map, sparklines — all natural additions to the linked-plots + library. ## Design note worth recording