some more features

CLAUDE.md

@@ -83,6 +83,9 @@ clean-up-workspaces.py          # Cron-style cleanup of stale workspace director
 docker/entrypoint.sh            # Canonical container entrypoint (copied to /app/entrypoint.sh in the image);
 entrypoint.sh                   #   auto-detects a read-only root (apptainer/HPC) and moves runtime state to /tmp
 gdpr_consent/                   # GDPR consent Streamlit component (prebuilt JS bundle)
+mq_dir_upload/                   # JS folder-picker shim: filters a picked MaxQuant folder to the PTXQC-relevant
+                                #   files and injects ONLY those into a native st.file_uploader (HTTP transport,
+                                #   scales to GBs; never uses setComponentValue). Python wrapper: src/common/mq_dir_upload.py
 docs/                           # User/developer + deployment docs
 k8s/                            # Kustomize: base/ + components/memory-tier-{low,high}/ + overlays/prod/
 Dockerfile_simple               # The single shipped image (linux/amd64): pyOpenMS (pip) + R + PTXQC + pandoc
@@ -110,7 +113,7 @@ Pages are registered in `app.py` under named sidebar sections (the dict key is t
 
 The whole app is one `WorkflowManager` subclass implementing the four template methods. This is the example to copy when changing app behavior:
 
-- `upload()` — a reactive **Data type** selector (`MaxQuant directory` / `MaxQuant files` / `mzTab file`) drives which `self.ui.upload_widget(...)` is shown (folder upload vs. multi-file vs. single).
+- `upload()` — a reactive **Data type** selector (`MaxQuant directory` / `MaxQuant files` / `mzTab file`) drives which uploader is shown. `MaxQuant files` / `mzTab file` use `self.ui.upload_widget(...)` (multi-file / single). `MaxQuant directory` uses `src/common/mq_dir_upload.filtered_directory_upload`: the `mq_dir_upload` JS **shim** lets the user pick a whole folder but injects **only** the PTXQC-relevant files into a native `st.file_uploader` (so multi-GB `allPeptides.txt` etc. never leave the user's machine — the stock directory uploader filters by extension only, which wouldn't). The native uploader transfers over HTTP (scales; a base64-over-websocket component value does **not** — it silently drops at a few hundred MB, the size real evidence/msms files reach). The wrapper mirrors the uploaded files into the same `input-files/txt-files/` dir the widget would, so `execution()`/`_gather_files` are unchanged. Note: `Workflow.show_file_upload_section()` is overridden to drop the template's generic "⬇️ Download files" button.
 - `configure()` (`@st.fragment`) — either an "upload a YAML config" path, or the manual advanced-settings grid built from the module-level `NUMBER_WIDGETS` list + a contaminants text field + a metrics multiselect populated from the installed PTXQC version. Degrades gracefully (a warning, defaults only) when R/PTXQC is unavailable.
 - `execution()` — gathers inputs, writes a JSON run-config (via `ptxqc_config.build_run_config`), then runs **R**: `self.executor.run_command(["Rscript", cfg.RUNNER, "run", "--config", ..., "--in", ..., "--type", "maxquant"|"mztab", "--out", ...])`. Appends a row to the shared usage log afterward.
 - `results()` (`@st.fragment`) — reads the wrapper's `ptxqc_result.json`, embeds the report HTML via `st.components.v1.html(...)` (with an injected "open in new tab" button), and offers PDF/HTML/YAML/log downloads.

Dockerfile_simple

@@ -119,6 +119,7 @@ COPY content/ /app/content
 COPY docs/ /app/docs
 COPY example-data/ /app/example-data
 COPY gdpr_consent/ /app/gdpr_consent
+COPY mq_dir_upload/ /app/mq_dir_upload
 COPY hooks/ /app/hooks
 COPY src/ /app/src
 COPY utils/ /app/utils

README.md

@@ -35,6 +35,57 @@ long-term maintainability.
 
 A hosted instance is available at **[ptxqc.webapps.openms.org](https://ptxqc.webapps.openms.org)**.
 
+## 💻 Run locally (without Docker)
+
+You can run the app straight from a checkout. Two layers are involved: the **Python/Streamlit**
+front end and the **R/PTXQC** engine that actually builds the reports.
+
+### 1. Python front end (minimum to launch the app)
+
+```bash
+git clone https://github.com/BioinformaticsSolutionCenter/PTXQC-web.git
+cd PTXQC-web
+python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
+pip install -r requirements.txt
+streamlit run app.py                                # opens http://localhost:8501
+```
+
+This is enough to launch every page. The committed `settings.json` runs in **local mode**
+(single user, no captcha, no Redis), so nothing else is required to start.
+
+> **Note:** `pip install` only gives you **pyOpenMS** — it does **not** install R or PTXQC. Without
+> them the app still loads and degrades gracefully: the *Configure* page warns, the live metric
+> list is empty, and actually running a report fails. To generate reports, install the R engine
+> below.
+
+### 2. R engine (needed to generate reports)
+
+Install **R** (4.x) and **pandoc** (used for the HTML report), then install the PTXQC package and
+the two helper packages the runner uses:
+
+```bash
+Rscript -e 'install.packages(c("PTXQC", "jsonlite", "yaml"), repos = "https://cloud.r-project.org")'
+```
+
+- **Windows / macOS** get precompiled CRAN binaries, so this is quick and needs no extra system
+  libraries. Install R from <https://cran.r-project.org> and pandoc from
+  <https://pandoc.org/installing.html> (or `winget install RProject.R pandoc` /
+  `brew install r pandoc`).
+- **Linux** compiles PTXQC's dependencies from source unless you use precompiled binaries. The
+  fast path (same one the Docker build uses) is Posit Public Package Manager plus the system
+  `-dev` libraries — see the `install.packages` step and the `apt-get install` list in
+  [`Dockerfile_simple`](Dockerfile_simple) for the exact repository URL and package names.
+
+Make sure `Rscript` is on your `PATH` (the app shells out to `Rscript src/ptxqc_runner.R …`).
+Verify the engine is visible to the app with:
+
+```bash
+Rscript src/ptxqc_runner.R default-config --out /tmp/ptxqc-default.yaml
+```
+
+If that writes a YAML file, the *Configure* page will show the live metric list and report
+generation will work.
+
 ## 🐳 Run with Docker
 
 The app ships as a single prebuilt image (linux/amd64) on the GitHub Container Registry, so you

app.py

@@ -11,10 +11,10 @@
 if __name__ == '__main__':
     pages = {
         str(st.session_state.settings["app-name"]): [
-            st.Page(Path("content", "ptxqc_upload.py"), title="Upload Data", icon="📁"),
-            st.Page(Path("content", "ptxqc_configure.py"), title="Configure", icon="⚙️"),
-            st.Page(Path("content", "ptxqc_run.py"), title="Create Report", icon="🚀"),
-            st.Page(Path("content", "ptxqc_results.py"), title="Report", icon="📊"),
+            st.Page(Path("content", "ptxqc_upload.py"), title="1. Upload Data", icon="📁"),
+            st.Page(Path("content", "ptxqc_configure.py"), title="2. Configure", icon="⚙️"),
+            st.Page(Path("content", "ptxqc_run.py"), title="3. Create Report", icon="🚀"),
+            st.Page(Path("content", "ptxqc_results.py"), title="4. Report", icon="📊"),
         ],
         "Info": [
             st.Page(Path("content", "help.py"), title="Help", icon="❓"),

content/ptxqc_configure.py

@@ -6,3 +6,8 @@
 wf = Workflow()
 
 wf.show_parameter_section()
+
+# Configuration is optional — always offer the link to step 3.
+import streamlit as st
+st.divider()
+st.page_link("content/ptxqc_run.py", label="Next: **3. Create Report**", icon="🚀")

content/ptxqc_run.py

@@ -6,3 +6,9 @@
 wf = Workflow()
 
 wf.show_execution_section()
+
+# Once a report exists, link to step 4 to view it.
+if wf.has_report():
+    import streamlit as st
+    st.success("✅ Report ready. Next step:")
+    st.page_link("content/ptxqc_results.py", label="**4. Report**", icon="📊")

content/ptxqc_upload.py

@@ -6,3 +6,9 @@
 wf = Workflow()
 
 wf.show_file_upload_section()
+
+# Step 1 done once data is staged → offer the link to step 2.
+if wf.has_inputs():
+    import streamlit as st
+    st.success("✅ Data uploaded. Next step:")
+    st.page_link("content/ptxqc_configure.py", label="**2. Configure**", icon="⚙️")

docs/adr/0001-gui-tests-drive-real-navigation.md

@@ -0,0 +1,23 @@
+# GUI tests drive the real `app.py` navigation shell
+
+`test_gui.py` boots the actual app via `AppTest.from_file("app.py")` (which runs
+`st.navigation`) and then `switch_page`s to each page, instead of loading each
+page standalone with `AppTest.from_file("content/<page>.py")`.
+
+We do this because `st.page_link` resolves its target through the page registry
+that `st.navigation` builds. A standalone page run never calls `st.navigation`,
+so `st.page_link` raises `KeyError('url_pathname')` — a failure that only exists
+in the test, not in production. The tempting "fix" is to wrap every `page_link`
+in `try/except`, but that both papers over a test that isn't running pages the
+way they actually run *and* silently swallows a genuinely broken page link (a
+typo'd target passes CI green). Driving `app.py` + `switch_page` runs pages in
+their real navigation context, so `page_link` works without guards and a broken
+link fails the test.
+
+## Consequences
+
+- No defensive `try/except` around `st.page_link` in the `content/` pages.
+- Do **not** "simplify" `test_gui.py` back to per-page `AppTest.from_file` — it
+  reintroduces the `KeyError`, the guards, and the silent bug-hiding.
+- The fixture must inject `settings["test"] = True` (captcha bypass) and a
+  `workspace` secret, since it now goes through the app's real entrypoint.

docs/adr/0002-keep-template-layer-app-agnostic.md

@@ -0,0 +1,20 @@
+# Keep the inherited template layer (`src/workflow/`) app-agnostic
+
+`src/workflow/` is vendored from the OpenMS streamlit-template and is merged from
+the `template/main` remote periodically. App-specific behaviour must **not** be
+added to those files; it lives in the app layer (`src/Workflow.py`, `content/`,
+`src/*.py`). Otherwise every template sync produces conflicts and the shared
+framework accumulates one app's concerns.
+
+Concretely: report-generation preconditions such as "is `Rscript` present?" are
+guarded in `Workflow.execution()` (app layer, with an app-appropriate "install R
+/ use Docker" message), **not** in `CommandExecutor.run_command()`. A `try/except`
+that caught a missing executable and logged a PTXQC/R-specific message was removed
+from `CommandExecutor.py` for this reason — it was both redundant with the
+`execution()` guard and a source of divergence from `template/main`.
+
+## Consequences
+
+- Precondition checks and user-facing error messages belong in the app layer.
+- Changes under `src/workflow/` should be limited to what can be upstreamed to the
+  template; keep these files diff-clean against `template/main` where possible.

mq_dir_upload/README.md

@@ -0,0 +1,50 @@
+# mq_dir_upload — MaxQuant folder-picker shim
+
+A tiny bidirectional Streamlit custom component that lets the user pick a whole
+MaxQuant `txt` folder but upload **only** the PTXQC-relevant files
+(`evidence.txt`, `msms.txt`, `msmsScans.txt`, `parameters.txt`,
+`proteinGroups.txt`, `summary.txt`, `mqpar.xml`). The large irrelevant MaxQuant
+outputs (`allPeptides.txt`, `peptides.txt`, …) never leave the user's machine.
+
+## How it works (and why it's a *shim*)
+
+Streamlit's built-in `st.file_uploader` directory mode filters by file
+**extension** only, so it can't drop `allPeptides.txt` (it's a `.txt`).
+Filtering by file *name* must happen in the browser — which needs JavaScript.
+
+The naive custom-component approach (read the files, base64-encode them, and
+return them to Python via `setComponentValue`) **does not scale**: that value
+travels over the websocket, which chokes on the hundreds-of-MB sizes real
+MaxQuant files reach (the value is silently dropped → nothing uploads). The
+relevant files (`evidence`/`msms`/`msmsScans`) are themselves the big ones, so
+this isn't avoidable by filtering.
+
+So this component is a **shim**, not an uploader. On a folder pick it:
+
+1. filters the browser `FileList` down to the allow-listed names (in JS), then
+2. injects *only* those `File` objects into the page's native
+   `st.file_uploader` (`DataTransfer` → set `input.files` → dispatch a bubbling
+   `change` event).
+
+Streamlit's native uploader then transfers them over its **HTTP** endpoint
+(handles GBs, shows a file list / sizes / progress / per-file remove). The shim
+**never calls `setComponentValue`** — the heavy bytes never touch the websocket.
+
+It must be a *declared* component (not `components.html`) so it runs
+**same-origin** and can reach `window.parent.document` to find that native
+uploader input. The Python side (`src/common/mq_dir_upload.py`) renders the
+native uploader (drag-drop area hidden via CSS), mirrors its files into the
+workspace dir, and passes `hidden=True` to the shim once files are staged
+(uploading more into an existing set is disallowed; the user clears first).
+
+## Files
+
+- `index.html` — loads `main.js`.
+- `main.js` — **hand-written, no build step.** Speaks the Streamlit iframe
+  `postMessage` protocol directly (`componentReady` / `render` /
+  `setFrameHeight`) — there is nothing to `npm build`. Edit `main.js` and reload.
+
+## Args (from Python)
+
+- `allowed`: list of relevant file names (matched case-insensitively on basename).
+- `hidden`: when true, the picker button is hidden (files already staged).

mq_dir_upload/index.html

@@ -0,0 +1,8 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <script defer src="main.js"></script>
+</head>
+<body></body>
+</html>

mq_dir_upload/main.js

@@ -0,0 +1,139 @@
+/*
+ * mq_dir_upload — MaxQuant "txt" folder-picker shim.
+ *
+ * The user picks a whole MaxQuant txt folder; this shim filters the browser's
+ * FileList down to the PTXQC-relevant files (by name) and injects ONLY those
+ * into the page's native st.file_uploader (via DataTransfer + a change event).
+ * Streamlit then uploads them over its HTTP endpoint — so large files work and
+ * the irrelevant files (allPeptides.txt, …) never leave the machine.
+ *
+ * Why a shim instead of returning the bytes: a custom component can only send
+ * data back over the websocket (setComponentValue), which chokes on hundreds of
+ * MB. The native uploader uses a separate HTTP transport that scales. This shim
+ * deliberately never calls setComponentValue — it only orchestrates the native
+ * widget. It must be a *declared* component (not components.html) so it runs
+ * same-origin and can reach window.parent.document.
+ */
+(function () {
+  "use strict";
+
+  function post(t, d) {
+    var m = { isStreamlitMessage: true, type: t };
+    for (var k in d) m[k] = d[k];
+    window.parent.postMessage(m, "*");
+  }
+  function h(px) { post("streamlit:setFrameHeight", { height: px }); }
+  function basename(n) { return (n || "").split(/[\\/]/).pop(); }
+
+  var built = false, allowed = [], hidden = false, label, input, status;
+
+  // The native st.file_uploader's file input, in the parent document. On the
+  // upload page there is exactly one file_uploader, so this is unambiguous.
+  function nativeInput() {
+    var pdoc = window.parent.document;
+    return (
+      pdoc.querySelector('[data-testid="stFileUploader"] input[type="file"]') ||
+      pdoc.querySelector('section[data-testid="stFileUploaderDropzone"] input[type="file"]') ||
+      pdoc.querySelector('input[type="file"]')
+    );
+  }
+
+  function onPick(ev) {
+    var all = Array.prototype.slice.call(ev.target.files || []);
+    if (!all.length) return;
+
+    var allow = {};
+    allowed.forEach(function (n) { allow[n.toLowerCase()] = true; });
+
+    // De-dupe by basename (a MaxQuant export may nest combined/txt).
+    var picked = {}, order = [];
+    all.forEach(function (f) {
+      var lc = basename(f.name).toLowerCase();
+      if (allow[lc] && !picked[lc]) { picked[lc] = f; order.push(lc); }
+    });
+    var matched = order.map(function (lc) { return picked[lc]; });
+
+    if (!matched.length) {
+      status.style.color = "#b00";
+      status.textContent =
+        "No PTXQC-relevant files found in that folder (scanned " + all.length + ").";
+      ev.target.value = "";
+      h(90);
+      return;
+    }
+
+    var nat = nativeInput();
+    if (!nat) {
+      status.style.color = "#b00";
+      status.textContent = "Could not find the upload target — please reload the page.";
+      h(90);
+      return;
+    }
+
+    // Hand the filtered File objects to the native uploader and let Streamlit
+    // upload them over HTTP. react-dropzone listens for a bubbling change event.
+    var dt = new DataTransfer();
+    matched.forEach(function (f) { dt.items.add(f); });
+    nat.files = dt.files;
+    nat.dispatchEvent(new Event("input", { bubbles: true }));
+    nat.dispatchEvent(new Event("change", { bubbles: true }));
+
+    status.style.color = "#444";
+    status.textContent = "Uploading " + matched.length + " file(s)…";
+    ev.target.value = "";
+    h(90);
+  }
+
+  function build() {
+    label = document.createElement("label");
+    label.textContent = "📁 Select MaxQuant txt folder";
+    label.style.cssText =
+      "display:inline-block;padding:.5rem 1rem;background:#ff4b4b;color:#fff;" +
+      "border-radius:.5rem;cursor:pointer;font-weight:600;line-height:1.4;" +
+      "font-family:'Source Sans Pro',sans-serif;";
+
+    input = document.createElement("input");
+    input.type = "file";
+    input.multiple = true;
+    input.setAttribute("webkitdirectory", "");
+    input.setAttribute("directory", "");
+    input.webkitdirectory = true;
+    input.style.display = "none";
+    input.addEventListener("change", onPick);
+    label.appendChild(input);
+
+    status = document.createElement("div");
+    status.style.cssText =
+      "margin-top:.5rem;font-size:.85rem;color:#444;line-height:1.4;" +
+      "font-family:'Source Sans Pro',sans-serif;";
+
+    document.body.style.margin = "0";
+    document.body.appendChild(label);
+    document.body.appendChild(status);
+    built = true;
+  }
+
+  function apply() {
+    if (!built) return;
+    // Hidden once files are staged: uploading more into an existing set is not
+    // allowed — the user must clear first.
+    label.style.display = hidden ? "none" : "inline-block";
+    status.textContent = hidden
+      ? ""
+      : "Only the PTXQC-relevant files are uploaded; everything else stays on your machine.";
+    h(hidden ? 1 : 90);
+  }
+
+  window.addEventListener("message", function (ev) {
+    var d = ev.data;
+    if (!d || d.type !== "streamlit:render") return;
+    if (!built) build();
+    var a = d.args || {};
+    if (Array.isArray(a.allowed)) allowed = a.allowed;
+    hidden = !!a.hidden;
+    apply();
+  });
+
+  post("streamlit:componentReady", { apiVersion: 1 });
+  h(90);
+})();