Feature FullAutoS0A

.github/workflows/lint.yml

@@ -0,0 +1,42 @@
+name: Lint
+
+# Lint runs on a cheap GitHub-hosted runner (the unit-test / cluster jobs use a
+# self-hosted Windows runner with lab hardware + Confluence secrets, which is
+# overkill for a style check and unavailable on forks). black and flake8 read
+# their config from pyproject.toml ([tool.black] / [tool.flake8] via
+# Flake8-pyproject); versions are pinned to match .pre-commit-config.yaml so CI
+# and the local pre-commit hook agree.
+
+on:
+  push:
+    branches:
+      - master
+      - develop
+  pull_request:
+    branches:
+      - master
+      - develop
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install linters
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install black==26.5.1 flake8==7.1.1 Flake8-pyproject
+
+      - name: black --check
+        run: black --check .
+
+      - name: flake8
+        run: flake8 .

.gitignore

@@ -162,6 +162,11 @@ cython_debug/
 
 .DS_Store
 
+# Claude Code: keep local settings and personal notes out of git, but the
+# shared standing context in CLAUDE.md IS tracked (do not ignore it).
+.claude/
+CLAUDE.local.md
+
 # Generated by setuptools-scm at install time — do not commit
 picasso_workflow/_version.py
 
@@ -171,5 +176,4 @@ temp/*
 
 spinna*
 nn_redistribution.py
-CLAUDE.md
-config.yaml
+config.yaml

.vscode/settings.json

@@ -17,5 +17,5 @@
     "picasso_workflow"
   ],
   "python.testing.unittestEnabled": false,
-  "python.testing.pytestEnabled": true
+  "python.testing.pytestEnabled": true,
 }

CHANGELOG.md

@@ -0,0 +1,79 @@
+# Changelog
+
+All notable changes to picasso-workflow are documented here.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+Versions are derived from git tags by setuptools-scm, so entries are collected
+under `[Unreleased]` until a tag is cut.
+
+This file was started after v0.5.6; earlier history is in the git log.
+
+## [Unreleased]
+
+### Added
+
+- Confluence error reports now identify the failing module by index and
+  name in a heading, list the parameters it was called with, name the
+  innermost picasso-workflow stack frame, and link the module result
+  folder and the preceding module's results. Previously only the
+  exception message and traceback were posted, so diagnosing a failure
+  meant guessing which parameter value had caused it.
+- Failed modules are recorded in `WorkflowRunner.yaml` with their
+  index, parameters, exception type, message and traceback.
+- Per-channel parameters: module parameters can differ between channels
+  of an aggregation workflow via a `("$$map", "<column>", <default>)`
+  command backed by a column in `single_dataset_tileparameters`. The
+  GUI shows the resolved per-channel values beneath the parameter row
+  and round-trips the dataset table through the generated script.
+
+### Changed
+
+- The Zeiss `.czi` reader (`aicsimageio` + `aicspylibczi` + `fsspec`)
+  moved from the base dependencies to an optional `formats` extra
+  (`pip install "picasso_workflow[formats]"`). aicspylibczi has no
+  aarch64 wheels and source-builds via a C++/cmake toolchain, and
+  aicsimageio dragged an old imagecodecs that failed to build on the
+  py3.10 arm64 container, so a bare `pip install -e .` now resolves
+  entirely from wheels. The `convert_zeiss_movie` module raises a clear
+  ImportError pointing at the extra when the reader is absent; no other
+  workflow is affected.
+- `estimate_density_from_neighbordists` now validates the
+  `[min_dist, max_dist]` window per neighbour order and fails with a
+  message naming the parameter and the surviving counts, e.g.
+  `min_dist=50, max_dist=300 leaves 0 of 530 k=4 nearest-neighbour
+  distances (observed range 338.3-2.264e+04)`.
+
+  BEHAVIOUR CHANGE: runs in which some neighbour order had no distances
+  inside the window previously fitted on the remaining orders and now
+  raise instead. Workflows believed to be healthy may start failing;
+  widen the window or reduce the number of neighbours fitted.
+- Tracebacks are posted inside a Confluence code macro, so their line
+  structure is preserved. They were previously HTML-escaped without a
+  wrapper and reflowed into a single unreadable paragraph.
+- CI now runs a `black --check` + `flake8` lint job on a GitHub-hosted
+  runner (`.github/workflows/lint.yml`), so style regressions are caught
+  in CI and not only by the local pre-commit hook. Linter versions are
+  pinned to match `.pre-commit-config.yaml`.
+
+### Fixed
+
+- An exception other than `AutoPicassoError` escaped
+  `WorkflowRunner.run()` before `save()`, so the failing module left
+  no trace in `WorkflowRunner.yaml`.
+- A module raising `AutoPicassoError` on the first iteration of
+  `run()` raised `UnboundLocalError` on `success`, masking the
+  real error.
+- `call_module` re-raised a `copy.copy()` of the exception, which
+  drops `__traceback__`; the propagated error stopped at the re-raise
+  rather than pointing at the code that failed.
+- `fit_csr` used truthiness to detect optional parameters, so
+  `min_dist=0`, `max_dist=0` and `bkg_fraction=0` were silently
+  replaced by defaults.
+- `nndistribution_from_csr` raised "zero-size array to reduction
+  operation maximum" on an empty distance array instead of returning an
+  empty result.
+- The GUI's "Remove Dataset" button did nothing, silently, when a
+  channel was selected, and its buttons were ordered inconsistently.
+- Commands could not be assigned to nested (dict) sub-parameters: the
+  `cmd` dialog raised `KeyError` on accept, and a nested command
+  value was discarded when the workflow was reloaded.

CLAUDE.md

@@ -0,0 +1,175 @@
+# CLAUDE.md — picasso-workflow
+
+Standing context for Claude Code (claude.ai/code) working in this repo. Read
+this first, then the design doc and the cross-repo pointers below. picasso-workflow
+is one repo in the **DNA-PAINT full-automation** stack (siblings: PycroFlow,
+monet, picasso, picasso-registry, picasso-agent).
+
+## What this repo is
+
+picasso-workflow automates and documents DNA-PAINT **analysis** workflows built
+on `picassosr` (the picasso localization/clustering library). It runs a picasso
+pipeline as an ordered list of named **modules** — two workflow kinds:
+
+- **Single-dataset** — one movie is loaded, localized, drift-corrected,
+  clustered, etc. (`WorkflowRunner`).
+- **Aggregation / investigation** — many datasets each run the same
+  single-dataset modules, then are combined by aggregation modules
+  (`AggregationWorkflowRunner`). Module parameters can vary per channel via a
+  `("$$map", "<column>", <default>)` command backed by a column in
+  `single_dataset_tileparameters` (see the README "Per-channel parameters").
+
+Each run documents itself to **Confluence** (`ConfluenceReporter`) and/or a
+local **HTML report** (`html_reporter`). A **PyQt6 GUI** (`picasso-workflow-gui`)
+builds, edits, and launches workflows, and can generate SLURM scripts to submit
+them to a cluster. The importable package is `picasso_workflow/`.
+
+See `README.md` for the full feature list, installation, per-channel parameters,
+the four-tier testing strategy, the SLURM cluster runner, CI, and the release
+workflow — this file is the short standing context, not a duplicate of it.
+
+## Current branch
+
+`feature-FullAutoS0A` — PRs target `master`. (Upstream also maintains a
+`develop` branch; the release workflow merges `develop` → `master` and tags on
+`master` — see README "Releasing".)
+
+## Commands
+
+```bash
+# Install (core; pulls picassosr and the PyQt6 GUI deps). Do this in a
+# python>=3.10 env (README recommends a conda env named picasso-workflow).
+pip install -e .
+pip install -e ".[cluster]"    # adds mpi4py for the MPI aggregation path
+
+# For a local development picasso, install it first, then picasso-workflow:
+#   cd /path/to/picasso && pip install -r requirements.txt
+
+# Run the GUI (console script and module form are equivalent)
+picasso-workflow-gui
+python -m picasso_workflow.gui
+
+# Test (a bare `pytest` is UNIT-ONLY — the `integration` mark is deselected by
+# default in [tool.pytest.ini_options] addopts). See README "Testing".
+pytest                         # tiers 1+2: unit + template validation (fast)
+pytest -v                      # verbose, still unit-only
+pytest -k <name>               # a single test by keyword
+pytest -m integration          # tier 3: real picasso pipeline (needs picassosr)
+pytest -m "integration and real_data"   # tier 4: real acquired data (PW_TEST_DATA_DIR)
+pytest -m ""                   # clear the default filter: run everything
+
+# Lint / format — pre-commit owns black + flake8 (both run in pre-commit's own
+# isolated envs; they are NOT direct package deps, so `pre-commit` is the
+# canonical path rather than a bare `black`/`flake8`).
+pre-commit install             # once, to run hooks on every commit
+pre-commit run --all-files     # run all hooks (trailing-ws, eof, yaml, black, flake8) now
+```
+
+## Conventions (aligned across the DNA-PAINT automation repos)
+
+This repo already matches the aligned target — no S0A-2 migration pending here.
+
+- **Style:** Black, line length **79** (Black owns line wrapping). Config in
+  `pyproject.toml [tool.black]`.
+- **Lint:** flake8 via **Flake8-pyproject**, config in
+  `pyproject.toml [tool.flake8]` with `extend-ignore = E203, E501, W503` — E501
+  is ignored because **Black owns line length** (it already wraps code; long
+  strings/comments/HTML it can't split are intentional). `max-line-length = 88`
+  there is informational only. Some experimental modules and the test-data
+  fixtures are `extend-exclude`d.
+- **Pre-commit:** `pre-commit install` once; hooks run trailing-whitespace,
+  end-of-file-fixer, check-yaml, check-added-large-files, **black**, and
+  **flake8** (Flake8-pyproject). isort / bandit / mypy are intentionally **not**
+  in the pre-commit run (see `.pre-commit-config.yaml`).
+- **Versioning:** **setuptools-scm** — **the tag IS the version**; there is no
+  version string to edit by hand. It writes `picasso_workflow/_version.py`
+  (gitignored, importable as `picasso_workflow.__version__`); fallback outside a
+  git checkout is `0.3.3.dev0`. Release = merge to `master`, then
+  `git tag vX.Y.Z && git push origin vX.Y.Z` (format `vMAJOR.MINOR.PATCH`).
+- **Changelog on release:** the changelog is `CHANGELOG.md` at the repo root
+  (Keep a Changelog, in Markdown, with `### Added` / `### Changed` / `### Fixed`
+  subsections — matching monet / PycroFlow / picasso-registry). Add an entry
+  under the top **`## [Unreleased]`** section in every PR; at release, promote
+  `[Unreleased]` to a dated, tagged section (e.g. `## [1.2.3] - YYYY-MM-DD`). Because the version comes from git tags, the changelog
+  is the human-facing record of what each tag contains.
+- **Packaging:** `pyproject.toml` only (no `setup.py` / `setup.cfg`). Runtime
+  deps and the `[cluster]` extra live there.
+- **Tests:** write/extend tests with every change; keep every tier green.
+  Picasso is fully mocked in the unit tier so it runs anywhere with no data or
+  network. Adding a workflow module touches
+  `util.AbstractModuleCollection`, `analyse.AutoPicasso`,
+  `confluence.ConfluenceReporter`, and the matching `tests/test_*` files; if a
+  snapshotted template references it, re-run `python tools/snapshot_templates.py`
+  (see README "Adding a new workflow module").
+
+## Architecture (short)
+
+`workflow.py` holds the orchestrators (`WorkflowRunner`,
+`AggregationWorkflowRunner`): each reads a module list, calls the corresponding
+analysis method, and records results/failures to `WorkflowRunner.yaml`.
+`analyse.py` (`AutoPicasso`) implements the actual picasso-backed modules;
+`util.py` provides `AbstractModuleCollection` (the module contract),
+`ParameterTiler` / `ParameterCommandExecutor` (the `$`/`$$map` per-channel
+parameter machinery), and typing helpers. `standard_singledataset_workflows.py`
+and `standard_aggregation_workflows.py` are the predefined recipes;
+`modulespec.py` is the `ModuleSpec` annotation/validation layer;
+`picasso_outpost.py` holds picasso-adjacent code not yet upstream. Reporting:
+`confluence.py` (`ConfluenceReporter` / `ConfluenceInterface`) and
+`html_reporter.py`. `_launcher.py` is the `picasso-workflow-gui` entry point;
+`__init__.py` configures loguru logging and deep-merges `config.yaml`
+(package → site → per-user). Full module map in `README.md`.
+
+## Standing pointers
+
+Paths so later sessions can `@`-reference them. Repo root is
+`/workspaces/DNA-PAINT-FullAutomation/repositories/picasso-workflow`; the shared
+workspace root is `/workspaces/DNA-PAINT-FullAutomation`.
+
+**Live (resolve today):** the shared planning docs live in `../../planning/`
+(workspace `planning/` folder); start from its document map.
+- Document map / reading order: `../../planning/README.md`
+- Design doc — recommendation & roadmap (strategy, prioritized initiatives #1–#9,
+  work packages WP-1–WP-16, Parts I–X):
+  `../../planning/DNA-PAINT_Automation-Recommendation.md`
+- **Playbook** — Claude Code implementation playbook (operating model, Step 0
+  foundations, style/repo alignment, gated dependency-ordered work orders):
+  `../../planning/DNA-PAINT_ClaudeCode-Implementation-Playbook.md`
+- **Work-order briefs** — self-contained, paste-ready briefs (S0A-1, S0A-2,
+  S0B-1/2, WP-1…WP-16); this task is S0A-1:
+  `../../planning/DNA-PAINT_Work-Order-Briefs.md`
+- **Progress tracker** — tick-off worksheet + gates for the work orders:
+  `../../planning/DNA-PAINT_Implementation-Progress-Tracker.md`
+- Module-annotations reference (the `ModuleSpec` layer, data dependencies,
+  capability registry):
+  `../../planning/picasso-workflow_Module-Annotations_Reference.md`
+- Dev-environment setup (OrbStack dev-container):
+  `../../planning/DNA-PAINT_ClaudeCode-DevEnvironment.md`
+- Sibling repo standing context:
+  - PycroFlow (experiment orchestration): `../PycroFlow/CLAUDE.md`
+  - monet (laser-power calibration/control): `../monet/CLAUDE.md`
+  - picasso-registry (provenance/metrics DB; **owns the schema/API contract**):
+    `../picasso-registry/CLAUDE.md`
+  - picasso-agent (agentic layer): `../picasso-agent/CLAUDE.md`
+  - picasso (upstream localization/clustering library — `picassosr`; no CLAUDE.md
+    yet): `../picasso`
+- Sibling repo roots: `../PycroFlow`, `../monet`, `../picasso`,
+  `../picasso-registry`, `../picasso-agent`
+
+**Forthcoming (planned; not yet in-tree — do not treat as resolvable):**
+- Cross-repo contracts (after S0B): the picasso-registry OpenAPI spec + generated
+  client and the shared schemas (metric-vector, workflow-YAML,
+  `localize_frames` signature, picasso-workflow `ModuleSpec`) — these will be
+  owned by picasso-registry; see `../picasso-registry/CLAUDE.md` and work orders
+  S0B-1 / S0B-2 in the briefs above. picasso-workflow's `ModuleSpec`
+  (`modulespec.py`) is part of that contract set.
+
+## Notes for editing
+
+- `.gitignore` **tracks this `CLAUDE.md`** (it is not ignored) but keeps
+  `.claude/` and `CLAUDE.local.md` ignored (local settings / personal notes) —
+  keep it that way.
+- `picasso_workflow/_version.py` is generated by setuptools-scm and gitignored —
+  never commit or hand-edit it.
+- `spinna_mle.py`, `spinna_mle_2.py`, and `nn_redistribution.py` are isolated
+  experimental modules (not imported by the package) and are flake8-excluded;
+  don't expect them to be linted.