Skip to content

Refactor: decouple stage submit config from settingsState#206

Open
pcolt wants to merge 5 commits into
feat/pipeline-orchestrationfrom
refactor/decouple-stage-settings
Open

Refactor: decouple stage submit config from settingsState#206
pcolt wants to merge 5 commits into
feat/pipeline-orchestrationfrom
refactor/decouple-stage-settings

Conversation

@pcolt

@pcolt pcolt commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Note: merge first #205 than re-target to main before merging

It closes #203

  • Update CHANGELOG

Overview

Main goal — decouple stage submission from settingsState. The single-stage submit primitives (submitCoralStageRemote/submitExecutableStageRemote) still read settingsState.remote directly for run-mechanics inputs (coralBinaryPath, coralPluginPath, executablePath) instead of taking them as explicit arguments. That's a real coupling problem for the pipeline driver in particular: runPipelineRemote has no business knowing about the app's settings store, so any hidden settings read inside a primitive it calls leaks an implicit dependency into what should be a pure "submit this stage" call.

Follow-on idea — config becomes the complete, serializable argument bag for a single job/stage submit. Removing the settings reads only works if every run-mechanics field a submit needs is already sitting on config. Today it isn't: parametersFileName lives in two places (ExecutablePipelineStage top level and optionally ExecutableJobConfig) with disagreeing authority, and CoralJobConfig has no equivalent of executablePath at all (coral stages silently assume one shared install via settings). This PR widens the config types to close that gap, removes the settings reads from the primitives, and fixes every call site that relied on the old fallback.

Bundled fix — unify per-run working directories. Unrelated to the above but bundled into this PR since it touches the same submit primitives: rapid back-to-back single runs currently clobber the shared graph.json/job.sh (and, for local executable runs, parameters.json) because they all reuse one working directory. This PR also gives every run its own subdir.

Implementation plan

  • Step 1 — Widen config types: add coralBinaryPath/coralPluginPath to CoralJobConfig and executablePath/parametersFileName to ExecutableJobConfig as required fields; update buildBatchScript/buildExecutableBatchScript to read them from config.
  • Step 2 — Decouple submit primitives: remove the settingsState.remote spread from submitCoralStageRemote/submitExecutableStageRemote; all run-mechanics inputs come from config. IPC calls and jobIdMapState stay as-is — they are the execution mechanism, not settings reads.
  • Step 3 — Fix single-stage call sites: JobConfigModal and the exportAndEval* wrappers read paths from settingsState once at the UI boundary and pass them as explicit args — both remote and local dispatch paths, plus the DEFAULT_*_CONFIG() factories.
  • Step 4 — Move fields off the stage top level: remove executablePath/parametersFileName from ExecutablePipelineStage; update pipeline.svelte.ts, ExecutableStageNode, and the orchestrator to read/write them via config. Also capture coralBinaryPath/coralPluginPath at coral stage creation in PipelineCanvas (symmetric to the existing executable capture).
  • Step 5 — Unify per-run working directories: give every single remote run a unique run-<ts> subdir (same isolation pipeline stages already have), fixing the graph.json/job.sh clobber on rapid back-to-back runs. For local executable runs, wrap each in run-<jobId>/ with bare-named files (subdir not filename prefix, to preserve deal.II path-sniffing). Drop the dead workingDirectory recorded for local runs (local resolves via the in-process localRuns registry, not getJobWorkingDirectory).
  • Step 6 — Regression + tests: npm run check, npm run check:electron, npm test; pipelineOrchestrator.test.ts mocks sshMessages's submit/poll exports via vi.mock to test the topo-sort + submit-loop driver without Electron; manual remote + local verification via Docker Slurm.

Also

  • docs: CHANGELOG updated under "Pipelines" — the working-directory clobber fix (step 5) is the only user-visible behaviour change here; the rest is an internal refactor.

Test plan

  • npm run check and npm run check:electron pass with the widened config types and the decoupled primitives.
  • npm test passes; pipelineOrchestrator.test.ts mocks sshMessages's submit/poll exports and exercises the driver (submit order, dependency wiring, terminal-state reporting) without Electron.
  • Manual: single-stage coral run (remote, Docker Slurm) still submits and polls correctly with the path now sourced from the modal.
  • Manual: single-stage executable run still works with executablePath passed explicitly from settings at the call site.
  • Manual: 3-stage pipeline (coral → executable → coral) with different MPI per stage submits with correct --dependency and distinct #SBATCH resources, identical to pre-refactor behaviour.
  • Manual: two single coral runs fired back-to-back (remote) land in distinct run-<ts> dirs and each job runs its own graph.json (no clobber); logs and node status still resolve in JobsTable for both.
  • Manual: two single executable runs (local) with the same parametersFileName land in distinct run-<jobId>/ subdirs with bare-named params files; logs resolve for both.
  • git grep -n "settingsState" src/lib/orchestration/ returns no matches except pipelineOrchestrator.ts's own use of settingsState.remote.workingDirectory for the pipeline base dir.
  • git grep -n "settingsState" src/lib/utils/sshMessages.ts shows no reads inside submitCoralStageRemote/submitExecutableStageRemote.

- refactor: widen CoralJobConfig/ExecutableJobConfig with coralBinaryPath/coralPluginPath/executablePath/parametersFileName as required fields, so config is the complete argument bag for a stage submit
- refactor: submitCoralStageRemote/submitExecutableStageRemote and buildBatchScript no longer read settingsState.remote directly — every run-mechanics input flows in via config
- refactor: JobConfigModal reads settingsState once at the UI boundary and passes paths explicitly, for both remote and local dispatch
- refactor: move executablePath/parametersFileName off ExecutablePipelineStage onto its config; PipelineCanvas captures coralBinaryPath/coralPluginPath at coral stage creation (symmetric to the existing executable capture)
- fix: give every single remote run its own run-<ts> working subdirectory, and every local executable run its own run-<jobId> subdirectory, fixing a clobber bug on rapid back-to-back runs
- test: pipelineOrchestrator.test.ts mocks sshMessages's submit/poll exports via vi.mock to test the topo-sort + submit-loop driver without Electron
- docs: update CHANGELOG
- docs: add issue links for #205/#206 and note #206's orchestration-decoupling direction
@pcolt
pcolt requested a review from luca-heltai July 17, 2026 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant