Feat: pipeline orchestrator MVP (visual DAG + remote Slurm execution)

CHANGELOG.md

@@ -4,6 +4,10 @@ See [docs/changelog-template.md](docs/changelog-template.md) for formatting your
 
 ## [Unreleased]
 
+### Pipelines
+
+- New pipeline editor: a higher-level visual DAG whose nodes are whole CORAL graphs or standalone executables, connected by ordering dependencies. A central "Pipeline editor" view (toggled from the top-right) lets you add stages (a CORAL graph from a file or the current canvas, or an executable), configure each stage's own resources (MPI nodes/tasks, time limit), and run the whole pipeline. On execution each stage is submitted as its own Slurm job chained after its predecessors, so stages run in the correct order, independent branches run in parallel, and the run continues even if the app is closed. Connections that would create a cycle are rejected. This first version runs in remote (Slurm) mode only and passes ordering between stages (no file hand-off yet).
+
 ## [1.4.1] - 2026-07-14
 
 ### Canvas-graph

CLAUDE.md

@@ -98,9 +98,21 @@ Stores use Svelte 5 runes (`.svelte.ts` files):
 - `auth.svelte.ts` - JWT token for coral-remote-server API. Methods: `setToken()`, `setUsername()`, `clearToken()`. Persisted in electron-store under `access_token` / `username`.
 - `settingsStore.svelte.ts` - User settings stored under a single `'settings'` key in electron-store. Exports named key constants (`SSH_PATH`, `URL_VISUALIZER`, `URL_REMOTE_SERVER`, `USE_MPI`) and a `settingsState` object with `getKey(key)` / `setKey(key, value)` methods.
 - `currentProjectStore.svelte.ts` - Current project metadata (`id`, `name`). Methods: `set()`, `clear()`, `updateName()`. Exports `ApiProject` interface for use in components.
-- `jobsStore.svelte.ts` - Slurm job tracking. `jobsState` has `isEmpty`/`oneOrLess` getters, `update()` refreshes from SSH. `jobIdMapState` maps Slurm scheduler IDs → internal incremental job IDs (used in touch-dir paths).
+- `jobsStore.svelte.ts` - Slurm job tracking. `jobsState` has `isEmpty`/`oneOrLess` getters, `update()` refreshes from SSH. `jobIdMapState` maps Slurm scheduler IDs → a `JobIdEntry` (`internalId` used in touch-dir paths, `backendKind`, and the per-job `workingDirectory`). The `workingDirectory` is the root for single runs and the per-stage subdirectory for pipeline stages; `getOutFileContent` / `getNodesExecutionStatus` resolve a job's `slurm-<id>.out` and `nodes-exec-status/` from it (via `getJobWorkingDirectory` / `getEntryByInternal`), so logs and node status work uniformly for both.
 - `toastsStore.svelte.ts` - Toast notifications. Use `toastState.add({ message, type })` to show a toast. Supports `'error'`/`'success'` types with auto-dismissal.
 - `dndStore.svelte.ts` - Drag-and-drop state (`dndNodeDataState.current`) for tracking which sidebar node is being dragged onto the canvas.
+- `viewModeStore.svelte.ts` - Which central editor is shown: `viewModeState.value` is `'graph' | 'pipeline'`, with `toggle()`. Transient (not persisted). `App.svelte` renders `PipelineCanvas` when `'pipeline'`, else the graph/parameters editor.
+- `pipeline.svelte.ts` - Transient store for the pipeline editor, holding the pipeline as xyflow `Node[]`/`Edge[]` (one node per stage, one edge per ordering dependency). Mirrors the get/set shape of `nodes.svelte.ts` (`getNodes`/`setNodes`/`getEdges`/`setEdges`) for canvas binding, plus `pipelineState` with `addCoralStage()`, `addExecutableStage()`, `updateStageData()`, `updateStageConfig()`, `setStageParameters()`, `removeStage()`, `clear()`, `toPipeline()`, and a `validation` getter.
+
+### Pipeline Orchestration
+
+The pipeline editor composes whole CORAL graphs and standalone executables as nodes of a higher-level DAG, where each edge means "target runs after source" (pure ordering — no data/artifact passing yet). This is separate from the CORAL canvas; coral itself runs one flat graph and cannot orchestrate across jobs.
+
+- `src/lib/types/pipelineTypes.ts` - `Pipeline`, `PipelineStage` (`coral` | `executable`, each carrying its own `CoralJobConfig`/`ExecutableJobConfig`), `PipelineEdge`, and the `StageData` stored on canvas nodes.
+- `src/lib/orchestration/executionOrder.ts` - Pure, unit-tested `resolveExecutionOrder(pipeline)` (Kahn topological sort; throws `PipelineCycleError` on cycles) and `parentsOf(stageId, edges)`. No app/IO dependencies — the contract a future extracted service would reimplement.
+- `src/lib/orchestration/pipelineOrchestrator.ts` - `runPipelineRemote(pipeline)`: topo-orders the DAG and submits each stage as its own Slurm job into `<workingDirectory>/pipeline-<ts>/stage-<id>`, chained via `--dependency=afterok` on its parents' Slurm ids, then polls all jobs concurrently for live feedback. Remote (Slurm) only for now.
+- Per-stage submit primitives live in `sshMessages.ts`: `submitCoralStageRemote()` / `submitExecutableStageRemote()` build+upload+submit a single stage (with `sbatch --parsable --kill-on-invalid-dep=yes [--dependency=afterok:…]`) and return the Slurm id without polling. The existing single-run remote functions route through these. `withMpiPlugin(network, useMpi)` injects the MPI plugin block; `jobPolling()` is exported for reuse.
+- UI: `PipelineCanvas.svelte` (a second `<SvelteFlow>` with a toolbar; an `isValidConnection` acyclicity guard replaces the coral canvas's type validation) plus the stage node components `nodes/CoralStageNode.svelte` and `nodes/ExecutableStageNode.svelte` (the per-stage config card is the node body).
 
 ### Node Type System
 

src/App.svelte

@@ -2,6 +2,7 @@
   import { SvelteFlowProvider } from '@xyflow/svelte'
   import FlowCanvas from './lib/components/FlowCanvas.svelte'
   import ParametersView from './lib/components/ParametersView.svelte'
+  import PipelineCanvas from './lib/components/PipelineCanvas.svelte'
   import Sidebar from './lib/components/layout/Sidebar.svelte'
   import SidebarButtons from './lib/components/layout/SidebarButtons.svelte'
   import JobsTable from './lib/components/layout/JobsTable.svelte'
@@ -12,8 +13,10 @@
   import ToastsWrapper from './lib/components/ToastsWrapper.svelte'
   import { settingsState } from './lib/stores/settingsStore.svelte'
   import { graphHistoryState } from './lib/stores/graphStack.svelte'
+  import { viewModeState } from './lib/stores/viewModeStore.svelte'
 
   let isCoralMode = $derived(settingsState.isCoralMode)
+  let viewMode = $derived(viewModeState.value)
   let executionLocation = $derived(settingsState.execution.location)
   let backendKind = $derived(settingsState.execution.backendKind)
 
@@ -63,7 +66,7 @@
   <ToastsWrapper></ToastsWrapper>
   <SvelteFlowProvider>
     <div id="app-container">
-      {#if isCoralMode}
+      {#if isCoralMode && viewMode === 'single'}
         <ButtonToggleMenu />
         <div id="sidebar-wrapper" bind:this={sidebarWrapperElem}>
           <Sidebar />
@@ -82,10 +85,21 @@
         </div>
         <div class="top-right-overlay">
           <ExecutionBadge location={executionLocation} {backendKind} />
+          <button
+            class="view-toggle"
+            onclick={() => viewModeState.toggle()}
+            title={viewMode === 'pipeline'
+              ? 'Open single stage view'
+              : 'Go to pipeline view'}
+          >
+            {viewMode === 'pipeline' ? 'Single stage' : 'Pipeline'}
+          </button>
           <ButtonToggleDarkMode />
         </div>
         <div class="flow-wrapper">
-          {#if isCoralMode}
+          {#if viewMode === 'pipeline'}
+            <PipelineCanvas />
+          {:else if isCoralMode}
             <FlowCanvas />
           {:else}
             <ParametersView />
@@ -167,4 +181,19 @@
     height: 100%;
     position: relative;
   }
+
+  .view-toggle {
+    background: var(--primary-color);
+    color: var(--ternary-color);
+    border: 1px solid var(--ternary-color);
+    border-radius: 8px;
+    padding: 0.4rem 0.7rem;
+    font-size: 0.8rem;
+    font-weight: bold;
+    cursor: pointer;
+  }
+
+  .view-toggle:hover {
+    border-color: var(--border-color-hover);
+  }
 </style>

src/lib/components/JobConfigModal.svelte

@@ -13,6 +13,7 @@
   import { parametersState } from '../stores/parametersStore.svelte'
   import { settingsState } from '../stores/settingsStore.svelte'
   import { toastState } from '../stores/toastsStore.svelte'
+  import { isValidSlurmTime, SLURM_TIME_HINT } from '../utils/slurmTime'
 
   interface Props {
     modalId: string
@@ -35,15 +36,8 @@
   )
   let totalProcesses = $derived(nodes * tasksPerNode)
 
-  // Slurm --time accepted formats: minutes | minutes:seconds | hours:minutes:seconds
-  // | days-hours | days-hours:minutes | days-hours:minutes:seconds | 0
-  // See: https://slurm.schedmd.com/sbatch.html#OPT_time
-  const SLURM_TIME_PATTERN =
-    /^(\d+|\d+:\d{2}(:\d{2})?|\d+-\d{2}(:\d{2}(:\d{2})?)?)$/
   let timeLimitError = $derived(
-    SLURM_TIME_PATTERN.test(timeLimit)
-      ? ''
-      : 'Use: minutes, minutes:seconds, HH:MM:SS, D-HH, D-HH:MM, D-HH:MM:SS'
+    isValidSlurmTime(timeLimit) ? '' : SLURM_TIME_HINT
   )
 
   const handleConfirm = () => {

src/lib/components/PipelineCanvas.svelte

@@ -0,0 +1,239 @@
+<script lang="ts">
+  import {
+    SvelteFlow,
+    Background,
+    Controls,
+    Panel,
+    type NodeTypes,
+    type Connection,
+    type Edge,
+  } from '@xyflow/svelte'
+  import '@xyflow/svelte/dist/base.css'
+  import {
+    getNodes,
+    getEdges,
+    setNodes,
+    setEdges,
+    getEdgesSnapshot,
+    pipelineState,
+  } from '../stores/pipeline.svelte'
+  import {
+    getNodesSnapshot as getGraphNodesSnapshot,
+    getEdgesSnapshot as getGraphEdgesSnapshot,
+  } from '../stores/nodes.svelte'
+  import { buildGraphPayload } from '../utils/sshMessages'
+  import { runPipelineRemote } from '../orchestration/pipelineOrchestrator'
+  import { resolveExecutionOrder } from '../orchestration/executionOrder'
+  import { settingsState } from '../stores/settingsStore.svelte'
+  import { colorModeState } from '../stores/colorModeStore.svelte'
+  import { toastState } from '../stores/toastsStore.svelte'
+  import { currentProjectState } from '../stores/currentProjectStore.svelte'
+  import Button from './layout/Button.svelte'
+  import CoralStageNode from './nodes/CoralStageNode.svelte'
+  import ExecutableStageNode from './nodes/ExecutableStageNode.svelte'
+
+  const nodeTypes: NodeTypes = {
+    coralStage: CoralStageNode as unknown as NodeTypes[string],
+    executableStage: ExecutableStageNode as unknown as NodeTypes[string],
+  }
+
+  let coralGraphInput: HTMLInputElement | undefined = $state()
+
+  let isRemote = $derived(settingsState.execution.location === 'remote')
+  let validation = $derived(pipelineState.validation)
+
+  /**
+   * Rejects connections that would create a self-loop, duplicate an existing edge,
+   * or introduce a cycle (which would make the pipeline unschedulable).
+   * TODO: cache by (source, target, edgesHash) if this becomes measurable on large pipelines.
+   */
+  const isValidConnection = (connection: Connection | Edge): boolean => {
+    const { source, target } = connection
+    // console.log('[isValidConnection]', { source, target })
+    if (!source || !target || source === target) {
+      console.warn(
+        `Source stage ${source} and target stage ${target} are invalid: self-loop or missing endpoint`
+      )
+      return false
+    }
+
+    const edges = getEdgesSnapshot()
+    if (edges.some((e) => e.source === source && e.target === target)) {
+      console.warn(
+        `Source stage ${source} and target stage ${target} are invalid: duplicate edge`
+      )
+      return false
+    }
+
+    const pipeline = pipelineState.toPipeline()
+    const candidate = {
+      ...pipeline,
+      edges: [...pipeline.edges, { source, target }],
+    }
+    try {
+      resolveExecutionOrder(candidate)
+      return true
+    } catch {
+      console.warn(
+        `Source stage ${source} and target stage ${target} are invalid: would introduce a cycle`
+      )
+      return false
+    }
+  }
+
+  const handleAddCoralFromFile = async () => {
+    const file = coralGraphInput?.files?.[0]
+    if (!file) return
+    try {
+      const graph = JSON.parse(await file.text())
+      if (!graph || typeof graph !== 'object' || !('workflow' in graph)) {
+        toastState.add({
+          message:
+            'This file does not look like a CORAL graph (no "workflow").',
+          type: 'error',
+        })
+        return
+      }
+      const name = file.name.replace(/\.json$/i, '')
+      pipelineState.addCoralStage({ name, graph })
+    } catch (error) {
+      toastState.add({
+        message:
+          error instanceof Error ? error.message : 'Failed to read graph file',
+        type: 'error',
+      })
+    } finally {
+      if (coralGraphInput) coralGraphInput.value = ''
+    }
+  }
+
+  const handleAddCoralFromCanvas = () => {
+    const graph = buildGraphPayload(
+      getGraphNodesSnapshot(),
+      getGraphEdgesSnapshot(),
+      false
+    )
+    pipelineState.addCoralStage({
+      name: currentProjectState.name || 'canvas graph',
+      graph,
+    })
+  }
+
+  const handleAddExecutable = () => {
+    pipelineState.addExecutableStage({
+      name: 'executable',
+      executablePath: settingsState.remote.executablePath,
+      parametersFileName:
+        settingsState.remote.parametersFileName || 'parameters.json',
+    })
+  }
+
+  const handleRun = () => {
+    if (!validation.runnable) {
+      toastState.add({
+        message: `Cannot run: ${validation.issues.join('; ')}`,
+        type: 'error',
+      })
+      return
+    }
+    const pipeline = pipelineState.toPipeline()
+
+    // Expose the pipeline snapshot for inspection.
+    console.log('[pipeline canvas]', JSON.parse(JSON.stringify(pipeline)))
+
+    runPipelineRemote(pipeline)
+  }
+</script>
+
+<div class="pipeline-canvas" data-testid="pipeline-canvas">
+  <SvelteFlow
+    bind:nodes={getNodes, setNodes}
+    bind:edges={getEdges, setEdges}
+    {nodeTypes}
+    {isValidConnection}
+    colorMode={colorModeState.value}
+    fitView
+  >
+    <Background />
+    <Controls />
+    <Panel position="bottom-right">
+      <div class="toolbar">
+        {#if !isRemote}
+          <div class="banner">
+            Pipelines run in remote mode only — switch in Settings.
+          </div>
+        {/if}
+        <div class="add-buttons">
+          <Button size="small" onclick={() => coralGraphInput?.click()}>
+            + Coral (file)
+          </Button>
+          <Button size="small" onclick={handleAddCoralFromCanvas}
+            >+ Coral (canvas)</Button
+          >
+          <Button size="small" onclick={handleAddExecutable}
+            >+ Executable</Button
+          >
+        </div>
+        <div class="actions">
+          <Button
+            size="small"
+            variant="action"
+            disabled={!isRemote || !validation.runnable}
+            onclick={handleRun}
+          >
+            Run pipeline
+          </Button>
+        </div>
+        {#if validation.issues.length > 0}
+          <ul class="issues">
+            {#each validation.issues as issue (issue)}
+              <li>{issue}</li>
+            {/each}
+          </ul>
+        {/if}
+      </div>
+    </Panel>
+  </SvelteFlow>
+
+  <input
+    bind:this={coralGraphInput}
+    type="file"
+    accept=".json"
+    style="display: none"
+    onchange={handleAddCoralFromFile}
+  />
+</div>
+
+<style>
+  .pipeline-canvas {
+    width: 100%;
+    height: 100%;
+  }
+  .toolbar {
+    display: flex;
+    flex-direction: column;
+    gap: 0.5rem;
+    background: var(--primary-color);
+    border: 1px solid var(--ternary-color);
+    border-radius: 10px;
+    padding: 0.6rem;
+    max-width: 320px;
+  }
+  .add-buttons,
+  .actions {
+    display: flex;
+    flex-direction: column;
+    gap: 0.4rem;
+    flex-wrap: wrap;
+  }
+  .banner {
+    color: var(--error-color, #e53935);
+    font-size: 0.8rem;
+  }
+  .issues {
+    margin: 0;
+    padding-left: 1.1rem;
+    font-size: 0.75rem;
+    color: var(--ternary-color);
+  }
+</style>

src/lib/components/layout/ExecutionBadge.svelte

@@ -3,19 +3,24 @@
     ExecutionLocation,
     BackendKind,
   } from '../../types/settingsTypes'
+  import { viewModeState } from '../../stores/viewModeStore.svelte'
 
   interface Props {
     location: ExecutionLocation
     backendKind: BackendKind
   }
 
   let { location, backendKind }: Props = $props()
+
+  let modeLabel = $derived(
+    viewModeState.value === 'pipeline' ? 'pipeline' : backendKind
+  )
 </script>
 
 <div class="execution-badge">
   <span>{location}</span>
   <span class="badge-sep">·</span>
-  <span>{backendKind}</span>
+  <span>{modeLabel}</span>
 </div>
 
 <style>