Skip to content

Feat: pipeline orchestrator MVP (visual DAG + remote Slurm execution)#205

Open
pcolt wants to merge 2 commits into
mainfrom
feat/pipeline-orchestration
Open

Feat: pipeline orchestrator MVP (visual DAG + remote Slurm execution)#205
pcolt wants to merge 2 commits into
mainfrom
feat/pipeline-orchestration

Conversation

@pcolt

@pcolt pcolt commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator
  • Update CHANGELOG

Overview

Part of #202 — adds a pipeline editor and execution engine that lets users chain whole CORAL graphs and standalone executables as nodes of a higher-level DAG. This PR is a minimum viable product — it proves the meta-graph model and the remote --dependency execution path end to end.

Each stage runs as its own Slurm job chained via --dependency=afterok, so ordering and parallelism are enforced by the cluster and the run survives the app closing. Cyclic connections are rejected live as the user draws them. This first version targets remote (Slurm) mode only.

Summary

This PR delivers the first slice of #202 — the meta-graph model, dependency resolution, remote execution, and the pipeline editor UI.

Data model & ordering

  • feat: pipelineTypes.tsPipeline / PipelineStage (coral | executable) / PipelineEdge data model; ordering-only edges, no artifact ports yet
  • feat: executionOrder.ts — pure Kahn topological sort + PipelineCycleError; zero app dependencies, unit-tested; reused live in PipelineCanvas to reject cyclic connections as the user draws them

Remote execution

  • feat: pipelineOrchestrator.tsrunPipelineRemote walks topo order, submits each stage with --dependency=afterok:<parent ids>, polls all concurrently; --kill-on-invalid-dep=yes cascades cancellation on upstream failure
  • refactor: sshMessages.ts — extract submitCoralStageRemote / submitExecutableStageRemote / jobPolling as reusable primitives shared by single-run and pipeline paths; single-run functions delegate with dependencyJobIds: []; submitSbatch adds --kill-on-invalid-dep=yes
  • refactor: jobsStorejobIdMapState.add accepts an optional workingDirectory so log and node-status lookups resolve correctly for both single and pipeline jobs; each pipeline stage registers as its own row in the job table with independent Logs and Node Status access

Pipeline editor UI

  • feat: PipelineCanvas.svelte — second SvelteFlow canvas with add-stage controls (coral from file, coral from canvas, executable) and a validation-gated Run button; viewModeStore.svelte.ts + App.svelte toggle between single-stage and pipeline views
  • feat: CoralStageNode.svelte / ExecutableStageNode.svelte — inline per-stage config cards (MPI toggle, nodes, tasks/node, time limit, parameter file loading)
  • feat: pipeline.svelte.ts — pipeline canvas store mirroring the nodes.svelte get/set shape; exposes toPipeline() for the orchestrator and a validation getter driving the Run button
  • feat: slurmTime.ts — HH:MM:SS ↔ seconds conversion utility

Out of scope (MVP boundaries)

The following are intentionally deferred to follow-up issues tracked in #202:

  • Local execution — no in-process DAG executor; Run is disabled outside remote mode.
  • Artifact passing — edges are ordering-only, no inter-stage file/param staging yet.
  • Stage submit config still reads settingsState at submit time instead of taking it as an explicit argument.
  • UI/UX rationalisation — mode switching and the two separate Run controls are provisional.
  • Persistence & run history — the pipeline store is transient.
  • E2E test — no Playwright spec for the pipeline view yet.

Test plan

  • npx vitest run src/lib/orchestration/executionOrder.test.ts — topo order for linear / diamond / disconnected DAGs; cycle rejection
  • docker compose up -d, then run a 3-stage pipeline (coral → executable → coral) with different MPI settings per stage (stage 1: MPI 1 nodes, stage 2: no MPI, stage 3: MPI 1 node). Confirm via sacct/squeue correct --dependency chains and distinct #SBATCH resource blocks per stage
  • Regression: existing single-graph and single-executable runs still work (npm run test, npm run check, npm run check:electron)

…bles

- feat: PipelineCanvas.svelte — SvelteFlow DAG editor for chaining stages; acyclicity guard rejects cycles via live topo-sort on every edge draw
- feat: CoralStageNode and ExecutableStageNode canvas node components with per-stage resource config cards
- feat: pipelineTypes.ts and pipeline.svelte.ts — Pipeline/PipelineStage model, default configs, validation getter, toPipeline()
- feat: viewModeStore.svelte.ts and App.svelte — single-stage ↔ pipeline view toggle in top-right overlay
- feat: executionOrder.ts — pure Kahn topological sort with cycle detection (PipelineCycleError); executionOrder.test.ts covers topo order, ties, and cycles
- feat: pipelineOrchestrator.ts — resolves DAG order and submits each stage as its own Slurm job with --dependency=afterok chaining; polls all jobs concurrently
- feat: slurmTime.ts — HH:MM:SS ↔ seconds conversion utility
- refactor: sshMessages.ts — extract submitCoralStageRemote / submitExecutableStageRemote / jobPolling as reusable primitives for both single runs and pipeline stages
- refactor: jobsStore — track per-job workingDirectory so log and node-status lookups resolve correctly for both single and pipeline jobs
- docs: CHANGELOG and CLAUDE.md updated to document pipeline architecture and new stores
# Conflicts:
#	CHANGELOG.md
#	src/lib/components/layout/SidebarButtons.svelte
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant