Feat: pipeline orchestrator MVP (visual DAG + remote Slurm execution)#205
Open
pcolt wants to merge 2 commits into
Open
Feat: pipeline orchestrator MVP (visual DAG + remote Slurm execution)#205pcolt wants to merge 2 commits into
pcolt wants to merge 2 commits into
Conversation
…bles - feat: PipelineCanvas.svelte — SvelteFlow DAG editor for chaining stages; acyclicity guard rejects cycles via live topo-sort on every edge draw - feat: CoralStageNode and ExecutableStageNode canvas node components with per-stage resource config cards - feat: pipelineTypes.ts and pipeline.svelte.ts — Pipeline/PipelineStage model, default configs, validation getter, toPipeline() - feat: viewModeStore.svelte.ts and App.svelte — single-stage ↔ pipeline view toggle in top-right overlay - feat: executionOrder.ts — pure Kahn topological sort with cycle detection (PipelineCycleError); executionOrder.test.ts covers topo order, ties, and cycles - feat: pipelineOrchestrator.ts — resolves DAG order and submits each stage as its own Slurm job with --dependency=afterok chaining; polls all jobs concurrently - feat: slurmTime.ts — HH:MM:SS ↔ seconds conversion utility - refactor: sshMessages.ts — extract submitCoralStageRemote / submitExecutableStageRemote / jobPolling as reusable primitives for both single runs and pipeline stages - refactor: jobsStore — track per-job workingDirectory so log and node-status lookups resolve correctly for both single and pipeline jobs - docs: CHANGELOG and CLAUDE.md updated to document pipeline architecture and new stores
# Conflicts: # CHANGELOG.md # src/lib/components/layout/SidebarButtons.svelte
16 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Part of #202 — adds a pipeline editor and execution engine that lets users chain whole CORAL graphs and standalone executables as nodes of a higher-level DAG. This PR is a minimum viable product — it proves the meta-graph model and the remote
--dependencyexecution path end to end.Each stage runs as its own Slurm job chained via
--dependency=afterok, so ordering and parallelism are enforced by the cluster and the run survives the app closing. Cyclic connections are rejected live as the user draws them. This first version targets remote (Slurm) mode only.Summary
This PR delivers the first slice of #202 — the meta-graph model, dependency resolution, remote execution, and the pipeline editor UI.
Data model & ordering
pipelineTypes.ts—Pipeline/PipelineStage(coral|executable) /PipelineEdgedata model; ordering-only edges, no artifact ports yetexecutionOrder.ts— pure Kahn topological sort +PipelineCycleError; zero app dependencies, unit-tested; reused live inPipelineCanvasto reject cyclic connections as the user draws themRemote execution
pipelineOrchestrator.ts—runPipelineRemotewalks topo order, submits each stage with--dependency=afterok:<parent ids>, polls all concurrently;--kill-on-invalid-dep=yescascades cancellation on upstream failuresshMessages.ts— extractsubmitCoralStageRemote/submitExecutableStageRemote/jobPollingas reusable primitives shared by single-run and pipeline paths; single-run functions delegate withdependencyJobIds: [];submitSbatchadds--kill-on-invalid-dep=yesjobsStore—jobIdMapState.addaccepts an optionalworkingDirectoryso log and node-status lookups resolve correctly for both single and pipeline jobs; each pipeline stage registers as its own row in the job table with independent Logs and Node Status accessPipeline editor UI
PipelineCanvas.svelte— second SvelteFlow canvas with add-stage controls (coral from file, coral from canvas, executable) and a validation-gated Run button;viewModeStore.svelte.ts+App.sveltetoggle between single-stage and pipeline viewsCoralStageNode.svelte/ExecutableStageNode.svelte— inline per-stage config cards (MPI toggle, nodes, tasks/node, time limit, parameter file loading)pipeline.svelte.ts— pipeline canvas store mirroring thenodes.svelteget/set shape; exposestoPipeline()for the orchestrator and avalidationgetter driving the Run buttonslurmTime.ts— HH:MM:SS ↔ seconds conversion utilityOut of scope (MVP boundaries)
The following are intentionally deferred to follow-up issues tracked in #202:
settingsStateat submit time instead of taking it as an explicit argument.Test plan
npx vitest run src/lib/orchestration/executionOrder.test.ts— topo order for linear / diamond / disconnected DAGs; cycle rejectiondocker compose up -d, then run a 3-stage pipeline (coral → executable → coral) with different MPI settings per stage (stage 1: MPI 1 nodes, stage 2: no MPI, stage 3: MPI 1 node). Confirm viasacct/squeuecorrect--dependencychains and distinct#SBATCHresource blocks per stagenpm run test,npm run check,npm run check:electron)