Feat: legible run/pipeline naming + Working Directory job-table column + Pipeline import/export#208
Open
pcolt wants to merge 4 commits into
Open
Conversation
- feat: optional run/pipeline name at submit time, slugified into the output folder (run-<slug>/pipeline-<slug>), falling back to today's timestamp when left blank - feat: ensureUniqueRemoteDir auto-suffixes with a timestamp on name collision instead of blocking submission or overwriting - feat: new PipelineRunNameModal.svelte for pipeline runs; JobConfigModal.svelte gains the same optional field for single runs - feat: JobsTable gains a "Working Directory" column (raw jobIdMap path, no parsing); truncates with a tooltip when collapsed, wraps when expanded; table widened 50vw -> 60vw - fix: pipeline run name was silently dropped — Modal's onClose reset the name before the confirm callback read it - test: new slugify.test.ts, sshMessages.test.ts (ensureUniqueRemoteDir); pipelineOrchestrator.test.ts extended with naming-specific cases - docs: CHANGELOG, CLAUDE.md updated
- docs: update the entry's link from the tracking issue to the PR that implements it, matching the existing convention for other pipeline entries
- feat: pipeline export/import now round-trips through the same shape runPipelineRemote consumes (Pipeline: flat { nodes, edges }), instead of the raw xyflow canvas snapshot
- feat: pipeline stages are keyed by `type` ('coralStage' | 'executableStage') instead of a separate `kind` field, matching the xyflow node type; CoralJobConfig/ExecutableJobConfig drop their unused `kind` discriminant entirely
- feat: PipelineFile (the download/import file format) wraps Pipeline under a `pipeline` key plus a version/author/date_time_utc envelope, mirroring the CORAL graph export shape; the wrapper lives only on PipelineFile so orchestrator/canvas code never sees a `pipeline.pipeline` stutter
- feat: new exportMeta.ts centralizes the version/author/date_time_utc envelope (previously inlined in graphParser.ts); both graph and pipeline exports now stamp it identically
- test: pipeline.svelte.test.ts covers load()/toPipeline() round-tripping, stage-id counter resync, and envelope-ignored-on-load; executionOrder/pipelineOrchestrator tests updated to the new stage shape
- docs: CHANGELOG entry for pipeline import/export
pcolt
marked this pull request as ready for review
July 3, 2026 15:30
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It closes #207
Overview
Phase 1: Naming logic
The problem — Directory names for single runs and pipelines were pure timestamps (
run-<ts>,pipeline-<ts>), which made it hard to find or inspect a specific run's output on disk or in coral-visualizer. Separately, the Jobs table did not show the working directory, making it even harder for the user to match a timestamp to the results of a specific run (the working directory was only sometimes recoverable by digging through the log file contents).The solution — This PR delivers Phase 1 of #207: users can now optionally name a run or pipeline at submit time, that name becomes the output folder (falling back to today's timestamp when left blank, and auto-suffixed on collision rather than overwriting), and the Jobs table surfaces each job's working directory directly. No new persistence is introduced in this phase.
How a user recognizes their jobs, and which ones belong to the same pipeline run
There's no dedicated grouping UI (deliberately — see
issue207-body.md's "Out of scope") — it's the shared folder prefix in the new "Working Directory" column that makes it visually obvious. For example, after running a standalone job named "poisson-check" and a 3-stage pipeline named "mesh-study", the Jobs table would show:/app/shared-data/run-poisson-check/app/shared-data/pipeline-mesh-study/stage-p0/app/shared-data/pipeline-mesh-study/stage-p1/app/shared-data/pipeline-mesh-study/stage-p2Job 101 is a standalone run (
run-prefix, no siblings). Jobs 102–104 all share thepipeline-mesh-study/prefix and differ only bystage-<id>— that shared prefix is what tells the user these three rows are one pipeline execution, without needing a dedicated group header or a separate "which jobs belong to which pipeline" lookup.Phase2: Pipeline import/export format
This part is separate and unrelated to any persistence mechanism; a broader question of whether/how to persist saved pipeline definitions was considered and deferred out of this issue entirely (now tracked in
issue202-body.md).What export/import round-trip through, and why: this PR makes export/import round-trip through exactly the shape the orchestrator consumes (via
pipelineState.toPipeline()), so the file format and the runtime format are the same schema. That's slightly different from the raw xyflow canvas snapshot — literally what SvelteFlow itself renders (measured,selected,dragging, etc.) — which is what xyflow expects, not what the orchestrator expects; serializing the xyflow snapshot instead would leave the file format and the orchestrator's actual input as two different schemas that happen to overlap.Getting to a clean "the file is the orchestrator's input, wrapped" statement is also why the orchestrator-facing
Pipeline/PipelineStagetypes themselves changed in this PR, not just the export code: the pre-existing orchestrator shape ({ stages: [...], edges: [...] }, stages carrying akindfield, notype/position) wasn't itself modeled on anything else in the app, so it's reshaped here to mirror the existing CORAL graph export format below — same wrapper-plus-envelope idea, adapted for a DAG of stages instead of a DAG of C++ objects.CORAL graph export (
test_files/network-mwe-simplified-qualified.json), for comparison — the format this reshape is modeled on:{ "workflow": { "nodes": { "0": { "type": "dealii::Triangulation<2, 2>", "position": { "x": 153, "y": 160 }, "qualified_id": "0" }, "1": { "type": "std::string", "position": { "x": 196, "y": 252 }, "value": "hyper_cube", "qualified_id": "1" }, "3": { "type": "GridGenerator::generate_from_name_and_arguments<2>", "position": { "x": 625, "y": 240 }, "qualified_id": "3" } }, "edges": { "0": { "source": 0, "target": 3, "source_output": 0, "target_input": 0 }, "1": { "source": 1, "target": 3, "source_output": 0, "target_input": 1 } } }, "version": 1, "author": "dealiix-platform", "date_time_utc": "2026-02-11T13:21:11.051Z" }Pipeline export/import (new, this PR) — same envelope idea, different body:
{ "pipeline": { "nodes": [ { "id": "p0", "type": "coralStage", "position": { "x": -13, "y": 47 }, "name": "network-mwe-simplified-qualified", "graph": { "workflow": { "nodes": {"...": "..."}, "edges": {"...": "..."} }, "version": 1, "author": "dealiix-platform", "date_time_utc": "..." }, "config": { "coralBinaryPath": "...", "coralPluginPath": "...", "nodes": 1, "tasksPerNode": 4, "timeLimit": "01:00:00", "useMpi": false } }, { "id": "p1", "type": "executableStage", "position": { "x": 315, "y": 77 }, "name": "step-70", "parameters": { "Stokes Immersed Problem": {"...": "..."} }, "config": { "executablePath": "...", "parametersFileName": "parameters.json", "timeLimit": "01:00:00" } } ], "edges": [{ "source": "p0", "target": "p1" }] }, "version": 1, "author": "dealiix-platform", "date_time_utc": "2026-07-03T09:20:00.000Z" }Differences from the graph format worth calling out:
pipeline, notworkflow. A pipeline node's owngraphfield already contains a full CORAL export with its ownworkflowkey — reusingworkflowat the pipeline level too would put the identical key at two nesting depths meaning two unrelated things (the meta-DAG of stages vs. one stage's actual computation graph), which is genuinely ambiguous if yougrep/scan the file rather than just misleading-once.nodes/edgesare arrays, not id-keyed objects like the graph format. A pipeline is a small DAG the topological sort iterates directly; there's nothing to key by id the way graph edges key by port index. Pipeline edges are pure ordering ({ source, target }), nosource_output/target_input.typereplaces the old per-stagekindfield as the discriminant, matching the xyflow node's owntype('coralStage'/'executableStage') instead of carrying a second, redundant value that had to be kept in sync with it.configno longer carries any discriminant at all —CoralJobConfig/ExecutableJobConfigdropped theirkindfield entirely. It was write-only (constructed but never read anywhere in the codebase); removing it makes a stage'sconfiga plain run-mechanics bag, symmetric with how a single (non-pipeline) job submission already passes a bare config while the backend choice lives with the caller, not the data.pipelinewrapper exists only on this file format — the in-memoryPipelinetype the orchestrator (and the canvas) actually consume is flat ({ nodes, edges }, nopipelinekey). Wrapping a type literally namedPipelineunder a key literally namedpipelineproduced apipeline.pipeline.nodesstutter at every call site, since any local holding aPipelineis naturally namedpipeline. The wrapper is added once at export and removed once at import; the orchestrator never sees it.Summary
Phase 1 — legible naming:
slugify.ts—slugify()/buildDirName(prefix, name?)for turning a user-entered name into a filesystem-safe folder segment, falling back to a timestamp when no name is givenensureUniqueRemoteDir()insshMessages.ts— creates the run/pipeline directory, and if the name collides with an existing one, auto-appends a timestamp suffix instead of blocking submission or silently overwriting the earlier runexportAndEvalCoralGraph/exportAndEvalExecutable(and their remote counterparts) andrunPipelineRemoteall accept an optionalrunName, used to build the output directoryPipelineRunNameModal.svelte— confirmation modal for naming a pipeline run before submit;JobConfigModal.sveltegains the equivalent optional field for single runsJobsTable.sveltegains a "Working Directory" column showing the raw, already-trackedjobIdMappath verbatim — no parsing, so the table stays decoupled from therun-*/pipeline-*naming convention. Truncates with a hover tooltip when the table is collapsed; wraps onto multiple lines when expanded (collapsed view has no scroll, so wrapping there would just get clipped). Table widened 50vw → 60vw to give the new column more room.PipelineRunNameModal's confirm handler calledclose()before reading the input, butModal'sonClosecallback resets the field synchronously duringclose(), so the name was always empty by the time it reachedonConfirmslugify.test.ts,sshMessages.test.ts(ensureUniqueRemoteDirretry/collision/exhaustion behavior);pipelineOrchestrator.test.tsextended with cases for slugified names, timestamp fallback, and collision-suffixed stage directoriesCHANGELOG.md,CLAUDE.mdupdatedPhase 2 — pipeline import/export:
PipelineCanvas.sveltedownload/reload a pipeline as a JSON file, round-tripping through the same shaperunPipelineRemoteconsumes (pipelineState.toPipeline()/load()) rather than a raw xyflow canvas snapshottype('coralStage'/'executableStage') instead of a separatekindfield (matching the xyflow node type), andCoralJobConfig/ExecutableJobConfigdrop their unusedkinddiscriminant entirely (it was never read){ nodes, edges }under apipelinekey plus aversion/author/date_time_utcenvelope, mirroring the graph export'sworkflow-wrapped-with-envelope shape. The orchestrator-facingPipelinetype itself stays flat/unwrapped (nopipelinekey) so orchestrator/canvas code never has to readpipeline.pipeline.nodesexportMeta.tscentralizes theversion/author/date_time_utcenvelope (previously inlined ingraphParser.ts); both graph and pipeline exports now stamp it identicallypipeline.svelte.test.tscoversload()/toPipeline()round-tripping, stage-id counter resync after import, and the metadata envelope being ignored on load;executionOrder.test.ts/pipelineOrchestrator.test.tsupdated to the new stage shapeTest plan
npm run lint,npm run check,npm run check:electron,npm run testall passdocs/run-coral-docker.md): submit a single job without a name →run-<timestamp>on disk (unchanged default)run-<slug>on disk; Jobs table's Working Directory column shows itpipeline-<slug>/stage-<id>on disk-<timestamp>-suffixed directory; first run's files untouched{ pipeline: { nodes: [...], edges: [...] }, version, author, date_time_utc }with nokindfield and no xyflow cruft (measured/selected/dragging).json(e.g. a plain CORAL graph) → rejected with a toast, canvas untouched