Skip to content

Feat: legible run/pipeline naming + Working Directory job-table column + Pipeline import/export#208

Open
pcolt wants to merge 4 commits into
refactor/decouple-stage-settingsfrom
feat/group-persist-pipeline
Open

Feat: legible run/pipeline naming + Working Directory job-table column + Pipeline import/export#208
pcolt wants to merge 4 commits into
refactor/decouple-stage-settingsfrom
feat/group-persist-pipeline

Conversation

@pcolt

@pcolt pcolt commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

This branch is stacked on refactor/decouple-stage-settings (#206, still open) since the naming logic builds directly on that PR's per-run directory isolation and widened job configs. Do not merge before #206 lands; once it does, retarget this PR's base to main.

It closes #207

  • Update CHANGELOG

Overview

Phase 1: Naming logic

The problem — Directory names for single runs and pipelines were pure timestamps (run-<ts>, pipeline-<ts>), which made it hard to find or inspect a specific run's output on disk or in coral-visualizer. Separately, the Jobs table did not show the working directory, making it even harder for the user to match a timestamp to the results of a specific run (the working directory was only sometimes recoverable by digging through the log file contents).

The solution — This PR delivers Phase 1 of #207: users can now optionally name a run or pipeline at submit time, that name becomes the output folder (falling back to today's timestamp when left blank, and auto-suffixed on collision rather than overwriting), and the Jobs table surfaces each job's working directory directly. No new persistence is introduced in this phase.

How a user recognizes their jobs, and which ones belong to the same pipeline run

There's no dedicated grouping UI (deliberately — see issue207-body.md's "Out of scope") — it's the shared folder prefix in the new "Working Directory" column that makes it visually obvious. For example, after running a standalone job named "poisson-check" and a 3-stage pipeline named "mesh-study", the Jobs table would show:

Job ID State Working Directory
101 COMPLETED /app/shared-data/run-poisson-check
102 COMPLETED /app/shared-data/pipeline-mesh-study/stage-p0
103 RUNNING /app/shared-data/pipeline-mesh-study/stage-p1
104 PENDING /app/shared-data/pipeline-mesh-study/stage-p2

Job 101 is a standalone run (run- prefix, no siblings). Jobs 102–104 all share the pipeline-mesh-study/ prefix and differ only by stage-<id> — that shared prefix is what tells the user these three rows are one pipeline execution, without needing a dedicated group header or a separate "which jobs belong to which pipeline" lookup.

Screenshot from 2026-07-03 12-20-19 Screenshot from 2026-07-03 09-18-29

Phase2: Pipeline import/export format

This part is separate and unrelated to any persistence mechanism; a broader question of whether/how to persist saved pipeline definitions was considered and deferred out of this issue entirely (now tracked in issue202-body.md).

What export/import round-trip through, and why: this PR makes export/import round-trip through exactly the shape the orchestrator consumes (via pipelineState.toPipeline()), so the file format and the runtime format are the same schema. That's slightly different from the raw xyflow canvas snapshot — literally what SvelteFlow itself renders (measured, selected, dragging, etc.) — which is what xyflow expects, not what the orchestrator expects; serializing the xyflow snapshot instead would leave the file format and the orchestrator's actual input as two different schemas that happen to overlap.

Getting to a clean "the file is the orchestrator's input, wrapped" statement is also why the orchestrator-facing Pipeline/PipelineStage types themselves changed in this PR, not just the export code: the pre-existing orchestrator shape ({ stages: [...], edges: [...] }, stages carrying a kind field, no type/position) wasn't itself modeled on anything else in the app, so it's reshaped here to mirror the existing CORAL graph export format below — same wrapper-plus-envelope idea, adapted for a DAG of stages instead of a DAG of C++ objects.

CORAL graph export (test_files/network-mwe-simplified-qualified.json), for comparison — the format this reshape is modeled on:

{
  "workflow": {
    "nodes": {
      "0": { "type": "dealii::Triangulation<2, 2>", "position": { "x": 153, "y": 160 }, "qualified_id": "0" },
      "1": { "type": "std::string", "position": { "x": 196, "y": 252 }, "value": "hyper_cube", "qualified_id": "1" },
      "3": { "type": "GridGenerator::generate_from_name_and_arguments<2>", "position": { "x": 625, "y": 240 }, "qualified_id": "3" }
    },
    "edges": {
      "0": { "source": 0, "target": 3, "source_output": 0, "target_input": 0 },
      "1": { "source": 1, "target": 3, "source_output": 0, "target_input": 1 }
    }
  },
  "version": 1,
  "author": "dealiix-platform",
  "date_time_utc": "2026-02-11T13:21:11.051Z"
}

Pipeline export/import (new, this PR) — same envelope idea, different body:

{
  "pipeline": {
    "nodes": [
      {
        "id": "p0",
        "type": "coralStage",
        "position": { "x": -13, "y": 47 },
        "name": "network-mwe-simplified-qualified",
        "graph": { "workflow": { "nodes": {"...": "..."}, "edges": {"...": "..."} }, "version": 1, "author": "dealiix-platform", "date_time_utc": "..." },
        "config": { "coralBinaryPath": "...", "coralPluginPath": "...", "nodes": 1, "tasksPerNode": 4, "timeLimit": "01:00:00", "useMpi": false }
      },
      {
        "id": "p1",
        "type": "executableStage",
        "position": { "x": 315, "y": 77 },
        "name": "step-70",
        "parameters": { "Stokes Immersed Problem": {"...": "..."} },
        "config": { "executablePath": "...", "parametersFileName": "parameters.json", "timeLimit": "01:00:00" }
      }
    ],
    "edges": [{ "source": "p0", "target": "p1" }]
  },
  "version": 1,
  "author": "dealiix-platform",
  "date_time_utc": "2026-07-03T09:20:00.000Z"
}

Differences from the graph format worth calling out:

  • Wrapper key is pipeline, not workflow. A pipeline node's own graph field already contains a full CORAL export with its own workflow key — reusing workflow at the pipeline level too would put the identical key at two nesting depths meaning two unrelated things (the meta-DAG of stages vs. one stage's actual computation graph), which is genuinely ambiguous if you grep/scan the file rather than just misleading-once.
  • nodes/edges are arrays, not id-keyed objects like the graph format. A pipeline is a small DAG the topological sort iterates directly; there's nothing to key by id the way graph edges key by port index. Pipeline edges are pure ordering ({ source, target }), no source_output/target_input.
  • type replaces the old per-stage kind field as the discriminant, matching the xyflow node's own type ('coralStage' / 'executableStage') instead of carrying a second, redundant value that had to be kept in sync with it.
  • config no longer carries any discriminant at allCoralJobConfig/ExecutableJobConfig dropped their kind field entirely. It was write-only (constructed but never read anywhere in the codebase); removing it makes a stage's config a plain run-mechanics bag, symmetric with how a single (non-pipeline) job submission already passes a bare config while the backend choice lives with the caller, not the data.
  • The pipeline wrapper exists only on this file format — the in-memory Pipeline type the orchestrator (and the canvas) actually consume is flat ({ nodes, edges }, no pipeline key). Wrapping a type literally named Pipeline under a key literally named pipeline produced a pipeline.pipeline.nodes stutter at every call site, since any local holding a Pipeline is naturally named pipeline. The wrapper is added once at export and removed once at import; the orchestrator never sees it.

Summary

Phase 1 — legible naming:

  • feat: slugify.tsslugify() / buildDirName(prefix, name?) for turning a user-entered name into a filesystem-safe folder segment, falling back to a timestamp when no name is given
  • feat: ensureUniqueRemoteDir() in sshMessages.ts — creates the run/pipeline directory, and if the name collides with an existing one, auto-appends a timestamp suffix instead of blocking submission or silently overwriting the earlier run
  • feat: exportAndEvalCoralGraph/exportAndEvalExecutable (and their remote counterparts) and runPipelineRemote all accept an optional runName, used to build the output directory
  • feat: PipelineRunNameModal.svelte — confirmation modal for naming a pipeline run before submit; JobConfigModal.svelte gains the equivalent optional field for single runs
  • feat: JobsTable.svelte gains a "Working Directory" column showing the raw, already-tracked jobIdMap path verbatim — no parsing, so the table stays decoupled from the run-*/pipeline-* naming convention. Truncates with a hover tooltip when the table is collapsed; wraps onto multiple lines when expanded (collapsed view has no scroll, so wrapping there would just get clipped). Table widened 50vw → 60vw to give the new column more room.
  • fix: the pipeline run name was silently dropped on every submission — PipelineRunNameModal's confirm handler called close() before reading the input, but Modal's onClose callback resets the field synchronously during close(), so the name was always empty by the time it reached onConfirm
  • test: new slugify.test.ts, sshMessages.test.ts (ensureUniqueRemoteDir retry/collision/exhaustion behavior); pipelineOrchestrator.test.ts extended with cases for slugified names, timestamp fallback, and collision-suffixed stage directories
  • docs: CHANGELOG.md, CLAUDE.md updated

Phase 2 — pipeline import/export:

  • feat: "Export pipeline"/"Import pipeline" buttons in PipelineCanvas.svelte download/reload a pipeline as a JSON file, round-tripping through the same shape runPipelineRemote consumes (pipelineState.toPipeline()/load()) rather than a raw xyflow canvas snapshot
  • refactor: the pipeline data model now mirrors the CORAL graph export shape — each stage is keyed by type ('coralStage'/'executableStage') instead of a separate kind field (matching the xyflow node type), and CoralJobConfig/ExecutableJobConfig drop their unused kind discriminant entirely (it was never read)
  • feat: the download/import file wraps { nodes, edges } under a pipeline key plus a version/author/date_time_utc envelope, mirroring the graph export's workflow-wrapped-with-envelope shape. The orchestrator-facing Pipeline type itself stays flat/unwrapped (no pipeline key) so orchestrator/canvas code never has to read pipeline.pipeline.nodes
  • feat: new exportMeta.ts centralizes the version/author/date_time_utc envelope (previously inlined in graphParser.ts); both graph and pipeline exports now stamp it identically
  • test: pipeline.svelte.test.ts covers load()/toPipeline() round-tripping, stage-id counter resync after import, and the metadata envelope being ignored on load; executionOrder.test.ts/pipelineOrchestrator.test.ts updated to the new stage shape

Test plan

  • npm run lint, npm run check, npm run check:electron, npm run test all pass
  • Manual (Docker Slurm, docs/run-coral-docker.md): submit a single job without a name → run-<timestamp> on disk (unchanged default)
  • Manual: submit a single job with a custom name → run-<slug> on disk; Jobs table's Working Directory column shows it
  • Manual: submit a pipeline with a custom name → pipeline-<slug>/stage-<id> on disk
  • Manual: submit the same custom name twice back-to-back → second run is not blocked and lands in a -<timestamp>-suffixed directory; first run's files untouched
  • Manual: confirm the entered pipeline run name is actually used end-to-end (regression check for the fix above)
  • Manual: open coral-visualizer and confirm the named folders are legible there
  • Manual: export a coral+executable pipeline, inspect the JSON matches { pipeline: { nodes: [...], edges: [...] }, version, author, date_time_utc } with no kind field and no xyflow cruft (measured/selected/dragging)
  • Manual: clear the canvas, import that file back → positions, configs, and connections are restored exactly; adding a new stage afterward doesn't collide with imported ids
  • Manual: import a non-pipeline .json (e.g. a plain CORAL graph) → rejected with a toast, canvas untouched

pcolt added 2 commits July 2, 2026 16:48
- feat: optional run/pipeline name at submit time, slugified into the output folder (run-<slug>/pipeline-<slug>), falling back to today's timestamp when left blank
- feat: ensureUniqueRemoteDir auto-suffixes with a timestamp on name collision instead of blocking submission or overwriting
- feat: new PipelineRunNameModal.svelte for pipeline runs; JobConfigModal.svelte gains the same optional field for single runs
- feat: JobsTable gains a "Working Directory" column (raw jobIdMap path, no parsing); truncates with a tooltip when collapsed, wraps when expanded; table widened 50vw -> 60vw
- fix: pipeline run name was silently dropped — Modal's onClose reset the name before the confirm callback read it
- test: new slugify.test.ts, sshMessages.test.ts (ensureUniqueRemoteDir); pipelineOrchestrator.test.ts extended with naming-specific cases
- docs: CHANGELOG, CLAUDE.md updated
- docs: update the entry's link from the tracking issue to the PR that implements it, matching the existing convention for other pipeline entries
@pcolt pcolt changed the title Feat: legible run/pipeline naming + Working Directory job-table column Feat: legible run/pipeline naming + Working Directory job-table column + Pipeline import/export Jul 2, 2026
- feat: pipeline export/import now round-trips through the same shape runPipelineRemote consumes (Pipeline: flat { nodes, edges }), instead of the raw xyflow canvas snapshot
- feat: pipeline stages are keyed by `type` ('coralStage' | 'executableStage') instead of a separate `kind` field, matching the xyflow node type; CoralJobConfig/ExecutableJobConfig drop their unused `kind` discriminant entirely
- feat: PipelineFile (the download/import file format) wraps Pipeline under a `pipeline` key plus a version/author/date_time_utc envelope, mirroring the CORAL graph export shape; the wrapper lives only on PipelineFile so orchestrator/canvas code never sees a `pipeline.pipeline` stutter
- feat: new exportMeta.ts centralizes the version/author/date_time_utc envelope (previously inlined in graphParser.ts); both graph and pipeline exports now stamp it identically
- test: pipeline.svelte.test.ts covers load()/toPipeline() round-tripping, stage-id counter resync, and envelope-ignored-on-load; executionOrder/pipelineOrchestrator tests updated to the new stage shape
- docs: CHANGELOG entry for pipeline import/export
@pcolt
pcolt marked this pull request as ready for review July 3, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant