Task: ComfyUI Workflow Graph — Final Data Wiring

[ash-bluepollution95]

TODO: ComfyUI Workflow Graph — Final Data Wiring
Status: UI tab mounted, component renders, backend extraction working
Issue: Graph viewer receives API-format workflow ({"1": {...}}) instead of Graph format ({ nodes, links })
Context:

• ComfyUI embeds TWO chunks: prompt (API/exec) + workflow (visual/graph)
• Backend currently assigns chunks.prompt to ai.comfyui_workflow
• LiteGraph viewer expects { nodes: [{ pos, size, type, ... }], links: [...] }

IF litegraph isn't the one to use there ARE things like:
https://www.npmjs.com/package/@xyflow/react

So far the CUSTOM COMPONENT exists, so that's a start.

[ash-bluepollution95]

@coderabbitai Any clue why we can't get litegraph to exist properly is it something stupid i did or otherwise - I used qwen to help me with this

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch the issue details
gh issue view 190 --repo Ktiseos-Nyx/Dataset-Tools

echo "---SEPARATOR---"

# Search for litegraph references in the codebase
rg -i "litegraph" --type-not=lock -n -C3

echo "---SEPARATOR---"

# Find files related to litegraph
fd -i "litegraph" --type f

echo "---SEPARATOR---"

# Look at package.json for litegraph dependency
fd "package.json" --max-depth 2 --exec cat {}

Length of output: 916

---

🏁 Script executed:

#!/bin/bash
# Find comfyui-related files
fd -i "comfyui" --type f
echo "---SEPARATOR---"
# Search for workflow-related backend code
rg -i "comfyui_workflow\|chunks\.prompt\|chunks\.workflow" -n -C5
echo "---SEPARATOR---"
# Search for graph component files
rg -i "graph\|workflow" --type vue -n -l 2>/dev/null || rg -i "WorkflowGraph\|GraphViewer\|LiteGraph" -n -C5
echo "---SEPARATOR---"
# Tree of the source directory
tree src/ 2>/dev/null || find . -maxdepth 4 -name "*.vue" -o -name "*.js" -o -name "*.ts" -o -name "*.py" | grep -v node_modules | grep -v ".git" | head -80

Length of output: 7777

---

🏁 Script executed:

#!/bin/bash
echo "=== vendored comfyui.py ==="
cat dataset_tools/vendored_sdpr/format/comfyui.py

echo "=== comfyui_workflow_analyzer.py ==="
cat dataset_tools/metadata_engine/extractors/comfyui_workflow_analyzer.py

echo "=== comfyui_extractors.py ==="
cat dataset_tools/metadata_engine/extractors/comfyui_extractors.py

echo "=== comfyui_enhanced_extractor.py ==="
cat dataset_tools/metadata_engine/extractors/comfyui_enhanced_extractor.py

Length of output: 167733

---

🏁 Script executed:

#!/bin/bash
echo "=== Search for comfyui_workflow in parser_definitions ==="
grep -r "comfyui_workflow\|raw_workflow\|workflow_graph\|graph_data" dataset_tools/parser_definitions/ -l

echo "---"
# Check comfyui.json parser definition for relevant fields
cat dataset_tools/parser_definitions/comfyui.json

echo "=== Search across all Python files for comfyui_workflow assignment ==="
rg -n "comfyui_workflow\|ai\.workflow\|\"workflow\"\s*:" --type py -C2

echo "=== Check how _info chunks are passed - look at image_metadata_reader ==="
cat dataset_tools/file_readers/image_metadata_reader.py

Length of output: 45828

---

@duskfallcrew — Yes, this is a real, identifiable bug, and it's rooted in a ComfyUI naming confusion that tripped up Qwen. Here's the full diagnosis:

---

🔍 Root Cause: The prompt vs workflow Chunk Mix-Up

In a ComfyUI PNG, there are two embedded JSON chunks with deceptive names:

PNG Chunk Key	Actual Content	Format
prompt	API/execution data	{"1": {"class_type": "KSampler", ...}, "2": {...}}
workflow	Visual/graph data	{"nodes": [...], "links": [...], "groups": [...]}

The naming is confusing — "prompt" sounds like text, but it's actually the node execution graph in API format. The "workflow" key is what LiteGraph (and XYFlow) can actually render.

---

🐛 Bug Location: dataset_tools/vendored_sdpr/format/comfyui.py

Qwen wrote the comments completely backwards, and the code follows suit.

In _process() (lines ~87–90):

# ❌ WRONG COMMENTS (Qwen got these backwards)
prompt_str = str(self._info.get("prompt", ""))       # Comment says "workflow JSON" — NO, this is API format
workflow_api_str = str(self._info.get("workflow", "")) # Comment says "API format" — NO, this is LiteGraph format

The heuristic label is also backwards:

# ❌ Label is wrong — nodes+links = LiteGraph format, NOT API format
if "nodes" in loaded_json and "links" in loaded_json:  # Heuristic for API format
    self._workflow_json = loaded_json

In _comfy_png_traverse_and_extract(), self._raw is set with the wrong priority:

# ❌ This feeds the GRAPH VIEWER the API format ({"1": {...}}) instead of LiteGraph format
if not self._raw:
    if self._info.get("prompt"):        # Sets raw = API format FIRST 😱
        self._raw = str(self._info.get("prompt"))
    elif self._info.get("workflow"):    # LiteGraph format only as fallback
        self._raw = str(self._info.get("workflow"))

---

✅ The Fix

In dataset_tools/vendored_sdpr/format/comfyui.py, apply these corrections:

def _process(self) -> None:
    self._logger.debug("Attempting to parse using %s logic.", self.tool)

    # ComfyUI embeds TWO chunks in PNG files:
    # - "prompt" key  → API/execution format: {"1": {"class_type": ..., "inputs": {...}}, "2": {...}}
    #                   (used for parameter extraction / node traversal)
    # - "workflow" key → Visual/LiteGraph format: {"nodes": [...], "links": [...], "groups": [...]}
    #                    (used for graph rendering in the UI)
    prompt_str = str(self._info.get("prompt", ""))    # API/execution format (for traversal)
    workflow_str = str(self._info.get("workflow", "")) # LiteGraph/visual format (for graph viewer)

    # Store the workflow (LiteGraph) chunk for the graph viewer BEFORE we decide
    # which source to use for extraction traversal
    if workflow_str:
        self._workflow_graph_json_str = workflow_str  # Keep for graph viewer
    
    # For traversal/extraction, prefer the API-format "prompt" chunk
    source_json_str_to_parse = None
    source_description = ""

    if prompt_str:
        source_json_str_to_parse = prompt_str
        source_description = "PNG 'prompt' chunk (API/execution format)"
    elif workflow_str:
        source_json_str_to_parse = workflow_str
        source_description = "PNG 'workflow' chunk (LiteGraph/visual format)"
    else:
        # ... rest of error handling unchanged

And fix _comfy_png_traverse_and_extract() to use the LiteGraph format for self._raw:

# In _comfy_png_traverse_and_extract(), at the bottom — flip the priority:
if not self._raw:
    # ✅ Prefer the "workflow" chunk (LiteGraph format) for self._raw — this is
    # what the graph viewer expects: {"nodes": [...], "links": [...]}
    if self._info.get("workflow"):
        self._raw = str(self._info.get("workflow"))
    elif self._info.get("prompt"):
        # Fallback to API format if visual workflow chunk is absent
        self._raw = str(self._info.get("prompt"))
    elif workflow_to_traverse:
        try:
            self._raw = json.dumps(workflow_to_traverse)
        except TypeError:
            self._raw = str(workflow_to_traverse)

Also fix the heuristic comment (cosmetic but prevents future confusion):

# ✅ Correct label: nodes+links = LiteGraph/visual format
if "nodes" in loaded_json and "links" in loaded_json:  # LiteGraph/visual graph format

---

📌 About the input_data in comfyui.json

Good news: the parser definition in dataset_tools/parser_definitions/comfyui.json already has the correct priority for source_options:

"source_options": [
    { "source_type": "pil_info_key", "source_key": "workflow" },  // ✅ LiteGraph first
    { "source_type": "pil_info_key", "source_key": "prompt" }     // API format as fallback
]

So raw_metadata (via comfyui_extract_raw_workflow) should already be receiving LiteGraph format when both chunks are present. The vendored_sdpr path is the one needing the fix.

---

🤔 LiteGraph vs. XYFlow?

Both are valid choices for the graph viewer:

• LiteGraph (@comfyorg/litegraph): ComfyUI itself uses this. Direct compatibility with the workflow chunk since it's the native format. More work to embed but pixel-perfect fidelity.
• @xyflow/react (React Flow): Much easier to integrate in a React app. You'd need a converter from the ComfyUI workflow JSON to XYFlow's node/edge format. Manageable since the data structure is well-documented.

The converter for XYFlow would be roughly:

# nodes[].id, nodes[].type, nodes[].pos → XYFlow nodes
# links[] → XYFlow edges

Either way, the fix above ensures the correct LiteGraph chunk is passed — the rest is a frontend rendering decision.

[ash-bluepollution95]

@coderabbitai I FORGOT TO NOTE IT WAS THE nextjs-rewrite branch not the pythonic one :D the python one DOES NOT use litegraph just the nextjs-rewrite branch - we'll switch to main ASAP with this branch just patching stuff but why in the nextjs branch is it not showing in our component?

[coderabbitai]

🧩 Analysis chain

Good catch! Let me switch to the nextjs-rewrite branch and inspect the LiteGraph component setup there.

---

🏁 Script executed:

#!/bin/bash
# Switch to the nextjs-rewrite branch
git fetch origin nextjs-rewrite
git checkout nextjs-rewrite

echo "=== package.json ==="
cat package.json

echo "=== SEPARATOR ==="
# Find all litegraph references
rg -i "litegraph" -n -C5 --type ts --type tsx 2>/dev/null || rg -i "litegraph" -n -C5

echo "=== SEPARATOR ==="
# Find all component files
find . -maxdepth 5 \( -name "*.tsx" -o -name "*.ts" \) | grep -v node_modules | grep -v ".git" | grep -v ".next"

echo "=== SEPARATOR ==="
# Look at the app directory structure
find . -maxdepth 5 -type f | grep -v node_modules | grep -v ".git" | grep -v ".next" | grep -v ".turbo"

Length of output: 28615

---

The sandbox cloned main — let me pull the Next.js branch files directly via the GitHub API.

---

🏁 Script executed:

#!/bin/bash
# List files in the nextjs-rewrite branch root
gh api repos/Ktiseos-Nyx/Dataset-Tools/git/trees/nextjs-rewrite?recursive=1 \
  --jq '.tree[] | select(.type=="blob") | .path' 2>/dev/null | head -100

echo "=== SEPARATOR: package.json ==="
gh api repos/Ktiseos-Nyx/Dataset-Tools/contents/package.json?ref=nextjs-rewrite \
  --jq '.content' | base64 -d 2>/dev/null

echo "=== SEPARATOR: app directory listing ==="
gh api repos/Ktiseos-Nyx/Dataset-Tools/git/trees/nextjs-rewrite?recursive=1 \
  --jq '.tree[] | select(.type=="blob") | .path' 2>/dev/null | grep -E "\.(tsx|ts|js|jsx|json|css|md)$" | grep -v node_modules

Length of output: 9718

---

Found it — litegraph.js is installed and components/ComfyUIWorkflowViewer.tsx exists. Let me read those files now.

---

🏁 Script executed:

#!/bin/bash
echo "=== ComfyUIWorkflowViewer.tsx ==="
gh api repos/Ktiseos-Nyx/Dataset-Tools/contents/components/ComfyUIWorkflowViewer.tsx?ref=nextjs-rewrite \
  --jq '.content' | base64 -d

echo "=== app/page.tsx ==="
gh api repos/Ktiseos-Nyx/Dataset-Tools/contents/app/page.tsx?ref=nextjs-rewrite \
  --jq '.content' | base64 -d

echo "=== metadata-panel.tsx ==="
gh api repos/Ktiseos-Nyx/Dataset-Tools/contents/components/metadata-panel.tsx?ref=nextjs-rewrite \
  --jq '.content' | base64 -d

Length of output: 43737

---

Found multiple bugs! Let me also check what comfyui_workflow actually contains from the metadata route before writing up the full diagnosis.

---

🏁 Script executed:

#!/bin/bash
echo "=== app/api/metadata/route.ts ==="
gh api repos/Ktiseos-Nyx/Dataset-Tools/contents/app/api/metadata/route.ts?ref=nextjs-rewrite \
  --jq '.content' | base64 -d

echo "=== types/metadata.ts ==="
gh api repos/Ktiseos-Nyx/Dataset-Tools/contents/types/metadata.ts?ref=nextjs-rewrite \
  --jq '.content' | base64 -d

Length of output: 75276

---

@duskfallcrew — Got it! This is a different (and actually simpler) problem than the Python one. There are four stacked bugs — and they each hide the next one. Let me go through them in order of impact:

---

🐛 Bug #1 — Key Name Mismatch (Nothing Ever Arrives)

This is the root cause of the blank viewer. In app/api/metadata/route.ts, the ComfyUI workflow is stored under:

// route.ts — line ~570
aiData.comfy_workflow = chunks.workflow ? JSON.parse(chunks.workflow) : workflow;
//         ^^^^^ "comfy_workflow" — no "ui"

But metadata-panel.tsx looks for a completely different key:

// metadata-panel.tsx
metadata.ai?.comfyui_workflow   // ← "comfyui_workflow" — with "ui"
ai.comfyui_workflow             // ← same, never matches

These will never match, so the condition is always false and the viewer never receives any data — you just get "No ComfyUI workflow data in this image" every time, even for valid ComfyUI PNGs.

Fix — one of two options:

Option A — rename in route.ts (recommended, keeps the component clean):

// route.ts ~line 570
aiData.comfyui_workflow = chunks.workflow ? JSON.parse(chunks.workflow) : workflow;
//         ^^^ add "ui"

Option B — rename in metadata-panel.tsx (2 places):

metadata.ai?.comfy_workflow   // basic check
ai.comfy_workflow             // both the debug div and the viewer condition

---

🐛 Bug #2 — Wrong Import (Component Crashes at Runtime)

litegraph.js has no default export. The current import:

import LiteGraph from 'litegraph.js'; // ❌ — LiteGraph is undefined at runtime

...gives you undefined, so new LiteGraph.LGraph() throws immediately. Change it to named imports:

// ComfyUIWorkflowViewer.tsx
import { LGraph, LGraphCanvas, LGraphNode, LiteGraph } from 'litegraph.js';

Then update all usages in the component:

// Before                          After
new LiteGraph.LGraph()          → new LGraph()
new LiteGraph.LGraphCanvas()    → new LGraphCanvas()
LiteGraph.registered_node_types → LiteGraph.registered_node_types  // ✅ (LiteGraph is still the namespace)
class MinimalNode extends LiteGraph.LGraphNode  → class MinimalNode extends LGraphNode

---

🐛 Bug #3 — fitView() Doesn't Exist (Triggers Error State)

graphCanvas.fitView() is React Flow's API, not litegraph's. Calling it throws TypeError: graphCanvas.fitView is not a function, which gets caught and puts the component into error state (red box).

// ❌ Remove this line:
graphCanvas.fitView(); // React Flow API, not litegraph

// ✅ Replace with litegraph's equivalent:
graphCanvas.ds.offset = [0, 0];
graphCanvas.ds.scale = 1;
graphCanvas.setDirty(true, true);

---

🐛 Bug #4 — Canvas Not Sized in Pixels (Renders Blank/Distorted)

The canvas element uses CSS classes only:

<canvas ref={canvasRef} className="w-full h-full" />

CSS dimensions do not set the canvas backing-store dimensions — the actual canvas.width / canvas.height attributes default to 300×150. LiteGraph renders to those pixel coordinates, so the graph is drawn into a tiny 300×150 buffer that then gets stretched by CSS.

Fix — set explicit pixel dimensions in the useEffect:

useEffect(() => {
  if (!canvasRef.current || !workflow?.nodes) { ... }
  
  const canvas = canvasRef.current;
  // ✅ Set backing-store to the actual rendered size
  const rect = canvas.getBoundingClientRect();
  canvas.width = rect.width || 800;
  canvas.height = rect.height || 400;
  
  const graph = new LGraph();
  // ... rest of setup
}, [workflow, readOnly]);

---

🐛 Bug #5 — graphCanvas.close() Doesn't Exist (Cleanup Warning)

Minor, but worth fixing:

// ❌ litegraph has no .close() method
return () => {
  graph.stop();
  graphCanvas.close(); // throws on unmount
};

// ✅ Just stop the graph:
return () => {
  graph.stop();
};

---

✅ Also: Remove the Debug Div

There's a debug block left in metadata-panel.tsx inside the AI tab that was clearly meant to be temporary:

{ai.comfyui_workflow && (
  <div className="text-[10px] font-mono bg-black/60 text-lime-400 p-2 rounded max-h-24 overflow-auto">
    typeof: {typeof ai.comfyui_workflow} | hasNodes: ...
  </div>
)}

This should be removed before shipping.

---

Summary

Priority	Bug	Effect
1️⃣	comfy_workflow vs comfyui_workflow key mismatch	Viewer never gets data
2️⃣	import LiteGraph from 'litegraph.js' (no default export)	Runtime crash
3️⃣	graphCanvas.fitView() doesn't exist	Error state shown
4️⃣	Canvas missing pixel width/height	Blank/distorted render
5️⃣	graphCanvas.close() doesn't exist	Cleanup console error

Fix #1 first — you'll immediately see whether the workflow data is actually making it through from the PNG. The debug div (which checks hasNodes) can actually help you verify that before you remove it!

(='.'=)
(")_(")  *sniffs out the bug trail*

---

✏️ Learnings added

Learnt from: duskfallcrew
URL: https://github.com/Ktiseos-Nyx/Dataset-Tools/issues/190

Timestamp: 2026-04-11T05:36:16.894Z
Learning: In the nextjs-rewrite branch of Ktiseos-Nyx/Dataset-Tools, the ComfyUI LiteGraph workflow viewer has a key name mismatch: `app/api/metadata/route.ts` stores the parsed ComfyUI workflow JSON as `aiData.comfy_workflow` (no "ui"), but `components/metadata-panel.tsx` and `components/ComfyUIWorkflowViewer.tsx` look for `metadata.ai?.comfyui_workflow` (with "ui"). The fix is to rename the key in route.ts to `comfyui_workflow`.

---

Learnt from: duskfallcrew
URL: https://github.com/Ktiseos-Nyx/Dataset-Tools/issues/190

Timestamp: 2026-04-11T05:36:16.894Z
Learning: In the nextjs-rewrite branch of Ktiseos-Nyx/Dataset-Tools, `components/ComfyUIWorkflowViewer.tsx` imports litegraph.js incorrectly as a default import (`import LiteGraph from 'litegraph.js'`) — litegraph.js has no default export, so `LiteGraph` is undefined at runtime. The correct import is named: `import { LGraph, LGraphCanvas, LGraphNode, LiteGraph } from 'litegraph.js'`. Also, `graphCanvas.fitView()` does not exist in litegraph.js (that is React Flow's API); use `graphCanvas.ds.offset = [0,0]; graphCanvas.ds.scale = 1; graphCanvas.setDirty(true, true)` instead. The canvas element also needs explicit pixel width/height set on it in the useEffect (CSS classes alone don't set canvas backing-store dimensions).

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

---

🧠 Learnings used

Learnt from: duskfallcrew
Repo: Ktiseos-Nyx/Dataset-Tools PR: 0
File: :0-0
Timestamp: 2026-04-09T05:51:02.743Z
Learning: The Dataset-Tools `nextjs-rewrite` branch (github.com/Ktiseos-Nyx/Dataset-Tools/tree/nextjs-rewrite) is a fully standalone Next.js 16 / React 19 / TypeScript app. It does NOT call the Python backend at all — all metadata extraction is implemented in TypeScript inside `app/api/metadata/route.ts`. State management uses plain React useState hooks in `app/page.tsx`; Zustand/Jotai is listed as an open architecture decision in ROADMAP.md but not yet installed. The `coercePromptValue()` function in route.ts and `toDisplayString()` in metadata-panel.tsx already handle dict-to-string coercion defensively.

[ash-bluepollution95]

@coderabbitai Ok well wer'e getting somewhere i'm going to take a stab at "NOWS A GOOD TIME TO LEAVE IT BEFORE I BREAK IT MORE" XD it's "ABOVE MY PAYGRADE" now and claude can poke it in a couple days or maybe Quadmoon can either way wer'e part way there with a custom component!

[coderabbitai]

@duskfallcrew — Totally the right call! Knowing when to put down the shovel is half the battle. 😄

Here's a quick handoff note for Claude/Quadmoon (or future-you) when they pick it up:

---

📋 Where things stand — nextjs-rewrite LiteGraph viewer

What's already been identified (all in this thread):

#	File	Fix needed
1️⃣	app/api/metadata/route.ts	Rename aiData.comfy_workflow → aiData.comfyui_workflow
2️⃣	components/ComfyUIWorkflowViewer.tsx	Change default import to named: import { LGraph, LGraphCanvas, LGraphNode, LiteGraph } from 'litegraph.js'
3️⃣	components/ComfyUIWorkflowViewer.tsx	Replace graphCanvas.fitView() with graphCanvas.ds.offset=[0,0]; graphCanvas.ds.scale=1; graphCanvas.setDirty(true,true)
4️⃣	components/ComfyUIWorkflowViewer.tsx	Set canvas.width/canvas.height in px in useEffect (CSS classes alone don't size the backing store)
5️⃣	components/ComfyUIWorkflowViewer.tsx	Remove graphCanvas.close() — that method doesn't exist in litegraph

Fix #1 first — it's a one-liner and will confirm whether workflow data is flowing through at all before diving into the canvas rendering bugs.

All of this is also saved in the knowledge base so it'll surface automatically when anyone opens a PR touching those files. 🧠

Good luck and don't stress — you got further than you think! 🐰✨

---

🧠 Learnings used

Learnt from: duskfallcrew
Repo: Ktiseos-Nyx/Dataset-Tools PR: 0
File: :0-0
Timestamp: 2026-04-11T05:36:16.894Z
Learning: In the nextjs-rewrite branch of Ktiseos-Nyx/Dataset-Tools, `components/ComfyUIWorkflowViewer.tsx` imports litegraph.js incorrectly as a default import (`import LiteGraph from 'litegraph.js'`) — litegraph.js has no default export, so `LiteGraph` is undefined at runtime. The correct import is named: `import { LGraph, LGraphCanvas, LGraphNode, LiteGraph } from 'litegraph.js'`. Also, `graphCanvas.fitView()` does not exist in litegraph.js (that is React Flow's API); use `graphCanvas.ds.offset = [0,0]; graphCanvas.ds.scale = 1; graphCanvas.setDirty(true, true)` instead. The canvas element also needs explicit pixel width/height set on it in the useEffect (CSS classes alone don't set canvas backing-store dimensions).

---

Learnt from: duskfallcrew
Repo: Ktiseos-Nyx/Dataset-Tools PR: 0
File: :0-0
Timestamp: 2026-04-11T05:36:16.894Z
Learning: In the nextjs-rewrite branch of Ktiseos-Nyx/Dataset-Tools, the ComfyUI LiteGraph workflow viewer has a key name mismatch: `app/api/metadata/route.ts` stores the parsed ComfyUI workflow JSON as `aiData.comfy_workflow` (no "ui"), but `components/metadata-panel.tsx` and `components/ComfyUIWorkflowViewer.tsx` look for `metadata.ai?.comfyui_workflow` (with "ui"). The fix is to rename the key in route.ts to `comfyui_workflow`.

---

Learnt from: duskfallcrew
Repo: Ktiseos-Nyx/Dataset-Tools PR: 0
File: :0-0
Timestamp: 2026-04-09T05:51:02.743Z
Learning: The Dataset-Tools `nextjs-rewrite` branch (github.com/Ktiseos-Nyx/Dataset-Tools/tree/nextjs-rewrite) is a fully standalone Next.js 16 / React 19 / TypeScript app. It does NOT call the Python backend at all — all metadata extraction is implemented in TypeScript inside `app/api/metadata/route.ts`. State management uses plain React useState hooks in `app/page.tsx`; Zustand/Jotai is listed as an open architecture decision in ROADMAP.md but not yet installed. The `coercePromptValue()` function in route.ts and `toDisplayString()` in metadata-panel.tsx already handle dict-to-string coercion defensively.