Pipeline Studio

A full-stack, visual pipeline builder: drag nodes onto a canvas, wire them into a graph, and execute the pipeline on a real backend with live progress. Built from a frontend take-home and extended into a working, secure application with authentication, per-user pipelines, run history, and an extensible node execution engine.

---

Table of Contents

1. About the App
2. Features
3. Tech Stack
4. Quick Start
5. Running in Development
6. Running in Production
7. Configuration
8. Usage Guide
9. Architecture
10. Node & Connection Model
11. Security
12. Testing
13. Project Layout
14. Troubleshooting

---

1. About the App

Pipeline Studio lets a user compose a data pipeline visually. Each node performs one step — read an input, format text, call an LLM, hit an API, do math, filter, transform, or capture output — and edges carry one node's output into the next node's input. When the pipeline is run, the backend executes the graph in topological order and streams live progress back to the canvas.

It is designed to be understandable by non-technical users (friendly node names, icons, descriptions, templates, inline results) while staying a clean, tested, secure codebase underneath.

2. Features

• Visual editor — drag-and-drop nodes, typed handle connections, pan/zoom, minimap, delete, dark mode, empty-state guidance, animated interactions.
• 9 node types — Input, Text ({{variables}}), LLM (Anthropic/OpenAI), API (real HTTP), Math, Filter, Transform, Output, Note.
• Node abstraction — every node is one config entry (nodes/registry.js): Factory + Registry + Strategy. Adding a node = one entry.
• Editable node labels — rename any node; the name shows in the canvas, progress panel, and results.
• Declarative validation — per-field rules (required / pattern / min / max / JSON / key-value / custom) shown inline; the whole pipeline (fields + DAG) is validated before a run.
• Typed connections — handles carry data types (text/number/json/any); incompatible connections are rejected with a hint.
• Real execution engine — backend runs the graph for real: text interpolation, math, filter routing, transforms, HTTP requests, LLM calls.
• Live progress (SSE) — nodes animate (pending → running → success/error) while the backend streams events; results shown inline and in a Results panel.
• Authentication — sign up / sign in (JWT); each user owns their data.
• Multiple saved pipelines — manage at /pipelines (open / rename / delete / save current canvas).
• Run history — every run is recorded per pipeline (/pipelines/:id/history) with inputs, outputs, status, timing — restore any run's exact graph onto the canvas. Last 50 runs kept per pipeline.
• 11 templates — across Basics / Text / Data / Logic / Math / API / LLM (e.g. Mad Libs, Grade Gate, Calculator, Article Summarizer, Translator, Draft & Refine).
• Import / export — pipelines round-trip as JSON (secrets stripped).

3. Tech Stack

Layer	Tech
Frontend	React 18, Vite 5, Tailwind 3, ReactFlow 11, Zustand 4, framer-motion, react-router
Backend	FastAPI, SQLAlchemy 2, SQLite, python-jose (JWT), passlib/argon2, httpx, sse-starlette
Tests	Vitest (frontend), pytest (backend)
Infra	Docker + Docker Compose (dev & prod), nginx (prod static serving)

4. Quick Start

Requires Docker (no host Node/Python needed — everything runs in containers).

docker compose up --build

• Frontend: http://localhost:3000
• Backend: http://localhost:8000

Sign up with any email + password (≥8 chars), open Templates, load Hello World, and click Run Pipeline.

5. Running in Development

docker compose up --build        # Vite hot-reload (:3000) + FastAPI (:8000)
docker compose down              # stop
docker compose exec backend python -m pytest   # backend tests
docker compose exec frontend npm run test       # frontend tests

Dev images include the toolchain (Vite dev server, pytest in-container). Source changes hot-reload.

6. Running in Production

Production uses slim multi-stage images (frontend served as a static bundle by nginx; backend with runtime-only deps, non-root). It requires a strong JWT_SECRET.

JWT_SECRET=$(openssl rand -hex 32) \
  docker compose -f docker-compose.prod.yml up --build

• Frontend (nginx): http://localhost:8080
• Backend: http://localhost:8000
• SQLite persists in the backend_data volume.

Image sizes: frontend ~77 MB (vs ~579 MB dev), backend ~400 MB (pytest-free, non-root). Set CORS_ORIGINS and VITE_API_URL to your real deployment URLs.

7. Configuration

All config is environment-driven (typed via pydantic-settings on the backend). Copy the examples and adjust:

cp backend/.env.example backend/.env
cp frontend/.env.example frontend/.env

Backend (backend/.env.example): APP_ENV, JWT_SECRET (mandatory in prod), JWT_ALGORITHM, ACCESS_TOKEN_TTL_MINUTES, CORS_ORIGINS, DATABASE_URL, MAX_RUNS_PER_PIPELINE, API_NODE_TIMEOUT_SECONDS, API_NODE_MAX_RESPONSE_BYTES, LLM_TIMEOUT_SECONDS, ANTHROPIC_BASE_URL, OPENAI_BASE_URL.

Frontend (frontend/.env.example): VITE_API_URL (backend base URL).

In a non-dev APP_ENV, the backend refuses to start on the default dev JWT_SECRET — set a strong value.

8. Usage Guide

1. Sign up / sign in.
2. Start from a template (Templates button) or drag nodes from the palette.
3. Wire nodes left→right; only type-compatible handles connect.
4. Configure nodes — double-click a node header to rename it. For an LLM node, pick provider/model and paste your own API key (never stored).
5. Run — click Run Pipeline. Watch nodes light up; see results inline and in the Results panel.
6. Save at /pipelines; view past runs at /pipelines/:id/history and Restore any of them to the canvas.

9. Architecture

flowchart LR
    subgraph Browser
      UI["React + ReactFlow canvas"]
      Store["Zustand store<br/>(graph / ui / auth / execution)"]
    end
    subgraph Backend["FastAPI (layered app/)"]
      API["routes: auth, pipelines, execution, runs"]
      SVC["services: auth, pipeline, history, graph, execution engine"]
      DB[("SQLite<br/>User, Pipeline, PipelineRun")]
    end
    UI <--> Store
    Store -->|JWT fetch / SSE| API
    API --> SVC --> DB
    SVC -->|API/LLM nodes| Ext["External APIs / LLM providers"]

Loading

Request lifecycle (a run): the frontend validates the graph → strips secrets into a per-request bag → POSTs to /pipelines/execute/stream → the engine topologically orders the graph, runs each node via its executor, and streams ProgressEvents back over SSE → the store reduces events into live node status → on completion a PipelineRun is recorded (graph + I/O + timing, secrets stripped).

Patterns: Registry + Factory + Strategy (nodes & executors mirror each other), Observer (store), Adapter (api client), layered backend with DI (api/deps.py). See .claude/skills/solid-and-patterns.md.

10. Node & Connection Model

• A node is described by a config in frontend/src/nodes/registry.js: { label, title, accent, icon, description, fields[], handles[], deriveHandles? }.
• Handle ids follow `${nodeId}-<role>`; the backend engine strips the prefix to map an upstream output role to a downstream input role.
• Handles carry a dataType (text | number | json | any); any matches all. Connections between incompatible types are rejected on the canvas.
• The backend executor for each node type self-registers (@register) — adding a node type is one config entry on the frontend + one executor on the backend.

11. Security

Security is the project's first priority (full checklist: .claude/skills/security.md). Highlights:

• User API keys are per-request and never persisted — no secret columns in the DB; the frontend strips secret:true fields (and uploaded file bodies) from localStorage, exports, saved pipelines, and run history; keys live in memory only during execution.
• Errors never leak internals — centralized exception handlers (incl. the SSE generator) return safe messages; full detail goes to logs only.
• SSRF guard on the API node — blocks private/loopback/link-local/metadata IPs (incl. IPv4-mapped IPv6), http(s) only, redirects disabled.
• No arbitrary code execution — Filter/Transform use fixed grammars, never eval.
• Auth — argon2 password hashing, JWT (require exp/sub), owner-scoped data (no cross-user access), non-enumerating login errors, fail-fast on a weak prod secret.
• DoS guards — graph size caps; bounded run history; request size limits.

12. Testing

docker compose exec backend python -m pytest   # 70 tests
docker compose exec frontend npm run test       # 104 tests

Backend (pytest): auth, error redaction, pipeline CRUD, graph algorithms, all node executors, the engine, SSE, run history, and a security-hardening suite (SSRF matrix, JWT forgery, secret-leak, payload caps). Frontend (Vitest): graph, validators, connection types, variable extraction, templates (each is a valid DAG and passes the connection validator), secret stripping, node-id collisions, output formatting, import validation, and SSE parsing/finalize/abort.

13. Project Layout

backend/
  app/
    main.py            app factory (CORS, routers, exception handlers, DB init)
    core/              config (pydantic-settings), security, errors, logging
    db/                models (User, Pipeline, PipelineRun) + session
    repositories/      owner-scoped data access
    schemas/           Pydantic (auth, pipeline, execution, history)
    api/routes/        health, auth, pipelines, execution (sync+SSE), runs
    services/          auth, pipeline, history, graph, execution/{engine,executors/*}
  tests/               pytest
  Dockerfile           dev   |  Dockerfile.prod  multi-stage prod
  requirements*.txt    .env.example

frontend/
  src/
    app/               AuthGate, PipelineApp (router)
    shared/            store (slices), api (client/sse/auth/pipeline/history), lib, ui
    nodes/             BaseNode, registry, nodeTypes, connections, fields/*
    features/          pipeline, auth, execution, templates, pipelines, history
    __tests__/         Vitest
  Dockerfile           dev   |  Dockerfile.prod  build->nginx  |  nginx.conf
  .env.example

docker-compose.yml          dev
docker-compose.prod.yml     prod

14. Troubleshooting

• Login error not showing / blank flash: hard-refresh to clear the cached bundle.
• Run stuck "running": fixed — the stream finalizes on completion/disconnect.
• LLM node fails: ensure a valid provider API key is entered (keys are never stored, so re-enter after a reload/restore).
• API node "URL is not allowed": the SSRF guard blocks internal/private addresses by design; use a public http(s) URL.
• Prod backend won't start: set a real JWT_SECRET (required when APP_ENV != dev).