From e809d5a6fbe56259407e93bf1aa4ec4cf739c35d Mon Sep 17 00:00:00 2001
From: mohabbis <101276427+mohabbis@users.noreply.github.com>
Date: Tue, 4 Aug 2026 16:21:41 -0500
Subject: [PATCH 1/9] =?UTF-8?q?feat(cloud):=20browser=20recording=20captur?=
=?UTF-8?q?e=20=E2=80=94=20Chrome=20extension=20+=20deterministic=20compil?=
=?UTF-8?q?e?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Priority 3. Ghost could execute and approve, but nobody could record — the
only way to get a workflow into Ghost was to upload a trace file produced by
some other tool. There was no capture at all, and `RecordingStatus.ACTIVE` was
never set because nothing recorded.
Per the capture decision in docs/ARCHITECTURE_DECISIONS.md §3, this ships a
Chrome extension for v1, execution stays entirely server-side.
## The trace contract (`@ghost/core/recording/trace`)
A Zod schema for what a recorder uploads: navigate/click/input/select/submit
events, each carrying a `TraceTarget` with accessible role, name, test id and
ordered CSS fallbacks. The defining property: **role and name are read at
capture time**, off the live DOM while the element is still on screen — not
inferred later from an opaque trace. Those are exactly the fields
`resolveLocator` already prefers, so nothing downstream had to change.
A secret is never captured, not encrypted or truncated — absent. `redacted:
true` records that something was typed without recording what. This is the
only place that redaction can be done honestly: a trace carrying the value
plus a "please ignore this" flag has already leaked it (see the P0-2 finding
in the architecture doc, about the previous design uploading traces whole to
a third party).
## The compiler (`@ghost/core/recording/compile`)
Trace to typed steps, deterministically. No model, no network, no configured
compiler. Three things it does beyond translation:
- collapses a run of keystrokes on one field into one `fill` with the final
value
- runs every produced step through `classifyStep` — the same deterministic
classifier the worker consults at run time — and inserts an `approval`
immediately before anything it would gate, so the authored workflow
agrees with what execution will actually do
- never carries a redacted value; a secret field becomes a `fill` marked
`sensitive` with a placeholder, and a note tells the reviewer to set it
This is what makes recording work with no compiler configured — the state
production runs in per the HarnessRouter decision (§1). A model still has a
job: naming steps, proposing gates beyond the obvious cases, flagging what it
could not resolve. Deciding which element was clicked is not a judgement call
and should never have been one.
24 tests, including a fixture shaped exactly like the extension's actual
output (`roundtrip.test.ts`) — the seam most likely to rot, since the
extension is untyped JS with nothing else to notice if its output drifts from
what the compiler expects.
## Ingest (`lib/recording-ingest.ts`)
Shared by the existing upload form and the extension, so upload limits,
filename sanitisation and audit events cannot drift between the two paths. A
structured Ghost trace compiles inline and lands `READY` in the same request;
anything else (HAR, Playwright zip) is stored and left for whatever compiler
is configured, unchanged from before.
`POST /api/agent/recordings` is new: the extension's ingest point, on the
already bearer-authenticated agent surface rather than the session-only
upload route, since the extension runs at a different origin and cannot rely
on the session cookie reaching it. `resolveAgentPrincipal` enforces the same
second factor on its session fallback as every other agent route (see the
P0-1 fix). Uploading only creates a proposal — nothing here publishes a
workflow or executes anything; the human still reviews compiled steps in the
editor and publishes through `POST /api/workflows`, which revalidates them.
## The extension (`apps/extension`)
Manifest V3. `content.js` captures clicks, typing, selects, submits and SPA
navigation, computing accessible name in roughly accname-spec order from
label/aria-label/aria-labelledby/placeholder/value/text. Secret fields are
detected by `type=password`, secret-shaped `autocomplete`, or a name/label/id
matching a word list (password, otp, cvv, card number, ssn, ...) — and the
value is never read for them, not merely withheld after reading.
`background.js` buffers events locally and uploads only on Stop, via a
revocable bearer token created in Ghost Settings. The popup is deliberately
thin — start, stop, where to send it — because a second place to edit steps
would be a second thing to keep in sync with the real editor.
## Validation
398 tests pass (up from 374 — 24 new), against a database migrated from
zero. `pnpm typecheck` and `pnpm build` both clean with `HR_API_KEY` unset.
Manually verified `ingestTrace` end-to-end against a real Postgres: a
structured trace lands `compileStatus: READY` with steps persisted, in one
request, no compiler configured.
## What is not yet covered
`GET /api/recordings` gained a `take: 50` (P2-2 for this route) as an
incidental fix while touching the file; the rest of that finding stands.
Extension host permissions are `` for v1; scoping to allowlisted
origins per organization is a natural follow-up once there is a customer to
scope it for. No component tests for the extension itself — Manifest V3
content scripts have no test harness in this repo, so `roundtrip.test.ts`
pinning its exact output shape is the coverage that exists.
Co-Authored-By: Claude Sonnet 5
---
cloud/apps/extension/README.md | 73 ++++
cloud/apps/extension/manifest.json | 18 +
cloud/apps/extension/src/background.js | 125 +++++++
cloud/apps/extension/src/content.js | 329 ++++++++++++++++++
cloud/apps/extension/src/popup.html | 63 ++++
cloud/apps/extension/src/popup.js | 86 +++++
.../web/src/app/api/agent/recordings/route.ts | 59 ++++
.../apps/web/src/app/api/recordings/route.ts | 78 ++---
cloud/apps/web/src/lib/recording-ingest.ts | 137 ++++++++
cloud/packages/core/package.json | 4 +-
.../core/src/recording/compile.test.ts | 205 +++++++++++
cloud/packages/core/src/recording/compile.ts | 243 +++++++++++++
.../core/src/recording/roundtrip.test.ts | 156 +++++++++
cloud/packages/core/src/recording/trace.ts | 119 +++++++
14 files changed, 1640 insertions(+), 55 deletions(-)
create mode 100644 cloud/apps/extension/README.md
create mode 100644 cloud/apps/extension/manifest.json
create mode 100644 cloud/apps/extension/src/background.js
create mode 100644 cloud/apps/extension/src/content.js
create mode 100644 cloud/apps/extension/src/popup.html
create mode 100644 cloud/apps/extension/src/popup.js
create mode 100644 cloud/apps/web/src/app/api/agent/recordings/route.ts
create mode 100644 cloud/apps/web/src/lib/recording-ingest.ts
create mode 100644 cloud/packages/core/src/recording/compile.test.ts
create mode 100644 cloud/packages/core/src/recording/compile.ts
create mode 100644 cloud/packages/core/src/recording/roundtrip.test.ts
create mode 100644 cloud/packages/core/src/recording/trace.ts
diff --git a/cloud/apps/extension/README.md b/cloud/apps/extension/README.md
new file mode 100644
index 00000000..5ec69b84
--- /dev/null
+++ b/cloud/apps/extension/README.md
@@ -0,0 +1,73 @@
+# Ghost Workflow Recorder (Chrome extension)
+
+Capture. The user demonstrates a workflow once in their own browser; Ghost
+compiles it into typed steps they review and publish.
+
+## Why an extension, and what it deliberately does not do
+
+A Ghost-hosted remote browser is the more elegant answer — recording in the
+same environment the worker replays in removes a whole class of production
+defect — and it lost on cost and time-to-users. See
+`cloud/docs/ARCHITECTURE_DECISIONS.md` §3 for the full comparison and for the
+environment gaps this design has to be engineered against.
+
+**It only records.** Execution stays server-side, in Ghost's workers. An
+extension that also executed would mean Chrome must stay open and the laptop
+awake, and nothing would run overnight — which is most of the value in
+back-office work.
+
+## What it captures
+
+Clicks, typing, dropdown selections, form submissions, and navigation
+(including SPA `pushState`). For each element it reads the **accessible role
+and name** off the live DOM, plus ordered CSS fallbacks.
+
+That is the design's whole point: those are the fields the worker's resolution
+chain prefers, so the trace compiles into steps *deterministically* — no model
+in the correctness path, and recording works in a deployment with no AI
+dependency configured at all.
+
+## What it never captures
+
+- Passwords, one-time codes, card numbers, and anything whose field name,
+ label, or `autocomplete` looks like a secret. The value is **not read**; the
+ step is marked `sensitive` with a placeholder for the reviewer to fill in.
+- Hidden inputs.
+- Session cookies.
+- Cross-origin iframe contents (`all_frames: false`).
+- Coordinates — there is no step field for them, and a coordinate-replayed
+ click is exactly what the semantic selector chain exists to avoid.
+
+Redaction happens at capture because that is the only place it can be done
+honestly. A trace carrying the value plus a "please ignore this" flag has
+already leaked it.
+
+## Install (unpacked)
+
+1. `chrome://extensions` → enable **Developer mode** → **Load unpacked** →
+ select this directory.
+2. Open the popup → **Connection** → set your Ghost URL and an API token
+ created in Ghost under **Settings → Agent credentials** (revocable there at
+ any time).
+3. **Start recording**, do the task, **Stop and send to Ghost**, then follow the
+ review link.
+
+A bearer token rather than the session cookie: the extension is a different
+origin, so a `SameSite=Lax` session cookie is not reliably sent on a cross-site
+POST, and a token can be revoked from Ghost without touching the browser.
+
+## Trust boundary
+
+Uploads go to `POST /api/agent/recordings` — the *propose* side. The extension
+creates a `Recording` whose compiled steps are a proposal. It cannot publish a
+workflow, start a run, or approve anything. A human reviews the steps in the
+editor and publishes through `POST /api/workflows`, which revalidates them.
+
+## The contract
+
+The trace format is `@ghost/core/recording/trace`; the compiler is
+`@ghost/core/recording/compile`. This extension is plain JavaScript and is not
+typechecked against them, so `packages/core/src/recording/roundtrip.test.ts`
+pins a fixture in exactly the shape `content.js` emits. If that test fails, the
+recorder and Ghost have disagreed about the format — fix one of them rather
+than relaxing the test.
diff --git a/cloud/apps/extension/manifest.json b/cloud/apps/extension/manifest.json
new file mode 100644
index 00000000..10fb0282
--- /dev/null
+++ b/cloud/apps/extension/manifest.json
@@ -0,0 +1,18 @@
+{
+ "manifest_version": 3,
+ "name": "Ghost Workflow Recorder",
+ "version": "0.1.0",
+ "description": "Record a business workflow once in your browser, then review and publish it in Ghost.",
+ "permissions": ["activeTab", "scripting", "storage", "tabs"],
+ "host_permissions": [""],
+ "background": { "service_worker": "src/background.js", "type": "module" },
+ "action": { "default_popup": "src/popup.html", "default_title": "Ghost Workflow Recorder" },
+ "content_scripts": [
+ {
+ "matches": [""],
+ "js": ["src/content.js"],
+ "run_at": "document_idle",
+ "all_frames": false
+ }
+ ]
+}
diff --git a/cloud/apps/extension/src/background.js b/cloud/apps/extension/src/background.js
new file mode 100644
index 00000000..6ba919df
--- /dev/null
+++ b/cloud/apps/extension/src/background.js
@@ -0,0 +1,125 @@
+/**
+ * Ghost recorder — collection and upload.
+ *
+ * Holds the event buffer for a recording session and, on stop, posts the trace
+ * to Ghost. Nothing is uploaded while recording: a partial trace is not a
+ * workflow, and streaming would mean a half-recorded session could be reviewed
+ * and published.
+ *
+ * Auth is a revocable bearer credential the user creates in Ghost Settings,
+ * not the session cookie. The extension is a different origin, so a
+ * `SameSite=Lax` session cookie is not reliably sent on a cross-site POST —
+ * and a token can be revoked from Ghost without touching the browser.
+ *
+ * It uploads to `/api/agent/recordings`, the "propose" side of the trust
+ * boundary. The extension cannot publish a workflow or approve anything; it
+ * hands over a proposal for a human to review.
+ */
+
+const STORAGE = chrome.storage.local;
+
+async function getState() {
+ return STORAGE.get(["recording", "events", "ghostUrl", "token", "startedAt"]);
+}
+
+chrome.runtime.onMessage.addListener((message, _sender, sendResponse) => {
+ if (message?.kind === "ghost-event") {
+ // Fire-and-forget from the page's perspective; the append is async.
+ void appendEvent(message.event);
+ return false;
+ }
+
+ if (message?.kind === "ghost-start") {
+ void start().then(sendResponse);
+ return true; // async response
+ }
+
+ if (message?.kind === "ghost-stop") {
+ void stop().then(sendResponse);
+ return true;
+ }
+
+ if (message?.kind === "ghost-status") {
+ void getState().then((s) =>
+ sendResponse({
+ recording: Boolean(s.recording),
+ count: (s.events || []).length,
+ configured: Boolean(s.ghostUrl && s.token),
+ }),
+ );
+ return true;
+ }
+
+ return false;
+});
+
+/** Cap the buffer so a forgotten recording cannot grow without bound. */
+const MAX_EVENTS = 5000;
+
+async function appendEvent(event) {
+ const { recording, events } = await STORAGE.get(["recording", "events"]);
+ if (!recording) return;
+ const next = events || [];
+ if (next.length >= MAX_EVENTS) return;
+ next.push(event);
+ await STORAGE.set({ events: next });
+ chrome.action.setBadgeText({ text: String(next.length) });
+ chrome.action.setBadgeBackgroundColor({ color: "#b45309" });
+}
+
+async function start() {
+ await STORAGE.set({ recording: true, events: [], startedAt: Date.now() });
+ chrome.action.setBadgeText({ text: "0" });
+ return { ok: true };
+}
+
+async function stop() {
+ const state = await getState();
+ await STORAGE.set({ recording: false });
+ chrome.action.setBadgeText({ text: "" });
+
+ const events = state.events || [];
+ if (events.length === 0) {
+ return { ok: false, error: "Nothing was recorded." };
+ }
+ if (!state.ghostUrl || !state.token) {
+ return { ok: false, error: "Set your Ghost URL and API token first." };
+ }
+
+ const trace = {
+ version: 1,
+ sessionId: `rec_${state.startedAt || Date.now()}`,
+ startUrl: events.find((e) => e.url)?.url,
+ capturedAt: Date.now(),
+ recorder: "ghost-chrome-extension/0.1.0",
+ events,
+ };
+
+ try {
+ const response = await fetch(`${state.ghostUrl.replace(/\/$/, "")}/api/agent/recordings`, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ Authorization: `Bearer ${state.token}`,
+ },
+ body: JSON.stringify(trace),
+ });
+
+ const body = await response.json().catch(() => null);
+ if (!response.ok) {
+ return { ok: false, error: body?.error || `Ghost returned ${response.status}.` };
+ }
+
+ await STORAGE.set({ events: [] });
+ return {
+ ok: true,
+ id: body.id,
+ stepCount: body.stepCount,
+ reviewUrl: `${state.ghostUrl.replace(/\/$/, "")}/recordings/${body.id}`,
+ };
+ } catch (err) {
+ // The events are deliberately kept so a failed upload can be retried
+ // rather than discarding a recording the user cannot easily repeat.
+ return { ok: false, error: `Could not reach Ghost: ${err.message}` };
+ }
+}
diff --git a/cloud/apps/extension/src/content.js b/cloud/apps/extension/src/content.js
new file mode 100644
index 00000000..6b3554f9
--- /dev/null
+++ b/cloud/apps/extension/src/content.js
@@ -0,0 +1,329 @@
+/**
+ * Ghost recorder — page-side capture.
+ *
+ * Emits the trace format in `@ghost/core/recording/trace`, whose defining
+ * property is that **accessible role and name are read here**, off the live
+ * element while it is still on screen. Those are the fields the worker's
+ * resolution chain prefers, so the trace compiles into steps deterministically
+ * with no model in the correctness path. Inferring them later from a HAR is
+ * guesswork; reading them now is not.
+ *
+ * Two rules this file must not break:
+ *
+ * 1. **A secret never enters the trace.** Not encrypted, not truncated —
+ * absent. Redaction happens at capture because it is the only place it can
+ * be done honestly: a trace carrying the value plus a "please ignore"
+ * flag has already leaked it. `redacted: true` records that something was
+ * typed without recording what.
+ * 2. **No coordinates, ever.** There is no step field for them, and a
+ * coordinate-replayed click is the failure mode the whole semantic
+ * selector chain exists to avoid. An element we cannot name is reported as
+ * unidentifiable rather than approximated.
+ *
+ * Runs in the top frame only (`all_frames: false`). Cross-origin iframe
+ * contents are deliberately out of scope.
+ */
+
+(() => {
+ if (window.__ghostRecorderInstalled) return;
+ window.__ghostRecorderInstalled = true;
+
+ let recording = false;
+
+ chrome.storage.local.get(["recording"], (state) => {
+ recording = Boolean(state.recording);
+ });
+
+ chrome.storage.onChanged.addListener((changes, area) => {
+ if (area === "local" && changes.recording) recording = Boolean(changes.recording.newValue);
+ });
+
+ const send = (event) => {
+ if (!recording) return;
+ try {
+ chrome.runtime.sendMessage({ kind: "ghost-event", event });
+ } catch {
+ // The service worker may be asleep or the extension reloading. Dropping
+ // one event is better than throwing inside a page's event handler.
+ }
+ };
+
+ // --- sensitivity -------------------------------------------------------
+
+ const SECRET_HINTS =
+ /pass(word|code)|otp|one[-_ ]?time|2fa|mfa|cvv|cvc|card[-_ ]?number|credit|account[-_ ]?number|routing|ssn|social[-_ ]?security|\bpin\b|secret|token|security[-_ ]?code/i;
+
+ const SECRET_AUTOCOMPLETE =
+ /current-password|new-password|one-time-code|cc-number|cc-csc|cc-exp/i;
+
+ /** Whether this field's *value* must never be captured. */
+ function isSecretField(el) {
+ if (!el) return false;
+ const type = (el.getAttribute("type") || "").toLowerCase();
+ if (type === "password") return true;
+ if (SECRET_AUTOCOMPLETE.test(el.getAttribute("autocomplete") || "")) return true;
+
+ const haystack = [
+ el.getAttribute("name"),
+ el.getAttribute("id"),
+ el.getAttribute("placeholder"),
+ el.getAttribute("aria-label"),
+ accessibleName(el),
+ ]
+ .filter(Boolean)
+ .join(" ");
+ return SECRET_HINTS.test(haystack);
+ }
+
+ /** Fields that are never recorded at all, value or otherwise. */
+ function isIgnorableField(el) {
+ const type = (el.getAttribute("type") || "").toLowerCase();
+ return type === "hidden";
+ }
+
+ // --- accessible role and name -----------------------------------------
+
+ function implicitRole(el) {
+ const tag = el.tagName.toLowerCase();
+ const type = (el.getAttribute("type") || "").toLowerCase();
+ if (tag === "a" && el.hasAttribute("href")) return "link";
+ if (tag === "button") return "button";
+ if (tag === "select") return "combobox";
+ if (tag === "textarea") return "textbox";
+ if (tag === "input") {
+ if (["submit", "button", "reset", "image"].includes(type)) return "button";
+ if (type === "checkbox") return "checkbox";
+ if (type === "radio") return "radio";
+ if (["text", "email", "tel", "url", "search", "password", "number", ""].includes(type)) {
+ return "textbox";
+ }
+ }
+ if (/^h[1-6]$/.test(tag)) return "heading";
+ return undefined;
+ }
+
+ function role(el) {
+ return el.getAttribute("role") || implicitRole(el);
+ }
+
+ function textOf(el) {
+ return (el.textContent || "").replace(/\s+/g, " ").trim();
+ }
+
+ /**
+ * Accessible name, in roughly the order the accname spec resolves it.
+ *
+ * Not a complete implementation — that is a large specification — but it
+ * covers the attributes real applications actually use, and it is computed
+ * from the same DOM Playwright's `getByRole(role, { name })` will later
+ * query, which is what matters for replay.
+ */
+ function accessibleName(el) {
+ if (!el || el.nodeType !== 1) return "";
+
+ const labelledBy = el.getAttribute("aria-labelledby");
+ if (labelledBy) {
+ const parts = labelledBy
+ .split(/\s+/)
+ .map((id) => document.getElementById(id))
+ .filter(Boolean)
+ .map(textOf)
+ .filter(Boolean);
+ if (parts.length) return parts.join(" ");
+ }
+
+ const ariaLabel = el.getAttribute("aria-label");
+ if (ariaLabel && ariaLabel.trim()) return ariaLabel.trim();
+
+ if (el.id) {
+ const label = document.querySelector(`label[for="${CSS.escape(el.id)}"]`);
+ if (label) {
+ const t = textOf(label);
+ if (t) return t;
+ }
+ }
+
+ const wrapping = el.closest("label");
+ if (wrapping) {
+ const t = textOf(wrapping);
+ if (t) return t;
+ }
+
+ const tag = el.tagName.toLowerCase();
+ if (tag === "input") {
+ const type = (el.getAttribute("type") || "").toLowerCase();
+ if (["submit", "button", "reset"].includes(type)) {
+ const v = el.getAttribute("value");
+ if (v && v.trim()) return v.trim();
+ }
+ const placeholder = el.getAttribute("placeholder");
+ if (placeholder && placeholder.trim()) return placeholder.trim();
+ }
+
+ if (tag === "img") {
+ const alt = el.getAttribute("alt");
+ if (alt && alt.trim()) return alt.trim();
+ }
+
+ const title = el.getAttribute("title");
+ if (title && title.trim()) return title.trim();
+
+ // Buttons and links are named by their own text.
+ if (["button", "a", "summary"].includes(tag) || el.getAttribute("role") === "button") {
+ const t = textOf(el);
+ if (t) return t.slice(0, 200);
+ }
+
+ return "";
+ }
+
+ // --- selector candidates ----------------------------------------------
+
+ function cssPath(el) {
+ const parts = [];
+ let node = el;
+ let depth = 0;
+ while (node && node.nodeType === 1 && depth < 5) {
+ let part = node.tagName.toLowerCase();
+ if (node.id) {
+ parts.unshift(`#${CSS.escape(node.id)}`);
+ break;
+ }
+ const parent = node.parentElement;
+ if (parent) {
+ const siblings = Array.from(parent.children).filter((c) => c.tagName === node.tagName);
+ if (siblings.length > 1) part += `:nth-of-type(${siblings.indexOf(node) + 1})`;
+ }
+ parts.unshift(part);
+ node = node.parentElement;
+ depth++;
+ }
+ return parts.join(" > ");
+ }
+
+ /**
+ * Ordered best-first CSS fallbacks. Several, not one, because the worker
+ * replays in a different browser and a single brittle selector is the
+ * failure that invites.
+ */
+ function selectorCandidates(el) {
+ const out = [];
+ const testId =
+ el.getAttribute("data-testid") ||
+ el.getAttribute("data-test-id") ||
+ el.getAttribute("data-test");
+ if (testId) out.push(`[data-testid="${CSS.escape(testId)}"]`);
+ if (el.id) out.push(`#${CSS.escape(el.id)}`);
+ const name = el.getAttribute("name");
+ if (name) out.push(`${el.tagName.toLowerCase()}[name="${CSS.escape(name)}"]`);
+ const path = cssPath(el);
+ if (path) out.push(path);
+ return out.slice(0, 5);
+ }
+
+ function describeTarget(el) {
+ return {
+ role: role(el) || undefined,
+ name: accessibleName(el) || undefined,
+ testId:
+ el.getAttribute("data-testid") ||
+ el.getAttribute("data-test-id") ||
+ el.getAttribute("data-test") ||
+ undefined,
+ text: textOf(el).slice(0, 120) || undefined,
+ selectorCandidates: selectorCandidates(el),
+ tagName: el.tagName.toLowerCase(),
+ inputType: (el.getAttribute("type") || "").toLowerCase() || undefined,
+ };
+ }
+
+ const now = () => Date.now();
+
+ // --- listeners ---------------------------------------------------------
+
+ // Capture phase: a page that calls stopPropagation on its own handlers
+ // would otherwise make the workflow unrecordable.
+ document.addEventListener(
+ "click",
+ (e) => {
+ const el = e.target instanceof Element ? e.target.closest("a,button,[role],input,summary,label") || e.target : null;
+ if (!el || el.nodeType !== 1) return;
+ if (isIgnorableField(el)) return;
+ send({ type: "click", url: location.href, timestamp: now(), target: describeTarget(el) });
+ },
+ true,
+ );
+
+ document.addEventListener(
+ "change",
+ (e) => {
+ const el = e.target;
+ if (!(el instanceof Element) || isIgnorableField(el)) return;
+ const tag = el.tagName.toLowerCase();
+
+ if (tag === "select") {
+ send({
+ type: "select",
+ url: location.href,
+ timestamp: now(),
+ target: describeTarget(el),
+ value: el.value ?? "",
+ });
+ return;
+ }
+
+ if (tag === "input" || tag === "textarea") {
+ const type = (el.getAttribute("type") || "").toLowerCase();
+ if (["checkbox", "radio", "submit", "button", "file"].includes(type)) return;
+
+ const secret = isSecretField(el);
+ send({
+ type: "input",
+ url: location.href,
+ timestamp: now(),
+ target: describeTarget(el),
+ // The whole point: on a secret field the value is not read at all.
+ ...(secret ? {} : { value: el.value ?? "" }),
+ redacted: secret,
+ });
+ }
+ },
+ true,
+ );
+
+ document.addEventListener(
+ "submit",
+ (e) => {
+ const el = e.target instanceof Element ? e.target : null;
+ send({
+ type: "submit",
+ url: location.href,
+ timestamp: now(),
+ ...(el ? { target: describeTarget(el) } : {}),
+ });
+ },
+ true,
+ );
+
+ // SPA navigation: pushState/replaceState fire no event of their own.
+ let lastUrl = location.href;
+ const reportNavigation = () => {
+ if (location.href === lastUrl) return;
+ lastUrl = location.href;
+ send({ type: "navigate", url: location.href, timestamp: now() });
+ };
+
+ for (const method of ["pushState", "replaceState"]) {
+ const original = history[method];
+ history[method] = function patched(...args) {
+ const result = original.apply(this, args);
+ reportNavigation();
+ return result;
+ };
+ }
+ window.addEventListener("popstate", reportNavigation);
+ window.addEventListener("hashchange", reportNavigation);
+
+ // The initial page of a recording, so the compiled workflow opens somewhere.
+ send({ type: "navigate", url: location.href, timestamp: now() });
+})();
diff --git a/cloud/apps/extension/src/popup.html b/cloud/apps/extension/src/popup.html
new file mode 100644
index 00000000..0209dcfe
--- /dev/null
+++ b/cloud/apps/extension/src/popup.html
@@ -0,0 +1,63 @@
+
+
+
+
+
+
+
+
Ghost Workflow Recorder
+
+
+
+
+
+ Connection
+
+
+
+
+
+ Create a revocable token in Ghost under Settings → Agent credentials.
+
+
+
+
+
+
diff --git a/cloud/apps/extension/src/popup.js b/cloud/apps/extension/src/popup.js
new file mode 100644
index 00000000..09334238
--- /dev/null
+++ b/cloud/apps/extension/src/popup.js
@@ -0,0 +1,86 @@
+/**
+ * Ghost recorder — popup.
+ *
+ * Deliberately thin: start, stop, and where to send it. The review step is in
+ * Ghost, not here. An extension that let you edit steps would be a second
+ * editor to keep in sync with the real one, and the whole point of the trust
+ * pipeline is that a proposal is reviewed and published in one place.
+ */
+
+const toggle = document.getElementById("toggle");
+const status = document.getElementById("status");
+const urlInput = document.getElementById("url");
+const tokenInput = document.getElementById("token");
+const settings = document.getElementById("settings");
+
+let recording = false;
+
+function say(html) {
+ status.innerHTML = html;
+}
+
+async function refresh() {
+ const state = await chrome.storage.local.get(["ghostUrl", "token"]);
+ urlInput.value = state.ghostUrl || "";
+ tokenInput.value = state.token || "";
+
+ chrome.runtime.sendMessage({ kind: "ghost-status" }, (s) => {
+ recording = Boolean(s?.recording);
+ toggle.textContent = recording ? "Stop and send to Ghost" : "Start recording";
+ toggle.classList.toggle("rec", recording);
+ if (recording) {
+ say(`Recording — ${s.count} event${s.count === 1 ? "" : "s"} captured.`);
+ } else if (!s?.configured) {
+ say("Set your Ghost URL and API token below to begin.");
+ settings.open = true;
+ } else {
+ say("Ready. Recording captures clicks, typing and navigation — never passwords.");
+ }
+ });
+}
+
+const persist = () =>
+ chrome.storage.local.set({
+ ghostUrl: urlInput.value.trim(),
+ token: tokenInput.value.trim(),
+ });
+
+urlInput.addEventListener("change", persist);
+tokenInput.addEventListener("change", persist);
+
+toggle.addEventListener("click", async () => {
+ await persist();
+ toggle.disabled = true;
+
+ if (!recording) {
+ chrome.runtime.sendMessage({ kind: "ghost-start" }, async () => {
+ // The content script is injected at document_idle, so a tab already open
+ // before the extension was installed has no recorder in it yet.
+ const [tab] = await chrome.tabs.query({ active: true, currentWindow: true });
+ if (tab?.id) {
+ await chrome.scripting
+ .executeScript({ target: { tabId: tab.id }, files: ["src/content.js"] })
+ .catch(() => undefined);
+ }
+ toggle.disabled = false;
+ refresh();
+ });
+ return;
+ }
+
+ say("Sending to Ghost…");
+ chrome.runtime.sendMessage({ kind: "ghost-stop" }, (result) => {
+ toggle.disabled = false;
+ if (result?.ok) {
+ say(
+ `Sent. ${result.stepCount} step${result.stepCount === 1 ? "" : "s"} proposed — ` +
+ `review in Ghost.`,
+ );
+ } else {
+ say(`Could not send: ${result?.error || "unknown error"}. Your recording was kept.`);
+ }
+ refresh();
+ });
+});
+
+refresh();
diff --git a/cloud/apps/web/src/app/api/agent/recordings/route.ts b/cloud/apps/web/src/app/api/agent/recordings/route.ts
new file mode 100644
index 00000000..b45c7def
--- /dev/null
+++ b/cloud/apps/web/src/app/api/agent/recordings/route.ts
@@ -0,0 +1,59 @@
+import { resolveAgentPrincipal } from "@/lib/agent-auth";
+import { ingestTrace, MAX_TRACE_BYTES } from "@/lib/recording-ingest";
+
+/**
+ * Recording ingest for non-interactive clients — the Chrome extension.
+ *
+ * Lives under `/api/agent` because that is the surface authenticated by a
+ * revocable bearer credential rather than a browser session. The extension
+ * runs outside Ghost's origin and cannot rely on the session cookie reaching
+ * it, so it holds a token the user creates in Settings and can revoke there.
+ *
+ * It sits on the "propose" side of the trust boundary, like every other agent
+ * route: uploading a recording creates a `Recording` whose compiled steps are
+ * a *proposal*. Nothing here publishes a workflow or executes anything. The
+ * human reviews the steps in the editor and publishes through
+ * `POST /api/workflows`, which revalidates them.
+ *
+ * `resolveAgentPrincipal` accepts a session too, and enforces the second
+ * factor on that path — see `lib/agent-auth.ts`.
+ *
+ * Body is JSON rather than multipart: the extension builds the trace in
+ * memory and has no file to attach.
+ */
+export async function POST(req: Request) {
+ const principal = await resolveAgentPrincipal(req);
+ if (!principal.ok) {
+ return Response.json({ error: principal.error }, { status: principal.status });
+ }
+
+ const raw = await req.text();
+ if (!raw) {
+ return Response.json({ error: "a JSON recording trace is required" }, { status: 400 });
+ }
+ if (Buffer.byteLength(raw) > MAX_TRACE_BYTES) {
+ return Response.json(
+ { error: `trace exceeds the ${MAX_TRACE_BYTES / (1024 * 1024)}MB limit` },
+ { status: 413 },
+ );
+ }
+
+ // Parsed only to reject obvious junk before it reaches storage. The
+ // authoritative validation is `parseRecordingTrace` inside `ingestTrace`,
+ // which decides whether this is a Ghost trace worth compiling.
+ try {
+ JSON.parse(raw);
+ } catch {
+ return Response.json({ error: "body was not valid JSON" }, { status: 400 });
+ }
+
+ const result = await ingestTrace({
+ orgId: principal.principal.orgId,
+ userId: principal.principal.userId,
+ filename: "extension-trace.json",
+ contentType: "application/json",
+ buffer: Buffer.from(raw, "utf8"),
+ });
+
+ return Response.json(result, { status: 201 });
+}
diff --git a/cloud/apps/web/src/app/api/recordings/route.ts b/cloud/apps/web/src/app/api/recordings/route.ts
index 9b88253a..9f84b6e3 100644
--- a/cloud/apps/web/src/app/api/recordings/route.ts
+++ b/cloud/apps/web/src/app/api/recordings/route.ts
@@ -1,23 +1,20 @@
-import { appendAuditEvent } from "@ghost/core/audit-log";
-import { artifactStore } from "@ghost/core/storage/artifacts";
import { auth } from "@/auth";
import { prisma } from "@/lib/db";
+import { ingestTrace, MAX_TRACE_BYTES } from "@/lib/recording-ingest";
-/** Recording traces are small event/HAR/trace-zip logs, not raw video — cap
- * well under Vercel's 100MB function body limit to keep uploads fast and
- * keep an accidental video upload from tying up the compile step. */
-const MAX_TRACE_BYTES = 25 * 1024 * 1024;
-
-function sanitizeFilename(name: string): string {
- const base = name.split(/[/\\]/).pop() || "recording-trace";
- return base.replace(/[^a-zA-Z0-9._-]/g, "_").slice(-200) || "recording-trace";
-}
-
-/** Upload a raw workflow-recording trace and create the `Recording` it will
- * be compiled from. The capture mechanism itself (server-side remote browser
- * vs. extension — see `docs/CURSOR_HANDOFF.md` Phase 2) is still an open
- * decision; this accepts a trace produced by whatever means already exists
- * (a JSON event log, a HAR export, a Playwright trace .zip). */
+/**
+ * Upload a workflow-recording trace and create the `Recording` it will be
+ * reviewed from.
+ *
+ * A trace produced by a Ghost recorder (`@ghost/core/recording/trace`) is
+ * compiled into typed steps deterministically, in this request — no model and
+ * no configured compiler, which is what makes recording work in a production
+ * deployment that has neither. Any other format (HAR, Playwright zip) is stored
+ * and left for whatever compiler is configured, if one is.
+ *
+ * The extension posts to `/api/agent/recordings` instead, which accepts a
+ * bearer credential; both share `ingestTrace`.
+ */
export async function POST(req: Request) {
const session = await auth();
if (!session?.user?.orgId) {
@@ -29,7 +26,7 @@ export async function POST(req: Request) {
const form = await req.formData().catch(() => null);
const file = form?.get("file");
if (!file || !(file instanceof File)) {
- return Response.json({ error: "a \"file\" upload is required" }, { status: 400 });
+ return Response.json({ error: 'a "file" upload is required' }, { status: 400 });
}
if (file.size === 0) {
return Response.json({ error: "the uploaded file is empty" }, { status: 400 });
@@ -41,41 +38,15 @@ export async function POST(req: Request) {
);
}
- const filename = sanitizeFilename(file.name || "recording-trace");
- const buffer = Buffer.from(await file.arrayBuffer());
-
- const recording = await prisma.recording.create({
- data: { orgId, status: "STOPPED" },
+ const result = await ingestTrace({
+ orgId,
+ userId,
+ filename: file.name || "recording-trace",
+ contentType: file.type,
+ buffer: Buffer.from(await file.arrayBuffer()),
});
- try {
- const key = `recordings/${recording.id}/trace-${filename}`;
- await artifactStore().put(key, buffer, file.type || "application/octet-stream");
- await prisma.$transaction(async (tx) => {
- await tx.recording.update({
- where: { id: recording.id },
- data: { rawTraceKey: key, rawTraceFilename: filename },
- });
- await appendAuditEvent(
- orgId,
- userId,
- {
- action: "recording.uploaded",
- entityType: "Recording",
- entityId: recording.id,
- metadata: { filename, bytes: buffer.byteLength },
- },
- tx,
- );
- });
- } catch (err) {
- // Storage failed after the row was created — don't leave an unusable
- // Recording with no trace behind for the org to trip over.
- await prisma.recording.delete({ where: { id: recording.id } }).catch(() => undefined);
- throw err;
- }
-
- return Response.json({ id: recording.id }, { status: 201 });
+ return Response.json(result, { status: 201 });
}
export async function GET() {
@@ -83,19 +54,18 @@ export async function GET() {
if (!session?.user?.orgId) {
return Response.json({ error: "unauthorized" }, { status: 401 });
}
-
const recordings = await prisma.recording.findMany({
where: { orgId: session.user.orgId },
orderBy: { createdAt: "desc" },
+ take: 50,
select: {
id: true,
+ createdAt: true,
status: true,
compileStatus: true,
rawTraceFilename: true,
workflowId: true,
- createdAt: true,
},
});
-
return Response.json({ recordings });
}
diff --git a/cloud/apps/web/src/lib/recording-ingest.ts b/cloud/apps/web/src/lib/recording-ingest.ts
new file mode 100644
index 00000000..61f8080e
--- /dev/null
+++ b/cloud/apps/web/src/lib/recording-ingest.ts
@@ -0,0 +1,137 @@
+import { appendAuditEvent } from "@ghost/core/audit-log";
+import { artifactStore } from "@ghost/core/storage/artifacts";
+import { Prisma } from "@ghost/core/db";
+import { compileTrace } from "@ghost/core/recording/compile";
+import { parseRecordingTrace } from "@ghost/core/recording/trace";
+import { prisma } from "@/lib/db";
+
+/**
+ * Storing an uploaded recording trace, shared by the browser upload form and
+ * the extension's ingest endpoint so the two cannot drift on limits,
+ * sanitisation, or what gets audited.
+ *
+ * The interesting part is what happens to a *structured* trace. A Ghost
+ * recorder reads the accessible role and name off each element while it is
+ * still on screen, so the trace already contains what the worker's resolution
+ * chain wants. `compileTrace` then turns it into typed steps deterministically
+ * — no model, no network, no configured compiler — and the recording lands
+ * `READY` for review in a single request.
+ *
+ * That is what makes recording work in production, where there is deliberately
+ * no compiler configured at all (see `docs/DEPLOY.md`). Anything that is *not*
+ * a structured trace — a HAR, a Playwright zip — is stored as before and left
+ * for whatever compiler exists, if any.
+ */
+
+/** Small event logs, not video. Well under Vercel's 100MB body limit. */
+export const MAX_TRACE_BYTES = 25 * 1024 * 1024;
+
+export function sanitizeFilename(name: string): string {
+ const base = name.split(/[/\\]/).pop() || "recording-trace";
+ return base.replace(/[^a-zA-Z0-9._-]/g, "_").slice(-200) || "recording-trace";
+}
+
+export interface IngestResult {
+ id: string;
+ /** READY when the trace compiled deterministically, NONE when it awaits a compiler. */
+ compileStatus: "READY" | "NONE";
+ stepCount: number;
+ notes: string[];
+}
+
+export async function ingestTrace(args: {
+ orgId: string;
+ userId: string | null;
+ filename: string;
+ contentType: string;
+ buffer: Buffer;
+}): Promise {
+ const { orgId, userId, buffer } = args;
+ const filename = sanitizeFilename(args.filename);
+
+ const recording = await prisma.recording.create({
+ data: { orgId, status: "STOPPED" },
+ });
+
+ try {
+ const key = `recordings/${recording.id}/trace-${filename}`;
+ await artifactStore().put(key, buffer, args.contentType || "application/octet-stream");
+
+ const compiled = tryCompile(buffer);
+
+ await prisma.$transaction(async (tx) => {
+ await tx.recording.update({
+ where: { id: recording.id },
+ data: {
+ rawTraceKey: key,
+ rawTraceFilename: filename,
+ ...(compiled
+ ? {
+ compileStatus: "READY" as const,
+ compiledSteps: compiled.steps as unknown as Prisma.InputJsonValue,
+ compileNotes: compiled.notes.length > 0 ? compiled.notes.join("\n\n") : null,
+ compileError: null,
+ }
+ : {}),
+ },
+ });
+ await appendAuditEvent(
+ orgId,
+ userId,
+ {
+ action: "recording.uploaded",
+ entityType: "Recording",
+ entityId: recording.id,
+ metadata: { filename, bytes: buffer.byteLength },
+ },
+ tx,
+ );
+ if (compiled) {
+ // Audited as a compile in its own right. A reviewer looking at how a
+ // workflow came to exist should see that a compiler ran, even though
+ // it ran inline and deterministically rather than as a queued task.
+ await appendAuditEvent(
+ orgId,
+ userId,
+ {
+ action: "recording.compile_ready",
+ entityType: "Recording",
+ entityId: recording.id,
+ metadata: { stepCount: compiled.steps.length, compiler: "deterministic" },
+ },
+ tx,
+ );
+ }
+ });
+
+ return {
+ id: recording.id,
+ compileStatus: compiled ? "READY" : "NONE",
+ stepCount: compiled?.steps.length ?? 0,
+ notes: compiled?.notes ?? [],
+ };
+ } catch (err) {
+ // Storage failed after the row was created — don't leave an unusable
+ // Recording with no trace behind for the org to trip over.
+ await prisma.recording.delete({ where: { id: recording.id } }).catch(() => undefined);
+ throw err;
+ }
+}
+
+/** Returns compiled steps for a structured Ghost trace, or null for anything else. */
+function tryCompile(buffer: Buffer): { steps: unknown[]; notes: string[] } | null {
+ let json: unknown;
+ try {
+ json = JSON.parse(buffer.toString("utf8"));
+ } catch {
+ return null; // a zip, a HAR that isn't ours, or not JSON at all
+ }
+ const parsed = parseRecordingTrace(json);
+ if (!parsed.ok) return null;
+
+ const { steps, notes } = compileTrace(parsed.trace);
+ // A trace with no replayable action compiles to just the opening navigate.
+ // Storing that as READY would present an empty workflow as a result.
+ if (steps.length <= 1) return null;
+ return { steps, notes };
+}
diff --git a/cloud/packages/core/package.json b/cloud/packages/core/package.json
index bea36179..a92a2b6b 100644
--- a/cloud/packages/core/package.json
+++ b/cloud/packages/core/package.json
@@ -26,7 +26,9 @@
"./roles": "./src/roles.ts",
"./invitations": "./src/invitations.ts",
"./mfa": "./src/mfa.ts",
- "./crypto/mfa-secret": "./src/crypto/mfa-secret.ts"
+ "./crypto/mfa-secret": "./src/crypto/mfa-secret.ts",
+ "./recording/trace": "./src/recording/trace.ts",
+ "./recording/compile": "./src/recording/compile.ts"
},
"scripts": {
"build": "prisma generate && tsc --noEmit",
diff --git a/cloud/packages/core/src/recording/compile.test.ts b/cloud/packages/core/src/recording/compile.test.ts
new file mode 100644
index 00000000..1a8d9ccc
--- /dev/null
+++ b/cloud/packages/core/src/recording/compile.test.ts
@@ -0,0 +1,205 @@
+import { describe, expect, it } from "vitest";
+import { compileTrace } from "./compile.js";
+import { parseRecordingTrace, type RecordingTrace } from "./trace.js";
+import { EDITABLE_STEP_TYPES, workflowSteps } from "../schema/step.js";
+import { classifyStep } from "../classifier/sensitive.js";
+
+/**
+ * The deterministic path from a captured trace to typed steps.
+ *
+ * This is what makes recording work with no compiler configured — the case
+ * production runs in. If these pass, a customer can record a workflow in a
+ * deployment that has no AI dependency of any kind.
+ */
+
+function trace(events: RecordingTrace["events"], startUrl = "https://shop.test/order"): RecordingTrace {
+ return { version: 1, sessionId: "rec_test", startUrl, events };
+}
+
+const target = (name: string, role = "button") => ({
+ role,
+ name,
+ selectorCandidates: [`#${name.replace(/\s+/g, "-").toLowerCase()}`],
+});
+
+describe("compileTrace", () => {
+ it("opens the starting page before anything else", () => {
+ const { steps } = compileTrace(trace([]));
+ expect(steps[0]).toMatchObject({ type: "navigate", url: "https://shop.test/order" });
+ });
+
+ it("produces steps that satisfy the published workflow schema", () => {
+ const { steps } = compileTrace(
+ trace([
+ {
+ type: "input",
+ url: "https://shop.test/order",
+ timestamp: 1,
+ target: { ...target("Full name", "textbox") },
+ value: "Ada Lovelace",
+ redacted: false,
+ },
+ { type: "click", url: "https://shop.test/order", timestamp: 2, target: target("Submit order") },
+ ]),
+ );
+
+ // The real gate: these steps go through the same validator the publish
+ // route uses, so a compiled workflow cannot be unpublishable.
+ expect(workflowSteps.safeParse(steps).success).toBe(true);
+
+ // And every emitted type must have an executor. `apiCall`/`sendEmail`
+ // parse and gate but do nothing, so a compiler that emitted one would
+ // produce a workflow that reports SUCCEEDED having skipped the step —
+ // exactly the failure `EDITABLE_STEP_TYPES` exists to prevent.
+ const editable = new Set(EDITABLE_STEP_TYPES);
+ for (const step of steps) expect(editable.has(step.type)).toBe(true);
+ });
+
+ it("prefers role and name over the CSS fallback", () => {
+ const { steps } = compileTrace(
+ trace([{ type: "click", url: "https://shop.test/x", timestamp: 1, target: target("Submit order") }]),
+ );
+ const click = steps.find((s) => s.type === "click");
+ expect(click).toMatchObject({ selector: { role: "button", name: "Submit order" } });
+ expect(click).not.toHaveProperty("selector.css");
+ });
+
+ it("falls back to CSS only when nothing semantic was captured", () => {
+ const { steps } = compileTrace(
+ trace([
+ {
+ type: "click",
+ url: "https://shop.test/x",
+ timestamp: 1,
+ target: { selectorCandidates: ["#mystery-button", ".btn"] },
+ },
+ ]),
+ );
+ expect(steps.find((s) => s.type === "click")).toMatchObject({
+ selector: { css: "#mystery-button" },
+ });
+ });
+
+ it("collapses a run of keystrokes on one field into a single fill", () => {
+ const field = { ...target("Full name", "textbox") };
+ const { steps } = compileTrace(
+ trace([
+ { type: "input", url: "https://shop.test/x", timestamp: 1, target: field, value: "A", redacted: false },
+ { type: "input", url: "https://shop.test/x", timestamp: 2, target: field, value: "Ad", redacted: false },
+ { type: "input", url: "https://shop.test/x", timestamp: 3, target: field, value: "Ada", redacted: false },
+ ]),
+ );
+ const fills = steps.filter((s) => s.type === "fill");
+ expect(fills).toHaveLength(1);
+ expect(fills[0]).toMatchObject({ value: "Ada" });
+ });
+
+ it("never carries a redacted value, and marks the step sensitive", () => {
+ const { steps, notes } = compileTrace(
+ trace([
+ {
+ type: "input",
+ url: "https://bank.test/login",
+ timestamp: 1,
+ target: { ...target("Password", "textbox"), inputType: "password" },
+ redacted: true,
+ },
+ ]),
+ );
+ const fill = steps.find((s) => s.type === "fill") as { value: string; sensitive: boolean };
+ expect(fill.sensitive).toBe(true);
+ expect(fill.value).not.toContain("hunter2");
+ expect(fill.value).toMatch(/redacted/i);
+ expect(notes.join(" ")).toMatch(/never captured/i);
+ });
+
+ it("puts an approval in front of an irreversible click", () => {
+ const { steps } = compileTrace(
+ trace([{ type: "click", url: "https://shop.test/x", timestamp: 1, target: target("Submit order") }]),
+ );
+ const submitIndex = steps.findIndex((s) => s.type === "click");
+ expect(submitIndex).toBeGreaterThan(0);
+ expect(steps[submitIndex - 1]).toMatchObject({ type: "approval" });
+ });
+
+ it("agrees with the engine: every gated step is preceded by an approval", () => {
+ const { steps } = compileTrace(
+ trace([
+ {
+ type: "input",
+ url: "https://shop.test/x",
+ timestamp: 1,
+ target: target("Full name", "textbox"),
+ value: "Ada",
+ redacted: false,
+ },
+ { type: "click", url: "https://shop.test/x", timestamp: 2, target: target("Delete account") },
+ { type: "click", url: "https://shop.test/x", timestamp: 3, target: target("Submit order") },
+ ]),
+ );
+
+ steps.forEach((step, i) => {
+ if (step.type === "approval") return;
+ if (classifyStep(step).sensitive) {
+ expect(steps[i - 1], `step ${i} (${step.type}) is gated but has no approval before it`)
+ .toMatchObject({ type: "approval" });
+ }
+ });
+ });
+
+ it("does not stack two approvals in a row", () => {
+ const { steps } = compileTrace(
+ trace([
+ { type: "click", url: "https://shop.test/x", timestamp: 1, target: target("Submit order") },
+ { type: "click", url: "https://shop.test/x", timestamp: 2, target: target("Confirm") },
+ ]),
+ );
+ for (let i = 1; i < steps.length; i++) {
+ expect(steps[i]!.type === "approval" && steps[i - 1]!.type === "approval").toBe(false);
+ }
+ });
+
+ it("says so when nothing looked irreversible, rather than staying quiet", () => {
+ const { notes } = compileTrace(
+ trace([{ type: "click", url: "https://shop.test/x", timestamp: 1, target: target("Next") }]),
+ );
+ expect(notes.join(" ")).toMatch(/no approval gate was added/i);
+ });
+
+ it("reports an unidentifiable click instead of guessing at coordinates", () => {
+ const { steps, notes } = compileTrace(
+ trace([{ type: "click", url: "https://shop.test/x", timestamp: 1, target: { selectorCandidates: [] } }]),
+ );
+ expect(steps.filter((s) => s.type === "click")).toHaveLength(0);
+ expect(notes.join(" ")).toMatch(/cannot be replayed/i);
+ });
+
+ it("ignores a navigation that did not change the page", () => {
+ const { steps } = compileTrace(
+ trace([
+ { type: "navigate", url: "https://shop.test/order", timestamp: 1 },
+ { type: "navigate", url: "https://shop.test/confirm", timestamp: 2 },
+ ]),
+ );
+ const urls = steps.filter((s) => s.type === "navigate").map((s) => (s as { url: string }).url);
+ expect(urls).toEqual(["https://shop.test/order", "https://shop.test/confirm"]);
+ });
+});
+
+describe("parseRecordingTrace", () => {
+ it("accepts a well-formed trace", () => {
+ const result = parseRecordingTrace({ version: 1, sessionId: "rec_1", events: [] });
+ expect(result.ok).toBe(true);
+ });
+
+ it("rejects a HAR or anything else that is not this format", () => {
+ const result = parseRecordingTrace({ log: { version: "1.2", entries: [] } });
+ expect(result.ok).toBe(false);
+ });
+
+ it("names what was wrong rather than failing opaquely", () => {
+ const result = parseRecordingTrace({ version: 1, sessionId: "rec_1", events: [{ type: "click" }] });
+ expect(result.ok).toBe(false);
+ if (!result.ok) expect(result.error).toMatch(/events/);
+ });
+});
diff --git a/cloud/packages/core/src/recording/compile.ts b/cloud/packages/core/src/recording/compile.ts
new file mode 100644
index 00000000..89a72940
--- /dev/null
+++ b/cloud/packages/core/src/recording/compile.ts
@@ -0,0 +1,243 @@
+import { classifyStep } from "../classifier/sensitive.js";
+import type { Selector, WorkflowStep, WorkflowSteps } from "../schema/step.js";
+import type { RecordingTrace, TraceEvent, TraceTarget } from "./trace.js";
+
+/**
+ * Turns a captured trace into typed workflow steps — **deterministically**.
+ *
+ * No model, no network, no inference. The recorder already read the accessible
+ * role and name off the live element, which are the fields the worker's
+ * resolution chain prefers, so compilation is a mechanical translation rather
+ * than a guess. That matters for more than purity: it means recording works
+ * with no compiler configured, in a deployment that has deliberately removed
+ * every AI dependency from the runtime (see `docs/DEPLOY.md`).
+ *
+ * A model still has a job here, just not this one. Naming steps readably,
+ * proposing where an approval belongs beyond the obvious cases, and flagging
+ * what it could not resolve are all judgement calls. Deciding *which element
+ * was clicked* is not, and should never have been.
+ *
+ * Three things it does beyond translation, all conservative:
+ *
+ * 1. **Collapses keystroke noise.** Consecutive `input` events on one field
+ * become one `fill` with the final value. A recorder that emits per
+ * keystroke would otherwise produce a step per character.
+ * 2. **Gates sensitive actions.** Every produced step is run through the same
+ * `classifyStep` the engine uses, and an `approval` step is inserted before
+ * any that gates. The workflow therefore arrives already stopping before it
+ * submits, rather than relying on whoever reviews it to notice.
+ * 3. **Never carries a secret.** A redacted input becomes a `fill` marked
+ * `sensitive` with a placeholder value. The real value was never in the
+ * trace; this makes its absence explicit in the step the human reviews.
+ */
+
+export interface CompileTraceResult {
+ steps: WorkflowSteps;
+ /** Human-facing notes: what was dropped, and what needs a decision. */
+ notes: string[];
+}
+
+/** Ghost never stores a captured secret. The reviewer replaces this. */
+const SECRET_PLACEHOLDER = "";
+
+function selectorFrom(target: TraceTarget): Selector | null {
+ const selector: Selector = {};
+ if (target.role) selector.role = target.role;
+ if (target.name) selector.name = target.name;
+ if (target.testId) selector.testId = target.testId;
+ // Text is only worth carrying when nothing stronger identified the element;
+ // as an extra field alongside role+name it adds no resolution power and
+ // makes the step noisier to read.
+ if (!selector.role && !selector.testId && target.text) selector.text = target.text;
+ if (!selector.role && !selector.testId && !selector.text) {
+ const css = target.selectorCandidates[0];
+ if (css) selector.css = css;
+ }
+ return Object.keys(selector).length > 0 ? selector : null;
+}
+
+function describe(target: TraceTarget): string | undefined {
+ if (target.name) return target.name;
+ if (target.text) return target.text;
+ return undefined;
+}
+
+/** Stable within one compile; the schema only requires uniqueness. */
+function makeIdFactory() {
+ let n = 0;
+ return (prefix: string) => `${prefix}${++n}`;
+}
+
+/**
+ * True when two consecutive input events are typing into the same field, so
+ * only the last value matters. Compared on the resolved selector rather than
+ * on object identity, since each event carries its own target object.
+ */
+function sameField(a: TraceEvent, b: TraceEvent): boolean {
+ if (a.type !== "input" || b.type !== "input") return false;
+ const sa = selectorFrom(a.target);
+ const sb = selectorFrom(b.target);
+ if (!sa || !sb) return false;
+ return JSON.stringify(sa) === JSON.stringify(sb);
+}
+
+export function compileTrace(trace: RecordingTrace): CompileTraceResult {
+ const id = makeIdFactory();
+ const steps: WorkflowStep[] = [];
+ const notes: string[] = [];
+
+ // Open the page the recording started on. Without this the workflow depends
+ // on whatever the browser happened to be showing.
+ const firstUrl = trace.startUrl ?? trace.events.find((e) => e.url)?.url;
+ if (firstUrl) {
+ steps.push({ id: id("nav"), type: "navigate", url: firstUrl, label: "Open the starting page" });
+ } else {
+ notes.push("The trace had no URL, so the workflow does not open a page first. Add a navigate step.");
+ }
+
+ let lastUrl = firstUrl;
+
+ for (let i = 0; i < trace.events.length; i++) {
+ const event = trace.events[i]!;
+
+ // Collapse a run of keystrokes on one field into its final value.
+ if (event.type === "input") {
+ let last = event;
+ while (i + 1 < trace.events.length && sameField(last, trace.events[i + 1]!)) {
+ last = trace.events[++i] as typeof event;
+ }
+ const selector = selectorFrom(last.target);
+ if (!selector) {
+ notes.push(
+ `Skipped typing into an element with nothing to identify it by (at ${last.url}). ` +
+ "The recorder could not read a role, name, test id or text for it.",
+ );
+ continue;
+ }
+ steps.push({
+ id: id("fill"),
+ type: "fill",
+ selector,
+ value: last.redacted ? SECRET_PLACEHOLDER : (last.value ?? ""),
+ sensitive: last.redacted,
+ ...(describe(last.target) ? { label: `Fill ${describe(last.target)}` } : {}),
+ });
+ if (last.redacted) {
+ notes.push(
+ `The value typed into "${describe(last.target) ?? "a field"}" was a password or ` +
+ "other secret, so it was never captured. Set it before running this workflow.",
+ );
+ }
+ continue;
+ }
+
+ if (event.type === "select") {
+ const selector = selectorFrom(event.target);
+ if (!selector) {
+ notes.push(`Skipped a dropdown selection with nothing to identify it by (at ${event.url}).`);
+ continue;
+ }
+ steps.push({
+ id: id("select"),
+ type: "select",
+ selector,
+ value: event.value,
+ ...(describe(event.target) ? { label: `Choose ${describe(event.target)}` } : {}),
+ });
+ continue;
+ }
+
+ if (event.type === "click") {
+ const selector = selectorFrom(event.target);
+ if (!selector) {
+ notes.push(
+ `Skipped a click on an element with nothing to identify it by (at ${event.url}). ` +
+ "Pixel coordinates are deliberately not recorded, so this click cannot be replayed.",
+ );
+ continue;
+ }
+ steps.push({
+ id: id("click"),
+ type: "click",
+ selector,
+ ...(describe(event.target) ? { description: describe(event.target) } : {}),
+ ...(describe(event.target) ? { label: `Click ${describe(event.target)}` } : {}),
+ });
+ continue;
+ }
+
+ if (event.type === "navigate") {
+ // Only when the page actually changed; a recorder may emit one per
+ // history event, including replaces that land on the same URL.
+ if (event.url && event.url !== lastUrl) {
+ steps.push({
+ id: id("nav"),
+ type: "navigate",
+ url: event.url,
+ label: "Follow the page change",
+ });
+ lastUrl = event.url;
+ }
+ continue;
+ }
+
+ // `submit` carries no reliable target of its own — the click that caused
+ // it is already recorded, and replaying a form submission separately would
+ // double the action.
+ if (event.type === "submit") {
+ notes.push(
+ `A form was submitted at ${event.url}. The click that submitted it is already a step; ` +
+ "check that an approval sits in front of it.",
+ );
+ continue;
+ }
+ }
+
+ const gated = insertApprovals(steps, id);
+
+ if (gated.inserted === 0) {
+ notes.push(
+ "No step in this recording looked irreversible, so no approval gate was added. " +
+ "If it sends, pays, deletes or submits anything, add one before publishing.",
+ );
+ }
+
+ return { steps: gated.steps as WorkflowSteps, notes };
+}
+
+/**
+ * Puts an `approval` immediately before every step the engine would gate.
+ *
+ * Uses `classifyStep` — the same deterministic classifier the worker consults
+ * at run time — so the authored workflow agrees with what execution will
+ * actually do. Inserting a gate here does not *create* the protection (the
+ * engine gates regardless); it makes the pause visible in the plan the human
+ * reviews, with a reason attached.
+ */
+function insertApprovals(
+ steps: WorkflowStep[],
+ id: (prefix: string) => string,
+): { steps: WorkflowStep[]; inserted: number } {
+ const out: WorkflowStep[] = [];
+ let inserted = 0;
+
+ for (const step of steps) {
+ const verdict = classifyStep(step);
+ if (verdict.sensitive) {
+ const previous = out[out.length - 1];
+ // Do not stack two gates: an approval immediately before is already the
+ // pause this would add.
+ if (!previous || previous.type !== "approval") {
+ out.push({
+ id: id("approve"),
+ type: "approval",
+ reason: verdict.reason ?? "This step changes something outside Ghost.",
+ });
+ inserted++;
+ }
+ }
+ out.push(step);
+ }
+
+ return { steps: out, inserted };
+}
diff --git a/cloud/packages/core/src/recording/roundtrip.test.ts b/cloud/packages/core/src/recording/roundtrip.test.ts
new file mode 100644
index 00000000..838ba59a
--- /dev/null
+++ b/cloud/packages/core/src/recording/roundtrip.test.ts
@@ -0,0 +1,156 @@
+import { describe, expect, it } from "vitest";
+import { compileTrace } from "./compile.js";
+import { parseRecordingTrace } from "./trace.js";
+import { classifyStep } from "../classifier/sensitive.js";
+
+/**
+ * The capture → compile contract, exercised against a trace shaped exactly the
+ * way `apps/extension/src/content.js` emits one.
+ *
+ * This is the seam most likely to rot: the extension is plain JavaScript with
+ * no type checking against `@ghost/core`, so nothing but this test notices if
+ * the recorder's output drifts from what the compiler expects. If it fails,
+ * the extension and Ghost have disagreed about the format — fix one of them,
+ * do not relax the test.
+ *
+ * The scenario is the wedge Ghost is aimed at: log into a portal, find a
+ * record, change it, submit.
+ */
+
+/** Verbatim in the shape `describeTarget()` produces. */
+const target = (over: Record) => ({
+ selectorCandidates: [],
+ ...over,
+});
+
+const extensionTrace = {
+ version: 1,
+ sessionId: "rec_1735000000000",
+ startUrl: "https://portal.test/login",
+ capturedAt: 1735000000000,
+ recorder: "ghost-chrome-extension/0.1.0",
+ events: [
+ { type: "navigate", url: "https://portal.test/login", timestamp: 1 },
+ {
+ type: "input",
+ url: "https://portal.test/login",
+ timestamp: 2,
+ target: target({
+ role: "textbox",
+ name: "Username",
+ selectorCandidates: ["#username", "input[name=\"user\"]"],
+ tagName: "input",
+ inputType: "text",
+ }),
+ value: "ops.team",
+ redacted: false,
+ },
+ {
+ // The recorder never read this value — the field was type=password.
+ type: "input",
+ url: "https://portal.test/login",
+ timestamp: 3,
+ target: target({
+ role: "textbox",
+ name: "Password",
+ selectorCandidates: ["#password"],
+ tagName: "input",
+ inputType: "password",
+ }),
+ redacted: true,
+ },
+ {
+ type: "click",
+ url: "https://portal.test/login",
+ timestamp: 4,
+ target: target({ role: "button", name: "Sign in", selectorCandidates: ["#signin"] }),
+ },
+ { type: "navigate", url: "https://portal.test/customers/4821", timestamp: 5 },
+ {
+ type: "select",
+ url: "https://portal.test/customers/4821",
+ timestamp: 6,
+ target: target({ role: "combobox", name: "Status", selectorCandidates: ["#status"] }),
+ value: "active",
+ },
+ {
+ type: "click",
+ url: "https://portal.test/customers/4821",
+ timestamp: 7,
+ target: target({ role: "button", name: "Submit changes", selectorCandidates: ["#save"] }),
+ },
+ { type: "submit", url: "https://portal.test/customers/4821", timestamp: 8 },
+ ],
+};
+
+describe("extension trace → workflow steps", () => {
+ const parsed = parseRecordingTrace(extensionTrace);
+
+ it("is accepted by the trace schema exactly as the extension emits it", () => {
+ expect(parsed.ok).toBe(true);
+ });
+
+ if (!parsed.ok) throw new Error(`fixture must parse: ${parsed.error}`);
+ const { steps, notes } = compileTrace(parsed.trace);
+
+ it("opens the page the recording started on", () => {
+ expect(steps[0]).toMatchObject({ type: "navigate", url: "https://portal.test/login" });
+ });
+
+ it("carries the password field through as sensitive, with no value", () => {
+ const password = steps.find(
+ (s) => s.type === "fill" && s.selector.name === "Password",
+ ) as { value: string; sensitive: boolean } | undefined;
+
+ expect(password).toBeDefined();
+ expect(password!.sensitive).toBe(true);
+ expect(password!.value).toMatch(/redacted/i);
+ // The whole trace must be free of it — no field anywhere holds a password.
+ expect(JSON.stringify(extensionTrace)).not.toMatch(/hunter2|correct-horse/i);
+ });
+
+ it("keeps the non-secret value it did capture", () => {
+ const username = steps.find((s) => s.type === "fill" && s.selector.name === "Username") as
+ | { value: string }
+ | undefined;
+ expect(username?.value).toBe("ops.team");
+ });
+
+ it("resolves elements semantically rather than by CSS", () => {
+ const signIn = steps.find(
+ (s) => s.type === "click" && s.selector.name === "Sign in",
+ ) as { selector: Record } | undefined;
+ expect(signIn?.selector).toMatchObject({ role: "button", name: "Sign in" });
+ expect(signIn?.selector.css).toBeUndefined();
+ });
+
+ it("stops for approval before submitting the change", () => {
+ const submitIndex = steps.findIndex(
+ (s) => s.type === "click" && s.selector.name === "Submit changes",
+ );
+ expect(submitIndex).toBeGreaterThan(0);
+ expect(steps[submitIndex - 1]).toMatchObject({ type: "approval" });
+ });
+
+ it("gates every step the engine would gate, and no step is left ungated", () => {
+ steps.forEach((step, i) => {
+ if (step.type === "approval") return;
+ if (classifyStep(step).sensitive) {
+ expect(steps[i - 1], `step ${i} is sensitive but ungated`).toMatchObject({
+ type: "approval",
+ });
+ }
+ });
+ });
+
+ it("tells the reviewer the password needs setting before a run", () => {
+ expect(notes.join(" ")).toMatch(/set it before running/i);
+ });
+
+ it("emits no coordinates anywhere", () => {
+ const serialized = JSON.stringify(steps);
+ expect(serialized).not.toMatch(/"x":\s*\d/);
+ expect(serialized).not.toMatch(/"y":\s*\d/);
+ expect(serialized).not.toMatch(/clientX|pageX|screenX/);
+ });
+});
diff --git a/cloud/packages/core/src/recording/trace.ts b/cloud/packages/core/src/recording/trace.ts
new file mode 100644
index 00000000..284a6f72
--- /dev/null
+++ b/cloud/packages/core/src/recording/trace.ts
@@ -0,0 +1,119 @@
+import { z } from "zod";
+
+/**
+ * The capture format: what a Ghost recorder uploads.
+ *
+ * This is a **contract**, not an internal shape. The Chrome extension in
+ * `apps/extension` produces it; `compileTrace` consumes it. Anything else that
+ * can produce this JSON — a future desktop recorder, a customer's own tooling —
+ * works without changing Ghost.
+ *
+ * The design decision that matters: a recorder emits **accessible role and
+ * name at capture time**, computed from the live DOM while the element is
+ * still on screen. Those are exactly the fields `resolveLocator` prefers, so
+ * the trace can be compiled into steps deterministically, with no model in the
+ * correctness path. Inferring `role`/`name` later from an opaque HAR or event
+ * log is guesswork; reading them off the element is not.
+ *
+ * `selectorCandidates` is ordered best-first and exists because the worker
+ * replays in a *different* browser from the one that recorded. A single
+ * brittle selector is the failure that design invites.
+ */
+
+/** What a recorder observed about the element an event happened on. */
+export const traceTargetSchema = z.object({
+ /** ARIA role, explicit or implicit. */
+ role: z.string().min(1).optional(),
+ /** Accessible name, computed the way a screen reader would. */
+ name: z.string().min(1).optional(),
+ /** `data-testid` and friends, when the page has them. */
+ testId: z.string().min(1).optional(),
+ /** Visible text, as a weaker fallback than role+name. */
+ text: z.string().min(1).optional(),
+ /**
+ * Ordered best-first CSS fallbacks. Only used when nothing semantic
+ * resolved — the resolution chain prefers role+name, then testId, then text.
+ */
+ selectorCandidates: z.array(z.string().min(1)).max(10).default([]),
+ /** Tag name, for deciding between `fill` and `select`. */
+ tagName: z.string().min(1).optional(),
+ /** `type` attribute of an input, for sensitivity decisions. */
+ inputType: z.string().min(1).optional(),
+});
+
+export type TraceTarget = z.infer;
+
+const base = z.object({
+ /** Page URL at the moment of the event. */
+ url: z.string().min(1),
+ /** Epoch milliseconds. Ordering is by array position; this is for humans. */
+ timestamp: z.number().int().nonnegative(),
+});
+
+/**
+ * `redacted` is set by the *recorder*, which is the only place it can be done
+ * honestly: the value never leaves the page. A trace that carried the secret
+ * plus a "please ignore this" flag would already have leaked it — see
+ * `docs/DEPLOY.md` on why a trace is customer data.
+ */
+export const traceEventSchema = z.discriminatedUnion("type", [
+ base.extend({
+ type: z.literal("navigate"),
+ }),
+ base.extend({
+ type: z.literal("click"),
+ target: traceTargetSchema,
+ }),
+ base.extend({
+ type: z.literal("input"),
+ target: traceTargetSchema,
+ /** Absent when `redacted` — the recorder never captured it. */
+ value: z.string().optional(),
+ redacted: z.boolean().default(false),
+ }),
+ base.extend({
+ type: z.literal("select"),
+ target: traceTargetSchema,
+ value: z.string(),
+ }),
+ base.extend({
+ type: z.literal("submit"),
+ target: traceTargetSchema.optional(),
+ }),
+]);
+
+export type TraceEvent = z.infer;
+
+export const recordingTraceSchema = z.object({
+ version: z.literal(1),
+ sessionId: z.string().min(1),
+ /** Where recording started, so the compiled workflow opens the right page. */
+ startUrl: z.string().min(1).optional(),
+ capturedAt: z.number().int().nonnegative().optional(),
+ /** Names the producer, so a bad recorder can be identified from a trace. */
+ recorder: z.string().min(1).optional(),
+ events: z.array(traceEventSchema),
+});
+
+export type RecordingTrace = z.infer;
+
+/**
+ * Whether some uploaded bytes are a structured Ghost trace.
+ *
+ * Used to decide between the deterministic compiler and the optional
+ * model-backed one. A HAR or a Playwright zip is not this shape and falls
+ * through to whatever compiler is configured, if any.
+ */
+export function parseRecordingTrace(
+ input: unknown,
+): { ok: true; trace: RecordingTrace } | { ok: false; error: string } {
+ const parsed = recordingTraceSchema.safeParse(input);
+ if (!parsed.success) {
+ const first = parsed.error.issues[0];
+ return {
+ ok: false,
+ error: first ? `${first.path.join(".") || "trace"}: ${first.message}` : "invalid trace",
+ };
+ }
+ return { ok: true, trace: parsed.data };
+}
From 748395084883e749a1ff6b9f14bbd6d38e9599df Mon Sep 17 00:00:00 2001
From: mohabbis <101276427+mohabbis@users.noreply.github.com>
Date: Wed, 5 Aug 2026 12:52:07 -0500
Subject: [PATCH 2/9] fix(worker): make production container build and boot
reliably
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The worker Dockerfile had never actually been built or run since it was
written. Auditing what was genuinely left on the cloud roadmap surfaced two
stacked bugs, neither caught by CI (`pnpm build` only bundles the worker with
tsup at the workspace level — it never exercises the Dockerfile's own COPY
list or executes the resulting dist/index.js):
- The deps stage copied packages/core/package.json but not its prisma/
directory. @ghost/core's postinstall runs a bare `prisma generate`, which
resolves the schema at the default ./prisma/schema.prisma path, so
`pnpm install` failed inside the image before the build stage (which does
copy the rest of packages/core) ever ran. The same gap silently dropped
tsconfig.base.json, breaking apps/worker/tsconfig.json's `extends`.
- Once building, the container crashed immediately on boot: bundling
@ghost/core (via tsup's `noExternal: [/^@ghost\//]`) pulls in
@prisma/client's generated CJS runtime, which dynamically requires native
query-engine files — esbuild's CJS-to-ESM interop can't represent that and
throws "Dynamic require of 'fs' is not supported" at the first call. Fixed
by keeping @prisma/client external in tsup.config.ts and adding it as a
direct dependency of @ghost/worker, since pnpm's strict linking won't
resolve a transitive dep at the worker's own require path otherwise.
Verified by rebuilding the image and running it against real Postgres/Redis
until it logged its startup line rather than crashing. Added a CI step that
builds and boot-smoke-tests the image on every PR so this class of bug can't
ship invisibly again.
Co-Authored-By: Claude Sonnet 5
---
.github/workflows/cloud.yml | 45 ++++++++++++++++++++++++++++++++
cloud/apps/worker/Dockerfile | 7 ++++-
cloud/apps/worker/package.json | 1 +
cloud/apps/worker/tsup.config.ts | 7 +++++
cloud/pnpm-lock.yaml | 3 +++
5 files changed, 62 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/cloud.yml b/.github/workflows/cloud.yml
index 390fb262..ab3ab0c8 100644
--- a/.github/workflows/cloud.yml
+++ b/.github/workflows/cloud.yml
@@ -160,3 +160,48 @@ jobs:
- name: Build
run: pnpm build
+
+ - name: Build worker container image
+ # `pnpm build` above only bundles the worker with tsup — it never
+ # exercises the Dockerfile's own multi-stage COPY list, and it never
+ # runs the resulting dist/index.js in an isolated, workspace-filtered
+ # node_modules the way the image's `pnpm install --filter
+ # @ghost/worker...` does. Both gaps let a genuinely broken image ship
+ # silently before: a missing `packages/core/prisma` COPY failed
+ # `pnpm install` inside the image, and bundling @prisma/client's CJS
+ # runtime into the ESM output crashed the container on boot with
+ # "Dynamic require of 'fs' is not supported". Build it here so a
+ # regression in either fails CI instead of surfacing at deploy time.
+ run: docker build -f apps/worker/Dockerfile -t ghost-worker-ci .
+
+ - name: Smoke-test worker container boots
+ # A build succeeding is not the same as the entrypoint running. Boot
+ # it against the same Postgres/Redis service containers this job
+ # already has and require the worker's own startup log line — not just
+ # "container still alive", which a crash-loop with a restart policy
+ # would also satisfy.
+ run: |
+ set -euo pipefail
+ container_id=$(docker run -d --network host \
+ -e DATABASE_URL="$DATABASE_URL" \
+ -e REDIS_URL="$REDIS_URL" \
+ -e GHOST_SESSION_KEY="$GHOST_SESSION_KEY" \
+ ghost-worker-ci)
+ trap 'docker logs "$container_id" || true; docker rm -f "$container_id" >/dev/null 2>&1 || true' EXIT
+
+ ok=""
+ for _ in $(seq 1 15); do
+ if docker logs "$container_id" 2>&1 | grep -q "Ghost worker started"; then
+ ok=1
+ break
+ fi
+ if [ "$(docker inspect -f '{{.State.Running}}' "$container_id")" != "true" ]; then
+ break
+ fi
+ sleep 1
+ done
+
+ if [ -z "$ok" ]; then
+ echo "worker container did not report startup within 15s" >&2
+ exit 1
+ fi
diff --git a/cloud/apps/worker/Dockerfile b/cloud/apps/worker/Dockerfile
index 65e12edc..1062d4bc 100644
--- a/cloud/apps/worker/Dockerfile
+++ b/cloud/apps/worker/Dockerfile
@@ -27,8 +27,13 @@ WORKDIR /app
# ---- dependencies -----------------------------------------------------------
FROM base AS deps
# Manifests first so a code-only change does not re-resolve the dependency graph.
-COPY package.json pnpm-lock.yaml pnpm-workspace.yaml turbo.json ./
+COPY package.json pnpm-lock.yaml pnpm-workspace.yaml turbo.json tsconfig.base.json ./
COPY packages/core/package.json packages/core/
+# packages/core's postinstall runs a bare `prisma generate`, which resolves
+# the schema at the default `./prisma/schema.prisma` path relative to the
+# package. Without it here, `pnpm install` fails inside this stage before the
+# build stage (which copies the rest of packages/core) ever runs.
+COPY packages/core/prisma packages/core/prisma
COPY apps/worker/package.json apps/worker/
RUN pnpm install --frozen-lockfile --filter @ghost/worker...
diff --git a/cloud/apps/worker/package.json b/cloud/apps/worker/package.json
index 5595d718..1acb88f0 100644
--- a/cloud/apps/worker/package.json
+++ b/cloud/apps/worker/package.json
@@ -13,6 +13,7 @@
},
"dependencies": {
"@ghost/core": "workspace:*",
+ "@prisma/client": "^6.1.0",
"bullmq": "^5.34.4",
"ioredis": "^5.4.2",
"playwright": "^1.55.0"
diff --git a/cloud/apps/worker/tsup.config.ts b/cloud/apps/worker/tsup.config.ts
index 3fcb94a6..ffc83301 100644
--- a/cloud/apps/worker/tsup.config.ts
+++ b/cloud/apps/worker/tsup.config.ts
@@ -8,4 +8,11 @@ export default defineConfig({
// Bundle the workspace package (it ships TS source via subpath exports) so the
// built worker runs on plain Node.
noExternal: [/^@ghost\//],
+ // @prisma/client's generated runtime is CJS that dynamically `require()`s
+ // native query-engine files by path. Inlining it into the ESM bundle makes
+ // esbuild rewrite that into a `require` shim that throws "Dynamic require of
+ // 'fs' is not supported" the instant the worker boots. Keep it (and the
+ // generated `.prisma/client`) external so Node loads it normally from
+ // node_modules instead.
+ external: ["@prisma/client", ".prisma/client"],
});
diff --git a/cloud/pnpm-lock.yaml b/cloud/pnpm-lock.yaml
index 88f55c9e..28a4b2af 100644
--- a/cloud/pnpm-lock.yaml
+++ b/cloud/pnpm-lock.yaml
@@ -115,6 +115,9 @@ importers:
'@ghost/core':
specifier: workspace:*
version: link:../../packages/core
+ '@prisma/client':
+ specifier: ^6.1.0
+ version: 6.19.3(prisma@6.19.3(typescript@5.9.3))(typescript@5.9.3)
bullmq:
specifier: ^5.34.4
version: 5.81.2
From edf10c7f1c31a824311896b4a34034b9390f83b4 Mon Sep 17 00:00:00 2001
From: mohabbis <101276427+mohabbis@users.noreply.github.com>
Date: Wed, 5 Aug 2026 12:52:19 -0500
Subject: [PATCH 3/9] docs(cloud): reconcile handoff and status documentation
with current implementation
CURSOR_HANDOFF.md described the WorkflowCompiler abstraction, four-eyes
approval, and S3 artifact serving as open work, framing a stale roadmap that
led to prioritizing already-shipped items. All three were fully implemented
and tested by earlier PRs (#403, #406) whose changes never made it back into
this file's "remaining work" section:
- WorkflowCompiler: apps/web/src/lib/compiler/{types,index,harness-router-compiler}.ts
already provides the interface, a swappable HarnessRouter adapter, and
normalized error types.
- Four-eyes approval: Membership/Role/Invitation plus
Organization.requireSeparateApprover already enforce requester != approver
server-side, covered by separation-of-duties.test.ts.
- S3 artifact serving: packages/core/src/storage/artifacts.ts already has a
working S3ArtifactStore with presigned URLs; only retention/cleanup is
genuinely still open, called out as such.
Also documents the worker container bugs found and fixed in the prior commit,
and corrects the stale "239 tests" figure (actual full green run is 398) and
the "typed step editor: remaining" status line (workflow-editor.tsx already
covers it) in both this file and README.md.
Co-Authored-By: Claude Sonnet 5
---
cloud/README.md | 4 +-
cloud/docs/CURSOR_HANDOFF.md | 102 ++++++++++++++++++++++++++++++-----
2 files changed, 90 insertions(+), 16 deletions(-)
diff --git a/cloud/README.md b/cloud/README.md
index cf725f4d..adcd0db1 100644
--- a/cloud/README.md
+++ b/cloud/README.md
@@ -144,7 +144,7 @@ and `turbo run lint` skips packages that define no `lint` script — silently, a
with a green summary. Treat `typecheck` as the real static gate until the other
three packages have configs.
-A full green run is **239 tests**. Roughly 90 of them are gated on
+A full green run is **398 tests**. Roughly 90 of them are gated on
`Boolean(process.env.DATABASE_URL)` and **skip silently** without it — so a
green run with no database covers none of the execution engine. If the worker
suite reports 45 tests rather than 90, the database is not being reached.
@@ -157,7 +157,7 @@ suite reports 45 tests rather than 90, the database is not being reached.
| 1.1 | Playwright engine, approval state machine, artifacts, hash-chained audit | **done** |
| 1.2 | Run trigger, live timeline, approve/reject resume | **done** |
| 1.3 | Durable execution (run journal), timeouts/retry, incidents + cancel, audit-verify, variable store | **done** |
-| 1.x | Typed step editor | remaining |
+| 1.x | Typed step editor | **done** (`workflow-editor.tsx`: type/selector/value, reorder, delete, validation, undo authoring) |
| 2 | Browser recording → editable steps | convert built (upload → compile → review), off by default; capture is next |
Phase 1.3 made run position a fold over an append-only, hash-chained journal
diff --git a/cloud/docs/CURSOR_HANDOFF.md b/cloud/docs/CURSOR_HANDOFF.md
index 6bfb22f8..bd859e14 100644
--- a/cloud/docs/CURSOR_HANDOFF.md
+++ b/cloud/docs/CURSOR_HANDOFF.md
@@ -305,7 +305,7 @@ Redis running** — CI catches it before merge either way, but a local loop is
faster than waiting on a run:
```bash
-cd cloud && pnpm typecheck && pnpm test && pnpm build # expect 239 tests
+cd cloud && pnpm typecheck && pnpm test && pnpm build # expect 398 tests
```
Without a database roughly 90 of those skip themselves and you learn nothing
@@ -363,15 +363,28 @@ page reload, confirmed the stream closes on a terminal status rather than
reconnecting forever (5 requests total, network panel stayed flat for 8s
after).
-1. **S3 serve path** — disk store works in dev; wire presigned URLs when S3 is on.
- Note the artifact route is now a positive allow-list (`step-`/`restore-` PNGs
- only); keep it that way, because the same prefix holds encrypted session
- blobs that must never be served.
-2. **Four-eyes approval** — `isOrgAdmin` above is a role check, not the RBAC
- `requireSeparateApprover` would need (which is about who may *approve*, not
- who may *administer*); still needs its own design, and orgs are still
- auto-created single-member on first sign-in with no invite flow, so it has
- no real subject to test against yet.
+Done: **S3 serve path**. `packages/core/src/storage/artifacts.ts` has a real
+`S3ArtifactStore` (put/get/delete/deletePrefix/presigned `signedUrl`) alongside
+the disk fallback; `artifactStore()` switches on `S3_BUCKET` +
+`S3_ACCESS_KEY_ID` + `S3_SECRET_ACCESS_KEY` being set, and `docs/DEPLOY.md`
+documents the silent-fallback failure mode. The artifact route is a positive
+allow-list (`step-`/`restore-` PNGs only) — keep it that way; the same prefix
+holds encrypted session blobs that must never be served. What's still actually
+missing is a retention/cleanup policy: nothing expires or purges old run
+artifacts, on disk or in the bucket, so storage grows unbounded over the life
+of an org. Not started.
+
+Done: **four-eyes approval**. `isOrgAdmin` above undersold this — a full
+membership model shipped alongside it: `Membership`/`Role`
+(`OWNER`/`ADMIN`/`MEMBER`) and `Invitation` in the schema, invite/accept routes
+under `app/api/invitations` and `app/api/settings/members`
+(email-match-enforced acceptance, not just token possession — see
+`checkInvitationRedeemable`), and `Organization.requireSeparateApprover`
+enforcing `requester_user_id !== approver_user_id` server-side when a workflow
+opts in. Covered end-to-end by
+`apps/web/src/app/api/runs/separation-of-duties.test.ts` (5 tests) and
+`apps/web/src/app/api/settings/members/members.test.ts` (9 tests) against a
+real Postgres. Nothing on this item is a gap anymore.
## Phase 2
@@ -395,10 +408,18 @@ pipeline, it only pre-fills the editor. `HR_API_KEY` (server-only,
infrastructure used to build Ghost; nothing a customer executes goes through
it. Compile is an optional authoring convenience that is unavailable when the
key is unset, and production is expected to leave it unset — see
-`docs/DEPLOY.md`. The next change here extracts a provider-neutral
-`WorkflowCompiler` interface so the HarnessRouter client is one replaceable
-adapter rather than a dependency reaching into the recording routes. Removing
-that adapter must disable compile and nothing else.
+`docs/DEPLOY.md`.
+
+Done: **the provider-neutral `WorkflowCompiler` boundary** (PR #406). The
+interface, its `HarnessRouterWorkflowCompiler` implementation, and the
+`workflowCompiler()`/`requireWorkflowCompiler()`/`compilerConfigured()`
+accessors live in `apps/web/src/lib/compiler/{types,index,harness-router-compiler}.ts`.
+`recording-compiler.ts` depends only on `WorkflowCompiler`; HarnessRouter-specific
+types stay inside `harness-router-compiler.ts` and never leak into it. Errors
+normalize to Ghost-owned `CompilerNotConfiguredError`/`CompilerRequestError`,
+and `compiler-optional.test.ts` swaps in a fake for tests. Adding a second
+adapter needs no change to the recording routes — only a new file and a branch
+in `workflowCompiler()`.
**Capture is decided: a Chrome extension for v1.** A Ghost-hosted remote
browser is the more elegant answer — it records in the same environment the
@@ -410,6 +431,59 @@ to exist, not a particular producer, so the extension slots in ahead of the
upload step and `POST /api/recordings` stays as it is. The manual upload form
is scaffolding, not the product.
+**Capture is also built, not just decided.** `cloud/apps/extension` is a
+working Chrome extension (records clicks/typing/selects/submits/navigation via
+accessible role+name, redacts secret-shaped fields at capture, uploads to
+`POST /api/agent/recordings` with a revocable bearer token) — see its own
+`README.md` for the trust boundary. It lands on `feat/browser-recording-extension`,
+not yet merged, so `README.md`'s Phase 2 status line ("capture is next") is
+correct as of `master` but stale the moment this branch merges. Update it in
+the same PR.
+
+## Worker container: built, but had never actually run
+
+Audited 2026-08-05, prompted by a stale roadmap in this very document (the S3
+and four-eyes items above, both already shipped). While verifying what was
+*actually* left, `docker build -f apps/worker/Dockerfile .` was run for the
+first time since this file's own "no deployment of Ghost exists yet" caveat
+in `docs/DEPLOY.md` was written — and it failed, then failed differently, then
+crashed on boot. Two independent bugs, both now fixed:
+
+1. The `deps` stage only `COPY`'d `packages/core/package.json`, not
+ `packages/core/prisma`. `@ghost/core`'s `postinstall` runs a bare `prisma
+ generate`, which resolves the schema at the default `./prisma/schema.prisma`
+ path — absent at that point in the build — so `pnpm install` itself failed
+ before the `build` stage (which does copy the rest of `packages/core`) ever
+ ran. Same root cause silently dropped `tsconfig.base.json`, so
+ `apps/worker/tsconfig.json`'s `extends` also failed (non-fatally — tsup
+ warned and fell back to its own defaults). Fixed by copying both ahead of
+ `pnpm install`.
+2. With the image building, the container crashed immediately on boot:
+ `Error: Dynamic require of "fs" is not supported`. `tsup.config.ts`'s
+ `noExternal: [/^@ghost\//]` bundles `@ghost/core` into the worker's ESM
+ output, which transitively pulled in `@prisma/client`'s generated CJS
+ runtime (it dynamically `require()`s native query-engine files) — esbuild's
+ CJS→ESM interop can't represent that and throws at the first call. Fixed by
+ marking `@prisma/client`/`.prisma/client` `external` in `tsup.config.ts`,
+ and adding `@prisma/client` as a direct dependency of `@ghost/worker`
+ (pnpm's strict linking won't resolve a transitive dep at the worker's own
+ require path otherwise).
+
+Verified by rebuilding the image and running it against a real Postgres +
+Redis with `docker run --network host`, confirming the `[worker] Ghost worker
+started...` log line rather than a crash. `.github/workflows/cloud.yml` now
+builds and boot-smoke-tests the image on every PR (`Build worker container
+image` / `Smoke-test worker container boots`) — before this, CI's `pnpm build`
+only ran `tsup` at the workspace level, which neither exercised the
+Dockerfile's own `COPY` list nor ever executed the bundled `dist/index.js`, so
+both bugs shipped invisibly across several PRs.
+
+This does not mean a deployment now exists — `docs/DEPLOY.md`'s runbook is
+still unexecuted against a real host, and the sign-in trap and domain trap it
+documents are still open. It means the one artifact that runbook assumed
+worked, and that a first deploy attempt would have discovered the hard way,
+now actually does.
+
## Repo conventions
- Validate from `cloud/`: `pnpm typecheck && pnpm test && pnpm build`.
From e6ba2ae9ac3b518754a15e4cbdf8804028fb7ef6 Mon Sep 17 00:00:00 2001
From: mohabbis <101276427+mohabbis@users.noreply.github.com>
Date: Wed, 5 Aug 2026 14:35:10 -0500
Subject: [PATCH 4/9] fix(web): stop signin from silently rendering with no way
to sign in
In production with no GitHub OAuth app configured, the signin page rendered
a card with a title and zero buttons -- indistinguishable from a bug. Whoever
hits this is more likely to be standing up the deployment than an end user,
so name the exact fix (AUTH_GITHUB_ID/AUTH_GITHUB_SECRET, the callback URL)
instead of failing silently. See docs/DEPLOY.md's "sign-in trap".
Co-Authored-By: Claude Sonnet 5
---
cloud/apps/web/src/app/signin/page.tsx | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/cloud/apps/web/src/app/signin/page.tsx b/cloud/apps/web/src/app/signin/page.tsx
index 60d650c3..62d71524 100644
--- a/cloud/apps/web/src/app/signin/page.tsx
+++ b/cloud/apps/web/src/app/signin/page.tsx
@@ -6,6 +6,12 @@ import { SOURCE_URL } from "@/lib/source-url";
const githubEnabled = Boolean(process.env.AUTH_GITHUB_ID && process.env.AUTH_GITHUB_SECRET);
const devEnabled = process.env.NODE_ENV !== "production";
+// Both false means production with no OAuth provider configured: no form
+// below has anything to render, and a card with a title and no buttons looks
+// like a bug rather than a missing deploy step. Whoever hits this is more
+// likely to be the person standing up the deployment than an end user, so
+// name the exact fix rather than failing silently. See docs/DEPLOY.md's "sign-in trap".
+const misconfigured = !githubEnabled && !devEnabled;
export default async function SignInPage({
searchParams,
@@ -27,6 +33,25 @@ export default async function SignInPage({
+ {misconfigured && (
+
+
+ No sign-in method is configured
+
+
+ This deployment has NODE_ENV=production and no GitHub OAuth
+ app configured, so there is no way to sign in. Set{" "}
+ AUTH_GITHUB_ID and AUTH_GITHUB_SECRET, with the
+ app's callback at{" "}
+ https://<this-domain>/api/auth/callback/github. See
+ docs/DEPLOY.md.
+