feat(crisp): link + confidence in summaries, pause draft replies, skip similar notes

.github/workflows/crisp-investigate-job.yml

@@ -123,6 +123,7 @@ jobs:
           CRISP_IDENTIFIER: ${{ secrets[format('CRISP_{0}_IDENTIFIER', matrix.account)] }}
           CRISP_KEY: ${{ secrets[format('CRISP_{0}_KEY', matrix.account)] }}
           CRISP_WEBSITE_ID: ${{ vars[format('CRISP_{0}_WEBSITE_ID', matrix.account)] }}
+          CONVERSATION_URL: https://app.crisp.chat/website/${{ vars[format('CRISP_{0}_WEBSITE_ID', matrix.account)] }}/inbox/${{ matrix.session_id }}/
         run: |
           set +e
           opencode run "$(cat "${{ steps.prompt.outputs.path }}")" \

prompts/crisp-triage-agent.md

@@ -82,40 +82,22 @@ You are triaging one customer support conversation for the `{{REPO}}` repository
     - Evidence: <ONE short line: at most two file:line references, or one short transcript quote; or what is missing if undetermined>
     - Bug: <what happened for the bug, if any -- "Filed: <url>", "Already tracked: <url>", or omit this line if no bug was filed/tracked (always omit it unless Category is product_bug)>
     - Feature request: <same pattern -- "Filed: <url>", "Already tracked: <url>", or omit this line if none found>
-
-    === DRAFT REPLY (for staff to edit and send -- the bot never messages the customer) ===
-    <the customer reply, see below>
-    === END DRAFT ===
+    - Confidence: <NN/100 -- the same honest score as in the issue body; for no-issue notes, your confidence in the Category>
     ```
-    The `Category` and `Evidence` lines and the draft block are always present, even when no bug or feature request was found. Keep the `=== DRAFT REPLY` and `=== END DRAFT ===` lines exactly as written; tooling splits on them. Internal details (file paths, issue URLs, issue numbers, confidence) belong only above the draft, never inside it.
-
-    **Keep the note easy to scan** (support staff read it in a narrow Crisp panel). The summary after "Investigation report:" is ONE plain sentence, with no internal jargon. The bullet lines are each one line. Use at most one blank line between sections, never two. Do not wrap or hard-break lines inside a paragraph.
+    The `Category`, `Evidence` and `Confidence` lines are always present, even when no bug or feature request was found.
 
-    **Draft layout.** At most about 80 words, in this order: a short greeting using the customer's first name if the transcript gives one; one or two sentences saying what we found, in plain words; then, if you need the customer to do something, a short numbered list with one action per line (maximum 3 items; each one a concrete thing a non-technical person can do; ask for at most two pieces of information); then one closing sentence. Use plain punctuation only: no em dashes, no markdown, no bullet symbols other than "1." style numbering. Do not repeat the same request in two places.
+    **Draft customer replies are PAUSED.** Do not write a draft reply and do not emit `=== DRAFT REPLY` / `=== END DRAFT ===` lines. (Tooling still tolerates them; the old guidance is in git history if drafts are re-enabled.)
 
-    **Draft reply guidelines.** This is a draft for a support person to edit and send; you never contact the customer yourself. Write it as a person would, in English (see Rules): short (a few sentences), warm, plain language a non-technical customer can follow, no jargon. No file paths, code, issue numbers or links to GitHub. No promised release dates or timelines. Never say a fix has shipped, is released, or is coming in a version unless you verified that; for a `product_bug`, say the team is looking into it and, if you know one, give a workaround. For `conflict`, `host`, and `user_error`, include concrete numbered steps or a workaround the customer can try (for example "deactivate the other plugin briefly and check again"; "ask your hosting provider to raise the PHP memory limit to 256M"). Only state facts the transcript or code support; if the category is `undetermined`, ask the one or two specific questions that would settle it (plugin/theme versions, an error message, a screenshot) instead of guessing. For a feature-request-only conversation, thank them and say it has been passed to the team, with no promise it will be built.
-
-    Good example (`conflict`):
-    ```
-    === DRAFT REPLY (for staff to edit and send -- the bot never messages the customer) ===
-    Hi Sam, thanks for the details, that helped a lot. It looks like another plugin on your site is getting in the way of the form. Could you try this?
-
-    1. Switch off your other plugins for a minute, then check whether the form works.
-    2. Turn them back on one at a time until it stops working. That one is the culprit.
-
-    Let us know which one it is and we will help from there.
-    === END DRAFT ===
-    ```
-    Bad example (do not do this): "Hi, this is a regression in `includes/class-form.php:214` tracked in themegrill/foo#312, fixed in 3.4.1 releasing Friday. Please update." -- it exposes internal paths and issue numbers, promises a release date, and claims a fix is shipped without verification.
+    **Keep the note easy to scan** (support staff read it in a narrow Crisp panel). The summary after "Investigation report:" is ONE plain sentence, with no internal jargon. The bullet lines are each one line. Use at most one blank line between sections, never two. Do not wrap or hard-break lines inside a paragraph.
 
-    If neither a bug nor a feature request was found, the note is the summary sentence explaining why (client-side issue, already fixed, etc.), the Category and Evidence lines, and the draft block, with no Bug/Feature request bullet lines.
+    If neither a bug nor a feature request was found, the note is the summary sentence explaining why (client-side issue, already fixed, etc.), the Category, Evidence and Confidence lines, with no Bug/Feature request bullet lines.
 
 ## Rules
 
 - Write everything you produce -- the Crisp note, the GitHub issue body, the issue comment -- in English, regardless of what language the transcript itself is in. Never mirror the customer's language.
 - At most one GitHub-side outcome (a new issue, or a comment on an existing one) *per distinct item* (bug, feature) -- never file and comment for the same item, but a bug and a feature request from the same conversation are separate items and can each independently result in their own outcome.
-- The Crisp note (step 5) always happens exactly once per run, regardless of how many GitHub-side outcomes occurred, and always uses the "Investigation report:" format above, including the Category line and the draft reply block.
+- The Crisp note (step 5) always happens exactly once per run, regardless of how many GitHub-side outcomes occurred, and always uses the "Investigation report:" format above, including the Category and Confidence lines.
 - Never file or comment on a GitHub issue for a bug unless its category is `product_bug`.
-- The draft reply is never sent by you: only the note command above posts anything, and only as a private note.
+- Only the note command above posts anything to Crisp, and only as a private note. Do not write draft customer replies (paused).
 - Never fabricate version numbers, error messages, or environment details the transcript doesn't actually contain.
 - If your confidence is genuinely low, say so in the confidence score rather than skipping the issue — a low-confidence tracked issue is more useful than silence, as long as it's honestly labeled as low-confidence.

scripts/crisp-classifier.mjs

@@ -16,13 +16,13 @@ async function classify(transcript) {
       "defect (bug) or a genuine feature request -- as opposed to a billing " +
       "question, how-to question, client-side misconfiguration, or anything " +
       "that isn't a product code issue. Respond with ONLY a JSON object: " +
-      '{"actionable": boolean, "kind": "bug" | "feature" | "none"}. ' +
+      '{"actionable": boolean, "kind": "bug" | "feature" | "none", "confidence": <integer 0-100, your honest confidence in this verdict>}. ' +
       "Be conservative: when genuinely unsure whether it's a real product " +
       'defect, prefer {"actionable": false, "kind": "none"} -- the next ' +
       "stage is expensive, so false positives cost real money and false " +
       "negatives just wait for a clearer report.",
     transcript,
-    { actionable: false, kind: "none" }
+    { actionable: false, kind: "none", confidence: null }
   );
 }
 
@@ -41,14 +41,14 @@ async function classifyWithProduct(transcript, productNames) {
       '"unknown"; (3) whether the transcript indicates the PRO/premium edition ' +
       "(mentions of a license, purchase, or Pro-only features) or the free edition " +
       "(the default when unclear). Respond with ONLY a JSON object: " +
-      '{"actionable": boolean, "kind": "bug" | "feature" | "none", ' +
+      '{"actionable": boolean, "kind": "bug" | "feature" | "none", "confidence": <integer 0-100, your honest confidence in actionable/kind>, ' +
       '"product": "<exact-name-from-list>" | "unknown", "edition": "free" | "pro"}. ' +
       "Be conservative on actionable/kind: when genuinely unsure whether it's a " +
       'real product defect, prefer {"actionable": false, "kind": "none"} -- the ' +
       "next stage is expensive, so false positives cost real money and false " +
       "negatives just wait for a clearer report.",
     transcript,
-    { actionable: false, kind: "none", product: "unknown", edition: "free" }
+    { actionable: false, kind: "none", product: "unknown", edition: "free", confidence: null }
   );
 }
 
@@ -63,10 +63,10 @@ async function classifyWithProduct(transcript, productNames) {
 // that call's own actionable verdict.
 export async function classifyAndRoute(accountConfig, conversation, transcript, { skipClassifier = false } = {}) {
   if (accountConfig.repo) {
-    const { data: { actionable, kind }, usage } = skipClassifier
+    const { data: { actionable, kind, confidence }, usage } = skipClassifier
       ? { data: { actionable: true, kind: "bug" }, usage: null }
       : await classify(transcript);
-    return { repo: accountConfig.repo, actionable, kind, usage };
+    return { repo: accountConfig.repo, actionable, kind, confidence: confidence ?? null, usage };
   }
 
   if (accountConfig.products) {
@@ -77,14 +77,14 @@ export async function classifyAndRoute(accountConfig, conversation, transcript,
     const repo = result.edition === "pro" && mapping.pro ? mapping.pro : mapping.free;
     return skipClassifier
       ? { repo, actionable: true, kind: result.kind === "feature" ? "feature" : "bug", usage }
-      : { repo, actionable: result.actionable, kind: result.kind, usage };
+      : { repo, actionable: result.actionable, kind: result.kind, confidence: result.confidence ?? null, usage };
   }
 
   const inboxKey = getInboxKey(conversation);
   const mapping = inboxKey && accountConfig.inboxes?.[inboxKey];
   if (!mapping) return { repo: null, unmappedKey: inboxKey };
-  const { data: { actionable, kind }, usage } = skipClassifier
+  const { data: { actionable, kind, confidence }, usage } = skipClassifier
     ? { data: { actionable: true, kind: "bug" }, usage: null }
     : await classify(transcript);
-  return { repo: mapping.repo, actionable, kind, usage };
+  return { repo: mapping.repo, actionable, kind, confidence: confidence ?? null, usage };
 }

scripts/crisp-classify.mjs

@@ -9,6 +9,7 @@
 // account before relying on it.
 import { readFile, writeFile } from "node:fs/promises";
 import {
+  conversationUrl,
   fetchResolvedConversationsSince,
   fetchActiveConversations,
   searchConversationsForManualTrigger,
@@ -26,8 +27,20 @@ const AUTO_ESCALATE_MAX_HOURS = 24 * 30; // past this, only a manual note escala
 // One entry per conversation that reached the classifier this run -- becomes
 // the `scan` event (see events.mjs). Ids, enums and costs only, no text.
 const scanRecords = [];
+// Escalated conversations for the step summary only (links + confidence);
+// kept apart from scanRecords because that becomes the event log.
+const escalatedRows = [];
 function recordScan(path, account, sessionId, result, { manual = false } = {}) {
   const escalated = !!result.repo && (manual || (result.actionable && result.kind !== "none"));
+  if (escalated) {
+    escalatedRows.push({
+      url: conversationUrl(credsForAccount(account), sessionId),
+      repo: result.repo,
+      kind: result.kind ?? "bug",
+      path: manual ? "manual note" : path,
+      confidence: manual ? null : (result.confidence ?? null),
+    });
+  }
   scanRecords.push({
     session_id: sessionId,
     account,
@@ -344,6 +357,12 @@ async function main() {
       `Fetched (resolved, recorded as seen, not investigated): ${totalFetched} · Actionable: ${dedupedMatrix.length} · Unmapped (skipped): ${skippedUnmapped.length} · Already handled while active (skipped): ${alreadyHandled}${duplicatesRemoved ? ` · Duplicate session_id across loops (deduped): ${duplicatesRemoved}` : ""}`,
       `Escalated from active: ${manualEscalations} manual, ${reopenEscalations} reopen-after-resolve, ${autoEscalations} stale-auto (${AUTO_ESCALATE_HOURS}h–${AUTO_ESCALATE_MAX_HOURS}h)`,
     ];
+    if (escalatedRows.length) {
+      lines.push(``, `| Crisp conversation | Repo | Kind | Via | Confidence |`, `| --- | --- | --- | --- | --- |`);
+      for (const r of escalatedRows) {
+        lines.push(`| [open in Crisp](${r.url}) | ${r.repo} | ${r.kind} | ${r.path} | ${r.confidence == null ? (r.path === "manual note" ? "n/a (human call)" : "n/a") : `${r.confidence}/100`} |`);
+      }
+    }
     if (skippedUnmapped.length) {
       lines.push(``, `Unmapped inbox keys / unidentified products seen (add these to config/inbox-to-repo.json if real):`);
       for (const s of skippedUnmapped) lines.push(`- [${s.account}] \`${s.inboxKey}\` (session ${s.session_id})`);

scripts/crisp-post-note.mjs

@@ -59,19 +59,56 @@ const issueUrls = [...report.matchAll(/https:\/\/github\.com\/[^\s)]+\/issues\/\
 const rawCategory = report.match(/^\s*[-*]\s*Category\s*:\s*([`*\w ]+)/im)?.[1]?.toLowerCase().replace(/[`*]/g, "").trim().split(/\s+/)[0];
 const category = rawCategory === "plugin_bug" ? "product_bug" : rawCategory; // legacy name
 if (!category) console.warn("Warning: note has no '- Category:' line (expected product_bug | conflict | host | user_error | undetermined | not_applicable).");
-if (!draftStart) console.warn("Warning: note has no '=== DRAFT REPLY ... ===' block for staff.");
-else if (!/^[ \t]*={2,}[ \t]*END DRAFT\b/im.test(draft)) console.warn("Warning: draft reply block is missing its '=== END DRAFT ===' line.");
+// Draft replies are paused (see prompts/crisp-triage-agent.md): no missing-draft warning.
+if (!/^\s*[-*]\s*Confidence\s*:\s*\d/im.test(report)) console.warn("Warning: note has no '- Confidence: NN/100' line.");
+if (draftStart && !/^[ \t]*={2,}[ \t]*END DRAFT\b/im.test(draft)) console.warn("Warning: draft reply block is missing its '=== END DRAFT ===' line.");
 if (category && category !== "product_bug" && /^\s*[-*]\s*Bug\s*:\s*(filed|already tracked)/im.test(report)) {
   console.warn(`Warning: category is ${category} but the note reports a bug issue as filed/tracked; only product_bug may be filed or commented on.`);
 }
 if (/https?:\/\/|issues?\s*#\d+|#\d{2,}|[\w./-]+\.(php|js|jsx|ts|tsx|css):\d+/i.test(draft)) {
   console.warn("Warning: draft reply contains a URL, issue number, or file path; customer drafts must not.");
 }
 
+// Similarity check for notes WITHOUT issue URLs (and as a second net for ones
+// with): a re-investigation of the same conversation often reaches the same
+// conclusion in slightly different words. Compare only the report part
+// (summary, category, evidence, bug/feature lines), ignoring confidence, which
+// legitimately varies run to run. Same category + >= 0.8 token overlap = same note.
+const SIMILARITY_THRESHOLD = 0.8;
+function reportTokens(text) {
+  const body = text.split(/^[ \t]*={2,}[ \t]*DRAFT REPLY\b/im)[0].replace(/^\s*[-*]\s*Confidence\s*:.*$/gim, "");
+  return new Set(body.toLowerCase().replace(/[^a-z0-9_./:#-]+/g, " ").split(" ").filter(Boolean));
+}
+function categoryOf(text) {
+  const c = text.match(/^\s*[-*]\s*Category\s*:\s*([`*\w ]+)/im)?.[1]?.toLowerCase().replace(/[`*]/g, "").trim().split(/\s+/)[0];
+  return c === "plugin_bug" ? "product_bug" : c;
+}
+function isSimilarNote(existing) {
+  if (!/investigation report:/i.test(existing)) return false; // only compare against our own notes
+  if (categoryOf(existing) !== category) return false;
+  const a = reportTokens(existing);
+  const b = reportTokens(report);
+  if (!a.size || !b.size) return false;
+  let inter = 0;
+  for (const t of a) if (b.has(t)) inter++;
+  return inter / (a.size + b.size - inter) >= SIMILARITY_THRESHOLD;
+}
+
 async function main() {
-  if (issueUrls.length > 0) {
+  let existingNotes = [];
+  try {
     const messages = await fetchRawMessages(creds, sessionId);
-    const existingNotes = messages.filter((m) => m.type === "note").map((m) => m.content ?? "");
+    existingNotes = messages.filter((m) => m.type === "note").map((m) => m.content ?? "");
+  } catch (err) {
+    // A lost note is worse than a duplicate: post anyway if the lookup fails.
+    console.warn(`Could not fetch existing notes for dedupe (posting anyway): ${err.message}`);
+  }
+  const similar = existingNotes.find(isSimilarNote);
+  if (similar) {
+    console.log(`Skipping note -- a similar investigation note (same category, near-identical content) already exists on conversation ${sessionId}`);
+    return;
+  }
+  if (issueUrls.length > 0) {
     const allAlreadyNoted = issueUrls.every((url) => existingNotes.some((content) => content.includes(url)));
     if (allAlreadyNoted) {
       // The skip above predates the draft reply. If this conversation has never

scripts/summarize-investigation.mjs

@@ -34,6 +34,10 @@ async function main() {
   }
 
   const lines = ["### Investigation"];
+  // Link + confidence up top so the summary is actionable without opening the note.
+  const url = process.env.CONVERSATION_URL;
+  const conf = lastText?.match(/^\s*[-*]?\s*Confidence\s*:\s*(\d{1,3})/im)?.[1];
+  if (url) lines.push(`**Crisp conversation:** [open in Crisp](${url})` + (conf ? ` · **Confidence:** ${conf}/100` : ""), "");
   lines.push(
     lastText ? lastText : "_The agent produced no final report -- check the raw log for what happened._"
   );