A -DEV run's tree and rule matches, on every pass

README.md

@@ -45,7 +45,10 @@ VS Code extension gives it. Click one and it fills the
 editor, read-only — a tree coloured as the extension colours trees, node names green, rewrites
 bold, offsets blue. Hover over a node to see the text it covers; go to
 definition (F12) opens the rule that built it, or selects a token's text in the input. Tick
-**Debug** to also keep the tree after every pass. Trees can be large, so the run server keeps
+**Debug** — the engine's `-DEV` — to also keep the tree after every pass: each pass in the
+sequence then carries two buttons, its parse tree and what its rules matched (the input with
+`<<<what a rule built>>>` and `((( what one matched )))` marked, as the extension's Display
+Matched Rules shows it). Trees can be large, so the run server keeps
 them for half an hour and sends one only when it is opened. Without the run server the studio still edits —
 colouring, hover, go to definition, completion, rename and problems all run in the browser,
 the language features in the NLP++ language server from `vscode-nlp` in a Web Worker.

analyzer-views/README.md

@@ -29,7 +29,7 @@ sidebar.append(sequence, kb);
 
 | Element | Lists | Extension view |
 |---|---|---|
-| `<nlp-sequence>` | The passes in `analyzer.seq`, numbered as the engine numbers them | Analyzer sequence |
+| `<nlp-sequence>` | The passes in `analyzer.seq`, numbered as the engine numbers them, each offering the tree and the rule matches a `-DEV` run left | Analyzer sequence |
 | `<nlp-knowledge-base>` | The `.dict` and `.kbb` files under `kb/` | KB |
 | `<nlp-output>` | The files a run wrote, by name | Output files |
 | `<nlp-trees>` | A run's parse trees: `final.tree`, then the tree after each pass | Output files (trees) |
@@ -52,6 +52,26 @@ Every list element takes:
 `<nlp-sequence>` also takes `sequence`, the text of `spec/analyzer.seq`. Its `passes`
 property returns the passes it read from that text.
 
+With `-DEV` on, the engine writes the parse tree after every pass. Hand those trees to
+`<nlp-sequence>` as `trees` and each pass that has one gets two buttons, as the extension's
+sequence view does: its **parse tree**, and **what its rules matched**. Both raise an event
+with the tree's name and the pass number; the page fetches that tree and shows it, or turns
+it into marked-up text with `ruleMatches()`:
+
+```js
+sequence.trees = [{ name: "ana003.tree", pass: 3, passName: "greeting" }];
+sequence.addEventListener("nlp-open-tree", (e) => openTree(e.detail.path));
+sequence.addEventListener("nlp-open-matches", async (e) => {
+  const tree = await fetchTree(e.detail.path);           // the pass's tree, from wherever the run kept it
+  show(ruleMatches(tree, inputText), "txxt");            // the input with every match marked
+});
+```
+
+`ruleMatches(treeText, inputText)` returns the input with `<<<built>>>` around what a rule
+built and `((( fired )))` around what it matched without building — the `.txxt` file the
+extension writes, which `<nlp-code language="txxt">` colors. `{ builtOnly: true }` marks
+only what built a node, and `matchesIn()` gives the spans themselves.
+
 `<nlp-trees>` takes `trees`, as `{ name, pass, passName, bytes }` objects where `pass` is
 `null` for `final.tree`, and `skipped`, the names of trees too large to keep:
 
@@ -132,6 +152,8 @@ follow the same light and dark choice.
   extension's token colors.
 - `problemsIn`, `problemsInLog`, `problemWhere` and `logLines` read the engine's `err.log` and
   `make_ana.log`.
+- `ruleMatches`, `matchesIn` and `matchCount` read a pass's tree for what its rules matched
+  in the text.
 - `filledFields` and `readOutput` read `output.json`.
 
 ## Develop

analyzer-views/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@visualtext/analyzer-views",
   "private": true,
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "An NLP++ analyzer's sequence, knowledge base, run results, values, code and log, shown the way the NLP++ extension for VS Code shows them, for any web page.",
   "type": "module",
   "repository": {

analyzer-views/src/elements.ts

@@ -19,14 +19,19 @@
 // Properties:  files      paths, or { path, bytes } to show sizes (all but <nlp-trees>)
 //              sequence   the text of spec/analyzer.seq (<nlp-sequence>)
 //              trees      { name, pass, passName, bytes } and skipped, names too large
-//                         to keep (<nlp-trees>)
+//                         to keep (<nlp-trees>); on <nlp-sequence>, the trees a -DEV run
+//                         wrote, which put a tree and a rule-matches button on their pass
 //              selected   the path open now, drawn as selected
 //              changed    paths with unsaved edits, marked, with a revert button when
 //                         the element has the `revertable` attribute
 // Attributes:  heading    the list's heading; heading="" for none
 //              revertable offer a revert button on a changed file
 // Events:      nlp-open   { path }  a file was clicked (a tree by its name)
 //              nlp-revert { path }  its revert button was clicked
+//              nlp-open-tree, nlp-open-matches  { path, pass }  a pass's tree, or what its
+//                         rules matched, was asked for: the tree file's name and its pass.
+//                         The page fetches that tree; ruleMatches() (matches.ts) turns it
+//                         into the marked-up text.
 //
 // <nlp-code> takes text, and a path or a language attribute to pick the grammar; <nlp-log>
 // takes problems and lines; <nlp-values> takes output (see their classes).
@@ -46,6 +51,15 @@ import { type Pass, SEQUENCE_FILE, passes, passLabel, passTooltip } from "./sequ
 export interface OpenDetail {
 	path: string;
 	line?: number;          // <nlp-log>: the line the problem is on
+	pass?: number;          // <nlp-sequence>: the pass whose tree or matches were asked for
+}
+
+// A button after a row's name: the tree a pass wrote, and what its rules matched.
+interface RowAction {
+	event: "nlp-open-tree" | "nlp-open-matches";
+	icon: IconName;
+	title: string;
+	detail: OpenDetail;
 }
 
 interface Row {
@@ -56,6 +70,7 @@ interface Row {
 	size?: string;
 	note?: string;          // a line under the name
 	classes?: string[];
+	actions?: RowAction[];
 }
 
 // A property set on an element before it was defined sits on the instance and hides the
@@ -155,6 +170,17 @@ abstract class AnalyzerList extends Base {
 			size.textContent = row.size;
 			line.append(size);
 		}
+		for (const action of row.actions ?? []) {
+			const button = document.createElement("button");
+			button.type = "button";
+			button.className = `nlp-act ${action.icon}`;
+			button.title = action.title;
+			button.setAttribute("aria-label", action.title);
+			button.append(iconElement(action.icon));
+			button.addEventListener("click", () => this.dispatchEvent(
+				new CustomEvent<OpenDetail>(action.event, { detail: action.detail, bubbles: true, composed: true })));
+			line.append(button);
+		}
 		const path = row.path;
 		if (path && name instanceof HTMLButtonElement) {
 			name.type = "button";
@@ -207,6 +233,7 @@ abstract class FileList extends AnalyzerList {
 export class NlpSequence extends FileList {
 	#sequence = "";
 	#passes: Pass[] | null = null;
+	#trees = new Map<number, TreeFile>();
 
 	protected readonly defaultHeading = "Analyzer Sequence";
 
@@ -220,6 +247,21 @@ export class NlpSequence extends FileList {
 		this.render();
 	}
 
+	// The trees a run kept, one per pass -- what the engine writes with -DEV on (the
+	// studio's Debug, the console's Log files). A pass with one gets two buttons: its
+	// tree, and what its rules matched. final.tree belongs to no pass and is ignored
+	// here; <nlp-trees> lists it.
+	get trees(): TreeFile[] {
+		return [...this.#trees.values()];
+	}
+	set trees(trees: Iterable<TreeFile> | null | undefined) {
+		this.#trees = new Map();
+		for (const tree of trees ?? []) {
+			if (tree.pass !== null && tree.pass !== undefined) this.#trees.set(tree.pass, tree);
+		}
+		this.render();
+	}
+
 	// The passes as listed: read from `sequence` against `files`.
 	get passes(): Pass[] {
 		return this.#passes ??= passes(this.#sequence, this.files.map((f) => f.path));
@@ -244,10 +286,23 @@ export class NlpSequence extends FileList {
 				icon: passIcon(p.kind, p.active),
 				title: p.active ? says : `${says} — switched off`,
 				classes,
+				actions: this.actions(p),
 			});
 		}
 		return rows;
 	}
+
+	// The tree the run wrote after this pass, and the text it matched in it.
+	private actions(pass: Pass): RowAction[] {
+		const tree = pass.n === null ? undefined : this.#trees.get(pass.n);
+		if (!tree) return [];
+		const detail: OpenDetail = { path: tree.name, pass: pass.n! };
+		const what = `pass ${pass.n} (${passLabel(pass)})`;
+		return [
+			{ event: "nlp-open-tree", icon: "tree", title: `Open the parse tree after ${what}`, detail },
+			{ event: "nlp-open-matches", icon: "matches", title: `Open what ${what} matched in the text`, detail },
+		];
+	}
 }
 
 export class NlpKnowledgeBase extends FileList {
@@ -621,5 +676,7 @@ declare global {
 	interface HTMLElementEventMap {
 		"nlp-open": CustomEvent<OpenDetail>;
 		"nlp-revert": CustomEvent<OpenDetail>;
+		"nlp-open-tree": CustomEvent<OpenDetail>;
+		"nlp-open-matches": CustomEvent<OpenDetail>;
 	}
 }

analyzer-views/src/icons.ts

@@ -10,7 +10,8 @@
 // both themes and the greyed state is the same shape rather than a second drawing.
 
 export type IconName =
-	"dna" | "dna-off" | "dnar" | "dot" | "folder" | "folder-off" | "dict" | "kbb" | "file" | "tree" | "json" | "log" | "blank";
+	"dna" | "dna-off" | "dnar" | "dot" | "folder" | "folder-off" | "dict" | "kbb" | "file" | "tree" | "matches"
+	| "json" | "log" | "blank";
 
 // dna.svg: the double helix, drawn as two strands.
 const HELIX = `<path fill="currentColor" d="M1.3,0c0,1.1,0.4,2.3,1.1,3.1c0.8,0.9,1.8,1.7,2.9,2.4C6.5,6.3,7.8,7.1,9.1,8c0.8,0.6,1.6,1.3,2.2,2.1c0.7,0.9,1.1,2.1,1.2,3.3c0,0.1,0,0.3,0,0.4h-1.2l-0.1-0.9c0-0.3-0.1-0.4-0.4-0.4H2.2l0.5-1c0.1-0.2,0.3-0.3,0.4-0.2h7.5l-0.9-1C9.7,10.1,9.5,10,9.3,10H4.1V9.9c0.5-0.3,1-0.7,1.4-1C5.7,8.8,5.9,8.8,6,8.8h1.7H8c-0.6-0.4-1.1-0.7-1.6-1C6.3,7.7,6.2,7.7,6.1,7.7C5.4,8.1,4.8,8.5,4.2,8.9c-0.8,0.5-1.4,1.2-2,1.9c-0.7,0.8-1,1.9-1,2.9H0c0.1-0.7,0.2-1.3,0.4-2C0.9,10.3,1.8,9.1,3,8.3c0.7-0.5,1.3-1,2-1.4C4.9,6.8,4.8,6.7,4.7,6.7C3.8,6,2.9,5.4,2.1,4.7C0.8,3.5,0,1.8,0,0H1.3z"/><path fill="currentColor" d="M12.5,0c0,1.2-0.4,2.4-1.1,3.4c-0.6,0.8-1.2,1.5-2,2.1C9,5.8,8.6,6.1,8.2,6.4c-0.1,0.1-0.3,0.1-0.4,0C7.5,6.2,7.2,6,6.9,5.8L8,5.1V5C7.4,5,6.8,5,6.3,5c-0.4,0-0.7-0.1-1-0.3C4.9,4.4,4.5,4.1,4.1,3.8V3.7h5.2c0.2,0,0.4-0.1,0.5-0.2c0.3-0.3,0.5-0.6,0.8-1V2.4H3.2C3,2.5,2.8,2.4,2.7,2.2l-0.5-1h8.6c0.3,0,0.4-0.1,0.4-0.4s0-0.6,0.1-0.9L12.5,0z"/>`;
@@ -36,6 +37,8 @@ const SVG: Record<Exclude<IconName, "blank">, string> = {
 	kbb: `<svg viewBox="0 0 487.57 507.95" aria-hidden="true"><path fill="currentColor" d="M486.05,154.02c-14.7-87.03-103.88-131.96-167.27-147.38S184.91,2.11,133.46,44.72c-51.45,42.61-43.27,141.43-43.27,141.43L0,333.9h99.93c.32,0,.61,72.52.61,72.52,0,0-8.52,46.89,12.79,47.14s92.46,3.63,92.46,3.63l2.76,50.77,196.6-.91-.92-174.05s96.52-91.95,81.83-178.98h-.01ZM157.34,245.24c-16.38,0-29.66-13.28-29.66-29.66s13.28-29.66,29.66-29.66,29.66,13.28,29.66,29.66-13.28,29.66-29.66,29.66Z"/></svg>`,
 	// file.svg
 	file: `<svg viewBox="0 0 16 16" aria-hidden="true"><path fill="currentColor" fill-rule="evenodd" clip-rule="evenodd" d="M10.57 1.14L13.85 4.44L14 4.8V14.5L13.5 15H2.5L2 14.5V1.5L2.5 1H10.22L10.57 1.14ZM10 5H13L10 2V5ZM3 2V14H13V6H9.5L9 5.5V2H3ZM11 7H5V8H11V7ZM5 9H11V10H5V9ZM11 11H5V12H11V11Z"/></svg>`,
+	// symbol-keyword.svg: the lines of text the extension's "Display Matched Rules" opens.
+	matches: `<svg viewBox="0 0 16 16" aria-hidden="true"><path fill="currentColor" d="M15 4H10V3H15V4ZM14 7H12V8H14V7ZM10 7H1V8H10V7ZM12 13H1V14H12V13ZM7 10H1V11H7V10ZM15 10H10V11H15V10ZM8 2V5H1V2H8ZM7 3H2V4H7V3Z"/></svg>`,
 	// tree.svg: the branching parse tree.
 	tree: `<svg viewBox="0 0 16 16" aria-hidden="true"><path fill="currentColor" d="M2.3,11.6L1.5,11l6.4-9.5L8.8,2L2.3,11.6z"/><path fill="currentColor" d="M5.4,14.1l-0.8-0.6l2.9-4.2l0.8,0.6L5.4,14.1z"/><path fill="currentColor" d="M7.6,2l0.8-0.6l6.4,9.5L14,11.6L7.6,2z"/><path fill="currentColor" d="M5.1,5.9l0.8-0.6l5.5,8.1L10.6,14L5.1,5.9z"/></svg>`,
 	// json.svg: the pair of braces.

analyzer-views/src/matches.ts

@@ -0,0 +1,101 @@
+// What a pass's rules matched, read off the tree it wrote: the input text with every
+// match marked, as the NLP++ extension for VS Code's "Display Matched Rules" writes it
+// (vscode-nlp treeFile.ts parseFireds and writeFiredText, the .txxt file).
+//
+// The engine writes no such file. It writes a tree after every pass when it runs with
+// -DEV -- output/ana003.tree is the tree as pass 3 left it -- and every node that a rule
+// matched carries the "fired" flag, with "blt" on the ones a rule built. The marked text
+// is those spans put back over the input:
+//
+//     I live in <<<San Diego>>>, in (((California))).
+//
+//   <<< >>>   a rule built a node here
+//   ((( )))   a rule matched here without building one
+//
+// which is the .txxt grammar's own markers, so the result colours as NLP++ colours a
+// rule-match file.
+//
+// OUTERMOST WINS. The dump is depth-first, parents before children, so a match inside one
+// already marked is skipped rather than nested -- as the extension does it. Nesting would
+// need markers the .txxt grammar does not have.
+//
+// OFFSETS. A node's ustart/uend count code points, inclusive; JavaScript counts UTF-16
+// units, which differ for everything past the Basic Multilingual Plane. They go through
+// utf16Offsets, so a match after an emoji still covers the words it matched.
+//
+// PURE: no DOM, so a server or a test can use it as well as a page.
+
+// One rule match, as [from, to) in UTF-16 units of the input text.
+export interface RuleMatch {
+	from: number;
+	to: number;
+	built: boolean;   // a rule built a node here, rather than only matching
+}
+
+// <indent><name> [start,end,ustart,uend,pass,ruleline,type<,flags>]. The name is
+// non-greedy, so the token for an open bracket in the text keeps "[" as its name.
+const LINE = /^(\s*)(.*?)\s+\[([^\]]*)\]\s*$/;
+
+export interface MatchOptions {
+	// Only the matches that built a node -- the extension's "Display Built Only".
+	builtOnly?: boolean;
+}
+
+// offsets[c] is where code point c starts in UTF-16 units; the last entry is the length.
+export function utf16Offsets(text: string): number[] {
+	const out: number[] = [];
+	let unit = 0;
+	for (const ch of text) {
+		out.push(unit);
+		unit += ch.length;
+	}
+	out.push(unit);
+	return out;
+}
+
+// The matches in one pass's tree, over `text` -- the input exactly as it was run.
+export function matchesIn(tree: string | null | undefined, text: string, options: MatchOptions = {}): RuleMatch[] {
+	const offsets = utf16Offsets(text);
+	const last = offsets.length - 1;
+	const found: RuleMatch[] = [];
+	let covered = -1;   // the last code point already inside a match
+	for (const line of String(tree ?? "").split(/\r?\n/)) {
+		const m = LINE.exec(line);
+		if (!m) continue;
+		const fields = m[3];
+		const cut = fields.indexOf("(");   // attributes, when a dump has them
+		const parts = (cut >= 0 ? fields.slice(0, cut) : fields).split(",").map((p) => p.trim()).filter(Boolean);
+		if (parts.length < 7) continue;
+		const flags = new Set(parts.slice(7));
+		if (!flags.has("fired")) continue;
+		const built = flags.has("blt");
+		if (options.builtOnly && !built) continue;
+		const ustart = Number.parseInt(parts[2], 10);
+		const uend = Number.parseInt(parts[3], 10);
+		if (!Number.isFinite(ustart) || !Number.isFinite(uend)) continue;
+		if (uend <= covered || ustart < 0 || ustart > last || uend < ustart) continue;
+		covered = uend;
+		found.push({ from: offsets[ustart], to: offsets[Math.min(uend + 1, last)], built });
+	}
+	return found;
+}
+
+// The input text with the pass's matches marked: what the extension opens as
+// output/ana003.txxt. Text the pass matched nothing in comes back unchanged.
+export function ruleMatches(tree: string | null | undefined, text: string, options: MatchOptions = {}): string {
+	let out = "";
+	let at = 0;
+	for (const match of matchesIn(tree, text, options)) {
+		const [open, close] = match.built ? ["<<<", ">>>"] : ["(((", ")))"];
+		out += text.slice(at, match.from) + open + text.slice(match.from, match.to) + close;
+		at = match.to;
+	}
+	return out + text.slice(at);
+}
+
+// How many matches a pass made, for a line under its name: "3 matches, 2 built".
+export function matchCount(matches: readonly RuleMatch[]): string {
+	const built = matches.filter((m) => m.built).length;
+	const n = `${matches.length} match${matches.length === 1 ? "" : "es"}`;
+	return built ? `${n}, ${built} built` : n;
+}

analyzer-views/src/rules.ts

@@ -6,5 +6,6 @@ export * from "./runs.js";
 export * from "./languages.js";
 export * from "./colors.js";
 export * from "./logs.js";
+export * from "./matches.js";
 export * from "./values.js";
 export { type IconName, fileIcon, iconSvg, passIcon } from "./icons.js";

analyzer-views/style.css

@@ -38,6 +38,7 @@
 	--nlp-icon-file: #424242;
 	--nlp-icon-log: #424242;
 	--nlp-icon-tree: #424242;
+	--nlp-icon-matches: #424242;
 }
 @media (prefers-color-scheme: dark) {
 	:root:not([data-theme="light"]) {
@@ -47,14 +48,15 @@
 		--nlp-icon-dna: #edec81; --nlp-icon-dna-off: #939598; --nlp-icon-dot: #b5aaff; --nlp-icon-folder: #c5c5c5;
 		--nlp-icon-folder-off: #7a7a7a; --nlp-icon-dict: #00aeef; --nlp-icon-kbb: #ffe680; --nlp-icon-json: #00aeef;
 		--nlp-icon-file: #c5c5c5; --nlp-icon-log: #e6e6e6; --nlp-icon-tree: #c5c5c5;
+		--nlp-icon-matches: #c5c5c5;
 	}
 }
 :root[data-theme="dark"] {
 	--nlp-muted: #9aa0a6; --nlp-hover: #243553; --nlp-selected: #243553; --nlp-selected-ink: #a9c7ff;
 	--nlp-changed: #fdd663; --nlp-error: #f28b82; --nlp-line: rgba(255, 213, 0, .16);
 	--nlp-icon-dna: #edec81; --nlp-icon-dna-off: #939598; --nlp-icon-dot: #b5aaff; --nlp-icon-folder: #c5c5c5;
 	--nlp-icon-folder-off: #7a7a7a; --nlp-icon-dict: #00aeef; --nlp-icon-kbb: #ffe680; --nlp-icon-json: #00aeef;
-	--nlp-icon-file: #c5c5c5; --nlp-icon-log: #e6e6e6; --nlp-icon-tree: #c5c5c5;
+	--nlp-icon-file: #c5c5c5; --nlp-icon-log: #e6e6e6; --nlp-icon-tree: #c5c5c5; --nlp-icon-matches: #c5c5c5;
 }
 
 nlp-sequence, nlp-knowledge-base, nlp-output, nlp-trees { display: block; }
@@ -93,6 +95,7 @@ nlp-sequence[hidden], nlp-knowledge-base[hidden], nlp-output[hidden], nlp-trees[
 .nlp-icon.file { color: var(--nlp-icon-file); }
 .nlp-icon.log { color: var(--nlp-icon-log); }
 .nlp-icon.tree { color: var(--nlp-icon-tree); }
+.nlp-icon.matches { color: var(--nlp-icon-matches); }
 
 .nlp-heading { font-size: 11px; text-transform: uppercase; letter-spacing: .06em; color: var(--nlp-muted);
 	margin: 12px 4px 4px; }
@@ -115,6 +118,14 @@ span.nlp-name { color: var(--nlp-muted); }
 .nlp-item.skipped .nlp-note { padding-left: 4px; }
 .nlp-size { flex: none; font-size: 11.5px; color: var(--nlp-muted); padding: 0 4px; }
 
+/* What a -DEV run left on a pass: its parse tree, and what its rules matched. Faint until
+   the row is under the mouse, since they sit on every pass a debug run wrote a tree for. */
+.nlp-act { flex: none; display: inline-flex; align-items: center; border: 0; background: none; cursor: pointer;
+	padding: 3px 4px; border-radius: 5px; color: var(--nlp-muted); opacity: .55; }
+.nlp-row:hover .nlp-act, .nlp-act:focus-visible { opacity: 1; }
+.nlp-act:hover { background: var(--nlp-hover); }
+.nlp-act .nlp-icon { color: inherit; }
+
 /* A changed file: a mark after its name, and its revert button. */
 .nlp-item.changed button.nlp-name::after { content: " ●"; color: var(--nlp-changed); }
 .nlp-revert { display: none; border: 0; background: none; padding: 2px 6px; color: var(--nlp-muted); font-size: 13px;