diff --git a/AGENTS.md b/AGENTS.md index 952675ebe9..ba02b2e0c2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,3 +28,33 @@ React component library for Telefonica's Design System. - Read [CONTRIBUTING](./CONTRIBUTING.md) before creating a PR or contributing to Mistica repo. - Always add the `AI` label to PRs where the code was written by an AI agent. + +# Documentation + +- **Do NOT read all docs upfront** +- When working on a task, use this map to find and read only the docs relevant to your task: + +``` +doc +├── analytics.md # event tracking and analytics props +├── components.md # component catalog and usage (read .d.ts for props) +├── design-tokens.md # skinVars tokens, colors, skin/color-scheme rules +├── fonts.md # font setup and loading +├── forms.md # form example, form fields, validation +├── images +│ └── layout # layout diagrams and SVG assets referenced by layout.md +├── layout.md # Box, Stack, Inline, grid, responsive layout +├── llms +│ ├── agents +│ │ └── figma-verifier.md # Figma verifier agent instructions +│ ├── commits # commit message guidance +│ ├── figma-mcp.md # Figma MCP workflow +│ └── llms.md # main entry point, critical rules, doc index +├── lottie.md # optimizing bundle size with Lottie animations +├── migration-guide.md # migration guide for the new Cards ecosystem (16.x) +├── patterns.md # page composition patterns and best practices +├── sheet.md # Sheet modal component and predefined sheets +├── testing.md # unit/visual test setup and conventions +├── texts.md # customizing default component texts via theme +└── theme-config.md # ThemeContextProvider and ThemeConfig options +``` diff --git a/eval/doc-map/.gitignore b/eval/doc-map/.gitignore new file mode 100644 index 0000000000..c8e15d948d --- /dev/null +++ b/eval/doc-map/.gitignore @@ -0,0 +1,3 @@ +results/ +runs.csv +*.err diff --git a/eval/doc-map/README.md b/eval/doc-map/README.md new file mode 100644 index 0000000000..fa9ba164d0 --- /dev/null +++ b/eval/doc-map/README.md @@ -0,0 +1,64 @@ +# Doc-map experiment + +Measures whether the `# Documentation` map in `AGENTS.md` improves agentic +performance, and whether the per-line descriptions earn their token cost. + +## Arms + +The arms differ only in the content of `AGENTS.md`: + +- `A-no-map` — no documentation map (the `master` baseline). +- `B-bare-tree` — the `doc/` tree with filenames only, no descriptions. +- `C-described` — the `doc/` tree with a one-line description per entry (current). + +Three arms separate two questions: does a map help at all (C and B versus A), +and do the descriptions add value (C versus B). + +## What is measured + +Per `(arm, task)` cell, averaged over repetitions: + +- `inputTotal` — input tokens processed (raw + cache read + cache creation); the primary cost signal. +- `output`, `cost$`, `turns` — output tokens, USD cost, conversation turns. +- `tools` — number of tool calls. +- `docsTouched` — distinct `doc/*.md` files the run opened. +- `readExpected%` — share of runs that opened the task's expected doc (doc-dependent tasks only). +- `wrongDocs` — doc files opened that were not the expected one. + +Tasks are split by tag. `doc-dependent` tasks require a specific doc; the map +should help. `control` tasks need no doc; they expose the map's fixed overhead. + +## Running + +Requires the `claude` CLI and `jq` on `PATH`. + +```bash +cd eval/doc-map +REPS=3 MODEL=sonnet ./run.sh # first pass +node parse.mjs # tabulate; also writes runs.csv +``` + +Set `MODEL=opus` to measure your real working model. Set `REPS` higher to +reduce noise from model nondeterminism. Restrict arms with `ARMS="A-no-map C-described"`. + +The runner swaps `AGENTS.md` per arm and always restores it on exit. It warns +if a task writes files outside `AGENTS.md` and this directory. + +## Reading the result + +- On `doc-dependent` tasks, C (and B) should show lower `inputTotal` and + `tools`, higher `readExpected%`, and lower `wrongDocs` than A. +- On `control` tasks, all arms should be close; a large gap means the map costs + tokens without benefit. +- If C clearly beats B, the descriptions justify themselves; if not, the bare + tree is the better trade. + +## Caveats + +- **Quality is not auto-scored.** Token savings are meaningless if answers get + worse. Grade answer correctness separately (manual rubric or a blind + LLM-judge pass over `results/*.jsonl`) before drawing conclusions. +- **Prompt caching** can mask token differences; keep `REPS` high and compare averages. +- **Nondeterminism** requires repetition; a single run proves nothing. +- For strict isolation against accidental writes, run the experiment inside a + throwaway `git worktree`. diff --git a/eval/doc-map/arms/A-no-map.AGENTS.md b/eval/doc-map/arms/A-no-map.AGENTS.md new file mode 100644 index 0000000000..952675ebe9 --- /dev/null +++ b/eval/doc-map/arms/A-no-map.AGENTS.md @@ -0,0 +1,30 @@ +# Mistica Web + +React component library for Telefonica's Design System. + +**Package manager**: `yarn` + +**Tooling**: yarn storybook (components live doc), yarn playroom (prototyping tool) + +**Critical rules**: + +- NEVER import `@vanilla-extract/css` in `.tsx` files (only in `.css.ts`) +- NEVER import `**/sprinkles.css` in `.tsx` files +- Always namespace React hooks: `React.useState`, `React.useEffect` +- Wrap unit tests with `ThemeContextProvider` + `makeTheme()` +- Prefer semantic queries (`getByRole`, `getByLabelText`) over `getByTestId` + +**Conventions**: `type` over `interface`, `export type` for types, `'use client';` for client components + +**Components**: + +- If you create a new component always create a snippet for playroom. +- If you add props to a component always update stories accordingly and add the new props to args and + argTypes. + +--- + +## GitHub conventions + +- Read [CONTRIBUTING](./CONTRIBUTING.md) before creating a PR or contributing to Mistica repo. +- Always add the `AI` label to PRs where the code was written by an AI agent. diff --git a/eval/doc-map/arms/B-bare-tree.AGENTS.md b/eval/doc-map/arms/B-bare-tree.AGENTS.md new file mode 100644 index 0000000000..a82f6ac810 --- /dev/null +++ b/eval/doc-map/arms/B-bare-tree.AGENTS.md @@ -0,0 +1,60 @@ +# Mistica Web + +React component library for Telefonica's Design System. + +**Package manager**: `yarn` + +**Tooling**: yarn storybook (components live doc), yarn playroom (prototyping tool) + +**Critical rules**: + +- NEVER import `@vanilla-extract/css` in `.tsx` files (only in `.css.ts`) +- NEVER import `**/sprinkles.css` in `.tsx` files +- Always namespace React hooks: `React.useState`, `React.useEffect` +- Wrap unit tests with `ThemeContextProvider` + `makeTheme()` +- Prefer semantic queries (`getByRole`, `getByLabelText`) over `getByTestId` + +**Conventions**: `type` over `interface`, `export type` for types, `'use client';` for client components + +**Components**: + +- If you create a new component always create a snippet for playroom. +- If you add props to a component always update stories accordingly and add the new props to args and + argTypes. + +--- + +## GitHub conventions + +- Read [CONTRIBUTING](./CONTRIBUTING.md) before creating a PR or contributing to Mistica repo. +- Always add the `AI` label to PRs where the code was written by an AI agent. + +# Documentation + +- **Do NOT read all docs upfront** +- When working on a task, use this map to find and read only the docs relevant to your task: + +``` +doc +├── analytics.md +├── components.md +├── design-tokens.md +├── fonts.md +├── forms.md +├── images +│ └── layout +├── layout.md +├── llms +│ ├── agents +│ │ └── figma-verifier.md +│ ├── commits +│ ├── figma-mcp.md +│ └── llms.md +├── lottie.md +├── migration-guide.md +├── patterns.md +├── sheet.md +├── testing.md +├── texts.md +└── theme-config.md +``` diff --git a/eval/doc-map/arms/C-described.AGENTS.md b/eval/doc-map/arms/C-described.AGENTS.md new file mode 100644 index 0000000000..ba02b2e0c2 --- /dev/null +++ b/eval/doc-map/arms/C-described.AGENTS.md @@ -0,0 +1,60 @@ +# Mistica Web + +React component library for Telefonica's Design System. + +**Package manager**: `yarn` + +**Tooling**: yarn storybook (components live doc), yarn playroom (prototyping tool) + +**Critical rules**: + +- NEVER import `@vanilla-extract/css` in `.tsx` files (only in `.css.ts`) +- NEVER import `**/sprinkles.css` in `.tsx` files +- Always namespace React hooks: `React.useState`, `React.useEffect` +- Wrap unit tests with `ThemeContextProvider` + `makeTheme()` +- Prefer semantic queries (`getByRole`, `getByLabelText`) over `getByTestId` + +**Conventions**: `type` over `interface`, `export type` for types, `'use client';` for client components + +**Components**: + +- If you create a new component always create a snippet for playroom. +- If you add props to a component always update stories accordingly and add the new props to args and + argTypes. + +--- + +## GitHub conventions + +- Read [CONTRIBUTING](./CONTRIBUTING.md) before creating a PR or contributing to Mistica repo. +- Always add the `AI` label to PRs where the code was written by an AI agent. + +# Documentation + +- **Do NOT read all docs upfront** +- When working on a task, use this map to find and read only the docs relevant to your task: + +``` +doc +├── analytics.md # event tracking and analytics props +├── components.md # component catalog and usage (read .d.ts for props) +├── design-tokens.md # skinVars tokens, colors, skin/color-scheme rules +├── fonts.md # font setup and loading +├── forms.md # form example, form fields, validation +├── images +│ └── layout # layout diagrams and SVG assets referenced by layout.md +├── layout.md # Box, Stack, Inline, grid, responsive layout +├── llms +│ ├── agents +│ │ └── figma-verifier.md # Figma verifier agent instructions +│ ├── commits # commit message guidance +│ ├── figma-mcp.md # Figma MCP workflow +│ └── llms.md # main entry point, critical rules, doc index +├── lottie.md # optimizing bundle size with Lottie animations +├── migration-guide.md # migration guide for the new Cards ecosystem (16.x) +├── patterns.md # page composition patterns and best practices +├── sheet.md # Sheet modal component and predefined sheets +├── testing.md # unit/visual test setup and conventions +├── texts.md # customizing default component texts via theme +└── theme-config.md # ThemeContextProvider and ThemeConfig options +``` diff --git a/eval/doc-map/parse.mjs b/eval/doc-map/parse.mjs new file mode 100644 index 0000000000..11f5a3af8b --- /dev/null +++ b/eval/doc-map/parse.mjs @@ -0,0 +1,146 @@ +#!/usr/bin/env node +// +// Tabulates the doc-map experiment results. +// +// Reads every results/*.jsonl stream-json log, extracts token usage, cost, +// turns and tool activity per run, then prints a per-arm summary split by task +// tag (doc-dependent vs control) and writes runs.csv for further analysis. +// +// Usage: node parse.mjs + +import {readFileSync, readdirSync, writeFileSync, existsSync} from 'node:fs'; +import {dirname, join} from 'node:path'; +import {fileURLToPath} from 'node:url'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const RESULTS = join(HERE, 'results'); + +const tasks = new Map( + readFileSync(join(HERE, 'tasks.jsonl'), 'utf8') + .split('\n') + .filter(Boolean) + .map((line) => JSON.parse(line)) + .map((task) => [task.id, task]) +); + +const docRefs = (text) => { + const out = new Set(); + for (const match of String(text).matchAll(/doc\/[\w/.-]+\.md/g)) out.add(match[0]); + return out; +}; + +const parseRun = (file) => { + const [arm, id, repWithExt] = file.split('__'); + const rep = repWithExt.replace('.jsonl', ''); + const task = tasks.get(id); + const lines = readFileSync(join(RESULTS, file), 'utf8').split('\n').filter(Boolean); + + let result = null; + let toolCalls = 0; + const docsTouched = new Set(); + + for (const line of lines) { + let event; + try { + event = JSON.parse(line); + } catch { + continue; + } + if (event.type === 'result') result = event; + if (event.type === 'assistant' && Array.isArray(event.message?.content)) { + for (const block of event.message.content) { + if (block.type !== 'tool_use') continue; + toolCalls += 1; + for (const ref of docRefs(JSON.stringify(block.input ?? {}))) docsTouched.add(ref); + } + } + } + + const usage = result?.usage ?? {}; + const inputTokens = usage.input_tokens ?? 0; + const cacheRead = usage.cache_read_input_tokens ?? 0; + const cacheCreate = usage.cache_creation_input_tokens ?? 0; + const expected = task?.expect_doc ?? null; + + return { + arm, + id, + rep, + tag: task?.tag ?? 'unknown', + ok: result?.subtype === 'success', + inputTokens, + cacheRead, + cacheCreate, + inputTotal: inputTokens + cacheRead + cacheCreate, + outputTokens: usage.output_tokens ?? 0, + costUsd: result?.total_cost_usd ?? 0, + numTurns: result?.num_turns ?? 0, + durationMs: result?.duration_ms ?? 0, + toolCalls, + docsTouchedCount: docsTouched.size, + readExpected: expected ? docsTouched.has(expected) : null, + wrongDocs: expected ? [...docsTouched].filter((d) => d !== expected).length : docsTouched.size, + }; +}; + +if (!existsSync(RESULTS)) { + console.error(`No results/ directory yet. Run ./run.sh first.`); + process.exit(1); +} +const files = readdirSync(RESULTS).filter((f) => f.endsWith('.jsonl')); +if (files.length === 0) { + console.error(`No result logs in ${RESULTS}. Run ./run.sh first.`); + process.exit(1); +} + +const runs = files.map(parseRun); + +// runs.csv +const cols = [ + 'arm', 'id', 'tag', 'rep', 'ok', 'inputTotal', 'inputTokens', 'cacheRead', + 'cacheCreate', 'outputTokens', 'costUsd', 'numTurns', 'durationMs', + 'toolCalls', 'docsTouchedCount', 'readExpected', 'wrongDocs', +]; +writeFileSync( + join(HERE, 'runs.csv'), + [cols.join(','), ...runs.map((r) => cols.map((c) => r[c]).join(','))].join('\n') + '\n' +); + +const mean = (xs) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : 0); +const rate = (xs) => { + const defined = xs.filter((x) => x !== null); + return defined.length ? defined.filter(Boolean).length / defined.length : null; +}; +const fmt = (n) => (Number.isFinite(n) ? n.toLocaleString('en-US', {maximumFractionDigits: 2}) : '-'); + +const arms = [...new Set(runs.map((r) => r.arm))].sort(); +const tags = ['doc-dependent', 'control']; + +for (const tag of tags) { + console.log(`\n## ${tag}`); + const header = ['arm', 'n', 'inputTotal', 'output', 'cost$', 'turns', 'tools', 'docsTouched', 'readExpected%', 'wrongDocs']; + console.log(header.join('\t')); + for (const arm of arms) { + const cell = runs.filter((r) => r.arm === arm && r.tag === tag && r.ok); + if (cell.length === 0) continue; + const expectedRate = rate(cell.map((r) => r.readExpected)); + console.log([ + arm, + cell.length, + fmt(mean(cell.map((r) => r.inputTotal))), + fmt(mean(cell.map((r) => r.outputTokens))), + fmt(mean(cell.map((r) => r.costUsd))), + fmt(mean(cell.map((r) => r.numTurns))), + fmt(mean(cell.map((r) => r.toolCalls))), + fmt(mean(cell.map((r) => r.docsTouchedCount))), + expectedRate === null ? '-' : fmt(expectedRate * 100), + fmt(mean(cell.map((r) => r.wrongDocs))), + ].join('\t')); + } +} + +const failed = runs.filter((r) => !r.ok); +if (failed.length) { + console.log(`\n${failed.length} failed run(s): ${failed.map((r) => `${r.arm}/${r.id}/${r.rep}`).join(', ')}`); +} +console.log(`\nWrote ${join(HERE, 'runs.csv')} (${runs.length} runs).`); diff --git a/eval/doc-map/run.sh b/eval/doc-map/run.sh new file mode 100755 index 0000000000..145a6eb430 --- /dev/null +++ b/eval/doc-map/run.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# +# Runs the doc-map A/B/C experiment. +# +# For each arm it swaps the repo AGENTS.md with the arm fixture, then runs every +# task REPS times through `claude -p` in headless mode, capturing the full +# stream-json event log per run under results/. The canonical AGENTS.md is +# always restored on exit. +# +# Usage: +# REPS=3 MODEL=sonnet ./run.sh +# +# Environment: +# REPS repetitions per (arm, task) cell (default: 3) +# MODEL model alias passed to claude --model (default: sonnet) +# Use the model you actually work with (e.g. opus) to measure your +# real workflow; sonnet is the cheaper default for a first pass. +# ARMS space-separated arm names to run (default: all in arms/) +# +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(git -C "$HERE" rev-parse --show-toplevel)" +AGENTS="$ROOT/AGENTS.md" +RESULTS="$HERE/results" +TASKS="$HERE/tasks.jsonl" + +REPS="${REPS:-3}" +MODEL="${MODEL:-sonnet}" + +command -v claude >/dev/null || { echo "claude CLI not found on PATH" >&2; exit 1; } +command -v jq >/dev/null || { echo "jq not found on PATH" >&2; exit 1; } + +mkdir -p "$RESULTS" + +# Back up the canonical AGENTS.md and guarantee restore on any exit. +BACKUP="$(mktemp)" +cp "$AGENTS" "$BACKUP" +restore() { cp "$BACKUP" "$AGENTS"; rm -f "$BACKUP"; echo "Restored canonical AGENTS.md"; } +trap restore EXIT + +if [[ -n "${ARMS:-}" ]]; then + arm_files=(); for a in $ARMS; do arm_files+=("$HERE/arms/$a.AGENTS.md"); done +else + arm_files=("$HERE"/arms/*.AGENTS.md) +fi + +echo "Model: $MODEL | Reps: $REPS | Results: $RESULTS" + +for fixture in "${arm_files[@]}"; do + arm="$(basename "$fixture" .AGENTS.md)" + echo "=== Arm: $arm ===" + cp "$fixture" "$AGENTS" + + jq -c '.' "$TASKS" | while IFS= read -r task; do + id="$(jq -r '.id' <<<"$task")" + prompt="$(jq -r '.prompt' <<<"$task")" + + for rep in $(seq 1 "$REPS"); do + out="$RESULTS/${arm}__${id}__${rep}.jsonl" + printf ' %-20s rep %s ... ' "$id" "$rep" + + before="$(cd "$ROOT" && git status --porcelain | grep -vE 'AGENTS.md|eval/doc-map/' || true)" + + ( cd "$ROOT" && claude -p "$prompt" \ + --output-format stream-json --verbose \ + --model "$MODEL" \ + --permission-mode bypassPermissions ) > "$out" 2> "${out%.jsonl}.err" || true + + after="$(cd "$ROOT" && git status --porcelain | grep -vE 'AGENTS.md|eval/doc-map/' || true)" + if [[ "$before" != "$after" ]]; then + echo "WARNING: working tree changed outside AGENTS.md/eval (a task wrote files). Inspect before trusting results." >&2 + fi + echo "done" + done + done +done + +echo "All runs complete. Parse with: node $HERE/parse.mjs" diff --git a/eval/doc-map/tasks.jsonl b/eval/doc-map/tasks.jsonl new file mode 100644 index 0000000000..1c9c9f5b5e --- /dev/null +++ b/eval/doc-map/tasks.jsonl @@ -0,0 +1,12 @@ +{"id": "forms-validation", "tag": "doc-dependent", "expect_doc": "doc/forms.md", "prompt": "Following this repo's conventions, show the code for a form with email and password fields that validates on submit. Output the code in your reply only; do NOT modify, create, or delete any files."} +{"id": "analytics-tracking", "tag": "doc-dependent", "expect_doc": "doc/analytics.md", "prompt": "Show how to add analytics event tracking to a ButtonPrimary press in this repo. Output the code in your reply only; do NOT modify, create, or delete any files."} +{"id": "design-tokens-color", "tag": "doc-dependent", "expect_doc": "doc/design-tokens.md", "prompt": "Show how to set a Text color using this repo's design tokens instead of a hardcoded hex value. Output the code in your reply only; do NOT modify, create, or delete any files."} +{"id": "sheet-radiolist", "tag": "doc-dependent", "expect_doc": "doc/sheet.md", "prompt": "Show how to open a RadioList sheet in this repo. Output the code in your reply only; do NOT modify, create, or delete any files."} +{"id": "theme-custom-text", "tag": "doc-dependent", "expect_doc": "doc/texts.md", "prompt": "Show how to customize a component's default text via the theme in this repo. Output the code in your reply only; do NOT modify, create, or delete any files."} +{"id": "layout-responsive", "tag": "doc-dependent", "expect_doc": "doc/layout.md", "prompt": "Show how to build a responsive two-column layout that stacks to one column on mobile using this repo's layout primitives. Output the code in your reply only; do NOT modify, create, or delete any files."} +{"id": "lottie-bundle", "tag": "doc-dependent", "expect_doc": "doc/lottie.md", "prompt": "Explain how this repo recommends reducing bundle size when using Lottie animations. Answer in prose; do NOT modify, create, or delete any files."} +{"id": "testing-setup", "tag": "doc-dependent", "expect_doc": "doc/testing.md", "prompt": "Show the boilerplate to unit test a component in this repo, including the required providers. Output the code in your reply only; do NOT modify, create, or delete any files."} +{"id": "migration-cards", "tag": "doc-dependent", "expect_doc": "doc/migration-guide.md", "prompt": "Summarize how to migrate to the new Cards ecosystem in Mistica 16.x per this repo. Answer in prose; do NOT modify, create, or delete any files."} +{"id": "ctrl-package-manager", "tag": "control", "expect_doc": null, "prompt": "What package manager does this repo use, and which command starts Storybook? Answer in one sentence; do NOT modify, create, or delete any files."} +{"id": "ctrl-type-vs-interface", "tag": "control", "expect_doc": null, "prompt": "Per this repo's conventions, should a new prop type use `type` or `interface`? Answer in one sentence; do NOT modify, create, or delete any files."} +{"id": "ctrl-hook-namespacing", "tag": "control", "expect_doc": null, "prompt": "Per this repo's critical rules, how should React hooks be referenced in code? Answer in one sentence; do NOT modify, create, or delete any files."}