test - mbertamini/doc-map-eval

AGENTS.md

@@ -28,3 +28,33 @@ React component library for Telefonica's Design System.
 
 - Read [CONTRIBUTING](./CONTRIBUTING.md) before creating a PR or contributing to Mistica repo.
 - Always add the `AI` label to PRs where the code was written by an AI agent.
+
+# Documentation
+
+- **Do NOT read all docs upfront**
+- When working on a task, use this map to find and read only the docs relevant to your task:
+
+```
+doc
+├── analytics.md              # event tracking and analytics props
+├── components.md             # component catalog and usage (read .d.ts for props)
+├── design-tokens.md          # skinVars tokens, colors, skin/color-scheme rules
+├── fonts.md                  # font setup and loading
+├── forms.md                  # form example, form fields, validation
+├── images
+│   └── layout                # layout diagrams and SVG assets referenced by layout.md
+├── layout.md                 # Box, Stack, Inline, grid, responsive layout
+├── llms
+│   ├── agents
+│   │   └── figma-verifier.md # Figma verifier agent instructions
+│   ├── commits               # commit message guidance
+│   ├── figma-mcp.md          # Figma MCP workflow
+│   └── llms.md               # main entry point, critical rules, doc index
+├── lottie.md                 # optimizing bundle size with Lottie animations
+├── migration-guide.md        # migration guide for the new Cards ecosystem (16.x)
+├── patterns.md               # page composition patterns and best practices
+├── sheet.md                  # Sheet modal component and predefined sheets
+├── testing.md                # unit/visual test setup and conventions
+├── texts.md                  # customizing default component texts via theme
+└── theme-config.md           # ThemeContextProvider and ThemeConfig options
+```

eval/doc-map/.gitignore

@@ -0,0 +1,3 @@
+results/
+runs.csv
+*.err

eval/doc-map/README.md

@@ -0,0 +1,64 @@
+# Doc-map experiment
+
+Measures whether the `# Documentation` map in `AGENTS.md` improves agentic
+performance, and whether the per-line descriptions earn their token cost.
+
+## Arms
+
+The arms differ only in the content of `AGENTS.md`:
+
+- `A-no-map` — no documentation map (the `master` baseline).
+- `B-bare-tree` — the `doc/` tree with filenames only, no descriptions.
+- `C-described` — the `doc/` tree with a one-line description per entry (current).
+
+Three arms separate two questions: does a map help at all (C and B versus A),
+and do the descriptions add value (C versus B).
+
+## What is measured
+
+Per `(arm, task)` cell, averaged over repetitions:
+
+- `inputTotal` — input tokens processed (raw + cache read + cache creation); the primary cost signal.
+- `output`, `cost$`, `turns` — output tokens, USD cost, conversation turns.
+- `tools` — number of tool calls.
+- `docsTouched` — distinct `doc/*.md` files the run opened.
+- `readExpected%` — share of runs that opened the task's expected doc (doc-dependent tasks only).
+- `wrongDocs` — doc files opened that were not the expected one.
+
+Tasks are split by tag. `doc-dependent` tasks require a specific doc; the map
+should help. `control` tasks need no doc; they expose the map's fixed overhead.
+
+## Running
+
+Requires the `claude` CLI and `jq` on `PATH`.
+
+```bash
+cd eval/doc-map
+REPS=3 MODEL=sonnet ./run.sh      # first pass
+node parse.mjs                    # tabulate; also writes runs.csv
+```
+
+Set `MODEL=opus` to measure your real working model. Set `REPS` higher to
+reduce noise from model nondeterminism. Restrict arms with `ARMS="A-no-map C-described"`.
+
+The runner swaps `AGENTS.md` per arm and always restores it on exit. It warns
+if a task writes files outside `AGENTS.md` and this directory.
+
+## Reading the result
+
+- On `doc-dependent` tasks, C (and B) should show lower `inputTotal` and
+  `tools`, higher `readExpected%`, and lower `wrongDocs` than A.
+- On `control` tasks, all arms should be close; a large gap means the map costs
+  tokens without benefit.
+- If C clearly beats B, the descriptions justify themselves; if not, the bare
+  tree is the better trade.
+
+## Caveats
+
+- **Quality is not auto-scored.** Token savings are meaningless if answers get
+  worse. Grade answer correctness separately (manual rubric or a blind
+  LLM-judge pass over `results/*.jsonl`) before drawing conclusions.
+- **Prompt caching** can mask token differences; keep `REPS` high and compare averages.
+- **Nondeterminism** requires repetition; a single run proves nothing.
+- For strict isolation against accidental writes, run the experiment inside a
+  throwaway `git worktree`.

eval/doc-map/arms/A-no-map.AGENTS.md

@@ -0,0 +1,30 @@
+# Mistica Web
+
+React component library for Telefonica's Design System.
+
+**Package manager**: `yarn`
+
+**Tooling**: yarn storybook (components live doc), yarn playroom (prototyping tool)
+
+**Critical rules**:
+
+- NEVER import `@vanilla-extract/css` in `.tsx` files (only in `.css.ts`)
+- NEVER import `**/sprinkles.css` in `.tsx` files
+- Always namespace React hooks: `React.useState`, `React.useEffect`
+- Wrap unit tests with `ThemeContextProvider` + `makeTheme()`
+- Prefer semantic queries (`getByRole`, `getByLabelText`) over `getByTestId`
+
+**Conventions**: `type` over `interface`, `export type` for types, `'use client';` for client components
+
+**Components**:
+
+- If you create a new component always create a snippet for playroom.
+- If you add props to a component always update stories accordingly and add the new props to args and
+  argTypes.
+
+---
+
+## GitHub conventions
+
+- Read [CONTRIBUTING](./CONTRIBUTING.md) before creating a PR or contributing to Mistica repo.
+- Always add the `AI` label to PRs where the code was written by an AI agent.

eval/doc-map/arms/B-bare-tree.AGENTS.md

@@ -0,0 +1,60 @@
+# Mistica Web
+
+React component library for Telefonica's Design System.
+
+**Package manager**: `yarn`
+
+**Tooling**: yarn storybook (components live doc), yarn playroom (prototyping tool)
+
+**Critical rules**:
+
+- NEVER import `@vanilla-extract/css` in `.tsx` files (only in `.css.ts`)
+- NEVER import `**/sprinkles.css` in `.tsx` files
+- Always namespace React hooks: `React.useState`, `React.useEffect`
+- Wrap unit tests with `ThemeContextProvider` + `makeTheme()`
+- Prefer semantic queries (`getByRole`, `getByLabelText`) over `getByTestId`
+
+**Conventions**: `type` over `interface`, `export type` for types, `'use client';` for client components
+
+**Components**:
+
+- If you create a new component always create a snippet for playroom.
+- If you add props to a component always update stories accordingly and add the new props to args and
+  argTypes.
+
+---
+
+## GitHub conventions
+
+- Read [CONTRIBUTING](./CONTRIBUTING.md) before creating a PR or contributing to Mistica repo.
+- Always add the `AI` label to PRs where the code was written by an AI agent.
+
+# Documentation
+
+- **Do NOT read all docs upfront**
+- When working on a task, use this map to find and read only the docs relevant to your task:
+
+```
+doc
+├── analytics.md
+├── components.md
+├── design-tokens.md
+├── fonts.md
+├── forms.md
+├── images
+│   └── layout
+├── layout.md
+├── llms
+│   ├── agents
+│   │   └── figma-verifier.md
+│   ├── commits
+│   ├── figma-mcp.md
+│   └── llms.md
+├── lottie.md
+├── migration-guide.md
+├── patterns.md
+├── sheet.md
+├── testing.md
+├── texts.md
+└── theme-config.md
+```

eval/doc-map/arms/C-described.AGENTS.md

@@ -0,0 +1,60 @@
+# Mistica Web
+
+React component library for Telefonica's Design System.
+
+**Package manager**: `yarn`
+
+**Tooling**: yarn storybook (components live doc), yarn playroom (prototyping tool)
+
+**Critical rules**:
+
+- NEVER import `@vanilla-extract/css` in `.tsx` files (only in `.css.ts`)
+- NEVER import `**/sprinkles.css` in `.tsx` files
+- Always namespace React hooks: `React.useState`, `React.useEffect`
+- Wrap unit tests with `ThemeContextProvider` + `makeTheme()`
+- Prefer semantic queries (`getByRole`, `getByLabelText`) over `getByTestId`
+
+**Conventions**: `type` over `interface`, `export type` for types, `'use client';` for client components
+
+**Components**:
+
+- If you create a new component always create a snippet for playroom.
+- If you add props to a component always update stories accordingly and add the new props to args and
+  argTypes.
+
+---
+
+## GitHub conventions
+
+- Read [CONTRIBUTING](./CONTRIBUTING.md) before creating a PR or contributing to Mistica repo.
+- Always add the `AI` label to PRs where the code was written by an AI agent.
+
+# Documentation
+
+- **Do NOT read all docs upfront**
+- When working on a task, use this map to find and read only the docs relevant to your task:
+
+```
+doc
+├── analytics.md              # event tracking and analytics props
+├── components.md             # component catalog and usage (read .d.ts for props)
+├── design-tokens.md          # skinVars tokens, colors, skin/color-scheme rules
+├── fonts.md                  # font setup and loading
+├── forms.md                  # form example, form fields, validation
+├── images
+│   └── layout                # layout diagrams and SVG assets referenced by layout.md
+├── layout.md                 # Box, Stack, Inline, grid, responsive layout
+├── llms
+│   ├── agents
+│   │   └── figma-verifier.md # Figma verifier agent instructions
+│   ├── commits               # commit message guidance
+│   ├── figma-mcp.md          # Figma MCP workflow
+│   └── llms.md               # main entry point, critical rules, doc index
+├── lottie.md                 # optimizing bundle size with Lottie animations
+├── migration-guide.md        # migration guide for the new Cards ecosystem (16.x)
+├── patterns.md               # page composition patterns and best practices
+├── sheet.md                  # Sheet modal component and predefined sheets
+├── testing.md                # unit/visual test setup and conventions
+├── texts.md                  # customizing default component texts via theme
+└── theme-config.md           # ThemeContextProvider and ThemeConfig options
+```