docs: add a docs-writing skill and fix structured-results against it (RES-1495)

[Baukebrenninkmeijer]

Why

The repo has four docs skills and all of them check prose without ever teaching it. docs-drift verifies claims, docs-coverage finds gaps, docs-autofill drafts unattended, mkdocs-material handles the build. Nothing said what a good page looks like, so every draft rediscovered the house voice from scratch — and docs-autofill in particular writes with no human in the loop and no voice guidance at all.

The skill

.claude/skills/docs-writing/SKILL.md holds:

• The assumed reader — an ML/AI engineer already shipping LLM apps, landing from a search result — and the jargon budget that follows: eval-domain terms defined on first use per page, general Python and LLM terms free.
• Three genres, each with an opening pattern and a length bound: tutorial (≤ 800 words), guide (≤ 2500), reference (uncapped, bounded by the surface).
• Four non-negotiables every page carries: a one-line definition, a scope line, a runnable example, a named failure mode.
• Eight voice rules, each anchored to a real before/after pair taken from guides/targets.md, tuning.md and guides/getting-started.md rather than invented.
• Page-level mechanics: fencing, the two mkdocs.yml entries a new page needs, naming a current model.
• A reviewer loop: three narrow sonnet lenses (reader, conformance, cutter) plus the opus content-evaluator agent, every finding validated before it is applied, an explicit blocking list so the loop terminates, and a three-iteration cap that ships a [BLOCKED] PR rather than looping.

CLAUDE.md hands off to it as step 3 of the docs checklist and gives up the two rules the skill now owns. Three rules stay in CLAUDE.md deliberately — they bind in a docstring, a commit message or a PR body, where no docs skill is loaded, so moving them would have silently stopped them binding.

The page

docs/structured-results.md was the first page run through the loop, over three rounds. What it found and the page now fixes:

• The page imported evaluatorq, defined a job, and called neither. A reader landing from search had three scorer functions and no way to run one. The first example is now a complete program that executes offline with no API key.
• The terminal never shows the sub-scores, and the page did not say how to get them. There is now a read-back section, with both of its guards explained rather than presented as tidiness — a float evaluator and a scorer that raised both put a non-cell in the same slot, so the isinstance check is what keeps the loop alive when the reader adds their second evaluator.
• EvaluationResultCellValue appeared in a signature and was never defined. It is now spelled out with its silent edges: bool coerces to 1, a list raises, an int below the top level reads back as a float, and a dict nested deeper than two levels raises.
• Nothing stated the trade. Sub-scores are never aggregated: three float evaluators give three averages, one cell with three keys gives a [structured] placeholder. That is now a section, and it is the reason the scope line at the top says what it says.
• The tracing note claimed the full value is preserved. It now names evaluator_type as the switch that decides — without it an evaluator gets exactly one span copy of the score, in an attribute Orq ingestion drops whole past 512 characters rather than truncating, with nothing logged.

Every code block on the page was executed in document order. The evaluator_type gate, both read-back guards, the bool/list/nesting coercions, and the [structured] placeholder were each verified against the running library rather than read off the source.

The example usage lines

Five example docstrings told the reader to run a path that does not exist — the files moved under examples/lib/ and the usage lines stayed behind. These lines render verbatim onto the generated example pages, so the broken path was published, not merely stale in the repo. The same five also invoked bare python, which in a uv project reaches whichever interpreter is first on PATH — usually one without evaluatorq installed — so a reader would have hit an ImportError immediately after the path was fixed. Both are corrected: real paths, uv run python, env-var prefixes kept ahead of the runner.

examples/README.md also listed langgraph_dataset_eval.py in its tree and gave a command to run it. That file has never existed — PR #54 added four files under langchain/ and a README describing five — so both references are removed rather than repaired. A dataset-based LangGraph example may still be worth having, but that is one to write, not a reference to fix.

Verified two ways rather than by eye: the documented command was executed for the rubric example, and every examples/**.py path referenced in any tracked file now resolves. The first version of that sweep searched only *.py files for references, so it could not see a dead link in a README — which is how the README one survived it.

What the dry run says about the loop itself

Worth recording, since it is the argument for the design:

• The opus editor found every Critical; the three narrow sonnet lenses found the mechanical conformance gaps it did not enumerate. Neither tier was redundant.
• Each round found a defect introduced by the previous round's fix. Round 2's rewrite asserted the span value is written twice, which is only true when evaluator_type is set and none of the page's evaluators set it. Round 3 then caught two more: the nesting depth was wrong in both directions, and the or [] explanation described a failure case that does not occur. All three were false statements about behaviour, not matters of taste — a single-pass review ships all three.
• The validation step earned its place immediately. One finding was dropped as not holding (a suggested Use it when table that would have partitioned nothing), two were downgraded as overstated, one upgraded. A reviewer-invented evaluator_type value was replaced with the codebase's own vocabulary.
• The lenses disagreed twice, and both disagreements were legitimate rather than noise. That is the case for perspective diversity over more reviewers of the same kind.

Checks

• uv run --group docs mkdocs build --strict — clean, exit 0.
• Every Python block on the changed page executed in document order, offline, with ORQ_API_KEY unset.
• No changes under src/.

One more CLAUDE.md rule

CLAUDE.md gains a rule requiring the ticket to be named by its id in the PR title and by no URL anywhere — docs: … (RES-1495). The id is what a reader greps, quotes in Slack and types into search, and a pasted Linear URL rots the moment a workspace or slug changes while the id never does. Linear links both directions from the id plus the PR attachment, so the URL buys nothing.

It is here rather than in its own PR because this PR is what exposed the problem, and because the rule states a cost that only shows up when you follow it: dropping the Closes <id> line is what stops a merge from closing the ticket, so whoever drops it moves the ticket by hand. This PR is the first case — RES-1495 is closed manually.

🤖 Generated with Claude Code

[Baukebrenninkmeijer]

Round 3 of the reviewer loop ran after this PR opened and found two claims the round 2 rewrite had introduced. Both are now fixed in a follow-up commit, and both were verified against the running library rather than read off the source.

The nesting rule was wrong in both directions. A sub-score dict may itself hold dicts of scalars, so {"a": {"b": {"c": 1}}} is accepted and only the fourth level raises. The page had said a third level raises, which would have stopped a reader from trying something legal and left them with an unexplained error when they went deeper.

The explanation of the two or [] guards described a case that does not occur. A job that raises still produces a JobResult with an empty evaluator_scores list (processings.py:180,190); job_results is None only when the datapoint fails before its jobs run at all (processings.py:103). The code was correct and the prose taught the wrong model of when results go missing.

Two overstatements narrowed while there: the blob-storage span copy escapes the 512-character attribute cap rather than being uncapped outright, and pass_ is the only aggregated signal a structured evaluator contributes to a CI gate, not to the summary table, which renders the placeholder either way.

That is three rounds, which is the cap the skill sets. Worth noting for anyone evaluating the loop itself: each round found a defect introduced by the previous round's fix, and every one of them was a false statement about behaviour rather than a matter of taste. A single-pass review would have shipped all three.

[github-actions]

Coverage report

This PR does not seem to contain any modification to coverable code.

[Baukebrenninkmeijer]

The example usage paths are fixed here after all, so the "Not fixed here" section of the PR description no longer applies.

It was five files rather than the three I first reported. The three structured_*_eval.py docstrings and path_organization.py all said examples/<file>.py while living under examples/lib/structured/, and country_unit_test.py had the same defect pointing at examples/ instead of examples/lib/datasets/. These lines render verbatim onto the generated example pages, so the broken path was published, not merely stale in the repo.

Verified two ways: the corrected command was executed for the rubric example rather than eyeballed, and a sweep confirms every examples/**.py path referenced anywhere under examples/ or docs/ now resolves to a real file.

Left alone deliberately: these five lines use bare python where most other examples use uv run python. That is a different call from fixing a wrong path, and normalising it across the tree belongs in its own change.

[currentlycodinng]

Ran every technical claim on the page against the library rather than reading it. Eight of nine hold. One does not, inline below.

Everything else checks out, confirmed by execution: True coerces to 1 as an int, False to 0. A list raises ValidationError. Nested int reads back 2.0. Both isinstance guards discriminate exactly as L89 describes, a 5-evaluator run put an EvaluationResultCell in one slot, three floats in three, and '' with error='scorer exploded' in the raised one, and a job that raised gave evaluator_scores=[]. Aggregation: three float evaluators rendered three averaged rows, the structured cell rendered [structured]. Tracing: without evaluator_type the span carried only orq.explanation and orq.score; a 10-key rubric with long names serialized to 539 characters, over the 512 cap; adding evaluator_type="python_eval" produced the second copy at orq.evaluation.output with all 10 keys intact, and spans.py:213 truncates the explanation but deliberately leaves orq.score alone, so the page is naming a real exposure. The first code block ran with ORQ_API_KEY and OPENAI_API_KEY both unset and produced the table shown, and the read-back block appended after it printed rubric rubric 0.44. mkdocs build --strict is clean, page is in mkdocs.yml nav and llmstxt sections. CI 13/13.

Two things outside this diff, so noting here instead of inline. The PR says every examples/**.py path referenced under examples/ or docs/ now resolves. One doesn't: examples/lib/integrations/langchain/langgraph_dataset_eval.py, referenced at examples/README.md:30 and :55. It's absent at the merge base too so this PR didn't break it, but the claim as written is wrong and the file is one line away from the five you did fix. Separately, examples/pairwise/bt_sigma_ranking.py:21 still invokes bare python rather than uv run python, the same defect the five corrected.

On the skill itself: coherent, and the voice anchors are real text. Checked targets.md:3, tuning.md:3, tuning.md:15 and targets.md:14 against the current files, all verbatim. Every page the genre table names exists. The CLAUDE.md handoff is what it says: 2 rules moved and both landed at SKILL.md:78-79, 3 stay and all three genuinely bind in docstrings and commit messages. One citation is off though: SKILL.md:58-60 attributes the gotcha-placement example to getting-started.md, but that uv add/uv init text lives in docs/index.md:17 and docs/faq.md:27; getting-started.md:8-11 has a different caveat under the install command. Not blocking, the pattern holds, just the pointer.

Also CLAUDE.md picks up a new rule about naming tickets by id in the PR title, unrelated to docs-writing and not mentioned in the description. Fine rule, just flagging it's a stowaway.

Fix L15 and I'm happy with this.

[Baukebrenninkmeijer]

Responses to the four points outside the diff. Three adopted, one deferred to you.

bt_sigma_ranking.py:21 — adopted, uv run python in f58c26a. Same defect as the five, and it was in the same grep output I had already run, so I simply stopped reading at the directory I was editing.

The SKILL.md citation — adopted. The quoted uv add / uv init text is in index.md:17 and faq.md:27, and getting-started.md carries a different install caveat, exactly as you say. index.md is now in the exemplar list.

The false sweep claim — adopted, and the cause is worth naming: the sweep ran with --include='*.py', so it searched Python files for path references and never opened a .md. It could not have found a dead link in a README. The PR body no longer makes the claim; it now reports the one live dead reference and why it is still there.

langgraph_dataset_eval.py — deferred rather than fixed, and I want a decision rather than a guess. It is referenced at examples/README.md:30 and :55, absent here and absent on main. Deleting the two references is the tidy move, but it also buries the question of whether that example was meant to survive a move and got lost; repointing them at langgraph_research_eval.py puts a reader in front of a different example than the one they were promised. Both are one-line changes and I would rather you pick.

On the stowaway — correct, and it is now called out in the PR description under its own heading rather than sitting in the diff unmentioned. Happy to pull it into its own PR if you would prefer the docs change stand alone.

Thanks for re-running the claims against the library instead of reading them; the 539-character rubric is a better demonstration of the 512 cap than anything on the page.

[Baukebrenninkmeijer]

Followed up on langgraph_dataset_eval.py, and the premise of my own question was wrong: nothing deleted it. git log --all -- "*langgraph_dataset_eval*" returns empty — the file has never existed in this repository. PR #54 added four files under langchain/ and a README describing five, and nothing since wrote the fifth.

So there was no lost file to weigh against tidiness, and both references are removed. Every examples/**.py path referenced in any tracked file now resolves. A dataset-based LangGraph example may still be worth having, but that is one to write rather than a reference to repair.

The CLAUDE.md ticket rule stays in this PR and the description now explains it rather than flagging it as unrelated baggage.

[currentlycodinng]

Re-checked against 5f6f8ef, not the commit titles.

L15: fixed. Now reads "one dict layer beyond that raises" with both the accepted and rejected examples spelled out. Matches what I got running it: {"a": {"b": {"c": 2}}} accepted, one level deeper raises.

examples/README.md: fixed. The langgraph_dataset_eval.py reference is gone from both the tree diagram and the usage block, so the claim about every referenced path resolving is now true.

bt_sigma_ranking.py:21: fixed, uv run python.

SKILL.md voice-anchor citation: fixed the right way. Rather than moving the getting-started.md attribution, it added index.md (and the same text is on faq.md:27) to the anchor list, and that uv add/uv init gotcha-placement example genuinely lives on both pages. Checked it against current content, verbatim.

Approving.