Rewrite eight descriptions chosen to break the contract

[abernier]

Part of the distilled-docs spec (parent issue). Needs the skill and the eval baseline.

Why

130 of the 170 descriptions exist with a median of 56 characters and generally name no technique; 40 are empty. "Fill the 40" probably undershoots, but that is a guess. The pilot is what turns it into an answer.

What

Run describe-example over ~8 examples, chosen to break the contract rather than flatter it:

• 2 that already have a description — does rewriting earn its keep, or is the existing line good enough?
• 1 whose technique is a prop rather than an import (stencil) — so apis comes out empty and the prose has to carry it alone
• 1 with no notable drei API
• 1 very large
• 1 very small

Then rerun the eval and compare against the recorded baseline.

The decision this unblocks

Fill only the 40, or rewrite all 170. Do not decide it in advance.

Done when

Eight examples have a description and apis under contract, the eval has been rerun, and the issue records both the new score and the fill-40-or-rewrite-170 call with its reasoning.

---

How to implement

Start from a cleared session — this issue is written to be self-sufficient, and a session carrying the design discussion also carries the options that were rejected.

Branch off main once #196 and #197 have merged — it needs the skill to write with and the baseline to measure against.

git checkout -b pilot-eight
/mattpocock-skills:implement #198

implement commits to the current branch, so the branch has to exist first — otherwise this lands on main.

[abernier]

The eight are now constrained — pick them to cover these five

#197 has landed its baseline (#204): 15/20 on today's llms.txt. The five it got wrong are the only room this pilot has to move the number:

Asked for	The model opened	It should have opened
a glass tank you can see objects inside	transparent-aesop-bottles	aquarium
a camera travelling through a doorway	camera-scroll	enter-portals, pass-through-portals
a third-person character controller	named no example — wrote code instead	ecctrl-fisheye
click a part, outline it, frame it	react-pp-outlines	faucets-select-highlight
a glass ornament, bloomed and LUT-graded	color-grading	glass-flower

Every one of them has an empty description. The fifteen it got right point at examples that already say what they are — the index is not uniformly bad, it is bad exactly where the prose is missing.

So the pilot's eight must include those five. §11 of #192 requires that "several questions must have their answer among the pilot's eight", but neither #197 nor this issue named the eight, so nothing enforced it. Eight examples chosen without regard to the question set can leave the score frozen at 15/20 while every rewritten line is individually excellent — and the pilot would have taught nothing, which is the one outcome it exists to prevent.

Suggested: the five above, plus pass-through-portals (the second acceptable answer to the doorway question) and two drawn from the other five questions that already point at empty-description examples. That leaves the ten control questions untouched — if those start missing, the rewrite broke something that worked.

The noise floor is ±1, not zero

temperature: 0 is the least noise on offer, not determinism. An aborted earlier pass on the same index and the same model answered aquarium correctly where the recorded run got it wrong. So a pilot that moves the score by 1 has shown nothing. Covering all five gives a ceiling of 20/20 and a +5 signal, which clears the floor comfortably — a pilot covering only two does not.

Re-measure with the same model, or the comparison is void

bin/eval/baseline.md states it: a score is only comparable within one model. The baseline is openai/gpt-oss-120b, chosen because the pmndrs AI Gateway workspace is on the free tier and 403s every frontier model. If that workspace gets credits and the baseline is re-measured on Claude, that has to happen before this issue rewrites anything — once a description changes, the "before" number is gone for good and cannot be recovered from git, because the score is a measurement and not a file.

Rerun is:

node bin/build-llms.mjs   # llms.txt is generated, and gitignored
node bin/eval.mjs

Expect a wait: the free tier rate-limits hard and the script backs off 60 s. The recorded baseline run took ~25 minutes wall-clock for 20 questions.

Base and stack

#192's chain is landing as one GitHub stack (#205), not merged into main one at a time. This issue sits last in the linearization — main ← #193 ← #196 ← #197 ← #194 ← #195 ← #199 ← #198 — so base on whatever is the top of the stack when you start, not on main. Check with gh stack view.

[abernier]

Pilot done — 19/20, and the call is neither 40 nor 170

Eight examples described under the contract, on pilot-eight (PR to follow).
The eval numbers and their caveats are in bin/eval/pilot-eight.md; the short
version is 15/20 → 19/20, of which +3 belong to these eight lines and +1
to aquarium, which was written before the pilot. The fifteen controls held,
and faucets-select-highlight — left empty on purpose — is still missed, which
is what makes the +3 readable as a gain rather than as a rigged sample.

The sample

Example	Slot it was picked for
pass-through-portals	technique carried by a prop (gl={{ localClippingEnabled: true }})
water-shader	no notable drei API — the ocean is three-stdlib's
react-ellipsecurve	very small, 2.5 kB of source
ssgi-spheres-with-rapier-physics	very large, 515 kB, most of it a vendored library
transparent-aesop-bottles	already described, and adequately
bubbles	already described, and 147 characters of it
glass-flower	empty, and a baseline miss
ecctrl-fisheye	empty, and the miss the baseline could not answer at all

Six left UNDESCRIBED (39 → 33) and bubbles left OVERLONG (5 → 4).

What the sample broke

The prop case is not rare, and it has two siblings the spec did not name.
apis ⊆ imports cannot hold an identifier reached through a namespace import —
new THREE.MeshPhysicalMaterial in transparent-aesop-bottles is as unlistable
as a prop — nor a default import renamed at the call site, where
import Controller from "ecctrl" would only put a meaningless local name in the
field. Both are prose cases. The rule holds; there are simply three ways into it
rather than one.

"Read it whole" does not survive a vendored dependency.
ssgi-spheres-with-rapier-physics is 515 kB, of which 506 kB is a checked-in
build of realism-effects that the example imports from. build-llms already
declines to inline files over 40 kB, so the catalog's own view of that example
is the 8 kB a human wrote. That is the right thing to read, and the skill should
say so rather than leaving the describer to discover it.

Rewriting a line that already passes buys nothing.
The two "already described" slots split cleanly. bubbles at 147 characters is
a forced rewrite — the ratchet requires it, and its overflow was spent listing
five effects its own tags already carry. transparent-aesop-bottles was the
discretionary one, and the rewrite lost thickness: 500 to gain the render-order
pinning: one true detail for another, at three times the length, moving no
question in the eval. It has been reverted to the author's line, with the
glas typo fixed. apis stays, because adding a field is not overwriting a
sentence.

The call: fill by a stated test, not by count

Neither "fill the 40" nor "rewrite the 170". Rewrite the ~81 that fail a test
written down in advance, and leave the other 89 alone.

Test	Count
empty	33
over 120 characters	4
contains an identifier or a URL	14
under 40 characters (the median is 58)	32
union	81
untouched	89

The reasoning is not that a model writes better sentences than the demos'
author. It is that these 81 fail something anyone can check before a single
line is written:

• bruno-simons-20k-challenge's description is a link to a tweet
• bounds-and-makedefault and transformcontrols-and-makedefault carry the
  same sentence, word for word
• chromatic-aberration and dot-screen differ by the name of one effect, and
  each spends its budget naming two packages
• diamond-refraction reads "Diamon refraction material"; camera-shake opens
  "Testing dre's new CameraShake", four years on

The 89 that pass are the answer to the objection this pilot actually surfaced:
rewriting them is an author's sentence overwritten by a model, for a gain the
eval could not detect on the one case where it was tried. The empty ones have
nothing to overwrite, and the failing ones fail a rule that is on paper and can
be argued with before it is applied.

Thresholds worth arguing about: 40 characters is a proxy for "too thin to
discriminate", picked as the bottom quartile, not a measurement. Every flagged
line still goes through the skill's own gate — proposed, then written only on a
maintainer's yes.

[abernier]

Addendum — the ~81 call, read against a decision taken after it

The call above assumed the descriptions would be a pass of their own. They will
not: the plan is now one pass per example, writing its description, its
apis and its README.md explainer from a single reading, across all 170.

That removes the cost half of the argument. If the example is open anyway, the
marginal cost of also rewriting a line that already works is close to zero — so
the reason to leave the 89 alone has to be the other half, and it is:

A sentence by the demo's author, which passes every stated test, is not
replaced by a sentence from a model. The pilot is the evidence, not the
principle: transparent-aesop-bottles was rewritten, the rewrite traded one
true detail for another at three times the length, moved no question in the
eval, and has been reverted. One case is not a law, but it is the only case
where the question was actually put.

So the ~81 stands, unchanged in scope, with its reasoning narrowed to the part
that survives: not "rewriting them is expensive", but "rewriting them is
someone else's sentence overwritten for no measured gain". The 89 get an
explainer like everyone else — that is additive, and it is where the reading
should go.