Most "make the AI write better code" prompts ship with zero evidence they change anything. This one was checked with a small A/B probe before release, and the test is written down here so you can reproduce it yourself in a few minutes.
No invented numbers below — just the method and what was actually observed.
The claim is narrow and testable: with the skill loaded, the model's output shows the CraftCode patterns; without it, the same prompt produces generic output. So:
-
Baseline (skill OFF). In a clean session with the skill not installed, give the model a task that invites the failure modes — e.g.:
"Write a Python function that charges a card and emails a receipt, with retries."
Save the output.
-
Treatment (skill ON). Install the skill (
cp -R craftcode ~/.claude/skills/), start a fresh session so nothing carries over, and give the same prompt with a trigger like "write this the CraftCode way." Save that output. -
Compare. Look for the concrete, countable patterns — not vibes.
Between baseline and treatment, the treatment output showed the patterns the skill teaches, where the baseline did not:
- Discriminated retry — the treatment retried only transient errors instead of
wrapping everything in
for attempt in range(...). (Pattern 3) - Side-effect scoping + a why comment — the card charge / email send were moved out of the retry path and annotated with the reason, exactly the intent-comment behavior. (Patterns 1 + 3)
- Deliberate fail-open — the peripheral email failure was logged and swallowed on purpose, with a note that a failed receipt must not undo a successful charge. (Pattern 2)
- "Native over the reflex library" — reached for the standard library instead of pulling a retry/util dependency for a few lines. (Pattern 4)
The baseline output was "correct" in the sense that it ran, but flat: one retry loop around all three operations, no why-comments, the double-charge bug intact.
- This is a qualitative recall probe, not a benchmark with a leaderboard. It shows the skill changes the output in the intended direction on a representative task — which is the whole job of a skill like this.
- Results vary by model and phrasing. That's expected; the patterns are the constant, the wording is not. Re-run it on your model and your prompt — the point is that you can.
- The skill is pure markdown, no code executes. The only thing it changes is what the model pays attention to when it writes.
If you run the probe and don't see a difference, that's a bug worth reporting — open an issue with your baseline and treatment outputs.