docs: record three-model L3 brainstorm sample

[El-Fitz]

Summary

• documents the 2026-06-09 two-prompt, three-model L3 prompt-reframing benchmark sample
• updates TODO 013, the benchmark plan, and handoff with the new results and next-step recommendation

Notes

This is docs-only. The sample compared --prompt-variants off vs per-model with Codex, GLM, and Kimi-for-coding. Per-model improved the small-sample quality floor/disagreement but both expanded runs degraded due to evaluator failures, so no default change is recommended.

[gemini-code-assist]

Code Review

This pull request documents the results of a two-prompt L3 comparison using three Pi-routed models (Codex, GLM, and Kimi-for-coding) with the prompt-reframing expansion. It updates the project handoff, research plan, and active todos to reflect these findings. The feedback points out a minor rounding error in both the handoff and the new brainstorm document, where the average lexical overlap for the per-model variant (0.0735) should be rounded to 0.074 instead of 0.073.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The average lexical overlap for per-model is calculated as (0.102 + 0.045) / 2 = 0.0735. Standard rounding to three decimal places should yield 0.074 instead of 0.073.

Suggested change

	| `per-model` | 8.25 | 8.00 | 0.75 | 0.073 | 0.00 |
	| `per-model` | 8.25 | 8.00 | 0.75 | 0.074 | 0.00 |

[gemini-code-assist]

The average lexical overlap for per-model is (0.102 + 0.045) / 2 = 0.0735. Standard rounding to three decimal places should yield 0.074 instead of 0.073.

Suggested change

- 2026-06-09 three-model L3 sample completed (`todos/013`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`) after PR #44 merged. Compared `--prompt-variants off` vs `per-model` on product and technical prompts using `pi/openai-codex/gpt-5.4:off`, `pi/zai/glm-5.1:off`, and `pi/kimi-coding/kimi-for-coding:off`, serial with `--max-concurrent 1`. Baseline runs completed clean (`18` calls each, ~7-8m). Per-model runs completed with full 12-candidate final sets but degraded evaluation status (`75` calls each, ~30-38m): product had one GLM invalid eval score; technical had one Codex SSE response-header timeout and one GLM invalid eval score. `controversy_floor_7` two-prompt averages improved mean quality `7.83 → 8.25`, min quality `7.00 → 8.00`, disagreement `0.33 → 0.75`; lexical overlap also rose `0.056 → 0.073`; meta-preamble stayed `0.0`. Promising but not enough for default changes because both expanded runs degraded.

- 2026-06-09 three-model L3 sample completed (`todos/013`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`) after PR #44 merged. Compared `--prompt-variants off` vs `per-model` on product and technical prompts using `pi/openai-codex/gpt-5.4:off`, `pi/zai/glm-5.1:off`, and `pi/kimi-coding/kimi-for-coding:off`, serial with `--max-concurrent 1`. Baseline runs completed clean (`18` calls each, ~7-8m). Per-model runs completed with full 12-candidate final sets but degraded evaluation status (`75` calls each, ~30-38m): product had one GLM invalid eval score; technical had one Codex SSE response-header timeout and one GLM invalid eval score. `controversy_floor_7` two-prompt averages improved mean quality `7.83 → 8.25`, min quality `7.00 → 8.00`, disagreement `0.33 → 0.75`; lexical overlap also rose `0.056 → 0.074`; meta-preamble stayed `0.0`. Promising but not enough for default changes because both expanded runs degraded.

[coderabbitai]

Warning

Review limit reached

@El-Fitz, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 45 minutes and 9 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9abd37e5-8750-4b4c-9ac7-cbd1a8528e6c

📥 Commits

Reviewing files that changed from the base of the PR and between ffa366e and 9bd927a.

📒 Files selected for processing (2)

• docs/HANDOFF.md
• docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md

📝 Walkthrough

Walkthrough

This PR documents a 2026-06-09 L3 three-model brainstorm benchmark (comparing --prompt-variants off vs per-model routing on Codex, GLM, and Kimi models), records the run configuration and results metrics, and updates cross-referenced planning and handoff documents with findings and revised next-step guidance.

Changes

L3 Brainstorm Three-Model Sample Benchmark Documentation and Reference Updates

Layer / File(s)	Summary
New L3 three-model brainstorm sample results docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md	Recorded benchmark run for two-prompt, per-model vs prompt-variants-off comparison on Codex, GLM, Kimi-for-coding. Includes configuration (max rounds, panel size, quality floor, concurrency), prompt suite definitions, run-execution outcomes (status, timing, provider failures), production-selector metrics tables (controversy_floor_7 view, two-prompt averages), qualitative observations on cost/timing/feasibility, specific degradation modes (GLM invalid eval scores, Codex SSE timeout), and recommendation to maintain defaults while proceeding with additional samples or evaluation hardening.
TODO 013 latest L3 sample and next-step guidance todos/013-brainstorm-strategy-benchmarks.md	Updated metadata date to 2026-06-09. The "Latest L3" section now documents the three-model sample findings (per-model improvements in controversy_floor_7, evaluation degradations with GLM and Codex), and expands next-step options to include human/calibrated judge pass, 2–4 additional L3 prompt-reframing runs with the same panel, or GLM evaluation-score hardening/triage.
Research plan addendum and verification docs/plans/2026-05-23-001-research-brainstorm-strategy-benchmarks-plan.md	Added dated 2026-06-09 addendum describing the three-model L3 comparison and linking to the brainstorm sample doc. Reworded "Next Implementation Step" to enumerate alternative actions for todos/013 (judge pass, more L3 prompts, GLM triage) with explicit budget/scaling constraints. Extended "Verification" checklist with entry confirming three-model L3 sample documentation.
Handoff document updates with recent context and next session docs/HANDOFF.md	Refreshed "Last updated" timestamp to 2026-06-09. Added new "Recent Context" entry for 2026-06-09 describing three-model L3 sample completion, setup (--prompt-variants off vs per-model), observed degradations, and metric deltas. Rewrote TODO 013 status/next-step summary and revised "Next Clean Session" step 3 to reference the brainstorm sample doc and specify alternative next actions (further L3 prompt-reframing runs with panel or GLM evaluation-score triage).

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

• Lightless-Labs/refinery#38: Both PRs modify docs/HANDOFF.md to update "Last updated" and "Recent Context" entries with latest brainstorm progress and adjust "Next Clean Session" guidance.
• Lightless-Labs/refinery#43: Both PRs refresh docs/HANDOFF.md with new "Last updated" timestamp and add dated "Recent Context" entries documenting brainstorm/benchmark results.
• Lightless-Labs/refinery#37: The main PR's L3 three-model sample results and controversy_floor_7 guidance in todos/013 and docs/HANDOFF.md directly document the running configurations and evaluation behavior effects introduced in the retrieved PR.

Poem

🐰 Three models joined to test the ways,
With prompts refined through brainstorm days,
Codex, GLM, Kimi combined—
Results recorded, findings aligned,
The research path now clearer designed! 🌟

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: recording a three-model L3 brainstorm sample with clear, specific language.
Description check	✅ Passed	The description is directly related to the changeset, explaining the benchmark sample being documented and the files/todos being updated.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch docs/l3-three-model-sample

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}