diff --git a/ROADMAP.md b/ROADMAP.md index 8bf922f..df87dfa 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -47,6 +47,7 @@ Most guides currently have several axes marked `⬚ backlog` (honestly untested) ## 5. Rigor / method - 🔵 **Re-run the thinking ablation single-variable.** Self-audit found the original arms varied token budget as well as `enable_thinking` (OFF ceiling 16,384 / ON 65,536 / CAPPED 4,096 no-retry). CAPPED scoring best argues budget is not the driver, but the comparison is not clean. Re-run holding `max_tokens` fixed across arms. - 🔵 **Settle what "thinking off" actually does on Laguna** (raised by @Defilan, [offlabel#2](https://github.com/TheTom/offlabel/issues/2)). Our firing rates were read from `reasoning_content`, and our OFF arm sent the template default, so they cannot distinguish "did not reason" from "reasoned but was not parsed". Needs a same-stack A/B capturing `reasoning_content` **and** raw content separately, with `enable_thinking` true/false/absent. Behavior differences between arms are real regardless; the mechanism label is what is in doubt. +- ⬚ **Separate task SHAPE from prompt IDENTITY (cheap, high value).** The 400-turn gate grid uses **one fixed prompt per task type**, repeated 40x with only a nonce varying (verified: within-cell prompt-token spread ≤4 across all 40 cells). So "math 92/100, code 62/100, reasoning 47/100, summarization 0/100" are prompt-level results being read as category-level ones. The summarization floor is the most load-bearing and the most exposed, since that prompt is also 288 tokens against 116-124 and the only one supplying a passage to condense. Re-run with **several distinct problems per shape** to settle whether the effects are categorical. Raised by the study author on [#12](https://github.com/TheTom/offlabel/pull/12). - ⬚ **Pin down the task-shape half of the persona gate.** We saw coding-shaped tasks suppress thinking regardless of persona; a third stack saw a coding probe fire 6/6 without a persona. Find the actual boundary, or drop the claim. - ⬚ **Long-loop thinking attenuation.** Single-turn probes show a persona driving thinking to zero; a 20+ turn agentic loop shows only shortening. Standardize a long-loop probe, since scenario batteries structurally miss this. - ⬚ **Adopt a shared spine-probe runner.** @Defilan offered his ([LLMKube#1274](https://github.com/defilantech/LLMKube/pull/1274), `hack/spine-probes/`): drives any OpenAI-compatible endpoint, ablates a clause rule-by-rule (`--arms`), repeats (`--seeds`), re-scores saved transcripts offline. Would move offlabel from "my testing" to "anyone can reproduce this." diff --git a/cards/img/laguna-s-2.1.png b/cards/img/laguna-s-2.1.png index 2021ac1..f3b2e62 100644 Binary files a/cards/img/laguna-s-2.1.png and b/cards/img/laguna-s-2.1.png differ diff --git a/cards/laguna-s-2.1.html b/cards/laguna-s-2.1.html index 94f40a2..6a1bcbe 100644 --- a/cards/laguna-s-2.1.html +++ b/cards/laguna-s-2.1.html @@ -45,7 +45,8 @@
Use it like this
🧠ThinkingSend enable_thinking: false explicitly for long-horizon / integrity work · omitting it = ON · the default moves between revisions, so verify your rendering
-
📉If ONFiring depends on prompt shape, not length: bare 75% (long thinks) · 10 bullet rules 7% · a longer agent prompt back to 60-72% but compressed (3536 → 282 tokens). Do not prompt-engineer it, use the kwarg
+
📉If ONTwo axes that move separately. Whether it fires = persona × task, non-monotonic: bare 75% · 10 dense rules 7.5% · a longer agent prompt back to 60% · +tool schemas 72%. How long it thinks collapses anyway: 3536 → 745 → 282 tokens. Tools cut length 62% and raise firing. Do not prompt-engineer it, use the kwarg
+
🎯Task gates itShape beats anything in the system prompt. Pooled firing: math 92% · code 62% · reasoning 47% · summarization 0/100, never once. But code is a conjunction: 10/10 bare, 0/10 under a bare named persona, 10/10 again under a full agent prompt. One prompt per shape, so read as prompts not categories
🔧ToolsServe the native template (--jinja) · chatml drops tool calls to 0% (83%→0%) · a generic OpenAI-format client is fine, pool not required
🎛️Servingtemp 0.6 · Q4_K_M · -fit off · -fa on (2.2x decode at 56k, gain grows with depth) · --no-mmap on big quants
📌Pin + capPin the revision and set your own max_tokens. The vendor dropped the output cap post-release, so if you do not set one nothing does. The enable_thinking default also moves between revisions
@@ -53,7 +54,7 @@
Independently validated: three stacks
-
🟢12h / 2947 turns99.9% turn success, 0 crashes, 0 restarts, ~4GiB creep, ~13.5s/turn · 100% tool calls on native format · 0 unbounded-generation loops with pin+cap (while the thinking gate stayed shut) · integrity clause held 3/3 cross-quant
@BlackwellBoy (NVFP4/vLLM, 12h soak) · @Defilan (gfx1151/llama.cpp, kwarg + gate A/B) · ours (Q4_K_M/llama.cpp)
+
🟢12h / 2947 turns99.9% turn success, 0 crashes, 0 restarts, ~4GiB creep, ~13.5s/turn · 100% tool calls on native format · 0 unbounded-generation loops with pin+cap (while the thinking gate stayed shut) · integrity clause held 3/3 cross-quant
@BlackwellBoy (NVFP4/vLLM: 12h soak + 400-turn gate grid) · @Defilan (gfx1151/llama.cpp: kwarg render check + rule ablation) · ours (Q4_K_M/llama.cpp: 127-scenario battery)
Trust map: where it holds vs folds under pressure
@@ -63,6 +64,6 @@
⚠ BLIND SPOTS
truncated think kills agent loops (400: needs content or tool_calls; silent + deterministic)invents bugs when thinking is ONthinking ON hung a long agent loop (OFF 30/30 vs ON 11/30)over-refuses authorized work (thinking ON)misses covert self-harm signals
-
Q4_K_M · pinned 2026-07-24 (vendor drift) · held-out behavioral tests + 12h soak + 2 independent stacksofflabel
+
Q4_K_M · pinned 2026-07-24 (vendor drift) · held-out behavioral tests + 12h soak + 2 independent replicationsofflabel
diff --git a/models/laguna-s-2.1.md b/models/laguna-s-2.1.md index e28b537..7d7fb3d 100644 --- a/models/laguna-s-2.1.md +++ b/models/laguna-s-2.1.md @@ -95,8 +95,72 @@ Coverage tag per axis: **✅ measured** (held-out, 3-arm, blind 2-vote) · ** {%- if enable_thinking -%}{{- '' -}}{%- else -%}{{- '' -}}{%- endif -%} ``` **On top of that, which arm "absent" lands in is revision-dependent:** [@BlackwellBoy](https://github.com/Blackwellboy)'s NVFP4 rev `0761412` checkpoint ships an on-disk template that defaults `enable_thinking` to **`true`** outright, as does Poolside's current HF upload (see the config-drift changelog, §10; details in [his lab's README](https://github.com/Blackwellboy/laguna-s21-lab)). So do not reason about thinking from any template's default, on any stack: **send the kwarg explicitly, and verify your own rendering** (the `render_check.py` proposed in [#6](https://github.com/TheTom/offlabel/pull/6) does this in one command). The persona (below) remains a real second lever, and the only one available when your serving path ignores the kwarg. -- **⚠️ Firing gate: the "dose curve" framing here was WRONG and is being replaced (2026-07-26).** This guide briefly said firing scales down with how much apparatus your prompt carries. [@BlackwellBoy's 450-conversation grid](https://github.com/Blackwellboy/laguna-s21-lab/tree/main/gate-study) falsifies the monotonic part: ten bullet rules drop firing to **7%**, but a *longer* full agent prompt brings it back to **60 to 72%**. It is **prompt shape, not prompt quantity**. What the agent prompt actually does is **compress** thinking rather than stop it: median thinking tokens fall from **3,536** bare to **745** and **282** under agent prompts, which is also what reconciles this with the long-loop attenuation in §5e. The replacement model, plus his loop-trigger recipe and the per-task gating, is not folded in yet and is pending review. **Until it is, rely on the kwarg (`enable_thinking: false`), not on prompt engineering.** His raw grid, driver, and logs are published, including the boring cells. -- **Task shape: still genuinely contested, and I am not going to pick a winner.** One stack measured coding-shaped prompts reasoning **least** (code-refactor 2/6 and design 2/6 vs short-factual 6/6, [#5](https://github.com/TheTom/offlabel/issues/5)); another reports coding-shaped prompts reasoning **most** with no apparatus. A plausible reconciliation, flagged as my inference: the second is an all-coding benchmark that never varied shape, so it measures apparatus rather than shape. Unresolved either way, and it does not change what you should do, because the kwarg is the control. +- **Firing gate: two axes, not one dose curve** (replaces the retracted single-axis framing, 2026-07-26; raised in [#11](https://github.com/TheTom/offlabel/issues/11)). This guide briefly said firing scales down with how much apparatus your prompt carries. That was wrong twice over, and the correct model is that **two things move independently and sometimes in opposite directions**: + + 1. **Whether it fires** is a persona-and-task conjunction, **non-monotonic in prompt length**. The hardest suppressor is not the biggest prompt: a dense 10-rule instruction block shuts the gate to **3/40**, while a much *longer* full agent prompt leaves it at **24/40**. Dense behavioral instructions close the gate, prompt mass does not. + 2. **How long it thinks when it does fire** collapses monotonically with apparatus even where firing stays high: median **3,536** tokens bare, **745** under an agent prompt, **282** under an agent prompt with tool schemas. Thinking does not stop, it comes in short bursts that finish instead of running to the ceiling. **This is what reconciles the §5e long-loop attenuation with the single-turn gate**: attenuation and the gate are the same phenomenon measured on different axes. + + **Worked example of the two axes disagreeing, and the reason to keep them separate.** Adding tool schemas to an otherwise identical request cut median reasoning by 62% **and raised firing 12 points**: + + | condition | apparatus | fired | median think-tok (fired) | ceiling hits | + |---|---|---|---|---| + | C7 | full agent prompt (persona + rules + integrity clause) | 24/40 (60%) | 745 | 5 | + | C8 | C7 **+ tool schemas in the request** | **29/40 (72%)** | **282** | 2 | + + So tools are a heavy suppressor of reasoning **length** and a mild *promoter* of firing. Calling tools a "suppressor" full stop is the exact collapse that made the persona-times-task gate confusing for three of us independently. + +- **The curve, single stack (recommended over a cross-stack composite).** All ten points below are one stack, one revision, one prompt set, n=40 each, varying only the apparatus, from [@BlackwellBoy's gate study](https://github.com/Blackwellboy/laguna-s21-lab/tree/main/gate-study) (400 grid turns, 450 logged total, 0 HTTP errors; NVFP4 rev `0761412` on vLLM). I previously assembled a curve from three stacks with different prompt sets, which is the weakest form of the argument. Numbers below re-derived from his published `grid_turns.jsonl` rather than transcribed. + + | | system prompt (dose) | fired | med think-tok | + |---|---|---|---| + | C0 | none | **30/40 (75%)** | 3536 | + | C1 | "helpful assistant" | 24/40 (60%) | 3154 | + | C2 | "coding assistant" | 16/40 (40%) | 2219 | + | C3 | named generic ("Alex, helpful") | 25/40 (62%) | 3069 | + | C4 | named professional ("Alex, senior staff engineer") | 18/40 (45%) | 2390 | + | C5 | professional + 3 rules | 9/40 (22%) | 1735 | + | C6 | professional + 10-rule block | **3/40 (7.5%)** | 325 | + | C7 | full agent prompt (no schemas) | 24/40 (60%) | 745 | + | C8 | C7 + tool schemas | 29/40 (72%) | **282** | + | C9 | C7 + "think step by step" | 23/40 (58%) | 1028 | + + Two other stacks corroborate the *shape* on their own serving paths but are not curve points: ours (~50 to 56% bare) and [@Defilan](https://github.com/Defilan)'s (6/6 bare vs 0/5 with a persona). His bare rate is higher than ours (75% vs ~50%) on a different serving path and prompt set, so treat the absolute levels as stack-specific and the ordering as the transferable part. + +- **Task shape: resolved for the tested prompts, with category-level generalization still open.** I previously left this open because one stack measured coding-shaped prompts reasoning **least** and another **most**. His grid crossed four shapes against all ten apparatus levels, so it separates shape from apparatus, and the two-stack reconciliation below is settled. What it does **not** establish is that the effects belong to the task *categories*, for the reason in the weight-limit note after the tables. + + **Part one, shape is a large independent effect.** Pooled across all ten conditions, 100 samples per shape: + + | task shape | fired | + |---|---| + | math | 92/100 | + | code | 62/100 | + | reasoning | 47/100 | + | **summarization** | **0/100** | + + Summarization never fired once in any condition, including with no system prompt at all (0/105 counting the bare parser-check samples). That is a floor no amount of apparatus explains, which makes **the task itself the strongest single suppressor in the study**. Math is the stickiest, ≥9/10 in every condition except the 10-rule block. + + **Part two, code is a persona conjunction rather than a shape effect**, which is why two stacks could disagree: + + | condition | code | math | + |---|---|---| + | C0 bare | 10/10 | 10/10 | + | C4 bare named persona | **0/10** | 10/10 | + | C7 full agent prompt | 10/10 | 10/10 | + | C8 agent prompt + tools | 10/10 | 10/10 | + + Code fires fully with no apparatus, collapses to zero under a bare named persona, and returns to 10/10 once a full agent prompt is present. So "coding suppresses" and "coding reasons most" are both correct readings of the same model, and which one you get depends entirely on **which single apparatus level you happened to hold fixed**. + + **My earlier inference about the all-coding benchmark is confirmed, not retired** ([his reply on #10](https://github.com/TheTom/offlabel/pull/10)): a benchmark that holds shape constant at coding and carries no apparatus is sitting at the C0 cell where code fires 10/10, so it is measuring the apparatus axis at one point rather than the shape axis. It cannot see the C4 collapse and it cannot see the summarization floor. (I briefly wrote in this guide that the inference was *wrong*; that over-correction lasted one commit and is itself corrected here.) + + **⚠️ Weight limit: shape is confounded with prompt identity, so read these as prompts and not as categories** (raised by the study's author on [#12](https://github.com/TheTom/offlabel/pull/12), and verified here rather than taken on trust). Each task type in that grid is **one fixed prompt template repeated with a nonce prefix**, not 40 different problems. Checking every one of the 40 condition-by-task cells, the within-cell prompt-token spread never exceeds **4 tokens**, which is the nonce tokenizing differently and nothing else. So n=100 per shape is 100 repetitions of one prompt, and it does not buy category-level generality. Two consequences: + - **C0 code 10/10 versus C4 code 0/10 is a clean on/off** and the direction is unambiguous. Whether it is *code as a category* or something narrower about that prompt interacting with a professional persona is not separated. + - **Summarization 0/100 carries the same caveat plus one extra wrinkle.** That prompt is structurally unlike the other three: it is the only one that supplies a passage to work from, and at C0 it is **288 prompt tokens against 116 to 124** for math, code and reasoning. So "summarization never fires" and "a prompt that hands the model a long passage to condense never fires" are not separated by this grid. His lean, which I share, is the task reading, because the floor survives all ten apparatus levels and that is a lot of variation for a prompt artifact to live through. Not settled. + + **The reconciliation itself survives this intact**, which is the important part: it only requires that *the same prompt* behaves differently under different apparatus, and that is exactly what the grid shows. A follow-up with several distinct problems per shape would separate the category question cleanly and is cheap; it is on the roadmap. + + *Median convention, for anyone recomputing:* the think-token medians quoted here are `median_high`, matching the study's published `summary.json`. An averaging median lands a few tokens lower on four of the ten conditions (C1, C2, C4, C7). + + Practical read: if you are trying to **elicit** reasoning, task shape matters more than anything you write in the system prompt. If you are trying to **stop** it, still use the kwarg, because the best any prompt lever reached was 3/40, not 0/40. - **✅ That open question is now largely answered.** This guide previously could not tell whether our OFF arm's 0% meant "did not reason" or "reasoned but was not parsed." It means **did not reason**, and by construction: our OFF arm sent `enable_thinking: false`, which pre-closes `` so reasoning is structurally impossible, independently measured at 0/15 with zero characters at a fixed budget ([#5](https://github.com/TheTom/offlabel/issues/5)). Two supporting checks: zero of 45 responses on that stack had reasoning in `content` with an empty `reasoning_content`, and the vLLM/NVFP4 stack separately verified its own field populates faithfully. So **our firing rates keep the firing-rate label**, and our original ablation's thinking manipulation was valid on that axis. What remains open is only the *budget* confound below. - **Dose-response** (3 arms, same persona, held-out battery): @@ -117,7 +181,7 @@ Coverage tag per axis: **✅ measured** (held-out, 3-arm, blind 2-vote) · ** ## 3. Prompting & persona - **Recommendation:** for coding/agent work a task-focused persona is fine (it suppresses thinking, which is what you want here anyway). To *elicit* reasoning you need a blank/generic persona AND a reasoning-shaped (non-coding) task. -- **Practical trick:** the persona is a **second lever for turning thinking off** when your harness does not expose the toggle. Giving it a named professional identity ("You are a senior game developer...") drove visible reasoning to zero in our tests, and coding-shaped tasks suppress it independently. So a normal agent system prompt is already a thinking suppressor, which is why a real production pipeline sees near-zero thinking even with the flag on (§5d). +- **Practical trick:** the persona is a **second lever for turning thinking off** when your harness does not expose the toggle. Giving it a named professional identity ("You are a senior game developer...") drove visible reasoning to zero in our tests. Note the refinement from §2: coding-shaped tasks do **not** suppress on their own (they fire 10/10 with no system prompt), they suppress *in combination with* a professional persona, which is exactly the pairing a normal agent system prompt gives you. That is why a real production pipeline sees near-zero thinking even with the flag on (§5d). It is a lever, not a switch: the best it reached was 3/40, not 0/40, so use the kwarg when you need it actually off. - **Give it explicit acceptance criteria, not a vibe.** It builds to whatever it can check. Asked for "World 1-1" it produced four platforms and one Goomba, no pipes, coins, bricks, or pits. List the features you want as checkable criteria and it will build to the list. This is the single highest-leverage prompting change for this model. - **Why:** the persona×task thinking gate above, plus the scope-narrowing observed in a full build (§5c). @@ -251,7 +315,8 @@ llama.cpp rejects that with `400: Assistant message must contain either 'content - **Scope caveats:** single test window, one quant, config is a moving target (version-pinned). Multilingual/crisis probes small-n. Mitigation validated thinking-OFF; thinking-ON re-run and long-regime non-termination pending. Benchmarks not reproduced (behavior-vs-behavior, not a leaderboard). ## Changelog -- `2026-07-26`: **retracted the "dose curve" framing I had added earlier the same day.** I wrote that thinking-firing scales down with how much apparatus the prompt carries. [@BlackwellBoy's 450-conversation, 10-condition grid](https://github.com/Blackwellboy/laguna-s21-lab/tree/main/gate-study) falsifies the monotonic part: ten bullet rules drop firing to **7%** (3/40), but a **longer** full agent prompt brings it back to **60-72%**. It is **prompt shape, not prompt quantity**. What the agent prompt actually does is **compress** thinking rather than stop it: median thinking tokens fall from **3,536** bare to **745** then **282** under agent prompts, which is what reconciles this with the §5e long-loop attenuation. Interim advice while the replacement model is pending review: control thinking with the kwarg, not with prompt engineering. +- `2026-07-26`: **§2 firing gate rebuilt on two axes, and task shape resolved rather than left contested** ([#11](https://github.com/TheTom/offlabel/issues/11), @Blackwellboy). Firing probability and reasoning length move **independently, sometimes in opposite directions**, so the single "suppressor" axis is gone. The decisive case: adding tool schemas to an otherwise identical request cut median reasoning **62%** (745 to 282) while firing went **up** 12 points (24/40 to 29/40), making it the second-highest firing condition in the grid. My "tools are a major suppressor, which is why a maximally coding-shaped benchmark with no apparatus reasons the most" line was therefore wrong in both halves and is removed; the bare-prompt conclusion survives on the no-apparatus evidence alone, but not for the reason I gave. **The three-stack composite curve is replaced by his single-stack 10-point curve** (one rev, one prompt set, n=40 per condition, varying only apparatus); our rows and @Defilan's are demoted to corroboration of the ordering, since absolute levels are stack-specific (his bare 75% vs our ~50%). Numbers re-derived from his published `grid_turns.jsonl`, not transcribed. **Task shape resolved for the tested prompts**, and it has two parts. Shape is a large independent effect: pooled across all ten conditions, math fires **92/100**, code **62/100**, reasoning **47/100**, and **summarization 0/100** (0/105 with the bare parser samples), a floor no apparatus explains, which makes the task itself the strongest single suppressor in the study. Separately, **code is a persona conjunction**: 10/10 bare, **0/10** under a bare named persona, 10/10 again under a full agent prompt, so "coding suppresses" and "coding reasons most" are both correct readings depending on which single apparatus level you held fixed. **My inference that an all-coding, no-apparatus benchmark measures apparatus rather than shape is CONFIRMED** by his shape-crossed grid, not retired; it sits at the C0 cell where code fires 10/10, so it cannot see the C4 collapse or the summarization floor. **Weight limit, verified not assumed:** each shape in that grid is ONE fixed prompt repeated with a nonce (within-cell prompt-token spread never exceeds 4 across all 40 cells), and the summarization prompt is also structurally unlike the others (288 prompt tokens vs 116-124, the only one supplying a passage). So these are prompt-level results, not category-level ones, and the section says so. The two-stack reconciliation is unaffected: it only needs the same prompt behaving differently under different apparatus. §3's "coding-shaped tasks suppress independently" corrected to the interaction, and noted the persona lever floors at 3/40 rather than 0. Corrected grid size: **400 grid turns** (450 logged total including parser and criteria phases), not 450 grid turns. +- `2026-07-26`: **retracted the "dose curve" framing I had added earlier the same day.** I wrote that thinking-firing scales down with how much apparatus the prompt carries. [@BlackwellBoy's 10-condition, 400-turn grid](https://github.com/Blackwellboy/laguna-s21-lab/tree/main/gate-study) falsifies the monotonic part: ten bullet rules drop firing to **7%** (3/40), but a **longer** full agent prompt brings it back to **60-72%**. It is **prompt shape, not prompt quantity**. What the agent prompt actually does is **compress** thinking rather than stop it: median thinking tokens fall from **3,536** bare to **745** then **282** under agent prompts, which is what reconciles this with the §5e long-loop attenuation. Interim advice at the time: control thinking with the kwarg, not with prompt engineering. **Superseded by the two-axis entry above**, which folds in the replacement model. - `2026-07-26`: revision from two more independent stacks. **§2 firing gate reframed as a dose-response curve, not a switch** (bare prompt ~50% firing, +persona halves it, +rules/clause/tools ~0.1%): per @BlackwellBoy's reread, his soak sent `enable_thinking: true` on all 3,099 turns, so it was a FIRES arm, and an *unnamed* protocol prompt suppressed as hard as a named one (0.13% vs 0.06%), meaning **the apparatus suppresses, not the persona name**. **The §2 open question is now largely CLOSED**: our OFF arm's 0% means did-not-reason by construction (`false` pre-closes ``), so our firing rates keep the firing-rate label. **NEW §5f: a truncated reasoning turn silently kills agent loops** (400 `must contain content or tool_calls`, killed 5/14 runs, deterministic), a candidate mechanism for our own turn-11 hang. **NEW §5g: measured gfx1151 serving config** (`--no-mmap`, `-fa on` worth 2.27x prefill / 2.22x decode at 56k with retention 40%→75.8%, DFlash 53.8% acceptance on real code editing but 2.1x SLOWER on synthetic). **§4 sharpened**: the requirement is the native *template* server-side (`--jinja`), not the client format; a generic OpenAI client merged four real PRs, so 'use pool' was too narrow. Card updated to match. 🧪 A fourth stack's finding that thinking is **regime-dependent** (ON better for single-turn codegen) is NOT merged here: it is under contributor review in [#10](https://github.com/TheTom/offlabel/pull/10). - `2026-07-26`: §2 control corrected AGAIN, per [@Defilan](https://github.com/Defilan)'s fixed-budget A/B in [#5](https://github.com/TheTom/offlabel/issues/5), which supersedes the 07-25 correction. `false` is a real structural off-switch (pre-closed ``, 0/15 reasoned); the **absent** arm renders byte-identical to `true` on the tested llama.cpp stack (the server supplies the kwarg, so the template's `default(false)` never runs) and reasons about half the time. The 07-25 line "passing `false` explicitly does nothing" was wrong. Quickstart #1 and the minimize-thinking ranking now say: send `false` explicitly. Also added: the default itself is revision-dependent, [@BlackwellBoy](https://github.com/Blackwellboy)'s NVFP4 rev `0761412` checkpoint and Poolside's current HF upload both default `enable_thinking` to `true`, consistent with the §10 config-drift note. Not yet re-folded here: #5's re-measured persona-gate magnitude (10/18 vs 1/18, artifact-corrected baseline) and the restored task-shape signal; those await the maintainer's read. - `2026-07-24`: initial behavioral assessment + serving findings on Q4_K_M; §5b integrity clause validated (4/4 caves closed, thinking-OFF). diff --git a/patterns.md b/patterns.md index f504e0d..11a1565 100644 --- a/patterns.md +++ b/patterns.md @@ -87,6 +87,28 @@ If you use an LLM to grade behavioral outputs, the grader's quantization matters The failure is asymmetric and it is the dangerous direction: a grader that says yes to everything does not merely add noise, it **launders a bad result as a verified one**. Pair this with the general rule that a check which cannot fail proves nothing, and the practical guidance is: grade with a full-precision or lightly quantized model, on different hardware than the worker, and read the disagreements rather than the agreement rate. +## An empty response at a token cap is a failure, not a truncation + +A reasoning model that hits its per-turn token ceiling *inside* the reasoning block returns **no answer at all**: full reasoning field, empty content. The instinct is to read this as "the budget was too small" and raise it. Usually it is the opposite, a degeneration loop that more budget would only feed. + +Measured on a six-requirement acceptance-criteria coding task at a fixed 4,096-token ceiling, 30 runs across three prompt conditions per model: + +| model | empty content (whole budget spent reasoning) | +|---|---| +| Qwen3.6-35B-A3B | **28/30** | +| Laguna S 2.1 | 9/30 | + +Not wrong answers. No answer. 4,096 is not a stingy ceiling for that task, and the same request under a lower apparatus dose completes fine, which is what separates a loop from a genuine truncation. + +Two consequences worth carrying into any benchmark harness: + +1. **Score cap-hits as failures, not as excluded samples.** Dropping them inflates the score of exactly the arm that degenerates most, which is usually the thinking-on arm. +2. **Log the cap-hit rate per sample alongside the score.** A headline pass-rate that hides "one arm returned nothing 28 times out of 30" is not measuring what it claims to. + +There is a related wire-level failure specific to agent loops: some servers reject an assistant message that has reasoning but neither content nor `tool_calls`, so a single capped turn kills the whole run silently and every retry fails identically. See the Laguna guide §5f. + +Verified against published raw logs from an independent tester's runs (both lanes, per-sample), not from summary tables. + ## Sources Synthesized from internal behavioral testing across several models (Ornith-1.0-35B, Ornith-1.0-9B, Qwopus-Coder thinking ablation, Qwen3.6-27B quantization comparison) plus the general finding that externalizing