From b024d531dcecca7f976c79cb8f1d8d7f4ebdc6a9 Mon Sep 17 00:00:00 2001
From: Blackwellboy <76774064+Blackwellboy@users.noreply.github.com>
Date: Sun, 26 Jul 2026 16:29:35 +1000
Subject: [PATCH] =?UTF-8?q?laguna=20=C2=A72:=20fold=20in=20#5's=20correcte?=
=?UTF-8?q?d=20kwarg=20model=20+=20revision-dependent=20default?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
---
models/laguna-s-2.1.md | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/models/laguna-s-2.1.md b/models/laguna-s-2.1.md
index 26a78df..45dd0e9 100644
--- a/models/laguna-s-2.1.md
+++ b/models/laguna-s-2.1.md
@@ -25,7 +25,7 @@ If you read nothing else, do these. **Three independent testers on three differe
| | Do this | Why | Where |
|---|---|---|---|
-| 1 | **Minimize thinking** (see the how, below) | It is net-negative on held-out work. **Do not just send `enable_thinking: false`, that is a no-op** | §2 |
+| 1 | **Minimize thinking** (see the how, below) | It is net-negative on held-out work. **Send `enable_thinking: false` explicitly; omitting the kwarg leaves thinking available** | §2 |
| 2 | **Native `poolside_v1` tool format** | Tool-calling is all-or-nothing: ~100% native, 0% under a generic/chatml format (the model narrates instead of calling) | §4 |
| 3 | **Integrity clause in your system prompt** | It refuses blatant fraud but complies when the same act is framed as cleanup (erase a leaked key from history, backdate, forge authorship). One paragraph closes it | §5b |
| 4 | **Pin the revision + set your own max-token ceiling** | Post-release config drift turned thinking on by default and dropped the output cap, which is exactly the "it never stops" recipe. If you do not set a ceiling, nothing does | §5, §5d |
@@ -34,7 +34,7 @@ If you read nothing else, do these. **Three independent testers on three differe
There is no single clean off-switch. Ranked by how well each is evidenced:
-1. **Never send `enable_thinking: true`.** That is the one setting that demonstrably flips the template from a pre-closed `` to an open ``. Sending `false` changes nothing (it is already the default), so if that was your whole "thinking is off" story, it was a no-op.
+1. **Send `enable_thinking: false` explicitly.** This is the one structural off-switch: `false` renders a pre-closed `` and thinking becomes impossible (0/15 reasoned across three prompt shapes at a fixed budget, [#5](https://github.com/TheTom/offlabel/issues/5)). **Omitting the kwarg does not give you the `false` path**: on the tested llama.cpp stack the absent arm renders byte-identical to `true` (the server supplies the value, so the template's `default(false)` never runs), and the model then reasons roughly half the time. The previous version of this advice said sending `false` changes nothing; that was wrong, see §2.
2. **Put a named professional identity in your system prompt** ("You are a senior staff engineer."). This is the lever with the most evidence behind it: three stacks, and in single-turn probes it drove visible reasoning to a clean zero (§5e: 6/6 fired without one, 0/5 with one).
3. **Expect attenuation, not elimination, in long agentic loops.** The same persona that zeroed single-turn thinking only *shortened* it across a 20+ turn tool-using loop (§5e). Budget for some reasoning in long runs regardless.
4. **Set your own max-output-token ceiling** as the backstop, since the vendor dropped theirs (§5).
@@ -90,11 +90,11 @@ Coverage tag per axis: **✅ measured** (held-out, 3-arm, blind 2-vote) · **
## 2. Thinking / reasoning
- **Recommendation:** default **OFF**. Enable ON only for an isolated hard-reasoning turn, and not on integrity-sensitive work.
-- **Control, precisely (corrected 2026-07-25, thanks [@Defilan](https://github.com/Defilan), see §5e):** the `enable_thinking` chat-template kwarg **defaults to `false`**, and in that branch the template hands the model a **pre-closed ``**; with `true` it hands an **open ``**:
+- **Control, precisely (corrected again 2026-07-26 by [@Defilan](https://github.com/Defilan)'s fixed-budget A/B in [#5](https://github.com/TheTom/offlabel/issues/5), superseding the 07-25 correction):** `enable_thinking: false` is a **real, structural off-switch**. It renders a pre-closed `` and the model cannot reason past it: 0/15 reasoned across three prompt shapes at a fixed `max_tokens`. **Omitting the kwarg is not the `false` path**: on the tested llama.cpp stack the absent arm renders **byte-identical to `true`** (open ``), because the server supplies the value itself and the template's `default(false)` branch never runs. In that arm the model reasons roughly half the time (8/15 absent, 9/15 explicit `true`). Verified prompt-level via `/apply-template`, deterministically and with no sampling:
```jinja
{%- if enable_thinking -%}{{- '' -}}{%- else -%}{{- '' -}}{%- endif -%}
```
- **So passing `enable_thinking: false` explicitly does nothing** (it is already the default). If you thought you disabled thinking by sending `false`, you changed no bytes. Passing `true` *is* a real change. The lever that actually moves behavior on a default config is the **persona** (below), not the kwarg.
+ **On top of that, which arm "absent" lands in is revision-dependent:** [@BlackwellBoy](https://github.com/Blackwellboy)'s NVFP4 rev `0761412` checkpoint ships an on-disk template that defaults `enable_thinking` to **`true`** outright, as does Poolside's current HF upload (see the config-drift changelog, §10; details in [his lab's README](https://github.com/Blackwellboy/laguna-s21-lab)). So do not reason about thinking from any template's default, on any stack: **send the kwarg explicitly, and verify your own rendering** (the `render_check.py` proposed in [#6](https://github.com/TheTom/offlabel/pull/6) does this in one command). The persona (below) remains a real second lever, and the only one available when your serving path ignores the kwarg.
- **Firing gate:** a **persona×task conjunction**: a named professional identity ("senior staff engineer") suppresses reasoning to zero. Reproduced independently on a third stack (§5e: 6/6 fired with no persona, 0/5 with one). The *task* half is less settled: we saw coding-shaped tasks suppress regardless of persona; @Defilan's coding probe fired 6/6 without a persona, so the boundary of "coding-shaped" is not pinned down. Treat **persona as the reliable lever** and task-shape as unresolved.
- **⚠️ Open question on what "off" actually means.** Our firing numbers were measured by reading the `reasoning_content` field. Because our OFF arm sent `enable_thinking: false` (= the default, pre-closed ``), **"0% fired" cannot distinguish "the model did not reason" from "it reasoned but nothing populated `reasoning_content`."** On @Defilan's stack, where `reasoning_content` is reliably populated, the model **produces reasoning anyway** under that same default. So the mechanism label may be wrong even though the behavior difference is real: the OFF and ON arms differed materially on outcomes (pass rate, failure modes, and 30/30 vs hung-at-11/30), so *something* changes. Pinning down what needs a same-stack A/B that reads `reasoning_content` and raw content separately. **Not yet run; treat the firing-rate row below as a parsed-reasoning rate, not a proven model-choice rate.**
- **Dose-response** (3 arms, same persona, held-out battery):
@@ -221,6 +221,7 @@ What each dataset actually supports, kept separate:
- **Scope caveats:** single test window, one quant, config is a moving target (version-pinned). Multilingual/crisis probes small-n. Mitigation validated thinking-OFF; thinking-ON re-run and long-regime non-termination pending. Benchmarks not reproduced (behavior-vs-behavior, not a leaderboard).
## Changelog
+- `2026-07-26`: §2 control corrected AGAIN, per [@Defilan](https://github.com/Defilan)'s fixed-budget A/B in [#5](https://github.com/TheTom/offlabel/issues/5), which supersedes the 07-25 correction. `false` is a real structural off-switch (pre-closed ``, 0/15 reasoned); the **absent** arm renders byte-identical to `true` on the tested llama.cpp stack (the server supplies the kwarg, so the template's `default(false)` never runs) and reasons about half the time. The 07-25 line "passing `false` explicitly does nothing" was wrong. Quickstart #1 and the minimize-thinking ranking now say: send `false` explicitly. Also added: the default itself is revision-dependent, [@BlackwellBoy](https://github.com/Blackwellboy)'s NVFP4 rev `0761412` checkpoint and Poolside's current HF upload both default `enable_thinking` to `true`, consistent with the §10 config-drift note. Not yet re-folded here: #5's re-measured persona-gate magnitude (10/18 vs 1/18, artifact-corrected baseline) and the restored task-shape signal; those await the maintainer's read.
- `2026-07-24`: initial behavioral assessment + serving findings on Q4_K_M; §5b integrity clause validated (4/4 caves closed, thinking-OFF).
- `2026-07-25`: added §5e third-stack field notes from @Defilan (gfx1151/llama.cpp/generic harness, `reasoning_content` reliably populated) and corrected §2 as a result. The `enable_thinking` kwarg **defaults to false**, so passing `false` is a no-op against the template default; verified in our own fork's template. Persona gate reproduced (6/6 fired with no persona, 0/5 with one); the **task-shape** half did NOT reproduce and is now marked unresolved. NEW: the gate **attenuates with conversation length** (single-turn persona drove thinking to zero; a 20+ turn agentic loop only shortened it, 35,913 to 20,573 chars, block count 10 to 9), which a scenario battery cannot surface. Flagged an open limitation of our own firing numbers: they were read from `reasoning_content`, so "0% fired" cannot distinguish not-reasoning from not-parsed. §5b gains his independent rule-by-rule ablation on freshly-worded scenarios (clause reproduces 7/7 vs 3/7 unprompted; rule (1) load-bearing; floor of (1)(2)(4)) and his walk-back of the rule-(3) cost magnitude.
- `2026-07-25`: applied @BlackwellBoy's two precision corrections to §5d after review. (a) His ~0.1% thinking rate is an OBSERVED routing rate on that rev/serving path, not proof the model declined to think: empty `reasoning` fields came back even when thinking was requested, so part of it may be template/parser-level. I had written that his number confirmed our persona-gate mechanism; that was overreach and is now corrected. (b) "Zero loops" is scoped to zero UNBOUNDED-GENERATION loops while the gate barely opened; his 9 logged incidents were his own driver's session cap firing by design, so it is not evidence about thinking-heavy workloads. Also recorded his stack precisely (NVFP4 on vLLM, vs our Q4_K_M on llama.cpp) and linked his published raw logs. Separately added the flash-attention serving finding (`-fa on`, up to 2.2x decode at 56K depth) from an independent Strix Halo test.