Skip to content

NOTES.md

Latest commit

 

History

History
515 lines (439 loc) · 31.9 KB

NOTES.md

File metadata and controls

515 lines (439 loc) · 31.9 KB
Raw

ChannelQuant — Lab Notebook

Append-only journal. Every measurement carries provenance: seed, exact command line, model, n, wall time, host. No undated, un-sourced numbers. Corrections are appended, never edited in place. Claims ledger lives in CLAIMS.md.

Standing environment (record any change as a new entry):

  • Host: spark-bd9f — DGX Spark, GB10 (Grace-Blackwell, unified ~121 GiB), GPU shared with other users (check nvidia-smi --query-compute-apps, never kill others' processes; memory.free reads N/A on this box).
  • Python 3.12.3, torch 2.12.0+cu130, transformers 5.10.2.
  • HF_HOME=/home/chaithu/lhs/.hf_cache (root owns ~/.cache), venv /home/chaithu/lhs/.venv, MPLCONFIGDIR=/tmp/mpl for figures.
  • PPL is banned as a success metric (overstated TurboQuant+ 3×). Report HellaSwag/ARC acc_norm with Wilson 95% CIs. Screening n=250; headline n≥1000.

2026-06-22 — Phase 0: repo scaffold + README + notebook

Seeded the ChannelQuant repo as a self-contained sibling of kv-cache-engine (TurboQuant+, the documented predecessor) and adaptive-precision-attention (APA — where the c13–c19 diagnosis originated). Did not recreate the working dir; it arrived pre-seeded with REVAMP_SPEC.md, the prior-art landscape doc, and the decisive evidence artifacts (c17_*, c19_*, fig_quantizer_fix.png).

Actions:

  • Created directory layout per spec §6, mirroring the predecessor's analysis/ sw/reference_model/ rtl/tb/ openlane/ docs/ structure (chose sw/reference_model/ over §6's analysis/reference_model.py to match the established predecessor convention and the build-order's sw/reference_model reference; analysis/ holds experiment studies).
  • Wrote README.md: leads with KIVI/KVQuant attribution and the scoped hardware-novelty claim, then the c17 evidence table, fig_quantizer_fix.png, the c19 gate result, tiers, layout, build status, environment, lineage.
  • Started this NOTES.md and CLAIMS.md (claims ledger).

Provenance of the inherited evidence (carried from APA, to be re-derived in Phase 1 before any number becomes an RTL golden):

  • analysis/c17_{q05,q15,q7}_summary.json — HellaSwag acc_norm, n=250, Qwen2 {0.5B,1.5B,7B}, uniform quantizer, Wilson CIs in-file. Headline contrast: naive per-token INT4 collapses (1.5B 0.248, 7B 0.212 vs FP16 0.540/0.612); per-channel/KIVI/outlier recover ~FP16.
  • analysis/c19_{q05,q15,q7}_summary.json — static-outlier-ROM gate. Mean top-2 channel stability across 8 calibration batches: 0.958 / 0.986 / 0.984; layer-0 = 1.00 at every scale; concentration 5.4× / 7.8× / 8.0× median. GATE PASSED → build CQ-4+ (static mask, no runtime argsort in silicon).

Verification status of all numbers above: INHERITED from APA, not yet re-derived in this repo. They are reported as provisional in CLAIMS.md and become confirmed only when Phase 1 reproduces c17 from the clean reference model within ±0.02 (spec §7 step 1, the next blocking gate).

Checklist — Phase 0:

  • Directory scaffold (analysis/ sw/ rtl/ openlane/ docs/paper/ + testvectors)
  • README leads with KIVI/KVQuant credit + scoped HW-novelty claim
  • README carries c17 table, fig_quantizer_fix.png, c19 gate, §10 intro
  • NOTES.md started with standing-environment + provenance block
  • CLAIMS.md ledger started, inherited numbers marked provisional
  • (next, Phase 1) reproduce c17 from sw/reference_model within ±0.02

Decision / next action: pause for go-ahead before Phase 1. No RTL until the software reference reproduces c17 (gate). No GPU model runs yet — Phase 1 will validate small/CPU first, then monitored GPU.

2026-06-22 — Rescope: ChannelQuant = algorithm, KVCE = silicon (correction)

Correction to the entry above. Project hierarchy clarified: KVCE (the kv-cache-engine repo) is a silicon block of the Longhorn chip; ChannelQuant is only the quantization algorithm that block implements. The earlier Phase-0 entry created rtl/ and openlane/ and a 6-phase plan ending in synthesis — that silicon work does not belong in this repo. It lives in the KVCE block, revamped separately.

Actions taken to rescope:

  • Deleted rtl/ and openlane/ (never committed — clean removal). Phases 4 (RTL) and 5 (synthesis) removed from the build order. New plan: P0 scaffold · P1 reference + c17 repro + golden vectors · P2 calibrator + group-size Pareto · P3 generalization · P4 method paper.
  • Renamed sw/reference_model/ → reference/; golden vectors live in reference/testvectors/ and are now a first-class Phase-1 deliverable (inputs + expected compressed payload + expected decompressed output, for CQ-8/CQ-4/CQ-4+), which the KVCE block consumes for 3-way Python↔C++↔SV parity.
  • Converted spec §5 "Hardware delta" into docs/HW_CONTRACT.md — a precise algorithm→silicon interface contract (scaling axes, group size G + residual flush semantics, static outlier-mask format, exact INT4/INT8 quant rule [round-half-to-even, clamp [-8,7]], per-tier bit/packing layout, fp16 scale format, parity acceptance). It declares the reference model + golden vectors as ground truth where prose disagrees.
  • Updated README.md (added a Scope section, revised layout + build-status, RTL/synth marked out-of-scope → KVCE), CLAIMS.md (area/Fmax now forthcoming from KVCE, not this repo), docs/paper/outline.md (silicon as a forthcoming companion section), and REVAMP_SPEC.md (rescope banner; §5 redirected to HW_CONTRACT; §6 layout; §7 RTL/synth steps redirected).

Everything else binding: algorithm §3, evidence §2, compression math §4, gates §7 (steps 0–3), stat hygiene §8, KIVI/KVQuant credit — unchanged.

Decision / next action: committing Phase 0 (rescoped) immediately, then proceeding to Phase 1 (implement reference/channelquant_ref.py CQ-8/4/4+ per HW_CONTRACT, reproduce c17 within ±0.02 — small/CPU first then monitored GPU — and emit golden vectors).

2026-06-22 — Phase 1: reference model + c17 reproduction GATE + golden vectors

Built the bit-accurate reference codec, proved it reproduces the inherited c17 evidence, and emitted the KVCE golden vectors. GATE: PASS (bit-exact).

Artifacts (this repo):

  • reference/channelquant_ref.py — canonical numpy codec (compress/decompress for values per-token, keys per-channel-grouped + outlier sidecar; little-endian nibble packing) + torch fake-quant tier wrappers, per docs/HW_CONTRACT.md. Quant rule: signed symmetric INT, round-half-to-even (np.rint), clamp INT4 [-8,7] / INT8 [-128,127], fp16 scales, EPS=2^-14.
  • reference/test_reference.py — 7 unit tests, all pass: numpy round-trip (values; keys G∈{None,32,64,128} incl. partial group), int4 nibble pack inverts, clamp-not-wrap, and bit-exact parity torch-fakequant == numpy pack→unpack for values, keys, and cq4+ (outlier channels exactly fp16 originals).
  • reference/gen_testvectors.py → reference/testvectors/*.npz + manifest.json — 9 golden vectors (SEED=20260622), cases d64_T128_G64 (2 full groups), d64_T70_G64 (1 full + partial g=6), d128_T100_G128 (partial g=100, D=128), each × CQ-8/CQ-4/CQ-4+. Each carries inputs + expected compressed payload (canonical bytes, contract §5) + expected decompressed output, with payload sha16 + rms. Generator self-asserts numpy==torch keys for every vector. Outlier isolation visible: CQ-4+ K_rms ≈ 0.10 vs CQ-4 ≈ 0.16–0.22; CQ-8 ≈ 0.028.

The GATE (spec §7 step 1): reproduce c17 from the clean reference, ±0.02.

  • Harness: analysis/c20_reference_repro.py — routes HellaSwag acc_norm (c17 protocol: n=250, max_ctx_tokens=160, choices prefixed " ") through the reference fake-quant path via forward hooks on .k_proj(is_key)/.v_proj. Keys use G=None (whole-sequence per-channel) to match c17; finite-G is Phase 2.
  • Command (per model): HF_HOME=/home/chaithu/lhs/.hf_cache /home/chaithu/lhs/.venv/bin/python \ analysis/c20_reference_repro.py --model <M> --device cuda --n-items 250 --tag <t>
  • Provenance: host spark-bd9f, device cuda (fp16), n=250, SEED via fixed item order; q05 ~10s/variant, q15 ~21s, q7 ~70s; 7B run logged to analysis/c20_q7.log (finished 2026-06-22T23:10:44Z, exit 0). Summaries: analysis/c20_{q05,q15,q7}_summary.json.

Result — every variant c17 actually measured reproduces bit-exactly (Δ=0.000) across all three models (c20 / c17 acc_norm):

variant q05 q15 q7
fp16 0.416/0.416 0.540/0.540 0.612/0.612
cq8 (int8) 0.420/0.420 0.528/0.528 0.600/0.600
pertoken 0.372/0.372 0.248/0.248 0.212/0.212
perchannel 0.436/0.436 0.536/0.536 0.604/0.604
kivi 0.408/0.408 0.540/0.540 0.600/0.600
outlier(K&V) 0.428/0.428 0.552/0.552 0.616/0.616

Max |Δ| over 18 comparisons = 0.000. The collapse signature reproduces (per-token INT4: 1.5B 0.248, 7B 0.212 vs FP16 0.540/0.612); per-channel / KIVI / outlier all recover ~FP16. Gate PASS, with margin to spare.

Spec-tier numbers (new this phase, not in c17):

  • CQ-4 == KIVI by construction, confirmed numerically: q05 0.408, q15 0.540, q7 0.600 — identical to the kivi row at every scale.
  • CQ-4+ (outlier on K only + per-token V — the shippable tier) vs the c17 outlier row (outlier on both K&V): q05 0.432 (Δ +0.004), q15 0.536 (Δ −0.016), q7 0.584 (Δ −0.032). Honest caveat: the 7B point estimate is −0.032 from the c17 both-sides row, exceeding ±0.02 — but this is a different recipe, not a failed reproduction (CQ-4+ drops V-side outlier protection to stay streaming-cheap). The gap is well inside the n=250 Wilson CI (CQ-4+ [0.522, 0.643] vs outlier [0.554, 0.674] overlap heavily) → not a statistically significant regression. Per stat-hygiene §8 we do not claim a per-model ordering here. Flagged for the headline n≥1000 run (Phase 3) and the finite-G Pareto (Phase 2).

Checklist — Phase 1:

  • reference/channelquant_ref.py CQ-8/4/4+ per HW_CONTRACT (numpy + torch)
  • Unit tests pass (7/7): round-trip + bit-exact fakequant↔pack parity
  • Golden vectors emitted (9) + manifest, generator self-checks numpy==torch
  • c17 reproduction GATE: bit-exact (Δ=0.000) on all measured variants × 3 models
  • CQ-4 == KIVI confirmed; CQ-4+ tier characterized (7B caveat logged honestly)
  • NOTES + CLAIMS updated; Phase 1 committed + pushed

Decision / next action: pause for go-ahead before Phase 2. Phase 2 = offline calibrator (real per-(layer,head) outlier masks, replacing the gen_testvectors amax stand-in) + finite-G group-size accuracy/bits Pareto (analysis/group_size_sweep.py). Open item to settle there/Phase 3: CQ-4+ vs CQ-4 at headline n on 7B (is K-only outlier protection sufficient, or does 7B want the V-side lane back?).

2026-06-22 — Phase 2: outlier calibrator + group-size Pareto → knee G=128

Two deliverables: the static outlier-mask ROM (real models) and the finite-G accuracy/bits Pareto that picks the shipped group size.

Calibrator — analysis/outlier_calibration.py

Emits the static per-(layer, KV-head) top-k key-channel mask (the c19 consensus statistic — mean_t |k[:,c]| summed over a calibration set — turned into ROM content). Deterministic: contexts in dataset order, n_calib=128, k=2.

  • Command: … outlier_calibration.py --model <M> --device cuda --tag <t> --topk 2
  • Provenance: host spark-bd9f, cuda/fp16, n_calib=128 HellaSwag contexts, ≤160 ctx tokens. Artifacts: reference/masks/{q05,q15,q7}_k2.npz (indices [L,n_kv,k] + D-wide bitmask, contract §4 format) + .json summaries.
  • Cross-validation: concentration (outlier mag / median) = 5.43 / 7.76 / 8.0 for 0.5B/1.5B/7B — independently reproduces c19's 5.4 / 7.8 / 8.0. The mask is the real ROM the CQ-4+ accuracy run consumes; it also seeds the eventual KVCE per-model ROM image. (The synthetic planted-channel mask in gen_testvectors.py stays — it tests codec mechanics, not a real model.)

Group-size Pareto — analysis/c22_group_size_sweep.py

HellaSwag acc_norm (c17 protocol, n=250) vs combined effective bits/value (contract §6) over G ∈ {32,64,128,256,full}, tiers CQ-4 and CQ-4+. CQ-4+ uses the calibrated static mask applied per (layer, head) — the real ROM, not on-the-fly top-k (this is strictly more faithful to the HW than c20's global top-k, and slightly better: 7B CQ-4+ full 0.592 vs c20's 0.584). Summaries: analysis/c22_{q05,q15,q7}_sweep.json; figure analysis/fig_group_size_pareto.png.

acc_norm / combined eff-bits (n=250):

model (D) fp16 CQ-4 G128 CQ-4+ G128 CQ-4 G32 CQ-4 full
0.5B (64) 0.416 0.408 / 4.19 0.428/4.38 0.416 0.408/4.13
1.5B (128) 0.540 0.540/4.13 0.528 / 4.22 0.520 0.540/4.06
7B (128) 0.612 0.600 / 4.13 0.592 / 4.22 0.604 0.600/4.06

Knee = G=128. Within every tier/model the acc_norm spread across G is inside overlapping n=250 Wilson CIs (CI half-width ~0.06 ≫ any G-to-G gap) — accuracy is statistically flat in G. So pick on bits + hardware: G=128 sits at the bits floor that still streams cheaply (cq4 4.13b; only ~0.06b above G=256/full, which need a bigger residual buffer and aren't worth it); below 128 (G=64/32) buys no measurable accuracy but costs +0.06–0.19b. G=128 is also KIVI-standard and matches the residual-buffer area already assumed in the KVCE revamp (G×D×FP16). The lone visible dip is CQ-4 G=32 at 1.5B (0.520), still inside fp16's CI — noise, not signal.

Headline acceptance (spec §7) at G=128, static calibrated mask — PASS:

  • CQ-4: 1.5B 0.540 (Δ 0.000), 7B 0.600 (Δ −0.012). ✓ ≤4.13b.
  • CQ-4+: 1.5B 0.528 (Δ −0.012), 7B 0.592 (Δ −0.020, exactly at the −0.02 bound). ✓ ≤4.22b ≤ 4.5b.

Honest finding — the "+" lane shows no measurable benefit at D=128. At D=64 (0.5B) CQ-4+ clearly helps (0.428 vs CQ-4 0.408). [CORRECTION, n=1000 headline: this D=64 benefit did NOT hold — the c23 headline shows CQ-4+ 0.415 vs CQ-4 0.421 (paired Δ−0.006, p=0.50), a slight loss. The screening "helps" was noise; CQ-4 is the default at every head dim. See CLAIMS C12′ / analysis/c23_q05_headline.json.] At D=128 (1.5B/7B) CQ-4+ point estimates sit at or marginally below plain CQ-4 (1.5B 0.528<0.540; 7B 0.592<0.600) — all CIs overlap, so no significant ordering either way, but there is no signal that isolating 2/128 channels in FP16 helps at scale, and it costs ~0.09b + the sidecar datapath. Working hypothesis: at D=128 the per-channel INT4 scale already localizes outlier error well enough that 2 FP16 channels are redundant; at D=64 the coarser per-channel granularity leaves more for the "+" to recover. Decision deferred to Phase 3 (n≥1000 on 1.5B/7B): if CQ-4+ does not beat CQ-4 at headline n on D=128, recommend CQ-4 as the D=128 default and keep CQ-4+ as a D=64 / small-head option — which would also let KVCE drop the outlier ROM+sidecar for the large-model deployments (an area win to note for the joint paper).

Checklist — Phase 2:

  • Calibrator emits static per-(layer,head) mask ROMs (3 models); concentration reproduces c19
  • Group-size Pareto n=250 × 3 models × {CQ-4, CQ-4+} × G∈{32,64,128,256,full}
  • Knee identified: G=128 (bits floor at flat accuracy; streaming-friendly)
  • Headline acceptance PASS at G=128 (CQ-4 and CQ-4+ ≥ fp16−0.02 at 1.5B & 7B, ≤4.5b)
  • Pareto figure analysis/fig_group_size_pareto.png
  • G=128 pinned in HW_CONTRACT.md §3/§6 (resolves KVCE "final G" open item — re-sync on next vendor)
  • NOTES + CLAIMS updated; Phase 2 committed + pushed

Decision / next action: pause for go-ahead before Phase 3. Phase 3 = generalization: one non-Qwen GQA model (Llama-3.x-8B or Mistral-7B) + ARC-Challenge, and the headline n≥1000 run on 1.5B/7B that settles CQ-4 vs CQ-4+ at D=128 (the open finding above). KVCE re-syncs the contract (G=128 pinned) at its convenience; the vendored bundle stays valid (G is a config, not a vector change).

2026-07-01 — Phase 3 START interrupted by a session crash; harnesses reconstructed

What happened. A Phase 3 working session (on a different host — chaithu@spark-bd9f, the DGX Spark GB10) crashed mid-run. It had authored the two Phase-3 harnesses and launched background jobs (headline n=1000 on 1.5B/7B; a Mistral-7B weight download) but committed nothing — the crash lost all of it. Confirmed on recovery: origin/main and the local clone are both at 661cfb8 (Phase 2); no c23_*/c24_* files, no Phase-3 result JSON, no Mistral mask. No Phase-3 numbers were ever saved — none are reconstructed or invented. (The KVCE handoff from Phase 1/2 was unaffected: the vendored bundle is present and KVCE has advanced to a passing golden-vector parity harness.)

Recovery host (this session). shadeform@brev-lvhh8zh1 — a fresh box: NVIDIA H100 PCIe 80 GB, but a bare environment (no torch/transformers, no HF cache, no models downloaded; system python3 has numpy 2.5.0 only). Not yet capable of running the GPU harnesses without setup (install torch/tf + datasets, download Qwen2-1.5B/7B + Mistral-7B, HF cache). The prior host's /home/chaithu/lhs/{.venv,.hf_cache} do not exist here.

Reconstructed (code only — faithful to the c20/c22 patterns, no data):

  • analysis/c23_headline.py — headline n≥1000 HellaSwag at the pinned G=128, variants fp16/cq4/cq4plus on 1.5B & 7B. Adds the piece screening lacked: scores cq4 and cq4+ on the same items and runs a paired test (McNemar exact + bootstrap 95% CI on the per-item acc delta) to settle C16, and stores per-item correctness vectors so the paired stat is auditable. Eff-bits helper reproduces the committed Phase-2 values (cq4 4.125, cq4+ 4.219) exactly; stat helpers unit-checked under numpy.
  • analysis/c24_generalization.py — spec §7 step 3. Non-Qwen GQA (Mistral-7B-v0.1) × ARC-Challenge (+ HellaSwag), variants fp16/pertoken (collapse control)/cq4/cq4plus at G=128. Gracefully skips cq4+ if no calibrated mask exists for --tag. ARC scored with the same token-length-normalized-NLL protocol as HellaSwag for internal comparability.
  • docs/paper/draft.md — the maintained paper draft the prior session was asked to keep current: all confirmed/screening results slotted in with CLAIMS IDs; every Phase-3 cell explicitly [PENDING] against the harness that fills it.

Both harnesses py_compile clean. They are ready to run but not yet run (env not provisioned on this host).

To execute Phase 3 (when a provisioned host is available):

# masks for Qwen already exist (reference/masks/q15_k2.npz, q7_k2.npz)
python analysis/c23_headline.py --model Qwen/Qwen2-1.5B --device cuda --tag q15 --n-items 1000
python analysis/c23_headline.py --model Qwen/Qwen2-7B   --device cuda --tag q7  --n-items 1000
# generalization: calibrate the non-Qwen mask first, then run arc (+ hellaswag)
python analysis/outlier_calibration.py --model mistralai/Mistral-7B-v0.1 --device cuda --tag mistral --topk 2
python analysis/c24_generalization.py  --model mistralai/Mistral-7B-v0.1 --task arc       --device cuda --tag mistral
python analysis/c24_generalization.py  --model mistralai/Mistral-7B-v0.1 --task hellaswag --device cuda --tag mistral

Decision / next action: commit the reconstructed harnesses + paper draft (no data), then await go-ahead on whether to provision this H100 box (install torch/tf + datasets, download ~30 GB of weights) and run Phase 3 here, vs. running on a host that already has the stack. Phase-3 result numbers, NOTES tables, and CLAIMS promotions to confirmed land only after a real run.

2026-07-01 — Phase 3 RESULTS: headline n=1000 + generalization (executed)

Provisioned the recovery host and ran the reconstructed harnesses. New standing environment (differs from the DGX Spark of Phases 0–2): host shadeform@brev-lvhh8zh1, NVIDIA H100 PCIe 80 GB, venv /ephemeral/cqvenv (torch 2.11.0+cu128, transformers 5.12.1, datasets 5.0.0, Python 3.12.13), HF_HOME=/ephemeral/hf_cache. Seeds: fixed dataset item order; paired bootstrap seed 20260622. Logs: /ephemeral/c23_run.log, /ephemeral/c24_run.log.

Headline — HellaSwag acc_norm, n=1000, G=128 (analysis/c23_{q15,q7}_headline.json)

Command: c23_headline.py --model <M> --device cuda --tag <t> --n-items 1000 (q15 ~4.5 min, q7 ~4.5 min + 15 GB download).

model (D) fp16 CQ-4 (Δ) CQ-4+ (Δ) paired CQ-4+ − CQ-4
1.5B (128) 0.522 0.505 (−0.017) 0.517 (−0.005) +0.012 CI[−0.001,0.025] McNemar p=0.088
7B (128) 0.614 0.615 (+0.001) 0.613 (−0.002) −0.002 CI[−0.017,0.013] McNemar p=0.897
  • Acceptance gate (spec §7) — PASS, now at n=1000. CQ-4+ ≥ FP16−0.02 at both 1.5B (Δ−0.005) and 7B (Δ−0.002), ≤4.22 eff-bits. CQ-4 also passes (1.5B −0.017, 7B +0.001). Promotes C12/C14-headline to confirmed; the Phase-1 7B CQ-4+ caveat (−0.032 vs a different both-sides recipe at n=250) is resolved — at n=1000 CQ-4+ 7B = 0.613 vs fp16 0.614.
  • C16 SETTLED (the whole point of the paired test). At D=128 the CQ-4+ vs CQ-4 paired delta's 95% CI includes 0 at both scales (7B dead flat p=0.90; 1.5B a weak non-significant lean toward CQ-4+, p=0.088 — which, if anything, softens the screening hint that "+" hurts). Discordant pairs: 1.5B 15/27, 7B 31/29. Verdict: no significant CQ-4+ vs CQ-4 difference at D=128 → CQ-4 is the sound D=128 default; keep "+" for D=64. Fed to KVCE (contract v0.2 §4.1: lane optional at D=128). Note the n=1000 point estimates flipped vs the n=250 screening (screening had CQ-4 ≥ CQ-4+; headline has them tied/CQ-4+-leaning) — exactly why headline n matters and screening orderings were never claimed.

Generalization (spec §7 step 3) — non-Qwen GQA + non-HellaSwag task

Calibrated the Mistral mask (reference/masks/mistral_k2.npz; 32 L, 8 KV heads, D=128, k=2; concentration median 7.62 [p10 5.94, p90 10.48] — reproduces the c19 outlier-concentration phenomenon on a non-Qwen architecture, extending C15). Then c24_generalization.py --model mistralai/Mistral-7B-v0.1 --task {arc,hellaswag} --tag mistral --n-items 1000 (analysis/c24_mistral_{arc,hellaswag}.json).

Mistral-7B acc_norm, n=1000, G=128:

task fp16 pertoken (Δ) CQ-4 (Δ) CQ-4+ (Δ)
HellaSwag 0.634 0.613 (−0.021) 0.636 (+0.002) 0.633 (−0.001)
ARC-Challenge 0.522 0.500 (−0.022) 0.496 (−0.026) 0.503 (−0.019)
  • The recover / near-lossless property GENERALIZES (the headline claim). CQ-4+ passes the FP16−0.02 gate on both new cells (HellaSwag Δ−0.001, ARC Δ−0.019), i.e. on a new model and a new task. CQ-4 is lossless on Mistral×HellaSwag (+0.002) and −0.026 on ARC (just past the gate; CQ-4+ recovers it to −0.019). Tier orderings on Mistral are all within overlapping CIs — no significant ordering claimed.
  • HONEST FINDING — the collapse is model-dependent, NOT universal. On Qwen2 naive per-token INT4 craters (1.5B/7B Δ≈−0.29/−0.40 vs FP16). On Mistral-7B it does not collapse — only −0.021 (HellaSwag) / −0.022 (ARC), staying well above chance. And this is despite Mistral's outlier concentration (7.62) being as high as Qwen2-7B's (8.0): channel-magnitude concentration alone does not predict the accuracy collapse. So we cannot market "naive INT4 collapses at scale" as a universal law — it is a (dramatic) Qwen2 phenomenon. What is robust and general: CQ-4/CQ-4+ are a safe near-lossless default everywhere (they prevent the collapse where it happens and never regress where it doesn't). This reframes the paper's value prop honestly (see CLAIMS C18/C19, draft §4.5).
  • Caveat: ARC absolute numbers (FP16 0.522) sit below published lm-eval Mistral ARC-Challenge (~0.60) because our protocol differs (token-length-normalized NLL, zero-shot "Question:…\nAnswer:", ≤256-ctx-token subset, n=1000) — valid for the relative collapse-vs-recover contrast, not comparable as an absolute ARC score.

Checklist — Phase 3:

  • Reconstructed harnesses executed on a provisioned H100 (env recorded above)
  • Headline n=1000 on 1.5B & 7B: acceptance PASS; C12/C14-headline → confirmed
  • C16 settled via paired McNemar/bootstrap: no sig. CQ-4+ vs CQ-4 at D=128
  • Generalization: recover generalizes (Mistral + ARC); collapse shown model-dependent (honest limitation), concentration reproduced on non-Qwen
  • KVCE contract v0.2 pinned + pushed (G/ROM/fp32/EPS) — orthogonal cross-lane
  • (next) fold into paper draft §4; optional generalization figure; CLAIMS promotions

Decision / next action: write CLAIMS promotions + paper draft §4/§4.5, commit + push Phase 3, then pause — Phase 4 is the method-paper write pass. No contract /vector churn from Phase 3 (accuracy-only), so KVCE stays unblocked on v0.2.

2026-07-01 — Phase 4: method paper (LaTeX→PDF) + striking figures + GIFs

Same H100 host/env as the Phase-3 entry. Built the paper deliverable and its figures; all figures read numbers from committed JSON/npz (nothing hardcoded).

New artifact — the key-channel spectrum (analysis/c25_capture_key_spectra.py → analysis/c25_key_spectra_{q7,mistral}.npz, [L,n_kv,D] mean|k|): c19 stored only aggregates, so the figures needed the full spectrum. Same statistic as the calibrator; argmax channels match the shipped ROM. This capture surfaced a refinement of C18 worth recording: median per-head concentration is ~identical across the two models (top-2/median 6.0 Qwen vs 6.1 Mistral; ~9% of key energy in 2 channels both), so a median statistic cannot distinguish the collapsing model from the robust one. The difference is entirely in the tail — Qwen has extreme heads (top-2 up to 247× median, 35% of key energy in 2 channels) that Mistral lacks (max 16×, 18%). Per-token scaling fails on those extreme heads; per-channel scaling is immune to any concentration. C18 updated accordingly (median-vs-tail, mechanistic) — this strengthens the honesty finding, doesn't walk it back.

Figures (analysis/c26_paper_figs.py → docs/paper/figs/*.{pdf,png}, vector for LaTeX + PNG for preview): fig1 collapse-cliff (c20/c23), fig2 outlier spectra Qwen vs Mistral (c25), fig3 accuracy–bits frontier (c20/c23), fig4 model-dependent collapse (c20/c24), fig5 concentration tail (c25 — the mechanistic backing for C18).

GIFs (analysis/c27_paper_gifs.py → docs/paper/figs/gif_{collapse,outliers}.gif, ~0.26 MB each): animated reveals of committed data for the repo/README/talks (a PDF can't animate) — the naive-INT4 "drop" and a per-layer outlier sweep. No fabricated interpolation; endpoints are the measured values.

Paper (docs/paper/channelquant.tex → channelquant.pdf, 4 pp): academic two-column, compiled with tectonic 0.16.9 (self-contained; cache /ephemeral/tectonic_cache). Sections: intro (fig1), background (KIVI/KVQuant/ FP8/TurboQuant+ credit), method (tiers, streaming residual group, static ROM, fig2), setup, results (repro gate, headline Table 1 + fig3, G=128 Pareto, C16 paired test, generalization fig4+fig5), algorithm→silicon (the v0.2 pins), limitations, conclusion, refs. Special chars via math macros (not literal unicode) for a clean XeTeX build. Formatting verified page-by-page (PyMuPDF render): resolved an overfull equation (split to two aligned lines), a stray hyphen-space ("residual-group"), and ragged-right refs; residual warnings are cosmetic underfull hboxes (badness ~3k) that render fine. \documentclass uses article two-column; figure* spans fig2.

Checklist — Phase 4:

  • Spectrum capture artifact (c25) + C18 refined to median-vs-tail (mechanistic)
  • 5 paper figures (vector PDF + PNG), collision-free, numbers from artifacts
  • 2 GIFs (collapse reveal, outlier layer sweep) for repo/talks
  • Academic LaTeX paper compiled to a clean 4-page PDF (tectonic), no visible issues
  • (optional next) real author/affiliation block + a venue class; silicon companion numbers when KVCE lands them; promote C8/C9 out of provisional.

Decision / next action: commit + push Phase 4. ChannelQuant is feature-complete through the method paper; remaining work is the KVCE silicon companion (area/Fmax) and any venue-specific formatting when a target is chosen.

2026-07-20 — README accuracy pass (lab-notebook principle)

Doc-only pass after the CQ-3-rot WHT value tier merged to main (commit 8192dde, Abhiram Bandi + Chaithu Talasila). The README top line still read Status: Phase 0 (scaffold) and the Build-status table listed Phases 1–4 as pending, while in reality Phases 0–4 have all landed (reference model + c17 GATE PASS + golden vectors; outlier calibrator + G=128 knee; n=1000 headline + Mistral/ARC generalization; the 4-page method paper) and the CQ-3-rot tier is merged. No lie left standing.

Changes (README.md only + this entry):

  • Top status line → "Phases 0–4 landed … CQ-3-rot WHT value tier merged", date 2026-07-20.
  • Headline blurb → added the flat-3-bit WHT rotation (CQ-3-rot) and ~4.8× compression, qualified near-lossless (~0.005 of FP16 on Qwen2-0.5B/1.5B + Llama-3.2-1B).
  • "Evidence" section → "will be re-run at n≥1000" corrected to "were re-run in Phase 3".
  • Build-status table → Phases 1–4 marked done with honest sub-status; added a CQ-3-rot row (bit-exact here; accuracy screening, in-repo re-derive pending).
  • Repo-layout comment → reference now lists CQ-3-rot.

Honesty guards kept: CQ-3-rot bit-exactness is stated as confirmed here (348,160/348,160 vs the KVE HW ref; 9/9 ref tests), but the ~0.005 accuracy delta is flagged screening (measured in the kv-cache-engine repo, not yet re-derived in this repo's harness — per CLAIMS.md). KIVI/KVQuant prior-art credit and the WHT authorship (Bandi + Talasila) left intact. No numbers fabricated; torch is absent on this ephemeral box so tests were not re-executed here — 9/9 relied on the merge commit + CLAIMS + the 9 test fns in the file.

2026-07-21 — WHT value-rotation provenance/citation pass (doc-only)

Correction/refinement to the CQ-3-rot credit above. The 2026-07-20 entry recorded the WHT authorship as "Bandi + Talasila." That is right for ChannelQuant's application (the flat-INT3 CQ-3-rot tier + the finding that value rotation is the safe half), but it under-cited the origin of the Walsh–Hadamard rotation primitive itself, which is not original to ChannelQuant: it comes from TurboQuant+ (themoddedcube/turboquant-plus), where the Hadamard/RHT rotation was introduced by Chaithu Talasila — README §2 "Randomized Hadamard Transform (RHT) replaces QR rotation", turboquant/wht_kernel.py, docs/08_wht_kernel.md (verified via git log on that file: authored by Chaithu Talasila, 2026-04-16). That in turn replaced the dense QR rotation of upstream TurboQuant (Zandieh et al., arXiv:2504.19874). Abhiram Bandi does not appear anywhere in turboquant-plus (no commit, no credit) — his credit is specifically for the ChannelQuant CQ-3-rot application, jointly with Talasila. Reconciled and cited consistently across:

  • README.md — value-rotation section (new "Provenance & credit" callout), headline blurb, Lineage section (repo link added; "retains the WHT value rotation" noted).
  • CLAIMS.md — C14 + C15 credit lines.
  • reference/channelquant_ref.py — WHT block comment + compress_values_wht docstring.
  • docs/paper/channelquant.tex — footnote at the TurboQuant+ mention + two new bibitems (turboquant, turboquantplus); docs/paper/draft.md, docs/paper/outline.md.
  • REVAMP_SPEC.md §1. No numbers touched; doc-only, provenance-only pass.