feat: v0.9.12 — Q6_K codegen + AOT validation (closes the Q6_K arc)

Builds on v0.9.11's foundation: emits a Q6_K dot kernel into AOT
binaries, wires is_q6k / lm_is_q6k dispatch through every projection /
lm_head emission site, threads --force-q6k through forge compile and
forge export-weights, and validates the codegen end-to-end via
--score-corpus on Llama-3.2-1B.

Result on the standard 4.9 KB factual corpus:
- AOT-Q6_K  --score-corpus     : ppl 4.2647, BPB 0.4000, 13 tok/s
- Interpreter Q8_0 bench-ppl   : ppl 4.2294, BPB 0.3976, 13 tok/s

Q6_K vs Q8_0 ppl: +0.8% — exactly the expected high-bit quantization
gap.  Better than AOT-Q4_K (4.57) by 7% as expected (Q6_K is 6.5
bits/elem vs Q4_K's 4.5).  Storage: Q6_K row is 23% smaller than Q8_0
(1680 vs 2176 bytes for k=2048).

Throughput is currently scalar-only (no NEON dot4 yet); ~3x slower
than the AOT-Q4_K NEON sdot path on the same model.  v0.9.13 will
add the NEON kernel + per-tensor dtype mixing for real Q4_K_M GGUFs.

339/339 workspace tests still pass.

Author: sauravpanda

CHANGELOG.md

@@ -2,6 +2,72 @@
 
 All notable changes to ForgeLLM are documented here.
 
+## [0.9.12] — 2026-05-01 — Q6_K codegen + AOT validation (closes the Q6_K arc)
+
+Builds on v0.9.11's foundation: emits a Q6_K dot kernel into AOT
+binaries, wires `is_q6k` / `lm_is_q6k` dispatch through every
+projection / lm_head emission site, threads `--force-q6k` through
+`forge compile` and `forge export-weights`, and validates the codegen
+end-to-end via `--score-corpus`.
+
+### Result — AOT Q6_K codegen mathematically validated
+
+Llama-3.2-1B-Q8_0 → `--force-q6k` → AOT-Q6_K, on 4.9 KB factual
+corpus:
+
+| path                                  | ppl    | BPB    | tok/s |
+| ------------------------------------- | ------ | ------ | ----- |
+| AOT-Q6_K  `--score-corpus`            | 4.2647 | 0.4000 | 13    |
+| Interpreter Q8_0 `bench-perplexity`   | 4.2294 | 0.3976 | 13    |
+
+Q6_K vs Q8_0 ppl: **+0.8%** — exactly the expected high-bit quant
+quality gap.  The +0.6% over AOT-Q4_K (4.5733) is also as expected
+(Q6_K is 6.5 bits/elem vs Q4_K's 4.5).  Storage: Q6_K row is 23%
+smaller than Q8_0 (1680 vs 2176 bytes for k=2048).
+
+Throughput is currently scalar-only (no NEON dot4 yet); ~3× slower
+than the AOT-Q4_K NEON sdot path on the same model.  v0.9.13 will
+add the NEON Q6_K kernel.
+
+### Added
+
+- **`emit_q6_k_kernel`** in codegen-cpu — emits scalar
+  `dot_q6_k_q8_0` into the AOT crate.  Reuses `f16_bits_to_f32` and
+  `quantize_to_q8_0_blocks_into` shared with the Q8_0/Q4_0/Q4_K kernel
+  families when any of them are also active.
+- **`emit_specialized_q6_k_matmul_functions`** — generates
+  `matmul_vec_q6_k_KxN` per (in, out) projection size.  Parallelizes
+  over output rows when total weight bytes > 1 MB.
+- **`is_q6k` / `lm_is_q6k` dispatch** in `emit_forward_function` and
+  `emit_prefill_function` (both QKV / o_proj / gate-up / down_proj /
+  lm_head sites + the `proj_type` / `lm_head_type` selection).
+- **Q6_K paths in `forgellm-codegen-cpu/src/project.rs`** — row-byte
+  helpers, weight-loading branch, weight-slicing for both regular and
+  `--embed-weights` builds.
+- **`forge compile --force-q6k`** and **`forge export-weights
+  --force-q6k`** — mirrors the `--force-q4k` flag.  Mutually exclusive
+  with `--force-q4k`.
+
+### Validated
+
+- All 339 workspace tests still pass; codegen tests still build
+  syntactically valid Rust for the Q6_K-containing emission.
+- Greedy continuation on Llama-3.2-1B-Q6_K produces coherent text
+  ("The capital of France is Paris. The capital of Germany is Berlin.
+  The capital of Italy is Rome.").
+- Per-chunk perplexity matches expected Q6_K quality: +0.8% over Q8_0,
+  and notably better than Q4_K (4.26 vs 4.57).
+
+### Deferred to v0.9.13
+
+- **NEON Q6_K dot kernel** — current scalar path runs at ~13 tok/s on
+  Llama-1B; AOT-Q4_K with NEON sdot runs at ~40 tok/s.  Adding
+  `dot4_q6_k_q8_0` with sdot ILP across 4 output rows should close
+  most of that gap.
+- **Per-tensor dtype mixing** — required to load real Q4_K_M GGUFs
+  (which mix Q4_K + Q6_K + F32 per tensor) without `--force-*`.
+  Currently codegen requires uniform projection dtype.
+
 ## [0.9.11] — 2026-05-01 — Native Q6_K quantizer + dot kernel (foundation)
 
 Q4_K_M GGUFs ship `attn_v` and `ffn_down` as Q6_K, which historically

Cargo.lock

@@ -13,7 +13,7 @@ members = [
 resolver = "2"
 
 [workspace.package]
-version = "0.9.11"
+version = "0.9.12"
 edition = "2021"
 license = "MIT"
 repository = "https://github.com/sauravpanda/forge-llm"
@@ -24,13 +24,13 @@ readme = "README.md"
 
 [workspace.dependencies]
 # Internal crates
-forgellm-frontend = { version = "0.9.11", path = "crates/forgellm-frontend" }
-forgellm-optimizer = { version = "0.9.11", path = "crates/forgellm-optimizer" }
-forgellm-codegen-cpu = { version = "0.9.11", path = "crates/forgellm-codegen-cpu" }
-forgellm-codegen-wasm = { version = "0.9.11", path = "crates/forgellm-codegen-wasm" }
-forgellm-codegen-gpu = { version = "0.9.11", path = "crates/forgellm-codegen-gpu" }
-forgellm-codegen-metal = { version = "0.9.11", path = "crates/forgellm-codegen-metal" }
-forgellm-runtime = { version = "0.9.11", path = "crates/forgellm-runtime" }
+forgellm-frontend = { version = "0.9.12", path = "crates/forgellm-frontend" }
+forgellm-optimizer = { version = "0.9.12", path = "crates/forgellm-optimizer" }
+forgellm-codegen-cpu = { version = "0.9.12", path = "crates/forgellm-codegen-cpu" }
+forgellm-codegen-wasm = { version = "0.9.12", path = "crates/forgellm-codegen-wasm" }
+forgellm-codegen-gpu = { version = "0.9.12", path = "crates/forgellm-codegen-gpu" }
+forgellm-codegen-metal = { version = "0.9.12", path = "crates/forgellm-codegen-metal" }
+forgellm-runtime = { version = "0.9.12", path = "crates/forgellm-runtime" }
 
 # Serialization
 serde = { version = "1", features = ["derive"] }

Cargo.toml

@@ -78,6 +78,12 @@ enum Commands {
         /// source GGUFs without needing a Q4_K_M file on disk.
         #[arg(long)]
         force_q4k: bool,
+
+        /// Force projection weights to be re-quantized to Q6_K (v0.9.12+).
+        /// Same purpose as --force-q4k but routes through the native Q6_K
+        /// kernel.  Mutually exclusive with --force-q4k.
+        #[arg(long)]
+        force_q6k: bool,
     },
 
     /// Export model weights as a flat binary file for AOT binaries
@@ -94,6 +100,10 @@ enum Commands {
         /// `forge compile --force-q4k`).
         #[arg(long)]
         force_q4k: bool,
+
+        /// Force projection weights to be re-quantized to Q6_K.
+        #[arg(long)]
+        force_q6k: bool,
     },
 
     /// Export a model to ONNX format
@@ -313,21 +323,31 @@ fn main() -> Result<()> {
             cross_target,
             lora,
             force_q4k,
-        } => cmd_compile(CompileArgs {
-            model_path: &model,
-            target: &target,
-            output_path: &output,
-            run,
-            prompt: prompt.as_deref(),
-            tokenizer_opt: &tokenizer,
-            embed_weights,
-            cross_target: cross_target.as_deref(),
-            lora_path: lora.as_deref(),
-            force_q4k,
-        })?,
+            force_q6k,
+        } => {
+            if force_q4k && force_q6k {
+                bail!("--force-q4k and --force-q6k are mutually exclusive");
+            }
+            cmd_compile(CompileArgs {
+                model_path: &model,
+                target: &target,
+                output_path: &output,
+                run,
+                prompt: prompt.as_deref(),
+                tokenizer_opt: &tokenizer,
+                embed_weights,
+                cross_target: cross_target.as_deref(),
+                lora_path: lora.as_deref(),
+                force_q4k,
+                force_q6k,
+            })?
+        }
 
-        Commands::ExportWeights { model, output, force_q4k } => {
-            cmd_export_weights_impl(&model, &output, None, force_q4k)?
+        Commands::ExportWeights { model, output, force_q4k, force_q6k } => {
+            if force_q4k && force_q6k {
+                bail!("--force-q4k and --force-q6k are mutually exclusive");
+            }
+            cmd_export_weights_impl(&model, &output, None, force_q4k, force_q6k)?
         }
 
         Commands::ExportOnnx { model, output } => {
@@ -771,6 +791,9 @@ struct CompileArgs<'a> {
     /// If true, override the source GGUF dtype to Q4_K and requant projection
     /// weights via `quantize_f32_to_q4_k`.
     force_q4k: bool,
+    /// If true, override the source GGUF dtype to Q6_K and requant projection
+    /// weights via `quantize_f32_to_q6_k`.
+    force_q6k: bool,
 }
 
 fn cmd_compile(args: CompileArgs<'_>) -> Result<()> {
@@ -785,6 +808,7 @@ fn cmd_compile(args: CompileArgs<'_>) -> Result<()> {
         cross_target,
         lora_path,
         force_q4k,
+        force_q6k,
     } = args;
     println!("Loading model config from {model_path}...");
     let mut config = load_model_config(model_path)?;
@@ -797,6 +821,15 @@ fn cmd_compile(args: CompileArgs<'_>) -> Result<()> {
         config.dtype = DType::Q4_K;
         config.lm_head_dtype = Some(DType::Q4_K);
     }
+    if force_q6k {
+        use forgellm_frontend::ir::DType;
+        println!(
+            "--force-q6k: overriding projection dtype {:?} → Q6_K (re-quantizing via in-tree quantizer)",
+            config.dtype
+        );
+        config.dtype = DType::Q6_K;
+        config.lm_head_dtype = Some(DType::Q6_K);
+    }
     config
         .validate()
         .map_err(|e| anyhow::anyhow!("invalid model config: {e}"))?;
@@ -818,7 +851,7 @@ fn cmd_compile(args: CompileArgs<'_>) -> Result<()> {
     if embed_weights {
         println!("Exporting weights for embedding...");
         let weights_path = output_dir.join("weights.bin");
-        cmd_export_weights_impl(model_path, &weights_path.to_string_lossy(), lora_path, force_q4k)?;
+        cmd_export_weights_impl(model_path, &weights_path.to_string_lossy(), lora_path, force_q4k, force_q6k)?;
 
         println!("Copying tokenizer for embedding...");
         let tokenizer_src = resolve_tokenizer(tokenizer_opt, model_path)?;
@@ -934,7 +967,7 @@ fn cmd_compile(args: CompileArgs<'_>) -> Result<()> {
         let weights_path = output_dir.join("weights.bin");
         let weights_path_str = weights_path.to_string_lossy().to_string();
         println!("[1/4] Exporting weights...");
-        cmd_export_weights_impl(model_path, &weights_path_str, lora_path, force_q4k)?;
+        cmd_export_weights_impl(model_path, &weights_path_str, lora_path, force_q4k, force_q6k)?;
 
         println!("[2/4] Resolving tokenizer...");
         let tokenizer_src = resolve_tokenizer(tokenizer_opt, model_path)?;
@@ -2343,14 +2376,15 @@ fn cmd_models(dir: &str) -> Result<()> {
 }
 
 fn cmd_export_weights(model_path: &str, output_path: &str) -> Result<()> {
-    cmd_export_weights_impl(model_path, output_path, None, false)
+    cmd_export_weights_impl(model_path, output_path, None, false, false)
 }
 
 fn cmd_export_weights_impl(
     model_path: &str,
     output_path: &str,
     lora_path: Option<&str>,
     force_q4k: bool,
+    force_q6k: bool,
 ) -> Result<()> {
     use forgellm_frontend::ir::DType;
     use forgellm_frontend::weight_loader::{load_from_file_mixed_with_target, WeightData};
@@ -2365,16 +2399,26 @@ fn cmd_export_weights_impl(
         config.dtype = DType::Q4_K;
         config.lm_head_dtype = Some(DType::Q4_K);
     }
+    if force_q6k {
+        eprintln!(
+            "--force-q6k: overriding projection dtype {:?} → Q6_K",
+            config.dtype
+        );
+        config.dtype = DType::Q6_K;
+        config.lm_head_dtype = Some(DType::Q6_K);
+    }
 
     let is_q8 = config.dtype == DType::Q8_0;
     let is_q4 = config.dtype == DType::Q4_0;
     let is_q4k = config.dtype == DType::Q4_K;
+    let is_q6k = config.dtype == DType::Q6_K;
 
-    if is_q8 || is_q4 || is_q4k {
+    if is_q8 || is_q4 || is_q4k || is_q6k {
         let quant_label = match config.dtype {
             DType::Q8_0 => "Q8_0",
             DType::Q4_0 => "Q4_0",
             DType::Q4_K => "Q4_K",
+            DType::Q6_K => "Q6_K",
             _ => "?",
         };
         if lora_path.is_some() {