GLM aggregates segfault in faer blocked-GEMM on DuckDB v1.4.4 unittest harness (fixtures ≥ ~150 rows)

[sipemu]

Symptom

Any GLM aggregate whose IRLS inner loop runs a weighted-least-squares QR (poisson_fit_agg, negbinom_fit_agg, tweedie_fit_agg, … — anything reaching faer's ColPivQr) segfaults deterministically when:

1. the fit has ≥ ~150 observations (crash threshold bisected to between n=100 and n=150), AND
2. the query runs inside the DuckDB v1.4.4 sqllogictest harness (build/release/test/unittest).

It does not reproduce:

• in the DuckDB CLI shell with identical data (default threads or threads=1),
• on DuckDB v1.5.3 (shell or harness),
• calling fit_negbinomial / fit_poisson directly from Rust (including on 256KB-stack threads).

Inside the harness it crashes with threads=1 too, so it is not a worker-thread race.

Backtrace

#0  private_gemm_x86::pack_lhs ()                       ← SIGSEGV, si_addr = 0x0 (NULL deref)
#1  private_gemm_x86::millikernel_colmajor ()
#2  private_gemm_x86::gemm ()
#3  faer::linalg::matmul::triangular::mat_x_mat_into_lower_impl_unchecked ()
#6  faer::linalg::householder::upgrade_householder_factor ()
#7  faer::linalg::solvers::ColPivQr<T>::new_imp ()
#8  anofox_regression::solvers::negative_binomial::NegativeBinomialRegressor::solve_weighted_ls ()
#9  NegativeBinomialRegressor::fit_irls ()
#10 anofox_negbinomial_fit ()
#11 duckdb::NegBinomAggFinalize(...)
#12 duckdb::RowOperations::FinalizeStates(...)

Faulting instruction is a mulsd (%r14,%r12,1),%xmm0 inside the packing loop with a NULL base pointer.

Size threshold

faer's QR uses an unblocked path for small matrices and switches to blocked Householder (which calls the gemm microkernels) for larger ones — only the blocked path crashes. That's why every pre-existing GLM test stayed green: all fixtures were < 150 rows.

n rows	harness v1.4.4
20 / 50 / 100	✅ pass
150 / 200	💥 SIGSEGV (3/3 runs)

Ruled out

• My new aggregate C++ (structural clone of poisson_aggregate.cpp; poisson_fit_agg itself crashes identically on the same fixture).
• FFI struct layout mismatch (C header vs #[repr(C)] verified field-by-field).
• Stack exhaustion (fault address is NULL, not a guard page; direct Rust runs pass on 256KB stacks).
• Worker-thread placement (crashes with SET threads=1 in the harness; crashed on main thread in one gdb run).
• Harness client settings (enable_progress_bar, PRAGMA enable_verification in the shell don't reproduce it).

Open leads

• The harness require → LOAD path dlopens the loadable .duckdb_extension while the statically linked copy is also present in libduckdb.so — two copies of faer/its dispatch state in one process. The shell auto-loads only the static copy. Explicit LOAD in the shell did not reproduce, but the harness load order/path differs (local repository, gzipped artifact) and was not fully replicated.
• v1.4.4 vs v1.5.3 extension-ci-tools differ in extension symbol visibility/linking, which would change whether the two copies' dynamic symbols interpose.

Workaround in place

All GLM test fixtures are capped at 100 rows (commits on the #88–#92 branches reference this issue). Production usage is unaffected — the crash needs the v1.4.4 harness specifically.

Repro recipe

git submodule update --init duckdb extension-ci-tools posthog-telemetry
(cd duckdb && git fetch --depth 1 origin tag v1.4.4 && git checkout v1.4.4)
(cd extension-ci-tools && git fetch --depth 1 origin v1.4.4 && git checkout FETCH_HEAD)
GEN=ninja make release
# any GLM agg with a 200-row fixture:
./build/release/test/unittest "test/sql/regression/test_negbinom_basic.test"  # with range(0,200)

🤖 Generated with Claude Code

[sipemu]

Status after the LTS bump (v1.4.4 → v1.4.5, merged via #94 / GLM PRs):

The DuckDB version bump that fixed the Windows fmt/MSVC break does not address this segfault, and is unlikely to resolve it:

• This crash was already characterised as specific to the v1.4 unittest-harness linking path (two copies of faer — static in libduckdb.so + dynamic in the dlopen'd extension), and explicitly did not reproduce on v1.5.3.
• The LTS lane now builds against DuckDB v1.4.5 but still uses the v1.4-andium extension-ci-tools branch — i.e. the same v1.4 LTS extension symbol-visibility/linking family as v1.4.4. The suspected root cause is in that linking path, not in the DuckDB patch release.

So the 100-row fixture caps remain in place on all five GLM test files (test/sql/regression/test_{negbinom,binomial,tweedie,gamma,logistic}_basic.test) and should stay until this is verified fixed. The v1.5.x lane is unaffected, and production usage (CLI / non-harness) was never impacted.

Keeping this open. Definitive check: temporarily raise one GLM fixture to ≥150 rows and run build/release/test/unittest built against the v1.4.5 LTS lane — if it still SIGSEGVs in faer blocked-GEMM, the issue persists on v1.4.5 as expected.

[sipemu]

Confirmed still present on v1.4.5; now reproducible locally with exact backtrace

Built the extension locally against DuckDB v1.4.5 and ran a GLM fit above the threshold through build/release/test/unittest — it still SIGSEGVs, so the v1.4.5 LTS bump does not fix this and the 100-row fixture caps must stay.

Minimal local repro (v1.4.4 and v1.4.5):

-- as a .test under test/sql/, run via build/release/test/unittest
CREATE TABLE big_glm AS
SELECT CAST(ROUND(EXP(0.5 + 0.3*(i%10)) + ((i*7)%5)) AS DOUBLE) AS y, (i%10)::DOUBLE AS x
FROM range(0, 250) t(i);
SELECT (negbinom_fit_agg(y, [x])).n_observations FROM big_glm;  -- or poisson_fit_agg

The data matters: the (i%10) design from test_negbinom_basic triggers it; some other 200–350-row fixtures I tried did not.

Backtrace (exact match to the original report):

#0 private_gemm_x86::pack_lhs ()          ← SIGSEGV, NULL deref   (in libduckdb.so)
#1 private_gemm_x86::millikernel_colmajor ()
#2 private_gemm_x86::gemm ()
#3 faer::linalg::matmul::triangular::mat_x_mat_into_lower_impl_unchecked ()
#6 faer::linalg::householder::upgrade_householder_factor ()
#7 faer::linalg::solvers::ColPivQr<T>::new_imp ()
#8 anofox_regression::...::NegativeBinomialRegressor::solve_weighted_ls ()
#11 duckdb::NegBinomAggFinalize(...)

Findings that refine the original hypotheses:

• The crash is in the gemm crate (private_gemm_x86::pack_lhs, v0.18.2), reached via faer 0.23.2 ColPivQr in the GLM IRLS weighted-LS.
• It reproduces with faer compiled into libduckdb.so (a single static copy) — so the "two copies of faer via dlopen" theory isn't required; an environment-sensitive gemm blocked-path bug is sufficient.
• Not faer/rayon parallelism: RAYON_NUM_THREADS=1 still crashes.
• Still environment-specific (harness, v1.4.x): doesn't reproduce in the CLI shell or on v1.5.x, consistent with the original report.

No in-repo fix available: gemm 0.18.2 / faer 0.23.2 are already the latest versions compatible with anofox-regression's constraint (cargo update -p gemm/faer finds nothing newer). A real fix requires anofox-regression to move to a faer/gemm release that fixes pack_lhs (or the gemm crate to fix it). Keeping this open and the fixture caps in place until then.