Skip to content

fix: avoid runtime compilation in regression target encoding (ENG-1093) - #1338

Merged
jmkuebler merged 4 commits into
mainfrom
jonas/eng-1093-fix-nightly-container-accuracy-failure-missing-c-compiler
Oct 1, 2026
Merged

jmkuebler merged 4 commits into
mainfrom
jonas/eng-1093-fix-nightly-container-accuracy-failure-missing-c-compiler

Conversation

@jmkuebler

@jmkuebler jmkuebler commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Generated by Codex.

Batched regression on Torch 2.13 can invoke an eager Triton kernel and fail when no C compiler is installed. Flatten scalar targets before the v3/v3.5 column and ICL linear projections, then restore their batch shape, preserving estimator batching. This adds no startup probe or compiler detection; a fresh-process GPU test rejects compiler invocation.

Related: https://linear.app/priorlabs/issue/ENG-1093/fix-nightly-container-accuracy-failure-missing-c-compiler

Validation and limits
  • RTX PRO 6000, extracted failing-image Python 3.13.15 / Torch 2.13.0+cu130 / Triton 3.7.1 runtime; same airfoil fold, v3.5-fast checkpoint and eight estimators.
  • Current main (ba6ea311) fails with compiler discovery disabled. Patched source passes with a fresh Triton cache and batching enabled.
  • New compiler regression test fails on unpatched main and passes with the fix. Two output/input-gradient cases cover v3 and v3.5, both target stages, and a batched non-contiguous input. The compiler-specific test runs only on CUDA with Torch 2.13+, avoiding an unnecessary Triton import on the lowest-dependency job.
  • Existing v3/v3.5 architecture tests plus the initial projection checks: 195 passed. Final trimmed projection test file: 3 passed in 3.12 seconds on RTX/Torch 2.13. Pre-commit passes.
  • One same-node timing comparison: constructor 24 → 23 microseconds; fit 117 → 114 ms; first predict 995 → 347 ms; warm predict 61.37 → 61.27 ms. This is a smoke comparison, not a broad performance benchmark. The baseline had a compiler available; the fix did not.
  • Predictions are not bitwise identical after changing the projection kernel: mean absolute difference 0.00513, maximum 0.03156 on airfoil predictions spanning 104.9–138.8. Full accuracy-suite and Docker/HTTP validation remain outstanding.
  • Logs/scripts retained at gcp-ml:/home/jonas_priorlabs_ai/strable-probe-results/container-compiler-20261001.

@jmkuebler
jmkuebler marked this pull request as ready for review October 1, 2026 12:05
@jmkuebler
jmkuebler requested a review from simo-prior October 1, 2026 12:06
simo-prior
simo-prior previously approved these changes Oct 1, 2026
@jmkuebler
jmkuebler enabled auto-merge October 1, 2026 12:11
Comment thread tests/test_architectures/test_regression_target_projection.py

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reapproving after a push, as this is a PR from an internal author and was already approved.

@jmkuebler
jmkuebler disabled auto-merge October 1, 2026 13:03
@jmkuebler
jmkuebler added this pull request to the merge queue Oct 1, 2026
Merged via the queue into main with commit 6e4ee78 Oct 1, 2026
17 checks passed
@jmkuebler
jmkuebler deleted the jonas/eng-1093-fix-nightly-container-accuracy-failure-missing-c-compiler branch October 1, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants