Skip to content

Work around a Julia 1.12 codegen bug that crashes ParamsWithStats#1440

Merged
yebai merged 1 commit into
mainfrom
pws-1.12-gc-fix
Jul 23, 2026
Merged

Work around a Julia 1.12 codegen bug that crashes ParamsWithStats#1440
yebai merged 1 commit into
mainfrom
pws-1.12-gc-fix

Conversation

@yebai

@yebai yebai commented Jul 22, 2026

Copy link
Copy Markdown
Member

Under allocation load, ParamsWithStats model re-evaluation could segfault on Julia 1.12 — a code-generation bug triggered by the Union-typed value init!! returns when the accumulator tuple is chosen before the call. It surfaced as a flaky multichain-sampling crash (AbstractMCMC.jl#214).

Both re-evaluation methods now route init!! through a shared @noinline barrier, so that value is concretely typed and the miscompile can't occur. Behaviour is unchanged. The regression test asserts the offending Union never reappears in either method's inferred IR — a deterministic, version-independent guard.

The bug is specific to Julia 1.12 at -O2 and reproduces with no Turing stack at all:

Julia -O2 -O1
1.10 clean (250M iters)
1.11 clean (280M iters)
1.12 segfault in gc_mark_obj8, within seconds clean (67M iters)
Pure-Base MWE (segfaults on Julia 1.12 -O2)
# Mechanism: `step` produces a 2-tuple `(nt, vi)` from one of two @noinline callees
# selected by a Bool (a ternary), giving a Union of two concrete tuple types. On
# Julia 1.12 at -O2, the union phi materializes the tuple as a HEAP box, but the
# stores initializing field 1 (a NamedTuple holding three GC pointers) are
# eliminated because field 1 is dead (only field 2 is read). The partially
# initialized box is rooted in the gcframe and a further allocation (the union
# re-box of `vi`) is a GC safepoint. A GC landing there scans the box's
# uninitialized pointer fields (recycled pool garbage) -> SIGSEGV in gc_mark_obj8.
# On Julia 1.11/1.10 the same tuple stays a stack alloca (never GC-scanned); at -O1
# the field-1 stores survive.
struct LogP; v::Float64; end
struct LogL; v::Float64; end
struct GetRaw end
struct VNT{T}; nt::T; end
struct RawAcc{F,V}; f::F; flag::Bool; vnt::V; end
struct OAVI{A}; accs::A; end

const VNTIn = VNT{NamedTuple{(:s, :m),Tuple{VNT{NamedTuple{(:p,),Tuple{Vector{Float64}}}},Vector{Float64}}}}
@noinline mkvnt() = VNT((s=VNT((p=fill(0.5, 2),)), m=zeros(2)))::VNTIn

@noinline function runmodelA(n::Int)
    nt = (s=fill(1.0, n), m=view(fill(2.0, n), 1:2), x=fill(3.0, n))
    vi = OAVI((LogPrior=LogP(1.0), RawValues=RawAcc(GetRaw(), true, mkvnt())))
    return (nt, vi)
end
@noinline function runmodelB(n::Int)
    nt = (s=fill(1.0, n), m=view(fill(2.0, n), 1:2), x=fill(3.0, n))
    vi = OAVI((
        LogPrior=LogP(1.0), LogLikelihood=LogL(2.0), RawValues=RawAcc(GetRaw(), true, mkvnt())
    ))
    return (nt, vi)
end

get_raw(vi::OAVI) = vi.accs.RawValues.vnt
@noinline dens(v::VNTIn) = length(v.nt.m)
@noinline lpof(vi::OAVI) = vi.accs.LogPrior.v

@noinline function step(fl::Bool, n::Int)
    t = fl ? runmodelA(n) : runmodelB(n)
    _, vi = t                              # field 1 (the NamedTuple) is dead
    x = dens(get_raw(vi))
    return x + lpof(vi)
end

# Heap churn keeps recycled pool pages full of stale non-NULL garbage so the
# uninitialized pointer fields are poisoned rather than zero.
function churn(bag, k)
    for i in 1:k
        push!(bag, (i, i + 1))
        push!(bag, view(rand(4), 1:2))
        push!(bag, (true, true, true))
    end
    length(bag) > 200_000 && empty!(bag)
    return nothing
end

function main()
    bag = Any[]
    acc = 0.0
    it = 0
    while true
        it += 1
        acc += step(true, 4)
        acc += step(false, 4)
        it % 50 == 0 && churn(bag, 100)
    end
end
main()

Fix TuringLang/AbstractMCMC.jl#214

Both `ParamsWithStats` model-reevaluation methods selected their accumulator
tuple with an `if` before calling `init!!`, so the `(retval, varinfo)` tuple
`init!!` returns was `Union`-typed. On Julia 1.12 `-O2` this union is heap-boxed
with the unused `retval` pointer fields left uninitialized across a GC safepoint;
a GC landing there corrupts the heap and segfaults in `gc_mark_obj8`. It only
triggers under concurrent allocation load, which is why it surfaced as a flaky
multichain-sampling crash (AbstractMCMC.jl#214).

Route both methods through a shared `@noinline _pws_eval` barrier so the
accumulator tuple -- and hence the `init!!` result -- is concretely typed at
every call site. Behaviour is unchanged.

The regression test asserts the union never reappears in the inferred IR of
either method: a deterministic, Julia-version-independent guard (only the crash
is 1.12-specific).

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

DynamicPPL.jl documentation for PR #1440 is available at:
https://TuringLang.github.io/DynamicPPL.jl/previews/PR1440/

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks @ d36a949

Performance Ratio: gradient time divided by log-density time.

For very small models these ratios are noisy across runs and machines; raw primal and gradient timings are more reliable. The benchmarks are aimed at DynamicPPL developers and mainly catch obvious allocation or type-stability regressions. See benchmark notes for details.

===================================================================================================
                                               eval                       gradient                 
                                            ----------  -------------------------------------------
Model                        dim    linked      primal     FwdDiff    RvsDiff    Mooncake    Enzyme
---------------------------------------------------------------------------------------------------
Simple assume observe*         1     false     3.27 ns       14.93    1754.54       51.54     15.38
Simple assume observe*         1      true     3.25 ns       16.34    2364.41       51.96     14.92
Smorgasbord                  201     false     5.24 μs       80.10     135.97        8.39      9.54
Smorgasbord                  201      true     6.84 μs       79.73     141.40        7.19      6.49
Loop univariate 1k          1000     false     13.0 μs     1239.09     351.59       11.29      9.76
Loop univariate 1k          1000      true     18.2 μs     1404.45     260.40        8.07      6.96
Multivariate 1k             1000     false     22.2 μs      377.62      70.25        7.97      2.40
Multivariate 1k             1000      true     20.8 μs      460.30      76.46        8.59      2.67
Loop univariate 10k        10000     false    122.0 μs    19665.59     410.47       11.63      9.90
Loop univariate 10k        10000      true    169.0 μs    16742.26     307.49        8.26      7.28
Multivariate 10k           10000     false    175.0 μs     8205.46      90.84       11.14      2.13
Multivariate 10k           10000      true    176.0 μs     8587.67      88.76       11.06      2.23
Dynamic                       15     false     1.58 μs         err      48.60       18.30     13.39
Dynamic                       10      true      2.1 μs        2.16      64.33       22.14     24.20
Submodel*                      1     false     3.27 ns       15.64    2477.10       53.22     15.88
Submodel*                      1      true     3.27 ns       16.52    2583.66       53.58     15.95
LDA                           12      true     20.4 μs        0.59       2.84       34.67       err
===================================================================================================
Main @ 29778df
===================================================================================================
                                               eval                       gradient                 
                                            ----------  -------------------------------------------
Model                        dim    linked      primal     FwdDiff    RvsDiff    Mooncake    Enzyme
---------------------------------------------------------------------------------------------------
Simple assume observe*         1     false     4.63 ns       12.61    1526.57       40.95     12.33
Simple assume observe*         1      true     4.63 ns       12.56    1538.92       39.63     12.50
Smorgasbord                  201     false     5.97 μs       70.78     134.02        6.95      9.63
Smorgasbord                  201      true     7.57 μs       76.30     148.02        6.32      6.90
Loop univariate 1k          1000     false     17.5 μs      936.35     310.64        8.27      6.74
Loop univariate 1k          1000      true     19.3 μs     1405.89     285.63        7.45      6.11
Multivariate 1k             1000     false     22.0 μs      354.00      78.48        9.34      2.89
Multivariate 1k             1000      true     29.4 μs      276.63      58.88        6.83      2.97
Loop univariate 10k        10000     false    172.0 μs    11655.41     322.61        8.50      6.81
Loop univariate 10k        10000      true    187.0 μs    11475.57     307.81        7.63      6.28
Multivariate 10k           10000     false    194.0 μs     5610.65      89.82       11.38      2.27
Multivariate 10k           10000      true    194.0 μs     5239.57      89.61       11.33      2.26
Dynamic                       15     false      1.4 μs         err      41.89       15.01     11.00
Dynamic                       10      true     1.92 μs        1.82      58.32       16.66     19.43
Submodel*                      1     false     4.63 ns       12.54    1651.23       40.38     12.65
Submodel*                      1      true     4.63 ns       12.64    1751.63       40.03     12.38
LDA                           12      true     22.0 μs        0.50       2.02       35.03       err
===================================================================================================
Environment
Julia Version 1.11.9
Commit 53a02c0720c (2026-02-06 00:27 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × INTEL(R) XEON(R) PLATINUM 8573C
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, sapphirerapids)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.63%. Comparing base (29778df) to head (d36a949).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1440      +/-   ##
==========================================
- Coverage   81.64%   81.63%   -0.01%     
==========================================
  Files          50       50              
  Lines        3579     3578       -1     
==========================================
- Hits         2922     2921       -1     
  Misses        657      657              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sunxd3

sunxd3 commented Jul 23, 2026

Copy link
Copy Markdown
Member

looks correct to me — thanks

@yebai
yebai merged commit 8415f12 into main Jul 23, 2026
25 checks passed
@yebai
yebai deleted the pws-1.12-gc-fix branch July 23, 2026 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-chain sample shares one initial_params object across chains — segfaults under ForwardDiff

2 participants