Adapt to GPUCompiler v2 and keep v1 support

[christiangnrd]

Builds on #3308 but keeps support for GPUCompiler v1

[christiangnrd]

This is now targeting @vchuravy's branch in #3308 since it doesn't seem like CI from forks runs

[codecov]

Codecov Report

❌ Patch coverage is 12.50000% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.06%. Comparing base (b6d2853) to head (f4f888e).

Files with missing lines	Patch %	Lines
src/compiler.jl	0.00%	7 Missing ⚠️

Additional details and impacted files

@@                  Coverage Diff                   @@
##           vc/gpucompiler-2.0    #3354      +/-   ##
======================================================
- Coverage               71.11%   71.06%   -0.06%     
======================================================
  Files                      66       66              
  Lines                   21986    21991       +5     
======================================================
- Hits                    15636    15628       -8     
- Misses                   6350     6363      +13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[christiangnrd]

This passes locally with GPUCompiler v1, but with v2, I'm getting the following error (output with some extra @show statements sprinkled in):

metadata(addr) = LLVM.GlobalMetadataDict(LLVM.UnknownMember => LLVM.MDTuple(Ptr{LLVM.API.LLVMOpaqueMetadata}(0x000000002cbdbc60)))
inst = LLVM.AddrSpaceCastInst(%1 = addrspacecast ptr %"jl_global#12889" to ptr addrspace(10), !dbg !9)
paddr = LLVM.GlobalVariable("jl_global#12889")
addr = nothing
mul_kernel: Error During Test at /home/christian/.julia/dev/Enzyme/test/cuda.jl:18
  Got exception outside of a @test
  MethodError: no method matching get_base_and_offset(::Nothing; offsetAllowed::Bool, inttoptr::Bool)
  The function `get_base_and_offset` exists, but no method is defined for this combination of argument types.

  Closest candidates are:
    get_base_and_offset(::LLVM.Value; offsetAllowed, inttoptr, inst, addrcast)
     @ Enzyme ~/.julia/dev/Enzyme/src/absint.jl:280

  Stacktrace:
    [1] try_replace_constant_load!(inst::LLVM.AddrSpaceCastInst; check_mutability::Bool, do_replace::Bool)
      @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler/validation.jl:370
    [2] check_ir!(interp::Enzyme.Compiler.Interpreter.EnzymeInterpreter{Nothing}, job::GPUCompiler.CompilerJob, errors::Vector{Tuple{String, Vector{Base.StackTraces.StackFrame}, Any}}, imported::Set{String}, f::LLVM.Function, deletedfns::Vector{LLVM.Function}, mod::LLVM.Module)
      @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler/validation.jl:455
    [3] check_ir!(interp::Enzyme.Compiler.Interpreter.EnzymeInterpreter{Nothing}, job::GPUCompiler.CompilerJob, errors::Vector{Tuple{String, Vector{Base.StackTraces.StackFrame}, Any}}, mod::LLVM.Module)
      @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler/validation.jl:328
    [4] check_ir
      @ ~/.julia/dev/Enzyme/src/compiler/validation.jl:297 [inlined]
    [5] compile_unhooked(output::Symbol, job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget{GPUCompiler.PTXCompilerTarget}, Enzyme.Compiler.EnzymeCompilerParams{CUDACore.CUDACompilerParams}})
      @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:5500
    [6] (::GPUCompiler.var"#deferred_codegen#emit_llvm##0")(dyn_job::GPUCompiler.CompilerJob, job::GPUCompiler.CompilerJob)
      @ GPUCompiler ~/.julia/packages/GPUCompiler/vpUQG/src/driver.jl:236
    [7] (::GPUCompiler.var"#emit_llvm##1#emit_llvm##2"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDACore.CUDACompilerParams}, Dict{String, Ptr{Nothing}}, Dict{Any, Any}, LLVM.Module, GPUCompiler.var"#deferred_codegen#emit_llvm##0", GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget{GPUCompiler.NativeCompilerTarget}, Enzyme.Compiler.EnzymeCompilerParams{Enzyme.Compiler.PrimalCompilerParams}}})()
      @ GPUCompiler ~/.julia/packages/GPUCompiler/vpUQG/src/driver.jl:245
    [8] get!(default::GPUCompiler.var"#emit_llvm##1#emit_llvm##2"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDACore.CUDACompilerParams}, Dict{String, Ptr{Nothing}}, Dict{Any, Any}, LLVM.Module, GPUCompiler.var"#deferred_codegen#emit_llvm##0", GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget{GPUCompiler.NativeCompilerTarget}, Enzyme.Compiler.EnzymeCompilerParams{Enzyme.Compiler.PrimalCompilerParams}}}, h::Dict{GPUCompiler.CompilerJob, String}, key::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget{GPUCompiler.NativeCompilerTarget}, Enzyme.Compiler.EnzymeCompilerParams{Enzyme.Compiler.PrimalCompilerParams}})
      @ Base ./dict.jl:458
    [9] emit_llvm(job::GPUCompiler.CompilerJob)
      @ GPUCompiler ~/.julia/packages/GPUCompiler/vpUQG/src/driver.jl:242
   [10] compile_unhooked(output::Symbol, job::GPUCompiler.CompilerJob)
      @ GPUCompiler ~/.julia/packages/GPUCompiler/vpUQG/src/driver.jl:76
   [11] compile(target::Symbol, job::GPUCompiler.CompilerJob)
      @ GPUCompiler ~/.julia/packages/GPUCompiler/vpUQG/src/driver.jl:58
   [12] #invoke_frozen#592
      @ ~/.julia/packages/CUDACore/Ch9Pj/src/initialization.jl:30 [inlined]
   [13] invoke_frozen
      @ ~/.julia/packages/CUDACore/Ch9Pj/src/initialization.jl:26 [inlined]
   [14] #compile##0
      @ ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/compilation.jl:382 [inlined]
   [15] JuliaContext(f::CUDACore.var"#compile##0#compile##1"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDACore.CUDACompilerParams}}; kwargs::@Kwargs{})
      @ GPUCompiler ~/.julia/packages/GPUCompiler/vpUQG/src/driver.jl:34
   [16] JuliaContext(f::Function)
      @ GPUCompiler ~/.julia/packages/GPUCompiler/vpUQG/src/driver.jl:25
   [17] compile(job::GPUCompiler.CompilerJob)
      @ CUDACore ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/compilation.jl:381
   [18] compile_or_lookup(job::GPUCompiler.CompilerJob)
      @ CUDACore ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/compilation.jl:562
   [19] macro expansion
      @ ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/execution.jl:576 [inlined]
   [20] macro expansion
      @ ./lock.jl:376 [inlined]
   [21] cufunction(f::typeof(grad_mul_kernel), tt::Type{Tuple{CuDeviceVector{Float32, 1}, CuDeviceVector{Float32, 1}}}; kwargs::@Kwargs{})
      @ CUDACore ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/execution.jl:571
   [22] cufunction
      @ ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/execution.jl:568 [inlined]
   [23] kernel_compile
      @ ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/execution.jl:62 [inlined]
   [24] #kernel_compile#757
      @ ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/execution.jl:126 [inlined]
   [25] kernel_compile(call::CUDACore.KernelCall{CUDACore.LLVMBackend, typeof(grad_mul_kernel), Tuple{CuDeviceVector{Float32, 1}, CuDeviceVector{Float32, 1}}, @NamedTuple{f::typeof(grad_mul_kernel), arguments::Tuple{CuArray{Float32, 1, CUDACore.DeviceMemory}, CuArray{Float32, 1, CUDACore.DeviceMemory}}}})
      @ CUDACore ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/execution.jl:124
   [26] top-level scope
      @ ~/.julia/dev/Enzyme/test/cuda.jl:19
   [27] macro expansion
      @ ~/.julia/juliaup/julia-1.12.6+0.x64.linux.gnu/share/julia/stdlib/v1.12/Test/src/Test.jl:1777 [inlined]
   [28] macro expansion
      @ ~/.julia/dev/Enzyme/test/cuda.jl:24 [inlined]
   [29] macro expansion
      @ ~/.julia/packages/CUDACore/Ch9Pj/src/compiler/execution.jl:278 [inlined]
   [30] include(mapexpr::Function, mod::Module, _path::String)
      @ Base ./Base.jl:307
   [31] top-level scope
      @ none:1
   [32] eval(m::Module, e::Any)
      @ Core ./boot.jl:489
   [33] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:283
   [34] _start()
      @ Base ./client.jl:550
Test Summary: | Error  Total   Time
mul_kernel    |     1      1  21.5s
RNG of the outermost testset: Random.Xoshiro(0xfb98cc14e19c82ee, 0x7ce1b16d0d999b9b, 0xfba36ff53d665069, 0x8fb091028c21e9d5, 0x4a43073812564c9f)
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /home/christian/.julia/dev/Enzyme/test/cuda.jl:18

I think it may be because of JuliaGPU/GPUCompiler.jl#872 but I'm not sure how to resolve it.

[christiangnrd]

Okay I've confirmed that reverting JuliaGPU/GPUCompiler.jl#872 (after also reverting JuliaGPU/GPUCompiler.jl#874) makes the metal tests pass. Obviously that's not a fix but hopefully it helps with identifying one.

[wsmoses]

yeah so we do our own relocation enzyme-side so I think the correct resolution is a gpucompiler config option that opts out of their relocation [at least for now]

[wsmoses]

another thing we should do is also teach enzyme's relocation about the gv_to_value I added to gpucompiler earlier

[wsmoses]

it may also need maleadt/CompilerCaching.jl#2 to land

[luraess]

Are the AMDGPU test in Enzyme.jl disabled on purpose?

[vchuravy]

Are the AMDGPU test in Enzyme.jl disabled on purpose?

I think we did that at one point when all the runners were down.