[AI] OptimizationBase: fix Enzyme out-of-place grad/fg wiring

[ChrisRackauckas-Claude]

Please ignore until reviewed by @ChrisRackauckas. Opened as a draft.

Rebased onto master now that #1283 has merged (as e254f94). This PR is now standalone: one commit, 2 source lines plus a testset.

Two pre-existing bugs in the OptimizationFunction{false} method of OptimizationEnzymeExt.instantiate_function, noticed while working on #1282. Neither is related to that regression.

1. Gradient fallback gated on the wrong flag

if g == true && f.grad === nothing
    ...
elseif fg == true          # <-- should be `g == true`
    grad = (θ, p = p) -> f.grad(θ, p)

g = true, fg = false with a supplied f.grad falls through to grad = nothing. Every other extension — and the in-place Enzyme method 400 lines up — gates on g.

2. fg! returns the wrong buffer

res_fg = zeros(eltype(x), size(x))
function fg!(θ, p = p)
    ...
    Enzyme.Duplicated(θ, res_fg),
    ...
    return y, res          # <-- should be `res_fg`

res belongs to the grad branch. With g = false it is never assigned and fg! throws; with g = true it silently returns the untouched gradient buffer.

Observed, before vs after

Driver: OptimizationFunction{false}(rosen, AutoEnzyme(); grad = rosen_grad), evaluated at [0.5, 0.7] where the true gradient is [-91.0, 90.0].

case	before	after
g=true, fg=false, user grad	optf.grad === nothing	optf.grad(z) ≈ [-91.0, 90.0]
g=false, fg=true	UndefVarError: res not defined	y ≈ 20.5, G ≈ [-91.0, 90.0]
g=true, fg=true	G = [0.0, 0.0] (silently wrong)	G ≈ [-91.0, 90.0]

The third row is the dangerous one — a zero gradient with no error.

Note these paths need both ChainRulesCore and Enzyme loaded to be reached, since the extension trigger is OptimizationEnzymeExt = ["ChainRulesCore", "Enzyme"]; with only Enzyme loaded, AutoEnzyme falls through to the generic DI extension, which is presumably why this went unnoticed.

OptimizationBase 5.2.3 → 5.2.4.

Test plan

New "Enzyme out-of-place grad/fg wiring" testset in lib/OptimizationBase/test/AD/adtests.jl covering all three cases above. All three fail on the parent branch.

Run locally on Julia 1.11:

• OPTIMIZATION_TEST_GROUP=AD Pkg.test() for OptimizationBase — 781 pass, 0 fail (re-run after the rebase, against the new master base)
• Confirmed the new testset fails without the two source-line changes
• Runic clean
• CI green

🤖 Generated with Claude Code

https://claude.ai/code/session_014Uj4Pu1LKHYqLSTLfPYDPo