[core] Export CppInterOp's dispatch entry point from libCore

[vgvassilev]

This Pull request:

Lets a CppInterOp client such as cppjit use ROOT's interpreter instead of loading its own copy of CppInterOp and LLVM. This is a first step. Sharing the interpreter works but is not reliable yet, and this draft is meant to discuss what is missing.

Changes or fixes:

A client attaches to an interpreter with Cpp::LoadDispatchAPI(lib), which loads lib and looks up CppGetProcAddress in it. In ROOT that function lives in libCling. TROOT loads libCling with RTLD_LOCAL and only when the interpreter starts, so a client running next to ROOT cannot find it. The client then loads a second interpreter, which costs about 128 MB and puts two LLVMs in one process.

With this change libCore exports CppGetProcAddress. It starts the interpreter if needed and forwards the call to libCling through the handle TROOT already holds. A client can pass $ROOTSYS/lib/libCore.so to Cpp::LoadDispatchAPI(). libCling is still loaded once and RTLD_LOCAL, and PyROOT is not affected. The new test TClingDispatch loads libCore the way a client would, declares a variable through the dispatched Process() and reads it back through gInterpreter. It fails without the change.

I tried this with upstream cppjit (3df629f) on Linux x86_64 and LLVM 22. I patched cppjit to load libCore when ROOTSYS is set and to skip C++ destructors while Python shuts down. Without import ROOT, cppjit used ROOT's interpreter: libCling was mapped once, no second CppInterOp was loaded, and ROOT classes such as TH1F and TFile worked.

Reliability

It is not reliable yet, and most of what is missing belongs in CppInterOp and cppjit rather than here.

The first problem is that the dispatch table is matched by name only. The released cppjit wheel (CppInterOp c8126249) and ROOT have functions with the same names but different signatures: LoadLibrary takes an extra argument in ROOT, and IsAllocator and GetEnumConstantValue return different types. Reusing ROOT from that wheel crashes inside LoadLibrary. Comparing LLVM versions does not catch this. CppInterOp should hash the signature of each table entry and have LoadDispatchAPI() check it. Doing it per entry keeps old clients working when functions are added, and only rejects a client when a signature it uses has changed.

cppjit also parses the CppInterOp headers it ships. When it reuses ROOT it has to use ROOT's headers from $ROOTSYS/etc/cppinterop, otherwise calls fail against ROOT's table.

Exit crashes when a TFile still holds a histogram. ROOT's EndOfProcessCleanups() then runs late, from a libCling static destructor. Skipping destructors during Python shutdown is not enough on its own, and neither is running the cleanup on its own. With both, as PyROOT does it from Python atexit, the case passed every run. Objects that ROOT deletes while the program runs are still a problem: f.Close(); del h deletes h twice. PyROOT avoids this with its memory regulator and by giving C++ the ownership of objects attached to a directory, and cppjit would need the same.

cppjit's own test suite shows 16 failures that only appear when it shares ROOT's interpreter. Three are tests marked as expected failures that now pass under cling. Four come from ROOT's global state: ROOT's names show up in cppjit.gbl, std is visible globally, and cppjit's -march=native is not applied. Eight are template instantiations with explicit arguments such as std::get<0>. One, test33_direct_base_methods, returns a wrong result. cppjit's ROOT branch may already handle some of these, and if so, reuse should require it.

This PR is the first step towards unifying cppjit and ROOT on a deeper level. Note that, until the table check, the headers and the exit handling are in place, cppjit should keep its bundled backend by default and reuse ROOT only when the user asks for it.

The rest of the required changes are needed elsewhere.

[github-actions]

Test Results

    24 files      24 suites   4d 1h 6m 20s ⏱️
 3 887 tests  3 886 ✅ 0 💤 1 ❌
83 201 runs  83 199 ✅ 0 💤 2 ❌

For more details on these failures, see this check.

Results for commit 7b0c6e3.

♻️ This comment has been updated with latest results.