build(before-code-freeze): Build against latest upstream container - #8918
Open
mc-nv wants to merge 19 commits into
Open
build(before-code-freeze): Build against latest upstream container#8918mc-nv wants to merge 19 commits into
mc-nv wants to merge 19 commits into
Conversation
The manylinux base container configures CPython with --disable-shared and its finalize.sh compresses every libpython*.a into static-libs-for-embedding-only.tar.xz before deleting the originals, so the image carries no linkable libpython at all. Since the pyenv build was dropped from change_default_python_version_rhel, nothing supplies one, and pybind11's FindPythonLibsNew silently falls back to the bare name "python3.12" -- which surfaces much later as "ld: cannot find -lpython3.12" when linking triton_python_backend_stub. Unpack that archive in the buildbase image. It lands in the interpreter's sysconfig LIBDIR, which is exactly where FindPythonLibsNew searches, so no CMake change is needed to find it. Linking libpython statically has two consequences handled here: - The stub now needs -Wl,--export-dynamic so Python C extension modules loaded at runtime resolve Py_* against the executable. pybind11 does not set ENABLE_EXPORTS for pybind11::embed, and a shared libpython did not need it. - Nothing has to ship beside the stub any more, so drop the copy of /usr/lib64/libpython<ver>* into the backend directory. That glob was fed by the removed pyenv step and now matches nothing, which would abort the generated build script under "set -e".
The embeddable CPython archive restored from the manylinux base image is built without -fPIC. Its x86-64 objects carry 15737 R_X86_64_32 and 3694 R_X86_64_32S relocations, which cannot appear in a position-independent executable, and gcc-toolset defaults to -pie -- so linking triton_python_backend_stub against it fails with "relocation R_X86_64_32S ... can not be used when making a PIE object". Link the stub with -no-pie there. The aarch64 archive uses PC-relative ADR_PREL_PG_HI21/ADD_ABS_LO12_NC addressing with no GOT indirection, which links into a PIE unchanged, so sbsa keeps the hardening. Relocation counts were read with readelf from the libpython3.12.a shipped in static-libs-for-embedding-only.tar.xz of cuda:13.4-devel-manylinux--26.08, for both architectures. Note this gives up ASLR on the stub executable for x86-64. Restoring a shared libpython instead would avoid both this and -Wl,--export-dynamic.
This reverts 2afa6bd. The -no-pie was added on the assumption that gcc-toolset-13 defaults to PIE, as RHEL's system gcc does. It does not: an x86-64 stub links successfully against the non-PIC libpython3.12.a with no -no-pie and no PIE-relocation errors, so the flag is a no-op. The relocation analysis behind 2afa6bd still holds -- the archive carries R_X86_64_32/32S and could not go into a PIE -- but the toolchain never asks for one, so there is nothing to opt out of and no ASLR trade-off to accept. -Wl,--export-dynamic is kept. It is confirmed necessary: a stub built without it links libpython3.12.a statically and exports zero PyExc_* symbols, leaving the 77 dlopen'ed lib-dynload modules -- which link no libpython -- with no way to resolve Py_* at import time.
The -Wl,--export-dynamic injected through CMAKE_EXE_LINKER_FLAGS is now set by python_backend itself, as an ENABLE_EXPORTS target property gated on libpython resolving to a static archive. Two things improve. The condition is the actual one -- static linkage -- rather than target_platform() == "rhel" standing in for it, so it stays correct if the RHEL flow returns to a shared libpython. And it is scoped to the stub instead of every executable in the backend build, and no longer overwrites the CMAKE_EXE_LINKER_FLAGS cache variable.
This reverts commit 7020028.
The hint existed because pybind would otherwise pick up an older interpreter present in the RHEL base container. It is no longer doing any work: - pybind11 sets Python_ADDITIONAL_VERSIONS "3.12;3.11;...;3.6" and prefers the newest entry, so an unconstrained search already lands on 3.12 (pybind11Tools.cmake:45-50). - The manylinux base container puts a single interpreter first on PATH. Every find_package site in the 26.08 RHEL build that carries no version constraint resolves to the same /opt/_internal 3.12.13 interpreter. - The non-RHEL path has never passed it and builds fine, so dropping it removes a platform divergence rather than creating one. - The pinned 3.12.3 had already drifted from the 3.12.13 the container ships. It passed only because find_package version matching means "at least"; a container on 3.12.2 would have failed the build on a number nobody was maintaining. With the argument gone python_cmake_args() returned an empty list, which is what the else branch of backend_cmake_args already produces, so both it and its dispatch entry go too. This removes the last reader of FLAGS.rhel_py_version, whose flag and version-map entry were already deleted -- build.py currently raises AttributeError for --target-platform=rhel --backend=python. qa/L0_backend_python/common.sh still passes PYBIND11_PYTHON_VERSION on purpose: it rebuilds the stub against a specific interpreter.
Every generated ONNX QA model fails to load: onnx runtime error 1: Load model from .../model.onnx failed: ValidateOpsetForDomain ... Opset 27 is under development and support for this is limited ... Current official support for domain ai.onnx is till opset 26. 320 models in a single L0_infer job, and the server then aborts with "failed to load all models", so every ONNX-backed test goes with it. ONNX_OPSET was 0, which does not mean "use what ONNX Runtime supports" as the original comment implied -- it means "omit opset_imports and let the onnx package stamp its newest". Per onnx's own VERSION_TABLE the pinned ONNX_VERSION=1.22.0 defaults to opset 27, while ONNX Runtime 1.28 accepts at most 26. The two pins drifted one release apart and nothing tied them together. Stamp 26 explicitly so the opset is decoupled from whichever onnx release the generator image happens to carry. The models are generated once and shared by every test job, so the value must not exceed what the oldest ONNX Runtime in the pipeline supports; the runtime names its maximum in the load error if this drifts again.
Nothing below 3.12 is supported any more, so the 3.11 conda environment and everything that existed only to serve it goes: the conda install and update, the env creation, conda-pack, the python_3_11 model, the teardown and the assertion that grepped for its version string. That assertion had to go with it. EXPECTED_VERSION_STRING was set only inside the 3.11 branch, so leaving the grep behind would have run `grep "" $SERVER_LOG`, matching every line and passing unconditionally -- a test that looks green while checking nothing. EXPECTED_VERSION_STRINGS (plural) was already dead and is removed too. The same-version custom environment path is unaffected: env/test.sh still covers it with create_conda_env "3.12". Its comment about Ubuntu 22.04/20.04 shipping 3.10/3.8 is corrected while here, since those mappings no longer describe anything we build.
Nothing below 3.12 is supported, so the PYTHON_ENV_VERSION == "8" arms are unreachable. Keep only the modern path -- numpy>=2 with torch 2.5.0 -- and drop the two `!= "8"` guards that skipped JAX, which dropped 3.8 support upstream. Leaves no PYTHON_ENV_VERSION references in this script.
setup_python_enviroment.sh had no reachable path left. It exits at the
top when PYTHON_ENV_VERSION is "12", which every config now sets after
the 3.11 environment and the blackwell 3.10 job were dropped. Its
remaining tail was also broken: it copies
python_backend/builddir/triton_python_backend_stub unconditionally, but
that directory was only ever produced by create_python_backend_stub()
inside the deleted 3.11 branch, so any non-12 value failed there.
The version assertion that followed the call goes with it. It grepped
python3 --version for 3.${PYTHON_ENV_VERSION}, which existed to confirm
the script had swapped the interpreter; with no script and the value
fixed at 12 it duplicated the "3.12" check a few lines above.
PYTHON_ENV_VERSION itself stays -- it still gates the env subtest.
This was referenced Aug 5, 2026
Contributor
Author
|
@greptileai review |
Greptile SummaryUpdates the build and QA configuration for the 26.08 upstream container.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Important Files Changed
Reviews (2): Last reviewed commit: "test: Remove the unreachable Python envi..." | Re-trigger Greptile |
mc-nv
changed the title
mc-nv
marked this pull request as ready for review
August 5, 2026 23:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does the PR do?
Build against the 26.08 upstream container. Main threads:
static-libs-for-embedding-only.tar.xzin the buildbase so the Pythonbackend stub can link libpython -- the manylinux base archives the static libs away.
PYBIND11_PYTHON_VERSIONpin, which had drifted (asked 3.12.3,container ships 3.12.13) and no longer does anything.
1.28 accepts at most 26, which made every generated ONNX model unloadable.
L0_backend_python.Checklist
<commit_type>: <Title>Commit Type:
Related PRs:
Where should the reviewer start?
build.py(restore_embeddable_python_lib_rhel,python_cmake_args) andqa/common/gen_qa_model_repository(the opset pin).Test plan:
Covered by the internal RHEL/manylinux pipeline for the 26.08 upstream bump.
Caveats:
Contains three debug commits (
extract single file, its revert,make extract verbose) and two non-conventional subjects; squash-on-merge keeps the mergedhistory clean.
The opset pin is correct for ONNX Runtime 1.28 (ceiling 26). If any job in the
pipeline consumes an ORT 1.27 image its ceiling is 25 and the pin must drop.
Background
Part of the 26.08 "build against latest upstream container" work: the base image
moved to
cuda:13.4-devel-manylinux--26.08, which changed the default GCCtoolset, the Python layout and the bundled OpenSSL.
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
- Resolves: TRI-1650
CI (internal): [#61271932](http://tritonserver.local/ci/pipelines/61271932)