Skip to content

[WIP][CI] Enable DCU C++20 toolchain baseline #79421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gouzil wants to merge 14 commits into
base: develop
Choose a base branch
from
Open

[WIP][CI] Enable DCU C++20 toolchain baseline #79421

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
6959f42
[CI] Enable DCU C++20 toolchain baseline
gouzil Jul 5, 2026
4e922db
[CI] let DCU docker image follow Dockerfile md5
gouzil Jul 8, 2026
c0adfdb
[CI] detect lib64 path when installing GCC
gouzil Jul 9, 2026
e9adce2
[CI] check libstdc++ path before GCC install
gouzil Jul 9, 2026
35e05b9
[CI] install DCU requirements from build context
gouzil Jul 9, 2026
ab2a559
[CI] restore DCU requirements download
gouzil Jul 9, 2026
7ccb519
更新 DTK 下载链接至新的源
gouzil Jul 9, 2026
dc37cc1
[CI] keep GCC runtimes isolated from system libraries
gouzil Jul 10, 2026
d9e3864
[CI] force remove stale DCU containers
gouzil Jul 13, 2026
709a5ac
[CI] clean up DCU containers after failures
gouzil Jul 13, 2026
451581e
[CI] simplify DCU container cleanup
gouzil Jul 13, 2026
d563602
[CI] run DCU cleanup commands directly
gouzil Jul 13, 2026
43b0636
[CI] keep DCU artifacts out of container layer
gouzil Jul 19, 2026
a665f2f
rollback Linux Dcu build clean
gouzil Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 11 additions & 10 deletions .github/workflows/_Linux-DCU.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ jobs:
GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }}
home_dir: ${{ github.workspace }}/../../../..
run: |
docker ps -aq --filter "name=^/${TASK}-" | xargs -r docker rm -f
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
echo "container_name=${container_name}" >> ${{ github.env }}
docker run --shm-size=128G \
Expand Down Expand Up @@ -155,8 +156,7 @@ jobs:
bash ${ci_scripts}/run_setup.sh bdist_wheel
EXCODE=$?
python -m pip install bce-python-sdk==0.8.74
rm -rf `find . -name "*.a"`
rm -rf `find . -name "*.o"`
find . -type f \( -name "*.a" -o -name "*.o" \) -delete
exit $EXCODE
'

Expand All @@ -169,14 +169,16 @@ jobs:
paddle_whl: paddlepaddle_dcu-0.0.0-cp310-cp310-linux_x86_64.whl
run: |
docker exec -t ${{ env.container_name }} /bin/bash -c '
set -e
if [ ! -f "${{ env.bos_file }}" ]; then
wget -q --no-proxy -O ${{ env.home_path }}/bos_new.tar.gz https://xly-devops.bj.bcebos.com/home/bos_new.tar.gz --no-check-certificate
mkdir ${{ env.home_path }}/bos
mkdir -p ${{ env.home_path }}/bos
tar xf ${{ env.home_path }}/bos_new.tar.gz -C ${{ env.home_path }}/bos
fi
apt install zstd -y
cd ..
tar --use-compress-program="pzstd -5" -cf build.tar.gz paddle
cd "${{ env.home_path }}"
trap "rm -f build.tar.gz ${{ env.paddle_whl }}" EXIT
tar --use-compress-program="pzstd -5" -cf build.tar.gz -C / paddle
echo "Uploading build.tar.gz"
python ${{ env.bos_file }} build.tar.gz paddle-github-action/PR/dcu/${{ env.PR_ID }}/${{ env.COMMIT_ID }}
echo "Uploading paddle_whl"
Expand Down Expand Up @@ -232,6 +234,7 @@ jobs:
GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }}
home_dir: ${{ github.workspace }}/../../../..
run: |
docker ps -aq --filter "name=^/${TASK}-" | xargs -r docker rm -f
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
echo "container_name=${container_name}" >> ${{ github.env }}
docker run --ulimit nofile=102400 --ulimit core=-1 --shm-size=32g -d -t --name ${container_name} \
Expand Down Expand Up @@ -301,8 +304,6 @@ jobs:
- name: Terminate and delete the container
if: always()
run: |
docker exec -t ${{ env.container_name }} /bin/bash -c '
rm -rf * .[^.]*
'
docker stop ${container_name}
docker rm ${container_name}
set -e
docker exec -t "${container_name}" /bin/bash -c 'rm -rf * .[^.]*'

@PaddlePaddle-bot PaddlePaddle-bot Jul 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 建议 test job 这里也有同样的清理顺序问题：docker exec 在 set -e 下失败时，后续的 docker rm -f 不会执行。

建议和 build job 保持一致，先容忍/记录 workspace 清理失败，再保证执行 docker rm -f "${container_name}"，避免失败测试留下同名容器影响后续 DCU 任务。

docker rm -f "${container_name}"

@risemeup1111 risemeup1111 Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

这里直接删容器会把 DCU test 的宿主机 workspace 留在原地：上面的 docker run 把 ${{ github.workspace }} bind 到 /paddle，而当前启动时只清理同名旧容器，不清理 workspace；Download build.tar.gz 也只执行 rm -rf ./*，不会删除 .git、.pytest_cache 等隐藏目录。这样一次失败/取消后的隐藏文件会跨 run 留在自托管 DCU runner 上，后续解包 build.tar.gz 时可能混入旧状态，导致测试结果不可复现。

请保留有上限的 workspace 清理，再强制删除容器，避免清理卡住但仍清掉 bind mount 内容。例如恢复成：

Suggested change
docker rm -f "${container_name}"
if [[ -n "${container_name:-}" ]]; then
timeout 10m docker exec -t "${container_name}" /bin/bash -c '
rm -rf * .[^.]*
' || true
docker rm -f "${container_name}"
fi

@risemeup1111 risemeup1111 Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

这版已经恢复了 bind mount workspace 的清理，但当前写法仍不满足“清理有上限且最终强制删除容器”的要求：set -e 之后先执行 docker exec ... rm -rf * .[^.]*，一旦这个清理命令失败或卡住，后面的 docker rm -f "${container_name}" 就不会执行。这样失败/取消路径仍可能留下同名 DCU 容器，并且 cleanup step 会被无上限的 workspace 删除阻塞。

请把 workspace 清理和强制删容器解耦：给 docker exec 加 timeout，记录清理状态，但无论清理是否失败都执行 docker rm -f。如果希望保留 workspace 清理失败状态，可以用类似下面的结构：

cleanup_status=0
timeout 10m docker exec -t "${container_name}" /bin/bash -c 'rm -rf * .[^.]*' || cleanup_status=$?
docker rm -f "${container_name}"
exit "${cleanup_status}"

@ShigureNyako ShigureNyako Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

补充 current head d563602 的实际 CI 证据：Linux-DCU / Build 的下载、编译和 BOS 上传步骤都已成功，Terminate and delete the container 从 17:46:16Z 开始后一直未完成；该 job 配置 timeout-minutes: 180，最终在约 180 分钟上限及取消宽限后于 20:23:47Z 被取消，Test 随后 skipped。

也就是说，当前无上限的 cleanup step 实际占满了剩余 job 时间，已经复现了这里描述的问题。仍需要给 workspace 清理加 timeout，并确保无论清理结果如何都会执行 docker rm -f。

@risemeup1111 risemeup1111 Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

确认，这个 CI 结果已经把风险从理论路径变成了实际复现：Linux-DCU / Build 的下载、编译和 BOS 上传步骤都已完成，job 最后停在 Terminate and delete the container，并因 3 小时上限失败，后续 Test 被跳过。

当前修复需要同时满足两点：给 workspace 清理设置有界 timeout，并且无论 docker exec 清理结果如何都执行 docker rm -f。可以按下面的结构处理，修复后请用新 head 的 Linux-DCU Build/Test 再确认 cleanup 不再耗尽 job timeout。

cleanup_status=0
timeout 10m docker exec -t "${container_name}" /bin/bash -c 'rm -rf * .[^.]*' || cleanup_status=$?
docker rm -f "${container_name}"
exit "${cleanup_status}"

@risemeup1111 risemeup1111 Jul 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

我复查了新 head 43b0636，这次只改了 build 产物打包和临时文件处理，Terminate and delete the container 这一段没有变化，所以你补充的 CI 结论仍然成立：cleanup 仍会占满 job 的剩余时间。

这里仍需要把 workspace 清理改成有界 timeout，并保证无论 docker exec 成功与否都执行 docker rm -f。建议继续按下面的结构修：

cleanup_status=0
timeout 10m docker exec -t "${container_name}" /bin/bash -c 'rm -rf * .[^.]*' || cleanup_status=$?
docker rm -f "${container_name}"
exit "${cleanup_status}"

@ShigureNyako ShigureNyako Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

复查了新 head a665f2f：Build teardown 回滚后不再删除完整 bind-mounted build workspace，这能避开前两次实际卡住的路径；但 cleanup P1 还没有闭环：

  • Build 现在恢复为顺序执行 docker stop、docker rm，仍不是“无论前一步结果如何都强制删除容器”；docker stop 失败时，默认 -e shell 会跳过后面的普通 docker rm。
  • Test teardown 完全未变，仍是无 timeout 的 docker exec ... rm -rf * .[^.]* 后再 docker rm -f。exec 卡住会继续耗尽 job timeout，exec 失败会因 set -e 跳过 force-remove。
  • /paddle 仍是宿主 workspace bind mount；如果 Test 被硬取消或 cleanup 未完成，下一次启动只有 rm -rf ./*，仍会漏掉 .git、.pytest_cache 等隐藏状态。

Build 可以直接保证执行 docker rm -f；Test 仍需使用有界 workspace cleanup，并无条件执行 docker rm -f，或者把完整（含 dotfiles）workspace 清理放到下一次解包前。修复后还需要新 head 的 Linux-DCU Build 和 Test 都完整通过。

@risemeup1111 risemeup1111 Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

你这里的判断是对的：Build 侧回滚后，前一次“完整 bind-mounted workspace 删除”造成的卡住路径已经收窄了，但这并没有把 cleanup 闭环。当前 head 里 Test 仍是无上限 docker exec ... rm -rf * .[^.]*，而且 set -e 会让后面的 docker rm -f 在 exec 失败或超时后直接跳过；/paddle 还是宿主 bind mount，隐藏状态也还会跨 run 残留。

因此这条 P1 仍需要继续修，不能只停在 Build 侧回滚。建议继续把 Test 改成“有界清理 + 无条件 force-remove”，或者把完整 workspace 清理前置到下一次解包之前：

cleanup_status=0
timeout 10m docker exec -t "${container_name}" /bin/bash -c 'rm -rf * .[^.]*' || cleanup_status=$?
docker rm -f "${container_name}"
exit "${cleanup_status}"

This comment was marked as outdated.

Sign in to view

@gouzil gouzil Jul 13, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ShigureNyako 喵子来看一下这里原来为啥要 rm -rf * .[^.]*, 直接把容器删了这不就一起带走了吗

@ShigureNyako ShigureNyako Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里的 /paddle 不是容器 writable layer，而是通过 -v ${{ github.workspace }}:/paddle 挂载的宿主目录；docker rm -f 只删除容器本身，不会删除宿主机 ${{ github.workspace }} 中的文件。

因此原来的 rm -rf * .[^.]* 是在 bind mount 内同时清理普通文件和隐藏文件。现在下载前的 rm -rf ./* 只匹配非隐藏项，.git、.pytest_cache 等仍会跨 run 留下。可以保留 docker rm -f，但需要在 teardown 做有界清理，或在解压前用 find . -mindepth 1 -maxdepth 1 -exec rm -rf -- {} + 清理完整 workspace。

@gouzil gouzil Jul 13, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里的 workspace 会在 github runner 结束的时候把数据删了吗

3 changes: 0 additions & 3 deletions .github/workflows/docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,9 +109,6 @@ jobs:
for name in "${!docker_files[@]}"
do
md5_value=`md5sum tools/dockerfile/${docker_files[$name]} | awk '{print $1}'`

@risemeup1111 risemeup1111 Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

这里仍然只用 Dockerfile.develop.dtk 的内容 md5 作为 docker_dcu 镜像 tag，但当前最新提交只修改了 tools/dockerfile/build_scripts/install_gcc.sh，Dockerfile.develop.dtk 的 md5 仍是上一轮的 881e30f7ca65be3eb3c04ea34e44c030。由于该 Dockerfile 通过 COPY tools/dockerfile/build_scripts /build_scripts 引入脚本，现有逻辑会继续拉取同一个 docker_dcu 镜像 tag；如果上一轮镜像已存在，docker pull 成功后就不会重新 build，install_gcc.sh 的 Kylin/lib64 修复不会进入实际 DCU 镜像，也无法验证本 PR 的目标环境。

建议把 docker_dcu 实际 COPY 的脚本也纳入 cache key，或采用等价方式确保 build_scripts 变更会触发 DCU 镜像重建。一个最小修复形态如下：

Suggested change
md5_value=`md5sum tools/dockerfile/${docker_files[$name]} | awk '{print $1}'`
if [ "$name" == "docker_dcu" ]; then
md5_value=`md5sum tools/dockerfile/${docker_files[$name]} tools/dockerfile/build_scripts/install_gcc.sh | md5sum | awk '{print $1}'`
else
md5_value=`md5sum tools/dockerfile/${docker_files[$name]} | awk '{print $1}'`
fi

@risemeup1111 risemeup1111 Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

补充当前 head ab2a5596fd175cdcda2edfac405d727be77b19cd 的复查结果：Dockerfile.develop.dtk 现在又回到 md5 881e30f7ca65be3eb3c04ea34e44c030，而 install_gcc.sh 仍是修改后的 a6ec0a16bc1ebde45ea6f329c87638d1。因此 docker_dcu 仍只会生成旧的 Dockerfile tag，若该 tag 已存在就会复用旧镜像，install_gcc.sh 的最新修复仍无法保证进入实际 DCU 镜像。请把 install_gcc.sh/build_scripts 纳入 docker_dcu hash，或用等价方式确保脚本变更触发新的镜像 tag。

This comment was marked as outdated.

Sign in to view

@risemeup1111 risemeup1111 Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

删除 docker_dcu 固定 md5 后，这里的 md5sum tools/dockerfile/${docker_files[$name]} 也会在 inputs.is_merge == "true" 的路径执行。该路径上面只下载 ci_dockerfile.sh 和 Dockerfile.ubuntu20，而 ci_dockerfile.sh 只生成 CPU/SOT/build/coverage 相关 Dockerfile，不会生成 Dockerfile.develop.dtk，也没有 tools/dockerfile/build_scripts；默认 task=cpu 的 docker_files 仍包含 docker_dcu。因此 merge-mode 会先得到空 md5，再尝试 docker build -f tools/dockerfile/Dockerfile.develop.dtk .，最终因为 Dockerfile/构建上下文缺失而失败。

请让 merge-mode 的 docker_dcu 路径也有完整输入，或在该模式下显式复用已有 DCU tag。一个可实现的修复形态如下：

if [ "${{ inputs.is_merge }}" == "true" ]; then
  mkdir -p tools/dockerfile/build_scripts
  cd tools/dockerfile
  wget -q --tries=5 --no-proxy \
    ${dockerfile_script} \
    ${dockerfile_origin} \
    https://raw.githubusercontent.com/PaddlePaddle/Paddle/refs/heads/develop/tools/dockerfile/Dockerfile.develop.dtk \
    --no-check-certificate
  cd build_scripts
  wget -q --tries=5 --no-proxy \
    https://raw.githubusercontent.com/PaddlePaddle/Paddle/refs/heads/develop/tools/dockerfile/build_scripts/install_gcc.sh \
    --no-check-certificate
  cd ..
else
  ...
fi

if [ $name == "docker_dcu" ]; then
md5_value="76937a563116f6008c8ca4cb4f592759"
fi
if [ $name == "docker_npu" ]; then
md5_value="a3793bdeea5ae881a0c1eaf4d7c30c64"
fi
Expand Down
2 changes: 1 addition & 1 deletion cmake/hip.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ list(APPEND HIP_CXX_FLAGS -Wno-missing-braces)
list(APPEND HIP_CXX_FLAGS -Wno-sometimes-uninitialized)
list(APPEND HIP_CXX_FLAGS -Wno-deprecated-copy)
list(APPEND HIP_CXX_FLAGS -Wno-pessimizing-move)
list(APPEND HIP_CXX_FLAGS -std=c++17)
list(APPEND HIP_CXX_FLAGS -std=c++20)

This comment was marked as outdated.

Sign in to view

list(APPEND HIP_CXX_FLAGS --gpu-max-threads-per-block=1024)
Comment on lines 200 to 202

if(CMAKE_BUILD_TYPE MATCHES Debug)
Expand Down
15 changes: 8 additions & 7 deletions tools/dockerfile/Dockerfile.develop.dtk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,13 @@ RUN yum install -y bzip2-devel openssh-server elfutils-devel diffutils libtool i
openssl-devel libjpeg-turbo-devel libpng-devel ninja-build pciutils libzstd-devel zstd

COPY tools/dockerfile/build_scripts /build_scripts
RUN bash /build_scripts/install_gcc.sh gcc82
RUN ln -sf /usr/local/gcc-8.2/bin/gcc /usr/local/bin/gcc
RUN ln -sf /usr/local/gcc-8.2/bin/g++ /usr/local/bin/g++
RUN ln -sf /usr/local/gcc-8.2/bin/gcc /usr/bin/gcc
RUN ln -sf /usr/local/gcc-8.2/bin/g++ /usr/bin/g++
ENV PATH=/usr/local/gcc-8.2/bin:$PATH
RUN bash /build_scripts/install_gcc.sh gcc122
RUN ln -sf /usr/local/gcc-12.2/bin/gcc /usr/local/bin/gcc
RUN ln -sf /usr/local/gcc-12.2/bin/g++ /usr/local/bin/g++
RUN ln -sf /usr/local/gcc-12.2/bin/gcc /usr/bin/gcc
RUN ln -sf /usr/local/gcc-12.2/bin/g++ /usr/bin/g++
ENV PATH=/usr/local/gcc-12.2/bin:$PATH
ENV LD_LIBRARY_PATH=/usr/local/gcc-12.2/lib64:/usr/local/gcc-12.2/lib:${LD_LIBRARY_PATH}

# workdir
WORKDIR /opt
Expand Down Expand Up @@ -100,7 +101,7 @@ RUN yum clean all && \
rm -rf /var/lib/yum/history

# Install DTK
RUN wget -q --no-proxy https://cancon.hpccube.com:65024/file/1/DTK-24.04.1/CentOS7.6/DTK-24.04.1-CentOS7.6-x86_64.tar.gz --no-check-certificate && \
RUN wget -q --no-proxy https://download.sourcefind.cn:65024/file/1/DTK-24.04.1/CentOS7.6/DTK-24.04.1-CentOS7.6-x86_64.tar.gz --no-check-certificate && \
tar zxf DTK-24.04.1-CentOS7.6-x86_64.tar.gz && rm -rf DTK-24.04.1-CentOS7.6-x86_64.tar.gz
# Replace if you use other device type, e.g. Z100, Z100L, K100
RUN wget -q https://paddle-device.bj.bcebos.com/dcu/hyhal-K100AI.tar.gz && \
Expand Down
1 change: 1 addition & 0 deletions tools/dockerfile/Dockerfile.release.ubuntu20
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ RUN ln -s /usr/local/gcc-12.1/bin/g++ /usr/local/bin/g++
RUN ln -s /usr/local/gcc-12.1/bin/gcc /usr/bin/gcc
RUN ln -s /usr/local/gcc-12.1/bin/g++ /usr/bin/g++
ENV PATH=/usr/local/gcc-12.1/bin:$PATH
ENV LD_LIBRARY_PATH=/usr/local/gcc-12.1/lib64:/usr/local/gcc-12.1/lib:${LD_LIBRARY_PATH}

RUN bash /build_scripts/install_cudnn.sh cudnn841
ENV CUDNN_VERSION=8.4.1
Expand Down
25 changes: 1 addition & 24 deletions tools/dockerfile/build_scripts/install_gcc.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,7 @@
# Stop at any error, show all commands
set -ex

if [ -f "/etc/redhat-release" ];then
lib_so_5=/usr/lib64/libgfortran.so.5
lib_so_6=/usr/lib64/libstdc++.so.6
lib_path=/usr/lib64
else
lib_so_5=/usr/lib/x86_64-linux-gnu/libstdc++.so.5
lib_so_6=/usr/lib/x86_64-linux-gnu/libstdc++.so.6
lib_path=/usr/lib/x86_64-linux-gnu
fi
# GCC is installed side-by-side. Callers select its runtime via LD_LIBRARY_PATH.

@risemeup1111 risemeup1111 Jul 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级：P1
处理要求：请针对该评论修复并提交新的 commit。

这里把 install_gcc.sh 改成只做 side-by-side 安装后，所有使用 GCC 12.x 的调用方都需要显式选择对应 runtime；但 tools/dockerfile/ci_dockerfile.sh 生成 Dockerfile.cuda123_cudnn9_gcc122_ubuntu20 时仍只把编译器路径从 GCC 8.2 改成 GCC 12.1，没有加入 /usr/local/gcc-12.1/lib64。我用当前脚本生成后看到该 Dockerfile 只有：

RUN bash /build_scripts/install_gcc.sh gcc121
ENV PATH=/usr/local/gcc-12.1/bin:$PATH

在本次提交之前，install_gcc.sh gcc121 会替换系统 libstdc++.so.6，所以这个生成路径还能拿到 GCC 12.1 runtime；现在该副作用被删除，CI 的 distribute 镜像会使用 GCC 12.1 编译器但运行/链接时仍落到 Ubuntu 20.04 系统 libstdc++，容易触发 GLIBCXX_* 符号缺失或运行时 ABI 不一致。

请同步修复 ci_dockerfile.sh 的 GCC 12.1 生成路径，例如在替换 PATH 时同时插入 runtime 路径：

sed -i 's#PATH=/usr/local/gcc-8.2/bin:$PATH#PATH=/usr/local/gcc-12.1/bin:$PATH#g' ${dockerfile_name}
sed -i '/ENV PATH=\/usr\/local\/gcc-12.1\/bin:$PATH/a ENV LD_LIBRARY_PATH=/usr/local/gcc-12.1/lib64:/usr/local/gcc-12.1/lib:${LD_LIBRARY_PATH}' ${dockerfile_name}

This comment was marked as outdated.

Sign in to view


if [ "$1" == "gcc82" ]; then
wget -q --no-proxy https://paddle-ci.gz.bcebos.com/gcc-8.2.0.tar.xz
Expand All @@ -43,12 +35,6 @@ if [ "$1" == "gcc82" ]; then
../gcc-8.2.0/configure --prefix=/usr/local/gcc-8.2 --enable-threads=posix --disable-checking --disable-multilib && \
make -j8 && make install
cd .. && rm -rf temp_gcc82 gcc-8.2.0 gcc-8.2.0.tar.xz
if [ -f "/etc/redhat-release" ];then
cp ${lib_so_6} ${lib_so_6}.bak && rm -f ${lib_so_6} &&
ln -s /usr/local/gcc-8.2/lib64/libgfortran.so.5 ${lib_so_5} && \
ln -s /usr/local/gcc-8.2/lib64/libstdc++.so.6 ${lib_so_6} && \
cp /usr/local/gcc-8.2/lib64/libstdc++.so.6.0.25 ${lib_path}
fi
elif [ "$1" == "gcc122" ]; then
wget -q --no-proxy https://paddle-ci.gz.bcebos.com/gcc-12.2.0.tar.gz
tar -xzf gcc-12.2.0.tar.gz && \
Expand All @@ -59,9 +45,6 @@ elif [ "$1" == "gcc122" ]; then
../gcc-12.2.0/configure --prefix=/usr/local/gcc-12.2 --enable-checking=release --enable-languages=c,c++ --disable-multilib && \
make -j8 && make install
cd .. && rm -rf temp_gcc122 gcc-12.2.0 gcc-12.2.0.tar.gz
cp ${lib_so_6} ${lib_so_6}.bak && rm -f ${lib_so_6} &&
ln -s /usr/local/gcc-12.2/lib64/libstdc++.so.6 ${lib_so_6} && \
cp /usr/local/gcc-12.2/lib64/libstdc++.so.6.0.30 ${lib_path}
elif [ "$1" == "gcc121" ]; then
wget -q --no-proxy https://paddle-ci.gz.bcebos.com/gcc-12.1.0.tar.gz
tar -xzf gcc-12.1.0.tar.gz && \
Expand All @@ -72,9 +55,6 @@ elif [ "$1" == "gcc121" ]; then
../gcc-12.1.0/configure --prefix=/usr/local/gcc-12.1 --enable-checking=release --enable-languages=c,c++ --disable-multilib && \
make -j8 && make install
cd .. && rm -rf temp_gcc121 gcc-12.1.0 gcc-12.1.0.tar.gz
cp ${lib_so_6} ${lib_so_6}.bak && rm -f ${lib_so_6} &&
ln -s /usr/local/gcc-12.1/lib64/libstdc++.so.6 ${lib_so_6} && \
cp /usr/local/gcc-12.1/lib64/libstdc++.so.6.0.30 ${lib_path}
elif [ "$1" == "gcc152" ]; then
GCC_VERSION=${GCC_VERSION:-15.2.0}
GCC_MAJOR_MINOR=$(echo ${GCC_VERSION} | cut -d. -f1,2)
Expand All @@ -89,8 +69,5 @@ elif [ "$1" == "gcc152" ]; then
../gcc-${GCC_VERSION}/configure --prefix=${GCC_PREFIX} --enable-checking=release --enable-languages=c,c++ --disable-multilib && \
make -j$(nproc) && make install
cd .. && rm -rf temp_gcc152 gcc-${GCC_VERSION} ${GCC_ARCHIVE}
cp ${lib_so_6} ${lib_so_6}.bak && rm -f ${lib_so_6} &&
ln -s ${GCC_PREFIX}/lib64/libstdc++.so.6 ${lib_so_6} && \
cp ${GCC_PREFIX}/lib64/libstdc++.so.6.* ${lib_path}

fi
14 changes: 7 additions & 7 deletions tools/dockerfile/centos7_manylinux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,42 +20,42 @@ REPO="${REPO:-paddledocker}"

function make_cuda112cudnn821trt8034() {
sed 's/<baseimg>/11.2.2-cudnn8-devel-centos7/g' Dockerfile.centos >Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc82 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-8.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-8.2/bin:\$PATH \nRUN yum remove -y libcudnn8-devel.x86_64 libcudnn8.x86_64 \nRUN bash build_scripts/install_cudnn.sh cudnn821 \nENV CUDNN_VERSION=8.2.1 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc82 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-8.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-8.2/bin:\$PATH \nENV LD_LIBRARY_PATH=/usr/local/gcc-8.2/lib64:/usr/local/gcc-8.2/lib:\$LD_LIBRARY_PATH \nRUN yum remove -y libcudnn8-devel.x86_64 libcudnn8.x86_64 \nRUN bash build_scripts/install_cudnn.sh cudnn821 \nENV CUDNN_VERSION=8.2.1 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#build_scripts/install_trt.sh#build_scripts/install_trt.sh trt8034#g" Dockerfile.tmp
sed -i '/CMD/iRUN ldconfig' Dockerfile.tmp
}

function make_cuda116cudnn840trt8406() {
sed 's/<baseimg>/11.6.2-cudnn8-devel-centos7/g' Dockerfile.centos >Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc82 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-8.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-8.2/bin:\$PATH \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc82 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-8.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-8.2/bin:\$PATH \nENV LD_LIBRARY_PATH=/usr/local/gcc-8.2/lib64:/usr/local/gcc-8.2/lib:\$LD_LIBRARY_PATH \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#build_scripts/install_trt.sh#build_scripts/install_trt.sh trt8406#g" Dockerfile.tmp
sed -i '/CMD/iRUN ldconfig' Dockerfile.tmp
}

function make_cuda117cudnn841trt8424() {
sed 's/<baseimg>/11.7.1-devel-centos7/g' Dockerfile.centos >Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc82 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-8.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-8.2/bin:\$PATH \nRUN bash build_scripts/install_cudnn.sh cudnn841 \nENV CUDNN_VERSION=8.4.1 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc82 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-8.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-8.2/bin:\$PATH \nENV LD_LIBRARY_PATH=/usr/local/gcc-8.2/lib64:/usr/local/gcc-8.2/lib:\$LD_LIBRARY_PATH \nRUN bash build_scripts/install_cudnn.sh cudnn841 \nENV CUDNN_VERSION=8.4.1 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#build_scripts/install_trt.sh#build_scripts/install_trt.sh trt8424#g" Dockerfile.tmp
sed -i '/CMD/iRUN ldconfig' Dockerfile.tmp
}

function make_cuda118cudnn860trt8531() {
sed 's/<baseimg>/11.8.0-devel-centos7/g' Dockerfile.centos >Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc82 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-8.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-8.2/bin:\$PATH \nRUN bash build_scripts/install_cudnn.sh cudnn860 \nENV CUDNN_VERSION=8.6.0 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc82 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-8.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-8.2/bin:\$PATH \nENV LD_LIBRARY_PATH=/usr/local/gcc-8.2/lib64:/usr/local/gcc-8.2/lib:\$LD_LIBRARY_PATH \nRUN bash build_scripts/install_cudnn.sh cudnn860 \nENV CUDNN_VERSION=8.6.0 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#build_scripts/install_trt.sh#build_scripts/install_trt.sh trt8531#g" Dockerfile.tmp
sed -i '/CMD/iRUN ldconfig' Dockerfile.tmp
}

function make_cuda120cudnn891trt8616() {
sed 's/<baseimg>/12.0.1-devel-centos7/g' Dockerfile.centos >Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc122 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-12.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-12.2/bin:\$PATH \nRUN bash build_scripts/install_cudnn.sh cudnn891 \nENV CUDNN_VERSION=8.9.1 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc122 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-12.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-12.2/bin:\$PATH \nENV LD_LIBRARY_PATH=/usr/local/gcc-12.2/lib64:/usr/local/gcc-12.2/lib:\$LD_LIBRARY_PATH \nRUN bash build_scripts/install_cudnn.sh cudnn891 \nENV CUDNN_VERSION=8.9.1 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#build_scripts/install_trt.sh#build_scripts/install_trt.sh trt8616#g" Dockerfile.tmp
sed -i '/CMD/iRUN ldconfig' Dockerfile.tmp
}

function make_cuda123cudnn900trt8616() {
sed 's/<baseimg>/12.3.1-devel-centos7/g' Dockerfile.centos >Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc122 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-12.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-12.2/bin:\$PATH \nRUN bash build_scripts/install_cudnn.sh cudnn900 \nENV CUDNN_VERSION=9.0.0 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc122 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-12.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-12.2/bin:\$PATH \nENV LD_LIBRARY_PATH=/usr/local/gcc-12.2/lib64:/usr/local/gcc-12.2/lib:\$LD_LIBRARY_PATH \nRUN bash build_scripts/install_cudnn.sh cudnn900 \nENV CUDNN_VERSION=9.0.0 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#build_scripts/install_trt.sh#build_scripts/install_trt.sh trt8616#g" Dockerfile.tmp
sed -i '/CMD/iRUN ldconfig' Dockerfile.tmp
}
Expand All @@ -70,7 +70,7 @@ function make_cuda124cudnn911trt8616() {

function make_cuda125cudnn911trt8616() {
sed 's/<baseimg>/12.5.1-cudnn-devel-rockylinux8/g' Dockerfile.rockylinux8 >Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc122 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-12.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-12.2/bin:\$PATH \nRUN bash build_scripts/install_cudnn.sh cudnn911 \nENV CUDNN_VERSION=9.1.1 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#RUN bash build_scripts/build.sh#RUN bash build_scripts/install_gcc.sh gcc122 \nRUN mv /usr/bin/cc /usr/bin/cc.bak \&\& ln -s /usr/local/gcc-12.2/bin/gcc /usr/bin/cc \nENV PATH=/usr/local/gcc-12.2/bin:\$PATH \nENV LD_LIBRARY_PATH=/usr/local/gcc-12.2/lib64:/usr/local/gcc-12.2/lib:\$LD_LIBRARY_PATH \nRUN bash build_scripts/install_cudnn.sh cudnn911 \nENV CUDNN_VERSION=9.1.1 \nRUN bash build_scripts/build.sh#g" Dockerfile.tmp
sed -i "s#build_scripts/install_trt.sh#build_scripts/install_trt.sh trt8616#g" Dockerfile.tmp
sed -i '/CMD/iRUN ldconfig' Dockerfile.tmp
}
Expand Down
3 changes: 3 additions & 0 deletions tools/dockerfile/ubuntu20_release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ function base_image(){
sed -i 's#<install_cpu_package>##g' ${dockerfile_name}
sed -i "s#<install_gcc>#WORKDIR /usr/bin ENV PATH=/usr/local/gcc-8.2/bin:\$PATH #g" ${dockerfile_name}
sed -i "s#gcc121#gcc82#g" ${dockerfile_name}
sed -i '/ENV LD_LIBRARY_PATH=\/usr\/local\/gcc-12.1\/lib64:\/usr\/local\/gcc-12.1\/lib:${LD_LIBRARY_PATH}/d' ${dockerfile_name}
sed -i "s#gcc-12.1#gcc-8.2#g" ${dockerfile_name}
sed -i 's#RUN bash /build_scripts/install_trt.sh#RUN bash /build_scripts/install_trt.sh trt8034#g' ${dockerfile_name}
sed -i 's#cudnn841#cudnn821#g' ${dockerfile_name}
Expand All @@ -33,6 +34,7 @@ function base_image(){
sed -i 's#<install_cpu_package>##g' ${dockerfile_name}
sed -i "s#<install_gcc>#WORKDIR /usr/bin ENV PATH=/usr/local/gcc-8.2/bin:\$PATH #g" ${dockerfile_name}
sed -i "s#gcc121#gcc82#g" ${dockerfile_name}
sed -i '/ENV LD_LIBRARY_PATH=\/usr\/local\/gcc-12.1\/lib64:\/usr\/local\/gcc-12.1\/lib:${LD_LIBRARY_PATH}/d' ${dockerfile_name}
sed -i "s#gcc-12.1#gcc-8.2#g" ${dockerfile_name}
sed -i 's#RUN bash /build_scripts/install_trt.sh#RUN bash /build_scripts/install_trt.sh trt8531#g' ${dockerfile_name}
sed -i 's#cudnn841#cudnn860#g' ${dockerfile_name}
Expand Down Expand Up @@ -71,6 +73,7 @@ function base_image(){
sed -i 's#<install_cpu_package>#RUN apt-get install -y gcc g++ make#g' ${dockerfile_name}
sed -i "s#<install_gcc>#WORKDIR /usr/bin ENV PATH=/usr/local/gcc-8.2/bin:\$PATH #g" ${dockerfile_name}
sed -i "s#gcc121#gcc82#g" ${dockerfile_name}
sed -i '/ENV LD_LIBRARY_PATH=\/usr\/local\/gcc-12.1\/lib64:\/usr\/local\/gcc-12.1\/lib:${LD_LIBRARY_PATH}/d' ${dockerfile_name}
sed -i "s#gcc-12.1#gcc-8.2#g" ${dockerfile_name}
sed -i 's#RUN bash /build_scripts/install_trt.sh##g' ${dockerfile_name}
sed -i 's#RUN bash /build_scripts/install_cudnn.sh cudnn841##g' ${dockerfile_name}
Expand Down
Loading