Skip to content

Improve build caching #879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Zelberor wants to merge 4 commits into
base: main
Choose a base branch
from
+34 −2
Open

Improve build caching #879

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b6583dc
Improve build caching
Zelberor Dec 8, 2025
c032f38
Merge branch 'main' into dependencies-build-cache-fix
Zelberor Dec 8, 2025
7edf374
Merge branch 'main' into dependencies-build-cache-fix
Zelberor Jan 9, 2026
70c5dfd
Fix dir name
Zelberor Jan 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ jobs:
PAF_UID: 1000
PAF_GID: 1000
BASE_FLAVOUR: cuda
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache,mode=max

- name: Build and push Docker image
id: build
Expand All @@ -68,6 +70,8 @@ jobs:
PAF_UID: 1000
PAF_GID: 1000
BASE_FLAVOUR: cuda
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache,mode=max

- name: Save pull request artifact
if: github.event_name == 'pull_request'
Expand Down
32 changes: 30 additions & 2 deletions build/docker/agent-ros2/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,31 @@ RUN echo source ${INTERNAL_WORKSPACE_DIR}/deploy.bashrc >> ~/.bashrc
WORKDIR ${INTERNAL_WORKSPACE_DIR}
CMD [ "ros2", "run", "leaderboard_launcher", "launch_leaderboard.deploy.sh" ]

##########################################################
##### The dependencies for agent-dev are pulled here #####
##########################################################
FROM ubuntu:24.04 AS agent-deps
# This separate stage only copies the dependency definitions from our src.
# When these dependency definitions are then used in the agent-dev stage,
# docker only checks for changes in the dependency definitions and only
# triggers a rebuild when dependencies changed (instead of any file change inside src).
# This avoids many unnecessary container rebuilds.

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update && apt-get install -y rsync

RUN --mount=type=bind,target=/workspace/code,source=code \
mkdir /agent-deps && \
rsync -r --prune-empty-dirs --include="*/" \
--include="requirements.txt" \
--include="requirements.*.txt" \
--include="package.xml" \
--include="setup.cfg" \
--include="setup.py" \
--include="CMakeLists.txt" \
--exclude="*" "/workspace/code" "/agent-deps/"

Comment on lines +342 to +356

@coderabbitai coderabbitai Bot Jan 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

# First, let's explore the repository structure
git ls-files | head -30

Repository: una-auxme/paf

Length of output: 768

🏁 Script executed:

# Check the Dockerfile context - find all files in or related to the agent-ros2 build
fd -t f 'Dockerfile|requirements|setup|pyproject|MANIFEST|CMakeLists' --max-results=50

Repository: una-auxme/paf

Length of output: 1808

🏁 Script executed:

# Get more context on the Dockerfile around line 393 mentioned in the comment
sed -n '390,400p' build/docker/agent-ros2/Dockerfile

Repository: una-auxme/paf

Length of output: 510

🏁 Script executed:

# Search for all dependency files that exist in the repo
echo "=== Python dependency files ===" && \
fd -t f -n 'pyproject\.toml|MANIFEST\.in' && \
echo -e "\n=== Requirements files ===" && \
fd -t f -n 'requirements.*\.txt|requirements\.txt' && \
echo -e "\n=== Setup files ===" && \
fd -t f -n 'setup\.py|setup\.cfg' && \
echo -e "\n=== Package manifests ===" && \
fd -t f -n 'package\.xml' && \
echo -e "\n=== CMake files ===" && \
fd -t f -n 'CMakeLists\.txt'

Repository: una-auxme/paf

Length of output: 336

🏁 Script executed:

# Fix the command - search for dependency files properly
echo "=== Python dependency files ===" && \
fd 'pyproject\.toml' && \
fd 'MANIFEST\.in' && \
echo -e "\n=== Requirements files ===" && \
fd 'requirements.*\.txt' && \
echo -e "\n=== Setup files ===" && \
fd 'setup\.py|setup\.cfg' && \
echo -e "\n=== Package manifests ===" && \
fd 'package\.xml' && \
echo -e "\n=== CMake files ===" && \
fd 'CMakeLists\.txt'

Repository: una-auxme/paf

Length of output: 2121

🏁 Script executed:

# Specifically check in code/ directory for dependency files
echo "=== All dependency-related files in code/ directory ===" && \
fd -t f '' code/ | grep -E '(requirements|setup|pyproject|MANIFEST|CMakeLists|package\.xml)' | head -50

Repository: una-auxme/paf

Length of output: 1180

🏁 Script executed:

# Check what files the rsync is supposed to copy - look at the code/ directory structure
echo "=== Directory structure of code/ ===" && \
fd -t f -d 2 '' code/ | head -40

Repository: una-auxme/paf

Length of output: 1232

🏁 Script executed:

# Verify which requirements files match the rsync pattern "requirements.*.txt"
echo "=== Testing rsync glob pattern 'requirements.*.txt' ===" && \
echo "Files that should match 'requirements.*.txt' pattern:" && \
cd code && ls -1 requirements*.txt && cd - && \
echo -e "\n=== Testing with bash glob matching ===" && \
python3 << 'EOF'
import fnmatch
import os

pattern = "requirements.*.txt"
test_files = [
    "requirements.txt",
    "requirements.cpu.txt",
    "requirements.cuda.txt", 
    "requirements.rocm.txt",
    "requirements_infrastructure.txt"
]

print(f"Pattern: {pattern}\n")
for f in test_files:
    matches = fnmatch.fnmatch(f, pattern)
    print(f"  {f:<35} -> {'✓ MATCHES' if matches else '✗ DOES NOT MATCH'}")
EOF

Repository: una-auxme/paf

Length of output: 143

🏁 Script executed:

# Check if requirements_infrastructure.txt is actually needed/used
echo "=== Checking usage of requirements_infrastructure.txt ===" && \
rg -l 'requirements_infrastructure' --type txt --type py --type sh

Repository: una-auxme/paf

Length of output: 177

🏁 Script executed:

# Look at the full Dockerfile to understand the context better
sed -n '342,400p' build/docker/agent-ros2/Dockerfile

Repository: una-auxme/paf

Length of output: 2693

🏁 Script executed:

# Check if requirements_infrastructure.txt is required for the build
cat build/docker/agent-ros2/scripts/install-python-requirements.sh

Repository: una-auxme/paf

Length of output: 2368

🏁 Script executed:

# Verify the exact rsync filter matching behavior
python3 << 'EOF'
import fnmatch

# Test files from the codebase
files = [
    "requirements.txt",
    "requirements.cpu.txt",
    "requirements.cuda.txt",
    "requirements.rocm.txt",
    "requirements_infrastructure.txt"
]

# Rsync patterns from the Dockerfile
patterns = [
    "requirements.txt",
    "requirements.*.txt"
]

print("Rsync filter matching analysis:\n")
for f in files:
    matches = any(fnmatch.fnmatch(f, p) for p in patterns)
    status = "✓ COPIED" if matches else "✗ NOT COPIED"
    print(f"  {f:<35} -> {status}")
    if not matches:
        print(f"    (not matched by any pattern: {patterns})")
EOF

Repository: una-auxme/paf

Length of output: 420

Add requirements_infrastructure.txt to rsync filters.

The rsync glob pattern requirements.*.txt only matches files with dots (e.g., requirements.cpu.txt), not underscores. However, requirements_infrastructure.txt is actively used in the build process and will not be copied to /agent-deps/code/, causing infrastructure dependencies to be missing during the build.

Update the rsync filters to explicitly include the infrastructure requirements file:

Fix rsync filters 
 RUN --mount=type=bind,target=/workspace/code,source=code \
   mkdir /agent-deps && \
   rsync -r --prune-empty-dirs --include="*/" \
   --include="requirements.txt" \
   --include="requirements.*.txt" \
+  --include="requirements_infrastructure.txt" \
   --include="package.xml" \
   --include="setup.cfg" \
   --include="setup.py" \
   --include="CMakeLists.txt" \
   --exclude="*" "/workspace/code" "/agent-deps/"

Alternatively, use requirements*.txt instead of both requirements.txt and requirements.*.txt to capture all variants (dot and underscore).

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update && apt-get install -y rsync
RUN --mount=type=bind,target=/workspace/code,source=code \
mkdir /agent-deps && \
rsync -r --prune-empty-dirs --include="*/" \
--include="requirements.txt" \
--include="requirements.*.txt" \
--include="package.xml" \
--include="setup.cfg" \
--include="setup.py" \
--include="CMakeLists.txt" \
--exclude="*" "/workspace/code" "/agent-deps/"
RUN --mount=type=bind,target=/workspace/code,source=code \
mkdir /agent-deps && \
rsync -r --prune-empty-dirs --include="*/" \
--include="requirements.txt" \
--include="requirements.*.txt" \
--include="requirements_infrastructure.txt" \
--include="package.xml" \
--include="setup.cfg" \
--include="setup.py" \
--include="CMakeLists.txt" \
--exclude="*" "/workspace/code" "/agent-deps/"
🤖 Prompt for AI Agents 
In @build/docker/agent-ros2/Dockerfile around lines 342 - 356, The rsync
invocation that builds /agent-deps currently includes patterns
"--include='requirements.txt'" and "--include='requirements.*.txt'", which
misses files with underscores like requirements_infrastructure.txt; update the
rsync include filters in that RUN block (the rsync command that copies from
"/workspace/code" to "/agent-deps/") to either add
"--include='requirements_infrastructure.txt'" or more generically replace the
two patterns with "--include='requirements*.txt'" so that both dotted and
underscored variants are copied.

###########################################
##### The agent-dev image starts here #####
###########################################
Expand Down Expand Up @@ -365,11 +390,12 @@ COPY --chmod=755 ${DOCKER_RESOURCE_BASE}/scripts/entrypoint-dev.sh \

ENTRYPOINT [ "/internal_workspace/scripts/entrypoint-dev.sh" ]

COPY --from=agent-deps --chown=${PAF_UID}:${PAF_GID} "/agent-deps/code" "${PAF_ROS_WS}/src"

WORKDIR ${PAF_ROS_WS}
# Install package.xml dependencies (but allow it to fail, because this is a dev container)
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
--mount=type=bind,target=${PAF_ROS_WS}/src,source=code \
bash -c " \
source ${INTERNAL_WORKSPACE_DIR}/env.bash && \
sudo apt-get update && \
Expand All @@ -380,13 +406,15 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \

# Install python requirements.txt dependencies (but allow it to fail, because this is a dev container)
RUN --mount=type=cache,target=${PIP_CACHE_DIR},sharing=locked,uid="${PAF_UID}",gid="${PAF_GID}" \
--mount=type=bind,target=${PAF_ROS_WS}/src,source=code \
bash -c " \
source ${INTERNAL_WORKSPACE_DIR}/dev.bashrc && \
((${INTERNAL_WORKSPACE_DIR}/scripts/install-python-requirements.sh |& tee ${INTERNAL_WORKSPACE_DIR}/pip_install.log; exit \${PIPESTATUS[0]}) \
|| echo \"echo -e \\\"\\\\e[31mERROR: pip install failed, check ${INTERNAL_WORKSPACE_DIR}/pip_install.log in the container\\\\e[0m\\\"\" >> ${INTERNAL_WORKSPACE_DIR}/dev.bashrc) \
"

# Clean source so no partial packages persist in the container
RUN rm -rf "${PAF_ROS_WS}/src"

RUN echo source ${INTERNAL_WORKSPACE_DIR}/dev.bashrc >> ~/.bashrc

VOLUME [ "/workspace" ]
Expand Down