Skip to content

Debug ci timed out, do not merge#1805

Open
TristanCacqueray wants to merge 2 commits into
containers:mainfrom
TristanCacqueray:debug-ci
Open

Debug ci timed out, do not merge#1805
TristanCacqueray wants to merge 2 commits into
containers:mainfrom
TristanCacqueray:debug-ci

Conversation

@TristanCacqueray

Copy link
Copy Markdown
Contributor

No description provided.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the playbooks/system-test-commands-options.yaml playbook to include additional flags (--show-output-of-passing-tests, --timing, and --trace) for the bats command when running the commands-options system tests. No review comments were provided, and there is no feedback to address.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@debarshiray

debarshiray commented Jun 11, 2026

Copy link
Copy Markdown
Member

This is where it gets stuck:

fedora-rawhide | ...
fedora-rawhide | $ _setup_docker_registry
fedora-rawhide | ...
fedora-rawhide | $$ run podman --root "${DOCKER_REG_ROOT}" run --detach --env REGISTRY_AUTH=htpasswd --env REGISTRY_AUTH_HTPASSWD_PATH="/auth/htpasswd" --env REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" --env REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt --env REGISTRY_HTTP_TLS_KEY=/certs/domain.key --name "${DOCKER_REG_NAME}" --privileged --publish 50000:5000 --rm --volume "${DOCKER_REG_AUTH_DIR}":/auth --volume "${DOCKER_REG_CERTS_DIR}":/certs "${IMAGES[docker-reg]}"
fedora-rawhide | ...
fedora-rawhide | $$ run podman login --authfile "${BATS_SUITE_TMPDIR}/authfile.json" --username user --password user "${DOCKER_REG_URI}"
fedora-rawhide | ...
fedora-rawhide | $$ run skopeo copy --dest-authfile "${BATS_SUITE_TMPDIR}/authfile.json" dir:"${IMAGE_CACHE_DIR}"/fedora-toolbox-34 docker://"${DOCKER_REG_URI}"/fedora-toolbox:34

The test suite runs its own Docker registry using podman ... run ..., and populates it with an old fedora-toolbox:34 image using skopeo copy .... The old image is already downloaded to disk in $IMAGE_CACHE_DIR, so it should be a local operation.

@debarshiray debarshiray left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added --show-output-of-passing-tests --timing --trace to playbooks/system-test-runtime-environment-arch-fedora.yaml and playbooks/system-test-runtime-environment-ubuntu.yaml to see why the other jobs aren't getting stuck.

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor

Maybe this is a race where the push starts before the registry container is actually ready. WDYT about introducing some way to wait on the registry? Maybe something like:

local -i retries
for ((retries = 0; retries < 30; retries++)); do
    if curl -fsk https://localhost:50000/v2/ > /dev/null 2>&1; then
        break
    fi
    sleep 1
done

@TristanCacqueray

Copy link
Copy Markdown
Contributor Author

It looks like the job is hanging on a skopeo command, perhaps this could also be reported upstream?

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor

It looks like the job is hanging on a skopeo command, perhaps this could also be reported upstream?

Yeah probably would be a good idea.

These skopeo calls should probably use --command-timeout at the very least so they can't hang forever.

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor

podman-container-tools/skopeo#2878 is potentially related, with podman-container-tools/container-libs#777 being the proposed solution there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants