SWT-Bench cold image builds regress badly under shared registry cache export

[simonrosenberg]

Summary

This is the SWT-Bench-specific sub-issue for the broader image build regression tracked in #504.

After going through the historical SWT runs carefully, the picture is:

1. The best historical cold SWT run we still have is much faster than current March runs.
2. SWT was already slower in early March before the shared-cache SDK change landed.
3. The latest and worst SWT collapse happens when the SDK starts exporting a shared registry cache from every parallel SWT build.
4. SWT is especially sensitive because it fans out across hundreds of mostly unique base images, so the shared cache is cold at the start and many workers try to write to the same cache ref concurrently.

Cold vs warm classification

These runs should not be used to measure cold-build throughput:

• 23000734281: fully warm, artifact manifest shows 433 skips / 0 builds.
• 22989698608: fully warm, artifact manifest shows 433 skips / 0 builds.
• 22985173172: fully warm, artifact manifest shows 433 skips / 0 builds.
• 22967516108: mixed, artifact manifest shows 288 skips / 145 builds.

These are the most useful comparison points:

• 21870877504 (2026-02-10, SDK cfe52af, best historical cold baseline available)
  • no already exists. Skipping build. lines in the run log
  • command used --max-workers 4
• 23001334237 (2026-03-12, SDK aa9df69, near-cold partial run)
  • manifest mostly real builds before cancellation
  • command used --max-workers 4
• 23008709458 (2026-03-12, SDK d0c1a39, near-cold partial run)
  • manifest shows only 12 skips in the first 170 completions
  • command used --max-workers 4
• 23017371689 (2026-03-12, SDK fc962c4, latest bad run)
  • manifest shows 44 builds / 27 skips before cancellation
  • command used --max-workers 16 --build-batch-size 50

Throughput regression

Useful milestone timings from the raw GitHub logs:

Run	SDK	Classification	50/433	100/433	150/433
21870877504	cfe52af	best cold baseline	18m29s	40m52s	55m41s
23001334237	aa9df69	near-cold	41m07s	1h20m24s	-
23008709458	d0c1a39	near-cold / lightly mixed	48m18s	1h41m39s	2h30m46s
23017371689	fc962c4	mixed, latest bad run	2h00m52s	-	-

Equivalent completion throughput:

• 21870877504: about 162 images/hour to 50, 147 images/hour to 100
• 23001334237: about 73 images/hour to 50, 75 images/hour to 100
• 23008709458: about 62 images/hour to 50, 59 images/hour to 100
• 23017371689: about 25 images/hour to 50

So there are really two regressions:

1. Feb cold baseline -> early March pre-shared-cache runs: roughly 2.0x to 2.6x slower.
2. Early March pre-shared-cache runs -> fc962c4 shared-cache run: another roughly 2.4x to 3.0x slower by the 50-image mark, even though the run was configured with higher nominal concurrency.

Relative to the best cold baseline, the latest fc962c4 run is about 4.9x slower at the 50-image mark.

What changed in code

Between the good February baseline and early March

The early March runs (aa9df69 / d0c1a39) are still using the older SWT workflow shape with --max-workers 4.

Those runs do include SDK build-path changes relative to cfe52af, including a heavier builder environment and then the prebuilt-sdist optimization. The prebuilt-sdist optimization helps remove repeated uv build --sdist, but it does not explain the remaining SWT slowdown by itself.

Between d0c1a39 and fc962c4

There are two important changes in the latest bad regime:

1. The SWT workflow was changed from default max-workers=4 to max-workers=16, and agent image submit batch size was raised to 50.
2. The SDK build path changed from exporting one registry cache to exporting two caches for pushed builds:
   • the image-specific cache ref
   • a shared cache ref

In openhands/agent_server/docker/build.py, fc962c4 introduces shared_cache_tag / shared_cache_tags and then emits two --cache-to type=registry,... destinations for a pushed build.

That change is the key new behavior for the worst SWT run.

Why SWT is hit harder than SWE

SWT expands to a very large set of distinct base images. On the current dataset, collect_unique_base_images() yields 433 unique base images.

That means the first wave of a cold SWT run gets very little value from a shared cache that has not been seeded yet. At the same time, with the fc962c4 behavior, many workers are all trying to export to the same shared registry cache ref.

So SWT pays both sides of the penalty:

• little cache reuse at the start, because the build graph is cold and the images are diverse
• heavy contention and upload cost on cache-to because many workers write the same shared ref

Direct evidence from the fc962c4 run

The run-level progress already looks bad, but the per-image logs make the root cause much clearer.

Representative examples from 23017371689:

• astropy-12907
  • image export itself took about 167s
  • the two registry cache exports then took about 870s and 874s
• django-10554
  • image export itself took about 79s
  • the two registry cache exports then took about 2294s and 2301s

In other words, some individual SWT builds are spending 15 to 38 minutes after the image is already built, almost entirely in exporting cache to registry.

That strongly points at docker buildx --cache-to type=registry,... as the current dominant bottleneck.

Root-cause hypothesis

Current best explanation:

1. There was already an earlier SWT slowdown between cfe52af and the early March SDKs (aa9df69 / d0c1a39). That older slowdown is likely due to SDK build-path changes that made the build heavier even before shared cache was introduced.
2. The latest severe SWT collapse is caused by the shared registry cache export added in fc962c4.
3. The workflow change to --max-workers 16 and --build-batch-size 50 likely amplifies the problem, because more cold SWT builds are now trying to export the same shared cache concurrently.
4. The old repeated-sdist issue is no longer the primary SWT bottleneck in these latest runs, because the slow shared-cache runs are already reusing a prebuilt sdist.

Implications

For SWT specifically, the main problem now is not the Dockerfile's main build steps. It is the cache export strategy under cold, high-fanout image building.

If we want to recover SWT throughput, we likely need to change the cache strategy, not just keep optimizing sdist creation.

Likely next steps

• Disable shared cache-to export for the normal parallel SWT worker pool.
• If we still want a shared cache, seed it once in a controlled warm-up step instead of having every worker write it.
• Re-test fc962c4 behavior at --max-workers 4 to separate pure shared-cache cost from the concurrency amplification.
• Separately isolate the earlier Feb -> early March slowdown, which predates shared_cache_tag.

References

• Parent tracker: Track root cause isolation and fix plan for the 500-image build regression after #456 #504
• Historical cold baseline: https://github.com/OpenHands/benchmarks/actions/runs/21870877504
• Early March near-cold run (aa9df69): https://github.com/OpenHands/benchmarks/actions/runs/23001334237
• Early March near-cold run (d0c1a39): https://github.com/OpenHands/benchmarks/actions/runs/23008709458
• Latest bad shared-cache run (fc962c4): https://github.com/OpenHands/benchmarks/actions/runs/23017371689

[simonrosenberg]

Root cause confirmed on March 12, 2026:

• software-agent-sdk commit fc962c4c8d7fd6eb0e91db06a5ec9911b29589ac added a shared registry cache-to export on pushed builds.
• On SWT-Bench cold runs, 433 mostly unique images fan out in parallel, so workers get little early reuse from the shared cache but all contend on the same shared registry cache ref.
• The slowest per-image logs spend most of their time after image export in exporting cache to registry, which matches the regression pattern from this issue.

Fix pushed into the existing feature PRs:

• OpenHands/software-agent-sdk#2399 (fix/prebuilt-sdist-api) now points at 447aa918b77ffaa1120b67c0cc9b55907eea7f82
  • change: keep shared cache-from, but disable shared registry cache-to by default; opt-in only via OPENHANDS_SHARED_REGISTRY_CACHE_EXPORT=1
• OpenHands/benchmarks#507 (fix/issue-504-cached-sdist) now points its submodule at that SDK commit via 22ea7b51cdfff717925c66c9efbad411c3fd0ad4

Full SWT image-build tracking run:

• workflow: Build SWT-Bench Images
• benchmarks ref: fix/issue-504-cached-sdist (22ea7b51cdfff717925c66c9efbad411c3fd0ad4)
• sdk ref: 447aa918b77ffaa1120b67c0cc9b55907eea7f82
• dataset: eth-sri/SWT-bench_Verified_bm25_27k_zsp
• split: test
• selected instances: 433 explicit instance IDs
• agent image settings: max-workers=16, agent-build-batch-size=50

This comment is the tracking anchor for the current image-building job.

[simonrosenberg]

Posted a follow-up fix after reviewing the logs from run 23028385721.

Refined root cause from the logs:

• The shared registry cache export regression is fixed in SDK 447aa91; the logs showed shared cache export=disabled on every build.
• Throughput still stayed collapsed because every cold SWT image build was still doing an image-specific registry --cache-to ... mode=max export. In the updated logs, 84/84 sampled docker buildx build commands still had registry cache-to, and many individual images were taking ~45-50 minutes end-to-end.
• So the remaining dominant bottleneck is registry cache export itself under the 433-image cold fan-out, not just the shared-cache variant.

Fixes pushed:

• SDK PR: fix: control buildkit cache exports software-agent-sdk#2399
  • new head: 8e8223b24dfd041875fe845b13326b0a1238b793
  • adds OPENHANDS_BUILDKIT_CACHE_MODE=off|min|max, keeps cache-from, and allows disabling all cache-to exports for cold image-build runs
• Benchmarks PR: fix: cache SDK sdist across image builds #507
  • new head: 277cb51be339c3f7fda1705be90cfa05840b3d81
  • sets OPENHANDS_BUILDKIT_CACHE_MODE=off for the SWT image-build workflow and updates the SDK submodule

Validation:

• uv run pytest tests/agent_server/test_docker_build.py
• uv run pre-commit run --files openhands-agent-server/openhands/agent_server/docker/build.py tests/agent_server/test_docker_build.py

Retriggered full 433-instance SWT image build with the updated branches:

• Run: https://github.com/OpenHands/benchmarks/actions/runs/23033839708
• Status: queued
• Benchmarks ref: fix/issue-504-cached-sdist @ 277cb51be339c3f7fda1705be90cfa05840b3d81
• SDK ref: 8e8223b24dfd041875fe845b13326b0a1238b793
• Inputs: dataset eth-sri/SWT-bench_Verified_bm25_27k_zsp, split test, explicit 433 instance IDs, max-workers=16, agent-build-batch-size=50

I’ll keep watching this rerun to see whether removing all registry cache exports restores cold-build throughput.

[simonrosenberg]

Monitoring update for the rerun with registry cache export fully disabled:

• Run: https://github.com/OpenHands/benchmarks/actions/runs/23033839708
• Benchmarks ref: fix/issue-504-cached-sdist @ 277cb51be339c3f7fda1705be90cfa05840b3d81
• SDK ref: 8e8223b24dfd041875fe845b13326b0a1238b793
• Result: the run was cancelled externally, not failed by a build error

What the logs show before cancellation:

• OPENHANDS_BUILDKIT_CACHE_MODE=off was active
• every build logged cache-to=disabled, shared cache export=disabled
• progress milestones improved sharply versus the previous run:
  • 50/433 at 2026-03-13 03:18:15 UTC (30m 02s)
  • 100/433 at 2026-03-13 04:03:34 UTC (1h 15m 21s)
  • 150/433 at 2026-03-13 04:50:28 UTC (2h 02m 15s)
  • 200/433 at 2026-03-13 05:35:26 UTC (2h 47m 13s)
• BuildKit usage at 200/433 was 37.11% (331.65 GiB / 893.66 GiB)

That is a substantial recovery from the prior run, which only reached 50/433 after about 2h 05m. So disabling all registry cache exports appears to address the main cold-build slowdown.

Next useful step is to rerun the full 433-instance workflow without cancellation to confirm end-to-end completion time under the new settings.

[simonrosenberg]

Updated the image-build workflows to remove the 180-minute job timeout for both SWE-Bench and SWT-Bench image building.

Benchmarks PR update:

• fix: cache SDK sdist across image builds #507
• new head: 039aebd2fa315bcbf738cbbf5c92ad8a90bb4c75

What changed:

• removed timeout-minutes: 180 from .github/workflows/build-swebench-images.yml
• removed timeout-minutes: 180 from .github/workflows/build-swtbench-images.yml

Retriggered the full 433-instance SWT image build on the updated branch:

• Run: https://github.com/OpenHands/benchmarks/actions/runs/23043936501
• Status: queued
• Benchmarks ref: fix/issue-504-cached-sdist @ 039aebd2fa315bcbf738cbbf5c92ad8a90bb4c75
• SDK ref: 8e8223b24dfd041875fe845b13326b0a1238b793
• Inputs: dataset eth-sri/SWT-bench_Verified_bm25_27k_zsp, split test, explicit 433 instance IDs, max-workers=16, agent-build-batch-size=50

This rerun should now be able to finish without the previous 3-hour cancellation.

[simonrosenberg]

I re-checked the timeout evidence currently cited by PR #520 and it is not a valid cold-build reference.

Why the current timeout note is wrong

The workflow comment points at run 23043936501 and says a full 433-image SWT-Bench build completed in 4:10:33.

After checking the actual run metadata and raw job log:

• Run: https://github.com/OpenHands/benchmarks/actions/runs/23043936501
• Created: 2026-03-13 09:09:56 UTC
• Job started: 2026-03-13 09:10:07 UTC
• Job completed: 2026-03-13 13:42:04 UTC
• The Build and push SWT-Bench images step ran from 2026-03-13 09:13:32 UTC to 2026-03-13 13:41:49 UTC, which is about 4h28m17s, not 4:10:33.

More importantly, this run was not cold:

• The run log contains 203 lines of Image ... already exists. Skipping build.
• The manifest has 433 successful rows, but that includes skips.
• So this run built about 230 images and skipped 203.

That means the PR comment is using a mixed warm run and treating successful as built.

Why the run was warm

The same SDK image namespace had already been seeded earlier:

• Cancelled precursor: 23033839708 on 2026-03-13 using SDK 8e8223b
• Mixed run cited by PR: 23043936501 on 2026-03-13 using the same SDK 8e8223b

So 23043936501 could reuse images pushed earlier under the same tag prefix.

The same pattern appears in later "fast" runs too:

• 23076867726 on 2026-03-14 looks fast by wall clock, but the job log shows 433 skip lines and 0 actual builds.
• 23143244162 on 2026-03-16 built only 5 images.
• 23143972500 on 2026-03-16 then skipped all 5 of those images.

These runs should not be used to estimate full cold-build duration.

Cold / mixed runs that are actually useful

From the raw run logs and local manifests:

• 23001334237 (2026-03-12)
  • total manifest rows: 113
  • skip lines: 1
  • actual builds: 112
  • job runtime: about 1h35m32s
• 23008709458 (2026-03-12)
  • total manifest rows: 170
  • skip lines: 12
  • actual builds: 158
  • job runtime: about 2h58m01s
• 23017371689 (2026-03-12, shared-cache regression run)
  • total manifest rows: 71
  • skip lines: 27
  • actual builds: 44
  • job runtime: about 3h01m27s
• 23043936501 (2026-03-13, run currently cited by PR Increase SWT-Bench image build timeout to 9 hours #520)
  • total manifest rows: 433
  • skip lines: 203
  • actual builds: about 230
  • build-step runtime: about 4h28m17s

Full cold-build estimate

If we correct 23043936501 for the skipped images, it implies:

• 230 real builds in 4h28m17s
• projected to 433 cold builds: about 8h25m

That matches the near-cold March runs reasonably well:

• 23001334237 implies something in the high-7h to low-8h range
• 23008709458 projects to about 8h08m

So the defensible operational estimate for a full cold SWT-Bench build on the current degraded path is roughly 8h to 8.5h, not 4h10m.

Practical implication for workflow timeout

The workflow should not justify its timeout using run 23043936501 as a full cold build.

A safer timeout is 10h (600 minutes), with the comment pointing back to this issue and explicitly noting that:

• the previous evidence was contaminated by skipped prebuilt images
• recent fast runs were warm or tiny-subset runs
• full cold builds currently appear to need about 8h to 8.5h

This keeps the timeout comfortably above the observed degraded cold-build envelope while the underlying regression is still being fixed.

[simonrosenberg]

A/B Test Results: Cache Export Mode Comparison

I conducted an A/B test to validate whether disabling cache export (as suggested in this comment) would improve SWT-Bench image build throughput.

Test Configuration

Parameter	Run 1 (mode=min)	Run 2 (mode=off)
Run ID	23198836788	23198839061
Branch	fix/swtbench-build-parallelism	fix/swtbench-build-parallelism-cache-off
SDK Commit	c34cb27	c34cb27
Cache Mode	OPENHANDS_BUILDKIT_CACHE_MODE=min	OPENHANDS_BUILDKIT_CACHE_MODE=off
Max Workers	16	16
Build Batch Size	50	50

The SDK changes add support for the OPENHANDS_BUILDKIT_CACHE_MODE environment variable (PR #2479), which controls BuildKit cache export behavior:

• max (default): Export all layers to registry cache
• min: Export only final stage layers
• off: Disable cache export entirely

Results Summary

Both runs timed out after 10 hours, building identical counts:

Metric	Mode=min	Mode=off
Images Built	343	348
Images Skipped	7	1
Images Failed	0	1
Total Processed	350/433 (80.8%)	350/433 (80.8%)
Avg Duration/Image	1,461s	1,432s
Throughput	~35 images/hour	~35 images/hour

Key Finding: Significant Build Time Degradation During Runs

Both runs showed ~82% slowdown from batch 1 to batch 6:

Batch	Mode=off	Mode=min	Slowdown vs Batch 1
1 (images 1-50)	1,038s	1,067s	baseline
2 (images 51-100)	1,200s	1,418s	+16% / +33%
3 (images 101-150)	1,435s	1,240s	+38% / +16%
4 (images 151-200)	1,407s	1,608s	+36% / +51%
5 (images 201-250)	1,537s	1,337s	+48% / +25%
6 (images 251-300)	1,886s	1,948s	+82% / +83%
7 (images 301-350)	1,502s	1,478s	+45% / +39%

Root Cause Analysis

1. BuildKit Failures Requiring Resets

From the logs, I found significant BuildKit instability:

Run	Buildx Resets	Full Resets	Partial Resets
Mode=off	10	2	8
Mode=min	3	?	?

Example reset events from run 23198839061:

15:47:11 - Resetting buildx (full) after BuildKit failure
15:53:01 - Resetting buildx (full) after BuildKit failure

2. Disk Pressure and Prune Operations

Batch	Disk Usage	Notes
After Batch 1	28.64%	No prune needed
After Batch 2	39.74%	No prune needed
After Batch 3	51.20%	No prune needed
After Batch 4	62.68%	Triggered prune (>60% threshold)

The prune command initially failed:

19:06:10 - Pruning BuildKit cache: docker buildx prune --all --force --max-storage 60g
19:06:11 - unknown flag: --max-storage
19:06:11 - Pruning BuildKit cache: docker buildx prune --all --force --keep-storage 60g

3. Cache Export Had Zero Measured Impact

Both runs showed sdk_cache_export_seconds: 0 in the manifest, meaning:

• The cache export mode changes were working (no time spent on cache export)
• But throughput was still poor, indicating cache export was not the primary bottleneck in these test runs

Comparison to Historical Baselines

Run	SDK	Images/hour (to 50)	Relative Speed
21870877504	cfe52af (Feb baseline)	~162	1.0x (baseline)
23198839061	c34cb27 (mode=off test)	~52	3.1x slower
23198836788	c34cb27 (mode=min test)	~50	3.2x slower

Conclusions

1. Cache export mode (min vs off) had negligible impact - Both achieved identical results (350/433 images in 10 hours)

2. The slowdown is NOT primarily caused by cache export - Even with cache export completely disabled (mode=off), builds are still ~3x slower than the Feb baseline

3. BuildKit instability is a major factor - 10 buildx resets in mode=off run, correlating with batch slowdowns

4. Disk pressure contributes to degradation - Batch 6 (slowest at +82%) occurred after prune was triggered at 62% disk

5. The --max-storage flag bug should be fixed - prune command failed initially

Recommendations

1. Investigate BuildKit failures - Why are there 10+ resets per run? This is the most actionable finding.

2. Consider reducing parallelism - max-workers=16 may be overwhelming BuildKit, causing failures

3. Fix the prune command - The --max-storage flag is not supported; ensure fallback works correctly

4. Increase timeout temporarily - To complete 433 images at current rate (~35/hour), need ~12.4 hours

5. The cache mode fix can still be merged - While it didn't solve the problem, mode=off eliminates unnecessary cache export overhead and showed marginally better results (348 vs 343 built, fewer skips)

Artifacts

Build manifests are available in the run artifacts:

• build-logs-23198836788 (mode=min)
• build-logs-23198839061 (mode=off)

Related PRs

• SDK: OpenHands/software-agent-sdk#2479 - Adds OPENHANDS_BUILDKIT_CACHE_MODE support
• Benchmarks: OpenHands/benchmarks#525 - Updates workflow with cache mode and parallelism settings