Split base image from SDK layer to avoid full 9h+ rebuilds on SDK changes

[VascoSch92]

Context

SDK PR #2522 (ARG ordering fix) restored registry cache hits for apt-get/npm layers across SDK bumps, cutting per-image build time from ~322s to ~154s (validation in #544). This recovers the Feb baseline of ~5-6 hours for 433 images.

However, each image still runs a full docker buildx build that pulls the SWE-bench base image, resolves registry cache, and copies the builder output. With 433 images, this adds up.

Proposal: pre-built base images

Split the build into two independently-tagged layers:

Layer 1 — Repo base (rarely changes):

ghcr.io/openhands/eval-base:{SWEBENCH_IMAGE_ID}

Contains everything from the SWE-bench base through apt/npm setup. Only rebuilds when the Dockerfile base layers or upstream SWE-bench image change.

Layer 2 — SDK agent (changes per SDK commit):

FROM ghcr.io/openhands/eval-base:${BASE_TAG}
COPY --from=builder /agent-server /agent-server
ENTRYPOINT ["/agent-server/.venv/bin/python", "-m", "openhands.agent_server"]

Just a COPY onto the cached base — ~5-10s per image.

Expected impact

Scenario	Current (with ARG fix)	With base split
Same SDK (all cached in GHCR)	3-12 min	3-12 min
New SDK, same SWE-bench instances	~5-6 hours	~30-45 min
New SDK + new SWE-bench base	~5-6 hours	~5-6 hours

The common case (new SDK commit, same instances) would go from hours to under an hour.

Trade-offs

• Adds operational complexity: two image registries, invalidation logic for base images
• Requires changes to build_utils.py and the workflow
• The ARG fix already handles the regression; this is a further optimization

Related

• Investigation: Root cause of per-image build time doubling (SDK cefaebf → eab666f) #544 — Root cause investigation (ARG ordering)
• SWT-Bench image building slowness: root cause analysis and fix plan #531 — Master tracker
• SDK #2522 — ARG fix (merged)

[simonrosenberg]

Implementation & Initial Results

PRs

• SDK: feat: support pre-built base images for faster rebuilds software-agent-sdk#2542 — Adds Dockerfile ARGs (SOURCE_MINIMAL_BASE, SOURCE_BASE), prebuilt_base field on BuildOptions, and base-image-minimal / base-image as valid build targets
• Benchmarks: feat: split base image from SDK layer for faster SWE-bench rebuilds #550 — New build_base_images.py script, --use-prebuilt-bases flag, two-phase workflow

How it works

Phase 1 builds and pushes ghcr.io/openhands/eval-base:{swebench_image_id} for each SWE-bench instance using --target base-image-minimal from the SDK Dockerfile. These are SDK-independent and only need rebuilding when upstream SWE-bench images change.

Phase 2 passes pre-built base refs via `BuildOptions.prebuilt_base` → SDK Dockerfile's `SOURCE_MINIMAL_BASE` ARG, causing Docker to skip the `base-image-minimal` stage entirely and pull the cached base from GHCR instead.

Validation: 100-image run

Two-phase build (run 23367721674):

Phase	Duration	Notes
Phase 1 (base images)	8 seconds	All 100 bases already in registry → skipped
Phase 2 (agent images)	42 minutes	Pre-built bases + cache export disabled
Total	~42 minutes

Telemetry from first run (before cache export fix, run 23363226647):
Each image was spending 67–109s on cache_export and 72–168s on image_export — the builder stage was cached (14/14 steps), but registry I/O dominated. Setting OPENHANDS_BUILDKIT_CACHE_MODE=off in Phase 2 cut per-image buildx wall-clock from ~650s to ~250s.

Baseline comparison

Run	Scenario	Images	Duration
23376566632	Single-phase, new SDK, all images	~433	5h 57min
23367721674	Two-phase, new SDK, pre-built bases	100	42 min

Extrapolated: ~433 images with two-phase should take ~3 hours (vs ~6h single-phase). Actual full run in progress now: run 23382573066.

Key insight

The bottleneck wasn't the Docker build itself — BuildKit cached 14/14 steps including the builder stage. The bottleneck was registry I/O: cache export (~100s) + image export (~160s) per image, multiplied by 12 concurrent workers creating massive contention. Disabling cache export in Phase 2 (safe because the builder is already cached from the first image) was the critical optimization.

[simonrosenberg]

Full-scale runs triggered

SWE-bench (all ~433 images)

• Two-phase: run 23382573066 — in progress
• Uses use-prebuilt-bases=true, base images already in registry from prior 100-image run

SWT-bench (all ~433 images)

• Two-phase: run 23382780015 — just started
• First run with two-phase for SWT-bench, so Phase 1 will build base images from scratch
• Baseline comparison: run 23354696949 took 5h 39min (single-phase)

Will update with timing results once both complete.

[simonrosenberg]

Full-scale results & honest assessment

Both full runs completed successfully (zero failures):

• SWE-bench: run 23382573066 — 500 images
• SWT-bench: run 23382780015 — 433 images

Per-image build time comparison

Run	Images built	Avg build/image	Phase 2 wall clock
Baseline single-phase (23376566632)	500 built, 0 skipped	352s	5h 57min
SWE-bench two-phase (23382573066)	321 built, 179 skipped	407s	5h 1min
SWT-bench two-phase (23382780015)	338 built, 95 skipped	424s	5h 6min

Assessment: marginal improvement, not the expected speedup

The per-image build time is actually slower with pre-built bases (407s vs 352s). Skipping base-image-minimal saves ~60s of apt-get/npm time, but this is offset by the overhead of pulling the pre-built base from GHCR.

The original proposal estimated "~5-10s per image" with pre-built bases. That assumed the COPY --from=builder would be the only cost. In reality, the per-image telemetry shows:

build_context_seconds: 0.04    # cached sdist — fast ✓
cache_import_seconds:  16      # pulling registry cache
image_export_seconds:  70-100  # pushing the unique combined image to GHCR ← bottleneck
push_layers_seconds:   20-50   # uploading layers
export_manifest_seconds: 5     # manifest push
cached_step_count:     11      # most build steps cached ✓
cache_export_seconds:  0       # disabled in Phase 2 ✓

The Docker build itself is fast (~5s, all cached). The bottleneck is registry I/O (~150s/image) — each of the 500 images is unique and must be pushed to GHCR. This is irreducible regardless of whether the base was pre-built or built from scratch.

Where the strategy does help

1. Phase 1 skip on subsequent runs: When base images already exist, Phase 1 completes in ~8 seconds (verified in run 23367721674)
2. Base-only refreshes: If upstream SWE-bench images change, bases can be rebuilt independently without rebuilding all agent images
3. Operational separation: Decouples base image lifecycle from SDK release cycle

What would actually move the needle

The real bottleneck is per-image registry I/O, not Docker build time. To get to the "30-45 min for 433 images" target from the original proposal, we'd need to either:

• Reduce the number of unique images pushed (e.g., shared builder layer as a separate image, with thin per-instance layers that only contain the base diff)
• Increase parallelism beyond 12 workers (limited by runner resources and GHCR rate limits)
• Reduce image size to speed up push time (~2GB per image currently)

Recommendation

The prebuilt-base PRs work correctly and the code is clean, but the performance improvement doesn't justify the added complexity for the common "new SDK, same instances" case. I'd recommend closing this unless the base-only refresh use case is valuable on its own.

PRs: OpenHands/software-agent-sdk#2542, #550

[simonrosenberg]

Head-to-head comparison: two-phase vs three-phase (force-rebuild, all images)

All runs use force-build=true so every image is rebuilt — no skipping.

Two-phase runs (branch: feat/prebuilt-base-two-phase)

Pre-built bases + cache export disabled, but still runs full docker buildx build per image (builder stage cached via registry).

• SWE-bench (500 images): run 23401832421
• SWT-bench (433 images): run 23401832742

Three-phase runs (branch: feat/prebuilt-base-images)

Pre-pushes the builder image too, then assembles final images via thin Dockerfile.agent-layer (FROM base + COPY --from=builder). GHCR cross-repo blob mounting should avoid re-uploading shared layers.

• SWE-bench (500 images): run 23407020785
• SWT-bench (433 images): run 23407021181

Baseline

Run	Images	Duration
SWE-bench single-phase (23376566632)	500	5h 57min
SWT-bench single-phase (23354696949)	433	5h 39min

Will update with final timing once all four complete.

[simonrosenberg]

Three-phase assembly with telemetry (100 images)

• SWT-bench three-phase: run 23409630540 — 100 images, force-rebuild, with BuildKit telemetry

Early telemetry shows ~580s/image for the thin Dockerfile assembly. The _parse_buildkit_telemetry captures image_export (~110s) and push_layers (~15s), but ~430s is unaccounted — likely layer pulls to the remote Blacksmith buildx driver. cached_step_count: 0 confirms nothing is cached on the remote builder between invocations.

Next experiment: try docker build with a local Docker daemon instead of the remote Blacksmith buildx driver.

[simonrosenberg]

Breakthrough: local docker build + push (three-phase)

Replaced docker buildx build --push (remote Blacksmith driver) with docker build + docker push (local daemon) for Phase 2 assembly.

Result: run 23409964288 — 100 images, 13 minutes total

Phase	Duration
Phase 0 (builder)	3s (cached)
Phase 1 (bases)	9s (cached)
Phase 2 (assembly, 100 images)	12min 26s
Total	~13 minutes

Per-image telemetry

Batch	total	build	push	Notes
First 12 (cold)	95s	63-71s	24-32s	Pulling base+builder to local daemon
Subsequent	30s	25s	5s	Layers cached locally

Why it's fast

The remote Blacksmith buildx driver was pulling ~3.5GB (base+builder) from GHCR to a remote builder for every image, then pushing ~2GB back. With local docker build, the base and builder images are pulled once and cached in the local Docker daemon. Subsequent builds reuse cached layers — only the new COPY layer + push is needed.

Projected full scale (433 images)

12 images × 95s + 421 images × 30s / 12 workers ≈ 18 minutes

Comparison

Approach	100 images	Projected 433
Baseline single-phase	~72min	~6h
Two-phase (prebuilt bases via buildx)	~42min	~5h
Three-phase (buildx remote driver)	~1.5h	~5h
Three-phase (local docker build)	13min	~18min

Branch: feat/prebuilt-base-local-build

[simonrosenberg]

Correction: full telemetry analysis for local docker build

My previous comment cherry-picked the fast cached results. Here's the actual distribution across all 100 images from run 23409964288:

Per-image timing (100 images, 0 failures)

Metric	Avg	Min	Max
Total	87s	30s	309s
Build (docker build)	79s	25s	305s
Push (docker push)	8s	3s	32s

Distribution

Bucket	Count
20-40s	5 images (locally cached base)
40-60s	27 images
60-100s	51 images
>100s	17 images (large/uncached bases)

Comparison (all approaches, "new SDK, same instances" scenario)

Approach	Avg/image	100 images	Projected 433
Baseline single-phase (buildx)	352s	~72min	~6h
Two-phase (buildx remote)	411s	~5h 57min*	~5h 57min
Three-phase (buildx remote)	580s	~1.5h	~5h
Three-phase (local docker)	87s	13min	~52min

* Two-phase force-rebuild, no images skipped

Why local docker is faster

• Build (79s avg): Local daemon caches pulled base/builder layers. First pull of each unique base is ~70-100s, subsequent images sharing layers are ~25-40s.
• Push (8s avg): docker push only uploads layers not already in GHCR. Base layers (from Phase 1) are mounted cross-repo. Only the COPY layer (~500MB) is uploaded.
• No remote driver overhead: Blacksmith buildx sends layers to/from a remote builder for each invocation (~400s overhead eliminated).

Key insight

The Blacksmith remote buildx driver is optimized for complex multi-stage builds where BuildKit cache stickiness matters. For thin assembly Dockerfiles with pre-built inputs, a local docker build + docker push is 4x faster because it avoids the remote network round-trip entirely.

Branch: feat/prebuilt-base-local-build