build: bundle iceberg extension in worker images#645
Merged
Conversation
Iceberg extension was downloaded on-demand at first use, unlike httpfs, ducklake, json, and postgres_scanner which the Dockerfiles pre-seed into the bundled extension cache. That on-demand INSTALL silently blocks the iceberg-only tenant activation past the ~60s activate-tenant deadline (observed on mw-dev with the per-step logging from #642: count-catalogs completes in ~1ms, load-iceberg-extension never returns, worker is retired at 60.7s with no DuckDB-level error). Iceberg+DuckLake tenants don't hit it because LoadExtensions(delta) runs first and primes DuckDB's extension subsystem. Bundle iceberg the same way as the others: curl the .duckdb_extension.gz from ${DUCKDB_EXTENSION_REPOSITORY} at build time, gunzip into /build/duckdb-extensions/v${DUCKDB_EXTENSION_VERSION}/linux_${TARGETARCH}/, and add it to the size-check loop. Applies to both the standalone Dockerfile and Dockerfile.worker so worker pods get a local cache hit on LoadExtensions("iceberg") instead of a CDN fetch. Eliminates the iceberg-only activation timeout and brings the activation cost of LoadExtensions("iceberg") in line with the other bundled extensions for every iceberg-using tenant (lakekeeper + s3tables backends alike).
benben
added a commit
that referenced
this pull request
Jun 1, 2026
LoadExtensions skips INSTALL whenever a .duckdb_extension binary is sitting in /app/extensions (preseeded from the Dockerfile). That works for the PostHog-fork extensions (httpfs, ducklake) and the stable stock ones (json, postgres_scanner): LOAD against the seeded extension_directory file just works. It does NOT work for iceberg. After #645 bundled iceberg, the bundle-skip turned db.Exec("LOAD iceberg") into the same ~60s silent hang the bundle was meant to fix: the bundled binary is in the cache, but without a prior INSTALL DuckDB's internal extension metadata leaves LOAD blocked indefinitely (observed live in mw-dev — worker logs the load-iceberg-extension step start and then no further activity until the activate-tenant deadline kills the worker). Special-case iceberg in shouldInstallExtension so INSTALL runs even when bundled. INSTALL with the binary already in extension_directory is a cheap no-op — DuckDB sees the cached file and skips the CDN download — so the bundle benefit is preserved while LOAD now finds the extension already installed. The upstream-overwrite risk that motivates skipping INSTALL for the PostHog forks doesn't apply: we bundle the same stock iceberg build the DuckDB repository would serve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Iceberg-only tenant activation silently exceeds the ~60s activate-tenant deadline and fails (
DeadlineExceeded), as observed on mw-dev (ben-{cnpg,ext,aur}-ice). Iceberg + DuckLake tenants are unaffected.Pinpointed via #642's per-step instrumentation on a real worker:
LoadExtensions(["iceberg"])blocks for the entire deadline.Why iceberg-only
The four DuckDB extensions duckgres relies on (
httpfs,ducklake,json,postgres_scanner) are pre-seeded into the bundled extension cache byDockerfile/Dockerfile.worker.icebergis not — it getsINSTALLed on-demand at first attach, fetching fromextensions.duckdb.org.LoadExtensions("iceberg")is the first network extension install of the worker process → silent cold install/download hangs past the activate deadline.LoadExtensions("delta")runs first (viaAttachDeltaCatalog), priming DuckDB's extension subsystem; the subsequentLoadExtensions("iceberg")finishes within the deadline (verified live with the existing "both" combos — iceberg writes succeed end-to-end).Worker→CDN reachability is fine (
extensions.duckdb.org:443OPEN from a born-as-worker pod) and worker→Lakekeeper is fine (#11444). The hang is inside DuckDB's INSTALL path.Fix
Bundle
icebergat image build time, identical to the other core-repo extensions:Applied to both
DockerfileandDockerfile.worker. SubsequentLoadExtensions("iceberg")becomes a local cache hit — same path as the other bundled extensions — for every iceberg-using tenant (Lakekeeper + S3Tables backends alike).Test
Once the new image is deployed to mw-dev I'll re-run
ben-ext-iceand confirm:step=load-iceberg-extensionfinishes in ms instead of timing outSELECT 1+CREATE TABLE iceberg.public.<t> / INSERT / SELECTround-trip succeeds🤖 Generated with Claude Code