Skip to content

Bug: Container registry proxy never caches artifacts due to missing auth token #43

@ignusws

Description

@ignusws

Description

When using the proxy as a caching layer for Docker / OCI registries, artifacts are always fetched from the upstream registry and never served from cache.

Expected behavior

The proxy should:

Fetch artifacts from upstream on first request

  • Store them in cache
  • Serve subsequent requests from cache without contacting upstream

Actual behavior

The proxy always fetches artifacts from upstream and never appears to cache them.

In the container handler implementation:

https://github.com/git-pkgs/proxy/blob/main/internal/handler/container.go#L78

the following function is called:

h.proxy.GetOrFetchArtifactFromURL(...)

However, this call does not include an authentication token.

As a result:

  • The upstream registry (e.g. Docker Hub) responds with 401 Unauthorized
  • GetOrFetchArtifactFromURL fails
  • The code falls back to: h.proxyBlobWithAuth(...)

This fallback path correctly includes the Bearer token and successfully fetches the artifact, but bypasses the caching mechanism entirely

Container logs with some debugging
proxy-postgres-1  | time=2026-03-22T13:34:06.702Z level=INFO msg="starting server" listen=:8080 base_url=http://localhost:8081 storage=/data/artifacts database=./cache/proxy.db
proxy-postgres-1  | time=2026-03-22T13:34:24.832Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000001 method=GET path=/v2/ status=200 duration=35.797µs remote=172.22.0.1:58158
proxy-postgres-1  | time=2026-03-22T13:34:24.833Z level=INFO msg="container manifest request" name=library/nginx reference=1.27
proxy-postgres-1  | time=2026-03-22T13:34:25.991Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000002 method=HEAD path=/v2/library/nginx/manifests/1.27 status=200 duration=1.15784739s remote=172.22.0.1:58164
proxy-postgres-1  | time=2026-03-22T13:34:26.014Z level=INFO msg="container manifest request" name=library/nginx reference=sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7d
proxy-postgres-1  | time=2026-03-22T13:34:26.444Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000003 method=GET path=/v2/library/nginx/manifests/sha256:6784fb0834aa7dbbe12e3d7471e69c290df3e6ba810dc38b34ae33d3c1c05f7d status=200 duration=429.981563ms remote=172.22.0.1:58176
proxy-postgres-1  | time=2026-03-22T13:34:26.469Z level=INFO msg="container manifest request" name=library/nginx reference=sha256:fafd5f55296511cd0f1c991efa0cb85f323449bf643f3b6d760c6542f020d6e5
proxy-postgres-1  | time=2026-03-22T13:34:26.921Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000004 method=GET path=/v2/library/nginx/manifests/sha256:fafd5f55296511cd0f1c991efa0cb85f323449bf643f3b6d760c6542f020d6e5 status=200 duration=452.203045ms remote=172.22.0.1:58192
proxy-postgres-1  | time=2026-03-22T13:34:26.937Z level=INFO msg="container blob request" name=library/nginx digest=sha256:1e5f3c5b981a9f91ca91cf13ce87c2eedfc7a083f4f279552084dd08fc477512
proxy-postgres-1  | time=2026-03-22T13:34:26.937Z level=INFO msg="container blob request" name=library/nginx digest=sha256:dad67da3f26bce15939543965e09c4059533b025f707aad72ed3d3f3a09c66f8
proxy-postgres-1  | time=2026-03-22T13:34:26.937Z level=INFO msg="container blob request" name=library/nginx digest=sha256:56b81cfa547d78176e719c9cd387e35fbd95ce6e8500efb26d9fbcbac4dbc4f6
proxy-postgres-1  | time=2026-03-22T13:34:26.937Z level=INFO msg="container blob request" name=library/nginx digest=sha256:3b00567da96412a0298328fad6b96aadd3bc9ca97e6683534da152b0b9e8a977
proxy-postgres-1  | time=2026-03-22T13:34:27.126Z level=INFO msg="fetching from upstream" ecosystem=oci name=library/nginx version=sha256:1e5f3c5b981a9f91ca91cf13ce87c2eedfc7a083f4f279552084dd08fc477512 url=https://registry-1.docker.io/v2/library/nginx/blobs/sha256:1e5f3c5b981a9f91ca91cf13ce87c2eedfc7a083f4f279552084dd08fc477512
proxy-postgres-1  | time=2026-03-22T13:34:27.132Z level=INFO msg="fetching from upstream" ecosystem=oci name=library/nginx version=sha256:3b00567da96412a0298328fad6b96aadd3bc9ca97e6683534da152b0b9e8a977 url=https://registry-1.docker.io/v2/library/nginx/blobs/sha256:3b00567da96412a0298328fad6b96aadd3bc9ca97e6683534da152b0b9e8a977
proxy-postgres-1  | time=2026-03-22T13:34:27.136Z level=INFO msg="fetching from upstream" ecosystem=oci name=library/nginx version=sha256:56b81cfa547d78176e719c9cd387e35fbd95ce6e8500efb26d9fbcbac4dbc4f6 url=https://registry-1.docker.io/v2/library/nginx/blobs/sha256:56b81cfa547d78176e719c9cd387e35fbd95ce6e8500efb26d9fbcbac4dbc4f6
proxy-postgres-1  | time=2026-03-22T13:34:27.149Z level=INFO msg="fetching from upstream" ecosystem=oci name=library/nginx version=sha256:dad67da3f26bce15939543965e09c4059533b025f707aad72ed3d3f3a09c66f8 url=https://registry-1.docker.io/v2/library/nginx/blobs/sha256:dad67da3f26bce15939543965e09c4059533b025f707aad72ed3d3f3a09c66f8
proxy-postgres-1  | time=2026-03-22T13:34:27.653Z level=ERROR msg="failed to fetch from cache" error="fetching from upstream: unexpected status 401: {\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required\",\"detail\":[{\"Type\":\"repository\",\"Class\":\"\",\"Name\":\"library/nginx\",\"Action\":\"pull\"}]}]}\n"
proxy-postgres-1  | time=2026-03-22T13:34:27.655Z level=ERROR msg="failed to fetch from cache" error="fetching from upstream: unexpected status 401: {\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required\",\"detail\":[{\"Type\":\"repository\",\"Class\":\"\",\"Name\":\"library/nginx\",\"Action\":\"pull\"}]}]}\n"
proxy-postgres-1  | time=2026-03-22T13:34:27.657Z level=ERROR msg="failed to fetch from cache" error="fetching from upstream: unexpected status 401: {\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required\",\"detail\":[{\"Type\":\"repository\",\"Class\":\"\",\"Name\":\"library/nginx\",\"Action\":\"pull\"}]}]}\n"
proxy-postgres-1  | time=2026-03-22T13:34:27.675Z level=ERROR msg="failed to fetch from cache" error="fetching from upstream: unexpected status 401: {\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required\",\"detail\":[{\"Type\":\"repository\",\"Class\":\"\",\"Name\":\"library/nginx\",\"Action\":\"pull\"}]}]}\n"
proxy-postgres-1  | time=2026-03-22T13:34:28.317Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000005 method=GET path=/v2/library/nginx/blobs/sha256:1e5f3c5b981a9f91ca91cf13ce87c2eedfc7a083f4f279552084dd08fc477512 status=200 duration=1.379954325s remote=172.22.0.1:58198
proxy-postgres-1  | time=2026-03-22T13:34:28.330Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000007 method=GET path=/v2/library/nginx/blobs/sha256:56b81cfa547d78176e719c9cd387e35fbd95ce6e8500efb26d9fbcbac4dbc4f6 status=200 duration=1.392982615s remote=172.22.0.1:58210
proxy-postgres-1  | time=2026-03-22T13:34:28.331Z level=INFO msg="container blob request" name=library/nginx digest=sha256:1bc5dc8b475d9de62c94cf69a139df17aaf7fe7bac76b5133d8048a058fb3c3b
proxy-postgres-1  | time=2026-03-22T13:34:28.529Z level=INFO msg="fetching from upstream" ecosystem=oci name=library/nginx version=sha256:1bc5dc8b475d9de62c94cf69a139df17aaf7fe7bac76b5133d8048a058fb3c3b url=https://registry-1.docker.io/v2/library/nginx/blobs/sha256:1bc5dc8b475d9de62c94cf69a139df17aaf7fe7bac76b5133d8048a058fb3c3b
proxy-postgres-1  | time=2026-03-22T13:34:28.704Z level=ERROR msg="failed to fetch from cache" error="fetching from upstream: unexpected status 401: {\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required\",\"detail\":[{\"Type\":\"repository\",\"Class\":\"\",\"Name\":\"library/nginx\",\"Action\":\"pull\"}]}]}\n"
proxy-postgres-1  | time=2026-03-22T13:34:29.439Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000009 method=GET path=/v2/library/nginx/blobs/sha256:1bc5dc8b475d9de62c94cf69a139df17aaf7fe7bac76b5133d8048a058fb3c3b status=200 duration=1.108420262s remote=172.22.0.1:58232
proxy-postgres-1  | time=2026-03-22T13:34:29.440Z level=INFO msg="container blob request" name=library/nginx digest=sha256:979e6233a40af9a51ee2c9894bd42ebf78a79c87cbed70cc6a3840a0952f3d57
proxy-postgres-1  | time=2026-03-22T13:34:29.631Z level=INFO msg="fetching from upstream" ecosystem=oci name=library/nginx version=sha256:979e6233a40af9a51ee2c9894bd42ebf78a79c87cbed70cc6a3840a0952f3d57 url=https://registry-1.docker.io/v2/library/nginx/blobs/sha256:979e6233a40af9a51ee2c9894bd42ebf78a79c87cbed70cc6a3840a0952f3d57
proxy-postgres-1  | time=2026-03-22T13:34:29.631Z level=ERROR msg="failed to fetch from cache" error="fetching from upstream: circuit breaker open for registry registry-1.docker.io: upstream registry unavailable"
proxy-postgres-1  | time=2026-03-22T13:34:30.066Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000010 method=GET path=/v2/library/nginx/blobs/sha256:979e6233a40af9a51ee2c9894bd42ebf78a79c87cbed70cc6a3840a0952f3d57 status=200 duration=625.73595ms remote=172.22.0.1:58244
proxy-postgres-1  | time=2026-03-22T13:34:30.067Z level=INFO msg="container blob request" name=library/nginx digest=sha256:d2a7ba8dbfee4f222961ad1449648ab46a6cea044e4cfb374450325ecee8f67f
proxy-postgres-1  | time=2026-03-22T13:34:30.262Z level=INFO msg="fetching from upstream" ecosystem=oci name=library/nginx version=sha256:d2a7ba8dbfee4f222961ad1449648ab46a6cea044e4cfb374450325ecee8f67f url=https://registry-1.docker.io/v2/library/nginx/blobs/sha256:d2a7ba8dbfee4f222961ad1449648ab46a6cea044e4cfb374450325ecee8f67f
proxy-postgres-1  | time=2026-03-22T13:34:30.262Z level=ERROR msg="failed to fetch from cache" error="fetching from upstream: circuit breaker open for registry registry-1.docker.io: upstream registry unavailable"
proxy-postgres-1  | time=2026-03-22T13:34:31.031Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000011 method=GET path=/v2/library/nginx/blobs/sha256:d2a7ba8dbfee4f222961ad1449648ab46a6cea044e4cfb374450325ecee8f67f status=200 duration=964.709555ms remote=172.22.0.1:58246
proxy-postgres-1  | time=2026-03-22T13:34:31.032Z level=INFO msg="container blob request" name=library/nginx digest=sha256:32e44235e1d55a34be577baa12271f067c0bf79ba4e1ca4e6ec484424540132a
proxy-postgres-1  | time=2026-03-22T13:34:31.229Z level=INFO msg="fetching from upstream" ecosystem=oci name=library/nginx version=sha256:32e44235e1d55a34be577baa12271f067c0bf79ba4e1ca4e6ec484424540132a url=https://registry-1.docker.io/v2/library/nginx/blobs/sha256:32e44235e1d55a34be577baa12271f067c0bf79ba4e1ca4e6ec484424540132a
proxy-postgres-1  | time=2026-03-22T13:34:31.229Z level=ERROR msg="failed to fetch from cache" error="fetching from upstream: circuit breaker open for registry registry-1.docker.io: upstream registry unavailable"
proxy-postgres-1  | time=2026-03-22T13:34:31.694Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000012 method=GET path=/v2/library/nginx/blobs/sha256:32e44235e1d55a34be577baa12271f067c0bf79ba4e1ca4e6ec484424540132a status=200 duration=662.176125ms remote=172.22.0.1:58260
proxy-postgres-1  | time=2026-03-22T13:34:34.099Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000006 method=GET path=/v2/library/nginx/blobs/sha256:dad67da3f26bce15939543965e09c4059533b025f707aad72ed3d3f3a09c66f8 status=200 duration=7.16207573s remote=172.22.0.1:58204
proxy-postgres-1  | time=2026-03-22T13:34:34.529Z level=INFO msg=request request_id=826f2ba24a49/b8R7Bdlk7n-000008 method=GET path=/v2/library/nginx/blobs/sha256:3b00567da96412a0298328fad6b96aadd3bc9ca97e6683534da152b0b9e8a977 status=200 duration=7.592170324s remote=172.22.0.1:58224

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions