driver/kubernetes: Add retry logic for transient TLS connection errors #3493
+68
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes docker#2668 When Kubernetes marks nodes as "Ready" before their Certificate Signing Requests (CSRs) are approved, the buildx kubernetes driver can fail to connect to builder pods with transient TLS errors like: - "tls: internal error" - "context deadline exceeded" - "use of closed network connection" - "i/o timeout" This is particularly problematic on EKS clusters with ARM64 nodes under heavy load, where multiple builders are being spawned simultaneously. This commit adds retry logic with exponential backoff to the Dial() function in the kubernetes driver. The implementation: - Attempts up to 5 connection retries - Uses exponential backoff starting at 500ms, capped at 10s - Only retries on known transient connection errors - Logs retry attempts to stderr for visibility - Respects context cancellation This allows buildx to gracefully handle the race condition where pods are marked as Running before their TLS certificates are fully ready. Signed-off-by: guimove <dasilva.guillaume@live.fr>
Guimove
force-pushed
the
fix-k8s-tls-retry-2668
branch
from
0780f28 to
71d3044
Compare
tonistiigi
reviewed
Oct 29, 2025
- Use logrus for logging instead of fmt.Fprintf - Remove unnecessary waitWithContext helper function - Refactor to use tryWithBackoff pattern with closure to avoid many parameters Co-authored-by: Tonis Tiigi <tonistiigi@gmail.com> Signed-off-by: guimove <dasilva.guillaume@live.fr>
Address gopls modernize analyzer suggestions: - Use range over int for retry loop (Go 1.22+) - Use built-in min() function instead of if statement Signed-off-by: Guillaume Da Silva <dasilva.guillaume@live.fr> Signed-off-by: guimove <dasilva.guillaume@live.fr>
3 tasks
AkihiroSuda
reviewed
Dec 24, 2025
| strings.Contains(errStr, "use of closed network connection") || | ||
| strings.Contains(errStr, "i/o timeout") || | ||
| strings.Contains(errStr, "connection refused") || | ||
| strings.Contains(errStr, "connection reset") |
Collaborator
There was a problem hiding this comment.
Ideally we should use errors.Is (or errors.As)
AkihiroSuda
approved these changes
Dec 24, 2025
Collaborator
|
Please squash the commits |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #2668
When Kubernetes marks nodes as "Ready" before their Certificate Signing Requests (CSRs) are approved, the buildx kubernetes driver can fail to connect to builder pods with transient TLS errors like:
This is particularly problematic on EKS clusters with ARM64 nodes under heavy load, where multiple builders are being spawned simultaneously.
This commit adds retry logic with exponential backoff to the Dial() function in the kubernetes driver. The implementation:
This allows buildx to gracefully handle the race condition where pods are marked as Running before their TLS certificates are fully ready.