experimental/ssh: upload connect binaries over HTTP/1.1#5703
Open
anton-107 wants to merge 1 commit into
Open
Conversation
`databricks ssh connect` uploads the ~14 MB CLI bundle to the workspace files import-file API during the "Uploading binaries..." step. Some corporate proxies reset large HTTP/2 request bodies with RST_STREAM(NO_ERROR), surfacing as `stream error: stream ID N; NO_ERROR; received from peer` and aborting the connection. A single large POST gains nothing from HTTP/2, so force HTTP/1.1 for this one upload client, which sidesteps the HTTP/2 interop entirely while leaving the rest of the connect flow on HTTP/2. Add filer.NewWorkspaceFilesClientWithClient so the upload can run on a dedicated HTTP/1.1 client without copying or mutating the shared config. Broaden the upload-error hint to also cover the HTTP/1.1 signatures of a genuine intermediary body-size limit (413, connection reset), since those (not the HTTP/2 reset) are what remains after forcing HTTP/1.1. See DECO-27497. Co-authored-by: Isaac
anton-107
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
anton-107
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Collaborator
Integration test reportCommit: 370cd2b
22 interesting tests: 13 SKIP, 7 KNOWN, 2 flaky
Top 3 slowest tests (at least 2 minutes):
|
Contributor
Approval status: pending
|
anton-107
requested review from
ilia-db,
pietern,
renaudhartert-db,
rugpanov and
simonfaltum
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
databricks ssh connectuploads the ~14 MB CLI bundle to the workspace filesimport-fileAPI during the "Uploading binaries…" step. Behind some corporatenetworks this fails with:
That is an HTTP/2 stream reset: an intermediary (corporate egress proxy / VPN /
WAF / load balancer) tears down the large request body mid-upload. The symptom
has two causes that look identical:
too, only a network-policy change fixes it.
RST_STREAM(NO_ERROR)mid-body) — Go can't retry a partially-sent non-idempotent POST, so it
surfaces a fatal error. HTTP/1.1 avoids this entirely. A customer
confirmed
GODEBUG=http2client=0(force HTTP/1.1) fixed the upload with nonetwork change.
A single large POST gains nothing from HTTP/2, so we can sidestep the whole
HTTP/2-reset class for this one upload.
What
upload; the rest of the connect flow keeps using HTTP/2.
filer.NewWorkspaceFilesClientWithClientso the upload can run on adedicated client without copying or mutating the shared
config.Config(it embeds a
sync.Mutex).newHTTP11Transportclones the resolved config's transport and disablesHTTP/2 via a non-nil, empty
TLSNextProtomap.isStreamResetError→isProxyUploadError)to also catch the HTTP/1.1 signatures of a genuine intermediary body-size
limit (
413, connection reset). Since the upload now always uses HTTP/1.1,the reworded message no longer points at HTTP/2 and instead tells the user to
ask their network admin to allow large uploads or switch networks.
Testing
isProxyUploadError(413 / connection-reset / HTTP/2 / negatives),newHTTP11Transportdisables HTTP/2,NewWorkspaceFilesClientWithClientwiring.
acceptance/ssh/connectionexercises the full upload over the newclient against the testserver; recorded output is unchanged.
go vetand golangci-lint are clean.See DECO-27497.
This pull request and its description were written by Isaac.