Skip to content

Tenant impersonation push stream#7475

Open
SungJin1212 wants to merge 6 commits intocortexproject:masterfrom
SungJin1212:Tenant-impersonation-pushStream
Open

Tenant impersonation push stream#7475
SungJin1212 wants to merge 6 commits intocortexproject:masterfrom
SungJin1212:Tenant-impersonation-pushStream

Conversation

@SungJin1212
Copy link
Copy Markdown
Member

@SungJin1212 SungJin1212 commented May 4, 2026

Problem statement

PushStream in the Ingester trusted req.TenantID from the stream payload without verifying it against the authenticated stream context. An attacker with direct access to the Ingester gRPC port could write metrics to any tenant.

Changes

  1. PushStream now rejects any request where the stream's authenticated orgID is a real tenant ID that doesn't match req.TenantID. It cannot prevent worker ID (ingester-<addr>-stream-push-worker-<N>) spoofing, see TestAttack_DirectGRPC_BypassWithWorkerID.

  2. Add a -distributor.sign-write-requests-keys flag. The Distributor signs each PushStream connection with the first key; the Ingester verifies the signature before accepting the stream. This blocks worker-ID spoofing even from callers who know the orgID pattern.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@dosubot dosubot Bot added the type/security label May 4, 2026
@SungJin1212 SungJin1212 force-pushed the Tenant-impersonation-pushStream branch 3 times, most recently from a4d294a to 2dc1ac9 Compare May 4, 2026 06:06
@pull-request-size pull-request-size Bot added size/XL and removed size/L labels May 4, 2026
@SungJin1212 SungJin1212 force-pushed the Tenant-impersonation-pushStream branch from 91cbdac to 64e9633 Compare May 4, 2026 10:17
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@SungJin1212 SungJin1212 force-pushed the Tenant-impersonation-pushStream branch from 64e9633 to bc012ac Compare May 4, 2026 10:22
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@SungJin1212 SungJin1212 force-pushed the Tenant-impersonation-pushStream branch from 10504c6 to c5053f2 Compare May 4, 2026 12:18
Comment thread pkg/ingester/pushstream_security_test.go Outdated
Comment thread pkg/distributor/distributor.go Outdated
@friedrichg friedrichg requested review from danielblando and yeya24 May 4, 2026 15:52
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@SungJin1212 SungJin1212 force-pushed the Tenant-impersonation-pushStream branch from be91dbb to 1566253 Compare May 5, 2026 04:03
Copy link
Copy Markdown
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing this!

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label May 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer size/XL type/security

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants