PRW2:Add bound checking for symbol referencs by SungJin1212 · Pull Request #7290 · cortexproject/cortex

SungJin1212 · 2026-02-23T00:35:24Z

I have found the Distributor gets stuck in panic when UnitRef exceeds the symbols array length in
https://github.com/cortexproject/cortex/blob/master/pkg/util/push/push.go#L220
This PR adds bound checking logic on the server side.

Panic log

Prometheus version: v3.9.1

2026/02/23 00:00:46 http: panic serving 192.168.119.93:60442: runtime error: index out of range [2484] with length 2160
goroutine 10056052 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1943 +0xd3
panic({0x3844be0?, 0xc00daff5a8?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1()
	/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:159 +0x169
panic({0x3844be0?, 0xc00daff5a8?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
github.com/cortexproject/cortex/pkg/util/push.convertV2RequestToV1(0xc00c9000c0, 0x0)
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:220 +0xc10
github.com/cortexproject/cortex/pkg/api.(*API).RegisterDistributor.Handler.func1.2()
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:114 +0x1e8
github.com/cortexproject/cortex/pkg/api.(*API).RegisterDistributor.Handler.func1({0x4139850, 0xc001e8aa80}, 0xc00044a000)
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:192 +0x6f0
net/http.HandlerFunc.ServeHTTP(0xc0006afe00?, {0x4139850?, 0xc001e8aa80?}, 0x413f2c8?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/weaveworks/common/middleware.init.func1.1({0x4139850, 0xc001e8aa80}, 0xc0006afe00)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/http_auth.go:17 +0x108
net/http.HandlerFunc.ServeHTTP(0x0?, {0x4139850?, 0xc001e8aa80?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/cortexproject/cortex/pkg/api.(*API).RegisterRoute.HTTPHeaderMiddleware.Wrap.func1({0x4139850, 0xc001e8aa80}, 0xc0166bf680?)
	/go/src/github.com/cortexproject/cortex/pkg/api/middlewares.go:50 +0x66
net/http.HandlerFunc.ServeHTTP(0xc0006afb80?, {0x4139850?, 0xc001e8aa80?}, 0x2?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0002bb5c0, {0x4139850, 0xc001e8aa80}, 0xc0006af7c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/gorilla/mux/mux.go:212 +0x1e2
github.com/weaveworks/common/middleware.(*Instrument).Wrap.Instrument.Wrap.func1.2({0x4139850?, 0xc001e8aa80?})
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:70 +0x33
github.com/felixge/httpsnoop.(*Metrics).CaptureMetrics(0xc001ba8570, {0x7f081fe2a9b0, 0xc00c900060}, 0xc00098f7f0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:84 +0x1e5
github.com/felixge/httpsnoop.CaptureMetricsFn({0x7f081fe2a9b0, 0xc00c900060}, 0xc00098f7f0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:39 +0x4e
github.com/weaveworks/common/middleware.(*Instrument).Wrap.Instrument.Wrap.func1({0x7f081fe2a9b0, 0xc00c900060}, 0xc0006af7c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:69 +0x28b
net/http.HandlerFunc.ServeHTTP(0x41349b0?, {0x7f081fe2a9b0?, 0xc00c900060?}, 0xc0166bf560?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/weaveworks/common/middleware.(*Log).Wrap.Log.Wrap.func1({0x41349b0, 0xc000476e60}, 0xc0006af7c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/logging.go:81 +0x209
net/http.HandlerFunc.ServeHTTP(0x41e045?, {0x41349b0?, 0xc000476e60?}, 0x1?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5({0x41326e0, 0xc00544a5a0}, 0xc0006af540)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:163 +0x49d
net/http.HandlerFunc.ServeHTTP(0x41e045?, {0x41326e0?, 0xc00544a5a0?}, 0xc00544a501?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
net/http.serverHandler.ServeHTTP({0x41250c0?}, {0x41326e0?, 0xc00544a5a0?}, 0x1?)
	/usr/local/go/src/net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc00425d0e0, {0x413f2c8, 0xc000a033b0})
	/usr/local/go/src/net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 196
	/usr/local/go/src/net/http/server.go:3493 +0x485
2026/02/23 00:00:46 http: panic serving 192.168.119.93:60438: runtime error: index out of range [2512] with length 2127
goroutine 10056051 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1943 +0xd3
panic({0x3844be0?, 0xc00dafe0c0?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1()
	/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:159 +0x169
panic({0x3844be0?, 0xc00dafe0c0?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
github.com/cortexproject/cortex/pkg/util/push.convertV2RequestToV1(0xc0085befc0, 0x0)
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:220 +0xc10
github.com/cortexproject/cortex/pkg/api.(*API).RegisterDistributor.Handler.func1.2()
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:114 +0x1e8
github.com/cortexproject/cortex/pkg/api.(*API).RegisterDistributor.Handler.func1({0x4139850, 0xc000ef5840}, 0xc00e0f4a00)
	/go/src/github.com/cortexproject/cortex/pkg/util/push/push.go:192 +0x6f0
net/http.HandlerFunc.ServeHTTP(0xc00e0f48c0?, {0x4139850?, 0xc000ef5840?}, 0x413f2c8?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/weaveworks/common/middleware.init.func1.1({0x4139850, 0xc000ef5840}, 0xc00e0f48c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/http_auth.go:17 +0x108
net/http.HandlerFunc.ServeHTTP(0x0?, {0x4139850?, 0xc000ef5840?}, 0x0?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/cortexproject/cortex/pkg/api.(*API).RegisterRoute.HTTPHeaderMiddleware.Wrap.func1({0x4139850, 0xc000ef5840}, 0xc000919d40?)
	/go/src/github.com/cortexproject/cortex/pkg/api/middlewares.go:50 +0x66
net/http.HandlerFunc.ServeHTTP(0xc00e0f4640?, {0x4139850?, 0xc000ef5840?}, 0x2?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0002bb5c0, {0x4139850, 0xc000ef5840}, 0xc00e0f4500)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/gorilla/mux/mux.go:212 +0x1e2
github.com/weaveworks/common/middleware.(*Instrument).Wrap.Instrument.Wrap.func1.2({0x4139850?, 0xc000ef5840?})
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:70 +0x33
github.com/felixge/httpsnoop.(*Metrics).CaptureMetrics(0xc0004dbe00, {0x7f081fe2a9b0, 0xc0085bef60}, 0xc0019e77f0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:84 +0x1e5
github.com/felixge/httpsnoop.CaptureMetricsFn({0x7f081fe2a9b0, 0xc0085bef60}, 0xc0019e77f0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:39 +0x4e
github.com/weaveworks/common/middleware.(*Instrument).Wrap.Instrument.Wrap.func1({0x7f081fe2a9b0, 0xc0085bef60}, 0xc00e0f4500)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:69 +0x28b
net/http.HandlerFunc.ServeHTTP(0x41349b0?, {0x7f081fe2a9b0?, 0xc0085bef60?}, 0xc000918d20?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/weaveworks/common/middleware.(*Log).Wrap.Log.Wrap.func1({0x41349b0, 0xc000966820}, 0xc00e0f4500)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/logging.go:81 +0x209
net/http.HandlerFunc.ServeHTTP(0x41e045?, {0x41349b0?, 0xc000966820?}, 0x1?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5({0x41326e0, 0xc0053f25a0}, 0xc00e0f43c0)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:163 +0x49d
net/http.HandlerFunc.ServeHTTP(0x41e045?, {0x41326e0?, 0xc0053f25a0?}, 0xc0053f2501?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
net/http.serverHandler.ServeHTTP({0x41250c0?}, {0x41326e0?, 0xc0053f25a0?}, 0x1?)
	/usr/local/go/src/net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc00425d050, {0x413f2c8, 0xc000a033b0})
	/usr/local/go/src/net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 196
	/usr/local/go/src/net/http/server.go:3493 +0x485

Which issue(s) this PR fixes:
Fixes #

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

dosubot · 2026-02-23T00:35:31Z

Related Documentation

Checked 0 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

yeya24 · 2026-02-24T16:32:14Z

pkg/util/push/push.go

 			return v1Req, fmt.Errorf("TimeSeries must contain at least one sample or histogram for series %v", lbs.String())
 		}

+		if int(v2Ts.Metadata.UnitRef) >= len(symbols) {


Interesting... Is it possible from existing remote write v2 sender? And how does Prometheus handle this today

It seems to be a bug in the use of sync.Pool. I made a PR to fix it in here #7299

pull-request-size bot added the size/L label Feb 23, 2026

dosubot bot added component/distributor type/bug labels Feb 23, 2026

SungJin1212 force-pushed the Add-unit/help-ref-bound-checking branch from 1f22d7d to 44c19ef Compare February 23, 2026 00:36

PRW2:Add bound checking for symbol referencs

9d73e9c

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

SungJin1212 force-pushed the Add-unit/help-ref-bound-checking branch from 44c19ef to 9d73e9c Compare February 24, 2026 06:35

yeya24 reviewed Feb 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PRW2:Add bound checking for symbol referencs#7290

PRW2:Add bound checking for symbol referencs#7290
SungJin1212 wants to merge 1 commit intocortexproject:masterfrom
SungJin1212:Add-unit/help-ref-bound-checking

SungJin1212 commented Feb 23, 2026 •

edited

Loading

Uh oh!

dosubot bot commented Feb 23, 2026

Uh oh!

yeya24 Feb 24, 2026

Uh oh!

SungJin1212 Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

SungJin1212 commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Panic log

Uh oh!

dosubot bot commented Feb 23, 2026

Uh oh!

yeya24 Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

SungJin1212 Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SungJin1212 commented Feb 23, 2026 •

edited

Loading