Skip to content

VT Proto migration#306

Open
gavins-db wants to merge 38 commits intodatabricks:db_mainfrom
gavins-db:gs/vt-migrate-squash
Open

VT Proto migration#306
gavins-db wants to merge 38 commits intodatabricks:db_mainfrom
gavins-db:gs/vt-migrate-squash

Conversation

@gavins-db
Copy link

Migrate from GoGo to VT proto

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

  • Migrate thanos from gogo/protobuf to vtproto with pointer-based labels ([]*Label and other types). This is the majority the changes.
  • A custom gRPC codegen plugin for pool-aware unmarshal.
  • VT protos also have label string interning during unmarshal which is a big reason for the memory improvements in this branch. A custom pkg/unique string interning package (xsync.Map + finalizer cleanup, with an optional no-GC variant via build tag) improves performance here compared with stdlib unique which itself was already an improvement on the previous string interning library.
  • Together these yield ~50% reduction in B/op and ~40% reduction in allocs/op with a minimal cpu increase on the gRPC receive synthetic benchmark. - This cpu increase may actually be the in the nature of the synthetic benchmark and performance may (I hope anyway) be better in a real-life scenario.
  • note: cortex code has not been touched.

This PR was primarily focused on optimizing the receive path. It is possible there are some regressions in other areas (e.g. Query) however, at this point we should just see if this is a viable way forward at all. If it isn't then we stop here. I can close this and follow up with some of the micro optimizations i've made (or planned to make) in a separate PR building on gogo. If this is a viable path forward then I/we can begin tracing down further regressions and making improvements such as enabling pooling in other places. and even re-thinking how the underlying TSDB works in prometheus to carry these changes all the way through -- which I believe could have a greater impact.

Verification

  • existing tests pass.
  • introduced additional benchmarks which I've used to check on things.
  • only a real test will tell...

This reverts commit 4463ebe.
@gavins-db
Copy link
Author

🔁

@gavins-db gavins-db closed this Mar 9, 2026
@gavins-db
Copy link
Author

reopening to force tests to re-run 😈

@gavins-db gavins-db reopened this Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants