Skip to content

Flatten distributeTimeseriesToReplicas data structures in receive/handler.go#309

Open
gavins-db wants to merge 3 commits intodatabricks:db_mainfrom
gavins-db:gs/receive-refactor
Open

Flatten distributeTimeseriesToReplicas data structures in receive/handler.go#309
gavins-db wants to merge 3 commits intodatabricks:db_mainfrom
gavins-db:gs/receive-refactor

Conversation

@gavins-db
Copy link

@gavins-db gavins-db commented Mar 6, 2026

In some previous profiling I learned that distributeTimeseriesToReplicas which returns a map[endpointReplica]map[string]trackedSeries, map[endpointReplica]map[string]trackedSeries is allocating more objects than necessary due to that pattern, similarly, allocating a replicas []uint64, slice rather than just iterating between two numbers is also unnecessarily wasted memory.
These are micro optimizations and are still largely overshadowed by other concerns but these changes are foundational to #306 (my vt migration) and really should just be pulled forward here. Furthermore, at least in my opinion, this makes what's happening under the hood significantly more easy to read and reason about.

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

  • Introduce distributionKey struct to replace nested map[endpointReplica]map[string]trackedSeries with flat map[distributionKey]*trackedSeries
  • Use replicaStart/replicaEnd range instead of []uint64 replicas slice
  • Remove remoteWriteParams; pass replica and timeseries directly
  • Update gatherWriteStats, sendWrites, sendLocalWrite, sendRemoteWrite signatures to use the flattened map and pointer trackedSeries
  • Update TestDistributeSeries for new signatures

Verification

  • make test-local

yuchen-db and others added 3 commits March 3, 2026 15:14
  - Introduce distributionKey struct to replace nested
    map[endpointReplica]map[string]trackedSeries with flat
    map[distributionKey]*trackedSeries
  - Use replicaStart/replicaEnd range instead of []uint64 replicas slice
  - Remove remoteWriteParams; pass replica and timeseries directly
  - Update gatherWriteStats, sendWrites, sendLocalWrite, sendRemoteWrite
    signatures to use the flattened map and pointer trackedSeries
  - Update TestDistributeSeries for new signatures
Port descriptive comments for replica, distributionKey, and
distributeTimeseriesToReplicas from the vtproto optimization branch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants