Skip to content

perf(index): optimize search index slice allocation#2074

Closed
MEKXH wants to merge 3 commits intoOpenListTeam:mainfrom
MEKXH:perf/optimize-search-allocation
Closed

perf(index): optimize search index slice allocation#2074
MEKXH wants to merge 3 commits intoOpenListTeam:mainfrom
MEKXH:perf/optimize-search-allocation

Conversation

@MEKXH
Copy link

@MEKXH MEKXH commented Feb 3, 2026

perf(search): optimize search index slice allocation

Description / 描述

优化搜索索引构建过程中的切片内存分配策略:

  1. build.go: 在 batchIndexMessages 函数中使用预分配切片替代 append 动态增长
  2. search.go: 在 BatchIndex 函数中同样采用预分配策略,移除 utils.MustSliceConvert 中间转换

核心变更

- var searchNodes []model.SearchNode
- for i := range objs {
-     searchNodes = append(searchNodes, model.SearchNode{...})
- }
+ searchNodes := make([]model.SearchNode, len(objs))
+ for i := range objs {
+     searchNodes[i] = model.SearchNode{...}
+ }

Motivation and Context / 背景

搜索索引批量构建是一个高频操作,尤其在初始化索引或批量更新时。原实现存在以下性能问题:

  1. append 动态扩容: 导致多次内存分配和数据拷贝
  2. utils.MustSliceConvert 中间切片: 产生额外内存分配开销

通过预分配确切大小的切片,可显著减少 GC 压力并提升批量索引性能。

Relates to performance optimization

How Has This Been Tested? / 测试

添加了 Benchmark 测试对比优化前后性能:

go test -bench=BenchmarkSearchAllocation -benchmem ./internal/search/

测试结果

数据规模 旧逻辑 (ns/op) 新逻辑 (ns/op) 内存分配次数 提升
100 ~5841 ~486 12 → 1 ~12x
1000 ~43277 ~15637 12 → 1 ~2.8x
10000 ~470000 ~177000 12 → 1 ~2.6x

Checklist / 检查清单

  • I have read the CONTRIBUTING document.
    我已阅读 CONTRIBUTING 文档。
  • I have formatted my code with go fmt or prettier.
    我已使用 go fmtprettier 格式化提交的代码。
  • I have added appropriate labels to this PR (or mentioned needed labels in the description if lacking permissions).
    我已为此 PR 添加了适当的标签(如无权限或需要的标签不存在,请在描述中说明,管理员将后续处理)。
    建议标签: performance, search
  • I have requested review from relevant code authors using the "Request review" feature when applicable.
    我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
  • I have updated the repository accordingly (If it's needed).
    我已相应更新了相关仓库(若适用)。

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes search index batch-building by reducing slice reallocations and avoiding intermediate slice conversions, with an added benchmark to quantify allocation improvements.

Changes:

  • Preallocate []model.SearchNode in BatchIndex to avoid append growth.
  • Replace utils.MustSliceConvert + BatchIndex in index building with a direct preallocated conversion path for MQ messages.
  • Add benchmarks to compare allocation behavior before/after the optimization.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
internal/search/search.go Preallocates searchNodes in BatchIndex and assigns by index to reduce allocations.
internal/search/build.go Introduces batchIndexMessages to build searchNodes directly from MQ messages and preallocates toAddObjs capacity.
internal/search/benchmark_test.go Adds benchmarks simulating old vs new slice allocation strategies.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@KirCute KirCute changed the title Perf/optimize search allocation perf(index): optimize search index slice allocation Feb 4, 2026
@MEKXH MEKXH closed this Feb 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant