fix(mem_wal): exact PK dedup for LSM vector search and point lookup#6856
Open
jackye1995 wants to merge 5 commits into
Open
fix(mem_wal): exact PK dedup for LSM vector search and point lookup#6856jackye1995 wants to merge 5 commits into
jackye1995 wants to merge 5 commits into
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
…point lookup Same primary key written multiple times into one memtable, or into both a memtable and an older generation, used to leak through to the user as distinct rows. Two paths were affected: - Vector search: HNSW indexes every insert as its own graph node, so KNN could return both V1 and V2 of the same PK from a single source. - Point lookup (active arm): `FilterExec + LIMIT 1` over an insert-ordered scan returned the oldest match among duplicates. Vector search now runs each per-source KNN through `LsmSourceTagExec`, which appends `(_memtable_gen, _freshness)`. A single `LsmGlobalPkDedupExec` over the union picks the row with the largest tuple per PK — newer generations win, ties fall to the normalized within-source order (larger `_rowid` for the active arm; flipped via `u64::MAX - _rowid` for flushed arms to compensate for the reverse-write convention). This replaces the older two-step `WithinSourceDedupExec` + bloom-based `FilterStaleExec` design and is exact (no false-positive recall loss, no top-k under-fill, no missing-bloom footgun). Point lookup keeps a `WithinSourceDedupExec(KeepMaxFreshness)` on the active arm only; `CoalesceFirstExec` already short-circuits cross-source selection so global dedup would conflict. Flushed and base arms still rely on `LIMIT 1` under the reverse-write / forward-write conventions respectively. Removed: `FilterStaleExec`, `GenerationBloomFilter`, and the `LsmVectorSearchPlanner::with_bloom_filter[s]` API — no longer needed now that dedup is exact. New tests pin: duplicate PK within one active memtable (both planners), duplicate PK across generations (vector search), and the partition coalesce ahead of `LsmGlobalPkDedupExec` that keeps active-memtable rows from being silently dropped.
jackye1995
force-pushed
the
jack/fix-lsm-pk-dedupe
branch
from
33cf21c to
a6e7d82
Compare
…gh LSM vector search The LSM vector search planner now accepts a base dataset reference and appends a TakeExec after the global PK dedup + sort. This allows the final top-k rows to materialize any user-projected columns that were not part of the per-source KNN output, fetching from the base dataset by _rowid. Also plumbs refine_factor as a parameter on plan_search() so callers can enable base-table refine (over-fetch k*factor candidates, re-rank with exact distances). Memtable arms use exact HNSW and are unaffected. Both Python and Java bindings are updated with the new parameter.
jackye1995
changed the title
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…point-lookup benchmarks Remove the duplicated criterion-based mem_wal_read.rs and mem_wal_vector.rs benchmarks. Replace with two standalone CLI benchmarks that produce JSON output for panel-style trend analysis: - mem_wal_vector_bench: KNN search across LSM levels using real 384-dim embeddings from lance-format/fineweb-edu, IVF-RQ base table index, recall verification against brute-force ground truth. - mem_wal_point_lookup_bench: PK-based point lookups across base table, flushed generations, and active memtable. Both accept --flushed-generations (0/1/2) and --max-memtable-rows (100k/500k/1M) for sweeping the full matrix. Results are written as individual JSON files for aggregation.
The hf:// URI scheme for accessing lance-format/fineweb-edu requires network access that may not be available or reliable on all environments. Switch to deterministic synthetic 384-dim embeddings using the same cluster+noise scheme as mem_wal_hnsw_bench.rs. This makes the bench self-contained with no external dependencies.
hamersaw
reviewed
May 20, 2026
Contributor
hamersaw
left a comment
hamersaw
left a comment
There was a problem hiding this comment.
IIUC this algorithm is basically parallelize top-K from each source and then dedup based on freshness. When exploring this elsewhere we noted the bug where a high ranking row is bumped out of the top-K by an un-fresh row and ends up with incorrect results. For a concrete example:
active memtable
PK=0, _distance=10
PK=1, _distance=11
PK=2, _distance=12
flushed memtable (l0 cache) gen 0
PK=0, _distance=1
PK=3, _distance=2
PK=4, _distance=3
If we we a top-K of 2 and dedup on this we have active memtable returning PK 0, 1 and flushed memtable returning PK 0, 3. the dedup results in keys 0, 3 returned. When really, it should be 3, 4 -- both from flushed memtable.
The various mitigations we discussed were:
(1) overfetching each source: we can have some bound that overfetches - don't love it.
(2) incremental refill: if a lower tier source has keys that are update we re-query it to ensure top-K non-updated keys - don't love it.
TBH I thought the bloom filter approach was reasonable to stop duplicates between sources. My thought was that within a source can we simply add a deletion vector to the flushed memtable (l0 cache) on write so that the read tooling automatically removes duplicates without having to rebuild indexes on write.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Duplicate primary keys written into one memtable or across generations leaked through as distinct rows in vector search and point lookup.
Exact PK dedup — replaces the bloom-filter-based
FilterStaleExecwith an exact pipeline:LsmSourceTagExectags each row with(_memtable_gen, _freshness)LsmGlobalPkDedupExecdoes single-pass cross-source PK dedup keeping the freshest row per PKWithinSourceDedupExec(KeepMaxFreshness)on the active armRemoves
FilterStaleExec,GenerationBloomFilter, and the bloom-filter building from transaction commit.Post-rerank TakeExec — the planner now accepts a base
Datasetreference. After global PK dedup + sort + top-k, aTakeExecmaterializes any user-projected columns not in the per-source KNN output by fetching from the base dataset via_rowid.Refine factor plumbing —
plan_search()now acceptsrefine_factorso callers can enable base-table refine (over-fetchk * factorcandidates, re-rank with exact distances). Memtable arms use exact HNSW and are unaffected. Exposed in both Python and Java bindings.