perf: make HNSW cheaper to load

[wombatu-kun]

Summary

Closes #6746. Loading an HNSW partition no longer reconstructs a per-node Vec<GraphBuilderNode> / Vec<OrderedNode> graph. The loaded graph is now backed directly by the on-disk Arrow buffers, with neighbor adjacency served as zero-copy &[u32] slices straight out of the __neighbors ListArray value buffer. This unblocks a future zero-copy CacheCodec (#6745).

Motivation

Per #6746, loading an HNSW partition required expensive per-node reconstruction, which makes a zero-copy IPC CacheCodec (#6745) infeasible. The fix is to keep the Arrow data and offsets as the graph's backing store while preserving current search behavior and performance.

What changed

• HnswCore now holds an HnswGraph enum instead of Arc<Vec<GraphBuilderNode>>: Built (in-memory, produced by the online builder / index_vectors — build path untouched) or Loaded (Arrow-backed, search-only).
• LoadedHnswGraph retains the full RecordBatch plus per-level zero-copy ListArray neighbor views and a tiny per-upper-level id -> row lookup; the geometrically-shrinking upper levels keep these maps negligible.
• Level 0 uses a Dense lookup (row == __vector_id, asserted in debug); upper levels use a Sparse map keyed by __vector_id value, exactly mirroring the old per-node load — including the known level_offsets quirk where the entry-point node is written by to_batch at every level but counted only at level 0, so upper-level slices are off-by-one and duplicate ids resolve last-write-wins.
• The search loop is single-sourced across both backends via a local macro, keeping the existing Graph / BorrowingGraph seam; search is unchanged.
• to_batch() on a loaded graph is a verbatim passthrough (re-stamped metadata only), so the IVF partition cache (ivf/partition_serde.rs, which re-serializes loaded indices) round-trips losslessly and Implement CacheCode for HNSW indices #6745 can write/read it through lance_arrow::ipc without rebuilding the graph.

Correctness & compatibility

• Loaded-graph search is bit-identical to the in-memory build across L2 / Dot / Hamming and graph sizes (single node, pair, multi-level 2048).
• Old load semantics are preserved bit-for-bit, including duplicate-id last-write-wins across a misaligned slice boundary; build -> to_batch -> load -> to_batch is byte-stable (b1 == b2).
• No public API signature change. HNSW::nodes() now panics on a loaded graph (documented; GraphBuilderNode is internal API and there are no in-tree callers).

Benchmarks

criterion --quick, 100000×128, L2, k=100, ef=300 (rust/lance-index/benches/hnsw.rs). The "before" load_hnsw was measured by running this same bench against the parent commit's reconstruction-based builder.rs (only that file swapped), so it is a like-for-like HNSW::load comparison.

Benchmark	Before (reconstruction load)	After (Arrow-backed)	Δ
load_hnsw(100000x128)	~127 ms	~90.8 µs	~1,400× faster
search_hnsw100000x128 (built, baseline)	~700.7 µs	~700.7 µs	unchanged
search_hnsw_loaded100000x128	n/a	~690.4 µs	on par with built (within noise)

Load drops from ~127 ms (allocating 100k GraphBuilderNodes + per-node OrderedNode adjacency) to ~91 µs (batch slice + tiny upper-level sparse maps), while search on the Arrow-backed graph stays on par with the in-memory build. Numbers are --quick/indicative; the ~3-orders-of-magnitude load delta is well outside noise. Re-run a full cargo bench before merge for headline figures.

Tests

All in rust/lance-index/src/vector/hnsw/builder.rs:

• test_loaded_search_parity_and_recall (rstest: L2 single / L2 pair / L2 2048 / Dot 2048) — built vs loaded parity plus recall ≥ 0.5.
• test_loaded_level_offsets_misalignment_invariant — pins the entry-point-written-at-every-level surplus (batch.num_rows() > sum(level_count)), the Dense level-0 precondition, and loaded↔built search parity despite the misalignment.
• test_loaded_empty_index — 0-row to_batch → load → empty graph round-trip.
• test_to_batch_roundtrip_loaded — the IVF partition-cache path: to_batch on a loaded index is byte-stable and reloads/searches identically.
• test_loaded_graph_is_arrow_backed — loaded graph is strictly lighter than the built representation.
• Pre-existing test_builder_write_load (2048, L2, file round-trip) and test_builder_write_load_binary_hamming (256, Hamming) continue to pass unchanged.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 88.96797% with 31 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/vector/hnsw/builder.rs	88.96%	24 Missing and 7 partials ⚠️

📢 Thoughts on this report? Let us know!

[Xuanwo]

Thank you for working on this!

[Xuanwo]

Is it safe to panic here?

[wombatu-kun]

Good question. The census found no in-tree callers of nodes() anywhere (rust/, python/, java/), and nodes() was already being modified in this PR. Rather than rely on the panic being unreachable, I changed the signature to Option<Arc<Vec<GraphBuilderNode>>> — it now returns None for a disk-loaded (Arrow-backed) graph instead of panicking. Done in 719460f.

[Xuanwo]

I prefer not to use macro_rules.

[wombatu-kun]

Done in 719460f

[Xuanwo]

Do we really need such function?

[wombatu-kun]

No. Removed it in 719460f