Skip to content

refactor(index): use arrow stats for scalar index stats#6808

Open
HaochengLIU wants to merge 1 commit into
mainfrom
replace-scalar-index-stats-arrow-stats
Open

refactor(index): use arrow stats for scalar index stats#6808
HaochengLIU wants to merge 1 commit into
mainfrom
replace-scalar-index-stats-arrow-stats

Conversation

@HaochengLIU
Copy link
Copy Markdown
Member

@HaochengLIU HaochengLIU commented May 16, 2026

Summary

  • Replace zonemap DataFusion min/max accumulators with lance_arrow_stats::StatisticsAccumulator.
  • Use arrow_stats to track bloom filter zone null counts.
  • Keep scalar-index NaN handling local to Lance instead of importing DataFusion math UDFs.
  • Add lance-arrow-stats to the lance-index dependency graph and update the Python lockfile.

Testing

  • All tests pass.

@github-actions
Copy link
Copy Markdown
Contributor

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@HaochengLIU HaochengLIU force-pushed the replace-scalar-index-stats-arrow-stats branch from d6405d3 to 9aa30a8 Compare May 16, 2026 15:45
@HaochengLIU HaochengLIU changed the title Use arrow stats for scalar index zone stats refactor(index): use arrow stats for scalar index stats May 16, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented May 16, 2026

Codecov Report

❌ Patch coverage is 70.11494% with 26 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-index/src/scalar/zonemap.rs 66.21% 18 Missing and 7 partials ⚠️
rust/lance-index/src/scalar/bloomfilter.rs 92.30% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@HaochengLIU HaochengLIU force-pushed the replace-scalar-index-stats-arrow-stats branch from 9aa30a8 to 921f623 Compare May 16, 2026 18:14
@HaochengLIU HaochengLIU force-pushed the replace-scalar-index-stats-arrow-stats branch 3 times, most recently from 6e67ae1 to 156695b Compare May 17, 2026 00:27
@HaochengLIU HaochengLIU marked this pull request as ready for review May 17, 2026 02:30
Copy link
Copy Markdown

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@HaochengLIU
Copy link
Copy Markdown
Member Author

@claude review once

Comment thread rust/lance-index/src/scalar/expression.rs Outdated
@HaochengLIU HaochengLIU force-pushed the replace-scalar-index-stats-arrow-stats branch from 156695b to 2f8a18d Compare May 20, 2026 00:56
@HaochengLIU
Copy link
Copy Markdown
Member Author

@westonpace @jackye1995 can any of you review this PR? ty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant