Epic: Stats and AggregateFns

[gatesn]

Goal

Make Vortex statistics pluggable by modeling pruning stats as aggregate-function state exposed through physical bound expressions. The concrete success case is demonstrating a Bloom-filter zone-map stat for UTF-8 equality pruning added through plugins: a custom aggregate function, scalar function, and rewrite rule, without changing built-in pruning logic.

Direction

Stats are aggregate-function partials/results when the aggregate has pruning semantics. Not every aggregate is a useful pruning stat; the rewrite path should depend only on aggregates that can prove bounds.

Keep expressions physical for this epic. Falsification turns concrete predicates into normal Vortex expressions over physical scalar functions and stat(expr, AggregateFnRef). We are not adding a logical expression layer yet.

Use stat(expr, AggregateFnRef) as the bound-expression primitive. It returns the stat value for the current stats scope, or null when unavailable. Falsification produces expressions containing stat(...); simplification/execution decides whether anything is proven.

Aggregate functions advertise whether a stored aggregate can satisfy a requested aggregate through AggregateFnRef::can_satisfy(...). Exact descriptor matches are preferred; compatible approximate aggregates, such as bounded max satisfying max, may be used when the stored aggregate is a sound bound.

Zone maps store aggregate-function descriptors, using Display for AggregateFnRef, and use those descriptors as stats-table column names. At read time, the zone map lowers bound expressions by matching available descriptors against requested aggregates.

Expression expansion is acceptable for now. Rewrites may produce multiple physical proof expressions, and each zone map can lower unavailable aggregates to null.

All new stats-facing APIs should live under vortex-array/src/stats/. Scalar function implementations may live with scalar functions, but should be re-exported through vortex_array::stats.

Phase 1: Bound Expressions and Pruning Aggregates

• Add StatFn / stat(expr, AggregateFnRef) under vortex-array/src/stats/. Add stats rewrite session API #7930 Thread scope dtype through stats rewrites #8024
  • Re-export through vortex_array::stats.
  • Use StatFn::new_expr(...) directly in tests and rewrites.
• Add pruning aggregate functions needed by the rewrite path. Add NullCount aggregate function #7933 Add pruning aggregate functions #8025
  • Include Min, Max, AllNull, AllNonNull, AllNan, AllNonNan, and bounded min/max variants for binary/string stats.
  • Keep NullCount as a legacy bridge for existing stats, not as a pruning proof aggregate.
• Add the aggregate satisfaction relation used to match stored stats to requested bounds. Add pruning aggregate functions #8025
  • AggregateFnRef::can_satisfy(...) returns whether a stored aggregate is exact, approximate, or unusable for the request.
• Add legacy mapping from pruning aggregate functions to existing Stat slots. Centralize aggregate stat bridge #7931 Add pruning aggregate functions #8025
  • Min and Max map separately to Stat::Min and Stat::Max.
• Make StatFn read existing legacy stats through the aggregate-to-Stat mapping. Centralize aggregate stat bridge #7931 Add pruning aggregate functions #8025
• Ensure StatFn works for flat arrays and chunked arrays. Remove chunked special case from stat execution #7928
• Add focused tests for StatFn::new_expr(...) reading legacy pruning stats. Add NullCount aggregate function #7933 Add pruning aggregate functions #8025

Phase 2: Rewrite Registry

• Add vortex-array/src/stats/session.rs for stats session state. Add stats rewrite session API #7930 Thread scope dtype through stats rewrites #8024
• Add vortex-array/src/stats/rewrite.rs for rewrite traits and helpers. Add stats rewrite session API #7930
• Register stats rewrite rules through VortexSession. Add stats rewrite session API #7930
• Support multiple rules for the same expression shape. Add stats rewrite session API #7930
  • Combine independent falsifiers with OR.
• Thread row-scope dtype through rewrite entrypoints. Thread scope dtype through stats rewrites #8024

Phase 3: Built-In Rewrite Rules

• Implement built-in rewrite rules directly against StatFn::new_expr(...). Add built-in stats rewrite rules #7935
  • Do not reuse existing stat_falsification implementations.
  • Do not introduce a fake StatsCatalog.
• Port comparison falsification. Add built-in stats rewrite rules #7935
• Add bounded min/max rewrite rules for binary/string pruning. Add built-in stats rewrite rules #7935
• Port boolean and / or rewrites. Add built-in stats rewrite rules #7935
• Port between. Add built-in stats rewrite rules #7935
• Port is_null and is_not_null. Add built-in stats rewrite rules #7935
• Port list_contains. Add built-in stats rewrite rules #7935
• Port safe cast handling where needed by the above rewrites. Add built-in stats rewrite rules #7935
• Add regression tests comparing new rewrites against current pruning behavior where practical. Add built-in stats rewrite rules #7935

LIKE pruning is tracked separately in #8026.

Phase 4: Zoned Layout Migration

• Teach ZoneMapLayout to lower StatFn bound expressions against existing zone stats. Teach zoned pruning to lower StatFn #7937
• Lower unavailable stats to nullable null results. Teach zoned pruning to lower StatFn #7937
• Move zoned pruning to the session-registered rewrite path. Teach zoned pruning to lower StatFn #7937
• Bridge richer null and NaN proof aggregates from existing legacy zone stats where exact. Teach zoned pruning to lower StatFn #7937
• Remove the old StatsCatalog zoned pruning path once StatFn lowering covers existing behavior. Teach zoned pruning to lower StatFn #7937

Phase 5: Aggregate-Function Zoned Stats

WARNING: this is the phase that changes the ZonedLayout serialized form

• Replace new zoned-layout stats configuration with aggregate-function descriptors. Use aggregate descriptors for zoned stats #7938
  • Configure stored zone stats with AggregateFnRef, not Stat enum values.
  • Use Display for AggregateFnRef as the descriptor string.
  • Use the descriptor string as the zone-map stats-table column name.
  • Keep Stat only as a compatibility bridge for existing array stats and legacy zoned metadata.
• Compute per-zone aggregate partials at write time. Use aggregate descriptors for zoned stats #7938
  • Build the auxiliary stats table from each aggregate function's partial/state dtype.
  • Use a custom strategy/configuration hook for selecting aggregates before adding broader policy machinery.
• Add a new zoned metadata format for aggregate-function stats. Use aggregate descriptors for zoned stats #7938
  • Current zoned metadata is raw bytes, not protobuf: zone_len followed by a legacy Stat bitset.
  • Add a version/magic marker so new metadata can be recognized as protobuf.
  • Store zone_len and present_aggregates: repeated string.
  • Preserve legacy metadata decoding by translating old Stat bitsets into built-in aggregate descriptor strings.
• Lower stat(expr, aggregate_fn) at read time by matching aggregate-function descriptors in the zone stats table. Use aggregate descriptors for zoned stats #7938
  • Match exact descriptors first.
  • Use AggregateFnRef::can_satisfy(...) when a stored aggregate is a sound exact or approximate substitute for the requested aggregate.
  • Unavailable aggregate stats continue to lower to nullable null results.
• Remove zoned-layout schema special cases that are only needed because stats are modeled as Stat enum values. Use aggregate descriptors for zoned stats #7938
  • Any auxiliary proof state should be represented by the aggregate partial itself or by a dedicated aggregate.

Phase 6: Plugin Bloom Proof

• Add a plugin-provided Bloom aggregate for UTF-8 values.
• Add the Bloom filter extension/storage type needed by the aggregate output.
• Add a plugin-provided bloom_might_contain(filter, value) scalar function.
• Register a plugin-provided rewrite for UTF-8 equality.
• Store/load the Bloom stat through the aggregate-function zone-map path.
• Demonstrate UTF-8 equality pruning without modifying built-in binary-expression pruning.

Phase 7: Satisfaction Follow-Up

• Add satisfaction rewrite APIs and rules as new behavior.
  • Combine independent satisfiers with OR.
• Teach filtering to use satisfied-zone masks to skip residual predicate evaluation for zones proven true.

Phase 8: Cleanup

• Remove duplicated legacy stat propagation once the new rewrite path is complete.
• Retire the old StatsCatalog pruning path.
• Move broader generic stats storage work to a follow-up epic if still needed.

Status

In progress.