Skip to content

feat(common): initial multi-network incrementalizer support#1954

Draft
Theodus wants to merge 1 commit intomainfrom
theodus/multi-network-incrementalizer
Draft

feat(common): initial multi-network incrementalizer support#1954
Theodus wants to merge 1 commit intomainfrom
theodus/multi-network-incrementalizer

Conversation

@Theodus
Copy link
Member

@Theodus Theodus commented Mar 11, 2026

Introduce lead-network incrementalization for streaming SQL queries spanning
multiple blockchain networks, interpolating non-lead _block_num values into
the lead network's block range using segment metadata.

  • Add IncrementalContext with single_network/multi_network constructors
    and interpolate_block_num_from_segment projection for non-lead tables
  • Extract validate_table_providers into a standalone plan walk and
    build_table_network_map from find_cross_network_join for reuse
  • Add Filter node above constrained table scans for providers without
    filter pushdown (e.g. MemTable)
  • Parse SETTINGS lead_network = '...' in helpers.rs and wire through
    flight.rsStreamingQuery::spawn
  • Add 5 incrementalizer unit tests + 3 lead_network_setting parser tests
  • Cross-network joins are not yet supported; when added, the lead network's
    _block_num will pass through to the output table directly

@Theodus Theodus force-pushed the theodus/multi-network-incrementalizer branch from 12b633d to 5abe1b3 Compare March 11, 2026 15:14
@Theodus Theodus mentioned this pull request Mar 11, 2026
16 tasks
@Theodus Theodus marked this pull request as draft March 11, 2026 19:18
Introduce lead-network incrementalization for streaming SQL queries spanning
multiple blockchain networks, interpolating non-lead `_block_num` values into
the lead network's block range using segment metadata.
- Add `IncrementalContext` with `single_network`/`multi_network` constructors
  and `interpolate_block_num_from_segment` projection for non-lead tables
- Extract `validate_table_providers` into a standalone plan walk and
  `build_table_network_map` from `find_cross_network_join` for reuse
- Add `Filter` node above constrained table scans for providers without
  filter pushdown (e.g. `MemTable`)
- Parse `SETTINGS lead_network = '...'` in `helpers.rs` and wire through
  `flight.rs` → `StreamingQuery::spawn`
- Add 5 incrementalizer unit tests + 3 `lead_network_setting` parser tests
- Cross-network joins are not yet supported; when added, the lead network's
  `_block_num` will pass through to the output table directly
@Theodus Theodus force-pushed the theodus/multi-network-incrementalizer branch from 5abe1b3 to bf7c781 Compare March 12, 2026 15:55
@Theodus Theodus changed the title feat(common): add multi-network block number to incrementalizer feat(common): initial multi-network incrementalizer support Mar 12, 2026
@Theodus
Copy link
Member Author

Theodus commented Mar 12, 2026

Properly resolving the set of networks associated with derived tables will be added in another PR.

@Theodus
Copy link
Member Author

Theodus commented Mar 12, 2026

This PR does not attempt to infer which network is "fastest" using the segment metadata because:

  • Different networks may be faster or slower across segments, if for example the faster chain pauses.
  • This makes it clear to the user which network will be chosen, and gives them a setting to make the choice themselves.

@Theodus Theodus marked this pull request as ready for review March 12, 2026 16:01
@Theodus Theodus marked this pull request as draft March 12, 2026 17:00
@Theodus Theodus marked this pull request as draft March 12, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant