feat(common): initial multi-network incrementalizer support#1954
Draft
feat(common): initial multi-network incrementalizer support#1954
Conversation
12b633d to
5abe1b3
Compare
Introduce lead-network incrementalization for streaming SQL queries spanning multiple blockchain networks, interpolating non-lead `_block_num` values into the lead network's block range using segment metadata. - Add `IncrementalContext` with `single_network`/`multi_network` constructors and `interpolate_block_num_from_segment` projection for non-lead tables - Extract `validate_table_providers` into a standalone plan walk and `build_table_network_map` from `find_cross_network_join` for reuse - Add `Filter` node above constrained table scans for providers without filter pushdown (e.g. `MemTable`) - Parse `SETTINGS lead_network = '...'` in `helpers.rs` and wire through `flight.rs` → `StreamingQuery::spawn` - Add 5 incrementalizer unit tests + 3 `lead_network_setting` parser tests - Cross-network joins are not yet supported; when added, the lead network's `_block_num` will pass through to the output table directly
5abe1b3 to
bf7c781
Compare
Member
Author
|
Properly resolving the set of networks associated with derived tables will be added in another PR. |
Member
Author
|
This PR does not attempt to infer which network is "fastest" using the segment metadata because:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce lead-network incrementalization for streaming SQL queries spanning
multiple blockchain networks, interpolating non-lead
_block_numvalues intothe lead network's block range using segment metadata.
IncrementalContextwithsingle_network/multi_networkconstructorsand
interpolate_block_num_from_segmentprojection for non-lead tablesvalidate_table_providersinto a standalone plan walk andbuild_table_network_mapfromfind_cross_network_joinfor reuseFilternode above constrained table scans for providers withoutfilter pushdown (e.g.
MemTable)SETTINGS lead_network = '...'inhelpers.rsand wire throughflight.rs→StreamingQuery::spawnlead_network_settingparser tests_block_numwill pass through to the output table directly