[VL] Reduce Velox scan SQL metrics by default to mitigate driver OOM#12127
Draft
lifulong wants to merge 1 commit into
Draft
[VL] Reduce Velox scan SQL metrics by default to mitigate driver OOM#12127lifulong wants to merge 1 commit into
lifulong wants to merge 1 commit into
Conversation
|
Run Gluten Clickhouse CI on x86 |
lifulong
force-pushed
the
gluten_driver_oom_while_spark_ok_use_same_driver_memory
branch
from
86f7772 to
09c0f07
Compare
|
Run Gluten Clickhouse CI on x86 |
lifulong
force-pushed
the
gluten_driver_oom_while_spark_ok_use_same_driver_memory
branch
2 times, most recently
from
a8e8cab to
67c52c7
Compare
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
lifulong
force-pushed
the
gluten_driver_oom_while_spark_ok_use_same_driver_memory
branch
from
67c52c7 to
6bbb6e8
Compare
|
Run Gluten Clickhouse CI on x86 |
lifulong
force-pushed
the
gluten_driver_oom_while_spark_ok_use_same_driver_memory
branch
from
6bbb6e8 to
4bb6c9a
Compare
lifulong
force-pushed
the
gluten_driver_oom_while_spark_ok_use_same_driver_memory
branch
from
4bb6c9a to
c621483
Compare
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Gluten jobs on the Velox backend are more prone to driver memory pressure than vanilla Spark in some production workloads. Investigation points to scan operators registering too many SQL metrics (accumulators).
Each BatchScanExecTransformer / FileSourceScanExecTransformer / HiveTableScanExecTransformer previously registered 30+ executor-side metrics per scan node.
Vanilla Spark is much leaner—for example, BatchScanExec only exposes numOutputRows (+ connector customMetrics), and FileSourceScanExec adds a small set of driver metrics (numFiles, metadataTime, etc.).
This gap increases driver heap usage and can contribute to driver OOM, especially on scan-heavy queries.
(Driver heap dump analysis while oom, the largest memory-consuming object is LiveStageMetrics)
(Gluten has been failed in first scan stage, while vanilla spark finished successfully with same driver memory 12g.)
Introduce a Velox-only minimal scan metrics set by default, with an opt-in switch for full metrics collection (debugging / advanced troubleshooting).
spark.gluten.sql.scan.detailedMetrics.enabled
ClickHouse backend is unchanged—this config does not affect CH scan metrics.
Default minimal metrics (Velox)
BatchScan (9 executor metrics):
rawInputRows, rawInputBytes, numOutputRows, outputBytes, scanTime, wallNanos, peakMemoryBytes, ioWaitTime, storageReadBytes
FileSourceScan / HiveTableScan — above plus Spark-aligned driver metrics:
numFiles, metadataTime, filesSize, numPartitions, pruningTime
Moved to full collection only (when detailed metrics enabled)
Examples include: numInputRows, inputVectors, inputBytes, outputVectors, cpuCount, numMemoryAllocations, skippedSplits, processedSplits, numDynamicFiltersAccepted, loadLazyVectorTime, skippedStrides, processedStrides, connector timing (preloadSplits, pageLoadTime, dataSourceAddSplitTime, dataSourceReadTime), storage cache details (storageReads, localReadBytes, ramReadBytes), etc.
How was this patch tested?
WIP on our produce envriment
Was this patch authored or co-authored using generative AI tooling?
co-authored using cursor.