[SPARK-56840][SQL] Avoid unresolved NullIf type lookup#55838
Open
sunchao wants to merge 3 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
NULLIFbuilds its replacement expression before analysis has resolved all child expressions.For nested field references, the existing implementation can read the left operand's data type
too early while constructing the null branch, which can fail analysis even though the SQL shape
is valid.
SPARK-56840 tracks this analyzer failure.
What changes were proposed in this PR?
NULLIFnull branch with a lazy typed-null placeholder so construction does not eagerlyread the unresolved left operand type, while
NullIf.replacement.dataTyperemains valid once theoperand type is available.
RuntimeReplaceable, soReplaceExpressionsrestores an ordinary typedLiteral(null, ...)before later optimizer rules run and existing null-literal simplificationscontinue to apply.
nullif(c.provider, lower(...))analysis in bothALWAYS_INLINE_COMMON_EXPRmodes;NullIfreplacement type reporting before type coercion;Does this PR introduce any user-facing change?
Yes. Valid
NULLIFexpressions over unresolved nested field references that could fail duringanalysis now resolve and execute successfully.
How was this patch tested?
build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.NullExpressionsSuite -- -z "NullIf replacement preserves its data type before type coercion"'build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizerSuite -- -z "NullIf typed null branch is replaced with a null literal"'build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z "nullif function"'build/sbt 'sql/testOnly org.apache.spark.sql.ExplainSuite -- -z "explain for these functions; use range to avoid constant folding"'Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5.5)