[CALCITE-6451] Improve Nullability Derivation for Intersect and Minus

[xiedeyantu]

Jira Link

CALCITE-6451

Changes Proposed

SetOp overrides deriveRowType() and computes the output row type to be the least restrictive across all inputs here.

So for example given

Input 1: (I64, I64, I64?, I64?)
Input 2: (I64, I64?, I64, I64?)
where ? denotes nullable, the least restrictive output computes:

Output: (I64, I64?, I64?, I64?)
For UNION operations, these nullabilities are accurate.

However for MINUS and INTERSECT there is room for improvement.

MINUS only returns rows from the first input, as such its output nullability should always match that of its first input:

Output: (I64, I64, I64?, I64?)
INTERSECT only returns rows that match across all inputs. If a column is not nullable in any of the inputs, then it is not nullable in the output because no rows can be emitted in which that column is null:

Output: (I64, I64, I64, I64?)

[xiedeyantu]

Related PR #3845.

[mihaibudiu]

you have some checker failures

[mihaibudiu]

Does this work around the problems in the other PR?

[xiedeyantu]

Does this work around the problems in the other PR?

Are you referring to #3845? I noticed that you had approved this PR before, but there were some conflicts. Since it's been a long time, the CI status is no longer visible, and it's unclear if there were other issues back then. I think it's a good PR, so I’m trying to finish it up.

[mihaibudiu]

Yes, the discussion in JIRA was about causing problems with other rules.

[xiedeyantu]

Yes, the discussion in JIRA was about causing problems with other rules.

I didn’t see any discussion in the Jira. Are you referring to the discussion in the original PR? I have resolved the rule conflicts.

[mihaibudiu]

yes, the original PR

[xiedeyantu]

According to this disscusion #3845 (comment) .
I think we don't need to dwell on this issue too much. If we transform INTERSECT into UNION, then we can simply use UNION's type inference. It's like LEFT JOIN (though this example might not be entirely appropriate), which can also change the nullability of columns from the right table. I'm not sure if my understanding is correct.

[xiedeyantu]

@mihaibudiu I'm not sure if you agree with the current simplified processing logic. If you have time, please review this PR to see if there are any other concerns.

[mihaibudiu]

@vbarua what do you think of this approach?

[silundong]

Do you know why this plan changed?

[xiedeyantu]

I haven't been able to pinpoint how the old plan included AGG in the join, but the new plan seems fine. I might need to investigate this difference in more detail later when I have more time.

[silundong]

There is a relBuilder.convert(intersect.getRowType(), false) call at the end, is this logic necessary?
Calcite doesn't have the ability to refine type inference from predicates, this rewrite is semantically equivalent, but it is unrelated to the issue-6451.

[xiedeyantu]

Removed

[silundong]

I'm not sure that whether onMatchAggregateOnUnion need this.

[xiedeyantu]

I agree, it would be safer that way.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[mihaibudiu]

What is the status of this PR?

[xiedeyantu]

What is the status of this PR?

There may only be one problem left now, which is that I haven't found the root cause of the changes in the plan mentioned by silun above.

[xiedeyantu]

Current PR Change Management Process:

1. MinusMergeRule
   Merges the nested binary Minus operators into a single multi-input LogicalMinus:

LogicalMinus(all=[false])
  t1
  t2
  t3

2. MinusToAntiJoinRule
   Rewrites the multi-input Minus into a chain of anti joins with a top-level distinct aggregate:

LogicalAggregate(group=[{0}])
  LogicalJoin(joinType=[anti])
    LogicalJoin(joinType=[anti])
      t1
      t2
    t3

3. JoinPushExpressionsRule
   Extracts the CAST expressions from the join condition into projection columns (A0), allowing the join to use direct column references:

LogicalProject(A=[$0])
  LogicalJoin(condition=[=($1, $3)], joinType=[anti])
    LogicalProject(A=[$0], A0=[CAST($0)])
    LogicalProject(A=[$0], A0=[CAST($0)])

4. EnumerableJoinRule
   Converts the logical anti joins into physical join implementations. The inner anti join becomes a nested-loop join, while the outer anti join is implemented as a hash join:

EnumerableHashJoin(joinType=[anti])
  EnumerableProject(A=[$0], A0=[CAST($0)])
    EnumerableNestedLoopJoin(joinType=[anti])
      ...

5. EnumerableAggregateRule
   Converts the top-level logical aggregate into a physical enumerable aggregate:

EnumerableAggregate(group=[{0}])
  EnumerableHashJoin(joinType=[anti])
    ...

The final selected plan is:

EnumerableAggregate
  EnumerableHashJoin (anti)
    EnumerableProject(A0=CAST(...))
      EnumerableNestedLoopJoin (anti)
        ...

The original plan took this specific form because, within Volcano, every rule generates a corresponding candidate RelNode. A particular scenario arises with nested Minus operations: for each level of nesting, an equivalent Aggregation-plus-Join structure is generated. Subsequently, during the final cost-based selection phase, this specific structure was deemed to have the lowest cost and was therefore selected. Please refer to the plan diagram below.

This is my current understanding; please take another look to see if it is clearly explained. @silundong @mihaibudiu

[mihaibudiu]

If the plan is semantically equivalent, it's not a problem from my pov.
If @silundong cannot complete the review I can try to take a look.