[CALCITE-7551] Project/Filter/Join transpose and merge rules should not duplicate non-deterministic expressions (e.g. RAND()) by darpan-e6 · Pull Request #4970 · apache/calcite

darpan-e6 · 2026-05-27T06:54:05Z

Jira Link

Changes Proposed

Several optimizer and RelBuilder-level transformations inline a lower-projection expression into multiple positions of an upper expression without checking whether the inlined expression is non-deterministic. When the lower expression is non-deterministic (e.g. RAND()), this silently turns one evaluation into many, changing query semantics.

Reproduction

SELECT a, a + 1 AS b FROM (SELECT rand() AS a)

Before the fix, SqlToRelConverter produces:

LogicalProject(A=[RAND()], B=[+(RAND(), 1)])
  LogicalValues(tuples=[[{ 0 }]])

The two RAND() calls are evaluated independently, so B - A is no longer always 1. After the fix the inner projection is preserved and A, B share a single evaluation:

LogicalProject(A=[$0], B=[+($0, 1)])
  LogicalProject(A=[RAND()])
    LogicalValues(tuples=[[{ 0 }]])

Scope

The bug only affects operators that can return different values across call sites within a single statement — i.e. operators with isDeterministic() == false (SqlRandFunction, SqlRandIntegerFunction). Per-statement-stable operators such as CURRENT_TIMESTAMP, CURRENT_DATE, CURRENT_USER, etc. resolve to the same value at every occurrence in one execution, so duplicating them in the plan is observationally a no-op; they don't need to be blocked.

The guard uses RexUtil.isDeterministic(...), which consults SqlOperator.isDeterministic() and correctly partitions the two sets.

Affected code paths and fixes

The guard only kicks in when the transform would actually duplicate a non-deterministic evaluation — not whenever a non-deterministic column merely exists. The precise condition depends on whether the lower project is consumed by the transform or re-emitted above the new node:

Component	Mechanism	Guard
`RelOptUtil.pushPastProjectUnlessBloat`	`pushShuttle(Project)` substitutes every `RexInputRef` with the underlying projection. Only `RexOver` was guarded; determinism was not. The bottom project is consumed (inlined away).	Count `RexInputRef` uses per bottom slot; return `null` only if a non-deterministic bottom expression is referenced more than once (a single reference yields a single inlined copy, which is safe). Covers `ProjectMergeRule` and `FilterProjectTransposeRule`'s helper call, plus the `RelBuilder` bloat merge that `SqlToRelConverter` relies on.
`FilterProjectTransposeRule`	Pushes the filter below the project, but re-emits the original project above the new filter. So even one reference splits a single evaluation in two (one inlined into the pushed-down filter, one in the re-emitted project).	Skip only when the filter condition references a non-deterministic projected column (`RelOptUtil.InputFinder.bits`). Filters on deterministic columns are pushed even if the project computes an unrelated `RAND()`.
`JoinProjectTransposeRule`	Uses `RexProgramBuilder.mergePrograms` + `mergedProgram.expandLocalRef`, inlining projected expressions into the new join condition while re-emitting the project above.	Skip a side only when the join condition references one of that side's non-deterministic projected columns (left at offset 0, right at `nLeftFields`).
`SemiJoinProjectTransposeRule`	Same `mergePrograms` + `expandLocalRef` pattern as `JoinProjectTransposeRule`.	Skip only when the semi-join condition references a non-deterministic projected column on the LHS.

The "consumed vs re-emitted" distinction is why the threshold differs: ProjectMergeRule blocks at > 1 references (the bottom project disappears, so one reference = one surviving copy), while the transpose rules block at >= 1 (the project survives on top, so a single referenced occurrence already yields two evaluations).

Tests

Blocked cases (transform must not fire / not duplicate):

RelOptRulesTest.testProjectMergeShouldIgnoreNonDeterministic
RelOptRulesTest.testFilterProjectTransposeShouldIgnoreNonDeterministic
RelOptRulesTest.testJoinProjectTransposeShouldIgnoreNonDeterministic
RelOptRulesTest.testSemiJoinProjectTransposeShouldIgnoreNonDeterministic
SqlToRelConverterTest.testRandNotDuplicatedInProjectionMerge

Allowed cases (transform still fires when the non-deterministic column is unrelated to the condition):

RelOptRulesTest.testFilterProjectTransposeWithUnrelatedNonDeterministic
RelOptRulesTest.testJoinProjectTransposeWithUnrelatedNonDeterministic
RelOptRulesTest.testSemiJoinProjectTransposeWithUnrelatedNonDeterministic

Confirmed unaffected (verified by additional probing during investigation)

CalcMergeRule (RexProgram preserves CSE via local refs), ProjectReduceExpressionsRule, ProjectValuesMergeRule, ProjectFilterTransposeRule, ProjectJoinTransposeRule, ProjectCorrelateTransposeRule, AggregateProjectMergeRule, FilterSetOpTransposeRule (textual duplication only — each row flows through one branch of a UNION ALL, so per-row evaluation count is unchanged), ProjectToWindowRule.

…ot duplicate non-deterministic expressions (e.g. RAND())

mihaibudiu · 2026-05-27T18:18:04Z

+        return null;
+      }
+    }.visitEach(nodes);
+    for (int i = 0; i < refs.length; i++) {


As discussed in Jira, this is a bit too conservative, since it will not distinguish CURRENT_TIMESTAMP from RAND. But fixing that may be in the scope of a separate PR - we need really two separate notions of nondeterminism.

Agreed. We should complete the fine-grained judgment of the deterministic function first before completing this PR.

As far as I understand, CURRENT_TIMESTAMP is actually different than non-deterministic functions like RAND().

Non-deterministic function: it may return different values for every evaluations.
1.1 Returns false to isDeterministic().
1.2 Returns false to isDynamicFunction().

Dynamic Function: It will return same value at every call site within one statement; can differ across executions
2.1 Returns true to isDeterministic().
2.2 Returns true to isDynamicFunction().

And this path only blocks when we get isDeterministic() method response as false. Dynamic functions like CURRENT_TIMESTAMP can be duplicated and actually it is safe to duplicate them.

The JavaDoc of isDynamicFunction says:

/** * Returns whether it is unsafe to cache query plans referencing this * operator; false is assumed by default. */ public boolean isDynamicFunction() {

which is not exactly what you seem to imply.

You're right, that comment mixed up the two flags. The "same value at every call site within one statement" property I attached to isDynamicFunction() is not in its JavaDoc; I generalized from CURRENT_TIMESTAMP's behavior to isDynamicFunction as a whole, which is wrong, isDynamicFunction is overloaded (e.g. SqlRandIntegerFunction also returns true), and its actual contract is just plan-cache invalidation.

Scoping back to this PR: the bug only affects operators that can return different values across call sites within a single statement, i.e. operators with isDeterministic() == false, like RAND and RAND_INTEGER. The other operators that override isDynamicFunction() to true (SqlAbstractTimeFunction, SqlCurrentDateFunction, SqlBaseContextVariable) are per-statement-stable: duplicating them in the plan is observationally a no-op, because every occurrence resolves to the same value within an execution. So they are not affected by this bug and don't need to be blocked.

The guard in the fix uses RexUtil.isDeterministic(...), which is exactly the right discriminator for the affected set. I'll update the PR description to drop the misleading framing of isDynamicFunction().

That is fine, but I think we have no method to discriminate between RAND and CURRENT_TIMESTAMP. Maybe we need to introduce such a tool?

Yes, makes sense.

That should probably be a separate JIRA issue and PR. Coming up with a good name for this property of CURRENT_TIMESTAMP may be difficult.

mihaibudiu · 2026-05-27T18:24:46Z

    sql(sql).withConformance(SqlConformanceEnum.PRESTO).ok();
  }
+
+  /** Test case of


why is SqlToRel connected to these rules?

Because SqlToRelConverter builds its RelNode tree through RelBuilder, and RelBuilder eagerly merges adjacent projects at construction time using the same helper that the planner rules use.

Concretely, RelBuilder.project_ method uses the method RelOptUtil.pushPastProjectUnlessBloat(nodeList, project, config.bloat()) which we are fixing as part of this PR, so I thought of putting a test here as well.

xuzifu666 · 2026-05-28T02:37:13Z

+    // non-deterministic evaluation (e.g. RAND()) into two: one consumed by
+    // the new filter condition, and the original still produced by the
+    // project above. Refuse to transpose in that case.
+    if (!project.getProjects().stream().allMatch(RexUtil::isDeterministic)) {


Blocks transpose (because RAND() exists in project)? eg:

SELECT * FROM ( SELECT RAND() as r, col1 as b FROM emp ) WHERE col1 > 0

…ferences a non-deterministic projected column

…ondition references a non-deterministic projected column

sonarqubecloud · 2026-05-28T06:27:34Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
95.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[CALCITE-7551] Project/Filter/Join transpose and merge rules should n…

61088ed

…ot duplicate non-deterministic expressions (e.g. RAND())

mihaibudiu reviewed May 27, 2026

View reviewed changes

xuzifu666 reviewed May 28, 2026

View reviewed changes

darpan-e6 added 2 commits May 28, 2026 10:13

[CALCITE-7551] Only block filter-project transpose when the filter re…

88d3ad7

…ferences a non-deterministic projected column

[CALCITE-7551] Only block join/semi-join project transpose when the c…

b851108

…ondition references a non-deterministic projected column

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CALCITE-7551] Project/Filter/Join transpose and merge rules should not duplicate non-deterministic expressions (e.g. RAND())#4970

[CALCITE-7551] Project/Filter/Join transpose and merge rules should not duplicate non-deterministic expressions (e.g. RAND())#4970
darpan-e6 wants to merge 3 commits into
apache:mainfrom
darpan-e6:CALCITE-7551

darpan-e6 commented May 27, 2026 •

edited

Loading

Uh oh!

mihaibudiu May 27, 2026

Uh oh!

xiedeyantu May 28, 2026

Uh oh!

darpan-e6 May 28, 2026

Uh oh!

mihaibudiu Jun 1, 2026

Uh oh!

darpan-e6 Jun 2, 2026 •

edited

Loading

Uh oh!

mihaibudiu Jun 2, 2026

Uh oh!

darpan-e6 Jun 3, 2026

Uh oh!

mihaibudiu Jun 3, 2026

Uh oh!

mihaibudiu May 27, 2026

Uh oh!

darpan-e6 May 28, 2026 •

edited

Loading

Uh oh!

xuzifu666 May 28, 2026

Uh oh!

sonarqubecloud Bot commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

darpan-e6 commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Link

Changes Proposed

Reproduction

Scope

Affected code paths and fixes

Tests

Confirmed unaffected (verified by additional probing during investigation)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darpan-e6 Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darpan-e6 May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented May 28, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

darpan-e6 commented May 27, 2026 •

edited

Loading

darpan-e6 Jun 2, 2026 •

edited

Loading

darpan-e6 May 28, 2026 •

edited

Loading