Skip to content

Restore Sort unparser guard for correct ORDER BY placement#20658

Open
krinart wants to merge 4 commits intoapache:mainfrom
spiceai:viktor/fix-unparser-origin
Open

Restore Sort unparser guard for correct ORDER BY placement#20658
krinart wants to merge 4 commits intoapache:mainfrom
spiceai:viktor/fix-unparser-origin

Conversation

@krinart
Copy link
Contributor

@krinart krinart commented Mar 3, 2026

Which issue does this PR close?

Restore the already_projected() guard in the Sort case of select_to_sql_recursively. Without it, queries with ORDER BY expressions generate invalid SQL when the Sort node follows an outer Projection.

Problem

Two regressions in the DuckDB federation unparser after upgrading to DataFusion 52:

  1. clickbench q25: Queries like SELECT "SearchPhrase" ... ORDER BY to_timestamp("EventTime") LIMIT 10 produce ORDER BY outside the subquery, referencing table names ("hits") that are out of scope.
  2. tpcds q36: ORDER BY with GROUPING() expressions loses the derived_sort subquery alias, placing sort references outside their valid scope.

Root Cause

DF52's optimizer merges Limit into Sort as a fetch parameter, changing the plan shape from Projection → Limit → Sort → ... to Projection → Sort(fetch) → .... The Sort case previously had a guard that wrapped the sort into a derived_sort subquery when already_projected() was true. This guard was removed in DF52, causing ORDER BY and LIMIT to be placed on the outer query where inner table references are invalid.

Fix

Re-add the guard at the top of the Sort match arm in select_to_sql_recursively:

 if select.already_projected() {
     return self.derive_with_dialect_alias("derived_sort", plan, relation, false, vec![]);
 }

@alamb alamb added the regression Something that used to work no longer does label Mar 10, 2026
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @krinart

// the inner Projection's raw expressions. The optimizer may add
// sort expressions to the inner Projection without aliases, while
// the Sort node's expressions carry aliases from the original plan.
let mut expr = sort.expr.clone();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It apparently isn't quite the same, but I am not sure if the different matters. Let's leave it like this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to use it here - but test_complex_order_by_with_grouping test failed.

@alamb
Copy link
Contributor

alamb commented Mar 12, 2026

@krinart -- this PR also has some conflicts that need to be resolved

Maybe you can also address the comment as well? I would like to backport this to the 52 branch for the next patch release

Screenshot 2026-03-12 at 8 54 43 AM

@alamb
Copy link
Contributor

alamb commented Mar 12, 2026

I also filed #20905 to track this issue so we can merge it back to 52

@alamb
Copy link
Contributor

alamb commented Mar 13, 2026

We are trying to prepare a 52 and 53 release, and since this issue is a regression I would like to fix it today. I can't push to your branch (as it is in the spiceai fork) so I will likely open a new PR to get it ready

@alamb
Copy link
Contributor

alamb commented Mar 13, 2026

Here is a proposed fix: #20923

@krinart krinart closed this Mar 14, 2026
@alamb alamb reopened this Mar 15, 2026
@alamb
Copy link
Contributor

alamb commented Mar 15, 2026

No worries, thank you @krinart -- I reopened this PR and as it is approved I'll merge it when it passes

@alamb
Copy link
Contributor

alamb commented Mar 16, 2026

Hi @krinart -- there appears to be a few test failures. Any chance you can resolve them (You can use the changes on #20923 as a guide)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

regression Something that used to work no longer does sql SQL Planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incorrect ORDER BY in unparser after DF 52 upgrade

2 participants