Extended tests now take over 2 hours to run

[alamb]

Describe the bug

@nuno-faria and @xudong963 noticed on #19918 that the extended tests now take well over 2 hours to run

The ci takes so long to run.

Updated the branch to see what will happen

It looks like the extended tests workflow takes one extra hour since a few days ago:

STATUS  TITLE                                                                                                BRANCH                            EVENT         ID           ELAPSED   AGE
✓       Improve error messages with nicer formatting of Date and Time types                                  emilk/less-debug-formatting       pull_request  21289130254  2h28m49s  about 4 days ago
✓       Disallow positional struct casting when field names don’t overlap                                    positional-fallback-19841         pull_request  21285795721  2h43m17s  about 5 days ago
✓       feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations  plann-recursive-subquery-cleanup  pull_request  21280272900  2h49m11s  about 5 days ago
✓       Fix struct casts to align fields by name (prevent positional mis-cast…                               main                              push          21274179200  2h41m15s  about 5 days ago
✓       Fix struct casts to align fields by name (prevent positional mis-casts)                              struct-casting-17285              pull_request  21270852353  2h43m42s  about 5 days ago
✓       Support API for "pre-image" for pruning predicate evaluation (#19722)                                main                              push          21267996805  1h28m7s   about 5 days ago
✓       feat(datafusion-spark): implement spark compatible  `unhex` function …                               main                              push          21266842171  1h27m21s  about 5 days ago
✓       Refactor ListArray hashing to consider only sliced values (#19500)                                   main                              push          21266824967  1h32m50s  about 5 days ago
✓       Improve error message when string functions receive Binary types (#19…                               main                              push          21266584166  1h27m34s  about 5 days ago
✓       fix: union should retrun error instead of panic when input schema's l…                               main                              push          21266396262  1h28m41s  about 5 days ago

To Reproduce

This seems to have started after this PR

• Fix struct casts to align fields by name (prevent positional mis-casts) #19674

I verified the timings via the github UI too:

https://github.com/apache/datafusion/commits/main/?after=20d5f06a554f39407e52dd1d4848267dbc1467ce+34

Expected behavior

Extended tests should not take so long

Additional context

No response

[Tushar7012]

Hi @alamb , I'd like to work on this issue!

Root Cause Analysis

The regression starts after PR #19674, which introduced name-based struct field matching. The key overhead comes from:

1. cast_struct_column() in nested_struct.rs - performs field-by-name matching with recursive struct handling
2. validate_struct_compatibility() - comprehensive compatibility checks on every struct cast
3. Additional validation in ColumnarValue::cast_to - routes all struct casts through new logic

Proposed Approach

1. Profile extended tests to identify which tests are most affected
2. Optimize hot paths:
   • Add fast-path when source/target schemas are identical
   • Skip redundant re-validation when already verified at planning time
   • Consider #[inline] hints for frequently-called casting functions
3. Reduce overhead:
   • Early bailout in validate_struct_compatibility() for identical types
   • Lazy evaluation for expensive field matching operations

Next Steps

1. Set up local profiling to identify exact bottlenecks
2. Compare test durations before/after PR Fix struct casts to align fields by name (prevent positional mis-casts) #19674
3. Submit targeted optimization PR
   Could I please be assigned to this issue?