`SPARK-17091: Convert IN predicate to Parquet filter push-down"` fails for both native_iceberg_compat

[parthchandra]

Describe the bug

The test from Spark SQL tests fails in native_iceberg_compat
The reason for the failure is that Spark does not filter out all the records in the scan while datafusion does, leading to a difference in the number of records actually returned by the scan. The test explicitly checks the number of records returned by the scan.
Eventually both return correct (i.e. identical) results.

The test needs to be fixed for native_iceberg_compat so it does not fail.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[kazuyukitanimura]

Looks like there is a minimum number of records that Spark filter push down always returns, probably for performance reasons. then later the filter exec actually filters again to make correct results
Comet filters everything with the pushdown. Perhaps copying the Spark logic improves the performance

[parthchandra]

Predicate pushdown into scans is pretty much best effort so there we have to have a Filter to ensure that the correct results are returned. In general the number of records filtered by a scan is totally dependent on the scan implementation so checking the number of records returned needs to be implementation specific.
Performance wise there is an issue in Datafusion's implementation, but in general it is always better to evaluate the predicate as early as possible.

[andygrove]

Closing as obsolete: the native_iceberg_compat scan implementation was removed (see #4019, #4362, #4363), so this test no longer runs.