[AURON #2172] Add parquet suite wrappers for Spark versions#2281
Open
myandpr wants to merge 8 commits into
Open
[AURON #2172] Add parquet suite wrappers for Spark versions#2281myandpr wants to merge 8 commits into
myandpr wants to merge 8 commits into
Conversation
6 tasks
slfan1989
approved these changes
May 20, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Extends Auron’s Spark Parquet correctness coverage beyond the existing Spark 3.3 module by adding version-specific wrapper suites and enabling them in each Spark test module’s AuronSparkTestSettings, while omitting suites that do not exist in older Spark versions.
Changes:
- Added
AuronParquet*wrapper test suites for Spark 3.1, 3.2, 3.4, 3.5, 4.0, and 4.1 test modules. - Enabled the new Parquet wrapper suites in each module’s
AuronSparkTestSettings. - Version-gated suite coverage by only adding/enabling wrappers for suites present in that Spark version.
Reviewed changes
Copilot reviewed 188 out of 188 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| auron-spark-tests/spark31/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala | Import parquet package + enable Spark 3.1 Parquet wrapper suites |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark31/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala | Spark 3.1 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala | Import parquet package + enable Spark 3.2 Parquet wrapper suites |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark32/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala | Spark 3.2 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala | Import parquet package + enable Spark 3.4 Parquet wrapper suites |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark34/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala | Spark 3.4 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala | Import parquet package + enable Spark 3.5 Parquet wrapper suites |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark35/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala | Spark 3.5 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala | Import parquet package + enable Spark 4.0 Parquet wrapper suites |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala | Spark 4.0 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala | Import parquet package + enable Spark 4.1 Parquet wrapper suites |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetAvroCompatibilitySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetColumnIndexSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompatibilityTest.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetCompressionCodecPrecedenceSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetEncodingSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdIOSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFieldIdSchemaSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV1Suite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetFileFormatV2Suite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetInteroperabilitySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetIOSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetPartitionDiscoverySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetProtobufCompatibilitySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetQuerySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV1Suite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetRebaseDatetimeV2Suite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaInferenceSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaPruningSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetSchemaSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetTest.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetThriftCompatibilitySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1FilterSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1PartitionDiscoverySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1QuerySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV1SchemaPruningSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2FilterSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2PartitionDiscoverySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2QuerySuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetV2SchemaPruningSuite.scala | Spark 4.1 Parquet suite wrapper |
| auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/AuronParquetVectorizedSuite.scala | Spark 4.1 Parquet suite wrapper |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ShreyeshArangath
requested changes
May 20, 2026
Contributor
ShreyeshArangath
left a comment
ShreyeshArangath
left a comment
There was a problem hiding this comment.
@myandpr I think the tests right now are not running, before we merge more PRs do you mind enabling the correctness tests for the other Spark versions?
auron-spark-tests/pom.xmlneeds six new profiles (spark-3.1, spark-3.2, spark-3.4, spark-3.5, spark-4.0, spark-4.1) each wiring their sparkXX submodule, mirroring the existing spark-3.3 profile.github/workflows/tpcds.ymlneeds sparktests: true added to the test-spark-31, test-spark-32, test-spark-34, test-spark-35-, test-spark-40-, test-spark-41-* jobs. Currently only test-spark-33 has it
You can file a new issue if you'd like
Contributor
Author
Hi @ShreyeshArangath Thanks for pointing this out. Updated the TPC-DS workflow to enable spark correctness tests for the affected Spark versions. The spark test profiles already exist in auron-spark-tests/pom.xml. Could you please take another look? Thanks! |
Contributor
|
Thanks for fixing it! Change looks fine, do you mind looking at the failures? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #2172
Rationale for this change
Spark 3.3 already has Parquet correctness test wrappers. This PR extends the same Parquet correctness coverage to the other supported Spark test modules requested by the issue.
What changes are included in this PR?
AuronSparkTestSettings.Are there any user-facing changes?
No. This PR only adds test coverage.
How was this patch tested?