Implement tiered CI approach

[andygrove]

What is the problem the feature request solves?

CI is getting too painful.

I would like to explore a tiered CI approach:

• PR CI: fast subset (Latest 3.x and 4.x Spark versions, smoke tests, lint, Comet Scala + Comet SQL tests)
• Merge queue: Test all Spark versions
• Nightly/weekly: full matrix, fuzz tests, benchmarks, long-running correctness suites

Describe the potential solution

No response

Additional context

No response

[andygrove]

Proposing a minimal first step toward tiered CI: split the existing spark_sql_test.yml matrix so PRs run a 2-version subset and the GitHub merge queue covers the remaining versions. Push to main and workflow_dispatch continue to run all four versions.

Proposed split

Trigger	Spark versions	Jobs (modules x Sparks)
pull_request	3.5.8, 4.1.1	7 x 2 = 14
merge_group	3.4.3, 4.0.2	7 x 2 = 14
push to main	3.4.3, 3.5.8, 4.0.2, 4.1.1	7 x 4 = 28 (unchanged)

Net effect per PR: -14 spark-sql-test/* jobs from this workflow. The merge queue adds 14 jobs, but each PR triggers merge_group only once when it enters the queue.

Mechanism

1. Add merge_group: to the workflow on: triggers (no paths: filter so queue runs always execute).
2. Replace the static config: matrix with an event-conditional \${{ ... }} expression that selects the Spark version subset based on github.event_name.
3. Add a single rollup job spark-sql-test-status (needs: spark-sql-test, if: always()) that becomes the sole required status check. This decouples branch protection from matrix shape so future reshapes do not require re-editing required checks.

Branch protection coordination

The per-version status check names (spark-sql-{module}/spark-{full}-jdk{java}) will no longer be produced on PRs after this change. Branch protection must be updated to require Spark SQL Tests Status instead, before merging. Suggested sequence: remove per-version required checks, merge the workflow change, verify a merge_group run, then add the rollup as the sole required check.

Scope

In scope: one file only, .github/workflows/spark_sql_test.yml.

Out of scope (future follow-ups): pr_build_linux.yml (Comet's own tests still run all five Spark profiles on PR), iceberg_spark_test.yml, macOS, benchmarks, fuzz, nightly cron tier. Each of these can be tackled as its own follow-up.

Branch

The proposed change is on andygrove:ci/tiered-spark-sql-test-matrix. Not opening a PR yet, pending feedback on the approach and the branch-protection coordination plan above.

[kazuyukitanimura]

Merge queue looks promising +1

[comphead]

Merge queue looks promising +1

with MQ there are some drawbacks, so effectively you running CI twice and if CI fails during MQ cycle your PR is removed from the queue.

I'm still playing with #4292 to offboard heavyweight tasks, but need some time, straightfwd CI port from Spark to Comet doesn't properly work

[andygrove]

Merge queue looks promising +1

with MQ there are some drawbacks, so effectively you running CI twice and if CI fails during MQ cycle your PR is removed from the queue.

I'm still playing with #4292 to offboard heavyweight tasks, but need some time, straightfwd CI port from Spark to Comet doesn't properly work

There are some nice advantages to #4292 approach:

• contributors do not have to wait for maintainers to approve workflows
• contributors get their own resource allocation so do have to wait with tasks queued due to shared ASF resources
• we can probably use larger runners since it runs in contributor account
• also we could parameterize it so users can choose runners (paid or free) by overrideing variable:

jobs:
  build:
    runs-on: ${{ vars.RUNNER || 'ubuntu-latest' }}