Support extra field for Coordinator

[jason810496]

Why

While verifying KuberntesExecutor setup for Multi-Lang feature, I found that we need a new config mapping to represent coordinator -> pod_template_file. For example, KubernetesExecutor needs image with JVM runtime for Java-Task.

How

Here're several directions that I went through:

1. (Current choice) Generic extra object on the [sdk] coordinators entry. Regarding the extra field naming, I also considered executor, deployment, etc. But I prefer as extra to make it more extendable as user might introduce other optional fields in extra for non-deployment purpose (e.g. referencing conn_id).

{
  "jdk-17": {
    "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
    "kwargs": {"java_executable": "/usr/lib/jvm/java-17-openjdk/bin/java", "jvm_args": ["-Xmx1024m"]},
    "extra": {"pod_template_file": "/opt/airflow/pod_templates/java.yaml"}
  }
}

2. An optional pod_template_file field at first level -- Cons: coupling the optional / provider-level field at first level, needs coordinator code change for every new field.

{
  "jdk-17": {
    "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
    "kwargs": {"java_executable": "/usr/lib/jvm/java-17-openjdk/bin/java", "jvm_args": ["-Xmx1024m"]},
    "pod_template_file": "/opt/airflow/pod_templates/java.yaml"
  }
}

3. An optional pod_template_file in kwargs level -- Same cons as point 2.

{
  "jdk-17": {
    "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
    "kwargs": {
      "java_executable": "/usr/lib/jvm/java-17-openjdk/bin/java",
      "jvm_args": ["-Xmx1024m"],
      "pod_template_file": "/opt/airflow/pod_templates/java.yaml"
    },
  }
}

4. A fully decouple config setup -- AIRFLOW__KUBERNETES__COORDINATOR_TO_POD_TEMPLATE_FILE

AIRFLOW__SDK__COORDINATORS='{"jdk-17": {"classpath": "airflow.sdk.coordinators.java.JavaCoordinator", "kwargs": {"jars_root": ["/files/java-bundles"]}}}'
AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "jdk-17"}'
AIRFLOW__KUBERNETES__COORDINATOR_TO_POD_TEMPLATE_FILE='{"jdk-17": "/opt/airflow/pod_templates/java.yaml"}'

The cons is quite obvious, user have to understand the queue -> coordinator -> pod_template relationship to setup the Multi-Lang with KubernetesExecutor properly. IMO, it's a too complicated, and it's fine to store the optional metadata in core level config but parsing the optional info at provider level.

What

• Add an extra field to _CoordinatorSpec (kept separate from kwargs) and CoordinatorManager.extra_for_queue.
  • The CoordinatorManager.extra_for_queue method will not instantiate the coordinator instance.

Next

• Decide pod_template and image based on Coordinator for lang-SDK tasks on KubernetesExecutor #68713

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Opus 4.8) following the guidelines

[jason810496]

Hi all,
Would appreciate review on this one to get in beta2 when you have a moment, thanks.

[jason810496]

The Mypy provider CI failure is not related so I will merge this.

[github-actions]

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-3-test