Fix KubernetesExecutor scheduler crash from unpicklable pod_override by ephraimbuddy · Pull Request #68831 · apache/airflow

ephraimbuddy · 2026-06-22T09:34:37Z

When the scheduler runs in-cluster, the kubernetes client installs a process-global default Configuration whose refresh_api_key_hook is a local closure. Under kubernetes-client v36, deserializing a V1Pod copies that Configuration onto the pod and every nested model. A task that sets a V1Pod pod_override therefore produces a pod that pickle cannot serialize, and the KubernetesExecutor pickles each task onto a multiprocessing queue synchronously in the scheduler loop, so the scheduler crashes in a loop and no task can be launched.

Deserialize pods through an ApiClient built with a fresh Configuration so that neither the pod nor any nested model captures the in-cluster global. This is applied at the point the config gets attached, so every consumer of a deserialized pod stays picklable: airflow-core's serialization and unpickling repair paths, and the cncf provider's worker-side reconstruction.

Provider counter part of this PR: #68848

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Claude Opus 4.8

phanikumv

The same ApiClient() deserialize code still lives in the below two paths.

  - providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/job.py — L381–L382 (api_client = ApiClient() → _ApiClient__deserialize_model(job, k8s.V1Job))
  - providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py — L39–L40 (api_client = ApiClient() → _ApiClient__deserialize_model(obj, new_class))

Neither is on the path where a deserialized kubernetes model gets pickle-serialized, however worth fixing them as a follow up in another PR..?

ephraimbuddy · 2026-06-22T11:11:47Z

The same ApiClient() deserialize code still lives in the below two paths.
  - providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/job.py — L381–L382 (api_client = ApiClient() → _ApiClient__deserialize_model(job, k8s.V1Job))
  - providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py — L39–L40 (api_client = ApiClient() → _ApiClient__deserialize_model(obj, new_class))
Neither is on the path where a deserialized kubernetes model gets pickle-serialized, however worth fixing them as a follow up in another PR..?

Yeah, we can do a follow up since those are not on the pickle path

eladkal

Can we split this PR to core and providers?

ephraimbuddy · 2026-06-22T15:35:13Z

Can we split this PR to core and providers?

~~The core change is what's important. Provider change can be released much later. i don't think we should split.~~

Splitting

ephraimbuddy · 2026-06-22T15:59:16Z

Can we split this PR to core and providers?

~~The core change is what's important. Provider change can be released much later. i don't think we should split.~~

Splitting

PR: #68848

phanikumv

Both the PRs are currently missing the airflow-core scheduler fix. Please restore yesterday's serialized_objects.py + sqlalchemy.py changes

Code changed post approval.

amoghrajesh

The fix is fine for pods deserialized from yaml / dicst but misses directly instantiated V1Pod objects via the executor_config.

When a user constructs a V1Pod directly in their dag, it never goes through `ApiClient.__deserialize_model. Requesting changes to avoid accidental merge.

ephraimbuddy · 2026-06-23T07:51:21Z

The fix is fine for pods deserialized from yaml / dicst but misses directly instantiated V1Pod objects via the executor_config.

When a user constructs a V1Pod directly in their dag, it never goes through `ApiClient.__deserialize_model. Requesting changes to avoid accidental merge.

Yeah, there was a mix up. I have pushed the actual change.

Copilot

Pull request overview

This PR prevents scheduler crashes in KubernetesExecutor when a task sets pod_override to a kubernetes.client.models.V1Pod that becomes unpicklable in-cluster (due to kubernetes-client v36 attaching the process-global in-cluster Configuration with an unpicklable refresh_api_key_hook closure). It centralizes pod dict deserialization through an ApiClient constructed with a fresh Configuration, ensuring deserialized pods (and nested models) remain picklable across core serialization and the “repair after unpickling” path.

Changes:

Add deserialize_pod_dict() that deserializes pod dicts via ApiClient(configuration=Configuration()) to avoid capturing the in-cluster global Configuration.
Update both BaseSerialization.deserialize() (POD branch) and ensure_pod_is_valid_after_unpickling() to use the new safe deserialization helper.
Add regression tests covering both the serialization/deserialization path and the unpickling repair path for picklability in-cluster.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
`airflow-core/src/airflow/utils/sqlalchemy.py`	Introduces safe pod dict deserialization with a fresh Kubernetes `Configuration` and reuses it in the unpickling repair path.
`airflow-core/src/airflow/serialization/serialized_objects.py`	Switches POD deserialization to the safe helper to keep deserialized pods picklable.
`airflow-core/tests/unit/utils/test_sqlalchemy.py`	Adds a test ensuring the unpickling repair path produces a picklable pod even with a dirty in-cluster default `Configuration`.
`airflow-core/tests/unit/serialization/test_serialized_objects.py`	Adds a test ensuring serialized/deserialized pods do not capture an unpicklable in-cluster default `Configuration`.

phanikumv

LGTM now after the last commit

amoghrajesh

Some nits, otherwise LGTM!

amoghrajesh · 2026-06-23T09:05:03Z

+        the pod it must round-trip through a fresh ``Configuration`` so it stays picklable onto the
+        KubernetesExecutor queue.
+        """
+        from kubernetes.client import Configuration


Same here (but this one can stay actually)

amoghrajesh · 2026-06-23T09:07:09Z

+        assert decoded.local_vars_configuration.refresh_api_key_hook is None
+        assert decoded.spec.containers[0].local_vars_configuration.refresh_api_key_hook is None
+
+        _has_kubernetes.cache_clear()


Is this really needed? I guess its bleeding from other tests but yeah just flagging

When the scheduler runs in-cluster, the kubernetes client installs a process-global default Configuration whose refresh_api_key_hook is a local closure. Under kubernetes-client v36, deserializing a V1Pod copies that Configuration onto the pod and every nested model. A task that sets a V1Pod pod_override therefore produces a pod that pickle cannot serialize, and the KubernetesExecutor pickles each task onto a multiprocessing queue synchronously in the scheduler loop, so the scheduler crashes in a loop and no task can be launched. Deserialize pods through an ApiClient built with a fresh Configuration so that neither the pod nor any nested model captures the in-cluster global. This is applied where the config gets attached -- airflow-core's serialization and unpickling repair paths -- so every consumer of a deserialized pod stays picklable.

Move the deserialize_pod_dict import to module top-level, hoist pickle and Configuration to top-level imports in the serialization test, and drop the redundant _has_kubernetes cache_clear calls that were copied from a sibling test.

ephraimbuddy requested review from ashb, bolkedebruin, hussein-awala, jedcunningham and jscheffl as code owners June 22, 2026 09:34

boring-cyborg Bot added area:DAG-processing area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Jun 22, 2026

phanikumv reviewed Jun 22, 2026

View reviewed changes

vatsrahul1001 mentioned this pull request Jun 22, 2026

Fix KubernetesExecutor scheduler crash when pod_override is queued in-cluster #68819

Closed

1 task

ephraimbuddy force-pushed the fix-v1pod-unpicklable-config-in-cluster branch from 736cbee to f029434 Compare June 22, 2026 12:31

phanikumv previously approved these changes Jun 22, 2026

View reviewed changes

ephraimbuddy force-pushed the fix-v1pod-unpicklable-config-in-cluster branch from f029434 to 902fbf3 Compare June 22, 2026 15:25

eladkal approved these changes Jun 22, 2026

View reviewed changes

ephraimbuddy requested review from amoghrajesh and kaxil as code owners June 22, 2026 15:58

ephraimbuddy force-pushed the fix-v1pod-unpicklable-config-in-cluster branch 2 times, most recently from 153aedd to 5490346 Compare June 22, 2026 16:01

phanikumv reviewed Jun 23, 2026

View reviewed changes

amoghrajesh requested changes Jun 23, 2026

View reviewed changes

amoghrajesh mentioned this pull request Jun 23, 2026

Make cncf.kubernetes model deserialization picklable in-cluster #68848

Open

1 task

ephraimbuddy force-pushed the fix-v1pod-unpicklable-config-in-cluster branch from 5490346 to f8c5f27 Compare June 23, 2026 07:50

ephraimbuddy requested a review from Copilot June 23, 2026 07:51

Copilot started reviewing on behalf of ephraimbuddy June 23, 2026 07:52 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

phanikumv approved these changes Jun 23, 2026

View reviewed changes

phanikumv requested a review from amoghrajesh June 23, 2026 09:03

amoghrajesh approved these changes Jun 23, 2026

View reviewed changes

ephraimbuddy force-pushed the fix-v1pod-unpicklable-config-in-cluster branch from f8c5f27 to fb76af2 Compare June 23, 2026 09:58

Address review feedback on pod deserialization fix

154a011

Move the deserialize_pod_dict import to module top-level, hoist pickle and Configuration to top-level imports in the serialization test, and drop the redundant _has_kubernetes cache_clear calls that were copied from a sibling test.

ephraimbuddy force-pushed the fix-v1pod-unpicklable-config-in-cluster branch from fb76af2 to 154a011 Compare June 23, 2026 10:08

Conversation

ephraimbuddy commented Jun 22, 2026 • edited by phanikumv Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Was generative AI tooling used to co-author this PR?

Uh oh!

phanikumv left a comment

Choose a reason for hiding this comment

Uh oh!

ephraimbuddy commented Jun 22, 2026

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

ephraimbuddy commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ephraimbuddy commented Jun 22, 2026

Uh oh!

phanikumv left a comment

Choose a reason for hiding this comment

Uh oh!

amoghrajesh left a comment

Choose a reason for hiding this comment

Uh oh!

ephraimbuddy commented Jun 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

phanikumv left a comment

Choose a reason for hiding this comment

Uh oh!

amoghrajesh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amoghrajesh Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

amoghrajesh Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ephraimbuddy commented Jun 22, 2026 •

edited by phanikumv

Loading

ephraimbuddy commented Jun 22, 2026 •

edited

Loading