Add opt-in workaround for OCM CA bundle race condition in acm-mch step

[ccardenosa]

Summary

This PR adds an opt-in workaround to the acm-mch step to handle a race condition in the OCM cluster-manager controller that causes MultiClusterHub deployments to fail intermittently.

⚠️ Important: This workaround is disabled by default and must be explicitly enabled via ENABLE_WORKAROUND_LIST environment variable.

Opt-in Mechanism

Workarounds are controlled via the ENABLE_WORKAROUND_LIST environment variable:

env:
  ENABLE_WORKAROUND_LIST: "[72976]"  # Enable this workaround

Setting	Behavior
ENABLE_WORKAROUND_LIST: "[]"	Default - No workarounds enabled
ENABLE_WORKAROUND_LIST: "[72976]"	Enable OCM CA bundle race condition workaround
ENABLE_WORKAROUND_LIST: "[72976, 12345]"	Enable multiple workarounds

Benefits of Opt-in Design

• Explicit - Workarounds must be deliberately enabled
• Traceable - Each workaround identified by its PR number
• Safe - No unexpected behavior in production jobs
• Easy cleanup - Remove PR number from list once upstream fix is released

Currently Enabled For

Config File	Branch	Purpose
openshift-kni-eco-ci-cd-ztp-left-shifting-kpi__ci-4.21.yaml	ztp-left-shifting-kpi	Virtualised hub deployment
openshift-kni-eco-ci-cd-main__ci-4.21.yaml	main	Metal hub deployment

Related Issues

Issue	Repository	Status
Upstream Fix	open-cluster-management-io/ocm#1309	🔄 Open

Problem

The cluster-manager controller has a race condition where it may create CRDs (ClusterManagementAddOn, ManagedClusterAddOn) before the cert rotation controller creates the CA bundle ConfigMap. When this happens:

1. CRDs are created with caBundle: cGxhY2Vob2xkZXI= (base64 of literal string "placeholder")
2. Webhook conversion fails with InvalidCABundle error
3. CRDs remain in Established: False state
4. API endpoints are not registered
5. MCH fails with: "no matches for kind 'ClusterManagementAddOn' in version 'addon.open-cluster-management.io/v1alpha1'"

Evidence from Failed Prow Jobs

Job Run	Date	ACM Version	MCE Version
#2005051399989104640	Dec 27, 2025	2.16.0-113	2.11.0-142
#2005219283428184064	Dec 28, 2025	2.16.0-114	2.11.0-143

Solution

When ENABLE_WORKAROUND_LIST includes 72976, the workaround only triggers if the initial 30-minute wait for MCH fails:

Normal Flow (workaround disabled or upstream fix merged):
  Apply MCH → Wait 30min → Success ✓

Workaround Flow (enabled + race condition hit):
  Apply MCH → Wait 30min → Fail → Check if 72976 enabled → Detect race condition → Apply workaround → Wait 30min → Success ✓

Workaround Steps (when enabled and race condition detected)

1. Detect - Check if CRDs have the placeholder CA bundle (cGxhY2Vob2xkZXI=)
2. Patch Services - Add service.beta.openshift.io/serving-cert-secret-name annotation to webhook services
3. Wait for Secrets - Let service-ca-operator create TLS certificates
4. Create ConfigMap - Create ca-bundle-configmap from serving cert secret
5. Patch CRDs - Extract real CA bundle from secrets and update CRDs
6. Force Reconciliation - Restart cluster-manager and MCE operator

Design Decisions

Decision	Rationale
Opt-in via env var	No impact on jobs that don't explicitly enable it
Workaround only on failure	Doesn't add latency to normal deployments
Specific detection	Only triggers for this exact issue (placeholder CA bundle)
PR number as identifier	Easy to track and remove once fixed

Cleanup Path

Once ocm#1309 is merged and released in ACM/MCE:

1. Remove 72976 from ENABLE_WORKAROUND_LIST in affected configs
2. The workaround functions become dead code (never triggered)
3. They can be removed in a future cleanup PR

Testing

• Bash syntax check passes
• Workaround successfully applied manually on live cluster (sno-vhub-0)
• MCH reached Running status (22/22 components) after workaround
• Rehearsal job verified workaround works in CI

Changes

• ci-operator/step-registry/acm/mch/acm-mch-commands.sh
  • Added ENABLE_WORKAROUND_LIST opt-in mechanism
  • Added workaround functions for OCM CA bundle race condition
  • Modified main wait logic to check if workaround is enabled before applying
• ci-operator/step-registry/acm/mch/acm-mch-ref.yaml
  • Documented ENABLE_WORKAROUND_LIST environment variable
• ci-operator/config/openshift-kni/eco-ci-cd/openshift-kni-eco-ci-cd-ztp-left-shifting-kpi__ci-4.21.yaml
  • Added ENABLE_WORKAROUND_LIST: "[72976]"
• ci-operator/config/openshift-kni/eco-ci-cd/openshift-kni-eco-ci-cd-main__ci-4.21.yaml
  • Added ENABLE_WORKAROUND_LIST: "[72976]"

/cc @openshift/openshift-team-edge-ztp

[openshift-ci]

@ccardenosa: GitHub didn't allow me to request PR reviews from the following users: openshift/openshift-team-edge-ztp.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

Summary

This PR adds a workaround to the acm-mch step to handle a race condition in the OCM cluster-manager controller that causes MultiClusterHub deployments to fail intermittently.

Related Issues

Issue	Repository	Status
Upstream Fix	open-cluster-management-io/ocm#1309	🔄 Open

Problem

The cluster-manager controller has a race condition where it may create CRDs (ClusterManagementAddOn, ManagedClusterAddOn) before the cert rotation controller creates the CA bundle ConfigMap. When this happens:

1. CRDs are created with caBundle: cGxhY2Vob2xkZXI= (base64 of literal string "placeholder")
2. Webhook conversion fails with InvalidCABundle error
3. CRDs remain in Established: False state
4. API endpoints are not registered
5. MCH fails with: "no matches for kind 'ClusterManagementAddOn' in version 'addon.open-cluster-management.io/v1alpha1'"

Evidence from Failed Prow Jobs

Job Run	Date	ACM Version	MCE Version
#2005051399989104640	Dec 27, 2025	2.16.0-113	2.11.0-142
#2005219283428184064	Dec 28, 2025	2.16.0-114	2.11.0-143

Solution

This PR adds a workaround that only triggers if the initial 30-minute wait for MCH fails:

Normal Flow (upstream fix merged):
 Apply MCH → Wait 30min → Success ✓

Workaround Flow (race condition hit):
 Apply MCH → Wait 30min → Fail → Detect race condition → Apply workaround → Wait 30min → Success ✓

Workaround Steps

1. Detect - Check if CRDs have the placeholder CA bundle (cGxhY2Vob2xkZXI=)
2. Patch Services - Add service.beta.openshift.io/serving-cert-secret-name annotation to webhook services
3. Wait for Secrets - Let service-ca-operator create TLS certificates
4. Patch CRDs - Extract real CA bundle from secrets and update CRDs
5. Force Reconciliation - Restart MCE operator to pick up changes
6. Retry Wait - Wait again for MCH to reach Running status

Design Decisions

Decision	Rationale
Workaround only on failure	Doesn't add latency to normal deployments
Specific detection	Only triggers for this exact issue (placeholder CA bundle)
Dead code after fix	Once upstream PR #1309 is merged, detection returns false and workaround never runs
Clear documentation	Functions are well-commented with links to upstream PR

Cleanup Path

Once ocm#1309 is merged and released in ACM/MCE:

1. The first 30min wait will always succeed
2. The workaround functions become dead code
3. They can be removed in a future cleanup PR

Testing

• Bash syntax check passes
• Workaround successfully applied manually on live cluster (sno-vhub-0)
• MCH reached Running status after workaround

Changes

• ci-operator/step-registry/acm/mch/acm-mch-commands.sh
• Added workaround functions for OCM CA bundle race condition
• Modified main wait logic to detect and remediate the issue on failure

/cc @openshift/openshift-team-edge-ztp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-ztp-left-shifting-kpi-ci-4.21-telcov10n-virtualised-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-main-ci-4.21-telcov10n-metal-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-main-ci-4.21-telcov10n-metal-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-ztp-left-shifting-kpi-ci-4.21-telcov10n-virtualised-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

✅ Workaround Verified Working

The rehearsal job confirms the workaround successfully resolves the OCM CA bundle race condition:

Successful run: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/72976/rehearse-72976-periodic-ci-openshift-kni-eco-ci-cd-ztp-left-shifting-kpi-ci-4.21-telcov10n-virtualised-single-node-hub-ztp/2005316456039845888

Execution Summary

MCH did not reach Running status in the first attempt.
Checking for known issues and applying workarounds if needed...

============================================================
Applying OCM CA Bundle Race Condition Workaround
Upstream fix: https://github.com/open-cluster-management-io/ocm/pull/1309
============================================================

Checking for OCM CA bundle race condition (PR #1309)...
DETECTED: clustermanagementaddons CRD has placeholder CA bundle

Step 1/6: Patching webhook services with serving-cert-secret-name annotation...
  ✓ All 3 webhook services patched

Step 2/6: Waiting for service-ca-operator to create secrets...
  ✓ All 3 secrets created by service-ca-operator

Step 3/6: Creating ca-bundle-configmap from serving cert...
  ✓ ConfigMap created with real CA bundle

Step 4/6: Patching CRDs with real CA bundles...
  (CRDs auto-updated after configmap creation)

Step 5/6: Verifying CRDs are now Established...
  ✓ clustermanagementaddons.addon.open-cluster-management.io: Established=True
  ✓ managedclusteraddons.addon.open-cluster-management.io: Established=True

Step 6/6: Restarting cluster-manager and forcing reconciliation...
  ✓ cluster-manager deployment restarted
  ✓ multicluster-engine-operator restarted
  ✓ multiclusterengine annotated for reconciliation

============================================================
Workaround applied successfully!
============================================================

multiclusterhub.operator.open-cluster-management.io/multiclusterhub condition met
MCH reached Running status after applying workaround!
Success! ACM 2.16.0-114 is Running

This workaround will be needed until the upstream fix (open-cluster-management-io/ocm#1309) is merged and released in a future ACM/MCE version.

[ccardenosa]

/assign @sg-rh

Could you please review this workaround?

[ccardenosa]

/pj-rehearse ack

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/assign @vboulos

Could you please review this workaround?

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-main-ci-4.21-telcov10n-metal-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-main-ci-4.21-telcov10n-metal-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-main-ci-4.21-telcov10n-metal-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse ack

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-main-ci-4.21-telcov10n-metal-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-ztp-left-shifting-kpi-ci-4.21-telcov10n-virtualised-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse ack

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ccardenosa
Once this PR has been reviewed and has the lgtm label, please ask for approval from sg-rh. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift-kni/eco-ci-cd/OWNERS [ccardenosa]
• ci-operator/step-registry/acm/mch/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-main-ci-4.21-telcov10n-metal-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

/pj-rehearse periodic-ci-openshift-kni-eco-ci-cd-ztp-left-shifting-kpi-ci-4.21-telcov10n-virtualised-single-node-hub-ztp

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[ccardenosa]

🔬 Bug Still Exists - Proved via Binary Analysis (Jan 6, 2026)

Background

Rehearsal job #2008550209683984384 succeeded without triggering the workaround. To verify whether the bug still exists or was fixed upstream, I performed binary analysis on the deployed cluster-manager.

Cluster Status

Item	Value
Cluster	sno-vhub-0 (OCP 4.21.0-rc.0)
ACM Version	2.16.0-125
MCE Version	2.11.0-154
cluster-manager image	registry.redhat.io/multicluster-engine/registration-operator-rhel9@sha256:02a4c491fc3d36022f7ab3a8847b7dca8a9d4471312ca7b72bf6fe629b365602

Binary Analysis

I searched the running cluster-manager binary for the buggy "placeholder" string:

$ oc exec -n multicluster-engine $POD -- grep -ao "placeholder" /registration-operator | wc -l
172

Result: The buggy code is still compiled into the binary (172 occurrences of "placeholder").

CRD State Check

Despite the bug existing in code, the CRDs have real certificates:

$ oc get crd clustermanagementaddons.addon.open-cluster-management.io \
    -o jsonpath="{.spec.conversion.webhook.clientConfig.caBundle}" | base64 -d | head -c 50

-----BEGIN CERTIFICATE-----
MIIDPzCCAiegAwIBAgIIBhmpSdaTem8wDQYJKoZIhvcNAQE

✅ Real certificate (not "placeholder")

Conclusion

Evidence	Result
"placeholder" string in binary	172 occurrences → Bug code path exists
"ca-bundle-configmap" in binary	2 references → ConfigMap lookup exists
CRD's actual CA bundle	-----BEGIN CERTIFICATE----- → Real cert
Was bug triggered?	❌ No

The race condition didn't trigger because certRotationController happened to create ca-bundle-configmap BEFORE clustermanagerController tried to read it.

┌────────────────────────────────────────────────────────────────────┐
│  BINARY PROOF: Bug exists (172x "placeholder" in code)             │
├────────────────────────────────────────────────────────────────────┤
│  RUNTIME STATE: Bug did NOT trigger (real cert in CRD)             │
├────────────────────────────────────────────────────────────────────┤
│  REASON: Race won by timing luck, not by code fix                  │
│          certRotationController was faster this time               │
└────────────────────────────────────────────────────────────────────┘

Why This Workaround is Still Needed

1. ✅ Upstream fix (ocm#1309) NOT merged - still under review
2. ✅ MCE 2.11.0-154 has NOT been patched - confirmed via binary analysis
3. ✅ Race condition is non-deterministic - can still trigger under:
   • Slower infrastructure (storage, network, CPU)
   • Higher cluster load during deployment
   • Unlucky Kubernetes pod scheduling

The workaround remains necessary until the upstream fix is merged and released in ACM/MCE.

[openshift-ci-robot]

[REHEARSALNOTIFIER]
@ccardenosa: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
periodic-ci-stolostron-policy-collection-main-ocp4.20-interop-opp-aws	N/A	periodic	Registry content changed
periodic-ci-stolostron-acmqe-autotest-main-acm-ocp4.17-lp-interop-acm-interop-aws	N/A	periodic	Registry content changed
periodic-ci-openshift-kni-eco-ci-cd-ztp-left-shifting-kpi-ci-4.21-telcov10n-virtualised-single-node-hub-ztp	N/A	periodic	Ci-operator config changed
periodic-ci-RedHatQE-interop-testing-master-acm-cnv-ocp4.19-p2p-interop-acm-cnv-p2p-aws-419	N/A	periodic	Registry content changed
periodic-ci-stolostron-acmqe-autotest-main-acm-ocp4.16-lp-interop-acm-interop-aws	N/A	periodic	Registry content changed
periodic-ci-openshift-kni-eco-ci-cd-main-ci-4.21-telcov10n-metal-single-node-hub-ztp	N/A	periodic	Ci-operator config changed
periodic-ci-stolostron-acmqe-autotest-main-acm-ocp4.15-lp-interop-acm-interop-aws	N/A	periodic	Registry content changed
periodic-ci-stolostron-acmqe-autotest-main-acm-ocp4.14-lp-interop-acm-interop-aws	N/A	periodic	Registry content changed
periodic-ci-stolostron-policy-collection-main-ocp4.21-interop-opp-aws	N/A	periodic	Registry content changed
periodic-ci-stolostron-policy-collection-main-ocp4.21-interop-opp-vsphere	N/A	periodic	Registry content changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[ccardenosa]

/pj-rehearse ack

[openshift-ci-robot]

@ccardenosa: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.