OCPBUGS-77056: Release lock before WaitForCacheSync in secret monitor

[bentito]

Summary

Routes using spec.tls.externalCertificate trigger a call to
AddSecretEventHandler for each route at router startup. The previous
implementation held the global write mutex (s.lock) for the entire duration
of the call, including the WaitForCacheSync round-trip to the API server and
etcd (~100–600ms per secret). This caused all N secret registrations to run
strictly serially:

Scale	etcd latency	Startup time
112 secrets	~587ms/secret	65s
1,738 secrets	~1,414ms/secret	40.9 minutes

The customer reports ~2,000 external cert routes causing router pods to fail
their 120s startup probe repeatedly and enter CrashLoopBackOff.

Root Cause

In addSecretEventHandler, WaitForCacheSync was called inside
defer s.lock.Unlock(), meaning the global write lock was held while waiting
for each secret's informer to complete its initial LIST to etcd. No other
registration could begin until the previous one finished.

Fix

Write the new monitoredItem into s.monitors before releasing the lock
(so concurrent callers for the same secret safely reuse the same informer),
then release the lock before WaitForCacheSync. This allows concurrent
AddSecretEventHandler calls for different secrets to perform their etcd
round-trips in parallel, reducing startup time from O(N × api_latency) to
O(api_latency).

On sync failure the map entry is cleaned up so a future caller can retry with
a fresh informer.

Testing

All existing unit tests in pkg/secret/... pass.

A concurrent registration test should be added (tracked separately).

Fixes: OCPBUGS-77056

[openshift-ci-robot]

@bentito: This pull request references Jira Issue OCPBUGS-77056, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Routes using spec.tls.externalCertificate trigger a call to
AddSecretEventHandler for each route at router startup. The previous
implementation held the global write mutex (s.lock) for the entire duration
of the call, including the WaitForCacheSync round-trip to the API server and
etcd (~100–600ms per secret). This caused all N secret registrations to run
strictly serially:

Scale	etcd latency	Startup time
112 secrets	~587ms/secret	65s
1,738 secrets	~1,414ms/secret	40.9 minutes

The customer reports ~2,000 external cert routes causing router pods to fail
their 120s startup probe repeatedly and enter CrashLoopBackOff.

Root Cause

In addSecretEventHandler, WaitForCacheSync was called inside
defer s.lock.Unlock(), meaning the global write lock was held while waiting
for each secret's informer to complete its initial LIST to etcd. No other
registration could begin until the previous one finished.

Fix

Write the new monitoredItem into s.monitors before releasing the lock
(so concurrent callers for the same secret safely reuse the same informer),
then release the lock before WaitForCacheSync. This allows concurrent
AddSecretEventHandler calls for different secrets to perform their etcd
round-trips in parallel, reducing startup time from O(N × api_latency) to
O(api_latency).

On sync failure the map entry is cleaned up so a future caller can retry with
a fresh informer.

Testing

All existing unit tests in pkg/secret/... pass.

A concurrent registration test should be added (tracked separately).

Fixes: OCPBUGS-77056

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bentito]

/jira refresh

[openshift-ci-robot]

@bentito: This pull request references Jira Issue OCPBUGS-77056, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lihongan

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[Miciah]

/assign

[bentito]

/assign @everettraven

[bentito]

@everettraven Could you give this proposed fix a look from the standpoint of whether it makes logical sense or whether we might just be moving the problem to the API server? Thanks!

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bentito
Once this PR has been reviewed and has the lgtm label, please ask for approval from miciah. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/route/OWNERS
• pkg/secret/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bentito]

/retest

Note: The ci/prow/unit failure appears to be in pkg/operator/staticpod/installerpod (TestCopyContent) which is a pre-existing test failure unrelated to this PR's changes. This PR only modifies pkg/secret/secret_monitor.go, pkg/secret/fake/fake_secret_monitor.go, and pkg/route/secretmanager/manager*.go. All tests in those packages pass locally with -race -count=5.

[openshift-ci]

@bentito: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.