Add to 10.1.2: Fix cache retry assert on ServerAddrSe

[DawidKowalczyk]

Hello,
could you add this changes to v. 10.1.2 ?
#12972

Unortunetly, we see a lot of crush without this changes.

[bryancall]

@DawidKowalczyk We plan to pull this in for our next 10.1.3 release. Thank you