Sudden SSH Issue

[chrigi]

Reconciliation with my GitHub Repo I bootstrapped using flux bootstrap github --owner="xxx" --repository="yyys" --path="cluster" --personal days ago worked fine until it suddenly, without any changes on the host or cluster, failed with: "failed to checkout and determine revision: unable to clone 'ssh://git@github.com/xxx/yyys': ssh: handshake failed: read tcp 11.11.1.11:45906->140.82.121.4:22: read: connection timed out

I tired re-bootstrapping and it actually updated my fluxcd deployment to the latest version and pushed the changes to my github repo but still it got stuck at the reconciliation with the same error:

flux bootstrap github --owner="xxx" --repository="yyys" --path="cluster" --personal
► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/xxx/yyy.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
✔ source secret up to date
► generating sync manifests
✔ generated sync manifests
✔ sync manifests are up to date
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
^C

After a flux uninstall and then a new bootstrap, this time including --token-auth it worked again.

[matheuscscp]

Which version of flux are you running

[chrigi]

Sorry, flux check:

► checking prerequisites
✔ Kubernetes 1.34.4+k3s1 >=1.33.0-0
► checking version in cluster
✔ distribution: flux-v2.8.0
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v1.5.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.8.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.8.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.8.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1
✔ externalartifacts.source.toolkit.fluxcd.io/v1
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1
✔ helmreleases.helm.toolkit.fluxcd.io/v2
✔ helmrepositories.source.toolkit.fluxcd.io/v1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

[stefanprodan]

Given the error read tcp 11.11.1.11:45906->140.82.121.4:22: read: connection timed out this looks like a DNS issue.

[chrigi]

I checked with nslookup github.com and nc -vz github.com 22 from the host and from the source-controlle pod and both were successfull. Flux managed to clone the repo initially on re-bootstrap, write the updated resources to my repo but as soon as I tried to reconcile it failed.

[matheuscscp]

I wonder if you only upgraded to 2.8 after you tried to fix the issue by rerunning bootstrap. Did you first see the issue in Flux 2.7? If you're using flux bootstrap to manage the installation I think you first saw the issue in 2.7 because this method does not upgrade Flux automatically. Can you please confirm if this was the case?

[matheuscscp]

Ah you already said it upgraded

[matheuscscp]

Yeah if it stopped working without upgrading then it's likely to be an environment issue...