Prune stale per-container version metrics to stop unbounded Prometheus series growth

[Copilot]

version-checker was retaining superseded version_checker_is_latest_version series for active containers. In clusters that run long enough and observe image/version churn, this causes Prometheus label cardinality to grow over time and can present as steady memory growth/OOMs.

• Root cause

  • AddImage writes version_checker_is_latest_version with current_version and latest_version in the label set.
  • For the same pod/container, each new observed version created a new gauge series, while the previous series remained registered until pod/container removal.

• Change

  • Before recording the latest image state for a container, remove the existing per-container gauge series for:
    • version_checker_is_latest_version
    • version_checker_last_checked
  • Keep duration and error metrics unchanged; this only de-duplicates the “current state” metrics.

• Result

  • Each pod/container now has a single current version/check timestamp series instead of an ever-growing set of historical version-labeled gauges.
  • This bounds metric memory usage for long-lived workloads while preserving the intended exported state.

• Regression coverage

  • Add a focused metrics test that updates the same container twice and asserts the stale version-labeled series is no longer present in the registry.

labels := buildContainerPartialLabels(namespace, pod, container, containerType)

m.containerImageVersion.DeletePartialMatch(labels)
m.containerImageChecked.DeletePartialMatch(labels)

m.containerImageVersion.With(
	buildFullLabels(namespace, pod, container, containerType, imageURL, currentVersion, latestVersion),
).Set(isLatestF)

[Copilot]

Pull request overview

This PR addresses unbounded Prometheus series growth by pruning stale per-container “current state” gauge series when the observed image/version for a container changes, keeping metric cardinality bounded for long-lived clusters.

Changes:

• In AddImage, delete existing per-container gauge series for is_latest_version and last_checked before recording the newest values.
• Add a regression test to ensure stale version-labeled is_latest_version series are removed when the same container is updated multiple times.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
pkg/metrics/metrics.go	Prunes existing per-container gauge series prior to writing updated “current state” metrics to prevent series buildup.
pkg/metrics/metrics_test.go	Adds a regression test asserting stale version-labeled gauge series are removed after updates.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.