feat(monitor): operator Prometheus metrics with mTLS by rene-dekker · Pull Request #4558 · tigera/operator

rene-dekker · 2026-03-16T23:15:31Z

Summary

Add configurable Prometheus metrics endpoint to the operator via METRICS_HOST, METRICS_PORT, and METRICS_SCHEME env vars
mTLS support when METRICS_SCHEME=https: server cert from tigera-operator-tls, client auth trusts tigera-ca-private CA
Monitor controller creates Service, ServiceMonitor, and server TLS cert for automatic Prometheus discovery
Custom Prometheus collector exposes operator_installation_status and operator_tigera_status gauges

Test plan

make build passes
make ut UT_DIR=./pkg/controller/metrics — 10 tests pass
make ut UT_DIR=./pkg/render/monitor — 20 tests pass
make ut UT_DIR=./pkg/controller/monitor — 17 tests pass
Manual: deploy with METRICS_HOST/METRICS_PORT set, verify metrics scraped
Manual: deploy with METRICS_SCHEME=https, verify mTLS handshake with Prometheus

🤖 Generated with Claude Code

Example alerts:

Example metrics:

$ curl --cert client.crt --key client.key --cacert ca.crt https://tigera-operator-metrics.tigera-operator:9484/metrics | grep tigera_operator_tls_certificate
tigera_operator_tls_certificate_expiry_timestamp_seconds Unix timestamp of certificate expiry for operator-managed TLS secrets.
# TYPE tigera_operator_tls_certificate_expiry_timestamp_seconds gauge
tigera_operator_tls_certificate_expiry_timestamp_seconds{issuer="byo-signer",name="calico-apiserver-certs",namespace="calico-system"} 1.774828114e+09
tigera_operator_tls_certificate_expiry_timestamp_seconds{issuer="tigera-operator-signer",name="calico-apiserver-certs",namespace="tigera-operator"} 1.844609326e+09

$ curl --cert client.crt --key client.key --cacert ca.crt https://tigera-operator-metrics.tigera-operator:9484/metrics | grep tigera_operator_component_status
tigera_operator_component_status{component="apiserver",condition="available"} 1
tigera_operator_component_status{component="apiserver",condition="degraded"} 0
tigera_operator_component_status{component="apiserver",condition="progressing"} 0
tigera_operator_component_status{component="calico",condition="available"} 1
tigera_operator_component_status{component="calico",condition="degraded"} 0
tigera_operator_component_status{component="calico",condition="progressing"} 0


$ curl --cert client.crt --key client.key --cacert ca.crt https://tigera-operator-metrics.tigera-operator:9484/metrics | grep tigera_operator_license
tigera_operator_license_expiry_timestamp_seconds{package="Enterprise"} 2.051337599e+09
# HELP tigera_operator_license_valid Whether the Tigera license is valid (including grace period). 1 = valid, 0 = invalid.
tigera_operator_license_valid{package="Enterprise"} 1

rene-dekker · 2026-03-17T23:35:13Z

pkg/render/monitor/monitor.go

+		rules = append(rules,
+			monitoringv1.Rule{
+				Alert:  "TLSCertExpiringWarning",
+				Expr:   intstr.FromString("tigera_operator_tls_certificate_expiry_timestamp_seconds - time() < 29 * 24 * 3600"),


I picked 29 days, because we automatically renew our own certs 30d before expiry. I wanted to exclude that day to generate fewer alerts.

Add operator metrics endpoint with configurable mTLS via METRICS_SCHEME, METRICS_HOST, and METRICS_PORT env vars. The monitor controller creates a server cert, Service, and ServiceMonitor for Prometheus integration. Client auth trusts the tigera-ca-private CA rather than individual leaf certs. Includes a custom Prometheus collector for operator status gauges. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace the implicit metrics-enabled detection (METRICS_HOST/PORT set) with an explicit METRICS_ENABLED=true env var. Default METRICS_HOST to 0.0.0.0 and METRICS_PORT to 8484 when enabled. Log a helpful message when mTLS is enabled but the server certificate is not yet available. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Use port 9484 instead of 8484 to reduce the chance of conflicts on host-networked nodes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Set SecureServing: true so controller-runtime actually serves TLS instead of plain HTTP when METRICS_SCHEME=https. Add egress rule in calicoSystemPrometheusPolicy to allow Prometheus to reach the operator metrics Service. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

caseydavenport · 2026-03-27T18:23:38Z

cmd/main.go

+						MinVersion:     tls.VersionTLS12,
+					}, nil
+				}
+				cfg.MinVersion = tls.VersionTLS12


Do we need a configurable min TLS version (and configurable client cert checking?)

I think we have that in the other metrics endpoints (at least for OSS Calico)

caseydavenport · 2026-03-27T18:24:56Z

cmd/main.go


+// metricsEnabled returns true when the operator metrics endpoint is enabled.
+func metricsEnabled() bool {
+	return strings.EqualFold(os.Getenv("METRICS_ENABLED"), "true")


Technically a breaking change, since someone might be using metrics today without this set?

But probably not a big deal - easy fix.

caseydavenport · 2026-03-27T18:25:25Z

cmd/main.go

 }
+
+// metricsTLSEnabled returns true when the operator metrics endpoint should use mTLS.
+func metricsTLSEnabled() bool {


might be worth a metrics.go in the main package to put these helpers in to keep main.go a bit less of a "God file"

caseydavenport · 2026-03-27T18:26:52Z

cmd/main.go

+// dynamicCertLoader dynamically loads TLS certificates from Kubernetes secrets
+// for the metrics endpoint. The monitor controller creates the server cert, and
+// the client CA is loaded from the Prometheus client TLS secret.
+type dynamicCertLoader struct {


I wonder, do we need all of this complexity?

Couldn't we just use an optional secret file mount, and have kubelet auto-load changes to the mounted paths?

caseydavenport · 2026-03-27T18:29:05Z

pkg/controller/metrics/collectors.go

+
+func conditionLabel(ct operatorv1.StatusConditionType) string {
+	switch ct {
+	case operatorv1.ComponentAvailable:


Isn't this just effectively strings.ToLower(ct)?

caseydavenport · 2026-03-27T18:29:22Z

pkg/controller/metrics/collectors.go

+
+		expiry, err := time.Parse(expiryFormat, expiryStr)
+		if err != nil {
+			continue


Maybe a debug log?

caseydavenport · 2026-03-27T18:29:32Z

pkg/controller/metrics/collectors.go

+			continue
+		}
+
+		issuer := s.Annotations["certificates.operator.tigera.io/issuer"]


Do we have a const we can use? Should we have one?

caseydavenport · 2026-03-27T18:30:00Z

pkg/controller/metrics/collectors.go

+}
+
+func (c *OperatorCollector) collectLicense(ctx context.Context, ch chan<- prometheus.Metric) {
+	license, err := utils.FetchLicenseKey(ctx, c.client)


Is this going to generate an API call for OSS clusters?

caseydavenport · 2026-03-27T18:30:41Z

pkg/controller/monitor/monitor_controller.go

 }
+
+// metricsEnabled returns true when the operator metrics endpoint is enabled.
+func metricsEnabled() bool {


Looks like a duplicate of the one in the main package - would be nice to have a common impl.

caseydavenport · 2026-03-27T18:32:41Z

pkg/render/monitor/monitor.go

 	bearerTokenFile       = "/var/run/secrets/kubernetes.io/serviceaccount/token"
 	KubeControllerMetrics = "calico-kube-controllers-metrics"
+
+	OperatorMetricsSecretName  = "tigera-operator-tls"


It's a bit confusing that this is "tigera-operator-tls" but the var is "metrics" focused - I think it is correct for the secret name to NOT include the word metrics, but perhaps the variable name should match.

Would be worth some comments explaining as well. If we do end up using the cert in the future for anything else, we might need to move its creation out of this controller?

caseydavenport · 2026-03-27T18:33:38Z

pkg/render/monitor/monitor.go

+}
+
+// serviceMonitorOperator creates a ServiceMonitor for the operator's metrics endpoint.
+func (mc *monitorComponent) serviceMonitorOperator() *monitoringv1.ServiceMonitor {


we have "opertorMetricsService" but "serviceMonitorOperator" - naming is a bit inconsistently formatted.

caseydavenport · 2026-03-27T18:33:50Z

pkg/render/monitor/monitor.go

+		ObjectMeta: metav1.ObjectMeta{
+			Name:      OperatorMetricsServiceName,
+			Namespace: common.TigeraPrometheusNamespace,
+			Labels:    map[string]string{"team": "network-operators"},


What is this label?

I think this stems all the way back from when everything was manifest based installed. It may even have come from the prometheus example bundle. I think we should delete them all, everywhere probably.

marvin-tigera added this to the v1.42.0 milestone Mar 16, 2026

marvin-tigera added docs-pr-required release-note-required labels Mar 16, 2026

rene-dekker commented Mar 17, 2026

View reviewed changes

rene-dekker force-pushed the EV-6493 branch from 0ab85f9 to 69c317f Compare March 18, 2026 00:01

danudey modified the milestones: v1.42.0, v1.43.0 Mar 20, 2026

rene-dekker and others added 4 commits March 27, 2026 09:04

Change default metrics port to 9484

e6cf024

Use port 9484 instead of 8484 to reduce the chance of conflicts on host-networked nodes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

rene-dekker force-pushed the EV-6493 branch from 69c317f to 606be04 Compare March 27, 2026 18:19

rene-dekker marked this pull request as ready for review March 27, 2026 18:19

rene-dekker requested a review from a team as a code owner March 27, 2026 18:19

caseydavenport reviewed Mar 27, 2026

View reviewed changes

Conversation

rene-dekker commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rene-dekker commented Mar 16, 2026 •

edited

Loading