Skip to content

fix(prometheus-rules): escape PromQL $labels for Helm rendering#527

Closed
bussyjd wants to merge 1 commit into
feat/x402-marketplace-metricsfrom
fix/prometheus-rules-helm-template-escape
Closed

fix(prometheus-rules): escape PromQL $labels for Helm rendering#527
bussyjd wants to merge 1 commit into
feat/x402-marketplace-metricsfrom
fix/prometheus-rules-helm-template-escape

Conversation

@bussyjd
Copy link
Copy Markdown
Collaborator

@bussyjd bussyjd commented May 24, 2026

Why

PrometheusRule annotations use {{ $labels.X }} which Prometheus evaluates at alert-firing time. When this file is rendered through Helm (via chart: ./base in helmfile.yaml), Helm's Go-template engine tries to evaluate $labels at chart-render time and fails:

Error: UPGRADE FAILED: parse error at
(base-infra/templates/x402-prometheus-rules.yaml:N):
undefined variable "$labels"

This blocks obol stack up whenever the file is re-applied.

Before

summary: "x402 payment failures > 10% on {{ $labels.offer_namespace }}/..."
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  Helm parses this as a Go template action,
                  fails because $labels is not a Helm variable

After

summary: "x402 payment failures > 10% on {{ "{{" }} $labels.offer_namespace {{ "}}" }}/..."
                                          ^^^^^^^^                          ^^^^^^^^
                  Helm emits literal `{{ $labels.offer_namespace }}`
                  Prometheus then evaluates it at alert-fire time

Surfaced by

The 14-PR integration test campaign — full stack-up of integration branch combining all 15 PRs. go test ./... doesn't render Helm so it didn't catch this; nor did the per-PR validation since none ran helm template.

Recommendation for follow-up

Add a CI smoke that pipes every embedded *.yaml template through helm template ./base to catch this class going forward.

Stacks on

PR #513 (introduced the file in commit 27e1ac5). Once this lands, PRs #515/#516/#519/#524 (also stacked on #513) automatically pick up the fix when they rebase.

Test plan

  • go build ./... clean
  • helm template works on the fixed file (validated during the integration test stack-up — see plans/integration-test-L7-paid-flow-20260524.md)
  • CI: existing checks should pass; add helm-template smoke as a separate PR

Bug #1 of 4 surfaced by integration testing

See plans/integration-test-results-final-20260524.md for the full 4-bug list.

PrometheusRule annotations use {{ $labels.X }} which Prometheus
evaluates at alert-firing time. When this file is rendered through
Helm (via chart: ./base in helmfile.yaml), Helm's Go-template engine
tries to evaluate $labels at chart-render time and fails with:

  Error: UPGRADE FAILED: parse error at
  (base-infra/templates/x402-prometheus-rules.yaml:N):
  undefined variable "$labels"

Wrap each templated brace pair as {{ "{{" }}...{{ "}}" }} so Helm
emits literal Prometheus template syntax verbatim into the YAML
output, where Prometheus picks it up at alert-eval time.

Bug surfaced by integration-branch full stack-up; not caught by
`go test ./...` (unit tests don't render Helm) nor by the agent
worktree validation (which only checked Go-side compilation).

Recommend adding a CI smoke that pipes embedded *.yaml templates
through `helm template ./base` to catch this class going forward.

Stacks on PR #513 (which introduced the file in commit 27e1ac5).
@bussyjd
Copy link
Copy Markdown
Collaborator Author

bussyjd commented May 24, 2026

Superseded by bundle PR #536 — closing in favor of the consolidated merge target. Original branch and history preserved.

@bussyjd bussyjd closed this May 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant