feat(health): Add NVOS Streaming telemetry by mkoci · Pull Request #1975 · NVIDIA/infra-controller

mkoci · 2026-05-28T02:10:58Z

Description

gNMI collector ([collectors.nvue.gnmi], disabled by default) subscribes to NVUE gNMI ON_CHANGE /system-events and SAMPLE paths for:

/components/component
/interfaces/interface
/system-events/*

It uses long-lived gRPC streams with reconnection (exp. back-off + jitter).

Builds on #711 (SSE streaming + OtlpSink for logs). Protos vendored for reproducible offline builds, same as #711.

Currently supports PrometheusSink and OtlpSink

Type of Change

Add - New feature or capability
Change - Changes in existing functionality
Fix - Bug fixes
Remove - Removed features or deprecated functionality
Internal - Internal changes (refactoring, tests, docs, etc.)

Related Issues

Refactor builds on feat(health): SSE streaming log support. OtlpSink. FileSync #711
Per-stream metric accumulation tracked in feat: (health) Consolidate metric registries to avoid per-stream accumulation and leaks #989 (separate)

Testing

Unit tests added/updated
Integration tests added/updated
Manual testing performed
No testing required (docs, internal refactor, etc.)

Additional Notes

IN TEST - currently validating correct Switch host credentials.
This PR is stacked onto fix(health): address improper deserialization of num-gpus for Switch host sdn partition response #1962
Currently being tested with new changes for Switch Host credentials from feat(health): Switch BMC/Host roles for telemetry #1745

Future

It would be useful to port this to support HealthAlerts especially for ON_CHANGE from /system-events - there is extra information available here which is not available via the Switch BMC.

Signed-off-by: mkoci <mkoci@nvidia.com>

…ming Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

…ls on switch hosts Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

… monitoring Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

mkoci · 2026-05-29T11:38:51Z

I would like it be merged after #1913, as it has many changes to how auth/endpoints work.

Makes sense - I'll look over that PR and make sure I understand the changes. Perhaps I can stack it on this in test.

There's more work to do for Switch Hosts outside this PR. Will have to properly map TLS certificate support for both NICo and static config.

# Conflicts: # Cargo.lock # crates/health/src/collectors/runtime.rs # crates/health/src/discovery/spawn.rs # docs/architecture/health_aggregation.md

kensimon

I haven't reviewed the whole change, so I don't know how important "label per instance" is to the concept, but we can't do it that way: Labels need to be bounded and well-known, we don't want to create prometheus labels for individual objects like instances.

See comments

## Description Splits Health service prometheus endpoint between /metrics and /telemetry. Also removes backward compatability `/` endpoint for metrics, as it was confusing and abused via health probes. This issue was raised after discussion on PR #1975 ## Type of Change  - [ ] **Add** - New feature or capability - [x] **Change** - Changes in existing functionality - [ ] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) ## Related Issues (Optional)  ## Breaking Changes - [x] This PR contains breaking changes Telemetry no longer exposed on `/metrics` Default `/ `endpoint no longer exposes metrics ## Testing  - [x] Unit tests added/updated - [ ] Integration tests added/updated - [x] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) ## Additional Notes

coderabbitai · 2026-06-03T03:20:58Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b000451b-98f9-48a7-8386-2607ac34a873

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-03T03:22:00Z

🌿 Preview your docs: https://nvidia-preview-pull-request-1975.docs.buildwithfern.com/infra-controller

…as it is redundant and doesn't exist in most versions

mkoci · 2026-06-07T18:53:25Z

@Coco-Ben , @yoks, @kensimon - This should be in better shape now. I've merged in both #1913 and #2056

As this is an optional feature in config, I can address any bugs found in testing in subsequent PRs. This is becoming unwieldy 😅

mkoci and others added 30 commits May 14, 2026 14:20

feat(health): add machine placement metadata to events

0e1d965

Signed-off-by: mkoci <mkoci@nvidia.com>

feat(health): forward machine metadata from API

0053968

Signed-off-by: mkoci <mkoci@nvidia.com>

feat(health): expose machine metadata as Prometheus labels

037dd50

Signed-off-by: mkoci <mkoci@nvidia.com>

feat(health): emit machine metadata as OTLP attributes

40c3d49

Signed-off-by: mkoci <mkoci@nvidia.com>

docs(health): document hardware health metadata surfaces

f22528b

Signed-off-by: mkoci <mkoci@nvidia.com>

feat(health): expose switch placement metadata

83149ec

Signed-off-by: mkoci <mkoci@nvidia.com>

Merge branch 'main' into feature/hw-health-machine-metadata

c52200e

refactor(health): rename SseConnectionGuard to StreamingConnectionGuard

6331ef8

Signed-off-by: mkoci <mkoci@nvidia.com>

feat(health): vendor openconfig/gnmi protos for reproducible builds

89acbe5

Signed-off-by: mkoci <mkoci@nvidia.com>

feat(health): add NVUE gNMI streaming collector

57f9b0b

Signed-off-by: mkoci <mkoci@nvidia.com>

feat(health): add OTLP metrics export via MetricsService

8c0f234

Signed-off-by: mkoci <mkoci@nvidia.com>

fix(health): adapt nvos streaming to metadata stack

c8e3c83

Signed-off-by: mkoci <mkoci@nvidia.com>

Merge remote-tracking branch 'mk-origin/main' into feature/nvos_strea…

7f1445a

…ming Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

fix(health): add support for positional metadata in telemetry

3aafde5

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

Merge branch 'NVIDIA:main' into feature/nvos_streaming

cc59345

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

3885ae9

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

feat(health): restore nvue gnmi on-change events

37371fb

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

fix(health): typo

543be1a

Merge remote-tracking branch 'NVIDIA/main' into feature/nvos_streaming

11efebf

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

Merge branch 'feature/nvos_streaming' into feature-nvos-health

57221f2

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

lint(health): tidy up

0f3fd25

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

bug(health): gate switch hosts and bmcs in spawn to avoid redfish cal…

438019d

…ls on switch hosts Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

feat(health): add SwitchEndpointRole to distinguish switch BMC from Host

db6dca4

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

feat(health): add static config shape for switch bmc/host

3fbc376

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

feat(health): gate switch collection by endpoint role (host/bmc)

817c2f3

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

feat(api): expose switch host endpoints and nvos credentials for host…

83f05fe

… monitoring Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

feat(health): wire switch host endpoint to health discovery

039a020

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

feat(health): discover switch bmc and host endpoints, respectively

58e8c24

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

fix(health): remove unnecessary helpers

4f8e492

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>

Merge branch 'NVIDIA:main' into bug_1744_no_redfish_for_switch_hosts

7277eef

mkoci added 9 commits May 30, 2026 03:24

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

4a73a5d

# Conflicts: # Cargo.lock # crates/health/src/collectors/runtime.rs # crates/health/src/discovery/spawn.rs # docs/architecture/health_aggregation.md

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

f2482c0

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

2f58454

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

69d6764

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

d28395a

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

9302ba9

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

7280575

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

4b63232

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

ce886e8

kensimon previously requested changes Jun 1, 2026

View reviewed changes

Comment thread crates/health/src/collectors/nvue/gnmi/on_change_processor.rs

Coco-Ben approved these changes Jun 1, 2026

View reviewed changes

yoks mentioned this pull request Jun 1, 2026

chore: separate metrics and telemetry #2056

Merged

10 tasks

Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health

9db8814

mkoci added 9 commits June 3, 2026 06:53

Merge branch 'main' into feature-nvos-health

9306fb7

fix(health): wire gNMI auth through credential provider

11ca126

fix(health): export switch OTLP resource identity

ce43969

fix(health): key OTLP metric queue by sample identity

985e54f

fix(health): remove NVOS gNMI path for platform-general/leak-sensors …

ccd610d

…as it is redundant and doesn't exist in most versions

Merge branch 'main' into feature-nvos-health

6b7e9db

Merge branch 'main' into feature-nvos-health

81f69a5

Merge branch 'main' into feature-nvos-health

935fd0a

Merge branch 'main' into feature-nvos-health

7983fc8

mkoci requested review from Coco-Ben and kensimon June 7, 2026 18:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(health): Add NVOS Streaming telemetry#1975

feat(health): Add NVOS Streaming telemetry#1975
mkoci wants to merge 65 commits into
NVIDIA:mainfrom
mkoci:feature-nvos-health

mkoci commented May 28, 2026 •

edited

Loading

Uh oh!

mkoci commented May 29, 2026

Uh oh!

kensimon left a comment

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading

Review skipped

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

mkoci commented Jun 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mkoci commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Related Issues

Testing

Additional Notes

Future

Uh oh!

mkoci commented May 29, 2026

Uh oh!

kensimon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

mkoci commented Jun 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mkoci commented May 28, 2026 •

edited

Loading

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading