Add tracer spans inside list path so slow ps calls are visible#208
Draft
sjmiller609 wants to merge 1 commit intomainfrom
Draft
Add tracer spans inside list path so slow ps calls are visible#208sjmiller609 wants to merge 1 commit intomainfrom
sjmiller609 wants to merge 1 commit intomainfrom
Conversation
The /instances Server span has no children today, so a slow ps is opaque in traces. Instrument the listing pipeline: - instances.list (ListInstances) - instances.list.persist_boot_markers (post-loop pass) - instances.list_metadata (listInstances) + per-id hydrate_one - instances.derive_state - instances.hydrate_boot_markers - instances.parse_boot_markers (records log_paths count) - instances.persist_boot_markers (under per-instance lock) Plumb context.Context through hydrateBootMarkersFromLogs and parseBootMarkers so child spans hang off the right parent. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
/instancesServer span has no children today, so a slowhypeman ps(and the apiListInstancescalls from reaper workflows) is completely opaque in traces. The handler walks every metadata file, derives state, scans rotated app logs for boot markers, and then runs a second pass that takes per-instance write locks to persist any markers it just hydrated — but none of that work emits spans. When something stalls inside the loop, traces show a single multi-second server span with nothing under it.This adds tracer.Start spans throughout the listing pipeline:
instances.list— wrapsListInstancesinstances.list.persist_boot_markers— wraps the post-loop persist pass (records how many instances actually persisted)instances.list_metadata— wrapslistInstances(recordsmetadata_filesandinstancescounts)instances.list_metadata.hydrate_one— per-instance, withinstance_idinstances.derive_state— wrapsderiveStateWithOptionsinstances.hydrate_boot_markers— wrapshydrateBootMarkersFromLogsinstances.parse_boot_markers— wrapsparseBootMarkers(recordslog_pathscount, since the rotated-log scan size matters here)instances.persist_boot_markers— wrapsmaybePersistBootMarkers(so we can see lock-wait time)To make those spans nest correctly,
hydrateBootMarkersFromLogsandparseBootMarkersnow take acontext.Context; the only non-test caller already had ctx in scope, and the two test callers passt.Context().Test plan
go test ./lib/instances/ -run='TestParseBootMarkers|TestHydrateBootMarkers'go test ./cmd/api/api/ -run='TestList'GET /instancestrace in SigNoz shows the newinstances.*child spans and that the slow region is now attributable.🤖 Generated with Claude Code