Revamp runtime OpenTelemetry tracing: attempt-based spans with early publishing

[slinkydeveloper]

Summary

Our current tracing experience has several issues that make it unsuitable for debugging live systems. As more customers request end-to-end tracing, we need to revisit how the runtime publishes OpenTelemetry spans.

Current Problems

1. Spans per journal entry are confusing

• For ctx.run, the span duration includes internal retry timer backoffs and is only published when completed
• If an invocation is stuck in a retry loop, the span looks fine — you can't tell it's retrying
• Runtime-generated spans for ctx.run are not correlatable with user-created spans in the SDK

2. Parent span published too late

• Until the entire invocation completes, OTel traces are not correctly shown/correlated because the parent span is published at the end
• This makes tracing useless for debugging things that are actively on fire
• The problem compounds with e2e tracing — more info is uncorrelated

Root Cause

The current tracing was designed pre-UI and optimized for post-hoc viewing (a "logical view" of the invocation). But the OpenTelemetry spec doesn't support giving a logical view while the invocation is still running.

Proposal

Shift philosophy to representing what's physically happening, while it's happening.

1. Focus on invocation attempts, not the whole invocation
2. Publish spans as soon as an invocation attempt ends — this helps correlation and allows introspection of in-progress invocations
3. Correlate all invocation attempts under a parent span (e.g. a "started" span)
4. Use events instead of spans for journal entries inside the invocation attempt span (e.g. ctx.run becomes an event, not a child span)
5. Still generate service-to-service spans as children (or linked — TBD) to enable navigating between service calls

Expected Outcome

• Users get a physical view of what happened during the invocation via OTel traces
• The logical view remains the Restate UI
• Users can search spans by invocation ID (navigate between Restate UI ↔ Jaeger and vice versa)

Example

An invocation retried 3 times with a 3-second retry interval, doing a ctx.run then a call, would show:

• A parent span covering the full invocation
• 3 child attempt spans (one per attempt), each published as soon as the attempt ends
• Events within each attempt span for journal entries
• A service-to-service child/linked span for the outgoing call

[slinkydeveloper]

Issue opened while investigating what is possible with otel open-telemetry/opentelemetry-specification#4993

[slinkydeveloper]

Standardizing spans across SDKs

While reviewing restatedev/sdk-python#187 (comment), I noticed we should standardize span attributes across SDKs. Standardizing them makes it easy to sort/correlate traces across runtime and SDKs.

Each SDK tracing integration should create a "per attempt" span, with the following attributes:

Attribute	Description
restate.invocation.id	The invocation ID
restate.invocation.target	The invocation target (e.g. greeter/greet for unkeyed, greeter/myKey/greet for keyed)

If the SDK generates spans per ctx.run, it should contain the following attributes:

Attribute	Description
restate.run.name	The run name

Runtime uses also the following attributes, which for now dont seem relevant to the SDK:

Attribute	Description
restate.invocation.error.stacktrace	Error stacktrace of the invocation
restate.error.code	Restate error code
restate.deployment.id	Deployment ID
restate.deployment.address	Deployment address
restate.deployment.service_protocol_version	Service protocol version of the deployment

[slinkydeveloper]

More context: https://restatedev.slack.com/archives/C04KZRLE1SM/p1774374968526679

[tillrohrmann]

Is the plan to replace the existing tracing implementation?

[slinkydeveloper]

@tillrohrmann that was my idea. would there be any reason to even keep the old approach? existing services doing downstream tracing should not be affected by this.

[tillrohrmann]

I am mainly wondering whether existing users see value in the old way of tracing and whether we might inadvertently break a feature for existing users by removing/replacing it. Unfortunately, I don't have any data on that.

[slinkydeveloper]

Discussed this offline with @igalshilman. Our take is that the concern is likely moot: span shape is a bit like log lines — in theory you could treat them as an API, but in practice no one does, so reshaping the internal spans we emit shouldn't be a breaking change for anyone.

What does matter, and what we'd preserve, is the behavior at the boundaries:

• Upstream: we continue to respect upstream traceparent propagation, same as today.
• Downstream: we continue to propagate a traceparent to the SDK, so spans the user emits inside their handler still attach to the right trace.

So existing distributed-tracing setups (a caller tracing into Restate, or a handler emitting its own spans) keep working as-is — only the internal structure/attributes of the spans Restate itself produces change.