From d11fbc1161b11749e0502be08b77e4b50f3fe928 Mon Sep 17 00:00:00 2001 From: "skill-sync[bot]" Date: Tue, 2 Jun 2026 20:59:42 +0000 Subject: [PATCH 01/16] Finalize draft for 0022-opentelemetry-plugins --- references/integrations.md | 2 + references/python/advanced-features.md | 2 +- references/python/integrations/google-adk.md | 1 - references/python/integrations/langgraph.md | 2 +- references/python/integrations/langsmith.md | 1 - .../python/integrations/openai-agents-sdk.md | 1 - .../python/integrations/opentelemetry.md | 171 +++++++++++++++++ references/python/workflow-streams.md | 1 - references/ruby/gotchas.md | 1 - .../typescript/integrations/opentelemetry.md | 176 ++++++++++++++++++ 10 files changed, 351 insertions(+), 7 deletions(-) create mode 100644 references/python/integrations/opentelemetry.md create mode 100644 references/typescript/integrations/opentelemetry.md diff --git a/references/integrations.md b/references/integrations.md index 71b3169..6202a9c 100644 --- a/references/integrations.md +++ b/references/integrations.md @@ -19,3 +19,5 @@ Temporal ships and supports a growing set of integrations with third-party frame | LangSmith tracing (`temporalio.contrib.langsmith`) | Python | Experimental Temporal Plugin that propagates LangSmith trace context across Worker boundaries; lets `@traceable` run inside Workflows and Activities | `references/python/integrations/langsmith.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | LangGraph (`temporalio.contrib.langgraph`, Pre-release) | Python | Runs LangGraph Graph-API and Functional-API code as Temporal Workflows - nodes/tasks can execute as either in-workflow or as Activities | `references/python/integrations/langgraph.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | Google ADK (`temporalio[google-adk]`) | Python | Durable Google ADK agents: model calls run through `TemporalModel`-wrapped Activities, tools via `activity_tool`, MCP toolsets via `TemporalMcpToolSet` | `references/python/integrations/google-adk.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | +| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing via the new `OpenTelemetryPlugin` (recommended, experimental) or legacy `TracingInterceptor`; propagates W3C TraceContext + Baggage across Client, Workflow, Activity (including Standalone), Child Workflow, and Nexus boundaries | `references/python/integrations/opentelemetry.md` | `references/python/observability.md`, `references/python/standalone-activities.md` | +| OpenTelemetry (`@temporalio/interceptors-opentelemetry`) | TypeScript | Distributed tracing via `OpenTelemetryPlugin` plus per-tier Client/Activity/Nexus/Workflow interceptors; Workflow spans exported through an injected Sink; pass the plugin to both `bundleWorkflowCode` and `Worker.create` | `references/typescript/integrations/opentelemetry.md` | `references/typescript/observability.md`, `references/typescript/standalone-activities.md` | diff --git a/references/python/advanced-features.md b/references/python/advanced-features.md index 6ad8ae8..4cb4ad6 100644 --- a/references/python/advanced-features.md +++ b/references/python/advanced-features.md @@ -134,7 +134,7 @@ client = await Client.connect( - The only field is `resolution_interval_millis: int = 30000` — how often to re-resolve DNS, in milliseconds. - `DnsLoadBalancingConfig.default` is a pre-built instance with the default 30-second interval. -- `dns_load_balancing_config` defaults to 30 seconds if you don't pass anything explicitly. +- `dns_load_balancing_config` defaults to 30 seconds if you don't pass anything explicitly. - Pass `dns_load_balancing_config=None` to disable DNS load balancing entirely. ### Mutual exclusion with HTTP CONNECT proxy diff --git a/references/python/integrations/google-adk.md b/references/python/integrations/google-adk.md index 4d59f4d..3e0e015 100644 --- a/references/python/integrations/google-adk.md +++ b/references/python/integrations/google-adk.md @@ -9,7 +9,6 @@ The integration is built on the Python SDK [Plugin system](https://docs.temporal > [!NOTE] > This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. - For general Temporal AI/LLM patterns (retries, rate limits, multi-agent orchestration) see `references/core/ai-patterns.md` and `references/python/ai-patterns.md`. ## Prerequisites diff --git a/references/python/integrations/langgraph.md b/references/python/integrations/langgraph.md index 2675672..4237ca7 100644 --- a/references/python/integrations/langgraph.md +++ b/references/python/integrations/langgraph.md @@ -214,4 +214,4 @@ For LangSmith tracing of LangGraph nodes and Temporal Activities together, use t - `references/python/ai-patterns.md` — Python AI/LLM patterns (Pydantic data converter, LLM Activity design, retry/error classification). - `references/core/ai-patterns.md` — language-agnostic AI/LLM patterns. -- `references/python/integrations/langsmith.md` - Companion LangSmith plugin. +- `references/python/integrations/langsmith.md` - Companion LangSmith plugin. diff --git a/references/python/integrations/langsmith.md b/references/python/integrations/langsmith.md index 98db967..a0ab26a 100644 --- a/references/python/integrations/langsmith.md +++ b/references/python/integrations/langsmith.md @@ -100,7 +100,6 @@ The plugin makes `@traceable` replay-safe in the Workflow sandbox. You do not ne - The plugin injects metadata using `workflow.now()` for timestamps and `workflow.random()` for UUIDs instead of `datetime.now()` and `uuid4()`. - LangSmith HTTP calls run on a background thread pool that does not interfere with deterministic Workflow execution. - ## Context propagation Trace context flows automatically across Client → Workflow → Activity → Child Workflow → Nexus via Temporal headers. Do not pass context manually. diff --git a/references/python/integrations/openai-agents-sdk.md b/references/python/integrations/openai-agents-sdk.md index 3807eb7..8ddf36a 100644 --- a/references/python/integrations/openai-agents-sdk.md +++ b/references/python/integrations/openai-agents-sdk.md @@ -164,7 +164,6 @@ Note that the initial run context comes from the `context=...` argument you pass In addition, since a `@function_tool` runs in the workflow, they can also call Temporal activities or other durable primitives themselves. - **Don't put I/O, system clock, or sources of randomness inside a `@function_tool` body.** Make it an `@activity.defn` and wrap with `activity_as_tool` instead. ### Picking between the two diff --git a/references/python/integrations/opentelemetry.md b/references/python/integrations/opentelemetry.md new file mode 100644 index 0000000..d1f414e --- /dev/null +++ b/references/python/integrations/opentelemetry.md @@ -0,0 +1,171 @@ +# Temporal OpenTelemetry Integration (Python) + +## Overview + +`temporalio.contrib.opentelemetry` ships two ways to wire OpenTelemetry tracing into Temporal: +the new `OpenTelemetryPlugin` (recommended, experimental) and the legacy `TracingInterceptor` (stable). + +Both propagate W3C TraceContext + W3C Baggage through Temporal headers across Client, Workflow, Activity, Child Workflow, and Nexus boundaries. + +For non-OTel observability (metrics, logging, telemetry runtime) read `references/python/observability.md`. +For trace propagation through `client.start_activity` / `client.execute_activity` see `references/python/standalone-activities.md`. + +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +`OpenTelemetryPlugin` is marked experimental in its docstring; `TracingInterceptor` is stable. + +## Install + +```bash +pip install temporalio[opentelemetry] +``` + +The extra is `opentelemetry`. `uv add temporalio[opentelemetry]` works too. + +## Public API + +All five symbols are re-exported from `temporalio.contrib.opentelemetry`. + +| Symbol | Purpose | +|---|---| +| `OpenTelemetryPlugin` | New, recommended Client plugin; installs `OpenTelemetryInterceptor` and adds `opentelemetry` sandbox passthrough. | +| `OpenTelemetryInterceptor` | Underlying interceptor used by `OpenTelemetryPlugin`; requires the global tracer provider be a `ReplaySafeTracerProvider`. | +| `TracingInterceptor` | Legacy stable interceptor — register on `Client.connect(interceptors=...)`. | +| `TracingWorkflowInboundInterceptor` | Workflow-side half of the legacy interceptor; subclass it only when customizing header key, propagator, or payload converter. | +| `create_tracer_provider` | Builds a `ReplaySafeTracerProvider`; required when using `OpenTelemetryPlugin`. | + +## Pick a registration path + +- Use `OpenTelemetryPlugin` (new, recommended) for new code: accurate workflow span durations and direct `opentelemetry.trace.get_tracer(...)` usage inside workflows. +- Use `TracingInterceptor` (legacy, stable) when you need immediate span visibility before workflow completion or a non-experimental API. +- Do not register both on the same Client. + +## `OpenTelemetryPlugin` (recommended) + +Build a `ReplaySafeTracerProvider` with `create_tracer_provider()`, attach span processors, set it as the global tracer provider, then connect with `plugins=[OpenTelemetryPlugin()]`. Workers created from the returned Client inherit the plugin automatically. + +```python +import opentelemetry.trace +from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor +from temporalio.client import Client +from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_provider +from temporalio.worker import Worker + +provider = create_tracer_provider() +provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) +opentelemetry.trace.set_tracer_provider(provider) + +client = await Client.connect( + "localhost:7233", + plugins=[OpenTelemetryPlugin()], +) + +worker = Worker( + client, + task_queue="my-task-queue", + workflows=[MyWorkflow], + activities=[my_activity], +) +``` + +Constructor: `OpenTelemetryPlugin(*, add_temporal_spans: bool = False)`. Keyword-only; `add_temporal_spans=True` adds `StartWorkflow`, `RunWorkflow`, `StartActivity`, `RunActivity` (and similar) spans on top of application spans. + +`create_tracer_provider(sampler=None, resource=None, shutdown_on_exit=True, active_span_processor=None, id_generator=None, span_limits=None) -> ReplaySafeTracerProvider`. The provider must be set with `opentelemetry.trace.set_tracer_provider(...)` before `Client.connect`; otherwise the workflow interceptor factory raises `ValueError("When using OpenTelemetryPlugin, the global trace provider must be a ReplaySafeTracerProvider. Use init_tracer_provider to create one.")`. + +The plugin's `workflow_runner` callback applies `with_passthrough_modules("opentelemetry")` to `SandboxedWorkflowRunner.restrictions`. Do not add this passthrough manually. + +Inside a Workflow, use standard OpenTelemetry APIs — durations are accurate. + +```python +from datetime import timedelta +from opentelemetry.trace import get_tracer +from temporalio import workflow + +@workflow.defn +class MyWorkflow: + @workflow.run + async def run(self) -> None: + tracer = get_tracer(__name__) + with tracer.start_as_current_span("workflow-operation"): + await workflow.execute_activity( + my_activity, + start_to_close_timeout=timedelta(seconds=30), + ) +``` + +## `TracingInterceptor` (legacy) + +Register on the Client. Workers built from that Client inherit the interceptor. + +```python +from temporalio.client import Client +from temporalio.contrib.opentelemetry import TracingInterceptor +from temporalio.worker import Worker + +client = await Client.connect( + "localhost:7233", + interceptors=[TracingInterceptor()], +) + +worker = Worker( + client, + task_queue="my-task-queue", + workflows=[MyWorkflow], + activities=[my_activity], +) +``` + +Constructor: `TracingInterceptor(tracer: opentelemetry.trace.Tracer | None = None, *, always_create_workflow_spans: bool = False)`. `tracer` defaults to `opentelemetry.trace.get_tracer(__name__)`. `always_create_workflow_spans=True` creates workflow spans even without an upstream client span (risk: orphaned spans after replay). + +Inside a Workflow, standard `opentelemetry.trace` APIs do not work correctly. Use `temporalio.contrib.opentelemetry.workflow.completed_span(...)`. Workflow spans have zero duration. + +```python +import temporalio.contrib.opentelemetry.workflow as otel_workflow +from temporalio import workflow + +@workflow.defn +class MyWorkflow: + @workflow.run + async def run(self) -> None: + otel_workflow.completed_span( + "workflow-operation", + attributes={"business.unit": "payments", "request.id": "req-123"}, + ) +``` + +## Standalone Activities + +Trace context propagates through `client.start_activity` and `client.execute_activity` via Temporal headers, the same way it propagates from a Workflow. The client outbound's `start_activity(input: StartActivityInput)` opens a `StartActivity:{activity_type}` span (kind=CLIENT) and injects context into `input.headers`; the activity-side `_TracingActivityInboundInterceptor.execute_activity` extracts that context from `input.headers` and opens a `RunActivity:{activity_type}` span (kind=SERVER). + +See `references/python/standalone-activities.md`. + +## Nexus + +`OpenTelemetryInterceptor.intercept_nexus_operation` (and the legacy `TracingInterceptor`'s equivalent) wraps inbound Nexus handlers with `RunStartNexusOperationHandler:{service}/{operation}` and `RunCancelNexusOperationHandler:{service}/{operation}` spans (kind=SERVER), extracting context from Nexus operation headers. + +See `https://github.com/temporalio/samples-python/tree/main/open_telemetry`. + +## Migration from `TracingInterceptor` to `OpenTelemetryPlugin` + +1. Replace `interceptors=[TracingInterceptor()]` on `Client.connect` with `plugins=[OpenTelemetryPlugin()]`, and add `create_tracer_provider()` + `opentelemetry.trace.set_tracer_provider(provider)` before connecting. +2. Replace `temporalio.contrib.opentelemetry.workflow.completed_span("my-span")` calls inside workflows with `tracer = get_tracer(__name__)` + `with tracer.start_as_current_span("my-span"):`. +3. Re-verify any monitoring or analysis tools that depend on the legacy trace structure — span shapes change. + +## Common mistakes + +- **Registering the same plugin or interceptor on both Client and Worker.** Register on the Client only; Workers inherit. +- **Mixing `OpenTelemetryPlugin` and `TracingInterceptor`.** They are two paths; pick one. +- **Calling `Client.connect` before `opentelemetry.trace.set_tracer_provider(provider)`.** With `OpenTelemetryPlugin`, the workflow interceptor factory raises `ValueError("When using OpenTelemetryPlugin, the global trace provider must be a ReplaySafeTracerProvider. Use init_tracer_provider to create one.")`. +- **Building a plain `opentelemetry.sdk.trace.TracerProvider` and passing it to `set_tracer_provider`.** `OpenTelemetryPlugin` requires `ReplaySafeTracerProvider` — build it via `create_tracer_provider(...)`. +- **Using `addTemporalSpans=True` or other camelCase.** The parameter is `add_temporal_spans` (Python snake_case). +- **Passing the install extra as `temporalio[otel]`.** The extra is `opentelemetry`. +- **Calling `get_tracer().start_as_current_span` inside a Workflow with only `TracingInterceptor` registered.** That path requires `OpenTelemetryPlugin`; under the legacy interceptor use `temporalio.contrib.opentelemetry.workflow.completed_span(...)` instead. +- **Adding `with_passthrough_modules("opentelemetry")` to a `SandboxedWorkflowRunner` manually when using the plugin.** The plugin already does this. + +## Resources + +- Temporal Python observability docs: `https://docs.temporal.io/develop/python/platform/observability` +- Python SDK OTel contrib README: `https://github.com/temporalio/sdk-python/blob/main/temporalio/contrib/opentelemetry/README.md` +- Python samples — OpenTelemetry: `https://github.com/temporalio/samples-python/tree/main/open_telemetry` +- SDK metrics reference: `references/python/observability.md` diff --git a/references/python/workflow-streams.md b/references/python/workflow-streams.md index 84ecbd7..b53915b 100644 --- a/references/python/workflow-streams.md +++ b/references/python/workflow-streams.md @@ -11,7 +11,6 @@ Use it for modest fan-out progress streaming: AI-agent runs, order pipelines, mu Only available in the Python SDK today; cross-language is on the roadmap. - ## When to use / not to use - Use it for: updating a UI as an AI agent works; surfacing status from a payment or order pipeline; reporting intermediate results from a data job. diff --git a/references/ruby/gotchas.md b/references/ruby/gotchas.md index c2e962c..e56391d 100644 --- a/references/ruby/gotchas.md +++ b/references/ruby/gotchas.md @@ -59,7 +59,6 @@ require_relative 'activities/my_activity' Transient network errors should be retried. Authentication errors should not be. See `references/ruby/error-handling.md` to understand how to classify errors with `non_retryable: true` and `non_retryable_error_types`. - ## Heartbeating ### Forgetting to Heartbeat Long Activities diff --git a/references/typescript/integrations/opentelemetry.md b/references/typescript/integrations/opentelemetry.md new file mode 100644 index 0000000..593fe04 --- /dev/null +++ b/references/typescript/integrations/opentelemetry.md @@ -0,0 +1,176 @@ +# Temporal OpenTelemetry Integration (TypeScript) + +## Overview + +`@temporalio/interceptors-opentelemetry` is a contrib package for the Temporal TypeScript SDK that ships an `OpenTelemetryPlugin` plus per-tier interceptors for tracing Workflow Executions, Child Workflows, Activity invocations, Nexus Operations, and Client `start`/`signal` calls with OpenTelemetry. + +Workflow-side spans are emitted out of the Workflow isolate through an injected Sink (`makeWorkflowExporter`) that hands serialized spans to a host-side `SpanProcessor`. + +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +For non-OTel observability (metrics, runtime logger, sinks) read `references/typescript/observability.md`. For Standalone Activities (client-scheduled Activities) read `references/typescript/standalone-activities.md`. + +## Install + +```bash +npm i @temporalio/interceptors-opentelemetry +``` + +Peer packages this integration normally needs: + +- `@opentelemetry/api` — global propagator, tracer, context. +- `@opentelemetry/sdk-trace-base` — `SpanProcessor`, `BatchSpanProcessor`, `BasicTracerProvider`. +- `@opentelemetry/resources` — `Resource` attached to exported spans. +- An exporter, e.g. `@opentelemetry/exporter-trace-otlp-grpc`. + +## Public API + +| Export | Kind | From | +|---|---|---| +| `OpenTelemetryPlugin` | class extending `SimplePlugin` | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryPluginOptions` | interface | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryWorkflowClientInterceptor` | class (Client) | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryWorkflowClientCallsInterceptor` | `@deprecated` alias of above | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryActivityInboundInterceptor` | class (Activity) | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryActivityOutboundInterceptor` | class (Activity) | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryNexusInboundInterceptor` | class (Nexus) | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryNexusOutboundInterceptor` | class (Nexus) | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryInboundInterceptor` | class (Workflow inbound) | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryOutboundInterceptor` | class (Workflow outbound) | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryInternalsInterceptor` | class (Workflow internals) | `@temporalio/interceptors-opentelemetry` | +| `makeWorkflowExporter` | function | `@temporalio/interceptors-opentelemetry` | +| `SpanName` | enum | `@temporalio/interceptors-opentelemetry` | +| `SPAN_DELIMITER` | const (`':'`) | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetrySinks` | type | `@temporalio/interceptors-opentelemetry` | +| `OpenTelemetryWorkflowExporter` | type | `@temporalio/interceptors-opentelemetry` | +| `SerializableSpan` | type | `@temporalio/interceptors-opentelemetry` | + +## Register via the plugin (recommended) + +Construct one `OpenTelemetryPlugin` instance and pass it to BOTH `bundleWorkflowCode({ plugins })` and `Worker.create({ plugins })` so the Workflow-side interceptors are included in the bundle. + +```ts +import { Resource } from '@opentelemetry/resources'; +import { BasicTracerProvider, BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; +import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; +import { NativeConnection, Worker, bundleWorkflowCode } from '@temporalio/worker'; +import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; + +const resource = new Resource({ 'service.name': 'orders-worker' }); +const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter()); + +const provider = new BasicTracerProvider({ resource }); +provider.addSpanProcessor(spanProcessor); +provider.register(); + +const plugin = new OpenTelemetryPlugin({ resource, spanProcessor }); + +const bundle = await bundleWorkflowCode({ + workflowsPath: require.resolve('./workflows'), + plugins: [plugin], +}); + +const connection = await NativeConnection.connect(); +const worker = await Worker.create({ + connection, + taskQueue: 'orders', + workflowBundle: bundle, + activities: { /* ... */ }, + plugins: [plugin], +}); +await worker.run(); +``` + +The Client accepts the same plugin via the standard plugin path; the plugin contributes `OpenTelemetryWorkflowClientInterceptor` so client-side `start` and `signal` calls are traced. + +```ts +import { Client, Connection } from '@temporalio/client'; +import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; + +const client = new Client({ + connection: await Connection.connect(), + plugins: [plugin], +}); +``` + +## Constructor options + +`new OpenTelemetryPlugin(otelOptions: OpenTelemetryPluginOptions)`. + +| Field | Type | Required | Purpose | +|---|---|---|---| +| `resource` | `Resource` (`@opentelemetry/resources`) | Yes | Resource attributes attached to exported spans. | +| `spanProcessor` | `SpanProcessor` (`@opentelemetry/sdk-trace-base`) | Yes | Receives Workflow, Activity, Client, and Nexus spans. | +| `tracer` | `otel.Tracer` (`@opentelemetry/api`) | No | Override the tracer used by Client/Activity/Nexus interceptors; defaults to `otel.trace.getTracer('@temporalio/interceptor-client'|'-activity'|'-nexus')`. | + +## Trace context propagation + +The TypeScript SDK uses the global OpenTelemetry propagator; the default is W3C TraceContext + W3C Baggage. + +To extend (e.g. add Jaeger), call `propagation.setGlobalPropagator(new CompositePropagator({ propagators: [...] }))` at the top level of your Workflow code BEFORE the Worker bundles it. + +```ts +import { propagation } from '@opentelemetry/api'; +import { CompositePropagator, W3CBaggagePropagator, W3CTraceContextPropagator } from '@opentelemetry/core'; +import { JaegerPropagator } from '@opentelemetry/propagator-jaeger'; + +propagation.setGlobalPropagator( + new CompositePropagator({ + propagators: [ + new W3CTraceContextPropagator(), + new W3CBaggagePropagator(), + new JaegerPropagator(), + ], + }), +); +``` + +## Span names + +`SpanName` enum values; emitted spans are formed as `${SpanName.X}${SPAN_DELIMITER}${suffix}` where `SPAN_DELIMITER = ':'`. + +| Enum | String | Where | +|---|---|---| +| `WORKFLOW_START` | `StartWorkflow` | Client `start` | +| `WORKFLOW_SIGNAL` | `SignalWorkflow` | Client `signal`, Workflow `signalWorkflow` outbound | +| `WORKFLOW_EXECUTE` | `RunWorkflow` | Workflow inbound `execute` | +| `CHILD_WORKFLOW_START` | `StartChildWorkflow` | Workflow outbound `startChildWorkflowExecution` | +| `ACTIVITY_START` | `StartActivity` | Workflow outbound `scheduleActivity` / `scheduleLocalActivity` | +| `ACTIVITY_EXECUTE` | `RunActivity` | Activity inbound `execute` | +| `CONTINUE_AS_NEW` | `ContinueAsNew` | Workflow outbound `continueAsNew` | +| `NEXUS_OPERATION_START` | `StartNexusOperation` | Workflow outbound `startNexusOperation` | +| `NEXUS_START_OPERATION_EXECUTE` | `RunStartNexusOperation` | Nexus inbound `startOperation` | +| `NEXUS_CANCEL_OPERATION_EXECUTE` | `RunCancelNexusOperation` | Nexus inbound `cancelOperation` | + +## Standalone Activities + +`OpenTelemetryActivityInboundInterceptor` extracts the parent span context from `input.headers` regardless of whether the Activity was scheduled by a Workflow or directly by a Client (Standalone Activities). The same plugin registration traces both paths. + +For Standalone Activities themselves read `references/typescript/standalone-activities.md`. + +## Nexus + +`OpenTelemetryNexusInboundInterceptor` wraps inbound `startOperation` and `cancelOperation`, extracting parent context from `input.ctx.headers` and tagging spans with `NEXUS_SERVICE_ATTR_KEY` / `NEXUS_OPERATION_ATTR_KEY`. `OpenTelemetryNexusOutboundInterceptor` emits `trace_id` / `span_id` / `trace_flags` on log attributes and metric tags when a valid span context is active. + +OpenTelemetry is the supported tracing path for Nexus in TypeScript; see the [`interceptors-opentelemetry`](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) sample. + +## Log and metric correlation + +When a valid OTel span context is active during an Activity, Workflow, or Nexus call, the outbound interceptors merge three keys — `trace_id`, `span_id`, `trace_flags` — into the result of `getLogAttributes` (used by `log.*`) and `getMetricTags` (used by worker metric tags). `trace_flags` is formatted as `0${spanContext.traceFlags.toString(16)}`. + +## Common mistakes + +- Don't pass only `resource` or only `spanProcessor`. Both are required on `OpenTelemetryPluginOptions`. +- Don't call `new OpenTelemetryPlugin()` with no argument; the constructor requires `otelOptions`. +- Don't pass the plugin to `Worker.create` but skip `bundleWorkflowCode` — Workflow-side interceptors must be in the bundle. +- Don't install `@temporalio/opentelemetry`. The package is `@temporalio/interceptors-opentelemetry`. +- Don't use the `makeWorkflowExporter(spanExporter, resource)` overload — it is `@deprecated`. Pass a `SpanProcessor` instead. +- Don't expect non-default propagators (e.g. Jaeger) to work without setting the global propagator before `bundleWorkflowCode` runs. + +## Resources + +- TypeScript observability guide: +- Contrib package README: +- Sample: +- SDK metrics reference: From 8225809a30e4436c20e5cbe0ca7f314c5f9b706f Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Wed, 3 Jun 2026 17:15:59 -0400 Subject: [PATCH 02/16] Remove legacy TracingInterceptor content from OpenTelemetry docs Co-Authored-By: Claude Opus 4.8 (1M context) --- references/integrations.md | 2 +- .../python/integrations/opentelemetry.md | 75 +++---------------- .../typescript/integrations/opentelemetry.md | 2 +- 3 files changed, 11 insertions(+), 68 deletions(-) diff --git a/references/integrations.md b/references/integrations.md index 6202a9c..c306a83 100644 --- a/references/integrations.md +++ b/references/integrations.md @@ -19,5 +19,5 @@ Temporal ships and supports a growing set of integrations with third-party frame | LangSmith tracing (`temporalio.contrib.langsmith`) | Python | Experimental Temporal Plugin that propagates LangSmith trace context across Worker boundaries; lets `@traceable` run inside Workflows and Activities | `references/python/integrations/langsmith.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | LangGraph (`temporalio.contrib.langgraph`, Pre-release) | Python | Runs LangGraph Graph-API and Functional-API code as Temporal Workflows - nodes/tasks can execute as either in-workflow or as Activities | `references/python/integrations/langgraph.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | Google ADK (`temporalio[google-adk]`) | Python | Durable Google ADK agents: model calls run through `TemporalModel`-wrapped Activities, tools via `activity_tool`, MCP toolsets via `TemporalMcpToolSet` | `references/python/integrations/google-adk.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | -| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing via the new `OpenTelemetryPlugin` (recommended, experimental) or legacy `TracingInterceptor`; propagates W3C TraceContext + Baggage across Client, Workflow, Activity (including Standalone), Child Workflow, and Nexus boundaries | `references/python/integrations/opentelemetry.md` | `references/python/observability.md`, `references/python/standalone-activities.md` | +| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing via the `OpenTelemetryPlugin` (experimental); propagates W3C TraceContext + Baggage across Client, Workflow, Activity (including Standalone), Child Workflow, and Nexus boundaries | `references/python/integrations/opentelemetry.md` | `references/python/observability.md`, `references/python/standalone-activities.md` | | OpenTelemetry (`@temporalio/interceptors-opentelemetry`) | TypeScript | Distributed tracing via `OpenTelemetryPlugin` plus per-tier Client/Activity/Nexus/Workflow interceptors; Workflow spans exported through an injected Sink; pass the plugin to both `bundleWorkflowCode` and `Worker.create` | `references/typescript/integrations/opentelemetry.md` | `references/typescript/observability.md`, `references/typescript/standalone-activities.md` | diff --git a/references/python/integrations/opentelemetry.md b/references/python/integrations/opentelemetry.md index d1f414e..86db472 100644 --- a/references/python/integrations/opentelemetry.md +++ b/references/python/integrations/opentelemetry.md @@ -2,10 +2,9 @@ ## Overview -`temporalio.contrib.opentelemetry` ships two ways to wire OpenTelemetry tracing into Temporal: -the new `OpenTelemetryPlugin` (recommended, experimental) and the legacy `TracingInterceptor` (stable). +`temporalio.contrib.opentelemetry` wires OpenTelemetry tracing into Temporal through the `OpenTelemetryPlugin`. -Both propagate W3C TraceContext + W3C Baggage through Temporal headers across Client, Workflow, Activity, Child Workflow, and Nexus boundaries. +It propagates W3C TraceContext + W3C Baggage through Temporal headers across Client, Workflow, Activity, Child Workflow, and Nexus boundaries. For non-OTel observability (metrics, logging, telemetry runtime) read `references/python/observability.md`. For trace propagation through `client.start_activity` / `client.execute_activity` see `references/python/standalone-activities.md`. @@ -13,7 +12,7 @@ For trace propagation through `client.start_activity` / `client.execute_activity > [!NOTE] > This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. -`OpenTelemetryPlugin` is marked experimental in its docstring; `TracingInterceptor` is stable. +`OpenTelemetryPlugin` is marked experimental in its docstring. ## Install @@ -25,23 +24,15 @@ The extra is `opentelemetry`. `uv add temporalio[opentelemetry]` works too. ## Public API -All five symbols are re-exported from `temporalio.contrib.opentelemetry`. +All three symbols are re-exported from `temporalio.contrib.opentelemetry`. | Symbol | Purpose | |---|---| -| `OpenTelemetryPlugin` | New, recommended Client plugin; installs `OpenTelemetryInterceptor` and adds `opentelemetry` sandbox passthrough. | +| `OpenTelemetryPlugin` | Client plugin; installs `OpenTelemetryInterceptor` and adds `opentelemetry` sandbox passthrough. | | `OpenTelemetryInterceptor` | Underlying interceptor used by `OpenTelemetryPlugin`; requires the global tracer provider be a `ReplaySafeTracerProvider`. | -| `TracingInterceptor` | Legacy stable interceptor — register on `Client.connect(interceptors=...)`. | -| `TracingWorkflowInboundInterceptor` | Workflow-side half of the legacy interceptor; subclass it only when customizing header key, propagator, or payload converter. | | `create_tracer_provider` | Builds a `ReplaySafeTracerProvider`; required when using `OpenTelemetryPlugin`. | -## Pick a registration path - -- Use `OpenTelemetryPlugin` (new, recommended) for new code: accurate workflow span durations and direct `opentelemetry.trace.get_tracer(...)` usage inside workflows. -- Use `TracingInterceptor` (legacy, stable) when you need immediate span visibility before workflow completion or a non-experimental API. -- Do not register both on the same Client. - -## `OpenTelemetryPlugin` (recommended) +## `OpenTelemetryPlugin` Build a `ReplaySafeTracerProvider` with `create_tracer_provider()`, attach span processors, set it as the global tracer provider, then connect with `plugins=[OpenTelemetryPlugin()]`. Workers created from the returned Client inherit the plugin automatically. @@ -94,46 +85,6 @@ class MyWorkflow: ) ``` -## `TracingInterceptor` (legacy) - -Register on the Client. Workers built from that Client inherit the interceptor. - -```python -from temporalio.client import Client -from temporalio.contrib.opentelemetry import TracingInterceptor -from temporalio.worker import Worker - -client = await Client.connect( - "localhost:7233", - interceptors=[TracingInterceptor()], -) - -worker = Worker( - client, - task_queue="my-task-queue", - workflows=[MyWorkflow], - activities=[my_activity], -) -``` - -Constructor: `TracingInterceptor(tracer: opentelemetry.trace.Tracer | None = None, *, always_create_workflow_spans: bool = False)`. `tracer` defaults to `opentelemetry.trace.get_tracer(__name__)`. `always_create_workflow_spans=True` creates workflow spans even without an upstream client span (risk: orphaned spans after replay). - -Inside a Workflow, standard `opentelemetry.trace` APIs do not work correctly. Use `temporalio.contrib.opentelemetry.workflow.completed_span(...)`. Workflow spans have zero duration. - -```python -import temporalio.contrib.opentelemetry.workflow as otel_workflow -from temporalio import workflow - -@workflow.defn -class MyWorkflow: - @workflow.run - async def run(self) -> None: - otel_workflow.completed_span( - "workflow-operation", - attributes={"business.unit": "payments", "request.id": "req-123"}, - ) -``` - ## Standalone Activities Trace context propagates through `client.start_activity` and `client.execute_activity` via Temporal headers, the same way it propagates from a Workflow. The client outbound's `start_activity(input: StartActivityInput)` opens a `StartActivity:{activity_type}` span (kind=CLIENT) and injects context into `input.headers`; the activity-side `_TracingActivityInboundInterceptor.execute_activity` extracts that context from `input.headers` and opens a `RunActivity:{activity_type}` span (kind=SERVER). @@ -142,26 +93,18 @@ See `references/python/standalone-activities.md`. ## Nexus -`OpenTelemetryInterceptor.intercept_nexus_operation` (and the legacy `TracingInterceptor`'s equivalent) wraps inbound Nexus handlers with `RunStartNexusOperationHandler:{service}/{operation}` and `RunCancelNexusOperationHandler:{service}/{operation}` spans (kind=SERVER), extracting context from Nexus operation headers. +`OpenTelemetryInterceptor.intercept_nexus_operation` wraps inbound Nexus handlers with `RunStartNexusOperationHandler:{service}/{operation}` and `RunCancelNexusOperationHandler:{service}/{operation}` spans (kind=SERVER), extracting context from Nexus operation headers. See `https://github.com/temporalio/samples-python/tree/main/open_telemetry`. -## Migration from `TracingInterceptor` to `OpenTelemetryPlugin` - -1. Replace `interceptors=[TracingInterceptor()]` on `Client.connect` with `plugins=[OpenTelemetryPlugin()]`, and add `create_tracer_provider()` + `opentelemetry.trace.set_tracer_provider(provider)` before connecting. -2. Replace `temporalio.contrib.opentelemetry.workflow.completed_span("my-span")` calls inside workflows with `tracer = get_tracer(__name__)` + `with tracer.start_as_current_span("my-span"):`. -3. Re-verify any monitoring or analysis tools that depend on the legacy trace structure — span shapes change. - ## Common mistakes -- **Registering the same plugin or interceptor on both Client and Worker.** Register on the Client only; Workers inherit. -- **Mixing `OpenTelemetryPlugin` and `TracingInterceptor`.** They are two paths; pick one. +- **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit. - **Calling `Client.connect` before `opentelemetry.trace.set_tracer_provider(provider)`.** With `OpenTelemetryPlugin`, the workflow interceptor factory raises `ValueError("When using OpenTelemetryPlugin, the global trace provider must be a ReplaySafeTracerProvider. Use init_tracer_provider to create one.")`. - **Building a plain `opentelemetry.sdk.trace.TracerProvider` and passing it to `set_tracer_provider`.** `OpenTelemetryPlugin` requires `ReplaySafeTracerProvider` — build it via `create_tracer_provider(...)`. - **Using `addTemporalSpans=True` or other camelCase.** The parameter is `add_temporal_spans` (Python snake_case). - **Passing the install extra as `temporalio[otel]`.** The extra is `opentelemetry`. -- **Calling `get_tracer().start_as_current_span` inside a Workflow with only `TracingInterceptor` registered.** That path requires `OpenTelemetryPlugin`; under the legacy interceptor use `temporalio.contrib.opentelemetry.workflow.completed_span(...)` instead. -- **Adding `with_passthrough_modules("opentelemetry")` to a `SandboxedWorkflowRunner` manually when using the plugin.** The plugin already does this. +- **Adding `with_passthrough_modules("opentelemetry")` to a `SandboxedWorkflowRunner` manually.** The plugin already does this. ## Resources diff --git a/references/typescript/integrations/opentelemetry.md b/references/typescript/integrations/opentelemetry.md index 593fe04..46d3950 100644 --- a/references/typescript/integrations/opentelemetry.md +++ b/references/typescript/integrations/opentelemetry.md @@ -46,7 +46,7 @@ Peer packages this integration normally needs: | `OpenTelemetryWorkflowExporter` | type | `@temporalio/interceptors-opentelemetry` | | `SerializableSpan` | type | `@temporalio/interceptors-opentelemetry` | -## Register via the plugin (recommended) +## Register via the plugin Construct one `OpenTelemetryPlugin` instance and pass it to BOTH `bundleWorkflowCode({ plugins })` and `Worker.create({ plugins })` so the Workflow-side interceptors are included in the bundle. From c4c356cd74d5a0c23d950fcd067fc0bf2b1e8c4d Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Wed, 3 Jun 2026 17:23:52 -0400 Subject: [PATCH 03/16] Remove Nexus content from OpenTelemetry docs Co-Authored-By: Claude Opus 4.8 (1M context) --- references/integrations.md | 4 ++-- .../python/integrations/opentelemetry.md | 8 +------- .../typescript/integrations/opentelemetry.md | 19 ++++--------------- 3 files changed, 7 insertions(+), 24 deletions(-) diff --git a/references/integrations.md b/references/integrations.md index c306a83..70b61ec 100644 --- a/references/integrations.md +++ b/references/integrations.md @@ -19,5 +19,5 @@ Temporal ships and supports a growing set of integrations with third-party frame | LangSmith tracing (`temporalio.contrib.langsmith`) | Python | Experimental Temporal Plugin that propagates LangSmith trace context across Worker boundaries; lets `@traceable` run inside Workflows and Activities | `references/python/integrations/langsmith.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | LangGraph (`temporalio.contrib.langgraph`, Pre-release) | Python | Runs LangGraph Graph-API and Functional-API code as Temporal Workflows - nodes/tasks can execute as either in-workflow or as Activities | `references/python/integrations/langgraph.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | Google ADK (`temporalio[google-adk]`) | Python | Durable Google ADK agents: model calls run through `TemporalModel`-wrapped Activities, tools via `activity_tool`, MCP toolsets via `TemporalMcpToolSet` | `references/python/integrations/google-adk.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | -| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing via the `OpenTelemetryPlugin` (experimental); propagates W3C TraceContext + Baggage across Client, Workflow, Activity (including Standalone), Child Workflow, and Nexus boundaries | `references/python/integrations/opentelemetry.md` | `references/python/observability.md`, `references/python/standalone-activities.md` | -| OpenTelemetry (`@temporalio/interceptors-opentelemetry`) | TypeScript | Distributed tracing via `OpenTelemetryPlugin` plus per-tier Client/Activity/Nexus/Workflow interceptors; Workflow spans exported through an injected Sink; pass the plugin to both `bundleWorkflowCode` and `Worker.create` | `references/typescript/integrations/opentelemetry.md` | `references/typescript/observability.md`, `references/typescript/standalone-activities.md` | +| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing via the `OpenTelemetryPlugin` (experimental); propagates W3C TraceContext + Baggage across Client, Workflow, Activity (including Standalone), and Child Workflow boundaries | `references/python/integrations/opentelemetry.md` | `references/python/observability.md`, `references/python/standalone-activities.md` | +| OpenTelemetry (`@temporalio/interceptors-opentelemetry`) | TypeScript | Distributed tracing via `OpenTelemetryPlugin` plus per-tier Client/Activity/Workflow interceptors; Workflow spans exported through an injected Sink; pass the plugin to both `bundleWorkflowCode` and `Worker.create` | `references/typescript/integrations/opentelemetry.md` | `references/typescript/observability.md`, `references/typescript/standalone-activities.md` | diff --git a/references/python/integrations/opentelemetry.md b/references/python/integrations/opentelemetry.md index 86db472..29abc0f 100644 --- a/references/python/integrations/opentelemetry.md +++ b/references/python/integrations/opentelemetry.md @@ -4,7 +4,7 @@ `temporalio.contrib.opentelemetry` wires OpenTelemetry tracing into Temporal through the `OpenTelemetryPlugin`. -It propagates W3C TraceContext + W3C Baggage through Temporal headers across Client, Workflow, Activity, Child Workflow, and Nexus boundaries. +It propagates W3C TraceContext + W3C Baggage through Temporal headers across Client, Workflow, Activity, and Child Workflow boundaries. For non-OTel observability (metrics, logging, telemetry runtime) read `references/python/observability.md`. For trace propagation through `client.start_activity` / `client.execute_activity` see `references/python/standalone-activities.md`. @@ -91,12 +91,6 @@ Trace context propagates through `client.start_activity` and `client.execute_act See `references/python/standalone-activities.md`. -## Nexus - -`OpenTelemetryInterceptor.intercept_nexus_operation` wraps inbound Nexus handlers with `RunStartNexusOperationHandler:{service}/{operation}` and `RunCancelNexusOperationHandler:{service}/{operation}` spans (kind=SERVER), extracting context from Nexus operation headers. - -See `https://github.com/temporalio/samples-python/tree/main/open_telemetry`. - ## Common mistakes - **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit. diff --git a/references/typescript/integrations/opentelemetry.md b/references/typescript/integrations/opentelemetry.md index 46d3950..a9ab92b 100644 --- a/references/typescript/integrations/opentelemetry.md +++ b/references/typescript/integrations/opentelemetry.md @@ -2,7 +2,7 @@ ## Overview -`@temporalio/interceptors-opentelemetry` is a contrib package for the Temporal TypeScript SDK that ships an `OpenTelemetryPlugin` plus per-tier interceptors for tracing Workflow Executions, Child Workflows, Activity invocations, Nexus Operations, and Client `start`/`signal` calls with OpenTelemetry. +`@temporalio/interceptors-opentelemetry` is a contrib package for the Temporal TypeScript SDK that ships an `OpenTelemetryPlugin` plus per-tier interceptors for tracing Workflow Executions, Child Workflows, Activity invocations, and Client `start`/`signal` calls with OpenTelemetry. Workflow-side spans are emitted out of the Workflow isolate through an injected Sink (`makeWorkflowExporter`) that hands serialized spans to a host-side `SpanProcessor`. @@ -34,8 +34,6 @@ Peer packages this integration normally needs: | `OpenTelemetryWorkflowClientCallsInterceptor` | `@deprecated` alias of above | `@temporalio/interceptors-opentelemetry` | | `OpenTelemetryActivityInboundInterceptor` | class (Activity) | `@temporalio/interceptors-opentelemetry` | | `OpenTelemetryActivityOutboundInterceptor` | class (Activity) | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryNexusInboundInterceptor` | class (Nexus) | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryNexusOutboundInterceptor` | class (Nexus) | `@temporalio/interceptors-opentelemetry` | | `OpenTelemetryInboundInterceptor` | class (Workflow inbound) | `@temporalio/interceptors-opentelemetry` | | `OpenTelemetryOutboundInterceptor` | class (Workflow outbound) | `@temporalio/interceptors-opentelemetry` | | `OpenTelemetryInternalsInterceptor` | class (Workflow internals) | `@temporalio/interceptors-opentelemetry` | @@ -101,8 +99,8 @@ const client = new Client({ | Field | Type | Required | Purpose | |---|---|---|---| | `resource` | `Resource` (`@opentelemetry/resources`) | Yes | Resource attributes attached to exported spans. | -| `spanProcessor` | `SpanProcessor` (`@opentelemetry/sdk-trace-base`) | Yes | Receives Workflow, Activity, Client, and Nexus spans. | -| `tracer` | `otel.Tracer` (`@opentelemetry/api`) | No | Override the tracer used by Client/Activity/Nexus interceptors; defaults to `otel.trace.getTracer('@temporalio/interceptor-client'|'-activity'|'-nexus')`. | +| `spanProcessor` | `SpanProcessor` (`@opentelemetry/sdk-trace-base`) | Yes | Receives Workflow, Activity, and Client spans. | +| `tracer` | `otel.Tracer` (`@opentelemetry/api`) | No | Override the tracer used by Client/Activity interceptors; defaults to `otel.trace.getTracer('@temporalio/interceptor-client'|'-activity')`. | ## Trace context propagation @@ -139,9 +137,6 @@ propagation.setGlobalPropagator( | `ACTIVITY_START` | `StartActivity` | Workflow outbound `scheduleActivity` / `scheduleLocalActivity` | | `ACTIVITY_EXECUTE` | `RunActivity` | Activity inbound `execute` | | `CONTINUE_AS_NEW` | `ContinueAsNew` | Workflow outbound `continueAsNew` | -| `NEXUS_OPERATION_START` | `StartNexusOperation` | Workflow outbound `startNexusOperation` | -| `NEXUS_START_OPERATION_EXECUTE` | `RunStartNexusOperation` | Nexus inbound `startOperation` | -| `NEXUS_CANCEL_OPERATION_EXECUTE` | `RunCancelNexusOperation` | Nexus inbound `cancelOperation` | ## Standalone Activities @@ -149,15 +144,9 @@ propagation.setGlobalPropagator( For Standalone Activities themselves read `references/typescript/standalone-activities.md`. -## Nexus - -`OpenTelemetryNexusInboundInterceptor` wraps inbound `startOperation` and `cancelOperation`, extracting parent context from `input.ctx.headers` and tagging spans with `NEXUS_SERVICE_ATTR_KEY` / `NEXUS_OPERATION_ATTR_KEY`. `OpenTelemetryNexusOutboundInterceptor` emits `trace_id` / `span_id` / `trace_flags` on log attributes and metric tags when a valid span context is active. - -OpenTelemetry is the supported tracing path for Nexus in TypeScript; see the [`interceptors-opentelemetry`](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) sample. - ## Log and metric correlation -When a valid OTel span context is active during an Activity, Workflow, or Nexus call, the outbound interceptors merge three keys — `trace_id`, `span_id`, `trace_flags` — into the result of `getLogAttributes` (used by `log.*`) and `getMetricTags` (used by worker metric tags). `trace_flags` is formatted as `0${spanContext.traceFlags.toString(16)}`. +When a valid OTel span context is active during an Activity or Workflow call, the outbound interceptors merge three keys — `trace_id`, `span_id`, `trace_flags` — into the result of `getLogAttributes` (used by `log.*`) and `getMetricTags` (used by worker metric tags). `trace_flags` is formatted as `0${spanContext.traceFlags.toString(16)}`. ## Common mistakes From cc6b68afb8b9bb03d3d6030a4f92128743c2d63a Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Thu, 4 Jun 2026 10:23:24 -0400 Subject: [PATCH 04/16] Tie OpenTelemetry tracing into the observability references Both observability.md files advertised "tracing" but had no tracing section and never linked to the OTel integration docs. Add a concise Distributed Tracing (OpenTelemetry) section to each, surface the trace/log/metric correlation, and cross-link so the OTel <-> observability relationship is bidirectional. Deep API stays canonical in integrations/opentelemetry.md. Co-Authored-By: Claude Opus 4.8 (1M context) --- references/python/observability.md | 31 +++++++++++++++++++++++++- references/typescript/observability.md | 28 ++++++++++++++++++++++- 2 files changed, 57 insertions(+), 2 deletions(-) diff --git a/references/python/observability.md b/references/python/observability.md index 0130d89..4ec126e 100644 --- a/references/python/observability.md +++ b/references/python/observability.md @@ -2,7 +2,9 @@ ## Overview -The Python SDK provides comprehensive observability through logging, metrics, tracing, and visibility (Search Attributes). +The Python SDK provides comprehensive observability through logging, metrics, tracing (OpenTelemetry), and visibility (Search Attributes). + +These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate health, **tracing** stitches a single request across Client/Workflow/Activity boundaries, and **Search Attributes** make executions queryable. Tracing also correlates with the other pillars — see [Distributed Tracing](#distributed-tracing-opentelemetry). ## Logging @@ -94,6 +96,32 @@ Runtime.set_default(runtime, error_if_already_set=True) - `temporal_activity_execution_latency` - Activity execution time - `temporal_workflow_task_replay_latency` - Replay duration +> [!TIP] +> For span-level timing that complements these aggregate metrics, enable Temporal spans on the tracing plugin (`OpenTelemetryPlugin(add_temporal_spans=True)`) — see [Distributed Tracing](#distributed-tracing-opentelemetry). + +## Distributed Tracing (OpenTelemetry) + +OpenTelemetry is the supported way to add distributed tracing to Temporal applications. The `OpenTelemetryPlugin` (from `temporalio.contrib.opentelemetry`) propagates W3C TraceContext + Baggage through Temporal headers across Client, Workflow, Activity (including Standalone), and Child Workflow boundaries, so one trace follows a request through your whole execution — with replay-safe, accurate span durations. + +```python +import opentelemetry.trace +from temporalio.client import Client +from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_provider + +provider = create_tracer_provider() +# provider.add_span_processor(...) # attach your exporter +opentelemetry.trace.set_tracer_provider(provider) + +client = await Client.connect("localhost:7233", plugins=[OpenTelemetryPlugin()]) +``` + +Workers created from this Client inherit the plugin automatically. Inside a Workflow you then use standard OpenTelemetry APIs (`get_tracer(...).start_as_current_span(...)`); pass `OpenTelemetryPlugin(add_temporal_spans=True)` to also emit `StartWorkflow` / `RunWorkflow` / `StartActivity` / `RunActivity` spans alongside the SDK metrics above. + +This is a deliberately minimal orientation. For the full public API, replay-safe tracer-provider setup, Standalone Activity propagation, and common mistakes, see `references/python/integrations/opentelemetry.md`. (That file links back here for the logging, metrics, and runtime topics above.) + +> [!NOTE] +> `OpenTelemetryPlugin` is in Public Preview / marked experimental. It is fine to use on a user's behalf, but tell them it is a Public Preview feature. + ## Search Attributes (Visibility) See the Search Attributes section of `references/python/data-handling.md` @@ -104,3 +132,4 @@ See the Search Attributes section of `references/python/data-handling.md` 2. Don't use print() in workflows - it will produce duplicate output on replay 3. Configure metrics for production monitoring 4. Use Search Attributes for business-level visibility +5. Use the `OpenTelemetryPlugin` for distributed tracing across Client/Workflow/Activity boundaries (see [Distributed Tracing](#distributed-tracing-opentelemetry)) diff --git a/references/typescript/observability.md b/references/typescript/observability.md index 211fbc6..f098809 100644 --- a/references/typescript/observability.md +++ b/references/typescript/observability.md @@ -2,7 +2,9 @@ ## Overview -The TypeScript SDK provides replay-aware logging, metrics, and integrations for production observability. +The TypeScript SDK provides replay-aware logging, metrics, and distributed tracing (OpenTelemetry) for production observability. + +These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate worker health, **tracing** stitches a single request across Client/Workflow/Activity boundaries, and **Search Attributes** make executions queryable. With the tracing plugin installed they also reinforce each other — trace IDs are injected into log metadata and metric tags (see [Distributed Tracing](#distributed-tracing-opentelemetry)). ## Replay-Aware Logging @@ -100,6 +102,29 @@ Runtime.install({ }); ``` +## Distributed Tracing (OpenTelemetry) + +OpenTelemetry is the supported way to add distributed tracing to Temporal applications. The `OpenTelemetryPlugin` (from `@temporalio/interceptors-opentelemetry`) traces Workflow Executions, Child Workflows, Activity invocations, and Client `start`/`signal` calls, propagating W3C TraceContext + Baggage across all of them. Workflow-side spans are exported out of the Workflow isolate through an injected Sink. + +Construct one plugin instance and pass it to the Client, to `bundleWorkflowCode`, and to `Worker.create` (the Workflow-side interceptors must be in the bundle): + +```typescript +import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; + +const plugin = new OpenTelemetryPlugin({ resource, spanProcessor }); + +const bundle = await bundleWorkflowCode({ workflowsPath, plugins: [plugin] }); +const worker = await Worker.create({ connection, taskQueue, workflowBundle: bundle, plugins: [plugin] }); +// const client = new Client({ connection, plugins: [plugin] }); +``` + +**Correlation with logging and metrics.** When a valid span context is active during an Activity or Workflow call, the plugin merges `trace_id` / `span_id` / `trace_flags` into the log metadata used by the `log.*` calls in [Workflow Logging](#workflow-logging) / [Activity Logging](#activity-logging) **and** into the worker metric tags from the [Metrics](#metrics) section — so traces, logs, and metrics share correlation IDs without extra wiring. + +This is a deliberately minimal orientation. For the full public API, constructor options, span names, propagator customization, and Standalone Activity propagation, see `references/typescript/integrations/opentelemetry.md`. (That file links back here for the logging, metrics, and sink topics above.) + +> [!NOTE] +> `OpenTelemetryPlugin` is in Public Preview. It is fine to use on a user's behalf, but tell them it is a Public Preview feature. + ## Search Attributes (Visibility) See the Search Attributes section of `references/typescript/data-handling.md` @@ -111,3 +136,4 @@ See the Search Attributes section of `references/typescript/data-handling.md` 3. Configure Winston or similar for production log aggregation 4. Monitor Prometheus metrics for worker health 5. Use Event History for debugging workflow issues +6. Use the `OpenTelemetryPlugin` for distributed tracing — it also stamps trace IDs onto your logs and metrics (see [Distributed Tracing](#distributed-tracing-opentelemetry)) From a653436157a7da8eddfcce9ca4c747ac3c9d600d Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Thu, 4 Jun 2026 11:13:14 -0400 Subject: [PATCH 05/16] Trim TypeScript OpenTelemetry doc to lean style Mirror the lean, example-driven style now used in the Python OTel doc: fold the Public API / Constructor options / Span names tables into inline comments and prose, compress propagator customization to a one-liner, and keep the log/metric correlation tie-in. Update the observability.md pointer so it no longer promises tables that were removed. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../python/integrations/opentelemetry.md | 42 ++----- references/python/observability.md | 2 +- .../typescript/integrations/opentelemetry.md | 117 +++--------------- references/typescript/observability.md | 2 +- 4 files changed, 28 insertions(+), 135 deletions(-) diff --git a/references/python/integrations/opentelemetry.md b/references/python/integrations/opentelemetry.md index 29abc0f..c8b980a 100644 --- a/references/python/integrations/opentelemetry.md +++ b/references/python/integrations/opentelemetry.md @@ -6,31 +6,14 @@ It propagates W3C TraceContext + W3C Baggage through Temporal headers across Client, Workflow, Activity, and Child Workflow boundaries. -For non-OTel observability (metrics, logging, telemetry runtime) read `references/python/observability.md`. -For trace propagation through `client.start_activity` / `client.execute_activity` see `references/python/standalone-activities.md`. +For non-OTel observability (metrics, logging) read `references/python/observability.md`. > [!NOTE] > This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. -`OpenTelemetryPlugin` is marked experimental in its docstring. - ## Install -```bash -pip install temporalio[opentelemetry] -``` - -The extra is `opentelemetry`. `uv add temporalio[opentelemetry]` works too. - -## Public API - -All three symbols are re-exported from `temporalio.contrib.opentelemetry`. - -| Symbol | Purpose | -|---|---| -| `OpenTelemetryPlugin` | Client plugin; installs `OpenTelemetryInterceptor` and adds `opentelemetry` sandbox passthrough. | -| `OpenTelemetryInterceptor` | Underlying interceptor used by `OpenTelemetryPlugin`; requires the global tracer provider be a `ReplaySafeTracerProvider`. | -| `create_tracer_provider` | Builds a `ReplaySafeTracerProvider`; required when using `OpenTelemetryPlugin`. | +Modify your dependency on the `temporalio` package to add the `opentelemetry` extra, e.g. `uv add temporalio[opentelemetry]` or whatever is appropriate for your package manager. ## `OpenTelemetryPlugin` @@ -44,11 +27,13 @@ from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_ from temporalio.worker import Worker provider = create_tracer_provider() -provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) +provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) # attach your span processors as normal for OTel opentelemetry.trace.set_tracer_provider(provider) client = await Client.connect( "localhost:7233", + # Pass add_temporal_spans=True to also emit `StartWorkflow`, `RunWorkflow`, + # `StartActivity`, `RunActivity` (and similar) spans on top of application spans. plugins=[OpenTelemetryPlugin()], ) @@ -60,13 +45,7 @@ worker = Worker( ) ``` -Constructor: `OpenTelemetryPlugin(*, add_temporal_spans: bool = False)`. Keyword-only; `add_temporal_spans=True` adds `StartWorkflow`, `RunWorkflow`, `StartActivity`, `RunActivity` (and similar) spans on top of application spans. - -`create_tracer_provider(sampler=None, resource=None, shutdown_on_exit=True, active_span_processor=None, id_generator=None, span_limits=None) -> ReplaySafeTracerProvider`. The provider must be set with `opentelemetry.trace.set_tracer_provider(...)` before `Client.connect`; otherwise the workflow interceptor factory raises `ValueError("When using OpenTelemetryPlugin, the global trace provider must be a ReplaySafeTracerProvider. Use init_tracer_provider to create one.")`. - -The plugin's `workflow_runner` callback applies `with_passthrough_modules("opentelemetry")` to `SandboxedWorkflowRunner.restrictions`. Do not add this passthrough manually. - -Inside a Workflow, use standard OpenTelemetry APIs — durations are accurate. +Then just use standard OpenTelemetry APIs in your workflow or activity code — durations are accurate. ```python from datetime import timedelta @@ -87,22 +66,15 @@ class MyWorkflow: ## Standalone Activities -Trace context propagates through `client.start_activity` and `client.execute_activity` via Temporal headers, the same way it propagates from a Workflow. The client outbound's `start_activity(input: StartActivityInput)` opens a `StartActivity:{activity_type}` span (kind=CLIENT) and injects context into `input.headers`; the activity-side `_TracingActivityInboundInterceptor.execute_activity` extracts that context from `input.headers` and opens a `RunActivity:{activity_type}` span (kind=SERVER). - -See `references/python/standalone-activities.md`. +Standalone activities (`references/python/standalone-activities.md`) are supported automatically. ## Common mistakes - **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit. - **Calling `Client.connect` before `opentelemetry.trace.set_tracer_provider(provider)`.** With `OpenTelemetryPlugin`, the workflow interceptor factory raises `ValueError("When using OpenTelemetryPlugin, the global trace provider must be a ReplaySafeTracerProvider. Use init_tracer_provider to create one.")`. - **Building a plain `opentelemetry.sdk.trace.TracerProvider` and passing it to `set_tracer_provider`.** `OpenTelemetryPlugin` requires `ReplaySafeTracerProvider` — build it via `create_tracer_provider(...)`. -- **Using `addTemporalSpans=True` or other camelCase.** The parameter is `add_temporal_spans` (Python snake_case). -- **Passing the install extra as `temporalio[otel]`.** The extra is `opentelemetry`. - **Adding `with_passthrough_modules("opentelemetry")` to a `SandboxedWorkflowRunner` manually.** The plugin already does this. ## Resources -- Temporal Python observability docs: `https://docs.temporal.io/develop/python/platform/observability` -- Python SDK OTel contrib README: `https://github.com/temporalio/sdk-python/blob/main/temporalio/contrib/opentelemetry/README.md` -- Python samples — OpenTelemetry: `https://github.com/temporalio/samples-python/tree/main/open_telemetry` - SDK metrics reference: `references/python/observability.md` diff --git a/references/python/observability.md b/references/python/observability.md index 4ec126e..995967a 100644 --- a/references/python/observability.md +++ b/references/python/observability.md @@ -117,7 +117,7 @@ client = await Client.connect("localhost:7233", plugins=[OpenTelemetryPlugin()]) Workers created from this Client inherit the plugin automatically. Inside a Workflow you then use standard OpenTelemetry APIs (`get_tracer(...).start_as_current_span(...)`); pass `OpenTelemetryPlugin(add_temporal_spans=True)` to also emit `StartWorkflow` / `RunWorkflow` / `StartActivity` / `RunActivity` spans alongside the SDK metrics above. -This is a deliberately minimal orientation. For the full public API, replay-safe tracer-provider setup, Standalone Activity propagation, and common mistakes, see `references/python/integrations/opentelemetry.md`. (That file links back here for the logging, metrics, and runtime topics above.) +This is a deliberately minimal orientation. For the full public API, replay-safe tracer-provider setup, Standalone Activity propagation, and common mistakes, see `references/python/integrations/opentelemetry.md`. > [!NOTE] > `OpenTelemetryPlugin` is in Public Preview / marked experimental. It is fine to use on a user's behalf, but tell them it is a Public Preview feature. diff --git a/references/typescript/integrations/opentelemetry.md b/references/typescript/integrations/opentelemetry.md index a9ab92b..bad85e3 100644 --- a/references/typescript/integrations/opentelemetry.md +++ b/references/typescript/integrations/opentelemetry.md @@ -2,51 +2,22 @@ ## Overview -`@temporalio/interceptors-opentelemetry` is a contrib package for the Temporal TypeScript SDK that ships an `OpenTelemetryPlugin` plus per-tier interceptors for tracing Workflow Executions, Child Workflows, Activity invocations, and Client `start`/`signal` calls with OpenTelemetry. +`@temporalio/interceptors-opentelemetry` wires OpenTelemetry tracing into Temporal through the `OpenTelemetryPlugin`. It traces Workflow Executions, Child Workflows, Activity invocations, and Client `start`/`signal` calls, propagating W3C TraceContext + Baggage across all of them. -Workflow-side spans are emitted out of the Workflow isolate through an injected Sink (`makeWorkflowExporter`) that hands serialized spans to a host-side `SpanProcessor`. +Workflow-side spans are emitted out of the Workflow isolate through an injected Sink that hands serialized spans to a host-side `SpanProcessor`. + +For non-OTel observability (metrics, runtime logger, sinks) read `references/typescript/observability.md`. > [!NOTE] > This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. -For non-OTel observability (metrics, runtime logger, sinks) read `references/typescript/observability.md`. For Standalone Activities (client-scheduled Activities) read `references/typescript/standalone-activities.md`. - ## Install -```bash -npm i @temporalio/interceptors-opentelemetry -``` +Install `@temporalio/interceptors-opentelemetry` plus the OpenTelemetry peer packages you use — typically `@opentelemetry/api`, `@opentelemetry/sdk-trace-base`, `@opentelemetry/resources`, and an exporter (e.g. `@opentelemetry/exporter-trace-otlp-grpc`). -Peer packages this integration normally needs: - -- `@opentelemetry/api` — global propagator, tracer, context. -- `@opentelemetry/sdk-trace-base` — `SpanProcessor`, `BatchSpanProcessor`, `BasicTracerProvider`. -- `@opentelemetry/resources` — `Resource` attached to exported spans. -- An exporter, e.g. `@opentelemetry/exporter-trace-otlp-grpc`. - -## Public API - -| Export | Kind | From | -|---|---|---| -| `OpenTelemetryPlugin` | class extending `SimplePlugin` | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryPluginOptions` | interface | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryWorkflowClientInterceptor` | class (Client) | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryWorkflowClientCallsInterceptor` | `@deprecated` alias of above | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryActivityInboundInterceptor` | class (Activity) | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryActivityOutboundInterceptor` | class (Activity) | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryInboundInterceptor` | class (Workflow inbound) | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryOutboundInterceptor` | class (Workflow outbound) | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryInternalsInterceptor` | class (Workflow internals) | `@temporalio/interceptors-opentelemetry` | -| `makeWorkflowExporter` | function | `@temporalio/interceptors-opentelemetry` | -| `SpanName` | enum | `@temporalio/interceptors-opentelemetry` | -| `SPAN_DELIMITER` | const (`':'`) | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetrySinks` | type | `@temporalio/interceptors-opentelemetry` | -| `OpenTelemetryWorkflowExporter` | type | `@temporalio/interceptors-opentelemetry` | -| `SerializableSpan` | type | `@temporalio/interceptors-opentelemetry` | - -## Register via the plugin - -Construct one `OpenTelemetryPlugin` instance and pass it to BOTH `bundleWorkflowCode({ plugins })` and `Worker.create({ plugins })` so the Workflow-side interceptors are included in the bundle. +## `OpenTelemetryPlugin` + +Construct one `OpenTelemetryPlugin` and pass it to the Client, `bundleWorkflowCode`, and `Worker.create`. It must reach `bundleWorkflowCode` so the Workflow-side interceptors are included in the bundle. Lifecycle spans (workflow / activity / client) are then created automatically. ```ts import { Resource } from '@opentelemetry/resources'; @@ -62,6 +33,8 @@ const provider = new BasicTracerProvider({ resource }); provider.addSpanProcessor(spanProcessor); provider.register(); +// `resource` and `spanProcessor` are required; pass an optional `tracer` to override +// the tracer used by the Client/Activity interceptors. const plugin = new OpenTelemetryPlugin({ resource, spanProcessor }); const bundle = await bundleWorkflowCode({ @@ -80,11 +53,10 @@ const worker = await Worker.create({ await worker.run(); ``` -The Client accepts the same plugin via the standard plugin path; the plugin contributes `OpenTelemetryWorkflowClientInterceptor` so client-side `start` and `signal` calls are traced. +Pass the same plugin to the Client so client-side `start` and `signal` calls are traced: ```ts import { Client, Connection } from '@temporalio/client'; -import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; const client = new Client({ connection: await Connection.connect(), @@ -92,74 +64,23 @@ const client = new Client({ }); ``` -## Constructor options - -`new OpenTelemetryPlugin(otelOptions: OpenTelemetryPluginOptions)`. - -| Field | Type | Required | Purpose | -|---|---|---|---| -| `resource` | `Resource` (`@opentelemetry/resources`) | Yes | Resource attributes attached to exported spans. | -| `spanProcessor` | `SpanProcessor` (`@opentelemetry/sdk-trace-base`) | Yes | Receives Workflow, Activity, and Client spans. | -| `tracer` | `otel.Tracer` (`@opentelemetry/api`) | No | Override the tracer used by Client/Activity interceptors; defaults to `otel.trace.getTracer('@temporalio/interceptor-client'|'-activity')`. | - -## Trace context propagation - -The TypeScript SDK uses the global OpenTelemetry propagator; the default is W3C TraceContext + W3C Baggage. - -To extend (e.g. add Jaeger), call `propagation.setGlobalPropagator(new CompositePropagator({ propagators: [...] }))` at the top level of your Workflow code BEFORE the Worker bundles it. - -```ts -import { propagation } from '@opentelemetry/api'; -import { CompositePropagator, W3CBaggagePropagator, W3CTraceContextPropagator } from '@opentelemetry/core'; -import { JaegerPropagator } from '@opentelemetry/propagator-jaeger'; - -propagation.setGlobalPropagator( - new CompositePropagator({ - propagators: [ - new W3CTraceContextPropagator(), - new W3CBaggagePropagator(), - new JaegerPropagator(), - ], - }), -); -``` - -## Span names - -`SpanName` enum values; emitted spans are formed as `${SpanName.X}${SPAN_DELIMITER}${suffix}` where `SPAN_DELIMITER = ':'`. - -| Enum | String | Where | -|---|---|---| -| `WORKFLOW_START` | `StartWorkflow` | Client `start` | -| `WORKFLOW_SIGNAL` | `SignalWorkflow` | Client `signal`, Workflow `signalWorkflow` outbound | -| `WORKFLOW_EXECUTE` | `RunWorkflow` | Workflow inbound `execute` | -| `CHILD_WORKFLOW_START` | `StartChildWorkflow` | Workflow outbound `startChildWorkflowExecution` | -| `ACTIVITY_START` | `StartActivity` | Workflow outbound `scheduleActivity` / `scheduleLocalActivity` | -| `ACTIVITY_EXECUTE` | `RunActivity` | Activity inbound `execute` | -| `CONTINUE_AS_NEW` | `ContinueAsNew` | Workflow outbound `continueAsNew` | +The SDK uses the global OpenTelemetry propagator (default: W3C TraceContext + Baggage). To use a non-default propagator (e.g. Jaeger), call `propagation.setGlobalPropagator(...)` at the top level of your Workflow code BEFORE the Worker bundles it. ## Standalone Activities -`OpenTelemetryActivityInboundInterceptor` extracts the parent span context from `input.headers` regardless of whether the Activity was scheduled by a Workflow or directly by a Client (Standalone Activities). The same plugin registration traces both paths. - -For Standalone Activities themselves read `references/typescript/standalone-activities.md`. +Standalone Activities (`references/typescript/standalone-activities.md`) are traced automatically — the same plugin registration extracts the parent span context whether the Activity was scheduled by a Workflow or directly by a Client. ## Log and metric correlation -When a valid OTel span context is active during an Activity or Workflow call, the outbound interceptors merge three keys — `trace_id`, `span_id`, `trace_flags` — into the result of `getLogAttributes` (used by `log.*`) and `getMetricTags` (used by worker metric tags). `trace_flags` is formatted as `0${spanContext.traceFlags.toString(16)}`. +When a valid OTel span context is active during an Activity or Workflow call, the plugin merges `trace_id`, `span_id`, and `trace_flags` into `getLogAttributes` (used by `log.*`) and `getMetricTags` (used by worker metric tags), so traces, logs, and metrics share correlation IDs. ## Common mistakes -- Don't pass only `resource` or only `spanProcessor`. Both are required on `OpenTelemetryPluginOptions`. -- Don't call `new OpenTelemetryPlugin()` with no argument; the constructor requires `otelOptions`. -- Don't pass the plugin to `Worker.create` but skip `bundleWorkflowCode` — Workflow-side interceptors must be in the bundle. -- Don't install `@temporalio/opentelemetry`. The package is `@temporalio/interceptors-opentelemetry`. -- Don't use the `makeWorkflowExporter(spanExporter, resource)` overload — it is `@deprecated`. Pass a `SpanProcessor` instead. -- Don't expect non-default propagators (e.g. Jaeger) to work without setting the global propagator before `bundleWorkflowCode` runs. +- **Passing only `resource` or only `spanProcessor`.** Both are required; `new OpenTelemetryPlugin()` with no argument throws. +- **Passing the plugin to `Worker.create` but not `bundleWorkflowCode`.** Workflow-side interceptors must be in the bundle. +- **Installing `@temporalio/opentelemetry`.** The package is `@temporalio/interceptors-opentelemetry`. +- **Expecting a non-default propagator (e.g. Jaeger) to work without setting the global propagator before `bundleWorkflowCode` runs.** ## Resources -- TypeScript observability guide: -- Contrib package README: -- Sample: -- SDK metrics reference: +- SDK metrics / observability reference: `references/typescript/observability.md` diff --git a/references/typescript/observability.md b/references/typescript/observability.md index f098809..ea92d80 100644 --- a/references/typescript/observability.md +++ b/references/typescript/observability.md @@ -120,7 +120,7 @@ const worker = await Worker.create({ connection, taskQueue, workflowBundle: bund **Correlation with logging and metrics.** When a valid span context is active during an Activity or Workflow call, the plugin merges `trace_id` / `span_id` / `trace_flags` into the log metadata used by the `log.*` calls in [Workflow Logging](#workflow-logging) / [Activity Logging](#activity-logging) **and** into the worker metric tags from the [Metrics](#metrics) section — so traces, logs, and metrics share correlation IDs without extra wiring. -This is a deliberately minimal orientation. For the full public API, constructor options, span names, propagator customization, and Standalone Activity propagation, see `references/typescript/integrations/opentelemetry.md`. (That file links back here for the logging, metrics, and sink topics above.) +This is a deliberately minimal orientation. For the full setup, constructor options, propagator customization, and Standalone Activity propagation, see `references/typescript/integrations/opentelemetry.md`. > [!NOTE] > `OpenTelemetryPlugin` is in Public Preview. It is fine to use on a user's behalf, but tell them it is a Public Preview feature. From cec317978e4261969491585aa483144ec9b6f382 Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Thu, 4 Jun 2026 17:53:34 -0400 Subject: [PATCH 06/16] Finalize Python file --- references/python/integrations/opentelemetry.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/references/python/integrations/opentelemetry.md b/references/python/integrations/opentelemetry.md index c8b980a..5d496fd 100644 --- a/references/python/integrations/opentelemetry.md +++ b/references/python/integrations/opentelemetry.md @@ -11,7 +11,7 @@ For non-OTel observability (metrics, logging) read `references/python/observabil > [!NOTE] > This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. -## Install +## Install the plugin Modify your dependency on the `temporalio` package to add the `opentelemetry` extra, e.g. `uv add temporalio[opentelemetry]` or whatever is appropriate for your package manager. @@ -45,7 +45,7 @@ worker = Worker( ) ``` -Then just use standard OpenTelemetry APIs in your workflow or activity code — durations are accurate. +Then just use standard OpenTelemetry APIs in your workflow code — durations are accurate. ```python from datetime import timedelta @@ -64,10 +64,6 @@ class MyWorkflow: ) ``` -## Standalone Activities - -Standalone activities (`references/python/standalone-activities.md`) are supported automatically. - ## Common mistakes - **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit. From fa9b1094d23751b24430d42d18b51571d900f547 Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 10:30:43 -0400 Subject: [PATCH 07/16] Simplify OpenTelemetry rows in integrations catalog Reduce both OTel rows to a purpose-only description, dropping mechanism detail (plugin names, interceptors, sinks, propagation specifics). Co-Authored-By: Claude Opus 4.8 (1M context) --- references/integrations.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/references/integrations.md b/references/integrations.md index 70b61ec..8222db0 100644 --- a/references/integrations.md +++ b/references/integrations.md @@ -19,5 +19,5 @@ Temporal ships and supports a growing set of integrations with third-party frame | LangSmith tracing (`temporalio.contrib.langsmith`) | Python | Experimental Temporal Plugin that propagates LangSmith trace context across Worker boundaries; lets `@traceable` run inside Workflows and Activities | `references/python/integrations/langsmith.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | LangGraph (`temporalio.contrib.langgraph`, Pre-release) | Python | Runs LangGraph Graph-API and Functional-API code as Temporal Workflows - nodes/tasks can execute as either in-workflow or as Activities | `references/python/integrations/langgraph.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | Google ADK (`temporalio[google-adk]`) | Python | Durable Google ADK agents: model calls run through `TemporalModel`-wrapped Activities, tools via `activity_tool`, MCP toolsets via `TemporalMcpToolSet` | `references/python/integrations/google-adk.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | -| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing via the `OpenTelemetryPlugin` (experimental); propagates W3C TraceContext + Baggage across Client, Workflow, Activity (including Standalone), and Child Workflow boundaries | `references/python/integrations/opentelemetry.md` | `references/python/observability.md`, `references/python/standalone-activities.md` | -| OpenTelemetry (`@temporalio/interceptors-opentelemetry`) | TypeScript | Distributed tracing via `OpenTelemetryPlugin` plus per-tier Client/Activity/Workflow interceptors; Workflow spans exported through an injected Sink; pass the plugin to both `bundleWorkflowCode` and `Worker.create` | `references/typescript/integrations/opentelemetry.md` | `references/typescript/observability.md`, `references/typescript/standalone-activities.md` | +| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing for Temporal apps with OpenTelemetry | `references/python/integrations/opentelemetry.md` | `references/python/observability.md`, `references/python/standalone-activities.md` | +| OpenTelemetry (`@temporalio/interceptors-opentelemetry`) | TypeScript | Distributed tracing for Temporal apps with OpenTelemetry | `references/typescript/integrations/opentelemetry.md` | `references/typescript/observability.md`, `references/typescript/standalone-activities.md` | From 0b6e2651471280e11151d5d687dd21641b76b49a Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 10:49:27 -0400 Subject: [PATCH 08/16] Finalize observability files other than code snippets --- references/python/observability.md | 16 ++++++++-------- references/typescript/observability.md | 14 ++++++-------- 2 files changed, 14 insertions(+), 16 deletions(-) diff --git a/references/python/observability.md b/references/python/observability.md index 995967a..c2dc9b5 100644 --- a/references/python/observability.md +++ b/references/python/observability.md @@ -4,7 +4,7 @@ The Python SDK provides comprehensive observability through logging, metrics, tracing (OpenTelemetry), and visibility (Search Attributes). -These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate health, **tracing** stitches a single request across Client/Workflow/Activity boundaries, and **Search Attributes** make executions queryable. Tracing also correlates with the other pillars — see [Distributed Tracing](#distributed-tracing-opentelemetry). +These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate health, **tracing** stitches a single request across Client/Workflow/Activity boundaries, and **Search Attributes** make executions queryable. ## Logging @@ -101,7 +101,10 @@ Runtime.set_default(runtime, error_if_already_set=True) ## Distributed Tracing (OpenTelemetry) -OpenTelemetry is the supported way to add distributed tracing to Temporal applications. The `OpenTelemetryPlugin` (from `temporalio.contrib.opentelemetry`) propagates W3C TraceContext + Baggage through Temporal headers across Client, Workflow, Activity (including Standalone), and Child Workflow boundaries, so one trace follows a request through your whole execution — with replay-safe, accurate span durations. +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +OpenTelemetry is the supported way to add distributed tracing to Temporal applications. The `OpenTelemetryPlugin` (from `temporalio.contrib.opentelemetry`, installed via the `temporalio[opentelemetry]` extra) propagates W3C TraceContext + Baggage through Temporal headers across Client, Workflow, Activity (including Standalone), and Child Workflow boundaries, so one trace follows a request through your whole execution — with replay-safe, accurate span durations. ```python import opentelemetry.trace @@ -115,12 +118,9 @@ opentelemetry.trace.set_tracer_provider(provider) client = await Client.connect("localhost:7233", plugins=[OpenTelemetryPlugin()]) ``` -Workers created from this Client inherit the plugin automatically. Inside a Workflow you then use standard OpenTelemetry APIs (`get_tracer(...).start_as_current_span(...)`); pass `OpenTelemetryPlugin(add_temporal_spans=True)` to also emit `StartWorkflow` / `RunWorkflow` / `StartActivity` / `RunActivity` spans alongside the SDK metrics above. +Workers created from this Client inherit the plugin automatically. Inside a Workflow you then use standard OpenTelemetry APIs (`get_tracer(...).start_as_current_span(...)`); pass `OpenTelemetryPlugin(add_temporal_spans=True)` to also emit `StartWorkflow` / `RunWorkflow` / `StartActivity` / `RunActivity` spans automatically alongside the SDK metrics above. -This is a deliberately minimal orientation. For the full public API, replay-safe tracer-provider setup, Standalone Activity propagation, and common mistakes, see `references/python/integrations/opentelemetry.md`. - -> [!NOTE] -> `OpenTelemetryPlugin` is in Public Preview / marked experimental. It is fine to use on a user's behalf, but tell them it is a Public Preview feature. +For the full setup and options, see `references/python/integrations/opentelemetry.md`. ## Search Attributes (Visibility) @@ -132,4 +132,4 @@ See the Search Attributes section of `references/python/data-handling.md` 2. Don't use print() in workflows - it will produce duplicate output on replay 3. Configure metrics for production monitoring 4. Use Search Attributes for business-level visibility -5. Use the `OpenTelemetryPlugin` for distributed tracing across Client/Workflow/Activity boundaries (see [Distributed Tracing](#distributed-tracing-opentelemetry)) +5. Use the `OpenTelemetryPlugin` for distributed tracing across Client/Workflow/Activity boundaries. diff --git a/references/typescript/observability.md b/references/typescript/observability.md index ea92d80..e606270 100644 --- a/references/typescript/observability.md +++ b/references/typescript/observability.md @@ -4,7 +4,7 @@ The TypeScript SDK provides replay-aware logging, metrics, and distributed tracing (OpenTelemetry) for production observability. -These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate worker health, **tracing** stitches a single request across Client/Workflow/Activity boundaries, and **Search Attributes** make executions queryable. With the tracing plugin installed they also reinforce each other — trace IDs are injected into log metadata and metric tags (see [Distributed Tracing](#distributed-tracing-opentelemetry)). +These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate worker health, **tracing** stitches a single request across Client/Workflow/Activity boundaries, and **Search Attributes** make executions queryable. ## Replay-Aware Logging @@ -104,6 +104,9 @@ Runtime.install({ ## Distributed Tracing (OpenTelemetry) +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + OpenTelemetry is the supported way to add distributed tracing to Temporal applications. The `OpenTelemetryPlugin` (from `@temporalio/interceptors-opentelemetry`) traces Workflow Executions, Child Workflows, Activity invocations, and Client `start`/`signal` calls, propagating W3C TraceContext + Baggage across all of them. Workflow-side spans are exported out of the Workflow isolate through an injected Sink. Construct one plugin instance and pass it to the Client, to `bundleWorkflowCode`, and to `Worker.create` (the Workflow-side interceptors must be in the bundle): @@ -118,12 +121,7 @@ const worker = await Worker.create({ connection, taskQueue, workflowBundle: bund // const client = new Client({ connection, plugins: [plugin] }); ``` -**Correlation with logging and metrics.** When a valid span context is active during an Activity or Workflow call, the plugin merges `trace_id` / `span_id` / `trace_flags` into the log metadata used by the `log.*` calls in [Workflow Logging](#workflow-logging) / [Activity Logging](#activity-logging) **and** into the worker metric tags from the [Metrics](#metrics) section — so traces, logs, and metrics share correlation IDs without extra wiring. - -This is a deliberately minimal orientation. For the full setup, constructor options, propagator customization, and Standalone Activity propagation, see `references/typescript/integrations/opentelemetry.md`. - -> [!NOTE] -> `OpenTelemetryPlugin` is in Public Preview. It is fine to use on a user's behalf, but tell them it is a Public Preview feature. +For the full setup and options, see `references/typescript/integrations/opentelemetry.md`. ## Search Attributes (Visibility) @@ -136,4 +134,4 @@ See the Search Attributes section of `references/typescript/data-handling.md` 3. Configure Winston or similar for production log aggregation 4. Monitor Prometheus metrics for worker health 5. Use Event History for debugging workflow issues -6. Use the `OpenTelemetryPlugin` for distributed tracing — it also stamps trace IDs onto your logs and metrics (see [Distributed Tracing](#distributed-tracing-opentelemetry)) +6. Use the `OpenTelemetryPlugin` for distributed tracing across Client/Workflow/Activity boundaries. From 87b4181ccab7111218bafe0a00df98c95fb3f1b7 Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 10:52:39 -0400 Subject: [PATCH 09/16] finalize python observaibility file --- references/python/observability.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/references/python/observability.md b/references/python/observability.md index c2dc9b5..61acc13 100644 --- a/references/python/observability.md +++ b/references/python/observability.md @@ -108,11 +108,12 @@ OpenTelemetry is the supported way to add distributed tracing to Temporal applic ```python import opentelemetry.trace +from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor from temporalio.client import Client from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_provider provider = create_tracer_provider() -# provider.add_span_processor(...) # attach your exporter +provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) # attach your span processors as normal for OTel opentelemetry.trace.set_tracer_provider(provider) client = await Client.connect("localhost:7233", plugins=[OpenTelemetryPlugin()]) From 4f1037ab8f8cca21784ad79353e4e8c0e312cb69 Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 11:01:59 -0400 Subject: [PATCH 10/16] Finalize TS observability file --- references/typescript/observability.md | 32 +++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/references/typescript/observability.md b/references/typescript/observability.md index e606270..7f86642 100644 --- a/references/typescript/observability.md +++ b/references/typescript/observability.md @@ -112,13 +112,39 @@ OpenTelemetry is the supported way to add distributed tracing to Temporal applic Construct one plugin instance and pass it to the Client, to `bundleWorkflowCode`, and to `Worker.create` (the Workflow-side interceptors must be in the bundle): ```typescript +import { Resource } from '@opentelemetry/resources'; +import { BasicTracerProvider, ConsoleSpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base'; +import { Client, Connection } from '@temporalio/client'; +import { NativeConnection, Worker, bundleWorkflowCode } from '@temporalio/worker'; import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; +const resource = new Resource({ 'service.name': 'my-worker' }); +const spanProcessor = new SimpleSpanProcessor(new ConsoleSpanExporter()); // swap in your own exporter + +const provider = new BasicTracerProvider({ resource }); +provider.addSpanProcessor(spanProcessor); +provider.register(); + const plugin = new OpenTelemetryPlugin({ resource, spanProcessor }); -const bundle = await bundleWorkflowCode({ workflowsPath, plugins: [plugin] }); -const worker = await Worker.create({ connection, taskQueue, workflowBundle: bundle, plugins: [plugin] }); -// const client = new Client({ connection, plugins: [plugin] }); +// In your worker process: +const bundle = await bundleWorkflowCode({ + workflowsPath: require.resolve('./workflows'), + plugins: [plugin], +}); +const worker = await Worker.create({ + connection: await NativeConnection.connect(), + taskQueue: 'my-task-queue', + workflowBundle: bundle, + plugins: [plugin], +}); +await worker.run(); + +// In your client process, pass the same plugin: +const client = new Client({ + connection: await Connection.connect(), + plugins: [plugin], +}); ``` For the full setup and options, see `references/typescript/integrations/opentelemetry.md`. From 3aa2751481fdfe25c3c30dd6473025ac463cc71c Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 11:05:14 -0400 Subject: [PATCH 11/16] cut correlation --- references/typescript/integrations/opentelemetry.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/references/typescript/integrations/opentelemetry.md b/references/typescript/integrations/opentelemetry.md index bad85e3..421d87a 100644 --- a/references/typescript/integrations/opentelemetry.md +++ b/references/typescript/integrations/opentelemetry.md @@ -70,10 +70,6 @@ The SDK uses the global OpenTelemetry propagator (default: W3C TraceContext + Ba Standalone Activities (`references/typescript/standalone-activities.md`) are traced automatically — the same plugin registration extracts the parent span context whether the Activity was scheduled by a Workflow or directly by a Client. -## Log and metric correlation - -When a valid OTel span context is active during an Activity or Workflow call, the plugin merges `trace_id`, `span_id`, and `trace_flags` into `getLogAttributes` (used by `log.*`) and `getMetricTags` (used by worker metric tags), so traces, logs, and metrics share correlation IDs. - ## Common mistakes - **Passing only `resource` or only `spanProcessor`.** Both are required; `new OpenTelemetryPlugin()` with no argument throws. From 132faaa398462fcac36fa7050f54ea241f52f34a Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 12:03:53 -0400 Subject: [PATCH 12/16] Move TypeScript OpenTelemetry docs to a separate PR The TypeScript material needs more work, so split it out (now on branch feat/ts-otel). This leaves PR #243 scoped to the Python OpenTelemetry integration only: removes the TS integration reference, reverts the TS observability tracing section, and drops the TS row from the catalog. Co-Authored-By: Claude Opus 4.8 (1M context) --- references/integrations.md | 1 - .../typescript/integrations/opentelemetry.md | 82 ------------------- references/typescript/observability.md | 52 +----------- 3 files changed, 1 insertion(+), 134 deletions(-) delete mode 100644 references/typescript/integrations/opentelemetry.md diff --git a/references/integrations.md b/references/integrations.md index 8222db0..7b543a4 100644 --- a/references/integrations.md +++ b/references/integrations.md @@ -20,4 +20,3 @@ Temporal ships and supports a growing set of integrations with third-party frame | LangGraph (`temporalio.contrib.langgraph`, Pre-release) | Python | Runs LangGraph Graph-API and Functional-API code as Temporal Workflows - nodes/tasks can execute as either in-workflow or as Activities | `references/python/integrations/langgraph.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | Google ADK (`temporalio[google-adk]`) | Python | Durable Google ADK agents: model calls run through `TemporalModel`-wrapped Activities, tools via `activity_tool`, MCP toolsets via `TemporalMcpToolSet` | `references/python/integrations/google-adk.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing for Temporal apps with OpenTelemetry | `references/python/integrations/opentelemetry.md` | `references/python/observability.md`, `references/python/standalone-activities.md` | -| OpenTelemetry (`@temporalio/interceptors-opentelemetry`) | TypeScript | Distributed tracing for Temporal apps with OpenTelemetry | `references/typescript/integrations/opentelemetry.md` | `references/typescript/observability.md`, `references/typescript/standalone-activities.md` | diff --git a/references/typescript/integrations/opentelemetry.md b/references/typescript/integrations/opentelemetry.md deleted file mode 100644 index 421d87a..0000000 --- a/references/typescript/integrations/opentelemetry.md +++ /dev/null @@ -1,82 +0,0 @@ -# Temporal OpenTelemetry Integration (TypeScript) - -## Overview - -`@temporalio/interceptors-opentelemetry` wires OpenTelemetry tracing into Temporal through the `OpenTelemetryPlugin`. It traces Workflow Executions, Child Workflows, Activity invocations, and Client `start`/`signal` calls, propagating W3C TraceContext + Baggage across all of them. - -Workflow-side spans are emitted out of the Workflow isolate through an injected Sink that hands serialized spans to a host-side `SpanProcessor`. - -For non-OTel observability (metrics, runtime logger, sinks) read `references/typescript/observability.md`. - -> [!NOTE] -> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. - -## Install - -Install `@temporalio/interceptors-opentelemetry` plus the OpenTelemetry peer packages you use — typically `@opentelemetry/api`, `@opentelemetry/sdk-trace-base`, `@opentelemetry/resources`, and an exporter (e.g. `@opentelemetry/exporter-trace-otlp-grpc`). - -## `OpenTelemetryPlugin` - -Construct one `OpenTelemetryPlugin` and pass it to the Client, `bundleWorkflowCode`, and `Worker.create`. It must reach `bundleWorkflowCode` so the Workflow-side interceptors are included in the bundle. Lifecycle spans (workflow / activity / client) are then created automatically. - -```ts -import { Resource } from '@opentelemetry/resources'; -import { BasicTracerProvider, BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; -import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; -import { NativeConnection, Worker, bundleWorkflowCode } from '@temporalio/worker'; -import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; - -const resource = new Resource({ 'service.name': 'orders-worker' }); -const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter()); - -const provider = new BasicTracerProvider({ resource }); -provider.addSpanProcessor(spanProcessor); -provider.register(); - -// `resource` and `spanProcessor` are required; pass an optional `tracer` to override -// the tracer used by the Client/Activity interceptors. -const plugin = new OpenTelemetryPlugin({ resource, spanProcessor }); - -const bundle = await bundleWorkflowCode({ - workflowsPath: require.resolve('./workflows'), - plugins: [plugin], -}); - -const connection = await NativeConnection.connect(); -const worker = await Worker.create({ - connection, - taskQueue: 'orders', - workflowBundle: bundle, - activities: { /* ... */ }, - plugins: [plugin], -}); -await worker.run(); -``` - -Pass the same plugin to the Client so client-side `start` and `signal` calls are traced: - -```ts -import { Client, Connection } from '@temporalio/client'; - -const client = new Client({ - connection: await Connection.connect(), - plugins: [plugin], -}); -``` - -The SDK uses the global OpenTelemetry propagator (default: W3C TraceContext + Baggage). To use a non-default propagator (e.g. Jaeger), call `propagation.setGlobalPropagator(...)` at the top level of your Workflow code BEFORE the Worker bundles it. - -## Standalone Activities - -Standalone Activities (`references/typescript/standalone-activities.md`) are traced automatically — the same plugin registration extracts the parent span context whether the Activity was scheduled by a Workflow or directly by a Client. - -## Common mistakes - -- **Passing only `resource` or only `spanProcessor`.** Both are required; `new OpenTelemetryPlugin()` with no argument throws. -- **Passing the plugin to `Worker.create` but not `bundleWorkflowCode`.** Workflow-side interceptors must be in the bundle. -- **Installing `@temporalio/opentelemetry`.** The package is `@temporalio/interceptors-opentelemetry`. -- **Expecting a non-default propagator (e.g. Jaeger) to work without setting the global propagator before `bundleWorkflowCode` runs.** - -## Resources - -- SDK metrics / observability reference: `references/typescript/observability.md` diff --git a/references/typescript/observability.md b/references/typescript/observability.md index 7f86642..211fbc6 100644 --- a/references/typescript/observability.md +++ b/references/typescript/observability.md @@ -2,9 +2,7 @@ ## Overview -The TypeScript SDK provides replay-aware logging, metrics, and distributed tracing (OpenTelemetry) for production observability. - -These pillars are complementary: **logging** (below) captures discrete events, **metrics** capture aggregate worker health, **tracing** stitches a single request across Client/Workflow/Activity boundaries, and **Search Attributes** make executions queryable. +The TypeScript SDK provides replay-aware logging, metrics, and integrations for production observability. ## Replay-Aware Logging @@ -102,53 +100,6 @@ Runtime.install({ }); ``` -## Distributed Tracing (OpenTelemetry) - -> [!NOTE] -> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. - -OpenTelemetry is the supported way to add distributed tracing to Temporal applications. The `OpenTelemetryPlugin` (from `@temporalio/interceptors-opentelemetry`) traces Workflow Executions, Child Workflows, Activity invocations, and Client `start`/`signal` calls, propagating W3C TraceContext + Baggage across all of them. Workflow-side spans are exported out of the Workflow isolate through an injected Sink. - -Construct one plugin instance and pass it to the Client, to `bundleWorkflowCode`, and to `Worker.create` (the Workflow-side interceptors must be in the bundle): - -```typescript -import { Resource } from '@opentelemetry/resources'; -import { BasicTracerProvider, ConsoleSpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base'; -import { Client, Connection } from '@temporalio/client'; -import { NativeConnection, Worker, bundleWorkflowCode } from '@temporalio/worker'; -import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry'; - -const resource = new Resource({ 'service.name': 'my-worker' }); -const spanProcessor = new SimpleSpanProcessor(new ConsoleSpanExporter()); // swap in your own exporter - -const provider = new BasicTracerProvider({ resource }); -provider.addSpanProcessor(spanProcessor); -provider.register(); - -const plugin = new OpenTelemetryPlugin({ resource, spanProcessor }); - -// In your worker process: -const bundle = await bundleWorkflowCode({ - workflowsPath: require.resolve('./workflows'), - plugins: [plugin], -}); -const worker = await Worker.create({ - connection: await NativeConnection.connect(), - taskQueue: 'my-task-queue', - workflowBundle: bundle, - plugins: [plugin], -}); -await worker.run(); - -// In your client process, pass the same plugin: -const client = new Client({ - connection: await Connection.connect(), - plugins: [plugin], -}); -``` - -For the full setup and options, see `references/typescript/integrations/opentelemetry.md`. - ## Search Attributes (Visibility) See the Search Attributes section of `references/typescript/data-handling.md` @@ -160,4 +111,3 @@ See the Search Attributes section of `references/typescript/data-handling.md` 3. Configure Winston or similar for production log aggregation 4. Monitor Prometheus metrics for worker health 5. Use Event History for debugging workflow issues -6. Use the `OpenTelemetryPlugin` for distributed tracing across Client/Workflow/Activity boundaries. From fc69e391a456d9986cf60133db32624998ceb2ec Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 12:18:40 -0400 Subject: [PATCH 13/16] Apply suggestions from code review Co-authored-by: Donald Pinckney --- references/python/observability.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/references/python/observability.md b/references/python/observability.md index 61acc13..6714c85 100644 --- a/references/python/observability.md +++ b/references/python/observability.md @@ -96,9 +96,6 @@ Runtime.set_default(runtime, error_if_already_set=True) - `temporal_activity_execution_latency` - Activity execution time - `temporal_workflow_task_replay_latency` - Replay duration -> [!TIP] -> For span-level timing that complements these aggregate metrics, enable Temporal spans on the tracing plugin (`OpenTelemetryPlugin(add_temporal_spans=True)`) — see [Distributed Tracing](#distributed-tracing-opentelemetry). - ## Distributed Tracing (OpenTelemetry) > [!NOTE] From ed96a5e4275e6de1b19a98428702cc58b0c69e5e Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 16:08:36 -0400 Subject: [PATCH 14/16] Apply suggestion from @donald-pinckney --- references/python/integrations/opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/references/python/integrations/opentelemetry.md b/references/python/integrations/opentelemetry.md index 5d496fd..4048992 100644 --- a/references/python/integrations/opentelemetry.md +++ b/references/python/integrations/opentelemetry.md @@ -67,7 +67,7 @@ class MyWorkflow: ## Common mistakes - **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit. -- **Calling `Client.connect` before `opentelemetry.trace.set_tracer_provider(provider)`.** With `OpenTelemetryPlugin`, the workflow interceptor factory raises `ValueError("When using OpenTelemetryPlugin, the global trace provider must be a ReplaySafeTracerProvider. Use init_tracer_provider to create one.")`. +- **Calling `Client.connect` before `opentelemetry.trace.set_tracer_provider(provider)`.**`OpenTelemetryPlugin`, will raise an exception if you do this - **Building a plain `opentelemetry.sdk.trace.TracerProvider` and passing it to `set_tracer_provider`.** `OpenTelemetryPlugin` requires `ReplaySafeTracerProvider` — build it via `create_tracer_provider(...)`. - **Adding `with_passthrough_modules("opentelemetry")` to a `SandboxedWorkflowRunner` manually.** The plugin already does this. From aa90bd98b4e280377d93d0d60d8f7596fd657d3c Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 16:31:39 -0400 Subject: [PATCH 15/16] Apply suggestion from @donald-pinckney --- references/python/integrations/opentelemetry.md | 1 - 1 file changed, 1 deletion(-) diff --git a/references/python/integrations/opentelemetry.md b/references/python/integrations/opentelemetry.md index 4048992..982e75c 100644 --- a/references/python/integrations/opentelemetry.md +++ b/references/python/integrations/opentelemetry.md @@ -69,7 +69,6 @@ class MyWorkflow: - **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit. - **Calling `Client.connect` before `opentelemetry.trace.set_tracer_provider(provider)`.**`OpenTelemetryPlugin`, will raise an exception if you do this - **Building a plain `opentelemetry.sdk.trace.TracerProvider` and passing it to `set_tracer_provider`.** `OpenTelemetryPlugin` requires `ReplaySafeTracerProvider` — build it via `create_tracer_provider(...)`. -- **Adding `with_passthrough_modules("opentelemetry")` to a `SandboxedWorkflowRunner` manually.** The plugin already does this. ## Resources From 7506681584f14e34edc9eb1917d49252b097e0a9 Mon Sep 17 00:00:00 2001 From: Donald Pinckney Date: Fri, 5 Jun 2026 16:39:36 -0400 Subject: [PATCH 16/16] Consolidate Python OpenTelemetry docs into observability Remove the standalone references/python/integrations/opentelemetry.md file and fold its unique content (Common mistakes, workflow custom-span example) into the Distributed Tracing section of observability.md. Repoint the integrations catalog row at the observability section. Co-Authored-By: Claude Opus 4.8 (1M context) --- references/integrations.md | 2 +- .../python/integrations/opentelemetry.md | 75 ------------------- references/python/observability.md | 23 +++++- 3 files changed, 23 insertions(+), 77 deletions(-) delete mode 100644 references/python/integrations/opentelemetry.md diff --git a/references/integrations.md b/references/integrations.md index 7b543a4..36196a8 100644 --- a/references/integrations.md +++ b/references/integrations.md @@ -19,4 +19,4 @@ Temporal ships and supports a growing set of integrations with third-party frame | LangSmith tracing (`temporalio.contrib.langsmith`) | Python | Experimental Temporal Plugin that propagates LangSmith trace context across Worker boundaries; lets `@traceable` run inside Workflows and Activities | `references/python/integrations/langsmith.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | LangGraph (`temporalio.contrib.langgraph`, Pre-release) | Python | Runs LangGraph Graph-API and Functional-API code as Temporal Workflows - nodes/tasks can execute as either in-workflow or as Activities | `references/python/integrations/langgraph.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | | Google ADK (`temporalio[google-adk]`) | Python | Durable Google ADK agents: model calls run through `TemporalModel`-wrapped Activities, tools via `activity_tool`, MCP toolsets via `TemporalMcpToolSet` | `references/python/integrations/google-adk.md` | `references/python/ai-patterns.md`, `references/core/ai-patterns.md` | -| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing for Temporal apps with OpenTelemetry | `references/python/integrations/opentelemetry.md` | `references/python/observability.md`, `references/python/standalone-activities.md` | +| OpenTelemetry (`temporalio[opentelemetry]`) | Python | Distributed tracing for Temporal apps with OpenTelemetry | `references/python/observability.md` (Distributed Tracing section) | | diff --git a/references/python/integrations/opentelemetry.md b/references/python/integrations/opentelemetry.md deleted file mode 100644 index 982e75c..0000000 --- a/references/python/integrations/opentelemetry.md +++ /dev/null @@ -1,75 +0,0 @@ -# Temporal OpenTelemetry Integration (Python) - -## Overview - -`temporalio.contrib.opentelemetry` wires OpenTelemetry tracing into Temporal through the `OpenTelemetryPlugin`. - -It propagates W3C TraceContext + W3C Baggage through Temporal headers across Client, Workflow, Activity, and Child Workflow boundaries. - -For non-OTel observability (metrics, logging) read `references/python/observability.md`. - -> [!NOTE] -> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. - -## Install the plugin - -Modify your dependency on the `temporalio` package to add the `opentelemetry` extra, e.g. `uv add temporalio[opentelemetry]` or whatever is appropriate for your package manager. - -## `OpenTelemetryPlugin` - -Build a `ReplaySafeTracerProvider` with `create_tracer_provider()`, attach span processors, set it as the global tracer provider, then connect with `plugins=[OpenTelemetryPlugin()]`. Workers created from the returned Client inherit the plugin automatically. - -```python -import opentelemetry.trace -from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor -from temporalio.client import Client -from temporalio.contrib.opentelemetry import OpenTelemetryPlugin, create_tracer_provider -from temporalio.worker import Worker - -provider = create_tracer_provider() -provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) # attach your span processors as normal for OTel -opentelemetry.trace.set_tracer_provider(provider) - -client = await Client.connect( - "localhost:7233", - # Pass add_temporal_spans=True to also emit `StartWorkflow`, `RunWorkflow`, - # `StartActivity`, `RunActivity` (and similar) spans on top of application spans. - plugins=[OpenTelemetryPlugin()], -) - -worker = Worker( - client, - task_queue="my-task-queue", - workflows=[MyWorkflow], - activities=[my_activity], -) -``` - -Then just use standard OpenTelemetry APIs in your workflow code — durations are accurate. - -```python -from datetime import timedelta -from opentelemetry.trace import get_tracer -from temporalio import workflow - -@workflow.defn -class MyWorkflow: - @workflow.run - async def run(self) -> None: - tracer = get_tracer(__name__) - with tracer.start_as_current_span("workflow-operation"): - await workflow.execute_activity( - my_activity, - start_to_close_timeout=timedelta(seconds=30), - ) -``` - -## Common mistakes - -- **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit. -- **Calling `Client.connect` before `opentelemetry.trace.set_tracer_provider(provider)`.**`OpenTelemetryPlugin`, will raise an exception if you do this -- **Building a plain `opentelemetry.sdk.trace.TracerProvider` and passing it to `set_tracer_provider`.** `OpenTelemetryPlugin` requires `ReplaySafeTracerProvider` — build it via `create_tracer_provider(...)`. - -## Resources - -- SDK metrics reference: `references/python/observability.md` diff --git a/references/python/observability.md b/references/python/observability.md index 6714c85..5ad5e18 100644 --- a/references/python/observability.md +++ b/references/python/observability.md @@ -118,7 +118,28 @@ client = await Client.connect("localhost:7233", plugins=[OpenTelemetryPlugin()]) Workers created from this Client inherit the plugin automatically. Inside a Workflow you then use standard OpenTelemetry APIs (`get_tracer(...).start_as_current_span(...)`); pass `OpenTelemetryPlugin(add_temporal_spans=True)` to also emit `StartWorkflow` / `RunWorkflow` / `StartActivity` / `RunActivity` spans automatically alongside the SDK metrics above. -For the full setup and options, see `references/python/integrations/opentelemetry.md`. +```python +from datetime import timedelta +from opentelemetry.trace import get_tracer +from temporalio import workflow + +@workflow.defn +class MyWorkflow: + @workflow.run + async def run(self) -> None: + tracer = get_tracer(__name__) + with tracer.start_as_current_span("workflow-operation"): + await workflow.execute_activity( + my_activity, + start_to_close_timeout=timedelta(seconds=30), + ) +``` + +**Common mistakes:** + +- **Registering the same plugin on both Client and Worker.** Register on the Client only; Workers inherit. +- **Calling `Client.connect` before `opentelemetry.trace.set_tracer_provider(provider)`.** `OpenTelemetryPlugin` raises an exception unless the global tracer provider is already set. +- **Building a plain `opentelemetry.sdk.trace.TracerProvider` and passing it to `set_tracer_provider`.** `OpenTelemetryPlugin` requires a `ReplaySafeTracerProvider` — build it via `create_tracer_provider(...)`. ## Search Attributes (Visibility)