Distributed tracing - Does ADK api server respect the traceparent header?

[markns]

Hi,

I'm trying to get distributed tracing working between the ADK api server and a client application of the api server.

The client application is sending the traceparent header, and it is being received in the run_sse endpoint of the api server. I know this is working, because I have modified the adk_web_server.py code to print the fastapi headers and I see the traceparent header.

I have enabled OTel data to the Google Cloud Observability services via the adk api_server --otel_to_cloud flag.
I also have these env vars set:

OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=true
OTEL_PROPAGATORS=tracecontext,baggage

The traces from both the client application and the adk api server are being received in Google Cloud Tracing, but unfortunately the Trace IDs are different between the client and server services, so the waterfall charts are not complete.

Does ADK api server not respect the traceparent header? Or am I missing some configuration to make this work?

[markns]

Seems to be a problem with the propagation on the ADK api server side.

With the following code hacked into adk_web_server.py

    @app.post("/run_sse")
    async def run_agent_sse(req: RunAgentRequest, request: Request) -> StreamingResponse:
      # SSE endpoint
      print(request.headers)
      print(trace.get_current_span())
      print("**********")

I get the following output:

Headers({'host': 'localhost:8001', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.28.1', 'content-length': '145', 'content-type': 'application/json', 'traceparent': '00-122e1b6c2af85fd32b43363017d01d1b-27c351f9413c5cca-01'})
NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))
**********

If I'm not mistaken, the fact that the traceparent is there should mean that the current span should have the same trace_id