[otel-advisor] OTel improvement: include otel.jsonl in agent artifact upload so the JSONL mirror fulfills its stated purpose

[github-actions]

📡 OTel Instrumentation Improvement: include otel.jsonl in agent artifact upload

Analysis Date: 2026-04-07
Priority: Medium
Effort: Small (< 2h)

Problem

send_otlp_span.cjs appends every OTLP span payload to /tmp/gh-aw/otel.jsonl for local mirroring. The file's own JSDoc comment states:

Every OTLP span payload is also appended here as a JSON line so that it can be inspected via GitHub Actions artifacts without needing a live collector.

Yet /tmp/gh-aw/otel.jsonl is never included in any artifact upload. The compiler assembles the agent artifact path list in pkg/workflow/compiler_yaml_main_job.go at line 472, where github_rate_limits.jsonl and safeoutputs.jsonl are explicitly included — but otel.jsonl is absent. This is confirmed by the golden test fixtures (pkg/workflow/testdata/TestWasmGolden_CompileFixtures/with-imports.golden:223) and all compiled .lock.yml workflows in .github/workflows/.

A DevOps engineer debugging a failed run without access to the live OTLP backend (Sentry/Grafana) cannot answer: "What trace ID was emitted? Which spans were sent? Did the collector receive them?" — because the file they would inspect is never available.

Why This Matters (DevOps Perspective)

• Offline debugging: When the OTLP collector is unreachable or unset, otel.jsonl is the only copy of span data. Today that copy is deleted with the runner.
• MTTR reduction: On-call engineers downloading the agent artifact to investigate a failure see github_rate_limits.jsonl and agent_output.json but no trace data. Adding otel.jsonl closes this gap — trace IDs, resource attributes, and error status messages are immediately visible without requiring backend access.
• Consistency with peer files: github_rate_limits.jsonl (also written to /tmp/gh-aw/ and read only in the conclusion span) is uploaded. otel.jsonl follows the same pattern but is inexplicably excluded.

Current Behavior

// actions/setup/js/send_otlp_span.cjs  (lines 160–183)
/**
 * Path to the OTLP telemetry mirror file.
 * Every OTLP span payload is also appended here as a JSON line so that it can
 * be inspected via GitHub Actions artifacts without needing a live collector.
 */
const OTEL_JSONL_PATH = "/tmp/gh-aw/otel.jsonl";

function appendToOTLPJSONL(payload) {
  try {
    fs.mkdirSync("/tmp/gh-aw", { recursive: true });
    fs.appendFileSync(OTEL_JSONL_PATH, JSON.stringify(payload) + "\n");
  } catch {
    // Mirror failures are non-fatal; do not propagate.
  }
}

// pkg/workflow/compiler_yaml_main_job.go  (lines 470–480)
// Collect GitHub API rate-limit log for observability.
artifactPaths = append(artifactPaths, "/tmp/gh-aw/"+constants.GithubRateLimitsFilename)

// otel.jsonl is NOT appended here — the file is silently discarded at runner cleanup.

if data.SafeOutputs != nil {
    artifactPaths = append(artifactPaths, "/tmp/gh-aw/"+constants.SafeOutputsFilename)
    artifactPaths = append(artifactPaths, "/tmp/gh-aw/"+constants.AgentOutputFilename)
}

Proposed Change

Step 1 — add a constant (pkg/constants/job_constants.go):

// OtelJsonlFilename is the filename of the OTLP span mirror written to /tmp/gh-aw/
// by send_otlp_span.cjs. Each line is a full OTLP/HTTP JSON traces payload.
// Included in the agent artifact so spans are available without a live collector.
const OtelJsonlFilename = "otel.jsonl"

Step 2 — include it in the artifact upload (pkg/workflow/compiler_yaml_main_job.go, alongside GithubRateLimitsFilename):

// Collect GitHub API rate-limit log for observability.
artifactPaths = append(artifactPaths, "/tmp/gh-aw/"+constants.GithubRateLimitsFilename)

// Collect OTLP span mirror — enables post-hoc trace debugging without a live collector.
artifactPaths = append(artifactPaths, "/tmp/gh-aw/"+constants.OtelJsonlFilename)

Step 3 — update the CLI artifact set (pkg/cli/logs_artifact_set.go): note in the ArtifactSetAgent description that otel.jsonl is now included (no code change needed unless the set has an explicit path list).

Step 4 — update golden fixtures: recompile or manually add the new path to pkg/workflow/testdata/TestWasmGolden_CompileFixtures/*.golden artifacts.

Secondary improvement (separate PR): appendToOTLPJSONL at send_otlp_span.cjs:318 writes the unsanitized payload to disk, while sendOTLPSpan sends the sanitized version to the collector. Once the file is uploaded, those differences are visible. Consider sanitizing before writing:

const sanitized = sanitizeOTLPPayload(payload);
appendToOTLPJSONL(sanitized);   // consistent with what the backend received
// ...
body: JSON.stringify(sanitized),

Expected Outcome

After this change:

• In the GitHub Actions artifact browser: the agent artifact will contain otel.jsonl alongside github_rate_limits.jsonl, agent_output.json, etc.
• In the JSONL mirror: the same full OTLP payloads already written — now actually accessible post-run.
• For on-call engineers: gh run download <run-id> -n agent pulls the trace data; jq '.resourceSpans[].scopeSpans[].spans[] | {name, traceId, status}' otel.jsonl gives an instant trace summary without backend access.
• In Grafana / Datadog: no change — the collector path is unaffected.

Implementation Steps

• Add OtelJsonlFilename = "otel.jsonl" to pkg/constants/job_constants.go
• Add artifactPaths = append(artifactPaths, "/tmp/gh-aw/"+constants.OtelJsonlFilename) in pkg/workflow/compiler_yaml_main_job.go near line 472 (alongside GithubRateLimitsFilename)
• Update golden test fixtures in pkg/workflow/testdata/TestWasmGolden_CompileFixtures/ to include the new path
• Run make test-unit (or go test ./pkg/workflow/...) to confirm golden tests pass
• Run make fmt to ensure formatting
• Recompile affected workflows (gh aw compile) and verify otel.jsonl appears in their Upload agent artifacts step
• Open a PR referencing this issue

Evidence from Live Sentry Data

No Sentry MCP tools are available in this environment — live span data could not be sampled. The gap is confirmed statically by cross-referencing:

• send_otlp_span.cjs:163 (stated design intent: "inspected via GitHub Actions artifacts")
• pkg/workflow/compiler_yaml_main_job.go:472 (missing otel.jsonl from artifact path list)
• pkg/workflow/testdata/TestWasmGolden_CompileFixtures/with-imports.golden:223 (golden fixture shows github_rate_limits.jsonl present, otel.jsonl absent)
• All .github/workflows/*.lock.yml upload-artifact steps (none include otel.jsonl)

Related Files

• actions/setup/js/send_otlp_span.cjs (writes the file; comment states it should be artifact-accessible)
• pkg/workflow/compiler_yaml_main_job.go (assembles artifact paths — fix goes here)
• pkg/constants/job_constants.go (add OtelJsonlFilename constant here)
• pkg/cli/logs_artifact_set.go (documents artifact contents — update description)
• pkg/workflow/testdata/TestWasmGolden_CompileFixtures/*.golden (update after fix)

---

Generated by the Daily OTel Instrumentation Advisor workflow

Generated by Daily OTel Instrumentation Advisor · ● 410.6K · ◷

• expires on Apr 14, 2026, 9:31 PM UTC