ci: upload system-tests results to Test Optimization and add tracer-release nightly by nccatoni · Pull Request #3660 · DataDog/dd-trace-php

nccatoni · 2026-02-18T14:16:02Z

Description

Two changes to the system-tests CI integration:

Upload all system-tests results to Test Optimization — adds an after_script to the .system_tests base job that copies JUnit XML files and uploads them via the existing silent-upload-junit-to-datadog.sh script.
Add tracer-release nightly — create a new system-tests workflow that uses the official system-tests reusable workflow to run the tracer-release scenario group on master.

This is meant to replace the system-tests-dashboard nightly.

Reviewer checklist

This is a CI-only change, no code changes.
Appropriate labels assigned.

Run the official system-tests reusable workflow daily at 4 AM GMT using dev mode against the latest commit on the default branch. Co-authored-by: Cursor <cursoragent@cursor.com>

datadog-official · 2026-02-18T14:22:44Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 1028 Tests failed

testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integration\PHPInstallerTest::testSearchPhpBinaries
Test code or tested code printed unexpected output: Searching for available php binaries, this operation might take a while.
testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Datadog) (Fix with Cursor)
Risky Test
phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:97
testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69b2fc1300000000fab19443c394ded9
tid: 69b2fc1300000000
hexProcessTraceId: fab19443c394ded9
hexProcessSpanId: 3a2e67a3b2b06c55
processTraceId: 18064382599124803289
processSpanId: 4192402255903747157
View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 85c13ba | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

Co-authored-by: Cursor <cursoragent@cursor.com>

codecov-commenter · 2026-02-18T14:35:08Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.21%. Comparing base (a9caaa0) to head (85c13ba).
⚠️ Report is 31 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3660   +/-   ##
=======================================
  Coverage   62.21%   62.21%           
=======================================
  Files         141      141           
  Lines       13387    13387           
  Branches     1753     1753           
=======================================
  Hits         8329     8329           
  Misses       4260     4260           
  Partials      798      798

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a9caaa0...85c13ba. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-authored-by: Cursor <cursoragent@cursor.com>

- Add after_script to .system_tests base job to upload JUnit results to Datadog Test Optimization using the existing upload script - Add tracer-release scenario group to configure_system_tests on schedule and master runs - Remove separate GitHub Actions nightly workflow Made-with: Cursor

The silent wrapper's grep returns exit code 1 when no summary lines are found, causing the after_script to fail. Use the non-silent version with || true to prevent after_script failures. Made-with: Cursor

Replace the generic upload-junit-to-datadog.sh with a custom after_script that: - Uses --service system-tests (not dd-trace-php-tests) - References logs directly from system-tests/logs*/ - Uses the datadog-ci standalone binary (no npm needed) - Uses Python for JSON/zip (guaranteed available in the image) - Provides explicit error messages instead of silent exit 0 Made-with: Cursor

pr-commenter · 2026-02-26T17:25:52Z

Benchmarks [ tracer ]

Benchmark execution time: 2026-03-12 18:55:43

Comparing candidate commit 85c13ba in PR branch add-nightly-system-tests with baseline commit a9caaa0 in branch master.

Found 2 performance improvements and 0 performance regressions! Performance is the same for 190 metrics, 2 unstable metrics.

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

🟩 execution_time [-7.495µs; -5.665µs] or [-6.712%; -5.073%]

scenario:TraceSerializationBench/benchSerializeTrace-opcache

🟩 execution_time [-20.434µs; -9.266µs] or [-5.819%; -2.639%]

Made-with: Cursor

nccatoni · 2026-03-05T12:06:47Z

For this to work two things need to be done:
Adding docker/login-action and nick-fields/retry to the list of authorized actions for the repo
Adding an Datadog API key as a github repo secret

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a65519dc59

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.github/workflows/system_tests.yml

.gitlab/generate-package.php

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b44c4ab3eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-12T14:42:00Z

.gitlab/generate-package.php

+
+      # Fetch API key from Vault
+      echo "Fetching API key from Vault..."
+      VAULT_JSON=$(/tmp/vault kv get --format=json "kv/k8s/gitlab-runner/dd-trace-php/datadoghq-api-key" 2>&1)


Keep Vault stderr out of parsed JSON

Capturing Vault output with 2>&1 mixes diagnostics into the JSON payload, so any warning emitted on stderr (for example token/lease notices) makes the subsequent JSON parse fail and leaves DATADOG_API_KEY unset. In that case the JUnit upload step runs without valid credentials and results are dropped, even though the secret fetch itself may have succeeded; read only stdout for JSON and handle stderr separately.

Useful? React with 👍 / 👎.

Leiyks · 2026-03-12T14:52:28Z

.gitlab/generate-package.php

+  after_script:
+    - |
+      set +e
+      echo "=== Uploading system-tests JUnit results to Test Optimization ==="
+
+      # Check that there are JUnit files to upload
+      ls system-tests/logs*/reportJunit.xml >/dev/null 2>&1
+      if [ $? -ne 0 ]; then echo "No JUnit XML files found, skipping upload"; exit 0; fi
+
+      # Download datadog-ci standalone binary
+      echo "Downloading datadog-ci..."
+      curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_linux-x64" -o /tmp/datadog-ci
+      if [ $? -ne 0 ]; then echo "Failed to download datadog-ci"; exit 0; fi
+      chmod +x /tmp/datadog-ci
+
+      # Download and extract Vault CLI
+      echo "Downloading Vault..."
+      VAULT_VERSION="1.20.0"
+      curl -L --fail "https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip" -o /tmp/vault.zip
+      if [ $? -ne 0 ]; then echo "Failed to download Vault"; exit 0; fi
+      python3 -c "import zipfile; zipfile.ZipFile('/tmp/vault.zip').extractall('/tmp/')"
+      chmod +x /tmp/vault
+
+      # Fetch API key from Vault
+      echo "Fetching API key from Vault..."
+      VAULT_JSON=$(/tmp/vault kv get --format=json "kv/k8s/gitlab-runner/dd-trace-php/datadoghq-api-key" 2>&1)
+      if [ $? -ne 0 ]; then echo "Failed to fetch API key from Vault: $VAULT_JSON"; exit 0; fi
+      export DATADOG_API_KEY=$(echo "$VAULT_JSON" | python3 -c "import sys,json; print(json.loads(sys.stdin.read())['data']['data']['key'])")
+      export DATADOG_SITE="datadoghq.com"
+
+      # Upload JUnit results
+      echo "Uploading JUnit results..."
+      /tmp/datadog-ci junit upload \
+        system-tests/logs*/reportJunit.xml \
+        --service system-tests \
+        --env ci \
+        --verbose \
+        --xpath-tag "test.codeowners=/testcase/properties/property[@name='test.codeowners']"
+      echo "=== Upload complete ==="


We have a script that basically do all of this in .gitlab/upload-junit-to-datadog.sh

You may need to tweak it a bit to add the xpath-tag thing tho

Add nightly system-tests workflow

17ff796

Run the official system-tests reusable workflow daily at 4 AM GMT using dev mode against the latest commit on the default branch. Co-authored-by: Cursor <cursoragent@cursor.com>

Enable push_to_test_optimization

10f7eab

Co-authored-by: Cursor <cursoragent@cursor.com>

nccatoni changed the title ~~Add nightly system-tests workflow~~ ci: add nightly system-tests workflow Feb 18, 2026

nccatoni and others added 6 commits February 18, 2026 15:40

Pass secrets explicitly to reusable workflow

2f73f4b

Co-authored-by: Cursor <cursoragent@cursor.com>

Only pass TEST_OPTIMIZATION_API_KEY secret

34d46d2

Co-authored-by: Cursor <cursoragent@cursor.com>

tmp: add pull_request trigger for testing

420b281

Co-authored-by: Cursor <cursoragent@cursor.com>

Set desired_execution_time to 300s

1185a36

Co-authored-by: Cursor <cursoragent@cursor.com>

Also pass DD_API_KEY secret

4af9b76

Co-authored-by: Cursor <cursoragent@cursor.com>

nccatoni changed the title ~~ci: add nightly system-tests workflow~~ ci: upload system-tests results to Test Optimization and add tracer-release nightly Feb 26, 2026

nccatoni added 2 commits February 26, 2026 16:54

ci: fix after_script by using upload script directly

42bcd97

The silent wrapper's grep returns exit code 1 when no summary lines are found, causing the after_script to fail. Use the non-silent version with || true to prevent after_script failures. Made-with: Cursor

nccatoni added 3 commits March 2, 2026 14:20

Run only tracer-release on schedule, it already includes other groups

34f79e3

Made-with: Cursor

Temporarily run tracer-release on all triggers for testing

bafec28

Made-with: Cursor

Tracer release scenario group on main

578e0b6

nccatoni force-pushed the add-nightly-system-tests branch from 9a812c9 to 578e0b6 Compare March 5, 2026 11:08

nccatoni added 2 commits March 5, 2026 12:11

Test

4db0655

Remove ref argument

a65519d

nccatoni marked this pull request as ready for review March 5, 2026 12:06

nccatoni requested a review from a team as a code owner March 5, 2026 12:06

chatgpt-codex-connector bot reviewed Mar 5, 2026

View reviewed changes

.github/workflows/system_tests.yml Outdated Show resolved Hide resolved

.gitlab/generate-package.php Outdated Show resolved Hide resolved

nccatoni marked this pull request as draft March 9, 2026 10:15

Every day schedule (include weekends)

b44c4ab

nccatoni marked this pull request as ready for review March 12, 2026 14:37

chatgpt-codex-connector bot reviewed Mar 12, 2026

View reviewed changes

Leiyks reviewed Mar 12, 2026

View reviewed changes

nccatoni added 2 commits March 12, 2026 17:52

Use upload-junit-to-datadog.sh

5159a71

Remove github workflow

85c13ba

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: upload system-tests results to Test Optimization and add tracer-release nightly#3660

ci: upload system-tests results to Test Optimization and add tracer-release nightly#3660
nccatoni wants to merge 18 commits intomasterfrom
add-nightly-system-tests

nccatoni commented Feb 18, 2026 •

edited

Loading

Uh oh!

datadog-official bot commented Feb 18, 2026 •

edited by datadog-prod-us1-3 bot

Loading

Uh oh!

codecov-commenter commented Feb 18, 2026 •

edited

Loading

Uh oh!

pr-commenter bot commented Feb 26, 2026 •

edited

Loading

Uh oh!

nccatoni commented Mar 5, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Mar 12, 2026

Uh oh!

Leiyks Mar 12, 2026

Uh oh!

Leiyks Mar 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nccatoni commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Reviewer checklist

Uh oh!

datadog-official bot commented Feb 18, 2026 • edited by datadog-prod-us1-3 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

codecov-commenter commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [ tracer ]

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

scenario:TraceSerializationBench/benchSerializeTrace-opcache

Uh oh!

nccatoni commented Mar 5, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Leiyks Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Leiyks Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nccatoni commented Feb 18, 2026 •

edited

Loading

datadog-official bot commented Feb 18, 2026 •

edited by datadog-prod-us1-3 bot

Loading

codecov-commenter commented Feb 18, 2026 •

edited

Loading

pr-commenter bot commented Feb 26, 2026 •

edited

Loading