Skip to content

Add helper rewrite in rust#3581

Open
cataphract wants to merge 1 commit intomasterfrom
glopes/helper-rust
Open

Add helper rewrite in rust#3581
cataphract wants to merge 1 commit intomasterfrom
glopes/helper-rust

Conversation

@cataphract
Copy link
Contributor

@cataphract cataphract commented Jan 16, 2026

Description

Passing integration and system-tests.

Further integration into sidecar and protocol changes pending.

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

@cataphract cataphract requested review from a team as code owners January 16, 2026 16:47
@cataphract cataphract marked this pull request as draft January 16, 2026 16:47
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Jan 16, 2026

⚠️ Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 1028 Tests failed

testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integration\PHPInstallerTest::testSearchPhpBinaries
Test code or tested code printed unexpected output: Searching for available php binaries, this operation might take a while.
testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69b44374000000008730e98ca452f657
tid: 69b4437400000000
hexProcessTraceId: 8730e98ca452f657
hexProcessSpanId: 191c0eae453e9c1a
processTraceId: 9741542784263976535
processSpanId: 1809337291944926234

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106
testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69b4439500000000d4476abb67a30a20
tid: 69b4439500000000
hexProcessTraceId: d4476abb67a30a20
hexProcessSpanId: 7fb4f4546a74dd5e
processTraceId: 15296312012517345824
processSpanId: 9202248582025239902
View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f71ed6b | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@codecov-commenter
Copy link

codecov-commenter commented Jan 16, 2026

Codecov Report

❌ Patch coverage is 84.80088% with 832 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.72%. Comparing base (147f3c2) to head (f71ed6b).

Files with missing lines Patch % Lines
appsec/helper-rust/src/client.rs 89.30% 101 Missing ⚠️
appsec/helper-rust/src/lib.rs 53.47% 87 Missing ⚠️
appsec/helper-rust/src/rc.rs 85.71% 68 Missing ⚠️
appsec/src/extension/ddappsec.c 25.31% 52 Missing and 7 partials ⚠️
appsec/helper-rust/src/service.rs 90.95% 57 Missing ⚠️
appsec/helper-rust/src/telemetry/sidecar.rs 79.10% 56 Missing ⚠️
appsec/src/extension/helper_process.c 8.19% 56 Missing ⚠️
appsec/helper-rust/src/lock.rs 68.37% 37 Missing ⚠️
appsec/helper-rust/src/service/sampler.rs 90.76% 34 Missing ⚠️
appsec/helper-rust/src/service/waf_diag.rs 88.38% 28 Missing ⚠️
... and 19 more

❌ Your patch status has failed because the patch coverage (84.80%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3581      +/-   ##
==========================================
+ Coverage   62.32%   68.72%   +6.40%     
==========================================
  Files         142      166      +24     
  Lines       13586    19030    +5444     
  Branches     1775     1797      +22     
==========================================
+ Hits         8467    13079    +4612     
- Misses       4311     5136     +825     
- Partials      808      815       +7     
Flag Coverage Δ
helper-rust-integration 78.82% <78.82%> (?)
helper-rust-unit 49.36% <49.36%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
appsec/helper-rust/src/service/metrics.rs 100.00% <100.00%> (ø)
appsec/src/extension/configuration.h 100.00% <ø> (ø)
appsec/src/extension/user_tracking.c 75.00% <100.00%> (+0.08%) ⬆️
appsec/src/helper/client.cpp 75.46% <100.00%> (+0.05%) ⬆️
appsec/src/helper/client.hpp 89.58% <ø> (ø)
appsec/src/helper/engine.cpp 91.73% <100.00%> (ø)
appsec/src/helper/engine.hpp 100.00% <ø> (ø)
appsec/src/helper/network/proto.hpp 93.22% <ø> (ø)
appsec/src/helper/subscriber/base.hpp 100.00% <ø> (ø)
appsec/src/helper/subscriber/waf.cpp 71.57% <100.00%> (+0.04%) ⬆️
... and 30 more

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 147f3c2...f71ed6b. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pr-commenter
Copy link

pr-commenter bot commented Jan 16, 2026

Benchmarks [ appsec ]

Benchmark execution time: 2026-03-13 17:35:25

Comparing candidate commit f71ed6b in PR branch glopes/helper-rust with baseline commit 147f3c2 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

@pr-commenter
Copy link

pr-commenter bot commented Jan 16, 2026

Benchmarks [ tracer ]

Benchmark execution time: 2026-03-04 12:25:07

Comparing candidate commit e162af9 in PR branch glopes/helper-rust with baseline commit 8cd0131 in branch master.

Found 2 performance improvements and 0 performance regressions! Performance is the same for 190 metrics, 2 unstable metrics.

scenario:MessagePackSerializationBench/benchMessagePackSerialization

  • 🟩 execution_time [-6.504µs; -5.736µs] or [-5.806%; -5.120%]

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

  • 🟩 execution_time [-3.486µs; -2.554µs] or [-3.201%; -2.345%]

@cataphract cataphract force-pushed the glopes/helper-rust branch 9 times, most recently from fb4432d to 8d1029e Compare January 18, 2026 03:49
@morrisonlevi
Copy link
Collaborator

morrisonlevi commented Jan 18, 2026

This PR is so large that github will only permit me to review it one file at a time. I didn't even know that was a thing! You're going to need to break it down into a series of smaller PRs, probably.

@cataphract cataphract force-pushed the glopes/helper-rust branch 2 times, most recently from a218cd6 to 332fd93 Compare January 20, 2026 12:39
@bwoebi
Copy link
Collaborator

bwoebi commented Jan 20, 2026

@morrisonlevi I've had success for very big PRs with the PHPStorm/CLion github integrations in the past. Doesn't matter for small PRs, but can definitely recommend it for extra-large PRs :-)

@cataphract cataphract changed the base branch from glopes/appsec-curl to master January 29, 2026 12:58
@cataphract cataphract force-pushed the glopes/helper-rust branch 2 times, most recently from 5d247e2 to 57bcc57 Compare February 10, 2026 17:49
@cataphract cataphract force-pushed the glopes/helper-rust branch 2 times, most recently from d4571c9 to dd440cb Compare February 20, 2026 16:03
@cataphract cataphract force-pushed the glopes/helper-rust branch 5 times, most recently from 0385bb7 to d06640c Compare February 27, 2026 17:34
@cataphract cataphract marked this pull request as ready for review March 4, 2026 14:29
@@ -0,0 +1,28 @@
services:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this "installation" folder a bit dealigned with the rest of example. Is it require?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's there to manually test whether the packages generated from CI are installable, since what's on CI is basic and can never test fully against staging. Whether it should be here or on another repository, it's open for discussion

@@ -0,0 +1,3 @@
int test_add(int a, int b) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's used in the test test_sidecar_symbol_resolve_and_call

@pr-commenter
Copy link

pr-commenter bot commented Mar 12, 2026

Benchmarks [ profiler ]

Benchmark execution time: 2026-03-12 15:21:08

Comparing candidate commit 0d79129 in PR branch glopes/helper-rust with baseline commit 237080d in branch master.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 28 metrics, 6 unstable metrics.

scenario:php-profiler-timeline-memory-control

  • 🟥 cpu_user_time [+28.023ms; +36.773ms] or [+4.594%; +6.028%]
  • 🟥 execution_time [+27.602ms; +33.008ms] or [+4.307%; +5.151%]

@DataDog DataDog deleted a comment from chatgpt-codex-connector bot Mar 13, 2026
@cataphract
Copy link
Contributor Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 025f3110d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

This introduces a complete Rust reimplementation of the AppSec helper,
a library loaded into the PHP sidecar that executes the Datadog WAF on
request data, handles remote configuration, collects telemetry, and
provides RASP capabilities. The Rust helper is built as a cdylib
(libddappsec-helper-rust.so) and ships alongside the existing C++
helper (libddappsec-helper.so), selectable at runtime.

The Rust crate lives under appsec/helper-rust/ and is structured around
a Tokio async runtime exposed via a C FFI entry point
(appsec_helper_main/appsec_helper_shutdown). The server module accepts
PHP extension connections over a Unix socket. Each connection is handled
by the client module, which implements the msgpack-based protocol codec
(client_init, config_sync, request_init, request_exec,
request_shutdown) matching the C++ wire format. The service module
manages WAF instances per service configuration with atomic updates via
arc-swap, and includes sub-modules for rate limiting, trace sampling,
WAF diagnostics, and ruleset management. The rc module reads remote
configuration from shared memory published by the sidecar, while
rc_notify registers for push-style RC update callbacks via FFI. The
telemetry module submits metrics and logs to the sidecar by resolving
symbols at runtime, and integrates with the logger so error-level
messages are automatically forwarded as telemetry logs.

The wire protocol between the PHP extension and the helper has been
revised. The request_exec command now sends data as an array followed
by a map of options (rasp_rule, subctx_id, subctx_last_call) instead
of sending rasp_rule as a positional field before the data. The
client_init response gains a sixth field, helper_runtime, which the
Rust helper sets to "rust" and the C++ helper sets to "cpp". Both
helpers have been updated to speak this new protocol.

In the PHP extension, helper_process.c gains a
DD_APPSEC_HELPER_RUST_REDIRECTION configuration option (INI setting
datadog.appsec.helper_rust_redirection). When enabled, the extension
looks for libddappsec-helper-rust.so next to the configured helper
path and loads it instead of the C++ binary. The extension tracks
which runtime is connected via a helper_runtime enum and reports it in
phpinfo() output as "Yes (Rust)" or "Yes (C++)". A new span tag
_dd.appsec.helper_runtime is set when using the Rust helper. A testing
function send_invalid_msg is added for protocol error testing.

The binary is built against musl using nightly Rust with -Z build-std
to rebuild the standard library with LLVM's libunwind, then patchelf
removes the musl libc NEEDED entry so the resulting .so runs on both
glibc and musl systems without modification. A glibc_compat.c shim
provides ceil/ceilf/fcntl64/dlopen/dlsym/dlclose implementations for
the musl build so it links without pulling in glibc symbols.

CI changes in .gitlab/generate-appsec.php add four new pipeline jobs:
"helper-rust build and test" (cargo test + format check), "helper-rust
code coverage" (unit test coverage uploaded to codecov), "helper-rust
integration coverage" (integration test coverage), and "appsec
integration tests (helper-rust)" (integration tests on PHP 7.4, 8.1,
8.3, 8.4-zts, 8.5-musl). The existing integration test job template is
refactored into a shared .appsec_integration_tests base. The package
pipeline in generate-package.php adds a "compile appsec helper rust"
job for amd64 and arm64, and the artifact scripts now include
libddappsec-helper-rust.so in both glibc and musl packages. The
Gradle build system (build.gradle) is extended with tasks for building,
testing, and coverage-instrumenting the Rust helper, and gains musl
test support via a new nginx-fpm-musl Docker image. A libddwaf-rust
git submodule is added under appsec/third_party/ for the WAF Rust
bindings. Codecov flags helper-rust-unit and helper-rust-integration
are configured in codecov.yml.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants