APPSEC-61588 - SDS scans returned in SDK response by AI Guard [system-tests]

[obordeau]

Motivation

Test that AI Guard return SDS tags in SDK response from evaluator API response.

APPSEC-61588

Changes

Adds Test_SDS_Findings_In_SDK_Response
Update weblogs for Python and Ruby to add SDS in the Abort Error and SDK Response

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/ai_guard/test_ai_guard_sdk.py                                     @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/python/flask/app.py                                  @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/ruby/rails52/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails61/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails72/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails80/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core

[datadog-datadog-prod-us1]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 5 Tests failed

tests.ai_guard.test_ai_guard_sdk.Test_SDS_Findings_In_SDK_Response.test_sds_in_response[express4] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: 'sds' not found in '{'action': 'ALLOW', 'reason': 'No rule match.', 'tags': []}'
assert 'sds' in {'action': 'ALLOW', 'reason': 'No rule match.', 'tags': []}

self = <tests.ai_guard.test_ai_guard_sdk.Test_SDS_Findings_In_SDK_Response object at 0x7f92d4283320>

    def test_sds_in_response(self):
        """Test SDS findings are returned in SDK response.
        Verifies that the SDK evaluation response contains sds findings.
        """
        assert self.r.status_code == 200
...

tests.ai_guard.test_ai_guard_sdk.Test_SDS_Findings_In_SDK_Response.test_sds_in_response[spring-boot] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: 'sds' not found in '{'action': 'ALLOW', 'reason': 'No rule match.', 'tags': []}'
assert 'sds' in {'action': 'ALLOW', 'reason': 'No rule match.', 'tags': []}

self = <tests.ai_guard.test_ai_guard_sdk.Test_SDS_Findings_In_SDK_Response object at 0x7f73804344d0>

    def test_sds_in_response(self):
        """Test SDS findings are returned in SDK response.
        Verifies that the SDK evaluation response contains sds findings.
        """
        assert self.r.status_code == 200
...

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Line_Probe_Snaphots.test_default_max_field_count[uds-rails] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: assert 'Probes did not reach INSTALLED status' is None
 +  where 'Probes did not reach INSTALLED status' = <built-in method join of str object at 0x7fefba22aa60>(['Probes did not reach INSTALLED status'])
 +    where <built-in method join of str object at 0x7fefba22aa60> = '\n'.join
 +    and   ['Probes did not reach INSTALLED status'] = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Line_Probe_Snaphots object at 0x7fefb01d5370>.setup_failures

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Line_Probe_Snaphots object at 0x7fefb01d5370>

    def test_default_max_field_count(self):
        """Test that the tracer uses default maxFieldCount=20 when capture property is omitted"""
>       many_fields = self._get_snapshot_locals_variable("manyFields")
...

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a5929e8 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}