Unset inherited JVM env vars in crashtracking scripts#10819
Draft
Unset inherited JVM env vars in crashtracking scripts#10819
Conversation
The OOME notifier and crash uploader scripts now unset JDK_JAVA_OPTIONS, JAVA_TOOL_OPTIONS, and _JAVA_OPTIONS before spawning a child JVM. This prevents port conflicts, memory contention, and lost diagnostics when the parent environment contains flags like JMX remote ports. Fixes #10766 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 65 metrics, 6 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~620ffe3c5a, baseline=1.61.0-SNAPSHOT~93c3816062
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.067 s) : 0, 1067142
Total [baseline] (11.057 s) : 0, 11056548
Agent [candidate] (1.062 s) : 0, 1061715
Total [candidate] (11.122 s) : 0, 11121606
section appsec
Agent [baseline] (1.25 s) : 0, 1250418
Total [baseline] (11.188 s) : 0, 11188084
Agent [candidate] (1.254 s) : 0, 1253686
Total [candidate] (11.131 s) : 0, 11130809
section iast
Agent [baseline] (1.229 s) : 0, 1229082
Total [baseline] (11.296 s) : 0, 11295951
Agent [candidate] (1.244 s) : 0, 1243694
Total [candidate] (11.343 s) : 0, 11342967
section profiling
Agent [baseline] (1.198 s) : 0, 1198029
Total [baseline] (11.014 s) : 0, 11014438
Agent [candidate] (1.202 s) : 0, 1201986
Total [candidate] (11.207 s) : 0, 11206666
gantt
title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~620ffe3c5a, baseline=1.61.0-SNAPSHOT~93c3816062
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.212 ms) : 0, 1212
crashtracking [candidate] (1.194 ms) : 0, 1194
BytebuddyAgent [baseline] (631.61 ms) : 0, 631610
BytebuddyAgent [candidate] (629.755 ms) : 0, 629755
AgentMeter [baseline] (29.403 ms) : 0, 29403
AgentMeter [candidate] (29.248 ms) : 0, 29248
GlobalTracer [baseline] (258.571 ms) : 0, 258571
GlobalTracer [candidate] (257.806 ms) : 0, 257806
AppSec [baseline] (31.79 ms) : 0, 31790
AppSec [candidate] (31.585 ms) : 0, 31585
Debugger [baseline] (59.919 ms) : 0, 59919
Debugger [candidate] (59.631 ms) : 0, 59631
Remote Config [baseline] (590.842 µs) : 0, 591
Remote Config [candidate] (587.455 µs) : 0, 587
Telemetry [baseline] (8.745 ms) : 0, 8745
Telemetry [candidate] (8.706 ms) : 0, 8706
Flare Poller [baseline] (9.117 ms) : 0, 9117
Flare Poller [candidate] (7.097 ms) : 0, 7097
section appsec
crashtracking [baseline] (1.202 ms) : 0, 1202
crashtracking [candidate] (1.203 ms) : 0, 1203
BytebuddyAgent [baseline] (659.125 ms) : 0, 659125
BytebuddyAgent [candidate] (662.748 ms) : 0, 662748
AgentMeter [baseline] (12.102 ms) : 0, 12102
AgentMeter [candidate] (12.092 ms) : 0, 12092
GlobalTracer [baseline] (259.705 ms) : 0, 259705
GlobalTracer [candidate] (259.734 ms) : 0, 259734
AppSec [baseline] (178.479 ms) : 0, 178479
AppSec [candidate] (178.157 ms) : 0, 178157
Debugger [baseline] (65.895 ms) : 0, 65895
Debugger [candidate] (66.009 ms) : 0, 66009
Remote Config [baseline] (579.399 µs) : 0, 579
Remote Config [candidate] (571.322 µs) : 0, 571
Telemetry [baseline] (9.117 ms) : 0, 9117
Telemetry [candidate] (9.051 ms) : 0, 9051
Flare Poller [baseline] (3.629 ms) : 0, 3629
Flare Poller [candidate] (3.594 ms) : 0, 3594
IAST [baseline] (24.158 ms) : 0, 24158
IAST [candidate] (24.075 ms) : 0, 24075
section iast
crashtracking [baseline] (1.196 ms) : 0, 1196
crashtracking [candidate] (1.224 ms) : 0, 1224
BytebuddyAgent [baseline] (797.45 ms) : 0, 797450
BytebuddyAgent [candidate] (808.625 ms) : 0, 808625
AgentMeter [baseline] (11.379 ms) : 0, 11379
AgentMeter [candidate] (11.667 ms) : 0, 11667
GlobalTracer [baseline] (247.65 ms) : 0, 247650
GlobalTracer [candidate] (249.691 ms) : 0, 249691
AppSec [baseline] (26.483 ms) : 0, 26483
AppSec [candidate] (26.687 ms) : 0, 26687
Debugger [baseline] (63.552 ms) : 0, 63552
Debugger [candidate] (64.469 ms) : 0, 64469
Remote Config [baseline] (526.055 µs) : 0, 526
Remote Config [candidate] (539.566 µs) : 0, 540
Telemetry [baseline] (15.158 ms) : 0, 15158
Telemetry [candidate] (14.875 ms) : 0, 14875
Flare Poller [baseline] (4.267 ms) : 0, 4267
Flare Poller [candidate] (4.084 ms) : 0, 4084
IAST [baseline] (25.159 ms) : 0, 25159
IAST [candidate] (25.314 ms) : 0, 25314
section profiling
ProfilingAgent [baseline] (94.653 ms) : 0, 94653
ProfilingAgent [candidate] (95.799 ms) : 0, 95799
crashtracking [baseline] (1.196 ms) : 0, 1196
crashtracking [candidate] (1.181 ms) : 0, 1181
BytebuddyAgent [baseline] (692.443 ms) : 0, 692443
BytebuddyAgent [candidate] (693.122 ms) : 0, 693122
AgentMeter [baseline] (8.755 ms) : 0, 8755
AgentMeter [candidate] (8.77 ms) : 0, 8770
GlobalTracer [baseline] (218.365 ms) : 0, 218365
GlobalTracer [candidate] (219.557 ms) : 0, 219557
AppSec [baseline] (32.51 ms) : 0, 32510
AppSec [candidate] (32.853 ms) : 0, 32853
Debugger [baseline] (63.56 ms) : 0, 63560
Debugger [candidate] (64.793 ms) : 0, 64793
Remote Config [baseline] (597.809 µs) : 0, 598
Remote Config [candidate] (585.971 µs) : 0, 586
Telemetry [baseline] (10.695 ms) : 0, 10695
Telemetry [candidate] (9.903 ms) : 0, 9903
Flare Poller [baseline] (3.554 ms) : 0, 3554
Flare Poller [candidate] (3.609 ms) : 0, 3609
Profiling [baseline] (95.225 ms) : 0, 95225
Profiling [candidate] (96.368 ms) : 0, 96368
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~620ffe3c5a, baseline=1.61.0-SNAPSHOT~93c3816062
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1063713
Total [baseline] (8.828 s) : 0, 8828332
Agent [candidate] (1.065 s) : 0, 1064714
Total [candidate] (8.851 s) : 0, 8851348
section iast
Agent [baseline] (1.226 s) : 0, 1225992
Total [baseline] (9.561 s) : 0, 9561136
Agent [candidate] (1.226 s) : 0, 1226280
Total [candidate] (9.587 s) : 0, 9587250
gantt
title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~620ffe3c5a, baseline=1.61.0-SNAPSHOT~93c3816062
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.215 ms) : 0, 1215
crashtracking [candidate] (1.218 ms) : 0, 1218
BytebuddyAgent [baseline] (632.097 ms) : 0, 632097
BytebuddyAgent [candidate] (631.26 ms) : 0, 631260
AgentMeter [baseline] (29.396 ms) : 0, 29396
AgentMeter [candidate] (29.363 ms) : 0, 29363
GlobalTracer [baseline] (258.424 ms) : 0, 258424
GlobalTracer [candidate] (258.867 ms) : 0, 258867
AppSec [baseline] (31.935 ms) : 0, 31935
AppSec [candidate] (31.852 ms) : 0, 31852
Debugger [baseline] (59.272 ms) : 0, 59272
Debugger [candidate] (59.319 ms) : 0, 59319
Remote Config [baseline] (593.397 µs) : 0, 593
Remote Config [candidate] (597.24 µs) : 0, 597
Telemetry [baseline] (8.785 ms) : 0, 8785
Telemetry [candidate] (8.8 ms) : 0, 8800
Flare Poller [baseline] (5.745 ms) : 0, 5745
Flare Poller [candidate] (7.25 ms) : 0, 7250
section iast
crashtracking [baseline] (1.189 ms) : 0, 1189
crashtracking [candidate] (1.191 ms) : 0, 1191
BytebuddyAgent [baseline] (795.209 ms) : 0, 795209
BytebuddyAgent [candidate] (795.132 ms) : 0, 795132
AgentMeter [baseline] (11.348 ms) : 0, 11348
AgentMeter [candidate] (11.338 ms) : 0, 11338
GlobalTracer [baseline] (247.654 ms) : 0, 247654
GlobalTracer [candidate] (247.981 ms) : 0, 247981
AppSec [baseline] (26.369 ms) : 0, 26369
AppSec [candidate] (26.404 ms) : 0, 26404
Debugger [baseline] (62.876 ms) : 0, 62876
Debugger [candidate] (62.681 ms) : 0, 62681
Remote Config [baseline] (535.485 µs) : 0, 535
Remote Config [candidate] (520.118 µs) : 0, 520
Telemetry [baseline] (15.004 ms) : 0, 15004
Telemetry [candidate] (15.081 ms) : 0, 15081
Flare Poller [baseline] (4.635 ms) : 0, 4635
Flare Poller [candidate] (4.555 ms) : 0, 4555
IAST [baseline] (25.153 ms) : 0, 25153
IAST [candidate] (25.148 ms) : 0, 25148
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 3 performance regressions! Performance is the same for 17 metrics, 15 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~620ffe3c5a, baseline=1.61.0-SNAPSHOT~93c3816062
dateFormat X
axisFormat %s
section baseline
no_agent (1.17 ms) : 1158, 1181
. : milestone, 1170,
iast (3.17 ms) : 3131, 3209
. : milestone, 3170,
iast_FULL (5.813 ms) : 5755, 5871
. : milestone, 5813,
iast_GLOBAL (3.537 ms) : 3481, 3593
. : milestone, 3537,
profiling (2.204 ms) : 2183, 2224
. : milestone, 2204,
tracing (1.815 ms) : 1799, 1831
. : milestone, 1815,
section candidate
no_agent (1.174 ms) : 1162, 1185
. : milestone, 1174,
iast (3.195 ms) : 3151, 3239
. : milestone, 3195,
iast_FULL (6.09 ms) : 6028, 6153
. : milestone, 6090,
iast_GLOBAL (3.753 ms) : 3684, 3823
. : milestone, 3753,
profiling (1.942 ms) : 1925, 1959
. : milestone, 1942,
tracing (1.782 ms) : 1769, 1796
. : milestone, 1782,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~620ffe3c5a, baseline=1.61.0-SNAPSHOT~93c3816062
dateFormat X
axisFormat %s
section baseline
no_agent (18.215 ms) : 18028, 18402
. : milestone, 18215,
appsec (19.121 ms) : 18932, 19311
. : milestone, 19121,
code_origins (17.882 ms) : 17704, 18060
. : milestone, 17882,
iast (17.742 ms) : 17564, 17919
. : milestone, 17742,
profiling (18.883 ms) : 18694, 19073
. : milestone, 18883,
tracing (17.824 ms) : 17645, 18002
. : milestone, 17824,
section candidate
no_agent (19.189 ms) : 18995, 19382
. : milestone, 19189,
appsec (19.051 ms) : 18856, 19247
. : milestone, 19051,
code_origins (17.811 ms) : 17634, 17988
. : milestone, 17811,
iast (17.794 ms) : 17615, 17972
. : milestone, 17794,
profiling (21.025 ms) : 20810, 21239
. : milestone, 21025,
tracing (18.028 ms) : 17849, 18206
. : milestone, 18028,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~620ffe3c5a, baseline=1.61.0-SNAPSHOT~93c3816062
dateFormat X
axisFormat %s
section baseline
no_agent (1.483 ms) : 1472, 1495
. : milestone, 1483,
appsec (3.807 ms) : 3585, 4030
. : milestone, 3807,
iast (2.27 ms) : 2201, 2339
. : milestone, 2270,
iast_GLOBAL (2.308 ms) : 2239, 2377
. : milestone, 2308,
profiling (2.528 ms) : 2360, 2696
. : milestone, 2528,
tracing (2.076 ms) : 2022, 2129
. : milestone, 2076,
section candidate
no_agent (1.485 ms) : 1474, 1497
. : milestone, 1485,
appsec (3.81 ms) : 3590, 4030
. : milestone, 3810,
iast (2.274 ms) : 2205, 2342
. : milestone, 2274,
iast_GLOBAL (2.312 ms) : 2243, 2381
. : milestone, 2312,
profiling (2.118 ms) : 2062, 2175
. : milestone, 2118,
tracing (2.076 ms) : 2022, 2129
. : milestone, 2076,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~620ffe3c5a, baseline=1.61.0-SNAPSHOT~93c3816062
dateFormat X
axisFormat %s
section baseline
no_agent (15.107 s) : 15107000, 15107000
. : milestone, 15107000,
appsec (14.853 s) : 14853000, 14853000
. : milestone, 14853000,
iast (18.087 s) : 18087000, 18087000
. : milestone, 18087000,
iast_GLOBAL (17.712 s) : 17712000, 17712000
. : milestone, 17712000,
profiling (14.995 s) : 14995000, 14995000
. : milestone, 14995000,
tracing (15.16 s) : 15160000, 15160000
. : milestone, 15160000,
section candidate
no_agent (14.968 s) : 14968000, 14968000
. : milestone, 14968000,
appsec (15.253 s) : 15253000, 15253000
. : milestone, 15253000,
iast (18.094 s) : 18094000, 18094000
. : milestone, 18094000,
iast_GLOBAL (16.964 s) : 16964000, 16964000
. : milestone, 16964000,
profiling (15.526 s) : 15526000, 15526000
. : milestone, 15526000,
tracing (14.995 s) : 14995000, 14995000
. : milestone, 14995000,
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Clears
JDK_JAVA_OPTIONS,JAVA_TOOL_OPTIONS, and_JAVA_OPTIONSenvironment variables before spawning a child JVM in the OOME notifier and crash uploader scripts (both.shand.batvariants). This prevents the child JVM from inheriting the parent's environment, which can cause port conflicts, memory contention, and lost diagnostics.Motivation
When the Datadog admission controller injects the agent via
JAVA_TOOL_OPTIONS, and the application sets JMX or memory flags viaJDK_JAVA_OPTIONS, the child JVM spawned bydd_oome_notifier.sh/dd_crash_uploader.shinherits these variables. This causes failures such asBindExceptionon JMX ports still held by the parent, or cgroup OOM kills from competing memory reservations — resulting in lost OOME/crash events.GH Ticket #10766
Additional Notes
notify_oome.sh,notify_oome.bat,upload_crash.sh,upload_crash.batupload_crash.*are coveredJDK_JAVA_OPTIONSwith JMX port-binding flags and asserting events are still receivedContributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueNote: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.