Skip to content

feat(mem): JDK 25 UseCompactObjectHeaders #35931

Open
wezell wants to merge 18 commits into
mainfrom
issue-33865-java25-memory-tuning
Open

feat(mem): JDK 25 UseCompactObjectHeaders #35931
wezell wants to merge 18 commits into
mainfrom
issue-33865-java25-memory-tuning

Conversation

@wezell

@wezell wezell commented Jun 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Big news in this PR - this turns on Java's new UseCompactObjectHeaders which can save up to 20% in memory usage across the board. With this change, we had to adjust the dotCMS's default cache sizing algo and tests to account for the smaller objects.

It also applies jaz's recommended GC tuning suggestions and adds some constraints for off-heap memory allocations by default ( we have been running unbounded) while also removing jaz as the default java launcher. jaz can be re-enabled by setting the env var JAZ_IGNORE_USER_TUNING=1

Container JVM (setenv.sh)

  • -XX:+UseCompactObjectHeaders (JEP 519, product flag on JDK 25; ~5–15% heap savings; jaz does not set it — verified via JAZ_DRY_RUN)
  • GC: ZGC → G1 with jaz-mirrored tuning (MinHeapFreeRatio=10, MaxHeapFreeRatio=50, G1PeriodicGCInterval=10000), MaxRAMPercentage=68
  • Off-heap OOM diagnostics: -XX:NativeMemoryTracking=summary, -Djdk.nio.maxCachedBufferSize=262144, and a dynamic, cgroup-derived -XX:MaxDirectMemorySize (20% clamped [512m,4096m], fail-safe, env-overridable)

Tests

  • surefire/failsafe argLine: add -XX:+UseCompactObjectHeaders (prod parity)
  • CacheSizingUtilTest: layout-agnostic assertions (COH shrinks headers 12→8B; the util reads live Unsafe offsets and adapts — only the hardcoded byte expectations were wrong)

TODO (next commit)

  • Refactor CacheSizingUtil to use Instrumentation.getObjectSize() (authoritative shallow size; Unsafe fallback when no agent) per discussion.

This PR fixes: #33865

wezell and others added 7 commits June 3, 2026 11:17
@wezell @claude
feat(java): move to Java 25 (runtime 25.0.2-ms, default bytecode 25)
6453610 
Reapplies the Java 25 part-2 changes on top of current main, keeping main's
configurable dotcms.core.compiler.release mechanism but flipping its default
from 11 to 25 so core compiles to Java 25 bytecode by default (override with
-Ddotcms.core.compiler.release=11 for older compatibility).

- .sdkmanrc / docker/java-base: build/runtime JDK -> 25.0.2-ms
- parent/pom.xml: dotcms.core.compiler.release default 11 -> 25,
  jdk.min.version -> 25, glowroot -> 0.14.5-beta.3-java25
- test-jmeter / test-karate: compiler source/target/release -> 25
- CLAUDE.md: runtime references 21 -> 25
- Add Java 25 problems/solutions and smoke-test docs

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wezell @claude
fix(java): address PR review — inherit compiler release in test modul…
57d2ce9 
…es, align docs

- test-jmeter / test-karate: use ${dotcms.core.compiler.release} instead of
  hardcoded 25 so the -Ddotcms.core.compiler.release override applies uniformly
- CLAUDE.md: resolve Java 11-syntax vs default-25-bytecode contradiction; core
  now compiles to Java 25 by default (override-able), runtime is Java 25

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wezell @claude
docs(java25): remove migration notes from repo root
e4d7ef8 
Drop JAVA25_PROBLEMS_AND_SOLUTIONS.md and JAVA25_SMOKE_TEST.md from the
source tree per review feedback — operational/migration notes belong outside
the versioned source.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wezell @claude
feat(java): enable JDK preview features + switch container GC to G1 w…
cd9ba38 
…ith compact object headers

Enable project-wide --enable-preview (JDK 25) so core can use java.lang.StableValue
(JEP 502), and update the container JVM flags.

Container runtime (setenv.sh):
- Replace ZGC (-XX:+UseZGC -XX:+ZGenerational) with -XX:+UseG1GC
- Add -XX:+UseCompactObjectHeaders (product flag on JDK 25, JEP 519)
- Add --enable-preview
- Drop -XX:+UnlockExperimentalVMOptions (no longer needed)

Build wiring (parent/pom.xml):
- New property maven.compiler.enablePreview=true, wired into maven-compiler-plugin
- Add --enable-preview and -XX:+UseCompactObjectHeaders to surefire + failsafe argLines
- .mvn/jvm.config carries --enable-preview so compile-phase plugins (swagger-maven-plugin)
  can classload preview-stamped (0xFFFF) core classes

Pinned-to-Java-11 modules set enablePreview=false (javac rejects --enable-preview at
--release 11): dotcms-cli, tika-api, tika-plugin.

Add StableValuePreviewTest smoke test (surefire): fails the build if preview support
regresses and asserts -XX:+UseCompactObjectHeaders is active under surefire.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wezell @claude
fix(java): JDK 25 container JVM tuning + off-heap diagnostics
a753fa3 
Container runtime (setenv.sh), validated against dotcms/dotcms:java-25 and the
certified mcr.microsoft.com/openjdk/jdk:25-ubuntu jaz build:

Heap/GC (mirrors jaz's profile, which it applies only when no -X/-XX flag is set;
since we set tuning flags explicitly, jaz is hands-off but still does crash-dump +
graceful SIGTERM):
- -XX:+UseG1GC, -XX:MaxRAMPercentage=66.0 (lowered from jaz's ~72% for more native
  headroom; fixed native costs — ~600 thread stacks, metaspace, code cache — are
  roughly constant and dominate small 4GB containers)
- -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=50 -XX:G1PeriodicGCInterval=10000
- -XX:+UseCompactObjectHeaders (JEP 519, ~5-15% heap savings; jaz never sets it)

Off-heap diagnostics (workloads run 4GB-30GB containers, OOM-killed by native/RSS
growth of unknown origin):
- -XX:NativeMemoryTracking=summary — jcmd VM.native_memory summary[.diff] to attribute
  JVM-internal native memory (metaspace, threads, code cache, direct buffers)
- -Djdk.nio.maxCachedBufferSize=262144 — bound the JDK's unbounded per-thread temp
  direct-buffer cache (a common RSS balloon with many I/O threads)
- Dynamic -XX:MaxDirectMemorySize derived from the cgroup limit (DOT_DIRECT_MEM_PCT%,
  default 20%, clamped [512m,4096m]; fail-safe to unset when unbounded). Caps ONLY NIO
  direct ByteBuffers — a rule-in/rule-out probe, not a catch-all (does not bound JNI/
  mmap/Unsafe/Netty); use NMT + jemalloc profiling for those.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wezell @claude
refactor(java): split container JVM tuning + off-heap diagnostics out…
b10e2a9 
… of Java 25 PR

Keep #35914 focused on the Java 25 migration only. The compact object headers
(-XX:+UseCompactObjectHeaders), G1/heap tuning, and off-heap OOM diagnostics
(NativeMemoryTracking, jdk.nio.maxCachedBufferSize, dynamic MaxDirectMemorySize) move to
a dedicated branch (issue-33865-java25-memory-tuning) since they are runtime tuning, not
the migration itself — and COH is what required the CacheSizingUtil changes.

setenv.sh: restore original (ZGC) container config, add only --enable-preview (required at
runtime to load preview-compiled classes, e.g. java.lang.StableValue / JEP 502).
parent/pom.xml: drop -XX:+UseCompactObjectHeaders from the surefire/failsafe argLine; keep
--enable-preview. This removes the CacheSizingUtilTest object-size failure that fail-fast
canceled the PR test matrix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wezell @claude
feat(mem): JDK 25 container memory tuning + off-heap OOM diagnostics
2509a7f 
Stacked on the Java 25 migration (#35914). Runtime memory tuning + diagnostics that were
split out of that PR:

- -XX:+UseCompactObjectHeaders (JEP 519, ~5-15% heap savings; jaz never sets it)
- Switch container GC ZGC -> G1 with jaz-mirrored tuning (MinHeapFreeRatio=10,
  MaxHeapFreeRatio=50, G1PeriodicGCInterval=10000) and MaxRAMPercentage=68
- Off-heap OOM diagnostics: -XX:NativeMemoryTracking=summary,
  -Djdk.nio.maxCachedBufferSize=262144, and a dynamic cgroup-derived
  -XX:MaxDirectMemorySize (rule-in/rule-out probe for direct ByteBuffers)
- surefire/failsafe argLine: add -XX:+UseCompactObjectHeaders (prod parity)
- CacheSizingUtilTest: layout-agnostic assertions (COH shrinks object headers 12->8;
  the util reads live Unsafe offsets so it adapts — only the hardcoded expectations were wrong)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot mentioned this pull request Jun 4, 2026
Closed
3 tasks
@wezell @claude
refactor(cache): size objects via Instrumentation.getObjectSize, agen…
25f75ff 
…t-enable unit tests

Replace CacheSizingUtil's manual Unsafe/header math with the authoritative
Instrumentation.getObjectSize() for shallow size (recursion for retained size kept).
getObjectSize is exact under compact object headers, compressed oops, and alignment —
no header/oops guesswork.

- Instrumentation is read ONLY via ByteBuddyAgent.getInstrumentation() (already-installed
  agent); never install() — its self-attach hangs on JDK 21+ in containers. Falls back to
  the Unsafe estimate when no agent is present.
- Fix latent Unsafe-fallback bug: sizeof() used f.getClass() (always Field.class) instead of
  f.getType(), under-counting when the highest-offset field is a long/double.
- Agent-enable dotcms-core unit tests: dependency-plugin 'properties' goal populates
  ${net.bytebuddy:byte-buddy-agent:jar}; a module-scoped ${bytebuddy.agent.argline} (empty
  elsewhere) preloads it via -javaagent in the shared surefire/failsafe argLine, so the
  getObjectSize path is exercised in CI. Verified: agent-presence test runs (not skipped).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Base automatically changed from issue-33865-java-25-rebase to main June 4, 2026 18:14
@github-actions github-actions Bot added Area : Backend PR changes Java/Maven backend code and removed AI: Safe To Rollback labels Jun 5, 2026
@wezell wezell marked this pull request as ready for review June 9, 2026 13:20
@wezell @claude
chore(mem): drop explicit -XX:+UseG1GC (G1 is the JDK 25 default)
85b5e18 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wezell @claude
chore(mem): drop -XX:NativeMemoryTracking=summary from JAVA_OPTS_MEMORY
cff6aa5 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@claude
Lower DOT_DIRECT_MEM_MIN_MB default from 512 to 128 to prevent OOM on…
6d40d90 
… small containers

With MaxRAMPercentage=72, a 1GB container allocates ~737MB to heap, leaving only
~287MB for metaspace, stack, code cache, and direct buffers. The 512MB floor caused
OOM kills on containers under ~2.5GB; 128MB fits comfortably within that budget.

https://claude.ai/code/session_015P4qyq2daq4w7WeYhbs1gw
@wezell wezell changed the title feat(mem): JDK 25 container memory tuning + off-heap OOM diagnostics feat(mem): JDK 25 UseCompactObjectHeaders Jun 9, 2026
@wezell
Clean up comments related to jaz JVM launcher
13a3479 
Removed detailed comments about jaz JVM launcher and tuning flags.
@claude
fix: invert JAZ_IGNORE_USER_TUNING condition so jaz is opt-in
c5d16ae 
The -z check enabled jaz when the variable was unset (the default),
which is the opposite of the documented intent. Change to -n so jaz
is only activated when JAZ_IGNORE_USER_TUNING=1 is explicitly set.

https://claude.ai/code/session_01HDzAEtJb7E3KmM2SKRs9Gn
@claude
fix(java25): restore --enable-native-access=ALL-UNNAMED to JAVA_OPTS_…
ede8856 
…BASE

This flag was present in the original JAVA_OPTS_BASE and was accidentally
dropped during the Java 25 memory tuning refactor.

https://claude.ai/code/session_01Au82uqQdNaYjAuQanj2rdB
@dotCMS dotCMS deleted a comment from claude Bot Jun 9, 2026
@claude

claude Bot commented Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Claude finished @wezell's task in 1m 44s —— View job

Rollback Safety Analysis

  • Read rollback-unsafe categories reference
  • Get full PR diff
  • Analyze changes against unsafe categories
  • Apply appropriate label

Result: ✅ Safe To Rollback

All 5 changed files were analyzed against the rollback-unsafe categories:

File Category Check Result
dotCMS/src/main/resources/container/tomcat9/bin/setenv.sh JVM startup flags only — no DB, ES, or API changes ✅ Safe
dotCMS/src/main/java/com/dotmarketing/business/cache/provider/CacheSizingUtil.java Internal utility class, not an OSGi-exported interface or REST API ✅ Safe
dotCMS/src/test/java/com/dotmarketing/business/cache/provider/CacheSizingUtilTest.java Test-only change ✅ Safe
dotCMS/pom.xml Build/test configuration only (byte-buddy agent wiring) ✅ Safe
parent/pom.xml Build/test configuration only (-XX:+UseCompactObjectHeaders in surefire argLine) ✅ Safe

No matches against any unsafe category (C-1 through M-4):

  • No database runonce tasks, schema changes, or data migrations
  • No Elasticsearch mapping changes
  • No REST/GraphQL API contract changes
  • No OSGi public interface changes (the retainedSize return type change from int → long is on a package-private utility, not an exported interface)
  • No contentlet_as_json model version bumps
  • No storage provider configuration changes
  • No push publishing bundle format changes

Label AI: Safe To Rollback has been applied.

@wezell wezell requested review from swicken and yolabingo June 10, 2026 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@swicken swicken Awaiting requested review from swicken

@yolabingo yolabingo Awaiting requested review from yolabingo

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

AI: Safe To Rollback Area : Backend PR changes Java/Maven backend code

Projects

dotCMS - Product Planning
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[SPIKE] Run on Java 25

2 participants

@wezell @claude