YARN-11964: Clamp negative available resources in AllocateResponse to zero by ryukobayashi · Pull Request #8523 · apache/hadoop

ryukobayashi · 2026-06-01T08:32:46Z

Description of PR

YARN RM can transiently report negative available resources (e.g. cluster_capacity - allocated goes negative due to overload or node failures). DefaultAMSProcessor now applies Resources.componentwiseMax with Resources.none() before setting available resources in AllocateResponse, ensuring the AM always receives a non-negative value.

This is a re-implementation of YARN-11964 (reverted in #8519). The previous fix applied the clamp inside Resource.castToIntSafely(), which was too broad and caused a regression in TestRLESparseResourceAllocation. This fix applies the clamp at the correct layer — in DefaultAMSProcessor where the AllocateResponse is built.

Contains content generated by Claude (Anthropic)

How was this patch tested?

Added TestDefaultAMSProcessor#testAvailableResourcesClampedToNonNegative which uses a custom scheduler that returns a negative resource limit and verifies the AllocateResponse always contains non-negative available resources.
Existing tests pass: TestApplicationMasterServiceCapacity, TestApplicationMasterServiceFair, TestApplicationMasterServiceInterceptor, TestRLESparseResourceAllocation.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

The PR includes the phrase "Contains content generated by Claude (Anthropic)"
My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html

hadoop-yetus · 2026-06-02T06:38:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 21s		trunk passed
+1 💚	compile	0m 50s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	0m 52s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	0m 51s		trunk passed
+1 💚	mvnsite	0m 55s		trunk passed
+1 💚	javadoc	0m 49s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 48s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 29s		trunk passed
+1 💚	shadedclient	16m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 36s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 30s		the patch passed
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	javadoc	0m 29s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 20s		the patch passed
+1 💚	shadedclient	15m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	91m 21s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		163m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/2/artifact/out/Dockerfile
GITHUB PR	#8523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 61aa8d2782f7 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a767604`
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/2/testReport/
Max. process+thread count	956 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/2/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

pan3793 · 2026-06-03T03:19:19Z

@ryukobayashi, the Test hdfs-rbf failure and hang can be ignored, it is caused by another PR, and the author is working on the solution, but the Test mr is expected to pass. Since the GHA job runs on your forked repo, you have permission to rerun it if you think it's a flaky test which is irrelevant to your change.

ryukobayashi · 2026-06-03T04:59:34Z

@pan3793 Yes, I tried again as you suggested, but it failed. So, could you review this?

… zero

hadoop-yetus · 2026-06-09T08:51:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 49s		trunk passed
+1 💚	compile	0m 51s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	0m 53s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	0m 47s		trunk passed
+1 💚	mvnsite	0m 55s		trunk passed
+1 💚	javadoc	0m 50s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 46s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 32s		trunk passed
+1 💚	shadedclient	16m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 30s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 16s		the patch passed
+1 💚	shadedclient	15m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	91m 16s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		162m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/3/artifact/out/Dockerfile
GITHUB PR	#8523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 3d011f3ac43b 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `03c7018`
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/3/testReport/
Max. process+thread count	956 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/3/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

pan3793 · 2026-06-09T09:13:09Z

@ryukobayashi the mr failure is related to your change, try debugging locally.

slfan1989 · 2026-06-10T01:15:38Z

@ryukobayashi I’ll take a look at this PR today and share my feedback later.
cc: @pan3793

ryukobayashi · 2026-06-10T02:46:21Z

@pan3793 Thanks, I identified and fixed the parts where NPE was occurring.
@slfan1989 Thanks.

hadoop-yetus · 2026-06-10T05:28:14Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	26m 19s		trunk passed
+1 💚	compile	0m 49s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	compile	0m 50s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	checkstyle	0m 45s		trunk passed
+1 💚	mvnsite	0m 56s		trunk passed
+1 💚	javadoc	0m 50s		trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 45s		trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 34s		trunk passed
+1 💚	shadedclient	16m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 29s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	javac	0m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 28s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
+1 💚	spotbugs	1m 18s		the patch passed
+1 💚	shadedclient	15m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	91m 7s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		162m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/4/artifact/out/Dockerfile
GITHUB PR	#8523
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ce988b2140bb 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `905a1a7`
Default Java	Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/4/testReport/
Max. process+thread count	927 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8523/4/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

github-actions Bot added YARN trunk labels Jun 1, 2026

ryukobayashi mentioned this pull request Jun 1, 2026

YARN-11964: Resource.castToIntSafely() should clamp negative values to 0 #8506

Merged

6 tasks

ryukobayashi added 2 commits June 9, 2026 15:08

YARN-11964: Clamp negative available resources in AllocateResponse to…

9c78211

… zero

YARN-11964: Trigger CI

03c7018

ryukobayashi force-pushed the YARN-11964 branch from a767604 to 03c7018 Compare June 9, 2026 06:08

pan3793 requested a review from slfan1989 June 9, 2026 06:15

YARN-11964: Handle null resourceLimit in DefaultAMSProcessor

905a1a7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11964: Clamp negative available resources in AllocateResponse to zero#8523

YARN-11964: Clamp negative available resources in AllocateResponse to zero#8523
ryukobayashi wants to merge 3 commits into
apache:trunkfrom
ryukobayashi:YARN-11964

ryukobayashi commented Jun 1, 2026

Uh oh!

hadoop-yetus commented Jun 2, 2026

Uh oh!

pan3793 commented Jun 3, 2026

Uh oh!

ryukobayashi commented Jun 3, 2026

Uh oh!

hadoop-yetus commented Jun 9, 2026

Uh oh!

pan3793 commented Jun 9, 2026

Uh oh!

slfan1989 commented Jun 10, 2026

Uh oh!

ryukobayashi commented Jun 10, 2026

Uh oh!

hadoop-yetus commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ryukobayashi commented Jun 1, 2026

Description of PR

How was this patch tested?

For code changes:

AI Tooling

Uh oh!

hadoop-yetus commented Jun 2, 2026

Uh oh!

pan3793 commented Jun 3, 2026

Uh oh!

ryukobayashi commented Jun 3, 2026

Uh oh!

hadoop-yetus commented Jun 9, 2026

Uh oh!

pan3793 commented Jun 9, 2026

Uh oh!

slfan1989 commented Jun 10, 2026

Uh oh!

ryukobayashi commented Jun 10, 2026

Uh oh!

hadoop-yetus commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants