CLDSRV-828: fix(rateLimit): dynamic limit updates not taking effect #6046

anurag4DSB · 2026-01-16T14:30:08Z

Note to reviewers: An integration test will be added with https://scality.atlassian.net/browse/INTGR-1063;
Reference slack discussion
Why we missed this

Root Cause:

When a bucket's rate limit was changed via PUT/DELETE API, the
system continued using the original limit because:

Token bucket cached limitConfig on first creation and never
updated it, causing GCRA to use stale interval calculations
Config cache (30s TTL) was not invalidated, delaying updates
and causing stale rateLimitSource in server access logs

Reported Scenarios:

Case 1: Limit 30→0→150, system stuck at 30
Case 2: Global=100, bucket=30, system stuck at 100
Case 3: Set limit before traffic worked (bucket created fresh)

Fix:

getTokenBucket() now updates limitConfig when limit changes
Add invalidateCachedConfig() for immediate cache invalidation
Add removeTokenBucket() for cleanup on DELETE
PUT handler invalidates cache after metadata update
DELETE handler invalidates cache and removes token bucket

bert-e · 2026-01-16T14:30:12Z

Hello anurag4dsb,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-01-16T14:30:17Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-828 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.2.15
9.3.0

Please check the Fix Version/s of CLDSRV-828, or the target
branch of this pull request.

codecov · 2026-01-16T14:31:54Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.37%. Comparing base (c8f4e95) to head (456bd7c).
⚠️ Report is 8 commits behind head on development/9.2.
✅ All tests successful. No failed tests found.

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/api/apiUtils/rateLimit/cache.js	`100.00% <100.00%> (ø)`
lib/api/apiUtils/rateLimit/tokenBucket.js	`79.56% <100.00%> (+1.16%)`	⬆️
lib/api/bucketDeleteRateLimit.js	`96.66% <100.00%> (+0.66%)`	⬆️
lib/api/bucketPutRateLimit.js	`100.00% <100.00%> (ø)`

@@                 Coverage Diff                 @@
##           development/9.2    #6046      +/-   ##
===================================================
+ Coverage            84.36%   84.37%   +0.01%     
===================================================
  Files                  206      206              
  Lines                12999    13014      +15     
===================================================
+ Hits                 10966    10981      +15     
  Misses                2033     2033

Flag	Coverage Δ
file-ft-tests	`67.44% <100.00%> (+0.03%)`	⬆️
kmip-ft-tests	`28.11% <20.00%> (-0.01%)`	⬇️
mongo-v0-ft-tests	`68.66% <100.00%> (+0.01%)`	⬆️
mongo-v1-ft-tests	`68.68% <100.00%> (+0.01%)`	⬆️
multiple-backend	`35.27% <20.00%> (-0.02%)`	⬇️
sur-tests	`35.58% <20.00%> (-0.05%)`	⬇️
sur-tests-inflights	`37.40% <20.00%> (-0.06%)`	⬇️
unit	`69.98% <100.00%> (+0.03%)`	⬆️
utapi-v2-tests	`34.27% <20.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

anurag4DSB · 2026-01-16T14:32:59Z

ping

bert-e · 2026-01-16T14:33:10Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

anurag4DSB · 2026-01-16T14:35:18Z

/create_integration_branches

bert-e · 2026-01-16T14:35:29Z

Conflict

A conflict has been raised during the creation of
integration branch w/9.3/bugfix/CLDSRV-828-fix-limit-override-bug with contents from bugfix/CLDSRV-828-fix-limit-override-bug
and development/9.3.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 git fetch
 git checkout -B w/9.3/bugfix/CLDSRV-828-fix-limit-override-bug origin/development/9.3
 git merge origin/bugfix/CLDSRV-828-fix-limit-override-bug
 # <intense conflict resolution>
 git commit
 git push -u origin w/9.3/bugfix/CLDSRV-828-fix-limit-override-bug

The following options are set: create_integration_branches

anurag4DSB · 2026-01-16T14:50:53Z

ping

bert-e · 2026-01-16T14:51:06Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

bert-e · 2026-01-19T09:05:09Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-828 contains:

9.2.15
9.3.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.2.16
9.3.0

Please check the Fix Version/s of CLDSRV-828, or the target
branch of this pull request.

The following options are set: create_integration_branches

package.json

bert-e · 2026-01-19T09:21:29Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-828 contains:

9.3.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.2.16
9.3.0

Please check the Fix Version/s of CLDSRV-828, or the target
branch of this pull request.

The following options are set: create_integration_branches

The token bucket was created once with the initial limitConfig and never updated when the rate limit changed dynamically. This caused the GCRA interval calculation to use stale limits. Changes: - getTokenBucket() now updates limitConfig when limit changes - Add removeTokenBucket() for cleanup on rate limit deletion - Add invalidateCachedConfig() for immediate cache invalidation Issue: CLDSRV-828

Without cache invalidation, rate limit changes took up to 30 seconds (cache TTL) to take effect. This also caused stale rateLimitSource in server access logs used by monitoring. Changes: - PUT: invalidate cache after metadata update - DELETE: invalidate cache and remove token bucket Issue: CLDSRV-828

Add unit tests covering the cache invalidation and token bucket update functionality introduced in the previous commits. Tests added: - getTokenBucket updates limitConfig when limit changes - getTokenBucket does not log when limit unchanged - removeTokenBucket removes existing bucket - removeTokenBucket returns false for non-existent bucket - invalidateCachedConfig removes specific bucket entry - invalidateCachedConfig returns false for non-existent Issue: CLDSRV-828

anurag4DSB · 2026-01-20T06:31:42Z

/reset

bert-e · 2026-01-20T06:31:51Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2026-01-20T06:32:01Z

Conflict

A conflict has been raised during the creation of
integration branch w/9.3/bugfix/CLDSRV-828-fix-limit-override-bug with contents from bugfix/CLDSRV-828-fix-limit-override-bug
and development/9.3.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 git fetch
 git checkout -B w/9.3/bugfix/CLDSRV-828-fix-limit-override-bug origin/development/9.3
 git merge origin/bugfix/CLDSRV-828-fix-limit-override-bug
 # <intense conflict resolution>
 git commit
 git push -u origin w/9.3/bugfix/CLDSRV-828-fix-limit-override-bug

The following options are set: create_integration_branches

anurag4DSB · 2026-01-20T06:46:08Z

ping

bert-e · 2026-01-20T06:46:19Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

anurag4DSB · 2026-01-20T07:06:54Z

/approve

bert-e · 2026-01-20T07:07:06Z

Build failed

The build for commit did not succeed in branch w/9.3/bugfix/CLDSRV-828-fix-limit-override-bug

The following options are set: approve, create_integration_branches

anurag4DSB · 2026-01-20T07:19:23Z

ping

bert-e · 2026-01-20T07:19:33Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/9.2
✔️ development/9.3

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_integration_branches

anurag4DSB · 2026-01-20T07:51:55Z

ping

bert-e · 2026-01-20T07:51:58Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/9.2
✔️ development/9.3

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1

Please check the status of the associated issue CLDSRV-828.

Goodbye anurag4dsb.

anurag4DSB requested a review from a team January 16, 2026 14:47

anurag4DSB assigned tmacro Jan 16, 2026

anurag4DSB unassigned tmacro Jan 16, 2026

anurag4DSB requested a review from tmacro January 16, 2026 14:57

BourgoisMickael approved these changes Jan 17, 2026

View reviewed changes

anurag4DSB requested a review from a team January 19, 2026 09:05

anurag4DSB commented Jan 19, 2026

View reviewed changes

package.json Outdated Show resolved Hide resolved

tmacro approved these changes Jan 19, 2026

View reviewed changes

anurag4DSB added 4 commits January 20, 2026 07:27

CLDSRV-828: bump package.json from 9.2.15 to 9.2.16

456bd7c

anurag4DSB force-pushed the bugfix/CLDSRV-828-fix-limit-override-bug branch from ca37d48 to 456bd7c Compare January 20, 2026 06:31

bert-e merged commit 9b02185 into development/9.2 Jan 20, 2026
29 checks passed

bert-e deleted the bugfix/CLDSRV-828-fix-limit-override-bug branch January 20, 2026 07:52

CLDSRV-828: fix(rateLimit): dynamic limit updates not taking effect #6046

CLDSRV-828: fix(rateLimit): dynamic limit updates not taking effect #6046

Conversation

anurag4DSB commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root Cause:

Reported Scenarios:

Fix:

Uh oh!

bert-e commented Jan 16, 2026

Hello anurag4dsb,

Uh oh!

bert-e commented Jan 16, 2026

Incorrect fix version

Uh oh!

codecov bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

anurag4DSB commented Jan 16, 2026

Uh oh!

bert-e commented Jan 16, 2026

Request integration branches

Uh oh!

anurag4DSB commented Jan 16, 2026

Uh oh!

bert-e commented Jan 16, 2026

Conflict

Uh oh!

anurag4DSB commented Jan 16, 2026

Uh oh!

bert-e commented Jan 16, 2026

Waiting for approval

Uh oh!

bert-e commented Jan 19, 2026

Incorrect fix version

Uh oh!

Uh oh!

bert-e commented Jan 19, 2026

Incorrect fix version

Uh oh!

anurag4DSB commented Jan 20, 2026

Uh oh!

bert-e commented Jan 20, 2026

Reset complete

Uh oh!

bert-e commented Jan 20, 2026

Conflict

Uh oh!

anurag4DSB commented Jan 20, 2026

Uh oh!

bert-e commented Jan 20, 2026

Waiting for approval

Uh oh!

anurag4DSB commented Jan 20, 2026

Uh oh!

bert-e commented Jan 20, 2026

Build failed

Uh oh!

anurag4DSB commented Jan 20, 2026

Uh oh!

bert-e commented Jan 20, 2026

In the queue

Uh oh!

anurag4DSB commented Jan 20, 2026

Uh oh!

bert-e commented Jan 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

anurag4DSB commented Jan 16, 2026 •

edited

Loading

codecov bot commented Jan 16, 2026 •

edited

Loading