HDDS-13474. Support abort incomplete multipart upload action#10052
HDDS-13474. Support abort incomplete multipart upload action#10052YutaLin wants to merge 4 commits intoapache:HDDS-8342from
Conversation
spacemonkd
changed the title
| while (mpuIterator.hasNext()) { | ||
| if (!shouldRun()) { |
There was a problem hiding this comment.
Let's add some similar metrics like numKeyIterated for abort incomplete multipart upload case.
| OmMultipartKeyInfo mpuKeyInfo = entry.getValue(); | ||
|
|
||
| // Extract multipart upload information from the key | ||
| OmMultipartUpload upload = OmMultipartUpload.from(entry.getKey()); |
There was a problem hiding this comment.
Catch the IllegalArgumentException and skip the MPU.
| // Check each rule to see if this upload should be aborted | ||
| for (OmLCRule rule : mpuRules) { | ||
| if (shouldAbortUpload(upload, keyName, rule)) { | ||
| expiredUploads.add(upload); |
There was a problem hiding this comment.
Let's try to break down the delete into multiple batch, see ozone.lifecycle.service.delete.batch-size and its usage in the normal key expiry flow (e.g. LimitedExpiredObjectList)
| return false; | ||
| } | ||
|
|
||
| // TODO: Add tag filtering support when multipart uploads support tags |
There was a problem hiding this comment.
Multipart uploads should already support tagging so we can support tag based filtering support.
| } | ||
|
|
||
| @Test | ||
| public void testGetLifecycleWithAbortIncompleteMultipartUpload() throws Exception { |
There was a problem hiding this comment.
Let's add or modify the test in AbstractS3SDKV1Tests and AbstractS3SDKV2Tests to test the E2E flow.
| * @param bucketInfo the bucket information | ||
| * @param ruleList list of lifecycle rules to evaluate | ||
| */ | ||
| private void processMultipartUploads(OmBucketInfo bucketInfo, List<OmLCRule> ruleList) { |
There was a problem hiding this comment.
Please add a test in TestKeyLifecycleService to test this behavior.
3 participants
What changes were proposed in this pull request?
Support AbortIncompleteMultipartUpload action
Please describe your PR in detail:
configure a lifecycle rule by using the AbortIncompleteMultipartUpload action to minimize storage costs.
What is the link to the Apache JIRA
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html
How was this patch tested?
Add unit tests and xml tests
CI Action (https://github.com/YutaLin/ozone/actions/runs/24017470716)