Fix handling of SSE-C keys when copying unencrypted to encrypted objects or objects with different keys

[ashovlin]

Issue #, if available: #6012, and supersedes #8345

Description of changes: When copying a file that originates in S3, the high level aws s3 commands make a HeadObject call for the file that is being copied.

When using server-side encryption with customer-provided keys (SSE-C), callers can specify the key for the source object(s) --sse-c-copy-source/--sse-c-copy-source-key and/or the destination object (--sse-c/--sse-c-key).

The CLI is incorrectly handling this for S3 -> S3 copies, by applying the destination key to the source object.

1. Copying an unencrypted object to an encrypted one: aws s3 cp s3://$bucket/file.txt s3://$bucket/ssec-copy-c.txt --sse-c AES256 --sse-c-key $key_1
2. Copying an object with different encryption keys: aws s3 cp s3://$bucket/ssec.txt s3://$bucket/ssec-copy-diff-key.txt --sse-c-copy-source AES256 --sse-c-copy-source-key $key_1 --sse-c AES256 --sse-c-key $key_2

Arguably Incorrect Behavior, Leaving for Backwards Compatibility

I think we're arguably handling S3 -> local copies as well. aws s3 cp s3://$bucket/ssec.txt decrypt.txt --sse-c AES256 --sse-c-key $key_1 works, but I can see a case for applying --sse-c-copy-source/--sse-c-copy-source-key to the object that's originating to S3. But I'm preserving this behavior for backwards compatibility, the updated logic only runs for S3 -> S3 copies.

Manual Testing
I didn't add new integration tests, but manually ran the following

# bucket name and local paths
bucket=""
single_file=""
directory=""
key_1=""
key_2=""

echo "\nLocal -> S3 encrypted"
aws s3 cp $single_file s3://$bucket/ssec.txt --sse-c AES256 --sse-c-key $key_1
aws s3 cp --recursive $directory s3://$bucket/ssec-files --sse-c AES256 --sse-c-key $key_1

echo "\nS3 encrypted -> Local"
aws s3 cp s3://$bucket/ssec.txt decrypt.txt --sse-c AES256 --sse-c-key $key_1
aws s3 cp --recursive s3://$bucket/ssec-files decrypt/ --sse-c AES256 --sse-c-key $key_1

echo "\nS3 encrypted -> S3 Unencrypted"
aws s3 cp s3://$bucket/ssec.txt s3://$bucket/ssec-copy-unencrypted.txt --sse-c-copy-source AES256 --sse-c-copy-source-key $key_1
aws s3 cp --recursive s3://$bucket/ssec-files s3://$bucket/ssec-files-unencrypted --sse-c-copy-source AES256 --sse-c-copy-source-key $key_1

echo "\nS3 encrypted-> S3 encrypted (same key)"
aws s3 cp s3://$bucket/ssec.txt s3://$bucket/ssec-copy-c.txt --sse-c-copy-source AES256 --sse-c-copy-source-key $key_1 --sse-c AES256 --sse-c-key $key_1
aws s3 cp --recursive s3://$bucket/ssec-files s3://$bucket/ssec-files-copy --sse-c-copy-source AES256 --sse-c-copy-source-key $key_1 --sse-c AES256 --sse-c-key $key_1

echo "\nS3 encrypted -> S3 encrypted (different keys)"
aws s3 cp s3://$bucket/ssec.txt s3://$bucket/ssec-copy-diff-key.txt --sse-c-copy-source AES256 --sse-c-copy-source-key $key_1 --sse-c AES256 --sse-c-key $key_2
aws s3 cp --recursive s3://$bucket/ssec-files s3://$bucket/ssec-files-diff-key --sse-c-copy-source AES256 --sse-c-copy-source-key $key_1 --sse-c AES256 --sse-c-key $key_2

echo "\nS3 unencrypted -> S3 encrypted"
aws s3 cp --recursive $directory s3://$bucket/$directory
aws s3 cp s3://$bucket/ssec-copy-unencrypted.txt s3://$bucket/ssec-copy-c.txt --sse-c AES256 --sse-c-key $key_1
aws s3 cp --recursive s3://$bucket/$directory s3://$bucket/ssec-copy-from-s3 --sse-c AES256 --sse-c-key $key_1

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[FilipPyrek]

Awesome! Thanks @ashovlin 👏

[FilipPyrek]

Hi @ashovlin when do you think we could get this merged? 🙂

[hssyoo]

Looks good. But for discussion:

aws s3 cp s3://$bucket/ssec.txt decrypt.txt --sse-c AES256 --sse-c-key $key_1 works, but I can see a case for applying --sse-c-copy-source/--sse-c-copy-source-key to the object that's originating to S3.

GetObject itself uses SSECustomerAlgorithm over CopySourceSSECustomerAlgorithm, so isn't the current S3->local behavior correct?

[ashovlin]

Looks good. But for discussion:

aws s3 cp s3://$bucket/ssec.txt decrypt.txt --sse-c AES256 --sse-c-key $key_1 works, but I can see a case for applying --sse-c-copy-source/--sse-c-copy-source-key to the object that's originating to S3.

GetObject itself uses SSECustomerAlgorithm over CopySourceSSECustomerAlgorithm, so isn't the current S3->local behavior correct?

Ah, maybe it's more open to interpretation then:

1. My mental model was that the --sse-c-copy-source pair is for the source object, and --sse-c for the destination object regardless of local vs. S3, so the current behavior slightly surprised me.
2. But if you understand that the--see-c pair as maps toSSECustomerAlgorithm, then this makes sense.

[m1so]

Hi @ashovlin & @hssyoo, following up on the PR after taking over from @FilipPyrek. Happy to provide any inputs to move this forward 👍

Our main usecase at Deepnote was summarized by Filip in this thread: #8345 (comment)