Persist checksums for parts, validate on multipart completion

[afranken]

While implementing support for checksums for objects (see #1123 ), I could not implement checksum support for parts as well without major refactorings.

S3Mock currently does not store metadata for parts during a multipart upload, multipart handling (in the MultipartService / MultipartStore) needs to be completely refactored so checksum support can be implemented.

See https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html#AmazonS3-ListParts-response-Parts https://docs.aws.amazon.com/AmazonS3/latest/API/API_Part.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompletedPart.html

[marshalhagen]

Is this work going to get finished?

Also, in your related work - https://github.com/adobe/S3Mock/issues/1123, you stated, "First I thought that the S3 backend would need to calculate the checksum, but it's actually the client that calculates the checksum and then sends it to the backend - as part of the request body."

This is partially correct. The back end should also calculate the checksum and throw an error if the supplied checksum doesn't match.

From https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/s3-checksums.html

...If Amazon S3 determines the checksum value is incorrect for the specified algorithm, the service returns an error response.

[afranken]

@marshalhagen Nobody asked for this, I just added this issue when I worked on general checksum persistence. I'm looking into StorageClass support right now, this could be the next issue to look into. Since S3Mock currently does not persist metadata for parts at all, it could be a lengthy process to add checksums for parts.

[afranken]

@marshalhagen with #1827, I implemented checksum validation for parts and uploads. Thanks for making me aware that I overlooked this at first.

We still do not persist checksums for parts, and do not validate on multipart completion yet.

[ocostello]

Just putting my oar in to see I'd appreciate this feature. The past few days I've been trying to work out why validation on multiparts was giving me false positives, only just thought to check the s3mock

[afranken]

fixed with #1864 and #1890

[afranken]

I just released 3.9.1 which correctly implements checksums for multipart uploads.