s3fs
s3fs copied to clipboard
Writing a zero-length object discards s3 kwargs
Description
Additional kwargs which are intended to be passed along to low-level S3 code are ignored when writing a zero-length file. This behavior causes headers such as ContentMD5 to be silently ignored.
Reproduction steps
s3fs = S3FileSystem()
s3_kwargs = {"ContentMD5": "BOGUS"}
with s3fs.open(path, "wb", **s3_kwargs) as fout:
fout.write(io.BytesIO(b"")
# An OSError should be raised when attempting to PUT zero bytes with a bogus MD5
Analysis
It appears S3File.touch() is called when the library handles writing a zero-length buffer, but this call does not include kwargs which might have been passed to S3FileSystem.open():
https://github.com/fsspec/s3fs/blob/2c074502c2d6a9be0d3f05eb678f4cc5add2e7e5/s3fs/core.py#L2297
It sounds like a simple fix should be possible, and easy to test too - would you like to provide it?
I believe this is fixed on main