s3fs icon indicating copy to clipboard operation
s3fs copied to clipboard

Writing a zero-length object discards s3 kwargs

Open wilsoniya opened this issue 2 years ago • 2 comments

Description

Additional kwargs which are intended to be passed along to low-level S3 code are ignored when writing a zero-length file. This behavior causes headers such as ContentMD5 to be silently ignored.

Reproduction steps

s3fs = S3FileSystem()
s3_kwargs = {"ContentMD5": "BOGUS"}
with s3fs.open(path, "wb", **s3_kwargs) as fout:  
  fout.write(io.BytesIO(b"")

# An OSError should be raised when attempting to PUT zero bytes with a bogus MD5

Analysis

It appears S3File.touch() is called when the library handles writing a zero-length buffer, but this call does not include kwargs which might have been passed to S3FileSystem.open():

https://github.com/fsspec/s3fs/blob/2c074502c2d6a9be0d3f05eb678f4cc5add2e7e5/s3fs/core.py#L2297

wilsoniya avatar Nov 03 '23 22:11 wilsoniya

It sounds like a simple fix should be possible, and easy to test too - would you like to provide it?

martindurant avatar Nov 04 '23 00:11 martindurant

I believe this is fixed on main

martindurant avatar Nov 20 '23 19:11 martindurant