s3fs icon indicating copy to clipboard operation
s3fs copied to clipboard

Using S3 list_objects_v2 call to check the existence of a bucket is too costly

Open yima77 opened this issue 10 months ago • 1 comments

Here, I refer to the following code lines in s3fs/core.py:

1107             try:
1108                 await self._call_s3(
1109                     "list_objects_v2", MaxKeys=1, Bucket=bucket, **self.req_kw
1110                 )
1111                 return True
1112             except Exception:
1113                 pass

The purpose of these code lines is to check if the bucket exists. This can be accomplished by calling "head_bucket", instead. Listing objects, even with max-keys=1, can be costly on a bucket whose index is managed in multiple shards, like ceph does.

yima77 avatar Apr 29 '25 13:04 yima77

This has been resolved in #961 and can be closed

jacobtomlinson avatar Oct 01 '25 10:10 jacobtomlinson