s5cmd icon indicating copy to clipboard operation
s5cmd copied to clipboard

LastModified filter on s3 listing

Open tooptoop4 opened this issue 4 years ago • 4 comments

as per https://stackoverflow.com/questions/45429556/how-list-amazon-s3-bucket-contents-by-modified-date

can s5cmd list only files modified after certain date (server side filtering not client side which is slow) ?

tooptoop4 avatar Dec 03 '21 12:12 tooptoop4

The link above does not mention there's a last-modified filtering on S3 side. ListObjectsV2 API has no server-side filtering based on LastModified date of objects.

If we implement this, filtering would be on the client side.

igungor avatar Jul 26 '23 13:07 igungor

Can you give your use case for this feature? I'm asking this to understand whether your problem can be solved by a combination of the current features or not.

denizsurmeli avatar Jul 27 '23 11:07 denizsurmeli

I need to check new logs but don't want to pay to list millions of old files in the same folder

tooptoop4 avatar Jul 27 '23 19:07 tooptoop4

Well, the implementation will list everything then filter the ones that are after the date you give, since there is no parameter in the request that will only list the items modified after some certain date. Hence, the same cost will occur. Maybe structuring your objects by date would help your problem, but the implementation you ask won't help you to reduce costs unfortunately.

denizsurmeli avatar Jul 27 '23 20:07 denizsurmeli