listdir limited to 1000 entries

Open SnapshotCiTy opened this issue 9 years ago • 1 comments

Simple S3 is liiting the output to 1000 file when using listdir.

http://stackoverflow.com/questions/40331681/python-simples3-listdir-limit-the-output-to-1000-files

Any way / plan to make it support more than 1000 - or an unlimited number of entries?

Jan 31 '17 14:01 SnapshotCiTy

I'll copy my response here for posterity

Author of simples3 here. I'm sorry this stopped working, but that's really on Amazon. As you can see in the implementation of S3Bucket.listdir there is support for pagination and it should be seamless. If it isn't, then that's because Amazon stopped returning the truncated flag or stopped returning markers to perform pagination.

You're welcome to do the job, but I have to question Amazon on this decision. Why break existing APIs?

Jan 31 '17 15:01 lericson