Retrieval Cost for AIS with GCP / AWS

Open HHenryD opened this issue 3 years ago • 1 comments

I am interested in using PT+AIS to iterate over Google Cloud buckets using the Iterable DataPipes API in PT 1.12, and was wondering if there were retrieval costs in using AIS. Aside from paying for the storage, how cost-friendly would it be to use AIS+Google Cloud (or other services like Webdataset) for training over long durations?

Sep 23 '22 16:09 HHenryD

Assuming you will use AISFileLister and loads AISFileLoader.

With AIS in front of your Cloud bucket(s), you save both cost and time as AIS issues cold GET requests only once on a per-object basis. Multiple alternative data pre-loading mechanisms are also supported. Users can store data according to the per-bucket configurable policies (erasure coding, LRU, etc.).

Screen Shot 2022-09-23 at 11 57 36 AM

Sep 23 '22 18:09 gaikwadabhishek