AIStore example added

Open davidbuniat opened this issue 4 years ago • 1 comments

AIStore natively does not support boto3, because of http redirect issues https://github.com/NVIDIA/aistore/blob/master/docs/s3compat.md#boto3-compatibility)

[ ] Need to add a storage connector inside hub to support AIStore storage without boto3

ds = hub.empty('s3://ubuntu/dataset', creds={'endpoint_url': 'http://localhost:51080/s3'})

Dec 23 '21 23:12 davidbuniat

WebDatasets has an implementation of https://github.com/webdataset/webdataset/blob/master/webdataset/gopen.py#L153-L192 of using curl

Dec 24 '21 21:12 davidbuniat