the normal success rate and downloading speed?

[Tycho-Xue]

Hi, somehow the success rate starts at 0.8 and then dropped to 0.2 after downloading for a while, what success rate should we expect to be healthy? if an average success rate is 0.5 does it mean after the downloading I have only downloaded 50% of the total dataset? Also, will the factors like the image resolution downloaded have a great impact on the success rate? And do you know if it's possible to download the datacomp-1B using a 96 CPU pods in a week? Thanks

[rom1504]

Are you using knot resolver yet ? Can you share a wandb of your img2dataset download?

On Thu, Oct 5, 2023, 6:21 AM Tycho-Xue @.***> wrote:

Hi, somehow the success rate starts at 0.8 and then dropped to 0.2 after downloading for a while, what success rate should we expect to be healthy? if an average success rate is 0.5 does it mean after the downloading I have only downloaded 50% of the total dataset? Also, will the factors like the image resolution downloaded have a great impact on the success rate? And do you know if it's possible to download the datacomp-1B using a 96 CPU pods in a week? Thanks

— Reply to this email directly, view it on GitHub https://github.com/mlfoundations/datacomp/issues/68, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAR437R4WJ6M7FEMVLBBWBTX5XONXAVCNFSM6AAAAAA5TJ3FWWVHI2DSMVQWIX3LMV43ASLTON2WKOZRHEZDOMBWGU3TCNI . You are receiving this because you are subscribed to this thread.Message ID: @.***>