fairseq2 icon indicating copy to clipboard operation
fairseq2 copied to clipboard

Revise DownloadManager API to handle concurrent calls

Open cbalioglu opened this issue 2 years ago • 1 comments

When multiple processes in a distributed job attempt to download the same asset, we do not handle it in a race-free way if the file system is shared. Think of a way to avoid it in FileDownloadManager

cbalioglu avatar Feb 07 '24 13:02 cbalioglu

@cbalioglu : actually I'm hitting this error when launching a lots of Sonar data pipeline in parallel on Stopes. Hopefully there's a retry mechanism, but otherwise it feel like P0 is a good priority for this issue !

artemru avatar Feb 07 '24 15:02 artemru