rakro101
rakro101
Using pip install mosaicml-streaming resolves the error above, maybe some dependencies should be added to litdata.
Using then zstd -> and exucting the stream.py => Finished data processing! ⚡ ~ /home/zeus/miniconda3/envs/cloudspace/bin/python /teamspace/studios/this_studio/stream.py 8200 ``` Traceback (most recent call last): File "/teamspace/studios/this_studio/stream.py", line 19, in print(f'{dataset[0]}') File...
@tchaton it works now, but the ending should be .zstd instead of .bin
Comment: When you are using multiple GPUs, avoid creating your datasets in the __init__ method of the DataModule. (Support will be added in the future)
The issue was partially resolved with a workaround and by modifying the dataloader, if I remember correctly.: Comment: When you are using multiple GPUs, avoid creating your datasets in the...
Solved using pl.LightningDataModule and define using def setup ...