Results
2
comments of
Yifan Ding
Hey, Thanks for raising the question. So basically hetseq generates "HDF5" files following the logic provided by NVIDIA at https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/LanguageModeling/BERT/data/create_datasets_from_start.sh with downloaded wikipedia. You may need to adatp the code...
You definately can do that. Since you have A100, probabily one A100 is good enough to train a large language model.