Can you give an example of the first stage of training?

[yangsenwxy]

Dear all, Thank you very much for your work, but I don't know how to train your first stage, can you give some examples how it works?

[Sleepychord]

pretrain_coglm.py This is used for pretraining the first stage, and the launching scripts should be similar to those in https://github.com/THUDM/SwissArmyTransformer. You need to seed hyper-parameters and environments.

[victorup]

Hi, I want to train the model on my own dataset. How should I process the dataset? Thanks!

[Sleepychord]

@victorup The dataset is binary files created by https://github.com/Sleepychord/cogdata IcetkImageTextTask, which is the fastest way to load data during training. You should prepare zip/rar/tar files for images and text, see https://github.com/Sleepychord/cogdata/tree/dev/downloads/testcase/test_image_text_tokenization_task for format examples.

[victorup]

Can you please share complete scripts for processing datasets and training? Because there are a lot of parameters in commands in https://github.com/Sleepychord/cogdata. I'm not very clear about that. And I also need a training script after getting the processed data. Thanks!