Li Yudong (李煜东)

Fine-tuning LLaMA with TencentPretrain and DeepSpeed

11

We implement LLaMA training on the [TencentPretrain framework](https://github.com/Tencent/TencentPretrain), the tutorial is as follows: 1. Clone the TencentPretrain project and install dependencies: PyTorch, DeepSpeed, SentencePiece ``` git clone https://github.com/Tencent/TencentPretrain.git ``` 2....

Thanks for your excellent work! In Chapter 3.5, you gave examples of outstanding text generation results. Could you provide more details about image-to-text generation model?

python3 pretrain.py --dataset_path dataset.pt \ --pretrained_model_path clip.bin \ --output_model_path clip-base16.bin \ --config_path models/clip/base-16_config.json \ --vocab_path vocab.json --merges_path merges.txt --tokenizer clip \ --world_size 4 --gpu_ranks 0 1 2 3 --data_processor clip

## convert python3 scripts/convert_falcon_from_huggingface_to_tencentpretrain.py --input_model_path ../falcon-7b/ --output_model_path models/falcon-7b.bin ## test python3 scripts/generate_lm.py \ --load_model_path models/falcon-7b.bin \ --vocab_path ../falcon-7b/ --tokenizer hfpretrained \ --config_path models/falcon/7b_config.json \ --test_path beginning.txt --prediction_path result.txt

目前页面上的显示：  正确的显示：