ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard
verified publisher icon hpcaitech

[PROPOSAL]: Can you write down the detailed steps and technical details of the second and third steps?

Open Kayce001 opened this issue 2 years ago • 2 comments

Proposal

Therefore, we have divided the training process into three stages:

Large-scale pre-training stage (Conducted by LLaMA-2): This initial stage is aimed at establishing the model's foundational capabilities from the ground up. It necessitates the use of a substantial dataset comprising no less than 1 trillion tokens. Chinese knowledge injection stage: In this stage, we introduce Chinese knowledge into the model. It requires access to a high-quality dataset rich in comprehensive knowledge relevant to the Chinese language. Knowledge replay stage: Knowledge is replayed through a question-answering (QA) mechanism, encompassing both the Chinese and English domains.

Self-service

  • [ ] I'd be willing to do some initial work on this proposal myself.

Kayce001 avatar Sep 30 '23 07:09 Kayce001

Hi, thanks for your interest.

Yes, we are planning to release a technical report of our training process. Please stay tuned. Thanks

TongLi3701 avatar Oct 03 '23 06:10 TongLi3701

Hi, thanks for your interest.

Yes, we are planning to release a technical report of our training process. Please stay tuned. Thanks

Will the technical report or other stages of training code be made public?

lyy-zz avatar Oct 23 '23 08:10 lyy-zz