Shengyang Sun

GPTSFTChatDataset loss_mask becomes all False when prompt length > max_seq_length

5

**Describe the bug** In the `GPTSFTChatDataset`, if the first prompt length exceeds `max_seq_length`, all following turns are truncated out. Then the `loss_mask` becomes all `False` for the example. https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_chat_dataset.py#L359 This...

read system token from data in gpt_sft_chat_dataset

5

# What does this PR do ? The original implementation of the prompt template in `gpt_sft_chat_dataset.py` uses a fix system token `System`. This PR enables to read the system token...