trueRosun

question of chat templates

6

nice work! starred already. sorry for asking, why replacing the bos_token with empty string? sample['positive'] = tokenizer.apply_chat_template( sample['chosen'], tokenize=False, add_generation_prompt=False).replace(tokenizer.bos_token, "") sample['negative'] = tokenizer.apply_chat_template( sample['rejected'], tokenize=False, add_generation_prompt=False).replace(tokenizer.bos_token, "")