xtuner
xtuner copied to clipboard
Resolved TODO in tokenize_ftdp_datasets.py
- [x] This PR addresses a TODO in the
tokenize_ftdp_datasets.pyfile.
Solution:
I used the lstrip method to remove all leading newline characters from the content string. This way, we can avoid situations where the string starts with \n\n.
content = content.lstrip('\n')