preprocessor
preprocessor copied to clipboard
Bug when using Tokenize with the URL option
Describe the bug I want to replace URLs in the text with a URL tag, it generally works well but with some input, my code seems to bug with no errors whatsoever.
To Reproduce Steps to reproduce the behavior:
import preprocessor as p
p.set_options(p.OPT.URL)
line = '.......https://www.youtube.com/watch?v=DVdUHfKovn4...........................................'
print(p.tokenize(line))
Expected behavior A clear and concise description of what you expected to happen. '.......$URL$...........................................'
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: Windows-10 64
- Python Version: 3.8.5
- preprocessor version 0.6.0
Additional context Nop