huggingface
Tokenizer training fails with no error message if passed a Path
Passing paths to a training file using pathlib library throws a TypeError without any helpful message.
Please see this gist to reproduce this bug.
More helpful error messages will be generated once PyO3 is upgraded to 0.12. In this particular case there is no implementation to extract a String in Rust from a pathlib Path. I'm not sure whether there is a simple and sane solution for tokenizers to support Paths, but the straight forward one is to pass a str to the failing function.
FYI for PyO3 0.14 you can write functions which take Rust Path objects which will accept both strings and pathlib.Path objects from Python.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}