Haris Jabbar
Haris Jabbar
I used version 0.10.1 of tokenizers and I didn't get the error. Hope that helps conda install tokenizers=0.10.1 -c huggingface
@MichalisLazarou Have you found an intuition or explanation for this function? Thanks
This library can be an intermediate solution : https://github.com/ChenghaoMou/text-dedup/tree/main
Hi. If by concurrency you meant multiprocessing, I already tried that, but it's still pretty slow. Can you recommend anything else for English? Thanks
@logan-markewich, @jordanparker6 I am coding up the collator for the masking in the three pretraining strategies. Maybe we can work together, and share it here afterwards for everyone else to...
.. 3 years later .. same problem!
> ```shell > ./update-tlmgr-latest.sh -- --update > ``` For noobs like me, first shell into sharelatex docker instance like so `docker exec -it sharelatex bash` and then run the commands...
@iamshnoo In `option_tokens = [o[1] for o in option_tokens]`, the token at position [1] is a space. `tokenizer.encode(' dogs')=[1, 29871, 26361]`. The space is common to all options, which is...
I just tried to run the code under 'no code execution' in quick start found [here](https://microsoft.github.io/autogen/0.2/docs/Getting-Started#quickstart). And I got the same error.