tokenizers
tokenizers copied to clipboard
Published
4 months ago
•
lenML
lenML
a lightweight no-dependency fork from transformers.js (only tokenizers)
Results
1
tokenizers issues
Sort by
recently updated
recently updated
newest added
现在官方已经提供一个纯净的 tokenizer 包了，我这里没必要维护了 并且官方版本还是 ts 版本，类型很友好值得迁移 https://github.com/huggingface/transformers.js/issues/982#event-20664113158 首先，本仓库将彻底转为以提供 tokenizer 预打包的仓库 提供一个预打包 tokenizer 有个最大的好处，可以无缝使用 npm 的 cdn 生态系统，某些项目会很方便 ref https://github.com/huggingface/tokenizers.js https://www.npmjs.com/package/@huggingface/tokenizers
