transformers icon indicating copy to clipboard operation
transformers copied to clipboard
verified publisher icon huggingface

adding user defined tokens #30824

Open itazap opened this issue 1 year ago • 2 comments

Fixes #30824 #30947

Tasks

  • [ ] fix converter to handle user_defined_symbols
  • [ ] create necessary flags for user_defined_symbols
  • [ ] update docs
  • [ ] test slow == fast
  • [ ] add test without sentencepiece

Who can review?

  • @ArthurZucker

itazap avatar May 21 '24 08:05 itazap

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

HuggingFaceDocBuilderDev avatar May 21 '24 08:05 HuggingFaceDocBuilderDev

We can close this as #31305 superseeds it right?

ArthurZucker avatar Jul 10 '24 12:07 ArthurZucker