Extend Nemo AutoTokenizer & SentencePieceTokenizer API for TensorRT-LLM & AMMO evaluation scripts usage

[janekl]

What does this PR do ?

Extending NeMo's AutoTokenizer and SentencePieceTokenizer for consistency with HuggingFace tokenizers API as it is used throughout TensorRT-LLM and AMMO in evaluation tools.

This is necessary to seamlessly enable model validation -- including quantized variants -- with the current scripts and to avoid code duplication.

Example use-cases to support:

• overall API link1
• encode: link1 and link2
• decode: link1
• batch_decode: link1
• batch_encode_plus link2
• setting pad_token_id: link1

Collection: NLP

Changelog

• adding different variants of encode/decode methods
• adding pad_token_id (with setter enabled for HF AutoTokenizer wrapper) and eos_token_id property attributes for HF tokenizer

Usage

See examples listed above.

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

• [x] Make sure you read and followed Contributor guidelines
• [ ] Did you write any new necessary tests?
• [ ] Did you add or update any necessary documentation?
• [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • [ ] Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• [x] New Feature
• [x] Bugfix
• [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed. Contributor guidelines contains specific people who can review PRs to various areas.

[janekl]

jenkins

[janekl]

jenkins

[aklife97]

LGTM, but can we check if autotokenizer and local sentencepiece outputs match for the new methods implemented before we merge?

it is probably fine if they're not identical, but we need to know where the differences arise and also document them if any

[github-actions[bot]]

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

[github-actions[bot]]

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

[github-actions[bot]]

This PR was closed because it has been inactive for 7 days since being marked as stale.