Tj

Context Aware Tokenization

8

**Is your feature request related to a problem? Please describe.** Continuing this thread from https://github.com/microsoft/semantic-kernel/issues/9793#:~:text=Context%20Tokenization . Migrating from python to C# is difficult enough, supporting the same feature set but...

Batch Tokenization Support

6

**Is your feature request related to a problem? Please describe.** Most AI systems use batching for performance reasons, needing all tokenized sentences being the same size and outputting a mask...