SharpToken

SharpToken copied to clipboard

Cohere has two tokenizers: [chat](https://huggingface.co/Cohere/Command-nightly) [embeddings](https://huggingface.co/Cohere/multilingual-22-12) It would be great to add them to this project.

omri-suissa-clearmash

Anthropic (claude) support

6

Can we use SharpToken for Anthropic? I could not find if claude is using "cl100k_base" or other encoding

omri-suissa-clearmash

Hi, Any chance on update for Gbt4o? Thanks!

daniell0gda

Support for o200k_base and gpt-4o (omni) model

2

Notes: - I added in the new test plans for o200k based on output from tiktoken - I added a SetupO200k() in CompareBenchmark (and renamed Setup to SetupCL200k), but it's...

winzig

o200k_base support

1

It would be great to add gpt-4o o200k_base encoding support

omri-suissa-clearmash

SharpToken.GptEncoding.GetEncoding(o200k_base) takes minutes

1

Hi I am using your library in MAUI android. For some reason method `SharpToken.GptEncoding.GetEncoding(o200k_base) ` first time takes minutes to respond What can be issue?

vovanb

add support for utf8 input and output

2

add support for utf8 input and output Proposed API is ```cs public List EncodeFromUtf8(ReadOnlySpan lineToEncode, ISet allowedSpecial = null, ISet disallowedSpecial = null); public byte[] DecodeToUtf8(IEnumerable inputTokensToDecode); ```

abdulkareemnalband

Is there a way to get the vocabulary size of an initialized tokenizer from `var encoding = GptEncoding.GetEncoding(encodingName);`?

zhuzihan728

Hello, are you planning to update list of supported models? https://github.com/dmitry-brazhenko/SharpToken/blob/5e377adb6bc1113ffd3c6184f607c762bc58fc12/SharpToken/Lib/Model.cs#L64 I'm looking for model `4.1` which is already on `tiktoken`: https://github.com/openai/tiktoken/blob/3591ff175d6a80efbe4fcc7f0e219ddd4b8c52f1/tiktoken/model.py#L13 :)

lyczkop