Add max number of token to count to the TokenCount function

[pgroene]

Is your feature request related to a problem? Please describe.

When processing large inputs, I need to count the tokens to ensure they do not exceed the maximum allowed for my request before sending it. Currently, the existing interfaces require counting all tokens in the input, even if I only need to know if the count exceeds a specific number. This results in unnecessary processing. It would be more efficient to have the ability to stop counting once a certain threshold is reached.

Describe the solution you'd like

Introduce an optional parameter in the CountTokens method that allows setting a maximum number of tokens to count. This way, the counting process can terminate early if the specified limit is reached, saving processing time and resources. This parameter should be added to the existing parameters for the counting method

Describe alternatives you've considered

Additional context

[tarekgh]

I want to highlight that the requested functionality can be achieved using the following API, which lowers the priority of this request.

public int GetIndexByTokenCount(string text, int maxTokenCount, out string? normalizedText, out int tokenCount, bool considerPreTokenization = true, bool considerNormalization = true)