❓Get stats (e.g. counts) about the merged pairs

[pietrolesci]

Hi there,

I was wondering whether there is an easy way to ask the tokeniser trainer to return the counts (or frequency) of the pair in the moment the merge decision is made. The ideal output would be having a new column with the frequency of the pair in the merges.txt file.

Tagging the line below as it seems the relevant part of the code that has the info (i.e., pair_counts)

https://github.com/huggingface/tokenizers/blob/25aee8b88c8de3c5a52e2f9cb6281d6df00ad516/tokenizers/src/models/bpe/trainer.rs#L465

Thanks a lot for your help!