Alexander Neubeck

Thanks for taking a look! It turns out that for another use case, we also needed access to the actual state id... Since we have another crate (https://github.com/github/rust-gems/tree/main/crates/bpe) which requires...

Thanks for the proposal! Here is the actual use case: https://github.com/github/rust-gems/blob/main/crates/bpe/src/appendable_encoder.rs#L39 The challenge is that we want to be able to go back to an earlier snapshot of the processing...

Hi, maybe our implementation https://github.com/github/rust-gems/tree/main/crates/bpe-openai is a solution to problematic inputs? We have found a novel algorithm to the BPE problem which has linear worst-case complexity. In our implementation, we...