Ben Brandt
Ben Brandt
Have this would be amazing. Any progress on merging this?
Hmm ok I am traveling today but I will try and take a look this weekend. It does seem strange, and I'm not sure if there is something about loading...
Hmm also the rust crate assumes the entire string is in memory. The rust crate at least only ever uses references to the single string, but on the Python side...
Ok good to know, just wasn't something I had tried before so I wasn't sure. Thanks!
@do-me So I have been doing some tests. If I bump the chunk size up enough, it does finish, and it does some to progress even at smaller sizes, including...
@do-me just want to give you a heads up so you know I didn't forget. I've been trying dozens of implementations, all with different tradeoffs, but I think I found...
Hi @do-me I have been thinking about this, and actually I think what you are doing is exactly what I would do as well. The issue is, because of how...
v0.19.1 added a `chunk_all` on the Python side that should support this use case. Going to mark as closed for now.
Hi @do-me thanks for this. I had this more scoped here: https://github.com/users/benbrandt/projects/2?pane=issue&itemId=57419637 Basically this issue is more that the current algorithm is "greedy", but if we know we are within...
That is because I haven't really scoped it out yet 😆 I don't think it is out-of-scope, but will likely be an opt-in. I think I would provide both options...