Hi, is this issue resolved? I've been running into the same issue. (I can confirm that it happens on 1.23 as well)
I've found that the TN FST is slow regardless of language (English too). It is not very practical with large data even using multiprocessing (normalize_list()). Any other ways to speed...
For python, I'm simply initializing the text normalizer and running it in a for loop ``` normalizer = Normalizer( input_case='cased', lang='en', whitelist='path/to/whitelist.tsv', overwrite_cache=False, cache_dir='./assets/' ) ``` for each line of...
> Do you have the "actual time estimates" for the C++ implementation `normalizer_main.cc` ? I don't have the numbers / the docker container open anymore but like I said it's...
I see let me try, thanks
I got this error trying to compile: ``` 79.22 libtool: link: g++ -g -O2 -w -std=c++11 -o .libs/normalizer_main normalizer_main.o ../lib/.libs/libsparrowhawk.so -L/usr/local/lib/fst -lthrax -lfstfar -lfst -lm -ldl -lprotobuf -l re2 79.29...
Thanks, I will try this again
> The docker container used for sparrowhawk is not optimized for production. You can build your own docker container and compile openfst-1.7.9, thrax-1.3.4 and sparrowhawk ([anand-nv/sparrowhawk@`nemo_tests`](https://github.com/anand-nv/sparrowhawk/tree/nemo_tests?rgh-link-date=2024-07-09T02%3A13%3A21Z)) with `CXXFLAGS=-g -O2` The...
Hey I searched around for these issues and resolved it with some extra configure flags. I'll start testing the speed now. Sorry I'm not well versed with C and make...
It worked! My with everything properly optimized and compiled, I got double the speed than the python implementation. The dockerfile modification for future reference: ``` RUN cd protobuf-2.5.0 && ./configure...