haystack
haystack copied to clipboard
Support gRPC interface of TEI
The gRPC interface of HuggingFace's Text Embeddings Inference (TEI) library is much, much faster than its HTTP interface. It would be fantastic to see Haystack add support for it.
https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#grpc
Nice idea!
I had a look and seems non-trivial because we are currently using the huggingface_hub library to send requests to TEI and this library does not seem to support gRPC.
Let's see if they will introduce this feature...