haystack icon indicating copy to clipboard operation
haystack copied to clipboard

Support gRPC interface of TEI

Open mhillebrand opened this issue 1 year ago • 2 comments

The gRPC interface of HuggingFace's Text Embeddings Inference (TEI) library is much, much faster than its HTTP interface. It would be fantastic to see Haystack add support for it.

https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#grpc

mhillebrand avatar Apr 09 '24 20:04 mhillebrand

Nice idea!

I had a look and seems non-trivial because we are currently using the huggingface_hub library to send requests to TEI and this library does not seem to support gRPC.

Let's see if they will introduce this feature...

anakin87 avatar Apr 10 '24 06:04 anakin87