haystack
haystack copied to clipboard
FAISS in OpenSearch: Support Product Quantization
As a user I want to use FAISS's product quanitzation feature with OpenSearchDocumentStore.
According to https://opensearch.org/docs/latest/search-plugins/knn/knn-index/ FAISS plug-in has two param for pq:
-
m: number of sub vectors to split the original vector -
code_size: the number of bits to encode a sub-vector into
We only support setting m as code_size is only relevant for IVF, static for HNSW.
- [ ] add param
product_quantization_subvectorswith default valueNone - [ ] if
product_quantization_subvectorsis notNone, set pq params with m=product_quantization_subvectors - [ ] add check whether existing index has requested setting