SPTAG icon indicating copy to clipboard operation
SPTAG copied to clipboard

double free or corruption (out) during Search

Open darae-lee opened this issue 1 year ago • 1 comments

I want to evaluate SPTAG on the streaming track of the big-ann-benchmarks. (https://github.com/harsha-simhadri/big-ann-benchmarks/blob/main/neurips23/README.md) However, during the Search process, there was a double free or corruption (out) error. (It has to process 10K queries, but an error occurs in the middle)

Is there a problem with memory allocation and deallocation in the library? Or do I need to set some parameters?

Below is the benchmark code.

    def setup(self, dtype, max_pts, ndim):
        self.index = SPTAG.AnnIndex("BKT", "Float", ndim)

        self.index.SetBuildParam("NumberOfThreads", str(self.insert_threads), "Index")
        self.index.SetBuildParam("DistCalcMethod", self.translate_dist_fn(self._metric), "Index")
        self.max_pts = max_pts
        print('Index class constructed and ready for update/search')
    
    def insert(self, X, ids):
        self.index.SetBuildParam("NumberOfThreads", str(self.insert_threads), "Index")
        self.index.SetBuildParam("DistCalcMethod", self.translate_dist_fn(self._metric), "Index")
        p_meta = ''
        for i in ids:
            p_meta += str(i+1) + '\n'
        p_meta.encode()
        res = self.index.AddWithMetaData(X, p_meta, X.shape[0], True, False)

    def delete(self, ids):
        p_meta = ''
        for i in ids:
            p_meta += str(i+1) + '\n'
        p_meta.encode()

        self.index.DeleteByMetaData(p_meta)

    def query(self, X, k):
        result1 = []
        result2 = []
                
        for t in range(X.shape[0]):
            result = self.index.SearchWithMetaData(X[t], k)
            result1.append(result[2])
            result2.append(result[1])

        self.res = np.array(result1).reshape(-1, 10)
        self.query_dist = np.array(result2).reshape(-1, 10)

darae-lee avatar May 20 '24 07:05 darae-lee

i have the same problem, have you fixed it?

wangchengdng avatar Sep 10 '24 09:09 wangchengdng