vocode-python icon indicating copy to clipboard operation
vocode-python copied to clipboard
verified publisher icon vocodedev

[Inputs Needed] Add New VectorDB PGVector

Open rahulbansal16 opened this issue 2 years ago • 9 comments

I am working on the issue.

Here is the current progress.

  1. Added the vecs SDK to query the PGVector.
  2. The functions are working as expected.
  3. I have tested them locally via the commented code in the python file.

Pending Items

  1. ~~Need inputs should we use vecs (It's new SDK from supabase) or use the ORM sqlalchemy.~~
  2. Add test for the code.
  3. ~~Make the upsert method for the add_text async.~~
  4. ~~The flow for creating indexs on the vector table.~~

rahulbansal16 avatar Dec 30 '23 10:12 rahulbansal16

Hello @rahulbansal16,

I think it's a good idea to use 'vecs', but keeping this in mind. https://github.com/supabase/vecs/issues/6

Are you in the Vocode Open Source Discord community?

arpagon avatar Jan 11 '24 02:01 arpagon

Hey @arpagon I have joined the discord.

They have the goal to make vecs async in the near future. so, I will wrap the call in a thread for a now.

And will remove the thread when vecs is done with the async support.

Let me know if you have a different approach.

rahulbansal16 avatar Jan 11 '24 16:01 rahulbansal16

I have updated the code and wrapped the insert and query method in the asyncio threads. Tested them locally and able to query and insert the data into the PGVector in Supabase.

Since it's postgres. It's recommended to create the indexes before querrying. I am not exposing the functionality to create indexes via the API. It's better to use the GUI for that.

rahulbansal16 avatar Jan 11 '24 17:01 rahulbansal16

I did not find any test for other vector db. Do I have to write test for the pg_vector?

rahulbansal16 avatar Jan 11 '24 18:01 rahulbansal16

While other vector db tests might be missing, writing tests for pg_vector is crucial for confidence, stability, and adoption. It benefits both your project and the community. Thorough tests build trust, encouraging wider use and contributions. Don't hesitate to ask for help getting started – the community is here to support you!

arpagon avatar Jan 15 '24 14:01 arpagon

@arpagon I will add them, I am occupied with other work but will be able to contribute on weekend.

rahulbansal16 avatar Jan 17 '24 16:01 rahulbansal16

Hello @rahulbansal16 I'm interested in using this patch. Is it in a stable enough state that I can use it?

rjheeta avatar May 19 '24 00:05 rjheeta

@rjheeta I have not written test for it but I was able to test the functionality and it was working.

rahulbansal16 avatar May 22 '24 13:05 rahulbansal16

Perfect, thank you. I will give this a shot.

@arpagon At a high level, what tests are you looking for in order to merge & support this change? Maybe I can try to help with writing some of these? 

rjheeta avatar May 22 '24 13:05 rjheeta