toluaina
PGsync does not Immediately send document to Elastic Index
PGSync version: 2.2.1
Problem Description: Increase Indexing speed
@toluaina , the Sync is working fine, just that documents sometimes take 5-10 seconds to show up in the Index. Is there any way to increase the Speed.
Seems like Redis holds to documents for some time before sending them to the index. Any suggestions?
Yes, you can try:
- increasing
NTHREADS_POLLDB(default is 1) - reducing
POLL_TIMEOUT(default is 0.1)
These are defined here
Thank you @toluaina , I will get back to you after testing it out.
Hi @toluaina ,
Thank you for the PGSync. The sync works fine though we have an issue regarding sync speed.
We have around 360k documents in ES and it takes around ~5mins. to sync new inserts. Do you think, the amount of data could be the issue for this delay?? We tried increasing nthreads_polldb=10 and reducing poll_timout=0.05 but nothing changes.
@chintal31
are you suggesting you are seeing no difference at all ? Can you confirm which version of pssync you are running otherwise can you try the latest or use the master branch.
10 threads is a lot. Please start from 4.
@chintal31
- I have been working on an async feature branch.
- You can pull the branch here.
- It is not ready for production use yet.
- but if you notice a performance improvement I can put more effort into completing it
thanks.
PS: I have integrated the async branch into the master branch So you can just pull master and enable async mode with the environment variable
USE_ASYNC=True
Hi @toluaina, we have done the changes you mentioned. We can definitely see improvements in syncing but there are times when it shows delay. Even when we updating a particular document the updated values also takes time(Not always though).
Is there anything else on the we can do on the Infra side? like polling or threading which can be tweaked to achieve optimum sync speed?