jpughcs
jpughcs
Seems to work fine with a single schema.
Scratch that, problem remains even with single schema ``` Syncing db Xlog: [22] => Db: [3,142] => Redis: [total = 1,001 pending = 2,141] => Elastic: [26,680] ... Syncing db...
Yes, this happens always. Let me know what you'd like to patch in for logging.
Absolutely. I will get this for you ASAP, likely tomorrow morning (evening for you). Thank you!
Attached is a full run. You can see there's one tick with `bulk_pop nsize: 1000` and then `bulk_pop` never seems to be called again. [log.txt](https://github.com/toluaina/pgsync/files/7952843/log.txt)
Scratch that, `bulk_pop` is being called again.... but almost exactly at 15 minutes intervals...
Here is my Env config ``` ELASTICSEARCH_TIMEOUT=60 ELASTICSEARCH_TIMEOUT=600 ELASTICSEARCH_QUEUE_SIZE=4 ELASTICSEARCH_THREAD_COUNT=4 QUERY_CHUNK_SIZE=5000 ELASTICSEARCH_CHUNK_SIZE=1000 ELASTICSEARCH_STREAMING_BULK=False REDIS_POLL_INTERVAL=0.01 ```
Just to note, I've experimented a bit with the configuration without luck. Even with `REDIS_CHUNK_SIZE=10000` it seems that the `bulk_pop`, processing of the records, and insertion of the documents in...
This is being run in docker. The interval appears to change proportionally with `REDIS_CHUNK_SIZE` I will experiment changing `REDIS_POLL_INTERVAL`
Changing `REDIS_POLL_INTERVAL` does not seem to make a difference. I am seeing a large amount of these `POST` and `DELETE` requests (seen below) that happen in between `bulk_pop`. I am...