Full vs. Incremental Loading

Open amigotune opened this issue 10 years ago • 2 comments

Hi!

It's not clear to me if Square does incremental or full loads. Is it able to do incremental ones? Where can I find more information about this?

Cheers!

Jan 07 '16 21:01 amigotune

hey @amigoricardo happy to help answer your question, but wanted to make sure I understand it a bit more: are you asking if you can use this framework to do data processing that continues to build from a given starting point or if you need to do a complete rebuild each time? We've used this to do both and the amount of data and destination structure has really been the deciding factor in picking one set up over the other. Let me know!

Jan 24 '16 23:01 jeffreyiacono

Hi! I want to sync two tables according to a specific query. Thus, I would like to Square to identify, at each run, the minimum amount of UPDATEs, INSERTs and DELETEs in order to sync them.

Jan 25 '16 10:01 amigotune