pramen
pramen copied to clipboard
Add a mode for initial ingestion of event data
Background
When an event table is ingested initially, the history can be quite long. Executing each event date one by one can have a big overhead and execute many queries against the source database.
Feature
Add a mode for initial ingestion of event data.
Example
TBD
Proposed Solution
What we can do is load full table in one go to a temporary directory (cache), and then run pipeline tasks for each date found in the database.