pramen icon indicating copy to clipboard operation
pramen copied to clipboard

Add a mode for initial ingestion of event data

Open yruslan opened this issue 2 years ago • 0 comments

Background

When an event table is ingested initially, the history can be quite long. Executing each event date one by one can have a big overhead and execute many queries against the source database.

Feature

Add a mode for initial ingestion of event data.

Example

TBD

Proposed Solution

What we can do is load full table in one go to a temporary directory (cache), and then run pipeline tasks for each date found in the database.

yruslan avatar Feb 02 '24 08:02 yruslan