Fix: Take batch_size into account when filling snapshots for INCREMENTAL_BY_UNIQUE_KEY models
This PR makes the batch_index available when filling snapshots, so EvaluationStrategy's can take it into account when making decisions.
Prior to this, for kind's like INCREMENTAL_BY_UNIQUE_KEY, if:
- You had a model that defines intervals (say
@daily) and runs them in multiple batches (say,batch_size=1, which generates aSnapshotper interval) - When running the
Snapshot's, SQLMesh would treat them all as "clear table -> insert data" rather than just the first one
After this change, for Snapshot's with no existing intervals at plan time, SQLMesh can check the index in the batch and just execute the "clear table" logic for the first snapshot and then the "merge into existing table" logic for subsequent snapshots
Fixes #2609
@izeigerman I had a go at implementing your suggestions on the Airflow scheduler code but I ran out of time to add a test case that proves I implemented them correctly.
I'm back on Monday and can resume this up then; otherwise feel free to finish it off if this bug is blocking someone