TobikoData
Application plan in airflow is broken
When I apply some changes in multiple models, the application plan is broken. Some expected tasks are missing and the tasks are not in the correct order. This affects the correctness of the data. I haven't managed to create a small reproduction of this, but I've met it multiple times across 0.101 and 0.104 versions.
Here I changed all my models' types to be incremental by partition instead of date (new model kind supporting bq).
The DAG is wrong. Most of the models are missing snapshots and task dependency is wrong.
Some relevant chats:
- https://tobiko-data.slack.com/archives/C044BRE5W4S/p1717611722606679
- https://tobiko-data.slack.com/archives/C044BRE5W4S/p1717749316118669
In the chat @izeigerman mentioned they suspected the sensor. I met the same problem again, tried multiple times planning prod getting the same weird behavior repeatedly. I changed the DAG leaving out the external sensor and it worked. I still don't know if this is incidental, but it may help track the issue down. Also, nobody else has this issue apart from me :P so that would be another reason to suspect the sensor.
Actually I can confirm it. While the DAG was running, I changed the dag python file to re-add the sensor and the DAG got messed up (pic 1). Removing again the sensor fixed the dag (pic 2). That's the exact same DAG in the middle of its run.