sqlmesh icon indicating copy to clipboard operation
sqlmesh copied to clipboard

Support "external signals" for built-in scheduler

Open plaflamme opened this issue 1 year ago • 0 comments

Currently, sqlmesh supports synchronizing with externally scheduled dependencies in the Airflow scheduler via "external signals". That is, one can define an "external signal" on a sqlmesh model which allows Airflow to add a node to the DAG to represent a dependency between that model and some external state.

This issue is about extending this capability to the "built-in scheduler" as well: allowing the declaration of external dependencies which are evaluated by the build-in scheduler to determine if a model is "ready" to be evaluated within a sqlmesh run.

A public abstraction similar to BaseSensorOperator could be defined by sqlmesh, and implemented by sqlmesh users. A "factory method" (equivalent to create_external_sensor in the Airflow scheduler) could be provided to the build-in scheduler at config time (i.e.: in config.py)

plaflamme avatar Mar 18 '24 20:03 plaflamme