airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Remove execution_date and logical_date from arguments where function/API is used to look up a DAG run

Open sunank200 opened this issue 1 year ago • 0 comments

This PR removes the execution_date and logical_date arguments from functions and APIs that are used to retrieve DAG runs, aligning with the broader changes introduced in Airflow 2.2 and preparing for Airflow 3.0. The functions now use run_id as the sole identifier for DAG runs, simplifying the process and eliminating deprecated behaviour.

Motivation:

In Airflow, execution_date has historically been used to distinguish different DAG run instances. However, the introduction of run_id and the DAG run concept in Airflow 2.2 shifts away from using execution_date as an identifier. Continuing to rely on execution_date introduces limitations, such as the inability to handle multiple DAG runs at the same logical time, especially in cases like TriggerDagRunOperator when dynamic runs are generated.

By removing execution_date in favor of run_id, this PR eliminates these limitations. This also removes the unique constraint on execution_date at the database level, paving the way for a cleaner and more flexible scheduling system in Airflow 3.0.

Key Changes:

  1. API and Function Changes:

    • [ ] The execution_date and logical_date arguments have been removed from all public APIs and Python functions related to DAG run lookups.
    • [ ] run_id is now the exclusive identifier for DAG runs in these contexts.
    • [ ] Deprecation warnings for execution_date and logical_date are no longer necessary and have been removed.
  2. Database Migration:

    • [x] The unique constraint on execution_date in the database has been dropped, as run_id now ensures the uniqueness of DAG runs as part of #41818

Rationale:

Removing execution_date is necessary to enable more flexible DAG run management. For example, dynamic runs created by TriggerDagRunOperator can now be correctly identified and managed without awkward workarounds as discussed in this doc. This change makes subsequent DAG run lookups easier and more robust, while also simplifying the database schema by removing the unique constraint on execution_date.

Additionally, users will still be able to view execution_date for reference, renamed as logical_date, and paired with run_id for clarity in the web UI, making it easier to distinguish between DAG runs.


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

sunank200 avatar Sep 23 '24 05:09 sunank200