astronomer
Update existing astro deploy command
Update Existing command
astro deploy
This command needs to make the following changes.
DO NOT pack the /dags in the image Implement a similar flow like above for uploading dags to Azure as a tar ball. update dag deployment status to astrohub-api via reportDagDeploymentStatus. We need to differentiate a full deployment from the dag-only deployment via this API.
More info here - https://www.notion.so/astronomerio/Engineering-Design-DRAFT-a93d9fc59c414691a4e2474cb920a465
As per the discussion on making sure we are backwards compatible
the takeaways are: for astro deploy , we will create an image without dags, and an image tar ball. If users uses this command to manage airflow deploy, they will get the option to do dag only deploy. This is the approach we recommend users to follow For backward compatibility and special user cases, we still keep the image-based deploy via astro deploy -i . However, if users opt to use this approach (customized-build image), they will not be able to do dag-only deploy
Implementation Proposal: when deploying an Airflow instance, we only add the dag downloader (sidecar) if users follow the astro deploy path (harmony change). In this way, image-based deploy will work the same as it works today.
Next Iteration: if we see a strong use case of image-based deploy after the launch, we will explore ways to enable dag only deploy for this use case. but it will take more efforts to maneuver the images and deploy process. (involved run time changes, such as, entrypoint script, store dags in a non-default location in the image etc.)
We will be doing a small POC to validate the above theory