astro-cli
astro-cli copied to clipboard
DAG deployment: Astro CLI
CLI needs to introduce a new command and update an existing command
New command
astro deploy --dags
This command deploys DAGS only to an Airflow instance with the following flow
talk to astrohub-API via initiateDagDeployment to start a deployment process. This API will return a SAS link to Azure Blob if authorized.
generate a tar file for all files in /dag directories
upload the tar ball to azure blob via the SAS link. Refer to https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azblob#readme for Azure Blob API
report dag deployment status to astrohub-api via reportDagDeploymentStatus
Existing command
astro deploy
This command needs to make the following changes.
DO NOT pack the /dags in the image
Implement a similar flow like above for uploading dags to Azure as a tar ball.
update dag deployment status to astrohub-api via updateDagDeploymentStatus. We need to differentiate a full deployment from the dag-only deployment via this API.
Original Ticket - https://github.com/astronomer/cloud-cli/issues/216