astro-cli icon indicating copy to clipboard operation
astro-cli copied to clipboard

DAG deployment: Astro CLI

Open kushalmalani opened this issue 3 years ago • 1 comments

CLI needs to introduce a new command and update an existing command

New command

astro deploy --dags

This command deploys DAGS only to an Airflow instance with the following flow

talk to astrohub-API via initiateDagDeployment to start a deployment process. This API will return a SAS link to Azure Blob if authorized.
generate a tar file for all files in /dag directories
upload the tar ball to azure blob via the SAS link. Refer to https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azblob#readme for Azure Blob API
report dag deployment status to astrohub-api via reportDagDeploymentStatus

Existing command

astro deploy

This command needs to make the following changes.

DO NOT pack the /dags in the image
Implement a similar flow like above for uploading dags to Azure as a tar ball.
update dag deployment status to astrohub-api via updateDagDeploymentStatus. We need to differentiate a full deployment from the dag-only deployment via this API.

kushalmalani avatar May 25 '22 21:05 kushalmalani

Original Ticket - https://github.com/astronomer/cloud-cli/issues/216

kushalmalani avatar May 25 '22 21:05 kushalmalani