Create Baseline Release Pipeline

[devlace]

Create a simple baseline release pipeline that triggers of commits to main branch.

See here for potential reuse: https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/promisinganuj/722-observability-documentation/single_tech_samples/fabric/fabric_ci_cd/devops/azdo-fabric-cd-release.yml

DoD:

• [ ] AzDO yaml pipeline definition that triggers off commits to dev, stg, prod.
• [ ] Performs a git sync of the main branch to the corresponding (dev, stg, prod) workspace
• [ ] Updates any remaining items not supported by git sync (shortcuts, custom pool, environment)

[camaderal]

Here are some notes on ideas we have for the design of release/deploy cd pipeline.

• devops/azure-pipelines-ci-artifacts.yml ⇨ triggered on commit/merge to main/dev branch

  • Publish Fabric Environment Config Artifacts

    • Output

    fabric_env
    |_environment.yaml
    |_custom_libraries
       |_ddo_transform_standardize.py
       |_ddo_transform_transform.py
       |_otel_monitor_invoker.py
    

  • Publish ADLS Artifacts (create for dev, staging and prod environments)

    • Output

    adls
    |_config
       |_application.cfg
       |_lakehouse_ddls.yaml
    |_reference
       |_dim_date.csv
       |_dim_time.csv
    

• devops/azure-pipelines-cd-release.yml ⇨ triggered on completion of azure-pipelines-ci-artifacts pipeline.

  • Stages:
    • Deploy to Dev
    • — Run Test? + Approval —
    • Merge dev branch to staging branch
    • Deploy to Staging
    • Merge staging branch to prod branch
    • — Run Test? + Approval —
    • Deploy to Prod

• Deploy process:

  • Specify appropriate variable group based on where to deploy
  • Sync the workspace to the latest workspace files:
    • Sync fabric workspace with the fabric workspace folder in AzureDevOps
    • Discussion Point: Should we check for conflicts or always prefer remote?
  • Update Environment from Fabric Environment Config Artifacts
    • Modify and re-use devops/scripts/update_environment.py
      • Use the output of Publish Fabric Environment Config Artifacts for uploading the libraries (environment.yaml, custom libraries)
      • Delete the current libraries and re-upload (because there is no versioning)
    • Discussion Point: Updating the environment would result in git changes. Should we commit fabric workspace to git after updating environment? If not, it will always result in conflict in next deploy.
  • Upload ADLS Artifacts to storage container
    • Just use the output of Publish ADLS Artifacts for re-uploading the contents of ADLS storage container.

Something to think about:

• How is rollback handled? Might be useful to have the release pipeline do the deploy process on not only the latest version but to deploy a specific version. (Syncing workspace items on a specific commit hash, uploading fabric env artifacts of a specific version, re-uploading ADLS artifacts of a specific release version)