modern-data-warehouse-dataops icon indicating copy to clipboard operation
modern-data-warehouse-dataops copied to clipboard

Add functionality for "Config files upload to lakehouse" as part of CI/CD

Open promisinganuj opened this issue 1 year ago • 1 comments

As part of Fabric workspace integration with git, only the lakehouse metadata gets source-controlled and promoted.

To use config-driven data pipelines (CDDP), the config files are typically required to be uploaded in the "Files" section of the lakehouse. So, there has to be a mechanism to:

  • Source control these files in Azure Repo but outside the Fabric main folder.
  • Create a script to upload these files from the repo to Fabric lakehouse.
  • Include this script as part of CI/CD process.

Optional Functionality

  • Perform schema validations on these config files.
    • All the required attributes are present.

promisinganuj avatar May 21 '24 00:05 promisinganuj

The script has been added here - https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/main/single_tech_samples/fabric/fabric_ci_cd/scripts/lakehouse-file-upload/upload-file-to-lakehouse.py

Now, the CI/CD pipelines need to be updated to include this step.

promisinganuj avatar May 27 '24 10:05 promisinganuj

Closing to be succeeded by new issue to move script to utility folder

devlace avatar Oct 29 '24 20:10 devlace