modern-data-warehouse-dataops
modern-data-warehouse-dataops copied to clipboard
Add functionality for "Config files upload to lakehouse" as part of CI/CD
As part of Fabric workspace integration with git, only the lakehouse metadata gets source-controlled and promoted.
To use config-driven data pipelines (CDDP), the config files are typically required to be uploaded in the "Files" section of the lakehouse. So, there has to be a mechanism to:
- Source control these files in Azure Repo but outside the Fabric main folder.
- Create a script to upload these files from the repo to Fabric lakehouse.
- Include this script as part of CI/CD process.
Optional Functionality
- Perform schema validations on these config files.
- All the required attributes are present.
The script has been added here - https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/main/single_tech_samples/fabric/fabric_ci_cd/scripts/lakehouse-file-upload/upload-file-to-lakehouse.py
Now, the CI/CD pipelines need to be updated to include this step.
Closing to be succeeded by new issue to move script to utility folder