Fabric E2E Sample (Milestone 2)

[devlace]

Github Milestone: https://github.com/Azure-Samples/modern-data-warehouse-dataops/milestone/9 Milestone 2 target date: TBD

Automated Testing

• Integration Testing / Functional E2E testing (test/pre-prod environment - this is a permanent environment. Release is serial.)
• Unit testing of data pipelines and notebooks
• PR validation + Deployment to ephemeral pipeline (middle ground?)
  • This step takes substantial amount of time.
  • Requires a spike (can we go pure ephemeral or some are permanent; )

User Stories:

• [ ] #807
• [ ] #808
• [ ] #810
• [x] #806

Observability

• Open Telemetry standard

Python Package modernization

For Package development, update to latest / best practices. Align with Databricks + ADF update team on python package modernization.

• [ ] #811
• [x] #812

Consideration of Fabric functionality to showcases

• TBD: Functions, Python notebook

VNext Milestone ideas:

• Add streaming scenario
• Data Governance (Purview)