finops-toolkit
finops-toolkit copied to clipboard
Update data factory pipeline to FOCUS exports
🛠️ Description
Adds support for the new FOCUS export format in Azure Cost Management.
- Added dataset for manifest.json generated by Azure Cost Management
- Modified ingestion and msexports datasets to use blobpath rather than file and folder names.
- Modified ingestion pipeline & triggers to use manifest.json (better queueing)
- Modified managed exports feature to create FOCUS exports
- Modified ingestion ETL pipeline to convert FOCUS data
- Modified backfill pipeline to use existing monthly export rather than creating a new export (new feature in Azure Cost Management).
- Modified folder structure in ingestion container to include date hierarchy folders.
- Fixed file upload script's handling of scope object
📋 Checklist
🔬 How did you test this change?
- [x] 🤏 Lint tests
- [ ] 🤞 PS -WhatIf / az validate
- [x] 👍 Manually deployed + verified
- [ ] 💪 Unit tests
- [ ] 🙌 Integration tests
🙋♀️ Do any of the following that apply?
- [ ] 🚨 This is a breaking change.
- [ ] 🤏 The change is less than 20 lines of code.
📑 Did you update docs/changelog.md?
- [ ] ✅ Yes (required for
devPRs)- [x] ➡️ Will cover in a future PR (feature branch PRs only)
- [ ] ❎ Not needed (small/internal change)
📖 Did you update documentation?
- [ ] ✅ Public docs in
docs(required fordev)- [ ] ✅ Internal dev docs in
src(required fordev)- [x] ➡️ Will cover in a future PR (feature branch PRs only)
- [ ] ❎ Not needed (small/internal change)