finops-toolkit icon indicating copy to clipboard operation
finops-toolkit copied to clipboard

Update data factory pipeline to FOCUS exports

Open MSBrett opened this issue 2 years ago • 0 comments

🛠️ Description

Adds support for the new FOCUS export format in Azure Cost Management.

  • Added dataset for manifest.json generated by Azure Cost Management
  • Modified ingestion and msexports datasets to use blobpath rather than file and folder names.
  • Modified ingestion pipeline & triggers to use manifest.json (better queueing)
  • Modified managed exports feature to create FOCUS exports
  • Modified ingestion ETL pipeline to convert FOCUS data
  • Modified backfill pipeline to use existing monthly export rather than creating a new export (new feature in Azure Cost Management).
  • Modified folder structure in ingestion container to include date hierarchy folders.
  • Fixed file upload script's handling of scope object

📋 Checklist

🔬 How did you test this change?

  • [x] 🤏 Lint tests
  • [ ] 🤞 PS -WhatIf / az validate
  • [x] 👍 Manually deployed + verified
  • [ ] 💪 Unit tests
  • [ ] 🙌 Integration tests

🙋‍♀️ Do any of the following that apply?

  • [ ] 🚨 This is a breaking change.
  • [ ] 🤏 The change is less than 20 lines of code.

📑 Did you update docs/changelog.md?

  • [ ] ✅ Yes (required for dev PRs)
  • [x] ➡️ Will cover in a future PR (feature branch PRs only)
  • [ ] ❎ Not needed (small/internal change)

📖 Did you update documentation?

  • [ ] ✅ Public docs in docs (required for dev)
  • [ ] ✅ Internal dev docs in src (required for dev)
  • [x] ➡️ Will cover in a future PR (feature branch PRs only)
  • [ ] ❎ Not needed (small/internal change)

MSBrett avatar Jan 31 '24 15:01 MSBrett