Add the foundations for an extensible cost optimization recommendations pipeline framework

[helderpinto]

📝 Scenario

As a FinOps Practitioner using FinOps Hubs, I need to have a centralized list of cloud provider-generated and custom cost optimization recommendations with an historical perspective in order to be able to prioritize rate and usage optimization tasks based on potential savings and historical relevance.

💎 Solution

The Workload optimization and Rate optimization reports will include pages dedicated to cost optimization recommendations, not only coming from Azure Advisor, but also from custom recommendations based on Azure Resource Graph KQL queries (similar to the ones available in the Cost Optimization Workbook). Those pages will provide not only the latest available recommendations but also an historical perspective of how long those recommendations have been made.

📋 Tasks

Required tasks

• [x] Daily export of Azure Advisor and custom Azure Resource Graph-based recommendations into Azure Storage.
• [x] https://github.com/microsoft/finops-toolkit/issues/1412
• [ ] Hubs deployment documented manual steps to assign the required permissions to the Azure Data Factory identity.
• [ ] Implement Azure Storage and Azure Data Explorer-based Power BI pages for the Workload optimization and Rate Optimization reports.
• [ ] Update documentation with this new capability
• [ ] Deal with duplicate update policy after new Recommendations table version
• [x] https://github.com/microsoft/finops-toolkit/issues/1407

Stretch goals

• [ ] Hubs deployment automation to assign the required permissions to the Azure Data Factory identity.
• [ ] Enrich recommendations information presented in the reports by correlating with costs data.
• [ ] Document how to extend Hubs by bringing in externally generated recommendations

ℹ️ Additional context

The solution must allow for easily adding new custom recommendations in future releases.

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.