[Hubs] Start the ADX cluster when shut down

[flanakin]

📝 Scenario

As a FinOps hub admin, I need to start the ADX cluster in order to ingest data when the cluster is shut down

💎 Solution

When ingestion_ExecuteETL runs, if the ADX cluster isn't started, it will run for ~8 hours before it fails, then it needs to be manually rerun. This causes ingestion delays and extra charges for pipeline runs that don't actually ingest any data.

Instead, check if the cluster is started and start it, if needed. Starting a cluster can take 30-45m, so investigate options to re-schedule the ingestion run so the pipeline doesn't wait and incur charges during that time.

📋 Tasks

Required tasks

Preview Give feedback

1. Update ingestion_ExecuteETL to start the ADX cluster if not started
2. Document the behavior in the data processing docD

Stretch goals

Preview Give feedback

1. TODO: Add tasks

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.