Azure Data Factory - First ingestion_ETL_dataExplorer pipeline of ingestion_ExecuteETL will fail for daily export, subsequent runs succeed though

[ggzola]

🐛 Problem

ADF pipeline 'ingestion_ExecuteETL' is failing daily. Specifically, the first 'ingestion_ETL_dataExplorer' run will fail, but every run after will succeed. The error messages are also not helpful.

• Pipeline errors: Operation on target For Each Old File failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Ingestion Failed Error failed: Failed to interpret Ingestion Failed Error fail message or error code
• Activity errors:
  • Until Capacity Is Available: Activity failed because an inner activity failed
  • If has capacity: Activity failed because an inner activity failed
  • Ingest Data:
  • Ingestion Failed error: Failed to interpret Ingestion Failed Error fail message or error code
  • Pipeline output:

{
	"message": "Failed to interpret Ingestion Failed Error fail message or error code",
	"errorCode": "ErrorCodeNotString"
} 

👣 Repro steps

1. Deploy FinOps (0.7) with Azure Data Explorer Cluster
2. Grant the Azure Data Factory's Managed Identity "Department Reader" access at the Enterprise Agreement level
3. Configure the scope to the Enterprise Agreement billing account
4. Configure daily export of cost data (manual)
5. See failed pipeline run in ADF monitor

🤔 Expected

'ingestion_ExecuteETL' should not fail daily. Also, more detailed error messages should be provided.

📷 Screenshots

Putting this here to show the first job always points to the 0th parquet file (first file in the manifest).

ℹ️ Additional context

Daily export has been succeeding since the start of the month but a few days ago begun this behavior.

Also I have confirmed that data is missing in ADX.

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.

[ggzola]

Did a test where I renamed the files so they would execute in a different order. From the image the first job was a success. This infers there is something wrong with the first file generated by the export, and probably not an issue with the pipeline.

[flanakin]

This is strange. My first thought is to file a support request with Cost Management, but I have to be honest I don't know what to tell them. They don't support our pipeline and we don't know what's going on with the file right now 🤔

Can you open the failing file with a parquet reader to see if it's opening okay?

#needs-info

[ggzola]

No problem opening the file that failed versus a successful one, and the metadata is identical sans what you would expect.

[MSBrett]

Two things - first, is this for a recommendation or a cost import? I assume cost?
Second, could you run this command against the Ingestion database in Kusto:

.show ingestion failures