Handle missing costs and quantities

docs-mslearn/toolkit/changelog.md

@@ -35,6 +35,28 @@ The following section lists features and enhancements that are currently in deve
 
 <br><a name="latest"></a>
 
+## v0.10
+
+_Released April 2025_
+
+### [FinOps hubs](hubs/finops-hubs-overview.md) v0.10
+
+- **Fixed**
+  - Address new data quality issues with data ingested into Data Explorer:
+    - Fix x_EffectiveUnitPrice when it's calculated and there is a rounding error compared to x_BilledUnitPrice or ContractedUnitPrice.
+    - Calculate PricingQuantity and ConsumedQuantity when there is cost but no quantity.
+    - Set ListCost based on ContractedCost or ListUnitPrice when not specified.
+    - Replaced "-2" and "Unassigned" values in the x_InvoiceSectionId and x_InvoiceSectionName columns.
+    - Corrected x_EffectiveUnitPrice when it's calculated and has a rounding error.
+    - Add new x_SourceChanges checks for "MissingConsumedQuantity", "MissingPricingQuantity", and "XEffectiveUnitPriceRoundingError".
+
+> [!div class="nextstepaction"]
+> [Download](https://github.com/microsoft/finops-toolkit/releases/tag/v0.10)
+> [!div class="nextstepaction"]
+> [Full changelog](https://github.com/microsoft/finops-toolkit/compare/v0.9...v0.10)
+
+<br>
+
 ## v0.9 Update 1
 
 _Released April 7, 2025_
@@ -64,7 +86,7 @@ _Released April 4, 2025_
   - Added support for MCA reservation recommendation exports.
   - Added support for multiple reservation recommendation exports to support shared and single recommendations for all services and lookback periods.
   - Managed exports now create price, reservation detail, reservation transaction, and VM reservation recommendation exports.
-  - Address new data quality issues with ingested data:
+  - Address new data quality issues with data ingested into Data Explorer:
     - Change `BillingAccountId` to be lowercase in both the cost and price datasets.
     - Change `CommitmentDiscountId` to be lowercase in the cost dataset.
     - Handle `x_BillingProfileId` case-sensitivity for the cost/price join (without changing data).

docs-mslearn/toolkit/hubs/data-processing.md

@@ -106,6 +106,15 @@ Transforms:
   - Lowercase `BillingAccountId` to ensure the price join matches all rows.
   - Lowercase `CommitmentDiscountId` to avoid duplicate rows when aggregating data.
   - Add new `x_SourceChanges` checks for `ListCostLessThanContractedCost` and `ContractedCostLessThanEffectiveCost`.
+- v0.10+:
+  - Fix `x_EffectiveUnitPrice` when it's calculated and there is a rounding error compared to `x_BilledUnitPrice` or `ContractedUnitPrice`.
+  - Calculate PricingQuantity and ConsumedQuantity when there is cost but no quantity.
+  - Set `ContractedCost` to `EffectiveCost` when it's not set.
+  - Set `ListCost` to `ContractedCost` when it's not set.
+  - Remove "-2" in the `x_InvoiceSectionId` column.
+  - Remove "Unassigned" in the `x_InvoiceSectionName` column.
+  - Corrected `x_EffectiveUnitPrice` when it's calculated and has a rounding error.
+  - Add new `x_SourceChanges` checks for `MissingConsumedQuantity`, `MissingPricingQuantity`, and `XEffectiveUnitPriceRoundingError`.
 
 ### Price data transforms
 

src/templates/finops-hub/modules/scripts/IngestionSetup.kql

@@ -780,10 +780,15 @@ Costs_transform_v1_0()
         iff(ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountId) and EffectiveCost > 0,                                'InvalidEffectiveCost,',             ''),
         iff((isempty(ContractedCost)      or ContractedCost == 0)      and EffectiveCost != 0,                                      'MissingContractedCost,',            ''),
         iff((isempty(ContractedUnitPrice) or ContractedUnitPrice == 0) and x_EffectiveUnitPrice != 0,                               'MissingContractedUnitPrice,',       ''),
-        iff(ListCost < ContractedCost,                                                                                              'ListCostLessThanContractedCost,',      ''),
+        iff(ListCost < ContractedCost,                                                                                              'ListCostLessThanContractedCost,',   ''),
         iff(ContractedCost < EffectiveCost,                                                                                         'ContractedCostLessThanEffectiveCost,', ''),
         iff((isempty(ListCost)            or ListCost == 0)            and (ContractedCost != 0      or EffectiveCost != 0),        'MissingListCost,',                  ''),
         iff((isempty(ListUnitPrice)       or ListUnitPrice == 0)       and (ContractedUnitPrice != 0 or x_EffectiveUnitPrice != 0), 'MissingListUnitPrice,',             ''),
+        iff((isnotempty(x_BilledUnitPrice) and x_BilledUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice) and x_EffectiveUnitPrice != 0 and abs(x_BilledUnitPrice - x_EffectiveUnitPrice) < 0.0001)
+            or (isnotempty(ContractedUnitPrice) and ContractedUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice) and x_EffectiveUnitPrice != 0 and abs(ContractedUnitPrice - x_EffectiveUnitPrice) < 0.0001),
+            'XEffectiveUnitPriceRoundingError,',  ''),
+        iff(ConsumedQuantity == 0 and (EffectiveCost != 0 or BilledCost != 0),                                                      'MissingConsumedQuantity,',              ''),
+        iff(PricingQuantity == 0 and (EffectiveCost != 0 or BilledCost != 0),                                                       'MissingPricingQuantity,',              ''),
         iff(isempty(ProviderName),                                                                                                  'MissingProviderName,',              ''),
         iff(isempty(PublisherName),                                                                                                 'MissingPublisherName,',             ''),
         iff(ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountId) and isempty(ResourceId),                              'MissingResourceId,',                ''),
@@ -818,6 +823,20 @@ Costs_transform_v1_0()
         strcat(x_SourceChanges, iff(isempty(x_SourceChanges), '', ','), iff(x_SourceVersion == '', 'UnknownFocusVersion', 'LegacyFocusVersion'))
     )
     //
+    // Fix quantities
+    | extend PricingQuantity  = case(
+        PricingQuantity != 0 or (EffectiveCost == 0 and BilledCost == 0), PricingQuantity,
+        PricingQuantity == 0 and isnotempty(BilledCost) and BilledCost != 0 and isnotempty(x_BilledUnitPrice) and x_BilledUnitPrice != 0, BilledCost / x_BilledUnitPrice,
+        PricingQuantity == 0 and isnotempty(BilledCost) and BilledCost != 0 and isnotempty(x_BilledUnitPrice) and x_BilledUnitPrice != 0, BilledCost / x_BilledUnitPrice,
+        PricingQuantity == 0 and isnotempty(EffectiveCost) and EffectiveCost != 0 and isnotempty(x_EffectiveUnitPrice) and x_EffectiveUnitPrice != 0, EffectiveCost / x_EffectiveUnitPrice,
+        PricingQuantity
+    )
+    | extend ConsumedQuantity = case(
+        isnotempty(ConsumedQuantity) and ConsumedQuantity != 0, ConsumedQuantity,
+        ChargeCategory == 'Usage', PricingQuantity / coalesce(x_PricingBlockSize, 1),
+        ConsumedQuantity
+    )
+    //
     // Populate missing prices -- mapping to on-demand prices requires meter ID and offer ID
     | extend tmp_MissingPrices = ProviderName == 'Microsoft'
         and (ListUnitPrice == 0 or ContractedUnitPrice == 0)
@@ -834,11 +853,19 @@ Costs_transform_v1_0()
         | where x_SkuPriceType == 'Consumption' and tmp_ReservationPriceLookupKey in ((costsWithMissingPrices | summarize by tmp_ReservationPriceLookupKey))
         | summarize ListUnitPrice = min(ListUnitPrice), ContractedUnitPrice = min(ContractedUnitPrice) by tmp_ReservationPriceLookupKey, x_PricingBlockSize, PricingUnit
     ) on tmp_ReservationPriceLookupKey
+    //
     // Select the best price to use for each row
-    // TODO: Save values before changing -- | extend x_ms_ContractedUnitPrice = ContractedUnitPrice, x_ms_ListUnitPrice = ListUnitPrice, x_ms_ListCost = ListCost, x_ms_ContractedCost = ContractedCost
+    // TODO: Save values before changing -- | extend x_old_ContractedUnitPrice = ContractedUnitPrice, x_old_EffectiveUnitPrice = x_EffectiveUnitPrice, x_old_ListUnitPrice = ListUnitPrice, x_old_ListCost = ListCost, x_old_ContractedCost = ContractedCost
+    | extend x_EffectiveUnitPrice = case(
+        // If price is a rounding error away from the billed price, use the billed price
+        isnotempty(x_BilledUnitPrice) and x_BilledUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice) and x_EffectiveUnitPrice != 0 and abs(x_BilledUnitPrice - x_EffectiveUnitPrice) < 0.0001, x_BilledUnitPrice,
+        // If price is a rounding error away from the contracted price, use the contracted price
+        isnotempty(ContractedUnitPrice) and ContractedUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice) and x_EffectiveUnitPrice != 0 and abs(ContractedUnitPrice - x_EffectiveUnitPrice) < 0.0001, ContractedUnitPrice,
+        x_EffectiveUnitPrice
+    )
     | extend ContractedUnitPrice = case(
         // If price is already correct, keep that
-        ContractedUnitPrice != 0 or x_EffectiveUnitPrice == 0, ContractedUnitPrice,
+        (isnotempty(ContractedUnitPrice) and ContractedUnitPrice != 0) or (EffectiveCost == 0 and BilledCost == 0), ContractedUnitPrice,
         // If both prices use the same scale, use the new one
         PricingUnit == PricingUnit1 and x_PricingBlockSize == x_PricingBlockSize1, ContractedUnitPrice1 * x_BillingExchangeRate,
         // If prices are the same unit but not the same scale, use the new one but correct the scale
@@ -850,7 +877,7 @@ Costs_transform_v1_0()
     )
     | extend ListUnitPrice = case(
         // If price is already correct, keep that
-        ListUnitPrice != 0 or x_EffectiveUnitPrice == 0, ListUnitPrice,
+        (isnotempty(ListUnitPrice) and ListUnitPrice != 0) or (EffectiveCost == 0 and BilledCost == 0), ListUnitPrice,
         // If both prices use the same scale, use the new one
         PricingUnit == PricingUnit1 and x_PricingBlockSize == x_PricingBlockSize1, ListUnitPrice1 * x_BillingExchangeRate,
         // If prices are the same unit but not the same scale, use the new one but correct the scale
@@ -860,17 +887,31 @@ Costs_transform_v1_0()
     )
     // Calculate missing costs based on new prices -- If cost is already correct, keep that; if not and price is available, recalculate the cost; otherwise, keep the existing cost
     | extend ContractedCost = case(
-        ContractedCost != 0 or EffectiveCost == 0, ContractedCost,
-        ContractedUnitPrice == x_EffectiveUnitPrice, EffectiveCost,  // Use EffectiveCost if both prices are the same to avoid rounding errors
-        ContractedUnitPrice != 0, ContractedUnitPrice * PricingQuantity,
-        ContractedCost == 0 and ContractedUnitPrice == 0, EffectiveCost,  // Fall back to EffectiveCost when ContractedCost and ContractedUnitPrice are 0
+        // If not set or there's no cost, keep the original value
+        (isnotempty(ContractedCost) and ContractedCost != 0) or (EffectiveCost == 0 and BilledCost == 0), ContractedCost,
+        // ContractedCost is 0 in all other scenarios...
+        // If 0 and there's a billed cost and prices are the same, use BilledCost
+        isnotempty(BilledCost)    and BilledCost    != 0 and ContractedUnitPrice == x_BilledUnitPrice,    BilledCost,
+        // If 0 and there's a billed cost and prices are the same, use EffectiveCost
+        isnotempty(EffectiveCost) and EffectiveCost != 0 and ContractedUnitPrice == x_EffectiveUnitPrice, EffectiveCost,
+        // If 0 and there's a price, calculate the cost based on the price
+        isnotempty(ContractedUnitPrice) and ContractedUnitPrice != 0, ContractedUnitPrice * PricingQuantity,
+        // If 0 and there's no price, assume EffectiveCost
+        isempty(ContractedUnitPrice) or ContractedUnitPrice == 0, EffectiveCost,
+        // Fall back to the original value for any unhandled scenarios
         ContractedCost
     )
     | extend ListCost = case(
-        ListCost != 0 or EffectiveCost == 0, ListCost,
-        ListUnitPrice == ContractedUnitPrice, ContractedCost,  // Use ContractedCost if both prices are the same to avoid rounding errors
-        ListUnitPrice != 0, ListUnitPrice * PricingQuantity,
-        ListCost == 0 and ListUnitPrice == 0, ContractedCost,  // Fall back to ContractedCost when ListCost and ListUnitPrice are 0
+        // If not set or there's no cost, keep the original value
+        (isnotempty(ListCost) and ListCost != 0) or (EffectiveCost == 0 and BilledCost == 0), ListCost,
+        // ListCost is 0 in all other scenarios...
+        // If 0 and there's a contracted cost and prices are the same, use ContractedCost
+        isnotempty(ContractedCost) and ContractedCost != 0 and ListUnitPrice == ContractedUnitPrice, ContractedCost,
+        // If 0 and there's a price, calculate the cost based on the price
+        isnotempty(ListUnitPrice) and ListUnitPrice != 0, ListUnitPrice * PricingQuantity,
+        // If 0 and there's no price, assume ContractedCost
+        isempty(ListUnitPrice) or ListUnitPrice == 0, ContractedCost,
+        // Fall back to the original value for any unhandled scenarios
         ListCost
     )
     // Merge the rest of the unmodified cost records and remove excess columns
@@ -1012,8 +1053,14 @@ Costs_transform_v1_0()
         x_IngestionTime,             // Hubs add-on
         x_InvoiceId,                 // Azure 1.0-preview(v1)+
         x_InvoiceIssuerId,           // Azure 1.0-preview(v1)+
-        x_InvoiceSectionId,          // Azure 1.0-preview(v1)+
-        x_InvoiceSectionName,        // Azure 1.0-preview(v1)+
+        x_InvoiceSectionId = case(   // Azure 1.0-preview(v1)+
+            x_InvoiceSectionId == '-2', '',
+            x_InvoiceSectionId
+        ),
+        x_InvoiceSectionName = case(   // Azure 1.0-preview(v1)+
+            x_InvoiceSectionName == 'Unassigned', '',
+            x_InvoiceSectionName
+        ),
         x_ListCostInUsd,             // Azure 1.0-preview(v1)+
         x_Location,                  // GCP Jan 2024
         x_Operation,                 // AWS 1.0