[Hubs] msexports_ExecuteETL drops ReservationDetails ingestion when manifest dataRowCount is null (manifestVersion 2024-04-01)

[haflidif]

🐛 Problem

msexports_ExecuteETL silently skips all Cost Management ReservationDetails ingestion when the manifest emits dataRowCount: null (the default for manifestVersion: 2024-04-01). The Set Has No Rows SetVariable activity sets hasNoRows = true, which causes the downstream Detect Channel switch to route to "ignore", skipping every Copy/Ingest activity. The pipeline reports Succeeded, so there's no operator signal that data was dropped.

Affects FinOps hubs v14.0 (and likely earlier v13.x — same expression is in main).

Current expression in src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/app.bicep for the Set Has No Rows activity:

@or(
  equals(activity('Read Manifest').output.firstRow.blobCount, null),
  equals(activity('Read Manifest').output.firstRow.blobCount, 0),
  and(
    contains(activity('Read Manifest').output.firstRow, 'dataRowCount'),
    or(
      equals(activity('Read Manifest').output.firstRow.dataRowCount, null),   <-- treats null as zero
      equals(activity('Read Manifest').output.firstRow.dataRowCount, 0)
    )
  )
)

manifestVersion: 2024-04-01 emits dataRowCount: null to mean "row count not pre-computed", not "zero rows". blobCount and byteCount are always populated when data exists, so it's incorrect to treat null as zero rows.

👣 Repro steps

1. Create a Cost Management export of type ReservationDetails (dataVersion: 2023-03-01) on an EA billing account scope (Microsoft.Billing/billingAccounts/{enrollment}).
2. Let it run on its daily schedule, or trigger any "Export selected dates" run.
3. Inspect storage at msexports/billingAccounts/{enrollment}/{exportName}/{YYYYMMDD-YYYYMMDD}/{run}/{guid}/manifest.json — manifest shows manifestVersion: 2024-04-01, byteCount > 0, blobCount: 1, dataRowCount: null, with a real CSV alongside.
4. In ADF, the msexports_ExecuteETL run reports Succeeded. Drill into activity outputs:
   • Set Has No Rows → hasNoRows = true
   • Detect Channel → output "ignore"
   • Copy CSV → Parquet activities — skipped
5. Confirm CommitmentDiscountUsage_raw (and therefore CommitmentDiscountUsage_final_v1_2) in ADX stays empty despite multiple succeeded export runs.

Reproduced on FinOps hubs v14.0 with dataVersion: 2023-03-01, apiVersion: 2025-03-01, EA billing account scope.

🤔 Expected

The pipeline should ingest the data because real data exists (byteCount > 0, blobCount: 1, real CSV in storage). Either:

• Remove the dataRowCount check entirely and rely on blobCount (and optionally byteCount), or
• Stop treating dataRowCount: null as 0 — only the explicit numeric value 0 should mean "no rows".

Suggested replacement expression (simplest):

@or(
  equals(coalesce(activity('Read Manifest').output.firstRow.blobCount, 0), 0),
  equals(coalesce(activity('Read Manifest').output.firstRow.byteCount, 0), 0)
)

Alternative if you want to preserve the dataRowCount short-circuit for older manifest versions:

@or(
  equals(coalesce(activity('Read Manifest').output.firstRow.blobCount, 0), 0),
  and(
    contains(activity('Read Manifest').output.firstRow, 'dataRowCount'),
    equals(activity('Read Manifest').output.firstRow.dataRowCount, 0)   // explicit zero only
  )
)

🔧 Environment

• FinOps hub version: 14.0
• Billing account type: EA (Enterprise Agreement)
• Power BI report type: ADX (Azure Data Explorer)
• Cost Management export: ReservationDetails, dataVersion: 2023-03-01, apiVersion: 2025-03-01, manifestVersion: 2024-04-01

ℹ️ Additional context

Sample failing manifest:

{
  "manifestVersion": "2024-04-01",
  "byteCount": 1972417,
  "blobCount": 1,
  "dataRowCount": null,
  "exportConfig": {
    "exportName": "...-daily-reservationdetails",
    "dataVersion": "2023-03-01",
    "apiVersion": "2025-03-01",
    "type": "ReservationDetails",
    "timeFrame": "MonthToDate",
    "granularity": "Daily"
  },
  "blobs": [
    { "blobName": ".../<file>.csv", "byteCount": 1972417 }
  ]
}

Activity outputs from a failing pipeline run:

• Set Has No Rows → hasNoRows: true
• Detect Channel → "ignore"
• All downstream Copy/Ingest activities skipped (zero data flows through)

Workaround we applied: edited the Set Has No Rows expression in ADF Studio to use blobCount + byteCount only and dropped the dataRowCount check. After publishing, we re-triggered msexports_ExecuteETL per historical month folder via az datafactory pipeline create-run. ~11 months / 21 runs reprocessed successfully and CommitmentDiscountUsage_final_v1_2 populated as expected (~40k rows so far, monthly cadence of ~5,500–6,500 rows per month, ~65–99 distinct reservations). The manual edit will be overwritten by the next Deploy-FinOpsHub, so an upstream fix would be very welcome.

Why this only affected ReservationDetails in our hub: other EA-scope exports on the same factory (FocusCost, PriceSheet, ReservationRecommendations, ReservationTransactions) kept ingesting correctly. Either their manifests still emit a numeric dataRowCount, or they omit the field entirely (which makes contains(...) false and skips the third branch).

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. If you see CommitmentDiscountUsage_raw / CommitmentDiscountUsage_final_v1_2 empty in your hub despite the export running, drop a comment with your hub version, scope type, and the dataRowCount value from one of your ReservationDetails manifests.

[microsoft-github-policy-service]

Thank you for submitting this issue! We try to respond within 3 business days. In the meantime:

• 📖 Check the troubleshooting guide for common issues
• 💬 Ask questions in Discussions
• 📅 Join our biweekly office hours for live help from the team