Skip to content

Fix backstream data for private producers and add consumption since last invoice sensor - #508

Merged
GuyKh merged 8 commits into
mainfrom
fix/backstream-meter-kind-from-contract
Aug 10, 2026
Merged

Fix backstream data for private producers and add consumption since last invoice sensor#508
GuyKh merged 8 commits into
mainfrom
fix/backstream-meter-kind-from-contract

Conversation

@GuyKh

@GuyKh GuyKh commented Aug 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

Fixes #507 — backstream (energy production) data is no longer available or is zeroed in sensors. Also adds a new Consumption since last invoice sensor.

Root causes

Two separate problems caused backstream/export data to be missing or zeroed:

1. Meter kind was hardcoded to "Consumption"
Commit 808b967 (#465) hardcoded meter_kind="Consumption" for every device when working around the DeviceIn endpoint's recaptcha requirement. For cookie-less API clients like Home Assistant, the request's meterKind is the gate that determines whether backstream/export data is returned:

  • Request meterKind:"Consumption" → response has meterKind:1, all backstream/export fields zeroed
  • Request meterKind:"Backstream" → response has meterKind:2 with real backstream/export data (byte-identical to what the IEC web app receives)

2. Zeroed export data on monthly readings
Even with the correct meter kind, the IEC API returns zeroed export data for the current period unless lastInvoiceDate is the day after the last invoice. The coordinator only sent last_invoice_date on the first-of-month/Sunday path, so the regular monthly "latest reading" request sent the period start as lastInvoiceDate and the API zeroed out all export fields.

Changes

data_fetcher.py — _get_devices_by_contract_id no longer hardcodes meter_kind="Consumption". New _get_meter_kind_for_contract() derives the meter kind from the contract's from_private_producer flag, fetched cookie-less via get_contracts() (cached per refresh cycle; on IECError falls back to "Consumption"). The DeviceIn endpoint cannot be used here — that's exactly why #465 replaced it.

bill.py —

  • _map_meter_kind_to_remote_reading_param now normalizes Hebrew/English inputs to the exact API casing the IEC web app sends: "דו כיווני" / "דו-כיווני" → "Backstream" (was "BackStream" — capital S, which never matches the API), "צריכה" → "Consumption", numeric values (1 = Consumption, 2 = Backstream), and unknown values default to "Consumption".
  • _get_invoice_reading_dates now derives lastInvoiceDate from the most recent invoice's fullDate + 1 day (IEC's requirement for non-zeroed export data) instead of the lastDate field.
  • New _needs_future_consumption_fallback and _future_consumption_candidate_dates helpers.

coordinator.py —

  • Backstream meters are now classified from the request-side device.meter_kind (the contract's private-producer flag). The response meterKind flips between 1 and 2 for the same meter depending on the queried period, so it is not a reliable indicator of a bidirectional meter.
  • MONTHLY readings always send the invoice-derived lastInvoiceDate, keeping fromDate as the first of the current month.
  • The future-consumption fallback (_probe_future_consumption) walks progressively older windows (last 3 days → last completed week → first of month), skipping zero-export windows for backstream meters, instead of a single two-days-ago monthly fetch.

sensor.py — new meter-level Consumption since last invoice sensor (elec_consumption_since_last_invoice) reading future_consumption, mirroring the existing backstream production sensor. Translations exist in both en.json and he.json.

Tests — updated and extended (tests/test_bill.py, tests/coordinator/test_invoice_reading_dates.py) for the new casing/defaults, the fullDate + 1 contract, and the fallback helpers.

Test plan

  • Ruff lint passes (./scripts/lint)
  • Unit tests pass (pytest tests/ — 87 passed)
  • MyPy type check passes
  • Verified against a real private-producer (solar) contract: the monthly request with lastInvoiceDate=2026-07-02 and fromDate=2026-08-01 returns real export data (futureBackStream 3106.799, totalExport 39225.556), matching the IEC web app's displayed "Production: 3107" and "Last read: 39226".

@qodo-code-review

qodo-code-review Bot commented Aug 9, 2026

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@GuyKh GuyKh changed the title Fix backstream data unavailable for private producers (#507) Fix backstream data unavailable for private producers Aug 9, 2026
@GuyKh GuyKh changed the title Fix backstream data unavailable for private producers Fix backstream data for private producers and add consumption since last invoice sensor Aug 9, 2026
GuyKh and others added 8 commits August 10, 2026 09:13
@GuyKh @sisyphus-dev-ai
Fix meter kind mapping: use API 'Backstream' casing, default unknown …
dd81b15 
…to Consumption

Backstream detection for issue #507: normalize Hebrew/English meter kind inputs to the exact API casing ("Backstream", lowercase s — matching what the IEC web app sends) and default unknown values to "Consumption" instead of passing them through. Also accept the hyphenated "דו-כיווני" variant in _is_backstream_meter_kind.

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@GuyKh @sisyphus-dev-ai
Derive device meter kind from contract private-producer flag
90265e6 
Fix issue #507: commit 808b967 hardcoded meter_kind="Consumption" for every device, so backstream/export data was never returned for private producers (cookie-less API requests only return backstream when meterKind is "Backstream"). Replace the hardcoded value with a meter kind derived from the contract's from_private_producer flag (fetched cookie-less via get_contracts, cached per refresh cycle, IECError falls back to "Consumption").

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@GuyKh @sisyphus-dev-ai
Unify meter kind mapping: delegate backstream check to param mapper
45086d8 
DRY for issue #507: _is_backstream_meter_kind now delegates to _map_meter_kind_to_remote_reading_param, making the classification and the request parameter share a single source of truth. The mapping function learns the API's numeric meterKind values (1 = Consumption, 2 = Backstream) so int/enum inputs classify identically to strings.

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@GuyKh @sisyphus-dev-ai
Fix backstream meter classification: use request-side meter kind
84a4eae 
The API's response meterKind is not a stable meter property: for the same meter it returns 2 when export data exists for the queried period and 1 when it does not (observed on contract 348435866: Backstream request for 2026-08-05 returned meterKind 2 with totalExport 39225, same request for 2026-08-06 returned meterKind 1 with totalExport 0). Classifying backstream_meters from that field made backstream sensors flicker between refreshes. Classify instead from the request-side device.meter_kind, derived from the contract's from_private_producer flag, which is deterministic.

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@GuyKh @sisyphus-dev-ai
Add fallback probe helpers for zeroed IEC export data
14cba3b 
The IEC API zeroes backstream (export) fields in the current period's aggregations even for bidirectional meters. _needs_future_consumption_fallback triggers a fallback when a backstream meter's monthly export came back empty; _future_consumption_candidate_dates lists the reading windows to probe, newest first: the last three days, the most recent completed week (last Sunday; two Sundays ago on Sunday), and the first of the current month (previous month on the 1st). Ref issue #507.

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@GuyKh @sisyphus-dev-ai
Probe older reading windows for zeroed backstream totals
46f0dc9 
When the fallback is needed (missing future consumption, or a backstream meter whose monthly export is zeroed), walk the candidate reading windows via _probe_future_consumption instead of a single two-days-ago monthly fetch. Zero-export windows are skipped for backstream meters, the first window with real export wins, and the first valid window is kept as the bill-estimate baseline so consumption estimates are never lost.

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@GuyKh @sisyphus-dev-ai
Send last invoice date on monthly readings to fix zeroed export data
82c8ff6 
IEC zeroes the current period's export (backstream) data unless lastInvoiceDate is the day after the last invoice. Always pass the invoice-derived last invoice date for monthly readings while keeping fromDate as the first of the month; the from_date override still applies on the first-of-month and Sunday paths. _get_invoice_reading_dates now derives the value from the most recent invoice's full_date plus one day instead of the lastDate field.

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@GuyKh @sisyphus-dev-ai
Add consumption since last invoice sensor
5a6107d 
Add a meter-level sensor reading FutureConsumptionInfo.future_consumption, mirroring the existing backstream production sensor, so consumption for the current period since the last invoice is available alongside production.

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@GuyKh
GuyKh force-pushed the fix/backstream-meter-kind-from-contract branch from f16d92a to 5a6107d Compare August 10, 2026 06:13
@GuyKh
GuyKh merged commit e288885 into main Aug 10, 2026
8 checks passed
@GuyKh
GuyKh deleted the fix/backstream-meter-kind-from-contract branch August 10, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Energy production data is no longer available

1 participant

@GuyKh