Feat/forecast source filtering

[BelhsanHmida]

Description

• Expand regressors, future-regressors, and past-regressors to accept either plain sensor IDs or SensorReferenceSchema objects.
• Apply sources, source-types, exclude-source-types, and source-account filters when loading regressor beliefs.
• Preserve existing plain sensor ID behavior for API and CLI users.
• Parse JSON sensor references through the existing CLI regressor options without adding source-specific options.
• Serialize sensor references into primitive values so queued forecasting jobs can reconstruct the configuration.
• Include referenced regressor sensors in API authorization checks.
• Update the OpenAPI specification and changelog.
• Add schema, CLI, API, pipeline-loading, and queued-job test coverage.
• Added changelog item in documentation/changelog.rst

Look & Feel

Plain sensor IDs remain supported:

flexmeasures add forecasts \
  --sensor 2092 \
  --regressors 2093

Source-filtered sensor references can be passed as JSON through the existing options:

flexmeasures add forecasts \
  --sensor 2092 \
  --regressors '[{"sensor": 2093, "sources": [12, 13]}]'

The same structure supports the other SensorReferenceSchema filters:

{
  "sensor": 2093,
  "source-types": ["forecaster"],
  "exclude-source-types": ["user"],
  "source-account": [4]
}

How to test

Relevant automated tests:

uv run pytest -q flexmeasures/data/schemas/tests/test_forecasting.py
uv run pytest -q flexmeasures/cli/tests/test_data_add.py
uv run pytest -q flexmeasures/data/tests/test_forecasting_pipeline.py
uv run pytest -q flexmeasures/api/v3_0/tests/test_forecasting_api.py

Manual checks performed:

• Confirmed legacy sensor IDs and JSON sensor references are parsed by the CLI.
• Confirmed source filters select the expected regressor beliefs.
• Confirmed an empty sources list selects no beliefs.
• Ran a queued forecasting job with a filtered future regressor and confirmed that it persisted a forecast.
• Confirmed all three OpenAPI regressor fields accept either an integer or a sensor reference.

Further Improvements

• Define deterministic precedence when multiple selected sources contain beliefs with the same event and belief times.
• Canonicalize set-like filter lists before generating forecast data-source identities.

Related Items

Closes #2337

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or another license incompatible with FlexMeasures.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #33723980 | 📁 Comparing 2ebf6e0 against latest (5efa733)

  🔍 Preview build  

13 files changed · ± 13 modified

± Modified

• changelog.html
• genindex.html
• _autosummary/flexmeasures.api.v3_0.accounts.html
• _autosummary/flexmeasures.api.v3_0.sensors.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.base.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.predict.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.train.html
• _autosummary/flexmeasures.data.models.generic_assets.html
• _autosummary/flexmeasures.data.schemas.account.html
• _autosummary/flexmeasures.data.schemas.sensors.html
• and 3 more...

[BelhsanHmida]

Reviewed and manually tested this PR

Manual test commands:

export TARGET_SENSOR_ID=313
export REGRESSOR_SENSOR_ID=259
export SELECTED_SOURCE_ID=1212
export OTHER_SOURCE_ID=868
export TRAIN_START=2026-04-21T00:00:00+00:00
export FORECAST_START=2026-04-24T00:00:00+00:00

# Plain ID still works
uv run flexmeasures add forecasts --sensor "$TARGET_SENSOR_ID" --future-regressors "$REGRESSOR_SENSOR_ID" --train-start "$TRAIN_START" --start "$FORECAST_START" --duration PT1H --max-forecast-horizon PT1H --forecast-frequency PT1H

# JSON references work through all regressor options
uv run flexmeasures add forecasts --sensor "$TARGET_SENSOR_ID" --future-regressors "[{\"sensor\": $REGRESSOR_SENSOR_ID, \"sources\": [$SELECTED_SOURCE_ID]}]" --train-start "$TRAIN_START" --start "$FORECAST_START" --duration PT1H --max-forecast-horizon PT1H --forecast-frequency PT1H
uv run flexmeasures add forecasts --sensor "$TARGET_SENSOR_ID" --past-regressors "[{\"sensor\": $REGRESSOR_SENSOR_ID, \"exclude-source-types\": [\"forecaster\"]}]" --train-start "$TRAIN_START" --start "$FORECAST_START" --duration PT1H --max-forecast-horizon PT1H --forecast-frequency PT1H
uv run flexmeasures add forecasts --sensor "$TARGET_SENSOR_ID" --regressors "[{\"sensor\": $REGRESSOR_SENSOR_ID, \"sources\": [$SELECTED_SOURCE_ID, $OTHER_SOURCE_ID]}]" --train-start "$TRAIN_START" --start "$FORECAST_START" --duration PT1H --max-forecast-horizon PT1H --forecast-frequency PT1H

# Empty sources should not fall back to all sources
uv run flexmeasures add forecasts --sensor "$TARGET_SENSOR_ID" --future-regressors "[{\"sensor\": $REGRESSOR_SENSOR_ID, \"sources\": []}]" --train-start "$TRAIN_START" --start "$FORECAST_START" --duration PT1H --max-forecast-horizon PT1H --forecast-frequency PT1H --missing-threshold 0.5

# Queue boundary: terminal 1
uv run flexmeasures jobs run-worker --queue forecasting --without-scheduler --name manual-forecast-worker

# Queue boundary: terminal 2
uv run flexmeasures add forecasts --sensor "$TARGET_SENSOR_ID" --future-regressors "[{\"sensor\": $REGRESSOR_SENSOR_ID, \"sources\": [$SELECTED_SOURCE_ID]}]" --train-start "$TRAIN_START" --start "$FORECAST_START" --duration PT1H --max-forecast-horizon PT1H --forecast-frequency PT1H --as-job

Results:

• Plain regressor IDs still work.
• JSON sensor references are accepted for future-regressors, past-regressors, and regressors.
• Successful runs saved forecast beliefs.
• sources: [] was accepted but produced 100.0% missing values for the regressor, confirming it did not load all sources.
• The queued job completed in the worker with JobStatus.FINISHED.

Targeted automated checks also passed: 96 passed for schema/CLI/pipeline/API tests.

[BelhsanHmida]

@Flix6x follow-up question: should we warn when selected regressor sources collide?

For example, if a config uses sources: [1212, 868] and both sources contain beliefs for the same regressor with the same event_start and belief_time, the pipeline currently has no explicit source precedence rule.

Suggestion:

• Detect collisions after loading regressor beliefs.
• Log a warning that multiple selected sources provide equally-timed beliefs for the same event.
• Include the regressor sensor ID and source IDs in the warning.
• Keep the forecast running for now, but make the ambiguity visible.

[Flix6x]

Without a precedence rule, what is the current behaviour, does it crash?

[BelhsanHmida]

Without a precedence rule, what is the current behaviour, does it crash?

It does not crash.

When two selected sources provide values for the same regressor with the same event_start and belief_time, the pipeline currently keeps running and picks the first row it sees. Reversing the row order changes the selected value, so the behavior is order-dependent rather than explicitly source-prioritized.