Model Validation Entry - #285
Conversation
✅ Deploy Preview for wonderful-mousse-057100 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
willu47
left a comment
willu47
left a comment
There was a problem hiding this comment.
Hi. I've made a load of non-specific discussion points. Probably not so helpful.
From a project management perspective, then I believe this should be raised as a key milestone and deliverable in every modelling project, and discussed in the planning phase with clients. It is also important to plan resource allocation to take into account the expectations of the client.
|
|
||
| - **Capacity expansion models** | ||
| Focus: long-term investment decisions, technology mix, costs, emissions. | ||
| Validation: benchmark against published scenarios, check plausibility of technology shares, cost ranges, and policy consistency. |
There was a problem hiding this comment.
In my experience, it can be problematic when long-term investment planning or capacity expansion models are presented to a market or operational modelling audience. This is because these long-term models cannot be validated in the way that operational and dispatch models can e.g. in reproducing historical patterns. Instead, comparison of the results, and key sensitivities are a more usual approach.
There was a problem hiding this comment.
Yes! However, as we often fix the investments and only let the model operate, main trends wrt. historical patterns are expected by the client, at least for the "base"year/starting year of the analysis.
|
|
||
| - **Status:** Decision on the core model configuration is complete (e.g. region/countries, temporal resolution, demand assumptions, technology scope). An initial model is created, but not necessarily solved. | ||
| - **PyPSA context:** For PyPSA-related projects, this means that a project configuration file is defined and a first model instance (power-only or sector-coupled) is prepared. | ||
| - **Validation focus:** **Inputs.** Check completeness, plausibility, and consistency of input data. Try to spot major deviations from historically reported values. |
There was a problem hiding this comment.
The scope of the analysis should be confirmed with the client. Are they happy with the regional scope, the temporal or spatial resolution; etc. What key scenario assumptions (if appropriate) do they wish to use? It is also useful to make a note of where assumptions or input data points differ from the commonly accepted values - together with a rationale.
There was a problem hiding this comment.
This is also where a consistent data format as an input to PyPSA-Eur would be useful as it would allow easier comparisons between model runs.
There was a problem hiding this comment.
To complement what Will mentioned, I would suggest that M1 should focus on assumptions (making sure that they are plausible and reasonable) and input data (making sure that they are traceable and consistent).
| - **Validation focus:** | ||
| - Confirm that the model runs successfully end-to-end and the CI passes. | ||
| - Capture / Visualise raw outputs without applying detailed interpretation yet. | ||
| - Check for obvious red flags (e.g. infeasible dispatch, unbounded variables, solver non-convergence). |
There was a problem hiding this comment.
Can we link to specific guidance on this? For example, dual variables can be very useful for identifying optimal, but implausible results. Likewise, if dummy or backstop technology are used, looking at costs or activity of these can give a quick insight into where there could be problem.
| - **PyPSA context:** Running the optimisation workflow with the prepared configuration. | ||
| - **Validation focus:** | ||
| - Confirm that the model runs successfully end-to-end and the CI passes. | ||
| - Capture / Visualise raw outputs without applying detailed interpretation yet. |
There was a problem hiding this comment.
I think a first "sense check" of the results is only possible with a visualisation that provides an overview of all key results. The PyPSA Explorer, for example, gives a quick overview of capacities and generation. However, this can also require country or context-specific knowledge of the domain being modelled, so this is something which can be conducted with country analysts and modellers together.
| - **Inaccurate or incomplete input data**, | ||
| - **Formulation or constraint issues**, | ||
| - **Geographical/temporal scope or resolution**, or | ||
| - **Other reasons**. |
There was a problem hiding this comment.
It is quite likely that a new bug or workflow issue can be identified, especially if any changes have been made.
There was a problem hiding this comment.
Should we mention the solver / solver settings?
| --- | ||
|
|
||
| ## When to Apply | ||
|
|
There was a problem hiding this comment.
Validation should be discussed with the client in the project planning stage. What are their expectations? What is a good quality modelling result? At what point in the project will the model be approved? Do the clients expect fully custom input data, or will they accept the existing PyPSA-Eur workflow as inputs? Against what results do they wish to compare the model outputs? How many model revisions are budgeted for once the validated model is approved?
| - **PyPSA context:** For PyPSA-related projects, this means the model can be re-run consistently by others using the reproducible workflow guide, ideally the benchmarking results are caputred by a CI (optional). | ||
| - **Validation focus:** | ||
| - Ensure full **reproducibility** of results (inputs, code, environment). | ||
| - Package results and code in a **versioned release** (tagged, archived). |
There was a problem hiding this comment.
Also include a versioned package of input data. What if data is commercially sensitive or private? How can it be version-controlled to ensure reproducibility?
| - Ensure full **reproducibility** of results (inputs, code, environment). | ||
| - Package results and code in a **versioned release** (tagged, archived). | ||
| - Document any **deviations, limitations, or open issues**. | ||
| - **Rationale:** At this stage, the model is considered *validated enough* for external review, reporting, or delivery. |
There was a problem hiding this comment.
I would suggest that model results and input data are presented to the client, and the "validation" is signed off by the client.
| --- | ||
|
|
||
| ### 5) Sensitivity & Uncertainty | ||
| - **Key levers:** demand growth, fuel/CO₂ price, CAPEX/OPEX, availability. |
There was a problem hiding this comment.
In capacity expansion models, key levers include:
- Discount rate;
- Demand
- CO_2 price
- technology availability;
- resource availability;
- storage cost;
- demand profile;
- technology costs
Levers depend on what output sensitivity you are measuring. For example, costs are very sensitive to discount rate; and discount rate has a more subtle effect on temporal distribution of capacity;
| - **Provenance:** dataset sources, versions, licenses recorded. | ||
| - **Record deviations:** imputations/corrections with rationale. | ||
|
|
||
| **Deliverables:** `inputs_audit.md` + automated check logs (if available). |
There was a problem hiding this comment.
This deliverable could be a checklist, where each of these categories can be marked against, and evidence provided. This can then be reviewed by the client and the internal reviewer before signing off the validated model.
eantonini
left a comment
eantonini
left a comment
There was a problem hiding this comment.
The document is already in a very good shape and covers the main steps of a validation process. I added a few minor comments.
|
|
||
| Model validation is the process of assessing whether a model is **fit for purpose** by checking that: | ||
|
|
||
| 1. **Inputs** are complete, consistent, and plausible. |
There was a problem hiding this comment.
Inputs should also come from widely-used and traceable data sources. I would use "verifiable" instead of "plausible".
|
|
||
| - **Status:** Decision on the core model configuration is complete (e.g. region/countries, temporal resolution, demand assumptions, technology scope). An initial model is created, but not necessarily solved. | ||
| - **PyPSA context:** For PyPSA-related projects, this means that a project configuration file is defined and a first model instance (power-only or sector-coupled) is prepared. | ||
| - **Validation focus:** **Inputs.** Check completeness, plausibility, and consistency of input data. Try to spot major deviations from historically reported values. |
There was a problem hiding this comment.
To complement what Will mentioned, I would suggest that M1 should focus on assumptions (making sure that they are plausible and reasonable) and input data (making sure that they are traceable and consistent).
| - **Inaccurate or incomplete input data**, | ||
| - **Formulation or constraint issues**, | ||
| - **Geographical/temporal scope or resolution**, or | ||
| - **Other reasons**. |
There was a problem hiding this comment.
Should we mention the solver / solver settings?
| - Document decisions: what was fixed, what remains open, and why. | ||
| - **Rationale:** Iterative improvement ensures that the validated model is reproducible, credible, and defensible for reporting or delivery. | ||
|
|
||
| > **Note:** During this stage of validation, **don’t save on model runs**. The more parameter sweeps you run, the deeper your insights will be. Test whether compromising on spatial or temporal resolution significantly affects results — this may allow you to use a simplified model to accelerate turnaround times while preserving credibility. |
There was a problem hiding this comment.
I would also add that the we need to check the assumptions defined in M1 and determine what impacts they have on results (sensitivity analysis).
| - **Scenario sweeps:** quantify impact on KPIs; identify non-robust conclusions. | ||
| - **Tornado/elasticity plots:** highlight most influential assumptions. | ||
|
|
||
| **Deliverables:** sensitivity summary + plots + takeaways. |
There was a problem hiding this comment.
Nice ways to visualize scenario sweeps are contour plots, where two key levers are on the two axes and the contours show the KPI.
4 participants
Closes # (if applicable).
Changes Proposed in This Pull Request
Checklist