Skip to content

Require NOT NULL iteration and stage keys in opchar input tables - #1431

Merged
anamileva merged 1 commit into
developfrom
am/validate-null-iteration-columns
Sep 2, 2026
Merged

Require NOT NULL iteration and stage keys in opchar input tables#1431
anamileva merged 1 commit into
developfrom
am/validate-null-iteration-columns

Conversation

@anamileva

Copy link
Copy Markdown
Member

Summary

The iteration-keyed opchar input tables (variable generator profiles, hydro operational chars, energy profiles, horizon shaping, load modifier profiles, load component shift bounds, exogenous state of charge, variable OM by timepoint) are read with equality filters on weather_iteration, hydro_iteration, and stage_id (e.g. weather_iteration = 0 for inputs that do not vary by weather iteration). NULL never satisfies an equality filter, so a blank key cell in an input CSV silently dropped the row: the project ended up with no data and the scenario failed at model load rather than at import or validation.

  • Declare these 24 columns NOT NULL in db/db_schema.sql, so a blank cell now fails at CSV import with NOT NULL constraint failed: <table>.<column>.
  • Add tests/test_db_schema_iteration_keys.py, which discovers the iteration-keyed tables from the schema (every inputs_project_* table with a companion _iterations table keyed by an opchar column) and asserts the constraint on each key column, so a future table of this kind cannot be added without it.

Notes

  • Databases created before this change keep their unconstrained tables until rebuilt.
  • No committed example CSV has a blank key cell (scanned all 188 key columns under db/csvs_test_examples); the examples database rebuilds from CSVs under the new schema and the example scenarios touching these tables pass.
  • The test was mutation-checked: stripping one constraint from the schema makes it fail.

🤖 Generated with Claude Code

The iteration-keyed opchar input tables (variable generator profiles,
hydro operational chars, energy profiles, horizon shaping, load modifier
profiles, load component shift bounds, exogenous state of charge,
variable OM by timepoint) are read with equality filters on
weather_iteration, hydro_iteration, and stage_id, e.g.
weather_iteration = 0 for inputs that do not vary by weather iteration.
NULL never satisfies an equality filter, so a blank key cell in an input
CSV silently dropped the row: the project ended up with no data and the
scenario failed at model load rather than at import or validation.

Declare these 24 columns NOT NULL so a blank cell fails at CSV import
with "NOT NULL constraint failed: <table>.<column>". Add a schema test
that discovers the iteration-keyed tables from the schema and asserts
the constraint on every key column, so a future table of this kind
cannot be added without it.

Databases created before this change keep their unconstrained tables
until rebuilt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@anamileva
anamileva merged commit 564574e into develop Sep 2, 2026
6 checks passed
@anamileva
anamileva deleted the am/validate-null-iteration-columns branch September 4, 2026 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant