Require NOT NULL iteration and stage keys in opchar input tables - #1431
Merged
Conversation
The iteration-keyed opchar input tables (variable generator profiles, hydro operational chars, energy profiles, horizon shaping, load modifier profiles, load component shift bounds, exogenous state of charge, variable OM by timepoint) are read with equality filters on weather_iteration, hydro_iteration, and stage_id, e.g. weather_iteration = 0 for inputs that do not vary by weather iteration. NULL never satisfies an equality filter, so a blank key cell in an input CSV silently dropped the row: the project ended up with no data and the scenario failed at model load rather than at import or validation. Declare these 24 columns NOT NULL so a blank cell fails at CSV import with "NOT NULL constraint failed: <table>.<column>". Add a schema test that discovers the iteration-keyed tables from the schema and asserts the constraint on every key column, so a future table of this kind cannot be added without it. Databases created before this change keep their unconstrained tables until rebuilt. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The iteration-keyed opchar input tables (variable generator profiles, hydro operational chars, energy profiles, horizon shaping, load modifier profiles, load component shift bounds, exogenous state of charge, variable OM by timepoint) are read with equality filters on
weather_iteration,hydro_iteration, andstage_id(e.g.weather_iteration = 0for inputs that do not vary by weather iteration). NULL never satisfies an equality filter, so a blank key cell in an input CSV silently dropped the row: the project ended up with no data and the scenario failed at model load rather than at import or validation.NOT NULLindb/db_schema.sql, so a blank cell now fails at CSV import withNOT NULL constraint failed: <table>.<column>.tests/test_db_schema_iteration_keys.py, which discovers the iteration-keyed tables from the schema (everyinputs_project_*table with a companion_iterationstable keyed by an opchar column) and asserts the constraint on each key column, so a future table of this kind cannot be added without it.Notes
db/csvs_test_examples); the examples database rebuilds from CSVs under the new schema and the example scenarios touching these tables pass.🤖 Generated with Claude Code