Skip to content

[DEV-15416] CSV header names interpolated into CREATE TABLE - #4712

Open
DavidMikolaKC wants to merge 7 commits into
qatfrom
mod/dev-15416-interpolate-csv-header-names
Open

[DEV-15416] CSV header names interpolated into CREATE TABLE #4712
DavidMikolaKC wants to merge 7 commits into
qatfrom
mod/dev-15416-interpolate-csv-header-names

Conversation

@DavidMikolaKC

@DavidMikolaKC DavidMikolaKC commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Description:

When creating new population data tables using incoming treasury csv files, the default headers are passed directly into a create_table command without any processing. While unlikely, checks should be added to account for the possibility of receiving a compromised file.

Technical Details:

loaders.py:75 interpolates each CSV header c as f"{c} TEXT" into a CREATE TABLE statement and executes it with no identifier quoting or allow-list. A simple allow-list is added using the column mappings present in the loader factories

Requirements for PR Merge:

  1. Unit & integration tests updated
  2. API documentation updated (examples listed below)
    1. API Contracts
    2. API UI
    3. Comments
  3. Data validation completed (examples listed below)
    1. Does this work well with the current frontend? Or is the frontend aware of a needed change?
    2. Is performance impacted in the changes (e.g., API, pipeline, downloads, etc.)?
    3. Is the expected data returned with the expected format?
  4. Appropriate Operations ticket(s) created
  5. Jira Ticket(s)
    1. DEV-15416

Explain N/A in above checklist:

# checks to see if the column is an expected value
approved_cols = [c for c in columns if c in self._columns_mapper.keys()]

if len(approved_cols) > 0:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this check that the length of the approved_cols is equal to the length of the supplied columns? Otherwise, it will attempt to process if only a single column is approved.

sethstoudenmier
sethstoudenmier previously approved these changes Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants