Skip to content

fix(V2): convert CSV batch upload staged data to snake_case - #1579

Open
TheLastCicada wants to merge 1 commit into
developfrom
fix/v2-csv-batch-snake-case
Open

fix(V2): convert CSV batch upload staged data to snake_case#1579
TheLastCicada wants to merge 1 commit into
developfrom
fix/v2-csv-batch-snake-case

Conversation

@TheLastCicada

@TheLastCicada TheLastCicada commented Apr 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fix: V2 CSV batch upload (POST /v2/project/batch and POST /v2/unit/batch) was staging data with camelCase field names from CSV headers, but the V2 commit pipeline expects snake_case. This caused commits to fail with a null primary key error because cad_trust_project_id could not be found in the camelCase header row.
  • Fix: Staged records were missing org_uid, which the XLSX upload path correctly injects. This could cause SequelizeUniqueConstraintError during sync.
  • Docs: Updated CSV batch upload documentation to clarify create vs update behavior and explicitly note that NEW-<n> placeholder IDs are XLSX-only (not supported in CSV).

Changes

File Change
src/utils/v2-xls.js Export existing toDbFieldNames() for reuse
src/models/v2/project-v2.model.js Convert CSV records to snake_case via toDbFieldNames(), inject org_uid, hoist getHomeOrg() out of per-row loop
src/models/v2/unit-v2.model.js Same fix; also reuse UnitV2.prepareXlsRow() instead of duplicating serial ID logic
docs/cadt_rpc_api_v2.md Document create/update behavior for CSV batch, note NEW-<n> is XLSX-only
tests/v2/integration/project-v2.spec.js 6 new tests: snake_case format verification (INSERT + UPDATE), pipe-separated arrays, JSON arrays, non-existent ID rejection, empty CSV rejection
tests/v2/integration/unit-v2.spec.js 5 new tests: snake_case format verification (INSERT + UPDATE), serial ID derivation, non-existent ID rejection, empty CSV rejection

Test plan

  • All 1396 V2 integration tests pass
  • All 112 V1 integration tests pass
  • New tests verify staged data uses snake_case field names (would have caught the original bug)
  • New tests verify org_uid is present in staged records
  • New tests verify staging uuid matches the generated primary key
  • Pre-push adversarial diff review: 0 critical findings

Note

Medium Risk
Changes how POST /v2/project/batch and POST /v2/unit/batch stage data for the commit pipeline (field naming, primary keys, org scoping), which can impact imports and downstream syncing if incorrect.

Overview
Fixes V2 CSV batch upload staging for project and unit so staged JSON uses snake_case DB column names, includes org_uid from the home org, and sets staging uuid to the effective primary key (generated on INSERT or provided on UPDATE). It also reuses existing XLSX-row preparation for units (including unitSerialId derivation) and exports toDbFieldNames() for shared CSV/XLSX field conversion.

Updates the API docs to clarify create-vs-update behavior for CSV uploads and that NEW-<n> placeholders are XLSX-only, and expands integration tests to assert staging format, array parsing (pipe/JSON), serial derivation, and correct 400s for empty CSVs or non-existent IDs.

Reviewed by Cursor Bugbot for commit 8a6bc0d. Bugbot is set up for automated code reviews on this repo. Configure here.

The V2 CSV batch upload endpoints for projects and units were staging
data with camelCase field names (from CSV headers), but the V2 commit
pipeline expects snake_case. This caused commits to fail because the
primary key field (e.g. cad_trust_project_id) could not be found in
the staged data headers.

- Reuse toDbFieldNames() from v2-xls.js to convert CSV records to
  snake_case before staging, matching the XLSX upload path
- Inject org_uid into staged records (consistent with XLSX path)
- Hoist getHomeOrg() call before the CSV stream loop (once per batch
  instead of once per row)
- Reuse UnitV2.prepareXlsRow() for serial ID derivation instead of
  duplicating the logic inline
- Add integration tests verifying staged data format (snake_case
  fields, org_uid presence, primary key correctness)
- Add tests for array field parsing (pipe-separated, JSON), error
  cases (non-existent ID, empty CSV), and serial ID derivation
- Update V2 API docs to clarify CSV batch create/update behavior and
  document that NEW-<n> placeholders are XLSX-only
Base automatically changed from v2-rc2 to develop April 15, 2026 17:29
@TheLastCicada
TheLastCicada changed the base branch from develop to v2-rc2 August 10, 2026 22:57
Base automatically changed from v2-rc2 to develop August 13, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant