Merge development (with our current Datawarehouse code) into the AWS branch.

[jgrantr]

Note

High Risk
Large changes to fact/dimension ingest SQL, delete semantics, and Redshift COPY/S3 paths affect production warehouse data correctness; misconfiguration of new flags can call process.exit() at startup.

Overview
Merges current data warehouse work into the AWS branch: Redshift-oriented staging and load paths, corrected delete/update merging, and tooling to publish the postgres connector on GitHub release.

Postgres / Redshift DW (dwconnect.js, connect.js) adds a full streamToTableFromS3 path (CSV to S3, then COPY with optional IAM role and S3 cleanup). When config.version === 'redshift', fact/dimension imports use Redshift-style staging (DISTSTYLE ALL, sort keys from v_dist_sort_key), skip ANALYZE where inappropriate, and can use delete + reinsert instead of heavy updates. New config knobs include hashedSurrogateKeys (farmFingerPrint64, bigint SKs), bypassSlowlyChangingDimensions, and deleteFlushCount; incompatible combinations of hashed keys and SCD exit the process. Fact loads no longer push __leo_delete__ rows into staging (deletes go through the delete handler first). Temp table drops are IF EXISTS with errors logged instead of failing the run.

Common DW pipeline fixes combine.js so consecutive rows for the same key treat either side as a delete correctly (a later update cancels an earlier delete). load.js rewires the validation error sink to ls.process / ls.toLeo and records payload.source_id on failed events.

Also adds .github/workflows/publish-postgres.yml (release-triggered npm publish from ./postgres), .gitignore entries, and version bumps for common, entity-table, and postgres packages (lockfiles updated).

^{Reviewed by Cursor Bugbot for commit 9738116. Bugbot is set up for automated code reviews on this repo. Configure here.}

[ch-snyk-sa]

⛔ Snyk checks have failed. 31 issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (31)
⛔	Open Source Security	0	24	6	1	31 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[cursor]

Bug: Incorrect Case-Sensitive Property Check Breaks Error Stream Initialization

Same incorrect property check as above. The code checks !errorStream.Writable but should check !errorStream.writable (lowercase w). This will cause the error stream initialization logic to be executed every time handleFailedValidation is called, potentially creating multiple pipelines for the same error stream.

 

[cursor]

Debug console.log left in production code

Medium Severity

A console.log(has delete_fix in-place) statement is left in the module initialization path of dwconnect.js. This will print to stdout every time the module is loaded, which is noisy in production and clearly debug/development code that wasn't intended to be committed.

 

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 4 potential issues.

There are 5 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9738116. Configure here.}

[cursor]

Bypass SCD wrong ingest path

High Severity

When bypassSlowlyChangingDimensions is true but hashedSurrogateKeys is false, importDimension still runs the delete/reinsert path that sets surrogate keys with farmFingerPrint64, which does not match the integer SCD path and can corrupt dimension history.

 

^{Reviewed by Cursor Bugbot for commit 9738116. Configure here.}

[cursor]

Missing row guard on view

Medium Severity

Redshift sort-key setup reads results[0] without checking that v_dist_sort_key returned a row, so a missing view entry causes a TypeError and aborts the ingest for that table.

 

^{Reviewed by Cursor Bugbot for commit 9738116. Configure here.}

[cursor]

Debug log left in module

Low Severity

A console.log announcing delete-fix status runs every time the warehouse client module loads, adding noise in production logs and Lambda output.

 

^{Reviewed by Cursor Bugbot for commit 9738116. Configure here.}

[cursor]

S3 stream init race

High Severity

streamToTableFromS3 can process rows before the S3 upload stream exists: a single pending callback is overwritten if multiple rows arrive early, and the write step calls stream.write when stream is still undefined, causing lost rows, hangs, or runtime errors.

 

^{Reviewed by Cursor Bugbot for commit 9738116. Configure here.}