feat: CRM import pipeline for people and orgs - #24
Open
khanal-samir wants to merge 1 commit into
Open
Conversation
Add a staged import pipeline that loads people and organizations from CSV, an inbound webhook, and connector-backed sources (Gmail, Google Calendar, Calendly, Google Sheets, PostHog, Outlook via Graph). Design follows the existing sequence infrastructure: - Extraction and loading are separated. A connector stages raw records; the user reviews the mapping and match preview; a loader then writes to the CRM. Fixing a mapping re-runs the load without re-fetching the source. - Identity is resolved through external_identities, keyed on (workspace, provider, external_id), which makes re-sync idempotent. Sources with no stable id (CSV rows, anonymous pushes) fall back to a content fingerprint so a re-upload updates rather than duplicates — closing the nullable-email uniqueness gap for imported records. - Background work is a DB-polled worker mirroring sequence-worker, with no new infrastructure. Records load in per-batch transactions; a bad row is flagged and skipped rather than failing the run. 429s reschedule from the persisted cursor; auth failures flag the connection for reconnect. - Connectors sit behind INTEGRATION_LIVE_FETCH_ENABLED, mirroring SEQUENCE_LIVE_SEND_ENABLED: off, they resolve from fixtures, so the whole pipeline runs end to end with no provider credentials. - Each provider seeds people.status by signal tier (payment/conversation/ interest); seeding applies on create only, never downgrading an existing record. Adds four tables (integration_connections, import_jobs, import_records, external_identities) plus integration_api_keys for webhook auth, workspace- scoped routes/controllers, shared validators, and the 0004 migration. Tests cover the DB-free boundary at 100%: CSV parsing, mapping and auto-detection, normalization and match resolution, conflict policy, fingerprint idempotency, every connector (pagination, 429, auth failure, dedupe), route wiring, and the worker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A staged ETL pipeline to load people and organizations into the CRM from:
Design
Follows the existing sequence patterns; no new infrastructure.
import_recordsand stops. The user reviews mapping + match preview, then a loader writes to the CRM. Fixing a mapping re-runs the load without re-fetching the source.external_identities, keyed on(workspace, provider, external_id). Sources with no stable id (CSV rows, anonymous pushes) fall back to a content fingerprint, so a re-upload updates instead of duplicating — this closes the nullable-email uniqueness gap for imported records.sequence-worker(worker:imports). Records load in per-batch transactions — a bad row is flagged and skipped, not fatal to the run.429s reschedule from the persisted cursor; auth failures flag the connectionreconnect_required.INTEGRATION_LIVE_FETCH_ENABLED(mirrorsSEQUENCE_LIVE_SEND_ENABLED): off, connectors resolve from fixtures, so the full pipeline runs end to end with no credentials.customer, conversation →qualified, interest →lead). Applies on create only — an import never downgrades an existing record.Schema
Migration
0004:integration_connections,import_jobs,import_records,external_identities,integration_api_keys;people_sourcegainsimport.gmail_integrationsis left untouched.Testing
pnpm test— 127 passpnpm test:coverage— 100% on the DB-free boundary (statements/branches/functions/lines)pnpm check-typesandpnpm lint— cleanCovered: CSV parsing, mapping + auto-detection, normalization + match resolution, conflict policy, fingerprint idempotency, every connector (pagination / 429 / auth failure / dedupe), route wiring, and the worker.
Follow-ups (not in this PR)
settings/integrations+ import wizard)env.exampledocuments the new vars; real values to be added separately🤖 Generated with Claude Code