Skip to content

feat: CRM import pipeline for people and orgs - #24

Open
khanal-samir wants to merge 1 commit into
feat/sequencefrom
feat/import-pipeline
Open

feat: CRM import pipeline for people and orgs#24
khanal-samir wants to merge 1 commit into
feat/sequencefrom
feat/import-pipeline

Conversation

@khanal-samir

Copy link
Copy Markdown
Owner

What

A staged ETL pipeline to load people and organizations into the CRM from:

  • CSV upload — no OAuth, universal fallback
  • Inbound webhook (API-key auth) — the generic entry point for Zapier / Make / n8n / scripts
  • Gmail & Google Calendar — mines sent-mail recipients and meeting attendees
  • Calendly — booking invitees with custom question answers
  • Google Sheets — re-syncable, same path as CSV
  • PostHog — product users via the Persons API
  • Outlook / Microsoft 365 — contacts, sent mail, and calendar through one Graph grant

Base is feat/sequence (not main): the import code reuses the sequence infra — token-crypto, encrypted-token storage — which only exists on that branch. Targeting feat/sequence keeps this PR to the import diff alone.

Design

Follows the existing sequence patterns; no new infrastructure.

  • Extract is separated from load. A connector stages raw records into import_records and stops. The user reviews mapping + match preview, then a loader writes to the CRM. Fixing a mapping re-runs the load without re-fetching the source.
  • Idempotent re-sync via external_identities, keyed on (workspace, provider, external_id). Sources with no stable id (CSV rows, anonymous pushes) fall back to a content fingerprint, so a re-upload updates instead of duplicating — this closes the nullable-email uniqueness gap for imported records.
  • DB-polled worker mirroring sequence-worker (worker:imports). Records load in per-batch transactions — a bad row is flagged and skipped, not fatal to the run. 429s reschedule from the persisted cursor; auth failures flag the connection reconnect_required.
  • Fixture mode behind INTEGRATION_LIVE_FETCH_ENABLED (mirrors SEQUENCE_LIVE_SEND_ENABLED): off, connectors resolve from fixtures, so the full pipeline runs end to end with no credentials.
  • Status seeding by signal tier (payment → customer, conversation → qualified, interest → lead). Applies on create only — an import never downgrades an existing record.

Schema

Migration 0004: integration_connections, import_jobs, import_records, external_identities, integration_api_keys; people_source gains import. gmail_integrations is left untouched.

Testing

  • pnpm test127 pass
  • pnpm test:coverage100% on the DB-free boundary (statements/branches/functions/lines)
  • pnpm check-types and pnpm lint — clean
  • Verified end-to-end against a seeded DB (CSV import, re-sync idempotency, webhook push, automated-sender rejection). That live run is what surfaced the emailless-row duplication bug now fixed by the fingerprint fallback.

Covered: CSV parsing, mapping + auto-detection, normalization + match resolution, conflict policy, fingerprint idempotency, every connector (pagination / 429 / auth failure / dedupe), route wiring, and the worker.

Follow-ups (not in this PR)

  • Web UI (settings/integrations + import wizard)
  • Live OAuth connect flows per provider (env wiring pending)
  • env.example documents the new vars; real values to be added separately

🤖 Generated with Claude Code

Add a staged import pipeline that loads people and organizations from CSV,
an inbound webhook, and connector-backed sources (Gmail, Google Calendar,
Calendly, Google Sheets, PostHog, Outlook via Graph).

Design follows the existing sequence infrastructure:
- Extraction and loading are separated. A connector stages raw records; the
  user reviews the mapping and match preview; a loader then writes to the CRM.
  Fixing a mapping re-runs the load without re-fetching the source.
- Identity is resolved through external_identities, keyed on
  (workspace, provider, external_id), which makes re-sync idempotent. Sources
  with no stable id (CSV rows, anonymous pushes) fall back to a content
  fingerprint so a re-upload updates rather than duplicates — closing the
  nullable-email uniqueness gap for imported records.
- Background work is a DB-polled worker mirroring sequence-worker, with no new
  infrastructure. Records load in per-batch transactions; a bad row is flagged
  and skipped rather than failing the run. 429s reschedule from the persisted
  cursor; auth failures flag the connection for reconnect.
- Connectors sit behind INTEGRATION_LIVE_FETCH_ENABLED, mirroring
  SEQUENCE_LIVE_SEND_ENABLED: off, they resolve from fixtures, so the whole
  pipeline runs end to end with no provider credentials.
- Each provider seeds people.status by signal tier (payment/conversation/
  interest); seeding applies on create only, never downgrading an existing
  record.

Adds four tables (integration_connections, import_jobs, import_records,
external_identities) plus integration_api_keys for webhook auth, workspace-
scoped routes/controllers, shared validators, and the 0004 migration.

Tests cover the DB-free boundary at 100%: CSV parsing, mapping and
auto-detection, normalization and match resolution, conflict policy,
fingerprint idempotency, every connector (pagination, 429, auth failure,
dedupe), route wiring, and the worker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant