Skip to content

Add database crawlers (PostgreSQL, MySQL, Snowflake, BigQuery) #2

Description

@jzonthemtn

Summary

Phinder has native crawlers for S3, GCS, Azure Blob, and local filesystems, but no connectors for relational databases or data warehouses. These are common locations for PII.

Proposal

Add database crawlers for:

  • PostgreSQL
  • MySQL
  • Snowflake
  • BigQuery
  • Generic JDBC connector for other databases

Crawlers should sample or scan column values, apply Philterd policies, and report findings per table/column.

Acceptance Criteria

  • Phinder scans PostgreSQL and MySQL databases for PII in text columns
  • Snowflake and BigQuery connectors supported
  • Connection configuration via Phinder config file (host, credentials, schema/table filters)
  • Scan results include: table, column, row count, entity types found, sample matches
  • Tests cover each database type with representative schemas
  • Documentation describes connection setup and credential management

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions