Skip to content
This repository was archived by the owner on Sep 4, 2025. It is now read-only.

[+] add python based iceberg sink - #101

Merged
destrex271 merged 19 commits into
destrex271:mainfrom
0xgouda:add-pyiceberg-sink
Aug 26, 2025
Merged

[+] add python based iceberg sink#101
destrex271 merged 19 commits into
destrex271:mainfrom
0xgouda:add-pyiceberg-sink

Conversation

@0xgouda

@0xgouda 0xgouda commented Aug 25, 2025

Copy link
Copy Markdown
Contributor
  • The server assumes a PostgreSQL catalog is used and creates pgwatch namespace and pgwatch.metrics table within it if they don't exist.
  • The table is partitioned by MetricName and DBName (in order).
  • Metrics are written in the local file system as Apache Arrow records with the following schema:
    Schema(
        NestedField(field_id=1, name="DBName", field_type=StringType(), required=True),
        NestedField(field_id=2, name="MetricName", field_type=StringType(), required=True),
        NestedField(field_id=3, name="Data", field_type=StringType(), required=True),
    )
  • Catalog configurations should be provided in .pyiceberg.yaml file under pgcatalog.

Note

This PR focused on providing a working PoC for a Python-based Iceberg receiver, so there are still a couple of Todos that may be added in the future to make this receiver more production-ready:
1. Use object storage instead of the local file system.
2. Support TLS over the gRPC connection.
3. Add authentication interceptor.
4. Cache measurements to minimize the number of Parquet files written.

@0xgouda
0xgouda force-pushed the add-pyiceberg-sink branch from b97aed9 to ef3d64e Compare August 25, 2025 06:39
@0xgouda
0xgouda force-pushed the add-pyiceberg-sink branch from ef3d64e to e615ba4 Compare August 25, 2025 06:40
@0xgouda
0xgouda force-pushed the add-pyiceberg-sink branch from 1cb1478 to b7ec3e1 Compare August 25, 2025 06:54
@destrex271

Copy link
Copy Markdown
Owner

awesome work!

@destrex271
destrex271 merged commit f3b40a5 into destrex271:main Aug 26, 2025
2 checks passed
@0xgouda
0xgouda deleted the add-pyiceberg-sink branch August 30, 2025 16:13
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants