feature(csdk): proxy agent support example - #615
Conversation
|
Sinchana Hegde seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
🧹 Python Code Quality Check✅ No issues found in Python Files. This comment is auto-updated with every commit. |
There was a problem hiding this comment.
Pull request overview
This PR adds a new “Proxy Agent” set of Connector SDK examples showing how to connect to PostgreSQL sources when traffic is routed through a Fivetran Proxy Agent, including single-host, multi-host failover, and a custom proxy host configuration key.
Changes:
- Added three new proxy-agent PostgreSQL connector examples (single host, multiple hosts failover, custom host key).
- Added per-example READMEs,
configuration.jsontemplates, andpsycopg2-binarydependency pins. - Updated the top-level
examples/README.mdto include the new Proxy Agent examples section.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| examples/README.md | Adds links to the new Proxy Agent examples. |
| examples/common_patterns_for_connectors/proxy_agent/simple_postgres_connection/requirements.txt | Pins psycopg2-binary for the single-host example. |
| examples/common_patterns_for_connectors/proxy_agent/simple_postgres_connection/README.md | Documents setup, deploy, and behavior for the single-host proxy-agent example. |
| examples/common_patterns_for_connectors/proxy_agent/simple_postgres_connection/connector.py | Implements the single-host PostgreSQL connector example behind a proxy agent. |
| examples/common_patterns_for_connectors/proxy_agent/simple_postgres_connection/configuration.json | Provides a placeholder configuration template for the single-host example. |
| examples/common_patterns_for_connectors/proxy_agent/multiple_hosts/requirements.txt | Pins psycopg2-binary for the multi-host failover example. |
| examples/common_patterns_for_connectors/proxy_agent/multiple_hosts/README.md | Documents setup and failover behavior for the multi-host example. |
| examples/common_patterns_for_connectors/proxy_agent/multiple_hosts/connector.py | Implements host iteration + connection failover for proxy-agent PostgreSQL connectivity. |
| examples/common_patterns_for_connectors/proxy_agent/multiple_hosts/configuration.json | Provides a placeholder configuration template using a hosts array. |
| examples/common_patterns_for_connectors/proxy_agent/custom_proxy_host_key/requirements.txt | Pins psycopg2-binary for the custom-key example. |
| examples/common_patterns_for_connectors/proxy_agent/custom_proxy_host_key/README.md | Documents using --proxy-host-config-key and a custom proxy host configuration key. |
| examples/common_patterns_for_connectors/proxy_agent/custom_proxy_host_key/connector.py | Implements the custom-key variant of proxy-agent PostgreSQL connectivity. |
| examples/common_patterns_for_connectors/proxy_agent/custom_proxy_host_key/configuration.json | Provides a placeholder configuration template for the custom-key example. |
Suppressed comments (4)
examples/common_patterns_for_connectors/proxy_agent/custom_proxy_host_key/connector.py:175
- The
update()docstring should match the template connector (URL + exact wording/format) for consistency with the rest of the repo.
"""
Define the update function, which is a required function, and is called by Fivetran during each sync.
See the technical reference documentation for more details on the update function
https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update
Args:
examples/common_patterns_for_connectors/proxy_agent/custom_proxy_host_key/connector.py:201
- The connector creation + local
__main__debug block should match the repo’s template connector so users get consistent guidance (especially aroundfivetran debug).
# This creates the connector object that will use the update and schema functions defined in this connector.py file.
connector = Connector(update=update, schema=schema)
# Check if the script is being run as the main module. This is Python's standard entry method allowing your script to
# be run directly from the command line or IDE 'run' button. This is useful for debugging while you write your code.
# Note this method is not called by Fivetran when executing your connector in production. Please test using the
# Fivetran debug command prior to finalizing and deploying your connector.
if __name__ == "__main__":
try:
with open("configuration.json", "r") as f:
configuration = json.load(f)
except FileNotFoundError:
configuration = {}
connector.debug(configuration=configuration)
examples/common_patterns_for_connectors/proxy_agent/multiple_hosts/connector.py:216
- The
update()docstring should match the template connector (URL + exact wording/format) so the example stays aligned with the documented API reference.
"""
Define the update function, which is a required function, and is called by Fivetran during each sync.
See the technical reference documentation for more details on the update function
https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update
Args:
examples/common_patterns_for_connectors/proxy_agent/multiple_hosts/connector.py:242
- The connector creation + local
__main__debug block should match the repo’s template connector so users get consistent guidance (especially aroundfivetran debug).
# This creates the connector object that will use the update and schema functions defined in this connector.py file.
connector = Connector(update=update, schema=schema)
# Check if the script is being run as the main module. This is Python's standard entry method allowing your script to
# be run directly from the command line or IDE 'run' button. This is useful for debugging while you write your code.
# Note this method is not called by Fivetran when executing your connector in production. Please test using the
# Fivetran debug command prior to finalizing and deploying your connector.
if __name__ == "__main__":
try:
with open("configuration.json", "r") as f:
configuration = json.load(f)
except FileNotFoundError:
configuration = {}
connector.debug(configuration=configuration)
fivetran-sahilkhirwal
left a comment
There was a problem hiding this comment.
Looks good
fivetran-JenasVimal
left a comment
There was a problem hiding this comment.
NIT comments , will approve one you resolve them
fivetran-dejantucakov
left a comment
There was a problem hiding this comment.
@fivetran-sinchanahegde Left a couple of more suggestions/comments
… proxy-example-csdk
5c01c92
0bfe5bc
fivetran-dejantucakov
left a comment
There was a problem hiding this comment.
@fivetran-sinchanahegde Re-approved
Jira ticket
Closes https://fivetran.atlassian.net/browse/RD-1202613
Description of Change
Example to use proxy agent in csdk
Testing
Checklist
Some tips and links to help validate your PR:
fivetran debugcommand.