Extendable External Sync Providers

[teelur]

Currently, sync provider integration is very specific to the provider and requires first-party integration into Budget Board.

Ideally we would have some sort of hierarchy for these providers:

• first-party integration
• template support
• third-party plugins

First party integration would be similar to what we already have: a turnkey integration that just requires the user to add an API key and get started.

Template support would provide a framework to handle sync provider integration that could be done via something like YAML. We would provide a markup that could be configured with things like endpoints for certain data, the data structure of the request and responses, and maybe some sort of logic flow.

Lastly would be some way to allow third-party plugin integration. This would allow users to create their own sync providers (or wrap existing ones that don't comply with the template solution) and either load as a plugin in the existing server process or spin up as their own process and interact with an API.

This is all just high level thoughts, need to investigate more.

[john-frandsen]

Nice breakdown — having spent a lot of time wiring bank-data providers into apps like this, a few things worth baking in before the template/plugin tiers solidify:

• Auth model is the real dividing line between your tiers. SimpleFIN-style static API keys fit the YAML template tier fine, but PSD2/open-banking providers need an interactive redirect consent flow (user authenticates at their bank, you get back a consent handle) plus a 90-day reconsent cycle. If the template schema models auth as just a "key" field it won't stretch; model it as a state machine (unlinked → consent-pending → active → expiring-soon → reauth-needed) with a per-provider consent lifetime instead.
• Transactions are not append-only. Pending entries mutate into booked ones under new IDs and banks re-issue recent history, so the sync engine needs idempotent upserts keyed on (account, bank transaction id) with a content-hash fallback (amount + date + counterparty), not insert-only imports.
• Most PSD2 banks expose no webhooks, so the scheduler needs per-provider polling with rate-limit awareness and backoff — worth putting a request budget into the provider contract from day one rather than retrofitting it.
• Reserve schema space for the classic edge cases: bookingDate vs valueDate, interim vs closing booked balances, and multi-currency accounts that return XXX/no currency (apply an account-level fallback rather than failing the whole account).
• For the plugin tier, out-of-process providers speaking a normalized REST contract are easier to sandbox and update independently — and a plugin that wraps a PSD2 aggregator means the core app never has to hold bank certificates or become a licensed TPP itself, which is the cheapest route to EU coverage in your third tier.

Disclosure: I build open-banking.io, a cert-free open-banking API.