Implement new protocol for checkpoint requests

[simolus3]

This adds an option making the JS SDKs use the new checkpoint request protocol described in https://github.com/orgs/powersync-ja/discussions/317. While that discussion contains all the relevant context, it basically boils down to flaws in the current /write-checkpoint2.json request used to request write checkpoints after uploads. Requested write checkpoints are embedded in the sync protocol, allowing clients to recognize when their local writes should have synced back down again, at which point downloaded changes can be applied.

/write-checkpoint2.json has two flaws fixed by the new protocol:

1. The sequence of write checkpoints needs to increment at every request for every (device_id, user_id) pair. This means the service can't ever delete write checkpoint state.
2. When a client switches users, that changes the (device_id, user_id) pair which means the next write checkpoint might disrupt the sequence. Nothing in the client is prepared for that, and this can lead to the sync client not applying changes.

The new protocol replaces GET /write-checkpoint2.json with POST /sync/checkpoint-request. The key change is that the client includes an expected counter in the request, the service uses that value or its own (whichever is higher). This fixes both problems, but requires a few implementation changes:

1. As older service versions don't support this, it needs to be opt-in.
2. By the design of the core extension, new checkpoint requests using this format can only be posted when a sync iteration is active and that iteration has posted its expected counter to the service (which makes sure that subsequent target checkpoints we request after uploads use the reconciled counter state).
3. We need to keep supporting custom checkpoints for team/enterprise customers. Because the SDK can now fetch these proactively, this adds an optional postCheckpointRequest to backend connectors.
4. Another change is that once we have a checkpoint request, we should keep posting it if it hasn't been applied yet. This is relatively cheap, ensures this shows up in service logs, and acts as a catch-all for network failures which might have gotten the checkpoint sequence out-of-sync.

Note that this does not add checkpoint requests, I will add support for those in a follow-up PR.

AI use disclosure: I used Claude Code to help with simpler tests and to self-review this.

[changeset-bot]

🦋 Changeset detected

Latest commit: a9da7c3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 10 packages

Name	Type
@powersync/shared-internals	Minor
@powersync/react-native	Minor
@powersync/capacitor	Minor
@powersync/common	Minor
@powersync/node	Minor
@powersync/nuxt	Minor
@powersync/web	Minor
@powersync/adapter-sql-js	Patch
@powersync/diagnostics-app	Patch
@powersync/tanstack-react-query	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

@simolus3 I wonder if there's a way to auto-detect the feature, instead of requiring opt-in?

Some options:

1. Let the service advertise that it supports the feature. Not sure where though - might be easier for websockets than http streams? Maybe a response header for http?
2. First attempt a call to /sync/checkpoint-request, then fall back to the old one if that produces a 404 (kinda ugly, but for example most PowerSync Cloud instances will just work automatically).
3. Automatically use requests if postCheckpointRequest is defined. (This is still a form of opt-in, just without the flag)

I haven't worked through the implications of any of those, just wondering if something like that is feasible?

[Chriztiaan]

Happy with the changes, Ralf's suggestion could be a blocker if we want to incorporate it.

[simolus3]

The implementation here mostly follows the Swift implementation, which also doesn't try to auto-detect the feature. Adding that sounds reasonable to me, perhaps as we start maturing the APIs (they're currently marked as @alpha)? It's also a bit tricky:

Let the service advertise that it supports the feature. Not sure where though - might be easier for websockets than http streams? Maybe a response header for http?

I could see an X-PowerSync-Service-Version header or a /version endpoint. But since we reconcile checkpoint request state in parallel to the /sync/stream connection, we can't rely on an existing request to give us required information first. So it would be an extra request, at which point we might as well try to reach /sync/checkpoint-request unconditionally and interpret the 404.

Automatically use requests if postCheckpointRequest is defined. (This is still a form of opt-in, just without the flag)

Note that postCheckpointRequest is only used for custom checkpoint requests on connectors, so that only works for team/enterprise deployments. We currently have a warning when that method is defined with the legacy option. Defaulting to the new checkpoint mode (and failing the sync when it's not available) sounds reasonable to me as well.

We also tell the core extension that we want to use request checkpoints which makes the sync client behave slightly differently, throwing when we try to request a checkpoint without completing the reconciliation first (IIRC). So we'd need a way to tell the core extension about that downgrade as well.

Finally, we want explicit sync via requestCheckpoint() to throw if the old write checkpoints are used. This is a lot easier to do when it's a static option, otherwise we'd have to inspect the internal client state both when requesting them and while waiting for them to sync.

cc @stevensJourney for more ideas

[rkistner]

Considering the alpha status, I guess it makes sense to keep the old behavior as the default for now. We can revisit changing the default to a kind of "auto" mode at a later stage.

[stevensJourney]

My thoughts align with what Simon mentioned. I think auto detection should be possible in the future. My general thoughts are that I'd like to gradually release this instead of instantly switching everyone over by default.