PR #13721 restores correctness by no longer moving watch_ctx.rev to a revision sampled on a separate connection after a watch timeout. It also reuses unchanged objects during compact recovery. One cost remains: recovery still issues a Range request for each config type in every worker, transfers all values, and has lua-resty-etcd deserialize them before load_full_data() can compare modifiedIndex.
Once etcd has returned compacted, skipping reconciliation is not safe. If the watcher last confirmed revision R and etcd has compacted through C, these two histories are indistinguishable after (R, C] has been removed:
- no key under the watched prefix changed;
- the watch stream silently missed a change or deletion.
A revision sampled by a separate Range request does not prove delivery on the watch stream. The revision in a created response is not a delivery barrier either: etcd sends the create response before flushing buffered historical events for that watcher.
WatchProgressRequest provides the barrier we need. etcd only emits a progress response after the watchers on that same stream are synchronized, so its header revision certifies that all earlier events have been delivered. An idle watcher can then advance safely without waiting until its old revision is compacted.
The reference implementation I think we should pursue is:
- Change
lua-resty-etcd to represent /v3/watch as a full-duplex session with operations such as recv(), request_progress(), and close(). The etcd HTTP gateway already accepts multiple WatchRequest messages from one streaming request body, so this should not require restoring APISIX's native gRPC etcd mode.
- When the APISIX watch read times out, send
WatchProgressRequest on the same stream and wait for a bounded response. If a progress response reports revision P, set the next revision to P + 1 and continue or reopen the watch.
- If sending the request fails or no progress response arrives, close the stream and reconnect from the unchanged revision. Never advance based on another connection.
- If etcd actually returns
compacted before a progress barrier is received, keep the snapshot reload as the correctness fallback; at that point it cannot be skipped safely.
Periodic progress_notify alone is not enough as a default solution. etcd's default progress interval is 10 minutes, while APISIX normally restarts an idle watch after about 50 seconds, so the stream is usually closed before a notification can arrive. An explicit request avoids requiring an etcd server-side interval change.
The tests should cover at least:
- an idle APISIX prefix while another prefix advances the global revision through aggressive compaction: progress responses should advance the watcher without a recovery Range request;
- a put or delete immediately before a progress request: the event must be delivered before the progress revision is accepted;
- a silently blackholed watch stream: APISIX must not advance, and must replay from the old revision or reload if that revision was compacted.
Relevant protocol references:
This should remove the normal idle-prefix compact recovery path without reintroducing the event-loss window fixed by #13721.
PR #13721 restores correctness by no longer moving
watch_ctx.revto a revision sampled on a separate connection after a watch timeout. It also reuses unchanged objects during compact recovery. One cost remains: recovery still issues a Range request for each config type in every worker, transfers all values, and haslua-resty-etcddeserialize them beforeload_full_data()can comparemodifiedIndex.Once etcd has returned
compacted, skipping reconciliation is not safe. If the watcher last confirmed revisionRand etcd has compacted throughC, these two histories are indistinguishable after(R, C]has been removed:A revision sampled by a separate Range request does not prove delivery on the watch stream. The revision in a
createdresponse is not a delivery barrier either: etcd sends the create response before flushing buffered historical events for that watcher.WatchProgressRequestprovides the barrier we need. etcd only emits a progress response after the watchers on that same stream are synchronized, so its header revision certifies that all earlier events have been delivered. An idle watcher can then advance safely without waiting until its old revision is compacted.The reference implementation I think we should pursue is:
lua-resty-etcdto represent/v3/watchas a full-duplex session with operations such asrecv(),request_progress(), andclose(). The etcd HTTP gateway already accepts multipleWatchRequestmessages from one streaming request body, so this should not require restoring APISIX's native gRPC etcd mode.WatchProgressRequeston the same stream and wait for a bounded response. If a progress response reports revisionP, set the next revision toP + 1and continue or reopen the watch.compactedbefore a progress barrier is received, keep the snapshot reload as the correctness fallback; at that point it cannot be skipped safely.Periodic
progress_notifyalone is not enough as a default solution. etcd's default progress interval is 10 minutes, while APISIX normally restarts an idle watch after about 50 seconds, so the stream is usually closed before a notification can arrive. An explicit request avoids requiring an etcd server-side interval change.The tests should cover at least:
Relevant protocol references:
WatchProgressRequestis available since etcd 3.4This should remove the normal idle-prefix compact recovery path without reintroducing the event-loss window fixed by #13721.