Instance federation protocol

[pudo]

Rather than just one-off dump/load, have a pub/sub model?

[simonwoerpel]

You mean in the sense of "master (r&w)" / "slave (r)" instances?
What do you see there as more features rather than master -> (partial diff) dump -> load -> slave 🤔

[pudo]

So I guess there's two use cases here:

1. If you're the admin on both instances and want to transfer bulk data from A to B, then probably load/dump is better. It has higher throughput and probably puts less load on both systems because it would be using bulk APIs both to read and write entities.

2. You're only a reader on one system (let's call it the collection upstream?) and want to transfer the contents of a collection into a second system that you have write access to. I also feel like this scenario is in general going to be better at incremental updates, e.g. pulling in a few new documents every week.

The gold prize here is avoiding an individual call to the /archive API for each document in a collection, which could be millions of calls for a large collection. There's also the issue that indexing data like this would not actually tell us which of the documents are source docs, and which ones are derived. So in effect, a downstream collection could not perform a re-ingest by itself. Maybe not a show stopper.