Push notifications and item status for submitted items

[thopkins32]

Currently its a bit tricky to track a single item through the queue. I have to:

1. Submit the item, get its item UID
2. Poll the queue, manually filter for my item UID
3. Poll the status (or get stream updates) to see if my item UID is running
4. Check the history to see if my item completed

It would be a great addition to add an item lifecycle endpoint that allows me to query for the status of my item by its UID.

I.e. item_status(item_uid) -> ItemState or something similar

And also a push-based notification mechanism would be great!

[tacaswell]

This was something we talked about in the first pass (e.g. setting up ws in the httpserver) but defered doing until we determined polling was not good enough.

I think there are two things here:

1. an item status end point
2. push-based update on (1)

Can we do 1 without doing 2 to start or do they have to come together? I we do (2) do we want a full websocket, server-side events, or a webhook?

[thopkins32]

We can do 1 first and then decide how to do 2 properly. I'm still getting familiar with the internals of queueserver, but as a user of the API these additions would be nice to have.

[mrakitin]

We should also look at the server side of bluesky-httpserver as the websockets support is there.

[thopkins32]

Just to document how we do item tracking now:

1. Setup RemoteDispatcher callback
2. Register a callback that filters Bluesky runs based on an injected UUID (we call it a correlation_uid) in RunStart document. This way, our callback only fires when we notice this plan is actually running
3. RemoteDispatcher.start executed in separate "listener" thread
4. Submit plan via REManagerAPI
5. Callback will notice the RunStart document with the correct UUID, processing events until the RunStop comes through

This way, our custom filtering callback notices when the plan that we submitted has started and ended.