FIX: reject invalid inputs up-front to avoid PUT timeouts

[JJL772]

Models that take a while to simulate can delay the put done callbacks until they timeout. caput just shows a stale value in its readback, and pvput displays "Put timeout".

Even if we move the done callbacks to before the Model.set call, updates can get stalled in the queue and timeout. For example, if you have a slow model and caput/pvput while the model is simulating.

I'm not sure that there's a good way to implement done callbacks that encompass exceptions thrown during model simulation. The best thing we can do is reject invalid values up front in the CA/PVA callbacks. Luckily PVA allows us to specify an error message, though CA doesn't (hence the LOG.warn).

Curious if you have any better ideas or feedback on this change @tangkong. Do you guys need the caput-waits-for-model-simulation behavior?

Closes #39
Closes #43

[tangkong]

The put-completion was important to us to allow scans to not have hard-coded sleep statements between points. Without this signal we're left guessing at how long a step will take. For simulations that take varying amounts of time, this helps us be more efficient (and not have scans with single long steps force our step-waits to be long across the board)

I might argue that the current behavior is correct. EPICS IOCs are terribly inconsistent in their behavior, but it's not uncommon for records to reject puts while processing is ongoing. We could have an additional PV similar to the PACT (processing active) field that publicizes when the IOC is locked out. (some EPICS docs) This is an alternate way of timing your scans, but still involves some amount of manual polling logic, rather than using built-in put-completion logic in e.g. pyepics.

Even if we move the done callbacks to before the Model.set call, updates can get stalled in the queue and timeout.

Doesn't a model need to finish simulating before any updates to PVs can be meaningful? Or do you mean that other clients attempting to read from the IOC cannot caget properly during a simulation? (am I misunderstanding the issue here?)

[JJL772]

The put-completion was important to us to allow scans to not have hard-coded sleep statements between points. Without this signal we're left guessing at how long a step will take. For simulations that take varying amounts of time, this helps us be more efficient (and not have scans with single long steps force our step-waits to be long across the board)

Makes sense. My original idea for gating involved monitoring output variables, which would only be posted with a new value after the simulation completes. That obviously has some caveats.

I'm going to add a generic status PV to indicate model state.

I might argue that the current behavior is correct.

In terms of rejecting puts while the model is simulating, I tend to agree. The issue I'm seeing though is all pvputs timing out with a slow model, because it takes so long to ACK it.

Doesn't a model need to finish simulating before any updates to PVs can be meaningful? Or do you mean that other clients attempting to read from the IOC cannot caget properly during a simulation? (am I misunderstanding the issue here?)

Since updates are queued along with their done callback, caput/pvput during slow model updates end up timing out too. This isn't really relevant though; I just initially tried to fix it by adding the done callback handling to before the Model.set.

I guess there are a couple of behavioral things we need to think about:

1. PV updates during model execution
   1. Should pvput/caput fail/timeout if they happen while the model is simulating?
   2. Should updates to these input variables be queued up for the next simulation pass? (Current behavior)
      1. This is problematic for variables that act as both inputs and outputs, since the post-simulation update will overwrite the new value received during the previous model run.
2. CA/PVA operation completion
   1. Should these be acknowledged up-front after a quick value validation? (what this PR implements)
   2. Should these stall until the model run completes?
      1. Again, with this, slow models will cause pvput timeouts and caput readback timeouts even if the model simulates fine.

Should we setup a meeting this afternoon to hash this out?

[JJL772]

There are intermittent test failures due to the STATUS PV not connecting. I have zero idea why this is the case; the tests work reliably locally.

[tangkong]

This seems reasonable. It's backwards compatible (so I'm happy). I mostly have questions and suggestions that we could look at later.

In IMMEDIATE mode, it seems like puts are queued up so that as soon as the first sim is complete the next will fire? This seems like it could lead to some confusing behavior where puts get automated and the model just runs sim after sim.

I think my preferred behavior would be to ignore puts while the sim is running, but perhaps that's not what the people wanted

[tangkong]

Looks good to me! Thanks for accommodating my insistence on put-completion 😆

[tangkong]

oh I forgot to mention, I think StrEnum is available in our versions of python (3.11). But it's not an important adjustment

[JJL772]

Fixed. I pretty much just copied LUME base here

[JJL772]

GitHub wasting CI time yet again.