EPICS: Allow reading partial waveforms (first N elements)

[PierreSchnizer]

Large EPICS waveforms (array-valued signals) are currently always transferred in full, even when only a small subset of the data is needed.

For high-rate acquisitions, data is buffered continuously but read out on trigger. In typical use, users select only a small portion of that buffer. Transferring the full array in these cases creates unnecessary network load and does not scale well.

This has a measurable impact in practice: at BESSY II, reading out all ~120 BPMs doubles measurement time due to the volume of data transferred. As a workaround, users are forced to limit the number of devices instead of using all available data sources.

This PR adds support for reading only the first N elements of a waveform (i.e. partial reads of array/spectrum data). It uses the EPICS Channel Access n_elements feature and exposes it via epics_options when defining a signal.

The implementation is currently EPICS-specific. It supports 1 or more elements for channel access and 2 or more elements for pva, as pva only supports only slices above 2 elements. PVA supports slices from start..end, which is limited to 0..end to be compliant what ca provides.

As an example use case, beam position monitors (e.g. Libera Spark, Brilliance) produce turn-by-turn data at ~1 µs intervals and store it in buffers of up to 65k samples. In many applications, only the first few samples are required, making full data transfer unnecessary.

[coretl]

I like the concept, but we need to make sure both CA and PVA can do this. count seems the right approach for CA, but there is no easy concept in PVA. Claude suggests the following approach:

For EPICS 7 IOC waveforms served via QSRV2, array slicing can be requested by the client by appending a subarray field modifier to the channel name in the form PV_NAME.[start:end] (e.g. MY:WF.[0:99]). This is processed server-side by the IOC's dbChannel filter infrastructure, which QSRV2 respects, so no server-side changes are needed — the full waveform is never transmitted. For p4p-based servers such as those created by FastCS, there is no equivalent native mechanism; SharedPV broadcasts the same value to all clients and does not interpret the channel name. Support would need to be added explicitly by registering a DynamicProvider alongside the existing StaticProvider, intercepting channel names with the .[start:end] suffix, and posting sliced numpy arrays to per-slice SharedPV instances whenever the underlying attribute value updates.

Would you be able to try this out for PVA as well please?

[PierreSchnizer]

Yes I will try to get it running with P4P

[PierreSchnizer]

Sorry I was detoured by other tasks.
I will try to implement the comment given above soon

[PierreSchnizer]

Sorry for the delay. I expected much more work to verify what Claude suggested.
Yes it works as suggested for

from p4p.client.thread import Context

pv_name = "MY:WF"
ctx = Context()

start, end = 10, 15
slice = ctx.get(f"{pv_name}.[{start}:{end}]")

I have a look how it could be added.

[PierreSchnizer]

Backends p4p and aioca are tested that conistent outputs are made for methods describe, read and get_value

I am not sure if set_callback should be addressed too, as it also uses the readpv: see line

https://github.com/hz-b/ophyd-async/blob/e8f37e0b1f085a89e470327829a74b05e874a889/src/ophyd_async/epics/core/_p4p.py#L473

[coretl]

Backends p4p and aioca are tested that conistent outputs are made for methods describe, read and get_value

I am not sure if set_callback should be addressed too, as it also uses the readpv: see line

https://github.com/hz-b/ophyd-async/blob/e8f37e0b1f085a89e470327829a74b05e874a889/src/ophyd_async/epics/core/_p4p.py#L473

Yes please, set_callback should also be updated to do the same.

Also, please could you put the tests in tests/system_tests/epics/core/test_epics_signal_mechanisms.py and modify src/ophyd_async/epics/testing/_epics_test_ca_records.db to have the records you require for the tests?

[PierreSchnizer]

set_callback implemented and tested

Moved tests to tests/system_tests/epics/core/test_epics_signal_mechanisms.py
Needed to implement more records than I would need to be compliant with expectations of other tests.

Now epics.testing._devices.EpicsTestCaDevice derives from StandardReadable. Thus this device needs to have a name when instaniated. It will not work without the name.

It would be nice if StandardReadble could check for that (but I don't know how to implement it)