TimeOut Error related to detector or Xspress3

[yannachen]

While using Vortex4_1mm and Ge 13-element detector through electronics Xspress3, I encountered the following error. The temporary solution is restarting the detector IOC; sometimes it does not work, and I have to restart the hardware and then the IOC. The error frequency is not so high, around ~ once per week.

Plan failed: <WatchableAsyncStatus, device: vortex_me4, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7f3d9deab100>, errored: TimeoutError("Timeout Error while waiting 11.0s to update ['ca://XSP_ME4_1mm:HDF1:NumCaptured_RBV']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7f3da5ff19d0>, 20)")>


Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/queues.py", line 186, in get
    await getter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_signal.py", line 552, in observe_signals_value
    item = await asyncio.wait_for(q.get(), iteration_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/tasks.py", line 506, in wait_for
    async with timeouts.timeout(timeout):
               ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_status.py", line 37, in wait_with_error_message
    return await awaitable
           ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_status.py", line 221, in _notify_watchers_from
    async for update in iterator:
    ...<6 lines>...
            self._update_watcher(watcher, self._last_update)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_detector.py", line 661, in trigger
    async for update in self._wait_for_index(
    ...<7 lines>...
        yield update
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_detector.py", line 532, in _wait_for_index
    async for sig, value in observe_signals_value(
    ...<15 lines>...
            break
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/ophyd_async/core/_signal.py", line 554, in observe_signals_value
    raise TimeoutError(
    ...<3 lines>...
    ) from exc
TimeoutError: Timeout Error while waiting 11.0s to update ['ca://XSP_ME4_1mm:HDF1:NumCaptured_RBV']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7f3da5ff19d0>, 20)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky_queueserver/manager/worker.py", line 298, in _execute_plan
    result = func()
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky_queueserver/manager/worker.py", line 488, in start_plan_func
    return self._RE(plan_func(*plan_args, **plan_kwargs), {"all": [self._run_reg_cb]}, **plan_meta)
           ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 976, in __call__
    plan_return = self._resume_task(init_func=_build_task)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1121, in _resume_task
    raise exc
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1756, in _run
    raise err
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 1610, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1399, in __call__
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1250, in baseline_wrapper
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 863, in monitor_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 925, in fly_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1399, in __call__
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 1250, in baseline_wrapper
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 863, in monitor_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 925, in fly_during_wrapper
    return (yield from plan2)
            ^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 188, in plan_mutator
    raise ex
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 141, in plan_mutator
    msg = plan_stack[-1].send(ret)
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/utils/__init__.py", line 1279, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/aps_suspenders.py", line 122, in install_suspenders
    yield from plan
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/utils/__init__.py", line 1279, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/open_shutters.py", line 88, in open_shutters_wrapper
    return_val = yield from new_plan
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/src/haven/preprocessors/open_shutters.py", line 159, in open_on_kickoff_wrapper
    response = yield msg
               ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/preprocessors.py", line 213, in plan_mutator
    inner_ret = yield msg
                ^^^^^^^^^
  [Previous line repeated 5 more times]
  File "/home/beams0/S25IDCUSER/src/haven/.pixi/envs/25idc/lib/python3.13/site-packages/bluesky/run_engine.py", line 2378, in _status_object_completed
    raise FailedStatus(ret) from exc
bluesky.utils.FailedStatus: <WatchableAsyncStatus, device: vortex_me4, task: <coroutine object AsyncStatusBase.__init__.<locals>.wait_with_error_message at 0x7f3d9deab100>, errored: TimeoutError("Timeout Error while waiting 11.0s to update ['ca://XSP_ME4_1mm:HDF1:NumCaptured_RBV']. Last observed signal and value were (<ophyd_async.core._signal.SignalR object at 0x7f3da5ff19d0>, 20)")>

[canismarko]

If restarting the IOC fixes it, then most likely it's not a haven bug, or else haven is setting some PV it shouldn't be. Either way, I think we should look at the the IOC for clues.

We should look at:

• Number if images taken by the driver
• Number if frames received by the HDF plugin
• Number if frames written by the HDF plugin

They should all be the same, but one or more will be less than they should be.

Also the IOC console might hold clues. Any errors reported there?

[canismarko]

Can this and #474 be closed, since they're most likely unrelated to haven?