Auto-refresh hints when accessibility tree changes

[gabrielecirulli]

Pre-flight checks

• I have searched existing issues and discussions and this has not been requested yet

Category

Navigation (hints, grid, recursive grid, scroll)

Target platform

macOS

Problem or use case

There's a setting that auto-refreshes hints when an action is committed: mouse_action_refresh_delay = 0.

The problem is some applications are slow to update their UI after an action, and sometimes flaky.

Proposed solution

If feasible from a performance perspective, it would be great for the hints to auto-update at any moment in their lifecycle when anything on screen changes, or maybe when the accessibility tree changes. There might be macOS APIs that allow us to detect this without having to poll for changes.

Alternatives considered

No response

Screenshots / mockups

mouse_action_refresh_delay can also work when set to a longer delay, but slows down your workflow and won't work in flaky scenarios like electron apps or web navigation

Another alternative is to bind a manual refresh key, in my case cmd+r

Contribution

• I'm willing to submit a PR for this feature

[y3owk1n]

Honestly, this is one of the pain point using accessibly API for hints. There's no easy way to detect whether the tree has changed, except polling it with certain interval. But traversing accessibility tree is very very very expensive, and no one would want to do that on every second. Even as good as homerow, it only supports timer based refresh as far as I remember.

slows down your workflow and won't work in flaky scenarios like electron apps or web navigation

And you definitely found the issue, contents like this normally depends on internet speed and so on, so it's not predictable. The best and most reliable way is to re-run the hints on demand.

[gabrielecirulli]

Fair points, and I won't argue from authority since you know the AX layer better than I do. I dug into this with Claude and a few things made me question the "not feasible" read. Passing them along to get your take, since I can't tell which survive contact with reality.

Its main claim: detecting a tree change and traversing the tree are separate operations, and the cost concern is really about the traversal. macOS has a push API for the detection half, AXObserver (AXObserverCreate / AXObserverAddNotification / AXObserverGetRunLoopSource), which delivers element notifications into a CFRunLoop callback. You register on the frontmost app element for a small set (kAXLayoutChanged, kAXCreated, kAXUIElementDestroyed, kAXWindowResized/Moved, kAXValueChanged, kAXFocusedUIElementChanged) and only get woken when one fires. No timer, no polling. It says registering on the app element covers descendants, so it's one observer per app rebuilt on app switch, and that this is what Hammerspoon and the AX window managers use. That's the part I most want you to sanity-check.

If that holds, the cost model inverts: you traverse only when the UI actually changed, running the same on-demand rescan neru already does, which is fewer traversals than a fixed timer, not more.

On web unpredictability it made a point I hadn't considered: that's the case for events, since a fixed mouse_action_refresh_delay guesses when the page settled while kAXLayoutChanged fires when the layout actually changes, network speed aside.

It was upfront about the limits, which is why I half-trust it. Browsers stay silent on scroll and some JS updates, and Electron/Chromium need AXManualAccessibility (which it noted neru already sets). For the silent cases it suggested a fallback rescan off the existing scroll/key event tap, so the whole thing is additive, not a replacement.

So my real question: is there something that kills this in practice the analysis is missing, like apps not posting reliably or the observer being costlier than it sounds? If it holds up I'll happily prototype it behind a flag to measure real idle cost, but I'd rather hear your read first.

[y3owk1n]

I've done some research on this and personal testing on kAXLayoutChanged which is what relevant here. I am unable to get any notification from it at all, seems like even apple native apps doesn't respect this and post nothing when they should... Feel free to try out some prototype and let me know if I am wrong.

Most softwares uses a more targeted notifications like kAXValueChangedNotification, kAXRowCountChangedNotification, kAXTitleChangedNotification and more, which in reality much more reliable, but they are very targeted to certain part of the layout, and i am able to get notifications for these properly.

apple docs -> https://developer.apple.com/documentation/applicationservices/kaxtitlechangednotification

i have to say that i did not mess around neru codebase yet for the testing, i have another smaller project that targeted to provide simple window and space management with notification bindings, and here's where i tested the layout changes. https://github.com/y3owk1n/mimi

if you're able to get a reliable way to detect content drift without the need to listen to 10 different sources, please share the implementation so that we can brainstorm a better way forward to achieve this goal.

[y3owk1n]

Tho if there's a really good and reliable way for content drift detection, I probably prefer to implement it in project mimi, as this seems like very beneficial for other use cases too instead of just for neru.

I am currently already using mimi for some neru operations personally.

[gabrielecirulli]

Got a POC working, albeit flaky. #1006

CleanShot.2026-07-10.at.00.26.26-converted.2.mp4