[PoC, do not merge] push-based auto-refresh on accessibility-tree changes

[gabrielecirulli]

Disclaimer: This code was written by Claude. I steered it toward a solution I thought sensible, and I am opening it as a proof of concept, not a merge candidate.

Status

Do not merge. This is a still-flaky proof of concept for push-based hint auto-refresh, opened to share the approach and gather feedback against #1002. It's a LOT of code. It should probably be done with less code. It is opt-in and off by default, and there is a known miss rate (see Known issues).

Rationale

neru scans the accessibility tree once when hints mode is entered and only re-scans on a few discrete events. If a window's contents change after that scan (a page finishes loading, a banner appears, a menu opens, an Electron view re-renders), the on-screen hints go stale until the user manually refreshes. macOS exposes a push API for exactly this, AXObserver, which delivers a callback when an app posts an accessibility notification, with no polling.

This PR watches the processes a hint scan resolved and re-runs the scan when they actually change, coalesced so bursts collapse into one refresh. The paramount design constraint is resource safety: zero background cost when hints mode is inactive, and no leaked, hanging, or doubled observers or threads.

How it works

• A dedicated, lazily started CFRunLoop thread services AXObserver callbacks. It is created on the first arm and stopped and joined on the last disarm, so an idle neru has no observer thread and no background cost.
• AXObserver create, register, and release, plus run-loop source add/remove, all run on that one thread. That serializes them against the observer's own callbacks, so releasing an observer can never race an in-flight callback, and a synchronous AX registration that hangs stalls only the observer thread, never a keystroke or shutdown.
• The observer callback does integer-only work (it rejects stale-session events by a packed epoch, then signals a coordinator) and retains nothing, so it is O(1).
• A Go Manager owns every observer as an actor: all mutations are commands processed on one goroutine, so Reconcile, DisarmAll, app-terminate, and Close can arrive from different threads without touching the observer map directly.
• Observers are bound to hints mode. Leaving hints mode disarms everything at a single site, and a refresh stays in hints mode so it re-targets across a front-app switch without tearing the thread down.
• The notification set is per-source: the front window watches structure and geometry, menu targets watch only menu and element lifecycle, Notification Center watches window create/destroy, and so on. kAXValueChanged is off by default (the noisiest one).

Changes

• New native bridge internal/core/infra/platform/darwin/axobserver.{h,m,go}.
• New actor package internal/core/infra/axobserver (manager, per-source masks, platform shim).
• Observation targets emitted by the actual scan (internal/core/infra/accessibility, internal/core/ports), so coverage tracks the existing include_*_hints settings with no per-source special-casing, and vision-strategy windows never emit a target.
• A refresh coordinator that debounces the observer and modifier-passthrough feeds into one call and defers a refresh while the user is mid-typing a hint label.
• Stable hint labels: a domain element now carries a native-hash identity, and a label-reuse pass keeps a persisting element's label across a refresh (prefix-free preserved) so labels do not reshuffle on every refresh.
• Web/Electron accessibility enablement reworked from a whitelist to a listless model (see below), gated by a config safeguard.
• Config surface under [hints.auto_refresh] and [hints.additional_ax_support].

Web/Electron accessibility enablement

An Electron or Chromium app exposes no accessibility tree, and posts no notifications, until an assistive tool sets an attribute on it. There are two attributes with very different risk:

• AXManualAccessibility wakes Electron. It is a safe no-op on non-Electron apps and has no window side effect.
• AXEnhancedUserInterface wakes Chromium browsers (Chrome, Arc) and Firefox. It is the attribute known to make some apps relayout or move their windows, and it adds real overhead in the target app.

The old code only tried AXManualAccessibility on a hardcoded Electron whitelist and otherwise went straight to the enhanced attribute. This PR attempts AXManualAccessibility on every activated app except neru itself, so Electron apps work without a whitelist, while escalation to AXEnhancedUserInterface stays gated behind a browser signal so it is never sprayed on native apps (which have no web area and would just get relayouted for nothing). The gate is additional_ax_support.escalate_enhanced (whitelist | off | all).

The whole additional-accessibility path is behind additional_ax_support.enable, off by default, because waking a browser tree is intrusive by nature (the enhanced attribute's window moves). Native apps need none of this and auto-refresh works on them out of the box.

Known issues

• Roughly one refresh in five does not catch fresh hints. Diagnosis: reading the tree makes some apps churn their own elements, which fed back into an infinite refresh loop (visible as continuous flicker after the first selection). The current fix suppresses observer refreshes during neru's own scan plus a short margin. That stops the flicker, but it is a lossy time window: a real change whose notification lands inside it is dropped, so the settled state is sometimes never re-scanned. Adding delay does not help, because the missed notification is discarded regardless of timing. The proper fix is a post-scan content fingerprint (element count plus a bounding-box hash) so self-induced churn can be distinguished from a real change without a lossy window, at which point the suppression window can be removed. Not implemented here.
• Web and Electron auto-refresh is best-effort. It depends on the app posting AX notifications, which Chromium and WebKit do inconsistently, and often not at all on scroll.
• Every refresh re-scans all sources; the observer only fixes detection, not scan scope.
• Under vision strategy the frontmost window does not auto-refresh, because AX gives no push signal for vision-detected pixels. Only AX-backed sources (menubar, Notification Center) refresh under that strategy.

Testing

Manager unit tests (self-exclusion, recycled-pid, idempotent double-disarm and double-close, stale-epoch drop, the generation scheme that voids a reconcile queued behind a disarm), all goleak-clean with a fake platform. A native darwin soak test runs many Reconcile/DisarmAll cycles and asserts balanced CFRelease counts and no run-loop thread at idle. Label-reuse tests assert the carried set stays prefix-free. Field-tested by hand on Electron (Claude), the Settings app, and Arc.

[gabrielecirulli]

@y3owk1n worth taking for a spin if you're interested. Much less flaky after the last few commits and there's also no flicker during updates. One issue is some of the hints change labels during their lifecycle (or appear to due to refresh). It's not able to keep track of their identity. For what it's worth, the same would happen if you refreshed manually, I think. I think this should probably be possible somehow.

[y3owk1n]

@y3owk1n worth taking for a spin if you're interested. Much less flaky after the last few commits and there's also no flicker during updates. One issue is some of the hints change labels during their lifecycle (or appear to due to refresh). It's not able to keep track of their identity. For what it's worth, the same would happen if you refreshed manually, I think. I think this should probably be possible somehow.

i havent actually tested it out manually but just looking through the code briefly, and ya, this is what i meant that it will need lots of different notification sources to sort of guess if the layout actually changes, and it seems like that's the only way to get a more reliable content drifts notification.

This PR is definitely huge, but what i see is that it includes several group of changes that worth to be split out:

• tips and tricks documentation (already covered in another PR that i will go through in a bit)
• improvements on electron's initialisation
• observer watchers initialisation
• re-trigger hints when a change is notified

The one big important thing is that, how should we architect this properly in a more declarative and modular way. If you notice most of our recent commits, we try to stay away from defining new configuration keys due to the fact that configuration controls the whole program and it's less flexible.

Potentially patterns like any of the following:

• "hints --action left_click --repeat --watch"
• `[ "action ax_watch_start", "hints" ]

I have no idea yet what are the best for now (need more thinking), but the goal is that if possible, we do not want to impose changes in config but find a way to make them modular so that users that are interest in this, can build the command out.

I also see that you have added some sort of stable ID and reusing labels across hints refreshes, i am not sure if we need to add these tho, unless the performance is the issue, or else, i think we can just rebuild the whole tree as it is and let the generator do it's thing, so that we don't need to have some magic diffing that is hard to debug later on. I think the label changes across refreshes is expected, we don't need to spend time to ensure their consistency across refreshes. If possible, I don't think we should touch anything on hints and element related code in core at all for this purpose (unless we have a very good reason to).

Overall, amazing work and it proves that this is something that can be work out. The rest are just more on architectural design, ux design and how can we implement this with as little code changes as possible, which requires more planning and thinking.

One last thing that worth for a deep thought is that, is auto refresh worth the hassle of all of these, when it's just 1 key bind away to refresh the hints manually. For this, i can't really provide any opinion, as I don't daily drive hints at all anymore.

[gabrielecirulli]

Thanks! To be fair, it took a while to get Claude set up with context and a plan, so I didn't end up reviewing the code it wrote. I'm not that familiar with the codebase yet, so I'll have to take a deeper dive before restructuring this.

At a high level I agree with you:

• Grouping the changes makes sense. We shouldn't touch the core, just build a mechanism to drive refreshes off notifications.
• If we're able to build a clean solution, I'd consider enabling watch mode by default without a setting, and possibly just exposing an argument to hints to disable it. If it turns out flaky, it should be an opt-in argument to hints.
• The one thing I'd change in core is letting labels stay on screen until the next refresh arrives, to prevent flicker.
• We might need to rework the macOS accessibility defaults. The way accessibility gets enabled for Electron apps, and some of the default mechanisms it relies on, don't seem to work with this approach. For example, hints.additional_ax_support had to be enabled as a setting.

One last thing that worth for a deep thought is that, is auto refresh worth the hassle of all of these, when it's just 1 key bind away to refresh the hints manually. For this, i can't really provide any opinion, as I don't daily drive hints at all anymore.

It is just a keystroke away, but for me it significantly cuts the mental effort of using the tool, since the labels are always where you need them.

[y3owk1n]

Overnight, i had given a bit more thought about the structure of this, and currently I am more inclined in the following way, let me know your thought.

config

[hints.auto_refresh]
enabled = true # default to false (opt-in basis)
debounce_ms = 150
# user should be able to configure what they want, by default we includes every notifications that we supports
# user can remove them from the list or comment it out
# this has to be validated against `enabled` field, if `auto_refresh` is enabled, `allowed_notifications` has to have at least 1 notification allowed.
# these allowed notifications should also control the observers, and we should only register the observers that are allowed by user config.
allowed_notifications = [
	"layout_changed",
	"menu_opened",
	# and more
]

We probably should not make additional_ax_support as mandatory to this feature. We can just document it in the docs where it work best when enabled together, especially on electron or chromium applications.

cli

• no cli flags required for this, as i think it make sense to be automatic when it's enabled in config.
• the only potential clashes might be the --repeat flag. We could either provide a warning when running the hints mode with --repeat flag where it will be ignored, or document it in docs.

docs

• replace all of the action sleep to point to this auto_refresh method.

lifecycle

• very important to ensure all the observers tear down during mode exit cleanup (ensure no memory leak on this part).
• lets not touch on hints generation core too, this should only be responsible to register observers, watch and react, and call the hints generator.

cross platform preparation

• we should also ensure that we had stubs for linux and windows for the observers registration with unsupported code, so that in future, contributors can easily plug in relevant observers for different platform.

---

A separated PR before this lands could be a scoped additional ax enhancements.

[gabrielecirulli]

Thanks, I will incorporate the feedback. I've got a set of stacked PRs coming but I cannot open them as stacked because I'm using a downstream repository of yours. I will try to open them as separate PRs and rebase them as they get merged progressively. Unless you wish to add me as a contributor, then I can make a proper stack. Up to you.

[y3owk1n]

Thanks, I will incorporate the feedback. I've got a set of stacked PRs coming but I cannot open them as stacked because I'm using a downstream repository of yours. I will try to open them as separate PRs and rebase them as they get merged progressively. Unless you wish to add me as a contributor, then I can make a proper stack. Up to you.

I am fine with separated PRs, will do my best to ensure they are merged as soon as I can. 🤝

[gabrielecirulli]

Closing in favor of the chain #1015 #1016 #1017 #1018

I'm not super happy with how it turned out, but I ran out of steam and kind of burned out a bit on working with AI on this, to be honest. In the end, it turns out that the accessibility system does notify on some changes, but not all. As a page progressively loads, you don't get new notifications, which means that we need to implement a kind of exponential back-off re-checking algorithm. That's what I did in #1018.

[gabrielecirulli]

@y3owk1n by the way, maybe don't go too deep on reviewing these just yet because I haven't had a chance to have a deep look into them (my tooling doesn't let me easily review code before this stage), and I don't wanna waste your time. I need a break from this for a while, but once I'm ready, I'll get back and take a look and clean up a bit.

[y3owk1n]

I'm not super happy with how it turned out, but I ran out of steam and kind of burned out a bit on working with AI on this, to be honest. In the end, it turns out that the accessibility system does notify on some changes, but not all.

Yes, this is always the quirk of working with accessibility API, especially with non native apps, and I can understand how annoying it is. No worries about it, it's a long process, especially on open source projects, take your time when you're ready.

by the way, maybe don't go too deep on reviewing these just yet because I haven't had a chance to have a deep look into them (my tooling doesn't let me easily review code before this stage), and I don't wanna waste your time. I need a break from this for a while, but once I'm ready, I'll get back and take a look and clean up a bit.

I'll leave some comments and thought scoped to each PR, feel free to read through when you're ready to continue.