Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions configs/default-config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,8 @@ visible_check_enabled = false
"Left" = "action move_mouse_relative --dx=-10 --dy=0"
"Right" = "action move_mouse_relative --dx=10 --dy=0"

[hints.additional_ax_support]
enable = false # Enable enhanced AX for Electron/Chromium/Firefox/WebKit apps
[hints.web_content_hints]
enabled = false # Chromium/Firefox web-page hints via AXEnhancedUserInterface. Electron works without it
additional_electron_bundles = []
additional_chromium_bundles = []
additional_firefox_bundles = []
Expand Down
4 changes: 2 additions & 2 deletions configs/hints-only-config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ ignore_clickable_check = false
# "Shift+L" = "__disabled__"
# }

[hints.additional_ax_support]
enable = true
[hints.web_content_hints]
enabled = true

additional_electron_bundles = []
additional_chromium_bundles = []
Expand Down
29 changes: 16 additions & 13 deletions docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -474,21 +474,21 @@ y_offset = 24
width = 320
```

### Additional AX Support
### Web Content Hints

Framework-specific accessibility improvements for Electron, Chromium, Firefox, and WebKit apps:
Makes web-page content inside browsers hintable. Electron apps (VS Code, Slack, and any other) are woken automatically on focus and do not need this setting.

| Option | Type | Default | Description |
| ----------------------------- | ----- | ------- | ---------------------------- |
| `enable` | bool | `false` | Enable additional AX support |
| `additional_electron_bundles` | array | `[]` | Bundle IDs of Electron apps |
| `additional_chromium_bundles` | array | `[]` | Bundle IDs of Chromium apps |
| `additional_firefox_bundles` | array | `[]` | Bundle IDs of Firefox apps |
| `additional_webkit_bundles` | array | `[]` | Bundle IDs of WebKit apps |
| Option | Type | Default | Description |
| ----------------------------- | ----- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `enabled` | bool | `false` | Set `AXEnhancedUserInterface` on Chromium and Firefox browsers so hints reach their web-page content |
| `additional_electron_bundles` | array | `[]` | Extra Electron bundle IDs, so neru recognises their web view and prunes the noisy DOM tree while scanning |
| `additional_chromium_bundles` | array | `[]` | Extra Chromium browser bundle IDs that should get web-content hints |
| `additional_firefox_bundles` | array | `[]` | Extra Firefox browser bundle IDs that should get web-content hints |
| `additional_webkit_bundles` | array | `[]` | Extra WebKit browser bundle IDs, so neru prunes their web DOM tree while scanning |

```toml
[hints.additional_ax_support]
enable = false
[hints.web_content_hints]
enabled = false
additional_electron_bundles = []
additional_chromium_bundles = []
additional_firefox_bundles = []
Expand All @@ -497,12 +497,15 @@ additional_webkit_bundles = []

Find bundle IDs: `osascript -e 'id of app "Safari"'`

> [!WARNING]
> `enabled` sets the `AXEnhancedUserInterface` attribute on Chromium and Firefox browsers. Under a tiling or window-snapping manager (yabai, Rectangle, Magnet, Amethyst), turning this attribute on can make macOS relayout or move the browser window. neru sets it only while `enabled` is on, and only on apps it treats as Chromium or Firefox browsers. Native apps never receive it, and Electron apps receive it only if you list them in one of the browser arrays yourself. Leave `enabled` off if the window-move side effect bothers you.

> [!TIP]
> To support installed PWA apps, add a wildcard bundle ID to the appropriate browser array. For example, to support Brave-installed PWAs, add `"com.brave.Browser.app.*"` to `additional_chromium_bundles`:
>
> ```toml
> [hints.additional_ax_support]
> enable = true
> [hints.web_content_hints]
> enabled = true
> additional_chromium_bundles = ["com.brave.Browser.app.*"]
> ```
>
Expand Down
53 changes: 26 additions & 27 deletions docs/TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,13 +122,13 @@ neru hints # CLI works?
log_level = "debug"
```

### Electron/Chromium/Firefox issues
### Browser web content has no hints

**Enable additional AX support:**
Chromium and Firefox keep their web-page accessibility tree asleep until an app asks for it. Turn on the enhanced attribute so hints reach page content (Electron apps do not need this):

```toml
[hints.additional_ax_support]
enable = true
[hints.web_content_hints]
enabled = true
```

### Menubar/Dock hints missing
Expand Down Expand Up @@ -163,35 +163,30 @@ tail -f ~/Library/Logs/neru/app.log

### Hints don't appear in Electron apps

**Electron apps need additional AX support.**
Electron apps (VS Code, Slack, Discord, and any other) are woken automatically the moment they gain focus, with no configuration. neru sets `AXManualAccessibility` on every focused app, which wakes the Electron accessibility tree.

**Solution:**

Edit `~/.config/neru/config.toml`:
If hints still do not appear, turn on debug logging and confirm the wake happened:

```toml
[hints.additional_ax_support]
enable = true

# If your app isn't auto-detected, add it:
additional_electron_bundles = [
"com.your.electronapp",
]
[logging]
log_level = "debug"
```

Restart Neru:
Restart Neru and watch the log:

```bash
pkill neru && neru launch
tail -f ~/Library/Logs/neru/app.log
```

**Check logs for:**
**Check the log for:**

```
App requires Electron support
Enabled AXManualAccessibility for: com.your.app
manual accessibility set
```

If instead you see `manual accessibility set failed`, macOS refused the attribute. For an Electron app that usually means neru is missing macOS Accessibility permission, so check that first. Ordinary native apps that do not implement the attribute log the same line harmlessly. If neither line appears for the app, confirm the app was frontmost when you opened hints.

### Hints don't appear in Chrome/Firefox content

**Browser needs additional AX support.**
Expand Down Expand Up @@ -434,14 +429,12 @@ pkill -9 neru

### VS Code: Hints don't appear in editor

**Electron AX support needed.**

**Solution:**
VS Code is an Electron app and is woken automatically on focus, so hints work on its buttons, tabs, and sidebar without configuration. The editor surface itself exposes a thin accessibility tree, so hints can be sparse inside the editing area. For that surface, switch VS Code to the Vision strategy, which detects elements from the rendered window instead of the AX tree:

```toml
[hints.additional_ax_support]
enable = true
# VS Code is auto-detected
[[hints.app_configs]]
bundle_id = "com.microsoft.VSCode"
strategy = "vision"
```

### Adobe apps: Hints misaligned or missing
Expand Down Expand Up @@ -632,9 +625,15 @@ grep "com.apple.Safari" ~/Library/Logs/neru/app.log

### Common log messages

**"App requires Electron support"** - Electron app detected, needs AX support enabled
**"manual accessibility set"** - neru woke an app's accessibility tree by setting AXManualAccessibility on it

**"manual accessibility set failed"** - macOS refused AXManualAccessibility; harmless for a native app that does not implement it, a permission problem for an app that should

**"enhanced accessibility set for web content"** - neru set AXEnhancedUserInterface on a Chromium/Firefox browser to expose its web page

**"enhanced accessibility set failed"** - macOS refused AXEnhancedUserInterface for a browser; its web-page content may not be hintable

**"Enabled AXManualAccessibility"** - Electron support activated successfully
**"enhanced accessibility cleared"** - neru removed AXEnhancedUserInterface from a browser after web-content hint support was turned off, so its tiling-window side effect does not persist

**"Hints mode activated"** - Hint overlay is active; includes hint count when available

Expand Down
78 changes: 24 additions & 54 deletions internal/app/lifecycle.go
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,13 @@ func (a *App) setupAppWatcherCallbacks() {
a.handleAppActivation(bundleID)
})

// Drop an app's cached accessibility state when it quits, so a later process
// reusing its pid has the attributes set again instead of inheriting stale
// per-process flags.
a.appWatcher.OnTerminate(func(_, bundleID string) {
electron.ForgetAppAccessibility(bundleID)
})

// Watch for display parameter changes (monitor unplug/plug, resolution changes)
a.appWatcher.OnScreenParametersChanged(func() {
a.handleScreenParametersChange()
Expand Down Expand Up @@ -423,66 +430,29 @@ func (a *App) handleAppActivation(bundleID string) {
}

if cfg.Hints.Enabled {
if cfg.Hints.AdditionalAXSupport.Enable {
a.handleAdditionalAccessibility(bundleID, cfg)
}
a.handleAdditionalAccessibility(bundleID, cfg)
}
}

// handleAdditionalAccessibility configures accessibility support for Electron/Chromium/Firefox applications.
// handleAdditionalAccessibility wakes the focused application's accessibility
// tree so hints can read it. It sets `AXManualAccessibility` on any focused app
// (a no-op on apps that do not implement it), as well as
// `AXEnhancedUserInterface` on Chromium and Firefox browsers when
// `hints.web_content_hints` is enabled. The latter is necessary in order to
// expose web-area content but can cause windows to move and prevent tiling
// window managers from working correctly, so it stays off every other app.
//
// The work runs on a background goroutine because EnsureAppAccessibility
// retries a freshly launched app with backoff and would otherwise block the
// activation handler while it waits.
func (a *App) handleAdditionalAccessibility(bundleID string, cfg *config.Config) {
config := cfg.Hints.AdditionalAXSupport

isElectron := electron.ShouldEnableElectronSupport(bundleID, config.AdditionalElectronBundles)
isChromium := electron.ShouldEnableChromiumSupport(bundleID, config.AdditionalChromiumBundles)
isFirefox := electron.ShouldEnableFirefoxSupport(bundleID, config.AdditionalFirefoxBundles)

if !isElectron && !isChromium && !isFirefox {
return
}

go func() {
// Apps may need time to initialize their accessibility tree after launch.
// We retry a few times to ensure the accessibility attributes are successfully set.
// Use exponential backoff to minimize latency for fast-booting apps while
// still accommodating slow-booting ones.
const (
maxRetries = 5
initialDelay = 100 * time.Millisecond
backoffFactor = 2
)

delay := initialDelay
for range maxRetries {
allSuccess := true
axCfg := cfg.Hints.WebContentHints

if isElectron {
if !electron.EnsureElectronAccessibility(bundleID, a.logger) {
allSuccess = false
}
}

if isChromium {
if !electron.EnsureChromiumAccessibility(bundleID, a.logger) {
allSuccess = false
}
}

if isFirefox {
if !electron.EnsureFirefoxAccessibility(bundleID, a.logger) {
allSuccess = false
}
}

if allSuccess {
return
}
useEnhanced := axCfg.Enabled &&
(electron.ShouldEnableChromiumSupport(bundleID, axCfg.AdditionalChromiumBundles) ||
electron.ShouldEnableFirefoxSupport(bundleID, axCfg.AdditionalFirefoxBundles))

// Wait before retrying
time.Sleep(delay)
delay *= backoffFactor
}
}()
go electron.EnsureAppAccessibility(bundleID, useEnhanced, a.logger)
}

// printStartupInfo displays startup information including registered hotkeys.
Expand Down
8 changes: 4 additions & 4 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -722,7 +722,7 @@ type HintsConfig struct {

AppConfigs []AppConfig `json:"appConfigs" toml:"app_configs"`

AdditionalAXSupport AdditionalAXSupport `json:"additionalAxSupport" toml:"additional_ax_support"`
WebContentHints WebContentHints `json:"webContentHints" toml:"web_content_hints"`

Hotkeys map[string]StringOrStringArray `json:"hotkeys" toml:"-"`
}
Expand Down Expand Up @@ -919,9 +919,9 @@ type HeldRepeatConfig struct {
Interval int `json:"interval" toml:"interval_ms"` // Interval between subsequent repeats (ms)
}

// AdditionalAXSupport defines accessibility support for specific application frameworks.
type AdditionalAXSupport struct {
Enable bool `json:"enable" toml:"enable"`
// WebContentHints configures accessibility for hints inside browser web-page content.
type WebContentHints struct {
Enabled bool `json:"enabled" toml:"enabled"`
AdditionalElectronBundles []string `json:"additionalElectronBundles" toml:"additional_electron_bundles"`
AdditionalChromiumBundles []string `json:"additionalChromiumBundles" toml:"additional_chromium_bundles"`
AdditionalFirefoxBundles []string `json:"additionalFirefoxBundles" toml:"additional_firefox_bundles"`
Expand Down
4 changes: 2 additions & 2 deletions internal/config/config_defaults.go
Original file line number Diff line number Diff line change
Expand Up @@ -437,8 +437,8 @@ func newDefaultConfig() *Config {

AppConfigs: []AppConfig{},

AdditionalAXSupport: AdditionalAXSupport{
Enable: false,
WebContentHints: WebContentHints{
Enabled: false,
AdditionalElectronBundles: []string{},
AdditionalChromiumBundles: []string{},
AdditionalFirefoxBundles: []string{},
Expand Down
39 changes: 39 additions & 0 deletions internal/config/service.go
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,45 @@ func (s *Service) LoadWithValidation(path string) *LoadResult {
return configResult
}

// [hints.additional_ax_support] is no longer a valid section; it was replaced
// by [hints.web_content_hints]. The loader ignores unknown keys, so a config
// still using the old name would silently lose its settings. Reject it and
// point at the new key.
if hints, ok := raw["hints"].(map[string]any); ok {
if _, present := hints["additional_ax_support"]; present {
configResult.ValidationError = derrors.New(
derrors.CodeInvalidConfig,
"[hints.additional_ax_support] has been renamed to [hints.web_content_hints]. "+
"Rename that section in your config to keep browser web-content hints.",
)
configResult.Config = DefaultConfig()

s.logger.Warn("Config uses the renamed key hints.additional_ax_support",
zap.Error(configResult.ValidationError))

return configResult
}

// web_content_hints.enable was renamed to web_content_hints.enabled. The
// loader ignores unknown keys, so the old field would silently disable
// web-content hints. Reject it and point at the new key.
if webContentHints, ok := hints["web_content_hints"].(map[string]any); ok {
if _, present := webContentHints["enable"]; present {
configResult.ValidationError = derrors.New(
derrors.CodeInvalidConfig,
"[hints.web_content_hints] enable has been renamed to enabled. "+
"Rename that key in your config to keep browser web-content hints.",
)
configResult.Config = DefaultConfig()

s.logger.Warn("Config uses the renamed key hints.web_content_hints.enable",
zap.Error(configResult.ValidationError))

return configResult
}
}
}

// Decode into typed config struct (separate pass for validation)
_, err := toml.DecodeFile(configResult.ConfigPath, configResult.Config)
if err != nil {
Expand Down
Loading