Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,13 +85,20 @@ Full env / prefs: [SETUP.md](SETUP.md). Never commit tokens.
## Features

- Radiolist / checklist; recommended option first; options capped at **8**
- Keyboard: **1–8** select · **Enter** OK · **Esc** cancel (footer hint;
see [docs/AGENTS.md — Dialog UX](docs/AGENTS.md#dialog-ux-humans))
- Keyboard: **1–8** select · **Enter** OK · **Esc** cancel · **Ctrl+V** image
(footer hint; see [docs/AGENTS.md — Dialog UX](docs/AGENTS.md#dialog-ux-humans))
- **Readable Confirm bodies:** first line (the ask) always fully visible;
Command / To+body / notes scroll underneath — so tall gates do not hide the
payload or push Cancel/OK off-screen
- Optional **image / images** preview in the dialog (Linux Gtk; click to
enlarge, **F** / header to maximize; primary monitor only)
enlarge, **F** / header / **double-click title** to maximize; primary
monitor only). Multi-image is a **carousel** (one still at a time; click
left/right of the still, Prev/Next, or ←/→)
- **Ctrl+V paste references** (Linux Gtk Nebula): thumbnails in-dialog;
returned as MCP image blocks (max 4, no lasting files)
- **Idle timeout hold:** default `timeout_sec=0` (waits). If a timeout is set,
typing / paste / select cancels auto-close until OK/Cancel
- Linux aesthetic: **Nebula** dark glass (inspired by the Windows WebView dialog)
- Remembers last dialog size (`prefs.window`; position on Windows; size-only
on typical Wayland)
- Windows: scrollable option list + same lead/detail Confirm layout
Expand Down Expand Up @@ -127,8 +134,7 @@ Full table: [Dialog UX (humans)](docs/AGENTS.md#dialog-ux-humans).
|-------|------|----|-------|--------|
| Ubuntu 24.04 + GNOME + PipeWire | Cursor | Yes | Yes | **Verified** (2026-07) |
| Same stack | Claude Code | Yes | Text-only | **Verified** (2026-07) — [@jackghx](https://github.com/jackghx) |
| Windows 10/11 + WebView2 | Cursor | Text (Nebula UI) | No | Phase 2 UI — **Anthony laptop** |
| Windows 10/11 + tkinter | Cursor | Text | No | Fallback if pywebview missing |
| Windows 10/11 + tkinter | Cursor | Text | No | Phase 1 — **not yet reported** |
| macOS / headless CI | — | No | No | Unsupported / N/A |

More rows & how to report: [CONTRIBUTING.md](CONTRIBUTING.md).
Expand Down
42 changes: 27 additions & 15 deletions docs/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,23 +76,25 @@ skill via `ask-question-install --skill` (`~/.cursor/skills/ask-multiple-choice`
| `speak` | bool | no | default `true` (honours mute env / missing TTS) |
| `title` | string | no | default `"Decide"` — short noun phrase |
| `agent` | string \| null | **strongly yes** | Window title prefix `[agent]` |
| `timeout_sec` | int | no | default `300`; `0` = no timeout |
| `timeout_sec` | int | no | default `0` (no idle auto-close — waits for the human). Positive = soft idle; typing / paste / select **holds** until OK/Cancel. Parent also respects engagement (absolute ceiling ~4h). |
| `entry_seed` | string \| null | no | Prefill Something else / entry |
| `image` | string \| null | no | Local PNG/JPEG (etc.) path or `file://` URI — Gtk preview above the question (Linux). Missing/unsupported files are skipped. |
| `images` | string[] \| null | no | Same as `image`, up to 4 paths (combined with `image`, deduped). Prefer one clear still when possible. |
| `image` | string \| null | no | Local PNG/JPEG (etc.) path or `file://` URI — preview above the question (Linux Gtk + Nebula). Missing/unsupported files are skipped. |
| `images` | string[] \| null | no | Same as `image`, up to 4 paths (combined with `image`, deduped). Gtk shows a **carousel** (one still at a time; click left/right of the still, Prev/Next, or ←/→). Prefer one clear still when possible. |

**Images in the dialog (Linux Gtk):** pass an absolute path or `file://` URI so
**Images in the dialog (Linux Gtk + Nebula):** pass an absolute path or `file://` URI so
Alex sees the still *inside* the MCQ (not only in chat). Chat `Read` of a PNG
does not put pixels in the dialog — use `image` / `images`. When images are
present the window opens large on the **primary** usable workarea (not the
largest / secondary 4K); click the preview to toggle compact (~320px) vs large,
and use the header maximize button or **F** for a soft-fill on the host panel.
**Multi-image (`images=`, max 4): the whole stack must fit ≤ primary usable
resolution** — previews share one height budget and scroll inside; never open a
window taller/wider than the primary (or smaller host) display. Text-only MCQs
stay compact. Windows Phase 1 ignores these args (text-only). Pattern:
`mcq-with-image` (signed-off — agents **must** pass `image=`/`images=` when the
human must judge a still).
and use the header maximize button, **F**, or **double-click the title bar**
for a soft-fill on the host panel.
**Multi-image (`images=`, max 4): carousel — one still visible at a time**
(click left/right of the still, Prev/Next, or ←/→); each still uses the full
single-image height budget. Do **not** dump several tiny stacked previews.
Text-only MCQs stay compact. Windows Phase 1 ignores these args (text-only).
Patterns: `mcq-with-image` + `mcq-images-one-at-a-time` (agents **must** pass
`image=`/`images=` when the human must judge a still; multi via carousel or
sequential single-image MCQs — never rely on reading chat pixels).

### Example (single choice)

Expand Down Expand Up @@ -172,12 +174,17 @@ footer hint. Useful when coaching a human or writing host docs.
| Input | Behaviour |
|-------|-----------|
| **1–8** (top row or keypad) | Select that option (1-based). Labels show `1 · …`. Multi-select **toggles**. Ignored while the Something else entry is focused. |
| **↑ / ↓** | Move highlight among options; **Enter** confirms (single-select also selects as you move). |
| **Enter** | Confirm OK after the arm delay (same as clicking OK). |
| **Esc** / window close | Cancel. |
| **Audio** (footer checkbox) | Persistent mute for TTS/STT (`prefs.audio_enabled`). Env `ASK_QUESTION_AUDIO=0` also mutes. |
| **Audio** (footer checkbox) | Persistent mute for TTS/STT (`prefs.audio_enabled`). Env `ASK_QUESTION_AUDIO=0` hard-mutes; `=1` does **not** override the checkbox. |
| **R** / **L** | Replay question / Listen (Linux voice only, when configured). |
| **Click preview** (image MCQs) | Toggle large vs compact (~320px) image scale. |
| **F** / header maximize (image MCQs) | Maximize / restore the window so the still can use most of the screen. |
| **Click preview** (image MCQs) | Single still: toggle large vs compact (~320px). Multi-image: **left half → previous**, **right half → next**. |
| **F** / header maximize / **double-click title bar** (image MCQs) | Maximize / restore the window so the still can use most of the screen. |
| **← / →** or **Prev / Next** (multi-image) | Carousel: show previous / next still (one visible at a time). |
| **Ctrl+V** (Linux Gtk + Nebula; Windows Nebula) | Paste clipboard images as in-dialog **References** (max 4). No lasting local files — pixels return in JSON `pasted_images`. |
| **Linux Nebula** | Visual SoT: Anthony’s Windows fork (`theoriginalcheese/ask-question-mcp`). Hosted by `linux_webview_ask.py` (WebKit). Frameless chrome drag uses bridge `begin_move` → `Gdk.Toplevel.begin_move` (WebKit ignores `pywebview-drag-region`). Freeform/refs stay **inside** scrolling `<main>` so Cancel/OK never clip. Voice via `linux_webview_voice.py`; Audio checkbox wins over `ASK_QUESTION_AUDIO=1` (`=0` remains hard mute). Listen needs `ASK_QUESTION_STT_URL`. |
| **Typing / paste / select** | First freeform keystroke, image paste, or option pick **cancels idle `timeout_sec` auto-close** until OK / Cancel / Esc. |

### Reading the question (lead + detail)

Expand Down Expand Up @@ -223,6 +230,11 @@ Parse the string before branching.

**Freeform:** same shape plus `"freeform": true` and `"freeform_text": "…"`.

**Human-pasted references (Ctrl+V):** lean JSON may include
`pasted_image_count` (and optional `pasted_image_notes`). The tool result then
includes MCP **Image** content blocks after the JSON string so the model can
see the stills.

**Cancelled:**

```json
Expand Down Expand Up @@ -284,7 +296,7 @@ Env cheat sheet: [SETUP.md](../SETUP.md#5-env-cheat-sheet).
|---------|------------|
| Tool missing / won’t start | Absolute `uv` path; check `REPO_ROOT`; reload; `check_setup` |
| No dialog | `check_setup` → `display` / `gtk_*`; host must inherit `DISPLAY` |
| Hang / timeout | Human must click; raise `timeout_sec`; off-screen window? |
| Hang / timeout | Default waits forever (`timeout_sec=0`). If you set a positive timeout, typing/paste/select holds it. Off-screen window? |
| Speaks without TTS URL | Local Piper / notify path — mute with Audio / `ASK_QUESTION_AUDIO=0` |
| No speech / no mic | `setup_guide` topic `tts` / `stt`, or mute env |
| Works in terminal, not IDE | Absolute `uv`; restart IDE after install |
273 changes: 273 additions & 0 deletions scripts/smoke_nebula_bridge.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,273 @@
#!/usr/bin/env python3
"""Automated smoke: launch Linux Nebula, wait for bridge, submit, verify result.

Uses system Python (/usr/bin/python3) for WebKitGTK. Does not need a human click.
Optional screenshot when ImageMagick ``import`` or ``gnome-screenshot`` exists.
"""

from __future__ import annotations

import json
import os
import re
import subprocess
import sys
import tempfile
import time
import urllib.error
import urllib.request
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
SCRIPT = ROOT / "src" / "ask_question_mcp" / "linux_webview_ask.py"
GTK_PY = "/usr/bin/python3"


def _post(origin: str, path: str, body: dict) -> dict:
req = urllib.request.Request(
f"{origin}{path}",
data=json.dumps(body).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req, timeout=5) as resp:
return json.loads(resp.read().decode("utf-8") or "{}")


def _try_screenshot(out: Path) -> bool:
env = {**os.environ, "DISPLAY": os.environ.get("DISPLAY", ":0")}
for cmd in (
["gnome-screenshot", "-f", str(out)],
["import", "-window", "root", str(out)],
):
try:
r = subprocess.run(cmd, env=env, capture_output=True, timeout=8)
if r.returncode == 0 and out.is_file() and out.stat().st_size > 1000:
return True
except (OSError, subprocess.TimeoutExpired):
continue
return False


def main() -> int:
if not Path(GTK_PY).is_file():
print("FAIL: /usr/bin/python3 missing", file=sys.stderr)
return 2
display = os.environ.get("DISPLAY", ":0")
runtime = Path(tempfile.mkdtemp(prefix="askq-nebula-smoke-"))
result_path = runtime / "result.json"
payload = {
"question": "Nebula bridge smoke — automated submit (Anthony Windows port).",
"title": "Nebula smoke",
"ids": ["yes", "no", "other"],
"labels": {
"yes": "Looks good (recommended)",
"no": "Still broken",
"other": "Something else",
},
"preselect": ["yes"],
"recommended_ids": ["yes"],
"danger_ids": [],
"dangerous": False,
"allow_multiple": False,
"allow_other": True,
"timeout_sec": 60,
"theme": "glass",
"arm_ms": 250,
"speak_enabled": False,
"voice_answer": False,
"result_path": str(result_path),
}
env = {
**os.environ,
"DISPLAY": display,
"GDK_BACKEND": "x11",
"ASK_QUESTION_GDK_BACKEND": "x11",
"ASK_QUESTION_LINUX_UI": "nebula",
"ASK_QUESTION_AUDIO": "0",
"WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS": "1",
"WEBKIT_DISABLE_DMABUF_RENDERER": "1",
}
print(f"smoke: launching {SCRIPT} display={display}", flush=True)
proc = subprocess.Popen(
[GTK_PY, str(SCRIPT)],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
env=env,
)
assert proc.stdin is not None
proc.stdin.write(json.dumps(payload))
proc.stdin.close()

origin = ""
stderr_buf = ""
deadline = time.time() + 20
while time.time() < deadline and proc.poll() is None:
# Non-blocking-ish read of stderr lines
assert proc.stderr is not None
line = proc.stderr.readline()
if line:
stderr_buf += line
sys.stderr.write(line)
sys.stderr.flush()
m = re.search(r"nebula: bridge=(http://127\.0\.0\.1:\d+)", line)
if m:
origin = m.group(1)
break
else:
time.sleep(0.05)

if not origin:
# Drain a bit more
time.sleep(0.5)
try:
rest = proc.stderr.read() if proc.stderr else ""
except Exception:
rest = ""
stderr_buf += rest or ""
m = re.search(r"nebula: bridge=(http://127\.0\.0\.1:\d+)", stderr_buf)
origin = m.group(1) if m else ""

if not origin:
proc.kill()
print("FAIL: no bridge URL in stderr", file=sys.stderr)
print(stderr_buf[-2000:], file=sys.stderr)
return 1

# Wait until get_payload works (page loaded + JS can talk).
ready = False
for _ in range(60):
if proc.poll() is not None:
break
try:
data = _post(origin, "/api", {"name": "get_payload", "args": []})
if data.get("result") and data["result"].get("ids"):
ready = True
break
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError):
pass
time.sleep(0.15)

if not ready:
proc.kill()
print("FAIL: bridge never served payload", file=sys.stderr)
return 1

# Give paint a beat; probe DOM layout (screenshots are flaky on Wayland).
time.sleep(0.8)
try:
_post(origin, "/event", {"name": "content_ready", "args": []})
except Exception:
pass
time.sleep(0.4)
probe = {}
try:
probe = _post(origin, "/api", {"name": "dom_probe", "args": []}) or {}
if isinstance(probe, dict) and "result" in probe:
# _post already returns parsed body; handler wraps in result.
pass
except Exception as exc:
print(f"FAIL: dom_probe error {exc}", file=sys.stderr)
proc.kill()
return 1

# apiCall-style responses are {"result": ...}; our _post returns full body.
layout = probe.get("result") if isinstance(probe, dict) else None
if layout is None and isinstance(probe, dict) and "ready" in probe:
layout = probe
print(f"smoke: layout={json.dumps(layout, ensure_ascii=False)}", flush=True)
if not isinstance(layout, dict) or layout.get("error"):
print(f"FAIL: bad layout probe {layout}", file=sys.stderr)
proc.kill()
return 1
checks = {
"ready": layout.get("ready") is True,
"theme_glass": layout.get("theme") == "glass",
"stars": int(layout.get("stars") or 0) >= 20,
"freeformInBody": layout.get("freeformInBody") is True,
"footerVisible": layout.get("footerVisible") is True,
"okVisible": layout.get("okVisible") is True,
}
bad = [k for k, ok in checks.items() if not ok]
if bad:
print(f"FAIL: layout checks failed: {bad} full={layout}", file=sys.stderr)
proc.kill()
return 1
print("smoke: layout ok", flush=True)

try:
_post(
origin,
"/api",
{"name": "submit", "args": [["yes"], None, []]},
)
except Exception as exc:
print(f"FAIL: submit error {exc}", file=sys.stderr)
proc.kill()
return 1

# stdin already closed; stderr partially consumed — wait + drain leftovers.
try:
proc.wait(timeout=15)
except subprocess.TimeoutExpired:
proc.kill()
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
pass
print("FAIL: dialog hung after submit", file=sys.stderr)
return 1

stdout = ""
try:
if proc.stdout is not None:
stdout = proc.stdout.read() or ""
except Exception:
stdout = ""
try:
if proc.stderr is not None:
stderr_buf += proc.stderr.read() or ""
except Exception:
pass

# Result file may appear slightly after process exit on hard-bail races.
file_raw = ""
for _ in range(20):
if result_path.is_file() and result_path.stat().st_size > 2:
file_raw = result_path.read_text(encoding="utf-8").strip()
break
time.sleep(0.1)

answer = None
for raw in (file_raw, (stdout or "").strip().splitlines()[-1] if stdout else ""):
if not raw:
continue
try:
answer = json.loads(raw)
break
except json.JSONDecodeError:
continue

print(f"smoke: rc={proc.returncode}", flush=True)
print(f"smoke: stdout={stdout!r}", flush=True)
print(f"smoke: result_file={file_raw!r}", flush=True)

if not answer:
print("FAIL: no JSON answer", file=sys.stderr)
print(stderr_buf[-1500:], file=sys.stderr)
return 1
if answer.get("cancelled"):
print(f"FAIL: cancelled {answer}", file=sys.stderr)
return 1
if answer.get("ids") != ["yes"]:
print(f"FAIL: unexpected ids {answer}", file=sys.stderr)
return 1

print("smoke_nebula_bridge: ok", flush=True)
return 0


if __name__ == "__main__":
raise SystemExit(main())
Loading