swing-jab-tester

Read and drive Java Swing / AWT desktop apps through the Java Access Bridge (JAB) — so an AI agent (or a plain Python test) can log in, read tables, click buttons, fill forms, and verify show/hide behaviour on a native desktop app, the same way it would on the web.

Windows UI Automation (and therefore tools like pywinauto/Playwright-for-desktop) cannot see Swing controls — an opaque Swing window shows only its native Win32 frame. The real accessibility tree lives behind the proprietary Java Access Bridge. This library reads and drives that tree via pyjab, and bakes in the handful of non-obvious workarounds you need to make it reliable (see Gotchas).

Companion repo: swing-jab-tester-sample — a full sample Swing app (multi-monitor stock trader) with a SQL-backed REST mock and a worked add → test → validate → e2e scenario you can copy from.

---

The host: GitHub Copilot

This library is the tool layer, not the agent. The agent is run by GitHub Copilot — the same GitHub Copilot SDK runtime that drives the computer-use Computer-Use Agent (CUA). Copilot runs the agentic loop (plan → act → observe); swing_tester is the agent's hands on a Java Swing app, exactly as computer-use exposes screen/mouse/keyboard actions as custom Copilot SDK tools. The .github/skills/ and AGENTS.md here are what a Copilot-hosted agent loads to operate as a Swing tester — and you can also call swing_tester directly from a plain Python test, with no agent at all.

---

What you get

• swing_tester.read_jab_tree(title) — snapshot a window's accessibility tree (roles, names, text, states) read-only.
• swing_tester.JabActor(title) — find controls by accessible name and drive them: click, click_mouse (modal-safe), select, set_text, plus nodes() / names() / has_visible(name) for assertions.
• swing-tester CLI — doctor, tree, names, click for quick manual inspection.
• .github/skills/ — three agent skills (setup, testing, test-authoring) so GitHub Copilot can do all of this for you.
• AGENTS.md — the operating guide for an agent acting as a "Swing tester".

Requirements

Need	Why
Windows	The Access Bridge is Windows-only
JDK 17+ (21 recommended) with JAVA_HOME set	Ships WindowsAccessBridge-64.dll and jabswitch
Python 3.10+	Runtime
pyjab (GPLv2)	The JAB client — pip install pyjab

Quick start

# 1. install
python -m venv .venv; .\.venv\Scripts\Activate.ps1
pip install -e .            # installs swing_tester + pyjab, adds the `swing-tester` CLI

# 2. enable the Access Bridge once (writes ~/.accessibility.properties), then restart your Java app
jabswitch -enable

# 3. verify the toolchain
swing-tester doctor

# 4. inspect a running Java window (keep the title glob-safe: no [ ] * ?)
swing-tester tree  --title "My Swing App"
swing-tester click --title "My Swing App" --name "OK"

Full step-by-step setup (including the bridge-DLL details) is in docs/INSTALL.md.

Use it from Python

from swing_tester import JabActor, read_jab_tree

# read
res = read_jab_tree("My Swing App")
print(res.n_interactive, "interactive /", res.n_nodes, "nodes")

# drive
app = JabActor("My Swing App")
app.click("Sign in")                 # programmatic JAB action
app.set_text("Quantity", "100")
assert app.has_visible("Order summary")   # JAB 'showing' state, not 'visible'
app.click_mouse("Confirm")           # real OS mouse click — required for MODAL dialogs

See docs/USAGE.md for the full workflow and docs/PATCHING.md for the package-level patches.

Gotchas already solved

These are the things that make raw pyjab painful — this library / its docs handle them:

1. Bridge-DLL path. pyjab's default looks under %JAVA_HOME%\jre\bin, which modern JDKs don't have. find_bridge_dll() locates WindowsAccessBridge-64.dll in System32 / JAVA_HOME\bin and passes it explicitly — so no admin System32 copy is needed.
2. Glob-safe window titles. pyjab matches windows with fnmatch, so a title containing [ ] * ? never matches. Keep frame titles plain (e.g. App - James Smith, not App [James Smith]).
3. showing vs visible. A control hidden via setVisible(false) on a parent still reports its own visible flag true; only the JAB showing state reflects what's on screen. has_visible() checks showing.
4. Modal dialogs hang programmatic clicks. A modal dialog's open action blocks the JAB action thread, so click() hangs. click_mouse() (a synthesized OS mouse click) drives modals fine.
5. Combo selection. pyjab.select() raises KeyError: 'label' on Swing JComboBox whose items render as labels — prefer radios/checkboxes/buttons for cascading triggers, or pick the default.

License

MIT (this project). pyjab is GPLv2 and installed separately — see LICENSE.