App name: WingmanAI
AI persona name: Goose
Version: 1.0
Stack: Flutter macOS + thin Swift method channel
Philosophy: Ambient, proactive, thermally respectful, 100% local by default
WingmanAI is a macOS desktop overlay app powered entirely by local AI inference. Its AI persona — Goose — watches your screen, understands what you are doing in any app, and surfaces proactive advice without you having to ask. When Goose wants to act on your behalf, it proposes an action and waits for your approval. You can talk to Goose directly by voice or text at any time.
Named after Goose from Top Gun — always watching your back, speaking up when it matters, executing when asked.
Nothing leaves your Mac. Ever.
- Ambient over reactive — Goose watches and speaks up. You don't have to go to it.
- Thermally respectful — inference only fires when the screen meaningfully changes. The GPU rests between captures.
- Human in control — every action Goose wants to take requires explicit approval. No surprises.
- Local first — bundled Ollama, bundled models, zero cloud dependency by default. Frontier models are an optional opt-in.
- Hookable — every major subsystem is behind an interface. Swap LLaVA for a better vision model, serial inference for parallel inference, stub features for real ones — without touching the rest of the codebase.
Goose lives in a compact always-on-top floating panel in your chosen screen corner (default: upper-right). It is always visible — on top of every other window, translucent by default, small enough to be unobtrusive.
Compact card — always visible, ~300px wide:
┌─────────────────────────────────────────┐
│ 🪿 WingmanAI ▶ ⏸ ⏹ 👁️ 🔕 ⚙️ │
│ ─────────────────────────────────────── │
│ Looks like you're writing an email. │
│ The subject line could be sharper. │
│ [✏️ Fix it] [💡 Explain] [Dismiss] │
│ ─────────────────────────────────────── │
│ 🧠 14K/128K ████░ RAM 38G CPU 12% │
└─────────────────────────────────────────┘
Expanded chat — full interactive panel, tap anywhere to open:
┌─────────────────────────────────────────┐
│ 🪿 WingmanAI ▶ ⏸ ⏹ 👁️ 🔕 ⚙️ ✕│
│ ─────────────────────────────────────── │
│ [scrollable conversation history] │
│ │
│ 🪿 Goose: The subject line... │
│ 👤 You: Rewrite it │
│ 🌩️ Claude: Here's a sharper version... │
│ │
│ ─────────────────────────────────────── │
│ [────── say anything ──────] 🎤 Send │
│ ─────────────────────────────────────── │
│ 🧠 14K/128K RAM 38G CPU 12% GPU 48% │
└─────────────────────────────────────────┘
Three always-visible buttons in the compact card header. These are the most important controls in the app — they let the user manage Goose's resource consumption at any moment.
| Button | Color | State | What happens |
|---|---|---|---|
| ▶ Play | Green | Active | Screen watching runs, inference enabled, models loaded in GPU RAM |
| ⏸ Pause | Amber | Paused | Capture stops, inference queue freezes, models stay loaded — instant resume |
| ⏹ Stop | Red | Stopped | Everything halts, models explicitly unloaded from VRAM — full resource release |
Pause vs Stop is the critical distinction:
- Pause keeps models warm in GPU RAM. Resuming takes milliseconds. Use it for a quick break.
- Stop evicts models from memory entirely via
OllamaClient.unloadModel(). Resuming takes several seconds while models reload. Use it when you need GPU RAM for something else — video editing, gaming, a heavy build.
State persists across app restarts. If Goose was paused when you quit, it opens paused.
The two highest-leverage features in WingmanAI — telling Goose exactly where your attention is.
Goose tracks NSEvent.mouseLocation continuously. When it analyzes a screenshot, it knows where your cursor is and treats that screen region as the highest-priority area. A terminal pane you're hovering over matters more than the browser window behind it.
When you highlight text anywhere on screen, Goose immediately:
- Reads the exact selected text via
AXSelectedText(Accessibility API — no OCR, exact characters) - Understands the context — which app, what type of content, your conversation history, your active project, your memory
- Reasons about what you most likely want — generates contextually intelligent suggestions using Granite
- Surfaces a selection action card in the overlay
The suggestions are not hardcoded buttons — Granite generates them based on everything it knows about what you've been doing. The same text highlighted in different contexts produces completely different suggestions:
- Highlighted in Mail while discussing a client proposal → "Rewrite formally", "Convert to action item", "Flag as blocker"
- Highlighted in Xcode while debugging → "Explain this code", "Find the bug", "Write a unit test"
- Highlighted in Terminal → "Explain this error", "Suggest a fix", "Search for this"
- Highlighted in Safari → "Summarize this", "Fact-check this", "Save to memory"
A free-text input is always present so you can ask anything beyond what Goose suggested.
When you approve an action, Goose rewrites the text and injects it back in-place — the selection is replaced in whatever app you're in, without you leaving it. The diff view shows old vs new before injection is confirmed.
Text injection methods:
- Primary: app-specific AppleScript (
tell application "Mail" to set selected text...) - Universal fallback: keyboard simulation via
CGEventPost(type over active selection)
This feature requires Accessibility permission — requested and explained at first launch.
| Role | Model | RAM | Context |
|---|---|---|---|
| Vision (screen reading) | llava:7b |
~4.7GB | — |
| Reasoning (chat, advice, tools) | granite3.1-dense:8b |
~5GB | 128K tokens |
Both models are downloaded at first launch. Both are user-configurable. Both ship via the bundled Ollama runtime — no external installation required.
The vision layer sits behind a VisionProvider abstract interface. Swapping LLaVA for a better model requires implementing one class. No other code changes.
VisionProvider (abstract)
OllamaVisionProvider ← default, llava:7b
MoondreamProvider ← hook: ~1.7GB, faster for screen reading
Qwen2VLProvider ← hook: better quality
FrontierVisionProvider ← hook: GPT-4o Vision or Claude Vision
Near-term swap candidate: moondream2 — already on Ollama, significantly faster than LLaVA for screen reading specifically.
When local Granite isn't getting the job done, Goose can optionally escalate to a cloud frontier model. The user configures which providers to use, in what priority order, and how many times per minute escalation is allowed.
Providers: ChatGPT (GPT-4o), Claude (claude-opus-4), Grok (grok-3)
Failback chain: Provider 1 → Provider 2 → Provider 3 → Granite silently
Attribution: "🌩️ Claude says:" vs "🪿 Goose says:" — always visible
API keys: Stored locally in AppSettings, never transmitted except to the provider
Goose captures a screenshot every N seconds (configurable: 3/5/10/30s, default 5s). Before sending it to the vision model, a perceptual hash diff checks whether the screen has meaningfully changed. If not — no inference. The GPU stays cold.
Capture scope options:
- Full screen
- Active window only
- User-defined region (draw a rectangle once, Goose watches only that area)
Privacy exclusions: User can list apps Goose never watches (e.g. 1Password, banking apps). If the frontmost app is on the exclusion list, capture is skipped entirely.
Vision transparency log: The last 20 captures are stored with Goose's conclusions. The user can open this log at any time to understand why Goose said what it said.
Always visible in the performance strip:
🧠 14,203 / 128,000 ████████░░░░░░░░░░ 11%
Color shifts from green → amber at 70% → red at 90%.
When the context window fills (at 90% threshold), Goose:
- Writes a concise summary of the current session — not a transcript, a "state of the world" document similar to Bob's
DESIGN.mdandplan.md - Starts a fresh context window
- Loads the most recent session files that fit within the token budget (most recent first)
One file per context window: gooseSession1-2025-07-14.md
Location: ~/Library/Application Support/WingmanAI/sessions/
The session file contains: what the user is working on, key context, what Goose has done, open threads, things Goose has learned. The user can read and edit these files directly.
Multiple tasks run simultaneously. All I/O (web searches, file reads, shell commands) executes in parallel. Inference is serialized — one at a time, queued. This gives the feel of concurrent work with zero thermal penalty.
ParallelTaskScheduler is built in v1 but hidden behind a settings flag. When the user enables frontier cloud models (no thermal concern) or on more powerful future hardware, they unlock it in Advanced Settings. No code changes — one configuration flag.
Always visible in the compact overlay. Single line in compact mode, full panel in expanded mode.
Compact:
🧠 14K/128K RAM 38G CPU 12% GPU 48%
Expanded: Progress bars for context, RAM, CPU, GPU + Ollama status + which models are loaded.
Auto-throttle: When CPU or GPU hits the red threshold, Goose automatically increases the capture interval. The user sees a small warning and can override.
Goose can search the web and fetch URLs as part of its reasoning — reused directly from deepThinkER.
[SEARCH: query]→ DuckDuckGo HTML endpoint[FETCH: url]→ direct URL fetch- Rate limited (configurable, default 10/min)
- Results injected into Granite's context automatically
- Shown as collapsible "🔍 Searched:" cards in the chat
Proactive research mode: Tell Goose to research something while you keep working. It runs background searches via the deepThinkER ResearchEngine pattern and surfaces a result card when done.
Goose can execute shell commands and control other apps via AppleScript — with your approval on every action.
Approval gate: Every proposed action appears as an approval card in the overlay before executing. You tap Approve or Deny. Approved actions are logged to the audit log.
Shell execution: Process.run — proven pattern from deepThinkER
AppleScript: Process.run('osascript', ['-e', script]) — controls any AppleScript-aware app
Keyboard simulation: CGEventPost — universal fallback for apps without AppleScript support
Requires Accessibility permission — requested at first launch with guided setup.
Persistent memory: Goose remembers facts across sessions — your preferences, your projects, things you've told it. Reused from deepThinkER's MemoryStore.
Project profiles: Save a named context bundle — project folder + pinned files + persona prompt + model overrides. Switch profiles with one tap in the compact overlay.
Auto-learn: Dismissals and thumbs-down are stored in memory and adjust future advice. Consistent thumbs-down on a category can trigger automatic frontier escalation for that category.
Full two-way voice — Goose speaks and listens.
- Speech → Text: macOS
SFSpeechRecognizer— 50+ languages, offline-capable - Text → Speech: macOS
AVSpeechSynthesizer— neural voices, user picks in Settings - Global hotkey: Press
⌘⇧Space(configurable) to activate voice input from anywhere - Wake word: "Hey Goose" — on/off toggle
- Both directions independently toggleable in Settings
1. Splash screen — bundled Ollama starts, live status
2. Hardware detection — RAM tier, GPU backend
3. Model download — llava:7b + granite3.1-dense:8b with progress bars
4. Permissions — Screen Recording + Accessibility guided setup with System Settings deep links
5. Corner picker — choose where Goose lives
6. Persona picker — Developer / Writer / Executive / Default presets
7. Interactive tutorial — how to summon, approve actions, use voice
8. Goose activates — introduces itself by voice and text
| Category | Settings |
|---|---|
| Window | Corner (4 options), opacity, free drag |
| Models | Reasoning model, vision model, model management |
| Screen | Capture interval, capture scope, region selector, privacy exclusions |
| Voice | Input on/off, output on/off, language, voice, hotkey, wake word |
| Search | On/off, rate cap |
| Frontier | On/off, API keys, priority order, throttle, auto-escalate triggers |
| Context | Pinned files, folder, RAG folder |
| Session | Quiet hours, DND, Focus mode |
| Persona | Prompt editor, preset picker |
| Advanced | Task concurrency (hidden v1), thermal override, update check |
| Accessibility | Font size, sound cues |
- LaunchAgent: WingmanAI auto-restarts on crash — installed at first launch
- Crash recovery: State snapshot written before each inference — resumable on next launch
- Audit log: Every tool execution logged with timestamp, tool, argument, result, approval status
- Local-only badge: Visible indicator in the overlay confirming nothing has left your Mac
- Update check: Version check on launch, notification + GitHub link if newer version available
WingmanAI occupies a position no other tool currently holds:
| Tool | What it is | Screen watching | Proactive | Local |
|---|---|---|---|---|
| Ollama | Inference runtime | ❌ | ❌ | ✅ |
| Jan.ai | Local chat UI | ❌ | ❌ | ✅ |
| LM Studio | Local model runner + dev API | ❌ | ❌ | ✅ |
| Unsloth | Model training platform | ❌ | ❌ | ✅ |
| OpenClaw | Messaging gateway | ❌ | ❌ | ❌ needs API keys |
| WingmanAI | Ambient desktop AI | ✅ | ✅ | ✅ |
Every other tool is reactive — you go to them. WingmanAI comes to you. That is the moat.
For a full competitive analysis including OpenClaw deep-dive, see PLAN.md Section 3.
WingmanAI.app
├── Swift (~200 lines)
│ ├── NSWindow floating + corner snap + opacity
│ ├── SFSpeechRecognizer + AVSpeechSynthesizer
│ ├── NSWorkspace active app detection
│ ├── NSEvent mouse location stream
│ ├── AXSelectedText accessibility polling
│ ├── CGEventPost text injection
│ └── FlutterMethodChannel bridge
│
└── Dart / Flutter (all business logic)
├── Bundled Ollama runtime
├── VisionProvider (abstract, LLaVA default)
├── ReasoningEngine (Granite + tools)
├── ScreenWatcher + PerceptualHasher
├── FocusChannel (mouse + selection)
├── SelectionDetector + SelectionActionCard
├── GooseStateManager (Play/Pause/Stop)
├── TaskScheduler (serial v1, parallel hook)
├── FrontierClient (OpenAI + Anthropic + xAI)
├── NetworkFetcher + ResearchEngine (reused)
├── MemoryStore + ProjectProfiles (reused)
├── SessionSummaryWriter + SessionLoader
├── ResourceMonitor + ThermalGuard (reused)
└── AuditLog (reused)
Reused directly from deepThinkER/deepThink: OllamaClient, OllamaLauncher, ModelManager, HardwareDetector, ResourceMonitor, NetworkFetcher, NetworkSearchTool, NetworkFetchTool, AgentTool framework, ToolRegistry, ToolCallInterceptor, FileReadTool, FileWriteTool, MemoryStore, AuditLog, ContextManager, ResearchEngine, AppSettings pattern, AppPaths pattern, entitlements, AppDelegate lifecycle.
Design document written by Bob. See PLAN.md for full implementation sub-tasks.