WingmanAI — Project Plan

App name: WingmanAI AI persona name: Goose Stack: Flutter macOS + thin Swift method channel (~200 lines) Location: /Users/daneyand/Documents/Bob/ProjectO/WingmanAI Philosophy: Ambient, proactive, thermally respectful, 100% local by default

What Is WingmanAI
Why Build This
Competitive Landscape
Architecture Overview
VisionProvider Abstraction
Task Queue Architecture
Play / Pause / Stop Controls
Focus Awareness
Sub-Tasks

1. What Is WingmanAI

WingmanAI is a macOS desktop overlay app powered by local AI inference. Its AI persona, Goose, watches your screen continuously, understands what you are doing in any app, and surfaces proactive advice — all without sending anything off your machine. When Goose wants to act, it proposes a shell command or AppleScript action and waits for your approval before executing. You can also talk to Goose directly, by voice or text, at any time.

Goose runs in a compact always-on-top floating panel in your chosen screen corner. One click expands it to a full interactive chat. It speaks aloud when it has something to say. It listens when you speak to it. It keeps working on tasks in the background while you focus on other things. When its context window fills, it writes a concise summary of what was happening and loads it fresh — so it never loses the thread of your work.

WingmanAI is named after Goose, Maverick's wingman in Top Gun — always watching your back, speaking up when it matters, executing when asked.

2. Why Build This

Every AI assistant tool available today is reactive — you open it, ask a question, get an answer, close it. None of them watch your screen. None of them understand what you are doing right now. None of them speak up without being asked. WingmanAI fills that gap.

The Hardware Moment

Two years ago this was not viable. Local vision models were too weak. Local reasoning models had insufficient context. Today:

llava:7b via Ollama can read a screenshot and understand it
granite3.1-dense:8b has 128K context — your entire codebase in one window
A 64GB Apple Silicon Mac can run both simultaneously without breaking a sweat
Ollama is mature, bundled, and proven in deepThinkER and deepThink

The Foundation Already Exists

Approximately 40% of WingmanAI's core is already written in the deepThinkER and deepThink projects:

OllamaClient — streaming inference, model management
OllamaLauncher — bundled runtime lifecycle
ModelManager — download, check, status
HardwareDetector — RAM tier, GPU detection
ResourceMonitor — CPU, RAM, process polling
AppSettings + AppPaths — settings persistence pattern
NetworkFetcher + NetworkSearchTool + NetworkFetchTool — web search
MemoryStore — persistent memory
AuditLog — append-only action logging
ContextManager — token tracking and reset
ResearchEngine — background research with phase transitions
AgentTool / ToolRegistry / ToolCallInterceptor — tool call framework
FileReadTool / FileWriteTool — file I/O tools
Entitlements, AppDelegate pattern, Ollama PID lifecycle — all solved

The Privacy Differentiator

Microsoft Copilot Recall, Google AI features, and Apple Intelligence all read your screen and send data to cloud servers. A significant number of developers actively want the opposite — full ambient AI awareness with zero data leaving the machine. WingmanAI owns that position entirely.

The Cost Differentiator

No API fees. No subscriptions. Inference is free after the hardware is paid for. Frontier model escalation is optional and throttled by the user.

3. Competitive Landscape

The Tools

Ollama A local inference runtime — the engine under the hood. Not a UI, not an assistant. You run ollama run granite3.1-dense:8b in a terminal. No screen watching, no overlay, no voice, no tools. WingmanAI uses Ollama as its bundled inference backend. Ollama is infrastructure, not competition.

Jan.ai A polished desktop chat app (6.3M downloads, 44K GitHub stars). Runs local models, has an OpenAI-compatible API server, MCP connectors, and an autonomous agent mode. Beautiful UI. Think local ChatGPT replacement — you open the window, type, get answers. No screen watching, no overlay, no proactive advice. You still have to go to it.

LM Studio (Bionic) Recently rebranded with "Bionic," their agent for work and code. Local model runner with a polished UI, model library, and an OpenAI-compatible local server. Strong developer focus — VS Code integrations, local API for tools. Again: you open the window and ask. No ambient awareness, no overlay, no vision of your screen.

Unsloth The most different from the rest. Its primary superpower is fine-tuning and training models 2× faster with 70% less VRAM. It also runs models, has web search, RAG, and "Unsloth Start" which bridges local models to Claude Code and Codex. Not an assistant — a training platform. The right tool when you want to make a model better, not when you want help while working.

OpenClaw A self-hosted Node.js messaging gateway. Connects chat apps (WhatsApp, Telegram, Discord, Slack, Signal, iMessage, and 20+ more) to AI coding agents. Remote-first — you message it from your phone or laptop and it responds. Has a macOS menu bar app and browser control UI, but its core interaction model is chat-in-a-messaging-app. Requires cloud API keys (Anthropic, OpenAI, etc.). No screen watching, no proactive advice, no local inference. Open source, MIT licensed, community-driven.

Master Comparison Table

Capability	Ollama	Jan.ai	LM Studio	Unsloth	OpenClaw	WingmanAI
Core purpose	Inference runtime	Local chat UI	Local chat + dev API	Model training	Messaging gateway	Ambient desktop AI
Screen watching	❌	❌	❌	❌	❌	✅ Unique
Proactive advice	❌	❌	❌	❌	❌	✅ Unique
Always-on overlay	❌	❌	❌	❌	Menu bar only	✅ Unique
App control / keyboard injection	❌	❌	❌	❌	❌	✅ Unique
Works in any app	❌	❌	❌	❌	❌	✅ Unique
Voice two-way	❌	❌	❌	❌	TTS only	✅
Live screen vision	❌	❌	❌	❌	❌	✅
Shell execution	❌	❌	❌	❌	✅	✅ + approval gate
Web search	❌	❌	❌	✅	✅	✅
Persistent memory	❌	❌	❌	❌	MEMORY.md	✅
Git awareness	❌	❌	❌	❌	❌	✅
Session continuity files	❌	❌	❌	❌	❌	✅ Unique
Model fine-tuning	❌	❌	❌	✅ Best-in-class	❌	❌
100% local / offline	✅	✅	✅	✅	❌	✅
Zero API cost	✅	✅	✅	✅	❌	✅ optional
Bundled — no setup	❌	✅	✅	✅	❌ npm + Node	✅
IBM Granite first-class	Runs it	Runs it	Runs it	Runs it	❌	✅ Default
Thermal management	❌	❌	❌	❌	N/A	✅ diff-gated
Frontier model escalation	❌	❌	❌	✅ cloud	✅ cloud only	✅ local-first + optional cloud
Parallel task queue	❌	❌	❌	❌	❌	✅

WingmanAI vs OpenClaw — Deep Dive

OpenClaw and WingmanAI solve fundamentally different problems. OpenClaw answers: "How do I reach my AI from anywhere — my phone, my messaging apps, remotely?" WingmanAI answers: "How do I have an AI that watches what I am doing right now, on this machine, and helps without me having to ask?"

OpenClaw's power comes from cloud models (Anthropic, OpenAI) — your conversations go to external servers. WingmanAI never sends anything off your Mac.

OpenClaw has zero screen watching capability. Its macOS presence is a menu bar icon and a browser dashboard. WingmanAI's entire value proposition is built on ambient screen awareness.

They are complementary, not competing. OpenClaw is the AI you reach from anywhere. WingmanAI is the AI that watches your back while you work.

WingmanAI's Genuine Position

Every tool above is reactive. You have to go to them. WingmanAI is ambient and proactive — it sees your screen, knows what app you are in, speaks up when it has something useful, and acts when you approve. No other tool in this space does this. That is the moat.

4. Architecture Overview

WingmanAI.app
├── Swift layer (~200 lines total)
│   ├── NSWindow.level = .floating       (always-on-top)
│   ├── NSWindow.setFrame()              (corner snapping)
│   ├── NSWindow.alphaValue              (opacity)
│   ├── SFSpeechRecognizer               (speech-to-text)
│   ├── AVSpeechSynthesizer              (text-to-speech)
│   ├── NSWorkspace.frontmostApplication (active app detection)
│   └── FlutterMethodChannel             (bridge to Dart)
│
└── Dart / Flutter layer
    ├── core/ (pure Dart, zero Flutter imports)
    │   ├── ollama/          OllamaClient, OllamaLauncher, ModelManager, HardwareDetector
    │   ├── vision/          VisionProvider (abstract), OllamaVisionProvider (default)
    │   ├── screen/          ScreenWatcher, PerceptualHasher, RegionSelector
    │   ├── reasoning/       ReasoningEngine (Granite chat + tool calls)
    │   ├── tasks/           TaskScheduler (abstract), SerialTaskScheduler (v1)
    │   │                    ParallelTaskScheduler (hook, dormant in v1)
    │   ├── tools/           AgentTool, ToolRegistry, ToolCallInterceptor
    │   │   ├── network/     NetworkSearchTool, NetworkFetchTool (reused)
    │   │   ├── shell/       ShellTool (enabled with approval gate)
    │   │   ├── applescript/ AppleScriptTool (new)
    │   │   └── file/        FileReadTool, FileWriteTool (reused)
    │   ├── network/         NetworkFetcher, RateLimiter (reused)
    │   ├── research/        ResearchEngine (reused from deepThinkER)
    │   ├── frontier/        FrontierClient, OpenAIProvider, AnthropicProvider, GrokProvider
    │   ├── memory/          MemoryStore (reused), ProjectProfile
    │   ├── session/         SessionSummaryWriter, SessionLoader
    │   ├── context/         ContextManager (reused)
    │   ├── system/          ResourceMonitor (reused), ThermalGuard, ActivityDetector
    │   ├── clipboard/       ClipboardWatcher
    │   ├── git/             GitInspector
    │   ├── rag/             RagIndexer (hook in v1)
    │   ├── snippets/        SnippetLibrary
    │   ├── feedback/        FeedbackLog, QualityTracker
    │   ├── audit/           AuditLog (reused)
    │   ├── settings/        AppSettings (new Goose fields)
    │   └── paths/           AppPaths (new WingmanAI paths)
    │
    └── ui/ (Flutter widgets)
        ├── overlay/         CompactCard, ExpandedChat, ActionApprovalCard
        ├── performance/     PerformanceStrip, PerformancePanel
        ├── tasks/           TaskQueuePanel, TaskCard
        ├── settings/        SettingsScreen (full parity + Goose-specific)
        ├── firstlaunch/     SplashScreen, DownloadScreen, PermissionsScreen
        │                    CornerPickerScreen, PersonaPickerScreen, TutorialScreen
        ├── transparency/    VisionTransparencyLog
        ├── snippets/        SnippetLibraryPanel
        └── theme/           AppTheme, AppColors (Goose palette)

State Management

Same pattern as deepThinkER — pure Dart streams, setState() in widgets, no external state management packages.

Entitlements Required

com.apple.security.app-sandbox = false (Ollama subprocess)
com.apple.security.automation.apple-events = true (AppleScript)
com.apple.security.network.client = true (web search, frontier APIs)
Screen Recording — user-granted at first launch via System Settings
Accessibility — user-granted at first launch via System Settings

5. VisionProvider Abstraction

LLaVA 7B is the default vision model but it is not the final word. The vision layer is abstracted behind an interface from day one so swapping to a better model requires changing one implementation, not touching the rest of the codebase.

abstract class VisionProvider {
  /// Analyze a screen capture and return a natural-language description.
  /// [imageBytes] is the raw PNG/JPEG bytes of the capture.
  /// [context] is optional hint text (active app name, user task, etc.)
  Future<String> analyzeScreen(Uint8List imageBytes, String context);

  /// Human-readable name shown in Settings.
  String get displayName;

  /// Whether this provider is currently available and healthy.
  Future<bool> get isAvailable;
}

Implementations

Class	Model	Notes
`OllamaVisionProvider`	`llava:7b`	Default — bundled, local, free
`MoondreamProvider`	`moondream2`	Hook — ~1.7GB, faster for screen reading
`Qwen2VLProvider`	`qwen2-vl:7b`	Hook — better quality vision
`FrontierVisionProvider`	GPT-4o Vision / Claude Vision	Hook — cloud, requires API key

Swap Guide

To replace LLaVA with a better model:

Implement VisionProvider in a new class
Register it in VisionProviderRegistry
Expose it in Settings vision model dropdown
Done — zero changes to ScreenWatcher, ReasoningEngine, or any UI

Near-Term Swap Candidates

moondream2 — already on Ollama at ~1.7GB, significantly faster than LLaVA for screen reading tasks specifically, likely first upgrade
qwen2-vl:7b — stronger general vision quality
GPT-4o Vision — frontier quality when accuracy matters more than privacy

6. Task Queue Architecture

v1 — Parallel I/O, Sequential Inference

Multiple tasks run simultaneously. All I/O (web searches, file reads, shell commands, AppleScript calls) executes in parallel across tasks. Inference (the GPU-heavy reasoning step) is serialized — one at a time, queued. This gives the feel of concurrent work with zero thermal penalty.

abstract class TaskScheduler {
  /// Max concurrent inference tasks. v1 default: 1 (serial, thermally safe).
  /// Future: set to N for true parallel inference with cloud models or
  /// more powerful hardware.
  int get maxConcurrentInference;

  Future<TaskResult> schedule(GooseTask task);
  void cancel(String taskId);
  void cancelAll();
  Stream<TaskEvent> get taskStream;
}

v1 Implementation: SerialTaskScheduler

maxConcurrentInference = 1
I/O steps run via Future.wait() across all queued tasks
Inference steps join a single queue, processed one at a time
Thermal guard: if CPU/GPU hits red threshold, I/O concurrency also throttles

Future Implementation: ParallelTaskScheduler

maxConcurrentInference = N (user configurable in Advanced Settings)
All steps — I/O and inference — run fully in parallel
Built and tested in v1, but not exposed in the Settings UI
Activated by: frontier cloud models enabled (thermals irrelevant) OR user explicitly unlocks in Advanced Settings
Thermal interlock: bypassed when all tasks route to cloud models

The Single Unlock

In AppSettings:

/// Hidden in v1 UI. Exposed in v2 Advanced Settings.
/// 1 = serial (default). N = parallel inference pool size.
final int maxConcurrentInferenceTasks; // default: 1

Progression Path

v1 today:    Local Granite, serial, maxConcurrent=1, thermally guarded
Near term:   User gets M4 Ultra, bumps to 2, still local, still cool
Cloud on:    Claude handles tasks, maxConcurrent=5, thermals irrelevant
Future:      Apple Silicon 10x faster, maxConcurrent=N, all local, all fast

7. Play / Pause / Stop Controls

Three always-visible transport controls in the compact overlay. They represent three distinct system states with clear resource implications.

Button	State	Models	Inference	Screen Capture	Resume Speed
▶ Play	Active	Loaded in GPU RAM	Enabled	Running	Instant
⏸ Pause	Paused	Stay loaded in GPU RAM	Frozen	Stopped	Instant — models already warm
⏹ Stop	Stopped	Unloaded from all memory	Disabled	Stopped	Seconds — models must reload

The Critical Distinction

Pause is a temporary hold — like putting a conversation on hold. Models stay resident in VRAM. Pressing Play resumes in milliseconds with zero reload cost. Use this when you need a thermal break or want to focus without Goose interrupting.

Stop is a full resource release — models are explicitly evicted from GPU RAM and system memory via OllamaClient.unloadModel() (POST to /api/generate with keep_alive=0). This is already implemented in deepThinkER. Use this when you need all your RAM and GPU for something else — a game, video editing, a heavy build. When you press Play again Goose reloads exactly as it did at first launch.

UI Placement

The three buttons live in the compact card header — always visible regardless of overlay state. They are large enough to tap without expanding the overlay. Color coded:

▶ Green when stopped/paused (actionable — press to activate)
⏸ Amber when active (press to pause)
⏹ Red when active or paused (press to fully stop)

State Persistence

The last state is saved to AppSettings. If Goose was paused when the app quit, it reopens paused. If it was stopped, models are not reloaded until the user presses Play.

8. Focus Awareness

Two complementary layers that tell Goose where the user's attention is and what they want help with.

Layer 1 — Mouse Position Awareness

Goose tracks NSEvent.mouseLocation continuously via the Swift method channel. When analyzing a screenshot, the mouse coordinates are passed as context to the VisionProvider. The vision model is instructed to treat the region around the cursor as the highest-priority area — if the mouse is hovering over a terminal pane in a split editor, that pane is more important than the browser window behind it.

This is cheap — a simple coordinate stream from Swift to Dart, no additional inference cost.

Layer 2 — Text Selection Awareness (Key Feature)

When the user highlights text anywhere on screen, Goose:

Detects the selection via the macOS Accessibility API (AXSelectedText attribute on the focused element) — exact text, no OCR needed
Reads the full selected text directly
Understands the context — which app, what type of content (email, code, document, chat), conversation history, active project, memory
Reasons about what the user most likely wants — generating contextually intelligent suggestions specific to that text in that moment, informed by everything Goose knows about what the user has been doing
Surfaces a selection action card in the compact overlay immediately
Executes the chosen action and injects the result back into the app in-place, replacing the selected text

The Selection Action Card

The suggestions on the card are not hardcoded buttons — they are generated by Granite reasoning over the selected text plus full conversation context. The card always includes a free-text input so the user can ask anything beyond what Goose suggested.

Example — user has been discussing a client presentation for 20 minutes, then highlights a paragraph in Mail:

┌─────────────────────────────────────────────┐
│ 🪿 You selected:                            │
│ "We need to circle back on the Q3           │
│  deliverables ASAP"                         │
│                                             │
│ Given your presentation context:            │
│  • Rewrite for the client (formal tone)     │
│  • Convert to a clear action item           │
│  • Flag this as a blocker in your notes     │
│  • Translate to French for Marie            │
│                                             │
│ Or ask me anything about it...              │
│ [_________________________________________] │
└─────────────────────────────────────────────┘

The suggestions change entirely based on context:

Same text highlighted in Xcode → "Explain this code", "Find the bug", "Write a test for this"
Same text highlighted in Terminal → "Explain this error", "Suggest a fix", "Search for this"
Same text highlighted in a browser → "Summarize this", "Fact-check this", "Save to memory"

Text Injection Back Into the App

When Goose rewrites or transforms the selected text, it injects the result back in-place:

Primary method — app-specific AppleScript:

tell application "Mail"
    set selected text of front message to rewrittenText
end tell

Universal fallback — keyboard simulation:

CGEventPost: type rewritten text over selection
(selection already active — typing replaces it)

The diff view (already in the feature set) shows old vs new before injection is confirmed, so the user always has final approval.

Requires

Accessibility permission — already in the first launch permission setup flow
AXSelectedText polling — fires SelectionEvent to Dart when selection changes
NSEvent.mouseLocation — already part of Sub-Task 3 Swift method channel

What This Enables

This is the highest-leverage feature in WingmanAI. It transforms Goose from a passive observer into an active collaborator in any app — Mail, Xcode, VS Code, Pages, Notion, Terminal, browser, Slack — without requiring those apps to have any AI integration of their own. The user's selection is the universal interface.

9. Sub-Tasks

Sub-Task 1 — Project Scaffold

Status: [x] done

Intent Create the WingmanAI Flutter macOS project from scratch, mirroring the deepThinkER project structure. Establish the core directory layout, pubspec.yaml with all required dependencies, base entitlements, AppDelegate, and the folder structure for all planned modules. No business logic yet — just the skeleton every subsequent sub-task builds on.

Expected Outcomes

flutter create project exists at /Users/daneyand/Documents/Bob/ProjectO/WingmanAI
lib/core/ and lib/ui/ directory structure matches the architecture in Section 4
pubspec.yaml includes all dependencies
macOS entitlements configured (sandbox off, Apple Events, network)
AppPaths and AppSettings scaffolded with all WingmanAI-specific fields
App builds and launches to a blank window without errors

Todo List

Dependencies to add to pubspec.yaml

http: ^1.2.0              # Ollama API + web search + frontier APIs
path_provider: ^2.1.0     # App support directory
url_launcher: ^6.2.0      # Open URLs, System Settings deep links
flutter_local_notifications: ^18.0.0  # Desktop notifications
audioplayers: ^6.0.0      # Sound cues
archive: ^3.6.0           # Session export ZIP

Relevant Context

deepThinkER pubspec: deepThinkER/pubspec.yaml
deepThinkER AppDelegate: deepThinkER/macos/Runner/AppDelegate.swift
deepThinkER entitlements: deepThinkER/macos/Runner/DebugProfile.entitlements
deepThinkER AppPaths: deepThinkER/lib/core/paths/app_paths.dart
deepThinkER AppSettings: deepThinkER/lib/core/settings/app_settings.dart

Sub-Task 2 — Bundled Ollama + Model Management

Status: [x] done

Intent Port the bundled Ollama runtime from deepThinkER into WingmanAI. Copy the Ollama binary and libraries, adapt OllamaLauncher, OllamaClient, ModelManager, and HardwareDetector. Establish the two default models (llava:7b, granite3.1-dense:8b) in a new ModelRegistry for WingmanAI.

Expected Outcomes

Ollama binary ships in .app/Contents/Resources/ollama/
OllamaLauncher.start() spawns Ollama and writes PID file
OllamaClient.generateStream() works for both chat and vision (base64 image) requests
ModelManager.checkModels() detects llava:7b and granite3.1-dense:8b
HardwareDetector.detect() returns RAM tier and GPU backend
Model management panel: shows installed models, size, re-download, delete

Todo List

Copy macos/Runner/ollama/ directory from deepThinkER
Copy and adapt copy_ollama.sh build phase script
Port OllamaLauncher — update PID file name to wingmanai_ollama.pid
Port OllamaClient — add analyzeImage(base64, prompt) method for vision
Port ModelManager — update default models to llava:7b + granite3.1-dense:8b
Port HardwareDetector
Write ModelRegistry with WingmanAI model entries including Granite variants
Write model management panel UI
Confirm Ollama starts, models listed, health check passes

Relevant Context

deepThinkER OllamaLauncher: deepThinkER/lib/core/ollama/ollama_launcher.dart
deepThinkER OllamaClient: deepThinkER/lib/core/ollama/ollama_client.dart
deepThinkER ModelManager: deepThinkER/lib/core/ollama/model_manager.dart
deepThinkER ModelRegistry: deepThinkER/lib/core/ollama/model_registry.dart
deepThinkER copy_ollama.sh: deepThinkER/macos/Runner/copy_ollama.sh

Sub-Task 3 — Swift Method Channel

Status: [x] done

Intent Write the thin Swift layer (~200 lines) that gives Dart control over macOS-native capabilities: floating window positioning, opacity, speech recognition, text-to-speech, active app detection, text selection monitoring, mouse position tracking, and the macOS Services menu item.

Expected Outcomes

Window stays above all other windows (NSWindow.level = .floating)
Window snaps to any of 4 corners on demand from Dart
Window opacity controllable from Dart
SFSpeechRecognizer transcribes speech and sends text to Dart via method channel
AVSpeechSynthesizer speaks text received from Dart
NSWorkspace.shared.frontmostApplication polls active app name every 2s to Dart
NSEvent.mouseLocation streams cursor coordinates to Dart continuously
AXSelectedText polls focused UI element for selected text — fires SelectionEvent to Dart on change
macOS Services "Ask Goose" item registered and fires into Dart with selected text
macOS URL scheme wingmanai://ask?q=... registered and fires into Dart

Todo List

Relevant Context

deepThinkER MainFlutterWindow: deepThinkER/macos/Runner/MainFlutterWindow.swift
deepThinkER AppDelegate: deepThinkER/macos/Runner/AppDelegate.swift
Focus Awareness design: Section 8 of this plan

Sub-Task 4 — Screen Watcher + VisionProvider

Status: [x] done

Intent Build the screen capture pipeline, perceptual diff engine, screen region selector, and the VisionProvider abstraction with OllamaVisionProvider as the default implementation. This is the core sensing layer of WingmanAI.

Expected Outcomes

ScreenWatcher captures screenshots at configurable intervals via screencapture -x
PerceptualHasher computes image hash and skips inference when screen is unchanged
RegionSelector allows user to define a screen rectangle — only that region is captured
OllamaVisionProvider sends base64 image to llava:7b and returns description
VisionProvider interface is fully abstracted — swap requires one class change
Privacy exclusions respected — if frontmost app is excluded, capture is skipped
Multi-monitor aware — captures correct display
Vision transparency log records last N captures + conclusions
Screenshot annotation renders bounding boxes from LLaVA response when available
Manual snapshot trigger fires immediately on demand

Todo List

Relevant Context

VisionProvider abstraction rationale: Section 5 of this plan
deepThinkER shell pattern: deepThinkER/lib/core/ollama/ollama_launcher.dart lines 166-243
deepThinkER AppSettings: deepThinkER/lib/core/settings/app_settings.dart

Sub-Task 5 — Reasoning Engine + Tool Framework

Status: [x] done

Intent Build ReasoningEngine — the Granite-powered core that receives screen descriptions, user messages, and tool results, and produces advice or actions. Port the tool framework from deepThinkER and enable ShellTool, AppleScriptTool, and NetworkSearchTool with the approval gate for destructive tools.

Expected Outcomes

ReasoningEngine accepts screen context + user message + memory + session context and calls Granite
Tool call interception works — [SEARCH:], [FETCH:], [SHELL:], [APPLESCRIPT:], [REMEMBER:], [RECALL:], [FILE_READ:], [FILE_WRITE:] all intercepted and dispatched
ShellTool enabled with approval gate — proposes command, waits for user tap before executing
AppleScriptTool enabled with approval gate — same pattern
Network tools rate limited
ContextManager tracks token usage, triggers session summary write at 90% threshold
Confidence signal extracted from Granite response and surfaced on advice card
Frontier escalation triggered when confidence low (if enabled in settings)

Todo List

Relevant Context

deepThinkER AgentTool: deepThinkER/lib/core/tools/agent_tool.dart
deepThinkER ToolRegistry: deepThinkER/lib/core/tools/tool_registry.dart
deepThinkER ShellTool stub: deepThinkER/lib/core/tools/shell/shell_tool.dart
deepThinkER NetworkFetcher: deepThinkER/lib/core/network/network_fetcher.dart
deepThinkER ContextManager: deepThinkER/lib/core/context/context_manager.dart
deepThinkER AuditLog: deepThinkER/lib/core/audit/audit_log.dart

Sub-Task 6 — Overlay UI — Compact + Expanded

Status: [x] done

Intent Build the always-on-top floating overlay in both its compact card state and expanded full chat state. Wire the Swift method channel for window positioning and opacity. Implement the performance strip, token counter, status indicator, confidence indicator, diff view, advice history, Play/Pause/Stop transport controls, and the selection action card for Focus Awareness.

Expected Outcomes

Compact card visible in configured corner, always above other windows
▶ ⏸ ⏹ transport controls always visible in compact card header — color coded (Section 7)
Play activates screen watching + inference, Pause freezes inference with models loaded, Stop unloads all models
Goose status indicator: Watching / Thinking / Idle / Paused / Stopped / DND / Error
Performance strip: tokens used/total + RAM + CPU + GPU — color coded
Confidence indicator on each advice card
Thumbs up/down on each advice card
"Why did you say that?" button on each card
Action approval card — shows proposed shell/AppleScript command, Approve/Deny buttons
Diff view — old vs new side by side before approving text/code changes
Selection action card — appears immediately on text selection, shows AI-generated contextual suggestions + free-text input (Section 8)
Click anywhere expands to full chat with message history and input bar
Advice history scrollable in expanded view
Conversation branching — fork thread button
"?" button shows keyboard shortcut cheat sheet
Focus/DND toggle accessible from compact card
Quick profile switcher in compact card
Attribution label: "🪿 Goose" vs "🌩️ Claude" vs "🌩️ ChatGPT"

Todo List

Relevant Context

deepThinkER AiQuadrant widget patterns: deepThinkER/lib/ui/quadrants/ai_quadrant.dart
Swift method channel: Sub-Task 3 of this plan
Play/Pause/Stop design: Section 7 of this plan
Focus Awareness design: Section 8 of this plan
deepThinkER OllamaClient.unloadModel: deepThinkER/lib/core/ollama/ollama_client.dart

Sub-Task 7 — Voice: Speech-to-Text + Text-to-Speech

Status: [x] done

Intent Wire the Swift method channel voice capabilities into the Dart layer. Implement wake word detection, global hotkey for voice input, voice on/off settings, multi-language support, and the voice picker.

Expected Outcomes

Tapping microphone button or pressing global hotkey activates speech recognition
Transcribed text appears in input bar and auto-sends
"Hey Goose" wake word activates listening (when enabled in Settings)
Goose speaks advice aloud via AVSpeechSynthesizer (when voice output enabled)
Voice picker in Settings — lists available macOS neural voices
Language picker — 50+ languages via SFSpeechRecognizer
Voice on/off toggles independently for input and output
Wake word on/off toggle

Todo List

Write VoiceInputService — calls SpeechChannel.startListening(), handles transcription stream
Write VoiceOutputService — calls SpeechChannel.speak(), stop(), manages queue
Implement global hotkey registration (macOS NSEvent.addGlobalMonitorForEvents)
Implement wake word detection — lightweight keyword spotting on transcription stream
Write VoiceSettings model — input on/off, output on/off, language, voice, hotkey, wake word
Write voice picker UI — dropdown of available neural voices with preview button
Write language picker UI — dropdown of SFSpeechRecognizer supported locales
Wire voice input to chat input bar — auto-fill and send on recognition complete
Wire voice output to ReasoningEngine response stream — speak each new advice card

Relevant Context

Swift method channel: Sub-Task 3 of this plan
deepThinkER audioplayers pattern: deepThinkER/pubspec.yaml audioplayers dependency

Sub-Task 8 — Web Search + Proactive Research Mode

Status: [x] done

Intent Port web search from deepThinkER (DuckDuckGo search + direct URL fetch), integrate it with the tool framework, and implement the proactive research mode that runs background research while the user keeps working — reusing deepThinkER's ResearchEngine.

Expected Outcomes

[SEARCH: query] and [FETCH: url] work in Granite responses
Rate limiting enforced (configurable, default 10/min)
Search results shown as collapsible cards in expanded chat
"Research this for me" command queues a background research task
Research runs via ResearchEngine (gather → synthesize phases)
Result surfaces as an advice card when complete
User can cancel research task from task queue panel

Todo List

Port NetworkFetcher, NetworkSearchTool, NetworkFetchTool (already in Sub-Task 5)
Port simplified RateLimiter — flat rate cap, no trust tiers
Port ResearchEngine from deepThinkER — adapt for single-model (Granite only)
Write ProactiveResearchTask — wraps ResearchEngine as a GooseTask
Write search result card UI — collapsible, shows query + snippet + source URL
Add "Research this for me" intent detection in ReasoningEngine
Wire to task queue (Sub-Task 10)

Relevant Context

deepThinkER NetworkFetcher: deepThinkER/lib/core/network/network_fetcher.dart
deepThinkER ResearchEngine: deepThinkER/lib/core/research/research_engine.dart
deepThinkER RateLimiter: deepThinkER/lib/core/network/rate_limiter.dart

Sub-Task 9 — Frontier Model Escalation

Status: [x] done

Intent Build the optional cloud frontier model escalation layer. Off by default. Supports ChatGPT (GPT-4o), Claude (claude-opus-4), and Grok (grok-3) with user-configurable priority order, per-minute throttle, and automatic failback chain.

Expected Outcomes

FrontierClient abstract interface with three provider implementations
Priority order respected — tries provider 1, falls back to 2, then 3, then Granite
Per-minute throttle enforced via sliding window
API keys stored in AppSettings, never transmitted except to the respective API
GPT-4o Vision and Claude Vision usable for vision escalation
Three escalation triggers work: explicit user request, auto on low confidence, frustration signal
Attribution shown in overlay: "🌩️ Claude says:" vs "🪿 Goose says:"
Feature is completely off and invisible when disabled in Settings

Todo List

Relevant Context

deepThinkER RateLimiter sliding window: deepThinkER/lib/core/network/rate_limiter.dart
AppSettings pattern: deepThinkER/lib/core/settings/app_settings.dart

Sub-Task 10 — Task Queue

Status: [x] done

Intent Build the TaskScheduler infrastructure with SerialTaskScheduler as the v1 implementation and ParallelTaskScheduler as a built-but-dormant hook. Implement the task queue UI panel showing all active and queued tasks.

Expected Outcomes

SerialTaskScheduler queues inference tasks, runs them one at a time
I/O steps (Future.wait()) run in parallel across all queued tasks
ParallelTaskScheduler built and tested but maxConcurrentInferenceTasks hidden in UI
Task queue panel shows active tasks with progress, cancel button
Thermal guard reduces I/O concurrency when CPU/GPU hits red threshold
GooseTask model supports: research, shell, AppleScript, file read/write, web search
Timed task mode: task auto-cancels after user-specified duration
Scheduled tasks: cron-style scheduler fires tasks at configured times (hooks, full feature if stable)
Goose away message: tasks queued while idle, summary card on return

Todo List

Relevant Context

deepThinkER ResourceMonitor: deepThinkER/lib/core/system/resource_monitor.dart
Task queue architecture: Section 6 of this plan

Sub-Task 11 — Memory, Profiles + Session Continuity

Status: [x] done

Intent Port MemoryStore from deepThinkER, build project profiles, implement session summary file writing and loading, build the snippet library, and wire the context rollover logic that loads previous session summaries into a fresh context window.

Expected Outcomes

MemoryStore persists facts across sessions in ~/Library/Application Support/WingmanAI/memory/
Project profiles store: name, folder path, pinned files, persona prompt — switchable from overlay
Session summary file written continuously as Goose works — one file per context window
Format: gooseSession1-2025-07-14.md — living summary, not transcript
On context rollover: loads most recent session files until token budget reached
Snippet library: save, search, copy, delete — stored locally
Export session: one tap saves today's summary to ~/Desktop
Auto-learn: dismissals and thumbs-down stored in memory to adjust future advice

Todo List

Relevant Context

deepThinkER MemoryStore: deepThinkER/lib/core/memory/
deepThinkER ContextManager: deepThinkER/lib/core/context/context_manager.dart
deepThinkER AppPaths: deepThinkER/lib/core/paths/app_paths.dart

Sub-Task 12 — Git, Clipboard, Drag-and-Drop + Input Sources

Status: [x] done

Intent Build all input sources beyond screen watching and voice: git awareness, clipboard watcher, drag-and-drop into overlay, macOS Services integration, and URL scheme handler.

Expected Outcomes

GitInspector reads git status, git diff --stat, git log --oneline -10 from active project folder
Auto commit message generator fires when Goose detects uncommitted changes in active folder
ClipboardWatcher polls for clipboard changes, offers to act on copied text
Drag-and-drop: files, images, and PDFs dragged onto overlay are read and fed to Goose
"Ask Goose" macOS Services item sends selected text from any app to Goose
wingmanai://ask?q=... URL scheme fires query into Goose from any app

Todo List

Write GitInspector — Process.run('git', ...) with output parsing
Write commit message generator — triggered by GitInspector detecting staged changes
Write ClipboardWatcher — dart:isolate timer polling Clipboard.getData()
Write clipboard action card UI — "Rewrite?", "Translate?", "Explain?" options
Implement drag-and-drop target on overlay — DragTarget widget
Write file/image/PDF reader for drag-and-drop — pipe to ReasoningEngine
Wire macOS Services handler (registered in Sub-Task 3) to Goose chat input
Wire URL scheme handler (registered in Sub-Task 3) to Goose chat input
Write folder watcher — Directory.watch() on active project folder

Relevant Context

deepThinkER shell pattern for Process.run: deepThinkER/lib/core/ollama/hardware_detector.dart
deepThinkER FileReadTool: deepThinkER/lib/core/tools/file/

Sub-Task 13 — Intelligence Features

Status: [x] done

Intent Build the higher-order intelligence features: stuck detection, meeting mode, active app awareness, error message explainer, terminal suggester, draft improver, tone analyzer, translation, developer tools, daily brief hook, and reminder hook.

Expected Outcomes

Active app name from Swift channel used to tailor advice context
Stuck detection fires gentle prompt after same screen persists 3+ minutes
Meeting mode auto-activates DND when Zoom/Teams/Meet is frontmost
Idle detection pauses screen capture after N minutes of keyboard inactivity
Error message explainer fires when stack trace or compiler error detected on screen
Terminal command suggester fires when failed command detected in Terminal
Draft improver: select text → Goose rewrites → diff view → approve
Email tone analyzer: when Mail/Outlook frontmost, offers tone reading on visible email
Translation: "translate this" intent detected in user message
Code review: "review this file" intent triggers FileReadTool + structured review
Daily brief hook: stub that fires at configured time — full feature if scheduler is stable
Reminder hook: intent detected, stored in memory, timer fires TTS reminder

Todo List

Relevant Context

deepThinkER ResearchEngine: deepThinkER/lib/core/research/research_engine.dart
Sub-Task 10 scheduled tasks hook for daily brief and reminders

Sub-Task 14 — Performance Monitor + Thermal Management

Status: [x] done

Intent Build the performance monitoring display and thermal management system. Port ResourceMonitor and ContextManager from deepThinkER. Implement the auto-throttle that slows capture interval when the system runs hot.

Expected Outcomes

PerformanceStrip shows: context tokens used/total, RAM in GB, CPU %, GPU % — always visible
Color coding: green → amber (80%) → red (90%) for each metric
PerformancePanel shows full progress bars + Ollama/model status in expanded view
Auto-throttle: when CPU or GPU hits red threshold, capture interval increases automatically
User can override throttle with "Ignore thermal limits" toggle
Thermal guard feeds into task scheduler (Sub-Task 10) to limit concurrency
GPU % via powermetrics or Metal performance counters

Todo List

Port ResourceMonitor from deepThinkER — add GPU % polling via powermetrics
Port ContextManager from deepThinkER (already in Sub-Task 5, wire to UI here)
Write ThermalGuard — subscribes to ResourceMonitor, emits ThermalState (green/amber/red)
Write PerformanceStrip widget — single line, always in compact card
Write PerformancePanel widget — full detail in expanded view
Implement auto-throttle logic — ThermalGuard red → ScreenWatcher.increaseInterval()
Add "Ignore thermal limits" toggle in Settings
Wire thermal state to task scheduler concurrency cap

Relevant Context

deepThinkER ResourceMonitor: deepThinkER/lib/core/system/resource_monitor.dart
deepThinkER HardwareDetector: deepThinkER/lib/core/ollama/hardware_detector.dart

Sub-Task 15 — Settings Screen + First Launch Flow

Status: [x] done

Intent Build the full Settings screen covering all configurable options, and the complete first launch flow: splash, hardware detection, model downloads, permissions setup, corner picker, persona preset picker, and interactive onboarding tutorial.

Expected Outcomes

Settings screen covers all options listed in the feature set
Persona presets: Developer, Writer, Executive, Default — user picks at first launch, editable after
First launch flow navigates: Splash → Hardware → Download → Permissions → Corner → Persona → Tutorial → Live
Permission setup screen has guided instructions + deep links to System Settings for Screen Recording and Accessibility
Interactive tutorial walks through: how to summon Goose, how to approve actions, how to use voice, how to configure
Update check on launch — compares version, shows notification + GitHub link if newer version available
Quiet hours configurable — time range, Goose silent during those hours
RAG folder picker — selects folder to index (hook in v1, indexer stub)
Notification quiet hours enforced in ReasoningEngine before surfacing advice

Todo List

Relevant Context

deepThinkER WelcomeScreen: deepThinkER/lib/ui/screens/welcome_screen.dart
deepThinkER StartupConfigScreen: deepThinkER/lib/ui/screens/startup_config_screen.dart
deepThinkER ResourceGateScreen: deepThinkER/lib/ui/screens/resource_gate_screen.dart
deepThinkER AppSettings: deepThinkER/lib/core/settings/app_settings.dart

Sub-Task 16 — Reliability, Audit + Trust

Status: [x] done

Intent Build the reliability and trust layer: crash recovery, LaunchAgent auto-restart, audit log, local-only guarantee badge, feedback quality log viewer, and the vision transparency log UI.

Expected Outcomes

LaunchAgent plist installed at first launch — macOS auto-restarts WingmanAI on crash
State snapshot written before every inference call — recoverable if crash during inference
Audit log records every tool execution: timestamp, tool, argument, result, approved/denied
Audit log viewer accessible from Settings
Local-only badge visible in compact card — pulsing indicator that nothing has left the Mac
Feedback quality log viewer shows thumbs up/down history by category
Vision transparency log UI shows last 20 captures with Goose's conclusions and bounding boxes
Crash recovery on next launch: detects incomplete session, offers to resume

Todo List

Port AuditLog from deepThinkER — adapt for WingmanAI tool set
Write AuditLogViewer screen
Write StateSnapshot — serializes current session state to disk before each inference
Write crash recovery check in _AppLoader — detects stale snapshot, offers resume
Write LaunchAgent plist generator — installs ~/Library/LaunchAgents/ai.wingman.goose.plist
Write LocalOnlyBadge widget — animated indicator in compact card
Write FeedbackLogViewer screen — thumbs up/down by category, over time
Wire VisionTransparencyLog data (Sub-Task 4) to UI panel
Write update notification UI — banner in compact card when new version detected

Relevant Context

deepThinkER AuditLog: deepThinkER/lib/core/audit/audit_log.dart
deepThinkER AppDelegate PID lifecycle: deepThinkER/macos/Runner/AppDelegate.swift

Sub-Task 17 — Local RAG (Hook)

Status: [x] done

Intent Build the RAG (Retrieval-Augmented Generation) indexer as a v1 hook. The folder picker, index file format, and retrieval interface are fully implemented. The actual embedding generation is stubbed — if Granite embedding support proves stable, the stub is replaced with a real implementation. If not, the hook remains for v2.

Expected Outcomes

RagIndexer interface defined — indexFolder(path), query(text) → relevant chunks
RagIndexerStub returns empty results — safe no-op if indexing not available
GraniteRagIndexer attempts to use Granite embedding endpoint — activates if available
Index stored in ~/Library/Application Support/WingmanAI/rag/
RAG folder picker in Settings wired to RagIndexer.indexFolder()
ReasoningEngine calls RagIndexer.query() before each inference — injects relevant chunks if any

Todo List

Write RagIndexer abstract interface
Write RagIndexerStub — safe no-op, always returns empty
Write GraniteRagIndexer — attempts Granite /api/embeddings endpoint
Write RagStore — stores chunks + embeddings as NDJSON
Write RagRetriever — cosine similarity search over stored embeddings
Wire RAG query into ReasoningEngine system prompt builder
Test with real Granite embeddings — if stable, activate; if not, leave stub

Relevant Context

Granite 3.1 supports embedding generation via /api/embeddings endpoint in Ollama
deepThinkER tool pattern for extensibility: deepThinkER/lib/core/tools/agent_tool.dart

Feature Hooks Summary

The following features are built in v1 but may be left as stubs if they prove unstable during testing. Each has a clean interface so the stub can be replaced with a full implementation without touching other code.

Feature	Hook Interface	Activation Condition
Daily brief	`DailyBriefScheduler` stub	If scheduled tasks (Sub-Task 10) are stable
Reminder setting	`ReminderStore` + timer stub	If scheduled tasks are stable
Local RAG	`RagIndexer` abstract + stub	If Granite embeddings work reliably
Parallel inference	`ParallelTaskScheduler` built, hidden	User unlocks in Advanced Settings (v2)
Scheduled tasks	`ScheduledTaskStore` + cron hook	If cron-style scheduler is stable in v1
Screenshot annotation	Bounding box renderer stub	If LLaVA returns usable bounding box data
Goose away message	`AwayMessageTask` stub	If idle detection + task queue are stable
Conversation branching	`ThreadStore` stub	If session continuity is stable first

Plan written by Bob. Ready for implementation in Agent mode.

FilesExpand file tree

PLAN.md

Latest commit

History

PLAN.md

File metadata and controls

WingmanAI — Project Plan

Table of Contents

1. What Is WingmanAI

2. Why Build This

The Hardware Moment

The Foundation Already Exists

The Privacy Differentiator

The Cost Differentiator

3. Competitive Landscape

The Tools

Master Comparison Table

WingmanAI vs OpenClaw — Deep Dive

WingmanAI's Genuine Position

4. Architecture Overview

State Management

Entitlements Required

5. VisionProvider Abstraction

Implementations

Swap Guide

Near-Term Swap Candidates

6. Task Queue Architecture

v1 — Parallel I/O, Sequential Inference

v1 Implementation: SerialTaskScheduler

Future Implementation: ParallelTaskScheduler

The Single Unlock

Progression Path

7. Play / Pause / Stop Controls

The Critical Distinction

UI Placement

State Persistence

8. Focus Awareness

Layer 1 — Mouse Position Awareness

Layer 2 — Text Selection Awareness (Key Feature)

The Selection Action Card

Text Injection Back Into the App

Requires

What This Enables

9. Sub-Tasks

Sub-Task 1 — Project Scaffold

Sub-Task 2 — Bundled Ollama + Model Management

Sub-Task 3 — Swift Method Channel

Sub-Task 4 — Screen Watcher + VisionProvider

Sub-Task 5 — Reasoning Engine + Tool Framework

Sub-Task 6 — Overlay UI — Compact + Expanded

Sub-Task 7 — Voice: Speech-to-Text + Text-to-Speech

Sub-Task 8 — Web Search + Proactive Research Mode

Sub-Task 9 — Frontier Model Escalation

Sub-Task 10 — Task Queue

Sub-Task 11 — Memory, Profiles + Session Continuity

Sub-Task 12 — Git, Clipboard, Drag-and-Drop + Input Sources

Sub-Task 13 — Intelligence Features

Sub-Task 14 — Performance Monitor + Thermal Management

Sub-Task 15 — Settings Screen + First Launch Flow

Sub-Task 16 — Reliability, Audit + Trust

Sub-Task 17 — Local RAG (Hook)

Feature Hooks Summary