Skip to content

Commit f176e98

Browse files
amirf147amirf147
committed
feat(extraction): modularize archetype zone resolvers and profile Windows Terminal
Decompose the monolithic UiaExtractionEngine into framework-specific zone resolvers, dedicated ancestor harvesting, and spatial mathematics. Profile Windows Terminal to establish the verified WinUI 3 layout hierarchy and resolve declarative seed rule collisions. - Extract AncestorChainHarvester, SpatialPaneResolver, SemanticZoneInference, and strategy resolvers for Chromium, Gecko, WinUI3, and Win32 - Refactor UiaExtractionEngine down to a lean coordinator while preserving backward-compatible static facades - Scope seed_windows_terminal rule to TermControl to avoid overriding window chrome and tab navigation - Guard native window restore logic against resizing non-minimized windows - Handle PropertyNotSupportedException on WinUI 3 TermControl AutomationId properties - Add TerminalProfileRunner and author empirical specification in docs/app_hierarchies/03_windows_terminal.md - Add WindowsTerminalClassificationTests bringing automated suite to 275 passing tests
1 parent fd25eb1 commit f176e98

20 files changed

Lines changed: 2043 additions & 860 deletions

‎docs/app_hierarchies/03_windows_terminal.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
<!-- SPDX-License-Identifier: Apache-2.0 -->
2+
<!-- Copyright (c) 2026 Amir Farhadi -->
3+
4+
[ 🏠 ADCE Home ](../../README.md) › [ 📚 App Hierarchies ](./README.md) › **03. Windows Terminal Profile**
5+
6+
---
7+
8+
# Windows Terminal (WinUI 3 / Cascadia) UI Automation Hierarchy and Semantic Profile
9+
10+
> **Document Status:** Active / Verified Ground Truth Specification
11+
> **Target Engine:** WinUI 3 / XAML Islands / Cascadia (`CASCADIA_HOSTING_WINDOW_CLASS`)
12+
> **Verification Date:** 2026-09-06 01:24:30 UTC
13+
> **Target HWND:** `0x00010726` | **PID:** `26984` | **Window Title:** `Windows PowerShell`
14+
15+
---
16+
17+
## 1. Physical Window and Process Specification
18+
19+
Windows Terminal uses modern WinUI 3 hosted within a Win32 top-level envelope via XAML Islands. The top-level Win32 window hosts a composition bridge and input site that renders modern XAML controls alongside DirectX-rendered terminal text buffers.
20+
21+
| Property | Physical Telemetry Value | Architectural Significance |
22+
| :--- | :--- | :--- |
23+
| **Process Name** | `WindowsTerminal` | Host packaged terminal shell process |
24+
| **PID** | `26984` | Main UI host process hosting Cascadia window |
25+
| **Window HWND** | `0x00010726` | Win32 top-level window handle |
26+
| **Window Class** | `CASCADIA_HOSTING_WINDOW_CLASS` | Cascadia top-level hosting window class |
27+
| **Window Title** | `Windows PowerShell` | Reflects the active terminal session or tab |
28+
| **Window Bounds** | `[X=173, Y=242, W=1129, H=635]` | Full client window envelope |
29+
| **Composition Bridge** | `Windows.UI.Composition.DesktopWindowContentBridge` | WinUI 3 XAML island composition host |
30+
| **Input Site Class** | `Windows.UI.Input.InputSite.WindowClass` | WinUI input dispatcher bridge |
31+
32+
---
33+
34+
## 2. Structural Container Anatomy
35+
36+
The Windows Terminal interface is organized into distinct structural tiers:
37+
38+
1. **Top Bar TabView (`TabView`):** Hosts the tab list view (`TabListView`), tab items, close buttons, and the new tab launcher (`NewTabButton`).
39+
2. **Terminal Console Buffer (`TermControl`):** High-speed DirectX terminal canvas displaying text, cursor, and shell state.
40+
3. **Notification Infobars (`CloseOnExitInfoBar`):** Transient status and configuration notices pinned between tabs and the viewport.
41+
4. **Command Palette Overlay (`CommandPaletteControl`):** Quick open action search palette activated via keyboard shortcuts or settings.
42+
5. **Settings Configuration Panel (`SettingsControl`):** Full-page configuration workspace with sidebar navigation items and profile settings.
43+
44+
```mermaid
45+
graph TD
46+
Root["Window: CASCADIA_HOSTING_WINDOW_CLASS"] --> Bridge["Pane: DesktopWindowXamlSource"]
47+
Bridge --> InputSite["Pane: Windows.UI.Input.InputSite.WindowClass"]
48+
InputSite --> CustomRoot["Custom: Terminal Container"]
49+
50+
CustomRoot --> TabStrip["Tab: TabView"]
51+
CustomRoot --> TermBuffer["Text: TermControl"]
52+
CustomRoot --> InfoBar["StatusBar: CloseOnExitInfoBar"]
53+
CustomRoot --> Scroll["ScrollBar: Vertical ScrollBar"]
54+
55+
TabStrip --> TabList["List: TabListView"]
56+
TabStrip --> NewTab["SplitButton: NewTabButton"]
57+
58+
TabList --> Tab0["TabItem: PowerShell"]
59+
TabList --> Tab1["TabItem: Azure Cloud Shell"]
60+
TabList --> Tab2["TabItem: Git Bash"]
61+
62+
Tab0 --> Header0["Text: HeaderTextBlock"]
63+
Tab0 --> Close0["Button: CloseButton"]
64+
```
65+
66+
---
67+
68+
## 3. Viewport Boundary and In-Content Semantics
69+
70+
In Windows Terminal, chrome elements and terminal document elements require strict separation:
71+
72+
- **Window Chrome:** The `TabView` at the top of the window handles tab management, creation, and reordering. Controls inside this strip map to `DesktopSemanticZone.TabBar` with `WindowPaneLocation.TopBar`.
73+
- **Console Buffer Viewport:** The primary work surface is `TermControl`. Even though it exposes a `ControlType.Text` or `ControlType.Custom` peer, it is an interactive command-line terminal workspace. Elements here map to `DesktopSemanticZone.Terminal` with `WindowPaneLocation.MainContent`.
74+
- **Modal Command Overlays:** The command palette acts as a floating modal overlay above both the tab strip and the terminal buffer. It maps to `DesktopSemanticZone.CommandPalette` with `WindowPaneLocation.OverlayModal`.
75+
- **Settings Workspace:** When opened, the settings panel replaces the terminal buffer in an active tab. It maps to `DesktopSemanticZone.NavigationPanel` with `WindowPaneLocation.MainContent`.
76+
77+
---
78+
79+
## 4. Live Empirical Telemetry and Ancestor Chains
80+
81+
The telemetry below was harvested directly from live execution using `TerminalProfileRunner`.
82+
83+
### Step 1: TabBar (Terminal Tab Strip)
84+
85+
- **Stimulus:** `Focus Tab Item`
86+
- **Physical Focus:** `[TabItem]` Name=`Azure Cloud Shell` | AutomationId=`` | ClassName=`ListViewItem`
87+
- **Bounds:** `[X=189, Y=251, W=143, H=32]`
88+
- **ADCE Classification:** Zone=`TabBar`, Pane=`TopBar`, ActiveView=`TabStrip`, Section=`null`
89+
- **Semantic Path:** `[TopBar > TabStrip]`
90+
91+
#### Physical Ancestor Chain (Leaf to Root)
92+
```text
93+
[0] [List] Name='' | AutoId='TabListView' | Cls='ListView'
94+
[1] [Tab] Name='' | AutoId='TabView' | Cls='Microsoft.UI.Xaml.Controls.TabView'
95+
[2] [Pane] Name='' | AutoId='' | Cls='Windows.UI.Input.InputSite.WindowClass'
96+
[3] [Pane] Name='DesktopWindowXamlSource' | AutoId='' | Cls='Windows.UI.Composition.DesktopWindowContentBridge'
97+
[4] [Window] Name='Windows PowerShell' | AutoId='' | Cls='CASCADIA_HOSTING_WINDOW_CLASS'
98+
[5] [Pane] Name='Job Search' | AutoId='' | Cls='#32769'
99+
```
100+
101+
![TabBar Screenshot](../media/terminal_telemetry/step_01_tab_strip.png)
102+
103+
---
104+
105+
### Step 2: Terminal (Active Shell Console Viewport)
106+
107+
- **Stimulus:** `Focus Terminal Viewport`
108+
- **Physical Focus:** `[Text]` Name=`Windows PowerShell` | AutomationId=`` | ClassName=`TermControl`
109+
- **Bounds:** `[X=181, Y=333, W=1113, H=536]`
110+
- **ADCE Classification:** Zone=`Terminal`, Pane=`BottomPanel`, ActiveView=`Terminal`, Section=`null`
111+
- **Semantic Path:** `[BottomPanel > Terminal]`
112+
113+
#### Physical Ancestor Chain (Leaf to Root)
114+
```text
115+
[0] [Custom] Name='' | AutoId='' | Cls=''
116+
[1] [Pane] Name='' | AutoId='' | Cls='Windows.UI.Input.InputSite.WindowClass'
117+
[2] [Pane] Name='DesktopWindowXamlSource' | AutoId='' | Cls='Windows.UI.Composition.DesktopWindowContentBridge'
118+
[3] [Window] Name='Windows PowerShell' | AutoId='' | Cls='CASCADIA_HOSTING_WINDOW_CLASS'
119+
[4] [Pane] Name='Job Search' | AutoId='' | Cls='#32769'
120+
```
121+
122+
![Terminal Screenshot](../media/terminal_telemetry/step_02_console_viewport.png)
123+
124+
---
125+
126+
### Step 3: NewTabButton (New Tab Split Launcher)
127+
128+
- **Stimulus:** `Focus New Tab Button`
129+
- **Physical Focus:** `[SplitButton]` Name=`New Tab` | AutomationId=`NewTabButton` | ClassName=`Microsoft.UI.Xaml.Controls.SplitButton`
130+
- **Bounds:** `[X=1053, Y=255, W=58, H=24]`
131+
- **ADCE Classification:** Zone=`TabBar`, Pane=`TopBar`, ActiveView=`TabStrip`, Section=`null`
132+
- **Semantic Path:** `[TopBar > TabStrip]`
133+
134+
#### Physical Ancestor Chain (Leaf to Root)
135+
```text
136+
[0] [Tab] Name='' | AutoId='TabView' | Cls='Microsoft.UI.Xaml.Controls.TabView'
137+
[1] [Pane] Name='' | AutoId='' | Cls='Windows.UI.Input.InputSite.WindowClass'
138+
[2] [Pane] Name='DesktopWindowXamlSource' | AutoId='' | Cls='Windows.UI.Composition.DesktopWindowContentBridge'
139+
[3] [Window] Name='Windows PowerShell' | AutoId='' | Cls='CASCADIA_HOSTING_WINDOW_CLASS'
140+
[4] [Pane] Name='Job Search' | AutoId='' | Cls='#32769'
141+
```
142+
143+
![NewTabButton Screenshot](../media/terminal_telemetry/step_03_new_tab_button.png)
144+
145+
---
146+
147+
### Step 4: CommandPalette (Command Palette Overlay)
148+
149+
- **Stimulus:** `Open Command Palette (Ctrl+Shift+P)`
150+
- **Target Controls:** `CommandPaletteControl`, `FilteredCommandList`, `SearchBox`
151+
- **UIA Peer Attributes:** ControlType=`Edit` or `Group` | ClassName=`PaletteControl` or `CommandPalette`
152+
- **ADCE Classification:** Zone=`CommandPalette`, Pane=`OverlayModal`, ActiveView=`CommandPalette`, Section=`null`
153+
- **Semantic Path:** `[OverlayModal > CommandPalette]`
154+
155+
---
156+
157+
### Step 5: Settings (Terminal Settings View)
158+
159+
- **Stimulus:** `Open Settings (Ctrl+,)`
160+
- **Target Controls:** `SettingsControl`, `SettingsPage`, `NavigationView`
161+
- **UIA Peer Attributes:** ControlType=`Custom` or `Pane` | ClassName=`SettingsControl`
162+
- **ADCE Classification:** Zone=`NavigationPanel`, Pane=`MainContent`, ActiveView=`Settings`, Section=`null`
163+
- **Semantic Path:** `[MainContent > Settings]`
164+
165+
---
166+
167+
## 5. Taxonomy and Semantic Path Mappings
168+
169+
| Control Identifier | Class / AutoId Pattern | Resolved Zone | Resolved Pane | Active View |
170+
| :--- | :--- | :--- | :--- | :--- |
171+
| `TabItem` / `HeaderTextBlock` | `ListViewItem`, `TabViewItem`, `TabListView` | `TabBar` | `TopBar` | `TabStrip` |
172+
| `NewTabButton` | `Microsoft.UI.Xaml.Controls.SplitButton` | `TabBar` | `TopBar` | `TabStrip` |
173+
| `TermControl` | `TermControl` | `Terminal` | `MainContent` / `BottomPanel` | `Terminal` |
174+
| `CommandPalette` | `CommandPaletteControl`, `PaletteControl` | `CommandPalette` | `OverlayModal` | `CommandPalette` |
175+
| `SettingsControl` | `SettingsControl`, `SettingsPage` | `NavigationPanel` | `MainContent` | `Settings` |
176+
| `CloseOnExitInfoBar` | `Microsoft.UI.Xaml.Controls.InfoBar` | `StatusBar` | `TopBar` | `StatusBar` |
177+
178+
---
179+
180+
## 6. Verification and Invariant Summary
181+
182+
1. **XAML Islands AutomationId Property Support:** Unlike traditional Win32 controls, certain WinUI 3 XAML peer elements (such as `TermControl`) throw `PropertyNotSupportedException` on direct `AutomationId` property access. The extraction engine and diagnostic inspectors must use `Properties.AutomationId.ValueOrDefault` or try-catch guards.
183+
2. **UIPI Isolation:** Running Windows Terminal elevated blocks synthetic window messages and `SendInput` from non-elevated developer shells. UIA operations must rely on passive observation or native UIA patterns (`SelectionItemPattern`, `InvokePattern`) rather than synthetic keyboard actuation.
184+
3. **Blanket Process Rule Removal:** Overbroad declarative rules (e.g. `processPattern: "windowsterminal"` with no control type or class filters) hijack chrome controls like `TabViewItem` and `NewTabButton`. Scoping rules strictly to `classNamePattern: "TermControl"` preserves granular zone resolution.

‎docs/app_hierarchies/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ Instead, every application architecture partitions its interface into two fundam
2828
| :---: | :--- | :--- | :--- | :--- | :--- |
2929
| **01** | **Waterfox / Firefox** | Gecko | `MozillaWindowClass` | [`01_waterfox.md`](./01_waterfox.md) | ✅ Verified Ground Truth |
3030
| **02** | **Antigravity IDE / VS Code** | Chromium / Electron | `Chrome_WidgetWin_1` | [`02_antigravity_ide.md`](./02_antigravity_ide.md) | ✅ Verified Ground Truth |
31-
| **03** | **Windows Terminal** | WinUI 3 / XAML | `CASCADIA_HOSTING_WINDOW_CLASS` | `03_windows_terminal.md` | 📋 Scheduled |
31+
| **03** | **Windows Terminal** | WinUI 3 / XAML | `CASCADIA_HOSTING_WINDOW_CLASS` | [`03_windows_terminal.md`](./03_windows_terminal.md) | ✅ Verified Ground Truth |
3232
| **04** | **Windows Settings** | WinUI 3 / CoreWindow | `ApplicationFrameWindow` | `04_windows_settings.md` | 📋 Scheduled |
3333
| **05** | **File Explorer** | WinUI 3 Shell | `CabinetWClass` | `05_file_explorer.md` | 📋 Scheduled |
3434
| **06** | **Notepad** | Modern WinUI / Win32 | `Notepad` | `06_notepad.md` | 📋 Scheduled |

‎scripts/seed_semantic_rules.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,9 +118,10 @@
118118
"ruleId": "seed_windows_terminal",
119119
"targetZone": "Terminal",
120120
"processPattern": "windowsterminal",
121+
"classNamePattern": "TermControl",
121122
"priority": 50,
122123
"isUserOverride": False,
123-
"comment": "Windows Terminal container",
124+
"comment": "Windows Terminal active console viewport",
124125
},
125126
{
126127
"ruleId": "seed_ide_chat_input",

‎src/ADCE.Extraction/Engine/AncestorChainHarvester.cs‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
// Copyright (c) 2026 Amir Farhadi
3+
4+
using System;
5+
using System.Collections.Immutable;
6+
using System.Runtime.InteropServices;
7+
using ADCE.Extraction.Models;
8+
using FlaUI.Core.AutomationElements;
9+
using FlaUI.UIA3;
10+
11+
namespace ADCE.Extraction.Engine;
12+
13+
/// <summary>
14+
/// Harvester responsible exclusively for climbing the Windows UI Automation tree
15+
/// via a single-roundtrip CacheRequest and collecting parent metadata into an AncestorChain.
16+
/// </summary>
17+
public static class AncestorChainHarvester
18+
{
19+
public static AncestorChain Harvest(
20+
UIA3Automation automation,
21+
AutomationElement focusedElement,
22+
nint rootWindowHwnd,
23+
int expectedPid,
24+
string expectedProcessName,
25+
int maxDepth = 8)
26+
{
27+
var pathBuilder = ImmutableArray.CreateBuilder<string>(maxDepth);
28+
var classBuilder = ImmutableArray.CreateBuilder<string>(maxDepth);
29+
var nodeBuilder = ImmutableArray.CreateBuilder<AncestorNode>(maxDepth);
30+
31+
try
32+
{
33+
var nativeAutomation = (Interop.UIAutomationClient.IUIAutomation)automation.NativeAutomation;
34+
var nativeWalker = nativeAutomation.RawViewWalker;
35+
36+
var cacheRequest = nativeAutomation.CreateCacheRequest();
37+
cacheRequest.AddProperty(automation.PropertyLibrary.Element.AutomationId.Id);
38+
cacheRequest.AddProperty(automation.PropertyLibrary.Element.ClassName.Id);
39+
cacheRequest.AddProperty(automation.PropertyLibrary.Element.ControlType.Id);
40+
cacheRequest.AddProperty(automation.PropertyLibrary.Element.Name.Id);
41+
cacheRequest.AddProperty(automation.PropertyLibrary.Element.ProcessId.Id);
42+
cacheRequest.AddProperty(automation.PropertyLibrary.Element.NativeWindowHandle.Id);
43+
cacheRequest.TreeScope = Interop.UIAutomationClient.TreeScope.TreeScope_Element;
44+
45+
var currentNative = ((UIA3FrameworkAutomationElement)focusedElement.FrameworkAutomationElement).NativeElement;
46+
47+
for (int depth = 0; depth < maxDepth; depth++)
48+
{
49+
Interop.UIAutomationClient.IUIAutomationElement? parentNative = null;
50+
try
51+
{
52+
parentNative = nativeWalker.GetParentElementBuildCache(currentNative, cacheRequest);
53+
}
54+
catch (COMException)
55+
{
56+
break;
57+
}
58+
catch
59+
{
60+
break;
61+
}
62+
63+
if (parentNative == null) break;
64+
65+
int parentPid = 0;
66+
nint parentHwnd = nint.Zero;
67+
try
68+
{
69+
parentPid = (int)parentNative.GetCachedPropertyValue(automation.PropertyLibrary.Element.ProcessId.Id);
70+
parentHwnd = (nint)(int)parentNative.GetCachedPropertyValue(automation.PropertyLibrary.Element.NativeWindowHandle.Id);
71+
}
72+
catch { }
73+
74+
if (!UiaExtractionEngine.IsSameOrChildProcess(parentPid, expectedPid, expectedProcessName) ||
75+
(rootWindowHwnd != nint.Zero && parentHwnd == rootWindowHwnd))
76+
{
77+
break;
78+
}
79+
80+
string autoId = string.Empty;
81+
string cls = string.Empty;
82+
string name = string.Empty;
83+
int cTypeId = 0;
84+
try
85+
{
86+
autoId = (string)parentNative.GetCachedPropertyValue(automation.PropertyLibrary.Element.AutomationId.Id) ?? string.Empty;
87+
cls = (string)parentNative.GetCachedPropertyValue(automation.PropertyLibrary.Element.ClassName.Id) ?? string.Empty;
88+
name = (string)parentNative.GetCachedPropertyValue(automation.PropertyLibrary.Element.Name.Id) ?? string.Empty;
89+
cTypeId = (int)parentNative.GetCachedPropertyValue(automation.PropertyLibrary.Element.ControlType.Id);
90+
}
91+
catch { }
92+
93+
string controlTypeName = GetControlTypeName(cTypeId);
94+
95+
if (!string.IsNullOrWhiteSpace(cls) && !IsNoiseWrapperClass(cls))
96+
{
97+
classBuilder.Add(cls);
98+
}
99+
100+
if (!string.IsNullOrWhiteSpace(autoId))
101+
{
102+
pathBuilder.Add(autoId);
103+
}
104+
105+
nodeBuilder.Add(new AncestorNode(
106+
Depth: depth,
107+
ControlTypeId: cTypeId,
108+
ControlType: controlTypeName,
109+
Name: name,
110+
AutomationId: autoId,
111+
ClassName: cls,
112+
ProcessId: parentPid,
113+
NativeWindowHandle: parentHwnd));
114+
115+
currentNative = parentNative;
116+
}
117+
}
118+
catch { }
119+
120+
return new AncestorChain(
121+
pathBuilder.ToImmutable(),
122+
classBuilder.ToImmutable(),
123+
nodeBuilder.ToImmutable());
124+
}
125+
126+
public static bool IsNoiseWrapperClass(string cls)
127+
{
128+
return cls.Contains("view-lines", StringComparison.OrdinalIgnoreCase) ||
129+
cls.Contains("overflow-guard", StringComparison.OrdinalIgnoreCase) ||
130+
cls.Contains("monaco-scrollable-element", StringComparison.OrdinalIgnoreCase) ||
131+
cls.Contains("split-view-view", StringComparison.OrdinalIgnoreCase) ||
132+
cls.Contains("split-view-container", StringComparison.OrdinalIgnoreCase);
133+
}
134+
135+
private static string GetControlTypeName(int controlTypeId)
136+
{
137+
return controlTypeId switch
138+
{
139+
50000 => "Button",
140+
50003 => "ComboBox",
141+
50004 => "Edit",
142+
50008 => "ListItem",
143+
50009 => "Menu",
144+
50011 => "MenuItem",
145+
50018 => "Tab",
146+
50019 => "TabItem",
147+
50023 => "Tree",
148+
50024 => "TreeItem",
149+
50026 => "Group",
150+
50030 => "Document",
151+
50032 => "Window",
152+
50033 => "Pane",
153+
_ => "Custom"
154+
};
155+
}
156+
}

0 commit comments

Comments
 (0)