feat: agent map data + framebuffer canvas screenshot

TODO.md

@@ -18,7 +18,7 @@ Conventions:
 - **Entity-composition model (decided 2026-06-09)** — the agreed target content model lives in [`docs/concepts/entity-and-appearance-model.md`](docs/concepts/entity-and-appearance-model.md): explicit `components[]` on entity definitions + component registry (schema-generated inspectors) + editor **templates** replace the `ObjectKind`-switch + properties union; **Appearance** (sheet + animations) becomes a shared asset layer; `CharacterDefinition` folds in. Phases: **A** (landed — rename + deep-link); **B0** (landed — component registry + field DSL + 11 specs in `packages/engine/src/entity/`); **B1** (landed — objects refactor: `EntityDefinition.components[]`, registry-walk spawn in `entity/spawn-placement.ts`, `objectLibrary`→`entityLibrary`, `ObjectDefinition`/`ObjectKind` deleted, `games/*` migrated via `scripts/migrate-to-entity-components.mjs`); **B2** (landed — characters→`entityLibrary` (`character`-template), `isPlayer`→`playerActorId`, collab `entity.upsert`/`entity.remove`/`player.set`; `CharacterDefinition` kept as a non-persisted Cast view model via `entity/convert.ts`); **C1** (landed — generated component inspectors `ComponentInspector`/`EntityComponentsEditor`); **C2** (landed — `Objects` mode + library view + `ObjectsController` + `entity-templates.ts` + `win.new-object`/`open-object` + MCP `set_view objects`); **C3a** (landed — `PlaceObjectCommand`/`RemoveObjectCommand` + `MapScene.spawn`/`despawnPlacement`); **C3b** (landed — `object` tool + brush + `placeObjectAt` + DBus/MCP `place_object`; the scene-editor Objects-tab brush picker is a **visual sprite-thumbnail palette** + `win.set-inspector-tab` makes the inspector tabs driveable) (live spawn/despawn, undoable, collab); **C4** (landed — cast detail "all components" disclosure via `EntityComponentsEditor`; `mergeCharacterIntoEntity` preserves disclosure-added components on basic edits); **C5** (landed — Sheets-view unification: appearances + the animation editor relocate from Cast into the former Tiles view (now "Sheets"); Cast → Characters-only lens with `win.open-appearance` deep-link; `CastController` `appearances-changed` push, `win.open-sheet` removed); **C6** (landed — Objects = GENERAL lens over the whole `entityLibrary` incl. characters [drop the `!isCharacterEntity` filter] with a "Cast" badge; Cast = specialised character-only lens; cross-refresh via the `ProjectStore`'s `entity-library-changed` event; scene-editor object brush lists every entity except the player actor → Cast NPCs placeable; **"Cast member" toggle** in the Objects detail [`win.toggle-object-cast`] promotes/demotes an appearance-bearing entity into the Cast roster by flipping `editorData.template`↔'character' — unifies the Cast-NPC vs Objects-NPC confusion); **D** — declarative states (`EntityState` + `StateSystem`); **E** — script component + built-in code editor. *owner: engine + maker + gjs*
 - **Tileset editor follow-ups** — the Sheets view (former Tiles view) lists tilesets as cards (downscaled sheet thumbnails) with **import** (`win.new-tileset` → same `SpriteSetImportDialog`), **rename** (card ⋯ menu → `CardGallery` `rename-requested` → `ProjectStore.renameSpriteSet`, re-persists + broadcasts the descriptor), **drag-to-reorder** (`CardGallery.reorderable` DnD → `spriteset-reorder-requested` → `ProjectStore.reorderSpriteSets` rewrites `data.spriteSets[]` order; display-only, local — not collab-synced, joiners get it via the snapshot), and a **usage-aware delete** (the confirm names how many characters/maps still reference the set, via the shared `sprite-set-usage.ts` reverse-reference counter). Sprite-set CRUD + collab broadcast live in `ProjectStore` (the single project-level write pipeline; `apps/maker-gjs/src/services/project-store.ts`) — including **tile-property edits** (Solid / Surface switches: `TilesController` delegates to `ProjectStore.setTileSolid`/`setTileSurface`, which persist + broadcast `__project/spriteset.update.chunk`; receivers refresh live collision via `Engine.refreshTileSolidsForSpriteSet`). No tileset follow-ups outstanding. *owner: maker + gjs*
 - **`FloatingTopBar` breakpoint thresholds** — the 5 empirically-chosen stops (880sp split↔merged, 740 / 620 / 540 / 460sp disclosure inside merged) were picked while wiring the new chrome and should be re-measured once the inspector + library sidebars settle on final widths. Also verify the merged-pill doesn't wrap at 360sp phone-portrait widths. *owner: maker / gjs*
-- **MCP orchestrator / `org.pixelrpg.maker.Control` follow-ups** — `apps/mcp-bridge` is a dev-only orchestrator: status / screenshot / action-driving, headless project load (`OpenProject` / `ListRecentProjects` / `ListTemplates`), **multi-instance** (env `PIXELRPG_INSTANCE` → per-instance app-id/D-Bus name; `launch_instance` / `list_instances` / `stop_instance` + an `instance` selector on every tool), **collab session primitives** (`start_session` / `join_session` / `get_session_state`), **`paint_tile`** (operation-oriented → undo + collab op-sync; shared `tile-paint.service` with `TileEditorSystem`), **`set_zoom`**, **`present_window`**, **`resize_window`** (`ResizeWindow` Control method + preset phone/tablet/desktop sizes — for exercising the responsive breakpoints) (+ `engineReady`/`mapped` in status). Decided against (re-add only if a real need shows up): canvas-composite screenshot fallback; D-Bus/MCP push notifications (an MCP agent acts on its next turn anyway — `get_status` polling covers it). Still wanted: (a) **extract a reusable `@gjsify/*` helper** once a 2nd consumer appears (e.g. easy6502); (b) **data-driven component projection** — a generic `get_components` that serialises the session-singleton's ECS components (auto-tracks new ones) instead of the hand-curated `get_status`. **Capture caveat (environmental):** in-process `Screenshot` works (project views *and* the WebGL canvas captured fine in earlier sessions), but it needs the window to be actively rendering frames. A backgrounded / occluded / screen-off test window keeps its initial paint (so the welcome view captures) but stops getting frame-callbacks after a view switch, so `WidgetPaintable.snapshot → to_node()` comes back empty for project views. `present_window` doesn't reliably un-occlude on Wayland. For agent-driven responsive-layout screenshots the instance must be visible/top (or use `gnome-session-inhibit` + foreground it). *owner: maker / gjsify*
+- **MCP orchestrator / `org.pixelrpg.maker.Control` follow-ups** — `apps/mcp-bridge` is a dev-only orchestrator: status / screenshot / action-driving, headless project load (`OpenProject` / `ListRecentProjects` / `ListTemplates`), **multi-instance** (env `PIXELRPG_INSTANCE` → per-instance app-id/D-Bus name; `launch_instance` / `list_instances` / `stop_instance` + an `instance` selector on every tool), **collab session primitives** (`start_session` / `join_session` / `get_session_state`), **`paint_tile`** (operation-oriented → undo + collab op-sync; shared `tile-paint.service` with `TileEditorSystem`), **`set_zoom`**, **`present_window`**, **`resize_window`** (`ResizeWindow` Control method + preset phone/tablet/desktop sizes — for exercising the responsive breakpoints) (+ `engineReady`/`mapped` in status). Decided against (re-add only if a real need shows up): canvas-composite screenshot fallback; D-Bus/MCP push notifications (an MCP agent acts on its next turn anyway — `get_status` polling covers it). Still wanted: (a) **extract a reusable `@gjsify/*` helper** once a 2nd consumer appears (e.g. easy6502); (b) **data-driven component projection** — a generic `get_components` that serialises the session-singleton's ECS components (auto-tracks new ones) instead of the hand-curated `get_status`. Shipped 2026-06-12: **`get_map_data`** (agent-oriented walkability/placements/teleports projection, no engine/window needed) and the **framebuffer-based `canvas` screenshot scope** (gl.readPixels — occlusion-independent). Known quirk: the framebuffer capture can show a faint offset ghost of the scene (FBO double-buffer residue) — harmless for verification, revisit with the GL bridge. Also wanted: (c2) **blocking `open_scene`** — an optional wait-until-engineReady on the Control/MCP side would remove the agent's last gdbus polling loops. Still wanted on this front: (c) **`render_map`** — full-map offscreen render to PNG via the MapPreview Gsk pipeline (hand-built render nodes need no mapped widget), for visual map verification without any engine. **Capture caveat (environmental):** in-process `Screenshot` works (project views *and* the WebGL canvas captured fine in earlier sessions), but it needs the window to be actively rendering frames. A backgrounded / occluded / screen-off test window keeps its initial paint (so the welcome view captures) but stops getting frame-callbacks after a view switch, so `WidgetPaintable.snapshot → to_node()` comes back empty for project views. `present_window` doesn't reliably un-occlude on Wayland. For agent-driven responsive-layout screenshots the instance must be visible/top (or use `gnome-session-inhibit` + foreground it). *owner: maker / gjsify*
 - **Assistant pause / control-plane follow-ups** — pause is now ENFORCED at the Control layer (typed `assistant-paused` / `human-only-action` / `no-engine` / `nothing-to-undo|redo` D-Bus errors; window-side `AssistantStateService` + `engine-state-sync.ts` re-push survives engine recreation — see `docs/concepts/ai-collaborator.md` § Pause contract / § State ownership). Punted: (a) `ListActions` still lists `win.toggle-assistant-paused` as driveable although Control always rejects it — consider a `humanOnly` flag in `ActionDescriptor`; (b) `SetAssistantCursor` reports pause as boolean `false` (bridge hint text), not a typed error; (c) the human-facing zoom/undo OSD buttons still no-op silently without an engine (Control reports typed errors; the buttons are scene-editor-only, low priority); (d) the Props "Remove" button swallows a `removeObject` `false` (missing placement) with no toast. *owner: maker*
 - **Collab op-sync WORKS end-to-end (verified 2026-06-06).** Full pair-edit flow proven with two same-machine instances: `present_window` → host engine inits → `start_session` (hosting) → joiner discovers over LAN → WebRTC connects → **snapshot transfers** (`captureProjectSnapshot` loads lazy maps on demand — fixed) → joiner attaches its engine (`connected`) → **host paints → joiner's map syncs the tiles** (shared op-log) → **bidirectional cursors render** (joiner's cursor on the host, and the AI-assistant cursor relays to the joiner — see `docs/concepts/ai-collaborator.md` Phase 5). The earlier "WebGL + GStreamer can't coexist in one process" diagnosis was **wrong**: both engines run WebGL + `webrtcbin` together fine once init succeeds. The real issue is the next item.
 - **Surface AI op attribution in the receiving peer's UI** — assistant-initiated ops carry `Operation.origin = ASSISTANT_PEER_ID` on the wire (Control → `paintTileAt`/`placeObjectAt`/window `undoRedo` → `executeCommand`/`undo`/`redo`; see `docs/concepts/ai-collaborator.md` § Edit attribution), and receivers re-emit it via `EngineEvent.REMOTE_COMMAND_APPLIED { command, direction, origin }` — but nothing renders it yet. Follow-ups: (a) subscribe to `REMOTE_COMMAND_APPLIED` and flash/badge AI-attributed remote edits in the assistant's colour (mirror of the host-local `_flashAssistantTile`); (b) optional: per-entry origin on `UndoStackComponent` if "undo only the AI's changes" is ever wanted. *owner: maker + gjs*

apps/maker-gjs/src/services/assistant-pause-policy.spec.ts

@@ -26,6 +26,7 @@ const EXPECTED_KINDS: Record<string, ControlMethodKind> = {
   ListActions: 'read-only',
   ListRecentProjects: 'read-only',
   ListTemplates: 'read-only',
+  GetMapData: 'read-only',
   GetSessionState: 'read-only',
   PresentWindow: 'read-only',
   SetAssistantInfo: 'presence',

apps/maker-gjs/src/services/assistant-pause-policy.ts

@@ -74,6 +74,7 @@ export const CONTROL_METHOD_KINDS = {
   ListActions: 'read-only',
   ListRecentProjects: 'read-only',
   ListTemplates: 'read-only',
+  GetMapData: 'read-only',
   GetSessionState: 'read-only',
   PresentWindow: 'read-only',
   SetAssistantInfo: 'presence',

apps/maker-gjs/src/services/control-dbus.service.ts

@@ -52,6 +52,10 @@ const CONTROL_IFACE_XML = `
     <method name="ListRecentProjects">
       <arg type="s" direction="out" name="projects_json"/>
     </method>
+    <method name="GetMapData">
+      <arg type="s" direction="in" name="map_id"/>
+      <arg type="s" direction="out" name="map_json"/>
+    </method>
     <method name="ListTemplates">
       <arg type="s" direction="out" name="templates_json"/>
     </method>
@@ -143,12 +147,16 @@ export class ControlDbusService {
     return JSON.stringify(win.getDebugStatus())
   }
 
-  /** `Screenshot(scope) -> ay` — PNG bytes. `scope`: `window` | `canvas`. */
-  Screenshot(scope: string): Uint8Array {
+  /**
+   * `Screenshot(scope) -> ay` — PNG bytes. `scope`: `window` |
+   * `canvas`. The canvas scope reads the WebGL framebuffer during the
+   * next frame (occlusion-proof); async like `StartSession`.
+   */
+  async Screenshot(scope: string): Promise<Uint8Array> {
     const win = this.window
     if (!win) return new Uint8Array(0)
     const mode = scope === 'canvas' ? 'canvas' : 'window'
-    return win.captureScreenshot(mode) ?? new Uint8Array(0)
+    return (await win.captureScreenshot(mode)) ?? new Uint8Array(0)
   }
 
   /** `ListActions() -> s` — JSON of the `app.*` + `win.*` actions. */
@@ -206,6 +214,23 @@ export class ControlDbusService {
     win.openProject(path)
   }
 
+  /**
+   * `GetMapData(map_id) -> s` — agent-oriented JSON projection of a
+   * loaded map: walkability grid (`.` walkable / `#` solid / ` ` void),
+   * placements with resolved names + component types, spawn points and
+   * teleports with targets. Kilobytes instead of pixels; needs no open
+   * scene, engine or visible window. Read-only (allowed while paused).
+   */
+  GetMapData(mapId: string): string {
+    const data = this.requireWindow().agentMapData(mapId)
+    if (!data) {
+      throw new Error(
+        `unknown-map: "${mapId}" is not part of the loaded project (or no project is open) — see GetStatus.sceneIds`,
+      )
+    }
+    return JSON.stringify(data)
+  }
+
   /** `ListRecentProjects() -> s` — JSON list of recently opened projects. */
   ListRecentProjects(): string {
     return JSON.stringify(loadRecentProjects())

apps/maker-gjs/src/widgets/application-window.ts

@@ -4,12 +4,15 @@ import GLib from '@girs/glib-2.0'
 import GObject from '@girs/gobject-2.0'
 import Gtk from '@girs/gtk-4.0'
 import {
+  type AgentMapData,
   ASSISTANT_PEER_ID,
   type AwarenessPeerState,
+  buildAgentMapData,
   createMapEditorDataOp,
   type EditorTool,
   formatError,
   MapFormat,
+  type SpriteSetData,
   type SpriteSetKind,
 } from '@pixelrpg/engine'
 import {
@@ -1899,14 +1902,39 @@ export class ApplicationWindow extends Adw.ApplicationWindow {
    * the scene editor isn't open for `'canvas'`, or the window isn't
    * realised yet).
    */
-  captureScreenshot(scope: 'window' | 'canvas'): Uint8Array | null {
+  async captureScreenshot(scope: 'window' | 'canvas'): Promise<Uint8Array | null> {
     if (scope === 'canvas') {
       const engine = this._engineCtl.engine
+      // Framebuffer first: read during the next frame (postdraw), so
+      // it returns real content even when the window is occluded —
+      // the WidgetPaintable snapshot comes back empty then. Widget
+      // snapshot stays as the fallback for engines without a GL
+      // context yet (or with a paused frame clock: minimised).
+      const fromGl = (await engine?.captureCanvasPng()) ?? null
+      if (fromGl) return fromGl
       if (engine) return captureWidgetPng(engine)
     }
     return captureWidgetPng(this)
   }
 
+  /**
+   * Agent-oriented projection of a loaded map (walkability grid,
+   * placements, spawn points, teleport targets) — see the engine's
+   * `buildAgentMapData`. Works with no open scene and no engine:
+   * resolved purely from the loaded project data. Returns `null`
+   * when no project is open or the map id is unknown.
+   */
+  agentMapData(mapId: string): AgentMapData | null {
+    const resource = this._projectStore.resource
+    const mapResource = resource?.maps.get(mapId)
+    if (!resource || !mapResource?.mapData) return null
+    const sets = new Map<string, SpriteSetData>()
+    for (const [id, setResource] of resource.spriteSets) {
+      if (setResource.data) sets.set(id, setResource.data)
+    }
+    return buildAgentMapData(mapResource.mapData, sets, resource.data?.entityLibrary ?? [])
+  }
+
   /**
    * Load a project from a `game-project.json` path — the headless
    * equivalent of the welcome view's file picker, for external tooling

apps/mcp-bridge/src/index.ts

@@ -424,9 +424,12 @@ server.registerTool(
   'screenshot',
   {
     description:
-      'Capture a PNG screenshot. scope "window" (default) = whole app window (chrome + canvas); ' +
-      '"canvas" = just the map/engine canvas. Auto-raises the window and retries once if the first ' +
-      'capture is empty (an occluded / minimized / mid-resize window re-renders to nothing).',
+      'Capture a PNG screenshot. scope "window" (default) = whole app window (chrome + canvas) via the GTK ' +
+      'snapshot pipeline — needs a visible window (auto-raises + retries once when the capture comes back ' +
+      'empty). scope "canvas" = just the rendered engine content, read DIRECTLY from the WebGL framebuffer: ' +
+      'no UI chrome, and it returns the last rendered frame even when the window is occluded or minimised — ' +
+      'prefer it for verifying map/game content. For reasoning about a map (walkability, placements, ' +
+      'teleports) prefer get_map_data over any screenshot.',
     inputSchema: z.object({ scope: z.enum(['window', 'canvas']).optional(), ...instanceArg }),
   },
   async ({ scope, instance }) => {
@@ -505,6 +508,33 @@ server.registerTool(
   },
 )
 
+server.registerTool(
+  'get_map_data',
+  {
+    description:
+      'Agent-oriented JSON projection of a loaded map: a walkability grid (one string per row — "." walkable, ' +
+      '"#" solid, " " void/no tile), placements with resolved names + component types, spawn points, and ' +
+      'teleports with their target map/tile. Kilobytes instead of pixels — use this to plan where to walk or ' +
+      'paint BEFORE reaching for screenshots. Needs an open project (see get_status sceneIds for valid map ids) ' +
+      'but NO open scene, engine or visible window.',
+    inputSchema: z.object({ map_id: z.string(), ...instanceArg }),
+  },
+  async ({ map_id, instance }) => {
+    try {
+      const reply = await control(
+        instance,
+        'GetMapData',
+        GLib.Variant.new_tuple([GLib.Variant.new_string(map_id)]),
+        '(s)',
+      )
+      const [json] = reply.recursiveUnpack() as [string]
+      return ok(JSON.stringify(JSON.parse(json), null, 2))
+    } catch (error) {
+      return dbusError(error, instance)
+    }
+  },
+)
+
 // actions -------------------------------------------------------------------
 
 server.registerTool(

games/zelda-like/maps/kokiri-forest.json

@@ -124804,7 +124804,7 @@
       "id": "tp-treehouse-door-l",
       "layerId": "layer_functional",
       "tileX": 76,
-      "tileY": 66,
+      "tileY": 67,
       "inline": {
         "id": "tp-treehouse-door-l-def",
         "name": "Link's tree house door",
@@ -124831,7 +124831,7 @@
       "id": "tp-treehouse-door-r",
       "layerId": "layer_functional",
       "tileX": 77,
-      "tileY": 66,
+      "tileY": 67,
       "inline": {
         "id": "tp-treehouse-door-r-def",
         "name": "Link's tree house door",
@@ -124879,4 +124879,4 @@
     "atlasX": 46,
     "atlasY": 106
   }
-}
+}

games/zelda-like/maps/tree-house.json

@@ -679,7 +679,7 @@
             "type": "teleport",
             "targetMapId": "kokiri-forest",
             "targetTileX": 76,
-            "targetTileY": 67,
+            "targetTileY": 68,
             "facing": "down",
             "label": "Kokiri Forest"
           }
@@ -693,7 +693,7 @@
       "id": "spawn-player",
       "layerId": "layer_functional",
       "tileX": 5,
-      "tileY": 6,
+      "tileY": 7,
       "inline": {
         "id": "spawn-player-def",
         "name": "Player spawn",