diff --git a/TODO.md b/TODO.md index ed5fd81b..b9ca14cb 100644 --- a/TODO.md +++ b/TODO.md @@ -18,7 +18,7 @@ Conventions: - **Entity-composition model (decided 2026-06-09)** — the agreed target content model lives in [`docs/concepts/entity-and-appearance-model.md`](docs/concepts/entity-and-appearance-model.md): explicit `components[]` on entity definitions + component registry (schema-generated inspectors) + editor **templates** replace the `ObjectKind`-switch + properties union; **Appearance** (sheet + animations) becomes a shared asset layer; `CharacterDefinition` folds in. Phases: **A** (landed — rename + deep-link); **B0** (landed — component registry + field DSL + 11 specs in `packages/engine/src/entity/`); **B1** (landed — objects refactor: `EntityDefinition.components[]`, registry-walk spawn in `entity/spawn-placement.ts`, `objectLibrary`→`entityLibrary`, `ObjectDefinition`/`ObjectKind` deleted, `games/*` migrated via `scripts/migrate-to-entity-components.mjs`); **B2** (landed — characters→`entityLibrary` (`character`-template), `isPlayer`→`playerActorId`, collab `entity.upsert`/`entity.remove`/`player.set`; `CharacterDefinition` kept as a non-persisted Cast view model via `entity/convert.ts`); **C1** (landed — generated component inspectors `ComponentInspector`/`EntityComponentsEditor`); **C2** (landed — `Objects` mode + library view + `ObjectsController` + `entity-templates.ts` + `win.new-object`/`open-object` + MCP `set_view objects`); **C3a** (landed — `PlaceObjectCommand`/`RemoveObjectCommand` + `MapScene.spawn`/`despawnPlacement`); **C3b** (landed — `object` tool + brush + `placeObjectAt` + DBus/MCP `place_object`; the scene-editor Objects-tab brush picker is a **visual sprite-thumbnail palette** + `win.set-inspector-tab` makes the inspector tabs driveable) (live spawn/despawn, undoable, collab); **C4** (landed — cast detail "all components" disclosure via `EntityComponentsEditor`; `mergeCharacterIntoEntity` preserves disclosure-added components on basic edits); **C5** (landed — Sheets-view unification: appearances + the animation editor relocate from Cast into the former Tiles view (now "Sheets"); Cast → Characters-only lens with `win.open-appearance` deep-link; `CastController` `appearances-changed` push, `win.open-sheet` removed); **C6** (landed — Objects = GENERAL lens over the whole `entityLibrary` incl. characters [drop the `!isCharacterEntity` filter] with a "Cast" badge; Cast = specialised character-only lens; cross-refresh via the `ProjectStore`'s `entity-library-changed` event; scene-editor object brush lists every entity except the player actor → Cast NPCs placeable; **"Cast member" toggle** in the Objects detail [`win.toggle-object-cast`] promotes/demotes an appearance-bearing entity into the Cast roster by flipping `editorData.template`↔'character' — unifies the Cast-NPC vs Objects-NPC confusion); **D** — declarative states (`EntityState` + `StateSystem`); **E** — script component + built-in code editor. *owner: engine + maker + gjs* - **Tileset editor follow-ups** — the Sheets view (former Tiles view) lists tilesets as cards (downscaled sheet thumbnails) with **import** (`win.new-tileset` → same `SpriteSetImportDialog`), **rename** (card ⋯ menu → `CardGallery` `rename-requested` → `ProjectStore.renameSpriteSet`, re-persists + broadcasts the descriptor), **drag-to-reorder** (`CardGallery.reorderable` DnD → `spriteset-reorder-requested` → `ProjectStore.reorderSpriteSets` rewrites `data.spriteSets[]` order; display-only, local — not collab-synced, joiners get it via the snapshot), and a **usage-aware delete** (the confirm names how many characters/maps still reference the set, via the shared `sprite-set-usage.ts` reverse-reference counter). Sprite-set CRUD + collab broadcast live in `ProjectStore` (the single project-level write pipeline; `apps/maker-gjs/src/services/project-store.ts`) — including **tile-property edits** (Solid / Surface switches: `TilesController` delegates to `ProjectStore.setTileSolid`/`setTileSurface`, which persist + broadcast `__project/spriteset.update.chunk`; receivers refresh live collision via `Engine.refreshTileSolidsForSpriteSet`). No tileset follow-ups outstanding. *owner: maker + gjs* - **`FloatingTopBar` breakpoint thresholds** — the 5 empirically-chosen stops (880sp split↔merged, 740 / 620 / 540 / 460sp disclosure inside merged) were picked while wiring the new chrome and should be re-measured once the inspector + library sidebars settle on final widths. Also verify the merged-pill doesn't wrap at 360sp phone-portrait widths. *owner: maker / gjs* -- **MCP orchestrator / `org.pixelrpg.maker.Control` follow-ups** — `apps/mcp-bridge` is a dev-only orchestrator: status / screenshot / action-driving, headless project load (`OpenProject` / `ListRecentProjects` / `ListTemplates`), **multi-instance** (env `PIXELRPG_INSTANCE` → per-instance app-id/D-Bus name; `launch_instance` / `list_instances` / `stop_instance` + an `instance` selector on every tool), **collab session primitives** (`start_session` / `join_session` / `get_session_state`), **`paint_tile`** (operation-oriented → undo + collab op-sync; shared `tile-paint.service` with `TileEditorSystem`), **`set_zoom`**, **`present_window`**, **`resize_window`** (`ResizeWindow` Control method + preset phone/tablet/desktop sizes — for exercising the responsive breakpoints) (+ `engineReady`/`mapped` in status). Decided against (re-add only if a real need shows up): canvas-composite screenshot fallback; D-Bus/MCP push notifications (an MCP agent acts on its next turn anyway — `get_status` polling covers it). Still wanted: (a) **extract a reusable `@gjsify/*` helper** once a 2nd consumer appears (e.g. easy6502); (b) **data-driven component projection** — a generic `get_components` that serialises the session-singleton's ECS components (auto-tracks new ones) instead of the hand-curated `get_status`. **Capture caveat (environmental):** in-process `Screenshot` works (project views *and* the WebGL canvas captured fine in earlier sessions), but it needs the window to be actively rendering frames. A backgrounded / occluded / screen-off test window keeps its initial paint (so the welcome view captures) but stops getting frame-callbacks after a view switch, so `WidgetPaintable.snapshot → to_node()` comes back empty for project views. `present_window` doesn't reliably un-occlude on Wayland. For agent-driven responsive-layout screenshots the instance must be visible/top (or use `gnome-session-inhibit` + foreground it). *owner: maker / gjsify* +- **MCP orchestrator / `org.pixelrpg.maker.Control` follow-ups** — `apps/mcp-bridge` is a dev-only orchestrator: status / screenshot / action-driving, headless project load (`OpenProject` / `ListRecentProjects` / `ListTemplates`), **multi-instance** (env `PIXELRPG_INSTANCE` → per-instance app-id/D-Bus name; `launch_instance` / `list_instances` / `stop_instance` + an `instance` selector on every tool), **collab session primitives** (`start_session` / `join_session` / `get_session_state`), **`paint_tile`** (operation-oriented → undo + collab op-sync; shared `tile-paint.service` with `TileEditorSystem`), **`set_zoom`**, **`present_window`**, **`resize_window`** (`ResizeWindow` Control method + preset phone/tablet/desktop sizes — for exercising the responsive breakpoints) (+ `engineReady`/`mapped` in status). Decided against (re-add only if a real need shows up): canvas-composite screenshot fallback; D-Bus/MCP push notifications (an MCP agent acts on its next turn anyway — `get_status` polling covers it). Still wanted: (a) **extract a reusable `@gjsify/*` helper** once a 2nd consumer appears (e.g. easy6502); (b) **data-driven component projection** — a generic `get_components` that serialises the session-singleton's ECS components (auto-tracks new ones) instead of the hand-curated `get_status`. Shipped 2026-06-12: **`get_map_data`** (agent-oriented walkability/placements/teleports projection, no engine/window needed) and the **framebuffer-based `canvas` screenshot scope** (gl.readPixels — occlusion-independent). Known quirk: the framebuffer capture can show a faint offset ghost of the scene (FBO double-buffer residue) — harmless for verification, revisit with the GL bridge. Also wanted: (c2) **blocking `open_scene`** — an optional wait-until-engineReady on the Control/MCP side would remove the agent's last gdbus polling loops. Still wanted on this front: (c) **`render_map`** — full-map offscreen render to PNG via the MapPreview Gsk pipeline (hand-built render nodes need no mapped widget), for visual map verification without any engine. **Capture caveat (environmental):** in-process `Screenshot` works (project views *and* the WebGL canvas captured fine in earlier sessions), but it needs the window to be actively rendering frames. A backgrounded / occluded / screen-off test window keeps its initial paint (so the welcome view captures) but stops getting frame-callbacks after a view switch, so `WidgetPaintable.snapshot → to_node()` comes back empty for project views. `present_window` doesn't reliably un-occlude on Wayland. For agent-driven responsive-layout screenshots the instance must be visible/top (or use `gnome-session-inhibit` + foreground it). *owner: maker / gjsify* - **Assistant pause / control-plane follow-ups** — pause is now ENFORCED at the Control layer (typed `assistant-paused` / `human-only-action` / `no-engine` / `nothing-to-undo|redo` D-Bus errors; window-side `AssistantStateService` + `engine-state-sync.ts` re-push survives engine recreation — see `docs/concepts/ai-collaborator.md` § Pause contract / § State ownership). Punted: (a) `ListActions` still lists `win.toggle-assistant-paused` as driveable although Control always rejects it — consider a `humanOnly` flag in `ActionDescriptor`; (b) `SetAssistantCursor` reports pause as boolean `false` (bridge hint text), not a typed error; (c) the human-facing zoom/undo OSD buttons still no-op silently without an engine (Control reports typed errors; the buttons are scene-editor-only, low priority); (d) the Props "Remove" button swallows a `removeObject` `false` (missing placement) with no toast. *owner: maker* - **Collab op-sync WORKS end-to-end (verified 2026-06-06).** Full pair-edit flow proven with two same-machine instances: `present_window` → host engine inits → `start_session` (hosting) → joiner discovers over LAN → WebRTC connects → **snapshot transfers** (`captureProjectSnapshot` loads lazy maps on demand — fixed) → joiner attaches its engine (`connected`) → **host paints → joiner's map syncs the tiles** (shared op-log) → **bidirectional cursors render** (joiner's cursor on the host, and the AI-assistant cursor relays to the joiner — see `docs/concepts/ai-collaborator.md` Phase 5). The earlier "WebGL + GStreamer can't coexist in one process" diagnosis was **wrong**: both engines run WebGL + `webrtcbin` together fine once init succeeds. The real issue is the next item. - **Surface AI op attribution in the receiving peer's UI** — assistant-initiated ops carry `Operation.origin = ASSISTANT_PEER_ID` on the wire (Control → `paintTileAt`/`placeObjectAt`/window `undoRedo` → `executeCommand`/`undo`/`redo`; see `docs/concepts/ai-collaborator.md` § Edit attribution), and receivers re-emit it via `EngineEvent.REMOTE_COMMAND_APPLIED { command, direction, origin }` — but nothing renders it yet. Follow-ups: (a) subscribe to `REMOTE_COMMAND_APPLIED` and flash/badge AI-attributed remote edits in the assistant's colour (mirror of the host-local `_flashAssistantTile`); (b) optional: per-entry origin on `UndoStackComponent` if "undo only the AI's changes" is ever wanted. *owner: maker + gjs* diff --git a/apps/maker-gjs/src/services/assistant-pause-policy.spec.ts b/apps/maker-gjs/src/services/assistant-pause-policy.spec.ts index 341a57ea..94c634d2 100644 --- a/apps/maker-gjs/src/services/assistant-pause-policy.spec.ts +++ b/apps/maker-gjs/src/services/assistant-pause-policy.spec.ts @@ -26,6 +26,7 @@ const EXPECTED_KINDS: Record = { ListActions: 'read-only', ListRecentProjects: 'read-only', ListTemplates: 'read-only', + GetMapData: 'read-only', GetSessionState: 'read-only', PresentWindow: 'read-only', SetAssistantInfo: 'presence', diff --git a/apps/maker-gjs/src/services/assistant-pause-policy.ts b/apps/maker-gjs/src/services/assistant-pause-policy.ts index f4044a69..c4151658 100644 --- a/apps/maker-gjs/src/services/assistant-pause-policy.ts +++ b/apps/maker-gjs/src/services/assistant-pause-policy.ts @@ -74,6 +74,7 @@ export const CONTROL_METHOD_KINDS = { ListActions: 'read-only', ListRecentProjects: 'read-only', ListTemplates: 'read-only', + GetMapData: 'read-only', GetSessionState: 'read-only', PresentWindow: 'read-only', SetAssistantInfo: 'presence', diff --git a/apps/maker-gjs/src/services/control-dbus.service.ts b/apps/maker-gjs/src/services/control-dbus.service.ts index a67bbe05..8fd92e14 100644 --- a/apps/maker-gjs/src/services/control-dbus.service.ts +++ b/apps/maker-gjs/src/services/control-dbus.service.ts @@ -52,6 +52,10 @@ const CONTROL_IFACE_XML = ` + + + + @@ -143,12 +147,16 @@ export class ControlDbusService { return JSON.stringify(win.getDebugStatus()) } - /** `Screenshot(scope) -> ay` — PNG bytes. `scope`: `window` | `canvas`. */ - Screenshot(scope: string): Uint8Array { + /** + * `Screenshot(scope) -> ay` — PNG bytes. `scope`: `window` | + * `canvas`. The canvas scope reads the WebGL framebuffer during the + * next frame (occlusion-proof); async like `StartSession`. + */ + async Screenshot(scope: string): Promise { const win = this.window if (!win) return new Uint8Array(0) const mode = scope === 'canvas' ? 'canvas' : 'window' - return win.captureScreenshot(mode) ?? new Uint8Array(0) + return (await win.captureScreenshot(mode)) ?? new Uint8Array(0) } /** `ListActions() -> s` — JSON of the `app.*` + `win.*` actions. */ @@ -206,6 +214,23 @@ export class ControlDbusService { win.openProject(path) } + /** + * `GetMapData(map_id) -> s` — agent-oriented JSON projection of a + * loaded map: walkability grid (`.` walkable / `#` solid / ` ` void), + * placements with resolved names + component types, spawn points and + * teleports with targets. Kilobytes instead of pixels; needs no open + * scene, engine or visible window. Read-only (allowed while paused). + */ + GetMapData(mapId: string): string { + const data = this.requireWindow().agentMapData(mapId) + if (!data) { + throw new Error( + `unknown-map: "${mapId}" is not part of the loaded project (or no project is open) — see GetStatus.sceneIds`, + ) + } + return JSON.stringify(data) + } + /** `ListRecentProjects() -> s` — JSON list of recently opened projects. */ ListRecentProjects(): string { return JSON.stringify(loadRecentProjects()) diff --git a/apps/maker-gjs/src/widgets/application-window.ts b/apps/maker-gjs/src/widgets/application-window.ts index ab4d764a..de44d184 100644 --- a/apps/maker-gjs/src/widgets/application-window.ts +++ b/apps/maker-gjs/src/widgets/application-window.ts @@ -4,12 +4,15 @@ import GLib from '@girs/glib-2.0' import GObject from '@girs/gobject-2.0' import Gtk from '@girs/gtk-4.0' import { + type AgentMapData, ASSISTANT_PEER_ID, type AwarenessPeerState, + buildAgentMapData, createMapEditorDataOp, type EditorTool, formatError, MapFormat, + type SpriteSetData, type SpriteSetKind, } from '@pixelrpg/engine' import { @@ -1899,14 +1902,39 @@ export class ApplicationWindow extends Adw.ApplicationWindow { * the scene editor isn't open for `'canvas'`, or the window isn't * realised yet). */ - captureScreenshot(scope: 'window' | 'canvas'): Uint8Array | null { + async captureScreenshot(scope: 'window' | 'canvas'): Promise { if (scope === 'canvas') { const engine = this._engineCtl.engine + // Framebuffer first: read during the next frame (postdraw), so + // it returns real content even when the window is occluded — + // the WidgetPaintable snapshot comes back empty then. Widget + // snapshot stays as the fallback for engines without a GL + // context yet (or with a paused frame clock: minimised). + const fromGl = (await engine?.captureCanvasPng()) ?? null + if (fromGl) return fromGl if (engine) return captureWidgetPng(engine) } return captureWidgetPng(this) } + /** + * Agent-oriented projection of a loaded map (walkability grid, + * placements, spawn points, teleport targets) — see the engine's + * `buildAgentMapData`. Works with no open scene and no engine: + * resolved purely from the loaded project data. Returns `null` + * when no project is open or the map id is unknown. + */ + agentMapData(mapId: string): AgentMapData | null { + const resource = this._projectStore.resource + const mapResource = resource?.maps.get(mapId) + if (!resource || !mapResource?.mapData) return null + const sets = new Map() + for (const [id, setResource] of resource.spriteSets) { + if (setResource.data) sets.set(id, setResource.data) + } + return buildAgentMapData(mapResource.mapData, sets, resource.data?.entityLibrary ?? []) + } + /** * Load a project from a `game-project.json` path — the headless * equivalent of the welcome view's file picker, for external tooling diff --git a/apps/mcp-bridge/src/index.ts b/apps/mcp-bridge/src/index.ts index 498216ad..2d97e2e7 100644 --- a/apps/mcp-bridge/src/index.ts +++ b/apps/mcp-bridge/src/index.ts @@ -424,9 +424,12 @@ server.registerTool( 'screenshot', { description: - 'Capture a PNG screenshot. scope "window" (default) = whole app window (chrome + canvas); ' + - '"canvas" = just the map/engine canvas. Auto-raises the window and retries once if the first ' + - 'capture is empty (an occluded / minimized / mid-resize window re-renders to nothing).', + 'Capture a PNG screenshot. scope "window" (default) = whole app window (chrome + canvas) via the GTK ' + + 'snapshot pipeline — needs a visible window (auto-raises + retries once when the capture comes back ' + + 'empty). scope "canvas" = just the rendered engine content, read DIRECTLY from the WebGL framebuffer: ' + + 'no UI chrome, and it returns the last rendered frame even when the window is occluded or minimised — ' + + 'prefer it for verifying map/game content. For reasoning about a map (walkability, placements, ' + + 'teleports) prefer get_map_data over any screenshot.', inputSchema: z.object({ scope: z.enum(['window', 'canvas']).optional(), ...instanceArg }), }, async ({ scope, instance }) => { @@ -505,6 +508,33 @@ server.registerTool( }, ) +server.registerTool( + 'get_map_data', + { + description: + 'Agent-oriented JSON projection of a loaded map: a walkability grid (one string per row — "." walkable, ' + + '"#" solid, " " void/no tile), placements with resolved names + component types, spawn points, and ' + + 'teleports with their target map/tile. Kilobytes instead of pixels — use this to plan where to walk or ' + + 'paint BEFORE reaching for screenshots. Needs an open project (see get_status sceneIds for valid map ids) ' + + 'but NO open scene, engine or visible window.', + inputSchema: z.object({ map_id: z.string(), ...instanceArg }), + }, + async ({ map_id, instance }) => { + try { + const reply = await control( + instance, + 'GetMapData', + GLib.Variant.new_tuple([GLib.Variant.new_string(map_id)]), + '(s)', + ) + const [json] = reply.recursiveUnpack() as [string] + return ok(JSON.stringify(JSON.parse(json), null, 2)) + } catch (error) { + return dbusError(error, instance) + } + }, +) + // actions ------------------------------------------------------------------- server.registerTool( diff --git a/games/zelda-like/maps/kokiri-forest.json b/games/zelda-like/maps/kokiri-forest.json index 2c69eb58..a93c18bf 100644 --- a/games/zelda-like/maps/kokiri-forest.json +++ b/games/zelda-like/maps/kokiri-forest.json @@ -124804,7 +124804,7 @@ "id": "tp-treehouse-door-l", "layerId": "layer_functional", "tileX": 76, - "tileY": 66, + "tileY": 67, "inline": { "id": "tp-treehouse-door-l-def", "name": "Link's tree house door", @@ -124831,7 +124831,7 @@ "id": "tp-treehouse-door-r", "layerId": "layer_functional", "tileX": 77, - "tileY": 66, + "tileY": 67, "inline": { "id": "tp-treehouse-door-r-def", "name": "Link's tree house door", @@ -124879,4 +124879,4 @@ "atlasX": 46, "atlasY": 106 } -} \ No newline at end of file +} diff --git a/games/zelda-like/maps/tree-house.json b/games/zelda-like/maps/tree-house.json index 61bb4e98..3b5feab9 100644 --- a/games/zelda-like/maps/tree-house.json +++ b/games/zelda-like/maps/tree-house.json @@ -679,7 +679,7 @@ "type": "teleport", "targetMapId": "kokiri-forest", "targetTileX": 76, - "targetTileY": 67, + "targetTileY": 68, "facing": "down", "label": "Kokiri Forest" } @@ -693,7 +693,7 @@ "id": "spawn-player", "layerId": "layer_functional", "tileX": 5, - "tileY": 6, + "tileY": 7, "inline": { "id": "spawn-player-def", "name": "Player spawn", diff --git a/packages/engine/src/services/agent-map-data.spec.ts b/packages/engine/src/services/agent-map-data.spec.ts new file mode 100644 index 00000000..793e3917 --- /dev/null +++ b/packages/engine/src/services/agent-map-data.spec.ts @@ -0,0 +1,133 @@ +/** + * `buildAgentMapData` — the agent-oriented map projection. Pins the + * walkability fold (void / walkable / solid across visible layers, + * solid wins, hidden layers ignored, blocking placements stamp solid) + * and the placement summaries (canonical defId/inline resolution, + * spawn-point + teleport extraction). + */ + +import { describe, expect, it } from '@gjsify/unit' + +import type { EntityDefinition, MapData, SpriteSetData } from '../types/data/index.ts' +import { buildAgentMapData } from './agent-map-data.ts' + +function makeFixture(): { mapData: MapData; sets: Map; library: EntityDefinition[] } { + const sets = new Map([ + [ + 'terrain', + { + id: 'terrain', + sprites: [ + { id: 0, col: 0, row: 0 }, + { id: 1, col: 1, row: 0, solid: true }, + ], + } as unknown as SpriteSetData, + ], + ]) + const mapData = { + id: 'm1', + name: 'Test Map', + columns: 3, + rows: 2, + tileWidth: 16, + tileHeight: 16, + layers: [ + { + id: 'ground', + name: 'Ground', + visible: true, + sprites: [ + { x: 0, y: 0, spriteId: 0, spriteSetId: 'terrain' }, + { x: 1, y: 0, spriteId: 1, spriteSetId: 'terrain' }, + { x: 0, y: 1, spriteId: 0, spriteSetId: 'terrain' }, + ], + }, + { + id: 'hidden', + name: 'Hidden', + visible: false, + // Solid here must NOT count — the layer is hidden. + sprites: [{ x: 0, y: 0, spriteId: 1, spriteSetId: 'terrain' }], + }, + ], + objectPlacements: [ + { + id: 'p-door', + layerId: 'ground', + tileX: 2, + tileY: 1, + inline: { + id: 'door-def', + name: 'Door', + components: [ + { type: 'trigger', on: 'walk-onto' }, + { type: 'teleport', targetMapId: 'm2', targetTileX: 5, targetTileY: 6, label: 'Inside' }, + ], + }, + }, + { + id: 'p-spawn', + layerId: 'ground', + tileX: 0, + tileY: 1, + defId: 'spawn-def', + }, + { + id: 'p-rock', + layerId: 'ground', + tileX: 0, + tileY: 0, + inline: { id: 'rock-def', name: 'Rock', components: [{ type: 'blocking' }] }, + }, + ], + } as unknown as MapData + const library: EntityDefinition[] = [ + { + id: 'spawn-def', + name: 'Player spawn', + components: [{ type: 'spawn-point', spawnId: 'player', facing: 'down' }], + } as unknown as EntityDefinition, + ] + return { mapData, sets, library } +} + +export default async () => { + await describe('buildAgentMapData', async () => { + await it('folds walkability: walkable / solid / void, hidden layers ignored', async () => { + const { mapData, sets, library } = makeFixture() + const data = buildAgentMapData(mapData, sets, library) + + // (0,0) walkable tile but a blocking placement → '#' + // (1,0) solid sprite → '#'; (2,0) no tile → ' ' + expect(data.walkability[0]).toBe('## ') + // (0,1) walkable; (1,1) void; (2,1) void (door placement ≠ tile) + expect(data.walkability[1]).toBe('. ') + }) + + await it('summarises placements with resolved names + component types', async () => { + const { mapData, sets, library } = makeFixture() + const data = buildAgentMapData(mapData, sets, library) + + const door = data.placements.find((p) => p.id === 'p-door') + expect(door?.name).toBe('Door') + expect(door?.components).toStrictEqual(['trigger', 'teleport']) + // defId placements resolve through the library (the canonical resolver) + const spawn = data.placements.find((p) => p.id === 'p-spawn') + expect(spawn?.name).toBe('Player spawn') + }) + + await it('extracts teleports with targets and spawn points with facing', async () => { + const { mapData, sets, library } = makeFixture() + const data = buildAgentMapData(mapData, sets, library) + + expect(data.teleports.length).toBe(1) + expect(data.teleports[0].targetMapId).toBe('m2') + expect(data.teleports[0].targetTileX).toBe(5) + expect(data.teleports[0].label).toBe('Inside') + + expect(data.spawnPoints.length).toBe(1) + expect(data.spawnPoints[0].spawnId).toBe('player') + expect(data.spawnPoints[0].facing).toBe('down') + }) + }) +} diff --git a/packages/engine/src/services/agent-map-data.ts b/packages/engine/src/services/agent-map-data.ts new file mode 100644 index 00000000..c50e2c61 --- /dev/null +++ b/packages/engine/src/services/agent-map-data.ts @@ -0,0 +1,161 @@ +import { resolvePlacementDefinition } from '../entity/data-access.ts' +import type { EntityDefinition, GameProjectData, MapData, SpriteSetData } from '../types/data/index.ts' + +/** + * Compact, agent-oriented projection of a map — the data an external + * driver (MCP/D-Bus agent) needs to REASON about a map without + * reading pixels: where it can walk, what is placed where, and where + * doors lead. Kilobytes instead of screenshots, and available with no + * live engine, scene or visible window. + * + * Pure data-in/data-out (map JSON + sprite-set descriptors) so it is + * unit-testable and identical between editor, future game-browser + * runtime and headless tooling. + */ +export interface AgentMapData { + id: string + name: string + columns: number + rows: number + tileWidth: number + tileHeight: number + /** + * One string per row, one char per column: + * `.` walkable tile, `#` solid tile, ` ` void (no tile on any + * visible layer — out of bounds for gameplay purposes). + * Solidity folds every visible layer (any solid sprite at a cell + * makes it solid) plus blocking placements. + */ + walkability: string[] + placements: AgentPlacement[] + spawnPoints: Array<{ id: string; tileX: number; tileY: number; spawnId: string; facing?: string }> + teleports: Array<{ + id: string + tileX: number + tileY: number + targetMapId: string + targetTileX: number + targetTileY: number + label?: string + }> +} + +export interface AgentPlacement { + id: string + name: string + tileX: number + tileY: number + layerId: string + /** Component type ids, e.g. `["visual","trigger","teleport"]`. */ + components: string[] +} + +/** Look up a component of `type` on a resolved definition. */ +function componentOf(def: EntityDefinition | null, type: string): Record | null { + const found = def?.components?.find((c) => c.type === type) + return found ? (found as unknown as Record) : null +} + +/** + * Build the {@link AgentMapData} projection. + * + * `spriteSets` maps sprite-set id → descriptor (for the per-tile + * `solid` flags); `entityLibrary` resolves `defId` placements the + * same way spawning does ({@link resolvePlacementDefinition} — ONE + * resolver, see entity/data-access.ts). + */ +export function buildAgentMapData( + mapData: MapData, + spriteSets: ReadonlyMap, + entityLibrary: GameProjectData['entityLibrary'] = [], +): AgentMapData { + const columns = mapData.columns ?? 0 + const rows = mapData.rows ?? 0 + + // ── walkability fold: void → '.', any solid sprite → '#' + const VOID = 0 + const WALKABLE = 1 + const SOLID = 2 + const grid = new Uint8Array(columns * rows) + for (const layer of mapData.layers ?? []) { + if (!layer.visible || !layer.sprites) continue + for (const sprite of layer.sprites) { + if (sprite.x < 0 || sprite.y < 0 || sprite.x >= columns || sprite.y >= rows) continue + const i = sprite.y * columns + sprite.x + if (grid[i] === SOLID) continue + const def = spriteSets.get(sprite.spriteSetId)?.sprites?.find((s) => s.id === sprite.spriteId) + grid[i] = def?.solid ? SOLID : Math.max(grid[i], WALKABLE) + } + } + + // ── placements: resolve via the canonical resolver, collect specials + const placements: AgentPlacement[] = [] + const spawnPoints: AgentMapData['spawnPoints'] = [] + const teleports: AgentMapData['teleports'] = [] + for (const placement of mapData.objectPlacements ?? []) { + const def = resolvePlacementDefinition(placement, entityLibrary) + const types = def?.components?.map((c) => c.type) ?? [] + placements.push({ + id: placement.id, + name: def?.name ?? placement.id, + tileX: placement.tileX, + tileY: placement.tileY, + layerId: placement.layerId, + components: types, + }) + + const blocking = componentOf(def, 'blocking') + if ( + blocking && + placement.tileX >= 0 && + placement.tileY >= 0 && + placement.tileX < columns && + placement.tileY < rows + ) { + grid[placement.tileY * columns + placement.tileX] = SOLID + } + const spawn = componentOf(def, 'spawn-point') + if (spawn) { + spawnPoints.push({ + id: placement.id, + tileX: placement.tileX, + tileY: placement.tileY, + spawnId: typeof spawn.spawnId === 'string' ? spawn.spawnId : 'player', + facing: typeof spawn.facing === 'string' ? spawn.facing : undefined, + }) + } + const teleport = componentOf(def, 'teleport') + if (teleport) { + teleports.push({ + id: placement.id, + tileX: placement.tileX, + tileY: placement.tileY, + targetMapId: String(teleport.targetMapId ?? ''), + targetTileX: Number(teleport.targetTileX ?? 0), + targetTileY: Number(teleport.targetTileY ?? 0), + label: typeof teleport.label === 'string' ? teleport.label : undefined, + }) + } + } + + const chars = [' ', '.', '#'] + const walkability: string[] = [] + for (let y = 0; y < rows; y++) { + let row = '' + for (let x = 0; x < columns; x++) row += chars[grid[y * columns + x]] + walkability.push(row) + } + + return { + id: mapData.id, + name: mapData.name ?? mapData.id, + columns, + rows, + tileWidth: mapData.tileWidth ?? 16, + tileHeight: mapData.tileHeight ?? 16, + walkability, + placements, + spawnPoints, + teleports, + } +} diff --git a/packages/engine/src/services/index.ts b/packages/engine/src/services/index.ts index e3a3c6be..816cb1e9 100644 --- a/packages/engine/src/services/index.ts +++ b/packages/engine/src/services/index.ts @@ -1,5 +1,6 @@ // Auto-generated by `gjsify barrels` — do not edit by hand. +export * from './agent-map-data' export * from './command-dispatch' export * from './editor-view' export * from './layer-visibility' diff --git a/packages/engine/src/test.mts b/packages/engine/src/test.mts index 48d8a181..78b3c743 100644 --- a/packages/engine/src/test.mts +++ b/packages/engine/src/test.mts @@ -26,6 +26,7 @@ import entitySpawnPlacementSuite from './entity/spawn-placement.spec.js' import entityValidateSuite from './entity/validate.spec.js' import objectSystemValidationSuite from './format/object-system-validation.spec.js' import mapResourceSuite from './resource/map-resource.spec.js' +import agentMapDataSuite from './services/agent-map-data.spec.js' import layerVisibilitySuite from './services/layer-visibility.spec.js' import spriteValidatorSuite from './services/sprite.validator.spec.js' import spriteInfoResolverSuite from './services/sprite-info.resolver.spec.js' @@ -61,6 +62,7 @@ run({ layerVisibilitySuite, spriteInfoResolverSuite, spriteValidatorSuite, + agentMapDataSuite, awarenessSuite, chunkingSuite, collabIntegrationSuite, diff --git a/packages/gjs/src/widgets/engine/engine.ts b/packages/gjs/src/widgets/engine/engine.ts index fca04cf8..abd0e917 100644 --- a/packages/gjs/src/widgets/engine/engine.ts +++ b/packages/gjs/src/widgets/engine/engine.ts @@ -1,4 +1,6 @@ import Adw from '@girs/adw-1' +import GdkPixbuf from '@girs/gdkpixbuf-2.0' +import GLib from '@girs/glib-2.0' import GObject from '@girs/gobject-2.0' import type Gtk from '@girs/gtk-4.0' import { Canvas2DBridge } from '@gjsify/canvas2d' @@ -262,6 +264,74 @@ export class Engine extends Adw.Bin { return this._excalibur } + /** + * Capture the engine canvas as PNG bytes by reading the WebGL + * framebuffer DURING the next frame (`postdraw` — the only moment + * GTK's GLArea framebuffer is bound and holds the finished frame; + * an out-of-frame `gl.readPixels` returns blanks). No + * `Gtk.WidgetPaintable`, so it works while the window is occluded — + * GTK keeps ticking mapped-but-covered windows. Resolves `null` + * when no engine is running or no frame arrives within `timeoutMs` + * (e.g. minimised/unmapped: no frame clock — callers fall back to + * the widget-snapshot path). + */ + public captureCanvasPng(timeoutMs = 1000): Promise { + const area = this._widget + if (!area || !(area instanceof WebGLBridge)) return Promise.resolve(null) + return new Promise((resolve) => { + let done = false + let handlerId = 0 + const finish = (result: Uint8Array | null) => { + if (done) return + done = true + clearTimeout(timer) + if (handlerId) area.disconnect(handlerId) + resolve(result) + } + const timer = setTimeout(() => finish(null), timeoutMs) + // connect_after('render'): runs after the frame was drawn into + // the GLArea framebuffer, which is only bound inside ::render — + // an out-of-frame readPixels reads blanks. + handlerId = area.connect_after('render', () => { + finish(this._readFramebufferPng()) + return false + }) + // Force a frame even when the clock is idle (nothing animating). + area.queue_render() + }) + } + + /** `gl.readPixels` + row flip + PNG encode — call only inside a frame (`postdraw`). */ + private _readFramebufferPng(): Uint8Array | null { + const gl = (this._excalibur?.excalibur?.graphicsContext as unknown as { __gl?: WebGL2RenderingContext })?.__gl + if (!gl) return null + const width = gl.drawingBufferWidth + const height = gl.drawingBufferHeight + if (width <= 0 || height <= 0) return null + + const rowBytes = width * 4 + const pixels = new Uint8Array(rowBytes * height) + gl.readPixels(0, 0, width, height, gl.RGBA, gl.UNSIGNED_BYTE, pixels) + + // GL rows are bottom-up — flip into GdkPixbuf's top-down order. + const flipped = new Uint8Array(pixels.length) + for (let y = 0; y < height; y++) { + flipped.set(pixels.subarray(y * rowBytes, (y + 1) * rowBytes), (height - 1 - y) * rowBytes) + } + + const pixbuf = GdkPixbuf.Pixbuf.new_from_bytes( + GLib.Bytes.new(flipped), + GdkPixbuf.Colorspace.RGB, + true, + 8, + width, + height, + rowBytes, + ) + const [ok, buffer] = pixbuf.save_to_bufferv('png', [], []) + return ok && buffer ? new Uint8Array(buffer) : null + } + /** Current camera zoom, or `null` if the engine isn't running yet. */ public getCameraZoom(): number | null { const camera = this._excalibur?.excalibur?.currentScene?.camera