Add .agents/skills

.agents/skills/mineagent-dev-workflow/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: mineagent-dev-workflow
+description: >-
+  Dev workflow for MineAgent: Pixi environments and tasks, pytest, pre-commit
+  (ruff, pyright), and GitHub Actions. Use when running tests, formatting,
+  typechecking, building the Forge mod in CI, or setting up a contributor
+  environment.
+---
+
+# MineAgent dev workflow
+
+## Pixi
+
+- **Install**: `pixi install` (from repo root). Lockfile: `pixi.lock`.
+- **Platforms**: `pixi.toml` sets `platforms = ["linux-64"]` — resolving or running Pixi on other OS targets may require adjusting this for your machine or CI matrix.
+- **Default env**: runtime + `gradle`, `openjdk`, PyTorch stack; editable install `mineagent`.
+- **Dev env**: `pixi install -e dev` (adds Node for dev feature + `mineagent[dev]` extras: pytest, ruff, pre-commit, etc.).
+
+### Common commands
+
+| Goal | Command |
+|------|---------|
+| Run agent entrypoint | `pixi run mineagent` (optional: `-f config.yaml`, `-kvp key=value`) |
+| Pytest (dev extras) | `pixi run -e dev pytest ./tests` |
+| Forge client | `pixi run gradle-run-client` |
+| Forge build | `pixi run gradle-build` |
+| Forge Java tests | `pixi run gradle-test` |
+
+## Python tooling
+
+- **Lint / format**: `ruff check`, `ruff format` (see `.pre-commit-config.yaml`).
+- **Types**: `pyright` (`pyproject.toml` `[tool.pyright]`).
+- **Hooks**: `pre-commit install` then `pre-commit run --all-files` (local), or rely on CI.
+
+## CI (`.github/workflows/`)
+
+| Workflow | What it does |
+|----------|----------------|
+| `pytest.yml` | Checkout → `setup-pixi` (v0.62.2) → `pixi run -e dev pytest ./tests` |
+| `gradle-build.yml` | Pixi default env → `cd forge && pixi run gradle-build` |
+| `pre-commit.yml` | Repo hygiene hooks |
+
+## Package install (non-Pixi)
+
+`README.md` documents `pip install .` and optional conda; editable fork of MineDojo is mentioned for alternate stacks. **This repo’s CI and recommended path** center on **Pixi**.
+
+## Contributing checklist
+
+1. `pixi run -e dev pytest ./tests`
+2. `pixi run gradle-build` if Java or protocol changed
+3. `pre-commit run --all-files` (or let CI catch it)

.agents/skills/mineagent-forge-mod/SKILL.md

@@ -0,0 +1,54 @@
+---
+name: mineagent-forge-mod
+description: >-
+  Minecraft Forge mod in forge/: Gradle project, MC/Forge versions, Java package
+  com.mineagent, client networking and input bridge. Use when editing Java mod
+  code, Gradle, window sizing, or game-side observation/action behavior.
+---
+
+# MineAgent Forge mod (`forge/`)
+
+## Versions and build
+
+Defined in `forge/gradle.properties` (authoritative for the template):
+
+| Property | Typical value |
+|----------|----------------|
+| `minecraft_version` | 1.21.5 |
+| `minecraft_version_range` | `[1.21.5,1.22)` |
+| `forge_version` | 55.0.15 |
+| `mapping_channel` / `mapping_version` | official / 1.21.5 |
+| `mod_id` | `mineagent` |
+| `mod_group_id` | `com.mineagent` |
+
+**JDK**: Java **21** is pulled via Pixi in this repo (`pixi.toml` `openjdk`). **Gradle** is constrained to `<9` in Pixi for compatibility.
+
+## Pixi tasks (from repo root)
+
+| Task | Command | Purpose |
+|------|---------|---------|
+| Run Minecraft client with mod | `pixi run gradle-run-client` | `cd forge && gradle runClient` |
+| Build mod JAR | `pixi run gradle-build` | `cd forge && gradle build` |
+| Java tests | `pixi run gradle-test` | `cd forge && gradle test` |
+
+## Java source map (`forge/src/main/java/com/mineagent/`)
+
+| Class | Role |
+|-------|------|
+| `MineAgentMod` | `@Mod` entry; registers client setup, `Config`, `ClientEventHandler`; starts `NetworkHandler` on client |
+| `NetworkHandler` | Unix domain socket **servers** for observation stream + action stream; threads / executors |
+| `DataBridge` | Singleton between network thread and game: latest `RawInput`, `InputInjector`, “client connected” flag |
+| `InputInjector` | Applies agent input to the game (see class for MC integration) |
+| `RawInput` | Java record mirroring wire protocol (keys, mouse, buttons, scroll, text) |
+| `Observation` | Frame + reward passed toward network layer |
+| `ClientEventHandler` | Client tick / render hooks (capture path lives here) |
+| `Config` | Forge config spec (e.g. window dimensions; note log strings in `MineAgentMod` still mention TCP/UDP in places—**sockets are Unix domain**; trust `NetworkHandler` + Python `ConnectionConfig`) |
+
+## Tests
+
+JUnit under `forge/src/test/java/` (e.g. `RawInputTest`, `DataBridgeTest`).
+
+## When changing the mod
+
+- Keep **byte-level I/O** aligned with Python `mineagent/client/protocol.py` and `connection.py`—see **`mineagent-ipc`** skill.
+- After protocol or rendering changes, verify both **Gradle tests** and **Python tests** if Python parsing or env behavior is affected.

.agents/skills/mineagent-ipc/SKILL.md

@@ -0,0 +1,59 @@
+---
+name: mineagent-ipc
+description: >-
+  IPC contract between Python mineagent and the Forge mod: Unix socket paths,
+  observation framing (reward + frame length + RGB bytes), and RawInput action
+  serialization. Use when changing client/connection code, NetworkHandler, or
+  debugging desync between Java and Python.
+---
+
+# MineAgent IPC (Python ↔ Forge)
+
+## Transport
+
+- **Unix domain sockets** (not TCP in the current `NetworkHandler` / `AsyncMinecraftClient` defaults).
+- Default paths (must match **Python** `mineagent/client/connection.py` `ConnectionConfig` and **Java** `NetworkHandler` constants):
+  - Observations (server sends, Python reads): `/tmp/mineagent_observation.sock`
+  - Actions (Python sends, server reads): `/tmp/mineagent_action.sock`
+
+Changing paths requires updating **both** sides (or making Java read the same config source).
+
+## Observation wire format
+
+**Java** (`NetworkHandler.sendObservationImmediate`): one message per frame:
+
+1. `double` **reward**, big-endian (8 bytes)
+2. `int` **frameLength**, big-endian (4 bytes)
+3. **frameLength** bytes of raw **RGB** row-major pixel data (`height * width * 3`)
+
+**Python** (`AsyncMinecraftClient.receive_observation`): reads 12-byte header, then `frame_length` bytes; `parse_observation` in `mineagent/client/protocol.py` validates length against `ConnectionConfig.frame_height/frame_width`.
+
+If `frame_length == 0`, Python still returns an `Observation` with a **zero** frame matrix of the configured shape (see `connection.py`).
+
+## Action wire format (`RawInput`)
+
+**Python** `mineagent/client/protocol.py` `RawInput.to_bytes()`:
+
+1. `uint8` number of pressed keys `N`
+2. `N` × **big-endian int16** key codes (GLFW codes; canonical order for the Gym space is `KEY_LIST` / `NUM_KEYS` in the same module)
+3. `float32` **mouse_dx**, big-endian
+4. `float32` **mouse_dy**, big-endian
+5. `uint8` **mouse_buttons** (bit flags: left, right, middle)
+6. `float32` **scroll_delta**, big-endian
+7. `uint16` **text** UTF-8 length, big-endian
+8. `text` bytes (UTF-8), may be empty
+
+**Java** `NetworkHandler.handleActionClient` documents the same layout; parsing must stay byte-for-byte compatible.
+
+## Gymnasium action space vs wire format
+
+- `make_action_space()` builds a `Dict` space: `keys` (MultiBinary `NUM_KEYS`), `mouse_dx`, `mouse_dy`, `mouse_buttons`, `scroll_delta`.
+- `action_to_raw_input()` maps dict → `RawInput` → bytes.
+- **Focus / ROI** is **not** sent on the wire; it is internal to `AgentV1` / perception (see `mineagent-python` skill).
+
+## Parity checklist (when editing protocol)
+
+- [ ] Field order and endianness match in Python `RawInput.to_bytes` and Java read loop
+- [ ] Observation header 12 bytes; frame byte count equals `H*W*3` when non-zero
+- [ ] Default socket paths identical in `ConnectionConfig` and `NetworkHandler`
+- [ ] Frame dimensions: `MinecraftEnv` copies `engine.image_size` into `ConnectionConfig` / env config—keep consistent with mod capture resolution

.agents/skills/mineagent-overview/SKILL.md

@@ -0,0 +1,64 @@
+---
+name: mineagent-overview
+description: >-
+  Brief orientation for the MineAgent research repo: virtual intelligence in
+  Minecraft, current milestones, repo layout, and entrypoints. Use when
+  onboarding, summarizing the project, or deciding which deeper skill to read
+  next (Python agent, Forge mod, IPC, dev workflow).
+---
+
+# MineAgent overview
+
+## What this project is
+
+MineAgent is a **research codebase** for *virtual intelligence*: AI that exists and acts **inside** a virtual world. Minecraft is the primary testbed because it is diverse and challenging while still being a tractable approximation of rich environments.
+
+The project is **early-stage** (see `README.md`, `pyproject.toml` classifiers). The near-term research direction is **visual perception plus curiosity**, with the agent initially oriented toward **observation and head / attention control** (region-of-interest / camera-style movement) rather than full embodied locomotion mastery.
+
+## Repository map
+
+| Path | Role |
+|------|------|
+| `mineagent/` | Python package: Gymnasium env, RL agent, learning (PPO, ICM), monitoring, client |
+| `forge/` | Minecraft **Forge** mod (Java): frames, rewards, input injection, Unix-socket servers |
+| `config_templates/config.yaml` | Example YAML for `mineagent` CLI (`-f`) |
+| `tests/` | Pytest suite mirroring package structure |
+| `.github/workflows/` | CI: pytest (Pixi dev env), Gradle build, pre-commit |
+
+Entrypoint for running the loop: **`mineagent.engine:run`** (console script `mineagent` from `pyproject.toml`).
+
+## Runtime shape (one sentence)
+
+Python **`MinecraftEnv`** talks to the Forge mod over **Unix domain sockets** (observations in, actions out); **`AgentV1`** turns frames into actions and learns with **PPO** plus an **intrinsic curiosity (ICM)** signal.
+
+```mermaid
+flowchart LR
+  subgraph forgeMod [Forge_mod]
+    Net[NetworkHandler]
+    Inj[InputInjector]
+  end
+  subgraph pythonPkg [Python_mineagent]
+    Env[MinecraftEnv]
+    Client[AsyncMinecraftClient]
+    Ag[AgentV1]
+  end
+  Net -->|"observation_UDS"| Client
+  Client --> Env
+  Ag -->|"action_UDS"| Net
+  Env --> Ag
+```
+
+## Deeper skills (read as needed)
+
+Skills live under **`.agents/skills/`** in this repo. Other tools (Cursor, Opencode, etc.) may need you to **register or symlink** that path if they only auto-discover a default skills directory—point them at these folders or copy the `SKILL.md` trees they expect.
+
+| Skill | When to open it |
+|-------|-----------------|
+| `mineagent-python` | Changing agent, perception, learning, engine, or Gymnasium env |
+| `mineagent-forge-mod` | Changing the Java mod, Gradle, MC/Forge versions |
+| `mineagent-ipc` | Changing sockets, wire formats, or Python/Java protocol parity |
+| `mineagent-dev-workflow` | Pixi, pytest, pre-commit, CI |
+
+## External references (human)
+
+- Upstream-style homepage / issues URLs are in `pyproject.toml` `[project.urls]` (GitHub project name may differ from local folder name).

.agents/skills/mineagent-python/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: mineagent-python
+description: >-
+  Python package layout for MineAgent: Gymnasium MinecraftEnv, AgentV1
+  (perception, affector, PPO, ICM, ROI), engine loop, and YAML config. Use when
+  editing mineagent/, RL training logic, observation tensors, or run
+  configuration.
+---
+
+# MineAgent Python package
+
+## Layout (`mineagent/`)
+
+| Area | Path | Notes |
+|------|------|--------|
+| Run loop | `mineagent/engine.py` | Builds `Config`, `MinecraftEnv`, `AgentV1`; publishes monitoring events |
+| Env | `mineagent/env.py` | `MinecraftEnv` (`gymnasium.Env`), sync wrapper over `AsyncMinecraftClient` |
+| Config | `mineagent/config.py` | Dataclasses + YAML via dacite; CLI `-f` / `-kvp` |
+| Agent | `mineagent/agent/agent.py` | `AgentV1`: vision → affector → critic; PPO + ICM updates |
+| Perception | `mineagent/perception/visual.py` | Foveated + peripheral CNN branches, attention combiner |
+| Affector | `mineagent/affector/affector.py` | `LinearAffector` → `AffectorOutput` (env action distributions + focus head) |
+| Critic | `mineagent/reasoning/critic.py` | `LinearCritic` |
+| Dynamics | `mineagent/reasoning/dynamics.py` | `InverseDynamics`, `ForwardDynamics` (ICM) |
+| Learning | `mineagent/learning/ppo.py`, `icm.py`, `td.py` | PPO (Spinning Up–style), ICM; TD helper exists |
+| Memory | `mineagent/memory/trajectory.py` | `TrajectoryBuffer` (fixed maxlen) |
+| Client | `mineagent/client/` | Async UDS client, `protocol` (action space, `RawInput`) |
+| Monitoring | `mineagent/monitoring/` | Event bus, TensorBoard callbacks |
+| Utils | `mineagent/utils.py` | `sample_action`, `joint_logp_action`, hooks, tensorboard setup |
+
+## Engine loop (mental model)
+
+1. `get_config()` loads YAML and/or CLI overrides.
+2. `MinecraftEnv` connects on reset; observations are `uint8` HWC RGB.
+3. Each step: `agent.act(obs_tensor)` returns a **dict** action (keys, mouse deltas, buttons, scroll) for `env.step`.
+4. Monitoring: `event_bus` + optional TensorBoard writers.
+
+## AgentV1 (important semantics)
+
+- **Visual features**: `VisualPerception` takes full frame + ROI crop; ROI comes from previous **focus** output (or center crop initially).
+- **Two action streams**: `sample_action` returns (1) **environment** tensor used by env + ICM + PPO env loss, and (2) **focus** (2D ROI) with its own log-probs. PPO applies a separate REINFORCE-style term for focus; ICM inverse dynamics **excludes** focus (see `mineagent/learning/icm.py` comments).
+- **Updates**: When `TrajectoryBuffer` length hits `agent.max_buffer_size`, `ppo.update` and `icm.update` run.
+
+## Configuration
+
+- Full schema: dataclasses in `mineagent/config.py` (`EngineConfig`, `AgentConfig`, `MonitoringConfig`, nested PPO/ICM/TD).
+- Template: `config_templates/config.yaml`.
+- Run: `mineagent -f path/to.yaml` and/or `mineagent -kvp engine.max_steps=100 agent.ppo.actor_lr=1e-4` (nested keys with `.`).
+
+## Tests
+
+Mirror structure under `tests/`; run via dev workflow skill (`pixi run -e dev pytest ./tests`).