diff --git a/.agents/skills/mineagent-dev-workflow/SKILL.md b/.agents/skills/mineagent-dev-workflow/SKILL.md new file mode 100644 index 0000000..4c570c7 --- /dev/null +++ b/.agents/skills/mineagent-dev-workflow/SKILL.md @@ -0,0 +1,51 @@ +--- +name: mineagent-dev-workflow +description: >- + Dev workflow for MineAgent: Pixi environments and tasks, pytest, pre-commit + (ruff, pyright), and GitHub Actions. Use when running tests, formatting, + typechecking, building the Forge mod in CI, or setting up a contributor + environment. +--- + +# MineAgent dev workflow + +## Pixi + +- **Install**: `pixi install` (from repo root). Lockfile: `pixi.lock`. +- **Platforms**: `pixi.toml` sets `platforms = ["linux-64"]` — resolving or running Pixi on other OS targets may require adjusting this for your machine or CI matrix. +- **Default env**: runtime + `gradle`, `openjdk`, PyTorch stack; editable install `mineagent`. +- **Dev env**: `pixi install -e dev` (adds Node for dev feature + `mineagent[dev]` extras: pytest, ruff, pre-commit, etc.). + +### Common commands + +| Goal | Command | +|------|---------| +| Run agent entrypoint | `pixi run mineagent` (optional: `-f config.yaml`, `-kvp key=value`) | +| Pytest (dev extras) | `pixi run -e dev pytest ./tests` | +| Forge client | `pixi run gradle-run-client` | +| Forge build | `pixi run gradle-build` | +| Forge Java tests | `pixi run gradle-test` | + +## Python tooling + +- **Lint / format**: `ruff check`, `ruff format` (see `.pre-commit-config.yaml`). +- **Types**: `pyright` (`pyproject.toml` `[tool.pyright]`). +- **Hooks**: `pre-commit install` then `pre-commit run --all-files` (local), or rely on CI. + +## CI (`.github/workflows/`) + +| Workflow | What it does | +|----------|----------------| +| `pytest.yml` | Checkout → `setup-pixi` (v0.62.2) → `pixi run -e dev pytest ./tests` | +| `gradle-build.yml` | Pixi default env → `cd forge && pixi run gradle-build` | +| `pre-commit.yml` | Repo hygiene hooks | + +## Package install (non-Pixi) + +`README.md` documents `pip install .` and optional conda; editable fork of MineDojo is mentioned for alternate stacks. **This repo’s CI and recommended path** center on **Pixi**. + +## Contributing checklist + +1. `pixi run -e dev pytest ./tests` +2. `pixi run gradle-build` if Java or protocol changed +3. `pre-commit run --all-files` (or let CI catch it) diff --git a/.agents/skills/mineagent-forge-mod/SKILL.md b/.agents/skills/mineagent-forge-mod/SKILL.md new file mode 100644 index 0000000..aa4dcbf --- /dev/null +++ b/.agents/skills/mineagent-forge-mod/SKILL.md @@ -0,0 +1,54 @@ +--- +name: mineagent-forge-mod +description: >- + Minecraft Forge mod in forge/: Gradle project, MC/Forge versions, Java package + com.mineagent, client networking and input bridge. Use when editing Java mod + code, Gradle, window sizing, or game-side observation/action behavior. +--- + +# MineAgent Forge mod (`forge/`) + +## Versions and build + +Defined in `forge/gradle.properties` (authoritative for the template): + +| Property | Typical value | +|----------|----------------| +| `minecraft_version` | 1.21.5 | +| `minecraft_version_range` | `[1.21.5,1.22)` | +| `forge_version` | 55.0.15 | +| `mapping_channel` / `mapping_version` | official / 1.21.5 | +| `mod_id` | `mineagent` | +| `mod_group_id` | `com.mineagent` | + +**JDK**: Java **21** is pulled via Pixi in this repo (`pixi.toml` `openjdk`). **Gradle** is constrained to `<9` in Pixi for compatibility. + +## Pixi tasks (from repo root) + +| Task | Command | Purpose | +|------|---------|---------| +| Run Minecraft client with mod | `pixi run gradle-run-client` | `cd forge && gradle runClient` | +| Build mod JAR | `pixi run gradle-build` | `cd forge && gradle build` | +| Java tests | `pixi run gradle-test` | `cd forge && gradle test` | + +## Java source map (`forge/src/main/java/com/mineagent/`) + +| Class | Role | +|-------|------| +| `MineAgentMod` | `@Mod` entry; registers client setup, `Config`, `ClientEventHandler`; starts `NetworkHandler` on client | +| `NetworkHandler` | Unix domain socket **servers** for observation stream + action stream; threads / executors | +| `DataBridge` | Singleton between network thread and game: latest `RawInput`, `InputInjector`, “client connected” flag | +| `InputInjector` | Applies agent input to the game (see class for MC integration) | +| `RawInput` | Java record mirroring wire protocol (keys, mouse, buttons, scroll, text) | +| `Observation` | Frame + reward passed toward network layer | +| `ClientEventHandler` | Client tick / render hooks (capture path lives here) | +| `Config` | Forge config spec (e.g. window dimensions; note log strings in `MineAgentMod` still mention TCP/UDP in places—**sockets are Unix domain**; trust `NetworkHandler` + Python `ConnectionConfig`) | + +## Tests + +JUnit under `forge/src/test/java/` (e.g. `RawInputTest`, `DataBridgeTest`). + +## When changing the mod + +- Keep **byte-level I/O** aligned with Python `mineagent/client/protocol.py` and `connection.py`—see **`mineagent-ipc`** skill. +- After protocol or rendering changes, verify both **Gradle tests** and **Python tests** if Python parsing or env behavior is affected. diff --git a/.agents/skills/mineagent-ipc/SKILL.md b/.agents/skills/mineagent-ipc/SKILL.md new file mode 100644 index 0000000..030f46d --- /dev/null +++ b/.agents/skills/mineagent-ipc/SKILL.md @@ -0,0 +1,59 @@ +--- +name: mineagent-ipc +description: >- + IPC contract between Python mineagent and the Forge mod: Unix socket paths, + observation framing (reward + frame length + RGB bytes), and RawInput action + serialization. Use when changing client/connection code, NetworkHandler, or + debugging desync between Java and Python. +--- + +# MineAgent IPC (Python ↔ Forge) + +## Transport + +- **Unix domain sockets** (not TCP in the current `NetworkHandler` / `AsyncMinecraftClient` defaults). +- Default paths (must match **Python** `mineagent/client/connection.py` `ConnectionConfig` and **Java** `NetworkHandler` constants): + - Observations (server sends, Python reads): `/tmp/mineagent_observation.sock` + - Actions (Python sends, server reads): `/tmp/mineagent_action.sock` + +Changing paths requires updating **both** sides (or making Java read the same config source). + +## Observation wire format + +**Java** (`NetworkHandler.sendObservationImmediate`): one message per frame: + +1. `double` **reward**, big-endian (8 bytes) +2. `int` **frameLength**, big-endian (4 bytes) +3. **frameLength** bytes of raw **RGB** row-major pixel data (`height * width * 3`) + +**Python** (`AsyncMinecraftClient.receive_observation`): reads 12-byte header, then `frame_length` bytes; `parse_observation` in `mineagent/client/protocol.py` validates length against `ConnectionConfig.frame_height/frame_width`. + +If `frame_length == 0`, Python still returns an `Observation` with a **zero** frame matrix of the configured shape (see `connection.py`). + +## Action wire format (`RawInput`) + +**Python** `mineagent/client/protocol.py` `RawInput.to_bytes()`: + +1. `uint8` number of pressed keys `N` +2. `N` × **big-endian int16** key codes (GLFW codes; canonical order for the Gym space is `KEY_LIST` / `NUM_KEYS` in the same module) +3. `float32` **mouse_dx**, big-endian +4. `float32` **mouse_dy**, big-endian +5. `uint8` **mouse_buttons** (bit flags: left, right, middle) +6. `float32` **scroll_delta**, big-endian +7. `uint16` **text** UTF-8 length, big-endian +8. `text` bytes (UTF-8), may be empty + +**Java** `NetworkHandler.handleActionClient` documents the same layout; parsing must stay byte-for-byte compatible. + +## Gymnasium action space vs wire format + +- `make_action_space()` builds a `Dict` space: `keys` (MultiBinary `NUM_KEYS`), `mouse_dx`, `mouse_dy`, `mouse_buttons`, `scroll_delta`. +- `action_to_raw_input()` maps dict → `RawInput` → bytes. +- **Focus / ROI** is **not** sent on the wire; it is internal to `AgentV1` / perception (see `mineagent-python` skill). + +## Parity checklist (when editing protocol) + +- [ ] Field order and endianness match in Python `RawInput.to_bytes` and Java read loop +- [ ] Observation header 12 bytes; frame byte count equals `H*W*3` when non-zero +- [ ] Default socket paths identical in `ConnectionConfig` and `NetworkHandler` +- [ ] Frame dimensions: `MinecraftEnv` copies `engine.image_size` into `ConnectionConfig` / env config—keep consistent with mod capture resolution diff --git a/.agents/skills/mineagent-overview/SKILL.md b/.agents/skills/mineagent-overview/SKILL.md new file mode 100644 index 0000000..380f8ac --- /dev/null +++ b/.agents/skills/mineagent-overview/SKILL.md @@ -0,0 +1,64 @@ +--- +name: mineagent-overview +description: >- + Brief orientation for the MineAgent research repo: virtual intelligence in + Minecraft, current milestones, repo layout, and entrypoints. Use when + onboarding, summarizing the project, or deciding which deeper skill to read + next (Python agent, Forge mod, IPC, dev workflow). +--- + +# MineAgent overview + +## What this project is + +MineAgent is a **research codebase** for *virtual intelligence*: AI that exists and acts **inside** a virtual world. Minecraft is the primary testbed because it is diverse and challenging while still being a tractable approximation of rich environments. + +The project is **early-stage** (see `README.md`, `pyproject.toml` classifiers). The near-term research direction is **visual perception plus curiosity**, with the agent initially oriented toward **observation and head / attention control** (region-of-interest / camera-style movement) rather than full embodied locomotion mastery. + +## Repository map + +| Path | Role | +|------|------| +| `mineagent/` | Python package: Gymnasium env, RL agent, learning (PPO, ICM), monitoring, client | +| `forge/` | Minecraft **Forge** mod (Java): frames, rewards, input injection, Unix-socket servers | +| `config_templates/config.yaml` | Example YAML for `mineagent` CLI (`-f`) | +| `tests/` | Pytest suite mirroring package structure | +| `.github/workflows/` | CI: pytest (Pixi dev env), Gradle build, pre-commit | + +Entrypoint for running the loop: **`mineagent.engine:run`** (console script `mineagent` from `pyproject.toml`). + +## Runtime shape (one sentence) + +Python **`MinecraftEnv`** talks to the Forge mod over **Unix domain sockets** (observations in, actions out); **`AgentV1`** turns frames into actions and learns with **PPO** plus an **intrinsic curiosity (ICM)** signal. + +```mermaid +flowchart LR + subgraph forgeMod [Forge_mod] + Net[NetworkHandler] + Inj[InputInjector] + end + subgraph pythonPkg [Python_mineagent] + Env[MinecraftEnv] + Client[AsyncMinecraftClient] + Ag[AgentV1] + end + Net -->|"observation_UDS"| Client + Client --> Env + Ag -->|"action_UDS"| Net + Env --> Ag +``` + +## Deeper skills (read as needed) + +Skills live under **`.agents/skills/`** in this repo. Other tools (Cursor, Opencode, etc.) may need you to **register or symlink** that path if they only auto-discover a default skills directory—point them at these folders or copy the `SKILL.md` trees they expect. + +| Skill | When to open it | +|-------|-----------------| +| `mineagent-python` | Changing agent, perception, learning, engine, or Gymnasium env | +| `mineagent-forge-mod` | Changing the Java mod, Gradle, MC/Forge versions | +| `mineagent-ipc` | Changing sockets, wire formats, or Python/Java protocol parity | +| `mineagent-dev-workflow` | Pixi, pytest, pre-commit, CI | + +## External references (human) + +- Upstream-style homepage / issues URLs are in `pyproject.toml` `[project.urls]` (GitHub project name may differ from local folder name). diff --git a/.agents/skills/mineagent-python/SKILL.md b/.agents/skills/mineagent-python/SKILL.md new file mode 100644 index 0000000..6f336e5 --- /dev/null +++ b/.agents/skills/mineagent-python/SKILL.md @@ -0,0 +1,51 @@ +--- +name: mineagent-python +description: >- + Python package layout for MineAgent: Gymnasium MinecraftEnv, AgentV1 + (perception, affector, PPO, ICM, ROI), engine loop, and YAML config. Use when + editing mineagent/, RL training logic, observation tensors, or run + configuration. +--- + +# MineAgent Python package + +## Layout (`mineagent/`) + +| Area | Path | Notes | +|------|------|--------| +| Run loop | `mineagent/engine.py` | Builds `Config`, `MinecraftEnv`, `AgentV1`; publishes monitoring events | +| Env | `mineagent/env.py` | `MinecraftEnv` (`gymnasium.Env`), sync wrapper over `AsyncMinecraftClient` | +| Config | `mineagent/config.py` | Dataclasses + YAML via dacite; CLI `-f` / `-kvp` | +| Agent | `mineagent/agent/agent.py` | `AgentV1`: vision → affector → critic; PPO + ICM updates | +| Perception | `mineagent/perception/visual.py` | Foveated + peripheral CNN branches, attention combiner | +| Affector | `mineagent/affector/affector.py` | `LinearAffector` → `AffectorOutput` (env action distributions + focus head) | +| Critic | `mineagent/reasoning/critic.py` | `LinearCritic` | +| Dynamics | `mineagent/reasoning/dynamics.py` | `InverseDynamics`, `ForwardDynamics` (ICM) | +| Learning | `mineagent/learning/ppo.py`, `icm.py`, `td.py` | PPO (Spinning Up–style), ICM; TD helper exists | +| Memory | `mineagent/memory/trajectory.py` | `TrajectoryBuffer` (fixed maxlen) | +| Client | `mineagent/client/` | Async UDS client, `protocol` (action space, `RawInput`) | +| Monitoring | `mineagent/monitoring/` | Event bus, TensorBoard callbacks | +| Utils | `mineagent/utils.py` | `sample_action`, `joint_logp_action`, hooks, tensorboard setup | + +## Engine loop (mental model) + +1. `get_config()` loads YAML and/or CLI overrides. +2. `MinecraftEnv` connects on reset; observations are `uint8` HWC RGB. +3. Each step: `agent.act(obs_tensor)` returns a **dict** action (keys, mouse deltas, buttons, scroll) for `env.step`. +4. Monitoring: `event_bus` + optional TensorBoard writers. + +## AgentV1 (important semantics) + +- **Visual features**: `VisualPerception` takes full frame + ROI crop; ROI comes from previous **focus** output (or center crop initially). +- **Two action streams**: `sample_action` returns (1) **environment** tensor used by env + ICM + PPO env loss, and (2) **focus** (2D ROI) with its own log-probs. PPO applies a separate REINFORCE-style term for focus; ICM inverse dynamics **excludes** focus (see `mineagent/learning/icm.py` comments). +- **Updates**: When `TrajectoryBuffer` length hits `agent.max_buffer_size`, `ppo.update` and `icm.update` run. + +## Configuration + +- Full schema: dataclasses in `mineagent/config.py` (`EngineConfig`, `AgentConfig`, `MonitoringConfig`, nested PPO/ICM/TD). +- Template: `config_templates/config.yaml`. +- Run: `mineagent -f path/to.yaml` and/or `mineagent -kvp engine.max_steps=100 agent.ppo.actor_lr=1e-4` (nested keys with `.`). + +## Tests + +Mirror structure under `tests/`; run via dev workflow skill (`pixi run -e dev pytest ./tests`).