feat: add dev-mode hot reload, dev-stack orchestrator, and contributo…

.gitignore

@@ -148,3 +148,6 @@ __pycache__/
 .web
 *.py[cod]
 assets/external/
+
+# Local dev stack logs (scripts/dev.py, PYGWALKER_LOG_FILE)
+logs/

AGENTS.md

@@ -0,0 +1,215 @@
+# AGENTS.md — PyGWalker contributor & agent guide
+
+This is the fast-start map of the PyGWalker repo for both human contributors and coding
+agents. It explains how the project is put together, how to run it in **dev mode with live
+frontend reload**, and where all the logs go. Read this first — it is written to save you
+from re-deriving the architecture by grepping.
+
+> Deeper references: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) (how it is built),
+> [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) (dev workflow + troubleshooting),
+> [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) (validation & CI).
+
+---
+
+## 1. What PyGWalker is (30-second model)
+
+PyGWalker turns a pandas / polars / pyarrow dataframe into an interactive
+[Graphic Walker](https://github.com/Kanaries/graphic-walker) UI inside notebooks, Streamlit,
+and plain web servers. It has **two halves that ship together**:
+
+- **Python package** (`pygwalker/`) — public API (`walk`, `render`, `table`, `Walker`,
+  `to_html`), data parsing, and the transports that talk to the UI.
+- **Frontend app** (`app/`, React + Vite) — the UI. It is compiled into JavaScript bundles
+  that are checked into the wheel under `pygwalker/templates/dist/` and loaded by the Python
+  side at render time.
+
+The Python side never renders charts itself; it hands built JS + serialized data to a
+notebook/browser and then answers data/spec requests over a message channel.
+
+---
+
+## 2. Repo map
+
+| Path | What lives here |
+|------|-----------------|
+| `pygwalker/api/` | Public entry points. `adapter.py` picks jupyter vs webserver; `jupyter.py` = notebook dispatch; `walker.py` = the reusable `Walker`; `pygwalker.py` = the core `PygWalker`. |
+| `pygwalker/services/` | Rendering + display. `anywidget_widget.py` (default transport), `render.py` + `templates/*.html` (iframe transport), `global_var.py` (runtime globals), `jupyter_display.py`. |
+| `pygwalker/communications/` | Kernel⇄frontend transports: `anywidget_comm.py` (default), `hacker_comm.py` (iframe), `streamlit_comm.py`, `gradio_comm.py`, `reflex_comm.py`. `protocol.py` is the shared message schema. |
+| `pygwalker/data_parsers/` | Dataframe/connector adapters (pandas, polars, pyarrow, SQL, spark…). |
+| `pygwalker/templates/dist/` | **Build output** (git-ignored). The JS bundles the Python side loads. |
+| `pygwalker/utils/` | Helpers: `frontend_assets.py` (locate/load bundles), `log.py` (logging), encoders. |
+| `app/src/` | Frontend source. `index.tsx` = entry; `utils/communication.tsx` = transports; `dataSource/` = data ingest; `interfaces/comm.generated.ts` = **generated** protocol types; `store/` = MobX state. |
+| `scripts/` | `dev.py` (dev orchestrator), `compile.sh` (build frontend), `local_ci.py` (mirror CI), `generate_comm_protocol_ts.py` (regenerate protocol types). |
+| `tests/` | Python tests + `*.ipynb` notebooks run by `nbmake`. `app/tests/` holds Playwright smoke tests. |
+
+---
+
+## 3. How the two halves fit together (build & load model)
+
+```
+app/src/*  --(vite build)-->  pygwalker/templates/dist/*.js  --(read at runtime)-->  Python renders it
+```
+
+**Frontend build variants** (`app/vite.config.ts`, output to `pygwalker/templates/dist/`):
+
+| Bundle | Built from | Loaded by |
+|--------|-----------|-----------|
+| `pygwalker-app.es.js` | `src/index.tsx` | **anywidget** transport (the default `pyg.walk` path) |
+| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / static `to_html()` |
+| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |
+| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |
+
+`yarn build` builds all four (+ typecheck). `yarn build:app` builds only the two app
+bundles (fast, no typecheck) — good for a quick manual rebuild, not for CI.
+
+**The message protocol is generated, not hand-written.** Python Pydantic models in
+`pygwalker/communications/protocol.py` are the source of truth. Running
+`python scripts/generate_comm_protocol_ts.py` regenerates
+`app/src/interfaces/comm.generated.ts`. **If you change `protocol.py`, regenerate and rebuild
+the frontend.** Never edit `comm.generated.ts` by hand.
+
+**Transports.** The default notebook transport is **anywidget** (`env='JupyterAnywidget'`).
+`env='Jupyter'` / `env='JupyterWidget'` are deprecated aliases that are coerced to anywidget
+and slated for removal in 0.7.0. Streamlit/Gradio/Reflex/web-server have their own transports.
+
+---
+
+## 4. First-time setup
+
+Requires **Python 3.10+**, **Node.js 22.x**, **Yarn 1.x**.
+
+```bash
+# Python (editable install with dev extras)
+python -m venv venv && source venv/bin/activate   # Windows: venv\Scripts\activate
+pip install -e ".[dev]"
+
+# Frontend deps + one full build so pygwalker/templates/dist/ is populated
+cd app && yarn install && yarn build && cd ..
+```
+
+---
+
+## 5. Dev mode: edit the frontend and see it live (anywidget HMR)
+
+The default `pyg.walk(df)` uses the anywidget transport, which loads
+`pygwalker-app.es.js` from disk. In dev mode we (a) rebuild that bundle on every source
+change and (b) let anywidget hot-reload it into open widgets. **One command starts
+everything and captures all logs:**
+
+```bash
+source venv/bin/activate
+python scripts/dev.py
+```
+
+This launches, and tees the output of, two long-running processes:
+
+- **frontend** — `cd app && yarn dev:build` (`vite build --watch`) rebuilds the bundles into
+  `pygwalker/templates/dist/` on every edit under `app/src/`. → `logs/frontend.log`
+- **jupyter** — `jupyter lab` started with `PYGWALKER_DEV=1` and `ANYWIDGET_HMR=1` in its
+  environment (inherited by kernels). → `logs/jupyter.log`
+
+Then, in a notebook cell, **no special setup is needed** — just use PyGWalker normally:
+
+```python
+import pandas as pd, pygwalker as pyg
+pyg.walk(pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]}))
+```
+
+Edit a file under `app/src/`, wait for the rebuild to finish in `logs/frontend.log`
+(`built in …`), and the widget hot-reloads in place — usually without re-running the cell.
+For a heavy change you can always re-run the cell.
+
+**Why it works:** with `ANYWIDGET_HMR=1` (or `PYGWALKER_DEV=1`), `WalkerAnyWidget._esm` is set
+to a `pathlib.Path` pointing at the built `pygwalker-app.es.js` instead of an embedded string.
+anywidget then reads that file and watches it (via `watchfiles`), pushing new code to the
+frontend when `vite build --watch` rewrites it. With the flags **off** (normal installs), the
+bundle is embedded as a string exactly as before — production behavior is unchanged.
+
+Useful flags: `--no-jupyter` (only rebuild the frontend), `--no-frontend` (only Jupyter),
+`--jupyter-port N`, `--no-browser` (auto-enabled when output is not a TTY, e.g. agent runs),
+`--log-dir DIR`. Stop everything with **Ctrl+C** (services are torn down cleanly).
+
+> Alternative (Vite dev server + browser refresh, using the legacy iframe transport) is
+> documented in [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#appendix-vite-dev-server-iframe-transport).
+> Prefer the anywidget HMR path above.
+
+---
+
+## 6. Logs: one place to look
+
+`scripts/dev.py` writes everything under `logs/` (git-ignored) at the repo root:
+
+| File | Contents |
+|------|----------|
+| `logs/frontend.log` | Vite build/watch output — watch for `built in …` (rebuild done) and TypeScript errors. |
+| `logs/jupyter.log` | JupyterLab server output — **the URL + token to open the notebook is here**. |
+| `logs/pygwalker.log` | Kernel-side PyGWalker Python logs (set via `PYGWALKER_LOG_FILE`). |
+
+**Python log controls** (honored by `pygwalker/utils/log.py`, independent of the orchestrator):
+
+- `PYGWALKER_LOG_FILE=/path/to/file.log` — also append Python logs to a file.
+- `PYGWALKER_LOG_LEVEL=DEBUG` — raise/lower verbosity (default `INFO`).
+
+**Frontend runtime logs** (what happens in the browser, e.g. comm errors) appear in the
+**browser devtools console**, not in `logs/`. The frontend surfaces user-facing errors as
+in-app toast notifications rather than `console.log`.
+
+Agent tip: tail `logs/jupyter.log` for the server URL, `logs/frontend.log` to know when a
+rebuild finished, and `logs/pygwalker.log` for kernel-side errors.
+
+---
+
+## 7. Everyday commands
+
+```bash
+# Frontend (run from app/)
+yarn build            # full build: typecheck + all 4 bundles (CI-equivalent)
+yarn build:app        # fast: only the two app bundles, no typecheck
+yarn dev:build        # rebuild-on-change (what scripts/dev.py runs)
+yarn typecheck        # tsc --noEmit
+yarn test:front_end   # Playwright smoke test (run `yarn playwright install chromium` once)
+
+# Python (run from repo root, venv active)
+python -m ruff check pygwalker tests scripts bin pygwalker_tools
+python -m ruff format --check pygwalker tests scripts bin pygwalker_tools
+python -X faulthandler -W error::DeprecationWarning:pygwalker -m pytest -o faulthandler_timeout=60 tests
+python -m pytest --nbmake --nbmake-kernel=python tests/*.ipynb   # notebook tests
+
+# Regenerate the JS protocol types after editing pygwalker/communications/protocol.py
+python scripts/generate_comm_protocol_ts.py
+
+# Run the whole CI flow locally (frontend build + smoke test + notebooks + python)
+python scripts/local_ci.py            # add --skip-frontend / --skip-notebooks to narrow
+```
+
+---
+
+## 8. Rules & gotchas
+
+- **Do not commit `pygwalker/templates/dist/`** — it is generated and git-ignored. The wheel
+  build (and CI) rebuilds it via the Hatch jupyter-builder hook.
+- **Regenerate + rebuild after protocol changes.** Editing `communications/protocol.py`
+  without running `scripts/generate_comm_protocol_ts.py` and rebuilding the frontend leaves
+  Python and JS out of sync.
+- **`yarn build:app` skips typecheck** and the DSL bundles. Run full `yarn build` (or
+  `yarn typecheck`) before pushing frontend changes.
+- **anywidget is the transport of record.** Don't reach for the deprecated `env='Jupyter'` /
+  iframe path for new work; it is removed in 0.7.0.
+- **Dev-mode flags are opt-in.** `PYGWALKER_DEV` / `ANYWIDGET_HMR` only affect a dev session.
+  Never rely on them being set at runtime for end users.
+- **First run needs a build.** In dev mode the widget loads `dist/pygwalker-app.es.js` from
+  disk; if it is missing you'll get a clear "Missing PyGWalker frontend asset" error — run the
+  frontend build (or wait for `scripts/dev.py`'s first build to finish).
+
+---
+
+## 9. Where to look for X
+
+| Question | Start here |
+|----------|-----------|
+| How does `pyg.walk()` decide what to display? | `pygwalker/api/adapter.py` → `api/jupyter.py` (`env_display_map`) |
+| How is the frontend bundle loaded / dev-swapped? | `pygwalker/services/anywidget_widget.py`, `utils/frontend_assets.py` |
+| What messages can the frontend send the kernel? | `pygwalker/communications/protocol.py` ⇄ `app/src/interfaces/comm.generated.ts` |
+| How is data sent to the browser? | `pygwalker/services/data_communication.py`, `app/src/dataSource/` |
+| How is a chart exported to PNG/SVG/code? | `pygwalker/services/chart_export.py`, `app/src/tools/` |
+| How does the iframe/static-HTML path render? | `pygwalker/services/render.py` + `pygwalker/templates/*.html` |

CLAUDE.md

@@ -0,0 +1,24 @@
+# CLAUDE.md
+
+This project's contributor & agent guide is maintained in **[AGENTS.md](AGENTS.md)** — a
+single source of truth for the architecture, dev-mode workflow, and log locations. Read it
+before making changes.
+
+@AGENTS.md
+
+## TL;DR for a coding agent
+
+- **Setup:** `python -m venv venv && source venv/bin/activate && pip install -e ".[dev]"`,
+  then `cd app && yarn install && yarn build`.
+- **Run everything in dev mode with live reload + centralized logs:** `python scripts/dev.py`
+  (starts the frontend watch build + JupyterLab with `PYGWALKER_DEV=1` / `ANYWIDGET_HMR=1`).
+  Pass `--no-browser` for headless runs.
+- **Logs:** `logs/frontend.log` (build/watch), `logs/jupyter.log` (server URL + token),
+  `logs/pygwalker.log` (kernel-side Python logs). Frontend runtime logs are in the browser
+  console.
+- **In a notebook:** just `import pygwalker as pyg; pyg.walk(df)` — no special setup; edits
+  under `app/src/` hot-reload into the widget.
+- **Before pushing:** `python scripts/local_ci.py` (or, narrower, `yarn typecheck` + `yarn build`
+  in `app/`, and `ruff check`/`ruff format --check`/`pytest` from the root).
+- **Don't:** commit `pygwalker/templates/dist/`, hand-edit `app/src/interfaces/comm.generated.ts`,
+  or rely on `PYGWALKER_DEV`/`ANYWIDGET_HMR` at runtime for end users.

CONTRIBUTING.md

@@ -1,13 +1,17 @@
 # Contributing to PyGWalker
 
-The contributor workflow lives in [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md).
+New contributors (and coding agents) should start with [`AGENTS.md`](AGENTS.md): it maps the
+repo, explains how the Python and frontend halves are built, and documents the one-command dev
+stack (`python scripts/dev.py`) with live frontend reload and centralized logs.
 
-Start there for the supported local setup, including:
+The detailed contributor workflow lives in [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md),
+covering:
 
 - Python editable installs and test commands.
 - Frontend dependency installation, builds, type checks, and Playwright smoke tests.
 - Optional local Graphic Walker source builds through `app`'s `dev:preinstall` script.
-- Vite dev-server setup for JupyterLab hot reload.
+- The hot-reload dev workflow ([`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md)) and project
+  architecture ([`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)).
 - CI and package-build expectations.
 
 Keeping the detailed guide under `docs/` lets the development notes and

README.md

@@ -291,7 +291,14 @@ renderer.explorer()
 
 ## Development
 
-Refer it: [local-development](https://docs.kanaries.net/pygwalker/installation#local-development)
+Local development is documented in this repository:
+
+- [AGENTS.md](AGENTS.md) — repo map, the one-command dev stack (`python scripts/dev.py`), and log locations (written for both human and AI-agent contributors).
+- [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) — hot-reload dev workflow and troubleshooting.
+- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — how the Python and frontend halves are built and talk to each other.
+- [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md) — validation commands, CI, and package builds.
+
+See also the hosted guide: [local-development](https://docs.kanaries.net/pygwalker/installation#local-development).
 
 ## Tested Environments
 

app/package.json

@@ -7,6 +7,7 @@
   "scripts": {
     "build": "yarn typecheck && vite build && vite build --mode=dsl_to_workflow && vite build --mode=vega_to_dsl",
     "build:app": "vite build",
+    "dev:build": "vite build --watch --mode production",
     "typecheck": "tsc --noEmit --pretty false",
     "dev:preinstall": "(cd ../graphic-walker/packages/graphic-walker; yarn --frozen-lockfile && yarn build)",
     "dev:server": "vite --host",