Mahf708/package

.github/workflows/ci.yml

@@ -0,0 +1,64 @@
+name: aigci
+
+on:
+  push:
+    branches: ["main"]
+    paths:
+      - .github/workflows/ci.yml
+      - pyproject.toml
+      - src/**
+      - tests/**
+  pull_request:
+    branches: ["main"]
+    paths:
+      - .github/workflows/ci.yml
+      - pyproject.toml
+      - src/**
+      - tests/**
+  workflow_dispatch:
+
+concurrency:
+  group: aigci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.13"]
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          persist-credentials: false
+          show-progress: false
+      - uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+      # uv provides the interpreter, so the matrix version is the one actually
+      # tested. Installing into a venv also avoids the runner's /usr, which is
+      # externally managed and refuses --system.
+      - name: Install
+        run: |
+          uv venv --python ${{ matrix.python-version }}
+          uv pip install -e '.[dev]'
+      - name: Show interpreter
+        run: .venv/bin/python -VV
+      - name: Lint
+        run: .venv/bin/ruff check
+      - name: Test
+        run: .venv/bin/pytest
+      # The base tier must stay installable with no scientific stack at all.
+      # If this step starts needing numpy, the core/adapter boundary has leaked.
+      - name: Assert the base install is light
+        run: |
+          .venv/bin/python - <<'PY'
+          import sys
+          for heavy in ("numpy", "xarray", "netCDF4", "torch", "matplotlib", "pandas"):
+              assert heavy not in sys.modules, f"{heavy} imported at rest"
+          import xaig, xaig.caig  # noqa: F401
+          for heavy in ("numpy", "xarray", "netCDF4", "torch", "matplotlib", "pandas"):
+              assert heavy not in sys.modules, f"importing xaig pulled in {heavy}"
+          print("base tier is light")
+          PY

.gitignore

@@ -3,3 +3,11 @@ site
 
 # any cache
 .cache
+
+# python
+.venv/
+__pycache__/
+*.py[cod]
+*.egg-info/
+.pytest_cache/
+.ruff_cache/

AGENTS.md

@@ -0,0 +1,27 @@
+# aigroup
+
+Two peers live here. Neither exists to serve the other.
+
+- `docs/` — the MkDocs guide site, published to gh-pages
+- `src/xaig/` — the `xaig` Python package
+
+## Rules
+
+- Work on `user/topic` branches; merge to `main` via PR. Short lowercase imperative
+  commit subjects.
+- `mkdocs build --strict` must pass. New pages must be added to `nav` in `mkdocs.yml`
+  by hand.
+- `ruff check` and `pytest` must pass. Both run in `.github/workflows/ci.yml`.
+- `uv` is the tool of record. ACE itself pins Python 3.11.
+- Ship in ~1000-line increments. Each increment leaves the repo working and useful.
+
+## Where to look
+
+| Path | AGENTS.md covers |
+|---|---|
+| `src/xaig/` | package architecture, the core/adapter boundary, dependency tiers |
+| `src/xaig/core/` | the purity contract |
+| `src/xaig/adapters/` | writing a new adapter |
+| `src/xaig/{caig,daig,taig}/` | each subpackage's scope and non-goals |
+| `tests/` | fixture rules |
+| `docs/` | prose and nav conventions |

README.md

@@ -0,0 +1,38 @@
+# aigroup
+
+Docs, scripts, examples, and prototypes for E3SM AI efforts.
+
+This repo hosts two peers:
+
+- **`docs/`** — the guide site, published at <https://e3sm-project.github.io/aigroup>
+- **`src/xaig/`** — `xaig`, a light Python package for campaign tracking (`caig`),
+  emulator diagnostics (`daig`), and reusable neural blocks (`taig`)
+
+## Install
+
+```console
+$ uv venv --python 3.11 .venv
+$ uv pip install -e '.[dev]'
+```
+
+The base install pulls only PyYAML and Click. Framework-specific readers live behind
+extras (`fme`, `viz`, `toys`) so the core stays nimble.
+
+## Use
+
+```console
+$ xaig caig specs
+$ xaig caig ls --spec aug26 --source /path/to/runs/MANIFEST.tsv
+$ xaig caig check --spec aug26 --source /path/to/runs/MANIFEST.tsv
+```
+
+## Develop
+
+```console
+$ uv run ruff check
+$ uv run pytest
+$ uv run mkdocs build --strict
+```
+
+See `AGENTS.md` for how the pieces fit together, and the `AGENTS.md` in each
+subdirectory for that directory's rules.

docs/AGENTS.md

@@ -0,0 +1,20 @@
+# docs
+
+MkDocs Material, published to gh-pages. `mkdocs build --strict` must pass — broken
+links fail CI.
+
+## Conventions
+
+- Shell blocks are fenced as `console`, with `$` prompts.
+- Annotated YAML uses ` ```{ .yaml .annotate } ` with `(1)!` markers and a matching
+  ordered list beneath.
+- Long configs and scripts are collapsed in `??? example "title"`.
+- Admonitions use lowercase titles: `!!! tip "uv cache"`.
+- Guides end with a `## Remaining tasks` unchecked list.
+- Cross-links are relative (`python-envs.md`).
+- Quote real measured numbers and real NERSC paths rather than genericizing them.
+
+## Adding a page
+
+Add the file, then add it to `nav` in `mkdocs.yml` by hand — nav is explicit, not
+inferred. Package API pages are generated by `mkdocstrings` from docstrings.

docs/package/index.md

@@ -0,0 +1,154 @@
+# The `xaig` package
+
+`xaig` is a light, framework-agnostic package for working with AI campaigns. It lives in
+this repo alongside the guides, as a peer rather than an appendix.
+
+| Subpackage | Scope |
+| --- | --- |
+| `caig` | campaign tracking — offline, no server, no live streaming |
+| `daig` | emulator-vs-reference diagnostics *(skeleton)* |
+| `taig` | reusable neural blocks *(skeleton)* |
+
+!!! warning "research tool"
+
+    `xaig` is early. The `caig` surface described here works; `daig` and `taig` are
+    skeletons with their intended APIs sketched in their module docstrings.
+
+## Install
+
+```console
+$ uv venv --python 3.11 .venv
+$ uv pip install -e '.[dev]'
+```
+
+The base install pulls only PyYAML and Click. Anything that knows about a specific
+training framework lives behind an extra.
+
+| Extra | Pulls | Gets you |
+| --- | --- | --- |
+| `fme` | numpy, xarray, netCDF4 | the ACE/FME adapter |
+| `viz` | matplotlib | plots and reports |
+| `toys` | torch | `taig` |
+
+!!! tip "uv cache"
+
+    On NERSC, keep the cache off `$HOME`:
+
+    ```console
+    $ export UV_CACHE_DIR="$PSCRATCH/.cache/uv"
+    ```
+
+## Why it is built this way
+
+The group expects to move to systems profoundly unlike ACE/FME/Samudra. So `xaig.core`
+knows nothing about any framework: it holds a data model, four protocols, and generic
+algorithms, and depends only on the standard library and PyYAML. Everything that knows
+about a real file layout, log format or scheduler lives in `xaig.adapters`.
+
+Supporting a new system means writing a new adapter, never editing core. Adapters are
+found through the `xaig.adapters` entry-point group, so one can ship from a completely
+separate package.
+
+A *campaign spec* is a YAML file describing one campaign's conventions — its run-id
+grammar, its factors, its parent map, its metric. Adding a campaign does not mean writing
+Python.
+
+## Tracking a campaign
+
+Specs bundled with the package:
+
+```console
+$ xaig caig specs
+aug26  35 runs (25 atmosphere, 10 ocean) ablating aerosol, CO2, batch size, ...
+```
+
+List the runs of the `e3sm_hist_v20260812` campaign, reading its manifest:
+
+```console
+$ xaig caig ls --spec aug26 \
+    --source /pscratch/sd/m/mahf708/ace/configs/experiments/e3sm_hist_v20260812/runs/MANIFEST.tsv
+ID                                       STATUS   EXP  REALM  SEED  AEROSOL  BATCH  CO2  LR  OCEAN_STEP  WEIGHTS  AMP
+E01.aug26.atm.A0_B16_C0_L0_O5_W0_X0.S01  unknown  E01  atm    1     0        16     0    0   5           0        0
+E02.aug26.atm.A0_B16_C1_L0_O5_W0_X0.S01  unknown  E02  atm    1     0        16     1    0   5           0        0
+...
+
+35 run(s)
+```
+
+The factor word in each run id is decomposed into named attributes, so you can filter on
+them:
+
+```console
+$ xaig caig ls --spec aug26 --source .../MANIFEST.tsv --select realm=ocn -c exp,seed,ocean_step
+```
+
+`STATUS` reads `unknown` above because status probing arrives with the FME adapter — see
+[remaining tasks](#remaining-tasks).
+
+Show one run:
+
+```console
+$ xaig caig show E05.aug26.atm.A3_B16_C1_L0_O5_W0_X0.S01 --spec aug26 --source .../MANIFEST.tsv
+```
+
+## Checking a campaign
+
+A run id that disagrees with the config it names is the worst failure a campaign can
+have, because every table and plot is labelled by the id. `check` round-trips every id
+through the spec:
+
+```console
+$ xaig caig check --spec aug26 --source .../MANIFEST.tsv
+OK  35 run id(s) round-trip through spec 'aug26'
+```
+
+## Python API
+
+The API is a peer of the CLI, not a layer beneath it. Both sit on `xaig.core`.
+
+```python
+from xaig.caig import load_campaign
+from xaig.core import spec as spec_module
+
+spec = spec_module.load("aug26")
+campaign = load_campaign(spec, source=".../runs/MANIFEST.tsv")
+
+ocean = campaign.filter(realm="ocn")
+for run in ocean.sorted_by("exp", "seed"):
+    print(run.id, run.attrs["ocean_step"])
+```
+
+## Adding a campaign
+
+Write a spec. This one describes a grammar with nothing in common with `aug26`:
+
+```yaml
+name: toy
+id_pattern: '^run(?P<num>\d+)-(?P<mode>fast|slow)-(?P<knobs>[a-z0-9-]+)-s(?P<seed>\d+)$'
+id_template: 'run{num:03d}-{mode}-{knobs}-s{seed:d}'
+factor_field: knobs
+factor_separator: "-"
+factors:
+  - { key: d, name: depth, width: 1 }
+  - { key: w, name: width, width: 2 }
+discovery:
+  adapter: table
+  id_column: name
+```
+
+```console
+$ xaig caig ls --spec ./toy.yaml --source runs.csv
+```
+
+A campaign whose run ids carry no structure simply omits `id_pattern`; its adapter
+attaches attributes instead.
+
+## Remaining tasks
+
+- [ ] FME adapter: run-directory discovery, `out.log` parsing, joblog segment chains,
+      optional Slurm probe
+- [ ] Incremental scan index, so polling a live campaign is cheap
+- [ ] `caig compare`: the pre-registered seed-spread decision rule
+- [ ] `caig doctor`: machine-checked campaign guardrails
+- [ ] `daig`: bias and time-mean maps, spectra, zonal means
+- [ ] `taig`: the first reusable blocks

mkdocs.yml

@@ -57,6 +57,10 @@ theme:
   icon:
     repo: fontawesome/brands/github
 
+# AGENTS.md files are contracts for contributors, not published pages.
+exclude_docs: |
+  AGENTS.md
+
 plugins:
   - search
   - social:
@@ -132,4 +136,6 @@ nav:
       - ACE2-EAMv3 Inference: ace2-inference.md
       - SamudrACE-E3SMv3 Coupled Inference: samudrace-inference.md
       - E3SM branch runs: e3sm-branch-runs.md
+  - Package:
+      - Overview: package/index.md
 

pyproject.toml

@@ -0,0 +1,48 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "xaig"
+version = "0.1.0"
+description = "Light, framework-agnostic tooling for E3SM AI campaigns: tracking, diagnostics, toys"
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "BSD-3-Clause" }
+authors = [{ name = "E3SM AI Group" }]
+keywords = ["e3sm", "emulator", "climate", "machine-learning"]
+
+# Base tier stays deliberately tiny: see src/xaig/core/AGENTS.md
+dependencies = ["pyyaml>=6.0", "click>=8.1"]
+
+[project.optional-dependencies]
+fme = ["numpy>=1.24", "xarray>=2023.1", "netCDF4>=1.6"]
+viz = ["matplotlib>=3.7"]
+toys = ["torch>=2.0"]
+dev = ["pytest>=7.4", "ruff>=0.5"]
+
+[project.scripts]
+xaig = "xaig._cli:main"
+
+# Adapters are discovered through this group, so a future system ships as a
+# separate distribution without editing xaig at all.
+[project.entry-points."xaig.adapters"]
+table = "xaig.adapters.table:TableDiscoverer"
+
+[project.urls]
+Homepage = "https://e3sm-project.github.io/aigroup"
+Source = "https://github.com/E3SM-Project/aigroup"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/xaig"]
+
+[tool.ruff]
+line-length = 100
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B", "SIM"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"