Use template to generate DHS Madagascar Environment

notebooks/projects/prospectorDHSMadagascar/00_misc.qmd

@@ -1,17 +1,163 @@
 ---
-title: "Legacy: Miscellaneous Functions"
+title: "Miscellaneous Helper Functions"
 format: gfm
 execute:
   eval: false
+filters:
+    - sorting-hat
+    - ripper
+extensions:
+  ripper:
+    output-name: "helpers"
+    output-dir: "../../src/rse_workbench"
+  sorting-hat:
+        keep: python
 ---
 
-This numbered notebook has been superseded by the reusable project workflow.
+This notebook collects miscellaneous functions that are useful for the RSE 
+Workbench. 
 
-Create a current notebook with:
+Here are a couple of functions that allow us to quickly
+build our singularity image and iterate on it by executing commands inside the 
+image. These functions are:
+
+```{python}
+#| sorting-hat: keep
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from rich.columns import Columns
+from rich.console import Console
+from rich.panel import Panel
+from rich.syntax import Syntax
+
+console = Console()
+
+
+def spython_exec(
+    command: str,
+    image: str | Path,
+    options: list[str] | None = None,
+    writable: bool = False,
+    fakeroot: bool = False,
+    bind_project: str | Path | None = None,
+    return_result: bool = True,
+) -> dict[str, Any]:
+    """Execute a command in a Singularity image through spython."""
+
+    from spython.main import Client
+
+    exec_options = list(options or ["--no-home"])
+
+    if writable and "--writable" not in exec_options:
+        exec_options.append("--writable")
+
+    if fakeroot and "--fakeroot" not in exec_options:
+        exec_options.append("--fakeroot")
+
+    if bind_project:
+        exec_options += ["--bind", f"{Path(bind_project)}:/work"]
+
+    return Client.execute(
+        image=str(image),
+        command=["bash", "--noprofile", "--norc", "-c", command],
+        options=exec_options,
+        sudo=False,
+        return_result=return_result,
+    )
+
+
+def normalize_spython_result(result: Any) -> dict[str, Any]:
+    """Normalize spython's mixed result shapes into stdout/stderr/code."""
+
+    if not isinstance(result, dict):
+        return {"stdout": str(result), "stderr": "", "return_code": None, "raw": result}
+
+    message = result.get("message", "")
+    return_code = result.get("return_code", None)
+
+    if isinstance(message, list):
+        stdout = message[0] if len(message) > 0 else ""
+        stderr = message[1] if len(message) > 1 else ""
+    else:
+        stdout = message or ""
+        stderr = ""
+
+    return {
+        "stdout": stdout,
+        "stderr": stderr,
+        "return_code": return_code,
+        "raw": result,
+    }
+
+
+def show_spython(result: Any, *, tail: int | None = None) -> dict[str, Any]:
+    """Render a spython result in a notebook-friendly panel."""
+
+    normalized = normalize_spython_result(result)
+    stdout = normalized["stdout"]
+    stderr = normalized["stderr"]
+
+    if tail:
+        stdout = "\n".join(stdout.splitlines()[-tail:])
+        stderr = "\n".join(stderr.splitlines()[-tail:])
+
+    console.print(
+        Panel(
+            Columns(
+                [
+                    Panel(Syntax(stdout or "<empty>", "text"), title="stdout / message", border_style="green"),
+                    Panel(
+                        Syntax(stderr or "<empty>", "text"),
+                        title="stderr",
+                        border_style="red" if normalized["return_code"] else "yellow",
+                    ),
+                ]
+            ),
+            title=f"Exit code: {normalized['return_code']}",
+            border_style="green" if normalized["return_code"] == 0 else "red",
+        )
+    )
+
+    return normalized
 
-```bash
-cp notebooks/templates/rse-workbench-project-template.qmd notebooks/projects/prospectorDHSMadagascar/environment.qmd
 ```
 
-Then edit the Project Contract chapter. Helper logic now lives in
-`src/rse_workbench/`.
+
+To see the ongoing result of a job with `submitit`, we can use the following helper function:
+
+```{python}
+#| sorting-hat: keep
+"""Slurm display and launch-script helpers."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from rich.console import Console
+from rich.panel import Panel
+from rich.text import Text
+
+console = Console()
+
+
+def show_submitit_job(job: Any) -> Any:
+    """Show stdout/stderr for a submitit job and return its result when done."""
+
+    console.print(f"Job ID: {job.job_id}")
+    console.print(f"Done: {job.done()}")
+
+    stdout = job.stdout() or "<empty>"
+    stderr = job.stderr() or "<empty>"
+
+    console.print(Panel(Text(stdout[-8000:] or "<empty>"), title="stdout", border_style="green"))
+    console.print(Panel(Text(stderr[-8000:] or "<empty>"), title="stderr", border_style="red"))
+
+    if not job.done():
+        return None
+
+    return job.result()
+```

notebooks/projects/prospectorDHSMadagascar/01_project-contract.qmd

@@ -0,0 +1,98 @@
+---
+title: "Project Contract"
+format: gfm
+filters:
+    - sorting-hat
+    - ripper
+extensions:
+  ripper:
+    output-name: "project_contract"
+    output-dir: "../../src/rse_workbench"
+  sorting-hat:
+        keep: python
+execute: 
+  eval: false
+---
+
+The project contract is the initial source of truth for the
+project. In Python, it's fairly easy to work with capital
+letter "constants", toss them around with pathlib,
+and use them to generate files and folders. The contract
+section makes use of this to get the project started:
+
+```{python}
+#| sorting-hat: remove
+from pathlib import Path
+from pyprojroot import here
+
+PROJECT_NAME = "prospectorDHSMadagascar" # FILL ME IN
+assert PROJECT_NAME, "Please fill in the project name."
+PROJECT_DIR = Path("/n/holylabs/LABS/cgolden_lab/Lab/frontier/works/prospectors") / PROJECT_NAME
+WORKBENCH_DIR = here()
+RUNTIME_DIR = WORKBENCH_DIR / "projects" / PROJECT_NAME
+
+BASE_IMAGE = "docker://spack/rockylinux8:develop"
+
+R_VERSION = "4.5.1"
+PYTHON_VERSION = "3.14"
+
+ENABLE_DIRENV = False
+```
+
+We can dedicate a section to the Slurm variables for the typical
+runtime while we're building the project:
+
+```{python}
+#| sorting-hat: remove
+SLURM_PARTITION = "hsph"
+SLURM_CPUS = 12
+SLURM_MEM = "32G"
+SLURM_HOURS = 2 # this is the project build time, not the runtime for a single job
+SLURM_EMAIL = "aaaapr3lk7gfdswioulcmvi3te@harvard.org.slack.com" # aaaapr3lk7gfdswioulcmvi3te@harvard.org.slack.com
+```
+
+Here are a few more useful paths. These names are intentionally 
+explicit because they are the boundary between this 
+notebook, the project tree, Slurm, and the container.
+
+```{python}
+#| sorting-hat: remove
+SINGULARITY_DEF = PROJECT_DIR / ".devcontainer" / "Singularity.def"
+SINGULARITY_SIF = PROJECT_DIR / ".devcontainer" / "Singularity.sif"
+SPACK_YAML_PATH = PROJECT_DIR / "spack.yaml"
+RPROJECT_TOML_PATH = PROJECT_DIR / "rproject.toml"
+PYPROJECT_TOML_PATH = PROJECT_DIR / "pyproject.toml"
+CONTAINER_HOME = PROJECT_DIR / ".container-home"
+CONTAINER_TMP = PROJECT_DIR / ".container-tmp"
+SUBMITIT_DIR = PROJECT_DIR / "slurm" / "submitit"
+```
+
+Lastly, we can create a `ProjectContract` `dataclass` to help us
+store all of these values in one place:
+
+```{python}
+#| sorting-hat: remove
+
+from rse_workbench.project_contract import ProjectContract
+
+project_contract = ProjectContract(
+    project_name=PROJECT_NAME,
+    project_dir=PROJECT_DIR,
+    workbench_dir=WORKBENCH_DIR,
+    runtime_dir=RUNTIME_DIR,
+    base_image=BASE_IMAGE,
+    r_version=R_VERSION,
+    python_version=PYTHON_VERSION,
+    slurm_partition=SLURM_PARTITION,
+    slurm_cpus=SLURM_CPUS,
+    slurm_mem=SLURM_MEM,
+    slurm_hours=SLURM_HOURS,
+    enable_direnv=ENABLE_DIRENV,
+)
+project_contract.ensure_directories()
+```
+
+By the end of this chapter, you should be able to answer the question,
+"what is this project, and where does it live?"
+
+In the next chapter, you'll define the platform recipe.