Use template to generate DHS Madagascar Environment#2
Merged
Conversation
- refined environment variable settings for spack integration - added additional paths for user and misc cache - ensured compatibility with previous builds by addressing malformed variables
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates the prospectorDHSMadagascar project notebooks from legacy, superseded chapters to a newer template-based workflow for defining and building a reproducible Singularity/Spack-based environment, and aligns an import path in an existing project notebook.
Changes:
- Replaces multiple legacy DHS Madagascar notebooks with new, template-style chapters (contract → platform recipe → sysdeps → datascience → activation → build/install).
- Updates the DHS Madagascar “misc helpers” notebook to include execution and submitit display helpers intended for reuse.
- Switches
prospectorMaheryproject activation to importshow_spython/spython_execfromrse_workbench.helpers.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| notebooks/projects/prospectorMahery/05_project-activation.qmd | Updates helper import path for Singularity execution display utilities. |
| notebooks/projects/prospectorDHSMadagascar/setup.qmd | Removes legacy setup notebook (superseded). |
| notebooks/projects/prospectorDHSMadagascar/06_launch.qmd | Removes legacy launch notebook (superseded). |
| notebooks/projects/prospectorDHSMadagascar/05_recipe.qmd | Removes legacy container-build notebook (superseded). |
| notebooks/projects/prospectorDHSMadagascar/04_devtools.qmd | Removes legacy devtools notebook (superseded). |
| notebooks/projects/prospectorDHSMadagascar/03_datascience_lang.qmd | Removes legacy language-layer notebook (superseded). |
| notebooks/projects/prospectorDHSMadagascar/02_spack.qmd | Removes legacy Spack-layer notebook (superseded). |
| notebooks/projects/prospectorDHSMadagascar/01_base_platform.qmd | Removes legacy base-platform notebook (superseded). |
| notebooks/projects/prospectorDHSMadagascar/00_misc.qmd | Adds helper utilities for spython execution + submitit job output display. |
| notebooks/projects/prospectorDHSMadagascar/01_project-contract.qmd | Adds project contract chapter defining paths, versions, and Slurm parameters. |
| notebooks/projects/prospectorDHSMadagascar/02_platform-recipe.qmd | Adds platform recipe chapter for base image/tooling installation in container. |
| notebooks/projects/prospectorDHSMadagascar/03_sci-sysdeps.qmd | Adds Spack system-dependency layer chapter and spack.yaml generation. |
| notebooks/projects/prospectorDHSMadagascar/04_datascience.qmd | Adds language package layer chapter for R/Python dependency TOMLs. |
| notebooks/projects/prospectorDHSMadagascar/05_project-activation.qmd | Adds activation/bridge chapter to validate environment variables and Spack activation. |
| notebooks/projects/prospectorDHSMadagascar/06_build-launch.qmd | Adds Slurm/submitit build + in-container install orchestration chapter. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+49
to
+51
| SLURM_MEM = "32G" | ||
| SLURM_HOURS = 2 # this is the project build time, not the runtime for a single job | ||
| SLURM_EMAIL = "aaaapr3lk7gfdswioulcmvi3te@harvard.org.slack.com" # aaaapr3lk7gfdswioulcmvi3te@harvard.org.slack.com |
| from rse_workbench.platform_contract import compose_platform_recipe, recipe_to_definition | ||
| ``` | ||
|
|
||
| THe platform recipe section tells us what foundation to build |
Comment on lines
+146
to
+149
| #| sorting-hat: remove | ||
| from spython.main import Client | ||
|
|
||
| TMP_SANDBOX = Path(f"/tmp/{os.environ['USER']}-{PROJECT_NAME}-sandbox") |
Comment on lines
+13
to
+15
| ```{python} | ||
| show_spython( | ||
| spython_exec( |
Comment on lines
+19
to
+21
| from rse_workbench.helpers import show_submitit_job | ||
| from rse_workbench.build_install import install_in_container | ||
|
|
Comment on lines
+133
to
+145
| """Slurm display and launch-script helpers.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| from rich.console import Console | ||
| from rich.panel import Panel | ||
| from rich.text import Text | ||
|
|
||
| console = Console() | ||
|
|
Comment on lines
+76
to
+80
| "cairo", | ||
| "readline", | ||
| "libuv", | ||
| "libgit2" | ||
| ] |
Comment on lines
+86
to
+90
| "udunits", | ||
| "libuv", | ||
| "zlib", | ||
| "r-udunits2" | ||
| ]) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.