Devel/containerization

.github/workflows/deploy.yml

@@ -30,7 +30,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install pymarkdownlnt==0.9.36
+          pip install $(cat config/requirements.txt | grep -E ^pymarkdownlnt)
       - name: Check ProperDocs YAML configuration
         uses: ibiqlik/action-yamllint@v3
         with:

.github/workflows/test-build.yml

@@ -1,37 +1,37 @@
-# documentation: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax
-#
-name: Test build NCC course material website
-on:
-  pull_request:
-  push:
-    branches-ignore:
-      - main
-      - production
-  workflow_dispatch:
-permissions:
-  contents: read
-  pages: write
-  id-token: write
-jobs:
-  check:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v6
-        with:
-          lfs: true
-      - name: Read Python version
-        id: pyversion
-        run: echo "PYTHON_VERSION=$(cat .python)" >> $GITHUB_ENV
-      - name: Set up Python
-        uses: actions/setup-python@v6
-        with:
-          python-version: '${{ env.PYTHON_VERSION }}'
-          cache: 'pip'
-      - name: Install dependencies
-        run: |
+    # documentation: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax
+    #
+    name: Test build NCC course material website
+    on:
+      pull_request:
+      push:
+        branches-ignore:
+          - main
+          - production
+      workflow_dispatch:
+    permissions:
+      contents: read
+      pages: write
+      id-token: write
+    jobs:
+      check:
+        runs-on: ubuntu-latest
+        steps:
+          - name: Checkout
+            uses: actions/checkout@v6
+            with:
+              lfs: true
+          - name: Read Python version
+            id: pyversion
+            run: echo "PYTHON_VERSION=$(cat .python)" >> $GITHUB_ENV
+          - name: Set up Python
+            uses: actions/setup-python@v6
+            with:
+              python-version: '${{ env.PYTHON_VERSION }}'
+              cache: 'pip'
+          - name: Install dependencies
+            run: |
           python -m pip install --upgrade pip
-          pip install pymarkdownlnt==0.9.36
+          pip install $(cat config/requirements.txt | grep -E ^pymarkdownlnt)
       - name: Check ProperDocs YAML configuration
         uses: ibiqlik/action-yamllint@v3
         with:

README.md

@@ -14,6 +14,22 @@ source ~/environments/ncc/bin/activate
 uv pip install --requirements requirements.txt
 ```
 
+## Deploying the page locally
+
+The project is compiled and deployed locally with `properdocs`. Start by loading the development environment.
+
+```bash
+source ~/environments/ncc/bin/activate
+```
+
+The typeset and deploy the page with `properdocs`.
+
+```bash
+properdocs serve --strict
+```
+
+The `--strict` flag causes the deployment to fail in case of errors.
+
 ### Compiling the images in resource directories
 
 There is code for LaTeX generated figures in some sections. The source code is located in directories named `resources`. The compilation of the code requires [TeX Live](https://tug.org/texlive/) and the [`convert`](https://linux.die.net/man/1/convert) tool. Compile the images manually by running the command
@@ -24,3 +40,25 @@ make
 
 inside the `resource` directories.
 
+### Linting
+
+To lint the documentation locally, install in the environment the linting packages.
+
+```console
+source ~/environments/ncc/bin/activate
+uv pip install --requirements config/requirements.txt
+```
+
+Then, lint the markdown files with the following command.
+
+```console
+pymarkdown \
+    --config config/pymarkdown.yml --strict-config scan \
+    --recurse docs
+```
+
+Lint the ProperDocs configuration with the following command.
+
+```console
+yamllint --config-file config/yamllint.yml properdocs.yml
+```

config/requirements.txt

@@ -0,0 +1,2 @@
+pymarkdownlnt==0.9.38
+yamllint==1.38.0

docs/courses/containerization/README.txt

@@ -283,3 +283,98 @@ echo 'Welcome to chroot jail!' > ${HOME}/jail/home/myusername/hello.txt
 ## Building a container
 * References [1, Shipping Your Machine: Building a Container in 50 Lines of Code](https://dev.to/yechielk/shipping-your-machine-building-a-container-in-60-lines-of-code-part-1-14ma), [2, Build Your Own Docker with Linux Namespaces, cgroups, and chroot: Hands-on Guide ](https://akashrajpurohit.com/blog/build-your-own-docker-with-linux-namespaces-cgroups-and-chroot-handson-guide/)
 * 
+
+## Running Apptainer in Grid' 5000
+
+```console
+sudo-g5k
+sudo apt install libseccomp-dev squashfuse cryptsetup
+module use /home/gkafanas/ulhpc/easybuild/release/2025a/modules/all
+sudo mount -o remount,suid /srv/storage/ulhpc@storage1.luxembourg.grid5000.fr
+```
+
+## References
+
+- https://snarky.ca/how-virtual-environments-work/
+
+- Source of DIA-NN examples: https://mbite.mdhs.unimelb.edu.au/intro-to-proteomics/02diann.html
+
+- https://en.wikipedia.org/wiki/Linux_namespaces
+
+- https://www.redhat.com/en/blog/7-linux-namespaces
+- https://www.redhat.com/en/blog/pid-namespace
+- https://www.redhat.com/en/blog/linux-pid-namespaces
+
+- https://www.hackerstack.org/understanding-linux-namespaces/
+
+- https://www.toptal.com/developers/linux/separation-anxiety-isolating-your-system-with-linux-namespaces
+
+## Virtual environments
+
+### The `venv` in Python
+
+### The `renv` in R
+
+## Containers
+
+An [introduction to Docker](https://carpentries-incubator.github.io/docker-introduction/advanced-containers.html) is available in a carpentry course.
+
+### Singularity type containers
+
+### Using Singularity containers
+
+#### Existing files and artifacts
+
+#### Fetching from SHPC registries
+
+- https://docs.alcf.anl.gov/crux/containers/containers/
+- https://github.com/argonne-lcf/container-registry/tree/main/containers/shpc
+
+### Building Singularity containers
+
+#### Builds that do not require elevated privileges
+
+#### Builds requiring `fakeroot` or id mapping
+
+- https://apptainer.org/docs/admin/latest/user_namespace.html
+- https://apptainer.org/docs/admin/latest/installation.html#fakeroot-with-uid-gid-mapping-on-network-filesystems
+- https://apptainer.org/docs/admin/latest/configfiles.html
+
+## Containerization
+
+### Creating a `chroot` environment
+
+### Modern containers and `namegroups`
+
+### Creating a containerized environment from first principle
+
+## Advanced containerization concepts
+
+### Connecting with system libraries
+
+#### MPI network libraries
+
+#### Offloading libraries and drivers
+
+### Reproducibility
+
+### Building a reproducible containers
+
+### Balancing reproducibility and usage of efficient system libraries
+
+For instance, imagine that there is an MPI library that ensures binary reproducibility. Your container could ensure binary reproducibility if it linked with this external MPI library, and still ensure native level MPI performance in every system that supports this MPI library.
+
+- Introductory example
+   - chroot base example
+   - the linux hier & file system exporters
+- containers in HPC
+   - basic usage
+   - building contaiers
+      - simple
+      - requiring root
+- modern containers
+   - namespaces and cgroups
+   - advanced bindings
+      - using mpi: container, native, on the fly compiled
+      - using accelerators
+- reproducible builds