Automation/notebook ci

.github/workflows/mobile-price-classification-components.yaml

@@ -0,0 +1,53 @@
+name: Build Mobile Price Classification Components Image
+
+on:
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: "Image tag (e.g. v2, v3)"
+        required: true
+        default: "v2"
+
+env:
+  DOCKERFILE_PATH: "pipelines/lightweight-python-package/Dockerfile"
+  BUILD_CONTEXT_DIR: "pipelines/lightweight-python-package/"
+  IMAGE_NAME: "mobile-price-classification"
+  IMAGE_PUSH_PATH: "${{ secrets.GCP_ARTIFACT_REGISTRY_PATH }}"
+  IMAGE_PUSH_SECRET: "${{ secrets.GCP_SA_KEY }}"
+
+jobs:
+  build-image:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Delete huge unnecessary tools folder
+        run: rm -rf /opt/hostedtoolcache
+
+      - name: Checkout code
+        uses: actions/checkout@v7
+
+      - name: Auth GCloud CLI
+        uses: google-github-actions/auth@v3
+        with:
+          credentials_json: ${{ env.IMAGE_PUSH_SECRET }}
+
+      - name: Docker Login to Google Artifact Registry
+        uses: docker/login-action@v4
+        with:
+          registry: europe-west3-docker.pkg.dev
+          username: _json_key
+          password: ${{ env.IMAGE_PUSH_SECRET }}
+
+      - name: Build and push image
+        run: |
+          TAG=${{ github.event.inputs.tag }}
+          VERSIONED=${{ env.IMAGE_PUSH_PATH }}/${{ env.IMAGE_NAME }}:${TAG}
+          LATEST=${{ env.IMAGE_PUSH_PATH }}/${{ env.IMAGE_NAME }}:latest
+          COMMIT=${{ env.IMAGE_PUSH_PATH }}/${{ env.IMAGE_NAME }}:commit-${{ github.sha }}
+          BUILD_CONTEXT="${GITHUB_WORKSPACE}/${{ env.BUILD_CONTEXT_DIR }}"
+          docker build -t $VERSIONED -f ${{ env.DOCKERFILE_PATH }} "${BUILD_CONTEXT}"
+          docker tag $VERSIONED $LATEST
+          docker tag $VERSIONED $COMMIT
+          docker push $VERSIONED
+          docker push $LATEST
+          docker push $COMMIT
+          echo "Pushed: $VERSIONED"

README.md

@@ -7,15 +7,21 @@ For full platform documentation, see [docs.prokube.ai](https://docs.prokube.ai/)
 ```py
 .
 ├── .github          # workflows to build images
+├── ci               # CI orchestrator (run_all.py) and contributor conventions
 ├── hparam-tuning    # hyperparameter tuning examples (Katib)
 ├── images           # custom container images used by examples
 ├── mlflow           # MLflow experiment tracking examples
 ├── notebooks        # Jupyter notebook examples (Dask, MNIST VAE, etc.)
 ├── pipelines        # Kubeflow Pipelines examples
 ├── rstudio          # RStudio examples
+├── scripts          # shared utilities (credential setup, API key management)
 ├── serving          # model serving examples (KServe, vLLM, shadow deployments)
 ```
 
+Many serving examples include an `apply.py` (deploy + smoke-test) and a `cleanup.py`
+(teardown of Kubernetes resources) alongside the notebook. These are used by the CI
+orchestrator and can also be run manually.
+
 ## Note about storage
 Storage on Kubernetes is a complex topic and a deep dive is outside the scope of this repository. There are two types
 of storage you might encounter here — block and object storage.
@@ -43,6 +49,12 @@ Some examples require a minimum prokube platform version. If an example is not l
 |---|---|---|
 | `serving/minimal-s3-model` | v1.7.0 | Requires s3creds secret with KServe support |
 
+## CI
+
+Examples are tested end-to-end by `ci/run_all.py`, which runs inside a Kubeflow
+notebook pod. See [`ci/README.md`](ci/README.md) for how the orchestrator works and
+how to add a new example to CI.
+
 ## Contributing
 All code contributions should go via pull requests. Make sure your code is clearly documented and that it adheres
 to established standards (e.g. PEP).

ci/README.md

@@ -0,0 +1,156 @@
+# CI — contributor guide
+
+> This document is written primarily for AI agents making changes to this
+> repository.  It describes the conventions that must be followed for a new
+> example to be picked up by `ci/run_all.py` correctly.
+
+---
+
+## Adding a new example
+
+Register it in the `_EXAMPLES` list in `run_all.py`.  Everything else —
+phase scheduling, opt-in gating, cleanup, dry-run output, MLflow-credential
+skipping — is derived from that entry automatically.
+
+```python
+Example(
+    name="serving/my-new-example",       # display name and result key
+    steps=[
+        Step("script", "serving/my-new-example/apply.py"),
+        # or: Step("notebook", "serving/my-new-example/example.ipynb")
+        # chain multiple steps if needed (executed sequentially)
+    ],
+    phase=1,                             # see Phase rules below
+    cleanup="serving/my-new-example/cleanup.py",  # omit if no K8s resources
+    opt_in="include_foo",                # omit if always enabled
+    mlflow_dependent=False,              # True = skip when MLflow creds absent
+)
+```
+
+### Phase rules
+
+| Phase | When to use |
+|-------|-------------|
+| 1 | Self-contained: does not depend on anything else in CI |
+| 2 | Submits a KFP pipeline and returns fast; actual run is polled in Phase 4 |
+| 3 | Requires a model already registered by the Phase 2 mlflow-mobile-price pipeline |
+
+### Opt-in flag
+
+Add `opt_in="include_foo"` and a corresponding `--include-foo` argument in
+the `__main__` block of `run_all.py`.  Use opt-in when the example requires
+cluster add-ons (KEDA, postgres-operator, GPU nodes) that are not guaranteed
+to be present.
+
+---
+
+## apply.py and cleanup.py
+
+### cleanup.py — always add when K8s resources are created
+
+Any example that creates Kubernetes resources (InferenceService, Deployment,
+Service, CRD instance, …) must have a `cleanup.py` in its directory.
+CI runs all cleanup scripts in parallel in a `finally` block so they execute
+even on failure.
+
+A cleanup script must:
+- Delete resources idempotently (`--ignore-not-found`)
+- Never raise on failure (print a warning at most)
+- Read the namespace from the pod filesystem:
+
+```python
+with open("/var/run/secrets/kubernetes.io/serviceaccount/namespace") as f:
+    ns = f.read().strip()
+```
+
+**Do not** add a `cleanup.py` for examples that only create in-cluster
+transient objects managed by KFP (pipeline runs, artifacts) — those are
+cleaned up by KFP's own retention policy.
+
+### apply.py — only for serving examples driven by a script
+
+Add an `apply.py` only when the CI execution of a serving example is driven
+by a Python script rather than a notebook.  This is the case when:
+
+- The notebook's deploy/test section requires manual inputs that cannot be
+  automated, so a separate script owns the full deploy-wait-smoketest cycle.
+- The example has no notebook at all.
+
+An `apply.py` must exit non-zero on failure (CI relies on the return code).
+It must print a clean error message to stderr and suppress the full traceback:
+
+```python
+if __name__ == "__main__":
+    try:
+        deploy()
+    except Exception as exc:
+        print(str(exc), file=sys.stderr)
+        sys.exit(1)
+```
+
+If you add an `apply.py`, always add the matching `cleanup.py` as well.
+
+---
+
+## scripts/ utilities
+
+`scripts/` contains two cluster utilities importable from notebooks and apply
+scripts.  Both work identically whether called from a notebook cell or from an
+`apply.py`.
+
+### setup_mlflow_credentials.py
+
+One-time, interactive.  Stores MLflow credentials in the
+`mlflow-credentials` K8s secret.  **Requires human input** (the MLflow
+Personal Access Token cannot be obtained programmatically).  Run it once from
+a JupyterLab terminal:
+
+```bash
+python examples/scripts/setup_mlflow_credentials.py
+```
+
+Do not call this from CI.  CI validates the secret exists in the preflight
+check and skips MLflow-dependent examples if it does not.
+
+### get_or_create_api_key.py
+
+Idempotent, no human input.  Creates or retrieves the `examples-key`
+AIGatewayKey CR.  Call it from any notebook cell or script that needs an
+inference API key:
+
+```python
+import sys, subprocess, os
+_root = subprocess.check_output(["git", "rev-parse", "--show-toplevel"], text=True).strip()
+sys.path.insert(0, os.path.join(_root, "scripts"))
+from get_or_create_api_key import get_or_create_api_key
+
+API_KEY = get_or_create_api_key()
+```
+
+---
+
+## ci-skip cell tag
+
+Tag a notebook cell with `ci-skip` to have CI replace it with a no-op comment
+before execution.  Use this **only** for cells that genuinely cannot run
+headlessly:
+
+- Cells that require manual input (hardcoded paths, paste-your-token
+  placeholders, `input()` calls).
+- Interactive widget cells (`ipywidgets`, `ipykernel` display-only code).
+
+**Do not** use `ci-skip` to work around a fixable automation problem.  If a
+cell fails because it needs credentials or a namespace, fix it to load those
+automatically (see the MLflow credential pattern above).  `ci-skip` is a last
+resort, not a convenience.
+
+To tag a cell in JupyterLab: select the cell → Property Inspector (right
+panel) → add `ci-skip` under Cell Tags.
+
+In raw notebook JSON the tag appears as:
+
+```json
+"metadata": {
+    "tags": ["ci-skip"]
+}
+```