Automation/notebook ci#56
Draft
tmvfb wants to merge 48 commits into
Draft
Conversation
- scripts/get_or_create_api_key.py: create/retrieve AIGatewayKey (importable + CLI)
- scripts/setup_mlflow_credentials.py: one-time MLflow PAT → k8s secret (importable + CLI, optional --uri/--username/--password args)
- notebooks/dask/dask_example.ipynb: auto-detect namespace from SA token file
- mlflow/{quickstart,image}: read MLflow creds from mlflow-credentials k8s secret instead of hardcoded placeholders
- mlflow/{kfp,mobile-price}: update markdown to reference setup_mlflow_credentials.py
- serving/minimal-s3-model: auto-create API key via get_or_create_api_key
- serving/mlflow-kserve-inference-protocols: add setup cell that applies YAML templates with substituted namespace/username, waits for ISVC readiness, and auto-fetches URIs and API key
- serving/mlflow-kserve-minimal/apply.py: deploy InferenceService with substituted values, wait for ready, print URI
- pipelines/lightweight-python-package/submit-cluster.py: print KFP_RUN_ID for CI polling
- cleanup.py per example: idempotent kubectl delete for k8s resources only
- ci/run_all.py: papermill-based orchestrator with parallel phases, KFP run polling, and cleanup in finally block
- Auto-install papermill if missing (pip install at startup) - Disable papermill progress bars (progress_bar=False) to prevent interleaved tqdm output from parallel notebook executions - Print [START] when each notebook/script is submitted so names are visible immediately, not just after completion - Switch from sequential f.result() to as_completed() so results print as each finishes rather than in submission order - Extract structured PapermillExecutionError details: cell number, cell source (first 5 lines), innermost traceback frame - Include full stderr/stdout in script failure messages (was truncated) - Print error details inline after each [FAIL] line during execution - Print full 'Failed details' section at the bottom of the report - Fix phase 2 remaining results printing under Phase 3 header
- _poll_kfp_run: use run.state (KFP v2 V2beta1Run) with fallback to run.run.status (KFP v1) for compatibility; normalise to uppercase - Terminal states updated to KFP v2 set: SUCCEEDED/FAILED/ERROR/CANCELED/SKIPPED - Phase 4: wrap f.result() in try/except so a polling error records POLL_ERROR in the report instead of crashing the whole CI run - _print_report: compare status.upper() == 'SUCCEEDED' (case-insensitive)
- inference_protocol_version_example.ipynb: fix pre-existing bug where cell read v1_InferenceService.yaml (underscore) but file is v1-InferenceService.yaml (dash); same for v2 - ci/run_all.py: move summary table to the very end (after Failed details) so final verdict is the last thing visible in the terminal - ci/run_all.py: skip Phase 3 ISVC tests when mlflow/mobile-price-classification fails, avoiding a 10-minute ISVC timeout on a model that was never registered
scripts/get_or_create_api_key.py: - Add required 'owner' field to AIGatewayKey manifest (CRD validation was rejecting the resource without it); reads MLFLOW_TRACKING_USERNAME from the mlflow-credentials secret when available, falls back to notebook-examples@<namespace> ci/run_all.py: - Add _check_mlflow_credentials() pre-flight check: validates secret exists AND credentials work (quick MLflow REST API call); skips all MLflow-dependent tests with a clear message if either check fails - Phase 1 and Phase 2 skip notebooks already resolved by pre-flight - Fix Phase 3 timing: after mlflow-mobile-price notebook finishes, poll its KFP run inline (blocking) before deploying ISVCs — previously Phase 3 deployed immediately while the pipeline (and model registration) was still running, causing ISVC timeouts - Phase 4 skips run IDs already polled inline before Phase 3
…tions ci/run_all.py: - _poll_kfp_run now returns (status, error_detail); on FAILED state fetches run.error.message from the KFP v2 API so the user can see what went wrong without opening the KFP UI - Add poll_errors dict; Failed details section in the report shows the KFP error message under each failed run - Add _strip_ci_skip_cells(): preprocess notebooks before papermill runs by replacing cells tagged 'ci-skip' with a comment placeholder - _run_notebook() calls _strip_ci_skip_cells() transparently (cwd stays the original notebook directory so relative imports still work) pipelines/lightweight-components/mobile-price-classifications.ipynb: - Tag cells 26-30 (the 'Debugging with Lightweight Components' section) with 'ci-skip' — these cells hardcode specific S3 artifact UUIDs from past manual runs and cannot execute in CI
notebooks/dask/dask_example.ipynb: - Drop pinned dask-kubernetes==2024.9.0 and dask[complete]==2024.9.1 which require anyio<4. Unpinned install resolves versions compatible with the cluster's anyio 4.12.1 and the updated dask operator. ci/run_all.py: - Add _get_failed_task_logs(): when a KFP run is FAILED and run.error is None (common in KFP v2), walks task_details → child_tasks → pod_name and tails the last 10 lines of the failed task pod's logs via kubectl. Falls back silently if pods are already gone or logs are unavailable. - _poll_kfp_run: reads namespace from SA token file and passes it to _get_failed_task_logs so CI now surfaces the actual pipeline failure message rather than showing nothing.
…ponents The train_model and evaluate_model KFP components had mlflow==3.10.0 pinned in packages_to_install. That version does not exist (latest 3.x is 3.1.4), causing every pipeline run to fail at the package install step. Changed to mlflow>=3.0.0,<4 which installs the latest available 3.x release and stays compatible with the 3.x MLflow tracking server.
The pin was correct — 3.10.0 exists on public PyPI and matches the server. The failure is that KFP component pods (python:3.9 base) have restricted network access and can only see an older PyPI snapshot. Root cause: network policy issue for component pods, not a version mismatch. Needs a platform-level fix (open network access or use a base image that already has mlflow pre-installed).
mlflow>=3.2.0 requires Python >=3.10. With python:3.9 as the base image, pip silently excludes all mlflow versions that have Requires-Python>=3.10, leaving only versions up to 3.1.4 visible — which is why mlflow==3.10.0 appeared 'not found'. Upgrading to python:3.11 makes the full PyPI version range visible and satisfies the requirement. Affects all @dsl.component decorators in: - mlflow/mobile-price-classification/mlflow-mobile-price-classification.ipynb (7 components) - pipelines/lightweight-components/mobile-price-classifications.ipynb (7 components)
Dockerfile: python:3.9-slim → python:3.11-slim (same Python version issue as the KFP lightweight components — newer packages require Python >=3.10) pipeline.py: COMPONENTS_IMAGE default bumped to :v2; falls back to COMPONENTS_IMAGE env var so the image path can be overridden without a code change. .github/workflows/mobile-price-classification-components.yaml: new workflow_dispatch workflow to build and push the image. Accepts a tag input (default: v2) and pushes :v2, :latest, and :commit-<sha>.
httpx_ws (a dask-kubernetes dependency via kr8s) uses anyio.AsyncContextManagerMixin which was removed in anyio 4.0 and re-added as a compat alias in anyio 4.12+. The etst-0 CI notebook has anyio 4.6.0 (no alias) while test-0 has 4.12.1 (alias present). Adding anyio>=4.12 to the pip install ensures the compat alias is available regardless of which notebook image version is running.
Pin dask[complete] and dask-kubernetes to 2026.3.0 to match the latest available daskdev/dask Docker image (no 2026.6.0 image exists yet on Docker Hub). Mismatched scheduler versions between the client (2026.6.0 from an unpinned install) and the worker image caused: TypeError: Scheduler.identity() got an unexpected keyword argument 'n_workers' Also updates the KubeCluster image from daskdev/dask:2024.9.1-py3.11 to daskdev/dask:2026.3.0-py3.11.
…ponents, mlflow-kserve test pipelines/minimal-container-components/submit-cluster.py: - Print KFP_RUN_ID so CI can poll the pipeline run status notebooks/mnist-vae/visualizations.ipynb: - Tag the ipywidgets.interact cell with ci-skip (GUI-only, blocks headless CI) serving/mlflow-kserve-minimal/apply.py: - Add test() function: gets API key, runs test_inference_service.py with API_KEY and INFERENCE_SERVICE_URI env vars set, asserts success - Add deploy_and_test() which chains deploy() then test() - __main__ now calls deploy_and_test() so CI gets a full smoke test ci/run_all.py: - _run_script: accept extra_args for passing CLI flags (e.g. --max_epochs) - _submit_script: propagate extra_args - _submit_chain: new helper to run a sequential list of notebook/script steps as a single named CI result (used for mnist-vae train→visualize) - Phase 1: add notebooks/mobile-price-classification (standalone notebook) and notebooks/mnist-vae (training 3 epochs → visualizations chain) - Phase 2: add pipelines/minimal-container-components - Phase 3: mlflow-kserve-minimal now calls apply.py which does deploy_and_test; inference-protocols unchanged
pytorch_lightning is not installed in all notebook images. Running the example without it fails immediately with ModuleNotFoundError. Add --include-pytorch flag (default off); mnist-vae is skipped with a clear message unless the flag is passed. Same pattern as --include-keda.
The pytorch-full image ships PyTorch but not pytorch-lightning. Add a 'pip' step kind to _submit_chain and use it to install pytorch-lightning before running run_training.py when --include-pytorch is set. Also documents the numpy/sklearn issue: the pytorch-full image pins numpy to 1.26.4 but scikit-learn 1.6.1 requires numpy >=2.0 for the _signature_descriptor ABI. Fix is in the image (upgrade numpy to 2.x — PyTorch 2.5.1 supports it) not in the examples.
pytorch-lightning is not bundled in all notebook images. The script now installs it silently on first run if absent, so users and CI don't need to pre-install it. The explicit pip step in the CI chain is removed since the script handles its own dependency.
…name
The KFP pipeline registers models as 'mobile-price-svm-{local_user}'
using MLFLOW_TRACKING_USERNAME.split('@')[0] (cell 19 of the notebook).
Both apply.py and the inference-protocols setup cell were passing the
full email (e.g. 'developer1@prokube.cloud') as the model name suffix,
causing the ISVC storageUri to reference a non-existent model and timing
out waiting for readiness.
…nflict KServe v0.18.0 injects S3 env vars (S3_ENDPOINT, AWS_ENDPOINT_URL, S3_USE_HTTPS) into the storage-initializer for every ISVC whose SA references s3creds. The MLflow storage-initializer sets the same vars via valueFrom, producing a value+valueFrom conflict Kubernetes rejects at admission. Pods are never created and the ISVC times out. - Add ServiceAccount.yaml: mlflow-isvc-sa with regcred-prokube and regcred-dev imagePullSecrets but no s3creds reference - Set serviceAccountName: mlflow-isvc-sa in InferenceService.yaml - Update apply.py to apply the SA manifest before the ISVC - Document the root cause and fix in README.md
Same KServe S3 credential injection conflict fix as mlflow-kserve-minimal: the default SA references s3creds, causing a value+valueFrom env conflict for mlflow:// ISVCs that Kubernetes rejects at admission. - Add ServiceAccount.yaml: mlflow-isvc-sa with imagePullSecrets and no s3creds reference - Set serviceAccountName: mlflow-isvc-sa in v1 and v2 InferenceService manifests - Add README.md documenting the root cause, fix, deploy steps, and cleanup
The inference-protocols notebook deploys InferenceServices and makes inference calls. Its output contains UUIDs (ISVC UIDs, request IDs) that the broad UUID regex in _extract_run_ids_from_notebook picks up. Phase 4 then polls these as KFP run IDs, which returns 404. Add extract_run_ids flag to _submit_notebook (default True) and pass False for serving/mlflow-kserve-inference-protocols, which never submits KFP pipelines.
…nist-vae scipy 1.17+ is compiled against CXXABI_1.3.15 which the system libstdc++ lacks (tops out at 1.3.13). The conda-bundled libstdc++ provides it but is not on LD_LIBRARY_PATH by default. Re-exec the process with /opt/conda/lib prepended to LD_LIBRARY_PATH before any imports so the dynamic linker picks up the right library. Also add --max_epochs click option (default 50) so the CI can pass --max_epochs 3 for a fast smoke-test run; previously the option was silently ignored and the trainer always ran 50 epochs.
torchvision is absent from the base notebook image independently of pytorch-lightning. Extend the self-install guard to check both and install whichever packages are missing in a single pip call.
…loyment
Three previously untested examples now have apply.py + cleanup.py and
are wired into ci/run_all.py:
serving/hf-vllm-completion (always-on in Phase 1)
- Deploys distilbert-inf-serv (HuggingFace CPU backend)
- Smoke-tests via cluster-internal URL; no API key required
serving/kserve-keda-autoscaling (opt-in: --include-keda)
- Checks scaledobjects.keda.sh CRD; exits 0 with SKIP if absent
- Deploys OPT-125M ISVC + ScaledObject, waits for Active=True
- Smoke-tests via cluster-internal predictor Service
serving/minimal-example-shadow-deployment (opt-in: --include-shadow)
- Checks postgresclusters CRD; exits 0 with SKIP if absent
- Deploys PostgresCluster, waits for primary, creates schema via
one-shot psql pod, deploys doubler + tripler ISVCs
- Smoke-tests primary via cluster-internal URL (factor=2 check)
- Istio VirtualService mirroring is not tested in CI
Also fixes the dead --include-keda flag (was parsed but never passed
into run_all()) and adds --include-shadow on the same pattern.
The previous approach checked 'kubectl get crd postgresclusters...' which requires cluster-level RBAC that notebook SAs typically don't have, causing the check to fail and the example to silently skip regardless of whether the operator is installed. Instead, attempt to apply postgres-cluster.yaml and treat a 'no matches for kind' / 'server doesn't have a resource type' error as a graceful skip. Any other error (permissions, connectivity) is raised so CI marks it as FAIL rather than a false PASS. Also strengthen the smoke test to verify the doubler's actual output (FACTOR=2: [1,2,3] must produce [2,4,6]) rather than just checking that a predictions key exists.
Same RBAC issue as the shadow-deployment fix: notebook SAs lack cluster-level permission to 'kubectl get crd', so the KEDA check always returned False and the example silently skipped. Deploy the ISVC first (always), then attempt to apply the ScaledObject and treat 'no matches for kind' as a graceful skip. This also gives more useful CI signal: the ISVC smoke-test still runs even when KEDA is absent.
Replace the raw kubectl stderr blob with targeted messages: - 'no matches for kind': operator not installed - Forbidden on postgres-operator resources: SA lacks RBAC, with the exact grant needed (get/create/update/patch on postgresclusters) The example still FAILs in CI (correct) but the error line now tells the operator admin exactly what to fix.
tmvfb
force-pushed
the
automation/notebook-ci
branch
from
5d3ba86 to
e7665fe
Compare
Unhandled exceptions print a full Python traceback to stderr which the CI captures verbatim. Catch at __main__ level and print just str(exc) so CI shows the actionable message without the stack noise.
Replace ad-hoc per-phase logic with a single _EXAMPLES list. Adding a new example is now one entry in the table — no phase function to find, no cleanup list to remember separately, opt-in flag and mlflow-dependency declared inline. Key changes: - Step / Example dataclasses describe what to run - _EXAMPLES table is the single source of truth for all examples, their phase, cleanup scripts, opt-in gates and mlflow dependency - _make_work() generates a work closure from a list of Steps, replacing _submit_notebook / _submit_script / _submit_chain - _run_phase() filters _EXAMPLES by phase and submits matching work - _preflight() and cleanup list are derived from _EXAMPLES - _print_dry_run() is also derived from _EXAMPLES (stays current) - Fix --dry-run bug: preflight no longer runs in dry-run mode
… and ci-skip tags
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a notebook-pod CI orchestrator to run the repository’s automatable examples end-to-end (notebooks, KFP submissions + polling, and serving deployments), alongside example/script updates to make headless execution and cleanup reliable.
Changes:
- Introduces
ci/run_all.pywith a table-driven registry, phased execution model, MLflow credential preflight, KFP run polling, and always-on cleanup. - Adds/updates
apply.py+cleanup.pyhelpers and KServe ServiceAccounts for serving examples (including MLflow ISVC SA to avoid KServe S3 env injection conflicts). - Updates multiple notebooks/pipeline artifacts for CI compatibility (auto-namespace/credentials,
ci-skiptags, Python base image updates, idempotent API key utility).
Reviewed changes
Copilot reviewed 40 out of 40 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| serving/mlflow-kserve-minimal/ServiceAccount.yaml | Adds dedicated SA with imagePullSecrets and avoids s3creds injection path. |
| serving/mlflow-kserve-minimal/README.md | Documents why the dedicated SA is required and how to deploy it. |
| serving/mlflow-kserve-minimal/InferenceService.yaml | Uses mlflow-isvc-sa for MLflow ISVC predictor. |
| serving/mlflow-kserve-minimal/cleanup.py | Adds idempotent teardown for the minimal MLflow ISVC. |
| serving/mlflow-kserve-minimal/apply.py | Adds scripted deploy + wait + smoke test for the minimal MLflow ISVC. |
| serving/mlflow-kserve-inference-protocols/v2-InferenceService.yaml | Sets serviceAccountName: mlflow-isvc-sa for v2 protocol ISVC. |
| serving/mlflow-kserve-inference-protocols/v1-InferenceService.yaml | Sets serviceAccountName: mlflow-isvc-sa for v1 protocol ISVC. |
| serving/mlflow-kserve-inference-protocols/ServiceAccount.yaml | Adds dedicated SA mirroring the minimal MLflow serving example. |
| serving/mlflow-kserve-inference-protocols/README.md | Adds end-to-end docs and the SA rationale for the protocol comparison example. |
| serving/mlflow-kserve-inference-protocols/inference_protocol_version_example.ipynb | Updates YAML filenames and automates live-testing setup. |
| serving/mlflow-kserve-inference-protocols/cleanup.py | Adds idempotent teardown for both protocol ISVCs. |
| serving/mlflow-kserve-inference-protocols/apply.py | Adds scripted deploy + readiness wait + dual-protocol smoke tests. |
| serving/minimal-s3-model/minimal-s3-model.ipynb | Automates API key acquisition via shared utility. |
| serving/minimal-s3-model/cleanup.py | Adds idempotent teardown, optionally removing shared API key resources. |
| serving/minimal-example-shadow-deployment/cleanup.py | Adds idempotent teardown of ISVCs and PostgresCluster. |
| serving/minimal-example-shadow-deployment/apply.py | Adds scripted deploy of Postgres + shadow ISVCs + internal smoke test. |
| serving/kserve-keda-autoscaling/cleanup.py | Adds idempotent teardown for KEDA ScaledObject + ISVC. |
| serving/kserve-keda-autoscaling/apply.py | Adds scripted deploy + KEDA presence detection + readiness + smoke test. |
| serving/hf-vllm-completion/cleanup.py | Adds idempotent teardown for the HF/vLLM serving example. |
| serving/hf-vllm-completion/apply.py | Adds scripted deploy + readiness + internal smoke test for HF serving. |
| scripts/setup_mlflow_credentials.py | Adds interactive helper to create/update mlflow-credentials secret. |
| scripts/get_or_create_api_key.py | Adds idempotent helper to create/reuse a shared AIGatewayKey + Secret. |
| README.md | Documents new ci/ and scripts/ directories and CI entrypoint. |
| pipelines/minimal-container-components/submit-cluster.py | Normalizes formatting and prints KFP_RUN_ID for CI polling. |
| pipelines/lightweight-python-package/submit-cluster.py | Improves namespace read formatting and prints KFP_RUN_ID. |
| pipelines/lightweight-python-package/pipeline.py | Makes components image configurable via env; updates default tag. |
| pipelines/lightweight-python-package/Dockerfile | Updates base image to Python 3.11 slim. |
| pipelines/lightweight-components/mobile-price-classifications.ipynb | Updates component base images to Python 3.11 and adds ci-skip tags. |
| notebooks/mnist-vae/visualizations.ipynb | Adds ci-skip tag to a visualization cell for headless CI execution. |
| notebooks/mnist-vae/run_training.py | Adds re-exec workaround for libstdc++ ABI and installs missing deps; adds --max_epochs. |
| notebooks/dask/dask_example.ipynb | Automates namespace detection and updates Dask/Dask-Kubernetes versions. |
| notebooks/dask/cleanup.py | Adds idempotent cleanup of DaskCluster CR. |
| mlflow/mobile-price-classification/mlflow-mobile-price-classification.ipynb | Updates MLflow secret setup guidance and Python base images. |
| mlflow/mlflow-quickstart-example.ipynb | Switches to loading MLflow creds from K8s secret and removes deprecated sklearn arg. |
| mlflow/mlflow-kfp-example.ipynb | Updates MLflow setup instructions to use the new credentials script. |
| mlflow/mlflow-image-example.ipynb | Switches to loading MLflow creds from K8s secret. |
| hparam-tuning/minimal-mnist/cleanup.py | Adds idempotent cleanup for Katib experiment resources. |
| ci/run_all.py | Adds phased CI orchestrator for notebooks/scripts + KFP polling + cleanup. |
| ci/README.md | Documents CI conventions: registry, phases, opt-ins, apply/cleanup, ci-skip. |
| .github/workflows/mobile-price-classification-components.yaml | Adds workflow to build/push the lightweight-python-package components image. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
* Bump `actions/checkout` from v4 to v7 * Bump `google-github-actions/auth` from v2 to v3 * Bump `docker/login-action` from v3 to v4
* Accept an optional `ns` parameter in `get_or_create_api_key()`, falling back to the auto-detected namespace when not provided * Add `argparse`-based CLI with `--namespace`/`-n` flag when the script is run directly * Allows callers to override the target Kubernetes namespace without changing the default behavior
…SA namespace replace in minimal
Auto-detect the AIGatewayKey CRD; if absent (older clusters using a hardcoded EnvoyFilter), fall back to the INFERENCE_SERVICE_API_KEY env var injected by the admin, and finally to an interactive getpass prompt so notebooks degrade gracefully instead of crashing.
Notebook pod service accounts lack RBAC permission to read cluster-scoped CRD objects, so the previous 'kubectl get crd' check always returned False in-cluster. API discovery (kubectl api-resources) is allowed for all authenticated users and reliably detects whether the CRD is registered.
Member
Author
|
Temporarily converting to draft, as we haven't decided on kserve model auth yet. As a temporal solution we can remove all AIGatewayKey logic and references, and accept that obtaining API key script will just prompt for credentials :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI automation for notebook examples
Adds a CI orchestrator that runs all automatable examples end-to-end inside a Kubeflow notebook pod, and makes the necessary fixes to the examples themselves to make that possible.
ci/run_all.py_EXAMPLESregistry. Adding a new example is one entry — phase scheduling, opt-in gating, cleanup, and dry-run output are all derived automatically.mlflow-credentialssecret; skips MLflow-dependent examples with a clear message if absent or expired.Examples automated
All automatable examples in the repository are now wired into CI — notebooks, pipeline submissions, and serving deployments across
mlflow/,notebooks/,pipelines/, andserving/. Three serving examples (hf-vllm-completion,kserve-keda-autoscaling,minimal-example-shadow-deployment) are new additions to the repo created specifically for CI coverage. Three (kserve-keda-autoscaling,minimal-example-shadow-deployment,mnist-vae) are opt-in as they require cluster/notebook add-ons not guaranteed to be present.Notebook fixes
mlflow-credentialsK8s secret and the pod filesystemci-skiptag mechanism: cells that cannot run headlessly (interactive widgets, manual S3 paths) are replaced with a no-op comment by the CI runnermulti_classparameter removed (dropped in 1.6)pytorch-lightning,torchvision,tensorboard) where neededSupporting utilities
scripts/setup_mlflow_credentials.py— one-time interactive credential setup (the only step requiring human input)scripts/get_or_create_api_key.py— idempotent AIGatewayKey creation for inference testingDocumentation
ci/README.md— conventions for adding new examples, when to addapply.py/cleanup.py, how to usescripts/, and theci-skiptag contract.