One user, one node: co-location affinity plus RWO home volume creates multiple issues

[aktech]

All of a user's servers (JupyterLab plus every jhub-apps app) are forced onto a single node. Three failure modes:

1. A running app blocks GPU lab spawns entirely.
User has a jhub-apps app running on a CPU node, then starts JupyterLab with a GPU profile. The GPU profile requires a GPU node (node selector / nvidia.com/gpu resource limit), while the co-location affinity requires the CPU node where the app already runs. No node satisfies both, the pod stays Pending, and the spawn fails at startTimeout (300s). The cluster autoscaler cannot help: a freshly provisioned GPU node has no pod with the user's label, so the affinity can never be satisfied there. The only way out is stopping every running server first.

2. Apps ride along on GPU nodes and keep them alive.
User starts JupyterLab on a GPU profile, then creates a jhub-apps app (any size). The affinity forces the app onto the GPU node. When the user later stops the lab but leaves the app running, the app keeps the expensive GPU node from scaling down until the app itself is stopped or deleted.

3. One node's capacity caps a user's total footprint.
Everything a user runs concurrently must fit on the node their first pod landed on. Once that node's remaining allocatable CPU/memory cannot hold the next server's resource guarantee, the spawn stays Pending until startTimeout even though the rest of the cluster has room. The autoscaler cannot resolve it for the same reason as above. The effective per-user limit is not the deployment's quota but whatever slice of one node their pods happened to land on, and that node choice is invisible to the user.

1. GPU lab blocked	2. GPU node kept alive	3. Capacity cap

Why

Every user pod gets a required (hard) pod affinity to nodes already running one of the user's pods:

data-science-pack/config/jupyterhub/01-spawner.py

Lines 161 to 179 in a1f8c97

	c.KubeSpawner.extra_pod_config = {
	"affinity": {
	"podAffinity": {
	"requiredDuringSchedulingIgnoredDuringExecution": [
	{
	"labelSelector": {
	"matchExpressions": [
	{
	"key": "nebari.dev/colocate-user",
	"operator": "In",
	"values": ["{username}"],
	}
	]
	},
	"topologyKey": "kubernetes.io/hostname",
	}
	]
	}
	},

The affinity exists because the home volume is a single per-user ReadWriteOnce PVC mounted by all of the user's servers, and RWO allows attachment to only one node at a time:

data-science-pack/config/jupyterhub/01-spawner.py

Line 37 in a1f8c97

c.KubeSpawner.storage_access_modes = ["ReadWriteOnce"]

data-science-pack/config/jupyterhub/01-spawner.py

Line 49 in a1f8c97

c.KubeSpawner.pvc_name_template = "claim-{username}"

Removing the affinity alone fixes nothing: with RWO storage the second node hits a Multi-Attach error and the pod dies in ContainerCreating instead of Pending. The invariant comes from the storage access mode; the affinity only surfaces it at scheduling time. (#211 changes how the affinity value is computed, not this behavior.)

Per-server choice of access mode is not possible: all servers mount the same claim-{username} PVC, and Kubernetes PVC access modes are immutable after creation.

Proposal

Deployment-level knob:

• custom.storage-access-mode: ReadWriteOnce (default, current behavior) or ReadWriteMany
• custom.storage-class: required for RWX, must be an RWX-capable StorageClass (e.g. Longhorn), same pattern sharedStorage.storageClass already uses
• When RWX: skip the co-location affinity entirely. Pods schedule independently; all three failure modes disappear.

Trade-offs to document with the knob:

• RWX home is network file storage. Home workloads (pixi/conda envs, tens of thousands of small files) pay a metadata round trip per operation, so environment creation and interactive latency degrade substantially versus node-attached block storage. This is why RWO stays the default.
• Existing claim-{username} PVCs keep RWO forever, so flipping the knob on a live deployment breaks existing users (no affinity + RWO volume = Multi-Attach returns). Safe only for fresh deployments or with home data migration.

Need to think through this.

Workaround today

Stop all running servers, then start the GPU lab: the volume detaches and reattaches to the GPU node. For an app stranded on a GPU node, stop and start the app after the lab is gone; with no other pod carrying the user's label, the scheduler places it on a CPU node. There is no workaround for the capacity cap: respawning everything only moves the pods to another single node, which must still hold the entire set.