[History Server] Tune CPU, memory, and event logging for cold loads (#5095)

* [History Server] Fix cold-load CPU starvation and per-event logging

Loading a dead session is CPU-bound, but the sample manifest sets only
`limits.cpu: "500m"`, so Kubernetes pins requests there too and the load
saturates the quota for its entire duration. Measured on kind across 13 runs,
the container sat at 0.42-0.50 cores every time.

Raising the limit to 4 (requests stay at 500m, so scheduling cost is unchanged):

| tasks in session | 500m | 4 cores |
|---|---|---|
| 50,000 | 97.9s | 30.5s |
| 100,000 | 907.3s | 62.7s |

This also removes what looked like superlinear degradation past 50k tasks: at
500m the per-task cost went from 1.96ms to 9.07ms between 50k and 100k, while at
4 cores it is flat (0.61ms and 0.63ms). The load uses ~1.2 cores on average and
peaks at ~2.2, because Go's GC runs concurrently and needs cores of its own.

The second change drops the per-event log line in storeEvent to Debug. It runs
once per event, so a 100k-task session writes ~436,000 INFO lines per cold load,
and the binary never calls logrus.SetLevel, so there is no way to turn it off.
Worth ~13% of load time at 50k tasks (97.9s -> 85.3s).

* historyserver: leave CPU limit configurable

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* historyserver: size sample memory for session cache

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* historyserver: repair tests after cache changes

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* historyserver: size example cache memory

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* historyserver: document example memory sizing

Signed-off-by: Future-Outlier <eric901201@gmail.com>

---------

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Author: Future-Outlier

historyserver/config/historyserver-azureblob.yaml

@@ -46,6 +46,13 @@ spec:
         - --ray-root-dir=log
         ports:
         - containerPort: 8080
+        # Cache is soft-bounded at 8 GiB; 12 GiB leaves decode headroom.
+        # The 2 GiB request keeps examples schedulable; size production requests to residency.
+        args:
+        - --session-cache-max-bytes=8589934592
         resources:
-          limits:
+          requests:
             cpu: "500m"
+            memory: "2Gi"
+          limits:
+            memory: "12Gi"

historyserver/config/historyserver-gcs.yaml

@@ -44,6 +44,13 @@ spec:
         imagePullPolicy: IfNotPresent
         ports:
         - containerPort: 8080
+        # Cache is soft-bounded at 8 GiB; 12 GiB leaves decode headroom.
+        # The 2 GiB request keeps examples schedulable; size production requests to residency.
+        args:
+        - --session-cache-max-bytes=8589934592
         resources:
-          limits:
+          requests:
             cpu: "500m"
+            memory: "2Gi"
+          limits:
+            memory: "12Gi"

historyserver/config/historyserver.yaml

@@ -65,6 +65,13 @@ spec:
         # - --use-auth-token-mode=true
         ports:
         - containerPort: 8080
+        # Cache is soft-bounded at 8 GiB; 12 GiB leaves decode headroom.
+        # The 2 GiB request keeps examples schedulable; size production requests to residency.
+        args:
+        - --session-cache-max-bytes=8589934592
         resources:
-          limits:
+          requests:
             cpu: "500m"
+            memory: "2Gi"
+          limits:
+            memory: "12Gi"

historyserver/pkg/eventserver/eventserver.go

@@ -133,7 +133,7 @@ func (h *EventHandler) storeEvent(clusterSessionKey string, eventMap map[string]
 	}
 	eventType := types.EventType(eventTypeStr)
 
-	logrus.Infof("current eventType: %v", eventType)
+	logrus.Debugf("current eventType: %v", eventType)
 	switch eventType {
 	case types.TASK_DEFINITION_EVENT:
 		return h.handleTaskDefinitionEvent(eventMap, clusterSessionKey, false)