Jac Scale: node-exporter and Alloy are per-app-namespace DaemonSets, so agent pods scale as nodes x apps and exhaust the node max-pods cap

[udithishanka]

Summary

jac-scale deploys the observability agents per app namespace, and two of them are DaemonSets: _deploy_node_exporter and _deploy_alloy in monitoring.jac. Both carry a blanket NoSchedule toleration and no nodeSelector or affinity, so each deployed app claims one node-exporter pod and one Alloy pod on every node in the cluster, doing the same node-level work the previous app's copies already do. Agent pod count grows as nodes x apps while the work is inherently per-node.

On a 15-node production cluster with five apps running the pair, this is 90 of 610 pods, 15% of everything scheduled, and it has begun exhausting the kubelet max-pods cap: agent pods now sit Pending indefinitely and the next workload rejected on a full node will be a user app, not an agent.

Environment

jac	0.34.17 (Darwin arm64), released binary on PATH
repro host	macOS 25.3.0 arm64, no cluster required
also present on	main @ 9b02a4b2d (verified by reading, line refs below)
observed in production	EKS 15 nodes, us-east-2, VPC CNI

The code is unchanged between v0.34.17 and main: the blanket toleration is at monitoring.jac:599 (node-exporter) and monitoring.jac:2004 (Alloy) on main, and #8413 did not touch the DaemonSet shape.

Reproduction (executed, not hypothetical)

No cluster needed. MonitoringDeployer builds the manifests as plain dicts, so recording fakes are enough to show what it would create. _deploy_alloy reaches for a RbacAuthorizationV1Api the caller cannot inject, so the class itself is swapped.

# repro.jac
import from jaclang.scale.deploy.target.kubernetes.monitoring { MonitoringDeployer }
import from jaclang.scale.deploy.target.kubernetes.kubernetes_config { KubernetesConfig }
import from kubernetes { client }
import from kubernetes.client.rest { ApiException }
import json;

class Recorder {
    def init(self: Recorder) -> None { self.created = []; }
    def __getattr__(self: Recorder, name: str) -> any {
        created = self.created;
        if name.startswith("read_") {
            def _read(**kw: any) -> any {
                raise ApiException(status=404, reason="Not Found");
            }
            return _read;
        }
        def _write(**kw: any) -> any {
            body = kw.get("body");
            if isinstance(body, dict) and body.get("kind") == "DaemonSet" {
                created.append(body);
            }
            return None;
        }
        return _write;
    }
}

with entry {
    client.RbacAuthorizationV1Api = Recorder;
    cfg = KubernetesConfig(
        app_name="dev-flowline",
        namespace="jb-299afd1f-dev-flowline",
        monitoring_enabled=True,
        loki_enabled=True
    );
    d = MonitoringDeployer(cfg, None);
    apps = Recorder();
    core = Recorder();
    d._deploy_node_exporter("dev-flowline", "jb-299afd1f-dev-flowline", apps, core);
    d._deploy_alloy("dev-flowline", "jb-299afd1f-dev-flowline", apps, core);

    print(f"\nDaemonSets created for ONE app: {len(apps.created)}\n");
    for body in apps.created {
        spec = body["spec"]["template"]["spec"];
        hp = [
            v["hostPath"]["path"] for v in spec.get("volumes", []) if "hostPath" in v
        ];
        print(f"{body['metadata']['name']}  (namespace={body['metadata']['namespace']})");
        print(f"  kind          : {body['kind']}");
        print(f"  nodeSelector  : {spec.get('nodeSelector', 'ABSENT')}");
        print(f"  affinity      : {spec.get('affinity', 'ABSENT')}");
        print(f"  tolerations   : {json.dumps(spec.get('tolerations', 'ABSENT'))}");
        print(f"  hostPath mounts: {json.dumps(hp)}");
        print("");
    }
}

$ jac --version
jac 0.34.17  (Darwin arm64)

$ jac run repro.jac
DaemonSets created for ONE app: 2

dev-flowline-node-exporter  (namespace=jb-299afd1f-dev-flowline)
  kind          : DaemonSet
  nodeSelector  : ABSENT
  affinity      : ABSENT
  tolerations   : [{"operator": "Exists", "effect": "NoSchedule"}]
  hostPath mounts: ["/proc", "/sys"]

dev-flowline-alloy  (namespace=jb-299afd1f-dev-flowline)
  kind          : DaemonSet
  nodeSelector  : ABSENT
  affinity      : ABSENT
  tolerations   : [{"operator": "Exists", "effect": "NoSchedule"}]
  hostPath mounts: ["/var/log/pods", "/var/lib/docker/containers"]

$ echo $?
0

Two DaemonSets, namespaced to the app, unconstrained in placement. Deploy N apps to a cluster of M nodes and you get 2 x N x M agent pods.

There is no configuration that constrains them. A repo-wide sweep of the whole deploy surface finds no placement plumbing at all:

$ grep -rn "nodeSelector\|node_selector" jac/jaclang/scale/deploy/
(no matches)

Expected

Node-level agents should cost O(nodes), not O(nodes x apps). Either one shared agent per node serves every app on the cluster, or a per-app agent is not a DaemonSet at all and reads only its own namespace's pods.

At minimum, a per-app DaemonSet should carry the same placement constraints as the app's own workloads, so it does not claim slots on nodes the app can never be scheduled onto.

Actual

Every app pins a pod on every node. Measured on a 15-node production cluster with five namespaces running the pair (jachammer, all-in-one, and three jb-* app namespaces):

$ kubectl get pods -A --no-headers | wc -l
610
$ kubectl get pods -A --no-headers | grep -c node-exporter
45
$ kubectl get pods -A --no-headers | grep -c alloy
45

90 of 610 pods, 15% of the cluster, are duplicate agents. Every healthy node runs exactly 6 of them.

The max-pods cap is now binding. max-pods on EKS with the VPC CNI is the per-instance-type ENI/IP limit:

node	instance type	max-pods	non-terminated pods
ip-10-20-180-66	t3.large	35	34
ip-10-20-107-43	t3.large	35	33
ip-10-20-117-154	r5a.xlarge	58	52
ip-10-20-128-187	r5a.xlarge	58	52

On those four nodes the agents that did not get a slot have been Pending for 15 hours, retried by the scheduler roughly every 3.5 minutes:

0/16 nodes are available: 1 Too many pods, 15 node(s) didn't satisfy plugin(s) [NodeAffinity].
preemption: 0/16 nodes are available: 1 No preemption victims found for incoming pod, ...

A DaemonSet never gives up on a node, so this is permanent: about 2,200 failed scheduling attempts per 15 h per app in this state. The affected apps have no node metrics and no log shipping from those nodes, silently, because _deploy_observability errors are swallowed as non-fatal.

Root cause pointers, main @ 9b02a4b2d:

• monitoring.jac:580 _deploy_node_exporter builds kind: DaemonSet in the app namespace; :599 the blanket toleration; :634-635 hostPath /proc, /sys.
• monitoring.jac:1725 _deploy_alloy, same shape; :2004 the toleration; :2048-2053 hostPath /var/log/pods, /var/lib/docker/containers.
• monitoring.jac:2118 and :2134 are the call sites, gated on monitoring_enabled and loki_enabled or tracing_enabled respectively.

Alloy is a DaemonSet only because its generated config tails host files rather than reading through the API server:

# monitoring.jac:1741-1791
discovery.kubernetes "pods" { role = "pod" }
...
  target_label = "__path__"
  replacement  = "/var/log/pods/*$1/*.log"
local.file_match "pods" { path_targets = discovery.relabel.pods.output }
loki.source.file "pods" { targets = local.file_match.pods.targets ... }

Two things follow from that block. It forces the per-node shape, and discovery.kubernetes "pods" carries no namespace selector while the agent's ServiceAccount holds a cluster-wide pod-list ClusterRole (monitoring.jac:1923-1940), so a per-app agent discovers and tails pods well outside its own namespace.

Impact

• Pod-slot exhaustion is the binding constraint on tenancy. At five apps, 6 of a t3.large's 35 slots (17%) are redundant agents before any workload. The next thing a full node rejects is a user app pod or a sandbox pod, not an agent.
• Silent observability gaps. Affected apps lose metrics and logs from whichever nodes are full, with no error surfaced on the deploy.
• Cost. 32 agent pods per app on a 16-node cluster, at roughly 120Mi (Alloy) and 20Mi (node-exporter) each, before Prometheus and Loki.
• It compounds with [Bug]: Jac Scale: the default memory HPA trigger pins idle apps at max_replicas, because 80% of the 1Gi fallback request is below a jac server's idle footprint #9055. On the same cluster, 8 HPAs are pinned at max_replicas by the defaulted memory trigger, 28 excess pods above min_replicas serving no traffic, competing for the same slots.
• fix(scale): the monitoring switch decides whether the metrics stack deploys, instead of being overridden on the way to the deployer #8413 does not resolve it. After fix(scale): the monitoring switch decides whether the metrics stack deploys, instead of being overridden on the way to the deployer #8413, monitoring_enabled defaults false so node-exporter goes away for a logs-only app, but Alloy is gated on loki_enabled and still deploys. That halves the agent count and leaves the Pending pods exactly as they are.

Workaround (verified)

None that holds. [scale.monitoring] enabled = false (post-#8413) removes node-exporter, kube-state-metrics and Prometheus, which I verified by reading the gate at monitoring.jac:2118 and the enabled default of False in config_loader.impl.jac's get_monitoring_config. It does not remove Alloy, so an app that wants logs at all still pays one pod per node.

Turning [scale.microservices.logs] enabled = false removes Alloy but is not a workaround, it is giving up log collection.

Suggested fix

1. Make Alloy a 1-replica Deployment instead of a DaemonSet. Replace the local.file_match + loki.source.file hostPath pipeline with loki.source.kubernetes, which reads pod logs through the API server. For one app namespace with tens of pods this is well within budget, and it takes the per-app cost from M pods to 1. It also drops both hostPath mounts, and with them the namespace-privileged requirement that jac-scale: privileged-namespace labeling for hostPath DaemonSets is defeated by jac-scale's own baseline default #7766 had to work around, so that class of failure goes away rather than being patched.
2. Scope the Alloy discovery to the app's own namespace. discovery.kubernetes "pods" should carry a namespaces { names = [...] } block, and the ClusterRole should become a namespaced Role, so a per-app agent only ever sees its own app. This is required for correctness independent of item 1.
3. If a DaemonSet is kept anywhere, give it placement constraints. Drop tolerations: [{operator: Exists, effect: NoSchedule}] in favour of the same constraints the app's own Deployments carry, and add the nodeSelector plumbing the deploy surface currently lacks entirely. As it stands these pods target nodes the app can never run on.
4. Longer term, share the node-level agents. One node-exporter and one Alloy per cluster, one Prometheus and one Loki with per-namespace scoping, and a per-app Grafana pointing at the shared backends. That is the only shape whose cost tracks nodes rather than nodes x apps. Tracked product-side as jaseci-labs/jacBuilder#1531.

Regression test. A unit test in the shape of jac/jaclang/scale/tests/deploy/test_monitoring_switch.jac, which already drives MonitoringDeployer with an injected config and no cluster: assert that a deploy with loki_enabled = true creates no object of kind: DaemonSet, and that any DaemonSet the deployer does render carries a nodeSelector and no unbounded operator: Exists toleration. The repro above is most of that test already.

Related

• Jac Scale deploy surface: 15 validated defects across manifests, apply, ingress, monitoring, database wiring and teardown (tracking) #8410 the jac-scale deploy surface tracking issue, whose monitoring items are all about the stack's contents rather than its per-app multiplicity
• Jac Scale: half the monitoring stack is create-only and config changes never roll pods, so a redeploy leaves kube-state-metrics, node-exporter, Alloy and Grafana running the first deploy's config forever #8407 the same DaemonSets are create-only, so this shape is also frozen after the first deploy
• jac-scale: privileged-namespace labeling for hostPath DaemonSets is defeated by jac-scale's own baseline default #7766 the privileged-namespace workaround that item 1 would make unnecessary
• fix(scale): the monitoring switch decides whether the metrics stack deploys, instead of being overridden on the way to the deployer #8413 scoped monitoring_enabled correctly but leaves Alloy per-node
• backport(0.34.x): the KEDA scaling suite (#7620 #7899 #8008 #8424 #6908 #7709) for pinned jacBuilder deploys #8940 the 0.34.x backport line, which is where any of this reaches deployed apps
• [Bug]: Jac Scale: the default memory HPA trigger pins idle apps at max_replicas, because 80% of the 1Gi fallback request is below a jac server's idle footprint #9055 the defaulted memory HPA trigger, the other contributor to pod pressure on the same nodes
• jaseci-labs/jacBuilder#1711 the production report this came from, jaseci-labs/jacBuilder#1531 shared observability, jaseci-labs/jacBuilder#1530 per-app cost