diff --git a/docs/docs/reference/plugins/jac-scale-kubernetes.md b/docs/docs/reference/plugins/jac-scale-kubernetes.md index 0fa82a42419..ec55ac37857 100644 --- a/docs/docs/reference/plugins/jac-scale-kubernetes.md +++ b/docs/docs/reference/plugins/jac-scale-kubernetes.md @@ -395,6 +395,24 @@ The `"keda"` engine creates a `ScaledObject` custom resource instead of an HPA. !!! note KEDA must be installed on the cluster before using this engine. If KEDA CRDs are absent at deploy time, jac-scale emits an install warning with a link to the [KEDA installation docs](https://keda.sh/docs/latest/deploy/) and falls back to static replicas rather than failing the deploy. +!!! note "HTTP-activated workloads also require the KEDA HTTP Add-on" + Workloads scaled via `apply_http_activation` (HTTP request-driven scale-to-zero) require the [KEDA HTTP Add-on](https://keda.sh/docs/latest/deploy/#http-add-on) in addition to core KEDA. Call `KEDAAutoscaler.discover_capabilities()` to check both together: it returns a `KEDACapabilities` object that distinguishes a missing core install from a missing or outdated HTTP Add-on, an RBAC-denied check from a genuinely absent API, and a missing external-scaler or interceptor-proxy Service, each with its own diagnostic. Results are cached per cluster; pass `refresh=True` or call `invalidate_capabilities()` to force a recheck. `apply_http_activation` calls `discover_capabilities()` automatically and raises when the Add-on is installed but broken, instead of deploying a workload that will never receive traffic. If the Add-on is simply absent, it logs a warning and skips activation rather than failing the deploy. + + Install both with Helm: + ```bash + helm repo add kedacore https://kedacore.github.io/charts + helm repo update + helm install keda kedacore/keda -n keda --create-namespace --wait + helm install http-add-on kedacore/keda-add-ons-http -n keda --wait + ``` + + Upgrading an older HTTP Add-on install (pre-0.14, `HTTPScaledObject` only) to the current `InterceptorRoute` API: + ```bash + helm upgrade http-add-on kedacore/keda-add-ons-http -n keda --wait + ``` + + Chart names, flags, and required values can change over time, so every install/upgrade command surfaced by `discover_capabilities()` also links to the current getting-started guide as the authoritative fallback: [https://keda.sh/http-add-on/0.15/getting-started/](https://keda.sh/http-add-on/0.15/getting-started/). + **Switching between engines is safe.** Each engine removes the other engine's resource (`ScaledObject` or `HPA`) on apply, so two autoscalers never compete for `spec.replicas` on the same Deployment. !!! warning "CPU/memory triggers: scale-down always takes ~5 minutes" @@ -454,6 +472,98 @@ metadata = { queueName = "orders", mode = "QueueLength", value = "50", protocol host = { name = "rabbitmq-secret", key = "host" } ``` +#### HTTP Add-on Activation (Scale-to-Zero on Request) + +The KEDA engine above scales on CPU, memory, or any KEDA trigger, but none of those triggers can wake a workload from zero replicas in response to an incoming HTTP request itself. There is nothing listening on the Service to observe traffic when replicas are at zero. The [KEDA HTTP Add-on](https://keda.sh/http-add-on/0.15/) closes that gap: it intercepts HTTP traffic bound for the target, holds the request while a zero-replica workload starts, and forwards it only once the workload is ready. + +!!! note + The KEDA HTTP Add-on installs separately from KEDA core. It ships as its own Helm chart: + ```bash + helm repo add kedacore https://kedacore.github.io/charts + helm repo update + helm install keda kedacore/keda -n keda --create-namespace --wait + helm install http-add-on kedacore/keda-add-ons-http -n keda --wait + ``` + If the HTTP Add-on's CRDs are missing at deploy time, jac-scale logs a warning and skips creating the `InterceptorRoute`/`ScaledObject` rather than failing the deploy, matching the `"keda"` engine's own preflight fallback above. + +**Prerequisites** + +- KEDA core is installed on the cluster, as described above for the `"keda"` engine. +- The KEDA HTTP Add-on is installed, per the note above. +- The scale target exposes Kubernetes' `/scale` subresource. A `Deployment` or `StatefulSet` works out of the box; a standalone `Pod` is rejected with an error explaining the `/scale` requirement. A custom resource such as a `Rollout` also works, but only once `scale_target_plural` is set, so jac-scale can confirm it exists before applying anything. +- The target's Deployment or StatefulSet, and its Service, already exist. This feature manages the `InterceptorRoute` and `ScaledObject` around an existing workload; it does not create the workload or the Service. +- Exactly one of `target_port` or `target_port_name` is set, and at least one of `concurrency_target` or `request_rate_target` is set. Both are validated up front with an error that names the offending `jac.toml` key. + +**Interaction with the base autoscaler:** a target with `http_activation.enabled = true` is scaled entirely by its `ScaledObject` (min/max replicas, scale-to-zero). `jac start --scale` skips creating the base HPA/KEDA autoscaler for that same target automatically -- KEDA's admission webhook rejects a `ScaledObject` for a workload already managed by an HPA (or another `ScaledObject`), so both can't coexist on one target. This also means the deploy's usual post-deploy HTTP reachability check is skipped for that target (it may legitimately be sitting at 0 replicas with nothing to reach); a crash-loop check on the pods runs instead. + +**HTTP activation configuration (`[scale.kubernetes.http_activation]`):** + +| TOML Key | Default | Description | +|----------|---------|-------------| +| `enabled` | `false` | Master switch. Off by default. | +| `min_replicas` | `0` | Replica floor while inactive. `0` enables true scale-to-zero. | +| `max_replicas` | `1` | Replica ceiling once activated. | +| `polling_interval` | `30` | Seconds between HTTP metric evaluations. | +| `cooldown_period` | `300` | Seconds of inactivity before scaling back to `min_replicas`. | +| `target_port` / `target_port_name` | `null` | Container port on the app's Service. Set exactly one. | +| `concurrency_target` | `null` | In-flight-request concurrency target. Set this or `request_rate_target`. | +| `request_rate_target` | `null` | Requests-per-window target, as an alternative to `concurrency_target`. | +| `request_rate_window` / `request_rate_granularity` | `"1m"` / `"1s"` | Window and sampling granularity for `request_rate_target`. | +| `[[rules]]` | `[]` | Routing rules: `hosts` (list), `paths` (list), `headers` (list of `{name, value}`, `value` omitted matches any). Fields within one rule are AND'd; separate rules are OR'd. **Leaving this empty means no traffic matches** -- the interceptor never forwards anything and the target never wakes. At least one rule is required; use `hosts = ["*"]` for an explicit catch-all. | +| `cold_start_status_code` / `cold_start_body` / `cold_start_headers` | `503` / `null` / `{}` | Static placeholder response served while the target cold-starts. | +| `cold_start_fallback_service` / `cold_start_fallback_port` | `null` | Service to forward to while cold-starting, as an alternative to a static placeholder. | +| `timeout_readiness` / `timeout_request` / `timeout_response_header` | `null` | Duration strings (e.g. `"30s"`) the interceptor waits at each stage. | +| `scale_target_kind` / `scale_target_api_version` / `scale_target_plural` | `"Deployment"` / `"apps/v1"` / `null` | Only needed when activating a non-Deployment/StatefulSet target. | + +**To configure in `jac.toml` (monolith deploy):** + +```toml +[scale.kubernetes.http_activation] +enabled = true +target_port = 8000 +concurrency_target = 10 +min_replicas = 0 +max_replicas = 3 +cooldown_period = 300 + +[[scale.kubernetes.http_activation.rules]] +hosts = ["app.example.com"] +``` + +**Per-service, in microservice mode:** the same keys apply under `[scale.microservices.services..http_activation]`. The target Service is always the service's own generated Service; it is never user-set. Any key left unset falls back to `[scale.kubernetes.http_activation]`'s value. + +```toml +[scale.microservices.services.jac_coder_sv.http_activation] +enabled = true +target_port = 8000 +concurrency_target = 5 +min_replicas = 0 + +[[scale.microservices.services.jac_coder_sv.http_activation.rules]] +paths = ["/coder"] +``` + +**Traffic topology** + +```mermaid +graph TD + Client["Client"] -->|"HTTP request"| Interceptor["HTTP Add-on Interceptor
(matches InterceptorRoute rules)"] + Interceptor -->|"pending request count"| Scaler["External Scaler"] + Scaler -->|"external-push metric"| Operator["KEDA Operator"] + Operator -->|"scale 0 to 1"| Target["Deployment (0 replicas)"] + Target -->|"pod Ready"| Interceptor + Interceptor -->|"forward held request"| Target + Target -->|"response"| Client +``` + +jac-scale always reconciles the `InterceptorRoute` before the `ScaledObject`, because the external scaler resolves the target Service and scaling metric from the route when KEDA evaluates the trigger. Reconciling in the other order would leave the `ScaledObject` unable to find its metric source. + +!!! warning "Route inbound traffic through the interceptor yourself" + jac-scale creates the `InterceptorRoute` and `ScaledObject`, but does **not** rewire the gateway or Ingress to the interceptor proxy -- they still resolve the app's own Service directly. With `min_replicas = 0`, a request that reaches the Service instead of the interceptor is refused and never wakes the pod. Only enable `http_activation` on a service whose inbound traffic you have already pointed at the KEDA HTTP interceptor proxy. The gateway is exempt and never inherits a shared `enabled = true` default. + +!!! note "Programmatic API for dynamic activation" + A control-plane process that creates and tears down workloads on demand (for example, an IDE-preview orchestrator spinning up a per-session preview) has no fixed target to put in `jac.toml`. For that case, `HTTPActivationSpec` (`jaclang.scale.deploy.autoscale.http_activation`) and `KEDAAutoscaler.apply_http_activation` / `destroy_http_activation` (`jaclang.scale.deploy.autoscale.keda_autoscaler`) remain available as a direct API, unchanged by the `jac.toml` surface above. Use whichever entry point matches your workload's lifecycle: `jac.toml` for a known, standing service; the programmatic API for one created and destroyed at runtime. + --- ### Persistent Storage diff --git a/docs/docs/tutorials/production/kubernetes.md b/docs/docs/tutorials/production/kubernetes.md index 3a6912cccc5..9e015e5d944 100644 --- a/docs/docs/tutorials/production/kubernetes.md +++ b/docs/docs/tutorials/production/kubernetes.md @@ -221,6 +221,36 @@ autoscaler_initial_cooldown = 0 # default 0; seconds after deploy before scal For the full list of autoscaling options (including event triggers, polling intervals, cooldown tuning, and authenticated triggers), see the [Scale Reference](../../reference/plugins/jac-scale-kubernetes.md#autoscaling). +### Scale to zero on an HTTP request + +The KEDA engine above scales on metrics or events. To instead wake a +zero-replica workload on an incoming HTTP request, enable the KEDA HTTP +Add-on activation: + +```toml +[scale.kubernetes.http_activation] +enabled = true +target_port = 8000 +concurrency_target = 10 +min_replicas = 0 # true scale-to-zero +max_replicas = 3 + +[[scale.kubernetes.http_activation.rules]] +hosts = ["app.example.com"] +``` + +This reconciles a KEDA `InterceptorRoute` and `ScaledObject` for the target, +and works for both single-app and per-service (`[scale.microservices.services..http_activation]`) deploys. + +!!! note + This needs the KEDA HTTP Add-on installed alongside KEDA core. With + `min_replicas = 0`, route inbound traffic through the KEDA HTTP interceptor + proxy: jac-scale does not yet rewire the gateway or Ingress to it, so a + request that reaches the app Service directly will not wake a scaled-to-zero + pod. + +See the [Scale Reference](../../reference/plugins/jac-scale-kubernetes.md#http-add-on-activation-scale-to-zero-on-request) for the full HTTP activation config (routing rules, cold-start response, timeouts). + --- ## Local and Remote Clusters diff --git a/jac/jaclang/project/providers.jac b/jac/jaclang/project/providers.jac index 520f904526c..35d408221ae 100644 --- a/jac/jaclang/project/providers.jac +++ b/jac/jaclang/project/providers.jac @@ -13,6 +13,29 @@ import from jaclang.project.pyvenv { glob EMIT_EJECT: str = "eject", EMIT_PUBLISH: str = "publish"; +glob VENV_DEPS_MARKER: str = ".jac-deps-hash"; + + +def _venv_deps_hash(specs: list[str]) -> str { + import hashlib; + return hashlib.sha256("\n".join(sorted(specs)).encode("utf-8")).hexdigest(); +} + + +def venv_deps_satisfied(venv_dir: Path, specs: list[str]) -> bool { + try { + marker = (venv_dir / VENV_DEPS_MARKER).read_text(encoding="utf-8"); + } except OSError { + return False; + } + return marker.strip() == _venv_deps_hash(specs); +} + + +def write_venv_deps_marker(venv_dir: Path, specs: list[str]) { + (venv_dir / VENV_DEPS_MARKER).write_text(_venv_deps_hash(specs), encoding="utf-8"); +} + obj ResolvedDependency { has name: str, version: str = "", @@ -306,8 +329,14 @@ obj PythonProvider(EcosystemProvider) { return self._install_global(specs, ctx); } venv_dir = self._project_venv(config); - ensure_venv(venv_dir, ctx.verbose); is_dry_run = ctx.is_dry_run(); + # Checked before ensure_venv: a venv restored from a sealed bundle + # carries a marker but no host-runnable interpreter, so probing it + # with ensure_venv would classify it as corrupted and delete it. + if not is_dry_run and venv_deps_satisfied(venv_dir, specs) { + return True; + } + ensure_venv(venv_dir, ctx.verbose); (before_snapshot_ok, before_set) = ( self._snapshot_installed(config) if not is_dry_run else (False, set()) ); diff --git a/jac/jaclang/scale/config/plugin_config.jac b/jac/jaclang/scale/config/plugin_config.jac index 16c569bbf37..e1f82247730 100644 --- a/jac/jaclang/scale/config/plugin_config.jac +++ b/jac/jaclang/scale/config/plugin_config.jac @@ -1,3 +1,135 @@ +def _http_activation_nested_schema -> dict[str, any] { + """The `nested` fields shared verbatim by [scale.kubernetes.http_activation] + and [scale.microservices.services.NAME.http_activation] -- factored out so + the two call sites (top-level default, per-service override) can't drift + out of sync on a field's type/default the way two independent copies + would. + """; + return { + "enabled": { + "type": "bool", + "default": False, + "description": "Master switch. Off by default." + }, + "min_replicas": { + "type": "int", + "default": 0, + "description": "Replica floor while inactive. 0 enables true scale-to-zero." + }, + "max_replicas": { + "type": "int", + "default": 1, + "description": "Replica ceiling once activated." + }, + "polling_interval": { + "type": "int", + "default": 30, + "description": "How often KEDA evaluates the HTTP scaling metric, in seconds." + }, + "cooldown_period": { + "type": "int", + "default": 300, + "description": "Seconds of inactivity before scaling back to min_replicas." + }, + "target_port": { + "type": "int", + "default": None, + "description": "Container port on the target's own generated Service. Exactly one of target_port/target_port_name must be set when enabled." + }, + "target_port_name": { + "type": "string", + "default": None, + "description": "Named container port on the target's own generated Service, as an alternative to target_port." + }, + "concurrency_target": { + "type": "int", + "default": None, + "description": "In-flight-request concurrency target. Exactly one of concurrency_target/request_rate_target must be set when enabled." + }, + "request_rate_target": { + "type": "int", + "default": None, + "description": "Requests-per-window target, as an alternative to concurrency_target." + }, + "request_rate_window": { + "type": "string", + "default": "1m", + "description": "Window duration for request_rate_target." + }, + "request_rate_granularity": { + "type": "string", + "default": "1s", + "description": "Sampling granularity for request_rate_target." + }, + "cold_start_status_code": { + "type": "int", + "default": 503, + "description": "HTTP status the interceptor returns while the target cold-starts, if a placeholder response is configured." + }, + "cold_start_body": { + "type": "string", + "default": None, + "description": "Static placeholder response body served while the target cold-starts." + }, + "cold_start_body_config_map": { + "type": "dict", + "default": None, + "description": "ConfigMap reference to source the placeholder response body from, as an alternative to cold_start_body." + }, + "cold_start_headers": { + "type": "dict", + "default": {}, + "description": "Extra headers on the placeholder response." + }, + "cold_start_fallback_service": { + "type": "string", + "default": None, + "description": "Service name to forward requests to while the target cold-starts, as an alternative to a static placeholder response." + }, + "cold_start_fallback_port": { + "type": "int", + "default": None, + "description": "Port on cold_start_fallback_service." + }, + "timeout_readiness": { + "type": "string", + "default": None, + "description": "Duration (e.g. \"30s\") the interceptor waits for the target to become ready after scaling up." + }, + "timeout_request": { + "type": "string", + "default": None, + "description": "Duration the interceptor waits for the forwarded request to complete." + }, + "timeout_response_header": { + "type": "string", + "default": None, + "description": "Duration the interceptor waits for response headers from the target." + }, + "scale_target_kind": { + "type": "string", + "default": "Deployment", + "description": "Kind of the scale target. Deployment/StatefulSet work directly; other kinds require scale_target_plural." + }, + "scale_target_api_version": { + "type": "string", + "default": "apps/v1", + "description": "apiVersion of the scale target, used when scale_target_plural is set for a custom-resource target." + }, + "scale_target_plural": { + "type": "string", + "default": None, + "description": "Plural resource name for a custom-resource scale target (e.g. \"rollouts\"), enabling existence validation before writing anything." + }, + "rules": { + "type": "list", + "default": [], + "description": "[[...rules]] array-of-tables. Each entry: hosts (list[str]), paths (list[str]), headers (list of {name, value?}). Fields within one rule are AND'd; separate rules are OR'd. At least one rule is required -- leaving this empty raises an error at build time, since no traffic would ever match and the target would never wake. Use hosts = [\"*\"] for an explicit catch-all." + } + }; +} + + class JacScalePluginConfig { static def get_plugin_metadata -> dict[str, any] { return { @@ -226,7 +358,12 @@ class JacScalePluginConfig { "default": [], "description": "(KEDA only) Additional KEDA trigger dicts applied to every service. Each entry: `type` (str), `metadata` (dict[str,str]), optional `name` (str), optional `auth.secret_refs` (dict). For per-service triggers use [[scale.microservices.services.NAME.triggers]] instead." }, - + "http_activation": { + "type": "dict", + "default": {}, + "description": "KEDA HTTP Add-on scale-to-zero activation for the monolith app (or shared defaults inherited by microservices unless overridden per-service). Requires the KEDA HTTP Add-on installed separately from core KEDA. For per-service activation use [scale.microservices.services.NAME.http_activation] instead.", + "nested": _http_activation_nested_schema() + } } }, "secrets": { @@ -299,7 +436,7 @@ class JacScalePluginConfig { "services": { "type": "dict", "default": {}, - "description": "Per-service overrides keyed by module name. Subkeys (all optional): `rpc_timeout` (float, default 10s, inter-service sv-import calls), `http_forward_timeout` (float, default 30s, gateway-to-service forward), `replicas` (int, default 1, K8s Deployment.spec.replicas), `cpu_request` / `cpu_limit` (str, e.g. \"100m\", K8s container resources), `memory_request` / `memory_limit` (str, e.g. \"128Mi\"), `env` (dict[str,str], extra container env vars merged with auto-set JAC_SV_NAME), `image_tag` (str, override global image tag for canary), and nested `hpa` / `pdb` sub-tables. `hpa` keys: `enabled` (bool, default true), `min` (int, default 1), `max` (int, default 3), `cpu_target` (int percent, default 70), `memory_target` (int percent of memory request, default 80). `pdb` keys: `enabled` (bool, default true), `max_unavailable` (int, default 1). `deployment_overlay` (table): a raw Deployment-manifest fragment merged into the generated manifest at build time - maps merge recursively, lists whose members all carry `name` (containers, volumes, env, initContainers, volumeMounts) merge by that name, anything else replaces; identity fields (metadata.name/namespace, selector, the app/managed template labels) are reasserted after the merge. The main container's `name` is the k8s-safe service name (e.g. `builder-sv`; gateway is `gateway`). Use it for nodeSelector, probe tuning, extra volumes/env/init containers - anything the schema has no first-class key for - so pods are right on the FIRST rollout instead of being kubectl-patched into a second one. Overlays add and override only: there is no delete directive (setting a key to null stores null rather than removing the generated field). Gateway uses the `__gateway__` key. `[[services.NAME.triggers]]` array (KEDA only): per-service event-driven triggers; each entry has `type` (str), `metadata` (dict[str,str]), optional `name` (str), optional `auth.secret_refs` (dict). Requires `autoscaler_engine = \"keda\"` in [scale.kubernetes]. Example: [scale.microservices.services.llm_app] rpc_timeout = 120.0, replicas = 2, cpu_limit = \"2000m\", memory_limit = \"4Gi\", env = { LOG_LEVEL = \"DEBUG\" }, [scale.microservices.services.llm_app.hpa] max = 20, cpu_target = 60" + "description": "Per-service overrides keyed by module name. Subkeys (all optional): `rpc_timeout` (float, default 10s, inter-service sv-import calls), `http_forward_timeout` (float, default 30s, gateway-to-service forward), `replicas` (int, default 1, K8s Deployment.spec.replicas), `cpu_request` / `cpu_limit` (str, e.g. \"100m\", K8s container resources), `memory_request` / `memory_limit` (str, e.g. \"128Mi\"), `env` (dict[str,str], extra container env vars merged with auto-set JAC_SV_NAME), `image_tag` (str, override global image tag for canary), and nested `hpa` / `pdb` sub-tables. `hpa` keys: `enabled` (bool, default true), `min` (int, default 1), `max` (int, default 3), `cpu_target` (int percent, default 70), `memory_target` (int percent of memory request, default 80), `behavior` (table, default `{}`): a raw HPA `behavior` fragment (`scaleUp`/`scaleDown`, each with `stabilizationWindowSeconds`/`policies`/`selectPolicy`) deep-merged over the generated scale-rate defaults - same merge semantics as `deployment_overlay` (maps merge recursively, the `policies` list has no `name` key so it replaces wholesale rather than merging by entry); applies to both \"hpa\" and \"keda\" engines since both route through the same `Autoscaler._build_behavior` (for keda it only governs scaling above zero replicas - the drop to zero remains controlled by `autoscaler_cooldown`, the ScaledObject's cooldownPeriod). Use it to slow scale-down for connection-heavy services (e.g. `[services.NAME.hpa.behavior.scaleDown] stabilizationWindowSeconds = 600, selectPolicy = 'Min', policies = [{ type = 'Pods', value = 2, periodSeconds = 300 }]`) without imposing the same rate limit on every service via the app-global `autoscaler_cooldown`. `pdb` keys: `enabled` (bool, default true), `max_unavailable` (int, default 1). `deployment_overlay` (table): a raw Deployment-manifest fragment merged into the generated manifest at build time - maps merge recursively, lists whose members all carry `name` (containers, volumes, env, initContainers, volumeMounts) merge by that name, anything else replaces; identity fields (metadata.name/namespace, selector, the app/managed template labels) are reasserted after the merge. The main container's `name` is the k8s-safe service name (e.g. `builder-sv`; gateway is `gateway`). Use it for nodeSelector, probe tuning, extra volumes/env/init containers - anything the schema has no first-class key for - so pods are right on the FIRST rollout instead of being kubectl-patched into a second one. Overlays add and override only: there is no delete directive (setting a key to null stores null rather than removing the generated field). Gateway uses the `__gateway__` key. `[[services.NAME.triggers]]` array (KEDA only): per-service event-driven triggers; each entry has `type` (str), `metadata` (dict[str,str]), optional `name` (str), optional `auth.secret_refs` (dict). Requires `autoscaler_engine = \"keda\"` in [scale.kubernetes]. `http_activation` sub-table (KEDA HTTP Add-on only): per-service HTTP scale-to-zero activation, same keys as [scale.kubernetes.http_activation] (`enabled`, `min_replicas`, `max_replicas`, `target_port`/`target_port_name`, `concurrency_target`/`request_rate_target`, `[[rules]]`, cold-start and timeout keys, `scale_target_kind`/`scale_target_api_version`/`scale_target_plural`); the target Service is always the service's own generated Service, never user-set. Falls back to [scale.kubernetes.http_activation]'s values for any key left unset. Example: [scale.microservices.services.llm_app] rpc_timeout = 120.0, replicas = 2, cpu_limit = \"2000m\", memory_limit = \"4Gi\", env = { LOG_LEVEL = \"DEBUG\" }, [scale.microservices.services.llm_app.hpa] max = 20, cpu_target = 60" }, "rate_limit": { "type": "dict", diff --git a/jac/jaclang/scale/deploy/autoscale/autoscaler.jac b/jac/jaclang/scale/deploy/autoscale/autoscaler.jac index 7e07b858ebe..c1022fb0dac 100644 --- a/jac/jaclang/scale/deploy/autoscale/autoscaler.jac +++ b/jac/jaclang/scale/deploy/autoscale/autoscaler.jac @@ -26,7 +26,8 @@ obj AutoscalerSpec { cooldown_period: int = 300, initial_cooldown_period: int = 0, scale_up_stabilization: int = 60, - scale_up_max_pods: int = 2; + scale_up_max_pods: int = 2, + behavior_overlay: dict[str, any] = {}; } @@ -36,7 +37,7 @@ class Autoscaler { } static def _build_behavior(spec: AutoscalerSpec) -> dict[str, any] { - return { + default: dict[str, any] = { "scaleUp": { "stabilizationWindowSeconds": min( 3600, max(0, spec.scale_up_stabilization) @@ -54,6 +55,21 @@ class Autoscaler { "policies": [{"type": "Percent", "value": 50, "periodSeconds": 60}] } }; + if not spec.behavior_overlay { + return default; + } + import from jaclang.scale.deploy.target.kubernetes.manifest_builder { + merge_manifest_overlay + } + return merge_manifest_overlay(default, spec.behavior_overlay); + } + + def _build_manifests( + self: Autoscaler, spec: AutoscalerSpec + ) -> list[dict[str, any]] { + raise NotImplementedError( + "_build_manifests() must be implemented by a concrete Autoscaler subclass" + ); } def apply(self: Autoscaler, spec: AutoscalerSpec) -> bool { @@ -62,7 +78,7 @@ class Autoscaler { ); } - def destroy(self: Autoscaler, autoscaler_name: str, namespace: str) { + def destroy(self: Autoscaler, app_name: str, namespace: str) { raise NotImplementedError( "destroy() must be implemented by a concrete Autoscaler subclass" ); diff --git a/jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.impl.jac b/jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.impl.jac index 1fc12f1c43b..70750dc1d76 100644 --- a/jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.impl.jac +++ b/jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.impl.jac @@ -39,26 +39,31 @@ impl HPAAutoscaler._build_manifests{ } ); } - return { - "apiVersion": "autoscaling/v2", - "kind": "HorizontalPodAutoscaler", - "metadata": { - "name": hpa_name, - "namespace": spec.namespace, - "labels": {"app": spec.scale_target_name, ** Autoscaler._managed_labels()} - }, - "spec": { - "scaleTargetRef": { - "apiVersion": "apps/v1", - "kind": "Deployment", - "name": spec.scale_target_name + return [ + { + "apiVersion": "autoscaling/v2", + "kind": "HorizontalPodAutoscaler", + "metadata": { + "name": hpa_name, + "namespace": spec.namespace, + "labels": { + "app": spec.scale_target_name, + ** Autoscaler._managed_labels() + } }, - "minReplicas": spec.min_replicas, - "maxReplicas": spec.max_replicas, - "metrics": metrics, - "behavior": Autoscaler._build_behavior(spec) + "spec": { + "scaleTargetRef": { + "apiVersion": "apps/v1", + "kind": "Deployment", + "name": spec.scale_target_name + }, + "minReplicas": spec.min_replicas, + "maxReplicas": spec.max_replicas, + "metrics": metrics, + "behavior": Autoscaler._build_behavior(spec) + } } - }; + ]; } impl HPAAutoscaler.apply{ @@ -70,7 +75,7 @@ impl HPAAutoscaler.apply{ ); } } - manifest = self._build_manifests(spec); + manifest = self._build_manifests(spec)[0]; name = manifest["metadata"]["name"]; namespace = manifest["metadata"]["namespace"]; @@ -111,7 +116,7 @@ impl HPAAutoscaler.destroy{ autoscaling_v2 = AutoscalingV2Api(); delete_if_exists( autoscaling_v2.delete_namespaced_horizontal_pod_autoscaler, - autoscaler_name, + self.resource_name_for(app_name), namespace, 'HorizontalPodAutoscaler' ); diff --git a/jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.jac b/jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.jac index 946441fdfd4..47866e25700 100644 --- a/jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.jac +++ b/jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.jac @@ -13,9 +13,9 @@ import from jaclang.scale.deploy.target.kubernetes.utils.kubernetes_utils { obj HPAAutoscaler(Autoscaler) { has logger: (Logger | None) = None; - def _build_manifests(spec: AutoscalerSpec) -> dict[str, any]; + def _build_manifests(spec: AutoscalerSpec) -> list[dict[str, any]]; def apply(spec: AutoscalerSpec) -> bool; - def destroy(autoscaler_name: str, namespace: str) -> None; + def destroy(app_name: str, namespace: str) -> None; def destroy_collection(namespace: str, label_selector: str) -> None; def resource_name_for(target: str) -> str; } diff --git a/jac/jaclang/scale/deploy/autoscale/http_activation.jac b/jac/jaclang/scale/deploy/autoscale/http_activation.jac index 2c7f136771c..42725e2c29c 100644 --- a/jac/jaclang/scale/deploy/autoscale/http_activation.jac +++ b/jac/jaclang/scale/deploy/autoscale/http_activation.jac @@ -25,7 +25,7 @@ obj HTTPConcurrencyMetric { obj HTTPRequestRateMetric { has target_value: int, - window: str = "1m", + rate_window: str = "1m", granularity: str = "1s"; } diff --git a/jac/jaclang/scale/deploy/autoscale/http_activation_config.jac b/jac/jaclang/scale/deploy/autoscale/http_activation_config.jac new file mode 100644 index 00000000000..ecf25471077 --- /dev/null +++ b/jac/jaclang/scale/deploy/autoscale/http_activation_config.jac @@ -0,0 +1,188 @@ +import from jaclang.scale.deploy.autoscale.http_activation { + HTTPActivationSpec, + HTTPServiceTarget, + HTTPConcurrencyMetric, + HTTPRequestRateMetric, + HTTPStaticResponse, + HTTPColdStartSpec, + HTTPTimeoutSpec, + HTTPRoutingRule, + HTTPPathMatch, + HTTPHeaderMatch +} +import from jaclang.scale.deploy.autoscale.keda_autoscaler { KEDAAutoscaler } +import from jaclang.scale._optdeps.kubernetes { ApiException } + +def build_http_activation_spec( + cfg: dict[str, any], scale_target_name: str, namespace: str, service_name: str +) -> HTTPActivationSpec | None { + if not cfg.get("enabled", False) { + return None; + } + + target_port = cfg.get("target_port"); + target_port_name = cfg.get("target_port_name"); + if target_port is not None and target_port_name is not None { + raise ValueError( + "[*.http_activation]: set exactly one of target_port or " + "target_port_name, not both." + ); + } + if target_port is None and target_port_name is None { + raise ValueError( + "[*.http_activation]: set exactly one of target_port or " + "target_port_name." + ); + } + + concurrency_target = cfg.get("concurrency_target"); + request_rate_target = cfg.get("request_rate_target"); + if concurrency_target is None and request_rate_target is None { + raise ValueError( + "[*.http_activation]: set at least one of concurrency_target or " + "request_rate_target." + ); + } + + concurrency = None; + if concurrency_target is not None { + concurrency = HTTPConcurrencyMetric(target_value=concurrency_target); + } + request_rate = None; + if request_rate_target is not None { + request_rate = HTTPRequestRateMetric( + target_value=request_rate_target, + rate_window=cfg.get("request_rate_window", "1m"), + granularity=cfg.get("request_rate_granularity", "1s") + ); + } + + raw_rules: list[dict[str, any]] = cfg.get("rules", []); + if not raw_rules { + raise ValueError( + "[*.http_activation]: at least one [[rules]] entry is required. " + "Leaving rules empty means no traffic ever matches, so the " + "interceptor never forwards anything and the target never wakes; " + "use hosts = [\"*\"] for an explicit catch-all." + ); + } + + rules: list[HTTPRoutingRule] = []; + for rule in raw_rules { + headers: list[HTTPHeaderMatch] = []; + for h in rule.get("headers", []) { + if "name" not in h { + raise ValueError( + "[*.http_activation]: a [[rules.headers]] entry is " + f"missing its required 'name' key: {h}." + ); + } + headers.append(HTTPHeaderMatch(name=h["name"], value=h.get("value"))); + } + rules.append( + HTTPRoutingRule( + hosts=list(rule.get("hosts", [])), + paths=[HTTPPathMatch(value=p) for p in rule.get("paths", [])], + headers=headers + ) + ); + } + + placeholder = None; + has_placeholder = ( + "cold_start_body" in cfg + or "cold_start_body_config_map" in cfg + or "cold_start_headers" in cfg + or "cold_start_status_code" in cfg + ); + if has_placeholder { + placeholder = HTTPStaticResponse( + status_code=cfg.get("cold_start_status_code", 503), + body=cfg.get("cold_start_body"), + body_from_config_map=cfg.get("cold_start_body_config_map"), + headers=cfg.get("cold_start_headers", {}) + ); + } + fallback_service = None; + if "cold_start_fallback_service" in cfg { + fallback_service = HTTPServiceTarget( + service=str(cfg.get("cold_start_fallback_service", "")), + port=cfg.get("cold_start_fallback_port") + ); + } + cold_start = None; + if placeholder is not None or fallback_service is not None { + cold_start = HTTPColdStartSpec( + placeholder=placeholder, fallback_service=fallback_service + ); + } + + timeouts = None; + has_timeouts = ( + "timeout_readiness" in cfg + or "timeout_request" in cfg + or "timeout_response_header" in cfg + ); + if has_timeouts { + timeouts = HTTPTimeoutSpec( + readiness=cfg.get("timeout_readiness"), + request=cfg.get("timeout_request"), + response_header=cfg.get("timeout_response_header") + ); + } + + return HTTPActivationSpec( + name=scale_target_name, + namespace=namespace, + scale_target_name=scale_target_name, + target=HTTPServiceTarget( + service=service_name, port=target_port, port_name=target_port_name + ), + scale_target_api_version=cfg.get("scale_target_api_version", "apps/v1"), + scale_target_kind=cfg.get("scale_target_kind", "Deployment"), + scale_target_plural=cfg.get("scale_target_plural"), + rules=rules, + min_replicas=cfg.get("min_replicas", 0), + max_replicas=cfg.get("max_replicas", 1), + polling_interval=cfg.get("polling_interval", 30), + cooldown_period=cfg.get("cooldown_period", 300), + concurrency=concurrency, + request_rate=request_rate, + cold_start=cold_start, + timeouts=timeouts + ); +} + +def apply_http_activation_for_target( + cfg: dict[str, any], + scale_target_name: str, + namespace: str, + service_name: str, + keda: KEDAAutoscaler +) -> bool | None { + spec = build_http_activation_spec(cfg, scale_target_name, namespace, service_name); + if spec is None { + return None; + } + return keda.apply_http_activation(spec); +} + +def destroy_http_activation_for_target( + scale_target_name: str, namespace: str, keda: KEDAAutoscaler +) { + try { + keda.destroy_http_activation(scale_target_name, namespace); + } except ApiException as e { + if e?.status != 403 { + raise; + } + if keda.logger { + keda.logger.warn( + "Failed to clean up HTTP activation resources for " + f"'{scale_target_name}' in '{namespace}': {e}. Continuing " + "with the rest of teardown; manual cleanup of the " + "InterceptorRoute/ScaledObject may be required." + ); + } + } +} diff --git a/jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac b/jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac index ddc931a25b1..350cfe8abed 100644 --- a/jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac +++ b/jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac @@ -59,6 +59,13 @@ impl KEDAAutoscaler.preflight{ KEDAAutoscaler._preflight_cache[cluster_key] = False; return install_warning; } + if e.status == 403 { + return [ + "Permission denied (403) discovering KEDA core " + "(keda.sh/v1alpha1 ScaledObject). Grant RBAC list/get on " + "this resource to the jac-scale service account." + ]; + } raise; } } @@ -98,10 +105,22 @@ impl KEDAAutoscaler._build_trigger{ return keda_trigger; } +impl KEDAAutoscaler._owner_label_value{ + if len(name) <= 63 { + return name; + } + return f"so-{hashlib.sha256(name.encode()).hexdigest()[:16]}"; +} + +impl KEDAAutoscaler._scaled_object_name{ + return spec.autoscaler_name or self.resource_name_for(app_name); +} + impl KEDAAutoscaler._build_manifests{ - scaled_obj_name = spec.autoscaler_name or f"{spec.scale_target_name}-scaledobject"; - keda_triggers: list[dict[str, any]] = []; app_name = spec.app_name or spec.scale_target_name; + self._validate_triggers(spec, app_name); + scaled_obj_name = self._scaled_object_name(spec, app_name); + keda_triggers: list[dict[str, any]] = []; for (i, trigger) in enumerate(spec.triggers) { keda_triggers.append(self._build_trigger(trigger, app_name, i)); } @@ -140,30 +159,36 @@ impl KEDAAutoscaler._build_manifests{ if spec.initial_cooldown_period > 0 { scaled_obj_spec["initialCooldownPeriod"] = spec.initial_cooldown_period; } - return { - "name": scaled_obj_name, - "body": { - "apiVersion": "keda.sh/v1alpha1", - "kind": "ScaledObject", - "metadata": { - "name": scaled_obj_name, - "namespace": spec.namespace, - "labels": { - "app": spec.scale_target_name, - ** Autoscaler._managed_labels() - } - }, - "spec": scaled_obj_spec - } + + scaled_object = { + "apiVersion": "keda.sh/v1alpha1", + "kind": "ScaledObject", + "metadata": { + "name": scaled_obj_name, + "namespace": spec.namespace, + "labels": {"app": spec.scale_target_name, ** Autoscaler._managed_labels()} + }, + "spec": scaled_obj_spec }; + + manifests: list[dict[str, any]] = [scaled_object]; + for (i, trigger) in enumerate(spec.triggers) { + auth_manifest = self._build_trigger_auth_manifest( + trigger, spec.namespace, app_name, i, scaled_obj_name + ); + if auth_manifest is not None { + manifests.append(auth_manifest); + } + } + return manifests; } -impl KEDAAutoscaler._apply_trigger_auth{ +impl KEDAAutoscaler._build_trigger_auth_manifest{ if not trigger.auth { - return; + return None; } if not trigger.auth.secret_refs { - return; + return None; } auth_name = self._trigger_auth_name(app_name, trigger, trigger_index); secret_target_refs: list[dict[str, any]] = []; @@ -179,40 +204,43 @@ impl KEDAAutoscaler._apply_trigger_auth{ {"parameter": param, "name": ref.get("name", ""), "key": ref.get("key", "")} ); } - auth_body = { + return { "apiVersion": "keda.sh/v1alpha1", "kind": "TriggerAuthentication", "metadata": { "name": auth_name, "namespace": namespace, - "labels": Autoscaler._managed_labels() + "labels": { + "app": app_name, + "jac-scale/owner": self._owner_label_value(owner), + ** Autoscaler._managed_labels() + } }, "spec": {"secretTargetRef": secret_target_refs} }; +} + +impl KEDAAutoscaler._apply_custom_object{ try { api.get_namespaced_custom_object( - group="keda.sh", - version="v1alpha1", - namespace=namespace, - plural="triggerauthentications", - name=auth_name + group=group, version=version, namespace=namespace, plural=plural, name=name ); api.patch_namespaced_custom_object( - group="keda.sh", - version="v1alpha1", + group=group, + version=version, namespace=namespace, - plural="triggerauthentications", - name=auth_name, - body=auth_body + plural=plural, + name=name, + body=body ); } except ApiException as e { if e.status == 404 { api.create_namespaced_custom_object( - group="keda.sh", - version="v1alpha1", + group=group, + version=version, namespace=namespace, - plural="triggerauthentications", - body=auth_body + plural=plural, + body=body ); } else { raise; @@ -222,7 +250,7 @@ impl KEDAAutoscaler._apply_trigger_auth{ impl KEDAAutoscaler.apply{ app_name = spec.app_name or spec.scale_target_name; - self._validate_triggers(spec, app_name); + manifests = self._build_manifests(spec); warnings = self.preflight(); if warnings { if self.logger { @@ -244,53 +272,128 @@ impl KEDAAutoscaler.apply{ } } - for (i, trigger) in enumerate(spec.triggers) { - self._apply_trigger_auth(api, trigger, spec.namespace, app_name, i); + known_kinds = [kind for (kind, _) in KEDAAutoscaler._APPLY_ORDER]; + unknown_kinds = [ + manifest["kind"] + for manifest in manifests + if manifest["kind"] not in known_kinds + ]; + if unknown_kinds { + raise ValueError( + f"_build_manifests returned unhandled kind(s) {unknown_kinds}; " + "add them to KEDAAutoscaler._APPLY_ORDER." + ); } - manifests = self._build_manifests(spec); - name = manifests["name"]; - body = manifests["body"]; + for (kind, plural) in KEDAAutoscaler._APPLY_ORDER { + for manifest in manifests { + if manifest["kind"] == kind { + self._apply_custom_object( + api, + "keda.sh", + "v1alpha1", + spec.namespace, + plural, + manifest["metadata"]["name"], + manifest + ); + } + } + } + + owner = self._scaled_object_name(spec, app_name); + desired_auth_names: set[str] = set(); + for manifest in manifests { + if manifest["kind"] == "TriggerAuthentication" { + desired_auth_names.add(manifest["metadata"]["name"]); + } + } + self._prune_orphan_trigger_auths(api, spec.namespace, owner, desired_auth_names); + return True; +} + +impl KEDAAutoscaler._prune_orphan_trigger_auths{ + selector = f"jac-scale/owner={self._owner_label_value(owner)},managed=jac-scale"; try { - api.get_namespaced_custom_object( - group="keda.sh", - version="v1alpha1", - namespace=spec.namespace, - plural="scaledobjects", - name=name - ); - api.patch_namespaced_custom_object( + existing = api.list_namespaced_custom_object( group="keda.sh", version="v1alpha1", - namespace=spec.namespace, - plural="scaledobjects", - name=name, - body=body + namespace=namespace, + plural="triggerauthentications", + label_selector=selector ); } except ApiException as e { - if e.status == 404 { - api.create_namespaced_custom_object( + if e.status not in [404, 422] { + raise; + } + return; + } + for item in existing.get("items", []) { + item_name = item["metadata"]["name"]; + if item_name in desired_auth_names { + continue; + } + try { + api.delete_namespaced_custom_object( group="keda.sh", version="v1alpha1", - namespace=spec.namespace, - plural="scaledobjects", - body=body + namespace=namespace, + plural="triggerauthentications", + name=item_name ); - } else { - raise; + } except ApiException as e { + if e.status not in [404, 422] { + raise; + } } } - return True; } impl KEDAAutoscaler.destroy{ api = self._custom_api or client.CustomObjectsApi(); + scaled_object_name = self.resource_name_for(app_name); + selector = f"jac-scale/owner={self._owner_label_value( + scaled_object_name + )},managed=jac-scale"; + try { + existing = api.list_namespaced_custom_object( + group="keda.sh", + version="v1alpha1", + namespace=namespace, + plural="triggerauthentications", + label_selector=selector + ); + listing: dict[str, list[dict[str, any]]] = existing + if isinstance(existing, dict) + else {}; + for item in listing.get("items", []) { + item_name = item["metadata"]["name"]; + try { + api.delete_namespaced_custom_object( + group="keda.sh", + version="v1alpha1", + namespace=namespace, + plural="triggerauthentications", + name=item_name + ); + } except ApiException as e { + if e.status not in [404, 422] { + raise; + } + } + } + } except ApiException as e { + if e.status not in [404, 422] { + raise; + } + } + try { api.delete_namespaced_custom_object( group="keda.sh", version="v1alpha1", namespace=namespace, plural="scaledobjects", - name=autoscaler_name + name=scaled_object_name ); } except ApiException as e { if e.status not in [404, 422] { @@ -306,6 +409,9 @@ impl KEDAAutoscaler.destroy_collection{ ("keda.sh", "v1alpha1", "triggerauthentications"), ("http.keda.sh", "v1beta1", "interceptorroutes") ] { + tolerated = [404, 422, 403] + if (group == "http.keda.sh" or best_effort) + else [404, 422]; try { items = api.list_namespaced_custom_object( group=group, @@ -325,13 +431,13 @@ impl KEDAAutoscaler.destroy_collection{ name=item_name ); } except ApiException as e { - if e.status not in [404, 422] { + if e.status not in tolerated { raise; } } } } except ApiException as e { - if e.status not in [404, 422] { + if e.status not in tolerated { raise; } } @@ -365,33 +471,22 @@ impl KEDAAutoscaler.http_scaled_object_name_for{ impl KEDAAutoscaler.apply_http_activation{ self._validate_http_activation_spec(spec); - cluster_key = self._get_cluster_key(); - api = self._custom_api or client.CustomObjectsApi(); - if cluster_key not in KEDAAutoscaler._http_preflight_cache { - try { - api.list_cluster_custom_object( - group="http.keda.sh", version="v1beta1", plural="interceptorroutes" - ); - KEDAAutoscaler._http_preflight_cache[cluster_key] = True; - } except ApiException as e { - if e.status in [404, 422] { - KEDAAutoscaler._http_preflight_cache[cluster_key] = False; - } else { - raise; - } - } + caps = self.discover_capabilities(); + if caps.errors { + raise ValueError( + "KEDA HTTP Add-on capability check failed:\n" + "\n".join(caps.errors) + ); } - if not KEDAAutoscaler._http_preflight_cache[cluster_key] { + if not caps.core_available or not caps.http_addon_available { if self.logger { - self.logger.warn( - "KEDA HTTP Add-on CRDs not found on cluster. Install the HTTP " - "Add-on from the official docs, then redeploy: " - "https://keda.sh/docs/latest/deploy/#http-add-on" - ); + for warning in caps.warnings { + self.logger.warn(warning); + } } return False; } + api = self._custom_api or client.CustomObjectsApi(); apps_api = self._apps_api or client.AppsV1Api(); core_api = self._core_api or client.CoreV1Api(); @@ -520,6 +615,9 @@ impl KEDAAutoscaler.apply_http_activation{ impl KEDAAutoscaler.destroy_http_activation{ api = self._custom_api or client.CustomObjectsApi(); + + interceptor_error: (ApiException | None) = None; + scaled_object_error: (ApiException | None) = None; try { api.delete_namespaced_custom_object( group="http.keda.sh", @@ -530,7 +628,7 @@ impl KEDAAutoscaler.destroy_http_activation{ ); } except ApiException as e { if e.status not in [404, 422] { - raise; + interceptor_error = e; } } try { @@ -543,9 +641,20 @@ impl KEDAAutoscaler.destroy_http_activation{ ); } except ApiException as e { if e.status not in [404, 422] { - raise; + scaled_object_error = e; } } + + if interceptor_error is not None and interceptor_error.status != 403 { + raise interceptor_error; + } + if scaled_object_error is not None and scaled_object_error.status != 403 { + raise scaled_object_error; + } + pending_error = interceptor_error or scaled_object_error; + if pending_error is not None { + raise pending_error; + } } impl KEDAAutoscaler._build_routing_rules{ @@ -594,7 +703,7 @@ impl KEDAAutoscaler._build_interceptor_route{ } elif spec.request_rate is not None { scaling_metric["requestRate"] = { "targetValue": spec.request_rate.target_value, - "window": spec.request_rate.window, + "window": spec.request_rate.rate_window, "granularity": spec.request_rate.granularity }; } @@ -681,7 +790,7 @@ impl KEDAAutoscaler._build_http_scaled_object{ trigger_metadata["targetValue"] = str(spec.concurrency.target_value); } elif spec.request_rate is not None { trigger_metadata["targetValue"] = str(spec.request_rate.target_value); - trigger_metadata["window"] = spec.request_rate.window; + trigger_metadata["window"] = spec.request_rate.rate_window; trigger_metadata["granularity"] = spec.request_rate.granularity; } @@ -741,3 +850,170 @@ impl KEDAAutoscaler._validate_http_activation_spec{ ); } } + +impl KEDAAutoscaler._check_service_address{ + (host, _, __) = address.partition(":"); + parts = host.split("."); + name = parts[0]; + namespace = parts[1] if len(parts) > 1 else "keda"; + try { + core_api.read_namespaced_service(name=name, namespace=namespace); + return None; + } except ApiException as e { + if e.status == 403 { + return ( + f"Permission denied (403) checking Service '{name}' in " + f"namespace '{namespace}'. Verify RBAC grants get/list on " + f"services for the jac-scale service account." + ); + } + if e.status in [404, 422] { + return f"Service '{name}' not found in namespace '{namespace}' (address: '{address}')."; + } + raise; + } +} + +impl KEDAAutoscaler._capabilities_cache_key{ + return ( + f"{self._get_cluster_key()}::{self.http_scaler_address}" + f"::{self.interceptor_service_address}::{id(self._custom_api)}" + f"::{id(self._core_api)}" + ); +} + +impl KEDAAutoscaler.discover_capabilities{ + cluster_key = self._capabilities_cache_key(); + if not refresh and cluster_key in self._capabilities_cache { + return self._capabilities_cache[cluster_key]; + } + + api = self._custom_api or client.CustomObjectsApi(); + core_api = self._core_api or client.CoreV1Api(); + caps = KEDACapabilities(); + + keda_docs_url = "https://keda.sh/http-add-on/0.15/getting-started/"; + http_addon_docs_url = "https://keda.sh/http-add-on/0.15/getting-started/"; + install_core_cmd = ( + "helm repo add kedacore https://kedacore.github.io/charts && " + "helm repo update && helm install keda kedacore/keda -n keda " + f"--create-namespace --wait (see {keda_docs_url} for current instructions)" + ); + install_addon_cmd = ( + "helm install http-add-on kedacore/keda-add-ons-http -n keda --wait " + f"(see {http_addon_docs_url} for current instructions)" + ); + upgrade_addon_cmd = ( + "helm upgrade http-add-on kedacore/keda-add-ons-http -n keda --wait " + f"(see {http_addon_docs_url} for current instructions)" + ); + + try { + api.list_cluster_custom_object( + group="keda.sh", version="v1alpha1", plural="scaledobjects" + ); + caps.core_available = True; + } except ApiException as e { + if e.status == 403 { + caps.errors.append( + "Permission denied (403) discovering KEDA core " + "(keda.sh/v1alpha1 ScaledObject). Grant RBAC list/get on " + "this resource to the jac-scale service account." + ); + } elif e.status in [404, 422] { + caps.warnings.append( + f"KEDA core not found on cluster. Install it with: {install_core_cmd}" + ); + } else { + raise; + } + } + + http_addon_status: (int | None) = None; + for version in SUPPORTED_HTTP_INTERCEPTOR_API_VERSIONS { + try { + api.list_cluster_custom_object( + group="http.keda.sh", version=version, plural="interceptorroutes" + ); + caps.http_addon_available = True; + caps.interceptor_route_api_version = version; + break; + } except ApiException as e { + http_addon_status = e.status; + if e.status == 403 { + caps.errors.append( + "Permission denied (403) discovering the KEDA HTTP Add-on " + f"(http.keda.sh/{version} InterceptorRoute). Grant RBAC " + "list/get on this resource to the jac-scale service account." + ); + break; + } elif e.status not in [404, 422] { + raise; + } + } + } + + if not caps.http_addon_available and http_addon_status in [404, 422] { + try { + api.list_cluster_custom_object( + group="http.keda.sh", version="v1alpha1", plural="httpscaledobjects" + ); + caps.warnings.append( + "Legacy KEDA HTTP Add-on detected (http.keda.sh/v1alpha1 " + "HTTPScaledObject); jac-scale requires v1beta1 " + f"InterceptorRoute. Upgrade with: {upgrade_addon_cmd}" + ); + } except ApiException as e { + if e.status == 403 { + caps.errors.append( + "Permission denied (403) probing the legacy KEDA HTTP " + "Add-on API (http.keda.sh/v1alpha1 HTTPScaledObject) " + "while checking for an outdated install. Grant RBAC " + "list/get on this resource to the jac-scale service " + "account." + ); + } elif e.status in [404, 422] { + caps.warnings.append( + f"KEDA HTTP Add-on not found on cluster. Install it with: {install_addon_cmd}" + ); + } else { + raise; + } + } + } + + if caps.http_addon_available { + caps.external_scaler_address = self.http_scaler_address; + caps.interceptor_service_address = self.interceptor_service_address; + scaler_error = self._check_service_address(core_api, self.http_scaler_address); + if scaler_error is not None { + caps.errors.append(scaler_error); + } + interceptor_error = self._check_service_address( + core_api, self.interceptor_service_address + ); + if interceptor_error is not None { + caps.errors.append(interceptor_error); + } + } + + self._capabilities_cache[cluster_key] = caps; + return caps; +} + +impl KEDAAutoscaler.invalidate_capabilities{ + self._capabilities_cache.pop(self._capabilities_cache_key(), None); +} + +impl KEDAAutoscaler.format_capabilities{ + if caps.core_available and caps.http_addon_available and not caps.errors { + return "KEDA HTTP Add-on capabilities ready: core and HTTP Add-on both available."; + } + if caps.errors { + return f"KEDA HTTP Add-on not ready: {caps.errors[0]}"; + } + if caps.warnings { + return f"KEDA HTTP Add-on not ready: {caps.warnings[0]}"; + } + return "KEDA HTTP Add-on capabilities unknown."; +} diff --git a/jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac b/jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac index 33b78a4eefc..8c8ec24a1dc 100644 --- a/jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac +++ b/jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac @@ -11,6 +11,10 @@ import from jaclang.scale.deploy.autoscale.http_activation { HTTPActivationSpec, HTTPRoutingRule } +import from jaclang.scale.deploy.autoscale.keda_capabilities { + KEDACapabilities, + SUPPORTED_HTTP_INTERCEPTOR_API_VERSIONS +} import from jaclang.scale.observability.logger { Logger } import hashlib; @@ -20,10 +24,15 @@ obj KEDAAutoscaler(Autoscaler) { _apps_api: (any | None) = None, _core_api: (any | None) = None, http_scaler_address: str = "keda-add-ons-http-external-scaler.keda:9090", - logger: (Logger | None) = None; + interceptor_service_address: str = "keda-add-ons-http-interceptor-proxy.keda:8080", + logger: (Logger | None) = None, + _capabilities_cache: dict[str, any] = {}; static has _preflight_cache: dict[str, any] = {}, - _http_preflight_cache: dict[str, any] = {}; + _APPLY_ORDER: list[tuple[str, str]] = [ + ("TriggerAuthentication", "triggerauthentications"), + ("ScaledObject", "scaledobjects") + ]; def _get_cluster_key -> str; def _trigger_key(trigger: Trigger, trigger_index: int) -> str; @@ -33,15 +42,34 @@ obj KEDAAutoscaler(Autoscaler) { trigger: Trigger, app_name: str, trigger_index: int ) -> dict[str, any]; - def _apply_trigger_auth( - api: any, trigger: Trigger, namespace: str, app_name: str, trigger_index: int + static def _owner_label_value(name: str) -> str; + def _scaled_object_name(spec: AutoscalerSpec, app_name: str) -> str; + def _build_trigger_auth_manifest( + trigger: Trigger, namespace: str, app_name: str, trigger_index: int, owner: str + ) -> (dict[str, any] | None); + + def _apply_custom_object( + api: any, + group: str, + version: str, + namespace: str, + plural: str, + name: str, + body: dict[str, any] + ) -> None; + + def _prune_orphan_trigger_auths( + api: any, namespace: str, owner: str, desired_auth_names: set[str] ) -> None; - def _build_manifests(spec: AutoscalerSpec) -> dict[str, any]; + def _build_manifests(spec: AutoscalerSpec) -> list[dict[str, any]]; def preflight -> list[str]; def apply(spec: AutoscalerSpec) -> bool; - def destroy(autoscaler_name: str, namespace: str) -> None; - def destroy_collection(namespace: str, label_selector: str) -> None; + def destroy(app_name: str, namespace: str) -> None; + def destroy_collection( + namespace: str, label_selector: str, best_effort: bool = False + ) -> None; + def resource_name_for(target: str) -> str; def _validate_http_activation_spec(spec: HTTPActivationSpec) -> None; def _build_routing_rules(rules: list[HTTPRoutingRule]) -> list[dict[str, any]]; @@ -51,4 +79,9 @@ obj KEDAAutoscaler(Autoscaler) { def http_scaled_object_name_for(target: str) -> str; def apply_http_activation(spec: HTTPActivationSpec) -> bool; def destroy_http_activation(target: str, namespace: str) -> None; + def discover_capabilities(refresh: bool = False) -> KEDACapabilities; + def invalidate_capabilities -> None; + def _capabilities_cache_key -> str; + def _check_service_address(core_api: any, address: str) -> (str | None); + def format_capabilities(caps: KEDACapabilities) -> str; } diff --git a/jac/jaclang/scale/deploy/autoscale/keda_capabilities.jac b/jac/jaclang/scale/deploy/autoscale/keda_capabilities.jac new file mode 100644 index 00000000000..4da634b85ff --- /dev/null +++ b/jac/jaclang/scale/deploy/autoscale/keda_capabilities.jac @@ -0,0 +1,11 @@ +glob SUPPORTED_HTTP_INTERCEPTOR_API_VERSIONS: list[str] = ["v1beta1"]; + +obj KEDACapabilities { + has core_available: bool = False, + http_addon_available: bool = False, + interceptor_route_api_version: (str | None) = None, + external_scaler_address: (str | None) = None, + interceptor_service_address: (str | None) = None, + warnings: list[str] = [], + errors: list[str] = []; +} diff --git a/jac/jaclang/scale/deploy/target/kubernetes/kubernetes_config.jac b/jac/jaclang/scale/deploy/target/kubernetes/kubernetes_config.jac index 0ce03904c80..74dbefe17cc 100644 --- a/jac/jaclang/scale/deploy/target/kubernetes/kubernetes_config.jac +++ b/jac/jaclang/scale/deploy/target/kubernetes/kubernetes_config.jac @@ -35,6 +35,7 @@ obj KubernetesConfig(BaseConfig) { autoscaler_cooldown: int = 300, autoscaler_initial_cooldown: int = 0, extra_triggers: list[dict] = [], + http_activation: dict[str, any] = {}, health_check_path: str = '/docs', python_image: str = '', busybox_image: str = 'busybox:1.36', @@ -118,6 +119,7 @@ obj KubernetesConfig(BaseConfig) { 'autoscaler_cooldown': self.autoscaler_cooldown, 'autoscaler_initial_cooldown': self.autoscaler_initial_cooldown, 'extra_triggers': self.extra_triggers, + 'http_activation': self.http_activation, 'health_check_path': self.health_check_path, 'python_image': self.python_image, 'busybox_image': self.busybox_image, @@ -207,6 +209,7 @@ obj KubernetesConfig(BaseConfig) { autoscaler_cooldown=config.get('autoscaler_cooldown', 300), autoscaler_initial_cooldown=config.get('autoscaler_initial_cooldown', 0), extra_triggers=config.get('extra_triggers', []), + http_activation=config.get('http_activation', {}), health_check_path=config.get('health_check_path', '/docs'), python_image=config.get('python_image', ''), busybox_image=config.get('busybox_image', 'busybox:1.36'), diff --git a/jac/jaclang/scale/deploy/target/kubernetes/kubernetes_target.jac b/jac/jaclang/scale/deploy/target/kubernetes/kubernetes_target.jac index 475801fa6aa..8008a1bd7a8 100644 --- a/jac/jaclang/scale/deploy/target/kubernetes/kubernetes_target.jac +++ b/jac/jaclang/scale/deploy/target/kubernetes/kubernetes_target.jac @@ -6,14 +6,9 @@ import from jaclang.scale.observability.logger { Logger } import from jaclang.scale.deploy.target.kubernetes.kubernetes_config { KubernetesConfig } -import from jaclang.scale.injector.pvc_injector { bundle_pvc_name } import from jaclang.scale.deploy.target.kubernetes.utils.kubernetes_utils { COMPANION_DEFS, create_k8s_secret, - delete_k8s_secret, - delete_if_exists, - delete_namespace, - check_K8s_status, ensure_namespace_exists, get_cluster_provider, resize_pvc_if_needed @@ -29,8 +24,6 @@ import from jaclang.scale._optdeps.kubernetes { import from jaclang.scale.deploy.database.factory { DatabaseProviderFactory } import from jaclang.scale.deploy.target.kubernetes.monitoring { MonitoringDeployer } import from jaclang.scale.deploy.target.kubernetes.ingress { IngressDeployer } -import from jaclang.scale.deploy.autoscale.factory { AutoscalerFactory } -import time; obj KubernetesTargetBase(DeploymentTarget) { has config: KubernetesConfig, @@ -462,469 +455,6 @@ obj KubernetesTargetBase(DeploymentTarget) { } } - def _wait_for_deletion( - app_name: str, - namespace: str, - apps_v1: any, - core_v1: any, - max_wait: int = 60, - poll_interval: float = 1.0 - ) -> None { - elapsed = 0.0; - while elapsed < max_wait { - resources_exist = False; - - try { - apps_v1.read_namespaced_deployment(name=app_name, namespace=namespace); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - - try { - core_v1.read_namespaced_service( - name=f"{app_name}-service", namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - - if self.k8s_config.mongodb_enabled { - mongodb_name = f"{app_name}-mongodb"; - try { - apps_v1.read_namespaced_stateful_set( - name=mongodb_name, namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - try { - core_v1.read_namespaced_service( - name=f"{mongodb_name}-service", namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - } - - if self.k8s_config.redis_enabled { - redis_name = f"{app_name}-redis"; - try { - apps_v1.read_namespaced_deployment( - name=redis_name, namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - try { - core_v1.read_namespaced_service( - name=f"{redis_name}-service", namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - } - - if self.k8s_config.monitoring_enabled { - prometheus_name = f"{app_name}-prometheus"; - try { - apps_v1.read_namespaced_deployment( - name=prometheus_name, namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - try { - core_v1.read_namespaced_service( - name=f"{prometheus_name}-service", namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - grafana_name = f"{app_name}-grafana"; - try { - apps_v1.read_namespaced_deployment( - name=grafana_name, namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - try { - core_v1.read_namespaced_service( - name=f"{grafana_name}-service", namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - } - - try { - pvcs = core_v1.list_namespaced_persistent_volume_claim(namespace); - for pvc in pvcs.items { - if pvc.metadata.name.startswith(app_name) { - resources_exist = True; - break; - } - } - } except Exception as e { - if self.logger { - self.logger.warning( - f"Failed to list PVCs while checking resources for '{app_name}': {e}" - ); - } - } - - try { - core_v1.read_namespaced_pod( - name=f"{app_name}-code-sync", namespace=namespace - ); - resources_exist = True; - } except ApiException as e { - if e.status != 404 { - raise; - } - } - - if not resources_exist { - if self.logger { - self.logger.info( - f"All resources for '{app_name}' have been deleted" - ); - } - return; - } - time.sleep(poll_interval); - elapsed = elapsed + poll_interval; - } - - if self.logger { - self.logger.warn( - f"Timeout waiting for resources to be deleted after {max_wait} seconds" - ); - } - } - - def _destroy_databases(app_name: str, namespace: str, apps_v1: any, core_v1: any) { - self._destroy_database(app_name, namespace, apps_v1, core_v1); - self._destroy_cache(app_name, namespace, apps_v1, core_v1); - self._destroy_dashboard(app_name, namespace, apps_v1, core_v1); - } - - def _destroy_application( - app_name: str, namespace: str, apps_v1: any, core_v1: any - ) { - autoscaler = AutoscalerFactory.create( - self.k8s_config.autoscaler_engine, self.k8s_config.to_dict(), self.logger - ); - autoscaler.destroy(autoscaler.resource_name_for(app_name), namespace); - delete_if_exists( - apps_v1.delete_namespaced_deployment, app_name, namespace, 'Deployment' - ); - delete_if_exists( - core_v1.delete_namespaced_service, - f"{app_name}-service", - namespace, - 'Service' - ); - delete_k8s_secret(core_v1, namespace, f"{app_name}-secrets"); - - delete_if_exists( - core_v1.delete_namespaced_pod, f"{app_name}-bundle-loader", namespace, 'Pod' - ); - delete_if_exists( - core_v1.delete_namespaced_persistent_volume_claim, - bundle_pvc_name(app_name), - namespace, - 'PersistentVolumeClaim' - ); - delete_if_exists( - core_v1.delete_namespaced_service_account, - f"{app_name}-sa", - namespace, - 'ServiceAccount' - ); - networking_v1 = client.NetworkingV1Api(); - - network_policy_names = [ - f"{app_name}-network-policy", - f"{app_name}-mongodb-network-policy", - f"{app_name}-redis-network-policy", - f"{app_name}-prometheus-network-policy", - f"{app_name}-grafana-network-policy", - f"{app_name}-kube-state-metrics-network-policy", - f"{app_name}-node-exporter-network-policy", - f"{app_name}-mongo-express-network-policy", - f"{app_name}-redis-insight-network-policy" - ]; - for policy_name in network_policy_names { - delete_if_exists( - networking_v1.delete_namespaced_network_policy, - policy_name, - namespace, - 'NetworkPolicy' - ); - } - } - - def _destroy_database(app_name: str, namespace: str, apps_v1: any, core_v1: any) { - mongodb_name = f"{app_name}-mongodb"; - delete_if_exists( - apps_v1.delete_namespaced_stateful_set, - mongodb_name, - namespace, - 'StatefulSet' - ); - delete_if_exists( - core_v1.delete_namespaced_service, - f"{mongodb_name}-service", - namespace, - 'Service' - ); - delete_k8s_secret(core_v1, namespace, f"{app_name}-mongodb-secret"); - delete_if_exists( - core_v1.delete_namespaced_service_account, - f"{app_name}-mongodb-sa", - namespace, - 'ServiceAccount' - ); - } - - def _destroy_cache(app_name: str, namespace: str, apps_v1: any, core_v1: any) { - redis_name = f"{app_name}-redis"; - delete_if_exists( - apps_v1.delete_namespaced_deployment, redis_name, namespace, 'Deployment' - ); - delete_if_exists( - core_v1.delete_namespaced_service, - f"{redis_name}-service", - namespace, - 'Service' - ); - delete_if_exists( - core_v1.delete_namespaced_config_map, - f"{redis_name}-config", - namespace, - 'ConfigMap' - ); - delete_k8s_secret(core_v1, namespace, f"{app_name}-redis-secret"); - delete_if_exists( - core_v1.delete_namespaced_service_account, - f"{app_name}-redis-sa", - namespace, - 'ServiceAccount' - ); - } - - def _destroy_dashboard(app_name: str, namespace: str, apps_v1: any, core_v1: any) { - express_name = f"{app_name}-mongo-express"; - delete_if_exists( - apps_v1.delete_namespaced_deployment, express_name, namespace, 'Deployment' - ); - delete_if_exists( - core_v1.delete_namespaced_service, - f"{express_name}-service", - namespace, - 'Service' - ); - insight_name = f"{app_name}-redis-insight"; - delete_if_exists( - apps_v1.delete_namespaced_deployment, insight_name, namespace, 'Deployment' - ); - delete_if_exists( - core_v1.delete_namespaced_service, - f"{insight_name}-service", - namespace, - 'Service' - ); - delete_if_exists( - core_v1.delete_namespaced_config_map, - f"{insight_name}-nginx-config", - namespace, - 'ConfigMap' - ); - delete_if_exists( - core_v1.delete_namespaced_secret, - f"{insight_name}-auth", - namespace, - 'Secret' - ); - delete_if_exists( - core_v1.delete_namespaced_service_account, - f"{app_name}-mongo-express-sa", - namespace, - 'ServiceAccount' - ); - delete_if_exists( - core_v1.delete_namespaced_service_account, - f"{app_name}-redis-insight-sa", - namespace, - 'ServiceAccount' - ); - } - - def _destroy_component( - app_name: str, namespace: str, component: str, apps_v1: any, core_v1: any - ) { - if component == 'application' { - self._destroy_application(app_name, namespace, apps_v1, core_v1); - } elif component == 'database' { - self._destroy_database(app_name, namespace, apps_v1, core_v1); - } elif component == 'cache' { - self._destroy_cache(app_name, namespace, apps_v1, core_v1); - } elif component == 'monitoring' { - monitoring = MonitoringDeployer(self.k8s_config, self.logger); - monitoring.destroy(app_name, namespace, apps_v1, core_v1); - } elif component == 'dashboard' { - self._destroy_dashboard(app_name, namespace, apps_v1, core_v1); - } else { - raise ValueError( - f"Unknown component: '{component}'. Valid choices: application, database, cache, monitoring, dashboard" - ); - } - if self.logger { - self.logger.info(f"Component '{component}' of '{app_name}' destroyed"); - } - } - - def destroy(app_name: str, component: str = "") -> None { - if self.logger { - if component { - self.logger.info( - f"Destroying component '{component}' of '{app_name}' from Kubernetes" - ); - } else { - self.logger.info( - f"Destroying application '{app_name}' from Kubernetes" - ); - } - } - - try { - self._load_cluster_config(); - apps_v1 = client.AppsV1Api(); - core_v1 = client.CoreV1Api(); - namespace = self.k8s_config.namespace; - - if component { - self._destroy_component( - app_name, namespace, component, apps_v1, core_v1 - ); - return; - } - - self._destroy_application(app_name, namespace, apps_v1, core_v1); - - rbac_v1 = client.RbacAuthorizationV1Api(); - networking_v1 = client.NetworkingV1Api(); - ingress_deployer = IngressDeployer(self.k8s_config, self.logger); - ingress_deployer.destroy( - app_name, namespace, apps_v1, core_v1, rbac_v1, networking_v1 - ); - - monitoring = MonitoringDeployer(self.k8s_config, self.logger); - monitoring.destroy(app_name, namespace, apps_v1, core_v1); - - self._destroy_databases(app_name, namespace, apps_v1, core_v1); - - pvcs = core_v1.list_namespaced_persistent_volume_claim(namespace); - for pvc in pvcs.items { - if pvc.metadata.name.startswith(app_name) { - try { - core_v1.delete_namespaced_persistent_volume_claim( - name=pvc.metadata.name, namespace=namespace - ); - } except Exception as e { - if self.logger { - self.logger.warning( - f"Failed to delete PVC '{pvc.metadata.name}': {e}" - ); - } - } - } - } - - if self.logger { - self.logger.info( - f"Waiting for all resources to be deleted for '{app_name}'..." - ); - } - self._wait_for_deletion(app_name, namespace, apps_v1, core_v1); - - if namespace != 'default' { - delete_namespace(namespace); - if self.logger { - self.logger.info( - f"Waiting for namespace '{namespace}' to be deleted..." - ); - } - ns_deleted = False; - ns_elapsed = 0.0; - ns_max_wait = 120; - ns_poll = 2.0; - while ns_elapsed < ns_max_wait { - try { - core_v1.read_namespace(name=namespace); - time.sleep(ns_poll); - ns_elapsed = ns_elapsed + ns_poll; - } except ApiException as e { - if e.status == 404 { - ns_deleted = True; - break; - } - raise; - } - } - if self.logger { - if ns_deleted { - self.logger.info(f"Namespace '{namespace}' deleted"); - } else { - self.logger.warning( - f"Timed out waiting for namespace '{namespace}' to be deleted" - ); - } - } - } - - if self.logger { - self.logger.info(f"Application '{app_name}' destroyed successfully"); - } - } except Exception as e { - if self.logger { - self.logger.error(f"Error destroying application '{app_name}': {e}"); - } - raise; - } - } - def get_status(app_name: str) -> ResourceStatusInfo { try { self._load_cluster_config(); diff --git a/jac/jaclang/scale/deploy/target/kubernetes/manifest_builder.jac b/jac/jaclang/scale/deploy/target/kubernetes/manifest_builder.jac index eb91aa0c206..4384eee4574 100644 --- a/jac/jaclang/scale/deploy/target/kubernetes/manifest_builder.jac +++ b/jac/jaclang/scale/deploy/target/kubernetes/manifest_builder.jac @@ -11,6 +11,10 @@ import from jaclang.scale.deploy.autoscale.autoscaler { DEFAULT_CPU_UTILIZATION_TARGET, DEFAULT_MEMORY_UTILIZATION_TARGET } +import from jaclang.scale.deploy.autoscale.http_activation_config { + build_http_activation_spec +} +import from enum { StrEnum } glob GATEWAY_NAME: str = "__gateway__", @@ -53,6 +57,12 @@ def _is_transient_client_build_failure(output: str) -> bool { } +enum ScalingMode(StrEnum) { + NONE = "none", + METRIC = "metric", + HTTP_ACTIVATION = "http_activation" +} + def k8s_safe_name(svc_name: str) -> str { if svc_name == GATEWAY_NAME { return "gateway"; @@ -336,26 +346,26 @@ obj ManifestBuilder { pod_specs = self.generate_service_pod_specs(app_config, image); deployments: dict[str, dict[str, any]] = {}; services: dict[str, dict[str, any]] = {}; - autoscalers: dict[str, dict[str, any]] = {}; + scaling: dict[str, dict[str, any]] = {}; pdbs: dict[str, dict[str, any]] = {}; for (svc_name, pod_spec) in pod_specs.items() { deployments[svc_name] = self._build_deployment_manifest(svc_name, pod_spec); services[svc_name] = self._build_service_manifest(svc_name); - autoscaler_cfg = self._get_autoscaler_config(svc_name); - if autoscaler_cfg is not None { - autoscalers[svc_name] = autoscaler_cfg; - } - pdb = self._build_pdb_manifest(svc_name); + scaling[svc_name] = self._resolve_scaling( + svc_name, deployments[svc_name], services[svc_name] + ); + pdb = self._build_pdb_manifest(svc_name, scaling[svc_name].get("mode")); if pdb is not None { pdbs[svc_name] = pdb; } } + self._validate_http_activation_rules(scaling); bundle: dict[str, any] = { "deployments": deployments, "services": services, - "autoscalers": autoscalers, + "scaling": scaling, "pdbs": pdbs, "pod_specs": pod_specs }; @@ -935,6 +945,127 @@ obj ManifestBuilder { }; } + def _resolve_scaling( + svc_name: str, deployment: dict[str, any], service: dict[str, any] + ) -> dict[str, any] { + """Resolve a service's single, mutually-exclusive scaling decision. + + The one place a service's scaling mode is decided. The scale-target + name is read from the Deployment manifest just built (never rebuilt + from svc_name), so build, apply, and `jac plan` can never disagree on + it. The gateway is forced out of HTTP_ACTIVATION here: it is the + ingress entry point and must stay warm, so it must never inherit the + shared scale-to-zero default. + """; + scale_target: str = deployment["metadata"]["name"]; + service_name: str = service["metadata"]["name"]; + http: dict[str, any] | None = self._get_http_activation_config(svc_name); + if http is not None and svc_name == GATEWAY_NAME { + if self.logger { + self.logger.warn( + "Ignoring http_activation for the gateway: it is the ingress " + "entry point and must stay warm. Set http_activation " + "per-service instead of relying on the shared default." + ); + } + http = None; + } + if http is not None { + try { + build_http_activation_spec( + http, scale_target, self.k8s_config.namespace, service_name + ); + } except ValueError as e { + raise ValueError(f"[service '{svc_name}'] {e}") from e; + } + return { + "mode": ScalingMode.HTTP_ACTIVATION, + "scale_target_name": scale_target, + "service_name": service_name, + "autoscaler": {}, + "http_activation": http + }; + } + metric: dict[str, any] | None = self._get_autoscaler_config(svc_name); + if metric is not None { + return { + "mode": ScalingMode.METRIC, + "scale_target_name": scale_target, + "service_name": service_name, + "autoscaler": metric, + "http_activation": {} + }; + } + return { + "mode": ScalingMode.NONE, + "scale_target_name": scale_target, + "service_name": service_name, + "autoscaler": {}, + "http_activation": {} + }; + } + + def _canonical_rules_fingerprint(rules: list[dict[str, any]]) -> str { + """A rules fingerprint that ignores list ordering, so two services + with the same match criteria collide in _validate_http_activation_rules + even when `hosts`/`paths`/`headers`, or the rules themselves, are + written in a different order -- fields within a rule are AND'd and + separate rules are OR'd, so reordering never changes what a rule set + actually matches. + """; + import json; + canonical_rules = []; + for rule in rules { + headers = [ + (str(h.get("name", "")), str(h.get("value", ""))) + for h in rule.get("headers", []) + ]; + canonical_rules.append( + { + "hosts": sorted(str(h) for h in rule.get("hosts", [])), + "paths": sorted(str(p) for p in rule.get("paths", [])), + "headers": sorted(headers) + } + ); + } + canonical_rules.sort(key=lambda (r) { json.dumps(r, sort_keys=True); }); + return json.dumps(canonical_rules, sort_keys=True); + } + + def _validate_http_activation_rules(scaling: dict[str, dict[str, any]]) { + """Two services sharing the top-level http_activation default with no + per-service override resolve to identical routing rules; the interceptor + then has no way to tell which target a matching request belongs to. + """; + by_rules: dict[str, list[str]] = {}; + for (svc_name, plan) in scaling.items() { + if plan.get("mode") != ScalingMode.HTTP_ACTIVATION { + continue; + } + rules = plan.get("http_activation", {}).get("rules", []); + if not rules { + continue; + } + fingerprint = self._canonical_rules_fingerprint(rules); + by_rules.setdefault(fingerprint, []).append(svc_name); + } + for svc_names in by_rules.values() { + if len(svc_names) > 1 { + names = ", ".join(sorted(svc_names)); + raise ValueError( + f"[http_activation] {names} resolve to identical routing " + f"rules, most likely all inheriting " + f"[scale.kubernetes.http_activation]'s rules unchanged. The " + f"KEDA HTTP Add-on interceptor can't tell which target a " + f"matching request belongs to when two InterceptorRoutes " + f"share the same match criteria. Set a distinct " + f"[services.NAME.http_activation.rules] for at least all " + f"but one of these services." + ); + } + } + } + def _get_autoscaler_config(svc_name: str) -> dict[str, any] | None { svc: dict[str, any] = self._service_config(svc_name); @@ -948,6 +1079,16 @@ obj ManifestBuilder { raw_triggers.append(dict(item)); } } + behavior = hpa.get("behavior", {}); + if not isinstance(behavior, dict) { + if self.logger { + self.logger.warning( + f"Ignoring malformed hpa.behavior for '{svc_name}': expected " + f"a table, got {type(behavior).__name__}" + ); + } + behavior = {}; + } return { "min": int(hpa.get("min", 1)), "max": int(hpa.get("max", 3)), @@ -955,10 +1096,22 @@ obj ManifestBuilder { "memory_target": int( hpa.get("memory_target", DEFAULT_MEMORY_UTILIZATION_TARGET) ), - "triggers": raw_triggers + "triggers": raw_triggers, + "behavior": dict(behavior) }; } + def _get_http_activation_config(svc_name: str) -> dict[str, any] | None { + svc: dict[str, any] = self._service_config(svc_name); + per_service: dict[str, any] = svc.get("http_activation", {}); + merged: dict[str, any] = dict(self.k8s_config.http_activation); + merged.update(per_service); + if not merged.get("enabled", False) { + return None; + } + return merged; + } + def _build_ingress_manifest(ingress_cfg: dict[str, any]) -> dict[str, any] | None { cfg = self.k8s_config; enabled: bool = bool(ingress_cfg.get("enabled", False)); @@ -1039,11 +1192,25 @@ obj ManifestBuilder { }; } - def _build_pdb_manifest(svc_name: str) -> dict[str, any] | None { + def _build_pdb_manifest( + svc_name: str, mode: (ScalingMode | None) = None + ) -> dict[str, any] | None { cfg: dict[str, any] = self._service_config(svc_name).get("pdb", {}); if not cfg.get("enabled", True) { return None; } + if mode == ScalingMode.HTTP_ACTIVATION { + if self.logger and cfg { + self.logger.warn( + f"No PodDisruptionBudget emitted for '{svc_name}': it is " + f"HTTP-activated and legitimately scales to zero replicas, " + f"so a floor-based budget (pdb.max_unavailable / " + f"min_available) doesn't apply. Disable it explicitly with " + f"pdb.enabled = false to silence this warning." + ); + } + return None; + } k8s_name = k8s_safe_name(svc_name); max_u: int = int(cfg.get("max_unavailable", 1)); return { diff --git a/jac/jaclang/scale/deploy/target/kubernetes/target.jac b/jac/jaclang/scale/deploy/target/kubernetes/target.jac index de4d023e3f4..bda640801b2 100644 --- a/jac/jaclang/scale/deploy/target/kubernetes/target.jac +++ b/jac/jaclang/scale/deploy/target/kubernetes/target.jac @@ -22,7 +22,8 @@ import from jaclang.scale.deploy.target.kubernetes.manifest_builder { CONFIG_REVISION_ANNOTATION, ManifestBuilder, GATEWAY_NAME, - k8s_safe_name + k8s_safe_name, + ScalingMode } import from jaclang.scale.deploy.target.kubernetes.database_provisioner { DatabaseProvisioner @@ -35,6 +36,11 @@ import from jaclang.scale.deploy.autoscale.autoscaler { DEFAULT_CPU_UTILIZATION_TARGET, DEFAULT_MEMORY_UTILIZATION_TARGET } +import from jaclang.scale.deploy.autoscale.keda_autoscaler { KEDAAutoscaler } +import from jaclang.scale.deploy.autoscale.http_activation_config { + apply_http_activation_for_target, + destroy_http_activation_for_target +} import from jaclang.scale.deploy.target.kubernetes.monitoring { MonitoringDeployer } import from jaclang.scale.config.config_loader { get_scale_config } import from jaclang.scale.injector.pvc_injector { bundle_keys_from_pod_spec } @@ -165,6 +171,43 @@ def restorable_pod_template(template: dict[str, any]) -> dict[str, any] { return cleaned; } +def apply_http_activations_for_bundle( + bundle: dict[str, any], namespace: str, keda: KEDAAutoscaler +) { + scaling: dict[str, any] = bundle.get("scaling", {}); + for svc_name in scaling.keys() { + plan: dict[str, any] = dict(scaling.get(svc_name, {})); + if plan.get("mode") != ScalingMode.HTTP_ACTIVATION { + continue; + } + apply_http_activation_for_target( + dict(plan.get("http_activation", {})), + str(plan.get("scale_target_name", "")), + namespace, + str(plan.get("service_name", "")), + keda + ); + } +} + + +def reap_stale_scaling_resources_for_bundle( + bundle: dict[str, any], namespace: str, autoscaler: any, keda: KEDAAutoscaler +) { + scaling: dict[str, any] = bundle.get("scaling", {}); + for svc_name in scaling.keys() { + plan: dict[str, any] = dict(scaling.get(svc_name, {})); + mode = plan.get("mode"); + scale_target_name = str(plan.get("scale_target_name", "")); + if mode != ScalingMode.METRIC { + autoscaler.destroy(k8s_safe_name(svc_name), namespace); + } + if mode != ScalingMode.HTTP_ACTIVATION { + destroy_http_activation_for_target(scale_target_name, namespace, keda); + } + } +} + obj KubernetesTarget(KubernetesTargetBase) { has _deploy_context: DeployContext = DeployContext(), @@ -498,10 +541,16 @@ obj KubernetesTarget(KubernetesTargetBase) { import sys; import from kubernetes { client } import from jaclang.cli.console { console } + + scaling: dict[str, any] = bundle.get("scaling", {}); + http_activation_svc_names: set = { + str(svc_name) + for svc_name in scaling.keys() + if dict(scaling.get(svc_name, {})).get("mode") == ScalingMode.HTTP_ACTIVATION + }; names: list[str] = []; app_labels: list[str] = []; - for dep in bundle.get("deployments", {}).values() { - names.append(dep["metadata"]["name"]); + for (svc_name, dep) in bundle.get("deployments", {}).items() { app: str = ""; try { app = str(dep["spec"]["selector"]["matchLabels"]["app"]); @@ -514,8 +563,29 @@ obj KubernetesTarget(KubernetesTargetBase) { if app { app_labels.append(app); } + if svc_name not in http_activation_svc_names { + names.append(str(dep["metadata"]["name"])); + } } if not names { + self._load_cluster_config(); + core_v1 = client.CoreV1Api(); + ns = self.k8s_config.namespace; + crash = self._scan_for_crash_loop(core_v1, ns, set(), set(app_labels))[ + "crash" + ]; + if crash { + raise Exception( + "Deployment failed: application pods are restarting " + "(crash loop / bad image); not marking the fleet ready." + ); + } + if self.logger { + self.logger.info( + "Fleet ready: all services use HTTP activation " + "(scale to zero until first request); pods healthy." + ); + } return; } self._load_cluster_config(); @@ -559,9 +629,16 @@ obj KubernetesTarget(KubernetesTargetBase) { for name in names { try { d = apps_v1.read_namespaced_deployment(name=name, namespace=ns); - complete = deployment_rollout_complete(d) - if require_full - else (d.status.ready_replicas or 0) >= 1; + # Measurement scaffolding standing in for #8723: an + # autoscaler that parks a scale-to-zero service at 0 + # replicas is desired state, not a stuck rollout. + parked = (d.spec.replicas or 0) == 0; + complete = parked + or ( + deployment_rollout_complete(d) + if require_full + else (d.status.ready_replicas or 0) >= 1 + ); if complete { ready += 1; } @@ -928,6 +1005,114 @@ obj KubernetesTarget(KubernetesTargetBase) { return ctx; } + def _resolve_service_triggers(svc_name: str, cfg: dict[str, any]) -> list[Trigger] { + is_gateway = svc_name == GATEWAY_NAME; + has_memory_request = bool(cfg.get("has_memory_request", True)); + effective_extra = self.k8s_config.extra_triggers if is_gateway else []; + triggers: list[Trigger] = []; + if not cfg.get("triggers", []) and not effective_extra { + triggers = [ + Trigger( + type="cpu", + metadata={ + "averageUtilization": str( + cfg.get("cpu_target", DEFAULT_CPU_UTILIZATION_TARGET) + ) + } + ) + ]; + if has_memory_request { + triggers.append( + Trigger( + type="memory", + metadata={ + "averageUtilization": str( + cfg.get( + "memory_target", DEFAULT_MEMORY_UTILIZATION_TARGET + ) + ) + } + ) + ); + } + } + for raw in cfg.get("triggers", []) + effective_extra { + t = Trigger( + type=str(raw.get("type", "")), + metadata=dict(raw.get("metadata", {})), + name=raw.get("name") + ); + if t.type == "memory" and not has_memory_request { + if self.logger { + self.logger.warn( + f"Skipping 'memory' trigger for '{svc_name}': " + "memory_request resolves empty (explicitly set to " + "\"\"); HPA cannot compute memory utilization " + "without a request baseline" + ); + } + continue; + } + raw_auth = raw.get("auth"); + if raw_auth { + t.auth = TriggerAuth(secret_refs=dict(raw_auth.get("secret_refs", {}))); + } + triggers.append(t); + } + return triggers; + } + + def _autoscaler_spec( + svc_name: str, plan: dict[str, any], autoscaler: any, namespace: str + ) -> AutoscalerSpec { + """Build the AutoscalerSpec for a METRIC-mode service from its already- + resolved scaling plan, so the scale target here is the same Deployment + name apply_manifests actually applied -- never re-derived from svc_name, + which is exactly the drift that caused apply_http_activation to 404 on + a real cluster before _resolve_scaling existed. + """; + k8s_name = k8s_safe_name(svc_name); + cfg: dict[str, any] = dict(plan.get("autoscaler", {})); + return AutoscalerSpec( + scale_target_name=str(plan.get("scale_target_name", "")), + autoscaler_name=autoscaler.resource_name_for(k8s_name), + app_name=k8s_name, + namespace=namespace, + min_replicas=int(cfg.get("min", 1)), + max_replicas=int(cfg.get("max", 3)), + idle_replicas=self.k8s_config.idle_replicas, + polling_interval=self.k8s_config.autoscaler_polling_interval, + cooldown_period=self.k8s_config.autoscaler_cooldown, + initial_cooldown_period=self.k8s_config.autoscaler_initial_cooldown, + scale_up_stabilization=self.k8s_config.autoscaler_scale_up_stabilization, + scale_up_max_pods=self.k8s_config.autoscaler_scale_up_max_pods, + triggers=self._resolve_service_triggers(svc_name, cfg), + behavior_overlay=cfg.get("behavior", {}) + ); + } + + def render_autoscaler_manifests(bundle: dict[str, any]) -> list[dict[str, any]] { + out: list[dict[str, any]] = []; + scaling: dict[str, any] = bundle.get("scaling", {}); + for svc_name in scaling.keys() { + plan: dict[str, any] = dict(scaling.get(svc_name, {})); + if plan.get("mode") != ScalingMode.METRIC { + continue; + } + autoscaler = AutoscalerFactory.create( + self.k8s_config.autoscaler_engine, self.k8s_config.to_dict(), None + ); + out.extend( + autoscaler._build_manifests( + self._autoscaler_spec( + svc_name, plan, autoscaler, self.k8s_config.namespace + ) + ) + ); + } + return out; + } + def apply_manifests(bundle: dict[str, any]) { self._load_cluster_config(); apps_v1 = client.AppsV1Api(); @@ -1022,15 +1207,28 @@ obj KubernetesTarget(KubernetesTargetBase) { } } - for (svc_name, raw_manifest) in bundle["deployments"].items() { + scaling: dict[str, any] = bundle.get("scaling", {}); + + deployments: dict[str, dict[str, any]] = bundle["deployments"]; + for (svc_name, raw_manifest) in deployments.items() { dep_manifest = with_config_revision(raw_manifest, config_revision); + patch_body = None; + if dict(scaling.get(svc_name, {})).get("mode") == ScalingMode.HTTP_ACTIVATION { + patch_body = dict(dep_manifest); + patch_body["spec"] = { + k: v + for (k, v) in dep_manifest["spec"].items() + if k != "replicas" + }; + } self._apply_or_replace( apps_v1, "read_namespaced_deployment", "patch_namespaced_deployment", "create_namespaced_deployment", namespace, - dep_manifest + dep_manifest, + patch_body ); if self.logger { self.logger.info( @@ -1042,95 +1240,44 @@ obj KubernetesTarget(KubernetesTargetBase) { self.reap_superseded_deployments(apps_v1, namespace, bundle); - for svc_name in bundle.get("autoscalers", {}).keys() { - k8s_name = k8s_safe_name(svc_name); - cfg = dict(bundle.get("autoscalers", {}).get(svc_name, {})); - autoscaler = AutoscalerFactory.create( + reap_stale_scaling_resources_for_bundle( + bundle, + namespace, + AutoscalerFactory.create( self.k8s_config.autoscaler_engine, self.k8s_config.to_dict(), self.logger - ); - triggers = []; + ), + KEDAAutoscaler(logger=self.logger) + ); - is_gateway = svc_name == GATEWAY_NAME; - effective_extra = self.k8s_config.extra_triggers if is_gateway else []; - if not cfg.get("triggers", []) and not effective_extra { - triggers = [ - Trigger( - type="cpu", - metadata={ - "averageUtilization": str( - cfg.get("cpu_target", DEFAULT_CPU_UTILIZATION_TARGET) - ) - } - ), - Trigger( - type="memory", - metadata={ - "averageUtilization": str( - cfg.get( - "memory_target", DEFAULT_MEMORY_UTILIZATION_TARGET - ) - ) - } - ) - ]; - } - for raw in cfg.get("triggers", []) { - t = Trigger( - type=str(raw.get("type", "")), - metadata=dict(raw.get("metadata", {})), - name=raw.get("name") - ); - raw_auth = raw.get("auth"); - if raw_auth { - t.auth = TriggerAuth( - secret_refs=dict(raw_auth.get("secret_refs", {})) - ); - } - triggers.append(t); + for svc_name in scaling.keys() { + plan: dict[str, any] = dict(scaling.get(svc_name, {})); + if plan.get("mode") != ScalingMode.METRIC { + continue; } - for raw in effective_extra { - t = Trigger( - type=str(raw.get("type", "")), - metadata=dict(raw.get("metadata", {})), - name=raw.get("name") - ); - raw_auth = raw.get("auth"); - if raw_auth { - t.auth = TriggerAuth( - secret_refs=dict(raw_auth.get("secret_refs", {})) - ); - } - triggers.append(t); - } - autoscaler_name = autoscaler.resource_name_for(k8s_name); - autoscaler_applied = autoscaler.apply( - AutoscalerSpec( - scale_target_name=f"{k8s_name}-deployment", - autoscaler_name=autoscaler_name, - app_name=k8s_name, - namespace=namespace, - min_replicas=int(cfg.get("min", 1)), - max_replicas=int(cfg.get("max", 3)), - idle_replicas=self.k8s_config.idle_replicas, - polling_interval=self.k8s_config.autoscaler_polling_interval, - cooldown_period=self.k8s_config.autoscaler_cooldown, - initial_cooldown_period=self.k8s_config.autoscaler_initial_cooldown, - scale_up_stabilization=self.k8s_config.autoscaler_scale_up_stabilization, - scale_up_max_pods=self.k8s_config.autoscaler_scale_up_max_pods, - triggers=triggers - ) + autoscaler = AutoscalerFactory.create( + self.k8s_config.autoscaler_engine, + self.k8s_config.to_dict(), + self.logger ); + spec = self._autoscaler_spec(svc_name, plan, autoscaler, namespace); + autoscaler_applied = autoscaler.apply(spec); if self.logger and autoscaler_applied { self.logger.info( - f"Applied autoscaler for '{svc_name}'", {"name": autoscaler_name} + f"Applied autoscaler for '{svc_name}'", + {"name": spec.autoscaler_name} ); } } - for (svc_name, pdb_manifest) in bundle.get("pdbs", {}).items() { + apply_http_activations_for_bundle( + bundle, namespace, KEDAAutoscaler(logger=self.logger) + ); + + pdbs: dict[str, dict[str, any]] = bundle.get("pdbs", {}) or {}; + for (svc_name, pdb_manifest) in pdbs.items() { self._apply_or_replace( policy_v1, "read_namespaced_pod_disruption_budget", @@ -1565,12 +1712,24 @@ obj KubernetesTarget(KubernetesTargetBase) { update: str, create: str, namespace: str, - manifest: dict[str, any] + manifest: dict[str, any], + patch_body: (dict[str, any] | None) = None ) { + """patch_body, when given, is sent to `update` instead of `manifest` -- + e.g. a copy with `spec.replicas` stripped so a strategic-merge PATCH + leaves an externally-scaled field (KEDA's ScaledObject) alone, while a + fresh `create` still gets the full manifest including that field. + HTTP activation relies on exactly this: KEDA owns `replicas` once a + Deployment scales via HTTP_ACTIVATION, and the strategic-merge PATCH + this omission produces is what keeps this apply from resetting the + live value KEDA set. + """; name: str = manifest["metadata"]["name"]; try { getattr(api, read)(name=name, namespace=namespace); - getattr(api, update)(name=name, namespace=namespace, body=manifest); + getattr(api, update)( + name=name, namespace=namespace, body=patch_body or manifest + ); } except ApiException as e { if e.status == 404 { getattr(api, create)(namespace=namespace, body=manifest); @@ -1647,6 +1806,18 @@ obj KubernetesTarget(KubernetesTargetBase) { } } + if self.k8s_config.autoscaler_engine != "keda" { + try { + KEDAAutoscaler(logger=self.logger).destroy_collection( + namespace, ms_selector, best_effort=True + ); + } except ApiException as e { + if e.status != 404 { + raise; + } + } + } + try { policy_v1.delete_collection_namespaced_pod_disruption_budget( namespace=namespace, label_selector=ms_selector diff --git a/jac/jaclang/scale/injector/bundle.jac b/jac/jaclang/scale/injector/bundle.jac index 855e49c2848..ff2bc1db1b1 100644 --- a/jac/jaclang/scale/injector/bundle.jac +++ b/jac/jaclang/scale/injector/bundle.jac @@ -322,6 +322,124 @@ def _vendor_stage(stage: Path, sanitized_toml: (str | None), pod_binary: bytes) } +def _resolved_python_specs(stage: Path) -> list[str] { + import from jaclang.project.config { JacConfig } + import from jaclang.project.providers { get_python_provider, resolve_plan } + config = JacConfig.load(stage / "jac.toml"); + deps = resolve_plan(config).for_ecosystem("python"); + return get_python_provider().specs_for(deps); +} + + +def _strip_external_record_lines(site: Path) { + # Entry-point scripts under bin/ embed the staging path in their shebang, + # so their RECORD hash lines differ on every seal; bin/ never ships, and + # dropping its lines keeps RECORD (and the bundle digest) deterministic. + for record in sorted(site.`glob("*.dist-info/RECORD")) { + lines = record.read_text(encoding="utf-8").splitlines(keepends=True); + kept = [ + line + for line in lines + if not line.startswith("../") + ]; + if len(kept) != len(lines) { + record.write_text("".join(kept), encoding="utf-8"); + } + } +} + + +def staged_venv_files(stage: Path) -> list[Path] { + import from jaclang.project.providers { VENV_DEPS_MARKER } + import from jaclang.project.pyvenv { get_venv_site_packages } + venv = stage / ".jac" / "venv"; + marker = venv / VENV_DEPS_MARKER; + if not marker.is_file() { + return []; + } + site = get_venv_site_packages(venv); + if not site.is_dir() { + return []; + } + files = [marker]; + for path in sorted(site.rglob("*")) { + if ( + path.is_file() and path.suffix != ".pyc" and "__pycache__" not in path.parts + ) { + files.append(path); + } + } + return files; +} + + +def _prune_seed_pip(site: Path, specs: list[str]) { + # ensurepip seeds pip into every fresh venv, but a sealed venv never + # runs pip again (the marker short-circuits jac install), so shipping + # it only bloats the bundle. Kept when the app itself depends on pip. + import re; + for spec in specs { + if re.split(r"[<>=!~\[ ;@]", spec.strip())[0].lower() == "pip" { + return; + } + } + for item in sorted(site.iterdir()) { + if item.name == "pip" + or (item.name.startswith("pip-") and item.name.endswith(".dist-info")) { + shutil.rmtree(str(item), ignore_errors=True); + } + } +} + + +def _venv_stage(stage: Path, pod_binary: bytes, timeout_s: int = 900) { + jac_path = stage.parent / ".seal-venv-jac"; + jac_path.write_bytes(pod_binary); + jac_path.chmod(0o755); + env = release_binary_env(); + # The same offline install the pod bootstrap runs. Offline also keeps the + # staged venv a pure function of the vendored wheels, so the bundle + # digest stays deterministic across seals. + env["PIP_NO_INDEX"] = "1"; + env["PIP_FIND_LINKS"] = str(stage / "_vendor" / "wheels"); + env["PIP_PREFIX"] = str(stage / ".jac" / "venv"); + # The seal host's own runtime site may already satisfy some specs, which + # would make pip skip them and leave the staged prefix silently empty. + env["PIP_IGNORE_INSTALLED"] = "1"; + env["JAC_CLIENT_SKIP_NPM_INSTALL"] = "1"; + try { + result = subprocess.run( + [str(jac_path), "install"], + capture_output=True, + text=True, + timeout=timeout_s, + env=env, + cwd=str(stage) + ); + } finally { + jac_path.unlink(missing_ok=True); + } + if result.returncode != 0 { + raise RuntimeError( + f"seal-time venv build failed (host cannot install the " + f"pod-platform wheels?): " + f"{(result.stderr or result.stdout).strip()[-300:]}" + ); + } + venv = stage / ".jac" / "venv"; + import from jaclang.project.pyvenv { get_venv_site_packages } + site = get_venv_site_packages(venv); + if not site.is_dir() { + raise RuntimeError(f"seal-time venv build left no site-packages under {venv}"); + } + specs = _resolved_python_specs(stage); + _prune_seed_pip(site, specs); + _strip_external_record_lines(site); + import from jaclang.project.providers { write_venv_deps_marker } + write_venv_deps_marker(venv, specs); +} + + def pack_jab(project_dir: str, provisioned_database: str, pod_binary: bytes) -> bytes { if not pod_binary { raise ValueError( @@ -350,6 +468,20 @@ def pack_jab(project_dir: str, provisioned_database: str, pod_binary: bytes) -> ); } } + if ( + not os.environ.get("JAC_SEAL_SKIP_VENV") + and (staged / "_vendor" / "wheels").is_dir() + ) { + try { + _venv_stage(staged, pod_binary); + } except Exception as exc { + shutil.rmtree(str(staged / ".jac"), ignore_errors=True); + logger.warning( + f"Seal-time venv staging failed; pods will build the venv " + f"at boot instead. {exc}" + ); + } + } buf = io.BytesIO(); (tar, gz) = open_det_tar(buf); try { @@ -393,6 +525,30 @@ def pack_jab(project_dir: str, provisioned_database: str, pod_binary: bytes) -> f".jab; pods install python deps offline." ); } + venv_files = staged_venv_files(staged); + for path in venv_files { + rel = path.relative_to(staged); + tar.add(str(path), arcname=str(rel), filter=det_tar_member); + } + if venv_files { + logger.info( + f"Sealed a prebuilt venv ({len(venv_files)} files) " + f"into the .jab; pods skip the boot-time dependency " + f"install." + ); + } + venv_files = staged_venv_files(staged); + for path in venv_files { + rel = path.relative_to(staged); + tar.add(str(path), arcname=str(rel), filter=det_tar_member); + } + if venv_files { + logger.info( + f"Sealed a prebuilt venv ({len(venv_files)} files) " + f"into the .jab; pods skip the boot-time dependency " + f"install." + ); + } if sanitized is not None { data = sanitized.encode("utf-8"); info = tarfile.TarInfo(name="jac.toml"); diff --git a/jac/jaclang/scale/runtime/cli/diagnostics.jac b/jac/jaclang/scale/runtime/cli/diagnostics.jac index d63fca81de8..6502a93c151 100644 --- a/jac/jaclang/scale/runtime/cli/diagnostics.jac +++ b/jac/jaclang/scale/runtime/cli/diagnostics.jac @@ -1,5 +1,8 @@ import from typing { Any } -import from jaclang.scale.deploy.target.kubernetes.manifest_builder { GATEWAY_NAME } +import from jaclang.scale.deploy.target.kubernetes.manifest_builder { + GATEWAY_NAME, + ScalingMode +} import from jaclang.scale.runtime.cli._narrow { bget, dget, @@ -81,6 +84,7 @@ obj PlanValidator { out: list[Diagnostic] = []; routes = dget(self.ms_cfg, "routes"); deployments = dget(self.bundle, "deployments"); + scaling = dget(self.bundle, "scaling"); self._check_routes(routes, out); @@ -106,6 +110,12 @@ obj PlanValidator { ); continue; } + mode = dget(scaling, svc_name).get("mode"); + if mode == ScalingMode.HTTP_ACTIVATION { + self._check_resources(svc_name, deployment, out); + self._check_image(svc_name, deployment, out); + continue; + } self._check_hpa(svc_name, svc_cfg, deployment, out); self._check_pdb(svc_name, svc_cfg, deployment, out); self._check_resources(svc_name, deployment, out); diff --git a/jac/jaclang/scale/runtime/cli/plan.jac b/jac/jaclang/scale/runtime/cli/plan.jac index d3e4401d8be..84926d68f59 100644 --- a/jac/jaclang/scale/runtime/cli/plan.jac +++ b/jac/jaclang/scale/runtime/cli/plan.jac @@ -2,7 +2,10 @@ import yaml; import from typing { Any } import from jaclang.cli.console { console } import from jaclang.scale.config.app_config { AppConfig } -import from jaclang.scale.deploy.target.kubernetes.manifest_builder { GATEWAY_NAME } +import from jaclang.scale.deploy.target.kubernetes.manifest_builder { + GATEWAY_NAME, + ScalingMode +} import from jaclang.scale.runtime.cli.diagnostics { Diagnostic, PlanValidator, @@ -59,6 +62,31 @@ obj HPAView { } +obj HTTPActivationView { + has min_replicas: int, + max_replicas: int, + metric: str; + + static def from_manifest(http: Any) -> HTTPActivationView | None { + if not isinstance(http, dict) or not http { + return None; + } + if http.get("concurrency_target") is not None { + metric = f"concurrency={http.get('concurrency_target')}"; + } elif http.get("request_rate_target") is not None { + metric = f"request_rate={http.get('request_rate_target')}"; + } else { + metric = "(metric unset)"; + } + return HTTPActivationView( + min_replicas=iget(http, "min_replicas", 0), + max_replicas=iget(http, "max_replicas", 1), + metric=metric + ); + } +} + + obj PDBView { has max_unavailable: int | None; @@ -92,11 +120,17 @@ obj ServiceView { env_keys: list[str] = [], secret_ref_count: int = 0, hpa: HPAView | None = None, + http_activation: HTTPActivationView | None = None, pdb: PDBView | None = None, mounts: list[tuple[str, str]] = []; static def from_manifests( - name: str, deployment: Any, hpa: Any, pdb: Any, route_path: str | None + name: str, + deployment: any, + hpa: any, + http_activation: any, + pdb: any, + route_path: str | None ) -> ServiceView { containers = lget(pod_spec(deployment), "containers"); container: dict = {}; @@ -158,6 +192,7 @@ obj ServiceView { env_keys=env_keys, secret_ref_count=secret_refs, hpa=HPAView.from_manifest(hpa), + http_activation=HTTPActivationView.from_manifest(http_activation), pdb=PDBView.from_manifest(pdb), mounts=mounts ); @@ -174,6 +209,13 @@ obj ServiceView { f"{self.replicas} " f"(HPA: {h.min_replicas} -> {h.max_replicas} {target_str})" ); + } elif self.http_activation is not None { + a = self.http_activation; + replicas_line = ( + f"{self.replicas} " + f"(HTTP activation: {a.min_replicas} -> {a.max_replicas} " + f"on {a.metric}, scale-to-zero via KEDA HTTP Add-on)" + ); } else { replicas_line = str(self.replicas); } @@ -308,19 +350,33 @@ def _render_totals(bundle: Any) { n = len(dget(bundle, key)); return f"{n} {label}{'s' if n != 1 else ''}" if n else None; } + def _n_mode(mode: ScalingMode, label: str) -> str | None { + n = len( + [ + p + for p in dget(bundle, "scaling").values() + if isinstance(p, dict) and p.get("mode") == mode + ] + ); + return f"{n} {label}{'s' if n != 1 else ''}" if n else None; + } parts: list[str] = []; n_dep = len(dget(bundle, "deployments")); n_svc = len(dget(bundle, "services")); parts.append(f"{n_dep} deployment{'s' if n_dep != 1 else ''}"); parts.append(f"{n_svc} service{'s' if n_svc != 1 else ''}"); - for (key, label) in [ - ("autoscalers", "Autoscaler"), - ("pdbs", "PDB"), - ("pvcs", "PVC") - ] { - entry = _n(key, label); - if entry { - parts.append(entry); + autoscaler_entry = _n_mode(ScalingMode.METRIC, "Autoscaler"); + if autoscaler_entry { + parts.append(autoscaler_entry); + } + http_entry = _n_mode(ScalingMode.HTTP_ACTIVATION, "HTTP activation"); + if http_entry { + parts.append(http_entry); + } + for (key, label) in [("pdbs", "PDB"), ("pvcs", "PVC")] { + `entry = _n(key, label); + if `entry { + parts.append(`entry); } } if isinstance(bundle, dict) and bundle.get("ingress") { @@ -344,6 +400,10 @@ obj Plan { static def from_target(target: Any, app_config: AppConfig) -> Plan { bundle_raw = target.generate_manifests(app_config); bundle: dict = bundle_raw if isinstance(bundle_raw, dict) else {}; + render_autoscalers = target?.render_autoscaler_manifests; + if render_autoscalers is not None { + bundle["autoscaler_manifests"] = render_autoscalers(bundle); + } (ms_cfg, svc_configs) = Plan._load_ms_cfg(); routes_any: Any = ms_cfg.get("routes", {}); @@ -352,7 +412,7 @@ obj Plan { routes = routes_any; } deployments = dget(bundle, "deployments"); - autoscalers = dget(bundle, "autoscalers"); + scaling = dget(bundle, "scaling"); pdbs = dget(bundle, "pdbs"); user_routes: list[str] = [ @@ -378,11 +438,25 @@ obj Plan { services: list[ServiceView] = []; for svc in ordered { route_path: str | None = str(routes[svc]) if svc in routes else None; + plan_svc: Any = scaling.get(svc); + mode: Any = plan_svc.get("mode") if isinstance(plan_svc, dict) else None; + hpa: Any = ( + plan_svc.get("autoscaler") + if isinstance(plan_svc, dict) and mode == ScalingMode.METRIC + else None + ); + http: Any = ( + plan_svc.get("http_activation") + if isinstance(plan_svc, dict) + and mode == ScalingMode.HTTP_ACTIVATION + else None + ); services.append( ServiceView.from_manifests( name=svc, deployment=deployments.get(svc), - hpa=autoscalers.get(svc), + hpa=hpa, + http_activation=http, pdb=pdbs.get(svc), route_path=route_path ) @@ -532,6 +606,9 @@ obj Plan { out.append(m); } } + for m in lget(self.bundle, "autoscaler_manifests") { + out.append(m); + } if self.bundle.get("ingress") { out.append(self.bundle["ingress"]); } diff --git a/jac/jaclang/scale/runtime/docs.md b/jac/jaclang/scale/runtime/docs.md index b90639e8c99..3651e1a5cf6 100644 --- a/jac/jaclang/scale/runtime/docs.md +++ b/jac/jaclang/scale/runtime/docs.md @@ -322,6 +322,7 @@ code changes from local mode. | `rpc_timeout` | `10.0` | `sv import` httpx timeout (s) | | `http_forward_timeout` | `30.0` | gateway-to-service forward (s) | | `hpa.enabled` / `min` / `max` / `cpu_target` | `true` / `1` / `3` / `70` | autoscaler bounds (applies to both `"hpa"` and `"keda"` engines) | +| `hpa.behavior` | `{}` | Raw HPA `behavior` fragment (`scaleUp`/`scaleDown`) deep-merged over the generated scale-rate defaults - same merge rules as `deployment_overlay`. Applies to both engines; for `keda` it only governs scaling while replicas are above zero - the drop to zero is still controlled by `autoscaler_cooldown` (the ScaledObject's `cooldownPeriod`). See example below. | | `[[triggers]]` | `[]` | Per-service KEDA triggers (requires `autoscaler_engine = "keda"`). Each entry: `type` (required), `metadata` (default `{}`), `name` (default `null`), `auth.secret_refs` (default `{}`). Same shape as `[[scale.kubernetes.extra_triggers]]`. | | `pdb.enabled` / `max_unavailable` | `true` / `1` | PodDisruptionBudget | diff --git a/jac/jaclang/scale/tests/deploy/http_activation_test_support.jac b/jac/jaclang/scale/tests/deploy/http_activation_test_support.jac new file mode 100644 index 00000000000..05acd7fc3d0 --- /dev/null +++ b/jac/jaclang/scale/tests/deploy/http_activation_test_support.jac @@ -0,0 +1,90 @@ +"""Shared test helpers for jac-scale's KEDA HTTP activation tests. + +Canonical, copy-paste-free fixtures used by both the integration tests in +test_http_activation_microservices.jac and the jac.toml config-translation/ +wiring tests (test_http_activation_config.jac): a defaulted +HTTPActivationSpec, a defaulted jac.toml-shaped config dict, and a +KEDAAutoscaler wired to MagicMock Kubernetes clients. + +Example: + import from jaclang.scale.tests.deploy.http_activation_test_support { + default_http_activation_spec as _spec, + mocked_keda_autoscaler as _mocked_keda + } + spec = _spec(namespace="staging"); +""" + +import unittest.mock; +import from kubernetes.client.exceptions { ApiException } +import from jaclang.scale.deploy.autoscale.keda_autoscaler { KEDAAutoscaler } +import from jaclang.scale.deploy.autoscale.http_activation { + HTTPActivationSpec, + HTTPServiceTarget, + HTTPConcurrencyMetric +} + +# Base HTTPActivationSpec shared by most tests: a Deployment target on port +# 8080 with a concurrency=10 trigger. Override only what a given test cares +# about; pass concurrency=None when testing the metric-required check. +def default_http_activation_spec(**overrides: any) -> HTTPActivationSpec { + defaults = { + "name": "preview", + "namespace": "default", + "scale_target_name": "preview-deployment", + "target": HTTPServiceTarget(service="preview-svc", port=8080), + "concurrency": HTTPConcurrencyMetric(target_value=10) + }; + if "request_rate" in overrides { + defaults.pop("concurrency"); + } + defaults.update(overrides); + return HTTPActivationSpec(**defaults); +} + +# Base jac.toml [*.http_activation] config dict, the dict-side counterpart to +# default_http_activation_spec above: enabled, target_port=8080, +# concurrency_target=10, rules=[{hosts=["*"]}]. Override only what a given +# test cares about; pass request_rate_target=... to switch to the +# request-rate metric (drops concurrency_target the same way +# default_http_activation_spec drops concurrency when request_rate is +# supplied); pass rules=[] to exercise the empty-rules rejection itself. +def default_http_activation_config(**overrides: any) -> dict[str, any] { + defaults: dict[str, any] = { + "enabled": True, + "target_port": 8080, + "concurrency_target": 10, + "rules": [{"hosts": ["*"]}] + }; + if "request_rate_target" in overrides { + defaults.pop("concurrency_target"); + } + defaults.update(overrides); + return defaults; +} + +# KEDAAutoscaler wired to MagicMock clients, with the HTTP Add-on preflight +# check (list_cluster_custom_object) pre-configured as installed by default. +# Pass a pre-configured apps_api/core_api/custom_api to control target, +# Service, or route/ScaledObject existence for a specific test. Returns the +# mocks alongside the instance since KEDAAutoscaler's api fields are typed +# `any | None`, which the checker won't let a caller re-narrow after the fact. +def mocked_keda_autoscaler( + http_addon_installed: bool = True, + custom_api: any = None, + apps_api: any = None, + core_api: any = None +) -> tuple[KEDAAutoscaler, any, any, any] { + KEDAAutoscaler._preflight_cache.clear(); + custom_api = custom_api or unittest.mock.MagicMock(); + if http_addon_installed { + custom_api.list_cluster_custom_object.return_value = {}; + } else { + custom_api.list_cluster_custom_object.side_effect = ApiException(status=404); + } + apps_api = apps_api or unittest.mock.MagicMock(); + core_api = core_api or unittest.mock.MagicMock(); + keda = KEDAAutoscaler( + _custom_api=custom_api, _apps_api=apps_api, _core_api=core_api + ); + return (keda, custom_api, apps_api, core_api); +} diff --git a/jac/jaclang/scale/tests/deploy/keda_http_activation_real_e2e.sh b/jac/jaclang/scale/tests/deploy/keda_http_activation_real_e2e.sh index 19e6b1a4e1f..8b4d974cdaf 100755 --- a/jac/jaclang/scale/tests/deploy/keda_http_activation_real_e2e.sh +++ b/jac/jaclang/scale/tests/deploy/keda_http_activation_real_e2e.sh @@ -1,50 +1,97 @@ #!/usr/bin/env bash -# Real-cluster e2e for jac-scale KEDA HTTP Add-on activation (#7403/#7421). +# Real-cluster e2e for jac-scale KEDA HTTP Add-on activation (#7403/#7421), +# deployed through the jac.toml [scale.kubernetes.http_activation] wiring +# (#7475) instead of calling KEDAAutoscaler.apply_http_activation directly. # -# Starts from a zero-replica Deployment, applies HTTP activation twice (create -# then patch, to exercise both branches of apply_http_activation against a -# real API server), sends an HTTP request through the KEDA HTTP Add-on -# interceptor, waits for the target to become Ready, then confirms it scales -# back to zero after cooldown. Requires KEDA core + the HTTP Add-on already -# installed on the target cluster (this script does not install them -- see -# README.md in the fixture dir / the CI step that calls this script for the -# `helm install` invocations). +# Deploys the fixture app via `jac scale deploy`, confirms the resulting +# InterceptorRoute + ScaledObject reconcile to Ready, sends an HTTP request +# through the KEDA HTTP Add-on interceptor (should block on cold start, then +# respond), waits for the target to scale 0 -> 1 and become Available, then +# confirms it scales back to zero after cooldown. Requires KEDA core + the +# HTTP Add-on already installed on the target cluster (this script does not +# install them -- see README.md in the fixture dir for the `helm install` +# invocations). set -euo pipefail +# shellcheck source=../../scripts/e2e_lib.sh +source "$(cd "$(dirname "$0")/../../scripts" && pwd)/e2e_lib.sh" +e2e_timing_init + FIXTURE_DIR="${1:-$(cd "$(dirname "$0")/../fixtures/keda_http_activation_e2e" && pwd)}" -if [ ! -f "${FIXTURE_DIR}/fixture.yaml" ]; then - echo "FAIL: ${FIXTURE_DIR}/fixture.yaml not found" >&2 +if [ ! -f "${FIXTURE_DIR}/jac.toml" ]; then + echo "FAIL: ${FIXTURE_DIR}/jac.toml not found" >&2 echo "Usage: $0 [FIXTURE_DIR]" >&2 exit 1 fi -# This script lives at jac/jaclang/scale/tests/deploy/, so the repo root is -# five levels up. -REPO_ROOT="$(cd "$(dirname "$0")/../../../../.." && pwd)" -DRIVER="${REPO_ROOT}/jac/jaclang/scale/tests/deploy/keda_http_activation_verify.jac" -if [ ! -f "${DRIVER}" ]; then - echo "FAIL: driver script not found at ${DRIVER}" >&2 - exit 1 +# The manifest builder refuses to guess a RWX-capable StorageClass for the +# bundle PVC (see manifest_builder.jac): most cloud defaults are +# ReadWriteOnce-only, so it now requires bundle_storage_class set explicitly +# rather than trusting the cluster default. Local/kind runs edit jac.toml by +# hand per the fixture README; a CI lane on a different cluster type sets +# BUNDLE_STORAGE_CLASS instead, applied here so the file itself stays +# cluster-agnostic. +if [ -n "${BUNDLE_STORAGE_CLASS:-}" ]; then + python3 - "${FIXTURE_DIR}/jac.toml" "${BUNDLE_STORAGE_CLASS}" <<'PYEOF' +import sys + +path, storage_class = sys.argv[1], sys.argv[2] +with open(path) as f: + lines = f.readlines() +out, in_k8s, done = [], False, False +for line in lines: + stripped = line.strip() + if stripped.startswith("["): + in_k8s = stripped == "[scale.kubernetes]" + if in_k8s and stripped.startswith("bundle_storage_class"): + continue + out.append(line) + if in_k8s and not done and stripped == "[scale.kubernetes]": + out.append(f'bundle_storage_class = "{storage_class}"\n') + done = True +with open(path, "w") as f: + f.writelines(out) +PYEOF fi -export KEDA_HTTP_E2E_NAMESPACE="${KEDA_HTTP_E2E_NAMESPACE:-jac-http-e2e}" -export KEDA_HTTP_E2E_DEPLOYMENT="${KEDA_HTTP_E2E_DEPLOYMENT:-echo}" -export KEDA_HTTP_E2E_SERVICE="${KEDA_HTTP_E2E_SERVICE:-echo-svc}" -export KEDA_HTTP_E2E_ROUTE_HOST="${KEDA_HTTP_E2E_ROUTE_HOST:-echo.jac-http-e2e.local}" -export KEDA_HTTP_E2E_POLLING_INTERVAL="${KEDA_HTTP_E2E_POLLING_INTERVAL:-10}" -export KEDA_HTTP_E2E_COOLDOWN_PERIOD="${KEDA_HTTP_E2E_COOLDOWN_PERIOD:-60}" -NAMESPACE="${KEDA_HTTP_E2E_NAMESPACE}" -DEPLOYMENT="${KEDA_HTTP_E2E_DEPLOYMENT}" +CFG=$(cd "${FIXTURE_DIR}" && jac -c " +import tomllib +with open('jac.toml', 'rb') as f: + cfg = tomllib.load(f) +proj = cfg['project'] +k8s = cfg['scale']['kubernetes'] +act = k8s['http_activation'] +print(proj['name']) +print(k8s.get('namespace', 'default')) +print(act['rules'][0]['hosts'][0]) +print(act.get('polling_interval', 30)) +print(act.get('cooldown_period', 300)) +") +APP_NAME=$(echo "${CFG}" | sed -n '1p') +# The unified deploy path names every workload "-deployment" (even a solo +# app behind its gateway), and the KEDA InterceptorRoute/ScaledObject are named +# after that scale target, so they are "-deployment-http-{route,scaledobject}". +SCALE_TARGET="${APP_NAME}-deployment" +NAMESPACE=$(echo "${CFG}" | sed -n '2p') +ROUTE_HOST=$(echo "${CFG}" | sed -n '3p') +POLLING_INTERVAL=$(echo "${CFG}" | sed -n '4p') +COOLDOWN_PERIOD=$(echo "${CFG}" | sed -n '5p') + # Bare seconds, like every other *_TIMEOUT var here -- "s" is appended at the # call site. kubectl's --timeout requires a unit suffix (e.g. "120s"); baking # it into the default here would make DELETE_TIMEOUT the only timeout var # that breaks if overridden with a bare integer like the rest. DELETE_TIMEOUT="${DELETE_TIMEOUT:-120}" -READY_TIMEOUT="${READY_TIMEOUT:-90}" -# Bound the scale-down wait comfortably above cooldown + one poll tick so a +# A real jac-scale pod's cold start runs jac-pvc-bootstrap + jac-bootstrap +# init containers (installing deps, first-run compile) before it's Ready - +# tens of seconds, not the near-instant start of a bare container image. 90s +# (right for the old fixture.yaml's http-echo image) was too tight for that; +# 180s gives real headroom while still failing fast on an actual hang. +READY_TIMEOUT="${READY_TIMEOUT:-180}" +# Bound the scale-down wait comfortably above cooldown + three poll ticks so a # real hang fails loudly instead of the script exiting early on a fluke. -SCALE_DOWN_TIMEOUT="${SCALE_DOWN_TIMEOUT:-$(( KEDA_HTTP_E2E_COOLDOWN_PERIOD + KEDA_HTTP_E2E_POLLING_INTERVAL * 3 + 30 ))}" +SCALE_DOWN_TIMEOUT="${SCALE_DOWN_TIMEOUT:-$(( COOLDOWN_PERIOD + POLLING_INTERVAL * 3 + 30 ))}" echo "=== preflight: KEDA HTTP Add-on CRDs ===" if ! kubectl get crd interceptorroutes.http.keda.sh >/dev/null 2>&1; then @@ -62,9 +109,9 @@ dump_state() { kubectl get pods -n "${NAMESPACE}" -o wide || true kubectl describe pods -n "${NAMESPACE}" || true kubectl get events -n "${NAMESPACE}" --sort-by=.lastTimestamp || true - kubectl logs -n "${NAMESPACE}" -l "app=${DEPLOYMENT}" --tail=200 --all-containers=true || true - kubectl describe interceptorroute "${DEPLOYMENT}-http-route" -n "${NAMESPACE}" || true - kubectl describe scaledobject "${DEPLOYMENT}-http-scaledobject" -n "${NAMESPACE}" || true + kubectl logs -n "${NAMESPACE}" -l "app=${APP_NAME}" --tail=200 --all-containers=true || true + kubectl describe interceptorroute "${SCALE_TARGET}-http-route" -n "${NAMESPACE}" || true + kubectl describe scaledobject "${SCALE_TARGET}-http-scaledobject" -n "${NAMESPACE}" || true echo "--- HTTP Add-on component logs (namespace=keda) ---" kubectl logs -n keda -l app=keda-add-ons-http-interceptor --tail=100 || true kubectl logs -n keda -l app=keda-add-ons-http-external-scaler --tail=100 || true @@ -88,14 +135,12 @@ cleanup() { echo "=== e2e failed (rc=${rc}); KEEPING namespace '${NAMESPACE}' for inspection (set E2E_KEEP_NS_ON_FAIL=0 to force cleanup) ===" return fi - (cd "${REPO_ROOT}/jac" && jac run "${DRIVER}" destroy) || true + # Deleting the namespace sweeps the Deployment/Service and the namespaced + # InterceptorRoute/ScaledObject together; no separate destroy call needed. kubectl delete namespace "${NAMESPACE}" --ignore-not-found --timeout="${DELETE_TIMEOUT}s" || true } trap 'cleanup "$?"' EXIT -_T0=$(date +%s) -_t() { echo "[TIMING +$(( $(date +%s) - _T0 ))s] $1"; } - # Polls a resource's status.conditions[type=Ready].status, per the HTTP # Add-on's own "Autoscale an App" verify step (kubectl get and # check the READY column) -- done here as a jsonpath poll instead so the @@ -120,39 +165,29 @@ wait_for_ready() { return 1 } -_t "fixture apply start" -echo "=== apply zero-replica Deployment + Service fixture ===" -kubectl apply -f "${FIXTURE_DIR}/fixture.yaml" -REPLICAS=$(kubectl get deployment "${DEPLOYMENT}" -n "${NAMESPACE}" -o jsonpath='{.spec.replicas}') -if [ "${REPLICAS}" != "0" ]; then - echo "FAIL: fixture Deployment '${DEPLOYMENT}' did not start at 0 replicas (got ${REPLICAS})" >&2 - exit 1 -fi -echo " ${DEPLOYMENT} starts at 0 replicas" - -_t "apply_http_activation (create path)" -echo "=== apply_http_activation: reconcile InterceptorRoute + ScaledObject (create) ===" -if ! (cd "${REPO_ROOT}/jac" && jac run "${DRIVER}" apply); then - echo "FAIL: first apply_http_activation call (create path) errored" >&2 +_t "deploy start" +echo "=== deploy via jac scale deploy (jac.toml [scale.kubernetes.http_activation] wiring) ===" +if ! (cd "${FIXTURE_DIR}" && jac scale deploy app.jac); then + echo "FAIL: deploy failed" >&2 dump_state exit 1 fi -_t "apply_http_activation (patch path)" -echo "=== re-apply: both resources already exist, so this exercises the get-then-patch branch against the real API server (required behavior: idempotent create-or-patch) ===" -if ! (cd "${REPO_ROOT}/jac" && jac run "${DRIVER}" apply); then - echo "FAIL: second apply_http_activation call (patch path) errored" >&2 +_t "redeploy (idempotency check)" +echo "=== redeploy: confirms InterceptorRoute + ScaledObject reconcile is idempotent (get-then-patch, not create-or-duplicate) ===" +if ! (cd "${FIXTURE_DIR}" && jac scale deploy app.jac); then + echo "FAIL: redeploy failed" >&2 dump_state exit 1 fi _t "wait for InterceptorRoute + ScaledObject Ready" echo "=== confirm InterceptorRoute and ScaledObject reconciled to Ready ===" -if ! wait_for_ready interceptorroute "${DEPLOYMENT}-http-route"; then +if ! wait_for_ready interceptorroute "${SCALE_TARGET}-http-route"; then dump_state exit 1 fi -if ! wait_for_ready scaledobject "${DEPLOYMENT}-http-scaledobject"; then +if ! wait_for_ready scaledobject "${SCALE_TARGET}-http-scaledobject"; then dump_state exit 1 fi @@ -177,8 +212,9 @@ echo "=== send HTTP request through the interceptor (should block on cold start, RESP_BODY_FILE="$(mktemp)" RESP_CODE=$(curl -s -o "${RESP_BODY_FILE}" -w "%{http_code}" \ --max-time "${READY_TIMEOUT}" \ - -H "Host: ${KEDA_HTTP_E2E_ROUTE_HOST}" \ - "http://localhost:${INTERCEPTOR_LOCAL_PORT}/" || echo "000") + -X POST \ + -H "Host: ${ROUTE_HOST}" \ + "http://localhost:${INTERCEPTOR_LOCAL_PORT}/walker/echo" || echo "000") if [ "${RESP_CODE}" != "200" ]; then echo "FAIL: interceptor request returned '${RESP_CODE}' (expected 200) within ${READY_TIMEOUT}s" >&2 dump_state @@ -190,27 +226,27 @@ rm -f "${RESP_BODY_FILE}" _t "wait for readiness" echo "=== confirm the target scaled 0 -> 1 and became Ready ===" -if ! kubectl wait --for=condition=Available "deployment/${DEPLOYMENT}" \ +if ! kubectl wait --for=condition=Available "deployment/${SCALE_TARGET}" \ -n "${NAMESPACE}" --timeout="${READY_TIMEOUT}s"; then - echo "FAIL: ${DEPLOYMENT} did not become Available within ${READY_TIMEOUT}s" >&2 + echo "FAIL: ${APP_NAME} did not become Available within ${READY_TIMEOUT}s" >&2 dump_state exit 1 fi -READY_REPLICAS=$(kubectl get deployment "${DEPLOYMENT}" -n "${NAMESPACE}" \ +READY_REPLICAS=$(kubectl get deployment "${SCALE_TARGET}" -n "${NAMESPACE}" \ -o jsonpath='{.status.readyReplicas}') -echo " ${DEPLOYMENT} readyReplicas=${READY_REPLICAS}" +echo " ${APP_NAME} readyReplicas=${READY_REPLICAS}" kill "${PORT_FORWARD_PID}" 2>/dev/null || true PORT_FORWARD_PID="" _t "wait for scale-down after cooldown" -echo "=== stop traffic; wait up to ${SCALE_DOWN_TIMEOUT}s for scale-down after ${KEDA_HTTP_E2E_COOLDOWN_PERIOD}s cooldown ===" +echo "=== stop traffic; wait up to ${SCALE_DOWN_TIMEOUT}s for scale-down after ${COOLDOWN_PERIOD}s cooldown ===" SCALED_DOWN=0 ELAPSED=0 while [ "${ELAPSED}" -lt "${SCALE_DOWN_TIMEOUT}" ]; do - sleep "${KEDA_HTTP_E2E_POLLING_INTERVAL}" - ELAPSED=$(( ELAPSED + KEDA_HTTP_E2E_POLLING_INTERVAL )) - CURRENT_REPLICAS=$(kubectl get deployment "${DEPLOYMENT}" -n "${NAMESPACE}" \ + sleep "${POLLING_INTERVAL}" + ELAPSED=$(( ELAPSED + POLLING_INTERVAL )) + CURRENT_REPLICAS=$(kubectl get deployment "${SCALE_TARGET}" -n "${NAMESPACE}" \ -o jsonpath='{.spec.replicas}') echo " +${ELAPSED}s replicas=${CURRENT_REPLICAS}" if [ "${CURRENT_REPLICAS}" = "0" ]; then @@ -219,7 +255,7 @@ while [ "${ELAPSED}" -lt "${SCALE_DOWN_TIMEOUT}" ]; do fi done if [ "${SCALED_DOWN}" != "1" ]; then - echo "FAIL: ${DEPLOYMENT} did not scale back to 0 within ${SCALE_DOWN_TIMEOUT}s of cooldown" >&2 + echo "FAIL: ${APP_NAME} did not scale back to 0 within ${SCALE_DOWN_TIMEOUT}s of cooldown" >&2 dump_state exit 1 fi diff --git a/jac/jaclang/scale/tests/deploy/keda_http_activation_verify.jac b/jac/jaclang/scale/tests/deploy/keda_http_activation_verify.jac deleted file mode 100644 index d99517f7630..00000000000 --- a/jac/jaclang/scale/tests/deploy/keda_http_activation_verify.jac +++ /dev/null @@ -1,86 +0,0 @@ -"""Apply/destroy driver for the KEDA HTTP Add-on activation e2e -(keda_http_activation_real_e2e.sh). Talks to whatever cluster the current -kubeconfig/in-cluster context points at -- no mocking, this exercises -KEDAAutoscaler.apply_http_activation / destroy_http_activation for real. - - jac run keda_http_activation_verify.jac apply - jac run keda_http_activation_verify.jac destroy - -Target namespace/name/host are read from env vars (defaults match -../fixtures/keda_http_activation_e2e/fixture.yaml) so the orchestrating -shell script controls them without editing this file. -""" -import os; -import sys; -import from jaclang.scale._optdeps.kubernetes { config as k8s_config, ConfigException } -import from jaclang.scale.deploy.autoscale.keda_autoscaler { KEDAAutoscaler } -import from jaclang.scale.deploy.autoscale.http_activation { - HTTPActivationSpec, - HTTPServiceTarget, - HTTPRoutingRule, - HTTPConcurrencyMetric -} -import from jaclang.scale.observability.standard_logger { StandardLogger } - -glob NAMESPACE: str = os.environ.get("KEDA_HTTP_E2E_NAMESPACE", "jac-http-e2e"), - DEPLOYMENT: str = os.environ.get("KEDA_HTTP_E2E_DEPLOYMENT", "echo"), - SERVICE: str = os.environ.get("KEDA_HTTP_E2E_SERVICE", "echo-svc"), - ROUTE_HOST: str = os.environ.get( - "KEDA_HTTP_E2E_ROUTE_HOST", "echo.jac-http-e2e.local" - ), - POLLING_INTERVAL: int = int( - os.environ.get("KEDA_HTTP_E2E_POLLING_INTERVAL", "10") - ), - COOLDOWN_PERIOD: int = int(os.environ.get("KEDA_HTTP_E2E_COOLDOWN_PERIOD", "60")); - -def load_config { - # any: sidesteps a checker false positive on _optdeps' guarded import (NoneType inferred from its ImportError fallback branch). - kube_config: any = k8s_config; - try { - kube_config.load_kube_config(); - } except ConfigException { - kube_config.load_incluster_config(); - } -} - -def build_spec -> HTTPActivationSpec { - return HTTPActivationSpec( - name="e2e", - namespace=NAMESPACE, - scale_target_name=DEPLOYMENT, - target=HTTPServiceTarget(service=SERVICE, port=80), - rules=[HTTPRoutingRule(hosts=[ROUTE_HOST])], - min_replicas=0, - max_replicas=1, - polling_interval=POLLING_INTERVAL, - cooldown_period=COOLDOWN_PERIOD, - concurrency=HTTPConcurrencyMetric(target_value=1) - ); -} - -with entry { - if len(sys.argv) < 2 or sys.argv[1] not in ("apply", "destroy") { - print("usage: jac run keda_http_activation_verify.jac [apply|destroy]"); - sys.exit(1); - } - - load_config(); -} - -glob keda = KEDAAutoscaler(logger=StandardLogger()); - -with entry { - if sys.argv[1] == "apply" { - ok = keda.apply_http_activation(build_spec()); - if not ok { - print("apply_http_activation returned False (HTTP Add-on CRDs missing?)"); - sys.exit(1); - } - print(f"apply_http_activation -> {ok}"); - print(f"InterceptorRoute: {keda.interceptor_route_name_for(DEPLOYMENT)}"); - print(f"ScaledObject: {keda.http_scaled_object_name_for(DEPLOYMENT)}"); - } else { - keda.destroy_http_activation(DEPLOYMENT, NAMESPACE); - print("destroy_http_activation -> done"); - } -} diff --git a/jac/jaclang/scale/tests/deploy/test_deployment_overlay.jac b/jac/jaclang/scale/tests/deploy/test_deployment_overlay.jac index 2c4fad2cc97..2f6560d472d 100644 --- a/jac/jaclang/scale/tests/deploy/test_deployment_overlay.jac +++ b/jac/jaclang/scale/tests/deploy/test_deployment_overlay.jac @@ -10,6 +10,7 @@ import from jaclang.scale.deploy.target.kubernetes.kubernetes_config { KubernetesConfig } import from jaclang.scale.config.config_loader { JacScaleConfig } +import from jaclang.scale.deploy.autoscale.autoscaler { Autoscaler, AutoscalerSpec } def _builder(services: dict) -> ManifestBuilder { @@ -49,6 +50,93 @@ test "unnamed lists are replaced wholesale" { assert merged["args"] == ["c"]; } +test "build_behavior with no overlay returns the hardcoded default" { + spec = AutoscalerSpec(scale_target_name="x", namespace="ns"); + result = Autoscaler._build_behavior(spec); + assert result["scaleDown"]["policies"] == [ + {"type": "Percent", "value": 50, "periodSeconds": 60} + ]; + assert result["scaleUp"]["policies"] == [ + {"type": "Pods", "value": 2, "periodSeconds": 60} + ]; +} + +test "build_behavior with an empty overlay dict is a no-op" { + default = Autoscaler._build_behavior( + AutoscalerSpec(scale_target_name="x", namespace="ns") + ); + overlaid = Autoscaler._build_behavior( + AutoscalerSpec(scale_target_name="x", namespace="ns", behavior_overlay={}) + ); + assert overlaid == default; +} + +test "build_behavior overlay replaces scaleDown policy shape, leaves scaleUp default" { + spec = AutoscalerSpec( + scale_target_name="x", + namespace="ns", + behavior_overlay={ + "scaleDown": { + "stabilizationWindowSeconds": 600, + "selectPolicy": "Min", + "policies": [{"type": "Pods", "value": 2, "periodSeconds": 300}] + } + } + ); + result = Autoscaler._build_behavior(spec); + assert result["scaleDown"]["stabilizationWindowSeconds"] == 600; + assert result["scaleDown"]["selectPolicy"] == "Min"; + assert result["scaleDown"]["policies"] == [ + {"type": "Pods", "value": 2, "periodSeconds": 300} + ]; + assert result["scaleUp"]["stabilizationWindowSeconds"] == 60; + assert result["scaleUp"]["policies"] == [ + {"type": "Pods", "value": 2, "periodSeconds": 60} + ]; +} + +test "build_behavior overlay on scaleUp does not disturb the scaleDown default" { + spec = AutoscalerSpec( + scale_target_name="x", + namespace="ns", + behavior_overlay={ + "scaleUp": { + "policies": [{"type": "Percent", "value": 100, "periodSeconds": 15}] + } + } + ); + result = Autoscaler._build_behavior(spec); + assert result["scaleUp"]["policies"] == [ + {"type": "Percent", "value": 100, "periodSeconds": 15} + ]; + assert result["scaleUp"]["stabilizationWindowSeconds"] == 60; + assert result["scaleDown"]["policies"] == [ + {"type": "Percent", "value": 50, "periodSeconds": 60} + ]; +} + +test "build_behavior overlay policies list replaces wholesale, not merged by item" { + # Policy dicts carry no `name` key, so merge_manifest_overlay treats + # `policies` as a plain list and replaces it wholesale rather than + # merging entries by position or identity. + spec = AutoscalerSpec( + scale_target_name="x", + namespace="ns", + behavior_overlay={ + "scaleDown": { + "policies": [ + {"type": "Pods", "value": 1, "periodSeconds": 120}, + {"type": "Percent", "value": 10, "periodSeconds": 120} + ] + } + } + ); + result = Autoscaler._build_behavior(spec); + assert len(result["scaleDown"]["policies"]) == 2; + assert result["scaleDown"]["policies"][0]["value"] == 1; + assert result["scaleDown"]["policies"][1]["value"] == 10; +} + test "deployment manifest applies the service overlay on the first build" { b = _builder( { diff --git a/jac/jaclang/scale/tests/deploy/test_factories.jac b/jac/jaclang/scale/tests/deploy/test_factories.jac index bf1d1a1fbbb..36b9e86c953 100644 --- a/jac/jaclang/scale/tests/deploy/test_factories.jac +++ b/jac/jaclang/scale/tests/deploy/test_factories.jac @@ -267,46 +267,76 @@ test "hpa autoscaler build manifests produces correct dict" { triggers=[Trigger(type="cpu", metadata={"averageUtilization": "70"})] ); result = HPAAutoscaler()._build_manifests(spec); - expected = { - "apiVersion": "autoscaling/v2", - "kind": "HorizontalPodAutoscaler", - "metadata": { - "name": "my-service-hpa", - "namespace": "default", - "labels": {"app": "my-service-deployment", "managed": "jac-scale"} - }, - "spec": { - "scaleTargetRef": { - "apiVersion": "apps/v1", - "kind": "Deployment", - "name": "my-service-deployment" + expected = [ + { + "apiVersion": "autoscaling/v2", + "kind": "HorizontalPodAutoscaler", + "metadata": { + "name": "my-service-hpa", + "namespace": "default", + "labels": {"app": "my-service-deployment", "managed": "jac-scale"} }, - "minReplicas": 1, - "maxReplicas": 3, - "metrics": [ - { - "type": "Resource", - "resource": { - "name": "cpu", - "target": {"type": "Utilization", "averageUtilization": 70} - } - } - ], - "behavior": { - "scaleUp": { - "stabilizationWindowSeconds": 60, - "policies": [{"type": "Pods", "value": 2, "periodSeconds": 60}] + "spec": { + "scaleTargetRef": { + "apiVersion": "apps/v1", + "kind": "Deployment", + "name": "my-service-deployment" }, - "scaleDown": { - "stabilizationWindowSeconds": 300, - "policies": [{"type": "Percent", "value": 50, "periodSeconds": 60}] + "minReplicas": 1, + "maxReplicas": 3, + "metrics": [ + { + "type": "Resource", + "resource": { + "name": "cpu", + "target": {"type": "Utilization", "averageUtilization": 70} + } + } + ], + "behavior": { + "scaleUp": { + "stabilizationWindowSeconds": 60, + "policies": [{"type": "Pods", "value": 2, "periodSeconds": 60}] + }, + "scaleDown": { + "stabilizationWindowSeconds": 300, + "policies": [ + {"type": "Percent", "value": 50, "periodSeconds": 60} + ] + } } } } - }; + ]; assert result == expected; } +test "hpa autoscaler build manifests applies behavior_overlay onto the default shape" { + spec = AutoscalerSpec( + scale_target_name="my-service-deployment", + autoscaler_name="my-service-hpa", + namespace="default", + min_replicas=1, + max_replicas=3, + triggers=[Trigger(type="cpu", metadata={"averageUtilization": "70"})], + behavior_overlay={ + "scaleDown": { + "stabilizationWindowSeconds": 600, + "selectPolicy": "Min", + "policies": [{"type": "Pods", "value": 2, "periodSeconds": 300}] + } + } + ); + result = HPAAutoscaler()._build_manifests(spec)[0]; + behavior = result["spec"]["behavior"]; + assert behavior["scaleDown"]["stabilizationWindowSeconds"] == 600; + assert behavior["scaleDown"]["selectPolicy"] == "Min"; + assert behavior["scaleDown"]["policies"] == [ + {"type": "Pods", "value": 2, "periodSeconds": 300} + ]; + assert behavior["scaleUp"]["stabilizationWindowSeconds"] == 60; +} + test "autoscaler factory creates keda autoscaler for engine=keda" { autoscaler = AutoscalerFactory.create("keda", {}); assert isinstance(autoscaler, KEDAAutoscaler); diff --git a/jac/jaclang/scale/tests/deploy/test_http_activation_config.jac b/jac/jaclang/scale/tests/deploy/test_http_activation_config.jac new file mode 100644 index 00000000000..209efb1d5d6 --- /dev/null +++ b/jac/jaclang/scale/tests/deploy/test_http_activation_config.jac @@ -0,0 +1,241 @@ +import unittest.mock; +import from kubernetes.client.exceptions { ApiException } +import from jaclang.scale.deploy.autoscale.keda_autoscaler { KEDAAutoscaler } +import from jaclang.scale.deploy.autoscale.http_activation_config { + build_http_activation_spec, + apply_http_activation_for_target, + destroy_http_activation_for_target +} +import from jaclang.scale.tests.deploy.http_activation_test_support { + default_http_activation_config as _cfg, + mocked_keda_autoscaler as _mocked_keda +} + + +# --- Section A: enabled gating ----------------------------------------------- +test "disabled or missing enabled key returns None instead of a spec" { + assert build_http_activation_spec( + _cfg(enabled=False), "preview-deployment", "staging", "preview-svc" + ) is None; + assert build_http_activation_spec( + {}, "preview-deployment", "staging", "preview-svc" + ) is None; +} + + +# --- Section B: minimal config ----------------------------------------------- +test "minimal config (enabled, target_port, concurrency_target) builds a spec with correct defaults everywhere else" { + spec = build_http_activation_spec( + _cfg(), "preview-deployment", "staging", "preview-svc" + ); + assert spec is not None; + assert spec.scale_target_name == "preview-deployment"; + assert spec.namespace == "staging"; + assert spec.target.service == "preview-svc"; + assert spec.target.port == 8080; + assert spec.target.port_name is None; + assert spec.concurrency.target_value == 10; + assert spec.request_rate is None; + assert spec.min_replicas == 0; + assert spec.max_replicas == 1; + assert spec.polling_interval == 30; + assert spec.cooldown_period == 300; + assert len(spec.rules) == 1; + assert spec.rules[0].hosts == ["*"]; + assert spec.cold_start is None; + assert spec.timeouts is None; + assert spec.scale_target_kind == "Deployment"; + assert spec.scale_target_api_version == "apps/v1"; + assert spec.scale_target_plural is None; +} + + +# --- Section C: full config (every optional field, exercised once) ---------- +test "full config translates rules, cold_start, timeouts, request_rate, and custom target kind" { + cfg = _cfg( + min_replicas=1, + max_replicas=5, + polling_interval=15, + cooldown_period=120, + target_port=None, + target_port_name="http", + request_rate_target=50, + request_rate_window="5m", + request_rate_granularity="10s", + rules=[ + { + "hosts": ["preview.jac.dev"], + "paths": ["/app"], + "headers": [{"name": "x-preview-id", "value": "123"}] + } + ], + cold_start_status_code=503, + cold_start_body="warming up", + cold_start_headers={"Retry-After": "2"}, + cold_start_fallback_service="fallback-svc", + cold_start_fallback_port=8080, + timeout_readiness="30s", + timeout_request="60s", + timeout_response_header="5s", + scale_target_kind="StatefulSet" + ); + spec = build_http_activation_spec( + cfg, "preview-deployment", "staging", "preview-svc" + ); + assert spec.min_replicas == 1; + assert spec.max_replicas == 5; + assert spec.polling_interval == 15; + assert spec.cooldown_period == 120; + assert spec.target.port is None; + assert spec.target.port_name == "http"; + assert spec.concurrency is None; + assert spec.request_rate.target_value == 50; + assert spec.request_rate.rate_window == "5m"; + assert spec.request_rate.granularity == "10s"; + assert len(spec.rules) == 1; + assert spec.rules[0].hosts == ["preview.jac.dev"]; + assert spec.rules[0].paths[0].value == "/app"; + assert spec.rules[0].headers[0].name == "x-preview-id"; + assert spec.rules[0].headers[0].value == "123"; + assert spec.cold_start.placeholder.status_code == 503; + assert spec.cold_start.placeholder.body == "warming up"; + assert spec.cold_start.placeholder.headers == {"Retry-After": "2"}; + assert spec.cold_start.fallback_service.service == "fallback-svc"; + assert spec.cold_start.fallback_service.port == 8080; + assert spec.timeouts.readiness == "30s"; + assert spec.timeouts.request == "60s"; + assert spec.timeouts.response_header == "5s"; + assert spec.scale_target_kind == "StatefulSet"; +} + + +# --- Section D: jac.toml-relative validation errors -------------------------- +# One shape, four inputs: each bad cfg must raise ValueError (never a bare +# KeyError or anything else -- left uncaught here, so it fails the test loudly +# instead of silently) naming the jac.toml key(s) at fault. +def _assert_config_error(cfg: dict[str, any], *substrings: str) -> None { + try { + build_http_activation_spec(cfg, "preview-deployment", "staging", "preview-svc"); + } except ValueError as e { + msg = str(e); + for s in substrings { + assert s in msg , f"expected '{s}' in error message: {msg}"; + } + return; + } + assert False , "expected build_http_activation_spec to raise ValueError"; +} + +test "config with both target_port and target_port_name raises a jac.toml-relative error" { + _assert_config_error( + _cfg(target_port=8080, target_port_name="http"), + "target_port", + "http_activation" + ); +} + +test "config with neither concurrency_target nor request_rate_target raises a jac.toml-relative error" { + _assert_config_error( + _cfg(concurrency_target=None), "concurrency_target", "request_rate_target" + ); +} + +test "config with empty rules raises instead of silently deploying an unreachable target" { + _assert_config_error(_cfg(rules=[]), "rules"); +} + +test "a rule header missing its required 'name' key raises a jac.toml-relative error, not a bare KeyError" { + _assert_config_error(_cfg(rules=[{"headers": [{"value": "yes"}]}]), "name"); +} + + +# --- Section E: apply/destroy wiring helpers (injectable KEDAAutoscaler) ---- +# These back the monolith (kubernetes_target.jac) and microservices +# (manifest_builder.jac / microservice/target.jac) deploy-path call sites. +# kubernetes_target.jac's deploy() is a single large method that talks to a +# real cluster end to end with no isolated seam to unit test (it is only +# ever exercised against a live cluster, in test_deploy_k8s.jac) - these two +# functions exist so the HTTP-activation call itself stays testable without +# that whole method, by taking the KEDAAutoscaler as an explicit parameter. +test "apply_http_activation_for_target calls apply_http_activation with a correctly-targeted spec when enabled" { + (keda, custom_api, _, _) = _mocked_keda(); + custom_api.get_namespaced_custom_object.side_effect = ApiException(status=404); + result = apply_http_activation_for_target( + _cfg(), "preview-deployment", "staging", "preview-svc", keda + ); + assert result == True; + create_calls = custom_api.create_namespaced_custom_object.call_args_list; + assert len(create_calls) == 2; + route_body = create_calls[0][1]["body"]; + assert route_body["spec"]["target"]["service"] == "preview-svc"; + scaled_obj_body = create_calls[1][1]["body"]; + assert scaled_obj_body["metadata"]["name"] == keda.http_scaled_object_name_for( + "preview-deployment" + ); +} + +test "apply_http_activation_for_target is a no-op when disabled or unset" { + (keda, custom_api, _, _) = _mocked_keda(); + assert apply_http_activation_for_target( + _cfg(enabled=False), "preview-deployment", "staging", "preview-svc", keda + ) is None; + assert apply_http_activation_for_target( + {}, "preview-deployment", "staging", "preview-svc", keda + ) is None; + custom_api.create_namespaced_custom_object.assert_not_called(); +} + +test "destroy_http_activation_for_target always sweeps both resources, independent of current config" { + # Deliberately unconditional: a resource created while http_activation was + # enabled must still be cleaned up if the config later disables it or + # drops the block entirely, otherwise it is orphaned. destroy_http_ + # activation is 404-safe per-resource, so calling it unconditionally + # (mirroring the base autoscaler's own unconditional destroy() call) is + # always safe, whether or not anything was ever actually created. + mock_api = unittest.mock.MagicMock(); + keda = KEDAAutoscaler(_custom_api=mock_api); + destroy_http_activation_for_target("preview-deployment", "staging", keda); + delete_calls = mock_api.delete_namespaced_custom_object.call_args_list; + plurals = [c[1]["plural"] for c in delete_calls]; + assert "interceptorroutes" in plurals; + assert "scaledobjects" in plurals; +} + +test "destroy_http_activation_for_target swallows a permission error instead of aborting the rest of teardown" { + # The HTTP Add-on CRD group (http.keda.sh) is installed separately from + # core KEDA; a cluster's RBAC may cover keda.sh (which the base + # autoscaler already needs) without yet covering this newer group. A 403 + # here must not stop the Deployment/Service/PVC/ingress/monitoring/ + # database cleanup that runs after this call in the monolith destroy + # path, mirroring the "HTTP Add-on might not be fully available" warn- + # and-continue precedent already used by apply_http_activation's + # preflight check. + mock_api = unittest.mock.MagicMock(); + mock_api.delete_namespaced_custom_object.side_effect = ApiException(status=403); + keda = KEDAAutoscaler(_custom_api=mock_api); + raised = False; + try { + destroy_http_activation_for_target("preview-deployment", "staging", keda); + } except ApiException { + raised = True; + } + assert not raised , "A 403 must not propagate out of destroy_http_activation_for_target"; +} + +test "destroy_http_activation_for_target still propagates an unexpected error like a 500" { + # Only a 403 is treated as "HTTP Add-on RBAC not provisioned yet" and + # swallowed. A 500/502/503 is a genuinely different failure mode (the + # API server itself is having trouble, not a permission gap) and must + # still surface to the caller instead of silently leaving the + # InterceptorRoute/ScaledObject orphaned with no error reported. + mock_api = unittest.mock.MagicMock(); + mock_api.delete_namespaced_custom_object.side_effect = ApiException(status=500); + keda = KEDAAutoscaler(_custom_api=mock_api); + raised = False; + try { + destroy_http_activation_for_target("preview-deployment", "staging", keda); + } except ApiException { + raised = True; + } + assert raised , "A 500 must propagate out of destroy_http_activation_for_target"; +} diff --git a/jac/jaclang/scale/tests/deploy/test_http_activation_microservices.jac b/jac/jaclang/scale/tests/deploy/test_http_activation_microservices.jac new file mode 100644 index 00000000000..185669656aa --- /dev/null +++ b/jac/jaclang/scale/tests/deploy/test_http_activation_microservices.jac @@ -0,0 +1,1391 @@ +import from kubernetes.client.exceptions { ApiException } +import from jaclang.scale.deploy.target.kubernetes.manifest_builder { + ManifestBuilder, + GATEWAY_NAME, + ScalingMode +} +import from jaclang.scale.deploy.target.kubernetes.target { + KubernetesTarget, + apply_http_activations_for_bundle, + reap_stale_scaling_resources_for_bundle +} +import unittest.mock; +import from jaclang.scale.deploy.target.kubernetes.kubernetes_config { + KubernetesConfig +} +import from jaclang.scale.observability.standard_logger { StandardLogger } +import from jaclang.scale.deploy.autoscale.keda_autoscaler { KEDAAutoscaler } +import from jaclang.scale.deploy.autoscale.autoscaler { + AutoscalerSpec, + Trigger, + TriggerAuth +} +import from jaclang.scale.deploy.autoscale.http_activation { + HTTPRoutingRule, + HTTPPathMatch, + HTTPHeaderMatch, + HTTPServiceTarget, + HTTPRequestRateMetric, + HTTPStaticResponse, + HTTPColdStartSpec, + HTTPTimeoutSpec +} +import from jaclang.scale.tests.deploy.http_activation_test_support { + default_http_activation_config as _cfg, + default_http_activation_spec as _spec, + mocked_keda_autoscaler as _mocked_keda +} + + +def _builder( + services: dict, k8s_config: KubernetesConfig | None = None +) -> ManifestBuilder { + return ManifestBuilder( + k8s_config=k8s_config or KubernetesConfig(app_name="t", namespace="t-ns"), + microservices_config={"services": services}, + dry_run=True + ); +} + + +# Minimal Deployment/Service manifests: _resolve_scaling only reads +# metadata.name off each, so a full manifest is unnecessary here. +def _dep(name: str) -> dict { + return {"metadata": {"name": name}}; +} + + +def _svc(name: str) -> dict { + return {"metadata": {"name": name}}; +} + + +# --- Section A: ManifestBuilder._get_http_activation_config ----------------- +# Thin pass-through (config dict in, same merged dict or None out) - the actual +# dict-to-spec translation is build_http_activation_spec's job, already covered +# in test_http_activation_config.jac. This only proves the enabled gate and the +# per-service <- top-level inheritance merge; the gateway exclusion and the +# metric-vs-activation decision live one layer up, in _resolve_scaling. +test "service with http_activation.enabled=true returns the config dict unchanged" { + b = _builder( + {"billing_ops": {"http_activation": {"enabled": True, "target_port": 8080}}} + ); + cfg = b._get_http_activation_config("billing_ops"); + assert cfg is not None; + assert cfg["target_port"] == 8080; +} + +test "service with http_activation disabled or unset returns None" { + b = _builder( + {"billing_ops": {"http_activation": {"enabled": False}}, "other_svc": {}} + ); + assert b._get_http_activation_config("billing_ops") is None; + assert b._get_http_activation_config("other_svc") is None; +} + +test "per-service http_activation inherits unset keys from [scale.kubernetes.http_activation], down to no per-service block at all" { + b = _builder( + {"billing_ops": {"http_activation": {"enabled": True, "target_port": 8080}}}, + k8s_config=KubernetesConfig( + app_name="t", + namespace="t-ns", + http_activation={"concurrency_target": 10, "polling_interval": 15} + ) + ); + cfg = b._get_http_activation_config("billing_ops"); + assert cfg is not None; + assert cfg["target_port"] == 8080; + assert cfg["concurrency_target"] == 10; + assert cfg["polling_interval"] == 15; + + # Same merge at its edge: no per-service key at all still inherits + # `enabled` too -- absent and {} are the same input to the merge. + b2 = _builder( + {"other_svc": {}}, + k8s_config=KubernetesConfig( + app_name="t", + namespace="t-ns", + http_activation={ + "enabled": True, + "target_port": 8080, + "concurrency_target": 10 + } + ) + ); + cfg2 = b2._get_http_activation_config("other_svc"); + assert cfg2 is not None; + assert cfg2["target_port"] == 8080; + assert cfg2["concurrency_target"] == 10; +} + +test "per-service http_activation overrides [scale.kubernetes.http_activation] on shared keys" { + b = _builder( + { + "billing_ops": { + "http_activation": {"enabled": True, "concurrency_target": 5} + } + }, + k8s_config=KubernetesConfig( + app_name="t", + namespace="t-ns", + http_activation={"enabled": True, "concurrency_target": 10} + ) + ); + cfg = b._get_http_activation_config("billing_ops"); + assert cfg["concurrency_target"] == 5; +} + +test "a service can explicitly opt out with enabled=false even when the top-level default is enabled" { + b = _builder( + {"opted_out": {"http_activation": {"enabled": False}}}, + k8s_config=KubernetesConfig( + app_name="t", namespace="t-ns", http_activation={"enabled": True} + ) + ); + assert b._get_http_activation_config("opted_out") is None; +} + + +# --- Section B: _resolve_scaling (the single scaling decision) -------------- +test "_resolve_scaling: an http_activation service is HTTP_ACTIVATION, carries the {name}-deployment target, and emits no metric autoscaler" { + b = _builder( + { + "billing_ops": { + "http_activation": { + "enabled": True, + "target_port": 8080, + "concurrency_target": 10, + "rules": [{"hosts": ["*"]}] + } + } + } + ); + plan = b._resolve_scaling( + "billing_ops", _dep("billing-ops-deployment"), _svc("billing-ops-service") + ); + assert plan["mode"] == ScalingMode.HTTP_ACTIVATION; + # The scale target is the real Deployment name, never the bare k8s name - + # this is the exact drift that 404'd apply_http_activation on a real cluster. + assert plan["scale_target_name"] == "billing-ops-deployment"; + assert plan["service_name"] == "billing-ops-service"; + # Mutual exclusion: activation replaces the metric autoscaler, never coexists + # (KEDA rejects a second ScaledObject on an HPA-managed target). + assert plan["autoscaler"] == {}; +} + +test "_resolve_scaling: the gateway never inherits http_activation even when the top-level default enables it" { + b = _builder( + {}, + k8s_config=KubernetesConfig( + app_name="t", + namespace="t-ns", + http_activation={ + "enabled": True, + "target_port": 8080, + "concurrency_target": 10 + } + ) + ); + # A real logger (not None, not a MagicMock) so the gateway-exclusion branch + # actually calls logger.warn - this is what catches a wrong method name like + # .warning, which only surfaces on a live deploy otherwise. + b.logger = StandardLogger(); + plan = b._resolve_scaling( + GATEWAY_NAME, _dep("gateway-deployment"), _svc("gateway-service") + ); + # The gateway is the ingress entry point: scaling it to zero black-holes all + # inbound traffic, so it must never pick up the shared activation default. + assert plan["mode"] != ScalingMode.HTTP_ACTIVATION; + assert plan["http_activation"] == {}; +} + +test "_resolve_scaling: a plain service with the default hpa is METRIC, not activation" { + b = _builder({"worker": {}}); + plan = b._resolve_scaling( + "worker", _dep("worker-deployment"), _svc("worker-service") + ); + assert plan["mode"] == ScalingMode.METRIC; + assert plan["autoscaler"] != {}; + assert plan["http_activation"] == {}; + assert plan["scale_target_name"] == "worker-deployment"; +} + + +# --- Section C: apply_http_activations_for_bundle (per-service apply loop) --- +test "apply_http_activations_for_bundle applies only HTTP_ACTIVATION services, using the carried scale-target + service names" { + (keda, custom_api, _, _) = _mocked_keda(); + custom_api.get_namespaced_custom_object.side_effect = ApiException(status=404); + bundle = { + "scaling": { + "billing_ops": { + "mode": ScalingMode.HTTP_ACTIVATION, + "scale_target_name": "billing-ops-deployment", + "service_name": "billing-ops-service", + "autoscaler": {}, + "http_activation": _cfg() + }, + "worker": { + "mode": ScalingMode.METRIC, + "scale_target_name": "worker-deployment", + "service_name": "worker-service", + "autoscaler": {"min": 1, "max": 3}, + "http_activation": {} + } + } + }; + apply_http_activations_for_bundle(bundle, "t-ns", keda); + create_calls = custom_api.create_namespaced_custom_object.call_args_list; + # Two resources (InterceptorRoute + ScaledObject) for billing_ops only; the + # METRIC-mode worker is skipped entirely. + assert len(create_calls) == 2; + route_body = create_calls[0][1]["body"]; + assert route_body["spec"]["target"]["service"] == "billing-ops-service"; + assert route_body["metadata"]["namespace"] == "t-ns"; +} + +test "apply_http_activations_for_bundle is a no-op on an empty bundle" { + (keda, custom_api, _, _) = _mocked_keda(); + apply_http_activations_for_bundle({}, "t-ns", keda); + custom_api.create_namespaced_custom_object.assert_not_called(); +} + + +# --- Section D: reap_stale_scaling_resources_for_bundle (mode-switch cleanup) +# A service can switch scaling mode between deploys. Applying the new mode's +# resource never removes the old one on its own, so these prove the reap step +# deletes whichever mode a service is NOT in, for every mode transition - +# exactly what a fresh-deploy-only e2e cannot exercise. +test "a METRIC service has its stale HTTP activation resources reaped, autoscaler left alone" { + (keda, custom_api, _, _) = _mocked_keda(); + autoscaler = unittest.mock.MagicMock(); + bundle = { + "scaling": { + "worker": { + "mode": ScalingMode.METRIC, + "scale_target_name": "worker-deployment", + "service_name": "worker-service", + "autoscaler": {"min": 1, "max": 3}, + "http_activation": {} + } + } + }; + reap_stale_scaling_resources_for_bundle(bundle, "t-ns", autoscaler, keda); + autoscaler.destroy.assert_not_called(); + delete_calls = custom_api.delete_namespaced_custom_object.call_args_list; + assert len(delete_calls) == 2; + deleted_names = {c[1]["name"] for c in delete_calls}; + assert deleted_names == { + "worker-deployment-http-route", + "worker-deployment-http-scaledobject" + }; +} + +test "an HTTP_ACTIVATION service has its stale metric autoscaler destroyed, activation left alone" { + # destroy() resolves its own resource name internally (resource_name_for + # is called inside KEDAAutoscaler.destroy / HPAAutoscaler.destroy), so + # the reap call site must pass the plain k8s-safe service name, not a + # pre-resolved one -- a MagicMock with a canned resource_name_for return + # would stay green even if the call site started double-wrapping it, so + # this pins the literal argument destroy() receives instead. + (keda, custom_api, _, _) = _mocked_keda(); + autoscaler = unittest.mock.MagicMock(); + bundle = { + "scaling": { + "billing_ops": { + "mode": ScalingMode.HTTP_ACTIVATION, + "scale_target_name": "billing-ops-deployment", + "service_name": "billing-ops-service", + "autoscaler": {}, + "http_activation": _cfg() + } + } + }; + reap_stale_scaling_resources_for_bundle(bundle, "t-ns", autoscaler, keda); + autoscaler.destroy.assert_called_once_with("billing-ops", "t-ns"); + custom_api.delete_namespaced_custom_object.assert_not_called(); +} + +test "a NONE service has both its stale autoscaler and HTTP activation resources reaped" { + (keda, custom_api, _, _) = _mocked_keda(); + autoscaler = unittest.mock.MagicMock(); + bundle = { + "scaling": { + "idle_svc": { + "mode": ScalingMode.NONE, + "scale_target_name": "idle-svc-deployment", + "service_name": "idle-svc-service", + "autoscaler": {}, + "http_activation": {} + } + } + }; + reap_stale_scaling_resources_for_bundle(bundle, "t-ns", autoscaler, keda); + autoscaler.destroy.assert_called_once(); + assert len(custom_api.delete_namespaced_custom_object.call_args_list) == 2; +} + +test "reap_stale_scaling_resources_for_bundle is a no-op on an empty bundle" { + (keda, custom_api, _, _) = _mocked_keda(); + autoscaler = unittest.mock.MagicMock(); + reap_stale_scaling_resources_for_bundle({}, "t-ns", autoscaler, keda); + autoscaler.destroy.assert_not_called(); + custom_api.delete_namespaced_custom_object.assert_not_called(); +} + + +# --- Section E: _validate_http_activation_rules (interceptor routing collisions) +# Two services that both inherit the shared [scale.kubernetes.http_activation] +# default unchanged resolve to identical InterceptorRoute match rules; the +# interceptor then has no way to tell which target a matching request wakes. +test "two services sharing identical non-empty rules raise ValueError" { + b = _builder({}); + scaling = { + "billing_ops": { + "mode": ScalingMode.HTTP_ACTIVATION, + "http_activation": {"rules": [{"hosts": ["*"]}]} + }, + "worker": { + "mode": ScalingMode.HTTP_ACTIVATION, + "http_activation": {"rules": [{"hosts": ["*"]}]} + }, + "gateway_ok": {"mode": ScalingMode.METRIC, "http_activation": {}} + }; + raised = False; + try { + b._validate_http_activation_rules(scaling); + } except ValueError as e { + raised = True; + assert "billing_ops" in str(e); + assert "worker" in str(e); + } + assert raised; +} + +def _activated(hosts: list[str] | None = None) -> dict[str, any] { + rules = [{"hosts": hosts}] if hosts else []; + return {"mode": ScalingMode.HTTP_ACTIVATION, "http_activation": {"rules": rules}}; +} + +test "distinct rules, both-empty rules, and a lone HTTP_ACTIVATION peer never collide" { + b = _builder({}); + non_colliding_scalings = [ + # distinct rules per service: no shared match criteria to collide on + { + "billing_ops": _activated(["billing.example.com"]), + "worker": _activated(["worker.example.com"]) + }, + # both empty: already individually dead, not ambiguous with each other + {"billing_ops": _activated(), "worker": _activated()}, + # only one is HTTP_ACTIVATION; a METRIC peer's rules don't matter + { + "billing_ops": _activated(["*"]), + "worker": {"mode": ScalingMode.METRIC, "http_activation": {}} + } + ]; + for scaling in non_colliding_scalings { + b._validate_http_activation_rules(scaling); + } +} + +test "generate_manifests raises when two services inherit the shared catch-all rule unchanged" { + import from jaclang.scale.config.app_config { AppConfig } + import tempfile; + import from pathlib { Path } + services = {"billing_ops": {"http_activation": {"enabled": True}}, "worker": {}}; + with tempfile.TemporaryDirectory() as project { + for name in list(services.keys()) + ["main"] { + (Path(project) / f"{name}.jac").write_text(""); + } + b = ManifestBuilder( + k8s_config=KubernetesConfig( + app_name="t", + namespace="t-ns", + bundle_storage_class="standard", + http_activation={ + "enabled": True, + "target_port": 8080, + "concurrency_target": 10, + "rules": [{"hosts": ["*"]}] + } + ), + microservices_config={ + "routes": {name: f"/{name}" for name in services}, + "services": services + }, + dry_run=True + ); + raised = False; + try { + b.generate_manifests(AppConfig(code_folder=project, app_name="t"), ""); + } except ValueError as e { + raised = True; + assert "billing_ops" in str(e); + assert "worker" in str(e); + } + assert raised; + } +} + + +# --- Section F: _apply_or_replace's patch_body contract --------------------- +# HTTP activation's "a redeploy doesn't reset a scaled-to-zero target" guarantee +# depends entirely on _apply_or_replace sending patch_body (with replicas +# stripped) to `update` instead of the full manifest, while a fresh `create` +# still gets the full manifest. Pin that contract directly so a future change +# to _apply_or_replace can't silently break it. + +def _target -> KubernetesTarget { + return KubernetesTarget(config=KubernetesConfig(app_name="t", namespace="t-ns")); +} + + +test "_apply_or_replace sends patch_body, not the full manifest, to update when the target exists" { + mock_api = unittest.mock.MagicMock(); + manifest = { + "metadata": {"name": "svc-deployment"}, + "spec": {"replicas": 3, "template": {}} + }; + patch_body = {"metadata": {"name": "svc-deployment"}, "spec": {"template": {}}}; + _target()._apply_or_replace( + mock_api, + "read_namespaced_deployment", + "patch_namespaced_deployment", + "create_namespaced_deployment", + "t-ns", + manifest, + patch_body + ); + mock_api.patch_namespaced_deployment.assert_called_once_with( + name="svc-deployment", namespace="t-ns", body=patch_body + ); + mock_api.create_namespaced_deployment.assert_not_called(); +} + + +test "_apply_or_replace's create path gets the full manifest, replicas included, even when patch_body omits it" { + mock_api = unittest.mock.MagicMock(); + mock_api.read_namespaced_deployment.side_effect = ApiException(status=404); + manifest = { + "metadata": {"name": "svc-deployment"}, + "spec": {"replicas": 3, "template": {}} + }; + patch_body = {"metadata": {"name": "svc-deployment"}, "spec": {"template": {}}}; + _target()._apply_or_replace( + mock_api, + "read_namespaced_deployment", + "patch_namespaced_deployment", + "create_namespaced_deployment", + "t-ns", + manifest, + patch_body + ); + mock_api.create_namespaced_deployment.assert_called_once_with( + namespace="t-ns", body=manifest + ); + mock_api.patch_namespaced_deployment.assert_not_called(); +} + + +test "_apply_or_replace sends the full manifest to update when no patch_body is given" { + mock_api = unittest.mock.MagicMock(); + manifest = {"metadata": {"name": "svc-deployment"}, "spec": {"replicas": 3}}; + _target()._apply_or_replace( + mock_api, + "read_namespaced_deployment", + "patch_namespaced_deployment", + "create_namespaced_deployment", + "t-ns", + manifest + ); + mock_api.patch_namespaced_deployment.assert_called_once_with( + name="svc-deployment", namespace="t-ns", body=manifest + ); +} + + +# --- Section G: KEDA ScaledObject/TriggerAuthentication -- manifest shape, +# apply/redeploy sequencing, and destroy, exercised together end to end +# instead of one private helper at a time. Only the Kubernetes API client is +# mocked; KEDAAutoscaler.apply()/destroy() run for real. +def _as_any(v: any) -> any { + return v; +} + +def _keda_apply_mocks -> tuple[any, any]; + +impl _keda_apply_mocks -> tuple[any, any] { + KEDAAutoscaler._preflight_cache.clear(); + mock_api = unittest.mock.MagicMock(); + mock_api.list_cluster_custom_object.return_value = {}; + mock_api.get_namespaced_custom_object.side_effect = ApiException(status=404); + return (mock_api, unittest.mock.MagicMock()); +} + +test "apply() produces a complete multi-trigger ScaledObject, creates its TriggerAuthentication first with the right labels, and rejects a duplicate trigger identity before any write" { + (mock_api, mock_v2) = _keda_apply_mocks(); + keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=mock_v2); + spec = AutoscalerSpec( + scale_target_name="orders-deployment", + app_name="orders", + namespace="staging", + min_replicas=2, + max_replicas=10, + idle_replicas=0, + initial_cooldown_period=120, + behavior_overlay={ + "scaleDown": {"stabilizationWindowSeconds": 600, "selectPolicy": "Min"} + }, + triggers=[ + Trigger(type="cpu", metadata={"averageUtilization": "60"}), + Trigger(type="memory", metadata={}), + Trigger( + type="prometheus", + metadata={"serverAddress": "http://prom:9090", "threshold": 100} + ), + Trigger( + type="redis", + metadata={"address": "redis:6379"}, + name="cache", + auth=TriggerAuth( + secret_refs={"password": {"name": "redis-secret", "key": "pw"}} + ) + ) + ] + ); + result = keda.apply(spec); + assert result == True; + + create_calls = mock_api.create_namespaced_custom_object.call_args_list; + assert len(create_calls) == 2; + assert create_calls[0][1]["plural"] == "triggerauthentications"; + assert create_calls[1][1]["plural"] == "scaledobjects"; + + ta_body = create_calls[0][1]["body"]; + so_body = create_calls[1][1]["body"]; + assert ta_body["metadata"]["labels"]["app"] == "orders"; + assert so_body["metadata"]["name"] == "orders-scaledobject"; + assert ta_body["metadata"]["labels"]["jac-scale/owner"] == "orders-scaledobject"; + triggers = so_body["spec"]["triggers"]; + ref_name = triggers[3]["authenticationRef"]["name"]; + assert ta_body["metadata"]["name"] == ref_name; + + assert so_body["spec"]["scaleTargetRef"]["name"] == "orders-deployment"; + assert so_body["spec"]["minReplicaCount"] == 2; + assert so_body["spec"]["maxReplicaCount"] == 10; + assert so_body["spec"]["idleReplicaCount"] == 0; + assert so_body["spec"]["initialCooldownPeriod"] == 120; + assert [t["type"] for t in triggers] == ["cpu", "memory", "prometheus", "redis"]; + # cpu/memory resolve to KEDA's Utilization shape, memory defaulting to 80 + # when its metadata is empty; a non-resource type carries no metricType + # and coerces integer metadata values to strings for the KEDA API. + assert triggers[0]["metricType"] == "Utilization"; + assert triggers[0]["metadata"]["value"] == "60"; + assert triggers[1]["metricType"] == "Utilization"; + assert triggers[1]["metadata"]["value"] == "80"; + assert "metricType" not in triggers[2]; + assert triggers[2]["metadata"]["threshold"] == "100"; + assert triggers[3]["name"] == "cache"; + behavior = so_body["spec"]["advanced"]["horizontalPodAutoscalerConfig"]["behavior"]; + assert behavior["scaleDown"]["selectPolicy"] == "Min"; + + dup_spec = AutoscalerSpec( + scale_target_name="orders-deployment", + app_name="orders", + namespace="staging", + triggers=[ + Trigger(type="redis", metadata={}, name="cache"), + Trigger(type="rabbitmq", metadata={}, name="cache") + ] + ); + raised = False; + try { + keda.apply(dup_spec); + } except ValueError as e { + raised = True; + assert "cache" in str(e); + } + assert raised; + # The rejected spec must not have written anything beyond the first apply. + assert len(mock_api.create_namespaced_custom_object.call_args_list) == 2; +} + +test "redeploy patches the ScaledObject instead of duplicating it, prunes a dropped trigger's TriggerAuthentication, and keeps one still referenced" { + (mock_api, mock_v2) = _keda_apply_mocks(); + keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=mock_v2); + spec = AutoscalerSpec( + scale_target_name="svc-deployment", + app_name="svc", + namespace="default", + triggers=[ + Trigger( + type="redis", + metadata={"address": "redis:6379"}, + name="cache", + auth=TriggerAuth( + secret_refs={"password": {"name": "redis-secret", "key": "pw"}} + ) + ) + ] + ); + keda.apply(spec); + + # Redeploy dropping the auth trigger: the ScaledObject now exists (get + # succeeds) and the stale TriggerAuthentication is still on the cluster. + stale_ta_name = mock_api.create_namespaced_custom_object.call_args_list[0][1][ + "body" + ]["metadata"]["name"]; + mock_api.get_namespaced_custom_object.side_effect = None; + mock_api.get_namespaced_custom_object.return_value = { + "metadata": {"name": "svc-deployment-scaledobject"} + }; + mock_api.list_namespaced_custom_object.return_value = { + "items": [{"metadata": {"name": stale_ta_name}}] + }; + result = keda.apply( + AutoscalerSpec( + scale_target_name="svc-deployment", + app_name="svc", + namespace="default", + triggers=[Trigger(type="cpu", metadata={})] + ) + ); + assert result == True; + patch_calls = [ + c + for c in mock_api.patch_namespaced_custom_object.call_args_list + if c[1]["plural"] == "scaledobjects" + ]; + assert len(patch_calls) == 1; + prune_calls = [ + c + for c in mock_api.delete_namespaced_custom_object.call_args_list + if c[1]["plural"] == "triggerauthentications" + ]; + assert len(prune_calls) == 1; + assert prune_calls[0][1]["name"] == stale_ta_name; + + # A third deploy that keeps the same auth trigger must not prune it. + mock_api.delete_namespaced_custom_object.reset_mock(); + keda2 = KEDAAutoscaler(_custom_api=mock_api, _v2_api=mock_v2); + keep_spec = AutoscalerSpec( + scale_target_name="svc-deployment", + app_name="svc", + namespace="default", + triggers=[ + Trigger( + type="redis", + metadata={"address": "redis:6379"}, + name="cache", + auth=TriggerAuth( + secret_refs={"password": {"name": "redis-secret", "key": "pw"}} + ) + ) + ] + ); + desired_name = keda2._trigger_auth_name("svc", keep_spec.triggers[0], 0); + mock_api.list_namespaced_custom_object.return_value = { + "items": [{"metadata": {"name": desired_name}}] + }; + keda2.apply(keep_spec); + kept_prune_calls = [ + c + for c in mock_api.delete_namespaced_custom_object.call_args_list + if c[1]["plural"] == "triggerauthentications" + ]; + assert len(kept_prune_calls) == 0; +} + +test "switching the autoscaler engine to keda deletes a competing HPA (a no-op when none exists), and a missing CRD skips creation with a cached, logged diagnostic" { + (mock_api, mock_v2) = _keda_apply_mocks(); + mock_logger = unittest.mock.MagicMock(); + keda = KEDAAutoscaler( + _custom_api=mock_api, _v2_api=mock_v2, logger=_as_any(mock_logger) + ); + spec = AutoscalerSpec( + scale_target_name="svc-deployment", + app_name="svc", + namespace="default", + triggers=[Trigger(type="cpu", metadata={})] + ); + keda.apply(spec); + mock_v2.delete_namespaced_horizontal_pod_autoscaler.assert_called_once_with( + name="svc-hpa", namespace="default" + ); + + mock_v2.delete_namespaced_horizontal_pod_autoscaler.side_effect = ApiException( + status=404 + ); + keda2 = KEDAAutoscaler(_custom_api=mock_api, _v2_api=mock_v2); + assert keda2.apply(spec) == True , "engine-switch HPA deletion must be a no-op, not a failure, when no HPA exists"; + + KEDAAutoscaler._preflight_cache.clear(); + missing_crd_api = unittest.mock.MagicMock(); + missing_crd_api.list_cluster_custom_object.side_effect = ApiException(status=404); + keda3 = KEDAAutoscaler(_custom_api=missing_crd_api, logger=_as_any(mock_logger)); + assert keda3.apply(spec) == False; + missing_crd_api.create_namespaced_custom_object.assert_not_called(); + mock_logger.warn.assert_called_once(); + warn_msg = mock_logger.warn.call_args[0][0]; + assert "KEDA CRDs not found" in warn_msg; + assert "keda.sh/docs" in warn_msg; + keda3.preflight(); + assert missing_crd_api.list_cluster_custom_object.call_count == 1 , "the missing-CRD result must be cached, not re-queried"; +} + +test "destroy sweeps the ScaledObject and every TriggerAuthentication it owns, tolerating an already-gone resource or an absent TriggerAuthentication CRD; destroy_collection also sweeps competing HPAs and is silent when KEDA itself is not installed" { + mock_api = unittest.mock.MagicMock(); + mock_api.list_namespaced_custom_object.return_value = { + "items": [{"metadata": {"name": "my-svc-scaledobject-89abcdef-ta"}}] + }; + keda = KEDAAutoscaler(_custom_api=mock_api); + keda.destroy("my-svc", "staging"); + mock_api.list_namespaced_custom_object.assert_called_once_with( + group="keda.sh", + version="v1alpha1", + namespace="staging", + plural="triggerauthentications", + label_selector="jac-scale/owner=my-svc-scaledobject,managed=jac-scale" + ); + delete_calls = mock_api.delete_namespaced_custom_object.call_args_list; + assert len(delete_calls) == 2; + assert delete_calls[0][1]["plural"] == "triggerauthentications"; + assert delete_calls[0][1]["name"] == "my-svc-scaledobject-89abcdef-ta"; + assert delete_calls[1][1]["plural"] == "scaledobjects"; + assert delete_calls[1][1]["name"] == "my-svc-scaledobject"; + + # Already gone: a 404 on both deletes is tolerated, not raised. + mock_api2 = unittest.mock.MagicMock(); + mock_api2.list_namespaced_custom_object.return_value = {"items": []}; + mock_api2.delete_namespaced_custom_object.side_effect = ApiException(status=404); + KEDAAutoscaler(_custom_api=mock_api2).destroy("my-svc", "staging"); + + # TriggerAuthentication CRD absent entirely: the ScaledObject delete + # still runs. + mock_api3 = unittest.mock.MagicMock(); + mock_api3.list_namespaced_custom_object.side_effect = ApiException(status=404); + KEDAAutoscaler(_custom_api=mock_api3).destroy("my-svc", "staging"); + mock_api3.delete_namespaced_custom_object.assert_called_once_with( + group="keda.sh", + version="v1alpha1", + namespace="staging", + plural="scaledobjects", + name="my-svc-scaledobject" + ); + + # destroy_collection sweeps ScaledObjects, TriggerAuthentications, and + # InterceptorRoutes together, plus any competing HPA, and stays silent + # when the CRD group itself is missing (404/422). + mock_api4 = unittest.mock.MagicMock(); + mock_api4.list_namespaced_custom_object.return_value = { + "items": [{"metadata": {"name": "svc-scaledobject"}}] + }; + mock_v2 = unittest.mock.MagicMock(); + KEDAAutoscaler(_custom_api=mock_api4, _v2_api=mock_v2).destroy_collection( + "staging", "managed=jac-scale" + ); + plurals = [ + c[1]["plural"] for c in mock_api4.delete_namespaced_custom_object.call_args_list + ]; + assert set(plurals) == { + "scaledobjects", + "triggerauthentications", + "interceptorroutes" + }; + mock_v2.delete_collection_namespaced_horizontal_pod_autoscaler.assert_called_once_with( + namespace="staging", label_selector="managed=jac-scale" + ); + + for status in (404, 422) { + mock_api5 = unittest.mock.MagicMock(); + mock_api5.list_namespaced_custom_object.side_effect = ApiException( + status=status + ); + KEDAAutoscaler( + _custom_api=mock_api5, _v2_api=unittest.mock.MagicMock() + ).destroy_collection( + "staging", "managed=jac-scale" + ); + } +} + + +# --- Section H: KEDA HTTP Add-on activation -- InterceptorRoute/ScaledObject +# shape, apply ordering, target validation, capability discovery, and +# destroy, exercised together end to end. Only the Kubernetes API client is +# mocked; KEDAAutoscaler.apply_http_activation()/destroy_http_activation() +# run for real. +test "apply_http_activation encodes multi-rule routing (AND-within-rule, OR-between-rules, multi-path expansion), cold_start, timeouts, and request_rate, then creates InterceptorRoute before ScaledObject; a redeploy patches both instead of duplicating" { + (keda, custom_api, _, _) = _mocked_keda(); + custom_api.get_namespaced_custom_object.side_effect = ApiException(status=404); + spec = _spec( + namespace="staging", + rules=[ + HTTPRoutingRule( + hosts=["preview.jac.dev"], + paths=[HTTPPathMatch(value="/app"), HTTPPathMatch(value="/api")], + headers=[HTTPHeaderMatch(name="x-preview-id", value="123")] + ), + HTTPRoutingRule(hosts=["fallback.jac.dev"]) + ], + cold_start=HTTPColdStartSpec( + placeholder=HTTPStaticResponse(status_code=503, body="warming up"), + fallback_service=HTTPServiceTarget(service="fallback-svc", port=8080) + ), + timeouts=HTTPTimeoutSpec(readiness="30s", request="60s", response_header="5s"), + concurrency=None, + request_rate=HTTPRequestRateMetric( + target_value=50, rate_window="5m", granularity="10s" + ) + ); + result = keda.apply_http_activation(spec); + assert result == True; + + create_calls = custom_api.create_namespaced_custom_object.call_args_list; + assert len(create_calls) == 2; + assert create_calls[0][1]["plural"] == "interceptorroutes"; + assert create_calls[1][1]["plural"] == "scaledobjects"; + + route_spec = create_calls[0][1]["body"]["spec"]; + # The multi-path rule expands into two OR-grouped match blocks, each + # still carrying its shared hosts/headers (AND-within-rule); the + # single-host fallback rule stays its own separate block. + assert len(route_spec["rules"]) == 3; + app_rules = [ + r + for r in route_spec["rules"] + if r["hosts"] == ["preview.jac.dev"] + ]; + assert len(app_rules) == 2; + path_values = [r["path"]["value"] for r in app_rules]; + assert sorted(path_values) == ["/api", "/app"]; + for r in app_rules { + assert r["headers"][0]["name"] == "x-preview-id"; + } + fallback_rules = [ + r + for r in route_spec["rules"] + if r["hosts"] == ["fallback.jac.dev"] + ]; + assert len(fallback_rules) == 1; + assert "path" not in fallback_rules[0]; + + assert route_spec["coldStart"]["placeholder"]["response"]["statusCode"] == 503; + assert route_spec["coldStart"]["fallback"]["service"]["name"] == "fallback-svc"; + assert route_spec["timeouts"]["readiness"] == "30s"; + assert route_spec["timeouts"]["request"] == "60s"; + scaling_metric = route_spec["scalingMetric"]; + assert scaling_metric["requestRate"]["targetValue"] == 50; + assert scaling_metric["requestRate"]["window"] == "5m"; + assert "concurrency" not in scaling_metric; + + so_spec = create_calls[1][1]["body"]["spec"]; + trigger_metadata = so_spec["triggers"][0]["metadata"]; + assert trigger_metadata["targetValue"] == "50"; + assert trigger_metadata["window"] == "5m"; + assert trigger_metadata["granularity"] == "10s"; + assert so_spec["triggers"][0]["metadata"]["interceptorRoute"] == keda.interceptor_route_name_for( + spec.scale_target_name + ); + + # Redeploy: both resources now exist, so both get patched, not recreated. + custom_api.get_namespaced_custom_object.side_effect = None; + custom_api.get_namespaced_custom_object.return_value = { + "metadata": {"name": "existing"} + }; + keda.apply_http_activation(spec); + patch_calls = custom_api.patch_namespaced_custom_object.call_args_list; + assert len(patch_calls) == 2; + assert patch_calls[0][1]["plural"] == "interceptorroutes"; + assert patch_calls[1][1]["plural"] == "scaledobjects"; + assert len(custom_api.create_namespaced_custom_object.call_args_list) == 2 , "the redeploy must not create a second time"; +} + +test "apply_http_activation validates the spec itself (port XOR port_name, concurrency or request_rate) and its scale target and Service before writing anything: rejects a bare Pod, an unrecognized kind without scale_target_plural, and a missing target or Service; accepts a StatefulSet and a custom kind with scale_target_plural set" { + rejections = [ + ( + "bare Pod", + _spec(scale_target_name="preview-pod", scale_target_kind="Pod"), + "/scale" + ), + ( + "unrecognized kind, no plural", + _spec(scale_target_name="preview-rollout", scale_target_kind="Rollout"), + "scale_target_plural" + ), + ( + "both port and port_name set", + _spec( + target=HTTPServiceTarget( + service="preview-svc", port=8080, port_name="http" + ) + ), + "not both" + ), + ( + "neither port nor port_name set", + _spec(target=HTTPServiceTarget(service="preview-svc")), + "must set exactly one of port or port_name." + ), + ( + "neither concurrency nor request_rate set", + _spec(concurrency=None), + "concurrency or request_rate" + ) + ]; + for (label, spec, expected_substring) in rejections { + (keda, custom_api, _, _) = _mocked_keda(); + raised = False; + error_msg = ""; + try { + keda.apply_http_activation(spec); + } except ValueError as e { + raised = True; + error_msg = str(e); + } + assert raised , f"'{label}' must raise ValueError"; + assert expected_substring in error_msg , f"'{label}': expected '{expected_substring}' in: {error_msg}"; + custom_api.create_namespaced_custom_object.assert_not_called(); + } + + (keda2, custom_api2, _, _) = _mocked_keda(); + custom_api2.get_namespaced_custom_object.side_effect = ApiException(status=404); + assert keda2.apply_http_activation( + _spec(scale_target_name="preview-sts", scale_target_kind="StatefulSet") + ) == True; + + (keda3, custom_api3, _, _) = _mocked_keda(); + result = keda3.apply_http_activation( + _spec( + scale_target_name="preview-rollout", + scale_target_kind="Rollout", + scale_target_api_version="argoproj.io/v1alpha1", + scale_target_plural="rollouts" + ) + ); + assert result == True; + target_checks = [ + c + for c in custom_api3.get_namespaced_custom_object.call_args_list + if c[1].get("plural") == "rollouts" + ]; + assert len(target_checks) == 1; + assert target_checks[0][1]["group"] == "argoproj.io"; + + (keda4, custom_api4, _, _) = _mocked_keda(); + custom_api4.get_namespaced_custom_object.side_effect = ApiException(status=404); + raised = False; + try { + keda4.apply_http_activation( + _spec( + scale_target_name="missing-rollout", + scale_target_kind="Rollout", + scale_target_api_version="argoproj.io/v1alpha1", + scale_target_plural="rollouts" + ) + ); + } except ValueError { + raised = True; + } + assert raised , "a custom target with scale_target_plural set must still be checked for existence"; + + missing_target_apps = unittest.mock.MagicMock(); + missing_target_apps.read_namespaced_deployment.side_effect = ApiException( + status=404 + ); + (keda5, _, _, _) = _mocked_keda(apps_api=missing_target_apps); + raised = False; + try { + keda5.apply_http_activation(_spec(scale_target_name="missing-deployment")); + } except ValueError { + raised = True; + } + assert raised , "a missing Deployment scale target must raise"; + + missing_service_core = unittest.mock.MagicMock(); + missing_service_core.read_namespaced_service.side_effect = ApiException(status=404); + (keda6, _, _, _) = _mocked_keda(core_api=missing_service_core); + raised = False; + try { + keda6.apply_http_activation( + _spec(target=HTTPServiceTarget(service="missing-svc", port=8080)) + ); + } except ValueError { + raised = True; + } + assert raised , "a missing Service target must raise"; +} + +test "apply_http_activation returns False and creates nothing when the HTTP Add-on's CRDs are absent or core KEDA itself is missing, and raises when the Add-on is installed but genuinely broken" { + (keda, custom_api, _, _) = _mocked_keda(http_addon_installed=False); + assert keda.apply_http_activation(_spec()) == False; + custom_api.create_namespaced_custom_object.assert_not_called(); + + core_missing_api = unittest.mock.MagicMock(); + def core_missing_side_effect(*args: any, **kwargs: any) -> any { + if kwargs.get("group") == "keda.sh" { + raise ApiException(status=404); + } + return {}; + } + core_missing_api.list_cluster_custom_object.side_effect = core_missing_side_effect; + keda2 = KEDAAutoscaler( + _custom_api=core_missing_api, + _core_api=unittest.mock.MagicMock(), + _apps_api=unittest.mock.MagicMock() + ); + assert keda2.apply_http_activation(_spec()) == False , "core KEDA missing must skip activation even if the Add-on itself is healthy"; + core_missing_api.create_namespaced_custom_object.assert_not_called(); + + broken_scaler_core = unittest.mock.MagicMock(); + def broken_scaler_side_effect(name: str, *args: any, **kwargs: any) -> any { + if "scaler" in name { + raise ApiException(status=404); + } + return {}; + } + broken_scaler_core.read_namespaced_service.side_effect = broken_scaler_side_effect; + keda3 = KEDAAutoscaler( + _custom_api=unittest.mock.MagicMock(), + _core_api=broken_scaler_core, + _apps_api=unittest.mock.MagicMock() + ); + raised = False; + error_msg = ""; + try { + keda3.apply_http_activation(_spec()); + } except ValueError as e { + raised = True; + error_msg = str(e); + } + assert raised , "an installed but broken Add-on (missing scaler Service) must raise, not silently skip"; + assert "scaler" in error_msg.lower(); +} + +test "InterceptorRoute and ScaledObject names are deterministic and distinct; destroy_http_activation always sweeps both, tolerating a 403 on one while still attempting the other, and never masking a genuine 500 behind a later tolerable 403; destroy_collection sweeps InterceptorRoutes too but only tolerates a 403 there, not on keda.sh, unless best_effort" { + keda = KEDAAutoscaler(); + route_name = keda.interceptor_route_name_for("preview-deployment"); + scaled_obj_name = keda.http_scaled_object_name_for("preview-deployment"); + assert route_name != scaled_obj_name; + assert route_name == keda.interceptor_route_name_for("preview-deployment"); + + mock_api = unittest.mock.MagicMock(); + KEDAAutoscaler(_custom_api=mock_api).destroy_http_activation( + "preview-deployment", "staging" + ); + plurals = [ + c[1]["plural"] for c in mock_api.delete_namespaced_custom_object.call_args_list + ]; + assert set(plurals) == {"interceptorroutes", "scaledobjects"}; + + mock_api2 = unittest.mock.MagicMock(); + mock_api2.delete_namespaced_custom_object.side_effect = [ + ApiException(status=403), # InterceptorRoute delete forbidden + None # ScaledObject delete still attempted, and succeeds + ]; + status2 = None; + try { + KEDAAutoscaler(_custom_api=mock_api2).destroy_http_activation( + "preview-deployment", "staging" + ); + } except ApiException as e { + status2 = e.status; + } + assert status2 == 403 , "the 403 must still surface after both deletes are attempted"; + plurals2 = [ + c[1]["plural"] for c in mock_api2.delete_namespaced_custom_object.call_args_list + ]; + assert set(plurals2) == {"interceptorroutes", "scaledobjects"}; + + mock_api3 = unittest.mock.MagicMock(); + mock_api3.delete_namespaced_custom_object.side_effect = [ + ApiException(status=500), # a genuine server error on the first delete + ApiException(status=403) # a tolerable error on the second + ]; + status3 = None; + try { + KEDAAutoscaler(_custom_api=mock_api3).destroy_http_activation( + "preview-deployment", "staging" + ); + } except ApiException as e { + status3 = e.status; + } + assert status3 == 500 , "a 500 on the first delete must not be masked by a later tolerable 403"; + + # The same priority the other way around: a tolerable 403 on the first + # delete must not mask a genuine 500 on the second. + mock_api3b = unittest.mock.MagicMock(); + mock_api3b.delete_namespaced_custom_object.side_effect = [ + ApiException(status=403), + ApiException(status=500) + ]; + status3b = None; + try { + KEDAAutoscaler(_custom_api=mock_api3b).destroy_http_activation( + "preview-deployment", "staging" + ); + } except ApiException as e { + status3b = e.status; + } + assert status3b == 500 , "a 500 on the second delete must not be masked by a tolerable 403 on the first"; + + def destroy_collection_raised(mock: any, best_effort: bool = False) -> bool { + try { + KEDAAutoscaler( + _custom_api=mock, _v2_api=unittest.mock.MagicMock() + ).destroy_collection( + "staging", "managed=jac-scale", best_effort=best_effort + ); + } except ApiException { + return True; + } + return False; + } + + mock_api4 = unittest.mock.MagicMock(); + def list_side_effect(**kwargs: any) -> any { + if kwargs.get("group") == "http.keda.sh" { + raise ApiException(status=403); + } + return {"items": [{"metadata": {"name": "preview-deployment-scaledobject"}}]}; + } + mock_api4.list_namespaced_custom_object.side_effect = list_side_effect; + assert not destroy_collection_raised(mock_api4) , "a 403 listing InterceptorRoutes must not abort the sweep of the other resource kinds"; + swept_plurals = [ + c[1]["plural"] for c in mock_api4.delete_namespaced_custom_object.call_args_list + ]; + assert "scaledobjects" in swept_plurals; + assert "triggerauthentications" in swept_plurals; + + mock_api5 = unittest.mock.MagicMock(); + mock_api5.list_namespaced_custom_object.side_effect = ApiException(status=403); + assert destroy_collection_raised(mock_api5) , "a 403 on the core keda.sh group is not tolerated by default"; + assert not destroy_collection_raised(mock_api5, best_effort=True) , "best_effort=True must tolerate a 403 on every resource group, including keda.sh"; +} + +def _keda_for( + custom_api: any = None, core_api: any = None, apps_api: any = None +) -> KEDAAutoscaler { + return KEDAAutoscaler( + _custom_api=custom_api or unittest.mock.MagicMock(), + _core_api=core_api or unittest.mock.MagicMock(), + _apps_api=apps_api or unittest.mock.MagicMock() + ); +} + +def _custom_api_for( + core: (int | None) = None, + http_v1beta1: (int | None) = None, + legacy_v1alpha1: (int | None) = None +) -> any { + api = unittest.mock.MagicMock(); + def side_effect(*args: any, **kwargs: any) -> any { + status = None; + if kwargs.get("group") == "keda.sh" { + status = core; + } elif kwargs.get("version") == "v1beta1" { + status = http_v1beta1; + } elif kwargs.get("version") == "v1alpha1" { + status = legacy_v1alpha1; + } + if status is not None { + raise ApiException(status=status); + } + return {}; + } + api.list_cluster_custom_object.side_effect = side_effect; + return api; +} + +def _core_api_for( + scaler: (int | None) = None, interceptor: (int | None) = None +) -> any { + api = unittest.mock.MagicMock(); + def side_effect(name: str, *args: any, **kwargs: any) -> any { + status = interceptor if "interceptor" in name else scaler; + if status is not None { + raise ApiException(status=status); + } + return {}; + } + api.read_namespaced_service.side_effect = side_effect; + return api; +} + +test "discover_capabilities produces a distinct diagnostic for every install/permission combination" { + # core-vs-Add-on presence, per-service health, and RBAC-denied vs + # genuinely-absent must each report their own diagnosis -- an operator + # debugging a broken activation reads this message, not the code. + scenarios = [ + ( + "core-only", + _custom_api_for(http_v1beta1=404, legacy_v1alpha1=404), + _core_api_for() + ), + ("full-install", _custom_api_for(), _core_api_for()), + ("missing-external-scaler", _custom_api_for(), _core_api_for(scaler=404)), + ( + "missing-interceptor-service", + _custom_api_for(), + _core_api_for(interceptor=404) + ), + ( + "legacy-addon-needs-upgrade", + _custom_api_for(http_v1beta1=404), + _core_api_for() + ), + ("rbac-denied-on-core", _custom_api_for(core=403), _core_api_for()), + ( + "rbac-denied-on-http-addon", + _custom_api_for(http_v1beta1=403), + _core_api_for() + ), + ( + "rbac-denied-on-legacy-probe", + _custom_api_for(http_v1beta1=404, legacy_v1alpha1=403), + _core_api_for() + ) + ]; + for (label, custom_api, core_api) in scenarios { + keda = _keda_for(custom_api=custom_api, core_api=core_api); + caps = keda.discover_capabilities(); + msg = f"scenario '{label}' produced: warnings={caps.warnings} errors={caps.errors}"; + + if label == "core-only" { + assert caps.core_available == True , msg; + assert caps.http_addon_available == False , msg; + } elif label == "full-install" { + assert caps.core_available == True , msg; + assert caps.http_addon_available == True , msg; + assert caps.errors == [] , msg; + assert caps.interceptor_route_api_version == "v1beta1" , msg; + } elif label == "missing-external-scaler" { + assert any([("scaler" in e.lower()) for e in caps.errors]) , msg; + } elif label == "missing-interceptor-service" { + assert any([("interceptor" in e.lower()) for e in caps.errors]) , msg; + } elif label == "legacy-addon-needs-upgrade" { + assert any([("upgrade" in w.lower()) for w in caps.warnings]) , msg; + assert caps.http_addon_available == False , msg; + } elif label == "rbac-denied-on-core" { + assert caps.core_available == False , msg; + assert any( + [ + ("permission" in e.lower() or "rbac" in e.lower()) + for e in caps.errors + ] + ) , msg; + } elif label == "rbac-denied-on-http-addon" { + assert any( + [ + ("permission" in e.lower() or "rbac" in e.lower()) + for e in caps.errors + ] + ) , msg; + legacy_calls = [ + c + for c in custom_api.list_cluster_custom_object.call_args_list + if c[1].get("plural") == "httpscaledobjects" + ]; + assert legacy_calls == [] , ( + f"{label}: RBAC-denied v1beta1 must not fall through to the " + "legacy-version probe (that's only for a genuinely absent API)" + ); + } elif label == "rbac-denied-on-legacy-probe" { + assert any( + [ + ("permission" in e.lower() or "rbac" in e.lower()) + for e in caps.errors + ] + ) , ( + f"{label}: an RBAC-denied legacy probe must be reported as a " + "permission error, not silently treated as 'not installed'" + ); + } + } +} + +test "discover_capabilities caches per (cluster, service-address, client-identity) and only re-queries on refresh=True or invalidate_capabilities(); format_capabilities and preflight agree with the underlying state, and FQDN service addresses resolve to name and namespace" { + custom_api = _custom_api_for(); + keda = _keda_for(custom_api=custom_api, core_api=_core_api_for()); + keda.discover_capabilities(); + keda.discover_capabilities(); + assert custom_api.list_cluster_custom_object.call_count == 2 , "a second call must hit the cache (2 calls = one discovery round: core + http)"; + keda.discover_capabilities(refresh=True); + assert custom_api.list_cluster_custom_object.call_count == 4 , "refresh=True must bypass the cache"; + keda.invalidate_capabilities(); + keda.discover_capabilities(); + assert custom_api.list_cluster_custom_object.call_count == 6 , "invalidate_capabilities must force the next call to requery"; + + ready_keda = _keda_for(custom_api=_custom_api_for(), core_api=_core_api_for()); + ready_caps = ready_keda.discover_capabilities(); + assert "ready" in ready_keda.format_capabilities(ready_caps).lower(); + + not_ready_keda = _keda_for( + custom_api=_custom_api_for(http_v1beta1=404, legacy_v1alpha1=404), + core_api=_core_api_for() + ); + not_ready_caps = not_ready_keda.discover_capabilities(); + assert "helm" in not_ready_keda.format_capabilities(not_ready_caps).lower(); + + core_missing_keda = _keda_for( + custom_api=_custom_api_for(core=404), core_api=_core_api_for() + ); + core_missing_caps = core_missing_keda.discover_capabilities(); + assert core_missing_caps.http_addon_available == True; + assert "not ready" in core_missing_keda.format_capabilities( + core_missing_caps + ).lower() , ( + "a healthy Add-on with missing core KEDA must not be reported as ready" + ); + + # Two instances with the same cluster host but a different + # http_scaler_address must not share a cache entry. + keda_a = _keda_for(custom_api=_custom_api_for(), core_api=_core_api_for()); + caps_a = keda_a.discover_capabilities(); + assert caps_a.errors == []; + keda_b = KEDAAutoscaler( + _custom_api=_custom_api_for(), + _core_api=_core_api_for(scaler=404), + http_scaler_address="custom-scaler.other-ns:9090" + ); + caps_b = keda_b.discover_capabilities(); + assert any([("scaler" in e.lower()) for e in caps_b.errors]) , "a distinct http_scaler_address must not reuse instance A's cached result"; + + # Two instances with the same cluster key and service addresses but a + # different injected client must not share a cache entry either. + keda_c = _keda_for(custom_api=_custom_api_for(), core_api=_core_api_for()); + assert keda_c.discover_capabilities().errors == []; + keda_d = KEDAAutoscaler( + _custom_api=_custom_api_for(core=403), _core_api=_core_api_for() + ); + caps_d = keda_d.discover_capabilities(); + assert any( + [("permission" in e.lower() or "rbac" in e.lower()) for e in caps_d.errors] + ) , "a distinct (RBAC-restricted) client must not reuse instance C's cached result"; + + fqdn_core = _core_api_for(); + KEDAAutoscaler( + _custom_api=_custom_api_for(), + _core_api=fqdn_core, + _apps_api=unittest.mock.MagicMock(), + http_scaler_address="keda-add-ons-http-external-scaler.keda.svc.cluster.local:9090", + interceptor_service_address="keda-add-ons-http-interceptor-proxy.keda.svc.cluster.local:8080" + ).discover_capabilities(); + lookups = [ + (c[1]["name"], c[1]["namespace"]) + for c in fqdn_core.read_namespaced_service.call_args_list + ]; + assert lookups == [ + ("keda-add-ons-http-external-scaler", "keda"), + ("keda-add-ons-http-interceptor-proxy", "keda") + ] , f"FQDN service addresses must parse as name.namespace; got: {lookups}"; + + preflight_api = unittest.mock.MagicMock(); + preflight_api.list_cluster_custom_object.side_effect = ApiException(status=403); + raised = False; + try { + KEDAAutoscaler(_custom_api=preflight_api).preflight(); + } except ApiException { + raised = True; + } + assert not raised , "preflight must report a 403 as a warning, not an unhandled exception"; +} diff --git a/jac/jaclang/scale/tests/deploy/test_keda_autoscaler.jac b/jac/jaclang/scale/tests/deploy/test_keda_autoscaler.jac deleted file mode 100644 index 1bfa7d48199..00000000000 --- a/jac/jaclang/scale/tests/deploy/test_keda_autoscaler.jac +++ /dev/null @@ -1,540 +0,0 @@ -import unittest.mock; -import from kubernetes.client.exceptions { ApiException } -import from jaclang.scale.deploy.autoscale.keda_autoscaler { KEDAAutoscaler } -import from jaclang.scale.deploy.autoscale.autoscaler { - AutoscalerSpec, - Trigger, - TriggerAuth -} - - -test "cpu trigger is translated to KEDA Utilization format" { - keda = KEDAAutoscaler(); - result = keda._build_trigger( - Trigger(type="cpu", metadata={"averageUtilization": "70"}), "myapp", 0 - ); - assert result["type"] == "cpu"; - assert result["metricType"] == "Utilization"; - assert result["metadata"]["value"] == "70"; - - assert "metricType" not in result["metadata"]; -} - -test "memory trigger uses its own default 80 when metadata is empty" { - keda = KEDAAutoscaler(); - result = keda._build_trigger(Trigger(type="memory", metadata={}), "myapp", 0); - assert result["type"] == "memory"; - assert result["metricType"] == "Utilization"; - assert result["metadata"]["value"] == "80"; -} - -test "non-resource trigger metadata integer values are coerced to strings" { - keda = KEDAAutoscaler(); - result = keda._build_trigger( - Trigger( - type="prometheus", - metadata={"serverAddress": "http://prom:9090", "threshold": 100} - ), - "myapp", - 0 - ); - assert result["type"] == "prometheus"; - assert result["metadata"]["threshold"] == "100"; - assert result["metadata"]["serverAddress"] == "http://prom:9090"; - - assert "metricType" not in result; -} - -test "trigger name is propagated to KEDA trigger dict" { - keda = KEDAAutoscaler(); - result = keda._build_trigger( - Trigger(type="redis", metadata={}, name="redis-queue"), "myapp", 0 - ); - assert result["name"] == "redis-queue"; -} - -test "trigger with auth adds authenticationRef whose name matches the created TriggerAuthentication" { - keda = KEDAAutoscaler(); - result = keda._build_trigger( - Trigger( - type="redis", - metadata={}, - name="redis-queue", - auth=TriggerAuth( - secret_refs={"password": {"name": "redis-secret", "key": "pw"}} - ) - ), - app_name="myapp", - trigger_index=0 - ); - assert result["authenticationRef"] == {"name": "myapp-c3a558ec-ta"}; -} - -test "TriggerAuthentication names are scoped per service and per trigger position" { - # A shared name lets one deploy silently patch another's TriggerAuthentication with - # the wrong credentials. Names must differ across services (Gap 1) and across - # positions of an unnamed trigger within one spec (Gap 2). - keda = KEDAAutoscaler(); - trigger = Trigger( - type="redis", - metadata={}, - auth=TriggerAuth(secret_refs={"password": {"name": "secret", "key": "pw"}}) - ); - order = keda._build_trigger(trigger, "order-service", 0)["authenticationRef"][ - "name" - ]; - payment = keda._build_trigger(trigger, "payment-service", 0)["authenticationRef"][ - "name" - ]; - order_second = keda._build_trigger(trigger, "order-service", 1)["authenticationRef"][ - "name" - ]; - assert order.startswith("order-service-") and order.endswith("-ta"); - assert payment.startswith("payment-service-"); - assert order != payment; - assert order != order_second; -} - -test "duplicate trigger identities raise ValueError before any cluster write" { - # Two triggers resolving to the same identity would overwrite each other's - # TriggerAuthentication; apply() must reject the spec up front. - keda = KEDAAutoscaler(); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - app_name="order-service", - namespace="default", - triggers=[ - Trigger( - type="redis", - metadata={}, - name="cache", - auth=TriggerAuth( - secret_refs={"password": {"name": "redis-secret", "key": "pw"}} - ) - ), - Trigger( - type="rabbitmq", - metadata={}, - name="cache", - auth=TriggerAuth( - secret_refs={"password": {"name": "rmq-secret", "key": "pw"}} - ) - ) - ] - ); - raised = False; - try { - keda.apply(spec); - } except ValueError as e { - raised = True; - assert "cache" in str(e); - } - assert raised; -} - - -test "build manifests produces a complete and valid ScaledObject body" { - keda = KEDAAutoscaler(); - spec = AutoscalerSpec( - scale_target_name="orders-deployment", - namespace="staging", - min_replicas=2, - max_replicas=10, - triggers=[Trigger(type="cpu", metadata={"averageUtilization": "60"})] - ); - m = keda._build_manifests(spec); - assert m["name"] == "orders-deployment-scaledobject"; - assert m["body"]["apiVersion"] == "keda.sh/v1alpha1"; - assert m["body"]["kind"] == "ScaledObject"; - assert m["body"]["metadata"]["labels"]["managed"] == "jac-scale"; - assert m["body"]["spec"]["scaleTargetRef"]["name"] == "orders-deployment"; - assert m["body"]["spec"]["minReplicaCount"] == 2; - assert m["body"]["spec"]["maxReplicaCount"] == 10; - assert m["body"]["spec"]["triggers"][0]["metadata"]["value"] == "60"; -} - -test "build manifests adds idleReplicaCount zero for scale-to-zero" { - keda = KEDAAutoscaler(); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", namespace="default", idle_replicas=0 - ); - m = keda._build_manifests(spec); - assert m["body"]["spec"]["idleReplicaCount"] == 0; -} - -test "build manifests omits idleReplicaCount when not configured" { - keda = KEDAAutoscaler(); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", namespace="default", idle_replicas=None - ); - m = keda._build_manifests(spec); - assert "idleReplicaCount" not in m["body"]["spec"]; -} - -test "build manifests adds initialCooldownPeriod when set above zero" { - keda = KEDAAutoscaler(); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - namespace="default", - initial_cooldown_period=120 - ); - m = keda._build_manifests(spec); - assert m["body"]["spec"]["initialCooldownPeriod"] == 120; -} - -test "build manifests omits initialCooldownPeriod when zero" { - keda = KEDAAutoscaler(); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - namespace="default", - initial_cooldown_period=0 - ); - m = keda._build_manifests(spec); - assert "initialCooldownPeriod" not in m["body"]["spec"]; -} - -test "build manifests falls back to default cpu trigger when triggers list is empty" { - keda = KEDAAutoscaler(); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", namespace="default", triggers=[] - ); - m = keda._build_manifests(spec); - assert len(m["body"]["spec"]["triggers"]) == 1; - assert m["body"]["spec"]["triggers"][0]["type"] == "cpu"; -} - -test "build manifests includes every trigger in the ScaledObject spec" { - keda = KEDAAutoscaler(); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - namespace="default", - triggers=[ - Trigger(type="cpu", metadata={"averageUtilization": "50"}), - Trigger( - type="prometheus", - metadata={"threshold": "100", "query": "sum(rate(req[1m]))"} - ) - ] - ); - m = keda._build_manifests(spec); - assert len(m["body"]["spec"]["triggers"]) == 2; - assert m["body"]["spec"]["triggers"][0]["type"] == "cpu"; - assert m["body"]["spec"]["triggers"][1]["type"] == "prometheus"; -} - -test "build manifests with hpa bounds and non-cpu trigger produces ScaledObject with both triggers" { - keda = KEDAAutoscaler(); - spec = AutoscalerSpec( - scale_target_name="orders-deployment", - namespace="staging", - min_replicas=2, - max_replicas=10, - triggers=[ - Trigger(type="cpu", metadata={"averageUtilization": "60"}), - Trigger( - type="redis", - metadata={ - "address": "redis:6379", - "listName": "jobs", - "listLength": "10" - }, - name="redis-jobs" - ) - ] - ); - m = keda._build_manifests(spec); - assert m["body"]["spec"]["minReplicaCount"] == 2; - assert m["body"]["spec"]["maxReplicaCount"] == 10; - trigger_types = [t["type"] for t in m["body"]["spec"]["triggers"]]; - assert "cpu" in trigger_types; - assert "redis" in trigger_types; - assert len(trigger_types) == 2; -} - - -test "preflight returns empty list when KEDA CRDs are installed" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.return_value = {"items": []}; - keda = KEDAAutoscaler(_custom_api=mock_api); - assert keda.preflight() == []; -} - -test "preflight returns warning with kubectl install command when cluster returns 404" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.side_effect = ApiException(status=404); - keda = KEDAAutoscaler(_custom_api=mock_api); - warnings = keda.preflight(); - assert len(warnings) == 1; - assert "KEDA CRDs not found" in warnings[0]; - assert "keda.sh/docs" in warnings[0]; -} - -test "preflight treats 422 the same as 404" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.side_effect = ApiException(status=422); - keda = KEDAAutoscaler(_custom_api=mock_api); - warnings = keda.preflight(); - assert len(warnings) == 1; - assert "KEDA CRDs not found" in warnings[0]; -} - -test "preflight result is cached so the API is called only once across two calls" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.return_value = {"items": []}; - keda = KEDAAutoscaler(_custom_api=mock_api); - keda.preflight(); - keda.preflight(); - assert mock_api.list_cluster_custom_object.call_count == 1; -} - - -test "apply skips ScaledObject creation when KEDA CRDs are not installed" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.side_effect = ApiException(status=404); - keda = KEDAAutoscaler(_custom_api=mock_api); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - namespace="default", - triggers=[Trigger(type="cpu", metadata={})] - ); - result = keda.apply(spec); - assert result == False; - mock_api.create_namespaced_custom_object.assert_not_called(); - mock_api.patch_namespaced_custom_object.assert_not_called(); -} - -test "apply logs the install docs link when KEDA CRDs are not installed" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.side_effect = ApiException(status=404); - mock_logger = unittest.mock.MagicMock(); - keda = KEDAAutoscaler(_custom_api=mock_api, logger=mock_logger); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - namespace="default", - triggers=[Trigger(type="cpu", metadata={})] - ); - keda.apply(spec); - mock_logger.warn.assert_called_once(); - warn_msg = mock_logger.warn.call_args[0][0]; - assert "KEDA CRDs not found" in warn_msg; - assert "keda.sh/docs" in warn_msg; -} - -test "TriggerAuthentication name written to cluster matches authenticationRef name in ScaledObject" { - # _build_trigger (authenticationRef) and _apply_trigger_auth (the resource) must - # agree on the name end to end, or KEDA cannot resolve the secret at admission time. - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.return_value = {}; - mock_api.get_namespaced_custom_object.side_effect = [ - ApiException(status=404), # TriggerAuthentication does not exist yet - ApiException(status=404) # ScaledObject does not exist yet - ]; - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=unittest.mock.MagicMock()); - spec = AutoscalerSpec( - scale_target_name="order-service-deployment", - app_name="order-service", - namespace="default", - triggers=[ - Trigger( - type="redis", - metadata={"address": "redis:6379"}, - name="cache", - auth=TriggerAuth( - secret_refs={"password": {"name": "redis-secret", "key": "pw"}} - ) - ) - ] - ); - keda.apply(spec); - create_calls = mock_api.create_namespaced_custom_object.call_args_list; - assert len(create_calls) == 2; - ta_name = create_calls[0][1]["body"]["metadata"]["name"]; - ref_name = create_calls[1][1]["body"]["spec"]["triggers"][0]["authenticationRef"][ - "name" - ]; - assert ta_name == ref_name; -} - -test "apply creates TriggerAuthentication before ScaledObject" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.return_value = {}; - mock_api.get_namespaced_custom_object.side_effect = [ - ApiException(status=404), - ApiException(status=404) - ]; - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=unittest.mock.MagicMock()); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - namespace="default", - triggers=[ - Trigger( - type="redis", - metadata={"address": "redis:6379"}, - name="redis-queue", - auth=TriggerAuth( - secret_refs={"password": {"name": "redis-secret", "key": "pw"}} - ) - ) - ] - ); - result = keda.apply(spec); - assert result == True; - create_calls = mock_api.create_namespaced_custom_object.call_args_list; - assert len(create_calls) == 2; - assert create_calls[0][1]["plural"] == "triggerauthentications"; - assert create_calls[1][1]["plural"] == "scaledobjects"; -} - -test "apply creates ScaledObject when it does not yet exist on the cluster" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.return_value = {}; - mock_api.get_namespaced_custom_object.side_effect = ApiException(status=404); - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=unittest.mock.MagicMock()); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - namespace="default", - triggers=[Trigger(type="cpu", metadata={})] - ); - result = keda.apply(spec); - assert result == True; - create_calls = [ - c - for c in mock_api.create_namespaced_custom_object.call_args_list - if c[1].get("plural") == "scaledobjects" - ]; - assert len(create_calls) == 1; - mock_api.patch_namespaced_custom_object.assert_not_called(); -} - -test "apply patches ScaledObject when it already exists on the cluster" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.return_value = {}; - mock_api.get_namespaced_custom_object.return_value = { - "metadata": {"name": "svc-deployment-scaledobject"} - }; - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=unittest.mock.MagicMock()); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - namespace="default", - triggers=[Trigger(type="cpu", metadata={})] - ); - result = keda.apply(spec); - assert result == True; - patch_calls = [ - c - for c in mock_api.patch_namespaced_custom_object.call_args_list - if c[1].get("plural") == "scaledobjects" - ]; - assert len(patch_calls) == 1; -} - - -test "apply deletes competing HPA when switching from hpa engine to keda" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.return_value = {}; - mock_api.get_namespaced_custom_object.side_effect = ApiException(status=404); - mock_v2 = unittest.mock.MagicMock(); - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=mock_v2); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - app_name="svc", - namespace="default", - triggers=[Trigger(type="cpu", metadata={})] - ); - keda.apply(spec); - mock_v2.delete_namespaced_horizontal_pod_autoscaler.assert_called_once_with( - name="svc-hpa", namespace="default" - ); -} - -test "apply engine-switch HPA deletion is a no-op when no HPA exists" { - KEDAAutoscaler._preflight_cache.clear(); - mock_api = unittest.mock.MagicMock(); - mock_api.list_cluster_custom_object.return_value = {}; - mock_api.get_namespaced_custom_object.side_effect = ApiException(status=404); - mock_v2 = unittest.mock.MagicMock(); - mock_v2.delete_namespaced_horizontal_pod_autoscaler.side_effect = ApiException( - status=404 - ); - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=mock_v2); - spec = AutoscalerSpec( - scale_target_name="svc-deployment", - app_name="svc", - namespace="default", - triggers=[Trigger(type="cpu", metadata={})] - ); - result = keda.apply(spec); - assert result == True; -} - -test "destroy collection also sweeps competing HPAs" { - mock_api = unittest.mock.MagicMock(); - mock_api.list_namespaced_custom_object.return_value = {"items": []}; - mock_v2 = unittest.mock.MagicMock(); - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=mock_v2); - keda.destroy_collection("default", "managed=jac-scale"); - mock_v2.delete_collection_namespaced_horizontal_pod_autoscaler.assert_called_once_with( - namespace="default", label_selector="managed=jac-scale" - ); -} - -test "destroy calls delete with the correct plural and resource name" { - mock_api = unittest.mock.MagicMock(); - keda = KEDAAutoscaler(_custom_api=mock_api); - keda.destroy("my-svc-scaledobject", "staging"); - mock_api.delete_namespaced_custom_object.assert_called_once_with( - group="keda.sh", - version="v1alpha1", - namespace="staging", - plural="scaledobjects", - name="my-svc-scaledobject" - ); -} - -test "destroy is a no-op when the ScaledObject is already gone" { - mock_api = unittest.mock.MagicMock(); - mock_api.delete_namespaced_custom_object.side_effect = ApiException(status=404); - keda = KEDAAutoscaler(_custom_api=mock_api); - keda.destroy("my-svc-scaledobject", "staging"); -} - - -test "destroy collection removes all labeled ScaledObjects and TriggerAuthentications" { - mock_api = unittest.mock.MagicMock(); - mock_api.list_namespaced_custom_object.return_value = { - "items": [{"metadata": {"name": "svc-scaledobject"}}] - }; - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=unittest.mock.MagicMock()); - keda.destroy_collection("staging", "managed=jac-scale"); - delete_calls = mock_api.delete_namespaced_custom_object.call_args_list; - assert len(delete_calls) == 3; - plurals = [c[1]["plural"] for c in delete_calls]; - assert "scaledobjects" in plurals; - assert "triggerauthentications" in plurals; - assert "interceptorroutes" in plurals; -} - -test "destroy collection is silent when KEDA CRDs are not installed (404)" { - mock_api = unittest.mock.MagicMock(); - mock_api.list_namespaced_custom_object.side_effect = ApiException(status=404); - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=unittest.mock.MagicMock()); - keda.destroy_collection("staging", "managed=jac-scale"); -} - -test "destroy collection is silent when cluster returns 422 for absent CRD group" { - mock_api = unittest.mock.MagicMock(); - mock_api.list_namespaced_custom_object.side_effect = ApiException(status=422); - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=unittest.mock.MagicMock()); - keda.destroy_collection("staging", "managed=jac-scale"); -} diff --git a/jac/jaclang/scale/tests/deploy/test_keda_http_activation.jac b/jac/jaclang/scale/tests/deploy/test_keda_http_activation.jac deleted file mode 100644 index 2c5bf77e20e..00000000000 --- a/jac/jaclang/scale/tests/deploy/test_keda_http_activation.jac +++ /dev/null @@ -1,493 +0,0 @@ -import unittest.mock; -import from kubernetes.client.exceptions { ApiException } -import from jaclang.scale.deploy.autoscale.keda_autoscaler { KEDAAutoscaler } -import from jaclang.scale.deploy.autoscale.http_activation { - HTTPActivationSpec, - HTTPRoutingRule, - HTTPPathMatch, - HTTPHeaderMatch, - HTTPServiceTarget, - HTTPConcurrencyMetric, - HTTPRequestRateMetric, - HTTPStaticResponse, - HTTPColdStartSpec, - HTTPTimeoutSpec -} - -# Base HTTPActivationSpec shared by most tests below: a Deployment target on -# port 8080 with a concurrency=10 trigger. Override only what a given test -# cares about; pass concurrency=None when testing the metric-required check. -def _spec(**overrides: any) -> HTTPActivationSpec { - defaults = { - "name": "preview", - "namespace": "default", - "scale_target_name": "preview-deployment", - "target": HTTPServiceTarget(service="preview-svc", port=8080), - "concurrency": HTTPConcurrencyMetric(target_value=10) - }; - if "request_rate" in overrides { - defaults.pop("concurrency"); - } - defaults.update(overrides); - return HTTPActivationSpec(**defaults); -} - -# KEDAAutoscaler wired to MagicMock clients, with the HTTP Add-on preflight -# check (list_cluster_custom_object) pre-configured as installed by default. -# Pass a pre-configured apps_api/core_api/custom_api to control target, -# Service, or route/ScaledObject existence for a specific test. Returns the -# mocks alongside the instance since KEDAAutoscaler's api fields are typed -# `any | None`, which the checker won't let a caller re-narrow after the fact. -def _mocked_keda( - http_addon_installed: bool = True, - custom_api: any = None, - apps_api: any = None, - core_api: any = None -) -> tuple[KEDAAutoscaler, any, any, any] { - KEDAAutoscaler._http_preflight_cache.clear(); - custom_api = custom_api or unittest.mock.MagicMock(); - if http_addon_installed { - custom_api.list_cluster_custom_object.return_value = {}; - } else { - custom_api.list_cluster_custom_object.side_effect = ApiException(status=404); - } - apps_api = apps_api or unittest.mock.MagicMock(); - core_api = core_api or unittest.mock.MagicMock(); - keda = KEDAAutoscaler( - _custom_api=custom_api, _apps_api=apps_api, _core_api=core_api - ); - return (keda, custom_api, apps_api, core_api); -} - - -# --- Section A: typed model validation ------------------------------------- -test "validate rejects a Service target with both port and port_name set" { - keda = KEDAAutoscaler(); - spec = _spec( - target=HTTPServiceTarget(service="preview-svc", port=8080, port_name="http") - ); - raised = False; - try { - keda._validate_http_activation_spec(spec); - } except ValueError as e { - raised = True; - } - assert raised , "Expected ValueError when both port and port_name are set"; -} - -test "validate rejects a Service target with neither port nor port_name set" { - keda = KEDAAutoscaler(); - spec = _spec(target=HTTPServiceTarget(service="preview-svc")); - raised = False; - try { - keda._validate_http_activation_spec(spec); - } except ValueError as e { - raised = True; - } - assert raised , "Expected ValueError when neither port nor port_name is set"; -} - -test "validate requires at least one of concurrency or request_rate" { - keda = KEDAAutoscaler(); - spec = _spec(concurrency=None); - raised = False; - try { - keda._validate_http_activation_spec(spec); - } except ValueError as e { - raised = True; - } - assert raised , "Expected ValueError when neither concurrency nor request_rate is set"; -} - -test "request_rate defaults window to 1m and granularity to 1s when unset" { - metric = HTTPRequestRateMetric(target_value=100); - assert metric.window == "1m"; - assert metric.granularity == "1s"; -} - - -# --- Section B: routing-rule grouping (AND-within-rule / OR-between-rules) - -test "single rule with host, path, and header collapses into one AND-grouped match block" { - keda = KEDAAutoscaler(); - rules = [ - HTTPRoutingRule( - hosts=["preview.jac.dev"], - paths=[HTTPPathMatch(value="/app")], - headers=[HTTPHeaderMatch(name="x-preview-id", value="123")] - ) - ]; - result = keda._build_routing_rules(rules); - assert len(result) == 1; - assert result[0]["hosts"] == ["preview.jac.dev"]; - assert result[0]["path"]["value"] == "/app"; - assert result[0]["headers"][0]["name"] == "x-preview-id"; -} - -test "two rules in the list produce two separate OR-grouped match blocks" { - keda = KEDAAutoscaler(); - rules = [ - HTTPRoutingRule(hosts=["a.jac.dev"]), - HTTPRoutingRule(hosts=["b.jac.dev"]) - ]; - result = keda._build_routing_rules(rules); - assert len(result) == 2; - assert result[0]["hosts"] == ["a.jac.dev"]; - assert result[1]["hosts"] == ["b.jac.dev"]; -} - -test "rule with only hosts omits empty path and headers keys" { - keda = KEDAAutoscaler(); - rules = [HTTPRoutingRule(hosts=["a.jac.dev"])]; - result = keda._build_routing_rules(rules); - assert "hosts" in result[0]; - assert "path" not in result[0]; - assert "headers" not in result[0]; -} - -test "rule with multiple paths expands into one OR-grouped match block per path" { - keda = KEDAAutoscaler(); - rules = [ - HTTPRoutingRule( - hosts=["preview.jac.dev"], - paths=[HTTPPathMatch(value="/app"), HTTPPathMatch(value="/api")], - headers=[HTTPHeaderMatch(name="x-preview-id", value="123")] - ) - ]; - result = keda._build_routing_rules(rules); - assert len(result) == 2; - path_values = [entry["path"]["value"] for entry in result]; - assert "/app" in path_values; - assert "/api" in path_values; - for entry in result { - assert entry["hosts"] == ["preview.jac.dev"]; - assert entry["headers"][0]["name"] == "x-preview-id"; - } -} - - -# --- Section C: InterceptorRoute manifest builder --------------------------- -test "build_interceptor_route produces a complete and valid InterceptorRoute body" { - keda = KEDAAutoscaler(); - spec = _spec( - namespace="staging", rules=[HTTPRoutingRule(hosts=["preview.jac.dev"])] - ); - m = keda._build_interceptor_route(spec); - body = m["body"]; - assert body["apiVersion"] == "http.keda.sh/v1beta1"; - assert body["kind"] == "InterceptorRoute"; - assert body["metadata"]["namespace"] == "staging"; - assert body["metadata"]["labels"]["managed"] == "jac-scale"; - assert body["spec"]["target"]["service"] == "preview-svc"; - assert body["spec"]["target"]["port"] == 8080; - assert body["spec"]["scalingMetric"]["concurrency"]["targetValue"] == 10; - assert len(body["spec"]["rules"]) == 1; -} - -test "build_interceptor_route uses port_name instead of port when port is unset" { - keda = KEDAAutoscaler(); - spec = _spec(target=HTTPServiceTarget(service="preview-svc", port_name="http")); - m = keda._build_interceptor_route(spec); - assert m["body"]["spec"]["target"]["portName"] == "http"; - assert "port" not in m["body"]["spec"]["target"]; -} - -test "build_interceptor_route includes cold_start placeholder only when set" { - keda = KEDAAutoscaler(); - with_cold_start = keda._build_interceptor_route( - _spec( - cold_start=HTTPColdStartSpec( - placeholder=HTTPStaticResponse(status_code=503, body="warming up") - ) - ) - ); - without_cold_start = keda._build_interceptor_route(_spec()); - assert with_cold_start["body"]["spec"]["coldStart"]["placeholder"]["response"][ - "statusCode" - ] == 503; - assert "coldStart" not in without_cold_start["body"]["spec"]; -} - -test "build_interceptor_route includes cold_start fallback_service only when set" { - keda = KEDAAutoscaler(); - spec = _spec( - cold_start=HTTPColdStartSpec( - fallback_service=HTTPServiceTarget(service="fallback-svc", port=8080) - ) - ); - m = keda._build_interceptor_route(spec); - assert m["body"]["spec"]["coldStart"]["fallback"]["service"]["name"] == "fallback-svc"; - assert "placeholder" not in m["body"]["spec"]["coldStart"]; -} - -test "build_interceptor_route encodes request_rate as scalingMetric.requestRate" { - keda = KEDAAutoscaler(); - spec = _spec( - request_rate=HTTPRequestRateMetric( - target_value=50, window="5m", granularity="10s" - ) - ); - m = keda._build_interceptor_route(spec); - scaling_metric = m["body"]["spec"]["scalingMetric"]; - assert scaling_metric["requestRate"]["targetValue"] == 50; - assert scaling_metric["requestRate"]["window"] == "5m"; - assert scaling_metric["requestRate"]["granularity"] == "10s"; - assert "concurrency" not in scaling_metric; -} - -test "build_interceptor_route includes timeouts only when set, omitted otherwise" { - keda = KEDAAutoscaler(); - with_timeouts = keda._build_interceptor_route( - _spec( - timeouts=HTTPTimeoutSpec( - readiness="30s", request="60s", response_header="5s" - ) - ) - ); - without_timeouts = keda._build_interceptor_route(_spec()); - assert with_timeouts["body"]["spec"]["timeouts"]["readiness"] == "30s"; - assert with_timeouts["body"]["spec"]["timeouts"]["request"] == "60s"; - assert "timeouts" not in without_timeouts["body"]["spec"]; -} - - -# --- Section D: ScaledObject external-push builder -------------------------- -test "build_http_scaled_object produces a complete external-push ScaledObject" { - keda = KEDAAutoscaler(); - spec = _spec(namespace="staging", min_replicas=0, max_replicas=5); - m = keda._build_http_scaled_object(spec); - body = m["body"]; - trigger = body["spec"]["triggers"][0]; - assert body["apiVersion"] == "keda.sh/v1alpha1"; - assert body["kind"] == "ScaledObject"; - assert body["spec"]["minReplicaCount"] == 0; - assert body["spec"]["maxReplicaCount"] == 5; - assert trigger["type"] == "external-push"; - assert trigger["metadata"]["interceptorRoute"] == keda.interceptor_route_name_for( - "preview-deployment" - ); - assert "scalerAddress" in trigger["metadata"]; -} - -test "build_http_scaled_object encodes concurrency metric as targetValue" { - keda = KEDAAutoscaler(); - spec = _spec(concurrency=HTTPConcurrencyMetric(target_value=25)); - m = keda._build_http_scaled_object(spec); - metadata = m["body"]["spec"]["triggers"][0]["metadata"]; - assert metadata["targetValue"] == "25"; - assert "window" not in metadata; -} - -test "build_http_scaled_object encodes request_rate metric with window and granularity" { - keda = KEDAAutoscaler(); - spec = _spec( - request_rate=HTTPRequestRateMetric( - target_value=50, window="5m", granularity="10s" - ) - ); - m = keda._build_http_scaled_object(spec); - metadata = m["body"]["spec"]["triggers"][0]["metadata"]; - assert metadata["targetValue"] == "50"; - assert metadata["window"] == "5m"; - assert metadata["granularity"] == "10s"; -} - - -# --- Section E: standalone-Pod rejection ------------------------------------ -test "apply_http_activation rejects a standalone Pod target with a /scale explanation" { - keda = KEDAAutoscaler(); - spec = _spec(scale_target_name="preview-pod", scale_target_kind="Pod"); - raised = False; - error_msg = ""; - try { - keda.apply_http_activation(spec); - } except ValueError as e { - raised = True; - error_msg = str(e); - } - assert raised , "Expected ValueError when scale_target_kind is Pod"; - assert "/scale" in error_msg , "Error message must explain the /scale requirement"; -} - -test "apply_http_activation accepts a StatefulSet target without raising" { - (keda, custom_api, _, _) = _mocked_keda(); - custom_api.get_namespaced_custom_object.side_effect = ApiException(status=404); - spec = _spec( - scale_target_name="preview-statefulset", scale_target_kind="StatefulSet" - ); - result = keda.apply_http_activation(spec); - assert result == True; -} - -test "apply_http_activation raises for an unrecognized scale_target_kind when scale_target_plural is unset" { - (keda, custom_api, apps_api, _) = _mocked_keda(); - spec = _spec(scale_target_name="preview-rollout", scale_target_kind="Rollout"); - raised = False; - error_msg = ""; - try { - keda.apply_http_activation(spec); - } except ValueError as e { - raised = True; - error_msg = str(e); - } - assert raised , "Expected ValueError when scale_target_kind is unrecognized and scale_target_plural is unset"; - assert "Rollout" in error_msg; - assert "scale_target_plural" in error_msg; - apps_api.read_namespaced_deployment.assert_not_called(); - apps_api.read_namespaced_stateful_set.assert_not_called(); - custom_api.create_namespaced_custom_object.assert_not_called(); -} - -test "apply_http_activation validates a custom target when scale_target_plural is set" { - (keda, custom_api, apps_api, _) = _mocked_keda(); - spec = _spec( - scale_target_name="preview-rollout", - scale_target_kind="Rollout", - scale_target_api_version="argoproj.io/v1alpha1", - scale_target_plural="rollouts" - ); - result = keda.apply_http_activation(spec); - assert result == True; - apps_api.read_namespaced_deployment.assert_not_called(); - apps_api.read_namespaced_stateful_set.assert_not_called(); - target_check_calls = [ - c - for c in custom_api.get_namespaced_custom_object.call_args_list - if c[1].get("plural") == "rollouts" - ]; - assert len(target_check_calls) == 1; - assert target_check_calls[0][1]["group"] == "argoproj.io"; - assert target_check_calls[0][1]["version"] == "v1alpha1"; - assert target_check_calls[0][1]["name"] == "preview-rollout"; -} - -test "apply_http_activation raises when a custom target with scale_target_plural is missing" { - (keda, custom_api, _, _) = _mocked_keda(); - custom_api.get_namespaced_custom_object.side_effect = ApiException(status=404); - spec = _spec( - scale_target_name="missing-rollout", - scale_target_kind="Rollout", - scale_target_api_version="argoproj.io/v1alpha1", - scale_target_plural="rollouts" - ); - raised = False; - try { - keda.apply_http_activation(spec); - } except ValueError as e { - raised = True; - } - assert raised , "Expected ValueError when a custom target with scale_target_plural is missing"; - custom_api.create_namespaced_custom_object.assert_not_called(); -} - - -# --- Section F: naming ------------------------------------------------------- -test "InterceptorRoute and ScaledObject names are deterministic and distinct" { - keda = KEDAAutoscaler(); - route_name = keda.interceptor_route_name_for("preview-deployment"); - scaled_obj_name = keda.http_scaled_object_name_for("preview-deployment"); - assert route_name != scaled_obj_name; - assert route_name == keda.interceptor_route_name_for("preview-deployment"); - assert scaled_obj_name == keda.http_scaled_object_name_for("preview-deployment"); -} - - -# --- Section G: apply ordering, validation, and preflight ------------------- -test "apply_http_activation creates InterceptorRoute before ScaledObject when neither exists" { - (keda, custom_api, _, _) = _mocked_keda(); - custom_api.get_namespaced_custom_object.side_effect = ApiException(status=404); - result = keda.apply_http_activation(_spec()); - assert result == True; - create_calls = custom_api.create_namespaced_custom_object.call_args_list; - assert len(create_calls) == 2; - assert create_calls[0][1]["plural"] == "interceptorroutes"; - assert create_calls[1][1]["plural"] == "scaledobjects"; -} - -test "apply_http_activation patches InterceptorRoute and ScaledObject when both already exist" { - (keda, custom_api, _, _) = _mocked_keda(); - custom_api.get_namespaced_custom_object.return_value = { - "metadata": {"name": "existing"} - }; - result = keda.apply_http_activation(_spec()); - assert result == True; - patch_calls = custom_api.patch_namespaced_custom_object.call_args_list; - assert len(patch_calls) == 2; - assert patch_calls[0][1]["plural"] == "interceptorroutes"; - assert patch_calls[1][1]["plural"] == "scaledobjects"; - custom_api.create_namespaced_custom_object.assert_not_called(); -} - -test "apply_http_activation raises when the referenced scale target or Service does not exist" { - custom_api = unittest.mock.MagicMock(); - - missing_target_apps = unittest.mock.MagicMock(); - missing_target_apps.read_namespaced_deployment.side_effect = ApiException( - status=404 - ); - (keda, _, _, _) = _mocked_keda(custom_api=custom_api, apps_api=missing_target_apps); - raised_for_missing_target = False; - try { - keda.apply_http_activation(_spec(scale_target_name="missing-deployment")); - } except ValueError as e { - raised_for_missing_target = True; - } - assert raised_for_missing_target , "Expected ValueError when the scale target is missing"; - - missing_service_core = unittest.mock.MagicMock(); - missing_service_core.read_namespaced_service.side_effect = ApiException(status=404); - (keda2, _, _, _) = _mocked_keda( - custom_api=custom_api, core_api=missing_service_core - ); - raised_for_missing_service = False; - try { - keda2.apply_http_activation( - _spec(target=HTTPServiceTarget(service="missing-svc", port=8080)) - ); - } except ValueError as e { - raised_for_missing_service = True; - } - assert raised_for_missing_service , "Expected ValueError when the Service is missing"; - custom_api.create_namespaced_custom_object.assert_not_called(); -} - -test "apply_http_activation returns False and skips creation when HTTP Add-on CRDs are not installed" { - (keda, custom_api, _, _) = _mocked_keda(http_addon_installed=False); - result = keda.apply_http_activation(_spec()); - assert result == False; - custom_api.create_namespaced_custom_object.assert_not_called(); -} - - -# --- Section H: destroy / partial-apply cleanup ------------------------------ -test "destroy_http_activation removes both InterceptorRoute and ScaledObject" { - mock_api = unittest.mock.MagicMock(); - keda = KEDAAutoscaler(_custom_api=mock_api); - keda.destroy_http_activation("preview-deployment", "staging"); - delete_calls = mock_api.delete_namespaced_custom_object.call_args_list; - plurals = [c[1]["plural"] for c in delete_calls]; - assert "interceptorroutes" in plurals; - assert "scaledobjects" in plurals; -} - -test "destroy_http_activation is a no-op for the InterceptorRoute when it is already gone" { - mock_api = unittest.mock.MagicMock(); - mock_api.delete_namespaced_custom_object.side_effect = [ - ApiException(status=404), - None - ]; - keda = KEDAAutoscaler(_custom_api=mock_api); - keda.destroy_http_activation("preview-deployment", "staging"); - assert mock_api.delete_namespaced_custom_object.call_count == 2; -} - -test "destroy_collection sweeps InterceptorRoutes alongside ScaledObjects and TriggerAuthentications" { - mock_api = unittest.mock.MagicMock(); - mock_api.list_namespaced_custom_object.return_value = { - "items": [{"metadata": {"name": "preview-deployment-http-route"}}] - }; - keda = KEDAAutoscaler(_custom_api=mock_api, _v2_api=unittest.mock.MagicMock()); - keda.destroy_collection("staging", "managed=jac-scale"); - delete_calls = mock_api.delete_namespaced_custom_object.call_args_list; - plurals = [c[1]["plural"] for c in delete_calls]; - assert "interceptorroutes" in plurals; -} diff --git a/jac/jaclang/scale/tests/deploy/test_memory_trigger_guard.jac b/jac/jaclang/scale/tests/deploy/test_memory_trigger_guard.jac new file mode 100644 index 00000000000..e54e8bc6832 --- /dev/null +++ b/jac/jaclang/scale/tests/deploy/test_memory_trigger_guard.jac @@ -0,0 +1,102 @@ +import from jaclang.scale.deploy.target.kubernetes.manifest_builder { + ManifestBuilder, + GATEWAY_NAME +} +import from jaclang.scale.deploy.target.kubernetes.kubernetes_config { + KubernetesConfig +} + + +def _builder( + services: dict, k8s_config: KubernetesConfig | None = None +) -> ManifestBuilder { + return ManifestBuilder( + k8s_config=k8s_config or KubernetesConfig(app_name="t", namespace="t-ns"), + microservices_config={"services": services}, + dry_run=True + ); +} + +test "_get_autoscaler_config surfaces the per-service hpa.behavior fragment" { + b = _builder( + { + "billing_ops": { + "hpa": { + "behavior": { + "scaleDown": { + "stabilizationWindowSeconds": 600, + "policies": [ + {"type": "Pods", "value": 2, "periodSeconds": 300} + ] + } + } + } + } + } + ); + cfg = b._get_autoscaler_config("billing_ops"); + assert cfg["behavior"]["scaleDown"]["stabilizationWindowSeconds"] == 600; +} + +test "_get_autoscaler_config defaults behavior to an empty dict when unset" { + b = _builder({"billing_ops": {}}); + cfg = b._get_autoscaler_config("billing_ops"); + assert cfg["behavior"] == {}; +} + +test "hpa.behavior on one service does not leak onto another" { + b = _builder( + { + "billing_ops": { + "hpa": {"behavior": {"scaleDown": {"selectPolicy": "Min"}}} + }, + "orders_ops": {} + } + ); + assert b._get_autoscaler_config("billing_ops")["behavior"] == { + "scaleDown": {"selectPolicy": "Min"} + }; + assert b._get_autoscaler_config("orders_ops")["behavior"] == {}; +} + +test "hpa.enabled false still short-circuits before behavior is considered" { + b = _builder( + { + "billing_ops": { + "hpa": { + "enabled": False, + "behavior": {"scaleDown": {"selectPolicy": "Min"}} + } + } + } + ); + assert b._get_autoscaler_config("billing_ops") is None; +} + + +test "hpa.behavior works for the gateway via the __gateway__ service key" { + b = _builder( + {GATEWAY_NAME: {"hpa": {"behavior": {"scaleDown": {"selectPolicy": "Min"}}}}} + ); + assert b._get_autoscaler_config(GATEWAY_NAME)["behavior"] == { + "scaleDown": {"selectPolicy": "Min"} + }; +} + +test "malformed hpa.behavior warns and falls back to the empty fragment" { + import unittest.mock; + + logger = unittest.mock.MagicMock(); + b = ManifestBuilder( + k8s_config=KubernetesConfig(app_name="t", namespace="t-ns"), + microservices_config={ + "services": {"billing_ops": {"hpa": {"behavior": "gentle"}}} + }, + dry_run=True, + logger=logger + ); + cfg = b._get_autoscaler_config("billing_ops"); + assert cfg["behavior"] == {}; + logger.warning.assert_called_once(); + assert "billing_ops" in str(logger.warning.call_args[0][0]); +} diff --git a/jac/jaclang/scale/tests/deploy/test_show_yaml_autoscalers.jac b/jac/jaclang/scale/tests/deploy/test_show_yaml_autoscalers.jac new file mode 100644 index 00000000000..debd914d480 --- /dev/null +++ b/jac/jaclang/scale/tests/deploy/test_show_yaml_autoscalers.jac @@ -0,0 +1,132 @@ +"""--show-yaml must print the autoscaler objects a deploy would apply. The +bundle carries autoscaler CONFIGS, not manifests; render_autoscaler_manifests +turns them into the same objects the apply path builds, from the same spec.""" + +import from jaclang.scale.deploy.target.kubernetes.kubernetes_config { + KubernetesConfig +} +import from jaclang.scale.deploy.target.kubernetes.target { KubernetesTarget } +import from jaclang.scale.deploy.target.kubernetes.manifest_builder { ScalingMode } + + +test "hpa engine renders the HPA manifest the apply path would build" { + target = KubernetesTarget(config=KubernetesConfig(app_name="t", namespace="t-ns")); + bundle = { + "scaling": { + "api": { + "mode": ScalingMode.METRIC, + "scale_target_name": "api-deployment", + "autoscaler": {"min": 2, "max": 7, "cpu_target": 60} + } + } + }; + manifests = target.render_autoscaler_manifests(bundle); + assert len(manifests) == 1; + m = manifests[0]; + assert m["kind"] == "HorizontalPodAutoscaler"; + assert m["metadata"]["namespace"] == "t-ns"; + assert m["spec"]["scaleTargetRef"]["name"] == "api-deployment"; + assert m["spec"]["minReplicas"] == 2; + assert m["spec"]["maxReplicas"] == 7; +} + + +test "keda engine renders the ScaledObject body, not the name/body wrapper" { + target = KubernetesTarget( + config=KubernetesConfig( + app_name="t", namespace="t-ns", autoscaler_engine="keda" + ) + ); + bundle = { + "scaling": { + "api": { + "mode": ScalingMode.METRIC, + "scale_target_name": "api-deployment", + "autoscaler": {"min": 1, "max": 4} + } + } + }; + manifests = target.render_autoscaler_manifests(bundle); + assert len(manifests) == 1; + m = manifests[0]; + assert m["kind"] == "ScaledObject"; + assert m["spec"]["scaleTargetRef"]["name"] == "api-deployment"; + assert m["spec"]["minReplicaCount"] == 1; + assert m["spec"]["maxReplicaCount"] == 4; +} + + +test "the scale target comes from the resolved plan, not re-derived from the service name" { + target = KubernetesTarget(config=KubernetesConfig(app_name="t", namespace="t-ns")); + bundle = { + "scaling": { + "api": { + "mode": ScalingMode.METRIC, + "scale_target_name": "not-the-k8s-safe-name-deployment", + "autoscaler": {"min": 1, "max": 3} + } + } + }; + manifests = target.render_autoscaler_manifests(bundle); + assert manifests[0]["spec"]["scaleTargetRef"]["name"] == "not-the-k8s-safe-name-deployment"; +} + + +test "a service scaled by http_activation renders no autoscaler manifest" { + target = KubernetesTarget(config=KubernetesConfig(app_name="t", namespace="t-ns")); + bundle = { + "scaling": {"api": {"mode": ScalingMode.HTTP_ACTIVATION, "autoscaler": {}}} + }; + assert target.render_autoscaler_manifests(bundle) == []; +} + + +test "a bundle with no scaling renders nothing" { + target = KubernetesTarget(config=KubernetesConfig(app_name="t", namespace="t-ns")); + assert target.render_autoscaler_manifests({"deployments": {}}) == []; +} + + +test "keda engine renders the TriggerAuthentication apply() would create for an authenticated trigger" { + # --show-yaml only rendered the ScaledObject, so a config with an + # auth-bearing trigger dry-ran clean and then apply() created a + # TriggerAuthentication the preview never showed. + target = KubernetesTarget( + config=KubernetesConfig( + app_name="t", namespace="t-ns", autoscaler_engine="keda" + ) + ); + bundle = { + "scaling": { + "api": { + "mode": ScalingMode.METRIC, + "scale_target_name": "api-deployment", + "autoscaler": { + "min": 1, + "max": 4, + "triggers": [ + { + "type": "redis", + "name": "cache", + "metadata": {"address": "redis:6379"}, + "auth": { + "secret_refs": { + "password": {"name": "redis-secret", "key": "pw"} + } + } + } + ] + } + } + } + }; + manifests = target.render_autoscaler_manifests(bundle); + assert len(manifests) == 2; + assert manifests[0]["kind"] == "ScaledObject"; + auth_name = manifests[0]["spec"]["triggers"][0]["authenticationRef"]["name"]; + assert manifests[1]["kind"] == "TriggerAuthentication"; + assert manifests[1]["metadata"]["name"] == auth_name; + assert manifests[1]["spec"]["secretTargetRef"] == [ + {"parameter": "password", "name": "redis-secret", "key": "pw"} + ]; +} diff --git a/jac/jaclang/scale/tests/fixtures/keda_http_activation_e2e/README.md b/jac/jaclang/scale/tests/fixtures/keda_http_activation_e2e/README.md index 2471a7a8e04..55dca97a117 100644 --- a/jac/jaclang/scale/tests/fixtures/keda_http_activation_e2e/README.md +++ b/jac/jaclang/scale/tests/fixtures/keda_http_activation_e2e/README.md @@ -1,39 +1,41 @@ # KEDA HTTP Add-on activation e2e fixture -Zero-replica `echo` Deployment (`hashicorp/http-echo`) plus its Service, -applied as a raw manifest so the e2e script can drive a real KEDA HTTP -Add-on scale-from-zero cycle against it. +A minimal Jac app (`app.jac`) deployed via `jac scale deploy`, with +`[scale.kubernetes.http_activation]` enabled in `jac.toml` so the deploy +wires up the KEDA HTTP Add-on's `InterceptorRoute` + `ScaledObject` for real +scale-to-zero activation (#7475). ``` keda_http_activation_e2e/ - fixture.yaml Namespace + Deployment (replicas: 0) + Service + app.jac A single public walker (`echo`) reporting a JSON message. + jac.toml [scale.kubernetes.http_activation] config; deploys as app "echo". ``` -There is no `jac.toml` here: this fixture is not a Jac app, just the -Kubernetes objects the KEDA HTTP Add-on scales, so nothing builds or -runs a client for it. +This fixture is a real Jac app deployed through the normal CLI path, not a +raw manifest -- `jac scale deploy` builds the Deployment/Service and, from +the `http_activation` config, the `InterceptorRoute`/`ScaledObject` too. See +`../http_activation_toml_e2e/` for the sibling fixture this one follows the +same pattern as. ## What the e2e covers [`../deploy/keda_http_activation_real_e2e.sh`](../deploy/keda_http_activation_real_e2e.sh) -drives [`../deploy/keda_http_activation_verify.jac`](../deploy/keda_http_activation_verify.jac), -which calls `KEDAAutoscaler.apply_http_activation` / `destroy_http_activation` -directly against whatever cluster the current kubeconfig points at. No -mocking. The flow: - -1. Apply this fixture; confirm `echo` starts at 0 replicas. -2. Call `apply_http_activation` twice: once to create the - `InterceptorRoute` + `ScaledObject`, once more to exercise the - get-then-patch branch against a real API server. -3. Poll both resources' `status.conditions[type=Ready]` until each - reports Ready, so a reconciliation problem fails here with a clear - message instead of surfacing later as an opaque interceptor timeout. -4. Port-forward the HTTP Add-on interceptor and send a request through - it; this should block on the cold start, then return 200 once the - target is Ready. +drives the deploy against whatever cluster the current kubeconfig points at. +No mocking. The flow: + +1. Deploy via `jac scale deploy app.jac` from this directory. +2. Redeploy once more, to confirm the `InterceptorRoute`/`ScaledObject` + reconcile is idempotent (get-then-patch, not create-or-duplicate) against + a real API server. +3. Poll both resources' `status.conditions[type=Ready]` until each reports + Ready, so a reconciliation problem fails here with a clear message + instead of surfacing later as an opaque interceptor timeout. +4. Port-forward the HTTP Add-on interceptor and `POST /walker/echo` through + it; this should block on the cold start, then return 200 once the target + is Ready. 5. Wait for `echo` to scale 0 to 1 and become Available. -6. Stop traffic and wait for the cooldown period to elapse, then - confirm `echo` scales back down to 0. +6. Stop traffic and wait for the cooldown period to elapse, then confirm + `echo` scales back down to 0. ## Prerequisites @@ -77,6 +79,44 @@ components in a Running state. The script also preflight-checks for the `interceptorroutes.http.keda.sh` CRD and fails immediately with the commands above if it is missing. +`jac.toml`'s `[dev] jaclang_source` points at this checkout's own `jac/` +source, so the deploy runs your in-tree code, not a published release. + +On `kind`, the cluster has no RWX-capable default `StorageClass` for the +bundle PVC (its built-in `standard` class is `ReadWriteOnce`), so you need +to provision one first: + +```bash +kubectl apply -f - <