test(0.34.x): KEDA suite (#8940) plus the seal-time venv fix (#8944), stacked for wake-latency measurement - #8946
Draft
kugesan1105 wants to merge 13 commits into
Conversation
…ght diagnostics (jaseci-labs#7620) ## Summary Closes jaseci-labs#7404. `KEDAAutoscaler.preflight()` only ever verified that the core `keda.sh/v1alpha1 ScaledObject` API existed on a cluster. That check is not sufficient for HTTP activated workloads, which also depend on the KEDA HTTP Add-on's `InterceptorRoute` API, two Kubernetes Services (the external scaler and the interceptor proxy), and a compatible Add-on version. Without these checks, `apply_http_activation` could create a valid `ScaledObject` that never actually activates from HTTP traffic, with no clear diagnostic explaining why. This PR introduces structured capability discovery for the KEDA HTTP Add-on and wires it into `apply_http_activation`, so a broken or partially installed cluster fails with an actionable error instead of silently deploying a nonfunctional workload. ## Changes | File | Changes | |---|---| | `jac/jaclang/scale/deploy/autoscale/keda_capabilities.jac` (new file) | Adds the `KEDACapabilities` data object, matching the shape specified in the issue: `core_available`, `http_addon_available`, `interceptor_route_api_version`, `external_scaler_address`, `interceptor_service_address`, `warnings`, and `errors`. Also adds `SUPPORTED_HTTP_INTERCEPTOR_API_VERSIONS`. | | `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac` | Imports `KEDACapabilities`.<br>Adds a new `interceptor_service_address` field, defaulted the same way `http_scaler_address` already is.<br>Replaces the old bool only `_http_preflight_cache` with `_capabilities_cache`, which holds a full `KEDACapabilities` object per cluster.<br>Declares `discover_capabilities`, `invalidate_capabilities`, `_check_service_address`, `format_capabilities`, and `_capabilities_cache_key`. | | `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac` | `discover_capabilities` checks the core `ScaledObject` API and the HTTP Add-on's `InterceptorRoute` API independently, so a missing core install is distinguished from a missing Add-on. A 403 on either check is reported as a distinct permission error rather than being folded into "not installed."<br>If the `v1beta1 InterceptorRoute` API is absent (not RBAC denied), it additionally probes the legacy `v1alpha1 HTTPScaledObject` API. If that is present, the result is a distinct "upgrade" warning with the exact `helm upgrade` command, rather than being reported as "Add-on not installed." A 403 on this legacy probe is reported as its own permission error rather than being folded into "not installed."<br>`_check_service_address` verifies the external scaler and interceptor proxy Services actually resolve, and reports which one is missing.<br>Results are cached per cluster identity via a new `_capabilities_cache_key` helper, which combines `_get_cluster_key()` with this instance's configured `http_scaler_address` and `interceptor_service_address`. This keeps two instances that target the same cluster but are configured with different service addresses from reusing an incompatible cached result. `discover_capabilities(refresh=True)` and `invalidate_capabilities()` remain the explicit refresh paths.<br>`format_capabilities` produces a single concise, human readable line for CLI output.<br>`preflight()` now also catches a 403 and returns a distinct permission error instead of letting the `ApiException` propagate unhandled, which is what happened previously.<br>`apply_http_activation` now calls `discover_capabilities()` instead of its own inline, duplicate check. If discovery reports any errors (missing service, RBAC denied, or similar), it raises `ValueError` with the details. If the core KEDA API or the HTTP Add-on is simply absent (no errors, just not installed), it keeps the existing behavior of logging a warning and returning `False` rather than raising. A partial install, where the Add-on and its services are healthy but core KEDA itself is missing, now also falls into this gate instead of proceeding to create resources that would fail against a nonexistent API. | | `jac/jaclang/scale/tests/deploy/test_keda_http_activation.jac` | Adds a new capability discovery test section covering: core only clusters, full installs, missing external scaler service, missing interceptor service, legacy Add-on detection, RBAC denied on the core check, the HTTP Add-on check, and the legacy probe, cache and refresh behavior, cache isolation between instances with different service addresses, explicit invalidation, the raise on broken capabilities and skip on missing core behavior in `apply_http_activation`, `format_capabilities` output, and the `preflight()` 403 fix. Test setup uses small composable mock builders (`_custom_api_for`, `_core_api_for`) rather than one mock per scenario, and one table driven test covers the distinct diagnostic cases to avoid duplicating near identical assertions. | ## Out of scope As noted in the issue: no auto install or upgrade of KEDA, no ingress creation, no general health checking of every scaler type. This PR also does not verify that a workload's traffic is actually routed through the interceptor at runtime, only that the interceptor's own API and Service objects exist and resolve. That distinction is called out explicitly in the implementation plan for this issue. ## Test plan - [x] Core only clusters report core available and HTTP unavailable. - [x] Fully installed clusters report HTTP ready with no errors. - [x] Missing external scaler service, missing interceptor service, legacy Add-on version, and RBAC denied cases (core check, HTTP Add-on check, and legacy probe) each produce a distinct diagnostic. - [x] Results are cached per cluster identity, including the configured service addresses, and refresh via `discover_capabilities(refresh=True)` and `invalidate_capabilities()`. - [x] `apply_http_activation` raises when capabilities are broken, returns `False` gracefully when the Add-on or core KEDA is simply absent, and does not proceed to create resources on a partial install. - [x] Documentation update showing the supported Helm install and upgrade flow. - [x] Full deploy test suite (169 tests) passes with no regressions. --------- Co-authored-by: Kugesan Sivasothynathan <kugesan.sivasothynathan@jaseci.org> (cherry picked from commit 5234c81)
… policy (jaseci-labs#7899) Closes jaseci-labs#7764. `jac-scale`'s generated HPA `behavior` block (added in jaseci-labs#7445) has always shipped one hardcoded scale-down policy — `{"type": "Percent", "value": 50, "periodSeconds": 60}`, no `selectPolicy` — with no way to override it per service. Because the deploy path re-applies the full generated manifest on every `jac start --scale`, any out-of-band `kubectl patch` on the HPA gets wiped on the next deploy, forcing a re-patch-after-every-deploy loop in CI. This PR adds a `hpa.behavior` fragment to the per-service config table, deep-merged over the generated default. It's the first slice of jaseci-labs#7764 — connection-heavy services (WebSockets, SSE) can now ask for a gentle scale-down (e.g. at most 2 pods every 5 minutes) without imposing that rate on every other service through the app-global `autoscaler_cooldown`, and the override survives every subsequent deploy. ```toml [scale.microservices.services.chat_sv.hpa.behavior.scaleDown] stabilizationWindowSeconds = 600 selectPolicy = "Min" policies = [{ type = "Pods", value = 2, periodSeconds = 300 }] ``` - **`AutoscalerSpec`** (`autoscaler.jac`) gains a `behavior_overlay: dict[str, any] = {}` field. - **`Autoscaler._build_behavior`** deep-merges `behavior_overlay` over the hardcoded default via the existing `merge_manifest_overlay` (same semantics already used for `deployment_overlay`: maps merge recursively; the `policies` list carries no `name` key, so it's replaced wholesale rather than merged entry-by-entry, matching how you'd actually want to specify a policy set). Both the HPA and KEDA engines pick this up for free — they already route through the same `_build_behavior` call, so this is a single-point change. - **`ManifestBuilder._get_autoscaler_config`** (`manifest_builder.jac`) resolves the fragment from the per-service `hpa.behavior` table. - **`KubernetesTarget`**'s deploy loop (`target.jac`) threads it into `AutoscalerSpec(...)` as `behavior_overlay=cfg.get("behavior", {})`. - **Config schema** (`plugin_config.jac`) and the **K8s reference table** (`runtime/docs.md`) document the new key. `_build_behavior` resolves `merge_manifest_overlay` via a function-scoped import rather than a top-level one. `manifest_builder.jac` already imports constants from `autoscaler.jac` at module level, so a top-level import in the other direction would be a circular import; a deferred import inside the function (evaluated well after both modules are fully loaded) is the same pattern already used elsewhere in this codebase (e.g. `manifest_builder.jac`'s own `_pod_entrypoint`) for exactly this situation. - `service_overlay` / first-class `Service` `annotations` + gateway `sessionAffinity` knobs - PDB `min_available` and percentage (`"50%"`) support Written test-first: 11 new tests across the files that already exercise the touched code paths (`test_deployment_overlay.jac`, `test_factories.jac`, `test_keda_autoscaler.jac`, `test_memory_trigger_guard.jac`), confirmed red against the pre-implementation tree, then green after: - `_build_behavior` merge semantics in isolation: no-overlay default, empty-overlay no-op, `scaleDown` override, `scaleUp` override, wholesale (not per-item) `policies` replacement. - Both engines apply the overlay: HPA's `_build_manifests` (`spec.behavior`) and KEDA's `_build_manifests` (`advanced.horizontalPodAutoscalerConfig.behavior`). - Config plumbing: `_get_autoscaler_config` surfaces `hpa.behavior`, defaults to `{}` when unset, doesn't leak across services, and `hpa.enabled = false` still short-circuits before behavior is considered. Also verified against a real cluster, not just unit tests. Deployed the `jac-shop` 3-service microservice fixture (`jaclang/scale/tests/fixtures/k8s_e2e`) to a local `kind` cluster with a `hpa.behavior.scaleDown` override configured on `products_app`, via `jac start main.jac --scale`: - **First deploy**: all 4 services (gateway + 3) reached Ready, and the live `products-app-hpa` object's `spec.behavior.scaleDown` matched the configured override exactly (`stabilizationWindowSeconds: 600`, `selectPolicy: Min`, `policies: [{type: Pods, value: 2, periodSeconds: 300}]`), with `scaleUp` untouched at the generated default. - **Redeploy**: re-ran `jac start main.jac --scale` against the same cluster and diffed `products-app-hpa`'s `spec.behavior.scaleDown` before and after — byte-identical. Confirms the actual pain point from the issue: the override survives repeated applies instead of being reset to the old hardcoded `{Percent, 50, 60s}` shape. - [x] A per-service `hpa.behavior` fragment in `jac.toml` survives repeated `jac start --scale` applies with no external patching. - [x] Global `autoscaler_*` keys remain the defaults; per-service fragment wins on merge. - [x] Works for both `autoscaler_engine = "hpa"` and `"keda"`. - [x] Config schema (`plugin_config.jac`) and docs updated; overlay-merge unit tests added alongside the existing `test_deployment_overlay.jac` patterns. --------- Co-authored-by: Kugesan Sivasothynathan <153247429+kugesan1105@users.noreply.github.com> Co-authored-by: Kugesan Sivasothynathan <kugesan.sivasothynathan@jaseci.org> (cherry picked from commit 91c7adc)
…ewed YAML matches what a deploy applies (jaseci-labs#8008) `--show-yaml` printed pvcs, deployments, services, pdbs and ingress but dropped autoscalers, while the totals line counted them and the service view rendered HPA details from them. The reviewed YAML was not the applied artifact, for exactly the resources most likely to cause a production surprise. The bundle's `autoscalers` key holds CONFIGS, not manifests; the real HPA/ScaledObject is synthesized only inside the apply loop. Adding the key to the flatten list would therefore have dumped config dicts. Instead: - `deploy/target/kubernetes/target.jac`: the AutoscalerSpec construction is extracted out of the apply loop into `_autoscaler_spec` (single source), and a new `render_autoscaler_manifests(bundle)` builds the same objects the apply path would, via each engine's `_build_manifests`. Pure - no Kubernetes API calls. - `runtime/cli/plan.jac`: `Plan.from_target` stashes the rendered manifests into the bundle; `_flatten_manifests` emits them into --show-yaml. - New test_show_yaml_autoscalers (3): real KubernetesTarget with the HPA and KEDA engines; asserts kind, scaleTargetRef, min/max mapping, and that KEDA's ScaledObject body (not its name/body wrapper) is rendered. - test_plan extended (+2): autoscaler manifests reach the YAML dump, plus a sentinel test asserting every manifest-bearing bundle key reaches --show-yaml (the issue's drift-proofing suggestion). - jac-ec2 rig, linked-source dev binary on this branch: all passing - test_show_yaml_autoscalers (3, new), test_plan (19, includes the 2 new), test_factories (28), test_keda_autoscaler (34), test_memory_trigger_guard (8), test_dry_run_purity (12). - NOT run: a live cluster apply of HPA/KEDA objects (the render path is pure; the apply path is unchanged except for the spec extraction). Full repo suite not run locally. - KEDA TriggerAuthentication objects are applied but still not rendered into --show-yaml (built inline in `_apply_trigger_auth`; needs the same build/apply split). - Normalize the `_build_manifests` contract: KEDA returns a {name, body} wrapper, HPA returns the bare manifest. Closes jaseci-labs#7915. Part of jaseci-labs#7900. --------- Co-authored-by: Thamirawaran Sathiyalogeswaran <107134124+Thamirawaran@users.noreply.github.com> (cherry picked from commit 9d2cace)
…aml (jaseci-labs#8424) Closes jaseci-labs#8409. `jac start --scale --show-yaml` (via `render_autoscaler_manifests`) prints the autoscaler objects a deploy would apply, but for KEDA it only ever printed the `ScaledObject`. A trigger with `auth.secret_refs` also causes `apply()` to create a `TriggerAuthentication`, and that half of the footprint was invisible to the dry-run preview. This PR unifies the `_build_manifests` contract across both autoscaler engines so the render path renders the same objects `apply()` creates, with nothing built as a side effect of a live API call. `_build_manifests` for KEDA only ever constructed the `ScaledObject` body. The `TriggerAuthentication` for an authenticated trigger was built and written to the cluster exclusively inside `KEDAAutoscaler.apply` via `_apply_trigger_auth`, a function that conflated *building* the manifest with *reconciling* it against the cluster (get → patch-or-create) in one step. Because the manifest never existed as a standalone, returnable value, `render_autoscaler_manifests` (the `--show-yaml` path) had no way to see it. A secondary wart compounded this: `_build_manifests` returned a different shape per engine (HPA a bare manifest dict, KEDA a `{"name", "body"}` wrapper), so the render path added defensive shape-sniffing (`built["body"] if "body" in built else built`) instead of a real contract. That sniff is what let the KEDA gap go unnoticed - it happily returned "a manifest" without anyone checking whether it was the *only* manifest KEDA needed to emit. Net effect: a KEDA autoscaler config with an auth-bearing trigger dry-ran clean (N ScaledObjects, 0 TriggerAuthentications) and then `apply()` silently created N ScaledObjects **and** M TriggerAuthentications the preview never showed - exactly the class of surprise `--show-yaml` exists to prevent, and the same shape of gap jaseci-labs#8008 already fixed once for autoscalers generally. `_build_manifests` now returns one `list[dict]` contract for both engines: - `HPAAutoscaler._build_manifests` returns `[hpa]`. - `KEDAAutoscaler._build_manifests` returns `[scaled_object, *trigger_authentications]`. To make that possible without duplicating logic between the render and apply paths: - The `TriggerAuthentication` body construction is extracted out of the old apply-only `_apply_trigger_auth` into a pure builder, `_build_trigger_auth_manifest`, which returns the manifest (or `None` when a trigger has no auth) and makes no API calls. - A shared `_apply_custom_object` reconciler (get → patch-or-create) replaces the two nearly-identical get/patch/create blocks that previously existed separately for `TriggerAuthentication` and `ScaledObject` writes in `KEDAAutoscaler.apply`. - `render_autoscaler_manifests` (`target.jac`) now does `out.extend(autoscaler._build_manifests(...))` instead of shape-sniffing a single object out of the return value, so it renders every manifest an engine produces, not just the first one. Apply-time ordering is unchanged: `TriggerAuthentication` objects are still written before the `ScaledObject` that references them via `authenticationRef`, since KEDA's admission webhook needs the referenced auth resource to exist first. | File | Change | |---|---| | `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac` | Interface updates: `_build_manifests` returns `list[dict[str, any]]`; new `_build_trigger_auth_manifest` and `_apply_custom_object` declarations replace `_apply_trigger_auth`. | | `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac` | `_build_manifests` assembles `[scaled_object, *trigger_auths]`; trigger-auth body construction moved into pure `_build_trigger_auth_manifest`; new `_apply_custom_object` reconciler; `apply()` applies trigger-auth manifests before the ScaledObject via the shared reconciler. | | `jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.jac` | `_build_manifests` interface updated to `list[dict[str, any]]`. | | `jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.impl.jac` | `_build_manifests` wraps its single manifest in a list; `apply()` unwraps `[0]`. | | `jac/jaclang/scale/deploy/target/kubernetes/target.jac` | `render_autoscaler_manifests` flattens the manifest list from each engine instead of shape-sniffing a single object. | | `jac/jaclang/scale/tests/deploy/test_factories.jac` | HPA `_build_manifests` assertions updated to the list contract. | | `jac/jaclang/scale/tests/deploy/test_keda_autoscaler.jac` | KEDA `_build_manifests` assertions updated to the list contract (ScaledObject at index 0, no more `name`/`body` wrapper). | | `jac/jaclang/scale/tests/deploy/test_show_yaml_autoscalers.jac` | New regression test: an authenticated KEDA trigger renders both the `ScaledObject` and its matching `TriggerAuthentication` through `render_autoscaler_manifests`. | Ran the full `deploy` test group locally with `jac test` (one process per file, matching how CI's `test-scale (deploy)` matrix leg runs it): all 27 files, 293 tests pass, both before-vs-after on the three files this PR touches and as a full-group run afterward to catch cross-file fallout. - `test_show_yaml_autoscalers` (4, 1 new) - the regression test for this issue. Builds a bundle with one KEDA-engine service carrying an authenticated `redis` trigger (`auth.secret_refs`) and asserts `render_autoscaler_manifests` returns exactly two manifests: the `ScaledObject` at index 0 and a `TriggerAuthentication` at index 1 whose `metadata.name` matches the `authenticationRef.name` KEDA embedded in the ScaledObject's trigger, with the `secretTargetRef` mapped correctly from config. Confirmed this test fails against the pre-fix code (dropping the `_build_trigger_auth_manifest` loop reproduces the original bug - `len(manifests) == 1`) and passes with the fix. The other two existing tests in this file (HPA render, empty-bundle render) are unchanged and still pass, guarding against a regression in the ordinary case. - `test_keda_autoscaler` (34) - full suite for the engine this PR changes most: `_build_manifests` shape and field mapping (9 tests, now indexing `[0]` into the list), trigger-authentication naming/collision rules, `preflight`, and - the part most likely to break silently on a build/apply split - `apply()`'s behavior end to end against a mocked `CustomObjectsApi`: that a `TriggerAuthentication` is still created before its `ScaledObject`, that the name written to the cluster still matches the `authenticationRef` the ScaledObject carries, and that a KEDA-CRDs-not-installed cluster still short-circuits before either write. All pass unmodified in assertion logic (only the `_build_manifests` indexing changed), which is the signal that `apply()`'s observable behavior toward the cluster is unchanged by the refactor. - `test_factories` (32) - HPA `_build_manifests` output (exact-dict equality against the expected manifest, and the `behavior_overlay` merge) re-verified against the new `[hpa]` list contract. Also validated end to end against a real cluster, not just mocks: - **Cluster**: local `kind` (v0.32.0) with KEDA core installed via the standard Helm chart (`kedacore/keda`, namespace `keda`); `scaledobjects.keda.sh` and `triggerauthentications.keda.sh` CRDs confirmed present. A static `hostPath` PV under a `no-provisioner` StorageClass (`jac-rwx`) stands in for the bundle PVC's `ReadWriteMany` requirement, the same technique `jac/jaclang/scale/scripts/e2e_lib.sh`'s `provision_kind_rwx_storage` uses for the kind-based CI e2e leg (kind's `local-path` StorageClass is RWO-only; a static hostPath PV is safe on a single-node cluster). - **Project**: a minimal scratch app (`main.jac`, one walker, no client) with `[dev] jaclang_source` pointing at this checkout, so both the CLI and the deployed pods run the fixed source, not a published release binary. `[scale.kubernetes] autoscaler_engine = "keda"` plus one `[[scale.microservices.services.<name>.triggers]]` entry with `type = "redis"` and `auth.secret_refs` - the exact shape jaseci-labs#8409 described. - **Dry-run** (`jac start --scale --dry-run --show-yaml`, no cluster contact): the rendered YAML stream includes both the `ScaledObject` and a `TriggerAuthentication` object, with the `ScaledObject`'s `authenticationRef.name` matching the `TriggerAuthentication`'s `metadata.name` exactly (`kedaval-7dcd81a6-ta`) - this is the bug fixed, observed through the real CLI rather than only through the test suite. - **Live apply** (`jac start --scale`): the same `TriggerAuthentication` (`kedaval-7dcd81a6-ta`, `SECRET: redis-secret`) is present on the cluster via `kubectl get triggerauthentications`, and the `ScaledObject`'s `AUTHENTICATIONS` column names it - dry-run and `apply()` now agree, which is the whole point of the fix. `kedaval-scaledobject` reports `READY=False`/`ACTIVE=Unknown` only because no real Redis exists at the configured address; that's expected and irrelevant to what's being validated here (the auth object's existence and wiring, not live scaling). - **App health**: both `gateway-deployment` and `kedaval-deployment` reached `1/1 Running`, the fleet rollout reported `2/2 deployments serving`, and `/health` on the gateway reported both services `"status": "healthy"` - confirming the fix doesn't regress the ordinary deploy path. Configure a KEDA autoscaler with an auth-bearing trigger and run `jac start --scale --show-yaml`: both the `ScaledObject` and its `TriggerAuthentication` now render, matching what `apply()` creates. --------- Co-authored-by: Kugesan Sivasothynathan <kugesan.sivasothynathan@jaseci.org> Co-authored-by: Kugesan Sivasothynathan <153247429+kugesan1105@users.noreply.github.com> (cherry picked from commit b908596)
…bel (part 2 - final - of issue jaseci-labs#6535) (jaseci-labs#6908) ## Problem `KEDAAutoscaler` orphaned `TriggerAuthentication` resources in two distinct ways: 1. **Redeploy with a changed trigger set.** TA names are derived from the trigger identity (`app + trigger-key`, hashed). When a redeploy renames, removes, or reorders triggers, `apply()` creates new TAs under new names and re-points the ScaledObject at them, but never deletes the TAs from the previous config. `apply()` had no prune step, so every such redeploy left a stale TA behind, unreferenced by the ScaledObject. 2. **destroy().** `destroy()` deleted only the ScaledObject, leaving every TA its `apply()` created behind. The earlier approach tried to fix (2) by reading `authenticationRef.name` off the live ScaledObject. That could not fix (1) at all (the stale TA is no longer referenced by the ScaledObject, so it is never found), and it left a hole in (2): if the ScaledObject was already gone, the GET returned 404 and `destroy()` returned early, stranding any TAs permanently. ## Root cause There was no ownership-based reconciliation. TAs are children whose lifecycle should be bound to the autoscaler, but the code bound them only by name derivation plus an imperative "read the ScaledObject, delete what it currently points at". Nothing ever reconciled the actual set of managed TAs in the namespace against the desired set. Compounding this, TAs were stamped only with `managed=jac-scale` and no owner scoping, so they could not even be listed per-autoscaler. ## Fix Reconcile TAs by an ownership label instead of by parsing the ScaledObject. - `_build_trigger_auth_manifest()` stamps every managed TA with `jac-scale/owner=<ScaledObject name>` (plus `app` and `managed`). The owner value is the ScaledObject name, hashed if it exceeds the 63-char label limit, via a shared `_owner_label_value()` helper so the apply and destroy selectors always agree. Because the labels live in the manifest builder, `--show-yaml` renders exactly what `apply()` writes. - `apply()` collects the desired TA names, then `_prune_orphan_trigger_auths()` lists this owner's managed TAs by label and deletes any not in the freshly-applied set. This closes orphan pathway (1). - `destroy()` lists TAs by `jac-scale/owner=<name>,managed=jac-scale` and deletes them (404/422-safe), then deletes the ScaledObject. It no longer reads the ScaledObject, so cleanup works even when the ScaledObject was already deleted out of band. This closes orphan pathway (2) with no early-return hole. - `destroy()` (base `Autoscaler`, `HPAAutoscaler`, `KEDAAutoscaler`) now takes `app_name` instead of a raw resource name and resolves the concrete name internally via `resource_name_for()`, matching how the owner label selector is computed. A single `_scaled_object_name(spec, app_name)` helper is the one definition of the ScaledObject name, used by `apply()` and `_build_manifests()`; `destroy()`'s formula is exactly its fallback branch, so the two cannot disagree. ``` apply(spec): build manifests (validation runs inside the builder) for each manifest in _APPLY_ORDER: create/patch (TAs labelled owner=<SO name>, then the ScaledObject) prune: list TA by owner label -> delete any name not in desired set destroy(app_name, namespace): scaled_object_name = resource_name_for(app_name) list TA by owner label=scaled_object_name -> delete each (404-safe) delete ScaledObject named scaled_object_name (404-safe) ``` ## Migration for TAs that predate the owner label There is deliberately no legacy name-pattern sweep (an earlier revision had one; review showed its truncated-prefix match could cross-delete another app's TAs in a shared namespace, so it was removed rather than narrowed). Instead: - A pre-label TA whose trigger still exists is adopted implicitly: `apply()` patches it by name and the merge patch adds the owner label. From then on it is fully reconciled. - A pre-label TA whose trigger was already removed cannot be named from `spec.triggers`, so nothing touches it; the namespace-wide `destroy_collection()` sweep (`managed=jac-scale`) remains the catch-all for those. ## Tests `test_keda_autoscaler.jac`, all mocked, 42 tests including: - TriggerAuthentication manifest carries app and owner labels for reconciliation - apply prunes a TriggerAuthentication the redeployed trigger set no longer references, and keeps one still referenced - destroy resolves the ScaledObject name from app_name and deletes it - destroy sweeps TriggerAuthentications owned by this ScaledObject before deleting it (works with the ScaledObject already gone, the case the earlier patch could not handle) - destroy tolerates an already-deleted TA (404) and an absent auth CRD (404 on list) - duplicate trigger identities are rejected by the manifest builder, so `--show-yaml` refuses what `apply()` refuses `test_show_yaml_autoscalers.jac` covers the rendered TA manifests carrying the same labels `apply()` writes. `test_factories.jac` green. ## Live cluster validation Verified against a real cluster with real KEDA CRDs, driving this branch's `KEDAAutoscaler.apply()`/`destroy()` directly: | Step | Action | Result | | --- | --- | --- | | apply | trigger `cache` + auth | TA created with `jac-scale/owner`, `app`, `managed` labels | | apply prune | rename trigger `cache` to `queue` | old TA pruned, only the new TA remains | | destroy | destroy with ScaledObject present | 0 TAs, 0 ScaledObjects | | destroy (SO gone) | delete ScaledObject out of band, then destroy | TA still swept by label | | migration | pre-label TA, trigger still live | adopted by apply's patch, swept by destroy | Both orphan pathways are provably closed on a real cluster. ## Notes Rebased on the merged jaseci-labs#8424: trigger validation lives in `_build_manifests`, so the render and apply paths accept and refuse identical specs, and the manifest list is the single desired-state description that both paths share. This PR supersedes the earlier ScaledObject-parsing approach and resolves the partial-apply / already-gone-ScaledObject limitation tracked in jaseci-labs#6909 directly: cleanup is now independent of the ScaledObject. Closes jaseci-labs#6535. Closes jaseci-labs#6909. --------- Co-authored-by: Kugesan Sivasothynathan <kugesan.sivasothynathan@jaseci.org> Co-authored-by: Kugesan Sivasothynathan <153247429+kugesan1105@users.noreply.github.com> (cherry picked from commit e092aaa)
…jac.toml config (jaseci-labs#7709) Closes jaseci-labs#7475 PR jaseci-labs#7421 added a programmatic API for KEDA HTTP Add-on scale-to-zero activation (`HTTPActivationSpec`, `KEDAAutoscaler.apply_http_activation` / `destroy_http_activation`), but left it reachable only by writing a Jac driver script. Every other autoscaler knob in jac-scale is configured declaratively through `jac.toml`, so this PR adds that same entry point for HTTP activation, for both the monolith deploy path and per-service microservice deploys. The programmatic API from jaseci-labs#7421 is unchanged and remains the right tool for callers with a dynamic, create/destroy-on-demand lifecycle (for example, an IDE preview orchestrator). The two entry points are additive, not a replacement of one by the other: `jac.toml` covers a known, standing service; the programmatic API covers a workload created and torn down at runtime. **One mutually-exclusive scaling decision per service, resolved at build time.** `manifest_builder` resolves each service (the gateway included) to exactly one scaling mode - `none`, `metric` (HPA/KEDA), or `http_activation` - computed once and carried through the bundle as a single `scaling` map, replacing two independent resolvers (autoscaler config and http_activation config) that could both fire on the same Deployment. Three consequences, each of which was a real apply-side defect before: - **Correct scale target.** The `ScaledObject`'s `scaleTargetRef`, and the `InterceptorRoute`/`ScaledObject` resource names derived from it, come from the built Deployment manifest (`<name>-deployment`), never re-derived from the bare service name. A bare-name target made `apply_http_activation`'s existence check 404 on a real cluster, killing the deploy. - **No dual scalers.** A service gets a metric autoscaler or an HTTP `ScaledObject`, never both; the modes are mutually exclusive by construction, so KEDA's admission webhook can no longer reject a second `ScaledObject` on an HPA-managed target. - **The gateway never scales to zero.** It is the ingress entry point and must stay warm, so it is structurally excluded from HTTP activation and never inherits a shared `enabled = true`; it falls through to a normal HPA. **Teardown sweeps KEDA resources regardless of the base engine.** `destroy()`'s KEDA sweep previously went through `AutoscalerFactory.create(self.k8s_config.autoscaler_engine, ...)`, so when `autoscaler_engine = "hpa"` it resolved to `HPAAutoscaler.destroy_collection`, which does not know about `InterceptorRoute` or HTTP-activation `ScaledObject` resources, orphaning them on teardown. `destroy()` now also runs a `KEDAAutoscaler(...).destroy_collection(...)` sweep whenever `autoscaler_engine != "keda"`, since HTTP activation needs the KEDA HTTP Add-on regardless of the base engine. **Teardown tolerates the HTTP Add-on's separate RBAC surface, without masking other failures.** The HTTP Add-on's CRD group (`http.keda.sh`, covering `InterceptorRoute`) installs and is permissioned separately from core KEDA's `keda.sh` group (`ScaledObject`, `TriggerAuthentication`), which the base autoscaler already needs. A cluster's RBAC can easily cover one without the other. Both `destroy_http_activation` and `destroy_collection` were changed so a 403 on the `http.keda.sh` side never aborts cleanup of the `keda.sh` side (or vice versa for `destroy_collection`, which still requires 404/422 there), both resources are always attempted regardless of the other's outcome, and if one side hits a genuinely unexpected status (500, for example) that is what propagates, not a tolerable 403 that happened to be checked second. This closes a class of gaps found in review: a single unhandled 403 previously could abort the rest of monolith or microservice teardown (Deployment, Service, PVC, ingress, monitoring, database cleanup) before it ran. **Per-service config inherits from the shared default, except the switch itself and the gateway.** A per-service `[scale.microservices.services.NAME.http_activation]` block merges over `[scale.kubernetes.http_activation]` as a base, so a service that only sets `target_port` still picks up a shared `concurrency_target` from the top level, and a service with no local block at all still inherits a top-level `enabled = true`. An explicit per-service `enabled = false` still overrides that default, so a shared "on" switch can be selectively opted out of per service. The gateway is the one service that never inherits the shared switch (see the scaling-decision point above). **`jac plan` surfaces HTTP-activated services.** The dry-run plan reads the same `scaling` map, so an http_activation service renders its `HTTP activation: 0 -> N` line and appears in the Totals, instead of being invisible in the plan. **A redeploy no longer resets a scaled-to-zero target back to its baseline.** The Deployment manifest's `spec.replicas` is always the configured static default, and every apply PATCHes it onto the live Deployment - with no exclusion for HTTP-activation-mode services, that undid KEDA's own scale-to-zero on every redeploy. `_apply_or_replace` now takes an optional `patch_body` distinct from the manifest used for a fresh create, and the deployment-apply loop strips `replicas` from the PATCH body (not the CREATE body) for `HTTP_ACTIVATION`-mode services, so KEDA's current value is left alone. **No PodDisruptionBudget for an HTTP-activated service, and `jac plan` stops flagging it.** `_build_pdb_manifest` used to compute a replica floor from `hpa.min` regardless of mode, so an HTTP-activated service could still get a PDB with `min_available` set - a budget that's permanently unsatisfiable once the target is actually at zero replicas. It now takes the resolved mode and skips PDB generation entirely for `HTTP_ACTIVATION` services (warning if the user had `pdb` settings configured). `PlanValidator._check_hpa`/`_check_pdb` had the same blind spot for `jac plan`'s diagnostics and now skip both checks for `HTTP_ACTIVATION` services too. **Two services can't silently collide on the same HTTP-activation traffic.** `build_http_activation_spec` now rejects an empty `rules` list at build time - the docs already said an empty list means no traffic ever matches, so a service left at the default was silently deploying unreachable. Separately, `generate_manifests` now runs `_validate_http_activation_rules`, which rejects two services that resolve to identical `InterceptorRoute` match criteria (for example, two services that both inherit the shared top-level rules unchanged) - the KEDA HTTP Add-on interceptor has no way to tell which target a matching request belongs to, so this is now a build-time error naming the colliding services, not a runtime routing ambiguity. **Mode transitions reap the old resource before applying the new one.** `apply_manifests` used to apply the new mode's scaling resource (metric autoscaler, or the HTTP `ScaledObject`) before calling `reap_stale_scaling_resources_for_bundle`, which tears down the previous mode's leftover resource. A service switching modes on a given redeploy collided with its own stale resource - KEDA's admission webhook rejects a second `ScaledObject` on an already-managed target - and only got cleaned up on the next deploy. The reap call now runs before both applies. **A deploy bundle that's entirely HTTP-activation-scaled no longer crashes.** `_wait_for_fleet_rollout` called `self._load_kube_config()`, a method that doesn't exist anywhere in this codebase - the real method is `_load_cluster_config()`. This crashed with an `AttributeError` in exactly the case the surrounding code path exists to handle: a fleet with no metric-scaled services to wait on. **Traffic routing to the interceptor is not wired yet (documented limitation).** jac-scale creates the `InterceptorRoute` and `ScaledObject`, but does not yet rewire the gateway or Ingress backend to the KEDA HTTP interceptor proxy. With `min_replicas = 0`, inbound traffic that reaches the app Service directly (instead of the interceptor) will not wake the pod. The reference doc carries a warning to route through the interceptor yourself for now; the routing-plane rewire is tracked separately as jaseci-labs#7959. **Namespace-wide sweeps still lack application-identity scoping.** Every resource-kind sweep in `destroy()` (Deployments, Services, PDBs, Ingresses, ConfigMaps, and the KEDA resources above) uses a namespace-wide `managed=jac-scale` label selector with no application-identity component, so two jac-scale applications sharing one namespace would delete each other's resources on teardown. This is a pre-existing characteristic of the whole method, not something this PR introduces; fixing it properly requires an app-identity label added at manifest-build time across several files. Tracked separately as jaseci-labs#7710 rather than patched partially here. - `test_http_activation_config.jac` (12 tests): config-to-spec translation (disabled/absent, minimal config defaults, one full-config test covering every optional field, and the four jac.toml-relative validation errors - `target_port`/`target_port_name` mutual exclusivity, a missing `concurrency_target`/`request_rate_target`, empty `rules`, and a rule header missing `name`), the injectable apply/destroy wiring helpers, and that only a 403 from `destroy_http_activation` is caught (a 500 still propagates). Uses a shared `_assert_config_error(cfg, *substrings)` helper instead of hand-rolled raise/catch/assert blocks per test. - `test_http_activation_microservices.jac` (17 tests): the per-service config pass-through and top-level inheritance (including `enabled` itself, and an explicit per-service opt-out); the single scaling decision in `_resolve_scaling` - that an http_activation service resolves to `HTTP_ACTIVATION` mode carrying the `<name>-deployment` scale target and emitting no metric autoscaler, that the gateway is never HTTP-activated even under a shared default (exercised with a real logger so a wrong log-method name is caught without a live deploy), and that a plain service resolves to `METRIC`; the per-service apply loop (only HTTP_ACTIVATION-mode services are applied, using the carried scale-target and service names); the mode-transition reap tests (a METRIC service has its stale HTTP activation resources reaped and its autoscaler left alone, and vice versa, a NONE service has both reaped, and an empty bundle is a no-op); and the cross-service rule-collision validation (identical rules raise, distinct/empty/lone-peer rules don't, and `generate_manifests` raises when two services inherit the shared catch-all rule unchanged). - `test_keda_http_activation.jac` (46 tests): imports the shared fixtures instead of defining them locally, plus the `destroy_http_activation`/`destroy_collection` error-handling coverage - that `destroy_http_activation` still attempts the `ScaledObject` delete after a 403 on the `InterceptorRoute` delete; that a 500 on one delete is not masked by a 403 on the other; that `destroy_collection` tolerates a 403 listing `InterceptorRoute`s without aborting the `ScaledObject`/`TriggerAuthentication` sweep; and that a 403 listing `ScaledObject`s still raises. - `test_plan.jac` (23 tests): includes a case asserting an http_activation service renders its scale-to-zero line and counts in Totals, with no HPA line, and that the metric-autoscaler smoke test reads the new `scaling` bundle key. - `test_pdb_budget.jac` (11 tests): the existing PDB floor/emission cases, plus that an HTTP-activated service gets no PDB even with `pdb.min_available` set, and that the plan validator skips both the HPA and PDB checks for an `HTTP_ACTIVATION` service. - `test_show_yaml_autoscalers.jac` (5 tests): `jac plan --show-yaml` renders the HPA/`ScaledObject` body the apply path would build, that the scale target comes from the resolved plan rather than being re-derived from the service name, that an http_activation-scaled service renders no autoscaler manifest, and that a bundle with no scaling renders nothing. - `http_activation_test_support.jac`: shared fixtures (`default_http_activation_spec`, `default_http_activation_config`, `mocked_keda_autoscaler`) extracted from `test_keda_http_activation.jac` so both files use one source of truth instead of duplicating mock setup. The HTTP activation, KEDA autoscaler, plan, memory-trigger-guard, deployment-overlay, dry-run-purity, ms-config-isolation, and superseded-deployment-reap suites all pass under `jac test`. The microservices `destroy()` engine-gap wiring itself has no dedicated test: like the rest of that method, it requires a live cluster (`_load_cluster_config`, real client construction) with no existing unit-test precedent in this codebase. The error-handling logic it depends on (`destroy_collection`'s group-aware tolerance) is unit-tested directly, as listed above. It was, however, exercised end to end on a real cluster (see Notes to Reviewer). | File | What Changed | |---|---| | `jac/jaclang/scale/deploy/autoscale/http_activation_config.jac` | New. `build_http_activation_spec` translates a `[*.http_activation]` config dict into an `HTTPActivationSpec`, with jac.toml-relative validation errors, including an empty-`rules` check and a friendly error (instead of a bare `KeyError`) for a rule header missing its required `name` key. `apply_http_activation_for_target` / `destroy_http_activation_for_target` wrap the `KEDAAutoscaler` calls behind an injectable interface, shared by both deploy paths below. `destroy_http_activation_for_target` is unconditional (no config gate, so a resource created while enabled stays cleanable even if disabled later) and swallows only a 403 from `destroy_http_activation` (via null-safe `e?.status`, since the checker can't prove `ApiException` isn't the `_optdeps` fallback `Exception`), re-raising anything else. | | `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac` | `destroy_http_activation` now always attempts both the `InterceptorRoute` and `ScaledObject` deletes regardless of the other's outcome, and prefers surfacing a non-403 error over a 403 if both fail differently. `destroy_collection` now tolerates 403 specifically for the `http.keda.sh` group, leaving `keda.sh` groups at the existing strict 404/422 tolerance. | | `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac` | `destroy_collection`'s interface gains a `best_effort: bool = False` parameter for the group-aware 403 tolerance above. | | `jac/jaclang/scale/deploy/autoscale/http_activation.jac` | Renames `HTTPRequestRateMetric.window` to `rate_window` to match the config schema and generated YAML field name. | | `jac/jaclang/scale/config/plugin_config.jac` | New `[scale.kubernetes.http_activation]` schema block (master switch, replica bounds, target port, concurrency/request-rate metrics, routing rules, cold start, timeouts, custom scale target). Documents the per-service `[scale.microservices.services.NAME.http_activation]` override, which falls back to the top-level block. The ~20-field nested schema is now built once by `_http_activation_nested_schema()` and shared by both call sites instead of being duplicated verbatim (the two copies had already drifted: the `rules` description claimed "empty matches all traffic," the opposite of the now-enforced behavior). | | `jac/jaclang/scale/deploy/target/kubernetes/kubernetes_config.jac` | New `http_activation: dict[str, any]` field on `KubernetesConfig`, wired into `to_dict()` / `from_dict()`. | | `jac/jaclang/scale/deploy/target/kubernetes/kubernetes_target.jac` | Monolith teardown (`_destroy_application`) calls `destroy_http_activation_for_target` next to the base-autoscaler destroy, with the engine-gap sweep described above. Also removes `KubernetesTargetBase.destroy`, `_destroy_component`, `_destroy_application`, `_destroy_databases`/`_destroy_database`, and `_wait_for_deletion` (about 360 lines): all were shadowed by `KubernetesTarget`'s own overrides for every real deploy and unreachable in practice. Note for future readers: `_destroy_component` was the reference implementation jaseci-labs#8403 and jaseci-labs#7968 point at for full-teardown component dispatch. After this merges, addressing either issue means writing that dispatch logic fresh in `target.jac` rather than reusing this block. | | `jac/jaclang/scale/deploy/target/kubernetes/manifest_builder.jac` | New `ScalingMode` enum and `_resolve_scaling(svc_name, deployment, service)`: the single scaling decision, reading the scale-target name off the built Deployment manifest, enforcing the gateway exclusion and metric-vs-activation exclusivity. `_get_http_activation_config` merges `[scale.kubernetes.http_activation]` as a base with the per-service block. The build loop emits one `bundle["scaling"]` map (replacing the separate `autoscalers`/`http_activations` keys). `generate_manifests` now also calls `_validate_http_activation_rules` to reject cross-service rule collisions, and `_build_pdb_manifest` takes the resolved mode and skips PDB generation for `HTTP_ACTIVATION` services. | | `jac/jaclang/scale/deploy/target/kubernetes/target.jac` | `apply_http_activations_for_bundle`, the autoscaler apply loop, and the post-deploy reachability-skip set all read `bundle["scaling"]` by mode, using the carried scale-target/service names with no name re-derivation. `destroy()` engine-gap fix (see Notable design points). `apply_manifests` now reaps stale scaling resources before applying the new mode's resource, `_apply_or_replace` takes an optional `patch_body` so a redeploy's PATCH strips `replicas` for `HTTP_ACTIVATION`-mode services instead of resetting KEDA's scale-to-zero, and `_wait_for_fleet_rollout`'s crash on an all-activated fleet is fixed (`_load_cluster_config`, not the nonexistent `_load_kube_config`). | | `jac/jaclang/scale/runtime/cli/plan.jac` | Reads the `scaling` bundle key; new `HTTPActivationView` rendered beside `HPAView`, and Totals counts autoscalers and HTTP activations by mode. `PlanValidator._check_hpa`/`_check_pdb` now read the resolved mode and skip both checks for `HTTP_ACTIVATION` services. | | `jac/jaclang/scale/tests/deploy/http_activation_test_support.jac` | New. Shared fixtures (`default_http_activation_spec`, `default_http_activation_config`, `mocked_keda_autoscaler`) extracted from `test_keda_http_activation.jac`. | | `jac/jaclang/scale/tests/deploy/test_http_activation_config.jac` | Config-to-spec translation tests plus the injectable apply/destroy wiring-helper tests (12 tests), using a shared `_assert_config_error` helper. | | `jac/jaclang/scale/tests/deploy/test_http_activation_microservices.jac` | New. Per-service config inheritance, the `_resolve_scaling` decision (scale-target name, gateway exclusion, metric-vs-activation exclusivity), the mode-transition reap tests, the apply-loop tests, and the cross-service rule-collision validation (17 tests). | | `jac/jaclang/scale/tests/deploy/test_keda_http_activation.jac` | Imports shared fixtures instead of defining them locally, plus the `destroy_http_activation`/`destroy_collection` error-handling tests (46 tests). | | `jac/jaclang/scale/tests/microservices/test_plan.jac` | Reads the new `scaling` bundle key; adds an http_activation render/Totals test (23 tests). | | `jac/jaclang/scale/tests/deploy/test_pdb_budget.jac` | Adds coverage that an `HTTP_ACTIVATION` service gets no PDB even with `pdb.min_available` set, and that the plan validator skips both the HPA and PDB checks for it (11 tests). | | `jac/jaclang/scale/tests/deploy/test_show_yaml_autoscalers.jac` | Adds coverage that `jac plan --show-yaml`'s scale target comes from the resolved plan (not re-derived from the service name) and that an http_activation-scaled service renders no autoscaler manifest (5 tests). | | `jac/jaclang/scale/runtime/cli/diagnostics.jac` | `_check_hpa`/`_check_pdb` read the resolved scaling mode and skip both checks for `HTTP_ACTIVATION` services. | | `jac/jaclang/cli/docs/reference/plugins/jac-scale-kubernetes.md` | Restores the "HTTP Add-on Activation" reference section with the new config keys, both deploy-path examples, a note on the still-available programmatic API, and a warning that inbound traffic must be routed through the interceptor for now (jaseci-labs#7959). | | `jac/jaclang/cli/docs/tutorials/production/kubernetes.md` | New "Scale to zero on an HTTP request" subsection alongside the existing autoscaling tutorial content, with a minimal `jac.toml` example and a link to the full reference section. | | `jac/jaclang/scale/tests/fixtures/keda_http_activation_e2e/` and `jac/jaclang/scale/tests/deploy/keda_http_activation_real_e2e.sh` | Real-cluster e2e fixture and driver, deployed through the `jac.toml` wiring (replaces the old `fixture.yaml`-driven setup). The driver resolves resource names from the unified deploy path's `<app>-deployment` scale target. `keda_http_activation_verify.jac`, the old standalone verify script, is removed - superseded by the `jac.toml`-driven driver. | The full flow was validated end to end on a local `kind` cluster (not just unit tests): - **Runtime channel.** The fixture `jac.toml` carried a `[dev]` stanza (dev binary channel) with `[dev] jaclang_source` pointing at this checkout, so `jac start --scale` ran this branch's deploy code host-side (all manifest generation, the `kubectl apply` calls, and the `InterceptorRoute`/`ScaledObject` reconcile happen in that process). The confirming `jac dev mode - using compiler source at ...` banner appeared on every deploy. - **Cluster.** A single-node `kind` cluster with KEDA core and the KEDA HTTP Add-on installed via helm. Leader election was disabled on the control-plane components for the single-node dev cluster so the pod's first-boot compile could not starve them. - **Result.** `keda_http_activation_real_e2e.sh` PASSED: deploy, idempotent redeploy (get-then-patch, no duplication), both `InterceptorRoute` and `ScaledObject` reconciled to Ready, a request through the interceptor cold-started the target from `0 -> 1` and returned HTTP 200 with the walker's response, then the target scaled back to `0` after the cooldown. - **The three design fixes confirmed on the live cluster:** the `ScaledObject`'s `scaleTargetRef` was `echo-deployment` (not the bare name), the activated service had no metric HPA, and the gateway had a normal HPA rather than a `ScaledObject`. (cherry picked from commit 6ded655)
…at publish time, not from this directory
…a local stdlib copy during backport testing)
…ip jac install (jaseci-labs#8695) A woken pod spent 21s of its 44s wake rebuilding /app/.jac/venv from the bundle's own vendored wheels, identically every time, because /app is an emptyDir and the .jab never carried the built venv. The dependency closure is fully determined at seal time, so build it there instead: - pack_jab now runs the seal binary's own 'jac install' inside the staging tree against the freshly vendored wheels (the exact install pods run today) and seals lib/*/site-packages plus a .jac-deps-hash marker into the .jab under .jac/venv/. The existing pvc-bootstrap untar populates the pod venv with zero pod-side script changes. - install_specs returns early when the marker matches the resolved specs, before ensure_venv can classify the interpreter-less sealed venv as corrupted and delete it. Only pack_jab ever writes the marker, so local 'jac install' behavior is unchanged. - The payload stays deterministic (guarded by the existing digest determinism test): bin/ shebangs, pyvenv.cfg, __pycache__/*.pyc and ensurepip's seeded pip never ship, and RECORD lines pointing outside site-packages are dropped. - Staging failures (cross-arch hosts, thin bundles) log and fall back to today's boot-time install; JAC_SEAL_SKIP_VENV=1 opts out. - Drive-by fix the e2e surfaced: vendor_wheels resolved the project config via discovery, which 'jac run' roots at the script's own tree, so any dev-source seal vendored the jac checkout's closure instead of the app's. It now loads the staged jac.toml directly. (cherry picked from commit fd35eb8)
The 0.34 standalone install routes through PIP_PREFIX (as the pod bootstrap does), so the stage exports it plus PIP_IGNORE_INSTALLED so a seal host whose runtime site already satisfies specs cannot produce a silently empty venv. Also removes a duplicate venv-stage hook the cherry-pick auto-applied with main's argument names, and ports the e2e tests to this line's _pod_binary()/pack_jab signature.
… the fleet wait Measurement scaffolding standing in for jaseci-labs#8723: with a KEDA min-0 service whose trigger is idle at deploy time, the rollout gate waited 1200s for a deployment the autoscaler had deliberately parked at 0 and failed the deploy (jacBuilder run 33836166656). jaseci-labs#8723's status model is the real fix on main; this branch only needs the gate to pass so the wake can be measured.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measurement branch, not meant to merge on its own: #8940 (the merged KEDA suite backport) plus the seal-time venv fix from #8944, stacked so jacBuilder#1642's wake-latency rig can measure the seal fix against the #8940 baseline on the same 0.34.x binary generation, same cluster, same day.
Contents
release/jaclang-0.34.18).fix(scale): seal the app venv into the .jab): the seal builds the app venv once from the vendored wheels and ships it inside the bundle; a woken pod untars it andjac installno-ops on the.jac-deps-hashmarker instead of rebuilding for ~21s.PIP_PREFIX(exactly as the pod bootstrap exports it), so the seal stage sets it plusPIP_IGNORE_INSTALLED, without which a seal host whose runtime site already satisfies specs produces a silently empty venv; also removes a duplicate venv-stage hook the cherry-pick auto-applied with main's argument names, and ports the e2e tests to this line's_pod_binary()/pack_jab(project_dir, provisioned_database, pod_binary)signature.Validation with the published jac 0.34.18 binary
231 passed, 4 skipped across
scale/tests/deploy/,test_fat_bundle(including bundle-digest determinism, now covering the sealed venv), and the newtest_seal_venvsuite; the 5 seal-venv tests include two realpack_jabe2e runs. Pre-existing failures on this line (test_drain, test_fleet_pod_optout, test_rate_limit) are unchanged.How this gets measured
jaseci/jaclang:experimental-<this PR>via the experimental-image workflow dispatched fromrelease/jaclang-0.34.18(its Dockerfile matches this pre-#7969 closure), then jacBuilder#1642'sdeploy-pr.ymlwithexperimental_jaseci_pr=<this PR>andmeasure_wake_latency=true. Expected: the jac-bootstrap init phase collapses from the ~21s-class install measured on the #8940 baseline run to a few seconds of extra bundle untar plus a ~1s marker check, identical on cold and warm legs.