Skip to content

backport(0.34.x): the KEDA scaling suite (#7620 #7899 #8008 #8424 #6908 #7709) for pinned jacBuilder deploys - #8940

Open
kugesan1105 wants to merge 10 commits into
jaseci-labs:release/jaclang-0.34.18from
kugesan1105:backport/keda-suite-0.34.18
Open

backport(0.34.x): the KEDA scaling suite (#7620 #7899 #8008 #8424 #6908 #7709) for pinned jacBuilder deploys#8940
kugesan1105 wants to merge 10 commits into
jaseci-labs:release/jaclang-0.34.18from
kugesan1105:backport/keda-suite-0.34.18

Conversation

@kugesan1105

@kugesan1105 kugesan1105 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Why

jacBuilder runs the pinned 0.34.x release line, and its KEDA scale-to-zero work (jacBuilder#1514, the #8695 wake-latency measurements) needs the declarative KEDA configuration that only exists on main. This PR brings the complete KEDA scaling suite to release/jaclang-0.34.18 so a pinned deploy can drive HTTP scale-to-zero, per-service scale-rate policy, and TriggerAuthentication reconciliation straight from jac.toml.

What is backported (in main's merge order)

PR Commit on main What it adds
#7620 5234c81 KEDA HTTP Add-on capability discovery and preflight diagnostics
#7899 91c7adc per-service hpa.behavior overlay for HPA/KEDA scale-rate policy
#8008 9d2cace render autoscaler manifests into --show-yaml (prerequisite: #8424 extends exactly this rendering path, it cannot apply without it)
#8424 b908596 render KEDA TriggerAuthentication manifests into --show-yaml
#6908 e092aaa reconcile KEDA TriggerAuthentications by ownership label (part 2 of #6535)
#7709 6ded655 declarative KEDA HTTP Add-on activation via jac.toml, one mutually-exclusive scaling decision per service

All six are cherry-pick -x, so each commit names its main-side original. #7709's test restructuring is mirrored faithfully: test_keda_autoscaler.jac, test_keda_http_activation.jac and keda_http_activation_verify.jac are deleted here exactly as on main, superseded by test_http_activation_config.jac, test_http_activation_microservices.jac and the shared http_activation_test_support.jac.

Conflict resolutions, and what deliberately did NOT come along

The 0.34.x line diverged from main at 0.34.6 (July 25), so every conflict was resolved by applying only the named PR's own delta and rejecting unrelated main drift:

  • plugin_config.jac / runtime/docs.md: only the hpa.behavior and http_activation schema additions were grafted onto the 0.34 text; main's unrelated schema drift (nested_each service schemas, file subkey, changed cpu_target default) was not taken.
  • manifest_builder.jac: the ScalingMode enum, _resolve_scaling, the scaling-map bundle shape, and _build_pdb_manifest's HTTP_ACTIVATION early return came in; main's replica-floor/pdb-budget machinery (_replica_floor, pdb_spec, min_available) is separate main work and was not taken, so test_pdb_budget.jac (which tests that machinery) is also not ported.
  • target.jac: fix(scale): render autoscaler manifests into --show-yaml, so the reviewed YAML matches what a deploy applies #8008's _autoscaler_spec/render_autoscaler_manifests extraction and feat(jac-scale): support KEDA HTTP Add-on activation via declarative jac.toml config #7709's scaling-map apply loop were taken; the extracted _resolve_service_triggers keeps 0.34 semantics (its has_memory_request read defaults to true, which reproduces this branch's existing behavior exactly).
  • kubernetes_target.jac: feat(jac-scale): support KEDA HTTP Add-on activation via declarative jac.toml config #7709's deletion of the legacy imperative destroy/wait path applied cleanly once resolved; no orphaned references remain. One deliberate exception to the no-main-drift rule rode along here: KubernetesTargetBase also loses the whole legacy destroy family (destroy, _destroy_application, _destroy_component, _destroy_database, _destroy_databases, _destroy_cache, _destroy_dashboard, _wait_for_deletion, ~470 lines) that main removed separately. It is dead code on this line too: KubernetesTarget.destroy in target.jac overrides the base and references none of the removed helpers, so nothing could reach them before or after.
  • test_pdb_budget.jac itself is not ported (it tests main's replica-floor machinery), but the machinery-free half of its coverage, the _build_pdb_manifest HTTP_ACTIVATION early return and the plan validator's HTTP_ACTIVATION skip, is rehomed as two tests in test_http_activation_microservices.jac (now 32 passing).
  • test_plan.jac: the new HTTP-activation render test was adapted to this branch's _run_plan(ms_cfg_pair=...) fixture (main's ms_toml parameter is later drift). One assert in test_memory_trigger_guard.jac uses str(...) around a MagicMock because this line's checker is stricter than main's; it landed one commit later than intended (in the fix(scale): render autoscaler manifests into --show-yaml, so the reviewed YAML matches what a deploy applies #8008 pick) during resolution.
  • .github/workflows/ci.yml: feat(jac-scale): support KEDA HTTP Add-on activation via declarative jac.toml config #7709's test-scale-k8s e2e lane was NOT ported. It depends on main-only CI infrastructure (jac-kit action, build-kit job, blacksmith runners) that this branch's CI does not have. The real-cluster e2e script itself (keda_http_activation_real_e2e.sh) ships and can be run manually.
  • Release-note fragments are dropped in a final commit, matching this branch's convention (the 0.34.x line writes its notes at publish time, not from release_notes/unreleased/).

Validation

Everything below ran with the published jac 0.34.18 linux-x86_64 release binary against this branch's source:

  • jac/jaclang/scale/tests/deploy/: 212 passed, 4 skipped after the final pick (232 passed, 4 skipped including test_plan.jac after the fmt pass).
  • The new suites specifically: test_http_activation_config 12 passed, test_http_activation_microservices 30 passed, plus test_show_yaml_autoscalers, test_memory_trigger_guard, test_factories, test_deployment_overlay, test_plan all green.
  • jac/jaclang/scale/tests/microservices/: 217 passed; the 12 failures in test_drain, test_fleet_pod_optout and test_rate_limit fail identically on the untouched branch head (verified side by side at 529fb29), so they are pre-existing on this line and unrelated to the backport.
  • jac fmt with the 0.34.18 binary over every touched file, committed separately per this branch's convention.

Developer-Linus and others added 9 commits September 4, 2026 04:44
…ght diagnostics (jaseci-labs#7620)

## Summary

Closes jaseci-labs#7404.

`KEDAAutoscaler.preflight()` only ever verified that the core
`keda.sh/v1alpha1 ScaledObject` API existed on a cluster. That check is
not sufficient for HTTP activated workloads, which also depend on the
KEDA HTTP Add-on's `InterceptorRoute` API, two Kubernetes Services (the
external scaler and the interceptor proxy), and a compatible Add-on
version. Without these checks, `apply_http_activation` could create a
valid `ScaledObject` that never actually activates from HTTP traffic,
with no clear diagnostic explaining why.

This PR introduces structured capability discovery for the KEDA HTTP
Add-on and wires it into `apply_http_activation`, so a broken or
partially installed cluster fails with an actionable error instead of
silently deploying a nonfunctional workload.

## Changes

| File | Changes |
|---|---|
| `jac/jaclang/scale/deploy/autoscale/keda_capabilities.jac` (new file)
| Adds the `KEDACapabilities` data object, matching the shape specified
in the issue: `core_available`, `http_addon_available`,
`interceptor_route_api_version`, `external_scaler_address`,
`interceptor_service_address`, `warnings`, and `errors`. Also adds
`SUPPORTED_HTTP_INTERCEPTOR_API_VERSIONS`. |
| `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac` | Imports
`KEDACapabilities`.<br>Adds a new `interceptor_service_address` field,
defaulted the same way `http_scaler_address` already is.<br>Replaces the
old bool only `_http_preflight_cache` with `_capabilities_cache`, which
holds a full `KEDACapabilities` object per cluster.<br>Declares
`discover_capabilities`, `invalidate_capabilities`,
`_check_service_address`, `format_capabilities`, and
`_capabilities_cache_key`. |
| `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac` |
`discover_capabilities` checks the core `ScaledObject` API and the HTTP
Add-on's `InterceptorRoute` API independently, so a missing core install
is distinguished from a missing Add-on. A 403 on either check is
reported as a distinct permission error rather than being folded into
"not installed."<br>If the `v1beta1 InterceptorRoute` API is absent (not
RBAC denied), it additionally probes the legacy `v1alpha1
HTTPScaledObject` API. If that is present, the result is a distinct
"upgrade" warning with the exact `helm upgrade` command, rather than
being reported as "Add-on not installed." A 403 on this legacy probe is
reported as its own permission error rather than being folded into "not
installed."<br>`_check_service_address` verifies the external scaler and
interceptor proxy Services actually resolve, and reports which one is
missing.<br>Results are cached per cluster identity via a new
`_capabilities_cache_key` helper, which combines `_get_cluster_key()`
with this instance's configured `http_scaler_address` and
`interceptor_service_address`. This keeps two instances that target the
same cluster but are configured with different service addresses from
reusing an incompatible cached result.
`discover_capabilities(refresh=True)` and `invalidate_capabilities()`
remain the explicit refresh paths.<br>`format_capabilities` produces a
single concise, human readable line for CLI output.<br>`preflight()` now
also catches a 403 and returns a distinct permission error instead of
letting the `ApiException` propagate unhandled, which is what happened
previously.<br>`apply_http_activation` now calls
`discover_capabilities()` instead of its own inline, duplicate check. If
discovery reports any errors (missing service, RBAC denied, or similar),
it raises `ValueError` with the details. If the core KEDA API or the
HTTP Add-on is simply absent (no errors, just not installed), it keeps
the existing behavior of logging a warning and returning `False` rather
than raising. A partial install, where the Add-on and its services are
healthy but core KEDA itself is missing, now also falls into this gate
instead of proceeding to create resources that would fail against a
nonexistent API. |
| `jac/jaclang/scale/tests/deploy/test_keda_http_activation.jac` | Adds
a new capability discovery test section covering: core only clusters,
full installs, missing external scaler service, missing interceptor
service, legacy Add-on detection, RBAC denied on the core check, the
HTTP Add-on check, and the legacy probe, cache and refresh behavior,
cache isolation between instances with different service addresses,
explicit invalidation, the raise on broken capabilities and skip on
missing core behavior in `apply_http_activation`, `format_capabilities`
output, and the `preflight()` 403 fix. Test setup uses small composable
mock builders (`_custom_api_for`, `_core_api_for`) rather than one mock
per scenario, and one table driven test covers the distinct diagnostic
cases to avoid duplicating near identical assertions. |

## Out of scope

As noted in the issue: no auto install or upgrade of KEDA, no ingress
creation, no general health checking of every scaler type. This PR also
does not verify that a workload's traffic is actually routed through the
interceptor at runtime, only that the interceptor's own API and Service
objects exist and resolve. That distinction is called out explicitly in
the implementation plan for this issue.

## Test plan

- [x] Core only clusters report core available and HTTP unavailable.
- [x] Fully installed clusters report HTTP ready with no errors.
- [x] Missing external scaler service, missing interceptor service,
legacy Add-on version, and RBAC denied cases (core check, HTTP Add-on
check, and legacy probe) each produce a distinct diagnostic.
- [x] Results are cached per cluster identity, including the configured
service addresses, and refresh via `discover_capabilities(refresh=True)`
and `invalidate_capabilities()`.
- [x] `apply_http_activation` raises when capabilities are broken,
returns `False` gracefully when the Add-on or core KEDA is simply
absent, and does not proceed to create resources on a partial install.
- [x] Documentation update showing the supported Helm install and
upgrade flow.
- [x] Full deploy test suite (169 tests) passes with no regressions.

---------

Co-authored-by: Kugesan Sivasothynathan <kugesan.sivasothynathan@jaseci.org>
(cherry picked from commit 5234c81)
… policy (jaseci-labs#7899)

Closes jaseci-labs#7764.

`jac-scale`'s generated HPA `behavior` block (added in jaseci-labs#7445) has always
shipped one hardcoded scale-down policy — `{"type": "Percent", "value":
50, "periodSeconds": 60}`, no `selectPolicy` — with no way to override
it per service. Because the deploy path re-applies the full generated
manifest on every `jac start --scale`, any out-of-band `kubectl patch`
on the HPA gets wiped on the next deploy, forcing a
re-patch-after-every-deploy loop in CI.

This PR adds a `hpa.behavior` fragment to the per-service config table,
deep-merged over the generated default. It's the first slice of jaseci-labs#7764 —
connection-heavy services (WebSockets, SSE) can now ask for a gentle
scale-down (e.g. at most 2 pods every 5 minutes) without imposing that
rate on every other service through the app-global
`autoscaler_cooldown`, and the override survives every subsequent
deploy.

```toml
[scale.microservices.services.chat_sv.hpa.behavior.scaleDown]
stabilizationWindowSeconds = 600
selectPolicy = "Min"
policies = [{ type = "Pods", value = 2, periodSeconds = 300 }]
```

- **`AutoscalerSpec`** (`autoscaler.jac`) gains a `behavior_overlay:
dict[str, any] = {}` field.
- **`Autoscaler._build_behavior`** deep-merges `behavior_overlay` over
the hardcoded default via the existing `merge_manifest_overlay` (same
semantics already used for `deployment_overlay`: maps merge recursively;
the `policies` list carries no `name` key, so it's replaced wholesale
rather than merged entry-by-entry, matching how you'd actually want to
specify a policy set). Both the HPA and KEDA engines pick this up for
free — they already route through the same `_build_behavior` call, so
this is a single-point change.
- **`ManifestBuilder._get_autoscaler_config`** (`manifest_builder.jac`)
resolves the fragment from the per-service `hpa.behavior` table.
- **`KubernetesTarget`**'s deploy loop (`target.jac`) threads it into
`AutoscalerSpec(...)` as `behavior_overlay=cfg.get("behavior", {})`.
- **Config schema** (`plugin_config.jac`) and the **K8s reference
table** (`runtime/docs.md`) document the new key.

`_build_behavior` resolves `merge_manifest_overlay` via a
function-scoped import rather than a top-level one.
`manifest_builder.jac` already imports constants from `autoscaler.jac`
at module level, so a top-level import in the other direction would be a
circular import; a deferred import inside the function (evaluated well
after both modules are fully loaded) is the same pattern already used
elsewhere in this codebase (e.g. `manifest_builder.jac`'s own
`_pod_entrypoint`) for exactly this situation.

- `service_overlay` / first-class `Service` `annotations` + gateway
`sessionAffinity` knobs
- PDB `min_available` and percentage (`"50%"`) support

Written test-first: 11 new tests across the files that already exercise
the touched code paths (`test_deployment_overlay.jac`,
`test_factories.jac`, `test_keda_autoscaler.jac`,
`test_memory_trigger_guard.jac`), confirmed red against the
pre-implementation tree, then green after:

- `_build_behavior` merge semantics in isolation: no-overlay default,
empty-overlay no-op, `scaleDown` override, `scaleUp` override, wholesale
(not per-item) `policies` replacement.
- Both engines apply the overlay: HPA's `_build_manifests`
(`spec.behavior`) and KEDA's `_build_manifests`
(`advanced.horizontalPodAutoscalerConfig.behavior`).
- Config plumbing: `_get_autoscaler_config` surfaces `hpa.behavior`,
defaults to `{}` when unset, doesn't leak across services, and
`hpa.enabled = false` still short-circuits before behavior is
considered.

Also verified against a real cluster, not just unit tests. Deployed the
`jac-shop` 3-service microservice fixture
(`jaclang/scale/tests/fixtures/k8s_e2e`) to a local `kind` cluster with
a `hpa.behavior.scaleDown` override configured on `products_app`, via
`jac start main.jac --scale`:

- **First deploy**: all 4 services (gateway + 3) reached Ready, and the
live `products-app-hpa` object's `spec.behavior.scaleDown` matched the
configured override exactly (`stabilizationWindowSeconds: 600`,
`selectPolicy: Min`, `policies: [{type: Pods, value: 2, periodSeconds:
300}]`), with `scaleUp` untouched at the generated default.
- **Redeploy**: re-ran `jac start main.jac --scale` against the same
cluster and diffed `products-app-hpa`'s `spec.behavior.scaleDown` before
and after — byte-identical. Confirms the actual pain point from the
issue: the override survives repeated applies instead of being reset to
the old hardcoded `{Percent, 50, 60s}` shape.

- [x] A per-service `hpa.behavior` fragment in `jac.toml` survives
repeated `jac start --scale` applies with no external patching.
- [x] Global `autoscaler_*` keys remain the defaults; per-service
fragment wins on merge.
- [x] Works for both `autoscaler_engine = "hpa"` and `"keda"`.
- [x] Config schema (`plugin_config.jac`) and docs updated;
overlay-merge unit tests added alongside the existing
`test_deployment_overlay.jac` patterns.

---------

Co-authored-by: Kugesan Sivasothynathan <153247429+kugesan1105@users.noreply.github.com>
Co-authored-by: Kugesan Sivasothynathan <kugesan.sivasothynathan@jaseci.org>
(cherry picked from commit 91c7adc)
…ewed YAML matches what a deploy applies (jaseci-labs#8008)

`--show-yaml` printed pvcs, deployments, services, pdbs and ingress but
dropped autoscalers, while the totals line counted them and the service
view rendered HPA details from them. The reviewed YAML was not the
applied
artifact, for exactly the resources most likely to cause a production
surprise.

The bundle's `autoscalers` key holds CONFIGS, not manifests; the real
HPA/ScaledObject is synthesized only inside the apply loop. Adding the
key
to the flatten list would therefore have dumped config dicts. Instead:

- `deploy/target/kubernetes/target.jac`: the AutoscalerSpec construction
is
extracted out of the apply loop into `_autoscaler_spec` (single source),
and a new `render_autoscaler_manifests(bundle)` builds the same objects
  the apply path would, via each engine's `_build_manifests`. Pure - no
  Kubernetes API calls.
- `runtime/cli/plan.jac`: `Plan.from_target` stashes the rendered
manifests
  into the bundle; `_flatten_manifests` emits them into --show-yaml.

- New test_show_yaml_autoscalers (3): real KubernetesTarget with the HPA
and KEDA engines; asserts kind, scaleTargetRef, min/max mapping, and
that
  KEDA's ScaledObject body (not its name/body wrapper) is rendered.
- test_plan extended (+2): autoscaler manifests reach the YAML dump,
plus a
  sentinel test asserting every manifest-bearing bundle key reaches
  --show-yaml (the issue's drift-proofing suggestion).
- jac-ec2 rig, linked-source dev binary on this branch: all passing -
test_show_yaml_autoscalers (3, new), test_plan (19, includes the 2 new),
test_factories (28), test_keda_autoscaler (34),
test_memory_trigger_guard
  (8), test_dry_run_purity (12).
- NOT run: a live cluster apply of HPA/KEDA objects (the render path is
pure; the apply path is unchanged except for the spec extraction). Full
  repo suite not run locally.

- KEDA TriggerAuthentication objects are applied but still not rendered
into --show-yaml (built inline in `_apply_trigger_auth`; needs the same
  build/apply split).
- Normalize the `_build_manifests` contract: KEDA returns a {name, body}
  wrapper, HPA returns the bare manifest.

Closes jaseci-labs#7915. Part of jaseci-labs#7900.

---------

Co-authored-by: Thamirawaran Sathiyalogeswaran <107134124+Thamirawaran@users.noreply.github.com>
(cherry picked from commit 9d2cace)
…aml (jaseci-labs#8424)

Closes jaseci-labs#8409.

`jac start --scale --show-yaml` (via `render_autoscaler_manifests`)
prints
the autoscaler objects a deploy would apply, but for KEDA it only ever
printed the `ScaledObject`. A trigger with `auth.secret_refs` also
causes
`apply()` to create a `TriggerAuthentication`, and that half of the
footprint was invisible to the dry-run preview. This PR unifies the
`_build_manifests` contract across both autoscaler engines so the render
path renders the same objects `apply()` creates, with nothing built as a
side effect of a live API call.

`_build_manifests` for KEDA only ever constructed the `ScaledObject`
body.
The `TriggerAuthentication` for an authenticated trigger was built and
written to the cluster exclusively inside `KEDAAutoscaler.apply` via
`_apply_trigger_auth`, a function that conflated *building* the manifest
with *reconciling* it against the cluster (get → patch-or-create) in one
step. Because the manifest never existed as a standalone, returnable
value,
`render_autoscaler_manifests` (the `--show-yaml` path) had no way to see
it.

A secondary wart compounded this: `_build_manifests` returned a
different
shape per engine (HPA a bare manifest dict, KEDA a `{"name", "body"}`
wrapper), so the render path added defensive shape-sniffing
(`built["body"] if "body" in built else built`) instead of a real
contract.
That sniff is what let the KEDA gap go unnoticed - it happily returned
"a manifest" without anyone checking whether it was the *only* manifest
KEDA needed to emit.

Net effect: a KEDA autoscaler config with an auth-bearing trigger
dry-ran
clean (N ScaledObjects, 0 TriggerAuthentications) and then `apply()`
silently created N ScaledObjects **and** M TriggerAuthentications the
preview never showed - exactly the class of surprise `--show-yaml`
exists
to prevent, and the same shape of gap jaseci-labs#8008 already fixed once for
autoscalers generally.

`_build_manifests` now returns one `list[dict]` contract for both
engines:

- `HPAAutoscaler._build_manifests` returns `[hpa]`.
- `KEDAAutoscaler._build_manifests` returns
  `[scaled_object, *trigger_authentications]`.

To make that possible without duplicating logic between the render and
apply paths:

- The `TriggerAuthentication` body construction is extracted out of the
old
  apply-only `_apply_trigger_auth` into a pure builder,
  `_build_trigger_auth_manifest`, which returns the manifest (or `None`
  when a trigger has no auth) and makes no API calls.
- A shared `_apply_custom_object` reconciler (get → patch-or-create)
  replaces the two nearly-identical get/patch/create blocks that
  previously existed separately for `TriggerAuthentication` and
  `ScaledObject` writes in `KEDAAutoscaler.apply`.
- `render_autoscaler_manifests` (`target.jac`) now does
`out.extend(autoscaler._build_manifests(...))` instead of shape-sniffing
a single object out of the return value, so it renders every manifest an
  engine produces, not just the first one.

Apply-time ordering is unchanged: `TriggerAuthentication` objects are
still
written before the `ScaledObject` that references them via
`authenticationRef`, since KEDA's admission webhook needs the referenced
auth resource to exist first.

| File | Change |
|---|---|
| `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac` | Interface
updates: `_build_manifests` returns `list[dict[str, any]]`; new
`_build_trigger_auth_manifest` and `_apply_custom_object` declarations
replace `_apply_trigger_auth`. |
| `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac` |
`_build_manifests` assembles `[scaled_object, *trigger_auths]`;
trigger-auth body construction moved into pure
`_build_trigger_auth_manifest`; new `_apply_custom_object` reconciler;
`apply()` applies trigger-auth manifests before the ScaledObject via the
shared reconciler. |
| `jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.jac` |
`_build_manifests` interface updated to `list[dict[str, any]]`. |
| `jac/jaclang/scale/deploy/autoscale/hpa_autoscaler.impl.jac` |
`_build_manifests` wraps its single manifest in a list; `apply()`
unwraps `[0]`. |
| `jac/jaclang/scale/deploy/target/kubernetes/target.jac` |
`render_autoscaler_manifests` flattens the manifest list from each
engine instead of shape-sniffing a single object. |
| `jac/jaclang/scale/tests/deploy/test_factories.jac` | HPA
`_build_manifests` assertions updated to the list contract. |
| `jac/jaclang/scale/tests/deploy/test_keda_autoscaler.jac` | KEDA
`_build_manifests` assertions updated to the list contract (ScaledObject
at index 0, no more `name`/`body` wrapper). |
| `jac/jaclang/scale/tests/deploy/test_show_yaml_autoscalers.jac` | New
regression test: an authenticated KEDA trigger renders both the
`ScaledObject` and its matching `TriggerAuthentication` through
`render_autoscaler_manifests`. |

Ran the full `deploy` test group locally with `jac test` (one process
per
file, matching how CI's `test-scale (deploy)` matrix leg runs it): all
27
files, 293 tests pass, both before-vs-after on the three files this PR
touches and as a full-group run afterward to catch cross-file fallout.

- `test_show_yaml_autoscalers` (4, 1 new) - the regression test for this
  issue. Builds a bundle with one KEDA-engine service carrying an
  authenticated `redis` trigger (`auth.secret_refs`) and asserts
  `render_autoscaler_manifests` returns exactly two manifests: the
`ScaledObject` at index 0 and a `TriggerAuthentication` at index 1 whose
`metadata.name` matches the `authenticationRef.name` KEDA embedded in
the
ScaledObject's trigger, with the `secretTargetRef` mapped correctly from
config. Confirmed this test fails against the pre-fix code (dropping the
  `_build_trigger_auth_manifest` loop reproduces the original bug -
  `len(manifests) == 1`) and passes with the fix. The other two existing
  tests in this file (HPA render, empty-bundle render) are unchanged and
  still pass, guarding against a regression in the ordinary case.
- `test_keda_autoscaler` (34) - full suite for the engine this PR
changes
most: `_build_manifests` shape and field mapping (9 tests, now indexing
  `[0]` into the list), trigger-authentication naming/collision rules,
  `preflight`, and - the part most likely to break silently on a
  build/apply split - `apply()`'s behavior end to end against a mocked
  `CustomObjectsApi`: that a `TriggerAuthentication` is still created
  before its `ScaledObject`, that the name written to the cluster still
  matches the `authenticationRef` the ScaledObject carries, and that a
KEDA-CRDs-not-installed cluster still short-circuits before either
write.
  All pass unmodified in assertion logic (only the `_build_manifests`
  indexing changed), which is the signal that `apply()`'s observable
  behavior toward the cluster is unchanged by the refactor.
- `test_factories` (32) - HPA `_build_manifests` output (exact-dict
  equality against the expected manifest, and the `behavior_overlay`
  merge) re-verified against the new `[hpa]` list contract.

Also validated end to end against a real cluster, not just mocks:

- **Cluster**: local `kind` (v0.32.0) with KEDA core installed via the
standard Helm chart (`kedacore/keda`, namespace `keda`);
`scaledobjects.keda.sh`
  and `triggerauthentications.keda.sh` CRDs confirmed present. A static
`hostPath` PV under a `no-provisioner` StorageClass (`jac-rwx`) stands
in
  for the bundle PVC's `ReadWriteMany` requirement, the same technique
  `jac/jaclang/scale/scripts/e2e_lib.sh`'s `provision_kind_rwx_storage`
uses for the kind-based CI e2e leg (kind's `local-path` StorageClass is
  RWO-only; a static hostPath PV is safe on a single-node cluster).
- **Project**: a minimal scratch app (`main.jac`, one walker, no client)
with
`[dev] jaclang_source` pointing at this checkout, so both the CLI and
the
  deployed pods run the fixed source, not a published release binary.
  `[scale.kubernetes] autoscaler_engine = "keda"` plus one
  `[[scale.microservices.services.<name>.triggers]]` entry with
`type = "redis"` and `auth.secret_refs` - the exact shape jaseci-labs#8409
described.
- **Dry-run** (`jac start --scale --dry-run --show-yaml`, no cluster
contact): the rendered YAML stream includes both the `ScaledObject` and
a
  `TriggerAuthentication` object, with the `ScaledObject`'s
  `authenticationRef.name` matching the `TriggerAuthentication`'s
`metadata.name` exactly (`kedaval-7dcd81a6-ta`) - this is the bug fixed,
  observed through the real CLI rather than only through the test suite.
- **Live apply** (`jac start --scale`): the same `TriggerAuthentication`
(`kedaval-7dcd81a6-ta`, `SECRET: redis-secret`) is present on the
cluster
  via `kubectl get triggerauthentications`, and the `ScaledObject`'s
  `AUTHENTICATIONS` column names it - dry-run and `apply()` now agree,
  which is the whole point of the fix. `kedaval-scaledobject` reports
`READY=False`/`ACTIVE=Unknown` only because no real Redis exists at the
  configured address; that's expected and irrelevant to what's being
validated here (the auth object's existence and wiring, not live
scaling).
- **App health**: both `gateway-deployment` and `kedaval-deployment`
reached
`1/1 Running`, the fleet rollout reported `2/2 deployments serving`, and
`/health` on the gateway reported both services `"status": "healthy"` -
  confirming the fix doesn't regress the ordinary deploy path.

Configure a KEDA autoscaler with an auth-bearing trigger and run
`jac start --scale --show-yaml`: both the `ScaledObject` and its
`TriggerAuthentication` now render, matching what `apply()` creates.

---------

Co-authored-by: Kugesan Sivasothynathan <kugesan.sivasothynathan@jaseci.org>
Co-authored-by: Kugesan Sivasothynathan <153247429+kugesan1105@users.noreply.github.com>
(cherry picked from commit b908596)
…bel (part 2 - final - of issue jaseci-labs#6535) (jaseci-labs#6908)

## Problem

`KEDAAutoscaler` orphaned `TriggerAuthentication` resources in two
distinct ways:

1. **Redeploy with a changed trigger set.** TA names are derived from
the trigger identity (`app + trigger-key`, hashed). When a redeploy
renames, removes, or reorders triggers, `apply()` creates new TAs under
new names and re-points the ScaledObject at them, but never deletes the
TAs from the previous config. `apply()` had no prune step, so every such
redeploy left a stale TA behind, unreferenced by the ScaledObject.

2. **destroy().** `destroy()` deleted only the ScaledObject, leaving
every TA its `apply()` created behind.

The earlier approach tried to fix (2) by reading
`authenticationRef.name` off the live ScaledObject. That could not fix
(1) at all (the stale TA is no longer referenced by the ScaledObject, so
it is never found), and it left a hole in (2): if the ScaledObject was
already gone, the GET returned 404 and `destroy()` returned early,
stranding any TAs permanently.

## Root cause

There was no ownership-based reconciliation. TAs are children whose
lifecycle should be bound to the autoscaler, but the code bound them
only by name derivation plus an imperative "read the ScaledObject,
delete what it currently points at". Nothing ever reconciled the actual
set of managed TAs in the namespace against the desired set. Compounding
this, TAs were stamped only with `managed=jac-scale` and no owner
scoping, so they could not even be listed per-autoscaler.

## Fix

Reconcile TAs by an ownership label instead of by parsing the
ScaledObject.

- `_build_trigger_auth_manifest()` stamps every managed TA with
`jac-scale/owner=<ScaledObject name>` (plus `app` and `managed`). The
owner value is the ScaledObject name, hashed if it exceeds the 63-char
label limit, via a shared `_owner_label_value()` helper so the apply and
destroy selectors always agree. Because the labels live in the manifest
builder, `--show-yaml` renders exactly what `apply()` writes.
- `apply()` collects the desired TA names, then
`_prune_orphan_trigger_auths()` lists this owner's managed TAs by label
and deletes any not in the freshly-applied set. This closes orphan
pathway (1).
- `destroy()` lists TAs by `jac-scale/owner=<name>,managed=jac-scale`
and deletes them (404/422-safe), then deletes the ScaledObject. It no
longer reads the ScaledObject, so cleanup works even when the
ScaledObject was already deleted out of band. This closes orphan pathway
(2) with no early-return hole.
- `destroy()` (base `Autoscaler`, `HPAAutoscaler`, `KEDAAutoscaler`) now
takes `app_name` instead of a raw resource name and resolves the
concrete name internally via `resource_name_for()`, matching how the
owner label selector is computed. A single `_scaled_object_name(spec,
app_name)` helper is the one definition of the ScaledObject name, used
by `apply()` and `_build_manifests()`; `destroy()`'s formula is exactly
its fallback branch, so the two cannot disagree.

```
apply(spec):
  build manifests (validation runs inside the builder)
  for each manifest in _APPLY_ORDER: create/patch (TAs labelled owner=<SO name>, then the ScaledObject)
  prune: list TA by owner label -> delete any name not in desired set

destroy(app_name, namespace):
  scaled_object_name = resource_name_for(app_name)
  list TA by owner label=scaled_object_name -> delete each (404-safe)
  delete ScaledObject named scaled_object_name (404-safe)
```

## Migration for TAs that predate the owner label

There is deliberately no legacy name-pattern sweep (an earlier revision
had one; review showed its truncated-prefix match could cross-delete
another app's TAs in a shared namespace, so it was removed rather than
narrowed). Instead:

- A pre-label TA whose trigger still exists is adopted implicitly:
`apply()` patches it by name and the merge patch adds the owner label.
From then on it is fully reconciled.
- A pre-label TA whose trigger was already removed cannot be named from
`spec.triggers`, so nothing touches it; the namespace-wide
`destroy_collection()` sweep (`managed=jac-scale`) remains the catch-all
for those.

## Tests

`test_keda_autoscaler.jac`, all mocked, 42 tests including:

- TriggerAuthentication manifest carries app and owner labels for
reconciliation
- apply prunes a TriggerAuthentication the redeployed trigger set no
longer references, and keeps one still referenced
- destroy resolves the ScaledObject name from app_name and deletes it
- destroy sweeps TriggerAuthentications owned by this ScaledObject
before deleting it (works with the ScaledObject already gone, the case
the earlier patch could not handle)
- destroy tolerates an already-deleted TA (404) and an absent auth CRD
(404 on list)
- duplicate trigger identities are rejected by the manifest builder, so
`--show-yaml` refuses what `apply()` refuses

`test_show_yaml_autoscalers.jac` covers the rendered TA manifests
carrying the same labels `apply()` writes. `test_factories.jac` green.

## Live cluster validation

Verified against a real cluster with real KEDA CRDs, driving this
branch's `KEDAAutoscaler.apply()`/`destroy()` directly:

| Step | Action | Result |
| --- | --- | --- |
| apply | trigger `cache` + auth | TA created with `jac-scale/owner`,
`app`, `managed` labels |
| apply prune | rename trigger `cache` to `queue` | old TA pruned, only
the new TA remains |
| destroy | destroy with ScaledObject present | 0 TAs, 0 ScaledObjects |
| destroy (SO gone) | delete ScaledObject out of band, then destroy | TA
still swept by label |
| migration | pre-label TA, trigger still live | adopted by apply's
patch, swept by destroy |

Both orphan pathways are provably closed on a real cluster.

## Notes

Rebased on the merged jaseci-labs#8424: trigger validation lives in
`_build_manifests`, so the render and apply paths accept and refuse
identical specs, and the manifest list is the single desired-state
description that both paths share. This PR supersedes the earlier
ScaledObject-parsing approach and resolves the partial-apply /
already-gone-ScaledObject limitation tracked in jaseci-labs#6909 directly: cleanup
is now independent of the ScaledObject.

Closes jaseci-labs#6535.
Closes jaseci-labs#6909.

---------

Co-authored-by: Kugesan Sivasothynathan <kugesan.sivasothynathan@jaseci.org>
Co-authored-by: Kugesan Sivasothynathan <153247429+kugesan1105@users.noreply.github.com>
(cherry picked from commit e092aaa)
…jac.toml config (jaseci-labs#7709)

Closes jaseci-labs#7475

PR jaseci-labs#7421 added a programmatic API for KEDA HTTP Add-on scale-to-zero
activation (`HTTPActivationSpec`, `KEDAAutoscaler.apply_http_activation`
/ `destroy_http_activation`), but left it reachable only by writing a
Jac driver script. Every other autoscaler knob in jac-scale is
configured declaratively through `jac.toml`, so this PR adds that same
entry point for HTTP activation, for both the monolith deploy path and
per-service microservice deploys.

The programmatic API from jaseci-labs#7421 is unchanged and remains the right tool
for callers with a dynamic, create/destroy-on-demand lifecycle (for
example, an IDE preview orchestrator). The two entry points are
additive, not a replacement of one by the other: `jac.toml` covers a
known, standing service; the programmatic API covers a workload created
and torn down at runtime.

**One mutually-exclusive scaling decision per service, resolved at build
time.** `manifest_builder` resolves each service (the gateway included)
to exactly one scaling mode - `none`, `metric` (HPA/KEDA), or
`http_activation` - computed once and carried through the bundle as a
single `scaling` map, replacing two independent resolvers (autoscaler
config and http_activation config) that could both fire on the same
Deployment. Three consequences, each of which was a real apply-side
defect before:

- **Correct scale target.** The `ScaledObject`'s `scaleTargetRef`, and
the `InterceptorRoute`/`ScaledObject` resource names derived from it,
come from the built Deployment manifest (`<name>-deployment`), never
re-derived from the bare service name. A bare-name target made
`apply_http_activation`'s existence check 404 on a real cluster, killing
the deploy.
- **No dual scalers.** A service gets a metric autoscaler or an HTTP
`ScaledObject`, never both; the modes are mutually exclusive by
construction, so KEDA's admission webhook can no longer reject a second
`ScaledObject` on an HPA-managed target.
- **The gateway never scales to zero.** It is the ingress entry point
and must stay warm, so it is structurally excluded from HTTP activation
and never inherits a shared `enabled = true`; it falls through to a
normal HPA.

**Teardown sweeps KEDA resources regardless of the base engine.**
`destroy()`'s KEDA sweep previously went through
`AutoscalerFactory.create(self.k8s_config.autoscaler_engine, ...)`, so
when `autoscaler_engine = "hpa"` it resolved to
`HPAAutoscaler.destroy_collection`, which does not know about
`InterceptorRoute` or HTTP-activation `ScaledObject` resources,
orphaning them on teardown. `destroy()` now also runs a
`KEDAAutoscaler(...).destroy_collection(...)` sweep whenever
`autoscaler_engine != "keda"`, since HTTP activation needs the KEDA HTTP
Add-on regardless of the base engine.

**Teardown tolerates the HTTP Add-on's separate RBAC surface, without
masking other failures.** The HTTP Add-on's CRD group (`http.keda.sh`,
covering `InterceptorRoute`) installs and is permissioned separately
from core KEDA's `keda.sh` group (`ScaledObject`,
`TriggerAuthentication`), which the base autoscaler already needs. A
cluster's RBAC can easily cover one without the other. Both
`destroy_http_activation` and `destroy_collection` were changed so a 403
on the `http.keda.sh` side never aborts cleanup of the `keda.sh` side
(or vice versa for `destroy_collection`, which still requires 404/422
there), both resources are always attempted regardless of the other's
outcome, and if one side hits a genuinely unexpected status (500, for
example) that is what propagates, not a tolerable 403 that happened to
be checked second. This closes a class of gaps found in review: a single
unhandled 403 previously could abort the rest of monolith or
microservice teardown (Deployment, Service, PVC, ingress, monitoring,
database cleanup) before it ran.

**Per-service config inherits from the shared default, except the switch
itself and the gateway.** A per-service
`[scale.microservices.services.NAME.http_activation]` block merges over
`[scale.kubernetes.http_activation]` as a base, so a service that only
sets `target_port` still picks up a shared `concurrency_target` from the
top level, and a service with no local block at all still inherits a
top-level `enabled = true`. An explicit per-service `enabled = false`
still overrides that default, so a shared "on" switch can be selectively
opted out of per service. The gateway is the one service that never
inherits the shared switch (see the scaling-decision point above).

**`jac plan` surfaces HTTP-activated services.** The dry-run plan reads
the same `scaling` map, so an http_activation service renders its `HTTP
activation: 0 -> N` line and appears in the Totals, instead of being
invisible in the plan.

**A redeploy no longer resets a scaled-to-zero target back to its
baseline.** The Deployment manifest's `spec.replicas` is always the
configured static default, and every apply PATCHes it onto the live
Deployment - with no exclusion for HTTP-activation-mode services, that
undid KEDA's own scale-to-zero on every redeploy. `_apply_or_replace`
now takes an optional `patch_body` distinct from the manifest used for a
fresh create, and the deployment-apply loop strips `replicas` from the
PATCH body (not the CREATE body) for `HTTP_ACTIVATION`-mode services, so
KEDA's current value is left alone.

**No PodDisruptionBudget for an HTTP-activated service, and `jac plan`
stops flagging it.** `_build_pdb_manifest` used to compute a replica
floor from `hpa.min` regardless of mode, so an HTTP-activated service
could still get a PDB with `min_available` set - a budget that's
permanently unsatisfiable once the target is actually at zero replicas.
It now takes the resolved mode and skips PDB generation entirely for
`HTTP_ACTIVATION` services (warning if the user had `pdb` settings
configured). `PlanValidator._check_hpa`/`_check_pdb` had the same blind
spot for `jac plan`'s diagnostics and now skip both checks for
`HTTP_ACTIVATION` services too.

**Two services can't silently collide on the same HTTP-activation
traffic.** `build_http_activation_spec` now rejects an empty `rules`
list at build time - the docs already said an empty list means no
traffic ever matches, so a service left at the default was silently
deploying unreachable. Separately, `generate_manifests` now runs
`_validate_http_activation_rules`, which rejects two services that
resolve to identical `InterceptorRoute` match criteria (for example, two
services that both inherit the shared top-level rules unchanged) - the
KEDA HTTP Add-on interceptor has no way to tell which target a matching
request belongs to, so this is now a build-time error naming the
colliding services, not a runtime routing ambiguity.

**Mode transitions reap the old resource before applying the new one.**
`apply_manifests` used to apply the new mode's scaling resource (metric
autoscaler, or the HTTP `ScaledObject`) before calling
`reap_stale_scaling_resources_for_bundle`, which tears down the previous
mode's leftover resource. A service switching modes on a given redeploy
collided with its own stale resource - KEDA's admission webhook rejects
a second `ScaledObject` on an already-managed target - and only got
cleaned up on the next deploy. The reap call now runs before both
applies.

**A deploy bundle that's entirely HTTP-activation-scaled no longer
crashes.** `_wait_for_fleet_rollout` called `self._load_kube_config()`,
a method that doesn't exist anywhere in this codebase - the real method
is `_load_cluster_config()`. This crashed with an `AttributeError` in
exactly the case the surrounding code path exists to handle: a fleet
with no metric-scaled services to wait on.

**Traffic routing to the interceptor is not wired yet (documented
limitation).** jac-scale creates the `InterceptorRoute` and
`ScaledObject`, but does not yet rewire the gateway or Ingress backend
to the KEDA HTTP interceptor proxy. With `min_replicas = 0`, inbound
traffic that reaches the app Service directly (instead of the
interceptor) will not wake the pod. The reference doc carries a warning
to route through the interceptor yourself for now; the routing-plane
rewire is tracked separately as jaseci-labs#7959.

**Namespace-wide sweeps still lack application-identity scoping.** Every
resource-kind sweep in `destroy()` (Deployments, Services, PDBs,
Ingresses, ConfigMaps, and the KEDA resources above) uses a
namespace-wide `managed=jac-scale` label selector with no
application-identity component, so two jac-scale applications sharing
one namespace would delete each other's resources on teardown. This is a
pre-existing characteristic of the whole method, not something this PR
introduces; fixing it properly requires an app-identity label added at
manifest-build time across several files. Tracked separately as jaseci-labs#7710
rather than patched partially here.

- `test_http_activation_config.jac` (12 tests): config-to-spec
translation (disabled/absent, minimal config defaults, one full-config
test covering every optional field, and the four jac.toml-relative
validation errors - `target_port`/`target_port_name` mutual exclusivity,
a missing `concurrency_target`/`request_rate_target`, empty `rules`, and
a rule header missing `name`), the injectable apply/destroy wiring
helpers, and that only a 403 from `destroy_http_activation` is caught (a
500 still propagates). Uses a shared `_assert_config_error(cfg,
*substrings)` helper instead of hand-rolled raise/catch/assert blocks
per test.
- `test_http_activation_microservices.jac` (17 tests): the per-service
config pass-through and top-level inheritance (including `enabled`
itself, and an explicit per-service opt-out); the single scaling
decision in `_resolve_scaling` - that an http_activation service
resolves to `HTTP_ACTIVATION` mode carrying the `<name>-deployment`
scale target and emitting no metric autoscaler, that the gateway is
never HTTP-activated even under a shared default (exercised with a real
logger so a wrong log-method name is caught without a live deploy), and
that a plain service resolves to `METRIC`; the per-service apply loop
(only HTTP_ACTIVATION-mode services are applied, using the carried
scale-target and service names); the mode-transition reap tests (a
METRIC service has its stale HTTP activation resources reaped and its
autoscaler left alone, and vice versa, a NONE service has both reaped,
and an empty bundle is a no-op); and the cross-service rule-collision
validation (identical rules raise, distinct/empty/lone-peer rules don't,
and `generate_manifests` raises when two services inherit the shared
catch-all rule unchanged).
- `test_keda_http_activation.jac` (46 tests): imports the shared
fixtures instead of defining them locally, plus the
`destroy_http_activation`/`destroy_collection` error-handling coverage -
that `destroy_http_activation` still attempts the `ScaledObject` delete
after a 403 on the `InterceptorRoute` delete; that a 500 on one delete
is not masked by a 403 on the other; that `destroy_collection` tolerates
a 403 listing `InterceptorRoute`s without aborting the
`ScaledObject`/`TriggerAuthentication` sweep; and that a 403 listing
`ScaledObject`s still raises.
- `test_plan.jac` (23 tests): includes a case asserting an
http_activation service renders its scale-to-zero line and counts in
Totals, with no HPA line, and that the metric-autoscaler smoke test
reads the new `scaling` bundle key.
- `test_pdb_budget.jac` (11 tests): the existing PDB floor/emission
cases, plus that an HTTP-activated service gets no PDB even with
`pdb.min_available` set, and that the plan validator skips both the HPA
and PDB checks for an `HTTP_ACTIVATION` service.
- `test_show_yaml_autoscalers.jac` (5 tests): `jac plan --show-yaml`
renders the HPA/`ScaledObject` body the apply path would build, that the
scale target comes from the resolved plan rather than being re-derived
from the service name, that an http_activation-scaled service renders no
autoscaler manifest, and that a bundle with no scaling renders nothing.
- `http_activation_test_support.jac`: shared fixtures
(`default_http_activation_spec`, `default_http_activation_config`,
`mocked_keda_autoscaler`) extracted from `test_keda_http_activation.jac`
so both files use one source of truth instead of duplicating mock setup.

The HTTP activation, KEDA autoscaler, plan, memory-trigger-guard,
deployment-overlay, dry-run-purity, ms-config-isolation, and
superseded-deployment-reap suites all pass under `jac test`.

The microservices `destroy()` engine-gap wiring itself has no dedicated
test: like the rest of that method, it requires a live cluster
(`_load_cluster_config`, real client construction) with no existing
unit-test precedent in this codebase. The error-handling logic it
depends on (`destroy_collection`'s group-aware tolerance) is unit-tested
directly, as listed above. It was, however, exercised end to end on a
real cluster (see Notes to Reviewer).

| File | What Changed |
|---|---|
| `jac/jaclang/scale/deploy/autoscale/http_activation_config.jac` | New.
`build_http_activation_spec` translates a `[*.http_activation]` config
dict into an `HTTPActivationSpec`, with jac.toml-relative validation
errors, including an empty-`rules` check and a friendly error (instead
of a bare `KeyError`) for a rule header missing its required `name` key.
`apply_http_activation_for_target` /
`destroy_http_activation_for_target` wrap the `KEDAAutoscaler` calls
behind an injectable interface, shared by both deploy paths below.
`destroy_http_activation_for_target` is unconditional (no config gate,
so a resource created while enabled stays cleanable even if disabled
later) and swallows only a 403 from `destroy_http_activation` (via
null-safe `e?.status`, since the checker can't prove `ApiException`
isn't the `_optdeps` fallback `Exception`), re-raising anything else. |
| `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.impl.jac` |
`destroy_http_activation` now always attempts both the
`InterceptorRoute` and `ScaledObject` deletes regardless of the other's
outcome, and prefers surfacing a non-403 error over a 403 if both fail
differently. `destroy_collection` now tolerates 403 specifically for the
`http.keda.sh` group, leaving `keda.sh` groups at the existing strict
404/422 tolerance. |
| `jac/jaclang/scale/deploy/autoscale/keda_autoscaler.jac` |
`destroy_collection`'s interface gains a `best_effort: bool = False`
parameter for the group-aware 403 tolerance above. |
| `jac/jaclang/scale/deploy/autoscale/http_activation.jac` | Renames
`HTTPRequestRateMetric.window` to `rate_window` to match the config
schema and generated YAML field name. |
| `jac/jaclang/scale/config/plugin_config.jac` | New
`[scale.kubernetes.http_activation]` schema block (master switch,
replica bounds, target port, concurrency/request-rate metrics, routing
rules, cold start, timeouts, custom scale target). Documents the
per-service `[scale.microservices.services.NAME.http_activation]`
override, which falls back to the top-level block. The ~20-field nested
schema is now built once by `_http_activation_nested_schema()` and
shared by both call sites instead of being duplicated verbatim (the two
copies had already drifted: the `rules` description claimed "empty
matches all traffic," the opposite of the now-enforced behavior). |
| `jac/jaclang/scale/deploy/target/kubernetes/kubernetes_config.jac` |
New `http_activation: dict[str, any]` field on `KubernetesConfig`, wired
into `to_dict()` / `from_dict()`. |
| `jac/jaclang/scale/deploy/target/kubernetes/kubernetes_target.jac` |
Monolith teardown (`_destroy_application`) calls
`destroy_http_activation_for_target` next to the base-autoscaler
destroy, with the engine-gap sweep described above. Also removes
`KubernetesTargetBase.destroy`, `_destroy_component`,
`_destroy_application`, `_destroy_databases`/`_destroy_database`, and
`_wait_for_deletion` (about 360 lines): all were shadowed by
`KubernetesTarget`'s own overrides for every real deploy and unreachable
in practice. Note for future readers: `_destroy_component` was the
reference implementation jaseci-labs#8403 and jaseci-labs#7968 point at for full-teardown
component dispatch. After this merges, addressing either issue means
writing that dispatch logic fresh in `target.jac` rather than reusing
this block. |
| `jac/jaclang/scale/deploy/target/kubernetes/manifest_builder.jac` |
New `ScalingMode` enum and `_resolve_scaling(svc_name, deployment,
service)`: the single scaling decision, reading the scale-target name
off the built Deployment manifest, enforcing the gateway exclusion and
metric-vs-activation exclusivity. `_get_http_activation_config` merges
`[scale.kubernetes.http_activation]` as a base with the per-service
block. The build loop emits one `bundle["scaling"]` map (replacing the
separate `autoscalers`/`http_activations` keys). `generate_manifests`
now also calls `_validate_http_activation_rules` to reject cross-service
rule collisions, and `_build_pdb_manifest` takes the resolved mode and
skips PDB generation for `HTTP_ACTIVATION` services. |
| `jac/jaclang/scale/deploy/target/kubernetes/target.jac` |
`apply_http_activations_for_bundle`, the autoscaler apply loop, and the
post-deploy reachability-skip set all read `bundle["scaling"]` by mode,
using the carried scale-target/service names with no name re-derivation.
`destroy()` engine-gap fix (see Notable design points).
`apply_manifests` now reaps stale scaling resources before applying the
new mode's resource, `_apply_or_replace` takes an optional `patch_body`
so a redeploy's PATCH strips `replicas` for `HTTP_ACTIVATION`-mode
services instead of resetting KEDA's scale-to-zero, and
`_wait_for_fleet_rollout`'s crash on an all-activated fleet is fixed
(`_load_cluster_config`, not the nonexistent `_load_kube_config`). |
| `jac/jaclang/scale/runtime/cli/plan.jac` | Reads the `scaling` bundle
key; new `HTTPActivationView` rendered beside `HPAView`, and Totals
counts autoscalers and HTTP activations by mode.
`PlanValidator._check_hpa`/`_check_pdb` now read the resolved mode and
skip both checks for `HTTP_ACTIVATION` services. |
| `jac/jaclang/scale/tests/deploy/http_activation_test_support.jac` |
New. Shared fixtures (`default_http_activation_spec`,
`default_http_activation_config`, `mocked_keda_autoscaler`) extracted
from `test_keda_http_activation.jac`. |
| `jac/jaclang/scale/tests/deploy/test_http_activation_config.jac` |
Config-to-spec translation tests plus the injectable apply/destroy
wiring-helper tests (12 tests), using a shared `_assert_config_error`
helper. |
|
`jac/jaclang/scale/tests/deploy/test_http_activation_microservices.jac`
| New. Per-service config inheritance, the `_resolve_scaling` decision
(scale-target name, gateway exclusion, metric-vs-activation
exclusivity), the mode-transition reap tests, the apply-loop tests, and
the cross-service rule-collision validation (17 tests). |
| `jac/jaclang/scale/tests/deploy/test_keda_http_activation.jac` |
Imports shared fixtures instead of defining them locally, plus the
`destroy_http_activation`/`destroy_collection` error-handling tests (46
tests). |
| `jac/jaclang/scale/tests/microservices/test_plan.jac` | Reads the new
`scaling` bundle key; adds an http_activation render/Totals test (23
tests). |
| `jac/jaclang/scale/tests/deploy/test_pdb_budget.jac` | Adds coverage
that an `HTTP_ACTIVATION` service gets no PDB even with
`pdb.min_available` set, and that the plan validator skips both the HPA
and PDB checks for it (11 tests). |
| `jac/jaclang/scale/tests/deploy/test_show_yaml_autoscalers.jac` | Adds
coverage that `jac plan --show-yaml`'s scale target comes from the
resolved plan (not re-derived from the service name) and that an
http_activation-scaled service renders no autoscaler manifest (5 tests).
|
| `jac/jaclang/scale/runtime/cli/diagnostics.jac` |
`_check_hpa`/`_check_pdb` read the resolved scaling mode and skip both
checks for `HTTP_ACTIVATION` services. |
| `jac/jaclang/cli/docs/reference/plugins/jac-scale-kubernetes.md` |
Restores the "HTTP Add-on Activation" reference section with the new
config keys, both deploy-path examples, a note on the still-available
programmatic API, and a warning that inbound traffic must be routed
through the interceptor for now (jaseci-labs#7959). |
| `jac/jaclang/cli/docs/tutorials/production/kubernetes.md` | New "Scale
to zero on an HTTP request" subsection alongside the existing
autoscaling tutorial content, with a minimal `jac.toml` example and a
link to the full reference section. |
| `jac/jaclang/scale/tests/fixtures/keda_http_activation_e2e/` and
`jac/jaclang/scale/tests/deploy/keda_http_activation_real_e2e.sh` |
Real-cluster e2e fixture and driver, deployed through the `jac.toml`
wiring (replaces the old `fixture.yaml`-driven setup). The driver
resolves resource names from the unified deploy path's
`<app>-deployment` scale target. `keda_http_activation_verify.jac`, the
old standalone verify script, is removed - superseded by the
`jac.toml`-driven driver. |

The full flow was validated end to end on a local `kind` cluster (not
just unit tests):

- **Runtime channel.** The fixture `jac.toml` carried a `[dev]` stanza
(dev binary channel) with `[dev] jaclang_source` pointing at this
checkout, so `jac start --scale` ran this branch's deploy code host-side
(all manifest generation, the `kubectl apply` calls, and the
`InterceptorRoute`/`ScaledObject` reconcile happen in that process). The
confirming `jac dev mode - using compiler source at ...` banner appeared
on every deploy.
- **Cluster.** A single-node `kind` cluster with KEDA core and the KEDA
HTTP Add-on installed via helm. Leader election was disabled on the
control-plane components for the single-node dev cluster so the pod's
first-boot compile could not starve them.
- **Result.** `keda_http_activation_real_e2e.sh` PASSED: deploy,
idempotent redeploy (get-then-patch, no duplication), both
`InterceptorRoute` and `ScaledObject` reconciled to Ready, a request
through the interceptor cold-started the target from `0 -> 1` and
returned HTTP 200 with the walker's response, then the target scaled
back to `0` after the cooldown.
- **The three design fixes confirmed on the live cluster:** the
`ScaledObject`'s `scaleTargetRef` was `echo-deployment` (not the bare
name), the activated service had no metric HPA, and the gateway had a
normal HPA rather than a `ScaledObject`.

(cherry picked from commit 6ded655)
…a local stdlib copy during backport testing)

@MusabMahmoodh MusabMahmoodh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the six cherry-picks against their main originals rather than as one diff. The backport discipline is good: -x on every pick so each names its original, the exclusions are stated with reasons instead of being silent, and #7709's test restructuring is mirrored rather than half-applied. #8008 is mine, and test_show_yaml_autoscalers.jac came across intact, so the --show-yaml rendering keeps its coverage on this line. That was the thing I would have checked first.

Three things, and the first is the one I would hold the merge on.

1. Nothing tested this.

ci.yml triggers on pull_request: branches: [main] and push: branches: [main]. This PR targets release/jaclang-0.34.18, so none of it ran. The two checks on the PR are pre-commit.ci (green) and [code]smith (skipped). So 34 files and roughly 3,500 lines of scale changes are going onto the branch jacBuilder pins its deploys to, and the only thing that has looked at them is a formatter.

That is a different risk profile from the same code on main, because on main the next PR pays for a mistake and here a fleet does. The 2026-08-06 template regression is the shape I have in mind: config-level change, correct-looking diff, found in production.

What I would want before this merges is one of: a workflow_dispatch run of CI against this ref (the trigger is already there, it just needs pointing at the branch), or the scale suites run on a rig against a binary built from this head with the result pasted here. Either is cheap next to re-deriving it after a pinned deploy misbehaves.

2. The largest change in the diff is a 470-line deletion that the body does not mention.

8eb3b11b2 (the #7709 pick) takes KubernetesTargetBase from 1,318 to 848 lines, removing destroy, _destroy_application, _destroy_component, _destroy_database, _destroy_databases, _destroy_cache, _destroy_dashboard and _wait_for_deletion. kubernetes_target.jac still exists on main at full size, so this is not something #7709 did upstream.

I chased it and it is fine: KubernetesTarget.destroy in target.jac overrides the base and references none of the seven removed helpers, so the removed family was already unreachable on this branch and nothing calls into it. Good removal, in other words. But the body says every conflict was resolved by applying only the named PR's own delta and rejecting unrelated main drift, and this is unrelated main drift that was taken. On a release branch that is the sentence a release reviewer most needs, and right now they have to derive it the way I just did. One line in "what deliberately did NOT come along" saying the dead destroy family went too, and why it is dead, would close it.

3. A PDB behavior change ships here with its test left behind.

_build_pdb_manifest's HTTP_ACTIVATION early return came across, and test_pdb_budget.jac did not, because the replica-floor machinery it also covers is separate main work. That reasoning is right for the machinery and wrong for the early return: the branch now changes what a PDB looks like for an HTTP_ACTIVATION service with nothing asserting it, on a branch where point 1 means nothing asserts anything. If the budget half of that file is what does not port, the early-return half is small enough to bring over on its own.

@MusabMahmoodh

Copy link
Copy Markdown
Contributor

Filed the general case as #8947, since it is not yours to fix in a backport PR: no workflow in the repo triggers on a release/* branch or a PR targeting one, so this is the third backport in a row (with #8038 and #8701) to arrive with only pre-commit.ci having looked at it.

Point 1 of my review still stands for this PR specifically. A workflow_dispatch run of CI against this ref would settle it today without waiting on #8947.

…t test_pdb_budget left behind

The early-return in _build_pdb_manifest and the plan validator's
HTTP_ACTIVATION skip shipped in the jaseci-labs#7709 pick, but their tests lived in
test_pdb_budget.jac alongside main's replica-floor machinery that this
line does not carry. These two are the machinery-free half, rehomed in
test_http_activation_microservices.jac using its existing fixtures.
@kugesan1105

Copy link
Copy Markdown
Collaborator Author

@MusabMahmoodh all three addressed, in your order.

1. Test evidence, pasted, plus stronger system-level evidence than a CI dispatch would give.

A workflow_dispatch of upstream CI cannot point at this ref (the head lives on a fork, and dispatch refs must exist in this repo), so here is your second option, the suites run against this exact head (5de53909d) with the published jac 0.34.18 linux-x86_64 release binary:

  • jac/jaclang/scale/tests/deploy/: 223 passed, 4 skipped (the 4 are the k8s-gated e2e, no cluster on the rig).
  • jac/jaclang/scale/tests/microservices/: 273 passed, 45 failed, and all 45 are environment-limited, not backport regressions: 40 in test_gateway.jac and 5 in test_app_routes.jac, every one failing on scale/admin/ui/main.jac failed to compile (the admin-UI client build needs the client toolchain the rig lacks). Verified by running both files on the untouched branch point 529fb296e in the same environment: identical failures, identical counts (40 and 5). Same story as the pre-existing test_drain/test_fleet_pod_optout/test_rate_limit dozen already noted in the body.

Beyond the suites, this head has also been under a real fleet: jaseci/jaclang:experimental-8948 is this branch plus one test-only fleet-gate commit, and it was deployed twice to EKS via jacBuilder#1642 (runs 33841609863 and 34166918740, four days apart), full fleet rollout, KEDA ScaledObjects applied by the backported path, scale-to-zero wake measured both times (cold/warm 94/96s and 94/92s). The pinned-fleet risk you describe is exactly what those runs exercised.

2. The 470-line deletion is now in the body.

Added to "what deliberately did NOT come along", stating the whole legacy destroy family that left with the #7709 pick, that it is unrelated main drift which was taken, and why it is safe: KubernetesTarget.destroy in target.jac overrides the base and references none of the removed helpers, so the family was unreachable on this line before and after. You should not have had to derive that; thank you for chasing it.

3. The machinery-free half of the PDB coverage is ported.

5de53909d rehomes the two tests into test_http_activation_microservices.jac using its existing fixtures: an HTTP-activated service gets no PDB even with pdb.min_available set (through the real generate_manifests path) and the plan validator skips the HPA and PDB checks for an HTTP_ACTIVATION service. Suite is 32 passing with them. test_pdb_budget.jac itself still stays behind, since the rest of it tests the replica-floor machinery this line does not carry.

@MusabMahmoodh MusabMahmoodh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the new commit (5de5390). Rehoming the two machinery-free tests into test_http_activation_microservices.jac and reusing its existing fixtures is the right call over carrying test_pdb_budget.jac and the replica-floor machinery this line does not have. Both tests drive a real ManifestBuilder over a real temp project rather than stubbing the bundle, which is what I would have asked for.

Two things.

The PDB assertion passes for the wrong reason too. assert bundle.get("pdbs", {}).get("billing_ops") is None; holds when the PDB is correctly suppressed, and equally when pdbs is absent, when the service key never made it into the bundle, or when generate_manifests returned early for an unrelated reason. It cannot tell "suppressed because HTTP_ACTIVATION" from "PDB generation did not run".

A second service in the same services dict, same pdb.min_available = 1, without http_activation, fixes it: assert that one is in bundle["pdbs"] and billing_ops is not. Same builder call, one more dict entry, and then the test can only pass for the reason it claims.

The validator test asserts on message prose. "HPA" in m and "PodDisruptionBudget" in m or "pdb." in m go quiet the day someone rewords the diagnostic, and quiet in the direction of passing. Does the diagnostic object carry a code or a rule identifier alongside .service and .message? If it does, asserting on that is the version that survives a rewording.

Point 1 from my last review still stands, and is sharper now. The only check on this head is [code]smith, skipped. A test-only commit landed on a backport branch and nothing executed it. #8947 covers the general case, but for this PR a workflow_dispatch run of CI against the ref would settle it today rather than merging four test commits nothing has run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants